Help with cursor based paging

[tbirnseth]

I'm trying to use the newer cursor based paging.  I use a url similar to:

store/admin/api/2019-07/products.json?limit=5

The headers returned with after the call to the api have:

x-shopify-shop-api-call-limit: 1/40
x-shopify-api-version: 2019-07
link: ; rel="next"

There is no 'page_info' parameter for me to use to get the next page of results.  And how do I use a 'rel' attribute in an API call?  Is this passed as a 'rel' header?

Be really helpful if someone could post a code segment of looping through some pages.  The docs don't make a lot of sense to me since the response headers in the docs don't match the response headers from the api.

I'd think it would be something like:

$link_info='';
do {
  $new_data = Http:get("shop/admin/api/2019-07/products.json?limit=5".($link_info?"&page_info=$link_info":'') );
  $link_header = getLinkHeader();
  $link_info= $link_header;  // should contain something like page_info=xyz&rel=next
  if( $new_data )
    processNewData($new_data);
} while(!empty($new_data));

[Josh]

Hey @tbirnseth , 

It sounds like you're on the right track here, but maybe there's an issue with the way you're parsing the response/headers? 

You don't include the 'rel' part of the header in your next call, but the 'Link' header should return a value like : 

<https://shop-name.myshopify.com/admin/api/2019-07/products.json?limit=2&page_info=eyJsYXN0X2lkIjoxOTc3MjMxMDE1OTkyLCJsYXN0X3ZhbHVlIjoiQnVydG9uIEN1c3RvbSBGcmVlc3R5bGUgMTUxIiwiZGlyZWN0aW9uIjoibmV4dCJ9>; rel="next"

But you'd only want the part within <> for your next GET request. Is this being treated at HTML or something possibly, causing the part within <> to be stripped out? Are you able to replicate this using a cURL request?

[tbirnseth]

I included the relevant headers in my post.  The 'link' header has only " ; rel=next".  This is bizarre if I can't use the link header as my request for the next page.
Your example shows what the docs say, but there is no page_info parameter.  And if there was, what do I do with the "rel" on an API call?  I.e. I'm not doing html, I'm doing PHP.  Does 'rel' become a header I'm to send?

A coding example that works would be nice.

tony

[tbirnseth]

the headers I show come from a dump of the headers received.  You'll note that the 'rel' header has an empty space, followed by a semi-colon followed by rel="next".  This is what I get from my API call.

So how do I get it to respond with a proper 'rel' header so I can use the cursor-based paging?  The max limit of 250 will not support the client's product set.

Right now, this is unusable and removing the 'page=$page' method will break my addon and my customer will not be able to work with their products.  Do I need to submit a bug somewhere that the 'rel' header is incomplete?

[rediskapsts]

I have the same problem as tbirnseth, I am making a request to Api

	function testAction($shop_url)
	{
		$endpoint = '/admin/api/2019-07/products.json';

		$s = curl_init();
		curl_setopt($s, CURLOPT_URL, 'https://'.$shop_url.$endpoint.'?limit=1');
		curl_setopt($s, CURLOPT_HTTPHEADER, array('Content-type: application/json'));
		curl_setopt($s, CURLOPT_USERPWD, $apiKey.':'.$password); 
		curl_setopt($s, CURLOPT_RETURNTRANSFER, 1);
		curl_setopt($s, CURLOPT_SSL_VERIFYPEER, 0);
		curl_setopt($s, CURLOPT_SSL_VERIFYHOST, 0);
		curl_setopt($s, CURLOPT_HEADERFUNCTION, array($this, 'headerLine'));

		$shopify_response = curl_exec($s);
	}

	function headerLine($curl, $header_line)
	{
		echo "<br>".$header_line;
    	        return strlen($header_line);
	}

And get this response header

HTTP/1.1 200 OK
Date: Tue, 17 Sep 2019 20:41:27 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d168f958740bb0d267f2a984f725e577b91568752887; expires=Wed, 16-Sep-20 20:41:27 GMT; path=/; domain=.myshopify.com; HttpOnly
X-Sorting-Hat-PodId: 52
X-Sorting-Hat-ShopId: 1341227061
Vary: Accept-Encoding
Referrer-Policy: origin-when-cross-origin
X-Frame-Options: DENY
X-ShopId: 1341227061
X-ShardId: 52
X-Stats-UserId:
X-Stats-ApiClientId: 2343500
X-Stats-ApiPermissionId: 28783247413
X-Shopify-API-Terms: By accessing or using the Shopify API you agree to the Shopify API License and Terms of Use at https://www.shopify.com/legal/api-terms
HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT: 1/40
X-Shopify-Shop-Api-Call-Limit: 1/40
X-Shopify-API-Version: 2019-07
Link: ; rel="next"
Strict-Transport-Security: max-age=7889238
X-Request-Id: 4625077c-19e6-4d7e-bbc0-d6816cdce4ef
X-Shopify-Stage: production
Content-Security-Policy: default-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://* shopify-pos://*; block-all-mixed-content; child-src 'self' https://* shopify-pos://*; connect-src 'self' wss://* https://*; frame-ancestors 'none'; img-src 'self' data: blob: https:; script-src https://cdn.shopify.com https://cdn.shopify.cn https://checkout.shopifycs.com https://js-agent.newrelic.com https://bam.nr-data.net https://dme0ih8comzn4.cloudfront.net https://api.stripe.com https://mpsnare.iesnare.com https://appcenter.intuit.com https://www.paypal.com https://js.braintreegateway.com https://c.paypal.com https://maps.googleapis.com https://www.google-analytics.com https://v.shopify.com https://widget.intercom.io https://js.intercomcdn.com 'self' 'unsafe-inline' 'unsafe-eval'; upgrade-insecure-requests; report-uri /csp-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=4625077c-19e6-4d7e-bbc0-d6816cdce4ef
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
X-XSS-Protection: 1; mode=block; report=/xss-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=4625077c-19e6-4d7e-bbc0-d6816cdce4ef
X-Dc: gcp-us-central1,gcp-us-central1
NEL: {"report_to":"network-errors","max_age":2592000,"failure_fraction":0.01,"success_fraction":0.0001}
Report-To: {"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 517de6aad82d8acc-KBP 

As you can see, Link parameter is empty.

Why is it empty if I set a limit of 1, having 4 products in the store?

[Josh]

Hey again guys, 

I'm not a PHP developer, so I wouldn't be able to provide code examples to show how to parse this. There was an example posted here though that should help get you guys going in the right direction. 

Something to note though is that the header will never be blank, the only time I have seen it 'blank' before was when it was being output into an HTML page and read from there.

If pagination isn't required (ex. setting limit=100 on a shop with 50 products) then the header won't be present at all, otherwise it'll have at least a rel="next" or rel="previous" link available to follow to get to another page. 

If the 'Link' header is blank using your apps, I'd really strongly suggest trying another method (like a cURL request) to narrow down if this is an issue with a specific shop or the way the headers are being parsed.

[tbirnseth]

The header info I provided was from a log file, not from an browser page.

The 'link' header does not have the expected 'page_info' data so the API knows what set of data is being paged.

All the docs provide examples of an html anchor tag (href and rel attributes) which is pretty useless in real api operations.

It seems to be a bug that at least 2 of us who are trying to port to the next API version have encountered.

It would certainly make more sense to simply have a 'page_info' header that contained the key.  There's no need for any rel attribute since there are only 2 options (previous/next).  We certainly don't need the URL since we already have it. 

So please use any language you choose to demonstrate how to use with a link header of:

link: ; rel="next"

 Right now, I've not found any way to get to the next page with the header info returned with the first page.  Hence it's not usable.

[tbirnseth]

So Shopify Support says that I can't report it as a bug and that I have to get help here in the forum.

But Shopify people have yet to explain why the API is returning headers that do NOT match what's needed (I.e. no page_info value).

So how is one supposed to proceed when the API does not return the data needed to page appropriatly.  Am I supposed to limit clients to 250 products?

Please, can someone provide a real answer?  The trouble is that the API is not returning information for us to parse!

[Josh · Accepted solution]

Hello again @tbirnseth , 

Have you attempted any other method of sending these calls to eliminate potential causes yet? If this happens with a cURL request too it would be good information to have since it would point to a shop issue which is yet to come up. If that were the case I could set up a private app on whatever shop you're using to see if I can replicate the same result.

For example, this cURL should work if you swap in your own private app API credentials and shop URL, provided that you're using a private app that is. 

curl -I -X GET https://{private_app_api_key}:{private_app_password}@{shop_url}.myshopify.com/admin/api/2019-10/orders.json?limit=1 -H 'Content-Type: application/json'

If your app uses OAuth and has an access token instead, you can send the cURL to your shop URL with your access token as a header : 

curl -I -X GET https://{shop-url}.myshopify.com/admin/api/2019-10/orders.json?limit=1 -H "Content-Type: application/json" -H "X-Shopify-Access-Token: {app_access_token}"

If the link header is still missing after trying one of the above requests and you could post the result of the cURL here, it would help a lot. 

[tbirnseth]

Egg on my face.  I was using a browser to view my log files.  So the data is there but it's hidden because of your use of '<'s around the data.  I had to use the browser inspector to see the data.  Not sure who decided this syntax was a good idea.  Preference would be two headers that one can see and more easily parse since using link syntax is not relative to using an API.

My suggestion would be 2 headers:

X-Shopify-Page-Next: page_info_value (empty if no more pages)

X-Shopify-Page-Perv: page_info_value  (empty on first page or if there is no previous page).

Easy to parse and use.

But having this buried as an invalid xml tag, having them both in the same header and using 'rel=' syntax makes no sense at all from an API perspective.

[DebashisD]

I am not a fan of the way the cursor-based pagination has been implemented but we have to live with this.

The documentation clearly missing two vital things to confuse developers.

1. If there are fewer results than the limit the Link header will not be present.

2. The Link header can't be shown in the browser. If this chain was not there I would never get to know this to inspect the header.

Thanks @tbirnseth 

[Saso-WD]

Holly cow, spent 2 hours trying to figure this one out, had the same issue O.o

[aatmik]

I totally agree with you and feel like throwing the same egg on the person who decided the stupid syntax of using <>'s for the cursor-pagination links.

How immature to complicate things.

First the user will have to parse the link header itself and use explode to get value of the rel parameter and then use the link finally.

[DanteCullari]

I was confused by this for a bit too. When testing with curl, I was outputting the header info into the browser and Link: appeared empty, but when I viewed Inspect Element > Elements, I found the links were actually there in the html. Just need to parse the headers and grab them. This thread helped but accepted answer didn't mention this directly so I wanted to emphasize.

I personally like how Facebook does it with concatenating a json pagination object to the end of the api response. Good step in the right direction though, it does work!

[Manaspaul623]

Why it is marked as Answered.  I have the same issue.

[DanteCullari]

Please check your html. The links are there, they may not be rendering properly in the browser window when testing but they are there. Let us know what you've tried and what outcome you're currently getting if still having an issue!

[Manaspaul623]

OK. I found that. but how to get the link with php. Is there a simple way to get the link to a variable?

[DanteCullari]

You have to isolate the request headers into a variable. If using Curl, something like CURLOPT_HEADER=>1 in the request. Look up some code to isolate curl header into a variable and  then return it from a function. 

Something like:

// Shopify Get Products API Call

$response = $shopify->getShopifyRemoteUrlWithHeaders( $shopify_api_endpoint );

// Get Response Headers

$headers = $shopify->getHeadersFromCurlResponse($response);

Then just parse, something like:

$link_ex = explode( '<', $headers['Link'] );
$link_previous_ex = explode( '>;', $link_ex[1] );
$link_next_ex = explode( '>;', $link_ex[2] );
$link_previous = $link_previous_ex[0];
$link_next = $link_next_ex[0];

[WOTIO]

The parsing method Dante provided worked for me. The links are there in the response, but they will not show up in the browser rendered view of the page if you are printing out the response.

[tbirnseth]

I'm sure there's a more elegant solution. However this is what I use. 
The function parameter should be an empty string on the first call and
the desired direction on subsequent calls (I.e. 'next', 'prev').  It
will return the query string for the URL you're using.  If you want the
full URL then make appropriate changes by returning $url instead.

    private static function get_qs($kind='next') {
        if( empty($kind) )    {    // first pass
            return sprintf("limit=%d", self::$items_per_page);
        }

        $headers = Http::getHeaders();
        $headers_ar = explode("\r\n", $headers);    // explode each
line
        foreach($headers_ar as $_v) {    // loop through the lines
            if( strpos($_v, 'link:') !== false) {    // found the
'link' line
                // Each link header has one or more link values in <>'s
                $x = explode('<', $_v);
                // $x[0] contains the 'link:' header tag
                array_shift($x);    // throw it away
                foreach($x as $rel) {
                    if( strpos($rel, "rel=\"$kind\"")) {
                        // Found the kind ww're looking for
                        $z = explode('>', $rel);    // Now just get
the URL portion into $z[0]
                        $url = $z[0];
                        $q = explode('?', $url);
                        return empty($q[1]) ? '' : $q[1];
                    }
                }
            }
        }
        return '';
    }

[maltillo]

@tbirnseth @DebashisD @DanteCullari @Josh @Manaspaul623 

Hi guys.

I am not a developer but trying to troubleshoot with my developer about why since implementing the new cursor-based pagination in our app, users can't search partially for products when filtering.  Previously they could, and now I am considering reverting to the old api until this is figured out.

Is the solution shown here related in any way to our issue?

[DanteCullari]

Hey I'm not 100% sure what you mean by "can't search partially for products when filtering", but maybe Shopify's predictive search api will help:

https://shopify.dev/tutorials/add-predictive-search-to-your-shopify-theme#requesting-predictive-sear...

I think using the products endpoint for search would be very inefficient if you have a lot of products, although it's doable. There are a lot of solutions I'm sure your dev should be able to come up.

Couple solutions off top of my head - store all the products in your own database and maintain updates with daily syncs from Shopify - perform your existing call with all products as response. Or you could technically call the products endpoint, loop through all of the pages, store search matches in memory and return, but this again would be pretty inefficient. Or again you could sever filtering from search and use the predictive search api for the search portion. Again not really sure what you're trying to accomplish but I've actually gone with my own database that syncs daily and it has been very useful for a lot of other things.

[maltillo]

Our app is backend. We allow users to apply bulk price updates to their products. They first need to search and filter through to find the products they want to update. Before they could search 'Blue' and the search would bring up all products with 'blue' in the title. Now that doesn't work.

We are using REST API.

[Manaspaul623]

Shopify product searching by title is supported with exactly same product title name. Partially title searching is not supported by shopify in recent Rest Admin api version. But it is available in GraphQL api. You can use it.

---

@maltillo wrote:

Our app is backend. We allow users to apply bulk price updates to their products. They first need to search and filter through to find the products they want to update. Before they could search 'Blue' and the search would bring up all products with 'blue' in the title. Now that doesn't work.

We are using REST API.

---

[maltillo]

Thanks for your reply.

Our app is written in REST API

Does this mean re-writing the entire app in GraphQL api?

[Manaspaul623]

No, not at all. 

You can only do the searching operation with GraphQL whereas other operation in Rest API

---

@maltillo wrote:

Thanks for your reply.

Our app is written in REST API

Does this mean re-writing the entire app in GraphQL api?

---

.

[MarceloDias]

I was thinking that i had the same problem,

but the truth was, how the link is between "<>" and i tried to see it in a Browser

the Browser understand that everything is between the "<>" as a HTML tag.

But in Google Chrome is possible to use CTRL + U ( or right click -> show source code ) and there will be the link.