FROM CACHE - en_header

Help with cursor based paging

Solved
tbirnseth
Shopify Partner
9 0 12

I'm trying to use the newer cursor based paging.  I use a url similar to:

store/admin/api/2019-07/products.json?limit=5

The headers returned with after the call to the api have:

x-shopify-shop-api-call-limit: 1/40
x-shopify-api-version: 2019-07
link: ; rel="next"

There is no 'page_info' parameter for me to use to get the next page of results.  And how do I use a 'rel' attribute in an API call?  Is this passed as a 'rel' header?

Be really helpful if someone could post a code segment of looping through some pages.  The docs don't make a lot of sense to me since the response headers in the docs don't match the response headers from the api.

I'd think it would be something like:

$link_info='';
do {
  $new_data = Http:get("shop/admin/api/2019-07/products.json?limit=5".($link_info?"&page_info=$link_info":'') );
  $link_header = getLinkHeader();
  $link_info= $link_header;  // should contain something like page_info=xyz&rel=next
  if( $new_data )
    processNewData($new_data);
} while(!empty($new_data));
Accepted Solution (1)

Accepted Solutions
Josh
Shopify Staff
Shopify Staff
1134 84 226

This is an accepted solution.

Hello again @tbirnseth , 

 

Have you attempted any other method of sending these calls to eliminate potential causes yet? If this happens with a cURL request too it would be good information to have since it would point to a shop issue which is yet to come up. If that were the case I could set up a private app on whatever shop you're using to see if I can replicate the same result.

 

For example, this cURL should work if you swap in your own private app API credentials and shop URL, provided that you're using a private app that is. 

 

curl -I -X GET https://{private_app_api_key}:{private_app_password}@{shop_url}.myshopify.com/admin/api/2019-10/orders.json?limit=1 -H 'Content-Type: application/json'

If your app uses OAuth and has an access token instead, you can send the cURL to your shop URL with your access token as a header : 

 

curl -I -X GET https://{shop-url}.myshopify.com/admin/api/2019-10/orders.json?limit=1 -H "Content-Type: application/json" -H "X-Shopify-Access-Token: {app_access_token}"

If the link header is still missing after trying one of the above requests and you could post the result of the cURL here, it would help a lot. 

Josh | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

View solution in original post

Replies 26 (26)
Josh
Shopify Staff
Shopify Staff
1134 84 226

Hey @tbirnseth , 

 

It sounds like you're on the right track here, but maybe there's an issue with the way you're parsing the response/headers? 

 

You don't include the 'rel' part of the header in your next call, but the 'Link' header should return a value like : 

 

<https://shop-name.myshopify.com/admin/api/2019-07/products.json?limit=2&page_info=eyJsYXN0X2lkIjoxOTc3MjMxMDE1OTkyLCJsYXN0X3ZhbHVlIjoiQnVydG9uIEN1c3RvbSBGcmVlc3R5bGUgMTUxIiwiZGlyZWN0aW9uIjoibmV4dCJ9>; rel="next"

But you'd only want the part within <> for your next GET request. Is this being treated at HTML or something possibly, causing the part within <> to be stripped out? Are you able to replicate this using a cURL request?

Josh | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

tbirnseth
Shopify Partner
9 0 12
I included the relevant headers in my post.  The 'link' header has only " ; rel=next".  This is bizarre if I can't use the link header as my request for the next page.
Your example shows what the docs say, but there is no page_info parameter.  And if there was, what do I do with the "rel" on an API call?  I.e. I'm not doing html, I'm doing PHP.  Does 'rel' become a header I'm to send?

A coding example that works would be nice.

tony
tbirnseth
Shopify Partner
9 0 12

the headers I show come from a dump of the headers received.  You'll note that the 'rel' header has an empty space, followed by a semi-colon followed by rel="next".  This is what I get from my API call.

 

So how do I get it to respond with a proper 'rel' header so I can use the cursor-based paging?  The max limit of 250 will not support the client's product set.

 

Right now, this is unusable and removing the 'page=$page' method will break my addon and my customer will not be able to work with their products.  Do I need to submit a bug somewhere that the 'rel' header is incomplete?

rediskapsts
New Member
2 0 3

I have the same problem as I am making a request to Api

	function testAction($shop_url)
	{
		$endpoint = '/admin/api/2019-07/products.json';

		$s = curl_init();
		curl_setopt($s, CURLOPT_URL, 'https://'.$shop_url.$endpoint.'?limit=1');
		curl_setopt($s, CURLOPT_HTTPHEADER, array('Content-type: application/json'));
		curl_setopt($s, CURLOPT_USERPWD, $apiKey.':'.$password); 
		curl_setopt($s, CURLOPT_RETURNTRANSFER, 1);
		curl_setopt($s, CURLOPT_SSL_VERIFYPEER, 0);
		curl_setopt($s, CURLOPT_SSL_VERIFYHOST, 0);
		curl_setopt($s, CURLOPT_HEADERFUNCTION, array($this, 'headerLine'));

		$shopify_response = curl_exec($s);
	}

	function headerLine($curl, $header_line)
	{
		echo "<br>".$header_line;
    	        return strlen($header_line);
	}

And get this response header

HTTP/1.1 200 OK
Date: Tue, 17 Sep 2019 20:41:27 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d168f958740bb0d267f2a984f725e577b91568752887; expires=Wed, 16-Sep-20 20:41:27 GMT; path=/; domain=.myshopify.com; HttpOnly
X-Sorting-Hat-PodId: 52
X-Sorting-Hat-ShopId: 1341227061
Vary: Accept-Encoding
Referrer-Policy: origin-when-cross-origin
X-Frame-Options: DENY
X-ShopId: 1341227061
X-ShardId: 52
X-Stats-UserId:
X-Stats-ApiClientId: 2343500
X-Stats-ApiPermissionId: 28783247413
X-Shopify-API-Terms: By accessing or using the Shopify API you agree to the Shopify API License and Terms of Use at https://www.shopify.com/legal/api-terms
HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT: 1/40
X-Shopify-Shop-Api-Call-Limit: 1/40
X-Shopify-API-Version: 2019-07
Link: ; rel="next"
Strict-Transport-Security: max-age=7889238
X-Request-Id: 4625077c-19e6-4d7e-bbc0-d6816cdce4ef
X-Shopify-Stage: production
Content-Security-Policy: default-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://* shopify-pos://*; block-all-mixed-content; child-src 'self' https://* shopify-pos://*; connect-src 'self' wss://* https://*; frame-ancestors 'none'; img-src 'self' data: blob: https:; script-src https://cdn.shopify.com https://cdn.shopify.cn https://checkout.shopifycs.com https://js-agent.newrelic.com https://bam.nr-data.net https://dme0ih8comzn4.cloudfront.net https://api.stripe.com https://mpsnare.iesnare.com https://appcenter.intuit.com https://www.paypal.com https://js.braintreegateway.com https://c.paypal.com https://maps.googleapis.com https://www.google-analytics.com https://v.shopify.com https://widget.intercom.io https://js.intercomcdn.com 'self' 'unsafe-inline' 'unsafe-eval'; upgrade-insecure-requests; report-uri /csp-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=4625077c-19e6-4d7e-bbc0-d6816cdce4ef
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
X-XSS-Protection: 1; mode=block; report=/xss-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=4625077c-19e6-4d7e-bbc0-d6816cdce4ef
X-Dc: gcp-us-central1,gcp-us-central1
NEL: {"report_to":"network-errors","max_age":2592000,"failure_fraction":0.01,"success_fraction":0.0001}
Report-To: {"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 517de6aad82d8acc-KBP 

As you can see, Link parameter is empty.

Why is it empty if I set a limit of 1, having 4 products in the store?

Josh
Shopify Staff
Shopify Staff
1134 84 226

Hey again guys, 

 

I'm not a PHP developer, so I wouldn't be able to provide code examples to show how to parse this. There was an example posted here though that should help get you guys going in the right direction. 

 

Something to note though is that the header will never be blank, the only time I have seen it 'blank' before was when it was being output into an HTML page and read from there.

 

If pagination isn't required (ex. setting limit=100 on a shop with 50 products) then the header won't be present at all, otherwise it'll have at least a rel="next" or rel="previous" link available to follow to get to another page. 

 

If the 'Link' header is blank using your apps, I'd really strongly suggest trying another method (like a cURL request) to narrow down if this is an issue with a specific shop or the way the headers are being parsed.

Josh | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

tbirnseth
Shopify Partner
9 0 12

The header info I provided was from a log file, not from an browser page.

The 'link' header does not have the expected 'page_info' data so the API knows what set of data is being paged.

All the docs provide examples of an html anchor tag (href and rel attributes) which is pretty useless in real api operations.

 

It seems to be a bug that at least 2 of us who are trying to port to the next API version have encountered.

 

It would certainly make more sense to simply have a 'page_info' header that contained the key.  There's no need for any rel attribute since there are only 2 options (previous/next).  We certainly don't need the URL since we already have it. 

 

So please use any language you choose to demonstrate how to use with a link header of:

link: ; rel="next"

 Right now, I've not found any way to get to the next page with the header info returned with the first page.  Hence it's not usable.

tbirnseth
Shopify Partner
9 0 12

So Shopify Support says that I can't report it as a bug and that I have to get help here in the forum.

 

But Shopify people have yet to explain why the API is returning headers that do NOT match what's needed (I.e. no page_info value).

 

So how is one supposed to proceed when the API does not return the data needed to page appropriatly.  Am I supposed to limit clients to 250 products?

 

Please, can someone provide a real answer?  The trouble is that the API is not returning information for us to parse!

Josh
Shopify Staff
Shopify Staff
1134 84 226

This is an accepted solution.

Hello again @tbirnseth , 

 

Have you attempted any other method of sending these calls to eliminate potential causes yet? If this happens with a cURL request too it would be good information to have since it would point to a shop issue which is yet to come up. If that were the case I could set up a private app on whatever shop you're using to see if I can replicate the same result.

 

For example, this cURL should work if you swap in your own private app API credentials and shop URL, provided that you're using a private app that is. 

 

curl -I -X GET https://{private_app_api_key}:{private_app_password}@{shop_url}.myshopify.com/admin/api/2019-10/orders.json?limit=1 -H 'Content-Type: application/json'

If your app uses OAuth and has an access token instead, you can send the cURL to your shop URL with your access token as a header : 

 

curl -I -X GET https://{shop-url}.myshopify.com/admin/api/2019-10/orders.json?limit=1 -H "Content-Type: application/json" -H "X-Shopify-Access-Token: {app_access_token}"

If the link header is still missing after trying one of the above requests and you could post the result of the cURL here, it would help a lot. 

Josh | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

tbirnseth
Shopify Partner
9 0 12

Egg on my face.  I was using a browser to view my log files.  So the data is there but it's hidden because of your use of '<'s around the data.  I had to use the browser inspector to see the data.  Not sure who decided this syntax was a good idea.  Preference would be two headers that one can see and more easily parse since using link syntax is not relative to using an API.

 

My suggestion would be 2 headers:

X-Shopify-Page-Next: page_info_value (empty if no more pages)

X-Shopify-Page-Perv: page_info_value  (empty on first page or if there is no previous page).

 

Easy to parse and use.

But having this buried as an invalid xml tag, having them both in the same header and using 'rel=' syntax makes no sense at all from an API perspective.

DanteCullari
Tourist
4 0 3

I was confused by this for a bit too. When testing with curl, I was outputting the header info into the browser and Link: appeared empty, but when I viewed Inspect Element > Elements, I found the links were actually there in the html. Just need to parse the headers and grab them. This thread helped but accepted answer didn't mention this directly so I wanted to emphasize.

 

I personally like how Facebook does it with concatenating a json pagination object to the end of the api response. Good step in the right direction though, it does work!

DebashisD
Shopify Partner
6 0 2

I am not a fan of the way the cursor-based pagination has been implemented but we have to live with this.

 

The documentation clearly missing two vital things to confuse developers.

1. If there are fewer results than the limit the Link header will not be present.

2. The Link header can't be shown in the browser. If this chain was not there I would never get to know this to inspect the header.

 

Thanks @tbirnseth 

 

Manaspaul623
Tourist
5 0 3

Why it is marked as Answered.  I have the same issue.

DanteCullari
Tourist
4 0 3

Please check your html. The links are there, they may not be rendering properly in the browser window when testing but they are there. Let us know what you've tried and what outcome you're currently getting if still having an issue!

Manaspaul623
Tourist
5 0 3

OK. I found that. but how to get the link with php. Is there a simple way to get the link to a variable?

DanteCullari
Tourist
4 0 3

You have to isolate the request headers into a variable. If using Curl, something like CURLOPT_HEADER=>1 in the request. Look up some code to isolate curl header into a variable and  then return it from a function. 

 

Something like:

 

// Shopify Get Products API Call
$response = $shopify->getShopifyRemoteUrlWithHeaders( $shopify_api_endpoint );
 
// Get Response Headers
$headers = $shopify->getHeadersFromCurlResponse($response);
 
Then just parse, something like:
 

$link_ex = explode( '<', $headers['Link'] );
$link_previous_ex = explode( '>;', $link_ex[1] );
$link_next_ex = explode( '>;', $link_ex[2] );
$link_previous = $link_previous_ex[0];
$link_next = $link_next_ex[0];

tbirnseth
Shopify Partner
9 0 12
I'm sure there's a more elegant solution. However this is what I use. 
The function parameter should be an empty string on the first call and
the desired direction on subsequent calls (I.e. 'next', 'prev').  It
will return the query string for the URL you're using.  If you want the
full URL then make appropriate changes by returning $url instead.

    private static function get_qs($kind='next') {
        if( empty($kind) )    {    // first pass
            return sprintf("limit=%d", self::$items_per_page);
        }

        $headers = Http::getHeaders();
        $headers_ar = explode("\r\n", $headers);    // explode each
line
        foreach($headers_ar as $_v) {    // loop through the lines
            if( strpos($_v, 'link:') !== false) {    // found the
'link' line
                // Each link header has one or more link values in <>'s
                $x = explode('<', $_v);
                // $x[0] contains the 'link:' header tag
                array_shift($x);    // throw it away
                foreach($x as $rel) {
                    if( strpos($rel, "rel=\"$kind\"")) {
                        // Found the kind ww're looking for
                        $z = explode('>', $rel);    // Now just get
the URL portion into $z[0]
                        $url = $z[0];
                        $q = explode('?', $url);
                        return empty($q[1]) ? '' : $q[1];
                    }
                }
            }
        }
        return '';
    }

maltillo
Shopify Partner
13 0 0

@tbirnseth @DebashisD @DanteCullari @Josh @Manaspaul623 

 

Hi guys.

 

I am not a developer but trying to troubleshoot with my developer about why since implementing the new cursor-based pagination in our app, users can't search partially for products when filtering.  Previously they could, and now I am considering reverting to the old api until this is figured out.

 

Is the solution shown here related in any way to our issue?

DanteCullari
Tourist
4 0 3

Hey I'm not 100% sure what you mean by "can't search partially for products when filtering", but maybe Shopify's predictive search api will help:

 

https://shopify.dev/tutorials/add-predictive-search-to-your-shopify-theme#requesting-predictive-sear...

 

I think using the products endpoint for search would be very inefficient if you have a lot of products, although it's doable. There are a lot of solutions I'm sure your dev should be able to come up.

 

Couple solutions off top of my head - store all the products in your own database and maintain updates with daily syncs from Shopify - perform your existing call with all products as response. Or you could technically call the products endpoint, loop through all of the pages, store search matches in memory and return, but this again would be pretty inefficient. Or again you could sever filtering from search and use the predictive search api for the search portion. Again not really sure what you're trying to accomplish but I've actually gone with my own database that syncs daily and it has been very useful for a lot of other things.

maltillo
Shopify Partner
13 0 0

Our app is backend. We allow users to apply bulk price updates to their products. They first need to search and filter through to find the products they want to update. Before they could search 'Blue' and the search would bring up all products with 'blue' in the title. Now that doesn't work.

 

We are using REST API.