Help with cursor based paging

Solved
Shopify Partner
8 0 1

I'm trying to use the newer cursor based paging.  I use a url similar to:

store/admin/api/2019-07/products.json?limit=5

The headers returned with after the call to the api have:

x-shopify-shop-api-call-limit: 1/40
x-shopify-api-version: 2019-07
link: ; rel="next"

There is no 'page_info' parameter for me to use to get the next page of results.  And how do I use a 'rel' attribute in an API call?  Is this passed as a 'rel' header?

Be really helpful if someone could post a code segment of looping through some pages.  The docs don't make a lot of sense to me since the response headers in the docs don't match the response headers from the api.

I'd think it would be something like:

$link_info='';
do {
  $new_data = Http:get("shop/admin/api/2019-07/products.json?limit=5".($link_info?"&page_info=$link_info":'') );
  $link_header = getLinkHeader();
  $link_info= $link_header;  // should contain something like page_info=xyz&rel=next
  if( $new_data )
    processNewData($new_data);
} while(!empty($new_data));
0 Likes
Shopify Staff
Shopify Staff
1087 75 140

Hey @tbirnseth , 

 

It sounds like you're on the right track here, but maybe there's an issue with the way you're parsing the response/headers? 

 

You don't include the 'rel' part of the header in your next call, but the 'Link' header should return a value like : 

 

<https://shop-name.myshopify.com/admin/api/2019-07/products.json?limit=2&page_info=eyJsYXN0X2lkIjoxOTc3MjMxMDE1OTkyLCJsYXN0X3ZhbHVlIjoiQnVydG9uIEN1c3RvbSBGcmVlc3R5bGUgMTUxIiwiZGlyZWN0aW9uIjoibmV4dCJ9>; rel="next"

But you'd only want the part within <> for your next GET request. Is this being treated at HTML or something possibly, causing the part within <> to be stripped out? Are you able to replicate this using a cURL request?

0 Likes
Shopify Partner
8 0 1
I included the relevant headers in my post.  The 'link' header has only " ; rel=next".  This is bizarre if I can't use the link header as my request for the next page.
Your example shows what the docs say, but there is no page_info parameter.  And if there was, what do I do with the "rel" on an API call?  I.e. I'm not doing html, I'm doing PHP.  Does 'rel' become a header I'm to send?

A coding example that works would be nice.

tony
0 Likes
Shopify Partner
8 0 1

the headers I show come from a dump of the headers received.  You'll note that the 'rel' header has an empty space, followed by a semi-colon followed by rel="next".  This is what I get from my API call.

 

So how do I get it to respond with a proper 'rel' header so I can use the cursor-based paging?  The max limit of 250 will not support the client's product set.

 

Right now, this is unusable and removing the 'page=$page' method will break my addon and my customer will not be able to work with their products.  Do I need to submit a bug somewhere that the 'rel' header is incomplete?

1 Like
Highlighted
New Member
2 0 1

I have the same problem as I am making a request to Api

	function testAction($shop_url)
	{
		$endpoint = '/admin/api/2019-07/products.json';

		$s = curl_init();
		curl_setopt($s, CURLOPT_URL, 'https://'.$shop_url.$endpoint.'?limit=1');
		curl_setopt($s, CURLOPT_HTTPHEADER, array('Content-type: application/json'));
		curl_setopt($s, CURLOPT_USERPWD, $apiKey.':'.$password); 
		curl_setopt($s, CURLOPT_RETURNTRANSFER, 1);
		curl_setopt($s, CURLOPT_SSL_VERIFYPEER, 0);
		curl_setopt($s, CURLOPT_SSL_VERIFYHOST, 0);
		curl_setopt($s, CURLOPT_HEADERFUNCTION, array($this, 'headerLine'));

		$shopify_response = curl_exec($s);
	}

	function headerLine($curl, $header_line)
	{
		echo "<br>".$header_line;
    	        return strlen($header_line);
	}

And get this response header

HTTP/1.1 200 OK
Date: Tue, 17 Sep 2019 20:41:27 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d168f958740bb0d267f2a984f725e577b91568752887; expires=Wed, 16-Sep-20 20:41:27 GMT; path=/; domain=.myshopify.com; HttpOnly
X-Sorting-Hat-PodId: 52
X-Sorting-Hat-ShopId: 1341227061
Vary: Accept-Encoding
Referrer-Policy: origin-when-cross-origin
X-Frame-Options: DENY
X-ShopId: 1341227061
X-ShardId: 52
X-Stats-UserId:
X-Stats-ApiClientId: 2343500
X-Stats-ApiPermissionId: 28783247413
X-Shopify-API-Terms: By accessing or using the Shopify API you agree to the Shopify API License and Terms of Use at https://www.shopify.com/legal/api-terms
HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT: 1/40
X-Shopify-Shop-Api-Call-Limit: 1/40
X-Shopify-API-Version: 2019-07
Link: ; rel="next"
Strict-Transport-Security: max-age=7889238
X-Request-Id: 4625077c-19e6-4d7e-bbc0-d6816cdce4ef
X-Shopify-Stage: production
Content-Security-Policy: default-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://* shopify-pos://*; block-all-mixed-content; child-src 'self' https://* shopify-pos://*; connect-src 'self' wss://* https://*; frame-ancestors 'none'; img-src 'self' data: blob: https:; script-src https://cdn.shopify.com https://cdn.shopify.cn https://checkout.shopifycs.com https://js-agent.newrelic.com https://bam.nr-data.net https://dme0ih8comzn4.cloudfront.net https://api.stripe.com https://mpsnare.iesnare.com https://appcenter.intuit.com https://www.paypal.com https://js.braintreegateway.com https://c.paypal.com https://maps.googleapis.com https://www.google-analytics.com https://v.shopify.com https://widget.intercom.io https://js.intercomcdn.com 'self' 'unsafe-inline' 'unsafe-eval'; upgrade-insecure-requests; report-uri /csp-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=4625077c-19e6-4d7e-bbc0-d6816cdce4ef
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
X-XSS-Protection: 1; mode=block; report=/xss-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=4625077c-19e6-4d7e-bbc0-d6816cdce4ef
X-Dc: gcp-us-central1,gcp-us-central1
NEL: {"report_to":"network-errors","max_age":2592000,"failure_fraction":0.01,"success_fraction":0.0001}
Report-To: {"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 517de6aad82d8acc-KBP 

As you can see, Link parameter is empty.

Why is it empty if I set a limit of 1, having 4 products in the store?

1 Like
Shopify Staff
Shopify Staff
1087 75 140

Hey again guys, 

 

I'm not a PHP developer, so I wouldn't be able to provide code examples to show how to parse this. There was an example posted here though that should help get you guys going in the right direction. 

 

Something to note though is that the header will never be blank, the only time I have seen it 'blank' before was when it was being output into an HTML page and read from there.

 

If pagination isn't required (ex. setting limit=100 on a shop with 50 products) then the header won't be present at all, otherwise it'll have at least a rel="next" or rel="previous" link available to follow to get to another page. 

 

If the 'Link' header is blank using your apps, I'd really strongly suggest trying another method (like a cURL request) to narrow down if this is an issue with a specific shop or the way the headers are being parsed.

0 Likes
Shopify Partner
8 0 1

The header info I provided was from a log file, not from an browser page.

The 'link' header does not have the expected 'page_info' data so the API knows what set of data is being paged.

All the docs provide examples of an html anchor tag (href and rel attributes) which is pretty useless in real api operations.

 

It seems to be a bug that at least 2 of us who are trying to port to the next API version have encountered.

 

It would certainly make more sense to simply have a 'page_info' header that contained the key.  There's no need for any rel attribute since there are only 2 options (previous/next).  We certainly don't need the URL since we already have it. 

 

So please use any language you choose to demonstrate how to use with a link header of:

link: ; rel="next"

 Right now, I've not found any way to get to the next page with the header info returned with the first page.  Hence it's not usable.

0 Likes
Shopify Partner
8 0 1

So Shopify Support says that I can't report it as a bug and that I have to get help here in the forum.

 

But Shopify people have yet to explain why the API is returning headers that do NOT match what's needed (I.e. no page_info value).

 

So how is one supposed to proceed when the API does not return the data needed to page appropriatly.  Am I supposed to limit clients to 250 products?

 

Please, can someone provide a real answer?  The trouble is that the API is not returning information for us to parse!

0 Likes

Success.

Shopify Staff
Shopify Staff
1087 75 140

Hello again @tbirnseth , 

 

Have you attempted any other method of sending these calls to eliminate potential causes yet? If this happens with a cURL request too it would be good information to have since it would point to a shop issue which is yet to come up. If that were the case I could set up a private app on whatever shop you're using to see if I can replicate the same result.

 

For example, this cURL should work if you swap in your own private app API credentials and shop URL, provided that you're using a private app that is. 

 

curl -I -X GET https://{private_app_api_key}:{private_app_password}@{shop_url}.myshopify.com/admin/api/2019-10/orders.json?limit=1 -H 'Content-Type: application/json'

If your app uses OAuth and has an access token instead, you can send the cURL to your shop URL with your access token as a header : 

 

curl -I -X GET https://{shop-url}.myshopify.com/admin/api/2019-10/orders.json?limit=1 -H "Content-Type: application/json" -H "X-Shopify-Access-Token: {app_access_token}"

If the link header is still missing after trying one of the above requests and you could post the result of the cURL here, it would help a lot. 

0 Likes
Shopify Partner
8 0 1

Egg on my face.  I was using a browser to view my log files.  So the data is there but it's hidden because of your use of '<'s around the data.  I had to use the browser inspector to see the data.  Not sure who decided this syntax was a good idea.  Preference would be two headers that one can see and more easily parse since using link syntax is not relative to using an API.

 

My suggestion would be 2 headers:

X-Shopify-Page-Next: page_info_value (empty if no more pages)

X-Shopify-Page-Perv: page_info_value  (empty on first page or if there is no previous page).

 

Easy to parse and use.

But having this buried as an invalid xml tag, having them both in the same header and using 'rel=' syntax makes no sense at all from an API perspective.

0 Likes