cursor-based pagination

Solved
Highlighted
Shopify Partner
15 0 2

Got the email as others have about page no longer being supported via a rest API. I suspect prior to Shopify adding cusor-based pagination most people would get the count of records divide that count into page blocks and pass the page as parameter.

 

I've looked at the documentation but it's not clear to me given the above what is the execution flow. For example I have 350 products. In the past I would get the count, then determine the number of pages using the 250 limit. So I would just loop through the number of pages (2 in my example) calling get products specifying the page and limit

 

I'm not understanding why Shopify would do it this way. It seems now I have to parse the header to be able to extract a URL to determine where I am and if I'm done or not. Am I missing something?

 

1 Like
Highlighted
Shopify Staff
Shopify Staff
1129 82 179

This is an accepted solution.

Hey @JohnCole , 

 

It sounds like you have the right idea here - you submit your first GET request with your limit parameter (up to 250) and then use the value of the 'Link' header from the response to follow the rel=next link to the next page until there is no rel=next value left. Once that happens you know you've paged through all of a resource. 

 

We're making the change from the page parameter because page is known to have some issues such as the potential to miss data, and the database queries it generates create a lot of extra load while also being extremely slow and error-prone. With cursor-based pagination, many of the issues with page are resolved and it's also far, far more efficient, relieving load on Shopify and enabling app developers to have faster apps at the same time. 

 

I can understand changing from current behaviour isn't ideal when you've gotten used to how something works, but in my experience so far cursor-based pagination does create a fairly substantial speed boost.

0 Likes
Highlighted
Shopify Partner
15 0 2

Just confirming it works as proposed. In my case performance on my end was never an issue, but I get that Shopify is always looking for ways to optimize just not a fan of how it got implemented. Would have preferred the Link info just be included in the response body or if it has to be in the header that it was JSON payload where I didn't have to parse a string. I know I'm being nit picky as the parsing isn't difficult just sharing my thoughts.

 

Thanks for the help Josh

0 Likes
Highlighted
Shopify Staff
Shopify Staff
1129 82 179

Hey again @JohnCole , 

 

Thanks for getting back to me - and that's a fair assessment. 

 

I'm not sure why the decision to use a header was made in all honesty, but I would imagine it's because it is a bit less intrusive and helps avoid issues with apps that are parsing resources a specific way. I know in the past that making changes to the body of webhooks, for example, has caused a lot of headaches when people are consuming them and expecting the body to be formatted a specific way.

 

I do see where you're coming from as well though, and I'll pass the feedback along to the team behind cursors here in the event this becomes feedback we're hearing frequently. 

0 Likes
Highlighted
Shopify Partner
3 0 1

Update 2:

The saving grace here is that the previous link appears to always come first when it's there. So here's a Javascript snippet for anyone struggling with this - hope it helps!

if (response.headers.link && response.headers.link.indexOf(`rel="next"`) > -1) {
                        try {
                          // Try to parse our the string we need
                          let nextLink = response.headers.link;

                          // If there's a previous link, remove the first part of the string entirely
                          if (nextLink.indexOf(`rel="previous"`) > -1) {
                            nextLink = nextLink.substr(nextLink.indexOf(",") + 2, nextLink.length);
                            messageService.logSuccess("Removed previous link from returned header, resulting value is: " + nextLink);
                          } 

                          // Parse the remaining string for the actual link
                          nextLink = nextLink.substr(1, nextLink.indexOf(">") - 1);
                          messageService.logSuccess('SHOPIFY SERVICE - GETALLORDERS', `Found nextLink (${nextLink}), continuing.`)
                          // READY - CALL THE NEXT SET WITH NEXTLINK
                            
                        } catch(ex) {
                            messageService.logError('SHOPIFY SERVICE - GETALLORDERS', 'Failed to parse nextlink, printing headers:')
                            console.log(response.headers)
                        }
                    }
                    else {
                        messageService.logSuccess('SHOPIFY SERVICE - GETALLORDERS', 'No nextLink returned, continuing.')
                        // DONE, NO MORE ORDERS
                    }

 

Update 1: this is much more difficult than I expected. When you have previous and next pages, the two links are just jammed together into a string with a seemingly arbitrary delimiting structure:

 

<https://STORE.myshopify.com/admin/api/2019-10/orders.json?limit=20&page_info=PAGEINFO>; rel="previous", <https://STORE.myshopify.com/admin/api/2019-10/orders.json?limit=20&page_info=PAGEINFO>; rel="next"

How are you expected to parse that out without a super convoluted way of determining which link is correct? Does it follow some standard that I can write against? The guide makes it sound so easy, but I have never seen a pagination structure like this. Why not use something well-established like HATEOAS? Or even just pass it back as stringified JSON?

 

Original post:

Hey @Josh - any update on this? I'm working on this change now, and it feels... wrong to be parsing the string. Even if it were just passed back as a string without the additional cruft that would be excellent.

 

<https://STORE-NAME.myshopify.com/admin/api/2019-10/orders.json?limit=5&page_info=PAGE_INFO>; rel="next"

 

Another gripe - I'm using basic auth for the initial request, but the returned URL is the normal REST API call. I'll work on switching to token auth, but this seems like a significantly breaking change that wasn't mentioned in the release (as far as I saw)

 

Thanks!

Adam

1 Like
Highlighted
Shopify Staff
Shopify Staff
1129 82 179

Hey Adam, 

 

So that is a web linking header that you're seeing and we're following the spec outlined here : https://tools.ietf.org/html/rfc5988

 

With regards to the Basic Auth URL not being returned in the header though, that's most likely a bug. I'll file a bug report to get that taken care of. 

0 Likes
Highlighted
Shopify Partner
3 0 1

Hi Josh,

 

Thank you so much for getting back to me so quickly. I've switched over to token-based auth and gotten this to work with the Web Linking header. The spec is verbose, and I couldn't find any simplified explanations of the actual implementation to write against. It seems like the spec was never approved... even the MDN page is still in draft. It might help if some additional information was included on the API documentation, or even an example of how to parse it.

 

There were two other issues I ran into:

 

1. If I appended "status=any" into the initial call, I had to avoid appending it in subsequent (cursor-based) calls or I would get a 400 error. The docs mention:

A request that includes the page_info parameter can't include any other parameters except for limit.

However, I would imagine Shopify should just ignore additional parameters instead of throwing an error. Now I have to have special logic for handling if it's the initial call or a cursor-based call.

 

2. The rate limiting starts to slow down our calls if we're trying to pull back a large list of orders, then eventually fail entirely (500 status code). Another Shopify thread mentioned that lowering the page size (from 250 to 100) would help. The issue is our API runs on AWS Lambdas, so there's a 30 second timeout limit on our end due to API Gateway's hard upper limits. Prior to these changes, I could make multiple calls to OUR API, passing an auto-incremented page number each time (get page 1, get page 2, etc), which would get around the timeout our end. However, now that there's no way to call a specific page, I have to depend on our API to handle the pagination (which causes the timeout) or return the `page_info` to the client, then have the client pass that back in... which means it's nearly impossible to pull back a history of all of orders without our API timing out. I'm going to see if I can fix that on our end, but I can see this being a serious problem for people who don't have control of their infrastructure.

 

Thanks again for your help and quick responses!

 

Regards,

Adam

 

0 Likes
Highlighted

Hi, 

 

In the previous way, I know which page is the current page with the page parameter. Now how can I do so with the cursor-based pagination? I can only know the first, last and the middle pages? I am trying to do "Page X of Y", where X is the current page and Y is the total number of page. 

SPO - SEO App to research keywords & edit social link preview
0 Likes
Highlighted
Shopify Partner
5 0 0

HI @Josh 

Can i pick your brains as im a little stumped... 

i call my api for products and it works for the first page, 

Question 1: should i get the value from the link header to get to next page?  or do i simply call next again to get the next page? 

example:

https://nnnnn@linneyteststore.myshopify.com/admin/api/2020-01/products.json?fields=title,variants&limit=2;rel="next"

 

(its just that the link header value doesnt work in postman?

 

i thought it was a case of running the url for next until no results return, (im just doing a While loop at the mo and trying to get it to work in postman before implementing of course :-)

thanks

Liam

 

0 Likes
Highlighted
Shopify Partner
5 0 0

ok , figured this out , basic auth in postman and remove the braces etc..<> 

got mine working to look for Header.contains 'next' else end 

 

0 Likes