Shopify API - pagination is broken

[Travis_D]

Hi there, 

This is a pretty important issue. It seems as of right now, pagination is broken on Shopify for Variants. 

We are using the Python SDK, version 7.0.2 and API version 2020-01. 

Code:

next_url = None
all_results = []
# grab all of the variants
while True:
    if next_url:
        print(next_url)
        results = shopify.Variant.find(limit=250, from_=next_url)
    else:
        results = shopify.Variant.find(limit=250)
        print('Got {} results'.format(len(results)))

    if results.has_next_page():
        print('Yes, has another page')
        next_url = results.next_page_url
    else:
        next_url = None

    all_results += results
    if not next_url:
        break

Produces: 

Got 250 results
Yes, has another page
https://milspin.myshopify.com/admin/api/2020-01/variants.json?limit=250&page_info=eyJsYXN0X2lkIjo2Nzg2NTM4MjQyMDg3LCJsYXN0X3ZhbHVlIjoiNjc4NjUzODI0MjA4NyIsImRpcmVjdGlvbiI6Im5leHQifQ
...
pyactiveresource.connection.BadRequest: Response(code=400, body="b'{"errors":{"page_info":"Invalid value."}}'", headers={'Date': 'T

I've tried this on two different shops and got the same result. 

Any ideas? 

Update: tried the same code for Order and Product and they are broken as well. 

[Greg_Kujawa]

Worked for me, against the 2020-04 API. Details below!

GET https://{my_shop}.myshopify.com/admin/api/2020-04/products.json HTTP/1.1
Content-Type: application/json
Authorization: Basic {my_access_token}
User-Agent: PostmanRuntime/7.25.0
Accept: */*
Host: {my_shop}.myshopify.com
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Cookie: __cfduid=d474d283195031f2c442a1f3fcd4abe0d1591975172


HTTP/1.1 200 OK
Date: Tue, 16 Jun 2020 18:41:41 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
X-Sorting-Hat-PodId: 161
X-Sorting-Hat-ShopId: 3036253
Vary: Accept-Encoding
Referrer-Policy: origin-when-cross-origin
X-Frame-Options: DENY
X-ShopId: 3036253
X-ShardId: 161
X-Stats-UserId: 
X-Stats-ApiClientId: 309925
X-Stats-ApiPermissionId: 8304915
X-Shopify-API-Terms: By accessing or using the Shopify API you agree to the Shopify API License and Terms of Use at https://www.shopify.com/legal/api-terms
HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT: 1/40
X-Shopify-Shop-Api-Call-Limit: 1/40
X-Shopify-API-Version: 2020-04
Link: <https://{my_shop}.myshopify.com/admin/api/2020-04/products.json?limit=50&page_info=eyJsYXN0X2lkIjo0NDI1NzQ0Nzc3MjY4LCJsYXN0X3ZhbHVlIjoiMTRLIFdoaXRlIEdvbGQgSGFsbyBSaW5nIiwiZGlyZWN0aW9uIjoibmV4dCJ9>; rel="next"
Strict-Transport-Security: max-age=7889238
X-Shopify-Stage: production
Content-Security-Policy: default-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://* shopify-pos://*; block-all-mixed-content; child-src 'self' https://* shopify-pos://*; connect-src 'self' wss://* https://*; frame-ancestors 'none'; img-src 'self' data: blob: https:; script-src https://cdn.shopify.com https://cdn.shopifycdn.net https://cdn.shopify.cn https://checkout.shopifycs.com https://js-agent.newrelic.com https://bam.nr-data.net https://api.stripe.com https://mpsnare.iesnare.com https://appcenter.intuit.com https://www.paypal.com https://js.braintreegateway.com https://c.paypal.com https://maps.googleapis.com https://www.google-analytics.com https://v.shopify.com https://widget.intercom.io https://js.intercomcdn.com 'self' 'unsafe-inline' 'unsafe-eval'; upgrade-insecure-requests; report-uri /csp-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=225a7ef1-93fc-4685-b4b5-718f01ea1143
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
X-XSS-Protection: 1; mode=block; report=/xss-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=225a7ef1-93fc-4685-b4b5-718f01ea1143
X-Dc: gcp-us-central1,gcp-us-east1,gcp-us-east1
NEL: {"report_to":"network-errors","max_age":2592000,"failure_fraction":0.01,"success_fraction":0.0001}
Report-To: {"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}
X-Request-ID: 225a7ef1-93fc-4685-b4b5-718f01ea1143
NEL: {"report_to":"network-errors","max_age":2592000,"failure_fraction":0.01,"success_fraction":0.0001}
Report-To: {"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}
CF-Cache-Status: DYNAMIC
cf-request-id: 036008f3470000f22ab7381200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 5a46aa987997f22a-ORD
alt-svc: h3-27=":443"; ma=86400
Content-Length: 150512

{"products":[{"id":4423635632180,"title":"\"Be Own Kind of Beautiful\" Cuff","body_html":"..."}]}

------------------------------------------------------------------

GET https://{my_shop}.myshopify.com/admin/api/2020-04/products.json?limit=50&page_info=eyJsYXN0X2lkIjo0NDI1NzQ0Nzc3MjY4LCJsYXN0X3ZhbHVlIjoiMTRLIFdoaXRlIEdvbGQgSGFsbyBSaW5nIiwiZGlyZWN0aW9uIjoibmV4dCJ9 HTTP/1.1
Content-Type: application/json
Authorization: Basic {my_access_token}
User-Agent: PostmanRuntime/7.25.0
Accept: */*
Host: {my_shop}.myshopify.com
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Cookie: __cfduid=d474d283195031f2c442a1f3fcd4abe0d1591975172


HTTP/1.1 200 OK
Date: Tue, 16 Jun 2020 18:41:53 GMT
Content-Type: application/json; charset=utf-8
Connection: keep-alive
X-Sorting-Hat-PodId: 161
X-Sorting-Hat-ShopId: 3036253
Vary: Accept-Encoding
Referrer-Policy: origin-when-cross-origin
X-Frame-Options: DENY
X-ShopId: 3036253
X-ShardId: 161
X-Stats-UserId: 
X-Stats-ApiClientId: 309925
X-Stats-ApiPermissionId: 8304915
X-Shopify-API-Terms: By accessing or using the Shopify API you agree to the Shopify API License and Terms of Use at https://www.shopify.com/legal/api-terms
HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT: 1/40
X-Shopify-Shop-Api-Call-Limit: 1/40
X-Shopify-API-Version: 2020-04
Link: <https://{my_shop}.myshopify.com/admin/api/2020-04/products.json?limit=50&page_info=eyJkaXJlY3Rpb24iOiJwcmV2IiwibGFzdF9pZCI6NDQyNTc0NTIwMzI1MiwibGFzdF92YWx1ZSI6IjE0SyBXaGl0ZSBHb2xkIEhhbG8gUmluZyJ9>; rel="previous", <https://{my_shop}.myshopify.com/admin/api/2020-04/products.json?limit=50&page_info=eyJkaXJlY3Rpb24iOiJuZXh0IiwibGFzdF9pZCI6NDQ1MTk1ODI5MjUzMiwibGFzdF92YWx1ZSI6IjE0S1JHIEEuIEpBRkZFIERJQU1PTkQgRVRFUk5JVFkgQkFORCJ9>; rel="next"
Strict-Transport-Security: max-age=7889238
X-Shopify-Stage: production
Content-Security-Policy: default-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://* shopify-pos://*; block-all-mixed-content; child-src 'self' https://* shopify-pos://*; connect-src 'self' wss://* https://*; frame-ancestors 'none'; img-src 'self' data: blob: https:; script-src https://cdn.shopify.com https://cdn.shopifycdn.net https://cdn.shopify.cn https://checkout.shopifycs.com https://js-agent.newrelic.com https://bam.nr-data.net https://api.stripe.com https://mpsnare.iesnare.com https://appcenter.intuit.com https://www.paypal.com https://js.braintreegateway.com https://c.paypal.com https://maps.googleapis.com https://www.google-analytics.com https://v.shopify.com https://widget.intercom.io https://js.intercomcdn.com 'self' 'unsafe-inline' 'unsafe-eval'; upgrade-insecure-requests; report-uri /csp-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=25642b22-de01-4d0f-9836-fdd95b405ec6
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
X-XSS-Protection: 1; mode=block; report=/xss-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Fproducts&source%5Bsection%5D=admin_api&source%5Buuid%5D=25642b22-de01-4d0f-9836-fdd95b405ec6
X-Dc: gcp-us-central1,gcp-us-east1,gcp-us-east1
NEL: {"report_to":"network-errors","max_age":2592000,"failure_fraction":0.01,"success_fraction":0.0001}
Report-To: {"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}
X-Request-ID: 25642b22-de01-4d0f-9836-fdd95b405ec6
NEL: {"report_to":"network-errors","max_age":2592000,"failure_fraction":0.01,"success_fraction":0.0001}
Report-To: {"group":"network-errors","max_age":2592000,"endpoints":[{"url":"https://monorail-edge.shopifycloud.com/v1/reports/nel/20190325/shopify"}]}
CF-Cache-Status: DYNAMIC
cf-request-id: 03600920ef000003ec37ae5200000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 5a46aae1791903ec-ORD
alt-svc: h3-27=":443"; ma=86400
Content-Length: 95849

{"products":[{"id":4425745203252,"title":"14K White Gold Halo Ring","body_html":"..."}]}

------------------------------------------------------------------

[Travis_D]

Thanks for checking that. 

Apparently passing in the limit=250 is causing the problem. It works without it. 

This works:

        results = shopify.Variant.find(from_=next_url)

but this does not:

        results = shopify.Variant.find(limit=250, from_=next_url)

[Travis_D]

It's 250:

https://www.shopify.ca/partners/blog/relative-pagination

[Greg_Kujawa]

Requesting the next page using ?limit=250 and referencing the correct page_info value worked fine for me. I would ensure that the page_info value being used is correct, based on the previous response's header.

[Travis_D]

I'm using the Python SDK, which should be handling that. Sounds like an SDK bug then.

[Greg_Kujawa]

Sounds like it. When I run into quirks using my own code projects I usually drop down to using Postman and passing along API requests in raw data form. That way I can see if it's more an issue with what I'm requesting versus an issue with my code project internally.