Issue with getting more than 100 000 customers

Tourist
5 0 1

Hi all,

 

We are having an issue getting customers past 100 000, specifically we are making a call for '/admin/customers.json?limit=250&page=401', and it returns a blank result set, where the store does have over 100 000 customers.

 

We are not specifying an API version, so it should use version 2019-04 which should still allow index based pagination, and we do get the customers up to and including page 400, so it is a rather abrupt stop.

 

We have tried this for several stores and different apps with the same results. We have also tried for different page indices and limits, and all return empty result sets once it gets to customer 100 000; and specifically the whole result set is blank, not just customer 100 000 on, ex: '/admin/customers.json?limit=249&page=402' also returns an empty result set.

 

 

Issue: Shopify API (version 2019-04) returns no results from customer 100 000. 

 

We tried: Different stores, different api keys, different page and the results are the same.

 

Question: Has anyone else seen this behaviour, or are you able to reproduce it?

 

Thank you for any help. If anyone wants more info just let me know.

1 Like
Shopify Staff
Shopify Staff
1028 68 135

Hello @Samuel_Moddapps , 

 

There is a limit on our API endpoints that will refuse GET requests for any resource when the offset generated by the request is over 100,000 - it's documented here. So the request you are performing would be refused on any shop on the platform currently. 

 

Are you trying to fetch all of the customers on a shop, by chance? If so, you may want to consider using the since_id parameter instead of page, or migrating to cursor-based pagination instead. Either option will enable you to fetch over 100,000 of a resource.

 

 

0 Likes
Tourist
5 0 1

Hi Josh,

 

I was not aware of that additional limitation, that being said, the docs say it will return a 429 Too Many Requests when that occurs, but we are seeing a 200 as the response. In addition to that we were able to grab customers over the 100 000 offset up until about a week ago, and we are still able to grab past the 100 000 on other resources.

0 Likes
Shopify Staff
Shopify Staff
1028 68 135

Hey again @Samuel_Moddapps , 

 

That's interesting - do you log any response headers on your end? If so, would you be able to provide the value of the X-Request-Id header from one of these responses? 

 

Otherwise, if you wouldn't mind providing a shop ID and timeframe, I could check our logs to see what is happening.

0 Likes
Tourist
5 0 1

Hi Josh,

 

I sent you a pm with the request ids.

 

Samuel

0 Likes
Highlighted
Shopify Staff
Shopify Staff
1028 68 135

Hey again @Samuel_Moddapps , 

 

Thanks for doing that! And I owe you an apology, my response was not entirely accurate. We do throttle requests that generate an offset over 100,000, but we don't refuse the request outright - we just have a stricter call limit in place. 

 

For calls such as the ones that you are making, the throttle is just one call per second instead of the two calls per second that our regular call limit allows. So you can make these calls, just not nearly as quickly. I would still recommend having a look at our documentation about cursor-based pagination, because we do intend to deprecate the page parameter next year.

0 Likes
Tourist
5 0 1

Hi Josh,

 

Thank you for looking into this.

We do have plans to upgrade, but currently we have apps that are in use using the 2019-04 api, and we are still getting empty result sets from the customer calls past an offset of 100 000.

 

Samuel

0 Likes
Pathfinder
129 4 24

Just ran into this issue as well just now for a customer of our app. :(

 

They have over 400k records and the API is also only returning max of 100k records (page=401 returns 0 records with a 200 status code).

0 Likes
Shopify Staff
Shopify Staff
1028 68 135

Hey again guys, 

 

I've been able to replicate this and will file a bug report because I don't think it is intended behaviour. 

 

With that being said, I'd also like to take this opportunity to again point out our docs for cursor-based pagination. Since we will be deprecating the page parameter in the future and this will be its replacement, now would be a great time to look at migrating! It's not only far more performant, it'll allow you to get past this issue. Using the since_id parameter to page through a resource will get you past this problem with the page parameter as well. 

 

You can even use since_id only after you reach 100,000 customers by taking the last customer from the response from 'limit=250&page=400' and using that as the since_id value for your next API call. 

 

That's not to say this isn't a bug anyway since it does appear to be, so I'll file a report and hope that we can get this fixed - but the above can help you get past the issue in the meantime until a fix is in place. 

0 Likes
Pathfinder
129 4 24

Thanks for looking into this Josh!

 

We did have to push an emergency fix last night to use `since_id` which I can confirm doesn't have the 100k bug.

 

We have multiple apps so we tend to roll things out slowly by updating the one with the least number of users first (which we did last week), wait at least a week in case a weird bug pops up, and then move on to the others.  But now we have to rush things a bit due to the bug.

 

It's just a little frustrating when something like this happens as we're under the impression that the API versioning was introduced to prevent breaking changes like this to get pushed out to older API versions. :(

1 Like