Pull orders data using API

New Member
11 0 0

Hi Team,

 

We are trying to pull order's data for the period of 15 days. Order volume is close to 1,000,000. While pulling the data using REST admin api, we keep getting 500 error. 

Any suggestion on how to pull this huge amount of data?

 

Thanks.

0 Likes
Shopify Staff
Shopify Staff
1087 75 140

Hey @MeasuredInc , 

 

Do you have an example of the shop that this occurred on, and a rough timeframe of when you made this request? If you log request IDs and could provide one of those that would be even better if possible. Given that you're attempting to pull a million orders though, chances are good that this 500 error is a result of database issues that may not be easily avoided. 

 

Have you tried to pull these orders using our new cursor-based pagination? It's a far more performant method of pagination compared to the page parameter if that's what you have been using, and should do a much better job. 

0 Likes
New Member
11 0 0

Hi,

I executed it again and tried to pull the data for only one day, but got the same error.

Yes, we are using the said method, following is the URL which we are using

https://fnova.myshopify.com/admin/api/2019-10/orders.json?limit=250&status=any&updated_at_min=2019-0... asc

 

In response header we get url for next set of data. I executed this at 12:40PM EST on 18 September 2019.

 

Error:

Request failed with status code 500

0 Likes
Shopify Staff
Shopify Staff
1087 75 140

Hey again @MeasuredInc , 

 

Although it's just one day of orders, that's a single day that is a long time ago on a shop with an incredibly large number of orders. We still have to read all the orders to find a starting point to respond with, and that's just an extremely expensive query being performed. 

 

I'd suggest taking a look at our newer cursor-based pagination and giving it a shot - it's a lot faster and more reliable and should help with this. 

0 Likes
New Member
11 0 0

Hey @Josh ,

 

We are using cursor-based pagination , then also Error: 500 is thrown.

 

Thanks.

0 Likes
Shopify Staff
Shopify Staff
1087 75 140

Hey again @MeasuredInc , 

 

Interesting, the only other thing that could be causing this is the dates being set in your query. 

 

As a workaround given the volume of orders on this shop, I think it would be best to employ cursor-based pagination still, but watch updated_at timestamps manually. 

 

If you send a request to /admin/api/2019-10/orders.json it will return orders in descending order, giving you the most recent first, and least recent last. If you grab the Link header in this case and follow it to the next page, this pattern will continue and you can paginate backwards until you hit the updated_at point that you want to paginate to. 

 

This is a little more of a manual process admittedly because you'd have to watch the updated_at timestamps yourself and know when to stop, but it should enable you to get all of the orders you need. 

 

Please let me know if you're still hitting 500 errors doing this though, I think it would be a case that our team building out this feature would love to have a look at.

0 Likes
Highlighted
New Member
11 0 0

---

0 Likes
New Member
11 0 0

Hi Josh,

 

 

We got the following response below when we shared the latest API call logs (with 500 errors) with the Shopify rep for Fashion Nova.

 

"Index requests are using pagination however, pagination has not yet been released for Orders and you're using the upcoming API release for October, not the currently live release. Here is an overview of supported endpoints and release candidates happening in october: https://help.shopify.com/en/api/guides/paginated-rest-results#supported-endpoints"

0 Likes
Shopify Staff
Shopify Staff
1087 75 140

Hey again @MeasuredInc , 

 

Sorry for the delayed response - 

 

While cursor-based pagination for orders is technically not officially available until the 2019-10 version release, it does work. At this point the version is so close to being made available anyway, anything included in it should be in good working order at this point - if not, anything not working as expected should be reportable as a bug. 

 

It would technically still be possible to have errors on a shop as huge as Fashion Nova I suppose, but cursor-based pagination would be your best possible option in terms of avoiding them. If you're still seeing 500 errors returned, if you could provide a request ID from one of those 500 errors it would help to dig up some logs to share with our development team. 

0 Likes