Orders API response time fluctuates

Highlighted
Shopify Partner
7 0 1

Our app queries all orders in a store using the 2019-10 REST API, orders endpoint and page_info pagination for use in a BI software tool.

We are experiencing very fluctuating response times ranging from an initial sub-seconds response for the first n requests to 20 seconds or more after a threshold has been hit. We can't seem to pinpoint this threshold and it seems to be different every time.

 

We're testing against a premium store with >170K orders and thus should have a burst of 80 requests and a leak rate of 4 per second but from the returned headers I can see we're not hitting the API limits (always a response of 1/80). All requests are sent serially due to pagination. Our app is cleared to read all orders by Shopify.

 

We've run various tests with both small and large limits and the threshold seems to be hit after more requests when using a small limit. The actual data download happens in milliseconds but the TTFB (Time To First Byte) seems to increase after this threshold. We've also tested against the GraphQL API with similar results.

 

Fetching a full dataset or just specific fields didn't seem to have an impact.

This is the request we're making: https://{shopname}.myshopify.com/admin/api/2019-10/orders.json?created_at_min=2019-06-01T00:00:00+01:00&limit=250

 

In the below graphs you can see on the left the fast responses turning into the slow responses on the right. The green part of each bar is the TTFB, the blue part is the actual data download.

no limit eq 50 requestsno limit eq 50 requestslimit of 250 requestslimit of 250 requests

 

As we need to fetch all records for the store and after the threshold each request takes 20s it would take us 226 minutes to download the whole dataset whereas a 1 second response would take just 11 minutes.

 

Are others seeing this behavior as well? What can we do to prevent this?

Happy to supply further detail if needed to investigate.

0 Likes
Highlighted
Shopify Staff
Shopify Staff
51 10 5

Hi @mverstrepen,

 

I haven't yet been able to pinpoint the source of the issue that you have here but the information you have provided is very useful so far. Would you additionally be able to send over the `x-request-id` header of any of the responses from the time period in which responses slow down significantly? Ids from recent requests (i.e. within the past 3 days) would be ideal. This will allow me to see any patterns within our logs pertaining to slow responses and hopefully help me find the source of the issue. 

 

One recommendation is to switch to using Bulk Operations queries instead of requesting each order individually. A Bulk Operation query will prevent you running into any client side throttling and can be polled for progress such that you can keep track of the completion of the query. Information on bulk operations can be found here: https://help.shopify.com/en/api/guides/bulk-operations

 

0 Likes
Highlighted
Shopify Partner
7 0 1

Hi @sd_,

 

I've re-run the test and the first slower response has the following x-request-id: 948314bd-6ec4-41d1-87c8-4fa6d7ebeaf7 with a response time of 4.41s
The request just before that was 0caa1814-c91b-4497-89f0-42815c90ec5f with a response time of 982ms

 

I've set the pagesize to 50, when increasing this to 250 the wait time also increases and the delay occurs faster.

 

I'll have a look at the bulk operations but initially assumed they were for updates only, not download of the data.

0 Likes
Highlighted
Shopify Staff
Shopify Staff
51 10 5
Hi @mverstrepen,

Bulk Operations is actually specifically for the download of data. Upon
completion, you are provided with a file in the JSON Lines format.

Thanks for sending over the request-ids. I'll take a closer look.
0 Likes
Highlighted
Shopify Partner
7 0 1

hi @sd_ , have you been able to lookup the request-ids and see what's causing the delay?

I've since tried to use the bulk option to download our order data. Initially it seemed like a solution to our problem when we're using a limited amount of fields to query, but when we add all required fields it slows down so much that I am already waiting for 20 minutes and the status update tells me it has processed 10.000 records. At this pace I would take well over 4 hours to complete.

As a reference it's the following bulk operation gid://shopify/BulkOperation/3184328773 (still running when writing this) the x-request-id of starting the bulk operation is this: 3c987911-dfa0-4920-90df-f018827c86c2

Do you have any other suggestions?

0 Likes
Highlighted
Shopify Partner
7 0 1

hi @sd_,

I've run another set of tests where we don't create 1 big request for the entire date range but split it up into multiple requests, one for each day. We then burst these requests with an initial 30 at once and then move to 3 per second to run through all the dates (and later all the next pages for some of the dates). This to make sure we stay under the API Limit (I think we could do 80/4 on this site if needed).

 

However: we face a similar problem here, the response on a single request can take up to 4.7 minutes, and 1% results in a number of 500 internal server errors. Have you been able to pinpoint the cause of slow responses from my earlier posts or is there someone else withing the Shopify team we can talk to?

 

PS. Running the same batch of requests shortly after the previous set results in much faster load times with the longest request taking 31.09s (are we hitting your cache here?)

 

1 Like
Highlighted
Shopify Partner
7 0 1

Please ignore the last post. The reported slow loading in this last test was caused by the cors proxy forcing http1 and a Chrome limitation of 6 concurrent tcp requests. Switching to http2 allowed for much faster loading.

 

I would still love to know why the other request (longer timeframe) slow down and why batch download is so slow.

0 Likes
Highlighted
New Member
1 0 0
We've been running into this same issue recently trying to get older orders.

We also get 500 errors randomly for about 1% of requests and have to retry the request, but I'm less concerned about that since we just retry anyways.

The first 20 requested are completed in under 1s each, but then afterwards it slows down to about 30s per page of 250 orders. For some of our large merchants with over 100k orders this is problematic.

Did you ever figure out a solution?
0 Likes