Discussing APIs and development related to customers, discounts, and order management.
Our app queries all orders in a store using the 2019-10 REST API, orders endpoint and page_info pagination for use in a BI software tool.
We are experiencing very fluctuating response times ranging from an initial sub-seconds response for the first n requests to 20 seconds or more after a threshold has been hit. We can't seem to pinpoint this threshold and it seems to be different every time.
We're testing against a premium store with >170K orders and thus should have a burst of 80 requests and a leak rate of 4 per second but from the returned headers I can see we're not hitting the API limits (always a response of 1/80). All requests are sent serially due to pagination. Our app is cleared to read all orders by Shopify.
We've run various tests with both small and large limits and the threshold seems to be hit after more requests when using a small limit. The actual data download happens in milliseconds but the TTFB (Time To First Byte) seems to increase after this threshold. We've also tested against the GraphQL API with similar results.
Fetching a full dataset or just specific fields didn't seem to have an impact.
This is the request we're making: https://{shopname}.myshopify.com/admin/api/2019-10/orders.json?created_at_min=2019-06-01T00:00:00+01:00&limit=250
In the below graphs you can see on the left the fast responses turning into the slow responses on the right. The green part of each bar is the TTFB, the blue part is the actual data download.
As we need to fetch all records for the store and after the threshold each request takes 20s it would take us 226 minutes to download the whole dataset whereas a 1 second response would take just 11 minutes.
Are others seeing this behavior as well? What can we do to prevent this?
Happy to supply further detail if needed to investigate.
Hi @mverstrepen,
I haven't yet been able to pinpoint the source of the issue that you have here but the information you have provided is very useful so far. Would you additionally be able to send over the `x-request-id` header of any of the responses from the time period in which responses slow down significantly? Ids from recent requests (i.e. within the past 3 days) would be ideal. This will allow me to see any patterns within our logs pertaining to slow responses and hopefully help me find the source of the issue.
One recommendation is to switch to using Bulk Operations queries instead of requesting each order individually. A Bulk Operation query will prevent you running into any client side throttling and can be polled for progress such that you can keep track of the completion of the query. Information on bulk operations can be found here: https://help.shopify.com/en/api/guides/bulk-operations
To learn more visit the Shopify Help Center or the Community Blog.
Hi @sd_,
I've re-run the test and the first slower response has the following x-request-id: 948314bd-6ec4-41d1-87c8-4fa6d7ebeaf7 with a response time of 4.41s
The request just before that was 0caa1814-c91b-4497-89f0-42815c90ec5f with a response time of 982ms
I've set the pagesize to 50, when increasing this to 250 the wait time also increases and the delay occurs faster.
I'll have a look at the bulk operations but initially assumed they were for updates only, not download of the data.
To learn more visit the Shopify Help Center or the Community Blog.
hi @sd_ , have you been able to lookup the request-ids and see what's causing the delay?
I've since tried to use the bulk option to download our order data. Initially it seemed like a solution to our problem when we're using a limited amount of fields to query, but when we add all required fields it slows down so much that I am already waiting for 20 minutes and the status update tells me it has processed 10.000 records. At this pace I would take well over 4 hours to complete.
As a reference it's the following bulk operation gid://shopify/BulkOperation/3184328773 (still running when writing this) the x-request-id of starting the bulk operation is this: 3c987911-dfa0-4920-90df-f018827c86c2
Do you have any other suggestions?
hi @sd_,
I've run another set of tests where we don't create 1 big request for the entire date range but split it up into multiple requests, one for each day. We then burst these requests with an initial 30 at once and then move to 3 per second to run through all the dates (and later all the next pages for some of the dates). This to make sure we stay under the API Limit (I think we could do 80/4 on this site if needed).
However: we face a similar problem here, the response on a single request can take up to 4.7 minutes, and 1% results in a number of 500 internal server errors. Have you been able to pinpoint the cause of slow responses from my earlier posts or is there someone else withing the Shopify team we can talk to?
PS. Running the same batch of requests shortly after the previous set results in much faster load times with the longest request taking 31.09s (are we hitting your cache here?)
Please ignore the last post. The reported slow loading in this last test was caused by the cors proxy forcing http1 and a Chrome limitation of 6 concurrent tcp requests. Switching to http2 allowed for much faster loading.
I would still love to know why the other request (longer timeframe) slow down and why batch download is so slow.