500 {"errors":"Internal Server Error"} from simple /admin/orders.json requests

Shopify Staff
Shopify Staff
291 29 56

Hey Joerz,

 

I've taken a look at the request id's which you've provided and I can see that these 500 errors are caused by exceptional cases when there are a lot of queries being made to the database across multiple API clients.

 

In these instances, the recommended approach is to implement some sort of time delay and then to retry your request. Increasing the time between each subsequent retry can also be a useful approach to limit the overall number of API calls made, as it's not possible to tell when the database associated with the requested shop will be back under normal operating load.

0 Likes
Shopify Partner
6 0 0

Hi @Zameer 

 

Wow that's unfortunate. I have already built in a bunch of retry / back off logic.  Could work that same logic for 500's - tough to distinguish real errors vs. this "false" error. 

 

Our current logic is fine for small Shopify stores, but on days with lots of volume, its not possible.  We have a client which is averaging 44,000+ orders per day.  At this point I have NO idea how to do a full historical data load from 2015..  As for daily incremental loads, this MIGHT work, but I felt I would have to use webhooks vs. polling APIs.

 

Would graphQL help in these scenarios?

Would finding a starting point (order number) and just query "since_id" from that point forward?  That helps w/ historical load (not daily "updated_at" incrementals)

 

Thank you so much for your guidance.

 

0 Likes
Highlighted
Shopify Partner
1 0 0

Hi Joerz,

 

I have the same error but extracting more than 50000 products, pagination from 100 in 100, Re: 500 {"errors": "Internal Server Error"}

 

We insert all the fields that the shopify api brings to our database, it's frustrating, when I remove the body_html it works correctly, apparently it is the large amount of data that each product brings, presumably body_html brings a large amount of data.

 

Some help?

 

Thank you,

0 Likes
New Member
1 0 0

Hi sagitario

 

There isn't a solid solution from Shopify; including webhooks and GraphQL.  Now, my issue was related to orders, not products, and was related to the number of orders within a date range. My scenario was trying to receive orders that were updated in a 24hr timeframe.  It is definitely related to the amount of data *and* how busy the servers are.

 

In my case I was date based, so the biggest thing I've done that helps is breaking the time frame into smaller timespans:  1 day turned into 24 hour blocks.  Only retrieve data for the 1 hour blocks.  Thus it was a smaller amount of data and worked without issue.  I even perform the blocks in parallel (up to 8 at a time).

 

Other things I've implemented:

* I use the `since_id` parameter to help pull in an optimized way (as noted in discussions -- but this really didn't help much)

* when retrieving a throttled status (http 429), I sleep and wait as oppose to bailing out

* changing page size didn't help much either - but *could*

 

I've found, you need to rework your algorithm to pull the data.

0 Likes
Shopify Expert
703 0 65

We're experiencing the same thing with Customer API calls... Trying to count Customers with a certain tag, borks after page 40 (which may or may not coincidentally be the 10,000th record?): `/admin/customers/search.json?query=tag:tag&fields=id,tags&limit=250&page=41`

 

@Zameer  any advice here?

I'm a million different people
0 Likes
Tourist
6 0 2

We have a similar issue, but instead of 500 error we are getting a lot of 403 errors. It's not about permissions, because our integrations have been running a long time without issues and those requests succeed after retries(using Sidekiq).

1 Like
Shopify Staff
Shopify Staff
291 29 56

@rickydazla The issue is most likely being caused by the offset created by the query. I would recommend switching over to `since_id` pagination to help deal with the issue.

 

@milep Can you please provide me with a Request-Id for the 403 error you're seeing?

0 Likes
Tourist
6 0 2
We are in process of adding response headers logging. Any other way you can help before we get that ready?
0 Likes
Shopify Staff
Shopify Staff
291 29 56

What shop are you experiencing this on and what's the API key for the app?

0 Likes
Tourist
6 0 2
Api key: 99adb7d80e116b3a9ae12010fd8e50d0
Error happens in every installed shop(at least in those that have any considerable amount of requests to api). I can provide more details in private messages.

- Miika
0 Likes