High number of "500 Internal Server Errors" while paginating products over REST API

[Naren1]

Shopify team
Since Sept 10th, we've had multiple stores  fail with "500 Internal server" error. Even re-running them 12 hours later does not fix the issue. 
I've attached one of the request id  and timestamp for debugging by Shopify staff.
Already raised ticket id 50836811 but no response yet

x-request-id: 25a59fd6-22af-4bf1-91ce-6a7eeccd10fb-1726646623

date: Wed, 18 Sep 2024 08:04:02 GMT
API version: 2024-01

GET /admin/api/2024-01/products.json?limit=250&page_info=eyJzdGF0dXMiOiJhY3RpdmUiLCJkaXJlY3Rpb24iOiJuZXh0IiwibGFzdF9pZCI6NDQ2ODExOTMwNjM1MywibGFzdF92YWx1ZSI6IkNob29zZSBIYXBweSAtIE1ldGFsIEVuYW1lbCBXYWxsIFNpZ24gLSAxNC0xXC8yLWluIn0& HTTP/2

According to the error code description in the docs,  this is clearly an issue on shopify's end

500 Internal Server Error
An internal error occurred in Shopify. Simplify or retry your request. If the issue persists, then please record any error codes, timestamps and contact Partner Support so that Shopify staff can investigate.

All we are trying to do is paginate and get the products for the store.
Each time it is a different store that fails and each time the error happens on a different "page" number of the products pagination.
This has become a production support nightmare for us where we have to explain to customers why the app is not working only for certain stores. 

Yes I know the correct answer is to migrate to GraphQL before Feb 1 deadline.
We are working on that.
But it is hard to juggle the migration while also firefighting REST API failures when the current REST api is unstable well before the deprecation date.

[SealSubs-Roan]

Hi @Naren1 

There must be a system issue on Shopify side. If you raised a ticket already, it would be best to wait for a response from their team.

[Naren1]

yea. but the percentage of impacted stores has been growing each day. that's what is alarming. 
impacting real-time web reports and other app functionality.
I'm surprised no one else is facing this problem 

[SealSubs-Roan]

Hi @Naren1 

There must be a system issue on Shopify side. If you raised a ticket already, it would be best to wait for a response from their team.

[arsuceno]

Same issue here. Started with one store. Now I'm experiencing it in two other more. No changes from our side.

[Naren1]

@arsuceno 
Shopify Developer Support got back to me pretty quickly with a solution that worked!

 our Developer Support team advised that it appears to be due to a database timeout, which could be related to the fact that they're 250 products. Could you try reducing the page size to 50 and see if it solves the issue? 

I reduced the page limit to request only 50 products at a time and that worked without any errors.

[Naren1 · Accepted solution]

Update:
Shopify Developer Support got back to me pretty quickly with a solution that worked!

our Developer Support team advised that it appears to be due to a database timeout, which could be related to the fact that they're 250 products. Could you try reducing the page size to 50 and see if it solves the issue? 

I reduced the page limit to request only 50 products at a time and that worked without any errors.

[arsuceno]

I changed to 50 and it works too. But I wouldn't say this is a fix but a workaround due to Shopify problems...