Api pagination losing page_info

[MagnumFonseca]

I'm using the gem shopify_api v 9.5.1 and the REST API version 2021-10 to import a user order historical data, and It has over 1 million orders. As I use page_info to get to the next page, at some point I start to get HTTP status 500.

page_info="eyJkaXJlY3Rpb24iOiJuZXh0Iiwic3..."

ShopifyAPI::Order.find(:all, params: {limit: 50, page_info: page_info})

And after the error, even if I try to retrieve only one order I do still get HTTP 500.

Is Shopify losing the page_info reference? Is there anything I could do better to import this huge amount of data?

[Alan]

Hey @MagnumFonseca - this is a great question, thanks for reaching out. Your initial insight is right, for this amount of orders a 500 error could be expected, but with the limit and pagination settings you've set up, it still is odd that it's only returning the 500 error after you hit a certain page. 

Would you be able to share an X-Request-ID for a call where you received a 500 error? Sometimes there can be further information in our logs that can help us investigate further, which I'd be able to pull using that X-Request ID. 

Let us know if we can clarify anything on our end and hope to hear from you soon. Happy to see if we can figure out a way to import this data more easily for you. 

Cheers!

[MagnumFonseca]

Hey, @Alan I made a call using the page_info I had and this is what I got. 

X-Request-Id: - 6bcbdc09-338e-4f3e-91da-66e71dbd6a8a

[Alan]

Hey @MagnumFonseca - thanks for providing that Request ID. I was able to do a bit more testing on my end, and my thinking is that the query may be a little too complex, especially when you are reaching higher page numbers like you mentioned in your initial message. I was able to surface a bit more context for the 500 error though, and it appears it looks like the query itself timed out which can happen with large data sets like this one, even if they're paginated. 

I do have a suggestion for a workaround though if you wanted to use our GraphQL API to capture all of the orders at once. If you already have the right access scopes for the store, you'd just need to run a BulkOperation query using our GraphQL schema to generate a downloadable JSONL file with the relevant order information, which you could then use to import. 

I noticed the initial request you made which is attached to that Request ID was a GET request - so I wanted to make sure you first wanted to obtain the data and then import it somewhere else? Let me know if I'm understanding correctly - happy to keep digging into this with you. 

Hope this helps!