API BUG - GET PRODUCT USING PAGE AND LIMIT RANDOM FAIL - REPRODUCIBLE

Highlighted
New Member
16 0 0

PROBLEM -

Get 1 PRODUCT at a time, such as in a queue, using PAGE = X and LIMIT = 1. After Y PAGES, API will return NULL ERROR, even though 1) PAGE EXISTS and 2) CALL TIMELIMIT is NOT EXCEEDED.

 

REPRODUCE SCRIPT EXAMPLE - LET RUN UNTIL ERROR APPEARS. ERROR SHOULD APPEAR WITHIN 1 - 3 MINUTES

 

USING SHOP - debrabright.myshopify.com

$product = true;

while ( $product === true ) { //THIS WILL LOOP UNTIL ERROR FOUND

    $params = [
        "page"=>47,
	"limit"=>1
    ];

    $product = $User->ShopifyApi->apiGetProduct($params); //THIS FUNCTION CALLS API AND CHECKS CALL LIMIT AND PAUSES EVERY 1 SEC IF NEARING LIMIT

    if ( empty($product[0]['id']) ) {
        var_dump($product); //<--- WILL STOP AND SHOW ERROR 
        $product = false;
    } else {
        var_dump('Found Product'); //<--- WILL CONTINUE
        $product = true;
    }

}

OTHER NOTICE
- This bug may be affecting certain shops only - ALSO TRY ON THESE SHOPS- mediagiantdeals.myshopify.com, kit-oliver.myshopify.com
- This bug appears to be random, and appears to happen in clusters over a few seconds - such as the shopify server being overloaded
- This bug workaround may be 1) FOUND ERROR, 2) PAUSE UP TO X SECONDS, 3) SEARCH FOR PRODUCT AGAIN. The amount of time delay has been set to 1 second with 3 consecutative searches and issue still appears. Raising Time Delay to 2-4 seconds on error may allow proper workaround, but this is quite inconvenient and unexpected!

0 Likes
Shopify Staff
Shopify Staff
299 32 59

Hey Jason,

 

Would you please be able to provide me with a Request Id for one of these errors? You can find it in the response headers.

 

This may be due to either the query cost of the associated database read for your request, or something else entirely, but the request id will allow me to dig into it to find out what the problem is.

 

Cheers

0 Likes
New Member
16 0 0

Hi Zameer,

 

This is a big problem for me at the moment. I have hundreds of these request IDs. Here's 6 of them:

 

b288fc39-80cf-4e77-8140-220400615ef0
51d43186-4b67-4edc-8162-4810e6359936
ff2ad481-6db2-42d8-a940-1eeda1833cf8
92bba53b-006e-4d6c-9403-1f382cecf070
6897f058-b729-4d09-bc16-9f801bd7f66a
6787a5d7-a271-4dc2-9735-e419f9d65c49

 

Can you please let me know asap what is happening? Thank you

0 Likes
New Member
16 0 0

Hi Zameer, can you also please let me know what the issue is with this request-id?

 

395e0ac2-7407-4a72-93e8-d7c95cf3521b

 

 

0 Likes
New Member
16 0 0

And this one:

 

9119a0e0-3fba-4639-a858-a29595f17b55

 

0 Likes
Shopify Staff
Shopify Staff
299 32 59

Hey Jason,

 

The reason you're running into this issue is indeed due to the structure of your queries.

 

Is there a reason you are choosing to use a limit of 1, when you could instead be retrieving products in larger batches?

 

I see that some of your queries include page parameter values of 800 or greater. Pagination using the page parameter causes significant database strain. There are two recommendations I have that can help with the errors you are seeing.

 

1. (Easier - less effective) Increase your limit parameter value to 100 or even 250. This will allow you to make requests with much smaller page values. A safe rule of thumb would be to never let your page parameter value increase above 30. 

2. (Slightly harder - more effective) Refactor your implementation to instead make use of the since_id parameter. Paginating with the since_id parameter is significantly more efficient than its page counterpart. An implementation that would previously start with page = 0, and that would then increment page by 1 each time, would instead begin with since_id = 0. Then, instead of incrementing by 1, you would retrieve the largest product_id from the previous response, and set that as the new since_id parameter for your next request. This too can be used in conjunction with the limit parameter to retrieve larger batches of products in a single request.

 

Of the request ids you provided, the following all failed due to your page parameter being too high:

b288fc39-80cf-4e77-8140-220400615ef0

51d43186-4b67-4edc-8162-4810e6359936

92bba53b-006e-4d6c-9403-1f382cecf070

6897f058-b729-4d09-bc16-9f801bd7f66a

6787a5d7-a271-4dc2-9735-e419f9d65c49

 

And from what I can tell, these two 'DELETE' requests completed successfully and returned 200's, so I'm not sure what error you saw on your end:

395e0ac2-7407-4a72-93e8-d7c95cf3521b

9119a0e0-3fba-4639-a858-a29595f17b55

 

The following request was too far back for me to take a look: 

ff2ad481-6db2-42d8-a940-1eeda1833cf8

 

In summary, I wouldn't refer to the issue as a bug, but instead an API limitation in place to ensure the stability of the platform. Implementing since_id should allow you to accomplish whatever it is you're looking to do without any similar errors.

0 Likes
New Member
16 0 0

Zameer, You cannot possibly place the onus of 'too high page parameter' on me... This is an api issue. I've seen this issue with PAGE set to as little as 47. It should be noted that any error when using the api as documented is the error of the api, not the user. If there is a significant difference between SINCE_ID and PAGE, it should be documented.

 

I've been using  PAGE values ranging from 1 to 400000 for 2 years. No problems. This is a recent development. Can you escalate this issue to the dev team? Maybe your dev team knows why this is a recent problem and can correct it.

 

I use a queue system, I have massive queues that process the products one at a time, cycling thru stores. There are several reasons why it's designed this way. Upping the limit value would require a possible large rewrite of some core code.

 

I will try using the SINCE_ID along with LIMIT 1 and see if that works better. I don't need them to be in alphabetical order when processing anyways - so this might be an idea that works if the errors go away.

 

The other 2 IDs that you say are successful are returning a BLANK ARRAY... instead of a populated array.

0 Likes
Shopify Staff
Shopify Staff
299 32 59

Hey Jason,

 

You're right, we don't currently have documentation around the benefits of using since_id over page, but this is something that we are currently developing to share with our partner community. In the meantime, partners like yourself who are experiencing issues as a result of the page parameter, we are encouraging to make the transition to since_id sooner.

 

The reason you are recently seeing these failures is because we've been updating our load shedding implementation in the case of heavy database loads. This means requests can sometimes get stopped early instead of them inevitably hitting an SQL timeout anyway.  That being said, your exact use-case probably shouldn't have failed as it doesn't seem too harmful, but even with a limit of 1, when your page parameter begins to exceed 10,000 the queries can become costly. This is why the since_id implementation would be greatly advantageous for both reduced response times and decreased failures.

 

As for the 2 request ids with empty response body's, this is expected behaviour for HTTP 'DELETE' methods, which both those requests were. Our documentation for the requests you made confirms the same.

0 Likes
New Member
16 0 0

Thanks for being a bit more specific concerning the recent developments. An up-to-date changelog (that includes all upgrades to API) for developers would be so truly amazing. I hope you guys consider this type of initiative.

 

FYI, page=0 and page=1 are the same. You should be sure to advise users to start with page=1

 

I just created a looping script that fetches batches of products from a massive shops. SINCE_ID is significantly slower than PAGE. Also setting the limit from 100 to 250 changes the deceleration rate of the call responses (they get slower as you cycle higher). You'd have to see a video or script it yourself to fully understand... But basically what you see with both, is as the script progresses the calls take longer to process. PAGE appears to be faster, perhaps up to 4x faster (but the dropped call error rate is almost unbearable). 

 

Imagine having to create a script that counts all the product photos very fast, for some very large shops. Using SINCE_ID may literally take up to 30 minutes to count 100,000 products (it gets slower as it cycles higher up). PAGE also gets slower as it cycles up, but I can count the photos in probably 10 minutes max. Using different LIMIT vars ranging from 100-250 also affects the speed of the calls (in terms of DECELERATION or DB LOAD - not sheer bandwidth or call-rate limiting). Small shops with under a couple thousand products doesn't appear to be any real issue.

 

Having said that, I will still try to use SINCE_ID with LIMIT 1 to cycle and see if it stops generating the same dropped-call errors as PAGE. Fingers crossed!

 

As for the 2 other IDs... I'm like 99% certain they are not generated by DELETE requests. There must be a problem with the IDs, perhaps I'm logging the prior call's ID, as the request is failing and not updating the request-id properly. I'll have to double check all this and get back to you with more specifics. 

0 Likes
New Member
16 0 0

These ID should correspond to an INTERNAL SERVER ERROR: 

e6fbdd49-7119-417e-8b71-98aec9c08955

16a06f3b-b703-4a84-9574-355c47548a68

bdf0c457-71d6-4173-805b-54200d404dfb

fadeb8c0-991e-4ba3-bc85-78e2eecd8d97

bb419caf-3902-4544-8348-e9ccdff03ea5

dee803b2-1e3b-40e1-a6a0-e605d014f3a3

 

Is that the case? Do you know these errors occur?

 

Can you also look at all the Errors related to:

ombrestores.myshopify.com - specifically PID - 740245078070 - This store specifically has been generating lots of errors (while at the same time the other stores have not).

data-city-dk.myshopify.com - PID - 19551196043351

 

 

0 Likes