Can't retry when hit 429 error and some empty body returned- inventory_levels endpoint

Highlighted
New Member
3 0 0

Hi guys, 

I try to fetch the inventory_levels using REST API and meet some error here. 
When I met 429 error, tried to sleep for 60 seconds and more but can't retry successful. After some retries I got empty body with status 200, have no reason why?

Please ignore the inventory_ids, location_ids and rate limit, I know that the rate limit is just 2 requests/second. What I use here just for testing purposes. 
The empty content response message is used when I check the status is 200 but the body is empty.
Full log: 


2019-08-14 12:42:29.679 +0700 [INFO] (0015:task-0000): >>> /admin/api/2019-07/inventory_levels.json?limit=250&inventory_item_ids=bn,j,trw,w,dg,jh,fg,fv,zx,wsd,qwd,da,sa,t3,er,12,35,346,46,325,42,1,31,123,432,43,36,45,65,56&location_ids=2,34,23,42,523&updated_at_min=2017-06-29T00:28:51+00:00
2019-08-14 12:42:30.207 +0700 [INFO] (0015:task-0000): X-Shopify-Shop-Api-Call-Limit aaaa 39/40
2019-08-14 12:42:30.210 +0700 [INFO] (0015:task-0000): >>> /admin/api/2019-07/inventory_levels.json?limit=250&inventory_item_ids=bn,j,trw,w,dg,jh,fg,fv,zx,wsd,qwd,da,sa,t3,er,12,35,346,46,325,42,1,31,123,432,43,36,45,65,56&location_ids=46,72,dfgdf,wer,waf&updated_at_min=2017-06-29T00:28:51+00:00
2019-08-14 12:42:30.638 +0700 [INFO] (0015:task-0000): X-Shopify-Shop-Api-Call-Limit aaaa null
2019-08-14 12:42:30.638 +0700 [INFO] (0015:task-0000): Retry after 2.0
2019-08-14 12:42:30.638 +0700 [WARN] (0015:task-0000): Retrying 1/7 after 60 seconds. Message: Response not 2xx: 429 Too Many Requests {"errors":"Exceeded 2 calls per second for api client. Reduce request rates to resume uninterrupted service."}
2019-08-14 12:43:30.644 +0700 [INFO] (0015:task-0000): >>> /admin/api/2019-07/inventory_levels.json?limit=250&inventory_item_ids=bn,j,trw,w,dg,jh,fg,fv,zx,wsd,qwd,da,sa,t3,er,12,35,346,46,325,42,1,31,123,432,43,36,45,65,56&location_ids=46,72,dfgdf,wer,waf&updated_at_min=2017-06-29T00:28:51+00:00
2019-08-14 12:43:30.645 +0700 [INFO] (0015:task-0000): X-Shopify-Shop-Api-Call-Limit aaaa null
2019-08-14 12:43:30.645 +0700 [INFO] (0015:task-0000): Retry after 2.0
2019-08-14 12:43:30.647 +0700 [WARN] (0015:task-0000): Retrying 2/7 after 60 seconds. Message: Response not 2xx: 429 Too Many Requests 
2019-08-14 12:44:30.656 +0700 [INFO] (0015:task-0000): >>> /admin/api/2019-07/inventory_levels.json?limit=250&inventory_item_ids=bn,j,trw,w,dg,jh,fg,fv,zx,wsd,qwd,da,sa,t3,er,12,35,346,46,325,42,1,31,123,432,43,36,45,65,56&location_ids=46,72,dfgdf,wer,waf&updated_at_min=2017-06-29T00:28:51+00:00
2019-08-14 12:44:30.656 +0700 [INFO] (0015:task-0000): X-Shopify-Shop-Api-Call-Limit aaaa 1/40
2019-08-14 12:44:30.657 +0700 [INFO] (0015:task-0000): Retry after null
2019-08-14 12:44:30.661 +0700 [WARN] (0015:task-0000): Retrying 3/7 after 60 seconds. Message: Empty response content
2019-08-14 12:45:30.670 +0700 [INFO] (0015:task-0000): >>> /admin/api/2019-07/inventory_levels.json?limit=250&inventory_item_ids=bn,j,trw,w,dg,jh,fg,fv,zx,wsd,qwd,da,sa,t3,er,12,35,346,46,325,42,1,31,123,432,43,36,45,65,56&location_ids=46,72,dfgdf,wer,waf&updated_at_min=2017-06-29T00:28:51+00:00
2019-08-14 12:45:30.670 +0700 [INFO] (0015:task-0000): X-Shopify-Shop-Api-Call-Limit aaaa 1/40
2019-08-14 12:45:30.671 +0700 [INFO] (0015:task-0000): Retry after null
2019-08-14 12:45:30.671 +0700 [WARN] (0015:task-0000): Retrying 4/7 after 60 seconds. Message: Empty response content
2019-08-14 12:46:30.674 +0700 [INFO] (0015:task-0000): >>> /admin/api/2019-07/inventory_levels.json?limit=250&inventory_item_ids=bn,j,trw,w,dg,jh,fg,fv,zx,wsd,qwd,da,sa,t3,er,12,35,346,46,325,42,1,31,123,432,43,36,45,65,56&location_ids=46,72,dfgdf,wer,waf&updated_at_min=2017-06-29T00:28:51+00:00
2019-08-14 12:46:30.674 +0700 [INFO] (0015:task-0000): X-Shopify-Shop-Api-Call-Limit aaaa 1/40
2019-08-14 12:46:30.674 +0700 [INFO] (0015:task-0000): Retry after null
2019-08-14 12:46:30.674 +0700 [WARN] (0015:task-0000): Retrying 5/7 after 60 seconds. Message: Empty response content
2019-08-14 12:47:30.681 +0700 [INFO] (0015:task-0000): >>> /admin/api/2019-07/inventory_levels.json?limit=250&inventory_item_ids=bn,j,trw,w,dg,jh,fg,fv,zx,wsd,qwd,da,sa,t3,er,12,35,346,46,325,42,1,31,123,432,43,36,45,65,56&location_ids=46,72,dfgdf,wer,waf&updated_at_min=2017-06-29T00:28:51+00:00
2019-08-14 12:47:30.681 +0700 [INFO] (0015:task-0000): X-Shopify-Shop-Api-Call-Limit aaaa 1/40
2019-08-14 12:47:30.681 +0700 [INFO] (0015:task-0000): Retry after null
2019-08-14 12:47:30.682 +0700 [WARN] (0015:task-0000): Retrying 6/7 after 60 seconds. Message: Empty response content
2019-08-14 12:48:30.691 +0700 [INFO] (0015:task-0000): >>> /admin/api/2019-07/inventory_levels.json?limit=250&inventory_item_ids=bn,j,trw,w,dg,jh,fg,fv,zx,wsd,qwd,da,sa,t3,er,12,35,346,46,325,42,1,31,123,432,43,36,45,65,56&location_ids=46,72,dfgdf,wer,waf&updated_at_min=2017-06-29T00:28:51+00:00
2019-08-14 12:48:30.691 +0700 [INFO] (0015:task-0000): X-Shopify-Shop-Api-Call-Limit aaaa 1/40
2019-08-14 12:48:30.691 +0700 [INFO] (0015:task-0000): Retry after null
2019-08-14 12:48:30.692 +0700 [WARN] (0015:task-0000): Retrying 7/7 after 60 seconds. Message: Empty response content
2019-08-14 12:49:30.697 +0700 [INFO] (0015:task-0000): >>> /admin/api/2019-07/inventory_levels.json?limit=250&inventory_item_ids=bn,j,trw,w,dg,jh,fg,fv,zx,wsd,qwd,da,sa,t3,er,12,35,346,46,325,42,1,31,123,432,43,36,45,65,56&location_ids=46,72,dfgdf,wer,waf&updated_at_min=2017-06-29T00:28:51+00:00
2019-08-14 12:49:30.697 +0700 [INFO] (0015:task-0000): X-Shopify-Shop-Api-Call-Limit aaaa 1/40
2019-08-14 12:49:30.697 +0700 [INFO] (0015:task-0000): Retry after null
2019-08-14 12:49:30.701 +0700 [INFO] (0001:transaction): {done:  1 / 1, running: 0}

Error: org.eclipse.jetty.client.HttpResponseException: Response not 2xx: 429 Too Many Requests {"errors":"Exceeded 2 calls per second for api client. Reduce request rates to resume uninterrupted service."}
0 Likes
Shopify Staff
Shopify Staff
1118 78 156

Hello @hieud , 

 

Would you be able to let me know the shop that this occurred on? I could take a look at our logs and see if anything stands out. 

1 Like
New Member
3 0 0

Hi @Josh

This shop https://mountbaker.myshopify.com.
Just for testing purpose. 

Thanks.

0 Likes
Shopify Staff
Shopify Staff
1118 78 156

Hey @hieud , 

 

I was looking in our logs at one of the 429s that occurred and it looks legitimate, it was on 8/14 at 1:08:53 EDT.

 

There was an API call to /admin/inventory_items.json at 1:08:52, and then the next second was a burst of two calls at 1:08:53 to the same endpoint. 

 

I would suggest trying to keep your API call bucket at 37-38/40 and trying to make sure that a burst like this can't happen. Given that ~1% of your calls are seeing this out of thousands, it does seem like you're doing ok at managing your call limit as it is - this may just be some edge cases that are occurring.

0 Likes
New Member
3 0 0

Hi Josh, 

Thank you for your reply!!

Hmmm, So I should keep the request not to reach the rate limit? And I also try with `inventory_item` endpoint same problem happened. 
Does this mean the retry is in some cases fail? 

0 Likes
Shopify Staff
Shopify Staff
1118 78 156

Hey again, 

 

I'm not sure if I'm fully understanding, but I was just suggesting to not stay at the very edge of your 40 API call limit at all times if you wanted to try to avoid any429 errors. 

 

I can't really tell in our logs which of your API calls are retries, but I could see that when you encountered a 429 error you had simply made your calls too fast. Ultimately the rate that you are seeing 429 errors isn't actually that bad considering the thousands of API calls and only a handful of 429 errors, so you would likely be fine continuing as you are now as well. But if you never want to see a 429 error, try to stay at 37-38 of your 40 API call limit, and try to avoid making 2 calls in the same second. 

0 Likes