API rate limit with 429 status but with bucket not empty

Solved
Highlighted
Shopify Partner
5 0 3

I,

In a private App 

I do the following API REST call :
For each order retrieve from magento do :
1 : search customer by email
1.1 : if it do not exist
1.1.1 : create him/her
2 :  get it's orders
3 : check if the magento order is in this list 
3.1 : if not
3.1.1 : create it
3.1.1 : close it
4 : loop to next order
 

Each api call took more than 500ms, this code is executed sequentialy (to find the problem) 
Ater nearly 30 call done in 30s, I have a call returning a 429 status but with an header saying that I have still 39 calls available in my bucket (1/40) and with the header "retry after" empty.

I try on two test store with the same result.

 

I already use the same kind of code (with different call patern) to create and update more than 100K orders and everything was OK.

I don't understand, It is like this partern of call consum more call than the real one done.

 

Thanks in advance for any help or idea.

 

 

image.png

0 Likes
Highlighted
Shopify Partner
1777 211 370

Which version of REST API are you using?

Sergiu Svinarciuc | CTO @ visely.io
- Was my reply helpful? Click Like to let me know!
- Was your question answered? Mark it as an Accepted Solution!
- To learn more about the awesome stuff we do head over to visely.io or our blog
0 Likes
Highlighted
Shopify Expert
9925 103 1764

Is this on an active shop or a dev (affiliate) one? It would not be unexpected to see different throttling and resource limits on those. If possible, slow down your calls.

★ Winning Partner of the Build a Business competition. ★ http://freakdesign.com.au
0 Likes
Highlighted
Shopify Partner
5 0 3
The 2019-10 version on a dev store.
0 Likes
Highlighted
Shopify Partner
5 0 3

Hello,

The problem can be reproduce with the curl script I've added below.

The script loops 20 times and creates a new user and a new order.

The header.txt generated by the script shows that we do not exceed the maximal call rate : 

HTTP_X_SHOPIFY_SHOP_API_CALL_LIMIT: 1/40
X-Shopify-Shop-Api-Call-Limit: 1/40
X-Shopify-API-Version: 2019-10

 

The output of the script :

https://

retrieve location
....
call createUser USER_5
userid=2685966221384
creating order
curl: (22) The requested URL returned error: 429 Too Many Requests

 

#!/bin/bash

if [ $# -eq 0 ] 
then
  echo missing path to url : appShopifyUser:appShopifyPass@store.myshopify.com
  exit 1
fi

SHOP=$1
URL=https://${SHOP}/admin/api/2019-10

echo $URL


echo retrieve location
LOCATION_ID=$(curl -s -S ${URL}/locations.json | jq '.locations[0].id')

echo location=$LOCATION_ID

for i in {1..20} 
do
USER=USER_$i
# echo call getuser $USER
# curl --fail -s -S $URL/customers/search.json?query=email%3A%22${USER}%40gmail.com%22
echo call createUser $USER
USER_ID=$(curl -D ./header.txt --fail -s -S -X POST -d '{"customer":{"email":"'$USER'@gmail.com"}}' -H "Content-Type: application/json" $URL/customers.json  | jq '.customer.id')
echo userid=$USER_ID
echo creating order
ORDER_ID=$(curl -D ./header.txt --fail -s -S -X POST -d '{"order":{"billing_address":{"address1":"AZEAEZAEZ","city":" PARIS","country_code":"FR","country_name":"France","first_name":"'$USER'","last_
name":"any","phone":"0633322445","zip":"75000"},"currency":"EUR","customer":{"id":'$USER_ID'},"financial_status":"paid","fulfillments":[{"created_at":"2019-08-20T07:44:32+02:00[Europe/Paris]","locat
ion_id":'$LOCATION_ID',"status":"success","shipment_status":"delivered"}],"fulfillment_status":"fulfilled","line_items":[{"price":"12.0833","quantity":1,"title":"Mascara Eye Catching Extrême Volume"
,"name":"Mascara Eye Catching Extrême Volume","tax_lines":[{"title":"TVA","price":"2.4167","rate":0.2}],"total_discount":"0.0000"}],"processed_at":"2019-08-20T07:44:32+02:00[Europe/Paris]","shipping
_address":{"address1":"ZERZTV","city":" PARIS","country_code":"FR","country_name":"France","first_name":"'$USER'","last_name":"any","phone":"0644422334","zip":"75000"},"source_name":"historique","to
tal_price":"19.5","transactions":[{"status":"success","amount":19.5,"kind":"sale"}],"metafields":[{"key":"GCO","namespace":"logistics","value":"ADS_FAKE.xml","value_type":"string"}]}}' -H "Content-T
ype: application/json" $URL/orders.json | jq '.order.id')
#echo closing $ORDER_ID
#curl --fail -s -S -X POST -d'{}' -H "Content-Type: application/json" $URL/orders/${ORDER_ID}/close.json > /dev/null
echo "done"
done

Thanks,

Marc Guiot

 

0 Likes
Highlighted
Shopify Partner
1777 211 370

This is an accepted solution.

As @Jason pointed out, if that is a dev store Shopify might not necessarily favor the limits and can engage some extra throttling even if the bucket is not exhausted. 

Sergiu Svinarciuc | CTO @ visely.io
- Was my reply helpful? Click Like to let me know!
- Was your question answered? Mark it as an Accepted Solution!
- To learn more about the awesome stuff we do head over to visely.io or our blog
0 Likes
Highlighted
Shopify Partner
5 0 3

Thank you very much,

I launch the transfert on the live store and everything went well.

I neither read nor thought that shopify could have a different behaviour between prod and dev store!!

 

Thank again.

 

2 Likes
Highlighted
New Member
1 0 1

that's a real shame. . I wrote a migration script to move from Magento to Shopify but I want to test it first before pushing this into a live store..     guess I'll have to throttle it way back.  2 seconds per request wasn't slow enough..       extra sad is I wrote a retry function that uses the retry after header but its always empty so I was retrying even then the 2 seconds..   oh well..      thanks for this info though. . It saving me from spending to night trying to figure out of my code was broken. .

 

Gary

1 Like
Highlighted
Shopify Partner
10 0 2

Thanks, this helped me too. I was running into the same problem.

 

I guess I understand why they would want to limit development stores to avoid people hammering their servers, but it makes it very difficult to reliably test our applications, especially if the retry after header is empty. If we could at least get that header back we could build something that was flexible across both environments.

0 Likes
Highlighted
Shopify Partner
10 0 2

In case anyone else runs into this, I found the following statement in the docs:

 

Note

If you're using this endpoint with a trial or Partner development store, then you can create no more than 5 new orders per minute.

So those are our limits for orders. 

0 Likes