Persistent 429 errors, even with no activity

[stickfigure]

I’m getting the 429 “Exceeded 2 calls per second for api client. Reduce request rates to resume uninterrupted service” error with 100% persistence, even with no activity on the store or app. I waited an hour.

My store is a development store with no other apps installed. My app works fine against other stores. I’m making a simple request to:

GET https://orbital-test-harness.myshopify.com/admin/api/2020-01/fulfillment_services.json

I’m passing the X-Shopify-Access-Token header. The response doesn’t seem to include the X-Shopify-Shop-Api-Call-Limit header I read about in the documentation. Actually no responses seem to include that header, working or not. Was it removed? Retry-After always says 2.0. Here’s an example response:

HTTP/1.1 429 Too Many Requests
Date: Wed, 29 Jan 2020 00:35:43 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: <deleted>
X-Sorting-Hat-PodId: 135
X-Sorting-Hat-ShopId: 30008737928
Referrer-Policy: origin-when-cross-origin
X-Frame-Options: DENY
X-ShopId: 30008737928
X-ShardId: 135
X-Stats-UserId: 
X-Stats-ApiClientId: 3302505
X-Stats-ApiPermissionId: 206955577480
Retry-After: 2.0
Strict-Transport-Security: max-age=7889238
X-Request-Id: 59af454a-6a4a-49fc-84e8-ae967e86344c
X-Shopify-Stage: production
Content-Security-Policy: default-src 'self' data: blob: 'unsafe-inline' 'unsafe-eval' https://* shopify-pos://*; block-all-mixed-content; child-src 'self' https://* shopify-pos://*; connect-src 'self' wss://* https://*; frame-ancestors 'none'; img-src 'self' data: blob: https:; script-src https://cdn.shopify.com https://cdn.shopify.cn https://checkout.shopifycs.com https://js-agent.newrelic.com https://bam.nr-data.net https://api.stripe.com https://mpsnare.iesnare.com https://appcenter.intuit.com https://www.paypal.com https://js.braintreegateway.com https://c.paypal.com https://maps.googleapis.com https://www.google-analytics.com https://v.shopify.com https://widget.intercom.io https://js.intercomcdn.com 'self' 'unsafe-inline' 'unsafe-eval'; upgrade-insecure-requests; report-uri /csp-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Ffulfillment_platform%2Ffulfillment_services&source%5Bsection%5D=admin_api&source%5Buuid%5D=59af454a-6a4a-49fc-84e8-ae967e86344c
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Permitted-Cross-Domain-Policies: none
X-XSS-Protection: 1; mode=block; report=/xss-report?source%5Baction%5D=index&source%5Bapp%5D=Shopify&source%5Bcontroller%5D=admin%2Ffulfillment_platform%2Ffulfillment_services&source%5Bsection%5D=admin_api&source%5Buuid%5D=59af454a-6a4a-49fc-84e8-ae967e86344c
X-Dc: gcp-us-central1,gcp-us-central1
CF-Cache-Status: DYNAMIC
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Alt-Svc: h3-24=":443"; ma=86400, h3-23=":443"; ma=86400
Server: cloudflare
CF-RAY: 55c720b29ea99340-SJC

{
  "errors": "Exceeded 2 calls per second for api client. Reduce request rates to resume uninterrupted service."
}

Any idea what the issue might be?

Thanks in advance.

[chrisbutcher]

Hello stickfigure,

I had a look at API call activity from your app to that shop, and I can confirm you made about 106 thousand API calls for fulfillment orders in a roughly 10 hour period that were throttled and got a 429 response.

Over the same time period, however I see 66 thousand API calls for fulfillment orders which succeeded, with a 200 response.

The last of the successful, 200 response API calls had the Shopify Request ID: b6b65e33-c3ec-4e01-8599-9d6cf3d276a3. Can you confirm if you were in fact seeing 200 OK responses during this time?

[stickfigure]

Weird! I have no way of explaining those numbers. Our app isn’t live in production. I checked all three places they could possibly have been issued from (two dev machines and CI) and none are responsible.

Is it possible that callbacks from Shopify are included in that count? For a test fulfillmentservice we provided a bogus callback… which in retrospect could mean that Shopify is trying over and over to notify us about fulfillments, with accumulated vigor as we add more test fulfillments. I’ll try blackholing the callbacks.

Is there a way I can get access to that graph? I couldn’t find it anywhere in the admin consoles.

I switched to an alternate store yesterday and got my app working again. This morning the original store is working again.

[chrisbutcher]

Hello stickfigure,

I checked with the team behind Fulfillment services and they found that Shopify is not rapidly calling any of your callbacks at the moment, but at one point we did attempt to send you about a thousand fulfillment requests between 2020-01-28 21:21:00 UTC and 2020-01-29 19:07:16 UTC. They then think that you updated the callback, after which our systems stopped calling rapidly.

It turns out this is a known issue with a fix in the works. Apologies for the inconvenience!

We appreciate that you’re using this new API, and would love to hear any other feedback you have!

Regards,

Chris

[stickfigure]

It’s working well now… and my test harness is blackholing callbacks (200 OK for everything) now so that should no longer be a problem.

I do have one (unrelated) piece of feedback on FulfillmentOrders:

FulfillmentOrder has a ‘destination’ property which looks like Order.shipping_address, but is missing ‘province_code’ (‘province’ is present, but not what I need for shipping!). So I have to get the shipping address out of the Order anyway. Shouldn’t FulfillmentOrder.destination be the same as Order.shipping_address? And what’s the FulfillmentOrder.destination.id field useful for?

Thanks!

[dianavidal]

Hi, @stickfigure can we ask your kind help , to us wth this topic, it´s related. I hope !

https://community.shopify.com/post/1760260

Regards!