API throttled when we are nowhere near the limit

[IvSegal]

NOTE: This has been a problem for years but as of this week it has elevated to the point where it is getting out of hand.

The problem: Shopify API is throttling customers even though the Shopify account is nowhere near the limit.

Example good response:

date: "Tue, 05 Mar 2024 08:53:15 GMT"
http_x_shopify_shop_api_call_limit: "1/80"
cf-cache-status: "DYNAMIC"
cf-ray: "85f8dfde2fc77b06-DEN"
x-request-id: "26f902a4-2f8c-4627-a955-2740fa7628ac-1709628794"
x-shardid: "231"

Bad response immediately following good response:

date: "Tue, 05 Mar 2024 08:53:16 GMT"
cf-cache-status: "DYNAMIC"
cf-ray: "85f8dfec8c297c1a-DEN"
server: "cloudflare"
server-timing: "cfRequestDuration;dur=29.999971, ipv6"
transfer-encoding: "chunked"
x-content-type-options: "nosniff"
x-dc: "gcp-us-central1"
x-download-options: "noopen"
x-permitted-cross-domain-policies: "none"
x-request-id: "2ac1272c-00d9-4c1a-ab4a-2bcc631114fd-1709628796"

As can see in the headers, the limit is 1/80, we should not even be close to being throttled.

[IvSegal]

(Continued: I had to his submit because the page glitched.)

The first response was 200 OK. The second was this:

extensions: {
code: "TOO_MANY_REQUESTS"
}
message: "You have sent too many requests. Try again soon."
}

We have been getting hundreds of these per day, but in this last week the incident amount has spiked unusually high.

Is there anyone at Shopify who can help resolve this issue?

(Please note - generic support such as "your network is sending too many requests will not be helpful. Our network is public cloud. We have no control over what other Shopify customers are doing. This is purely a problem on Shopify's end, either a capacity problem or miscondiguration in CloudFlare.)

[IvSegal]

Update:
No improvement. Problem started on May 5 and is ongoing.

We have a number of stores in Shopify and some for over 2 years. The issue while has always been around has been tolerable, but now is elevated to unacceptable levels.

I've reached out to support and got an automated response with a ticket number but no actual person helping with this case.

[IvSegal]

Update:
No improvement and no response from Shopify Support.

We're getting 430 errors at random even during the low peak and low traffic times.

This issue is causing customers to see errors when they attempt to checkout.

The following graph shows when the problem elevated and has not gone down since.

 

[PhilipDelorenzo]

Are you using Google App Engine?

[ZK-PB]

We are using App Engine and seeing this

[PhilipDelorenzo]

App Engine pulls from a dynamic pool of IP addresses for egress traffic. The issue is, you're sharing that IP address with ALL of Google's customers, some of which are also making calls to Shopify. It isn't you, it's the IP address. You can make changes to your GCP environment, using CloudNAT to route your traffic from a single IP in lieu of that dynamic pool. That should take care of it.

[ZK-PB]

Yeah we gave this a shot, however the issue with this solution is that being restricted to a single /28 subset means it causes issues when the app is issuing any kind of significant egress traffic (our app makes frequent calls to external APIs) and outbound requests start failing. I have yet to find a good way to make this scalable but that would definitely be the ideal solution.

[PhilipDelorenzo]

https://cloud.google.com/run/docs/configuring/static-outbound-ip

This is a good resource to follow!

[IvSegal]

@Shopify can we get some support on this please?

1. Chat support refused to help and sent us here.

2. Community forum sent us to partner support.

3. Partner support is not responding.

4. It's been 2 almost 2 weeks and the problem is still ongoing.

Ticket number 46065298

[PhilipDelorenzo]

Are you using App Engine? I responded above to another customer, I think you're having the same issue. I do not work for Shopify, just FYI.

[Jamie_D]

Hello there.

Just wanted to pop in and say we're seeing exactly this problem with our app. It is getting rather frustrating!

[ZK-PB]

Just chiming in that we are seeing the exact same thing with our Google Cloud app. Lots of unexpected 430 errors starting on 3/5 and continuing through today. This is causing significant issues for our users.

[IvSegal]

Update: Shopify has responded to my ticket and I am providing information in a back-and-forth email thread.

I will keep the group posted.

I agree with others that the static IP workaround is not a long term solution: it's not officially documented by Shopify and its behavior is undefined. Furthermore, this behavior elevated at a fairly specific time, indicating that it was likely some configuration change on the Shopify CDN side.

It should not be the burden on the partner/customer to jump through such hoops, especially if they are paying for authenticated access.

[m90studio]

Hi @IvSegal , do you have any updates to share? We are going through a similar thing and wanted to check before we submitted a support request also. 

[IvSegal]

Below is the response I got.

If we would like to all confirm - everyone who is having this issue is not running from a dedicated host? Eg you are running from serverless (google cloud) or cloud functions, or shared hosting?

Post your failing timestamps and request IDs here if possible. I will let the specialist know about this thread.

Hey Team,
 
Thanks for reaching back out.
 
I personally really appreciate your questions here, as I was able to advocate for you and the other Partners experiencing this behaviour. This led to a deeper investigation of the scenario, and I can confirm that we have a specialized team of developers doing a thorough investigation to improve this experience.
 
A piece of advice that our devs shared with me is to see if you can have a dedicated /64 egress for your service from your hosting environment as that may help.
 
You can continue reporting new instances of this behaviour, specifically with date/time/timezone and request ID and IP address.
 
All the best,
 
Miguel | Shopify Partner Support Specialist

[m90studio]

Hi @IvSegal ,

Our issue with the 430 errors resolved after we assigned the egress to a static IP in Gcloud with a VPC/NAT. Seems like it was a noisy neighbor issue as stated by our Shopify rep.