(Continued: I had to his submit because the page glitched.)
The first response was 200 OK. The second was this:
extensions: {
code: "TOO_MANY_REQUESTS"
}
message: "You have sent too many requests. Try again soon."
}
We have been getting hundreds of these per day, but in this last week the incident amount has spiked unusually high.
Is there anyone at Shopify who can help resolve this issue?
(Please note - generic support such as "your network is sending too many requests will not be helpful. Our network is public cloud. We have no control over what other Shopify customers are doing. This is purely a problem on Shopify’s end, either a capacity problem or miscondiguration in CloudFlare.)
We have a number of stores in Shopify and some for over 2 years. The issue while has always been around has been tolerable, but now is elevated to unacceptable levels.
I’ve reached out to support and got an automated response with a ticket number but no actual person helping with this case.
Just chiming in that we are seeing the exact same thing with our Google Cloud app. Lots of unexpected 430 errors starting on 3/5 and continuing through today. This is causing significant issues for our users.
App Engine pulls from a dynamic pool of IP addresses for egress traffic. The issue is, you’re sharing that IP address with ALL of Google’s customers, some of which are also making calls to Shopify. It isn’t you, it’s the IP address. You can make changes to your GCP environment, using CloudNAT to route your traffic from a single IP in lieu of that dynamic pool. That should take care of it.
Yeah we gave this a shot, however the issue with this solution is that being restricted to a single /28 subset means it causes issues when the app is issuing any kind of significant egress traffic (our app makes frequent calls to external APIs) and outbound requests start failing. I have yet to find a good way to make this scalable but that would definitely be the ideal solution.
Update: Shopify has responded to my ticket and I am providing information in a back-and-forth email thread.
I will keep the group posted.
I agree with others that the static IP workaround is not a long term solution: it’s not officially documented by Shopify and its behavior is undefined. Furthermore, this behavior elevated at a fairly specific time, indicating that it was likely some configuration change on the Shopify CDN side.
It should not be the burden on the partner/customer to jump through such hoops, especially if they are paying for authenticated access.
If we would like to all confirm - everyone who is having this issue is not running from a dedicated host? Eg you are running from serverless (google cloud) or cloud functions, or shared hosting?
Post your failing timestamps and request IDs here if possible. I will let the specialist know about this thread.
Hey Team,
Thanks for reaching back out.
I personally really appreciate your questions here, as I was able to advocate for you and the other Partners experiencing this behaviour. This led to a deeper investigation of the scenario, and I can confirm that we have a specialized team of developers doing a thorough investigation to improve this experience.
A piece of advice that our devs shared with me is to see if you can have a dedicated /64 egress for your service from your hosting environment as that may help.
You can continue reporting new instances of this behaviour, specifically with date/time/timezone and request ID and IP address.
Our issue with the 430 errors resolved after we assigned the egress to a static IP in Gcloud with a VPC/NAT. Seems like it was a noisy neighbor issue as stated by our Shopify rep.
