Surge in 430 status code inspite of using Buy Identity header

[varun_agarwal]

I build custom apps for 100+ merchants. A lot of this functionality is through server to server (s2s) APIs using admin rest and storefront graphql. For the last 2 weeks I am getting a lot of 430 status codes from Shopify for both of these endpoints. Requests from my server will all come from a common pool of IP addresses so we use a delegate access token and Shopify-Storefront-Buyer-IP header in our outgoing requests. As per docs this tells shopify the originating IP which should help in reducing 429 and 430 status codes. But this has not helped at all.

Requests are CRUD operations to admin endpoint, queries and mutations to storefront graphql endpoints. Has anyone else faced this issue, what is the solution for this? We, and our Shopify plus merchants are losing a lot of money on a daily basis due to this.

[Liam]

Hi Varun_agarwal,

Based on this description, it seems like you're hitting the rate limits set by Shopify's API, which results in 430 status codes - Shopify Security Rejection. This typically occurs when Shopify deems your requests as potentially malicious.

You can tackle this issue in several ways:

1. Optimize your requests: Ensure your requests only fetch the data you need. Reducing the complexity of your requests can help you stay within the rate limits.

2. Use Caching: If your app uses data frequently, consider caching it to reduce the number of requests to the Shopify API.

3. Proper Error Handling: Implement error handling in your code to gracefully handle rate limit errors. If you keep making requests without addressing these errors, your app may not recover smoothly.

4. Queuing and Throttling: Implement a queuing system where you can line up the API calls and process them at a rate that is within the Shopify API rate limits. This distribution can help avoid hitting the rate limits.

5. Respecting the Retry-After Header: When Shopify returns a 429 or 430 status code, it includes a `Retry-After` header indicating how many seconds to wait before making another request. Respect this delay to avoid further rejections.

Hope this helps!

[varun_agarwal]

Hi Liam, thanks for the reply. Can you address Jjwarren's answer below? The Aug release which effectively removes rate limit on a single storefront token seems to have tightened up the anti spam/ botting software being employed. The anti-botting measures have even been mentioned in his link. Partners like us who have several Shopify+ merchants using our server side rendered storefronts are not happy with getting our API requests randomly blocked. We can try and optimize our code but is there any other suggestions you can make?

[bhr]

Thanks @Liam ! We've been getting more error status codes as well. To some extent, it's caused by this issue in the Shopify Admin Python SDK. Our workaround is to fetch only a subset of orders with each API call. 

[IvSegal]

There is no Retry-After Header when the response is 430 and the system (Cloudflare worker?) that throws 430 is not honoring the Shopify's own  documented allowed limit. In fact it can randomly throw a 430 when the client had not sent any traffic in a while at all.

This implies that the system that is throwing the 430 is doing so not on a per-api-key basis but based on some global heuristic.

The problem is that lately there is so much 430 throttling that it makes Shopify close to useless as a SaaS, even with all of the caching and proper error handling implemented.

(For example, the fact that createDraftOrder fails with 430 on first try it a major customer-facing problem, there's nothing to cache at this point).

Shopify, please fix this. I am happy to work with you if you need diagnostics.

[Filljoy]

@IvSegal this is the response I got back from Partner Support: 

"I've received a reply from our team regarding your recent question. In general, the primary reason for partners' apps experiencing intermittent 430 errors that they have observed is the use of a shared IP on the app's hosting service. Furthermore, they mentioned that the client IP address is among various factors that the platform's security system considers. Therefore, if you are utilizing a shared IP address, it is plausible to encounter 430 errors due to high volumes of requests originating from the same IP, potentially not directly from your specific app."

We were being plagued by this, and fixed this (at least for now) using a VPC Connector for our Google Cloud Functions: https://medium.com/@rtmacdonald/rate-limiting-and-cloud-run-dynamic-ip-address-pool-1a70b2f08077 

Note that we'd done this once already before, and then got hit with 430s again. But we recreated another VPC connector and for the past week have gotten zero 430 errors. Maybe Cloudfare has an equivalent product.

[IvSegal]

Thank you, this is helpful, but also highlights the problem:

Shopify is supposed to enforce rate limits on a per-store basis (read: per API credential) when using private API keys, not on a per-IP basis:

This is according to their own docs: https://shopify.dev/docs/api/usage/rate-limits

It should not be the customer's problem that they happen to have a noisy neighbor when running on the public cloud. If the customer is using private API key, it should not matter what IP its coming from. Any sane SaaS typically treats private API traffic with a separate predictable set of rules. In the case of Shopify their rules are well documented, but their actual configuration does not match the documentation.

It's also counter-intuitive that having a static IP is a solution, and especially concerning that this solution can break at any time and require some fiddling around to change a static IP.

What is likely happening is that Shopify is using same anti-bot configuration for private APIs that they use for public traffic. This cases a lot of unnecessary pain for partners and customers.

Shopify needs to fix this.

[IvSegal]

Ok I've implemented the static IP workaround (fingers crossed). Will post an update.

ATTN: Shopify Security team, Shopify Devs, Managers, if you are reading. Some feedback:

1. As mentioned - throttling private authenticated, paid-for API (such as Shopify Admin REST and GraphQL API) on some global IP-based level is harmful to customers that are running on serverless public cloud, and does nothing against DDoS attacks and botnets. It also does not match Shopify's documented limits and behaves arbitrary.

Please either honor your own limits or update your documentation to match with reality. https://shopify.dev/docs/api/usage/rate-limits

2. Customer support was a waste of time as they straight up refused to help and did not get us in touch with anyone who would help. There needs to be an escalation path.

3. Partner support did not respond and still has not responded after a week of waiting.

4. While this issue is not new, it accelerated specifically around earlier this March and has been ongoing.

[jjwarren]

You're not alone - we've also been experiencing the same issue since 01/08/2023. This appears to have coincided with Shopify "removing rate limits" from their Storefront API (https://www.shopify.com/editions/summer2023?product=storefront-api). The documentation now states "There are no rate limits applied on the number of requests that can be made to the API." however it seems in reality the new "bot protection" applies a much stricter rate limit than before.

This has practically made their Storefront API un-usable for large-scale server-side rendered headless storefronts, which inherently make a reasonable number of requests from the server-side.

We've been in contact with Shopify Plus support and unfortunately they're not aware of a solution either at this time, besides the Shopify-Storefront-Buyer-IP header which hasn't helped us much. Use the API less appears to be the primary solution, admittedly not ideal.

[varun_agarwal]

Thanks for bringing this up. If most of the API calls to the SSR storefront across all our merchants is for managing cart and checkout sessions, this becomes very difficult to optimize.

Shopify support asked for several request IDs but haven't gotten back either.

The whole point for adding a Buyer IP header is to allow SSR storefronts to handle larger traffic. We were (sadly) doing significantly better before they made this change on 1st Aug. While caching catalogs and some APIs are possible, we cannot cache mutations to create a storefront checkout or cart or applying coupons. Any suggestions on how you are managing that?

I was looking at frequent IP rotation as one alternative for my server clusters, have you tried something like that?

[Filljoy]

@varun_agarwal did you ever figure out a solution? We are using Google Cloud Functions on the server to make our API calls (mostly REST) and as of this week are seeing a spike in 430 errors. We tried using a VPC (our own IP address pool), still no. 

@Liam we are not seeing any retry-again header in our 430 responses. Can you please confirm the 430 also sends this?

[christianOrgs]

UPVOTE ! Were also facing this issue with our GCP-functions middleware for custom warehouse forwarding and order handling. So we can't guarantee our merchants to handle this properly ATM. We working since two days on a solution.

[schnoberts]

We have also been seeing the same issue recently. Our app is low volume, issuing less than a hundred requests a day with no more than 6 requests concurrent. We have a static unshared IP and there’s nothing else using this IP. I wonder if it’s not IP based but perhaps network route based.