Does Shopify Acknowledge Our "429 Too Many" HTTP Responses?

[GuestUser69]

Background: If you run a fulfillment service and manage the inventory for it, then after products are updated, Shopify tends to request the stock levels for the fulfillment service. If you do this en masse, then Shopify will make these requests en masse. 

If you do not track the individual product inventory levels that you've recently given Shopify, then you'll be responding to each of these requests with your full set of inventory. (Note: As API users, this is our problem.)

If you run an app and have dozens or stores, then updates to all products on all stores will produce a lot of these requests.

For example: If you manage 2 locations and 25,000 products total for 50 stores, then this is a rapid burst of about 2.5 million requests for your entire store inventory, after updating all inventory on all stores.

If your uncompressed inventory feeds total around 600 KB in size, then your system is piping... Wait, what? 1.5 terabytes (!?) of highly redundant data to Shopify each time you update your stores.

Question: One simple way to alleviate this would be to simply respond to Shopify with "429 Too Many" response headers. Does anyone know if Shopify acknowledges our "429 Too Many" responses?

[GuestUser69 · Accepted solution]

We're not sure about the behavior of Shopify with regards to the HTTP Responses, so we've implemented a system by which we only give Shopify stock levels if it's been 15 minutes between their last serviced request. Then, we'll answer a request for stock with all of the stock that we have. Any request over the next 15 minutes will receive an empty response.

This solves the data throughput issue and some of the hell raised within our infrastructure; but it doesn't mitigate the numerical volume of requests that we have to handle. (Which *cough* by the way seems highly redundant and supremely needless (See: Above))

Knowing all of this, with relatively minimal effort, we've been able to minimize the amount of work that our system has to do by tracking when the last time that we provided stock was. 

For any geeks out there: We're storing this application data in a NoSQL database; and we reduce the strain on it by caching those requests in the execution environment that's directly handling the requests. The burden is still carried by the API Gateway and the request handler, but those are designed for high throughput and are relatively cheap services to run. Add to this that there's no need to connect and query an RDBMS; and those requests can be serviced with relatively minimal overhead.

And we're sending 429 Too Many responses, anyway.

[GuestUser69 · Accepted solution]

We're not sure about the behavior of Shopify with regards to the HTTP Responses, so we've implemented a system by which we only give Shopify stock levels if it's been 15 minutes between their last serviced request. Then, we'll answer a request for stock with all of the stock that we have. Any request over the next 15 minutes will receive an empty response.

This solves the data throughput issue and some of the hell raised within our infrastructure; but it doesn't mitigate the numerical volume of requests that we have to handle. (Which *cough* by the way seems highly redundant and supremely needless (See: Above))

Knowing all of this, with relatively minimal effort, we've been able to minimize the amount of work that our system has to do by tracking when the last time that we provided stock was. 

For any geeks out there: We're storing this application data in a NoSQL database; and we reduce the strain on it by caching those requests in the execution environment that's directly handling the requests. The burden is still carried by the API Gateway and the request handler, but those are designed for high throughput and are relatively cheap services to run. Add to this that there's no need to connect and query an RDBMS; and those requests can be serviced with relatively minimal overhead.

And we're sending 429 Too Many responses, anyway.

[GuestUser69 · Accepted solution]

Within minutes of a bulk operation updating the cost of 16,000 products, fulfilled by two locations; culminated 35,000 44,000 /fetch_stock.json requests. HTTP 429 Responses were sent and ignored.