Rate Limiting Shopify Webhook Events on imports

[Andrew93]

Hello,

I have several application setup that are effectively using webhooks to sync with users stores.  With most webhooks we make one api request to the store to fetch the most recent information.  This works great the majority of the time except when the shop performs a large batch action such as an import.

When the shop performs a large batch action such as an import we effectively get a ddos attack from shopify webhooks.  In addition to the large number of requests we also are unable to take action if there are a large number of webhooks because when over 500 webhooks are called from one shop we run into rate limiting errors(429) when attempting to fetch the most recent information for the webhook object.

Does anyone have any ideas on better ways to deal with batch updates with webhooks.  Is there a way to get notified that the shop is performing a batch update in a single webhook?

Thank You,

-Andrew

[Justin_Stern1]

I don't believe there's any way of consolidating those webhooks, but one thing you should definitely be doing if you aren't already, is processing the webhook in a background worker.  This is the recommended practice by the Shopify docs, and reduces your webhook handler to almost nothing: just receive the webhook and schedule a job to deal with it later.  This sort of setup should allow you to handle a huge number of webhooks without any real issue.  The other things it avoids is the Shopify servers automatically cutting off your servers from receiving webhooks, which can happen if you don't respond quickly enough to the webhook request.

Course how you go about doing this all depends on what language/platform you built on, we use heroku/rails and there's a great open source project called Sidekiq http://sidekiq.org/ that we use for our background processing that I'd recommend looking into if you share a similar tech stack

[Andrew93]

We are proceeing webhooks in the background and the rate limit is not really the issue as the jobs are created it is just that if we receive a large import it can take a very long time to run all the jobs since we run into rate limiting errors accessing the Shopify Api for each job and they need to be retried.

[HunkyBill]

This is interesting. In the early days of the Shopify API this exact problem was supposedly dealt with. When a merchant uses built-in Shopify methods to import products, there should not actually be a webhook fired off for every new product, as an example. Shopify was supposed to differentiate between imports and regular one-off operations so as to avoid this kind of crazy webhook exchange. 

That does leave wide-open the question of how to gauge whether a store is now functioning with a new inventory if they did an import, and no webhooks were triggered to alert external Apps. I am not sure anyone has every actually solved that. 

I had an App get crushed recently by the same thing. One merchant important thousands of products, and my webhook for products/create was hammered to the point of destruction. Given that webhooks have recently been experiencing timing issues, perhaps these are connected, and the behaviour of the last few years is toast. 

Just another case of operating in the somber late evening light of Shopify API as opposed to the clarity of noon hour sunlight. 

[Gavinator]

+` to what the other guys already said.

we effectively get a ddos attack from shopify webhooks

Ha! Yes, True. It would be great (in my case at least) if you could add a filter for when hooks would be fired so that I don't get an onslaught in notifications. Anyways, not happening soon so moving on... 

What I do is push these notifications on to a queue for later processing and then get back to the requesting thread ASAP with a 200 (basically what Justin said). I only push the product id. I've enhanced it a little to do look ahead so that if there is another update with the same product id then effectively the current background process does a no-op.