Dealing with multiple hits from Shopify to our webhooks for the same event

Tourist
12 0 2

This isn't really question, but I just want to give back some info here to the Shopify developer community about something we ran into that had me stumped for a long time.

It turns out that the Shopify API will sometimes make multiple calls to your webhooks (perhaps by error?) for the same webhook event. This isn't an issue if you're running a single web application server, but if you're running 2 or more web servers (like we do) then there is a possibility for a race condition when both servers check your database (or key-value store) for whether the event has already been handled, both get a negative response at the exact same time, and so both proceed to try to process the same event and cause duplication.

I read somewhere in the forms that someone said the best solution was to simply setup a queue and process events in the queue sequentially, making sure not to process the same event twice. However, that's not a great solution because what if you're using multiple event processing worker servers (like we do at Banana Stand)? If both event processing servers pull events off the queue at the exact same time, there you go - same issue with duplicated processing. 

After much thought we finally came up with the only reasonable solution we could figure out and I felt I should share it: add a unique constraint in your database and rescue (or catch) the database "value not unique" errors, then retry. In Ruby you can just retry the event processing if the event fails due to the unique constraint, since the second attempt should notice that the event has already been processed and move on.

We could have implemented our own lock-wait system, but using the database's lock-wait was a much easier idea.

Has anyone else run into this issue and come up with a better, but equally valid solution?

I don't expect many people to say yes, but I figure this is a good "knowledge dump" for someone running into the same "head-scratcher"in the future.

Shameless plug: Banana Stand is a social proof and urgency platform that helps increase conversions. Check it out!
0 Likes
Highlighted
Shopify Partner
5 0 0

@Jay_El-Kaake great question and solution as well. I'm also following your suggested solution in our app.I created composite keys for columns in a table
shopify_updated_at
shop_id
topic_name
so if Shopify triggers the same event multiple times I first insert a MySQL DB record in the above-mentioned columns, so all duplicate entries go in catch block while I create a job for all successful entries. Later I just clean the duplicate_entries table.
Now there comes another problem
All Shopify webhooks data don't contain shopify_updated_at column so it will be great if u suggest columns that are better for composite keys.

0 Likes