Missing A Single Order Created Webhook

[phil-sebastian]

I have a puzzling problem. I've been catching order created webhooks for many months, and I've always received them. Because I'm still in development mode, I log every http POST that I receive.  Last night at 7:29pm MST, I can see in my Shopify dashboard that a new order was received, but I can see from my logs that nothing was received.  I had an order at 7:26pm which I received a webhook for, and one at 7:42, so both before and after.  It's just the one missing, and no attempt to resend (i.e. the order ID doesn't appear in my logs anywhere).  Seems like a Shopify glitch?  Has anyone seen this before?  Thanks!  

[Deeps_Lady]

---

Hi Try this link below, If this doesnt work try to unintall webhooks and install again

https://community.shopify.com/c/Shopify-APIs-SDKs/Shopify-Not-Sending-Out-Webhooks/td-p/464704

---

[phil-sebastian]

---

@Deeps_Lady wrote:

---

Hi Try this link below, If this doesnt work try to unintall webhooks and install again

https://community.shopify.com/c/Shopify-APIs-SDKs/Shopify-Not-Sending-Out-Webhooks/td-p/464704

---

---

Thanks for the reply.  I'm receiving web hooks still, but I just missed the one, so uninstalling doesn't make sense to me.  I've raised this issue with technical support, I just wanted to see if others have experienced this?  

[waiwai]

Hi Phil, I encountered the same exact problem. Have you solved it after you raised it with technical support?

[phil-sebastian]

Hi waiwai,

I raised the issue with tech support, and they were very helpful to identify the problem (it was my problem).  Note that webhooks can be delayed.  I've seen them delayed up to 6 hours before. I'm actually dealing with another missed webhook issue right now, which I don't believe it my issue, but that's TBD!

[waiwai]

Hi phil,

So for your case, Shopify did send the webhook? But you failed to handle the webhook somehow. Am I getting it right?

For my problem, the server runs in AWS lambda and that missing webhook was not captured at all. Orders before it and orders after it behaves normally. I thought there will be retry for the missing webhook, but it seems that no retry of sending the webhook from Shopify at all.

[phil-sebastian]

Hi waiwai,

Yes. In my last case, it was a Google issue, and not a Shopify one, and I raised a bug report with them (Google).  However, I'm presently chasing an issue much like the one you're describing.  I'm successfully catching and processing hundreds of Order Create webhooks each day (511 since Monday), but I'm missing 4 of them.  I rxed everything fine before and after these missing 4, and it seems that there was no attempt to resend, just like you've indicated.  To my knowledge, these are the first 4 webhooks I haven't received in 10,000s of orders.  I have an open ticket with Shopify tech support presently investigating it. Shopify seems to have solid backend logging of their webhook txs and acks, which is great. I suggest you open a ticket with them to investigate.  I'll let you know what I discover in my case, if you would please let me know what you determine?

[_JB]

Hey @waiwai,

If you provide some example order_ids, I can check our logs to see what happened with the webhooks.

When Shopify sends a webhook, your server is required to respond with a 200 OK within 5 seconds or the webhook delivery is considered failed and is retried with an exponential backoff time. Shopify automatically deletes the webhook after 30 unsuccessful retries. Due to these measures it's odd that you are transiently missing hooks, since the assumption is your server acknowledged receipt. With that being said, it's also possible that a problem on our end prevented the webhook being sent. This happens very rarely, but is a possibility and why we recommend not relying solely on webhooks as a source of shop data. The logs can confirm if we attempted delivery.

[waiwai]

Hi @_JB,

That would be great!

I'll provide you the "id" of orders I fetch from with API /admin/api/2019-07/orders/{order_id}.json

The normal order ids: 2372229857358 (#1035)  and 2374267797582 (#1037)

The order id of missing webhook: 2372485742670 (#1036)

Orders before #1036 and after #1036 have no problems at all.

I'm not sure if the above info is enough for you to check logs as I'm not sure if the ids are unique across different stores. It would be perfect if I don't need to show my customer's store in public. Actually I have had an only chat previously with another support staff and the ticket number is 18073337. Maybe you can also get details from that ticket.

By the way, you mentioned that it not recommended relying solely on webhooks. Could you please tell what alternatives do I have to have same effects if webhook goes wrong. The webhook itself is really helpful as we can simply get notice whenever an order is created in the store and take actions correspondingly.

Thanks you very much.

[_JB]

Hey @waiwai 

Just wanted to let you know our developers have identified a recent change that looks to be responsible for causing dropped webhooks. This change has been reverted, and we're currently monitoring to ensure the issue is resolved. So far the revert looks to be working as expected, but if you have any additional examples from this time forward for resources that didn't fire a webhook, please provide them here.

With regards to not relying on webhooks, one option is to create a job that runs every 30/60 minutes to pull new orders from the API, and compare the response to the number of webhooks you received for the same time frame. If you get a greater number of orders in the API response, this suggests a webhook may have been missed, and you can run a check to find the missing order.