Solved

Order paid webhooks do not always fire

DanAtBonify
Shopify Partner
54 4 32

The Problem

There are times when order/created, order/paid, and order/fulfilled webhooks do not fire. This is confusing for developers, and also leads to "gaps" where it is nearly impossible to accurately react when an order becomes paid (created and fulfilled events are more easily identifiable).

This happens if you are subscribed to the order/updated AND order/paid webhook and there are multiple events happening on an order in a short period of time. For example, if an order is updated and also paid within a short period of time (15 seconds or so), you will only receive the order/updated webhook.

The primary issue with not always sending the webhooks is that there is no fast and guaranteed way to react to order/paid events. For example, if you only receive an order/updated webhook and do not get an order/paid webhook, how can you tell when the order was paid? You could potentially check the order transactions, but that requires an API call and is not efficient for large apps. The only way I can see this working is by tracking order paid state for all orders over time, such that you can tell when an order has transitioned from non-paid to paid. However, this is not ideal for large-scale apps. I do not want to be storing the financial status for 10 million orders.

How to Fix

The simplest fix would be to include a "paid_at" timestamp as part of the order payload. That way if you receive an order/updated webhook and the "paid_at" timestamp is within 15 seconds of the "updated_at" timestamp, you know the order was recently paid. Technically this is still a bit of a hack but I assume always sending order/paid webhooks may cause problems for other apps that have already had to build hacks around this problem.

A Note About Other Order Webhooks

While the issue also happens with other order webhooks, they are solvable. For example, if you are subscribed to order/updated and order/fulfilled you may not actually get a fulfilled webhook. However, you can check if an order was recently fulfilled because the fulfillments information and timestamp are passed on the payload. If "transactions" were included in the order payload then we could also determine when an order was paid without having to use an API call.

Similarly, if we don't get an order/created webhook we can assume an order was recently created by just checking the "created_at" timestamp.

Accepted Solution (1)

DanAtBonify
Shopify Partner
54 4 32

This is an accepted solution.

This issue is not valid and should be ignored. The problem with the webhooks was caused by an internal programming error.

View solution in original post

Replies 11 (11)

HunkyBill
Shopify Expert
4845 60 547

One thing to note. Webhook are not some synchronous signal you can rely on in terms of temporal status. Like email, just because someone hit send, does not mean the message is delivered to some inbox. A lot of nodes have to pass along the data, and a postmaster has to process it. Same for webhooks. Just because an order gets marked as paid in the Shopify state machine, does not mean you will know right away. In other words, if you think you are getting notified an order was paid, and that that is an accurate timestamp, you are mistaken. 

Receiving a webhook for orders/paid or any other topic, is a message you can act on. To say that if you got it late, ruins your business logic implies your business logic is dependent on something it should not be dependent on. If you actually do care when an order was actually paid, for that you do need to look at the transaction that in fact accepted payment.

All I am saying is, most of the time, you actually will receive the webhooks you need. But in a temporal sense, when you receive them is never going to be accurate on any timeline except for the fact that the Shopify webhook machine was fed some data, and sent it out. Nothing more, nothing less. It is a convenience but not a contract. Build your App appropriate to that and you are better off.

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
JohnAtBonify
Shopify Partner
136 7 34

@HunkyBill 

I think the timestamp mentioned is just a red herring in this conversation. The timestamp is just mentioned as brainstorming a workaround for a flaw in Shopify's webhook processing. If the webhook event worked as advertised, the workaround suggested is not needed.

The root cause issue is that Order Paid webhooks are not always sent when also listening for Order Updated webhooks. The delta between when they are sent and not sent is not documented and entirely mysterious. If the delta was explained, and enough information was available, we could piece it together with duct tape and glue and come up with a workaround.

The bottom line is that order paid events should always send an order paid webhook always and forever, but that's not happening.

HunkyBill
Shopify Expert
4845 60 547

Just to prove your point then. You are able to demonstrate, time and time again, or at least with a reasonable set of data that:

1. you receive orders/update webhook payload for an order not paid

2. at some point in the future, you check that order status and notice it is paid.

3. you have no record of the orders/paid webhook

Since you had to have implemented point 2, to prove your original proposition, at that point, without the orders/paid webhook, you would be able to execute the action of having received the orders/paid webhook, because, well, you note the transition to paid happened.

Must've been quite the pain to have done all that! I feel your pain.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
JohnAtBonify
Shopify Partner
136 7 34

@HunkyBill Unfortunately, your analysis remains inaccurate.

We're subscribed to order webhook events on behalf of our merchant customers. The reason we're subscribed to both is irrelevant. Merchants want to respond to these events independently. It has nothing to do with the problem or our approach to handling the data.

Order paid events are not consistently sent by Shopify as webhooks when listening to both order updated and order paid webhooks on the same shop. We're not checking later, we're not missing anything. It's not a programming problem.

HunkyBill
Shopify Expert
4845 60 547

It's cool that you assert something without any proof. My "incomplete analysis" then nicely complements your disjointed and somewhat murky claim that you are missing webhooks. All good. Goes hand in hand I guess! Happy Easter!

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
JohnAtBonify
Shopify Partner
136 7 34

What proof would you like?

HunkyBill
Shopify Expert
4845 60 547

I pointed out a scenario that helps expose that you are missing your expected webhooks. While there are other ways to prove it ie) Shopify tells you you indeed missed it, my outline is simple enough to "do" and while it does not "prove" anything perfectly, it at least gives you a fighting chance at arguing your point.

Statistically speaking, if even 500,000 Shopify stores use webhooks for orders (could be a low estimate, who knows), and Shopify was failing to send orders/paid webhooks, your complaint would not be isolated, nor uncommon. But since few others are complaining, it makes your complaint special. Without some kind of effort on your end to expose the issue with data, how can anyone help you? Say for example you tell Shopify support, Order #78651 NEVER issued an orders/paid webhook and it should have. They would be able to look that up easily and validate your claim. We can only assume you never reached out to them, or in fact, you did, and they just never responded back.

This is not about me. I am just trying to steer you straight, as it is not always easy to debug these pesky platform issues. You may think you know what it is, but the reality can often be totally different. Just claiming things don't work is not proof.

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com

DanAtBonify
Shopify Partner
54 4 32

This is an accepted solution.

This issue is not valid and should be ignored. The problem with the webhooks was caused by an internal programming error.

JohnAtBonify
Shopify Partner
136 7 34

For the record, webhook payloads and hmacs can be identical when following multiple topics. When determining if a webhook has already been sent, the webhook topic must be considered in addition to payload and hmac. For example, order paid and order updated webhook payloads can be identical aside from the topic. Additionally, order updated webhooks may or may not be sent before order paid webhooks. When checking specifically for duplicated webhooks, take these nuances into account. 

@HunkyBill I could have done without the condescension, but thanks for the "encouragement" to prove it. 

HunkyBill
Shopify Expert
4845 60 547

@JohnAtBonifySo sorry you "read" condescension in there, but I am sure you are well aware of just how difficult it is to write english and not have 12 differing opinions on whether the person was being sarcastic, spastic, idiotic, mean, jolly, snide, or perfectly supportive and communicative with clarity, detail and good ideas. I meant no ill will, but I accept your opinion, as that is the thing to do.

As a person that has written thousands upon thousands of missives, offering free advice that would otherwise simply not have been there, I have taken my fair share of abuse from people that seemingly don't get it, that I am helping them to get to their happy place, but without fluff words of encouragement or platitudes. I would not bother otherwise.

So be it. I accept that you acknowledged that my intuition was correct and luckily you found your happy place. If you check the records, you will find that in most cases when I post my questions here, I usually answer them myself once I get over the hump, by myself and find my mistakes or misunderstandings. There are some beauties buried in here that show others have helped me too. But a lot of my "tough" questions go unanswered, I get the old crickets chirping, and thus, no satisfaction either. Open issues FTW.

So ya.. glad it all worked out.

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
peters
New Member
5 0 0

Hey @DanAtBonify , just so I understand, you consider the orders/paid webhook reliable, i.e. it's not in conflict with orders/updated?

Thanks!