fulfillments/create webhook not firing for some orders

[Andrew-Corknine]

Over the past few days we have had several shops complain that a few of their fulfillments are stuck in "In Progress".  We listen for the 'fulfillments/create' webhook and complete the fulfillment, however on these orders our app never received a fulfillments/create webhook even though it was set as the fulfillment service for these orders.

Is there a current issue with this webhook.  If so, when will it be fixed and what can be done about sending the webhooks that were not sent?

[Mark_Rowlands]

Hi Support,

I am also experiencing the same issue where random fulfilment are either not triggering the fulfilments/create webhook or the Shopify platform is failing to send it.  In my case I have a utility in my app that can identify the missing fulfilment requests; we can then send a request to resend the fulfilment request which works without issue.

[Mark_Rowlands]

I hope you don't mind Corknine - I don't mean to hijack your post but here's a screenshot of what I mean. 

Blue 'Imported' indicates that we received a fulfilments/create request,

Green 'Import' indicates that the Order exists in Shopify and has been 'paid' for and awaiting fulfilment, be we do not have it in our Shopify app.

[Andrew-Corknine]

This is happening to us on many shops, however only a few orders per shop.  I'll include a few affected order ID's from a specific shop below in case that helps.  We have verified that we do not receive any fulfillments/create webhook for these particular fulfillments, even though they are created but we are receiving a fulfillments/create webhook on every other order.

2440263630933
2440335065173
2440339292245

[ECGroupIT]

Hi Corknine,

I raised a partner support request (18093189) around 8am GMT this morning, I reference this message tread in the ticket so maybe the'll respond here. 

I haven't heard anything other than the autoresponse acknowledgement from Shopify support; I'll let you know when I do

[_JB]

Hey @ECGroupIT @Andrew-Corknine @Mark_Rowlands ,

Just wanted to let you know we're currently investigating this one, and I'm hoping to get a bit more information in order to get to the bottom of the issue. 

Your reports mentioned not receiving fulfillment#create webhooks for orders created using products assigned to your fulfillment service. Does this mean you're expecting to receive a webhook when the actual order is created (as opposed to the fulfillment)? I've tested this on my end by creating a fulfillment service, assigning a product to that service, and creating an order using that product. I also signed up to the fulfillment#create webhook, but I only receive that webhook when I've actually created a fulfillment for the order. On my end this looks expected, since I'm only expecting to see the webhook once a fulfillment object has been created. I tested using a fulfillmentService that is opted in to fulfillment orders.

In order to investigate further, please kindly let me know when you're expecting to receive the webhook during the fulfillment flow. As well, please provide an example order_id for the first order you noticed the issue on. I'll forward this information to our developers which will help identify the issue, please don't hesitate to reach out if there are any questions in the meantime.

[Andrew-Corknine]

Hi @_JB ,

For these orders we are not receiving a fulfillments/create webhook when the fulfillment is created.  When we investigate there is a pending fulfillment associated with our fulfillment service but we never receive a webhook notification for these particular fulfillments.  For most fulfillments we do, however it seems to be random ones that we are never notified about.

This is for a fulfillment service not opted into the fulfillment orders using API version 2020-04 and the same webhook version. We only expect this webhook when the fulfillment is created.  I've included below a list of recent fulfillment ids for which we never received a fulfillments/create webhook and can give more as the issue continues if needed.

2289170808935
2289230020711
2166849470558
2038470180907
2314597531757
2314602774637
2123897798754
2159077720123
2123927486562
2123969134690
2123970347106
2123970707554
2379051696202
2096820518960
2248697610326
2402594881687
2117117214766

[Todd_Trimakas]

Here is another example of an order id where we did not receive a request from Shopify:

2508496076956

[ECGroupIT]

Hi @_JB 

Thanks for looking into this. 

To be clear in addressing your response and questions I offer the following background and all discussions/responses from me should be read within the context of this background.

Background

1. The issue we are experiencing relates to an app that our clients install for fulfilments.
2. The Shop that I am using for the examples installed the App 3+ years ago and we have been receiving thousands of fulfilments monthly since then.
3. As part of the install process App subscribes to several Shopify webhooks, among these are the fulfilment/create and fulfilment/update.
4. All of our client’s shops have auto-fulfilment enabled.

Question response – you asked

For the purposes of discussion and by way of example please note the following two orders that displayed in the screenshot I provided 06-16-2020 05:52 PM.

Key	Shopify order id	Order created dateTime	Status	Issue
1	2507236900960	16-06-2020 22:36:10	Paid	fulfillments/create webhook NOT received.
2	2507231658080	16-06-2020 22:32:14	Paid	fulfilments/create webhook RECEIVED.

1. Your reports mentioned not receiving fulfillment#create webhooks for orders created using products assigned to your fulfillment service. Does this mean you're expecting to receive a webhook when the actual order is created (as opposed to the fulfillment)?
1. I am expecting to receive a webhook for order 1 in the same way I receive the webhook for order 2.
2. I've tested this on my end by creating a fulfillment service, assigning a product to that service, and creating an order using that product.
1. The issue we are experiencing appears to be random or at least we cannot find a specific pattern to it.  It does not appear to be product or shop customer based.
3. I also signed up to the fulfillment#create webhook, but I only receive that webhook when I've actually created a fulfillment for the order.
1. Our shops are configured to automatically create fulfilments.
2. As can be seen above and in the screenshot previously sent, webhooks are automatically being sent for some order shipments but not all.
4. On my end this looks expected, since I'm only expecting to see the webhook once a fulfillment object has been created. I tested using a fulfillmentService that is opted in to fulfillment orders.
1. I would agree that this would be expected for shops that require manual fulfilment creation, but all of our shops auto create fulfilments.
2. This does not explain why some fulfilments are created an others are not within the same shop.
5. In order to investigate further, please kindly let me know when you're expecting to receive the webhook during the fulfillment flow.
1. I would expect fulfilment to be triggered automatically for orders that have been confirmed as paid.
6. As well, please provide an example order_id for the first order you noticed the issue on.
1. I do not have this information as we have subsequently regenerated all of the missing fulfilments.  We are currently running our utility to regenerate missing fulfilments 3 time a day.  You can spot the orders that had to have a fulfilment regenerated as the fulfilment date is significantly different from the order creation date (more than 1 minute).
7. I'll forward this information to our developers which will help identify the issue, please don't hesitate to reach out if there are any questions in the meantime.
1. Thank you.  Please update this thread if you have any additional questions for us, or update the associated support ticket – 18093189.

Thanks

Mark

[_JB]

@ECGroupIT @Andrew-Corknine @Todd_Trimakas @Mark_Rowlands 

Thank you for providing these details, our developers have identified a recent change that looks to be responsible for causing dropped webhooks. This change has been reverted, and we're currently monitoring to ensure the issue is resolved. So far the revert looks to be working as expected, but if you have any additional examples from this time forward for resources that didn't fire a webhook, please provide them here.