Removing of Webhook bug,flaw or issue before 48 hours

[Toughees]

Our store has a Webhook on the Order Payment hook but every now and again, Shopify remove the Webhook. It happens about every two months and it really affects our ordering process. They replied to me on a support chat with the following "I checked your resources here and Shopify expects the receiving server to send back a "200 OK" response to Shopify after receiving the webhook, if a response is not sent from your server within 5 seconds of the webhook having been sent, or a response other than a "200 OK" has been sent back, the webhook will be tried again. If their server continues to send an incorrect response during the retries, or no response at all, Shopify will continue retrying a total of 19 times over a 48 hour period. If all retries fail, the webhook is removed from the shop."

Now big issue, I picked up a flaw or bug with this process. Our last order that went through the Webhook successfully was on Saturday 23rd July at 19:20 so the Webhook was working then and I have the order numbers and log files as confirmation. If its 48 hours as per their documentation then the Webhook would only be removed at 19:20 on Monday 25th July. Saturday 19:20 to Sunday 19:20 is 24 hours, Sunday 19:20 too Monday 19:20 is 24 hours. However the Webhook disappeared before then as I am writing this issue 10:44 on Monday morning. Whats worse is there is no communication to inform you that the Webhook has been removed. You have to check manually. 

So the big question is why is our Webhook being removed prior to 48 hours when their documentation clearly states only after 48 hours. PS our server that handles the API is up 100% of the time in a data centre and all logs confirm the server was up at all times this entire weekend. 

[Alan]

Hi @Toughees - thanks for getting in touch. On our end, I can confirm that we have no reports of a larger webhook-related service outage or delay at the moment. That said, we'd be happy to investigate further to see if we can track down why this is happening. 

Could you share the Webhook-ID from the response body we sent out for the last time the specific webhook subscription was sent out correctly? Using this I could take a look at our logs and narrow down what the issue is/see if we can find out if the webhooks were properly triggered on our end. 

Hope to hear from you soon - more than happy to look into this further. 

[Toughees]

Hi Alan

Thanks for the info. We busy trying to check the responses back to you as the last request Shopify sent it was a 500 error. On that last request I take it its a keep alive post as we didnt have an order then. Attached is a screenshot of the last successful order 3250 and then order 3251 was never receieved as the webhook was removed prior to this order

Last successful order
api.toughees.co.za:443 34.71.35.169 - - [23/Jul/2022:19:20:56 +0200] "POST /order/create HTTP/1.1" 201 5114 "-" "Shopify-Captain-Hook"

Last request and got a 500 back
api.toughees.co.za:443 34.71.211.51 - - [24/Jul/2022:13:04:06 +0200] "POST /order/create HTTP/1.1" 500 5425 "-" "Shopify-Captain-Hook"

[Toughees]

Hope that info is sufficient, if not please let me know and thank you, really appreciate your help to look into this matter. 

[Alan]

Hey Toughees - thanks for providing that info, super helpful for sure. You mentioned you received a 500 response on one of the API calls? Do you happen to have an X-Request-ID for that call? That would definitely help us narrow down what could be happening here with the hooks overall. 

Hope to hear back from you soon - definitely happy to look into this further. 

[Toughees]

Hi Alan, unfortunately not sorry. Our previously developer didn't add that level of debugging into the app. I only have the logs from Apache which was the attachment. The 500 error though might be because we are sending a 201. I don't know why the developer is sending a 201. It should be a 200 right?

[Toughees]

Thank you Alan, really appreciate your help. I have got the logs and hopefully this is enough to assist. We are busy in the process of updating the code to respond correctly as it looks like we are not responding correctly back as unfortunately previous developer is no longer within reach. I have also attached the two orders between when the webhook dissappeared/was removed. The last successfully order was #3250 , the next order did not get sent via the webhook to us. 

Last successful order
api.toughees.co.za:443 34.71.35.169 - - [23/Jul/2022:19:20:56 +0200] "POST /order/create HTTP/1.1" 201 5114 "-" "Shopify-Captain-Hook"

Last request and got a 500 back on the 24th and then the webhook was removed/deleted between the 23rd and 25th and then we recreated it on Monday morning. We did not get any further attempts from the time till Monday morning. With this order its calling our webhook but just want to confirm what info you are sending so we can respond correctly as there were no orders to send through and I assume its a keep alive request. 
api.toughees.co.za:443 34.71.211.51 - - [24/Jul/2022:13:04:06 +0200] "POST /order/create HTTP/1.1" 500 5425 "-" "Shopify-Captain-Hook"