Not receiving webhook notifications after "Your webhook is failing" emails

Solved
phuh
Shopify Partner
7 1 1

I had an issue with my webhook callback endpoint therefore I kept receiving "Your webhook is failing" emails.

I fixed the endpoint today and created new webhook subscriptions but Shopify does not seem to send me notifications anymore. I also tried uninstalling/reinstalling app.

 

Is it possible that the system automatically disabled webhook notifications due to earlier failures? Could someone  from Shopify team please have a look into my app (ID: 3747789) and re-enable it if the system blocked it? Thank you-

0 Likes
SBD_
Shopify Staff
Shopify Staff
1043 141 183

Is it possible that the system automatically disabled webhook notifications due to earlier failures?

Webhooks will be removed if they consistently fail over 48 hours. https://shopify.dev/tutorials/manage-webhooks#frequency

 

But they can be recreated - are they listed if you GET webhooks?

0 Likes
phuh
Shopify Partner
7 1 1

But they can be recreated - are they listed if you GET webhooks?

Yes, there are active subscriptions listed in webhook. Here are sample webhook subscription IDs. Please help-

 

[841475293243] carts/create
[841475326011] collection_listings/remove
[841475358779] checkouts/create
[841475391547] app/uninstalled
[841475424315] checkouts/update

0 Likes
phuh
Shopify Partner
7 1 1

I used Hookbin (https://hookbin.com/) to test a different endpoint and interestingly this endpoint outside my domain worked. (Subscription ID: 841792356411)

This was tested on same app, same shop, same access token.

 

Shopify's webhook notification server probably blacklisted my domain name entirely after many failed attempts. It is fixed on my end but I need your team's help whitelisting it on your end. Thank you-

0 Likes
phuh
Shopify Partner
7 1 1

I've just noticed this note on https://shopify.dev/tutorials/manage-webhooks#frequency and it seems related to my issue since I'm serving this with Cloudflare Workers (https://workers.cloudflare.com/):

 

If you are using Cloudflare on your domain, then you should disable it for the registered webhook URL.

I also received "Your webhook is failing" email again, so I confirmed the server did try to send out a webhook notification.

Is it the case that I am not able to use Cloudflare Workers as callback endpoint? I'm using Full SSL certificate on this domain and it is trusted by all major web browsers according to SSL checker. Is there any reason why Shopify blocks only Cloudflare?

0 Likes
SBD_
Shopify Staff
Shopify Staff
1043 141 183

Hey @phuh 

 

Cloudflare Workers should be fine.

 

Our logs show the webhooks are firing, but often receive a 500 response from the server. Are you able to set up additional logging on your end?

0 Likes
phuh
Shopify Partner
7 1 1

Thanks @SBD_ for checking for me!

 

I've been logging every request hitting the Cloudflare Workers server, and I've been closely monitoring the logs.

All I've received recently was one request hitting at "/webhook/carts/update" sent from postman (AU IP) about 20 minutes ago, and it returned status 200.

 

I am pretty sure there is something weird going on at the Shopify servers. I think you're right I must have returned 500 about a week ago when I tried it, but it is no longer sending me anything since I started logging all requests.

 

I am curious why the doc says Cloudflare (reverse proxy) needs to be disabled in order to receive webhook. Assuming it is certificate related, I purchased a new dedicated SSL certificate at Cloudflare that shows my domain name as common name. However I am not sure how long your servers cache the old universal certificate (SNI-based) that I was using at the time webhook failed about a week ago.

0 Likes
SBD_
Shopify Staff
Shopify Staff
1043 141 183

That was me testing the endpoint from Postman.

 

I just added something to cart on your demo store which triggered carts/create. A request was sent to https://<your subdomain>/webhook/carts/create at 2020-05-19 02:11:11.083 and returned a 500 - do you see a log of this?

0 Likes
phuh
Shopify Partner
7 1 1

@SBD_ Unfortunately no, I don't see any request logged since your postman test.

0 Likes
phuh
Shopify Partner
7 1 1

This is an accepted solution.

@SBD_ Ah! I've just tried copying request headers & body of "carts/create" notification received at Hookbin and manually sending it to the endpoint. I got some clue from an error code specific to Cloudflare. The issue could be on their end not the Shopify servers after all. Thanks for looking into this, I will investigate this issue with Cloudflare and update here on the progress. Thank you!