Webhook failing, but no incoming requests!

[homeyjd]

We have developed a private Shopify app and installed it on ~10 stores successfully. All webhooks work great EXCEPT for "shop/update".  I cannot debug this problem because I see no incoming REQUESTS to my back-end! 

I keep receiving this email:

Your webhook for shop/update at https://xxxxxxxx.xxx/webhook is failing to return a successful response.
This webhook has been attempted XX times. If your webhook continues to fail, it will be removed and your application will not receive any more notifications.

The endpoint is backed by AWS CloudFront terminated at an API Gateway + Lambda implementation. It works great for other webhooks.

I am logging full request/responses within CloudWatch.  I've scoured all available logs, but I never see any events triggered during the times Shopify claims to be triggering the shop/update webhook.  It's like Shopify never actually reaches the backend.

As described here, 10240 byte header size limitation and 10MB request body limit. But I doubt we're hitting those limits.
https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html

1. Is there any other debugging steps or re-creation steps regarding WHAT is failing?

2. Is the shop/update webhook triggered by a different Shopify system or in a different way that I'm missing in the documentation?

[emilbryggare]

Hey @homeyjd,

The request funnel should look something like this:

HTTP Request -> CloudFront -> API Gateway -> Lambda.

I know you said you have not found the requests in any logs, but verify this as well.

1. Send a manual request to your endpoint using curl, postman or whatever you prefer.

What happens? Can you find it in the logs?

2. Do the webhook requests reach CloudFront? 

Enable logs for CF Access Logs and check for requests.

If you find requests here, does CloudFront cache the requests?

3. Do the webhook requests reach API Gateway?

Enable logs for API Gateway (if not already enabled) and check for requests.

As you said, I doubt you are hitting the header size or request body limits.

[homeyjd]

Thanks for the reply @emilbryggare !

1. Send a manual request...

Yes, all requests show up in the request/response log.  The API Gateway hosts the installation / OAuth / webhook process, and all other webhooks work correctly using the same URL (I use the X-Shopify-Topic header to differentiate between webhook types instead of using separate URLs).

2. Do the webhook requests reach CloudFront?

Hmm... I don't have CloudFront logging turned on, just API Gateway.  You're right, I should try this...

3. Do the webhook requests reach API Gateway?

No, they definitely do NOT reach API Gateway.  I have full request/responses being logged and not a single log entry is recorded for these attempts.

The Shopify "webhook is failing" email has the following error:

Error: read tcp 10.228.25.240:33020->13.35.115.118:443: i/o timeout

If this happened for other webhook types, I would think that CloudFront is incompatible with Shopify's webhook delivery (TLS version incompatibility?)  However, since only one webhook type fails, this leads me to believe Shopify's delivery system for shop/update is different, or that the shop/update message is very different.

[emilbryggare]

@homeyjd strange... I hope you find it, CloudFront logs is probably your best bet.

[homeyjd]

@emilbryggare Your suggestion to turn on CloudFront logging helped!  I was able to locate failed CloudFront requests and see that the times from Shopify's email did not match my expectations.  By adjusting for time jitter / timezone differences and filtering by error type, I was able to match CloudFront to API Gateway logs.

Shopify was hanging up after 4.5 seconds even though API Gateway was returning a response after 6-7 seconds.  I could never find a failure because API Gateway never officially failed.  I had to manually match up CloudFront errors to API Gateway successes to find which requests failed.

Additionally, my endpoint was taking between 2 and 7 seconds to return because Shopify's API would sometimes take longer than expected.  So when Shopify got slower to respond, it would tell me my webhook was failing.  But from my perspective nothing was failing!

I wish the error message had a little more detail.  "i/o timeout" could have been "Timed out after 4.x seconds" or better "Request sent, but did not return in expected X seconds".

[fordey]

@homeyjd How did you actually fix this problem? I'm in the same situation!

Thanks.

[homeyjd]

In my application, I lowered my HTTP request timeout when talking to Shopify's API so that when the API got slow, my app would fail-and-retry.  My app uses mostly GET requests, so these were safe to retry.

I also added an overall timeout on my webhook handler.  If my handler took too long, I responded right away with a "Request Timed Out" response.  Shopify has an auto-retry for failed webhooks, so I could fall back on this behavior without affecting my code.