shop/update wehbooks getting out of control

[Recomatic]

Hello,

In the past 24 hours we've been receiving an unusually large number of shop/update wehbooks where no attributes seem to have actually been updated, just the same data over and over again. At this time 60% of all webhooks we are receiving are shop/update webhooks and it's putting unnecessary strain on our servers.

We've checked and made sure that our app immediately sends a 200 response so we know it's not the standard Webhooks retry mechanism but we keep getting them for no apparent reason.

Here's an example:

In 3 minutes between 2019-04-30 15:57:11 +0000 and 2019-04-30 16:00:36 +0000 we received 10 shop/update webhooks from a shop with ID 28201302 where no info was actually updated except the updated_at field.

Please advise.

[Dan_Sundy]

Just wanted to chime in and say that I am experiencing the same thing on my production server.

It has been happening in the past 24 hours and doesn't seem to be correlated to a release.

[Alex]

Thanks for the report. Are you guys seeing any specific data changes on your end for the shop object? We're looking into it.

Edit

Completely missed the detail where seemingly nothing changed, my mistake. Considering that, continuing investigation.

[Recomatic]

Hi Alex,

Thanks for your answer, any news on your investigation? It is still happening and here's a fresher example:

Same webhook received twice from shop with ID 6981058596 in less than 20 seconds interval (2019-05-02 15:31:15 +0000 and 2019-05-02 15:31:30 +0000) containing the exact same info expect the updated_at field that changed respectively from 2019-05-03T01:31:02+10:00 to 2019-05-03T01:31:16+10:00.

Curious to see what your logs say about that.

[Recomatic]

Any news on that? Still receiving the same shop update webhooks over and over again...

[Dan_Sundy]

Still an issue for me as well.

[Alex]

Apologies for the delayed update.

@Recomatic based on what we can see, this looks to have been caused by a surge in metafield update requests from your app. It appears to be an unusual spike, seeing that you appear to have gone from around 4 PUT or POST requests (mostly PUT) per hour to a bit over 1000. Are you able to provide some context on this one? It looks like you share a decent number of installs with @Dan_Sundy as well, so that's probably what they're seeing too.

It is worth clarifying: updating base shop metafields leads to shop/update webhooks.

[Dan_Sundy]

Thanks for the insight @Alex. 

@Recomatic I hope you are able to resolve the PUT request increase.

[Dan_Sundy]

Hi @Recomatic, are you working on this? I'm still getting constant shop/update webhook calls.

[Recomatic]

Hi @Alex, Thanks for the clarification about the Shop metafield Update resulting in a shop/update webhook, which was not the case before (our portion of code responsible for writing Shop metafields has not changed in the last year and yet the webhooks surge only started last week). Concurrently can you confirm that a product metafield update should result in a product/update webhook?

@Dan_Sundy, we've already taken measures to limit the amount of metafield updates we make but we're working on a permanent solution that should be deployed shortly.

[Alex]

@Recomatic I can confirm that updating product metafields fires a products/update webhook.

I can't see too far back in the past in my logs, but your app was generally averaging around 4 updates to these metafields per hour and these appear to have been triggering the webhooks in the days leading up to the surge, where the frequency shot up significantly when you started seeing the issues. That's just what I'm seeing in very short term logs though, so obviously I don't have a full picture of your app's history.

I'll re-confirm with the webhooks team whether this has always been the case or not (maybe it was always the intended case, but it qwasn't working for some reason), a detail we could have missed.

[Dan_Sundy]

@Recomatic Could you keep me posted when you get this resolved? I am still seeing constant shop/update activity.

Luckily, we don't do anything on our end if the values haven't changed, so it's not absolutely killing our server. However, We will definitely be trying to figure out a way to ditch this webhook now that we know this can happen.

Thanks all!

[Recomatic]

Hello @Dan_Sundy, we’ve made some changes in the way we use shop metafields and you should no longer see shop update webhooks being fired as a result of our action when no actual shop attribute were changed.

However, note that as many apps can write shop metafields for various reasons, this can still happen occasionally.

[Dan_Sundy]

Thanks @Recomatic. It looks like the requests have slowed down a little on my end.