shop/update wehbooks getting out of control

shop/update wehbooks getting out of control

Recomatic
Shopify Partner
45 0 11

Hello,

 

In the past 24 hours we've been receiving an unusually large number of shop/update wehbooks where no attributes seem to have actually been updated, just the same data over and over again. At this time 60% of all webhooks we are receiving are shop/update webhooks and it's putting unnecessary strain on our servers.

 

We've checked and made sure that our app immediately sends a 200 response so we know it's not the standard Webhooks retry mechanism but we keep getting them for no apparent reason.

 

Here's an example:

 

In 3 minutes between 2019-04-30 15:57:11 +0000 and 2019-04-30 16:00:36 +0000 we received 10 shop/update webhooks from a shop with ID 28201302 where no info was actually updated except the updated_at field.

 

Please advise.

Replies 13 (13)

Dan_Sundy
Shopify Partner
31 0 4

Just wanted to chime in and say that I am experiencing the same thing on my production server.

 

It has been happening in the past 24 hours and doesn't seem to be correlated to a release.

Alex
Shopify Staff
1561 81 342

Thanks for the report. Are you guys seeing any specific data changes on your end for the shop object? We're looking into it.

 

Edit

 

Completely missed the detail where seemingly nothing changed, my mistake. Considering that, continuing investigation.

Alex | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

Recomatic
Shopify Partner
45 0 11

Hi Alex,

 

Thanks for your answer, any news on your investigation? It is still happening and here's a fresher example:

 

Same webhook received twice from shop with ID 6981058596 in less than 20 seconds interval (2019-05-02 15:31:15 +0000 and 2019-05-02 15:31:30 +0000) containing the exact same info expect the updated_at field that changed respectively from 2019-05-03T01:31:02+10:00 to 2019-05-03T01:31:16+10:00.

 

Curious to see what your logs say about that.

Recomatic
Shopify Partner
45 0 11

Any news on that? Still receiving the same shop update webhooks over and over again...

Dan_Sundy
Shopify Partner
31 0 4

Still an issue for me as well.

Alex
Shopify Staff
1561 81 342

Apologies for the delayed update.

 

@Recomatic based on what we can see, this looks to have been caused by a surge in metafield update requests from your app. It appears to be an unusual spike, seeing that you appear to have gone from around 4 PUT or POST requests (mostly PUT) per hour to a bit over 1000. Are you able to provide some context on this one? It looks like you share a decent number of installs with @Dan_Sundy as well, so that's probably what they're seeing too.

 

It is worth clarifying: updating base shop metafields leads to shop/update webhooks.

Alex | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

Dan_Sundy
Shopify Partner
31 0 4

Thanks for the insight @Alex

 

@Recomatic I hope you are able to resolve the PUT request increase.

Dan_Sundy
Shopify Partner
31 0 4

Hi @Recomatic, are you working on this? I'm still getting constant shop/update webhook calls.

Recomatic
Shopify Partner
45 0 11

Hi @Alex, Thanks for the clarification about the Shop metafield Update resulting in a shop/update webhook, which was not the case before (our portion of code responsible for writing Shop metafields has not changed in the last year and yet the webhooks surge only started last week). Concurrently can you confirm that a product metafield update should result in a product/update webhook?

 

@Dan_Sundy, we've already taken measures to limit the amount of metafield updates we make but we're working on a permanent solution that should be deployed shortly.

Alex
Shopify Staff
1561 81 342

@Recomatic I can confirm that updating product metafields fires a products/update webhook.

 

I can't see too far back in the past in my logs, but your app was generally averaging around 4 updates to these metafields per hour and these appear to have been triggering the webhooks in the days leading up to the surge, where the frequency shot up significantly when you started seeing the issues. That's just what I'm seeing in very short term logs though, so obviously I don't have a full picture of your app's history.

 

I'll re-confirm with the webhooks team whether this has always been the case or not (maybe it was always the intended case, but it qwasn't working for some reason), a detail we could have missed.

Alex | Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

Dan_Sundy
Shopify Partner
31 0 4

@Recomatic Could you keep me posted when you get this resolved? I am still seeing constant shop/update activity.

 

Luckily, we don't do anything on our end if the values haven't changed, so it's not absolutely killing our server. However, We will definitely be trying to figure out a way to ditch this webhook now that we know this can happen.

 

Thanks all!

Recomatic
Shopify Partner
45 0 11

Hello @Dan_Sundy, we’ve made some changes in the way we use shop metafields and you should no longer see shop update webhooks being fired as a result of our action when no actual shop attribute were changed.

 

However, note that as many apps can write shop metafields for various reasons, this can still happen occasionally.

Dan_Sundy
Shopify Partner
31 0 4

Thanks @Recomatic. It looks like the requests have slowed down a little on my end.