Not receiving app/uninstalled webhooks anymore.

[ClementBR]

It appears our app is not receiving app/uninstalled webhooks anymore.

Is there a known issue at the moment?

[OnltrApp]

Same issue

[ClementBR]

Good to hear we're not going crazy.

@shopify ?

It's been broken for at least 48 hours and I had to raise multiple tickets but it's literally impossible to reach dev support these days.

[Ryan]

Hey @ClementBR @OnltrApp 

We are taking a look at this already.  Could you let me know a bit of info for your affected app?

- App name

- Merchants affected? (all, or just a subset)

- Webhook version

Cheers!
Ryan

[ClementBR]

Webhook version is 2019-10

I DM'ed you to provide more details.

We did receive a couple webhooks since a couple of hours ago so maybe the issue was fixed?

[Ryan]

Looks like we've identified an issue, and are working on resolving it! Please bear with us.

[ClementBR]

FYI it is still happening

[Ryan]

We've identified a fix and are implementing it now.  After the fix goes in we are going to restore any webhook subscriptions that were inadvertently lost (skipping ones that were re-created), and then reconcile any missed webhooks!

Please let me know if you have any more concerns on this.

[ClementBR]

Can you please expand on that last comment?

Has the issue impacted other topics beyond app/uninstalled?

[ClementBR]

" and then reconcile any missed webhooks!"

Does that mean that the all the webhooks that were not sent will be sent once the fix is deployed?

[ClementBR]

<rant>

It's really frustrating trying to raise important issues and no one listens for 48 hours.

Impossible to make it through the support barriers and hoops quickly enough even though our app is used by 1,000s of merchants...

I beg you to please establish some kind of official IRC / chat / support for app developers! Pretty please!

</rant>

[Ryan]

This only affected app/uninstalled , we are restoring the subscriptions to app/uninstalled that were removed when the apps were not actually uninstalled.  If a webhook should have been sent, but was not, this will be sent.

Couple of things to note:

1. If you intentionally deleted an app/uninstalled subscription during the affected time, this will restore it.
2. If you re-registered your own subscription, and then the app was uninstalled correctly, this will send out a duplicate app/uninstalled webhook

[Ryan]

---

@ClementBR wrote:

<rant>

It's really frustrating trying to raise important issues and no one listens for 48 hours.

Impossible to make it through the support barriers and hoops quickly enough even though our app is used by 1,000s of merchants...

I beg you to please establish some kind of official IRC / chat / support for app developers! Pretty please!

</rant>

---

You should report these issues through your partner dashboard at https://partners.shopify.com/current/support/form, this is where we first heard of the issue and started investigating.

[ClementBR]

Thank you.

Good to hear that the issue was limited to app/uninstalled.

I do hope that you can think about creating a chat room to report production issues (exclusively).

It would probably be more efficient for everyone (including Shopify support).

[ClementBR]

@Ryan  FYI another potential production bug? 

https://community.shopify.com/c/Shopify-APIs-SDKs/Bug-State-nonce-lost-when-redirected-back-to-app-d...

[ClementBR · Accepted solution]

As far as I can tell the bug is still present? Any updates?

[ClementBR]

Not sure why it's marked as a solution, I must have accidentally clicked the wrong button.

Can we please get an update?

[Recomatic]

Same here.

We're missing several uninstall webhooks from the past few days that were never sent, the last one only a few hours ago and I'm noticing the same thing when testing on a development store just now. So this issue is very much "not resolved", and an update from Shopify on this would indeed be very much appreciated.

+1 on @ClementBR's suggestion of having a better way to quickly reach the Shopify team for reporting production issues, as the forum is slow and very dependent on whether or not someone from Shopify is checking it out (it frequently takes hours or days to get an answer, if any), and the so called "partner chat support" does get you someone fast, but the person you talk to is part of the general Shopify support team and getting them to actually understand what you are talking about and redirecting you to the right person is a time consuming and frustrating process to say the least.

[Ryan]

The initial issue is solved, existing webhooks are all sending correctly and no more app/uninstall hooks are being removed.

We are working on backfilling the now missing subscriptions but that might take until tomorrow or Wednesday, but I will provide updates when I can. If you need these webhooks urgently I would suggest re-subscribing to those webhooks topics with your app - this will not affect the backfill.

[ClementBR]

Ah, good to hear @Ryan thanks for the update.