Not receiving app/uninstalled webhooks anymore.

[ClementBR]

It appears our app is not receiving app/uninstalled webhooks anymore.

Is there a known issue at the moment?

[OnltrApp]

Same issue

[ClementBR]

Good to hear we're not going crazy.

@shopify ?

It's been broken for at least 48 hours and I had to raise multiple tickets but it's literally impossible to reach dev support these days.

[Ryan]

Hey @ClementBR @OnltrApp 

We are taking a look at this already.  Could you let me know a bit of info for your affected app?

- App name

- Merchants affected? (all, or just a subset)

- Webhook version

Cheers!
Ryan

[ClementBR]

Webhook version is 2019-10

I DM'ed you to provide more details.

We did receive a couple webhooks since a couple of hours ago so maybe the issue was fixed?

[Ryan]

Looks like we've identified an issue, and are working on resolving it! Please bear with us.

[ClementBR]

FYI it is still happening

[Ryan]

We've identified a fix and are implementing it now.  After the fix goes in we are going to restore any webhook subscriptions that were inadvertently lost (skipping ones that were re-created), and then reconcile any missed webhooks!

Please let me know if you have any more concerns on this.

[ClementBR]

Can you please expand on that last comment?

Has the issue impacted other topics beyond app/uninstalled?

[ClementBR]

" and then reconcile any missed webhooks!"

Does that mean that the all the webhooks that were not sent will be sent once the fix is deployed?

[ClementBR]

<rant>

It's really frustrating trying to raise important issues and no one listens for 48 hours.

Impossible to make it through the support barriers and hoops quickly enough even though our app is used by 1,000s of merchants...

I beg you to please establish some kind of official IRC / chat / support for app developers! Pretty please!

</rant>

[Ryan]

This only affected app/uninstalled , we are restoring the subscriptions to app/uninstalled that were removed when the apps were not actually uninstalled.  If a webhook should have been sent, but was not, this will be sent.

Couple of things to note:

1. If you intentionally deleted an app/uninstalled subscription during the affected time, this will restore it.
2. If you re-registered your own subscription, and then the app was uninstalled correctly, this will send out a duplicate app/uninstalled webhook

[Ryan]

---

@ClementBR wrote:

<rant>

It's really frustrating trying to raise important issues and no one listens for 48 hours.

Impossible to make it through the support barriers and hoops quickly enough even though our app is used by 1,000s of merchants...

I beg you to please establish some kind of official IRC / chat / support for app developers! Pretty please!

</rant>

---

You should report these issues through your partner dashboard at https://partners.shopify.com/current/support/form, this is where we first heard of the issue and started investigating.

[ClementBR]

Thank you.

Good to hear that the issue was limited to app/uninstalled.

I do hope that you can think about creating a chat room to report production issues (exclusively).

It would probably be more efficient for everyone (including Shopify support).

[ClementBR]

@Ryan  FYI another potential production bug? 

https://community.shopify.com/c/Shopify-APIs-SDKs/Bug-State-nonce-lost-when-redirected-back-to-app-d...

[ClementBR · Accepted solution]

As far as I can tell the bug is still present? Any updates?

[ClementBR]

Not sure why it's marked as a solution, I must have accidentally clicked the wrong button.

Can we please get an update?

[Recomatic]

Same here.

We're missing several uninstall webhooks from the past few days that were never sent, the last one only a few hours ago and I'm noticing the same thing when testing on a development store just now. So this issue is very much "not resolved", and an update from Shopify on this would indeed be very much appreciated.

+1 on @ClementBR's suggestion of having a better way to quickly reach the Shopify team for reporting production issues, as the forum is slow and very dependent on whether or not someone from Shopify is checking it out (it frequently takes hours or days to get an answer, if any), and the so called "partner chat support" does get you someone fast, but the person you talk to is part of the general Shopify support team and getting them to actually understand what you are talking about and redirecting you to the right person is a time consuming and frustrating process to say the least.

[Ryan]

The initial issue is solved, existing webhooks are all sending correctly and no more app/uninstall hooks are being removed.

We are working on backfilling the now missing subscriptions but that might take until tomorrow or Wednesday, but I will provide updates when I can. If you need these webhooks urgently I would suggest re-subscribing to those webhooks topics with your app - this will not affect the backfill.

[ClementBR]

Ah, good to hear @Ryan thanks for the update.

[Recomatic]

Thanks @Ryan, tryed again on a dev shop and it seems to be working as expected now.

[ClementBR]

FYI it is still happening.

I'm assuming it's because you haven't resubscribe all the webhooks yet but just wanted to let you know...

[Recomatic]

Indeed, just tried again on our dev store with ID 5234605 and did not receive an uninstall webhook despite our partners account logging it at 6:07 am EDT.

Please advise.

[praagya_joshi]

Just wanted to chime in and mention that we have also noticed that the "app/uninstalled" webhooks has been unsubscribed for quite a few of our customers' stores.

@RyanThank you for the updates so far. If you can confirm that the missing webhooks will be re-subscribed for all affected stores, that would be wonderful.

[Kasirajan]

@Ryan I am still facing issues in receiving an uninstall webhook. We are facing this issue from March 13 and we are missing some sequence of operations based on this webhook.

[shoppinwithme]

At least one of your webhooks have used a deprecated webhook version in the last 30 days. Support for these deprecated webhook versions will be removed on April 1, 2022. Please update the webhooks listed below to API version 2021-07 or later to ensure your webhook-driven workflows are not disrupted. Learn how to update to a newer webhook version here Store: Shoppin' With Me Webhook: app/uninstalled Breaking changes: The force_ssl field has been removed from the Shop REST Admin API. View change. Why am I getting this error? I have followed directions to find this but I don't see anywhere to update the webhooks for this.

[Matt_Goodwin]

A number of our production and dev app instances are still missing the app/uninstalled webhook, although new installs are registering the webhook correctly.  I saw the note from Shopify that removed webhooks would be back-filled by tomorrow, but just wanted to clarify  few things:

1. @Ryan mentioned resubscribing to any webhooks that are urgently needed.  I thought the app/uninstalled webhook was automatically subscribed to when a user installs an app.  Is there a manual way to subscribe to this webhook?

2.  The only potential issue I see with sending any app/uninstalled webhooks that were triggered during the outage is what if the merchant has since re-installed the app?  Occasionally we see merchants uninstall our app and then a few hours later they will re-install.  Sending an uninstall webhook for a store that re-installed will obviously cause big problems... are there any plans to address this?  

As a side note a big +1 to @ClementBR 's suggestion on a better way to urgently access partner support.  I don't even bother using the chat tool anymore for partner issues as the folks on the other end don't seem to have any knowledge of the issues partners often deal with.  I understand the policy is that support can't give 'technical help' to app devs who are troubleshooting, but I find that partner chat takes this to an extreme and generally won't verify that everything is working properly on Shopify's end.   Even a direct email address for support staff that are familiar with the API, webhooks, etc. would be a huge improvement.

[grwthgrp]

This is still happening for us as well. It started on March 10th, and has been that way since. Quite a while for an essential webhook to not be sent back.. and no notice from Shopify anywhere - have to dig around the forums for an official statement. Why is there no better communication with partners?

[Ryan]

To clarify this a bit further, this was a one time issue where some app/uninstalled webhook subscriptions were deleted.  There were never any issues with webhook deliveries.  Any "missed" webhooks are simply because the subscription was deleted.  Any subscriptions that were re-made or made after this event have no issues.

The team is currently in the process of restoring these subscriptions, and then reconciling any uninstall events that occurred while they were deleted which will generate these uninstall hooks that were never sent because the subscription was gone.  This is expected to be resolved by tomorrow.  

In the meantime, if your app needs to perform an important action when receiving this uninstall webhook, it is suggested that you have your app re-subscribe to this topic.  This will not affect the reconciliation.  See the documentation on creating a webhook subscription here.

[ClementBR]

@Ryan will you be re-issuing the app/uninstalled webhook even if they store has since re-installed the application?

I would hope not.

How do I mark this topic as non solved?

[ClementBR]

Any update would be much appreciated.

[Ryan]

Webhooks will not be sent for any shop that has since reinstalled the app.

There are some issues restoring the subscriptions and the team is still working on it but as you can imagine everyone has a limited capacity these days. I would suggest again, if your app relies on these subscriptions to perform any task on an uninstall, you should register the subscription via the API.

[ClementBR]

Got it, thanks.

[grwthgrp]

Still happening here, and still haven't received the backfill events.

[ClementBR]

We gave up and ended up writing a script to resubscribe the webhook for all customers and manually uninstall customers that had uninstalled while the bug was occurring...

[praagya_joshi]

Yeah, we ended up doing the same thing. Also utilised the opportunity to upgrade the webhook version.

[Andrew-Corknine]

We are still having this issue with shops.  

We have a shop who uninstalled on March 26th and we received no webhook for this.  There is no way to re-register as they are now uninstalled.  Please fix this and send missing notifications as soon as possible as this affects our ability to have our apps integrate correctly.

[Ryan]

Hey all, 

Just wanted to throw in a final update and close off this issue.  After restoring all of the deleted app/uninstalled webhook subscriptions, we had queued up any missed webhooks to be sent in staggered batches as to not overload apps.  This finished late last week, and we've verified that these subscriptions are intact and the reconciliations done.

Thanks for your patience on this, and let me know if you still have any issues with regards to these webhooks.

Ryan

[Andrew-Corknine]

We are still have at least one(very possibly more) shops that were uninstalled that we never received an app/uninstalled webhook for.  If I view the shop in my partner dashboard it shows that it was uninstalled, and the access_token is no longer valid but we never received a webhook.  How can I go about reconciling this?

[Parcel_Intellig]

Hi @Ryan 

Is there a recurrence of this issue? i'm finding that the app/uninstalled webhook is no longer firing since may 30th.

Thanks

Min

[LinhD]

Since May 30th, we don't receive uninstall webhooks from Shopify either. It's likely because of SSL expiring issue from Sectigo's (aka Comodo's) root certificate. This issue is also shared pretty widely on Twitter. However, I would expect to receive "Your webhook is failing" notification emails from Shopify about the failed webhooks. We do receive notification emails for other very few webhooks failing due this SSL expiring issue.

For this SSL expire issue, there are 2 possible solutions:

1. The senders (Shopify in this case) to fix their certificate chain 

2. The receivers (Shopify apps) to re-issue their SSL certificate 

I guess it would take Shopify a while to fix the certificate chain in their servers so we are re-issuing our SSL certificate, which also takes a few days because "it's a manual process" (said the SSL issuer). 

[Parcel_Intellig]

Thanks LinhD,

This is good info, looks like you can just download the certificate again from sectigo and install the new CA bundle to get Shopify to recognise it.

[Ryan]

Hi all, thanks for the reports I'll take a look into this new occurrence.  Any confirmation if updating your certificate solved the problem?

[Parcel_Intellig]

Hi @Ryan 

yes it looks to have solved the problem.

the certificates were cross signed, browsers recognised the usertrust cert one level above the root as the root when the addtrust one expired however looks like Shopify's server didn't recognise the alternative certification path.

i've reloaded in the new CA bundles and it looks to have cleared up the issue.

Thanks

Min

[LinhD]

@Parcel_Intellig Thanks for the tip! Updating the CA bundles seems to solve the issue from our side too. 

[Extendons]

Hi @Ryan 

Hope you are doing great!

We are also facing this issue Since May 30th with our listed and unlisted apps.

After reading this thread we have updated the certificate and it did not resolve this problem for our apps.

Can you please check this issue for our listed apps ?

[Uladzimir]

Same issue. All our apps stoped receiving app/uninstalled webhook since May 30.

[Juan_Garcia1]

Same here. Not receiving any app uninstalled webhook even when they are properly registered.

[Ryan]

---

@Extendons wrote:

Hi @Ryan 

Hope you are doing great!

We are also facing this issue Since May 30th with our listed and unlisted apps.

After reading this thread we have updated the certificate and it did not resolve this problem for our apps.

Can you please check this issue for our listed apps ?

---

I still see your root certificate from Sectigo expired.  Check out their site for more details on fixing the issue: https://support.sectigo.com/Com_KnowledgeDetailPage?Id=kA03l00000117LT.

For anyone with the same issue please use one of the free tools online to inspect the certificate of your domain for issues.  The Sectigo change reflects as a chain issue or an issue with a root certificate, not with yours in particular.  In all the cases I've seen, fixing this resolved the webhook issues.

[Juan_Garcia1]

Thank you so much @Ryan. Updating the Certificates worked on our case.