On Thursday, August 20, we experienced an outage that prevented merchants and partners from logging into Shopify for approximately three and a half hours. While this outage didn’t impact online storefronts or checkouts, we recognize that being unable to access your admin for a lengthy period of time creates a significant disruption to your business, especially for those who rely on our POS system to sell in-person. We’re sorry we let you down.
Today, in addition to acknowledging the impact of this outage, we’re sharing the details of what happened and the actions we’re taking to prevent such issues from significantly disrupting your business in the future.
What happened? (Root cause)
While performing updates to our Identity (single sign on) service, we shipped code to improve the way emails are verified when logging into Shopify. This change unexpectedly generated a slow query, which led to login attempts timing out. This meant that merchants and partners were largely unable to log in to Shopify.
The criticality of the system and nature of the change made it difficult to diagnose. We initially identified a very similar query as the likely cause, which contributed to the unusually long duration of the incident. When reverting this query did not fix the issue, we reinvestigated to identify the correct cause and implemented a successful fix. Our post-incident Root Cause Analysis identified inefficiencies in the way our tooling interacts with this specific service, which contributed to this misdiagnosis.
What we’re doing next
As part of the immediate response to this incident, we are performing a full audit of our Identity service in order to identify visible risks and correct them safely. This audit is currently well underway
Additional medium and long-term actions we’re taking include:
We know that reliability is crucial when it comes to the platform you choose to run and grow your business. We're committed to continuously enhancing our resilience and providing the transparency needed so you can run your business effectively. Thanks for your continued trust.
Chief Technology Officer, Shopify
Thanks for putting out this note. It's helpful to learn that I am most likely not the ONLY store whose business has been shut down by this issue. I started communicating through support chat a week ago today, which was Sunday the 23rd of August. I have logged in every day since then except for one. Not one reply back from your business team on why I can't log into either of my two stores. My online store address is www-directsourceone-com.myshopify.com. My name is Doug Marrison. My support ticket, which I had to beg one of the support agents to give to me, is 19311494.
I have inventory of $50,000 for a promotion that was to run last weed on www.cbdsourceone.com through our affiliation with a nation e-commerce retailer, and I have had to postpone it two times. They have given me until Tuesday to get his site back up and running. I know we are a small company, but there is NO excuse for the lack of support and communication. You guys were better than this in the past.
I am asking you to please have someone from your company contact me and tell me why I am not up and running , and how soon I will be. I am at my wits end with your company.
Thank you. Doug Marrison
Thanks for your feedback and the ticket number.
I took a look and it is an email interaction ticket. It is already with the correct team who should be in touch with you, please reply to the email directly in this instance as we're unable to provide support for your ticket in the community.
All the best, Nick