A message from our Chief Technology Officer

Highlighted
Shopify Staff
Shopify Staff
1 0 5

On Thursday, August 20, we experienced an outage that prevented merchants and partners from logging into Shopify for approximately three and a half hours. While this outage didn’t impact online storefronts or checkouts, we recognize that being unable to access your admin for a lengthy period of time creates a significant disruption to your business, especially for those who rely on our POS system to sell in-person. We’re sorry we let you down.

Today, in addition to acknowledging the impact of this outage, we’re sharing the details of what happened and the actions we’re taking to prevent such issues from significantly disrupting your business in the future.

What happened? (Root cause)

While performing updates to our Identity (single sign on) service, we shipped code to improve the way emails are verified when logging into Shopify. This change unexpectedly generated a slow query, which led to login attempts timing out. This meant that merchants and partners were largely unable to log in to Shopify.

The criticality of the system and nature of the change made it difficult to diagnose. We initially identified a very similar query as the likely cause, which contributed to the unusually long duration of the incident. When reverting this query did not fix the issue, we reinvestigated to identify the correct cause and implemented a successful fix. Our post-incident Root Cause Analysis identified inefficiencies in the way our tooling interacts with this specific service, which contributed to this misdiagnosis.

What we’re doing next

As part of the immediate response to this incident, we are performing a full audit of our Identity service in order to identify visible risks and correct them safely. This audit is currently well underway

Additional medium and long-term actions we’re taking include:

  • Improvements to both internal and third-party tools, which will allow us to more efficiently diagnose and resolve these issues.
  • Automating the way our monitoring dashboards interact with other tooling, allowing us to proactively detect issues and diagnose their cause with greater accuracy.
  • Additional safety measures for changes to the Identity application, including additional reviews and tests that will detect problems and require resolution or manual override before implementing the change.
  • Several updates have also been made to our playbooks based on learnings from this outage to set us up for a better, faster response to potential future issues.

We know that reliability is crucial when it comes to the platform you choose to run and grow your business. We're committed to continuously enhancing our resilience and providing the transparency needed so you can run your business effectively. Thanks for your continued trust.

Jean-Michel Lemieux
Chief Technology Officer, Shopify

5 Likes
Highlighted
Excursionist
54 0 10

Perhaps you should carry out the advice you give us.

That is  not to make any changes to a live site before they have been fully tested on a Beta site

 

Just saying  !!

3 Likes
Highlighted
Tourist
11 0 3

Shopify is awful. No customer support. Sad.

3 Likes
Highlighted
Excursionist
54 0 10

I wouldn't go that far.

The customer is in general excellent and far better than I have experienced with other platforms like Magento

1 Like
Highlighted
New Member
1 0 0

Jean-Michel,

Thanks for putting out this note. It's helpful to learn that I am most likely not the ONLY store whose business has been shut down by this issue. I started communicating through support chat a week ago today, which was Sunday the 23rd of August. I have logged in every day since then except for one. Not one reply back from your business team on why I can't log into either of my two stores. My online store address is www-directsourceone-com.myshopify.com. My name is Doug Marrison. My support ticket, which I had to beg one of the support agents to give to me, is 19311494.

I have these two websites that are not working: www.cbdsourceone.com, and www.mynaughtynights. com. 

I have inventory of $50,000 for a promotion that was to run last weed on www.cbdsourceone.com through our affiliation with a nation e-commerce retailer, and I have had to postpone it two times. They have given me until Tuesday to get his site back up and running. I know we are a small company, but there is NO excuse for the lack of support and communication. You guys were better than this in the past. 

I am asking you to please have someone from your company contact me and tell me why I am not up and running , and how soon I will be. I am at my wits end with your company. 

Thank you. Doug Marrison

0 Likes
Highlighted
Community Moderator
Community Moderator
3244 245 556

Hi @Dmarrison,

Thanks for your feedback and the ticket number. 

I took a look and it is an email interaction ticket. It is already with the correct team who should be in touch with you, please reply to the email directly in this instance as we're unable to provide support for your ticket in the community. 

All the best, Nick

Nick | Community Moderator @ Shopify
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

1 Like