Thousands of historical orders updated at 2024-03-08

[fpicanzadirect]

Some thing has happened today on Shopify's back end as an automated query that I run a few times a day against the API to get Shopify orders updated today (like if someone did a cancellation or a refund) returned thousands of records instead of the usual handful.

This ran fine for years into sometime between midnight and now.  

As shown below, those orders are orders that have been last created or closed going back to 2022.

As is visible in the updatedAt column, they were pretty much all updated at once which seems to be to a batch maintenance or update. 

I could look into creating a work around but I wonder if this is temporary (as in it was accidental) and will be rolled back ?

[Kalvin_Mizzi]

I have observed the same and it caused a lot of issues.

[fpicanzadirect]

It scrapped my whole day between investigating what was happening to cause our system being suddenly so low (because of volume of updated records being a multitude bigger than usual) and then creating a temporary work around. 

Hoping they will address the situation.

[Bas_ter_Brugge]

Same here. Our apps are processing a lot of old orders unexpectedly at the moment. 

[Kalvin_Mizzi]

@Shopify can we get some sort of explanation / resolution on this?  I think it may still be happening.

[sweetelaine]

I am experiencing the same issue and it is adding VIP Points to rewards accounts for my customers. I get messages every few minutes from customers asking what is going on and I cannot make it stop. I'd love to know what to do.

[fpicanzadirect]

I've reached out to Shopify support. They asked me for an example of where I was seeing this UpdatedAt date. I gave them the following screenshot 

Hopefully they will get to the bottom of this rapidly but I've been in a chat for a while. 

Here is the latest

[fpicanzadirect]

 I guess this is it. Not quite a solution but an end to awaiting a fix as it doesn't look like one is coming. 

• Shopify Kat: Thanks so much for waiting! Alright, so the team has let me know that the reason why we're seeing this is because a back fill job was run that appears to have triggered the update order webhooks. It shouldn't present any issues on the order itself!

• Me: Fine but those bad UpdatedAt dates are causing chaos but now we, and the other stores, do not know what is an actual update that required us to take action or report on (such a a change in items or a refund or that type of thing) vs what was updated accidentally by Shopify's developers.

• Shopify Kat: I completely understand you on that! Could you please elaborate when you say it is causing chaos? If you're running a job or task in any way affected by this, looking at the creation date rather than the updated at date of those orders might be required.

• Me: The problem with using creation date when I query the API, is that it would give me the new orders for that date. But what if on another date, say March 12, someone removes an item from the order? Then any backend process such as fulfillment ones, will not pick up that order on the date where the item was removed if we look at the creation date rather than the updated date.

  Here is another example of the impact from another store owner who wrote on my post: "I am experiencing the same issue and it is adding VIP Points to rewards accounts for my customers. I get messages every few minutes from customers asking what is going on and I cannot make it stop. I'd love to know what to do."

• Shopify Kat: Okay I see, that makes sense François! I've had our technical team look into this, and this is the only work around that they've provided. It is something that we are aware of, and I will certainly pass this feedback on to our developers, as I can definitely see how this would impact any queries.

• One last thing before you go please. Can you confirm that this was an isolated incident from March 8 that should never take place again or whether this is now something that could occur once in a while from now on?

  I mean the accuracy of the updatedat property

• Shopify Kat: As much as I would love to be able to promise that, this not something I'm able to confirm. Our platform was built and continues to evolve to meet the needs of our merchants, and though I cannot guarantee this, I understand your concerns and I will certainly pass them along to our developers.

• Me: ok thanks. Have a nice end of day

• Shopify Kat: You too! You'll be sent an email transcript of our conversation today, so feel free to respond to that email, or reach back out if there's anything else we can do. Have an excellent day, and please don't hesitate to reach out if you have any more questions! We are always here to help!

[Kalvin_Mizzi]

I'm still experiencing webhooks coming in on old orders as of this a.m.

[fpicanzadirect]

Hi Kalvin,

From what you've been seeing, is the issue with the updatedAt still ongoing for you? In my case those horribles updates to updatedAt seem to have happened from March 7 to March 11. I do not find any more old orders from years pas as having the updatedAt date update either yesterday or today. Don't know if the stoppage is linked to my having a chat with the Shopify support on March 11. 

[Bas_ter_Brugge]

It stopped for us too on March 11th at approx. 22:30 CET.

[fpicanzadirect]

Thanks Bas_ter_Brugge. Fingers crossed that this won't happen again.

[Kalvin_Mizzi]

Actually it looks like it has stopped (for now)

[fpicanzadirect]

Thankfully 🙂