FROM CACHE - en_header

Urgent: Are you experiencing issues with PUT/POST Product API right now?

Solved
Jason27
Shopify Partner
84 2 29

The issue started a few hours ago for all my apps. Feels like the Shopify Rest Api is partly broken down. My apps process tens of thousands of products a day and is on autopilot for months. But this issue just started today when updating Products and it's not returning any error Status Code...?!

It seems to be related to product photos..

Accepted Solution (1)

Accepted Solutions
CalD
Shopify Staff
Shopify Staff
140 21 33

This is an accepted solution.

Following up - Cloudflare has deployed a rollback which, from our preliminary testing, looks to have fixed the issue.

Let us know if this behaviour lingers (and/or if you can confirm the issue is resolved)!

CalD | Developer Support @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

View solution in original post

Replies 26 (26)
MikeDe
Tourist
6 0 3

Same here, the issue started occurring at 8 AM UTC. Can't figure out what is causing it.

It is not related just to Products endpoint, the SubscriptionContracts and SellingPlanGroups are also affected.

Jason27
Shopify Partner
84 2 29

Shopify App Team must be heading back to the office within the next hour or so. Try to get a hold of them! I've had to put all my apps into repair mode until this issue is solved. This is a Code Red situation.

MikeDe
Tourist
6 0 3

For us, the issue happens with GraphQL requests larger than 1.3 MB. It seems as if nothing is returned from Shopify when this happens, as we experience a timeout and if we increase the timeout, we get this error (from curl):

OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104

But the issue could be something completely else.

The interesting thing is that GraphQL requests with payloads smaller than 1.3 MB are processed without any issues.

Please let me know if you figure out what the solution is.

 

Eric_Boisjoli
Shopify Partner
21 0 4

Are you hosting in GCP Northamerica?

Jason27
Shopify Partner
84 2 29

GCP = Google Cloud?

I'm using google cloud as an image server, and it seems to be working ok. The issue I'm facing seems to be using PUT on Product Image endpoing using the ATTACHMENT property (base64 encoding). Can anyone else test this?

 

 

policenauts
Shopify Partner
194 9 57

I got one internal server error 500 error on an API request this morning when trying to delete a product, but otherwise my REST / GraphQL Product API requests have been processing without issue. I am using Google Cloud Functions, us-central1. 

kyle_truong
Shopify Partner
50 6 8

Been getting it since the morning too, none of my requests in production to Shopify are working:

request to https://<mydomain>/admin/api/2021-07/graphql.json failed, reason: socket hang up","error":{"message":"request to https://<mydomain>/admin/api/2021-07/graphql.json failed, reason: socket hang up","type":"system","errno":"ECONNRESET","code":"ECONNRESET"}



MikeDe
Tourist
6 0 3

Thank you Kyle, good to know that we are not alone in this issue.

We are hosting in Canada, and all of our GraphQL queries are failing similarily as Kyle's. It seems that the GraphQL endpoint resets the connection without returning any response.
The interesting thing is that the Query goes through if we reduce the payload (the query itself) to be less than 1.3 kB.

 

Jason27
Shopify Partner
84 2 29

I think you're on to something with the 1.3KB limit idea. I'm using the Attachment property and pasting a ton of base64 chars, surely much larger than 1.3KB.

Anybody else with this issue??

Eric_Boisjoli
Shopify Partner
21 0 4

Cool I'm seeing something similar related to being in Google's Canada region and having a larger payload. smaller payloads and being out of the canada google region seems to work for me

Jason27
Shopify Partner
84 2 29

How does the Google Canada angle work? I'm storing my photos on Google (I think it's Northeast North America), but I pull the data from the saved image, which works fine, and encode it to base64 and then PUT to Shopify Api using product image endpoint inside attachment property. It seems to return nothing at all, like a timeout. Is Shopify using Google too? Personally, I feel like Shopify pushed some update to their servers today at 8am UTC and it's caused some issue. If I'm wrong, please explain

Eric_Boisjoli
Shopify Partner
21 0 4

if my api request originate from northamerica-northeast1 to shopify they fail. 

 

the same request done from my laptop succeeds or in a different region it also succeeds

heindrik
New Member
2 0 2

This might be a region thing. We had our servers hosted in Canada and moved it to US East and the requests are no longer timing out.

Kris_Doyle
Shopify Partner
100 0 25

We're experiencing the same issue. Regional seems very likely as we're deployed in Eastern Canada as well. Oddly enough, it affects GETs intermittently and POSTs consistently.

Update: We just deployed to a location in the US-East region and calls are succeeding now so it does seem to be a regional issue.

MikeDe
Tourist
6 0 3

It would be great if someone from Shopify can take a look at this, as this is seriously impacting the usability of the GraphQL API.

It doesn't seem to me that it is region specific. It looks more like a limit on certain IPs, which means that probably even if you move your server to a different region, you will start experiencing the same issue a few days later.

Jason27
Shopify Partner
84 2 29

There's somone at Shopify app support taking a look at this now. He's thinking it has to do with Cloudflare. I think he's coming to the conclusion based on the comments in this thread, since he's not able to reproduce the issue. Can anyone provide any way to reproduce? Can anyone confirm if moving server out of Canada works for sure? I have my app server on Digital Ocean TOR1, and it experiencing those intermittent API timeouts described above.

heindrik
New Member
2 0 2

@Jason27 I have a similar infrastructure where I have Cloudflare as the DNS and Digital Ocean on TOR1 and had the following error.

Errno::ECONNRESET (Connection reset by peer)

This is when we made a POST request to /orders, the GET request seemed to work fine.

As a temporary solution, I moved the app to Heroku (United States) and haven't had any connection issues yet. I was going to move my DO instance to NYC to see if that solved the issue.

Hope that helps you a bit.

CalD
Shopify Staff
Shopify Staff
140 21 33

Hi all,

Our engineering teams are aware and are currently investigating the issue. It seems to affect app servers in the region of eastern Canada/Toronto only and, as others have mentioned, redeploying to a nearby geography can serve as a workaround for now.

This issue is likely external to our systems, and we've escalated where we can to both Cloudflare and GCP support.

We'll provide an update here when possible.

CalD | Developer Support @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

CalD
Shopify Staff
Shopify Staff
140 21 33

This is an accepted solution.

Following up - Cloudflare has deployed a rollback which, from our preliminary testing, looks to have fixed the issue.

Let us know if this behaviour lingers (and/or if you can confirm the issue is resolved)!

CalD | Developer Support @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit Shopify.dev or the Shopify Web Design and Development Blog