Urgent: Are you experiencing issues with PUT/POST Product API right now?

[Jason27]

The issue started a few hours ago for all my apps. Feels like the Shopify Rest Api is partly broken down. My apps process tens of thousands of products a day and is on autopilot for months. But this issue just started today when updating Products and it's not returning any error Status Code...?!

It seems to be related to product photos..

[MikeDe]

Same here, the issue started occurring at 8 AM UTC. Can't figure out what is causing it.

It is not related just to Products endpoint, the SubscriptionContracts and SellingPlanGroups are also affected.

[Jason27]

Shopify App Team must be heading back to the office within the next hour or so. Try to get a hold of them! I've had to put all my apps into repair mode until this issue is solved. This is a Code Red situation.

[MikeDe]

For us, the issue happens with GraphQL requests larger than 1.3 MB. It seems as if nothing is returned from Shopify when this happens, as we experience a timeout and if we increase the timeout, we get this error (from curl):

OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104

But the issue could be something completely else.

The interesting thing is that GraphQL requests with payloads smaller than 1.3 MB are processed without any issues.

Please let me know if you figure out what the solution is.

[Eric_Boisjoli]

Are you hosting in GCP Northamerica?

[Jason27]

GCP = Google Cloud?

I'm using google cloud as an image server, and it seems to be working ok. The issue I'm facing seems to be using PUT on Product Image endpoing using the ATTACHMENT property (base64 encoding). Can anyone else test this?

[policenauts]

I got one internal server error 500 error on an API request this morning when trying to delete a product, but otherwise my REST / GraphQL Product API requests have been processing without issue. I am using Google Cloud Functions, us-central1. 

[kyle_truong]

Been getting it since the morning too, none of my requests in production to Shopify are working:

request to https://<mydomain>/admin/api/2021-07/graphql.json failed, reason: socket hang up","error":{"message":"request to https://<mydomain>/admin/api/2021-07/graphql.json failed, reason: socket hang up","type":"system","errno":"ECONNRESET","code":"ECONNRESET"}

[MikeDe]

Thank you Kyle, good to know that we are not alone in this issue.

We are hosting in Canada, and all of our GraphQL queries are failing similarily as Kyle's. It seems that the GraphQL endpoint resets the connection without returning any response.
The interesting thing is that the Query goes through if we reduce the payload (the query itself) to be less than 1.3 kB.

[Jason27]

I think you're on to something with the 1.3KB limit idea. I'm using the Attachment property and pasting a ton of base64 chars, surely much larger than 1.3KB.

Anybody else with this issue??

[Eric_Boisjoli]

Cool I'm seeing something similar related to being in Google's Canada region and having a larger payload. smaller payloads and being out of the canada google region seems to work for me

[Jason27]

How does the Google Canada angle work? I'm storing my photos on Google (I think it's Northeast North America), but I pull the data from the saved image, which works fine, and encode it to base64 and then PUT to Shopify Api using product image endpoint inside attachment property. It seems to return nothing at all, like a timeout. Is Shopify using Google too? Personally, I feel like Shopify pushed some update to their servers today at 8am UTC and it's caused some issue. If I'm wrong, please explain

[Eric_Boisjoli]

if my api request originate from northamerica-northeast1 to shopify they fail. 

the same request done from my laptop succeeds or in a different region it also succeeds

[heindrik]

This might be a region thing. We had our servers hosted in Canada and moved it to US East and the requests are no longer timing out.

[Kris_Doyle]

We're experiencing the same issue. Regional seems very likely as we're deployed in Eastern Canada as well. Oddly enough, it affects GETs intermittently and POSTs consistently.

Update: We just deployed to a location in the US-East region and calls are succeeding now so it does seem to be a regional issue.

[MikeDe]

It would be great if someone from Shopify can take a look at this, as this is seriously impacting the usability of the GraphQL API.

It doesn't seem to me that it is region specific. It looks more like a limit on certain IPs, which means that probably even if you move your server to a different region, you will start experiencing the same issue a few days later.

[Jason27]

There's somone at Shopify app support taking a look at this now. He's thinking it has to do with Cloudflare. I think he's coming to the conclusion based on the comments in this thread, since he's not able to reproduce the issue. Can anyone provide any way to reproduce? Can anyone confirm if moving server out of Canada works for sure? I have my app server on Digital Ocean TOR1, and it experiencing those intermittent API timeouts described above.

[heindrik]

@Jason27 I have a similar infrastructure where I have Cloudflare as the DNS and Digital Ocean on TOR1 and had the following error.

Errno::ECONNRESET (Connection reset by peer)

This is when we made a POST request to /orders, the GET request seemed to work fine.

As a temporary solution, I moved the app to Heroku (United States) and haven't had any connection issues yet. I was going to move my DO instance to NYC to see if that solved the issue.

Hope that helps you a bit.

[CalD]

Hi all,

Our engineering teams are aware and are currently investigating the issue. It seems to affect app servers in the region of eastern Canada/Toronto only and, as others have mentioned, redeploying to a nearby geography can serve as a workaround for now.

This issue is likely external to our systems, and we've escalated where we can to both Cloudflare and GCP support.

We'll provide an update here when possible.

[CalD · Accepted solution]

Following up - Cloudflare has deployed a rollback which, from our preliminary testing, looks to have fixed the issue.

Let us know if this behaviour lingers (and/or if you can confirm the issue is resolved)!

[Kris_Doyle]

Just pointed back to the Toronto instance and everything looks like it's back to normal. Thanks very much!

[kyle_truong]

Working as expected for my app as well. Thanks for fixing the issue!

[Jonathan-HA]

Just confirmed it's back to normal on our side now as well!

Thank you!!

[Jason27]

Confirmed. Thank you for rapidly working with us on this!

[MikeDe]

It works!

Thank you for taking care of this so fast. We appreciate it.

Have a great day!

[abhishektaparia]

Hi I am facing similar issue -
Please check - https://community.shopify.com/c/shopify-apps/facing-socket-hang-up-exception/m-p/1555416#M48785

[desoleary]

Do you mind outlying how switching regions can be achieved so quickly? Is it a kubernetes command etc.