API calls are being blocked

Amber_Walker
Excursionist
12 0 3

We have a script running on our production web server that syncs products and orders using Shopify's API.

This has been running smoothly for years, but starting over the weekend, all network traffic from our web server (and only our web server) started failing. The failures are so complete that even a simple ping of the server fails:

BrokenPing.png

My first guess was that traffic was being blocked by a firewall that was preventing outgoing traffic from the web server, but after a conversation with the hosting company's support and a few traceroute calls, it became clear that traffic was making it to shopify's servers and dying.

This indicates to me that Shopify has blacklisted our web server's IP address.

We need this block removed. 3-4 hours with Shopify support was thoroughly unproductive. Is there someone from Shopify that can possibly assist? It is a devastatingly blocking issue.

Replies 6 (6)
Amber_Walker
Excursionist
12 0 3

To provide additional information beyond what is in the screenshot, the public IP address of the server in question is 64.111.127.92.

DrT
Tourist
4 0 3

Hello Amber, did you figure anything out with this? I am experiencing the same issue and it started on the same day as yours. I've had a process running for years that suddenly stopped from my server. I am on Dreamhost Whitfield server and I can't seem to figure out what is going on PING fails and trace-route fails at Shopify. Same thing with Shopify support and no help on the forums yet. I see you posted a few days ago with no results. This is where Shopify support sent me and were very unhelpful.

Luckily I can run these jobs from my home machine - but having a cron job makes my life so much easier as I wear a lot of hats for our business.

 

 

Amber_Walker
Excursionist
12 0 3

That is very interesting that you're having the same problem, so thank you for sharing--it seems that means this is a larger problem that Shopify (for whatever reason) doesn't seem to think is a high enough priority to warrant a response.

Their quote-unquote "tech team" from chat support claimed that the IP was not blocked, but these are the same people that asked me if I had tried, "clearing my cache or using incognito mode," so I'm 100% confident that they had no f---ing idea what they were talking about.

Shopify needs to fix this problem. There is absolutely no excuse for it going on this long.

Amber_Walker
Excursionist
12 0 3

Once again, Shopify is doing this again.

 

Heads up to the developers at Shopify: The problem is CloudFlare blocking API calls coming from Dreamhost's servers. Here's the traceroute:

[troup]$ traceroute kids-consign-it.myshopify.com

traceroute to kids-consign-it.myshopify.com (23.227.38.74), 30 hops max, 60 byte packets

1  ip-64-111-124-1.dreamhost.com (64.111.124.1)  2.337 ms  2.222 ms  2.156 ms

2  ip-208-113-156-48.dreamhost.com (208.113.156.48)  1.167 ms ip-208-113-156-13.dreamhost.com (208.113.156.13)  0.214 ms ip-208-113-156-48.dreamhost.com (208.113.156.48)  1.051 ms

3  iad1-bdr-1.sd.dreamhost.com (208.113.156.193)  0.277 ms  0.224 ms  7.735 ms

4  ae-0-0-bdr1-iad1.dreamhost.com (208.113.156.2)  0.252 ms  0.405 ms  0.349 ms

5  be5698.rcr51.b037327-0.iad02.atlas.cogentco.com (38.122.62.253)  0.574 ms  0.564 ms  0.861 ms

6  be3563.ccr22.iad02.atlas.cogentco.com (154.54.25.225)  1.036 ms  1.170 ms  1.204 ms

7  38.32.185.178 (38.32.185.178)  2.230 ms  2.204 ms 38.88.214.142 (38.88.214.142)  1.668 ms

8  172.70.32.4 (172.70.32.4)  1.098 ms  1.183 ms 172.70.32.2 (172.70.32.2)  1.233 ms

9  * * *

10  * * *

11  * * *

12  * * *

13  * * *

14  *^C

 

Any chance you could fix this? Again?

DrT
Tourist
4 0 3

Oooof... I can't believe that is happening again. Same thing is happening to my Dreamhost scripts. Luckily I just moved them to the Google Cloud last week - so they are working on Gcloud and not on Dreamhost. I love dreamhost, but ended up moving because of the very weak mail system, the recent price increases, and this issue. Looking at it now, I'm glad I did. I'll be shutting down my dreamhost VM soon.

Amber_Walker
Excursionist
12 0 3

Looks like this was fixed again. A working traceroute shows the difference:

 

[troup]$ traceroute kids-consign-it.myshopify.com

traceroute to kids-consign-it.myshopify.com (23.227.38.74), 30 hops max, 60 byte packets

1  ip-64-111-124-1.dreamhost.com (64.111.124.1)  1.936 ms  1.869 ms  1.840 ms

2  ip-208-113-156-13.dreamhost.com (208.113.156.13)  0.201 ms ip-208-113-156-48.dreamhost.com (208.113.156.48)  0.242 ms ip-208-113-156-13.dreamhost.com (208.113.156.13)  0.165 ms

3  iad1-bdr-1.sd.dreamhost.com (208.113.156.193)  1.301 ms  0.232 ms  1.170 ms

4  ae-0-0-bdr1-iad1.dreamhost.com (208.113.156.2)  0.386 ms  0.381 ms  0.293 ms

5  be5698.rcr51.b037327-0.iad02.atlas.cogentco.com (38.122.62.253)  0.511 ms  0.487 ms  0.468 ms

6  be3563.ccr22.iad02.atlas.cogentco.com (154.54.25.225)  1.243 ms  1.061 ms  1.112 ms

7  38.88.214.142 (38.88.214.142)  1.452 ms  1.806 ms 38.32.185.178 (38.32.185.178)  11.140 ms

8  173.245.63.243 (173.245.63.243)  0.708 ms  0.665 ms 172.70.172.2 (172.70.172.2)  1.450 ms

9  23.227.38.74 (23.227.38.74)  0.476 ms  0.517 ms  0.506 ms

[troup]$

 

With the successful CDN trace looking like:

 

[troup]$ curl https://kids-consign-it.myshopify.com/cdn-cgi/trace

fl=413f358

h=kids-consign-it.myshopify.com

ip=64.111.127.92

ts=1642265905.838

visit_scheme=https

uag=curl/7.58.0

colo=IAD

http=http/2

loc=US

tls=TLSv1.3

sni=plaintext

warp=off

gateway=off

 

As it was the last time, the problem appears to have been be with Cloudflare blocking the traffic. It seems the problem came immediately after host 172.70.32.4.

 

The connection is now working because it is being routed through a different host (173.245.63.243) after which it makes it to its final destination--a CDN node in Ontario.

 

I'm going to keep using this thread as (a) a log of everything I discover when it is going wrong, and (b) a testament to the ineptitude of the Shopify developers.

 

That second one may seem harsh, but they are continuing (to this very day) insist that the problem MUST be on my end ... as though somehow my code extends beyond the Dreamhost datacenter, beyond Cogentco's network (a 3rd party ISP), and into Shopify's CDN network. They just don't want to investigate what is going on.