I'd actually be very curious if anyone else in this thread had issues on May 23rd. Ask your support staff & check your logs to check and see if people were complaning about shipping in checkout.
For me, here's the times when I noticed the issue:
2018-05-23 05:00 UTC
2018-05-23 09:00 UTC
4 Hours total. Check your logs if you have any to see if you guys also had this issue, so we can see if it's an issolated incident or if it also occured on May 23rd.
Hey everyone. We pushed out a change last night intended to mitigate a lot of the issue and it appears to have done so (although I can see some are still experiencing the issue).
For those who this is still affecting, could you let me know what type of server your rates engine is running on and, if possible, where it is located?
I also want to be completely sure that each case here is a matter of not receiving requests for shipping rates in general. Please tell me if otherwise.
Hey Dag, thanks.
Our engineers are exploring solutions to resolve this for remaining cases. Unfortunately it's not as simple as it sounds. I can now shed some light on the cause:
This is mostly occurring because we're failing to establish a connection to your server. At first, the timeout threshold was 2 seconds. Over the past 24 hours we have upped it 5, 10, and now 15 seconds. Your integration (and most of the remaining others) in particular is unable to have a connection established to it within that 15 seconds, and it doesn't appear to be occurring because of anything you're in control of, of course.
Thanks for the patience here, I get how strenuous this is on integrations like yours and we're aware of it. Your success is our merchants' success is Shopify's success. I'll post back with updates as they become available.
Thanks for the update. I know the fix might not be simple, but this has hopefully demonstrated the need to better monitoring and alerting in this area.
- Monitoring the running average percentage of connections broken for carrier services
- Any jump in this average should alert your technical staff
- And customers should be notified
Whenever I as a customer, have to tell you about such an issue, and then customer service not even taking it seriously, I lose a lot of trust in your ability to keep my shop online. And keeping it online is what I pay all this money for.
@Alex mine is hosted on AWS us-east-1. Using Lambda. Requests are served in under 50ms.
Should be able to reach that in 2 seconds, so I assume there's some issue on Shopify's end with outgoing connection load which is causing this to take longer than 2 seconds?
Could you clarify why connections are taking longer than 2 seconds all of a sudden, when I assume they did not before.
My last error window was 2018-05-31 00:00:00 UTC to 2018-05-31 02:00:00 UTC, since then it's been working it seems.
I've still got my temporary shiipping options in place and am looking for a firm "this is resolved" moment until I can remove them. So please let us know when we think this is actually resolved. Because it appears the issue is still happening after incident report said it was fixed.