Custom Carrier Service not requesting rates from our carrier callback url

Trailblazer
195 0 84

I'd actually be very curious if anyone else in this thread had issues on May 23rd. Ask your support staff & check your logs to check and see if people were complaning about shipping in checkout.

 

For me, here's the times when I noticed the issue:

 

2018-05-23 05:00 UTC

to

2018-05-23 09:00 UTC

4 Hours total. Check your logs if you have any to see if you guys also had this issue, so we can see if it's an issolated incident or if it also occured on May 23rd.

0 Likes
Shopify Partner
43 0 5

Three shops reporting issues so far today - at 13.15 local time. 

0 Likes
Shopify Partner
6 0 0

This is completely unprofessional by Shopify. No follow-up and declaring it as "resolved" when it clearly was not.

0 Likes
Shopify Partner
43 0 5

Wow, the internet should have stayed in bed today, now the carrier I'm forwarding calls to was down today too. So the three shops not working might have been caused by that. . 

0 Likes
Highlighted
Shopify Partner
6 0 0

I can confirm that both apps (yours and my custom app) keep falling out today as well, so same as yesterday.

Edit: Just to clarify, I do not forward any calls to Bring, I do all the calculations in code

0 Likes
Shopify Staff
Shopify Staff
1554 77 232

Hey everyone. We pushed out a change last night intended to mitigate a lot of the issue and it appears to have done so (although I can see some are still experiencing the issue).

For those who this is still affecting, could you let me know what type of server your rates engine is running on and, if possible, where it is located?

I also want to be completely sure that each case here is a matter of not receiving requests for shipping rates in general. Please tell me if otherwise.

0 Likes
Shopify Partner
6 0 0

Mine in running in AWS Frankfurt. It does not show up in my logging so definitely requests are not coming through to my server

0 Likes
Shopify Staff
Shopify Staff
1554 77 232

Hey Dag, thanks.

Our engineers are exploring solutions to resolve this for remaining cases. Unfortunately it's not as simple as it sounds. I can now shed some light on the cause:

This is mostly occurring because we're failing to establish a connection to your server. At first, the timeout threshold was 2 seconds. Over the past 24 hours we have upped it 5, 10, and now 15 seconds. Your integration (and most of the remaining others) in particular is unable to have a connection established to it within that 15 seconds, and it doesn't appear to be occurring because of anything you're in control of, of course.

Thanks for the patience here, I get how strenuous this is on integrations like yours and we're aware of it. Your success is our merchants' success is Shopify's success. I'll post back with updates as they become available.

Cheers.

0 Likes
Shopify Partner
6 0 0

Thanks for the update. I know the fix might not be simple, but this has hopefully demonstrated the need to better monitoring and alerting in this area.

I suggest:

- Monitoring the running average percentage of connections broken for carrier services
- Any jump in this average should alert your technical staff
- And customers should be notified

Whenever I as a customer, have to tell you about such an issue, and then customer service not even taking it seriously, I lose a lot of trust in your ability to keep my shop online. And keeping it online is what I pay all this money for.

0 Likes
Trailblazer
195 0 84

@Alex mine is hosted on AWS us-east-1. Using Lambda. Requests are served in under 50ms.

Should be able to reach that in 2 seconds, so I assume there's some issue on Shopify's end with outgoing connection load which is causing this to take longer than 2 seconds?

Could you clarify why connections are taking longer than 2 seconds all of a sudden, when I assume they did not before.

My last error window was 2018-05-31 00:00:00 UTC to 2018-05-31 02:00:00 UTC, since then it's been working it seems.

I've still got my temporary shiipping options in place and am looking for a firm "this is resolved" moment until I can remove them. So please let us know when we think this is actually resolved. Because it appears the issue is still happening after incident report said it was fixed.

0 Likes