Changes to CarrierService API rate request timeouts - 10s -> 5s

Solved
Highlighted
Shopify Partner
53 2 19

Hey @bligneri,

Andy here from Zapiet. I'm sure you have good reasoning for this change and discussed at length internally, but i'd be doing my merchants and the community a disservice if I didn't share my thoughts. 

In short this change is going to cause our merchants major issues!

As you know we have to act as a proxy sitting the middle of the Carrier Service API request and third parties. This is so that we have the ability to only return shipping rates when the customer selected shipping, delivery rates for delivery and of course free pickup when pickup is selected. 

By far our biggest merchant complaint is the red no rates found error of doom. Reducing the response time is inevitably going to cause this to happen far more frequently. Here are some examples of why:

  1. We have zero control over the speed at which FedEx, UPS and others respond to us.
  2. We have a large number of last mile delivery integrations all of which dynamically provide rates based on distance. Distance based calculations of course take time to process as they have to first geocode the addresses before then calculating driving distances. 
  3. We have 1,000+ merchants using our integrations with @Parcel_Intellig@Sharon-Reeds @Joel-Reeds and Advanced Shipping Rules. All of which have clearly express great concern that their apps natively likely won't be able to meet this 5 second rule. Let alone when they first have to wait for the carrier service request to route through our system before they can begin processing.

Here's an example:

1 second for Store Pickup + Delivery to receive the request and send onto Intuitive Shipping
3 seconds for Store Pickup + Delivery to receive a response from Intuitive Shipping
1 second to calculate our own fallback rates and return to Shopify if Intuitive Shipping don't return rates within the 3 second window 

Within that 3 second window Intuitive Shipping (and our other integrated apps) are expected to carry out all their ranking, 3d boxing, connections to third party couriers and provide a successful response. It's just not feasible.  

We do have our own fallback rates feature. But it's only currently supports a single delivery method. Meaning the merchant has to select between fallback rates been either shipping, pickup or delivery. If a customer has selected local delivery on the storefront and no rates could be returned from Roadie we'd present the fallback rate, but that would be either a much cheaper shipping rate or worse the free pickup rate. So then merchants are going to have to either absorb the cost of the delivery, or contact each customer who has been charged incorrectly and ask that they pay the difference.

Can you imagine merchants been happy about having this seemingly unnecessary additional administrative work to do? Let alone the frustration amongst their customers? Merchants have already had it so hard this year, let's please not make it worse for them. 

If you do decide to continue with your plans and ignore the communities advice then so be it. But i'd ask that you look at disabling the App Store review feature for effected apps until you release the new API as it's going to be one hell of a mess for us all.

Kindest,

Andy Cargill
Founder of Zapiet
www.zapiet.com

 

 

 

 

Founder - Zapiet - Store Pickup + Delivery (apps.shopify.com/click-and-collect)
Highlighted
Tourist
3 0 4

TL;DR - Rate request timeout change from 10s -> 5s  plus the empty rates caching from 30s -> 15min means pissed off customers that can't check out.

 

With the new 5 second rate request timeout, there is actually another underlining issue that no one noticed.  If the rate request timeout on a CarrierService, then Shopify will not attempt to get new rates until 15 minutes later.  So if a customer is at checkout, they won't be able to check out if the rate request times out.  Shopify caches the rates for 15 minutes if it returns shipping rates, but apparently, they also cache the empty rates and rate request timeout for 15 minutes also.  Back then, it cached the empty rates for only 30 seconds if there was a rate request timeout.  We will definitely see a lot of stores that will lose a lot of customers not just because of the 10s -> 5s rate request timeout but also the 15 minutes cache of empty rates.  This 30s -> 15min caching of no rates started in June 2020, but still no fix yet since it reported it. 

If the customer cannot checkout because of empty rates, guess who the customers are going to blame?  The merchant.  After that, they will basically buy the item elsewhere because they can't wait 15 minutes to get try to get. shipping rates again.  Fallback rates don't work and apparently, Shopify want merchants to use Shopify shipping rates instead of our own custom shipping rates.

If you are an app developer, feel free to test is this is true.  Open development console and check the network timing for "aysnc_shipping_rates".  You will see that after a rate request timeout, it doesn't even attempt to send the web hook to get new rates until 15 minutes later.  

 

Highlighted
Shopify Expert
4263 32 434

Touched on by a few, but not really articulated well by any.

Let us sprinkle fairy dust and exhale cryptic vocalizations while banging turtle shells and poof! Shopify is 100% providing Shipping Rates, with accuracy, from the wide world of carriers, where they are providing the crucial dimensional weights, box sizes, and using algorithms to fit products to boxes so as to ensure the best, most economical shipping experience.

With that out of the way, there is zero chance they could offer results in < 5 seconds 100% of the time, as they are at the mercy of the external world and external Carrier API calls.

So with that out the way, they are effectively choosing to take the heat from the minority while servicing the majority. That classic 80/20 split again. Not their first rodeo as they say.

When facing the checkout step SHIPPING RATES we understand that from the customer perspective, nothing is happening that would cause them to abandon their purchase except one of two things:

1. Shopify blocks the purchase because no shipping rates are available in a reasonable amount of time

2. The rates presented are stupid expensive, and the customer chooses to walk away in disgust

My point is, in terms of capacity to process checkouts smoothly, if we abstract away everything else, there is only the load on Shopify to calculate shipping rates.

If we eliminate #2 out of a sense that there are usually acceptable rates to present, where no one loses money at the very least, we are back to #1. At this point, Shopify is capitulating to external forces in some cases, with CarrierServices, and a reasonable window of time must be set on them. This is where it gets interesting. A simple mechanism of SHOWING the customer how much time has elapsed in the reasonable window would be a start. If you present a 30 second timer to have rates to the customer, they will stick around. Countdown to zero. No rates. OK. Now you show off the dumb, default rates. Better than nothing perhaps. And within that window, rates are returned from the N expected carriers, or at least a subset of them. So if get them in 3 seconds or 13 or 24, you still get them, they are accurate, and you present them.

So the only thing stopping Shopify there is apparently their capacity to do that. It does sound like a challenge, but for a company that can process 300 or 400 orders per second, does opening and monitoring a 30 second window of opportunity present such a technical hurdle to the engineering team that is a non-starter? In 2020, with mastery of the monolith, surely the team could be coerced into this as a feature to be proud of?

Since commerce is not going away, and Carriers are an essential variable, marrying the two seems like it ought to be a priority. Yet here we are, where the true challenge is being punted further down the timeline, and in the immediate vacuum, there will be some carnage. Avoidable? Perhaps.

Interesting nonetheless. Clearly the weakest link right now in ALL of this new-age commerce, from Plus, to DTC darlings, to grey and dour Amazon, it is packing the right products in the right boxes, and bags, and getting them to the customer. FexEx, USPS, UPS, DHL et al, are all pretty functional here, and not really the problem. The real culprit in shitty service is the person packing a size 8 Red in a bag that should have a size 4 Blue, and then heaving that bag into the outbox for delivery. Until that is fixed somehow, all commerce suffers returns, cancels, and the gross expense of pissed off customers that temper their desires to shop online in the face of poor service. Not for Shopify to fix, but ultimately, since they seemingly has no internal developments to help alleviate this, it is clearly a sore point that only a huge outside interest will likely solve.

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
Highlighted
Tourist
5 0 2

I'd like to echo everyone else's frustration here. Can you at least share what's actually driving this decision?

You're essentially choosing to fail any carrier service request between 5 and 10 seconds. If that is a large fraction of requests, then you are frustrating a large number of buyers and sellers on your platform. If that is a small fraction of requests, then it seems there will be pretty limited performance benefits at the cost of frustrating a small (but potentially vocal) number of customers. Computers don't leave bad reviews.

We, of course, will not pass this frustration on to customers, but this now means that if shipping can't be calculated, we're going to fall back to some kind of "guess" as you are suggesting, and now part of life as a Shopify user means "making a system to guess shipping costs when it takes slightly too long to get the actual data".

Please reconsider.

0 Likes
Highlighted
Shopify Partner
68 1 33

Perhaps the new multi-origin shipping is driving a lot of pain for Shopify's load? I've seen on multi-location stores with multi-origin shipping turned on, if there are no rates returned for a given location/item, Shopify seemingly tries every single combination of products/locations until it gives up (i've observed no actual pattern to this, it looks like it starts trying every single combination but stops after x tries), If this is what is driving significant increases in requests, and open connections a big impact on the platform, perhaps this behaviour could be fixed, or turned off by default? A large majority of merchants who have recently switched to my app have had to turn off multi-origin shipping anyway, either it prevents how they want to setup their rates, or broke existing functionality which they had previously.

We've been through several BFCM with success, assuming there is proactive capacity management in place, i don't see why this year is so different than the previous ones?

Highlighted
Shopify Partner
46 0 9

"A root cause of merchant, partner, and customer dissatisfaction is the dependence on external calls, including usage of the CarrierService API."

I have an one-man-shop app with approximately paying 500 customer-years on it. And I'ved yet to hear anyone complain about the checkout time.

Highlighted
Shopify Partner
161 9 53

To echo Min's point, the ability to switch a shop between single /multi origin from the shipping settings would certainly reduce the large number of retries and multiple requests for split carts profiles has introduced. 

I imagine what used to be a fairly minor issue in the past regarding carrier services timing out was compounded significantly with the introduction of multi-origin shipping, especially those carrier services who are hitting the 10s cutoff repeatedly.

One solution would be adding the setting above to switch a shop to single origin. I know this would make our lives at Intuitive Shipping a lot better as we get inundated with merchant complaints about split carts due to locations. 

A second solution is to drop the retry from a second location if the first request timed out. If a carrier service simply returned with no results, then the system can perform a request from the next location, etc. However, if the first request hit the timeout, there's no reason to believe a second try from another location would yield a different outcome. 

Also, a side note to locations. The location closest to the shipping destination should be the first call made, not based on a location priority list. This is a common problem we encounter with our merchants. It's become such a problem that Andy from Zapiet and I have built this functionality into our integrations to improve the merchant experience. 

Perhaps there is an opportunity here for Shopify to host a conference call with the shipping app developers on here to work through a better solution that won't have such a large, negative impact on merchants and customers.

This change to carrier services should be put on hold until a proper solution can be arranged, as I don't see this timeout change benefiting anyone except the Shopify engineering team. 

Co-Founder / CTO @ Intuitive Shipping Inc.
Intuitive Shipping | Smart Boxing | Automate Shipping Profiles
Highlighted
New Member
2 0 3

I think this will be very bad @Shopify, if I cannot respond within the 5 seconds my merchants be very sad they will have no customers

You may want to warn them before next Wednesday.


I did this! I tell my merchants they will have no more customers on wednesday but they say they really want more customers not less customers!?! so this will not be what they want I thought you said this what merchants want?

I'm very worried that your engineers have tricked you @Shopify I don't think any merchants want this did you check if engineers are just joking? maybe ask them again and see if they are just making a joke? tell them I think it's a funny joke

Highlighted
Shopify Partner
2 0 0

In future it would be great to get notified about this stuff during business days worldwide. This was posted Saturday here in Australia so we're only seeing it today, Monday Morning and have to do a fair bit of leg work by Wednesday, if we had found out on Friday we could've had a couple extra days to prep if we needed. Not a major issue but would be good to know for future breaking changes.

0 Likes
Highlighted
Shopify Partner
68 1 33

I have a thread here on the inconsistent behaviour of multi-origin shipping and CCS

https://community.shopify.com/c/Shopify-APIs-SDKs/inconsistent-behaviour-with-new-multi-location-shi...

At the time of the post, someone from Shopify said merchants could raise a ticket with Shopify Support and ask for multi-origin shipping to be disabled (i don't remember who said this or where), however in the last few days i'm getting reports from merchants saying this option is now no longer available.

If this is the case, then merchants are being restricted at both ends of the spectrum.

Can't do complex rates (timeout)

Can't do the simplest of rates either (i.e. rate that applies across the entire cart - multi-origin shipping)

If this is the case, then engineering hasn't really weighed up the business pros and cons and likely result in a massive churn of merchants away from Shopify.

 

Let's look at some numbers here.

I have roughly 125 PLUS merchants who will be impacted by this (and many more merchants on a non-plus plan)

let's say 20% of them decide to move away due to all these problems, that's $600k USD (at $2k/mth for the PLUS plan) in lost revenue every year (not to mention the additional spend from processing fees and app subscriptions etc). And that is just from merchants using my app, add on merchants using Zapiet, Intuitive Shipping, Advanced Shipping Rate etc, I'm sure this number will increase significantly, furthermore, likely turning these business owners into detractors (whom are likely to start multiple businesses using the same platform too). From an engineering point of view, $600k buys a lot of processing power.

 

I Agree with @Joel-Reeds, some partner feedback would be greatly advantageous, personally I've found it incredibly difficult to provide feedback to decision makers. We can post on the forums, in which case we may get some sort of engagement, then perhaps forgotten? never the less, there isn't really any follow up to make sure they really understand the problem either. We can raise it via partner support, and in most cases it hits a dead end because the support staff themselves have no ability to escalate the issue to anyone internally either. I fear the new CCS upgrades may fall short of providing what's actually needed to provide for what merchants want as well, an ability to workshop through the issues will greatly improve Shopify's overall capabilities as well as introduce some efficiencies.