UPDATE - Nov 3, 2020:
TL;DR: Shopify will be moving to a dynamic timeout based on the number of requests made per minute.
TL;DR - We will not ship this change. The timeout will remain at 10 seconds.
--- ORIGINAL POST BELOW (NO LONGER VALID) ---
TL;DR - The timeout for responding to rate requests from Shopify is being lowered to 5s on Wednesday, September 30th.
BFCM 2020 is exceptional. We have already experienced unprecedented traffic levels on our platform and as trusted partners, you certainly have had to scale your infrastructure, and are busy doing BFCM preparations to ensure a smooth and incident-free holiday season. As we continue to prepare for BFCM 2020, Shopify is looking to optimize the overall performance of checkout. A root cause of merchant, partner, and customer dissatisfaction is the dependence on external calls, including usage of the CarrierService API.
To protect buyers, merchants, partners, and the platform, we're lowering the timeout for CarrierService API rate requests from 10 seconds to 5 seconds.
Dealing with timeouts - Backup rates
We understand that custom rate calculations are complex, and can often depend on even more external factors that are invisible to Shopify. In order to gracefully handle the timeout, and avoid returning no rates at checkout, we recommend you configure your app to return a backup rate when approaching the time limit.
To prepare for returning a backup rate, we add a new field to our app’s configuration, which captures the backup rate the merchant would like to surface.
During rate calculation, if the total process for a specific rate request is approaching 5 seconds, we refer to our previous field and return that. Alternatively, if our rate calculation provides us with information as it processes, we can return our current best guess.
Both of these will return a less accurate rate than under optimal conditions, but will not timeout, allowing the checkout to complete.
We expect to ship this change to all our rate requests on Wednesday, September 30th.
We encourage you to implement fallback rates and test with our current timeout as part of your BFCM 2020 prep.
Solved! Go to the solution
This is an accepted solution.
TL;DR - We will not ship this change. The timeout will remain at 10 seconds.
We reviewed our analytics report showing rate request response times and found an error. This led us to believe that the impact of this change was much smaller than it was. This is unacceptable, and we will be reviewing our internal processes to ensure a better developer experience for our partners.
To that end, we are reconsidering our approach and will NOT be rolling this out.
The timeout will remain at 10s.
We understand that most carrier service integrations are using these 10 seconds to process important tasks for the merchants, and provide the richest experience. We are returning to the drawing board to identify better ways to improve the performance of the platform while still enabling apps to build the key features that the merchants need.
I want to thank everyone who provided extremely detailed feedback on this change. The breakdown of use cases that were posted in this thread showcased very valid concerns, and we’ve read through every reply to ensure a better understanding of the usage of this endpoint.
Is there any way to extend the date to give developers a bit more time here? Our average calculation time is 1 second or less for live rates & rules, so in a regular use case this is completely fine. However, we have a significant number of users who use Intuitive Shipping for very complex shipping setups, with a large proportion being Plus stores who require granular control. We've been given 4 days to create a timeout fallback in our apps, add it into our UIs, and to educate our affected users on the new changes and what an effective fallback rate would be in their store. It's simply not enough time to educate and implement this for merchants, especially considering we are entering the weekend.
Echo "We've been given 4 days to create a timeout fallback in our apps, add it into our UIs, and to educate our affected users on the new changes and what an effective fallback rate would be in their store. It's simply not enough time to educate and implement this for merchants, especially considering we are entering the weekend."
Ha. I can almost guarantee that this decision will result in the following.
Any merchants using and depending on Carrier Service requests that are at all complex and take 5001 ms or more to respond will simply die. And when they complain, the complaints will 100% be directed at Shopify. And so it goes.
There is a puny to approaching zero chance App developers will respond by implementing this quaint little pseudo-pattern as presented. Makes no difference to Plus or non-plus... it is simply hardcore.
As the author of ~20 CustomCarrier Apps of various age, ability and utility, I am pretty sure I am going to just ignore this change, and let the merchants scream at Shopify. Not my gig to jump to the pump for what has never been a problem to date. All the previous BFCM dates in the past came and went with exactly zero complaints. Well, let's see if 2020 is a repeat!
Making popcorn now.
Could you please reconsider this or consider making the request timeout configurable per shop? Not to mention the lack of notice here is terrible especially that this means many shops would timeout and show nothing for shipping and thus halt checkout without CarrierService developers even being able to show an error message.
Also realistically what is the point of this? Shopify currently retries CarrierService calls 3-4 times if it hits the timeout. You are really telling developers to create some sort of complex rate resumption/order lookup(btw we still don't get a checkout id when rates are requested) in order to get up to `timeout * max retries` to calculate a rate. We haven't had to implement this yet but it doesn't sound fun especially when you have to sync requests from multiple servers.
Our app FreightChick connects 20+ LTL carriers to Shopify and often requires complex 3d bin packing calculations to figure out the final dimensions packages/pallets/etc. Bin packing even with bleeding edge algorithms and methods can take several seconds when there are many items. This means you already only have 3-5 seconds on a bad(400-500+ item) order before you have to cut off any pending rates and just pick from the ones which returned quickly. We do this currently but it'd basically mean for the large orders we'd have almost no time. Also the LTL industry as a whole is well on its way towards only allowing volume/dimensional rates which require 3d bin packing.
We developed the Shopify integration for a Shopify Plus customer who also runs several other shops and we plan to launch our public app soon. You are making it harder and harder to integrate shipping with Shopify. We can't return an error message, get a checkout id, or stream rates as they come back from carriers(1 carrier service limit per app + you'd have to sync all the pre rate calculations like bin packing between multiple requests unless you actually support streamed responses).
You're seriously springing notice of a midweek API change on the preceding Friday afternoon?
That is management for you! Release the new code to production on Friday at 5:00pm! Live large!
I would've approached this by making it opt-out with a simple tick-box. Since Shopify knows if a shop has CarrierServices registered, they could simply offer a checkbox to allow the reduction to 5 second or NOT, especially since for a minority of important customers, that might be a deal breaker!
But alas, I am not in charge, so I sit by and once again marvel at the internal machinations. Simply marvelous!
I think this should be a opt out option as well, as an app that runs thousands of stores, I can tell you now you will have hundreds of merchants left out in the cold and unable the transact, many of them plus customers, you need to consider the fact that some merchants have complex rates and needs to look up several live carrier rates, this often takes over 5seconds.
Hey Shopify Shipping Team,
Unfortunately, at this point, we haven't been given enough notice to implement a fallback rates feature as a knee jerk reaction to this last-minute decision. All we can do now is reach out to the merchants who may be impacted and try to find ways to streamline their setup.
What's confusing is that one of the reasons for making this decision is for the betterment of the merchant. From experience, merchants who require complicated shipping expect the checkout to be slightly slower. The trade-off is accurate shipping calculations ensuring they remain profitable at the end of the month. If this decision was really in the best interest of the merchant, the decision would be left up to the merchant to make. If an app that they're using for shipping is too slow, they have every reason to switch to a faster app. This keeps app developers competitive.
This decision, however, will negatively impact merchants because they're not given a say in this process at all. This is being forced on them and the consequence will be that they're unable to make sales because their shipping solution may not surface rates fast enough. What tends to be overlooked with changes like this is that not all merchants fit inside a tidy little box. There are many outliers, and those outliers are the ones most likely to be using carrier services.
We have invested a significant amount of time and resources to streamline our entire rating process, but at the end of the day, we're still at the mercy of external services such as Shopify's Admin API and third-party carriers which consume over 90% of our current calculation times. If the goal is to reduce latency, we need a better version of carrier services that includes additional information such as customer tags, etc. This will reduce overhead both on carrier services as well as Shopify's Admin API.
Since we're discussing BFCM, has there been consideration to take into account that BFCM sees a spike in server demand for everyone, including third-party carriers, seeing an increase in their own response time. This naturally causes our response times to climb which would cause our system to be unable to return rates in time, preventing the end customer from completing checkout and the merchant losing a sale. This is not a better experience for either merchant or customer.
Ultimately, I believe a better approach would be to educate merchants on checkout speed. The faster the checkout, the better the customer experience, which results in a higher conversion rate.
FedEx and UPS endpoints have been quite flakey in the past month or so too, I can imagine the spike of users come BFCM will cause merchants that run under the limit now be impacted once this change is out through too. A lot of merchants is already dissatisfied with the level of support of shipping rates in Shopify, this will likely be the final nail in the coffin which forces then to leave Shopify for good, its a lose lose for everyone.
Merchants who are forced to simplify their rates will likely lose money on shipping too, or lose customers if their rates aren't competitive enough, multiply this by volume there are significant losses here.
There is lots that can be done to improve the carrier service API so these requests can be made more efficient too, however there has been absolutely no improvement in at least the past 7 years, It would be a much better approach to give us the right tools first before taking away the capabilities.
The funny thing is, if there is really dissatisfaction out there, it exhibits itself as spikes showing delays, which are connected to shops. So we know the shops exhibiting delay. Now we can analyze why those shops exhibit this delay.
So instead of throwing out the baby with the bathwater, why not work with the issue at the source, with those Apps and/or shops suffering?
And secondly as per the other complaints, over the past 14 years, other than releasing Carrier Services as a thing, pretty much nothing has ever been done to improve it from the Shopify end. The payload sent to Apps remains stripped of value (no clue about discounts for example), tags, or cart attributes. Additionally, when customers make errors, and the Carrier Service discovers it, there is zero way to transmit this important information back to the customer.
So while it all sounds nice to pump up this BFCM hype, this hatchet falling so soon, so fast, and without much recourse just screams amateur hour. Had Shopify been working on Improving Carrier Services for years, interacting with the partners and community serving that important niche, this NEVER would've been a thing.
Sorry for the short notice. We are doing this because of production engineering team, kernel team and shipping team believe that 10s is causing too much stress at the Shopify platform level.
-> With the crazy traffic we are all experiencing, a slow carrier service API user can impact negatively multiple stores and multiple key processes.
The backup rate solution is not mandatory and we are well aware that the timeline to implement, test and roll-out for all your customer can be problematic.
This being said, as stated in the message, we have no visibility on apps and what is very hard for an app could be easy to implement for another.
Other things that could be done:
- scale faster your infrastructure (if you have auto-scaling)
- adjust your alerts to ping you when rates start to slow down (4.5 seconds)
Thanks for your understanding.
Please let us know how we can support you further.
This is simply the worst global change Shopify could make to its developer community and the stock will suffer gravely for it. The reason why merchants pay the extra $20 monthly or upgrade their plan for the carrier rate option is because the default platform shipping is elementary, to say the least. Complex 3rd party solutions on average require over 5 seconds to return shipping rates .... especially for large orders - This will hurt the merchant ............the developers................THE STOCK!!!!!!!!!!
Sorry about the short timeline.
The fallback rate is a proposal: it may be possible for some partners to implement rapidly and it may be impossible/highly impractical for others this close to BFCM.
Please let us know how we can help supporting you.
I think you're missing the point that the timeouts aren't really related to our app platforms, but rather we grab rates from 3rd party carriers, or many 3rd party carriers, it is impossible for us to make our platforms run faster.
I've just run some numbers, and i can tell you now, by wednesday, you will have at least 500 very unhappy merchants complaining, many of which are PLUS merchants because they have lost tens of thousands of potential customers - or lost significant amounts of money because they have had to subsidise shipping with fallback rates. Now 500 might not seems like much in the grand scheme of things, however we can't just ignore the minority here..
A lot of merchants here do ship wholesale orders, hence the reason for a need to obtain multiple carrier rates and toe compare rates, shipping is highly competitive in ecommerce and is often the point of difference for whether a merchant makes a sale or loses out to a competitor.
Understand that there will be increased traffic, however it sounds more or less an engineering issue that has been put off for far too long and should have been addressed by the engineering team rather than such a knee jerk reaction at the last moment. Myself and other app developers have been trying to get partner engagement so we can provide feedback, however there is almost no avenue to do this other than the forums, and when we do voice our concerns, nothing happens. I've just been on partner support and when i raised the issue, no one over there has heard of this change, and ended up hitting a brick wall because there was literally no one they could escalate the issue to.
At the moment, we have no other choice but to notify all our merchants that their rates are likely to be impacted in less than 4 days, and they will need to come up with a new shipping strategy in less time than that.
Thanks for your feedback.
We are all part of the same ecosystem and it is bad when a checkout is broken because no shipping rate can be returned.
We are merchant obsessed and the decision to adopt a 5 s timeout was motivated by the greater good: platform performance and reliability.
Could you have a look at your metrics and assess how many merchants have complex carrier requests that takes more than 5s to complete?
-> You may want to warn them before next Wednesday.
A sub-optimal solution could be to have them use the shipping profiled (flat rates) until you have time to update the app in order to be under the 5s timeout?
The issue is not our app servers it is that carriers are slow to begin with. UPS, FedEx and many others especially regional carriers are also near or at this limit. The main carriers also slow down during high volume months. How is a carrier service supposed to return rates if the carriers take more than 5 seconds? This is starting to sound like a push to kill off carrier services in favor of your own limited shipping which Shopify collects brokerage fees on. This seriously needs to be reconsidered both for developers, business owners, and Shopify Plus customers who you say you wish to help.
We are sorry about the short delay.
We analyzed the data and 5s seems, statistically, like a very acceptable number but, as you explained, some use-case may be impossible to address with this new constraint.
Could you check your metrics and identify the at-risk merchants?
So what happens if these merchants who have rates that take longer than 5s to complete can't make a decision before wednesday on how they want to structure their rates? you can imagine that not all merchants will have capacity to drop everything and work on their shipping rates. With such large dynamics in orders coming through stores, setting up flat rates, even coming up with average rates or implementing some sort of caching on the app ends will be next to useless, you are effectively asking all these merchants to lose for the greater good. Such a change is effectively holding a gun to their heads.