CarrierService callback_url randomly stops working - callback_url gets hit, rates fire in logs

[Liquidator3358]

And still, somehow the checkout manages to not show my rates.

My CarrierService callback_url gets hit by Shopify, I see the endpoint get hit in the logs, I see the rates come through correctly in the logs, and then, for whatever reason, Shopify chooses not to display them and shows the default rates (a major problem).

Basically, I am chasing a silent bug... by all accounts, things appear to be good on my end, it works probably 70-80% of the time but I cannot find a pattern nor any rhyme or reason as to why it is breaking sporadically.

Can anyone help me on this deal?

[Liquidator3358]

I mean seriously, can one Shopify employee who has a little knowledge troubleshoot this with me?  It makes zero sense... Shopify Support is zero help when it comes to the API, so surely another dev has a the slightest clue as to what is going on? The absolute worst part is if Shopify doesn't display the rates, then the default shipping rate is cached for the remainder of it's duration (whatever it is) or if the order changes.  If they continue to try to checkout, they still will not be able to access the real rates.

[ShopifyDevSup]

Hey @Liquidator3358,

Our dev documentation on carrier services outlines the expected behaviour for caching rates. 

Some of the most common issues that come up with carrier service rates are requests timing out, or JSON formatting being invalid preventing the carrier service's rates from appearing, but we can't provide authenticated support in the forums or provide any insights into your issue without context on what troubleshooting you have tried. 

We still recommend setting fallback rates in the admin, but have you tried limiting the number of returned rates to see if that yields better results? If the issue could be with the connection, have you tested your carrier service on an ngrok tunnel to rule out an issue with your server? 

[Liquidator3358]

Sorry for the confusion @ShopifyDevSup  - I was detailing out how I am caching the rates on my end so that the rate retrieval process is done quickly.

Okay, so I took your advice and changed the callback URL back to an ngrok route... I literally CAN'T break it lol.  Thanks to you it is now, 100% with out a doubt, a hosting problem on MY end.  Now, I am adding delays to my own code to see how long before I can break it.  Currently, I am adding a timeout function on my callback JSON response, and even with a 5 second delay, it continues to work but there is a difference... The checkout is actually searching for the rates and the loading indicator is somewhat matching the intentional delay.  This is NOT the case in production... when it fails, it fails fast, as if it's not even trying to load, and then it quickly defaults to the backup rates.  From that limited knowledge, it certainly feels like the endpoint is not accessible on my end.

Do you have any suggestions?  I am using Vercel hosting for my NextJS headless web application but I do not have allegiance to anyone... I am about to launch and I want this callback endpoint to be rock steady.

[ShopifyDevSup]

Hey @Liquidator3358,

Thanks for following up, I'm glad to hear that you have narrowed down where the problem is coming from.

I am going to add an internal feature request for better logging/ error reporting to see if there's anything we can do to make the carrier service setup process easier in the future, but have you tried sending POST requests to the Vercel endpoint with an HTTP client like Postman to see if you can replicate any errors that way? 

Alternatively, is there logging built into the NextJS app to take a closer look into the requests that are failing to display rates in the Shopify Checkout?

If you can provide the endpoint and a timestamp for one of the failed requests I'll see if there's anything else I can find on our end to help with the investigation.

[Liquidator3358]

Hey @ShopifyDevSup  (Al),

Okay, so here is where I am at. I have tried all the logging imaginable on my end, and it seems to be all for naught because nothing gets logged when it fails.  So for instance, I just tested it 10 times and it finally failed on the 11th time.

Here is what I have:

Requests

1. COMPLETED SUCCESFULLY
• ENDPOINT HIT TWICE - NO ERROS IN LOGS
2. COMPLETED SUCCESFULLY
• ENDPOINT HIT ONCE - NO ERRORS IN LOGS
3. COMPLETED SUCCESFULLY
• ENDPOINT HIT TWICE - NO ERRORS IN LOGS
4. COMPLETED SUCCESFULLY
• ENDPOINT HIT TWICE - NO ERRORS IN LOGS
5. COMPLETED SUCCESFULLY
• ENDPOINT HIT TWICE - NO ERRORS IN LOGS
6. COMPLETED SUCCESFULLY
• ENDPOINT HIT ONCE - NO ERRORS IN LOGS
7. COMPLETED SUCCESFULLY
• ENDPOINT HIT TWICE - NO ERRORS IN LOGS
8. COMPLETED SUCCESFULLY
• ENDPOINT HIT TWICE - NO ERRORS IN LOGS
9. COMPLETED SUCCESFULLY
• ENDPOINT HIT TWICE - NO ERRORS IN LOGS
10. COMPLETED SUCCESFULLY
• ENDPOINT HIT TWICE - NO ERRORS IN LOGS
11. COMPLETED SUCCESFULLY
• ENDPOINT HIT ONCE - NO ERRORS IN LOGS
12. FAILED
• ENDPOINT HIT ONCE - NO ERRORS IN LOGS

Now, on that last failure, my console.logs() went through and I had no errors on my end.  Also, on the ONE request that Shopify sent during the failed run, this is what the Vercel's server logs had to say about it:

[POST] /api/checkout/shippingRates 10:34:38:02

Function Status:200

Edge Status:200

Duration:769.77 ms

Init Duration:N/A

Memory Used:119 MB

ID:cle1::iad1::9dls4-1677256478023-1dbb030d3f4d

User Agent:Ruby

It showed a 200 request, log statements fired correctly, and still, the rates never made it to checkout... This is why it has been challenging to pinpoint where the miscommunication is. Also, my API endpoint is set up using try/catch, so when the request fails, I expect an error to get logged, but as you already know, that is NOT the case.  It is failing silently, so the catch block never enters the equation (not once).

I know there's a key/value in the request that comes from Shopify that is consumed: booleanValue - maybe that is an indicator as to whether or not the it was successful but even if that is the case, it doesn't give me much information regarding why it is failing to begin with.

By the way, add liquidationdelivered.com to the above URL in the logs, and you have your timestamp/URL to test :D.  That would be ideal!

[Liquidator3358]

@ShopifyDevSup 

Hey Al,

Not meaning to press you on this but I wanted to pick your brain before I move on.  I have a ticket out with Vercel as well, so I am trying to spearhead this thing from all three sides (my code included).

[ShopifyDevSup]

Hi again @Liquidator3358 ,

I managed to find the 12 requests you mentioned. The last of the 12 events in that ~20 minute timeframe has a generic error saying that the connection to the remote server (your vercel endpoint) timed out. 
 

I haven't been able to find any more details, but hopefully that helps! 

[Liquidator3358]

@ShopifyDevSup 

My man!

That helps a ton.  I am talking to Vercel today, so I will pass this on.

Thanks a bunch for looking into it.

[Parcel_Intellig]

Hi @ShopifyDevSup 

I'm having similar issues our CCS endpoint, just wondering, what is the connection timeout Shopify applies to CCS calls?

Thanks

Min

[ShopifyDevSup]

Hi @Parcel_Intellig ,

The read timeouts are described in [the CarrierService documentation under 'Response Timeouts'].

Hope you have a great day

[Parcel_Intellig]

Hi, thanks for your reply, our issue is that we have orders that has used backup rates, however our logs also indicate that these rates were sent back in well under the time limits. I've been back and fourth with partner support for nearly a month now and they have basically refused to investigate if we can't product a x-request-id, which is impossible for us to get as shopify has initiated the request in the checkout, I realize that the checkout page load comes with its own x-request-id but again only the end customer sees this and thus impossible to get.

As such, we suspect that there may be other bugs here where the response may be lost after we have returned it to Shopify, however with partner support refusing to investigate or demanding we supply unreasonable information, we aren't able to get back to our merchants. Having said that, we haven't had any more recent reports of such types of issues, but in future if we see issues not relating to the normal timeouts, how do we get such issues investigated?

[ShopifyDevSup]

Hey @Parcel_Intellig - thanks for waiting on our reply here. We do generally have an eye on the forums here, but our team isn't available 24/7 to investigate issues like this. In terms of investigations for issue reporting, Partner Support is still the best point of contact. If you still have an ongoing email conversation with partner support, my suggestion would be to share a checkout ID (if possible) with Partner Support and share our reply in this thread. 
 

We can investigate further with a checkout ID on our end, but due to the merchant/customer data that's included in troubleshooting checkout issues we can't speak too much about the details on the forums here. I can't guarantee a turnaround time, but if you do need further assistance, I'd reach out to partner support again and share this reply along with a checkout ID. Partner Support is able to get in touch with our more technical support teams, and while my team is unable to assist directly through email/phones, we do often collaborate with Partner Support.

Hope this helps - let us know if we can clarify anything on our end.  

[Parcel_Intellig]

Hi, thanks for the additional details, given it's nearly 2 months since the original issue, i will park this one for now as the merchant has already moved on. however will keep this link handy in case it comes up again.

[Liquidator3358]

Alright... last bump.  Again, callback_url receives request from Shopify, rates are cached BEFORE the callback receives the request (you literally cannot proceed to checkout without having rates, there the rates are always ready to go), callback endpoint and rates are seen on server side logs AND randomly, the default shipping option is displayed and not my carrier rates.

This has to be a network issue... and I have no idea how to fix it.  Does ANYBODY have ANY idea on what I can do here?

I am operating on a 85% (I have had some luck optimizing my callback function, but nothing that is broadly fixing the problem) success rate, but given how complex my shipping is, there is no way that I can set up custom rates that will even begin to remotely do me or my customers justice.  That 15% failure rate is detrimental to our operation and I would hate to have to default to "Call For Quote" on 15 out of 100 orders.