Failed to send fulfillment request from shop Foobar Store (max retries reached)

[LionetChen]

Hi, I am a developer and have integrated my client's shopify store with my Fulfillment Service.

After integrating with a lot of clients without problem for months, today I received an email from mailer@shopify.com titled Failed to send fulfillment request from shop X Store

Here's the content:

Fulfillment requests from shop X Store to fulfillment service Y failed. The maximum number of retries (100,000) has been reached and the request will not be retried.
Find out about integrated fulfillments at the Shopify Help Center.

I assume this is bacause shopify cannot contact my callback_url somehow and retried for a long time without success. But it is weird because I used the same callback_url for all clients and no other alerts are raised for my other clients and they're all using  the same callback_url, which if failing should affect all my clients. 

I check my logs and there is no logging about any of the 100,000 request. Is there anywhere I could lookup why the requests failed? Since no logs are available maybe they're network failure (though it's unlikely because requests from other clients can come through) but I have no where to debug from.

Also does this means all future requests will not be attempted or only the request from that particular store or is it only requests for that particular order?

Thank you.

[samc]

Hi,

One thing you could check first is if the webhook URL are registered properly with the clients that are having issues?

[Jorge12]

Hello,

I started getting the email alerts like this too, one on March 2nd, and one today on March 8th 2021.    Shopify Support tried some light diagnosing but was unable to pinpoint the cause.  Their internal console did say that my domain was unable to be connected to.

We have a custom fulfillment service we created that is working fine as far as I can tell.  

The domain being unreachable had me thinking about webhooks also, like the poster above mentioned.  However our webhooks appear to be receiving data properly.  Also, we don't have any webhooks related to our custom fulfillment service (we poll the API and import data). 

I don't know by what mechanism  Shopify would be trying to connect/send a Fulfillment Request to my shop.

If anyone else has different experience, please share! 

[LionetChen]

@samc 

Yes. I used GET admin/api/2020-04/fulfillment_services.json to check the callback_url. It's identical with other working clients. I use the same code to register clients so it should be the same across all of them.

@Jorge12 

Could you tell me how do you contact Shopify IT support? I can't seem to find any Email/portal I can do that.

Based on the similarity of your description and my problem, I think the Shopify DNS server may have some problem resolving our domains. But it happens to only one store which is weird enough. And I'm using Cloudflare for DNS, the best I can find. I received the alert email on 4th March at 7:30 AM (Sydney).

I think this alert merchanism can be improved. It can tell me the time span in which the 100,000 retries happen. It can contain some diag info or at least open a log that we can query from. But at the end of the day if it really is a DNS problem there's not much I can think of to help at my end.

[Jorge12]

@LionetChen 

I agree, this error alert email could do with some more specifics to help those of us that receive it!

I wasn't able to contact IT directly, but I used the live chat feature at https://help.shopify.com/en/questions  (You may have to pretend to search the Knowledge Base before the Chat button appears at the bottom of the screen).  The regular support is fairly helpful, and it seems they were able to pull some level of logs.  However, they ultimately couldn't figure it out, and so suggested I post on the forums instead.

You make a good point about DNS being a common thread and point of failure.  We're using AWS' Route52, which should also be quite reliable. 

Since we don't appear to be  impacted by whatever failure Shopify is experiencing, I'll chalk this alert us to a false alarm for now.  I'll stay subscribed to this topic and see if any other posters find it via Google (as I did this morning) and share more experiences. 

[GElder185]

I started receiving these emails on 3/2/2021. I immediately opened a support ticket with Shopify support asking about the alert since it doesn't really give you enough information to identify how or why this alert was sent. Support had no idea why we started receiving these alerts, but I was able to confirm that our integration was still working smoothly so we agreed to close the ticket as a mystery for the time being. I have continued to receive these alerts from the multiple stores that we have our integration installed on, again with no real issue receiving fulfillment requests from those integrations. At this point I've resorted to ignoring these alerts, but I would love to know why we started receiving these alerts so we could do whatever we need to do to stop them. My main concern is that our clients (the store owners) are also receiving these alerts which is causing them unnecessary worry that the integration is having an problem even though we've confirmed everything is working fine...

[LionetChen]

In my support ticket, the shopify staff confirmed that the warning is for a particular order. As long as your have periodic polling, or the store has subsequent requests that trigger your polling you'll be fine.

Shopify Support: Once a single fulfillment request fails, that same request is retried 100,000 times. After the 100,000th time, the request is no longer retried and is stopped. 100,000 requests takes about 3 days to process for reference. This stoppage would not affect other orders' fulfillment requests, each order has it's own 100,000 limit.

---

The only thing that does not make sense is that all these 100K requests, which stretched out through 3 days, should not all fail. If my server were down for that long a time my boss would have my head already 😛

I think the retry mechanism needs some looking into, which is happening according to the support staff. Also as of now I've gone ~a month without receiving new warnings.

[Jorge12]

I believe I stumbled onto the explanation and fix!

When setting up a Fulfillment Service via REST API, a "callback_url" is provided.  (https://shopify.dev/api/admin/rest/reference/shipping-and-fulfillment/fulfillmentservice#create-2021...)  In our instance, we set a dummy URL.  Looking recently at our 404 logs for the server hosting our app, we see regular requests to the following URL.    Where our Fulfillment Service had a callback value of "www.test.com/callback", our logs show that someone (Shopify) is making a POST to the following URL:

[12/Jul/2021:09:49:21] "POST callback/fulfillment_order_notification HTTP/1.1" 404 48 "-" "Ruby"

We created an endpoint to live at that location, in fact to capture any URL in that format so we may catch any other types of callbacks that are made.    Since doing that we've seen these callbacks alerting us when the store has a new fulfillment request.  We could presumably stop polling and rely on this to pull in our latest orders at the time fulfillment is requested.

Most importantly, we've stopped getting the error emails from Shopify!

I hope this helps someone facing this problem.

[GElder185]

Wow! That makes a ton of sense.

As you mentioned, polling for new fulfillment requests was working fine for us so we never attempted to set up the webhook to receive them this way. I thought we had to opt in to fulfillment request pushes on the fulfillment service, similar to the "inventory_management" flag on the fulfillment service, where you receive "GET callback/fetch_stock.json" requests if the flag is true.

Looks like we're going to implement this endpoint for Shopify to push these to us and we'll keep the polling feature in case the push fails.

Thanks for the find. This was driving us and our clients insane!