Bulk Operations API stopped working after the first test

[NTuggy]

I'm writing a webhook-based data cache as a private app for our store, and needed a way to recover from any outages (and, possibly, backfill some amount of past data at the moment it goes to production). At first, I was planning to just use the REST API to query through from the timestamp of the last addition before an outage, but then I ran across this guide to the Bulk Operations API. Looks pretty good, but there's just one small problem: after the first test I ran completed successfully (after about 2 seconds) and I downloaded the result file, I couldn't start any others, even with a direct copy+paste from the tutorial. Every request is met with a terse "400 Bad Request".

What I've tried:

• Insomnia and the internal code I wrote for Shopify API calls — both work fine for any other GET or POST call.
• The original (working) query, and the copy+paste from the tutorial.
• Switching Insomnia between JSON and GraphQL internally — I know Shopify is persnickety about Content-Type, but here, the Content-Type header is forced to be application/json either way, and I've verified that in the logs, but it makes no difference. (The same headers work fine for other requests, of course.)
• Switching between Basic Authorization and X-Shopify-Access-Token — both work fine for other calls, but neither works here, and in any case that should be 401 or 403, not 400.
• Canceling the original bulk operation (gid://shopify/BulkOperation/306678726854). Fails, since it's already completed, but I expected that.
• Waiting (so far, for 24 hours). The completed operation will expire after 7 days, so that's another possibility, but obviously having to wait even 15 seconds after the previous operation finishes makes this a non-starter for any serious use.

[Kevin_A]

Hey @NTuggy 

Can you provide us with the x-request-id response header for one of the failed 400 requests?

[NTuggy]

Sure, @Kevin_A. The most recent one was 604dfae2-435a-4383-b355-cdfb2d0ce809. I just sent off a4ef4545-da83-430e-a995-7a8cd8ea0564 with Basic authorization again.

[_JCC_]

Hey @NTuggy ,

Are you still experiencing an issue? Looking at bulkOperationRunQuery over the last seven days I'm not seeing any 400 - Bad Requests. If you do, please provide a recent request-id and I can have a look. The previous request-id provided indicates a parsing error in the payload of the request but I didn't experience an issue copying and pasting it into Insomnia.

Regards,

John Cole

[NTuggy · Accepted solution]

Thanks for getting back to me, @_JCC_ . I figured out some of the issues although some of it is still mysterious.

There were at least three bad interactions of Insomnia with Shopify:

1. Insomnia doesn't transparently rewrite GraphQL-as-JSON to GraphQL when you change the body type; you have to manually re-enter it. This isn't a bug, but it is annoying.
2. Copy-paste from Insomnia to itself injects extra tab characters, which are visible only in the Timeline view (as "\t"), and Shopify rejects that whitespace. This is a bug at both ends: whitespace is whitespace, and passing a string in with newline and tab escape sequences should not be a problem. OTOH, silently injecting tab characters is just inexcusable.
3. Insomnia rewrites Content-Type headers under certain circumstances, which apparently affected both the original problem request and subsequent cross-checks. I mis-remembered what it was supposed to be, assumed the original requests were unchanged, and went with that. Shopify rejects application/json passed to GraphQL endpoints, and Insomnia rewrites to application/json at least whenever the Body type is changed and possibly at other times. (I suspected on duplicating requests, but I can't reproduce that.) Rewriting Content-Type to a more formally-standardized value isn't exactly a bug, but it sure is a pain, and rejecting an applicable general Content-Type in favor of a specific one is also dubious at best.

I still don't know why the cross-checks using my own code failed, but I've now been able to start successful operations again.