Our Partner & Developer boards on the community are moving to a brand new home: the .dev community forums! While you can still access past discussions here, for all your future app and storefront building questions, head over to the new forums.

Bulk Operations API stopped working after the first test

Solved

Bulk Operations API stopped working after the first test

NTuggy
Tourist
3 1 1

I'm writing a webhook-based data cache as a private app for our store, and needed a way to recover from any outages (and, possibly, backfill some amount of past data at the moment it goes to production). At first, I was planning to just use the REST API to query through from the timestamp of the last addition before an outage, but then I ran across this guide to the Bulk Operations API. Looks pretty good, but there's just one small problem: after the first test I ran completed successfully (after about 2 seconds) and I downloaded the result file, I couldn't start any others, even with a direct copy+paste from the tutorial. Every request is met with a terse "400 Bad Request".

What I've tried:

  • Insomnia and the internal code I wrote for Shopify API calls — both work fine for any other GET or POST call.
  • The original (working) query, and the copy+paste from the tutorial.
  • Switching Insomnia between JSON and GraphQL internally — I know Shopify is persnickety about Content-Type, but here, the Content-Type header is forced to be application/json either way, and I've verified that in the logs, but it makes no difference. (The same headers work fine for other requests, of course.)
  • Switching between Basic Authorization and X-Shopify-Access-Token — both work fine for other calls, but neither works here, and in any case that should be 401 or 403, not 400.
  • Canceling the original bulk operation (gid://shopify/BulkOperation/306678726854). Fails, since it's already completed, but I expected that.
  • Waiting (so far, for 24 hours). The completed operation will expire after 7 days, so that's another possibility, but obviously having to wait even 15 seconds after the previous operation finishes makes this a non-starter for any serious use.
Accepted Solution (1)
NTuggy
Tourist
3 1 1

This is an accepted solution.

Thanks for getting back to me, @_JCC_ . I figured out some of the issues although some of it is still mysterious.

There were at least three bad interactions of Insomnia with Shopify:

  1. Insomnia doesn't transparently rewrite GraphQL-as-JSON to GraphQL when you change the body type; you have to manually re-enter it. This isn't a bug, but it is annoying.
  2. Copy-paste from Insomnia to itself injects extra tab characters, which are visible only in the Timeline view (as "\t"), and Shopify rejects that whitespace. This is a bug at both ends: whitespace is whitespace, and passing a string in with newline and tab escape sequences should not be a problem. OTOH, silently injecting tab characters is just inexcusable.
  3. Insomnia rewrites Content-Type headers under certain circumstances, which apparently affected both the original problem request and subsequent cross-checks. I mis-remembered what it was supposed to be, assumed the original requests were unchanged, and went with that. Shopify rejects application/json passed to GraphQL endpoints, and Insomnia rewrites to application/json at least whenever the Body type is changed and possibly at other times. (I suspected on duplicating requests, but I can't reproduce that.) Rewriting Content-Type to a more formally-standardized value isn't exactly a bug, but it sure is a pain, and rejecting an applicable general Content-Type in favor of a specific one is also dubious at best.

I still don't know why the cross-checks using my own code failed, but I've now been able to start successful operations again.

View solution in original post

Replies 4 (4)

Kevin_A
Shopify Staff
318 42 61

Hey @NTuggy 

Can you provide us with the x-request-id response header for one of the failed 400 requests?

Kevin_A | Solutions Engineer @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

NTuggy
Tourist
3 1 1

Sure, @Kevin_A. The most recent one was 604dfae2-435a-4383-b355-cdfb2d0ce809. I just sent off a4ef4545-da83-430e-a995-7a8cd8ea0564 with Basic authorization again.

_JCC_
Shopify Staff
200 27 58

Hey @NTuggy ,

Are you still experiencing an issue? Looking at bulkOperationRunQuery over the last seven days I'm not seeing any 400 - Bad Requests. If you do, please provide a recent request-id and I can have a look. The previous request-id provided indicates a parsing error in the payload of the request but I didn't experience an issue copying and pasting it into Insomnia.

Regards,

John Cole

John C | Developer Support @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

NTuggy
Tourist
3 1 1

This is an accepted solution.

Thanks for getting back to me, @_JCC_ . I figured out some of the issues although some of it is still mysterious.

There were at least three bad interactions of Insomnia with Shopify:

  1. Insomnia doesn't transparently rewrite GraphQL-as-JSON to GraphQL when you change the body type; you have to manually re-enter it. This isn't a bug, but it is annoying.
  2. Copy-paste from Insomnia to itself injects extra tab characters, which are visible only in the Timeline view (as "\t"), and Shopify rejects that whitespace. This is a bug at both ends: whitespace is whitespace, and passing a string in with newline and tab escape sequences should not be a problem. OTOH, silently injecting tab characters is just inexcusable.
  3. Insomnia rewrites Content-Type headers under certain circumstances, which apparently affected both the original problem request and subsequent cross-checks. I mis-remembered what it was supposed to be, assumed the original requests were unchanged, and went with that. Shopify rejects application/json passed to GraphQL endpoints, and Insomnia rewrites to application/json at least whenever the Body type is changed and possibly at other times. (I suspected on duplicating requests, but I can't reproduce that.) Rewriting Content-Type to a more formally-standardized value isn't exactly a bug, but it sure is a pain, and rejecting an applicable general Content-Type in favor of a specific one is also dubious at best.

I still don't know why the cross-checks using my own code failed, but I've now been able to start successful operations again.