Graphql Bulk Operation - operation hangs with no errors

Matt_Goodwin
Explorer
46 1 42
I love the bulk operations feature in graphql, and we have used a specific query successfully 1000s of times.  Every once in a while, however, we have a particular shop that suddenly starts taking a really long time to run bulk operations in graphql.  The bulk oepration typically completes in 10-20 seconds, but every few weeks we find a store where the same bulk operation suddenly starts taking 20 minutes, an hour, or even more to run.  After a few days the issue seems to resolve itself and the shop begins running the bulk op in less than 10 seconds again.  No error is ever reported, on creation or when polling the running request
 
I found this thread on the forums reporting a similar issue, but no solution was provided. 
 
The issue:
- We kicked off a graphql bulk op, and polling returned an operation status of "CREATED" for about 15 minutes.
- After 15 mins the polling returned the status "RUNNING", and this was returned for over an hour. 
- It then returns the status "CANCELLING", which after another 10-15 mins changes to "CANCELLED"
- No errors are ever reported on the operation.
- x-request-id for an example operation with this issue: 11a40783-983b-4267-a124-be4133816693
- Here is the query we are running:
 

 

 

mutation {
   bulkOperationRunQuery(
      query: 
"""
{
    draftOrder(id: "gid://shopify/DraftOrder/${########}") {     //####### = draft order id
      lineItems() {
        edges {
          node {
            product {
              id
              collections() {
                edges {
                  node {
                    id
                  }
                }
              }
              tags
            }
            id
          }
        }
      }
    }
  }
"""
         ) {
            bulkOperation {
               id
               status
            }
         userErrors {
            field
            message
      }
   }
}

 

 

 
Just to reiterate this query works fine dozens, or hundreds of times a day, then once a month or so a few stores consistently have the problem for a few days. Not all stores are affected either, just a handful of shops at a time.
 
Any insight on this?
Replies 12 (12)
TroyFeedo
New Member
1 0 0

Hello,

I am noticing very slow Bulk querying for collections on a specific store. All other stores I have are working fine, but as I check the status for the Bulk Query, it seems to FLY in the beginning (from 0 - 700,000 objectCount in a few minutes), but then it gets slower and slower until it's trickling in at a few 100 objectCount every few minutes... almost never completing and instead hitting an INTERNAL_SERVER_ERROR.

Request ID: 0a5fd7cb-ab24-4ee0-b9c2-ba340360e866

Originally, I was having this issue while requesting all collections, it would get to 1,000,000 - 1,300,000 objectCount before trickling and failing. I switched to only getting published collections and it now gets to 800,000 - 900,000 objectCount before trickling and failing. Seems like the trickling and failing happens at the end of the bulk query regardless of size.


EugeneKim
Shopify Partner
58 3 15

@TroyFeedo how many nested connections are there your query? It seems hard to believe that the store has over a million collections.

If you have multiple nested connections then it may be worth it to split up your bulk operation query so that each query only contains one connection.

DMiradakis
Tourist
4 0 1

Confirmed as a problem still on 2021-10-27, GraphQL API v2021-10. And we are running a Shopify Plus store. See my Postman screenshot below.

DMiradakis_0-1635365851483.png

Shopify support, please respond to this thread, as this is important to understand what's happening.

To everyone else in the thread, I guess we will have to account for this ourselves for the time being. Maybe check the createdAt field for a bulk operation and auto cancel it if it lasts longer than X minutes or X hours.