Graphql Bulk Operation - operation hangs with no errors

Matt_Goodwin
Shopify Partner
54 1 70
I love the bulk operations feature in graphql, and we have used a specific query successfully 1000s of times.  Every once in a while, however, we have a particular shop that suddenly starts taking a really long time to run bulk operations in graphql.  The bulk oepration typically completes in 10-20 seconds, but every few weeks we find a store where the same bulk operation suddenly starts taking 20 minutes, an hour, or even more to run.  After a few days the issue seems to resolve itself and the shop begins running the bulk op in less than 10 seconds again.  No error is ever reported, on creation or when polling the running request
 
I found this thread on the forums reporting a similar issue, but no solution was provided. 
 
The issue:
- We kicked off a graphql bulk op, and polling returned an operation status of "CREATED" for about 15 minutes.
- After 15 mins the polling returned the status "RUNNING", and this was returned for over an hour. 
- It then returns the status "CANCELLING", which after another 10-15 mins changes to "CANCELLED"
- No errors are ever reported on the operation.
- x-request-id for an example operation with this issue: 11a40783-983b-4267-a124-be4133816693
- Here is the query we are running:
 

 

 

mutation {
   bulkOperationRunQuery(
      query: 
"""
{
    draftOrder(id: "gid://shopify/DraftOrder/${########}") {     //####### = draft order id
      lineItems() {
        edges {
          node {
            product {
              id
              collections() {
                edges {
                  node {
                    id
                  }
                }
              }
              tags
            }
            id
          }
        }
      }
    }
  }
"""
         ) {
            bulkOperation {
               id
               status
            }
         userErrors {
            field
            message
      }
   }
}

 

 

 
Just to reiterate this query works fine dozens, or hundreds of times a day, then once a month or so a few stores consistently have the problem for a few days. Not all stores are affected either, just a handful of shops at a time.
 
Any insight on this?
Replies 12 (12)

BGilbert
Shopify Expert
30 0 15

We are having the same issue. Here is an example bulk operation that is just stuck running for a very long time:

 

    "currentBulkOperation": {
      "id": "gid://shopify/BulkOperation/48480682046",
      "status": "RUNNING",
      "errorCode": null,
      "createdAt": "2020-07-27T14:49:53Z",
      "completedAt": null,
      "objectCount": "0",
      "fileSize": null,
      "url": null,
      "partialDataUrl": null
    }

Kevin_A
Shopify Staff
318 42 61

Hey @Matt_Goodwin 

Can you give us a more recent example of this happening (preferably this week if possible)? A request ID or even the bulk query ID would work. 

Kevin_A | Solutions Engineer @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

Groon
Shopify Partner
15 0 16

Hey, this is happening for me today, I have used several times the same query to test and was okay, but after some time the query did returned anything, and even cancelling the current query and querying again none Bulk operation works right now.

 

Here is the operation: info:

results: #<OpenStruct data=#<OpenStruct bulkOperationRunQuery=#<OpenStruct bulkOperation=#<OpenStruct id="gid://shopify/BulkOperation/66034466891", status="CREATED">, userErrors=[]>>, extensions=#<OpenStruct cost=#<OpenStruct requestedQueryCost=10, actualQueryCost=10, throttleStatus=#<OpenStruct maximumAvailable=1000.0, currentlyAvailable=990, restoreRate=50.0>>>>

 

And this it the query:

 

mutation {
bulkOperationRunQuery(
query: """
{
products {
edges {
node {
id
title
publishedAt
handle
featuredImage {
transformedSrc(maxWidth: 200, maxHeight: 200)
}
images {
edges {
node {
id
transformedSrc(maxWidth: 200, maxHeight: 200)
}
}
}
options {
id
name
position
values
}
variants {
edges {
node {
id
title
price
position
inventoryPolicy
compareAtPrice
image {
id
transformedSrc(maxWidth: 200, maxHeight: 200)
}
inventoryItem {
tracked
inventoryLevels {
edges {
node {
available
}
}
}
}
}
}
}
}
}
}
}
"""
) {
bulkOperation {
id
status
}
userErrors {
field
message
}
}
}

Kevin_A
Shopify Staff
318 42 61

Hey @Groon 

I can see that the bulk operation linked is currently still showing a status of "RUNNING":

gid://shopify/BulkOperation/66034466891

 

Based on the logs it looks like you tried to recreate the bulk operation a few times and got throttle errors that there was already a bulk operation running. I just polled the status of that bulk operation and successfully got a response:

{
  node(id: "gid://shopify/BulkOperation/66034466891") {
    ... on BulkOperation {
      id
      status
      errorCode
      createdAt
      completedAt
      objectCount
      fileSize
      url
      partialDataUrl
    }
  }
}

{
  "data": {
    "node": {
      "id": "gid://shopify/BulkOperation/66034466891",
      "status": "RUNNING",
      "errorCode": null,
      "createdAt": "2020-09-09T13:56:02Z",
      "completedAt": null,
      "objectCount": "0",
      "fileSize": null,
      "url": null,
      "partialDataUrl": null
    }
  },
  "extensions": {
    "cost": {
      "requestedQueryCost": 1,
      "actualQueryCost": 1
    }
  }
}

I would suggest waiting till this operation is complete before attempting to run another one. You should be able to successfully query its status using the query above. 

Kevin_A | Solutions Engineer @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

Groon
Shopify Partner
15 0 16

@Kevin_A 

Thanks for the answer, but it seems like a BIG problem that the operation is still running after almost 1 hour and I can't create another one o.O

My app depends on this operation to work fine, it needs to just work, if it takes 1 minute it's too long but is acceptable, but as it is right now is not reliable.

Ankit_lux
Visitor
1 0 0

I had same issue today, about an hour ago, bulk operation was running for long time, then i had to manually cancel and restart the operation couple of times. And suddenly it started to work normally now. I guess something some changes on Shopify side? 

Matt_Goodwin
Shopify Partner
54 1 70

@Kevin_Asorry for the delay.  Here are some recent x-request-ids:

- c0d698d6-8a7b-4aec-ba6b-f2979dd1ea44

- c5a1d9c1-74fd-40a6-8df4-fa6de2306a59

- cd16ff16-991b-4227-897d-d06a78ec60ce

Just as a note our app cancels bulk requests after a certain timeout period, so ultimately these requests might have been cancelled after a few minutes of no response.  This is something we added in after the problem first occurred since we wanted to try the request again.  Hopefully that doesn't get in the way of your debugging.

 

Thanks!

 

 

 

 

Kevin_A
Shopify Staff
318 42 61

Hey @Matt_Goodwin 

Thanks for the request IDs you provided. Based on what you provided everything appears to be functioning correctly. In regards to bulk operations, it is possible to experience a varying completion time because of how exponentially the data can grow from connections. In cases where an operation automatically cancels itself, it could be related to an error completing the operation but we would need a request ID within our logs retention period to investigate further. 

Kevin_A | Solutions Engineer @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

ellitt
Excursionist
18 0 10

Is there any reason you can give us about why this happens? I just had a bulk operation take almost 2 hours, when it normally takes only a minute or two.

TroyFeedo
Visitor
1 0 0

Hello,

I am noticing very slow Bulk querying for collections on a specific store. All other stores I have are working fine, but as I check the status for the Bulk Query, it seems to FLY in the beginning (from 0 - 700,000 objectCount in a few minutes), but then it gets slower and slower until it's trickling in at a few 100 objectCount every few minutes... almost never completing and instead hitting an INTERNAL_SERVER_ERROR.

Request ID: 0a5fd7cb-ab24-4ee0-b9c2-ba340360e866

Originally, I was having this issue while requesting all collections, it would get to 1,000,000 - 1,300,000 objectCount before trickling and failing. I switched to only getting published collections and it now gets to 800,000 - 900,000 objectCount before trickling and failing. Seems like the trickling and failing happens at the end of the bulk query regardless of size.


EugeneKim
Shopify Partner
60 3 27

@TroyFeedo how many nested connections are there your query? It seems hard to believe that the store has over a million collections.

If you have multiple nested connections then it may be worth it to split up your bulk operation query so that each query only contains one connection.

DMiradakis
Tourist
7 0 8

Confirmed as a problem still on 2021-10-27, GraphQL API v2021-10. And we are running a Shopify Plus store. See my Postman screenshot below.

DMiradakis_0-1635365851483.png

Shopify support, please respond to this thread, as this is important to understand what's happening.

To everyone else in the thread, I guess we will have to account for this ourselves for the time being. Maybe check the createdAt field for a bulk operation and auto cancel it if it lasts longer than X minutes or X hours.