Graphql Bulk Operation - operation hangs with no errors

Highlighted
Excursionist
40 0 28
I love the bulk operations feature in graphql, and we have used a specific query successfully 1000s of times.  Every once in a while, however, we have a particular shop that suddenly starts taking a really long time to run bulk operations in graphql.  The bulk oepration typically completes in 10-20 seconds, but every few weeks we find a store where the same bulk operation suddenly starts taking 20 minutes, an hour, or even more to run.  After a few days the issue seems to resolve itself and the shop begins running the bulk op in less than 10 seconds again.  No error is ever reported, on creation or when polling the running request
 
I found this thread on the forums reporting a similar issue, but no solution was provided. 
 
The issue:
- We kicked off a graphql bulk op, and polling returned an operation status of "CREATED" for about 15 minutes.
- After 15 mins the polling returned the status "RUNNING", and this was returned for over an hour. 
- It then returns the status "CANCELLING", which after another 10-15 mins changes to "CANCELLED"
- No errors are ever reported on the operation.
- x-request-id for an example operation with this issue: 11a40783-983b-4267-a124-be4133816693
- Here is the query we are running:
 

 

 

mutation {
   bulkOperationRunQuery(
      query: 
"""
{
    draftOrder(id: "gid://shopify/DraftOrder/${########}") {     //####### = draft order id
      lineItems() {
        edges {
          node {
            product {
              id
              collections() {
                edges {
                  node {
                    id
                  }
                }
              }
              tags
            }
            id
          }
        }
      }
    }
  }
"""
         ) {
            bulkOperation {
               id
               status
            }
         userErrors {
            field
            message
      }
   }
}

 

 

 
Just to reiterate this query works fine dozens, or hundreds of times a day, then once a month or so a few stores consistently have the problem for a few days. Not all stores are affected either, just a handful of shops at a time.
 
Any insight on this?
5 Likes
Highlighted
Shopify Expert
30 0 14

We are having the same issue. Here is an example bulk operation that is just stuck running for a very long time:

 

    "currentBulkOperation": {
      "id": "gid://shopify/BulkOperation/48480682046",
      "status": "RUNNING",
      "errorCode": null,
      "createdAt": "2020-07-27T14:49:53Z",
      "completedAt": null,
      "objectCount": "0",
      "fileSize": null,
      "url": null,
      "partialDataUrl": null
    }
4 Likes
Highlighted
Shopify Staff
Shopify Staff
152 20 26

Hey @Matt_Goodwin 

Can you give us a more recent example of this happening (preferably this week if possible)? A request ID or even the bulk query ID would work. 

Kevin_A | Developer Support @ Shopify
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Click Accept as Solution 

0 Likes
Highlighted
Tourist
5 0 2

Hey, this is happening for me today, I have used several times the same query to test and was okay, but after some time the query did returned anything, and even cancelling the current query and querying again none Bulk operation works right now.

 

Here is the operation: info:

results: #<OpenStruct data=#<OpenStruct bulkOperationRunQuery=#<OpenStruct bulkOperation=#<OpenStruct id="gid://shopify/BulkOperation/66034466891", status="CREATED">, userErrors=[]>>, extensions=#<OpenStruct cost=#<OpenStruct requestedQueryCost=10, actualQueryCost=10, throttleStatus=#<OpenStruct maximumAvailable=1000.0, currentlyAvailable=990, restoreRate=50.0>>>>

 

And this it the query:

 

mutation {
bulkOperationRunQuery(
query: """
{
products {
edges {
node {
id
title
publishedAt
handle
featuredImage {
transformedSrc(maxWidth: 200, maxHeight: 200)
}
images {
edges {
node {
id
transformedSrc(maxWidth: 200, maxHeight: 200)
}
}
}
options {
id
name
position
values
}
variants {
edges {
node {
id
title
price
position
inventoryPolicy
compareAtPrice
image {
id
transformedSrc(maxWidth: 200, maxHeight: 200)
}
inventoryItem {
tracked
inventoryLevels {
edges {
node {
available
}
}
}
}
}
}
}
}
}
}
}
"""
) {
bulkOperation {
id
status
}
userErrors {
field
message
}
}
}

0 Likes
Highlighted
Shopify Staff
Shopify Staff
152 20 26

Hey @Groon 

I can see that the bulk operation linked is currently still showing a status of "RUNNING":

gid://shopify/BulkOperation/66034466891

 

Based on the logs it looks like you tried to recreate the bulk operation a few times and got throttle errors that there was already a bulk operation running. I just polled the status of that bulk operation and successfully got a response:

{
  node(id: "gid://shopify/BulkOperation/66034466891") {
    ... on BulkOperation {
      id
      status
      errorCode
      createdAt
      completedAt
      objectCount
      fileSize
      url
      partialDataUrl
    }
  }
}

{
  "data": {
    "node": {
      "id": "gid://shopify/BulkOperation/66034466891",
      "status": "RUNNING",
      "errorCode": null,
      "createdAt": "2020-09-09T13:56:02Z",
      "completedAt": null,
      "objectCount": "0",
      "fileSize": null,
      "url": null,
      "partialDataUrl": null
    }
  },
  "extensions": {
    "cost": {
      "requestedQueryCost": 1,
      "actualQueryCost": 1
    }
  }
}

I would suggest waiting till this operation is complete before attempting to run another one. You should be able to successfully query its status using the query above. 

Kevin_A | Developer Support @ Shopify
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Click Accept as Solution 

0 Likes
Highlighted
Tourist
5 0 2

@Kevin_A 

Thanks for the answer, but it seems like a BIG problem that the operation is still running after almost 1 hour and I can't create another one o.O

My app depends on this operation to work fine, it needs to just work, if it takes 1 minute it's too long but is acceptable, but as it is right now is not reliable.

0 Likes
Highlighted
New Member
1 0 0

I had same issue today, about an hour ago, bulk operation was running for long time, then i had to manually cancel and restart the operation couple of times. And suddenly it started to work normally now. I guess something some changes on Shopify side? 

0 Likes
Highlighted
Excursionist
40 0 28

@Kevin_Asorry for the delay.  Here are some recent x-request-ids:

- c0d698d6-8a7b-4aec-ba6b-f2979dd1ea44

- c5a1d9c1-74fd-40a6-8df4-fa6de2306a59

- cd16ff16-991b-4227-897d-d06a78ec60ce

Just as a note our app cancels bulk requests after a certain timeout period, so ultimately these requests might have been cancelled after a few minutes of no response.  This is something we added in after the problem first occurred since we wanted to try the request again.  Hopefully that doesn't get in the way of your debugging.

 

Thanks!

 

 

 

 

0 Likes
Highlighted
Shopify Staff
Shopify Staff
152 20 26

Hey @Matt_Goodwin 

Thanks for the request IDs you provided. Based on what you provided everything appears to be functioning correctly. In regards to bulk operations, it is possible to experience a varying completion time because of how exponentially the data can grow from connections. In cases where an operation automatically cancels itself, it could be related to an error completing the operation but we would need a request ID within our logs retention period to investigate further. 

Kevin_A | Developer Support @ Shopify
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Click Accept as Solution 

0 Likes
Highlighted
Excursionist
17 0 6

Is there any reason you can give us about why this happens? I just had a bulk operation take almost 2 hours, when it normally takes only a minute or two.

0 Likes