Bulk Query INTERNAL_SERVER_ERROR

Highlighted
Tourist
12 0 3

Hello, I'm trying to run a bulk query to import one of our user's data into our system. I ran the following query:

        mutation {
          bulkOperationRunQuery(
            query:"""
            {
              products {
                edges {
                  node {
                    id
                    descriptionHtml
                    images {
                      edges {
                        node {
                          id
                          originalSrc
                        }
                      }
                    }
                    collections(query:"collection_type:custom") {
                      edges {
                        node {
                          id,
                          title
                        }
                      }
                    }
                    options {
                      name
                      position
                      values
                    }
                    productType
                    publishedAt
                    title
                    variants {
                      edges {
                        node {
                          id
                          barcode
                          compareAtPrice
                          fulfillmentService {
                            id
                            inventoryManagement
                          }
                          weight
                          weightUnit
                          inventoryItem {
                            id
                            unitCost {
                              amount
                              currencyCode
                            }
                            countryCodeOfOrigin
                            harmonizedSystemCode
                            provinceCodeOfOrigin
                            tracked
                          }
                          inventoryPolicy
                          inventoryQuantity
                          price
                          sku
                          taxable
                          taxCode
                          selectedOptions {
                            name
                            value
                          }
                          image {
                            originalSrc
                            id
                          }
                        }
                      }
                    }
                    vendor
                    totalInventory
                  }
                }
              }
            }
            """
          ) {
            bulkOperation {
              id
              status
            }
            userErrors {
              field
              message
            }
          }
        }

And I received the following:

{"data":{"bulkOperationRunQuery":{"bulkOperation":{"id":"gid:\/\/shopify\/BulkOperation\/3136061483","status":"CREATED"},"userErrors":[]}},"extensions":{"cost":{"requestedQueryCost":10,"actualQueryCost":10,"throttleStatus":{"maximumAvailable":1000.0,"currentlyAvailable":990,"restoreRate":50.0}}}}


However, when I go to poll the Bulk Job about 5 minutes later, I receive the following:


I polled the job about 5-10 minutes later and it was running fine. However, polling job a few hours later and I'm receiving the following response:

{"data":{"currentBulkOperation":{"id":"gid:\/\/shopify\/BulkOperation\/3136061483","status":"FAILED","errorCode":"INTERNAL_SERVER_ERROR","createdAt":"2020-02-06T09:06:16Z","completedAt":null,"objectCount":"948118","fileSize":null,"url":null,"partialDataUrl":"..."}},"extensions":{"cost":{"requestedQueryCost":1,"actualQueryCost":1,"throttleStatus":{"maximumAvailable":1000.0,"currentlyAvailable":999,"restoreRate":50.0}}}}


What is causing this to happen? I was able to run a similar query two nights ago and it ran successfully (object count of ~1.3M where this seems to be failing at ~900K). I can also run this exact same bulk query on my own test store and it completes successfully. 

Any help would be appreciated since this is a major blocker for us.

Highlighted
Shopify Staff (Retired)
Shopify Staff (Retired)
624 102 129

Hey @ShaneOH ,

 

This issue you are facing might be related to an on-going problem Shopify is currently working to solve in which queries for `products` in our core database for a Shopify store that has a large amount of products (e.g. > 1000 products) are failing. 

 

As I mentioned there is on-going work on our end to resolve this, but in the meantime as a workaround you can resolve this on your end by including a filter in your GraphQL query for the "created_at" date of the product. So for example your query could look like:

 

mutation {
  bulkOperationRunQuery(
    query:"""
    {
      products(query: "created_at:>=2020-02-01") {
        edges {
          node {
            id
...
}
}
}
}
}

Hassain | Developer Support Specialist @ Shopify
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Click Accept as Solution 

0 Likes
Highlighted
Tourist
12 0 3

Hey @hassain , thanks for the response! It's good to know that Shopify is aware of this issue and actively working on it. 

The created_at filter is a bit of a rough solution since the purpose of this is to import a user's entire catalog into our system so we can then begin managing their product catalog from our side. So we would have to automate bulk importing back from some arbitrary date and request bulk files month by month (or year by year, etc., basically just needing to guess at a query length that would pull an amount of products that won't 500 error on Shopify's end). It's definitely possible to do but would take much longer, be more complex and prone to error, and sort of defeats the entire purpose of the bulk query in the first place. 

With all that said, definitely understand the problems at scale you're dealing with and again appreciate the answer, hope you guys can solve it soon! For what it's worth, the same query properly processed a couple days later with no change on our end, so it seems to be an intermittent issue. 

0 Likes
Highlighted
New Member
4 0 0

Is this problem fix?

I'm trying the solution, using the create_at filter on my dev store, the store has 12 products, i using a simplified query as test:

 

mutation {
  bulkOperationRunQuery(
    query:"""
    {
      products(query: "created_at:>=2020-02-01") {
        edges {
          node {
            id
          }
       }
     }
    }
    """
  ) {
    bulkOperation {
      id
      status
    }
    userErrors {
      field
      message
    }
  }
}

 

 

it's created, i receive this response

{
  "data": {
    "bulkOperationRunQuery": {
      "bulkOperation": {
        "id": ":)",
        "status": "CREATED"
      },
      "userErrors": []
    }
  },
  "extensions": {
    "cost": {
      "requestedQueryCost": 10,
      "actualQueryCost": 10,
      "throttleStatus": {
        "maximumAvailable": 1000,
        "currentlyAvailable": 990,
        "restoreRate": 50
      }
    }
  }
}

 

i pull using 

query {
  currentBulkOperation  {
      id
      status
      errorCode
      createdAt
      completedAt
      objectCount
      fileSize
      url
  }
}

 

it runs fine some secs, first pools i receive:

{
  "data": {
    "currentBulkOperation": {
      "id": ":)",
      "status": "RUNNING",
      "errorCode": null,
      "createdAt": "2020-08-13T11:09:08Z",
      "completedAt": null,
      "objectCount": "0",
      "fileSize": null,
      "url": null
    }
  },
  "extensions": {
    "cost": {
      "requestedQueryCost": 1,
      "actualQueryCost": 1,
      "throttleStatus": {
        "maximumAvailable": 1000,
        "currentlyAvailable": 999,
        "restoreRate": 50
      }
    }
  }
}

 

and sudenlly fails:

 

{
  "data": {
    "currentBulkOperation": {
      "id": ":)",
      "status": "FAILED",
      "errorCode": "INTERNAL_SERVER_ERROR",
      "createdAt": "2020-08-13T11:09:08Z",
      "completedAt": null,
      "objectCount": "0",
      "fileSize": null,
      "url": null
    }
  },
  "extensions": {
    "cost": {
      "requestedQueryCost": 1,
      "actualQueryCost": 1,
      "throttleStatus": {
        "maximumAvailable": 1000,
        "currentlyAvailable": 999,
        "restoreRate": 50
      }
    }
  }
}

 

 

what i'm doing wrong? 

0 Likes
Highlighted
Tourist
10 0 2

  I just moved over to GraphQL from REST API to take advantage of the bulk operations. The shop I'm querying has about 1500 products and experiencing the same INTERNAL_SERVER_ERROR after a number of successful polls. The plan of attack has been ...

  1. Create bulk operation to query all products.
  2. Poll every 10 seconds for the COMPLETED status.
  3. Grab the JSONL data and process the products.

  Two days ago it ran without issue. Today it has a status of RUNNING for 15-25 polls, 10 second intervals, before returning a FAILED status with errorCode INTERNAL_SERVER_ERROR.

  Hey [removed tags as per our Code of Conduct] any recent updates on this issue with bulk operations and INTERNAL_SERVER_ERROR!?

Thanks,

sfsdev

0 Likes