Inconsistent Graphql Bulk Query

[achempak]

I'm having extremely inconsistent experience with the Shopify bulk query. I'm doing multiple different queries every night. There are times where I get a response within minutes. Other times, within hours (sometimes up to 6 hours!). Furthermore, lately my requests are never completed. When I poll them, after 10-30 minutes, the operation status is "CANCELED." There is no error code given (it's null). The amount of data being requested varies in size, but some relatively small requests also run into this problem. Our backend APIs rely on data from Shopify, but lately, because of this problem, I can't retrieve the data. Has anyone else run into this problem? Is there a fix? Below is an example of a query I am doing.

mutation {
  bulkOperationRunQuery(
    query:"""
    {
      products(query:"updated_at:>'2020-07-01 12:00:00'") {
				edges{
					node{
						id
						title
						handle
						descriptionHtml
						vendor
						productType
						createdAt
						updatedAt
						publishedAt
						tags
						totalInventory
						hasOutOfStockVariants
						featuredImage {
							id
						}
						images {
							edges {
								node {
									id
									altText
									originalSrc
								}
							}
						}
						variants {
							edges{
								node{
									id
									title
									sku
									barcode
									createdAt
									updatedAt
									inventoryPolicy
									inventoryQuantity
									inventoryItem{
										id
										createdAt
										updatedAt
										sku
										duplicateSkuCount
										harmonizedSystemCode
										unitCost {
											amount
										}
										provinceCodeOfOrigin
									}
									price
									fulfillmentService {
										handle
									}
									selectedOptions{
										name
									}
									taxable
									taxCode
									weight
									weightUnit
									image{
										id
									}
								}
							}
						}
					}
				}
			}
    }
    """
  ) {
    bulkOperation {
      id
      status
    }
    userErrors {
      field
      message
    }
  }
}

[Kevin_A]

Hey @achempak 

Do you happen to have a request-id available (x-request-id response header) for one of the cancelled bulk jobs? It seems fairly normal that there may be a time discrepancy on how long they run, but if they are getting cancelled I can try and look into that for you. 

[achempak]

Hi @Kevin_A ,

Thanks for the quick response! I'm not sure if this is the id you're looking for, but here are two examples from last night.

gid://shopify/BulkOperation/50948112446

gid://shopify/BulkOperation/50950045758

[Kevin_A]

Hey @achempak 

Are you still seeing bulk operations being cancelled? Can you provide any recent IDs where you are seeing this happen?

[Louise_Elmose_H]

Hi Kevin_A,

I am currently having the same issues as described in this thread.

I have an app which creates bulk job queries for different shops. It has been running fine, until the 13th of October where the bulk jobs start running with status "RUNNING" for a very long time (a couple of hours compared to 20-30 minutes), after which the job gets status "CANCELED". Please note that I do not have any code in the app which cancels a running job.

This happens for several of the stores using the app, but not all.

It is quite critical for the usage of the app.

Some id's on bulk jobs that behaves like this from today:

• gid://shopify/BulkOperation/55562174542
• gid://shopify/BulkOperation/212754530503
• id://shopify/BulkOperation/55443882062

Thanks,

-Louise