Hi, we're developing an app that we integrate with different Shopify merchants. We're using BulkOperation to pull some data from their stores.
Up till now, all of these BulkOperations were more or less similar in runtime. For example, 150k objects in about 30min. However, we have a new integration that is going very slow. It's now been ~36hrs, it's still RUNNING, it has gone through 900k objects.
Does anyone know what could be the reason for a much slower speed? Basically, we want to set the expectations for our clients and explain what they could look into to make this process faster. Is it as simple as Standard vs Shopify Plus account? Or are there more variables that affect these speeds?
Hey @godzilla ,
I can confirm that Standard vs Plus account is not a factor for Bulk Ops, as it is with the other API limits. The major variables that affect bulk operation speed are the amount of data that needs to be iterated through and the amount of regular and nested connections requested on the data.
You could try reducing the complexity of the data requested and/or splitting the Bulk Op tasks into smaller passes to fetch less total data with each Op?
The completion time doesn't necessarily scale linearly between the amount of objects retrieved due to their asynchronous nature, and there aren't any guarantees about how fast they'll complete. The intention behind Bulk Ops is more for convenience of use requesting a large amount of data, less so for critical tasks where latency is important (you should rely on the traditional Admin API cursor-based paginated data if so).
I found it took ~2 hours to download ~ 200,000 products with GQL bulk query, totally acceptable. The real downside is when I had to update those 200,000 there is no bulk operation so it takes 25 hours to do that.
I am hoping Shopify gets around to allowing more bulk operations for mutations than is currently supported. Would be nice to speed up that process by a factor of 100 just by allowing bulk updates.