Bulk Operation is slower than expected

New Member
3 0 1

Hi, we're developing an app that we integrate with different Shopify merchants. We're using BulkOperation to pull some data from their stores.

Up till now, all of these BulkOperations were more or less similar in runtime. For example, 150k objects in about 30min. However, we have a new integration that is going very slow. It's now been ~36hrs, it's still RUNNING, it has gone through 900k objects.

Does anyone know what could be the reason for a much slower speed? Basically, we want to set the expectations for our clients and explain what they could look into to make this process faster. Is it as simple as Standard vs Shopify Plus account? Or are there more variables that affect these speeds?

0 Likes
Highlighted
Shopify Partner
66 5 26

Reading this is really worrying for me. I'm working on an app that uses BulkOperations. Having them run for hours would be devastating for my app.

I would love a response from Shopify on this.

0 Likes
Highlighted
Shopify Staff
Shopify Staff
79 8 11

Hey @godzilla ,

I can confirm that Standard vs Plus account is not a factor for Bulk Ops, as it is with the other API limits. The major variables that affect bulk operation speed are the amount of data that needs to be iterated through and the amount of regular and nested connections requested on the data.

You could try reducing the complexity of the data requested and/or splitting the Bulk Op tasks into smaller passes to fetch less total data with each Op?

The completion time doesn't necessarily scale linearly between the amount of objects retrieved due to their asynchronous nature, and there aren't any guarantees about how fast they'll complete. The intention behind Bulk Ops is more for convenience of use requesting a large amount of data, less so for critical tasks where latency is important (you should rely on the traditional Admin API cursor-based paginated data if so).

Callum | Developer Support @ Shopify
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Click Accept as Solution 

0 Likes
Highlighted
Shopify Expert
4262 32 434

I found it took ~2 hours to download ~ 200,000 products with GQL bulk query, totally acceptable. The real downside is when I had to update those 200,000 there is no bulk operation so it takes 25 hours to do that.

I am hoping Shopify gets around to allowing more bulk operations for mutations than is currently supported. Would be nice to speed up that process by a factor of 100 just by allowing bulk updates.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes