[Bulk API] Fetching all products

Highlighted
Shopify Partner
8 0 1

Hello folks!

 

I'm working on a feature for an app for which I require to fetch all products belonging to a shop.

The new Bulk APIs seem like the perfect candidate to achieve this however I notice on the documentation page that `first`, `pageInfo` and `cursor` fields are not supported by the Bulk API.

What this effectively means is that I can't limit the number of products I receive through this.

 

My question is: how would this scale with stores which have hundres of thousands of products?

Would the Bulk API take care of fetching all the data and be able to process such a high number of products?

If not, would Bulk API not be relevant and do I need to stick with regular GraphQL based cursor pagination?

Many thanks!

0 Likes
Highlighted
Tourist
5 1 1

Same question as above. 

Will the Bulk Query allow us to fetch about 30,000 products in one go?

How long would this query take? Would we need to expect this to happen in 1 minute? in 2 minutes?

 

 

1 Like
Highlighted
Shopify Partner
8 0 1

Bumping this for an official answer.

0 Likes
Highlighted
Shopify Partner
13 0 1

Not an official answer

 

I have used the bulk query over a very large set (100's of thousands). It took several hours (6+).

 

The runtime seems to be highly variable. Jobs that take less than 1 min occasionally take 10-15 min. 

 

The runtime seems to correlate with output size -- pulling more fields seems to slow down the job. 

1 Like