I'm working on a feature for an app for which I require to fetch all products belonging to a shop.
The new Bulk APIs seem like the perfect candidate to achieve this however I notice on the documentation page that `first`, `pageInfo` and `cursor` fields are not supported by the Bulk API.
What this effectively means is that I can't limit the number of products I receive through this.
My question is: how would this scale with stores which have hundres of thousands of products?
Would the Bulk API take care of fetching all the data and be able to process such a high number of products?
If not, would Bulk API not be relevant and do I need to stick with regular GraphQL based cursor pagination?
Not an official answer
I have used the bulk query over a very large set (100's of thousands). It took several hours (6+).
The runtime seems to be highly variable. Jobs that take less than 1 min occasionally take 10-15 min.
The runtime seems to correlate with output size -- pulling more fields seems to slow down the job.