I need some help. I have an issue with product cursor based pagination not giving all products. Tried several API versions until 2020-07.
Making a product count /products/count.json?updated_at_min=2011-01-01T00:00:00
I get 557 products.
Getting them with /products.json?updated_at_min=2011-01-01T00:00:00
I get 381 products
Getting them with /products.json?limit=20&updated_at_min=2011-01-01T00:00:00
I get 147 products
The number of products returned is always the same each time.
Are you using the newer cursor-based pagination? https://shopify.dev/tutorials/make-paginated-requests-to-rest-admin-api. When I am looking for to pull an entire recordset, I tend to lean toward the Shopify GraphQL bulk operation --> https://shopify.dev/tutorials/perform-bulk-operations-with-admin-api. It typically completes faster, I don't have to worry about the pagination aspect, and I can just download a flat file and iterate through the results in one shot!
I don't recall seeing any final confirmation or acknowledgment from the Shopify staff regarding some of the "holes" that exist in the cursor-based paginated REST API responses that were pointed out in other forum posts. So I can't speak to that.
But in terms of your mentioning development costs, I assume you mean that you are paying a third-party development resource for maintaining the custom/private app? If you're handling it yourself, there is a different approach involved. Mainly issuing a web request to a GraphQL API endpoint, passing along the proper web request body, and then downloading the web response to parse it and get the required dataset.
I searched into forums and saw there were what seems the same problem for orders, which seemed resolved, but didn't find about products.
I saw the other way with GraphQL and thanks for pointing out this option. Yes I would make the changes, but this step getting products is part of a whole process, requiring having all products, which means async call is not possible without changing other process steps. This is where it hurts
What you're goal is would be difficult to attain hitting most API providers, not just Shopify's API endpoints. If you are looking to pull an entire product set in real-time to present to the user in total in a single shot isn't realistic, frankly. Even natively in Shopify's e-com, the Liquid code that provides the user a view of available products only presents them a "page" at a time. It's not like a shop with 10,000 products presents the user with all of them in a single shot.
Most use cases for what you are looking to do are accomplished by providing the user a page of records at a time. Using the paginated cursor values you can then allow the user to advance forward a page, or step back a page. It's that cut and dried. And if you need to narrow down each page of records, your code would discard specific records that didn't meet the search criteria (e.g. - product category, in-stock only, etc.) and hide them from the user.