Product pagination miss data

M_MSC
New Member
6 0 0

Hi,

I need some help. I have an issue with product cursor based pagination not giving all products. Tried several API versions until 2020-07.

Making a product count /products/count.json?updated_at_min=2011-01-01T00:00:00
I get 557 products.

Getting them with /products.json?updated_at_min=2011-01-01T00:00:00
I get 381 products

Getting them with /products.json?limit=20&updated_at_min=2011-01-01T00:00:00
I get 147 products 

The number of products returned is always the same each time.

0 Likes
Greg_Kujawa
Shopify Partner
951 79 203

Are you using the newer cursor-based pagination? https://shopify.dev/tutorials/make-paginated-requests-to-rest-admin-api. When I am looking for to pull an entire recordset, I tend to lean toward the Shopify GraphQL bulk operation --> https://shopify.dev/tutorials/perform-bulk-operations-with-admin-api. It typically completes faster, I don't have to worry about the pagination aspect, and I can just download a flat file and iterate through the results in one shot!

0 Likes
M_MSC
New Member
6 0 0

Yes I use the newer cursor-based pagination.

Using GraphQL would be an option. Unfortunately it would mean development costs to rewrite the service.

0 Likes
M_MSC
New Member
6 0 0

Nobody has a solution with the cursor based REST API ?

0 Likes
Greg_Kujawa
Shopify Partner
951 79 203

I don't recall seeing any final confirmation or acknowledgment from the Shopify staff regarding some of the "holes" that exist in the cursor-based paginated REST API responses that were pointed out in other forum posts. So I can't speak to that.

But in terms of your mentioning development costs, I assume you mean that you are paying a third-party development resource for maintaining the custom/private app? If you're handling it yourself, there is a different approach involved. Mainly issuing a web request to a GraphQL API endpoint, passing along the proper web request body, and then downloading the web response to parse it and get the required dataset.  

0 Likes
M_MSC
New Member
6 0 0

I searched into forums and saw there were what seems the same problem for orders, which seemed resolved, but didn't find about products.

I saw the other way with GraphQL and thanks for pointing out this option. Yes I would make the changes, but this step getting products is part of a whole process, requiring having all products, which means async call is not possible without changing other process steps. This is where it hurts

0 Likes
Greg_Kujawa
Shopify Partner
951 79 203

What you're goal is would be difficult to attain hitting most API providers, not just Shopify's API endpoints. If you are looking to pull an entire product set in real-time to present to the user in total in a single shot isn't realistic, frankly. Even natively in Shopify's e-com, the Liquid code that provides the user a view of available products only presents them a "page" at a time. It's not like a shop with 10,000 products presents the user with all of them in a single shot. 

Most use cases for what you are looking to do are accomplished by providing the user a page of records at a time. Using the paginated cursor values you can then allow the user to advance forward a page, or step back a page. It's that cut and dried. And if you need to narrow down each page of records, your code would discard specific records that didn't meet the search criteria (e.g. - product category, in-stock only, etc.) and hide them from the user. 

0 Likes
M_MSC
New Member
6 0 0

Hi, this is what I explain in my post: problem with pagination. I never try getting all products in single shot and already have that process working with many other solutions.

0 Likes