How to sort results? Paging with since_id returns incomplete list or duplicate items

Highlighted
Tourist
3 0 2

I am attempting to page througth the complete list of products using the since_id to get the following page.

However, I noticed that in some cases the results are NOT ordered by id and this completely breaks the paging using since_id. For testing I'm running this against a shop with 14 products in it, but depending on the limit I use I will get less than the expected products or duplicate products.  

Here is a gist with the source code and the output https://gist.github.com/kurtharriger/f1e5e600fef3515c466bcfc717cc18cb

This fix for this should be as simple as sorting products by id, but I don't see this capability in the api.  The only reliable workaround I can think of is to use a page limit of 1 but this isn't ideal since each request is rate limited a page size of 1 will make it take substantially longer to export all the products. 

 

 

 

0 Likes
Highlighted
Shopify Staff
Shopify Staff
619 48 94

Hi Kurt,

I wasn't able to replicate this using a raw call through PostMan. Perhaps it's an issue specific to the Node wrapper you are using? If you can hook me up with the request ID of a recent call that shows this issue, I can take a look and see about getting some more info here.

Cheers,

0 Likes
Highlighted
Tourist
3 0 2

The node wrapper doesn't do anything special just light weight wrapper to add credentials and params to the request and parse response.  The issue can also be replicated easily with curl: 

curl -s https://searchspring-api-feed.myshopify.com/admin/products.json -H "accept: application/json" -H "<auth>" | jq ".products[].id"

266914221
266914773
266914649
266914681
266915241
266915301
266915337
266914949
266914985
266915005
266916257
266915681
266915717
305105929

My assumption is that the list query does not have an ORDER BY id or is using a different ORDER BY clause such as created_at and the results are being returned in the order they are stored in your database.  This could be shop specific and depend on how the data was initially added.  

curl -s https://searchspring-api-feed.myshopify.com/admin/products.json -H "accept: application/json" -H "<auth>" |   jq ".products[].created_at"
"2014-03-17T13:53:59-04:00"
"2014-03-17T13:54:33-04:00"
"2014-03-17T13:54:27-04:00"
"2014-03-17T13:54:28-04:00"
"2014-03-17T13:54:56-04:00"
"2014-03-17T13:54:59-04:00"
"2014-03-17T13:55:01-04:00"
"2014-03-17T13:54:42-04:00"
"2014-03-17T13:54:44-04:00"
"2014-03-17T13:54:45-04:00"
"2014-03-17T13:55:49-04:00"
"2014-03-17T13:55:18-04:00"
"2014-03-17T13:55:20-04:00"
"2014-05-05T13:50:28-04:00"

If the list product query does not explicitly specify an order by id or is using order by created_at then paging using since_id will not work. In the result you will notice that 266914949 comes after 266915337 in the result set even though 266914949 < 266915337 and should have come first.  Thus if I request a page size of 8 266915337 will be repeated and if I use a page size of 5 the last seen id is 266915241and  266914949 is never returned at all.

0 Likes
Highlighted
Shopify Staff
Shopify Staff
619 48 94

Hey Kurt,

I'm still unable to replicate this on any of my test shops. In all cases I am receiving products in order by id. I've just emailed you so we can troubleshoot this together.

Cheers,

0 Likes
Highlighted
Tourist
3 0 2

I just noticed a small but significant diff in your first request.  When you use since_id=1 the results are ordered.  If you do not specify since_id at all the results are unordered.  Thus the following are different:

 

curl -s  "https://searchspring-api-feed.myshopify.com/admin/products.json?fields=id&since_id=1&limit=8" -H "accept: application/json" -H "<auth>" | jq ".products[].id"

266914221

266914649

266914681

266914773

266914949

266914985

266915005

266915241

 

curl -s  "https://searchspring-api-feed.myshopify.com/admin/products.json?fields=id&limit=8" -H "accept: application/json" -H "<auth>" | jq ".products[].id"

266914221

266914773

266914649

266914681

266915241

266915301

266915337

266914949

 

So I guess the short answer to my question is that you can force results to be ordered by id by specifying since_id. For first query use since_id=0 -- do not omit it.

2 Likes
Highlighted
Shopify Partner
1 0 0

Thanks for the tips @Kurt_Harriger !

0 Likes
Highlighted
Shopify Expert
3 0 2

Thank you @Kurt_Harriger 

 

Shopify Team, the need of `$params['since_id'] = 1;` in the first request should be mentioned somewhere in the docs (maybe I've missed it, but couldn't find anything related on the page https://shopify.dev/docs/admin-api/rest/reference/products/product for example).

0 Likes