I am attempting to page througth the complete list of products using the since_id to get the following page.
However, I noticed that in some cases the results are NOT ordered by id and this completely breaks the paging using since_id. For testing I'm running this against a shop with 14 products in it, but depending on the limit I use I will get less than the expected products or duplicate products.
Here is a gist with the source code and the output https://gist.github.com/kurtharriger/f1e5e600fef3515c466bcfc717cc18cb
This fix for this should be as simple as sorting products by id, but I don't see this capability in the api. The only reliable workaround I can think of is to use a page limit of 1 but this isn't ideal since each request is rate limited a page size of 1 will make it take substantially longer to export all the products.
The node wrapper doesn't do anything special just light weight wrapper to add credentials and params to the request and parse response. The issue can also be replicated easily with curl:
curl -s https://searchspring-api-feed.myshopify.com/admin/products.json -H "accept: application/json" -H "<auth>" | jq ".products.id"
My assumption is that the list query does not have an ORDER BY id or is using a different ORDER BY clause such as created_at and the results are being returned in the order they are stored in your database. This could be shop specific and depend on how the data was initially added.
curl -s https://searchspring-api-feed.myshopify.com/admin/products.json -H "accept: application/json" -H "<auth>" | jq ".products.created_at"
If the list product query does not explicitly specify an order by id or is using order by created_at then paging using since_id will not work. In the result you will notice that 266914949 comes after 266915337 in the result set even though 266914949 < 266915337 and should have come first. Thus if I request a page size of 8 266915337 will be repeated and if I use a page size of 5 the last seen id is 266915241and 266914949 is never returned at all.
I just noticed a small but significant diff in your first request. When you use since_id=1 the results are ordered. If you do not specify since_id at all the results are unordered. Thus the following are different:
curl -s "https://searchspring-api-feed.myshopify.com/admin/products.json?fields=id&since_id=1&limit=8" -H "accept: application/json" -H "<auth>" | jq ".products.id"
curl -s "https://searchspring-api-feed.myshopify.com/admin/products.json?fields=id&limit=8" -H "accept: application/json" -H "<auth>" | jq ".products.id"
So I guess the short answer to my question is that you can force results to be ordered by id by specifying since_id. For first query use since_id=0 -- do not omit it.
Thank you @Kurt_Harriger
Shopify Team, the need of `$params['since_id'] = 1;` in the first request should be mentioned somewhere in the docs (maybe I've missed it, but couldn't find anything related on the page https://shopify.dev/docs/admin-api/rest/reference/products/product for example).