Complete List of Products in Custom Collection - Pagination Problem

New Member
1 0 0

Community - 

I have searched this community as well as the StackOverflow and continue to be baffled by the following problem and I hope that you can assist.

I am using the Python APIs for Shopify and when I need to obtain a complete list of products in my store, I can use the following command to obtain a page of products which I can then interavly parse to extract the individual products:

curProdCnt = shopify.Product.count()
prodPages = int(curProdCnt/maxProdPerPage)+1

pageInfo = ''
for pageNum in range(1, prodPages+1):
    prodPage = shopify.Product.find(limit=250, page_info=pageInfo)

Note that there is a "page_info" argument that is passed into the shopify.Product.find function. The reason for this is that there is a max limit of 250 products per page that can be reported.  Therefore, I obtain the page information from the header, assign it the "pageInfo" and continue to with my for-loop until I am through all of the pages (and harvested all of the product information).

Just for reference, I use this ugly code to find "pageInfo:"

#  Header and PageInfo Collection Code
pageHeaderInfo = shopify.ShopifyResource.connection.response.headers.get('Link')
if pageHeaderInfo is not None:
    partsHeaderInfo = pageHeaderInfo.split(',')
    for part in partsHeaderInfo:
        if part.find('next') > 0:
            print("part:", part)
            pieces = part.split(';')
            smallpiece = pieces[0]
            smallpiece = smallpiece.strip('<>')
            subpieces = smallpiece.split('page_info=')
            pageInfo = subpieces[1] # this is what we are after.


All good so far and it all works well.   However, I also want to be able to do the same thing with the products in a Custom Collection.  And there is where I need your help.

Specifically, I have over 1100 products within a custom collection.  Since that number exceeds the page product limit of 250, I want to use the same approach as above.  But I keep hitting a limitation that I do not know how to overcome.

More specifically, when I try to use this line of code, I get a traceback:

prodPage = shopify.Product.find(collection_id=customCollectionID, limit=maxProdPerPage, page_info=pageInfo)

However, if I use the same piece of code, but remove the "page_info" argument, then it all works fine (no traceback):


But the drawback is that I only am able to collect the first 250 products within the custom collection (not all 1100).  

Does anybody know how to paginate through the products within a custom collection?

Thanks in advance for your asssitance.

FWIW, the first few lines of the traceback are as follows:

Traceback (most recent call last):
  File "E:\Dropbox\Silver Oak\Scripts and Utilities\", line 116, in <module>
    prodPage = shopify.Product.find(collection_id=customCollectionID, limit=maxProdPerPage, page_info=pageInfo)
  File "C:\Users\rrobi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\shopify\", line 211, in find
    collection = super(ShopifyResource, cls).find(id_=id_, from_=from_, **kwargs)
  File "C:\Users\rrobi\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\pyactiveresource\", line 386, in find
    return cls._find_every(from_=from_, **kwargs)
Not applicable
1625 0 0

In GraphQL you could use cursor based pagination. 

After each request you pass the new generated cursor (it is the last cursor in the list) to an new request. 

Let say you click next page:

1) You check if hasNextPage, if yes, then load data otherwise do nothing or disable button, what ever you like

2) For loading data you have a limit of 1000 cost in GraphQL. I recommend you to load 20 between 30 data per page. 

3) Save data in cache for previous loaded data. I am saving some data in MongoDB as helper.


const GET_PRODUCTS_BY_COLLECTION = `query ($name: String!, $after: String) {
    shop {
     collections(first: 1, query: $name) {
    edges {
      node {
        products(first: 15, after:$after) {
          pageInfo {
          edges {
            node {
              images(first: 12) {
                edges {
                  node {