cursor-based pagination

Solved
domsteil
Shopify Partner
7 0 1
I am trying to pass the net new nextPageToken value to this function and I am getting the same page back instead of the next page.

 

The First Next Page Token is: eyJsYXN0X2lkIjo0MzIyMTUzNTI5Mzc5LCJsYXN0X3ZhbHVlIjoiODBzIE92ZXJzaXplZCBcIlBoYXJhb2hzXCIgVGVlIC0gV2FzaGVkIEJsYWNrIiwiZGlyZWN0aW9uIjoibmV4dCJ9
First Page of Products Pushed to Array
Page: 3 Next Token is: eyJkaXJlY3Rpb24iOiJuZXh0IiwibGFzdF9pZCI6MjMzMTM3MzEwOTMwNiwibGFzdF92YWx1ZSI6IkNvbWZvcnQgU3RyZXRjaCBIaWdoIFJpc2UgU3RvdmUgUGlwZSAtIExpZ2h0IFN0b25lIn0
Page 3 of Products Pushed to Array
Page: 3 Next Token is: eyJkaXJlY3Rpb24iOiJuZXh0IiwibGFzdF9pZCI6MjMzMTM3MzEwOTMwNiwibGFzdF92YWx1ZSI6IkNvbWZvcnQgU3RyZXRjaCBIaWdoIFJpc2UgU3RvdmUgUGlwZSAtIExpZ2h0IFN0b25lIn0
Page 3 of Products Pushed to Array

 

 
Any ideas on what I might need to change?      
 

 

    let limit = null // no initial limit
    let nextPageToken = null; // no initial nextPageToken
    // Set Initial Page
    let page = 1;
    // Set Max Page Number
    let maxPage = 3;   
    // Create Array to hold Product Records
    let products = [];
  
    // Axios Config
    let axiosConfig = {
        headers: {
            "Access-Control-Allow-Origin": "*",
            "Access-Control-Allow-Methods": "POST,GET,OPTIONS, PUT, DELETE",
            "Accept-Encoding": "gzip, deflate",
            "Content-Type": "application/json"
        },
        params: {
          limit: 250,
          page_info: nextPageToken
        }
      };

// Get Shopify Product Function
      const getShopifyProducts = async (nextPageToken) => {
          page += 1
        while (nextPageToken !== undefined  && page < maxPage) {
            axios.get(`https://{id:{key}@{shopName}.myshopify.com/admin/api/2020-07/products.json?limit=${limit}&page_info=${nextPageToken}; rel="next"`, axiosConfig)
            .then(response => {
              // Capture Next Page Token 
              let nextPageToken = response.headers.link;
              // If there's a previous link, remove the first part of the string entirely
              if (nextPageToken.indexOf(`rel="previous"`) > -1) {
                nextPageToken = nextPageToken.substr(nextPageToken.indexOf(",") + 2, nextPageToken.length);
                // Parse the remaining string for the actual link
                nextPageToken = nextPageToken.substr(85, 147);
                // Console Next Page Token
                console.log('Page: ' + page + ' Next Token is: ' + nextPageToken)
                // Push to Product Array
                products.push(response.data.products);
                console.log('Page ' + page + ' of Products Pushed to Array');
              } else {
                console.log("Looped through all the Product Pages, that is all of the products.")
              }
              }).then(
                getShopifyProducts(nextPageToken)
            )
          }
        };

 

0 Likes
LiamsLin
Shopify Partner
5 0 0

Hi , the token you are using is the same, 

the next page will need the token from the response Header called 'Link' 

you have to get that value, and call the url with that ID, 

then keep going until there are no Headers with the word 'next' 

 heres a snippet of mine:

 

 var url = baseurl + "products.json?fields=title,variants&limit=250";

  while (!string.IsNullOrWhiteSpace(url))
                {
                    //url = baseurl + "products.json?fields=title,variants&limit=250&page=" + page;
                    var get = client.GetAsync(url);
                    var response = get.Result;
                    var content = response.Content.ReadAsAsync<Products>();
                    var p = content.Result;
                    products.AddRange(p.products);

                    var wait = CallLimitWait(response.Headers);
                    if (wait > 0)
                        Thread.Sleep(wait);


                    //***next page Link  read Link to next page (if there is one)

                    url = null;
                    if (response.Headers.Contains("Link"))
                    {
                        List<string> LinkValues = response.Headers.GetValues("Link").ToList();

                        if (LinkValues.Count == 1) // There should be only one Header
                        {
                            string LinkCsv = LinkValues.Last();

                            if (!String.IsNullOrWhiteSpace(LinkCsv))
                            {
                                string PageLink = LinkCsv.Split(',').Last();

                                string[] LinkParts = PageLink.Split(';');

                                string LinkPartUrl = LinkParts[0];

                                string LinkPartRel = LinkParts[1];

                                if (LinkPartRel.Contains("next"))
                                {
                                    // Reaching here means there are more pages to come

                                    int startChevronIndex = LinkPartUrl.IndexOf('<') + 1;
                                    int LengthOfSubString = LinkPartUrl.IndexOf('>') - startChevronIndex;
                                    url = LinkPartUrl.Substring(startChevronIndex, LengthOfSubString);

                                }

                            }

                        }
                    }
                    ///***end next page linkk
              

 

0 Likes
domsteil
Shopify Partner
7 0 1

Hi LiamsLin,

I solved my problem with this:

      function getShopifyProducts(nextPageToken) {
            axios.get(`https://{id}:{key}@{shop}.myshopify.com/admin/api/2020-07/products.json?limit=${limit}&page_info=${nextPageToken}; rel="next"`, axiosConfig)
            .then(function(response) {
              let nextPageToken = response.headers.link;
              // If there's a previous link, remove the first part of the string entirely
              if (nextPageToken !== undefined && nextPageToken.indexOf(`rel="previous"`) > -1 && page < maxPage) {
                nextPageToken = nextPageToken.substr(nextPageToken.indexOf(",") + 2, nextPageToken.length);
                // Parse the remaining string for the actual link
                nextPageToken = nextPageToken.substr(1, nextPageToken.indexOf(">") - 1);
                // Console Next Page Token
                console.log('Page: ' + page + ' Next Token is: ' + nextPageToken)
                // Push to Product Array
                products.push(response.data.products);
                console.log('Page ' + page + ' of Products Pushed to Array');
                page += 1
                return getShopifyProducts(nextPageToken);
              } else {
                console.log("Looped through all the Product Pages, that is all of the products.")
              }
            })
        };
LiamsLin
Shopify Partner
5 0 0

thanks for letting me know, 

what platform do you run the javascript on ? node?  

 

0 Likes
j0nd0n7
New Member
1 0 0

Just wanted to share this (JS) parsing code using regex.

const CURSOR_REGEX = /page_info=([^>]+)>; rel="([^"]+)/g;

function getPaging(response) {
  const paging = {};
  if (response.headers.link) {
    let match;
    while ((match = CURSOR_REGEX.exec(response.headers.link)) !== null) {
      // match[2] is "next" or "previous"
      // match[1] is the cursor (page_info)
      paging[match[2]] = match[1];
    }
  }
  // returns {next: 'the next cursor', previous: 'the previous cursor'}
  return paging;
}

 

0 Likes
j_medina
New Member
2 0 0

Hey domsteil  
Can you tell me how you get the initial "nextPageToken"? When I try to run this fuction it doesnt work because it cant find the token to start. 

0 Likes
domsteil
Shopify Partner
7 0 1
I just hard-coded the limit based on the number of products.
0 Likes