Trouble with pagination when fetching products - API - python requests.get()

Solved
New Member
7 0 0

Using python i'm trying to use a private app to pull current inventory levels for from my Shopify storefront. My issue is with following the rel=next link. I seem to always get a 401 error when using request.get() on the next link. 

 

the actual error I get is: 

{'errors': '[API] Invalid API key or access token (unrecognized login or wrong password)'}

however if I paste that exact link into a browser i get the product data that I'm looking for returned. Am I missing something obvious here? 

 

edit: also the only information I want (barcode, capacity), but i'm currently downloading all product attributes, pulling the product ID and then using that list of ID's to pull the UPC and quantity information that I need. Is there a more efficient way to approach this problem? 

0 Likes

Success.

Community Manager
Community Manager
492 21 53

Hi @saml12345,

 

The error you are seeing only ever appears when, you guessed it, your method of authentication (access token or API key) is unrecognized. How is your app authenticated? If you're using basic auth, are you ensuring to add the API key and secret to the next link?

 

As for only requesting the info you want, this is the default behaviour for Graphql, but will need you to specify the `fields` query param in your request. So in your case, you can append `?fields=barcode,inventory_quantity` to receive only those two params.

0 Likes
Highlighted
New Member
7 0 0

I had not been re-adding the credentials to the next link, but managed to figure that out on my own. If the need to do that was mentioned in the documentation, I must have missed it. 

 

It sounds like graphQL is the better way to approach this problem, I'll start playing with that next. 

 

Thanks !

 

in case anyone else stumbles across this, this (very improvable) script worked for me: 

# get dataframe with product information and on hand inventory. Assumes that you're using a private app. 

import time
import requests
import pandas as pd

def get_all_upc():
    out = []  # the final output list of dictionaries to be converted to a DF
    continueprocess = True
    account = 'https://<APIkey:APIPASSWORD>@<Storename>' #Enter your own values here 
    url_start = f"{account}.myshopify.com/admin/api/2019-07/products.json"
    next_link2 = ''
    outer_loopcount = 0
    while continueprocess is True:
        if outer_loopcount is 0:
            url_final = url_start
        else:
            url_final = next_link2
        response = requests.get(url_final)
        link_header = response.headers['Link'].split(',')
        if 'next' in response.headers['Link']:
            for link in link_header:
                if 'next' in link:
                    r = response.json()
                    next_link = link[link.find("<") + 1:link.find(">")].replace("https://<Storename>", f"{account}")
                    next_link2 = f"{next_link}"
                    outer_loopcount += 1
                    for product in r['products']:
                        time.sleep(1 / 2)
                        product_id = product['id']
                        url2 = f"{account}.myshopify.com/admin/api/2019-07/products/{product_id}/variants.json"
                        product_variant_request = requests.get(url2)
                        variant_response = product_variant_request.json()
                        for variant in variant_response['variants']:
                            newline = {'product_id': variant['product_id'], 'variant_id': variant['id'], 'upc': variant['barcode'],
                                       'quantity': variant['inventory_quantity']}
                            out.append(newline)
                        continueprocess = True
        else:
            for link in link_header:
                if 'previous' in link:
                    r = response.json()
                    next_link = link[link.find("<") + 1:link.find(">")].replace("https://<storename>", f"{account}")
                    next_link2 = f"{next_link}"
                    outer_loopcount += 1

                    for product in r['products']:
                        time.sleep(1 / 2)
                        product_id = product['id']
                        url2 = f"{account}.myshopify.com/admin/api/2019-07/products/{product_id}/variants.json"
                        product_variant_request = requests.get(url2)
                        variant_response = product_variant_request.json()
                        for variant in variant_response['variants']:
                            newline = {'product_id': variant['product_id'], 'variant_id': variant['id'],
                                       'upc': variant['barcode'],
                                       'quantity': variant['inventory_quantity']}
                            out.append(newline)
                continueprocess = False

    df = pd.DataFrame(out)
    return df


df_barcodes = get_all_upc()
0 Likes
Shopify Partner
2 0 0

Thanks for posting this question. I assumed that the token embedded in the NEXT link would take care of authentication, but apparently not. The documentation of cursor pagination has a gap here (for private apps).

Adding (or rather, inserting) authentication into the next url works. 

 

 

  def authenticate_next_url(self,raw_url)->str:
        raw_url1 = raw_url[8:]
        return  f"https://{self.API_KEY}:{self.PASSWORD}@{raw_url1}"

 

0 Likes
Shopify Staff
Shopify Staff
1365 54 201

I just wanted to chime in to say that including your credentials in the URL is just one way of accomplishing Basic auth. I personally recommend using an Authorization header instead:

 

Authorization: Basic WU9VUi1BUEktS0VZOllPVVItQVBJLVNFQ1JFVA==

 

The Base64 string you see in the value is decoded as YOUR-API-KEY:YOUR-API-SECRET

 

Following the above pattern will ensure you do not have to insert your credentials into the URL for each page. You can set this up in your session while using shopify_python_api with this expression:

 

shopify.ShopifyResource.headers['Authorization'] = 'Basic WU9VUl9BUElfS0VZOllPVVJfU0VDUkVU'

Cheers.

0 Likes