Get all metafields for products

[ReloadSEO]

This has been posted before, but the Shopify support hasn't answered it yet... And it is a good and reasonable question. 

I want to get all the metafields for all the products.

My understanding is that there is no API like:

/admin/products/metafields.json

The only one is like this:

/admin/products/#productid#/metafields.json

Therefore, the only way is to get all the metafields for all the products is to make one API call per product!

It is also impossible to cache the data, because it takes one API call per product just to check if the metafields for that product have been updated!  For large stores, it is handicapping API applications and forcing one call per product, resulting in extremely time-consuming fetches for tiny amounts of data.

Can you please add an API call that works like this?  /admin/products/metafields.json

It would return all the metafields for all the products, 250 at a time.  We can then page through in a reasonable amount of time.

Or, does anyone know some other workaround to get all the product metafields at once?

[Jamie_D_]

Hey ReloadSEO,

There's currently no way to achieve this through the current API. However, we're aware that this is something that developers are interested in and we're actively looking into finding a solution.

Keep an eye on the API Announcements forum for any updates regarding this.

Cheers,

Jamie

[John81]

Just run into this problem myself. Haven't been able to find a workaround sorry. I agree with your request to extend the API though and would like to add that /products should be able to return metafields_global_title_tag and metafields_global_description_tag (if it doesn't already, maybe there's a way to specify the fields value of the request for it?).

[Thomas47]

Hi Jamie,

please include the functionality so to get all metafields for certain ids (add ids[] paramter). I'd like to get the metafields for a list of products (i.e. metafields for all products in an order).

Must say I am very disappointed to see this lacking the API and also finding threads that are more than two and a half years old on this topic. 

Please commit to a production date (or sprint, release) for these updates.

Best regards,

Thomas

[Dan_Chow]

I am also interested in more information here. What is the intended use/meaning of this section of the docs pertaining to metafields? This makes it look like we should expect to be able to obtain product metafields at the /admin/metafields.json resource, and filter for conditions; however when I tried this on my store, only the 'store' metafields were returned. Accessing metafields a single product at a time is not at all an efficient use of the limited API call resource. Really need someway to obtain this information in a single, or a few, calls : 

https://texansaddles-com.myshopify.com/admin/metafields.json?metafield[owner_resource]=product

I was able to pull a single product metafields at the following address, but this is a duplicate of the resource at /admin/products/{id}/metafields.json:

https://texansaddles-com.myshopify.com/admin/metafields.json?metafield[owner_id]=6107959043&metafiel...

From the Docs:

https://docs.shopify.com/api/reference/metafield

GET/admin/metafields.json

Get metafields that belong to a product image

Get all metafields that belong to the images of a product

GET /admin/metafields.json?metafield[owner_id]=850703190&metafield[owner_resource]=product_image

Hide Response

HTTP/1.1 200 OK
{
  "metafields": [
    {
      "id": 625663657,
      "namespace": "translation",
      "key": "title_fr",
      "value": "tbn",
      "value_type": "string",
      "description": "French product image title",
      "owner_id": 850703190,
      "created_at": "2016-03-17T17:04:49-04:00",
      "updated_at": "2016-03-17T17:04:49-04:00",
      "owner_resource": "product_image"
    }
  ]
}

[Dan_Chow]

It was also frustrating to find that the metafields for a product are not included in that product's json when called at /admin/products/{id}.json, or as part of the list from /admin/products.json. This would have been an intuitive way to access metafields associated with products.

[devdeepakrawat]

Shopify must provide an option to include the metafields data in the product API response. It's really frustrating to make an extra call to fetch metafields for a  product after fetching the product.

[Mitchmaster]

I find it hard to believe that Shopify has been "researching a solution" to this problem for years. It should take 10 minutes of programming to solve given that identical capabilities already exist at other levels of the API. Having to do a separate curl call to find out if each product has a given metafiled makes this useless for a live page - it can take up to a minute to load. Please expedite a solution!

[shul]

And its 2021 and the "research" continues.. Such a strait forward  feature..

[Jonathan-HA]

+1 for this. Would also be nice to be able to specifiy a list of product_ids and get the metafields for them.

We have a CSV exporter app and one of our customers asked if we could include metafields in the report. With the current API implementation, it's too expensive to make an API call for each unique product included in the report.

[bolle]

Any updates here?
This is needed, otherwise the metafields are stuck in a far off land with the API call limit.

[bolle]

Thanks HunkyBill. 

Maybe you could help me figure out the logic for this.

I'm creating a private app for my client. I was hoping to use metafields to allow us to change a binary option (true/false) for each customer.

In my app I list all of the customers in the shop, and ideally I'd like to display this metafield within the list as well.

Ex. First Name, Last Name, Address, Binary Metafield, etc

I obviously have the /customers.json GET working properly with all customer info, now I'm working on the metafields.

How could I grab this metafield for each customer, without making x # of requests (depending on how many customers)?

I hope this isn't an extremely simple solution, but I can't think of any way around the API limit for this. 
I've just started using the Shopify API (yesterday) so I am a bit new at this.

Thanks HunkyBill!

[HunkyBill]

You get 250 customers per API call. for each customer, add a metafield. 

simple. 

You can modify the theme to display the metafield values too, in the customer Liquid files. 

[bolle]

Sorry, I meant I need to GRAB that metafield value for each customer in the API.

My original thought - grab customers.json , for each customer in there, grab the id and send another request to Shopify for metafields.json with that user id. 

That should work technically, but if there are 40+ customers I'm going to hit the limit quickly.

[HunkyBill]

You can GET 250 customers in one API call. You won't hit the limit if you page through them like that. And even if you did throw 40 calls out (10,000 customers later).. you still get 2/second. About the API limits... there is a ton of code out there about how to deal with limits in any language except probably COBOL... 

So get 250 customers at a time, and deal with their metafields... and if you hit the limit... big deal.. you wait a 0.5 seconds to make your next API call... 

[bolle]

Wonderful. Thanks I'll try it out.

Bad pseudocode below - 

Would something like this work (away from pc at the moment):
 

GET customers.json (lets say all 250 customers) {

for each customer in customer.json {

GET metafields.json?metafield[owner_id]=user.id&metafield[owner_resource]=customer {

return data;

}
}

[HunkyBill]

yes... that will work fine... 

[Paul_Younger]

If I understand correctly, I should be able to get up to 250 product metafields with this call:

admin/metafields.json?metafield[owner_resource]=product

However, I get this as a response: (metafields do exist for products)

 {"metafields":[]}

[HunkyBill]

I think you're better off getting a product, and asking for all it's metafields. Asking for metafields that have an owner is kind of chaotic. I think that only works if you also provide an ID. 

[Dennis2]

I would say something like this should work:

/admin/metafields.json?metafield[owner_resource]=product

This way you only request resources for product and you could even make namespace required for example. Or just providing a list of resource IDs (products IDs for example) would work as well, I guess. Al long as you don't have to download all metafields for each resource one at a time would work for me 🙂

[Paul_Younger]

Dennis, that did not work for me.  Note my previous post.  Results were empty.

[HunkyBill]

As I told you. Asking for metafields by using some generic resource owner like "product" is chaos. There is little to no chance Shopify would support that as an ad-hoc query as it is expensive to gather those resources easily and cheaply. 

So function in the normal way. Get your products of interest first, and then get their metafields. It is not complex and it is guaranteed to work.

[Dennis2]

@HunkyBill I know, but doing that would require thouthands of requests. And why would it be difficult to gather the resources? If you are ONLY downloading one type, like products?

In any case, if Shopify dev team is around, I'd like to hear what they say about this.

[Dennis2]

I don't want to start a war here... but querying 400 times 250 results ALWAYS needs less resources than querying 10000 times 1 result. I'm not talking about grabbing ALL metafields, but metafiels bij owner_resource. Or even by owner_resource/namespace combo.

[HunkyBill]

Computers are governed by math. As you wisely point out. You can request 100,000 products with a mere 400 API calls.  

So that person now has 100,000 products. And wow... that took all of what, 180 seconds? Holy inconvenience Batman!  Moving on, now that said coder has 100,000 products to play with, indeed, getting the Metafields for each one is a chore. Could mean 100,000 more calls right?

So you learn to layer and filter in your API calls, so as to minimize calls. In most cases, it works out fine.

No one wants to start a war. It is just that to move someone along quicker to their goal, we have to point out the way for them to go. And if they feel they don't like it, well, that is what we call tough bananas. At this time, metafields are not a simple resource you can just have, without doing a little extra work.  

[ClementG]

Upvoting this. I also need to bulk load meta fields.

[Ruslan]

I have many products (100 000 for example). But Shopify doesn't allow to store SKU (why?!?!?!?!?!?!?!?!?!?!) and as result I must store SKU inside metafields and any time I must fetch metafields for any products to check if product available in shop or not. Now I have API calls = products pages count + products count just to read available data. If I have 100 000 products - I must use more than 100 000 calls because I can't use pagination with next parameters:

/admin/metafields.json?metafield[key]=SKU&metafield[owner_resource]=product

Now API have limits: 2 requests per second.

100 000 products / 2 requests / 60 seconds / 60 minutes = 14 HOURS!

During this time my product list can be changed many times.

[Jason]

 But Shopify doesn't allow to store SKU

Sure it does. You store things like SKU code on the variant.

So when creating a product:

POST /admin/products.json
{
  "product": {
    "title": "Burton Custom Freestyle 151",
    "body_html": "<strong>Good snowboard!<\/strong>",
    "vendor": "Burton",
    "product_type": "Snowboard",
    "variants": [
      {
        "option1": "First",
        "price": "10.00",
        "sku": "123"
      },
      {
        "option1": "Second",
        "price": "20.00",
        "sku": "123"
      }
    ]
  }
}

[Ruslan]

Product SKU is differ than variants (combinations) SKU and must be saved in other place for products with variants. Surprice!

[Ruslan]

Variants count restriction is a problem too. 3 options is aren't enough.

[Channable]

Hello,

We are running into the same exact problem. For several of our customers we would like to retrieve and process the meta datafields every day. However, with the current restrictions this is just not an option because this could result in 100k+ requests every day. 

We hope there will be found a solution to this problem in the future.

[Jason]

^ What are you actually doing that would require you to have 100k API calls daily? I'd like to understand context as there may be smarter ways to approach things.

[Channable]

As a datafeed management company we currently download all the product data from Shopify, process this (e.g remove products which are out of stock, add the color of the product to the title etc) and send the modified data to "export channels" such as Amazon and Google Shopping. We don't push any data to Shopify, only retrieve it every day.

Some of our Shopify customers store data in the meta data fields, for example color, sku's etc. and our customers want to use this information as to better optimize their ads on the "export channels".

We streamline this process as much as possible as to make it as easy as possible for the customer. When a customer changes an item's description (or stock changes after a sell), this  will automatically be changed on the export channels. We want to do something similar for the meta data fields.

Some customers have a lot of products in their stores and thus getting all the meta_fields for all the products every day would be a huge amount of api calls.

We are aware that the amount of api calls could be reduced by giving a customer an option to fetch the meta_data fields in Channable instead of doing this every day. However, we would like to keep our strucuture the same for all ecommerce shops and this is not required for other shops. Furthermore, this would require a big change in our backend structure.

I hope this clears things up.

[HunkyBill]

This one made me giggle. Please Shopify, could you alter your systems to match mine so I don't have to work anymore? Would be swell!! Thanks and tata... have a nice day.

[Perice]

+1 for this

One call per product to get the metafields is way too much when you have to deal with thousands of products.

[HunkyBill]

Soon enough we'll get GraphQL calls and then we can all get the associated metafield resources needed, and this whole silly thread can go away forever.

[ClementG]

Do you have non public knowledge about this?

As far as I understand GraphQL is currently limited to "customer facing" entities.

It's not clear to me exactly where the boundary between customer facing and back office entities is, but I'd think that metafields are considered "back office"..

[HunkyBill]

No dude, I have no secret info. But that is almost certainly where things are going. Eventually we will have GraphQL access to objects and resources. Until then, we deal with the RESTful API and do 100,000 API calls in ~14 hours. YMMV. 

Therefore, whether we like it or not, we understand that using Metafields is non-performant when your call count is high. Use a different strategy. That is the secret. Knowing how to squeeze performance. 

[HunkyBill]

I get X calls per week from merchants that complain to me about Apps that bungle this performance issue, and there is zilch I can do for them, except to tell them they cannot use an App that is architected to fail on big product counts. 

Using a lowest fruit hanging from the tree approach (sometimes Metafields), is clearly easy, but has hazards. It is obvious from the get-go, so expecting magic and a sudden elimination of this performance is not likely. We could wake up tomorrow with metafields available, but then again, if it was easy, we would've had that 8 years ago. Clearly, it is best to wait for the internal wiring to catch up, and when it does, using Metafields will likely be suitable for a wide range of uses currently not practical.

[Joel-Reeds]

+1

Pulling in 1,000 products with their metafields currently takes around 10 minutes to complete because of the API call limit.

As others have suggested, the ability the pull metafields based on owner_resource and namespace would significantly reduce the number of calls required.

Something simple such as 4 metafields returned per product could reduce the total number of API calls from 1,004 to 20.

[HunkyBill]

It's easy to play a numbers game, but you know what will happen. Some joker will figure out that with the owner resource and namespace, they can jam 100, 500, heck even 1000+ things on another resource. So what is the point? You can ask for your metafields, and if you have 2 or 3 or 7 or some modest number sure, it's a simple join, la-dee-da, off we go. But you know people will abuse this. And so you are just offloading poor performance from one area to another. And granted, it may bridge certain gaps, it will almost certainly bog down others. 

Don't forget, peoplez can writez terrible codez if you let them, and flush Shopify internal performance down the drain, and so the wiring for this has to be done carefully... so ya.. it's a nice thought experiment.. but not realistic for reasons probably due to DB structure, and SQL efficiency, and the like.

[Jason]

Joel - why does 10 minutes matter? It's not like you're sitting there processing them by hand. Can you explain the pain point you're seeing? If this thread is going to keep being revived I'd like to at least understand the context / reasons / pain etc.

If you are grabbing that content all the time why not just cache that on your end and save the call entirely?

[ClementG]

I must say I don't always understand why many people seem to very often push back on feature requests.

This is just that, a feature request.

I don't expect Shopify to implement all of them. I trust they have the intelligence and  inside knowledge to prioritize the various requests based on the number of people asking for it, complexity of implementation, risk, peformance, etc... If they don't implement it that is fine, but are we allowed to at least express that we would like some enhancement?

Yes the work around is acceptable, but why wouldn't we strive for better? Again I don't expect Shopify to implement all requests, but it seems to me they are the one that have the knowledge to priortize. One important input is the number of people requesting it in the forums.

IMO Jamie's answer was good enough, but +1 are ok.

I really wish that Shopify would use something like uservoice to keep track feature requests.

Something like Xero does: https://xero.uservoice.com/forums/5528-xero-accounting-api

Regarding the question at hand, my own context is that I want to generate a large report that shows some metafields.

[Jason]

My comments are not push back, though I do like more info on the whys.
Context is important and this thread is seen internally.

[Joel-Reeds]

I cache the retrieved data once it's received so the call doesn't go out often.  However should the shop owner wish to resync their product data manually then it's a 10 minute wait while everything is re-cached.

For now to get around this long wait time I'm just storing all custom meta data on my end and skipping Shopify's metafields.

I can see HunkyBill's point that should the feature be introduced it could have a negative impact on Shopify as people might cram all their misc data into metafields.

[Jason]

However should the shop owner wish to resync their product data manually then it's a 10 minute wait while everything is re-cached.

If you're storing the product updated_at value you can use that as an early indicator if you even need to sync a product or not. If it matches your last cache time you know the metafields haven't changed. May save you a call or two.

[Joel-Reeds]

Good point, thanks.

[Ruslan]

Problem will never be solved because developers doesn't understand why other CMS have SKU by default in main table (like product id and product name). Attributes (include features and combinations) alwais contained in other tables but product SKU available as main field any time.
We have simple behaviour: user create new product. Fill all data one time. And next time just search product by SKU to update price and quantity.
We can't cache data because if user change something directly in shop: data from cache will lie and user will have invalid data.
Also local cache means that customer must write functional identical to your CMS and backend to it. There is no need to buy CMS if you must develop new CMS just to work with CMS.
Your CMS is not the first CMS with which we work and we know these problems.

[Ruslan]

Problem will never be solved because developers doesn't understand why other CMS have SKU by default in main table (like product id and product name). Attributes (include features and combinations) alwais contained in other tables but product SKU available as main field any time.
We have simple behaviour: user create new product. Fill all data one time. And next time just search product by SKU to update price and quantity.
We can't cache data because if user change something directly in shop: data from cache will lie and user will have invalid data.
Also local cache means that customer must write functional identical to your CMS and backend to it. There is no need to buy CMS if you must develop new CMS just to work with CMS.
Your CMS is not the first CMS with which we work and we know these problems.

[Tien_Do]

Initially I also wanted to get metafields with a call to products.json, but after read all replies here, I changed my mind 🙂

+1 HunkyBill 

[fadi]

Hey Guys,

Is there any update on this issue?

I am trying to read all products from Shopify Admin API. I also need to include the product metafields.

I have abou 7000 products and if I use the currentl solution it will took almost an hour to get these products, which is very very long time.

Kiflay

[HunkyBill]

@fadi. 7000 products is 28 API calls. You get about 40 without delay, so without dwelling on what that exactly means, consider it quick, not very very long time.

Additionally, now with GraphQL at your disposal, you are free to get your products with the metafield data you crave at the same time. So it seems you are one lucky customer! Not only can you get your get products fast, but you no longer need to make secondary calls for the metafields.

[fadi]

@HUnkyBIll Thanks for the quick response.

You right to get all products is 28 API call.

To get metafield for each product will be

(7000/2) seconds

I  have to fire a single request for each product. it Will take more than an hour.

So am not sure what do you mean it will be only 28 API

[HunkyBill]

@fadi.

You missed something crucial in my explanation. Read it again. Closely. With GraphQL you can retrieve a product AND its metafield data of interest to you. 

So your math is off. I told you you now get to skip those 7000 extra calls. Pay attention man! I waste no words. They all matter.

[fadi]

@HunkBill,

Look man I didn't ask you an advice with what I can and I can't do with GraphQL.My question is related to the REST API and metafields

You said 

"7000 products is 28 API calls. You get about 40 without delay, so without dwelling on what that exactly means, consider it quick, not very very long time." 

So how does getting the metafields for each product is not taking long calls for my example about an hour.

[Alex_B]

Allow me to interject here.

With the new GraphQL Admin API, it is possible to obtain metafields for products in a way that does not add to your REST Admin API call limit. It is possible to combine your existing REST calls, which gathers a ton of data at the Product level, and then use the "admin_graphql_api_id" from that call to query the metafields for those products using GraphQL.

Starting with the assumption that you already have the REST Admin API call to the Product endpoint, which gets you the "admin_graphql_api_id" of the individual products, you can use a GraphQL query similar to the following to get the metafields.

{
  product(id: "gid://shopify/Product/1321540321336"){
    metafields(first:10) {
      edges {
		node {
          id
          key
          value
          valueType
          namespace
        }
        cursor
      }
      pageInfo {
        hasNextPage
      }
    }
  }
}

The "cursor" and "pageInfo/hasNextPage" fields are there to help you determine if pagination is necessary on the metafields.

Performing this query in my test store on a product that has 2 metafields costs 5, which means with the GraphQL Admin APIs restoreRate of 50, you could get metafields on 10 products per second without getting throttled.

For more information on the GraphQL Admin API call limits, see the docs here.

[fadi]

Thanks Alex 

That could help.

To get product metafields  using the REST API there is no solution other than firing a single request for each product. Is this right Alex?

[HunkyBill]

@ Fadi, you get spoon-fed an answer that saves your bacon and you get upset? Chill out. If you can make a call to the RestAPI endpoints, you can make a GraphQL call. And as Alex and I told you, with ONE call you can get 250 products AND their metafields. 

So move on, write your code and be happy someone filled you in and gave you free correct advice. It is common courtesy to be gracious and not defensive, as if I did something to offend you.

[Bharath]

Hi All,

As everyone recommended GraphQL for better querying, we thought of trying this but from our initial testing, it seems like it's not giving a big advantage to us in querying.

using a REST API, we can get one product's 250 metafields in one call using "admin/products/{{id}}/metafields.json?limit=250" call. When it comes to GraphQL, the query cost is controlling us to have a maximum of 3 products at a time in the call i.e 

{
  products(first: 3) {
    edges {
      node {
        legacyResourceId
        metafields (first: 250) {
          edges {
            node {
              legacyResourceId
              value
              namespace
              key
            }
          }
        }
      }
    }
  }
}

1. Is there any better way of querying for this purpose?
2. Also, all resources in GraphQL don't have the main filter supported by REST API i.e ID. So how GraphQL is recommended if we have to fetch details for certain ids?
3. As per Shopify, admin_graphql_api_id is not constant and it's not recommended to prepare the gid structure programmatically. So Shopify is expecting us to make a REST API call first to know the gids and then use GraphQL for querying?

[Bharath]

Hi All, 

Can anyone respond to these queries?

[steffenmande]

Kind of amazed that this issue is not made easier after years.

In my use case i just need to get a list of all metafields, to know what the partner has created already, so i can update their content.

Getting all their metafields is a pain in the ass though.

Should be made possible to get all metafrields with a resource owner. ie. all metafields on a owner_resource=product, or variant or image whatever.

[AndrewVelo]

The point that has been made by all the contributors are totally valid and I view the last response from a so called expect as both narrow and unhelpful in the extreme. There is a clear requirement for this resource from a large number of customers, please address the problem rather than dismissing it in a banal manner.

[SigProdGroup]

+1.

Lazy developers trying to push Shopify to bend to their system? This is demeaning, and oblivious to the state of the global economy.

Shopify currently lives in the ecomm sector, a whopping 8% of the retail market. Shopify wants to be the platform we use to drive a movement of the remaining 92% of the market, that is big business retail, from brick and mortar to ecommerce. Wal-Mart, Target, etc. these are the kings, not Shopify. To capture that business, Shopify must accommodate big business ERP databases and the breadth of complexity that retail chains demand. That means keeping item data in sync as much as keeping orders in sync. This metafield API issue is one pain point that Shopify has simply failed to accommodate so far.

The industry is moving this way. If you view us as pushing Shopify around, that's because we are. If Shopify doesn't oblige, they will simply not be relevant in this small piece of the market. Massive API pulls against metadata fields is not abuse. It's use in this case. And I expect we'll see similar "abuse" against other endpoints before the push is over. It's up to Shopify to figure out how to manage that without folding to performance issues.

I hope to see Shopify manage this quickly so that I can rely on Shopify as more business moves onto this platform.

[sipsynthemesc]

Just here to credit SigProdGroup. 

Shopify is the wordpress of e-commerce and has been that since Tobi left as CTO. I only ever find myself stumbling across these posts when trying to get more from the platform. Quite frankly, Shopify nowadays is a f*cking mess, constantly publishing unfinished features to the system or as we bare witnessed with "Shopify Mail" an overhyped pile of trash. Fear not my fellow engineers for the blue haired genderless support team is on your side who have zero f*cking clue.

Anyway, if any of you stumble across this thread and read this post, take a look at Centra which is what I believe to be the future of e-commerce. It flaunts a real headless e-commerce solution, none of this bitchery you get from the Shopify cretins. 

[ScreenStaring]

While still time-consuming here is a script to download the metafields of all products in a store to a JSONL file: https://gist.github.com/sshaw/dea140dd7a6afc1a4de77e7158605ee7. It requires 2 supplementary programs. See the script for more information.