Get all metafields for products

[ReloadSEO]

This has been posted before, but the Shopify support hasn't answered it yet... And it is a good and reasonable question. 

I want to get all the metafields for all the products.

My understanding is that there is no API like:

/admin/products/metafields.json

The only one is like this:

/admin/products/#productid#/metafields.json

Therefore, the only way is to get all the metafields for all the products is to make one API call per product!

It is also impossible to cache the data, because it takes one API call per product just to check if the metafields for that product have been updated!  For large stores, it is handicapping API applications and forcing one call per product, resulting in extremely time-consuming fetches for tiny amounts of data.

Can you please add an API call that works like this?  /admin/products/metafields.json

It would return all the metafields for all the products, 250 at a time.  We can then page through in a reasonable amount of time.

Or, does anyone know some other workaround to get all the product metafields at once?

[Dennis2]

@HunkyBill I know, but doing that would require thouthands of requests. And why would it be difficult to gather the resources? If you are ONLY downloading one type, like products?

In any case, if Shopify dev team is around, I'd like to hear what they say about this.

[Dennis2]

I don't want to start a war here... but querying 400 times 250 results ALWAYS needs less resources than querying 10000 times 1 result. I'm not talking about grabbing ALL metafields, but metafiels bij owner_resource. Or even by owner_resource/namespace combo.

[HunkyBill]

Computers are governed by math. As you wisely point out. You can request 100,000 products with a mere 400 API calls.  

So that person now has 100,000 products. And wow... that took all of what, 180 seconds? Holy inconvenience Batman!  Moving on, now that said coder has 100,000 products to play with, indeed, getting the Metafields for each one is a chore. Could mean 100,000 more calls right?

So you learn to layer and filter in your API calls, so as to minimize calls. In most cases, it works out fine.

No one wants to start a war. It is just that to move someone along quicker to their goal, we have to point out the way for them to go. And if they feel they don't like it, well, that is what we call tough bananas. At this time, metafields are not a simple resource you can just have, without doing a little extra work.  

[ClementG]

Upvoting this. I also need to bulk load meta fields.

[Ruslan]

I have many products (100 000 for example). But Shopify doesn't allow to store SKU (why?!?!?!?!?!?!?!?!?!?!) and as result I must store SKU inside metafields and any time I must fetch metafields for any products to check if product available in shop or not. Now I have API calls = products pages count + products count just to read available data. If I have 100 000 products - I must use more than 100 000 calls because I can't use pagination with next parameters:

/admin/metafields.json?metafield[key]=SKU&metafield[owner_resource]=product

Now API have limits: 2 requests per second.

100 000 products / 2 requests / 60 seconds / 60 minutes = 14 HOURS!

During this time my product list can be changed many times.

[Jason]

 But Shopify doesn't allow to store SKU

Sure it does. You store things like SKU code on the variant.

So when creating a product:

POST /admin/products.json
{
  "product": {
    "title": "Burton Custom Freestyle 151",
    "body_html": "<strong>Good snowboard!<\/strong>",
    "vendor": "Burton",
    "product_type": "Snowboard",
    "variants": [
      {
        "option1": "First",
        "price": "10.00",
        "sku": "123"
      },
      {
        "option1": "Second",
        "price": "20.00",
        "sku": "123"
      }
    ]
  }
}

[Ruslan]

Product SKU is differ than variants (combinations) SKU and must be saved in other place for products with variants. Surprice!

[Ruslan]

Variants count restriction is a problem too. 3 options is aren't enough.

[Channable]

Hello,

We are running into the same exact problem. For several of our customers we would like to retrieve and process the meta datafields every day. However, with the current restrictions this is just not an option because this could result in 100k+ requests every day. 

We hope there will be found a solution to this problem in the future.

[Jason]

^ What are you actually doing that would require you to have 100k API calls daily? I'd like to understand context as there may be smarter ways to approach things.

[Channable]

As a datafeed management company we currently download all the product data from Shopify, process this (e.g remove products which are out of stock, add the color of the product to the title etc) and send the modified data to "export channels" such as Amazon and Google Shopping. We don't push any data to Shopify, only retrieve it every day.

Some of our Shopify customers store data in the meta data fields, for example color, sku's etc. and our customers want to use this information as to better optimize their ads on the "export channels".

We streamline this process as much as possible as to make it as easy as possible for the customer. When a customer changes an item's description (or stock changes after a sell), this  will automatically be changed on the export channels. We want to do something similar for the meta data fields.

Some customers have a lot of products in their stores and thus getting all the meta_fields for all the products every day would be a huge amount of api calls.

We are aware that the amount of api calls could be reduced by giving a customer an option to fetch the meta_data fields in Channable instead of doing this every day. However, we would like to keep our strucuture the same for all ecommerce shops and this is not required for other shops. Furthermore, this would require a big change in our backend structure.

I hope this clears things up.

[HunkyBill]

This one made me giggle. Please Shopify, could you alter your systems to match mine so I don't have to work anymore? Would be swell!! Thanks and tata... have a nice day.

[Perice]

+1 for this

One call per product to get the metafields is way too much when you have to deal with thousands of products.

[HunkyBill]

Soon enough we'll get GraphQL calls and then we can all get the associated metafield resources needed, and this whole silly thread can go away forever.

[ClementG]

Do you have non public knowledge about this?

As far as I understand GraphQL is currently limited to "customer facing" entities.

It's not clear to me exactly where the boundary between customer facing and back office entities is, but I'd think that metafields are considered "back office"..

[HunkyBill]

No dude, I have no secret info. But that is almost certainly where things are going. Eventually we will have GraphQL access to objects and resources. Until then, we deal with the RESTful API and do 100,000 API calls in ~14 hours. YMMV. 

Therefore, whether we like it or not, we understand that using Metafields is non-performant when your call count is high. Use a different strategy. That is the secret. Knowing how to squeeze performance. 

[HunkyBill]

I get X calls per week from merchants that complain to me about Apps that bungle this performance issue, and there is zilch I can do for them, except to tell them they cannot use an App that is architected to fail on big product counts. 

Using a lowest fruit hanging from the tree approach (sometimes Metafields), is clearly easy, but has hazards. It is obvious from the get-go, so expecting magic and a sudden elimination of this performance is not likely. We could wake up tomorrow with metafields available, but then again, if it was easy, we would've had that 8 years ago. Clearly, it is best to wait for the internal wiring to catch up, and when it does, using Metafields will likely be suitable for a wide range of uses currently not practical.

[Joel-Reeds]

+1

Pulling in 1,000 products with their metafields currently takes around 10 minutes to complete because of the API call limit.

As others have suggested, the ability the pull metafields based on owner_resource and namespace would significantly reduce the number of calls required.

Something simple such as 4 metafields returned per product could reduce the total number of API calls from 1,004 to 20.

[HunkyBill]

It's easy to play a numbers game, but you know what will happen. Some joker will figure out that with the owner resource and namespace, they can jam 100, 500, heck even 1000+ things on another resource. So what is the point? You can ask for your metafields, and if you have 2 or 3 or 7 or some modest number sure, it's a simple join, la-dee-da, off we go. But you know people will abuse this. And so you are just offloading poor performance from one area to another. And granted, it may bridge certain gaps, it will almost certainly bog down others. 

Don't forget, peoplez can writez terrible codez if you let them, and flush Shopify internal performance down the drain, and so the wiring for this has to be done carefully... so ya.. it's a nice thought experiment.. but not realistic for reasons probably due to DB structure, and SQL efficiency, and the like.

[Jason]

Joel - why does 10 minutes matter? It's not like you're sitting there processing them by hand. Can you explain the pain point you're seeing? If this thread is going to keep being revived I'd like to at least understand the context / reasons / pain etc.

If you are grabbing that content all the time why not just cache that on your end and save the call entirely?