We are running into the same exact problem. For several of our customers we would like to retrieve and process the meta datafields every day. However, with the current restrictions this is just not an option because this could result in 100k+ requests every day.
We hope there will be found a solution to this problem in the future.
As a datafeed management company we currently download all the product data from Shopify, process this (e.g remove products which are out of stock, add the color of the product to the title etc) and send the modified data to "export channels" such as Amazon and Google Shopping. We don't push any data to Shopify, only retrieve it every day.
Some of our Shopify customers store data in the meta data fields, for example color, sku's etc. and our customers want to use this information as to better optimize their ads on the "export channels".
We streamline this process as much as possible as to make it as easy as possible for the customer. When a customer changes an item's description (or stock changes after a sell), this will automatically be changed on the export channels. We want to do something similar for the meta data fields.
Some customers have a lot of products in their stores and thus getting all the meta_fields for all the products every day would be a huge amount of api calls.
We are aware that the amount of api calls could be reduced by giving a customer an option to fetch the meta_data fields in Channable instead of doing this every day. However, we would like to keep our strucuture the same for all ecommerce shops and this is not required for other shops. Furthermore, this would require a big change in our backend structure.
I hope this clears things up.
No dude, I have no secret info. But that is almost certainly where things are going. Eventually we will have GraphQL access to objects and resources. Until then, we deal with the RESTful API and do 100,000 API calls in ~14 hours. YMMV.
Therefore, whether we like it or not, we understand that using Metafields is non-performant when your call count is high. Use a different strategy. That is the secret. Knowing how to squeeze performance.
I get X calls per week from merchants that complain to me about Apps that bungle this performance issue, and there is zilch I can do for them, except to tell them they cannot use an App that is architected to fail on big product counts.
Using a lowest fruit hanging from the tree approach (sometimes Metafields), is clearly easy, but has hazards. It is obvious from the get-go, so expecting magic and a sudden elimination of this performance is not likely. We could wake up tomorrow with metafields available, but then again, if it was easy, we would've had that 8 years ago. Clearly, it is best to wait for the internal wiring to catch up, and when it does, using Metafields will likely be suitable for a wide range of uses currently not practical.
Pulling in 1,000 products with their metafields currently takes around 10 minutes to complete because of the API call limit.
As others have suggested, the ability the pull metafields based on owner_resource and namespace would significantly reduce the number of calls required.
Something simple such as 4 metafields returned per product could reduce the total number of API calls from 1,004 to 20.