First 10 products. First 100 variants. (Query failed - Query cost over 1,000)

Highlighted
Shopify Partner
12 0 8

Hey team,
While migrating to the GraphQL API, we're struggling with the ability to query for any significant amount of data similar to the REST API.  

I'll provide a simple example which demonstrates this issue I believe a lot of developers will have.

With the REST API, you can grab 250 products from a single collection, complete with all variants, images, etc.  Everything.

With the GraphQL API, you can't grab 10 products and their variant IDs, let alone all their associated information.  

This limitation can be seen in a lot of other contexts, but I'll keep this simple for now.
Here's a query that will not work because the Query Cost is over 1,000.

{
  products(first: 10) {
    edges {
      node {
        variants(first: 100) {
          edges {
            node {
              id
            }
          }
        }
      }
    }
  }
}

This example is inherently trivial, and I could provide greater context, but I think it's safe to say that this could be a worthwhile query, and is certainly possible on REST.

Here's another example:

{
  products(first: 50) {
    edges {
      node {
        variants(first: 20) {
          edges {
            node {
              id
            }
          }
        }
      }
    }
  }
}


Both queries will fail because they cost over 1,000.

While at Unite, there were many questions regarding the 1-1 migration between REST and GraphQL.  Even if these queries provided a cost at 1/10th of their current value, the efficiency of the REST API and the cost-refresh-rate would still seem to far outpace the GraphQL API.

I love the progress of the GraphQL API, and I'm hoping that there's something I'm overlooking, but at this point this seems like a roadblock to the development of a large chunk of data-rich applications.

Looking forward to a conversation around this and how it might be solved down the road.

Cheers

1 Like
Highlighted
Shopify Expert
4182 29 403

I think you are missing a basic philosophy here. While it is true that with one RestAPI call you can get 250 objects, there are some issues with that:

  1. it is getting slower and slower as object complexity rises. Your API calls will take a lot longer than they used to
  2. typically you never need all the attributes of an object to get work done, hence a lot of wasted bandwidth

So move over to GraphQL. Yes it is true the complexity calculation will stop you dead if you attempt something potentially expensive. Note that that word is potentially expensive. Shopify is tuning complexity calculations (we hope), to ensure the guesses get better.

 

You ask for what you actually need. First off, that is going to make scripting easier to test and develop. Ask and you receive. The main difference though is this:

 

You will want to ask for the first record in N records, and then page. So you write a slightly more complex query (paging with cursors) but you get what you exactly need. So now instead of paging through 250 objects with one call, you might have a tighter loop that makes 250 cheaper faster calls. So nothing has really changed except the paging with cursors. Examine the complexity guess and reality, and it turns out you can process your N products, with no loss of speed compared to the RestAPI.

 

Carry this a little further, and it turns out there is a lot of new complexity in variants, not addressed by RestAPI so much. With GraphQL you can actually work with inventory and locations now, provided you also page variants. Yes, you could complain this is crazy, but it really is not too bad. Say you code up a fragment to do detailed variant work. And now you page that. Suddenly your scripting is tight and fast. So I page products, page variants, and dig deep into the products, and it works! Fast! And I could do with RestAPI, but you know what, it sucks. It is slow. It is clunky.

 

So yes, you are missing something if you just slap a 10 here or a 100 there, without regards to the paging (cursors)... try them out, with some fragmets, and boom, it is a pretty sweet pattern. And yes, a lot of GraphQL scripting out in the wild relies on libraries that currently works in progress, and some kinda suck, but it is getting better. So hopefully this year, we'll see some improved tools, better complexity guesses, and more endpoints!

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes
Highlighted
Shopify Partner
12 0 8

Thanks Bill,
Fair enough regarding querying only what you need, when you need it.  I'm familiar with using GraphQL in the wild, and perhaps our use-cases fall under those that seem to be relatively impossible at the current moment.  Needing to get all potential variants for multiple products seems like a common concern. 

Let's say I'm developing a search application that performs quick evaluation of a customer's text query.  I might want to know the full details about a store's products, which includes variants.  Many apps collect this data in the background or even relatively on-the-fly.  It's possible on REST to get large chunks of this data.  I might take 500-700ms for a page of results, but it's possible.  

Or, let's say I'm trying to get the equivalent of the "first_avaialble_variant" of a product.  Seeing how, in GraphQL, I cannot use "query" on the "Variant" connection, and seeing as the definition of "first_available_variant" falls under a logical condition that isn't replicatable through graphql, I might then want to have all the variants.  Scaling this out, perhaps I want to collect the data for "first_available_variant" for many products ahead of time.

Or, let's say I want to get images of 10 products at once.  Maybe because I'm displaying a custom page of product results, complete with image sliders of the product images.  Maybe I want to store this data in my DB, so I can't just use liquid.  It's just not possible to gather that data in one call using GraphQL.  
I can get 9 products at a time, for an "actualQueryCost" of 91.  Then I can paginate for the next single product. 

The usefulness and practicality of GraphQL is obvious and awesome.  Getting "all subconnection X for 10 products" (subconnection being images, variants, etc) doesn't seem like a fringe use-case.  I would also recommend developers to find other more efficient means by which they can query for only the needed data, but as Shopify has a vested interest in making it easier for apps to develop data-rich applications, and as Shopify is pushing GraphQL adoption, I do feel as though there are an abundance of use-cases in which a notably necessary query simply costs too much.  

So if you want all the variants or images of a number of products, do 9 or less at a time.  This will also incur 1/10th of your available API limit.  Fair enough.



0 Likes
Highlighted
Shopify Expert
4182 29 403

I don't think anyone would be naive enough to develop a search application using API calls. Successful Apps in that space employ common vetted approaches like Elastic Search on a search space maintained outside of Shopify. Problem solved.

 

Having to ask for a product's first variant with available inventory seems doable without too much trouble. Contriving to want something but not really having a need? Like getting all images for 10 products. Again, can be done, without too much trouble, but in no way does RestAPI make this a snap. It just happens to be a mundane thing to want.

 

I never said getting all aspects of a resource was fringe. But if you were to analyze this with a mind towards efficiency, perhaps using RestAPI calls for something like that is appropriate, and certainly less typing. I can get 50,000 products with 200 API calls. < 10 lines of code. If my use case tells me that is the way to go, I go there!

 

I am not seeing the ALL IN OR NOTHING with moving to GraphQL... it certainly is handy for a lot. Maybe not for everything, but luckily we get to pick and choose our tools!

 

One thing you point out, and that I point out, and that we all would love, is some real-world advice on what kind of limits result in reasonable complexity. I can blow > 1000 complexity in a simple one product scenario of wanting inventory and locations. I have to page. So do I do it in 10 10's or 4 25's ? How do we work so that instead of 10 calls of complexity 30 we maybe do 3 of 75? That kind of deep-dive knowledge remains murky and hidden, but hopefully with enough back and forth here, there and elsewhere, actionable patterns emerge!

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes
Highlighted
Shopify Partner
12 0 8

I certainly didn't say that someone would develop a search application using only API calls.  An elastic search approach would be fantastic, of course, which would require the data to be stored, which would require the data to be retrieved, which would lend credence to my point that using the GraphQL API to fetch this data is a relative non-starter, or at the very least, highly inefficient from a query-cost perspective.  Let's say my hypothetical search application could also highly benefit from datapoints only available through GraphQL.  Well, then, that's a shame.  Hopefully this hypothetical application developer could find a workaround.  Shopify would like that, because the more successful they are, then everyone benefits.

I am not "Contriving to want something but not having a need".  I could state my exact use case, but considered that the necessity for a query that satisfies this need or a need similar to this would most certainly exist, and probably exist frequently, and the degree of query-cost inefficiency provided by the GraphQL API for these specific yet generalizable instances is at the least worth noting to the development team.  Maybe it's not something they plan on addressing.  Maybe it is.  It would be awesome if it is.  As I scale and run an application with over 10k installs, we need data like this and the additional datapoints provided by GraphQL makes it a very valuable tool.  It's unfortunate that we can't query for 10 products and get all their variants at once.  It's unfortunate that we can't query for 10 products and get all their photos at once.  

I agree that things are progressing, that there are murky and hidden details, and hence why I'm raising this concern.  It is my hope that this thing is addressed in one vein or another, because the datapoints returned by the GraphQL API are very valuable to us and to other apps.  While workarounds are possible (and we have found one), I'm hoping that we can find another, quicker way.  




0 Likes
Highlighted
Tourist
11 1 0

Unless I'm missing something, Shopify's own video to show how great this is for a custom product page isn't actually possible as shown.  Or maybe the cost was higher at the time of this video?  Oct 2017.  See https://www.youtube.com/watch?v=S4hTq_3yQrk at 23:32.  products(first: 20) with variants(first: 250)...cost is WAY too high.  Also, isn't the max possible at 100 variants?

 

I totally agree with @Stephen24.  The dynamic cost of the number of variants should be better taken into consideration here.  I know there's an argument here for paging and cursor usage, but the fact is that this is inefficient for something built for efficiency.

 

2020-06-09_17-08-04.png

0 Likes
Highlighted
Shopify Expert
4182 29 403

For almost any use case, you can just move over to the bulk query. You get all your data in one JSON file you can download and work with. This is for any use case where number of products > 1.

 

For any single product, making a GraphQL call and getting back 100 variants is easy, my current cost is ~ 200. Note that is not going deep dive on inventory.

 

What is a dynamic cost predicated on variants? You do not know apriori if you have just 1, or 100. So you have to take that into account. How would you adjust this?

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes
Highlighted
Shopify Partner
27 0 5

@HunkyBill wrote:

For almost any use case, you can just move over to the bulk query.

 


"For almost any use case, you can just move over to the bulk query." is a gross generalization.  If you need the data in real-time, for example, if you haven't parsed an entire shop into your database, which could literally have an infinite number of products, variants, images, etc -- and you want to display the product details, or you want to get the product details in order to render something like a list of it's variants, then you really can't use the bulk API.  This is one anecdotal example in the sea of many.  But to say "almost any use case" is a bit of a stretch.

 

The GraphQL API is fantastic.  The rate limiting is not.   When someone initiates a Google or SEMRush crawl, their pages are pinged quite fast.  If you have an app that works with anything like that, and renders some dynamic data on a page, using AJAX, you'll either have to pre-crawl their entire shop, or grab the data in real time.  Sure, go ahead and pre-crawl their entire shop (we've seen shops with 10s of thousands of products).  Then set up webhooks to update a product once it's updated (when you have thousands of users, you're likely to pay a pretty penny for that queue processing if it does anything reasonably difficult).  Go for it.  We've set that up, but it wasn't easy, and this is a barrier to entry for many apps, at least in my opinion. 

These are just examples of where GQL falls short, not due to it's fundamental limitations, but due to rate limiting.  So we're glad the Rest API is still supported.

0 Likes
Highlighted
Shopify Expert
4182 29 403

You actually made my point. If you are beyond a simple API call for a product, be a proper engineer, and design a fast system to support your use case. Do not be a dope and expect API calls to bail you out for a lack of ability. As you said, it is tough. Well, geez Louise. Duh.

 

All I said was anything non-trivial is done in the background, to support speed up front. Nothing in this thread is the fault of Shopify or GraphQL, it is just experience pointing out, it is harder than it looks.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes
Highlighted
Shopify Partner
27 0 5

Your point was that "for almost any task, you can use the Bulk API", which I did not prove, not in it's current state.  If your point was "For almost any task, you can build a sufficiently complex system to piggyback off of the API calls that Shopify makes available, and copy an entire store's contents into your own database so that you can access things faster", well then Geez, Loise, Duh. 

It also doesn't seem like it's in Shopify's financial interest to make things that difficult, and they're open to feedback, so constructive discussion here is helpful, instead of merely pointing out how complicated one can make a system.  Shopify takes their cut, and actively promotes apps which further the success of the ecosystem as a whole.  Evaluating the rate-limiting in such cases with radically accumulating costs once you grab nested items is actually a constructive discussion to have, and as @chris_dell pointed out, perhaps there's issues in documentation that Shopify hasn't entirely sorted out.    

0 Likes