Best practices for paginating nested query results?

[jeremywh]

Hi all,

I'm fairly new to GraphQL in general, and I'm looking for some guidance on how to paginate nested query results.  I've tried Googling around, but I guess haven't hit on the right combination of search terms yet as I haven't been able to find answers.

I was wondering how you handle queries that have nested lists that need to be paginated to get everything. By "how do you handle..?" I mean how do you keep track of the cursors and feed them to the "after" filter and ensure that you get the full set of records for an initial query?

I'm going to give a very simple example query, but I want to be clear that I'm not looking for ways to fix or improve this specific query. I'm looking more for high level concepts and best practices. I'm also not looking for new tools or libraries for sending queries to the server.  I'm just trying to understand this at the basic GraphQL query request level. 

Here's a simple orders/lineItems query, and I'm requesting the first 5 orders, and for each order I'm requesting the first 5 lines:

{
  orders(first: 5, query:"created_at:>=2020-09-14") {
    pageInfo {
      hasNextPage
    }
    edges {
      cursor
      node {
        id
        name
        lineItems (first: 5) {
          pageInfo {
            hasNextPage
          }
          edges {
            cursor
            node {
              sku
            }
          }
        }
      }
    }
  }
}

There is a potential here that all lists will need to be paginated to get the full set of results on each, and in this example, that would be 6 different lists: one for the top-level list of orders, and then each lineItems list in each order.

What is the best way to handle this?

I know one option would be to split it into multiple queries. I could first run the top level "orders" query without the lineItems, and then I could feed the id or name from each order into an "order" query to get the lineItems for just that order, paginating the line items there as needed. But that seems like it would be very slow in inefficient because I would be, in this case, sending at least 6 different queries to achieve the same result set as the original query.

Actually... is that the only option? With each set of lineItems having its own final cursor, I can't even wrap my head around what the next query would look like. How would I even supply 5 cursors in the "after" filter to get the next page of each set of lineItems?

Hopefully someone out there has some enlightenment for me, because I've been noodling around on this for quite some time, and I can't come up with how I would get the subsequent pages of orders and line_items.

Thanks,
Jeremy

[_Dirk_]

Maybe the answer of Dan Crews in this thread https://spectrum.chat/graphql/general/how-do-you-design-cursors-for-nested-pagination helps you:

Assuming Relay: this query is to build the initial view of the page. You wouldn't make this query to get the nested-level pagination. You would make a node query to look up that nested node by id, and then you would pass the cursor in. You wouldn't ever make a call based on a nested cursor like this.

This thread talks about the same solution: https://www.reddit.com/r/graphql/comments/ivg7eq/best_practices_for_consuming_nested_pagination/

It seems GraphQL is not designed to query nested lists directly by cursor, only step by step.

[jeremywh]

Thanks Dirk.  Funnily enough, that reddit thread is mine.  I posted it there to try to get more responses since I hadn't gotten any bites here or over at the ShipHero forum. 🙂

The first-level query and node query situation that Dan talks about was what I was trying to avoid.  I was hoping I could do it with the one query.  Bummer.  Thanks for the link, though.  I hadn't seen that one in my research.

[guest4]

One way to do this is not to paginate the results at all, and then use bulkOperationRunQuery, instead. 

With the bulkOperationRunQuery mutation:

• The first argument is optional and ignored if present, so it can be removed.
• The cursor and pageInfo fields are also optional and ignored if present, so they can be removed.

URL: https://shopify.dev/api/usage/bulk-operations/queries

[LouiseEH]

Bulk query seems simpler - but it does not handle if you want to query for e.g. a single order and get all line items and all fulfillment orders, as there is no id query on bulk queries.
I wonder how pagination works when there are multiple child edges to paginate as e.g. an order where you want to get all line items and all fulfillment orders - anybody knows?

[guest4]

We're using bulk query in some places and then paginating queries in other places.

For the paginated order queries, we pull the first nodes (first: 1) for fulfillment order line items and order line items (and a few other fields that have paging) along with the pageInfo (hasNextPage, hasPreviousPage, startCursor, and endCursor). 

Then we make successive queries in a do...while loop until hasNextPage is false.