How can I manage JSONL parsing for large data with bulk operations?

Shopify Discussion
1

Hi,
I’m checking new api changes (10-2019) for bulk operations , i want to get all products in bulk include (image, options, and variants) but when i get JSONL file we didn’t get response in nested form like images, options and variants should be
available in product json rather than separate , i also read the bulk operations document. It will work when product catalog size is less than 5k , and for large catalog i will result approx 50k indexes includes products, variants and images that will be more difficult to parse the array and perform operations.

Please let me know if there is any feasible solution to manage large data with bulk operations. Below graphql query and i’m using:

mutation {
  bulkOperationRunQuery(
    query:"""
    {
      products(query:"published_status:all") {
        edges 
        {
          node {
            id
            title
            description
            vendor
            description
            createdAt
            handle
            productType
            publishedAt
            tags
            images(first:20){
              edges{
                node{
                  id
                  originalSrc
                }
              }
            }
            totalVariants
            variants(first:100){
              edges{
                node{
                  id
                  sku
                  title
                  price
                  inventoryQuantity
                  inventoryPolicy
                  fulfillmentService {
                    id
                  }
                  position
                  barcode
                  image {
                    id
                    originalSrc
                  }
                  taxable
                  weight
                  weightUnit
                  compareAtPrice
                  requiresShipping
                  selectedOptions{
                    name
                    value
                  }
                  inventoryItem{
                    
                    id
                    tracked
                  }
                }
              }
            }
            options {
              name
              position
              values
            }
          }
        }
    } 
}
    """
  ) {
    bulkOperation {
      id
      status
    }
    userErrors {
      field
      message
    }
  }
}

And i m using PHP code to parse JSONL file into JSON array format that results large data of array for large catalog.

$fp = @fopen($filename, 'r');
if ($fp) {
      $json_arrays = explode("\n", fread($fp, filesize($filename)));
}

Thanks,
■■■■■■■

2

hi @ksh_verma

did you found any solution ?