Our Partner & Developer boards on the community are moving to a brand new home: the .dev community forums! While you can still access past discussions here, for all your future app and storefront building questions, head over to the new forums.

Product delete webook / reconciliation

Product delete webook / reconciliation

HelpfulPerson
Shopify Partner
16 1 8

Hi all,

We are hoping to keep our app in sync with the merchants product catalog.

 

We'll use a webhook notify us when a product is deleted from the catalog, but also want an alternative process to reconcile products in the catalog if there should be a comm error or the webhook is missed.

 

Understand that when a merchant deletes a product it is hard deleted.

 

Can anyone recommend a bullet proof way to validate and reconcile the product catalog at any point in time?

 

Thanks in advance.

 

Kind regards,

 

Anna

Replies 7 (7)

Zameer
Shopify Staff
297 31 90

Hey Anna,

 

The only implementation that comes to mind for such a reconciliation would be to poll the products endpoint to retrieve all active ids. You can then compare that with what you have stored on your servers to determine which products have been deleted.

 

The easiest way to retrieve all products would be to use the `since_id` parameter for pagination. Your first call would be of the form:

GET /admin/products.json?since_id=1&limit=250

You would then take the largest product id returned in that request and pass it in as the `since_id` parameter of the subsequent call until you have retrieved all products. Therefore, any products which you have stored locally which weren't returned during the API calls must be deleted.

To learn more visit the Shopify Help Center or the Community Blog.

HelpfulPerson
Shopify Partner
16 1 8
Hi,

Thanks for the reply and the suggested solution.

Isn’t there a risk that if not all the products are returned for some reason, like a comm error or other, then we might accidentally delete products which are actually still active?


Zameer
Shopify Staff
297 31 90

Hey Anna,

 

As long as the API call is successful (ie: 200 is returned), then there should never be a case where an active product is not included in the list when you're using the `since_id` parameter.

 

There are instances when products could be skipped when `page` is used for pagination, but for that reason, among other performance issues, we don't recommend using it.

To learn more visit the Shopify Help Center or the Community Blog.

wjames
Shopify Partner
3 0 0

Is this there still no better way?

ShopifyDevSup
Shopify Staff
1453 238 531

Hi @wjames,

 

The best way to check for deleted products will still be a combination of subscribing to the products/delete webhook and occasionally polling our API for a list of products on the store.

If you were not wanting to deal with paginating results when making a query for a list of products, you can always use a Bulk Operation Query to run an asynchronous GraphQL query to get a list of ALL products on the store at once, without needing to worry about pagination or api rate limits.

Here's a Shopify.dev article with more information on how to run Bulk Operation Queries.

Developer Support @ Shopify
- Was this reply helpful? Click Like to let us know!
- Was your question answered? Mark it as an Accepted Solution
- To learn more visit Shopify.dev or the Shopify Web Design and Development Blog

wjames
Shopify Partner
3 0 0

I'll give it a go. Are there any plans to expose deletions via API?

 

By which I mean:

- Keep the current developer experience

- Use soft deletions instead of hard deletions

- Add a `deleted_at` timestamp to records, null if not deleted

- An optional request parameter to include deleted records in API requests

- API requests continue to use `updated_at_min` filters to reconcile, but now include deleted records.

 

Soft deletions are good practice so I'm hopeful there's appetite for this.

wjames
Shopify Partner
3 0 0

Here is my code to pull all product IDs using bulk query. The bulk query itself takes about 5 seconds to complete, in which time a new product may have been created and will be subsequently deleted. I'm sure you will also agree that this is a lot of complex code for not very much gain. This race condition is guaranteed to eventually trip someone up.

 

export async function getAllProductIds(): Promise<bigint[]> {
  let { id } = await startBulkQlQuery(`
    query {
      products {
        edges {
          node {
            id
          }
        }
      }
    }
  `)
  let records = await pollForBulkQlQuery(id, (line: string) => {
    const match = line.match(/^{"id":"gid:\\\/\\\/shopify\\\/Product\\\/(\d+)"/)
    if (match) {
      return BigInt(match[1])
    }
    return null
  })
  records = records.filter((id) => !!id)
  return records
}

export async function startBulkQlQuery(query: string): Promise<{ id: string }> {
  const { data, errors, extensions } = await graphql.request(
    `
      mutation bulkOperationRunQuery($query: String!) {
        bulkOperationRunQuery(query: $query) {
          bulkOperation {
            id
          }
          userErrors {
            field
            message
          }
        }
      }
    `,
    {
      variables: {
        query,
      },
    },
  )

  if (errors) {
    throw new Error(JSON.stringify(errors.graphQLErrors))
  }
  if (data.bulkOperationRunQuery?.userErrors?.length) {
    throw new Error(JSON.stringify(data.bulkOperationRunQuery?.userErrors))
  }

  return {
    id: data.bulkOperationRunQuery?.bulkOperation?.id,
  }
}

export async function pollForBulkQlQuery(
  id: string,
  lineTransform: (line: string) => any,
): Promise<any[]> {
  let records: any[]
  let pollCount = 0
  while (records == null) {
    {
      pollCount++
      logger.info(
        { name: "pollForBulkQlQuery" },
        `Polling for bulkQueryOperation result ${id}. Attempt #${pollCount}.`,
      )
      const { data, errors } = await graphql.request(
        `
          query {
            currentBulkOperation(type: QUERY) {
              id
              url
              type
              status
              fileSize
              type
              objectCount
              rootObjectCount
              query
              partialDataUrl
              errorCode
            }
          }
        `,
      )
      if (errors) {
        throw new Error(JSON.stringify(errors.graphQLErrors))
      }

      const {
        currentBulkOperation: { id: currentBulkOperationId, status, url },
      } = data
      if (currentBulkOperationId == id && status != "RUNNING") {
        if (status == "COMPLETED") {
          const res: Response = await fetch(url)
          const lines = (await res.text()).split("\n")
          records = lines.map(lineTransform)
        } else {
          logger.error({ name: `pollForBulkQlQuery`, data }, "Process failure.")
          break
        }
      }

      await new Promise((resolve) => {
        setTimeout(resolve, 500)
      })
    }
  }
  return records
}