A space to discuss GraphQL queries, mutations, troubleshooting, throttling, and best practices.
Hi all,
We are hoping to keep our app in sync with the merchants product catalog.
We'll use a webhook notify us when a product is deleted from the catalog, but also want an alternative process to reconcile products in the catalog if there should be a comm error or the webhook is missed.
Understand that when a merchant deletes a product it is hard deleted.
Can anyone recommend a bullet proof way to validate and reconcile the product catalog at any point in time?
Thanks in advance.
Kind regards,
Anna
Hey Anna,
The only implementation that comes to mind for such a reconciliation would be to poll the products endpoint to retrieve all active ids. You can then compare that with what you have stored on your servers to determine which products have been deleted.
The easiest way to retrieve all products would be to use the `since_id` parameter for pagination. Your first call would be of the form:
GET /admin/products.json?since_id=1&limit=250
You would then take the largest product id returned in that request and pass it in as the `since_id` parameter of the subsequent call until you have retrieved all products. Therefore, any products which you have stored locally which weren't returned during the API calls must be deleted.
To learn more visit the Shopify Help Center or the Community Blog.
Hey Anna,
As long as the API call is successful (ie: 200 is returned), then there should never be a case where an active product is not included in the list when you're using the `since_id` parameter.
There are instances when products could be skipped when `page` is used for pagination, but for that reason, among other performance issues, we don't recommend using it.
To learn more visit the Shopify Help Center or the Community Blog.
Is this there still no better way?
Hi @wjames,
The best way to check for deleted products will still be a combination of subscribing to the products/delete webhook and occasionally polling our API for a list of products on the store.
If you were not wanting to deal with paginating results when making a query for a list of products, you can always use a Bulk Operation Query to run an asynchronous GraphQL query to get a list of ALL products on the store at once, without needing to worry about pagination or api rate limits.
Here's a Shopify.dev article with more information on how to run Bulk Operation Queries.
Developer Support @ Shopify
- Was this reply helpful? Click Like to let us know!
- Was your question answered? Mark it as an Accepted Solution
- To learn more visit Shopify.dev or the Shopify Web Design and Development Blog
I'll give it a go. Are there any plans to expose deletions via API?
By which I mean:
- Keep the current developer experience
- Use soft deletions instead of hard deletions
- Add a `deleted_at` timestamp to records, null if not deleted
- An optional request parameter to include deleted records in API requests
- API requests continue to use `updated_at_min` filters to reconcile, but now include deleted records.
Soft deletions are good practice so I'm hopeful there's appetite for this.
Here is my code to pull all product IDs using bulk query. The bulk query itself takes about 5 seconds to complete, in which time a new product may have been created and will be subsequently deleted. I'm sure you will also agree that this is a lot of complex code for not very much gain. This race condition is guaranteed to eventually trip someone up.
export async function getAllProductIds(): Promise<bigint[]> {
let { id } = await startBulkQlQuery(`
query {
products {
edges {
node {
id
}
}
}
}
`)
let records = await pollForBulkQlQuery(id, (line: string) => {
const match = line.match(/^{"id":"gid:\\\/\\\/shopify\\\/Product\\\/(\d+)"/)
if (match) {
return BigInt(match[1])
}
return null
})
records = records.filter((id) => !!id)
return records
}
export async function startBulkQlQuery(query: string): Promise<{ id: string }> {
const { data, errors, extensions } = await graphql.request(
`
mutation bulkOperationRunQuery($query: String!) {
bulkOperationRunQuery(query: $query) {
bulkOperation {
id
}
userErrors {
field
message
}
}
}
`,
{
variables: {
query,
},
},
)
if (errors) {
throw new Error(JSON.stringify(errors.graphQLErrors))
}
if (data.bulkOperationRunQuery?.userErrors?.length) {
throw new Error(JSON.stringify(data.bulkOperationRunQuery?.userErrors))
}
return {
id: data.bulkOperationRunQuery?.bulkOperation?.id,
}
}
export async function pollForBulkQlQuery(
id: string,
lineTransform: (line: string) => any,
): Promise<any[]> {
let records: any[]
let pollCount = 0
while (records == null) {
{
pollCount++
logger.info(
{ name: "pollForBulkQlQuery" },
`Polling for bulkQueryOperation result ${id}. Attempt #${pollCount}.`,
)
const { data, errors } = await graphql.request(
`
query {
currentBulkOperation(type: QUERY) {
id
url
type
status
fileSize
type
objectCount
rootObjectCount
query
partialDataUrl
errorCode
}
}
`,
)
if (errors) {
throw new Error(JSON.stringify(errors.graphQLErrors))
}
const {
currentBulkOperation: { id: currentBulkOperationId, status, url },
} = data
if (currentBulkOperationId == id && status != "RUNNING") {
if (status == "COMPLETED") {
const res: Response = await fetch(url)
const lines = (await res.text()).split("\n")
records = lines.map(lineTransform)
} else {
logger.error({ name: `pollForBulkQlQuery`, data }, "Process failure.")
break
}
}
await new Promise((resolve) => {
setTimeout(resolve, 500)
})
}
}
return records
}