Bulk product create duplicates the product if product is already present in shop

Shopify Discussion
1

Hi, I am facing an issue while trying to upload multiple products through bulk api of shopify.
https://shopify.dev/api/usage/bulk-operations/imports

if first time I send 100 products, and then again if I send those same 100 products… shopify bulk api to create product duplicates it.

Solution I came up with is to store the shopify id(from response) after sending the products in bulk(batch of 250 products). So if product has shopify id then i will update otherwise create. As we know to update a product we need shopify id. But still there is an issue as follows:
Suppose, First time when we send products in shop using the bulk api and then get the response and save shopify id of each product in db. By that time if some other request was generated which was queued will run and again upload same products as they won’t have shopify id, So there will be duplication of products again.

Please suggest a solution to avoid the duplication.

2

Sure, this is a common issue in many applications, just create a field that every time you select items to upload it marks that that item is in queue to be uploaded, and that when you pull items to upload you only take items that are not in queue. Once your bulk edit job finishes (or after every item) you remove the queue flag for those items.