Webhook for product added / removed from collection

[Bnaya_Livne]

Hi,

We would like to get a webhook for every time a product is removed / added to a dynamic collection.

Is that possible? (We sohuld get that when a product is update or a collection is updated I guess).

Thanks,

Bnaya

[Jamie_D_]

Hey Bnaya,

There aren't currently any webhooks fired when a product is added or removed to a Smart Collection.

[phannigan]

Hi Jamie - any updates here?

We have a client with 30K+ items in several of their SmartCollections.  We need to keep an up-to-date list of items in the collection (ie Collects). When we get a webhook that the SmartCollection has been updated, we are having to page through 30K+ Collects to find items that have been removed from the SmartCollection. This obviously takes a lot of time and API calls, so we'd love to implement a better solution.

What is the recommended way for finding out what Collects have been removed from a large SmartCollection over time?

Thanks

Patrick

[Kris_Doyle]

Hi All,

Just wanted to add that I would benefit greatly from a hook like this as well. Please 🙂

Cheers

[Ruslan_Starikov]

Hi Jamie, is this still the case? Is there a roadmap for creating this functionality?

Thanks!

[khoinguyen]

Hi Shopify team, 

I also got trouble with this problem, are there any solution or roadmap for this one

Thanks.

[danielpretorius]

We would really like this as well!!!

[growlytics]

I don't know why this webhook is not being called. Some times webhook is triggered, sometime it doesn't. It's not reliable.

If product_update or collection_update webhook is not being called, at least collect webhook should be added in that case.

The way they have provided collect APIs for linking products to a collection.

I think webhook should be also available for changes on collects.

[wb1]

+1 for this feature request

[HunkyBill]

This has never been available, for many reasons. Imagine a smart collection. One change to the rules, and boom, 1000 products could be added or removed. So what do you do? Wait till that operation finishes inside Shopify, and then 1 minute later deliver a webhook saying the collection changed? And then another tweak to the same collection, and boom, 1000 products are added or removed, so you wait till that finishes, and then send off a webhook saying something changed? So it kind of makes some sense that firing off a webhook for this is iffy.

I get it for a manual collection. Someone drags a new product in, or removes an old one, wow, handy to know the collection just changed! But still, they don't service that. Supposedly in 2020 they were revamping the whole backend of how collections are formed, and the newer architecture, assuming there is one, would likely be more amenable to some kind of flag you could monitor to say hey, since the last time you checked, things changed here!

[wb1]

@HunkyBill I like your idea of flag notifying that sort order has changed. They don't currently have anything like this except Collect, which is explicitly 1:1 mapping and not efficient at all for large product collections

Whether it's just a new `updated_at` value, or webhook, we need somthing...

[phannigan]

The problem right now is that there is no way to know what collect specifically in a collection has changed without paging through every single collect in entire collection. Providing a method (webhook or otherwise) to know what collects have changed would prevent a ton of unnecessary work (API paging).

We work with many, many merchants with over 100,000+ items in a given collection... so when something changes in that collection we have to page through all 100K+ collects just to find out that one collect was added, or one was deleted.

In the worst case scenario of "sort order changed, so every collect has changed", we do care about the diff to all 100K collects, and we're happy to save that info on our side. In fact, we're happy to page through an API or consume 100,000+ webhooks in this case! It's the (much more common) case of "one collect has changed" when we still need to page through all 100K just to find the collect that's been diffed that is really painful. Sending us a "this collect has been diffed" message over a webhook feels like the fastest way (vs polling an API) to get that change reflected from Shopify -> our system... but I'd happily settle for a pollable API for this information too!

We'd happily consume 100K+ webhooks when we have to, vs spending minutes/hours of compute time (and a bunch of API headroom) paging through API responses when we don't. I'd wager that most apps dealing with merchants on this level already have some pretty decent webhook-handling infra to consume realtime order updates etc anyways?

For now, we've had to optimize our paging/data processing codepaths and throw infra at the problem to keep track of collects since we have no other choice... We're not well versed in or well tooled for GraphQL right now, but maybe there's a solution to this in a GraphQL-equivalent API endpoint?

Anyone else handling this usecase and want to swap notes? patrick@hive.co  👋🏻

[wb1]

@phannigan I've found GQL and API to be comparable in terms of throughput. Kind of just depends which case. E.g. metafields are much better via GQL if applicable because metafields API cannot be paged

i haven't had to deal with 100K products in a single collection, haven't seen anything that large, but up to 10k total products spread across many collections, the sort order issue is still a  pain, to say the least

i also update content of all entity types, so my overhead is large before i even get to collection sort order. better collection sort visibility, and article/pages webhooks are my two big slow points right now

[123221]

For Smart Collections-

Rather than pulling data for all the collections there is a smarter way to sync data.

This is useful when the store has several collections.

When you change the order of the products "updated_at" parameter of the Collection Object is updated. Use this to identify the collections that need to be synced.

Use- https://STORE/admin/api/2023-01/smart_collections.json?updated_at_min=2023-10-17T20:01:07-07:00

Store the updated_at_min in your database at the time your sync starts.

Also - get the number of products in the collection and use this information to remove extra Products from your Database.