I’m validating a narrow monitoring problem before building anything.
In a small validation exercise, the same search checks were rerun on 16 Shopify storefronts about three weeks apart. The measured result counts changed on 15 of them, and four of the original anomalies had reversed. One store returned nearly its entire catalogue for unrelated searches for several days, then returned to a more normal state without any alert.
This does not prove 15 harmful regressions: catalogue updates and intentional merchandising changes can also move results. The narrower question is this: if your store relies on critical queries, SKUs or part numbers, how do you know when the product that appeared yesterday disappears today?
Would a read-only alert showing the query, the expected product, before/after evidence and the first detected change solve a recurring problem for you, or is your current search or monitoring app already enough?
I’m looking for merchant and agency experience before deciding whether this should be built. There is no finished app to promote.
Before you build the alerting, I’d tighten what counts as evidence, because two documented defaults move result counts on their own.
Storefront search does a partial word match on the last term by default. options[prefix] defaults to last, so a query ending in a short token matches everything that starts with it. That makes counts jumpy in a way that has nothing to do with the catalogue, and it is much worse for part numbers than for words, because part numbers share prefixes.
Unavailable products default to last, not hide. So something going out of stock does not drop out of the count, it sinks to the bottom. That default is changeable in Search Settings in the Search & Discovery app, and so is the default type. A merchant flipping either one changes result counts store-wide in a single click, with no catalogue edit and no deploy. That is probably a chunk of your 15.
Which is why the near-full-catalogue store is the interesting one. That shape is what you get when the query stops constraining at all, and a theme or app that drops or rewrites the q param produces exactly it. Worth checking whether that store had a search app installed or a theme published inside that window, because those are dated events you can line up against your first detection.
Practical bit, capture the full search URL and the raw response rather than the term. /search?q=X&type=product&options[prefix]=none&options[unavailable_products]=hide and the same query on defaults will disagree with each other on a healthy store, and only one kind of disagreement is a regression.
On the product itself, the alert I’d act on is not that a count moved. It is that a named SKU that used to appear stopped appearing at all. Count deltas will cry wolf every time someone runs a bulk edit. Did any of your 15 involve a specific product falling out entirely, or was it all counts?
That’s a fair distinction, and it exposes a limitation in the 15/16 figure.
That comparison was mostly based on rerunning the same result-count checks. So no, I can’t honestly say that the 15 changes represent 15 cases where a named SKU disappeared between snapshots.
We did have a separate case where a published product appeared to be unfindable by its own name, but the count-based rerun wasn’t enough to prove that product-level claim. It needed a product-by-product check.
So I think your framing is the better one: capture the exact request and response so configuration changes can be explained, but don’t alert just because a count moved. The meaningful assertion is more like:
“for this exact critical query, this expected product must still be present.”
The near-full-catalogue case is also why I’m interested in keeping before/after evidence, although I don’t have admin-side theme/app event history for that store, so I can’t attribute the cause.
Since you work with Shopify tools for merchants: have you actually seen cases on stores you support where a known SKU/product used to be findable and then stopped appearing? If so, how was it discovered — merchant/customer report, manual QA, or monitoring?