Pop-up opt-in rate vs revenue per session: what we measure now

Shopify Discussion
, , , ,
1

Disclosure first. I work at Claspo, we build on-site pop-ups and quizzes, a lot of them on Shopify. Everything below is aggregated across client tests, and I am not publishing per-client numbers the client has not published themselves.

For a long stretch we ranked opt-in widgets by opt-in rate. Wheel beat form, form beat banner, ship it, move on. Then we started reading revenue per session on the same tests, and some of the rankings flipped.

Quick note on the maths, because it is the whole reason the flip is possible. The split runs at session level, revenue per session is arm revenue over arm sessions, and opt-in rate comes out of that same denominator. That is what lets a widget collecting fewer addresses win the comparison.

Gamification wins the opt-in comparison almost every run we have logged, spin-to-win and scratch cards both. People like the animation, they spin, the address goes in, and the list grows on a schedule you can promise a client.

Product-finder quizzes collected fewer addresses per session and in several niches came out ahead on revenue per session anyway, which took a few tests to believe. Two stores in the same niche, comparable traffic, the same two widget types, opposite winners. So the quiz result is a thing that happened in our data, and it predicted nothing about the store one niche over.

Push opt-in rate up while AOV slides and you have run a winning test that pays you less money. Same story with repeat purchase rate, except that one surfaces a quarter later, when nobody is checking the widget report anymore.

The mechanism is still a hypothesis for me. A wheel plausibly pulls in discount hunters who redeem once and go quiet, while a couple of questions about what someone is actually shopping for filter for people mid-purchase and hand you zero-party data for the flows afterwards. I see the pattern in aggregate. Why it happens, I cannot prove.

One more thing we got wrong at Claspo for a long time, and it is cheap to fix. We measured a widget against the store’s own baseline instead of against what that class of widget normally does. On one UK garden store a wheel hit 3.8% against a 1.64% baseline, which reads as a clear win until you check that wheels average just under 9% opt-in in ecommerce. Same number, opposite conclusion. Swapping the mechanic class took that store to 21.3%, and polishing the wheel would never have got there.

So, what do you rank email capture against on your store, opt-in rate, first-order revenue, or LTV? And if you have had a gamified pop-up running for over a year, did those signups hold up?

2

@Taras_claspo
This is a good point. I think first order revenue and LTV are much better metrics than opt in rate alone.

A pop up can generate a lot of emails but if those customers only use a discount once and never come back the high opt in rate doesnt mean much.

probably track opt in rate first order revenue, repeat purchase/LTV and unsubscribe rate together. That gives a much better picture of whether the widget is actually bringing valuable customers rather than just growing the list.

3

Opt-in rate and revenue per session disagree a lot. A widget that wins emails on first visit can still tax returning sessions.

Practical split: keep the interrupt popup for unknowns only, delay it on mobile, cut frequency. Judge it on RPS and identified-customer rate, not just opt-in %.

Returning people and dismissers still need a non-modal path for restocks, drops, and a second signup so list growth does not depend on another overlay. Sync that into the ESP you already run.

I work on Atomato (Atomato Email, SMS & Upsell - Grow email and SMS lists and upsell from a feed... | Shopify App Store). It is a shopper-opened bell/feed for that second path, with email/SMS as one card in the feed, not an ESP. Keep Claspo/Klaviyo/the popup you already have. Only relevant if you want the widget. The targeting + frequency change on the existing popup is the bigger lever this week.

4

This is why I increasingly think a lot of ecommerce optimization gets trapped at the metric level.

A popup can “win” opt-ins while hurting revenue, just like a PDP change can improve one aggregate metric while being wrong for a large subset of shoppers.

The more interesting question to me is whether the experience should change based on what the shopper appears to need at that moment — first-time vs returning is one dimension, but there are others like fit uncertainty, price/value concern, delivery timing, etc.

Have you seen much experimentation where the trigger/content itself changes based on behavioral context rather than just visitor segment?

5

I’d treat the popup as a treatment in the funnel, not as an email-capture contest. Keep a small holdout with no popup (or the normal non-modal path), and compare exposed vs holdout visitors on first-order revenue per session, purchase rate, repeat purchase and unsubscribe—not just opt-in.

For the adaptive question, log the trigger, message, device, new/returning status and the last meaningful action before exposure. Then segment the downstream result by source and product intent. RPS gets noisy at low volume, so use a fixed attribution window and report confidence alongside the lift. If a mechanic wins opt-ins but loses checkout or repeat purchase, it is not the conversion winner.