Would you know if an AI shopper bought from your store?

AI Build Your Business
1

Last month I posted about whether AI assistants can see a Shopify loyalty program. That left the next question: if an assistant does send a shopper, does the order look different when it lands? (Disclosure: I build Stabile, a Shopify loyalty app.)

Mostly, no. Two of yesterday’s orders across the stores we monitor came from ChatGPT — both through Online Store, app id 580111, Referring: none. The only trace of origin was utm_source=chatgpt.com on the landing URL.

Three things we got wrong first:

1. The AI source has to be on the first landing page. We added utm_source to a checkout URL, saw it in the browser, and the order webhook still reported Landing: /, Referring: none. The same parameters on the product URL, in a clean session, worked.

2. URL shape is not evidence. A /cart/c/ permalink with a numeric source looks agentic, but abandoned-cart apps produce it too. Yesterday every order that looked agentic by shape was Shopify’s own Shop app or an Etsy integration. Since February, one of those ids has surfaced in 26 different stores’ order feeds, looking agentic every time, until the Admin API resolved it.

3. Match only where origin is declared — the referring hostname, the utm_source value, and Shopify’s source_name. Substring-matching the whole URL means a campaign named spring_chatgpt_test counts as ChatGPT.

Worth keeping separate: which assistant sent the demand, and whether an agent executed the checkout. Everything AI-attributed we see is the first.

Rather not maintain this by hand? Stabile does it — labels each order with platform and mechanism, keeps the landing URL, referrer and app id so you can check its work, and earns loyalty points on those orders. Free to install:

Happy to paste the exact hostnames and utm_source values we match on. Has anyone found AI-referred orders in their own reports — and what did you match on?

2

You usually would not know from the order alone. Online Store orders can look identical whether the shopper came from ChatGPT, Google, or a copied link.

What I’d do today:

  • Standardize AI links with utm_source, utm_medium, and utm_campaign on the first product or collection URL.
  • Run one test in an incognito window, land on that URL first, then complete checkout. Check the order’s Conversion summary and landing page.
  • Save the raw landing URL, referrer, source_name, and app ID before applying any tags.
  • Tag only exact source values such as chatgpt.com. Do not use URL-wide substring matching.

That identifies AI-referred demand. It still does not prove an AI agent completed the checkout.

3

Good thread. The split you both land on — “which assistant sent the demand” vs “did an agent execute the checkout” — is the right one, and it’s why I stopped trusting URL shape a while ago too.

What we match on for the first half, in order of how much I trust it: the referring hostname (chat.openai.com, chatgpt.com, copilot.microsoft.com, gemini.google.com), then the exact utm_source value on the first landing URL, then the order’s source_name/app id from the Admin API — never a substring of the whole URL, for exactly the spring_chatgpt_test reason. And we keep the raw landing URL + referrer + user-agent on the order so a merchant can check the label instead of taking our word.

For the second half — whether an agent can actually get through your checkout — you can’t read that off orders at all, because the ones that fail never become orders. The only way I’ve found is to send an agent through the store on purpose and watch where it stops. That’s a different measurement from attribution and I’d keep them in separate columns.
Happy to compare hostname lists — ours is short and I’d rather it be right than long.

4

Hey @YangW_Stabile, that trace is already sitting on the order itself. Shopify keeps the converting session’s landing page and referrer on the order record, so open any order and read the conversion summary: the utm_source that rode in on the first landing URL is right there next to the landing page. That’s the mechanism behind your first finding, and it’s also why some orders read Referring: none, since the summary comes back limited or empty when session data wasn’t captured, like cookie blocking, draft orders, or anything that didn’t come through Online Store. I’d pull one order you know came from ChatGPT and compare its conversion summary against what your webhook recorded before trusting either one.

5

the distinction between “ai referred the shopper” and “an ai agent actually completed the checkout” is a really useful one. i’ve seen people treat anything with a weird cart url or app id as proof of agent activity, when there are so many other integrations that can produce similar looking data.

the first-landing-page point makes sense too. if the tracking parameter isn’t present when the session starts, trying to reconstruct the source later from the checkout url is going to get pretty unreliable. i’d be interested to see the exact hostname/utm matching you’re using, especially how you’re handling cases where the same shopper gets referred by an ai source and then comes back through another channel before purchasing.

6

The distinction between AI referral and an actual agent completing the checkout is a really important one. I can see how the URL shape would be tempting to use as a shortcut, especially when you’re trying to work with a lot of order data, but that sounds like a pretty easy way to end up with false positives.

I’d probably stick to declared attribution too. The first landing page detail is interesting as well because I could see people testing this from checkout and assuming the tracking was working when the order data tells a different story.

I’ve been looking at the same general problem from the SEO side. It’s not enough to know that AI is sending traffic, you want to know which pages are actually getting that visibility and whether those visits are doing anything useful afterward. That’s one reason I like SiteGuru alongside the normal Shopify data, since it makes it easier to connect the search side back to specific product and collection pages.

The app ID example is probably the bit I’d be most careful with. If the same ID keeps appearing across stores, I’d definitely want to resolve what actually generated it before treating it as evidence of AI agent activity.

7

One extra control I’d add to the test: a person opens a copied product link that still carries utm_source=``chatgpt.com, without using ChatGPT. The tag survives, but it no longer proves where that particular shopper discovered the product.

For a merchant-facing report, I’d label that “AI-tagged landing” and keep the underlying evidence separate from a verified integration source. I’d also leave unattributed orders as unknown, rather than counting them as definitely non-AI. That gives merchants a useful baseline without turning tracking parameters into a stronger claim than they support.

A small test matrix could cover a genuine assistant referral, a copied tagged link, an untagged direct visit, and a later return through another channel. Publish the expected label beside the observed order fields. It would make the matching rules much easier for merchants to audit, alongside your existing separation of referral from checkout execution.