Robots.default_groups returns outdated rules, mismatch with the default robots.txt

Technical Q&A
,
1

Hello,

I’d like to report a mismatch between the robots.default_groups Liquid variable and the robots.txt file Shopify generates by default.

Following the official documentation ( Customize robots.txt ), I created a robots.txt.liquid template with the recommended code:

liquid

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}
  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

The file generated by this code doesn’t match the robots.txt file served by default on the same store (before creating the template). The robots.default_groups variable seems to return an older version of the rule set, while the current default file is noticeably more recent.

Here are the main differences I noticed on my store:

Present in the default file, but missing from the Liquid output:

  • The directives related to AI agents and agentic commerce (UCP/MCP, agents.md) in the header
  • All the explicit Allow rules (Allow: /, Allow: /account/login, etc.)
  • Some recent Disallow rules such as /sf_*, /services, /cart.js, /*/cart.js

Present in the Liquid output, but missing from the current default file:

  • Bot groups that no longer appear in the default file: Nutch, AhrefsBot, AhrefsSiteAudit, MJ12bot, Pinterest (with Crawl-delay directives)
  • Blocking of /policies/ and /search, which the current default file actually allows to be crawled

A merchant who follows the documentation to customize their robots.txt.liquid (for example, to add a single rule) unknowingly switches to a different rule set that’s more restrictive on certain pages, while thinking they’re simply starting from the default file. All of this without any warning.

robots.default_groups should be updated to reflect the current rules of the default robots.txt file.

Has anyone else noticed this behavior? Thanks in advance for your feedback.

2

Can’t speak to the exact Robots.default_groups mismatch, but worth knowing the practical implication either way: what matters for AI visibility isn’t the Liquid source, it’s whatever robots.txt actually renders live at /robots.txt - that’s what GPTBot, PerplexityBot, ClaudeBot etc. read, regardless of what the theme template intends. If you’re auditing this, fetch the live output and check it directly rather than trusting the Liquid source matches what ships.

3

Yeah, this is exactly the kind of thing I’d be careful with before shipping a custom robots template.

If robots.default_groups is lagging Shopify’s live default, I’d avoid cloning the whole default from Liquid just to add one rule. Keep the custom file as small as possible, fetch the final /robots.txt output after publishing, and compare it against the pre-custom version so you don’t accidentally block /search, policies, or newer AI/agent commerce directives.

For AI/search visibility the rendered robots file is the source of truth, not the helper variable. This is the broader checklist I use for that layer: Shopify AI Visibility Guide (2026)

4

I can’t confirm what’s happening internally, so I wouldn’t assume robots.default_groups is intentionally updated on a different cadence than the live robots.txt.

One useful data point would be to reproduce this on a brand new dev store with no theme or app customizations. If the mismatch is still there, that would strongly suggest it’s platform behavior rather than something specific to an individual store or theme.

Since you’ve already identified concrete differences between the rendered /robots.txt and robots.default_groups, I’d also report it through Shopify Support with those examples. If the documented helper doesn’t reflect the current default output, that’s something only Shopify can correct.

For now, I’d treat the rendered /robots.txt as the source of truth for crawler behavior, but I’d still consider the mismatch worth reporting because developers expect robots.default_groups to mirror the current default template.