What's the Default robots.txt.liquid Code?

Technical Q&A
1

Hello, I’d like to edit robots.txt, however I really just want to add some disallow rules to the bottom of my existing robots.txt. When I add the rules, it replaces the whole robots.txt, which I don’t want to do. Does anyone have the default robots.txt.liquid code for me to put in my own robots.txt.liquid file along with my own added rules?

3

tech_dev27 has the officially documented approach, but one current caveat is important: I would not describe robots.default_groups as reproducing Shopify’s live default file exactly.

There have been several recent reports where the rendered output from that loop differs from the automatically generated /robots.txt on the same store. Some newer rules are missing, while some older rules remain.

If you use the documented loop, add your rule inside the main crawler group, before the sitemap:

{%- if group.user_agent.value == '*' -%}
{{ 'Disallow: /pages/example' }}
{%- endif -%}

Before publishing:

  1. Save a copy of the current live /robots.txt.

  2. Publish the robots.txt.liquid template.

  3. Compare the newly rendered /robots.txt with the saved version.

If exact parity is essential, the fallback is to copy the current live robots.txt into the template and add your custom rule manually. The tradeoff is that Shopify’s future default changes will no longer be inherited automatically.

So the documented loop is still the normal starting point, but I would verify the rendered output rather than assuming it is identical to Shopify’s current generated file.

4

Thank you both! This is extremely helpful information and is just what I needed