How to edit robots.txt that doesn't appear in .liquid - fixing 403s

Shopify Discussion

Topic summary

Issue: A site owner using Ahrefs sees 403 (forbidden) errors on 100+ pages, mostly active product pages. Ahrefs advised whitelisting their IPs, but Shopify declined for security reasons.

Possible cause: Shopify support suggested robots.txt formatting problems (missing spaces between User-agent sections) might be contributing to bot blocking. An attached screenshot highlights the suspected missing spaces and is central to the concern.

Constraint: The user tried to fix robots.txt but most of what appears in the live robots.txt does not appear in robots.txt.liquid, implying much of the file is auto-generated and not directly editable through the theme.

Request: Guidance on how to edit or override the relevant parts of robots.txt without hiring an expert, and whether others have encountered and solved this. Status: Unresolved; user is seeking solutions and effort estimates to avoid unnecessary costs.

Summarized with AI on December 16. AI used: gpt-5.
1

Hi,

I use Ahrefs to work on my website’s SEO, and in my last crawl many issues relating to error 403 have appeared, where Ahrefs was blocked and couldn’t crawl 100+ pages, most of them if not all of them are active products that I have stock on. I spoke with Ahrefs support and they told me to whitelist their IP addresses, but Shopify support said they don’t do that for security reasons but that a possible issue in my robots.txt file is that there are missing spaces between UserAgents’ code, and that could possibly a factor. I wanted to try and add those spaces in myself but most of the content on my robots.txt page doesn’t appear in my shopify robots.txt.liquid, so I can’t edit that.

Is there a way I can edit that that I’m not aware of? Has anyone had the same issue and have any ideas of how I can solve this issue without hiring an expert?

robots.txt. - missing spaecs.png
robots.txt. - missing spaecs.png369×442 10.5 KB

I want to understand the issue and how difficult it is to fix, and avoid any unnecessary costs where possible.

Any info is appreciated,

Thanks