All things Shopify and commerce
Hi,
I use Ahrefs to work on my website's SEO, and in my last crawl many issues relating to error 403 have appeared, where Ahrefs was blocked and couldn't crawl 100+ pages, most of them if not all of them are active products that I have stock on. I spoke with Ahrefs support and they told me to whitelist their IP addresses, but Shopify support said they don't do that for security reasons but that a possible issue in my robots.txt file is that there are missing spaces between UserAgents' code, and that could possibly a factor. I wanted to try and add those spaces in myself but most of the content on my robots.txt page doesn't appear in my shopify robots.txt.liquid, so I can't edit that.
Is there a way I can edit that that I'm not aware of? Has anyone had the same issue and have any ideas of how I can solve this issue without hiring an expert?
I want to understand the issue and how difficult it is to fix, and avoid any unnecessary costs where possible.
Any info is appreciated,
Thanks
Discover how to increase customer engagement on your store with articles from Shopify A...
By Jacqui Apr 23, 2025Hey Community 👋 Did you know that March 15th is National Everything You Think Is W...
By JasonH Apr 1, 2025Discover how to increase the efficiency of commerce operations with Shopify Academy's l...
By Jacqui Mar 26, 2025