Have your say in Community Polls: What was/is your greatest motivation to start your own business?

How to edit robots.txt that doesn't appear in .liquid - fixing 403s

How to edit robots.txt that doesn't appear in .liquid - fixing 403s

MarketWine
Shopify Partner
1 0 0

Hi,

 

I use Ahrefs to work on my website's SEO, and in my last crawl many issues relating to error 403 have appeared, where Ahrefs was blocked and couldn't crawl 100+ pages, most of them if not all of them are active products that I have stock on. I spoke with Ahrefs support and they told me to whitelist their IP addresses, but Shopify support  said they don't do that for security reasons but that a possible issue in my robots.txt file is that there are missing spaces between UserAgents' code, and that could possibly a factor. I wanted to try and add those spaces in myself but most of the content on my robots.txt page doesn't appear in my shopify robots.txt.liquid, so I can't edit that.

 

Is there a way I can edit that that I'm not aware of? Has anyone had the same issue and have any ideas of how I can solve this issue without hiring an expert?

robots.txt. - missing spaecs.png

I want to understand the issue and how difficult it is to fix, and avoid any unnecessary costs where possible.

 

Any info is appreciated,

Thanks

 

Replies 0 (0)