All things Shopify and commerce
Hi,
I use Ahrefs to work on my website's SEO, and in my last crawl many issues relating to error 403 have appeared, where Ahrefs was blocked and couldn't crawl 100+ pages, most of them if not all of them are active products that I have stock on. I spoke with Ahrefs support and they told me to whitelist their IP addresses, but Shopify support said they don't do that for security reasons but that a possible issue in my robots.txt file is that there are missing spaces between UserAgents' code, and that could possibly a factor. I wanted to try and add those spaces in myself but most of the content on my robots.txt page doesn't appear in my shopify robots.txt.liquid, so I can't edit that.
Is there a way I can edit that that I'm not aware of? Has anyone had the same issue and have any ideas of how I can solve this issue without hiring an expert?
I want to understand the issue and how difficult it is to fix, and avoid any unnecessary costs where possible.
Any info is appreciated,
Thanks
Dropshipping, a high-growth, $226 billion-dollar industry, remains a highly dynamic bus...
By JasonH Nov 27, 2024Hey Community! It’s time to share some appreciation and celebrate what we have accomplis...
By JasonH Nov 14, 2024In today’s interview, we sat down with @BSS-Commerce to discuss practical strategies f...
By JasonH Nov 13, 2024