Just a quick question about robots.txt.
Typical shop looks like this:
# robots.txt file for www.shopify.com e-commerce engine
Disallow: / <-------------- why is there a disallow on the root?
Any insight most appreciated. Thanks
Solved! Go to the solution
Google send me a Mail with an 100% Error Rate by reading the robots.txt. Google is not crawling the Site:
"Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%. "
We've just an SEO report done on our website and one of the recommendations is to remove the /tag links e.g:
Both of these pages link to the same page but we haven't made a /tag page, I'm guessing its something Shopify has already made?
To stop this, the SEO company said we need to block pages with /tag with 'Disallow: /*/tag' in the robot.txt file, we need to access this as Google looks at these pages as duplicate contact.
If anyone can help we would really appreciate it.
You've got no control over the robots.txt but you can add noindex meta tags to the page or collection header if required.
If you must do this (and I am not convinced you do) you'd need to wrap the metatag in a liquid if statement making sure it ONLY appears when you want to disable indexing. Depending on your theme design it may need to go inside the theme.liquid itself. Don't add anything until you know the if statement is (you don't want to make an error!).