Hoping someone can please help. I am just taking a look at the robots.txt file for our page, as per below.
Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkout Disallow: /13672987/checkouts Disallow: /carts Disallow: /account Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b*
All makes sense, except for the blogs/pages.
Does this means all blog posts will not be crawled, and therefore, won't assist with SEO?
Are we able to edit the robots.txt file to remove these?
It isn't making sense to me??
Thanks in advance,
These lines means only URL which has + in them would not be crawled. Shopify uses + in the URL when collection list or list of articles in the blog is filtered with more then one tag.
But this does not prevent from crawling the full list, so there is nothing to worry about, especially because you are not allowed to change your robots.txt
(for example, you have tags gold and rings on some products in your jewellery collection, then the URL can be /collections/jewellery/gold+rings to show only products which bear both tags; this lcollection listing will not be crawled, but entire /collections/jewellery will be crawled)
As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here.
Due to the age of the topic, I will be locking this thread. If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.