Robots.txt file - disallowing blog posts?

Jo_McGuire
New Member
1 0 1

HI,

Hoping someone can please help.  I am just taking a look at the robots.txt file for our page, as per below.

Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkout
Disallow: /13672987/checkouts
Disallow: /carts
Disallow: /account
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*

All makes sense, except for the blogs/pages.

Does this means all blog posts will not be crawled, and therefore, won't assist with SEO?

Are we able to edit the robots.txt file to remove these?

It isn't making sense to me??

Thanks in advance, 

Jo

tim
Shopify Expert
2926 143 1021

Hi Jo.

These lines means only URL which has + in them would not be crawled. Shopify uses + in the URL when collection list or list of articles in the blog is filtered with more then one tag.


But this does not prevent from crawling the full list, so there is nothing to worry about, especially because you are not allowed to change your robots.txt

(for example, you have tags gold  and rings on some products in your jewellery collection, then the URL can be /collections/jewellery/gold+rings  to show  only products which bear both  tags; this lcollection listing will not be crawled, but entire /collections/jewellery will be crawled)

Want to hire me to tweak a theme? Mail me at tairli@yahoo.com!
My post solved your problem? Like it!
Trevor
Community Moderator
Community Moderator
2543 327 518

Hello!

As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here

Due to the age of the topic, I will be locking this thread. If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.

Trevor | Community Moderator @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

0 Likes