"Disallow" in code ?

LeFlo
Tourist
47 0 1

Hi,


I have to prevent the crawl of many pages, and as you know shopify refuses to let us integrate these functions on robots.txt.


So I put this in my code, in theme.liquid :
Disallow:/*?pr_prod_strat=collection
Disallow:/collections/new
Disallow:/collections/all/products
Disallow:/?page=

 

I wonder if I put it correctly, because now on the top bar on my shop these scripts appear, and in general if it appears it's because there is a code error.

 

NB: I've already looked at the https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines page, but this one gives noindex codes, so to prevent the index, and I'm talking about preventing the crawl, I need Disallow.

 

Thank you

0 Likes
FrankMoore
Explorer
44 10 11

Hi,

The robots.txt can't be modified. It's not actually possible to remove blocked pages from the automatically-generated robots.txt file created by your Shopify admin.

Shopify Developer from @Cadifa
0 Likes
LeFlo
Tourist
47 0 1

Hi,

I didn't say anything about removing blocked pages, that's not the point

0 Likes
Trevor
Community Moderator
Community Moderator
2899 379 584

Hello!

Just a quick update about the robot.txt file that was referenced in this post. As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here

If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.


Trevor | Community Moderator @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

0 Likes