We're moving the community! Starting July 7, the current community will be read-only for approx. 2 weeks. You can browse content, but posting will be temporarily unavailable. Learn more

How to block crawling of 1 specific page in robots.txt

Solved

How to block crawling of 1 specific page in robots.txt

svalkyrie23
Explorer
87 2 12

Hi All,

 

I want to block a specific page URL from being crawled.  I have tried using the theme.liquid and added this code but it didnt work:

 

{% if handle contains 'page-handle-you-want-to-exclude' %}
<meta name="robots" content="noindex">
{% endif %}



I tried adding this to the robots.txt file

disallow: /a/XXXX 

under user-agent, adsbot-google etc. 
But this didnt work either.

 

Finally I tried this code at the bottom of robots txt but it also didn't work.  Can someone help?

 

{%- if group.user_agent.value == ‘*’ -%}

{{ ‘Disallow: [URLPath]‘ }}

{%- endif -%}

 

I'm testing crawlability via this link and it keeps coming up saying it's crawled: https://search.google.com/test/rich-results/

svalkyrie23_0-1723790770302.png

 

Accepted Solution (1)
Dan-From-Ryviu
Shopify Partner
12073 2359 2539

This is an accepted solution.

Please add domain.com/page/page.json to get the ID of the page

After that, please add this code to theme.liquid file 

{% if page.id == page ID goes here %}
<meta name="robots" content="noindex, nofollow" />
{% endif %}

 

 

- Helpful? Like & Accept solution!
- Ryviu - Product Reviews & QA app: Collect customer reviews, import reviews from AliExpress, Amazon, Etsy, Walmart, Dhgate and CSV.
- Lookfy Gallery: Lookbook Image - Gain customers with photo gallery, video & shoppable image.
- Reelfy‑Shoppable Videos+Reels: Create shoppable videos to engage customers and drive more sales.
- Enjoy 1 month of Shopify for $1. Sign up now.

View solution in original post

Replies 7 (7)

Dan-From-Ryviu
Shopify Partner
12073 2359 2539

Could you share that page link? 

- Helpful? Like & Accept solution!
- Ryviu - Product Reviews & QA app: Collect customer reviews, import reviews from AliExpress, Amazon, Etsy, Walmart, Dhgate and CSV.
- Lookfy Gallery: Lookbook Image - Gain customers with photo gallery, video & shoppable image.
- Reelfy‑Shoppable Videos+Reels: Create shoppable videos to engage customers and drive more sales.
- Enjoy 1 month of Shopify for $1. Sign up now.

svalkyrie23
Explorer
87 2 12

I would prefer not to because of scammers/copiers etc.

It's just a URL on the site. like domain.com/page/page 

svalkyrie23
Explorer
87 2 12

@Dan-From-Ryviu can you please advise? you shouldn't need the URL.  Lets just give it a name https:// domain.com/page

Dan-From-Ryviu
Shopify Partner
12073 2359 2539

This is an accepted solution.

Please add domain.com/page/page.json to get the ID of the page

After that, please add this code to theme.liquid file 

{% if page.id == page ID goes here %}
<meta name="robots" content="noindex, nofollow" />
{% endif %}

 

 

- Helpful? Like & Accept solution!
- Ryviu - Product Reviews & QA app: Collect customer reviews, import reviews from AliExpress, Amazon, Etsy, Walmart, Dhgate and CSV.
- Lookfy Gallery: Lookbook Image - Gain customers with photo gallery, video & shoppable image.
- Reelfy‑Shoppable Videos+Reels: Create shoppable videos to engage customers and drive more sales.
- Enjoy 1 month of Shopify for $1. Sign up now.

svalkyrie23
Explorer
87 2 12

@Dan-From-Ryviu appreciate the response, it doesn't seem to work, I just get "error occured on client". The page directory is domain.com/a/faq which renders the page.  So if I add faq/page.json I get the error.  if I do faq.json it is a 404?

atishayawasthi
Shopify Partner
3 0 1

You can easily get the page id from admin.

Open Online store then pages

Open the page in admin & check the URL You can see something like 

https://admin.shopify.com/store/[your-store-name]/pages/1986285850919 

that number is your page.id

svalkyrie23
Explorer
87 2 12

Can anyone help please?