Page blocked by robots.txt

Solved

Page blocked by robots.txt

Sarah441
New Member
6 0 0

I have 2 pages that Google Search Console says are indexed, but blocked by robots.txt. I went through the shopify help article and got to the point shown in the screenshot. I need help with next steps. Shopify support couldn't help me.robotscode.pngrobots2.png

Accepted Solution (1)
ilanadavis
Shopify Partner
209 27 71

This is an accepted solution.

Hey @Sarah441 It wasn't meant for you to code something, sorry for my lack of clarity.

 

My comment was that there is nothing you need to do. Your policy pages should not be indexed and Shopify is doing exactly what they should be doing. What's happened here is that you're linking to your policy page from a product page that is indexed. Google will attempt to crawl all links they see. In this case, they "see" the policy page because it's linked to from the product page so Google indexed the policy page. But Shopify is saying "Wait, we don't want this indexed, please don't spend resources crawling this page."

 

So Google will then send you a notice (your screenshot from Search Console) that says we see this policy page because it's linked, but we're not doing anything else with it. If you don't want that, take action.

 

But Google is doing exactly what you want it to do, so there is no action required. You can ignore that particular issue.

 

Additionally, if you are not a developer, I wouldn't go futzing with creating a robots.txt file outside of Shopify's. You'd have to maintain that and any future changes.

 

In summary, leave it as it is and focus on more important areas of growing your business.

Ilana Davis, Shopify Superhero
JSON-LD for SEO, adds structured data to your theme, getting you more organic traffic with Rich Results.
Linking Llama, link complementary, comparable, or related products to improve SEO & UX.

View solution in original post

Replies 4 (4)

ilanadavis
Shopify Partner
209 27 71

Shopify blocks policy pages in the robots.txt file because they usually aren't helpful for people when searching for products.

 

If you go to your live robots.txt page, you'll see Shopify has the following which is expected.

Disallow: /policies/
Disallow: /*/policies/

 

Ilana Davis, Shopify Superhero
JSON-LD for SEO, adds structured data to your theme, getting you more organic traffic with Rich Results.
Linking Llama, link complementary, comparable, or related products to improve SEO & UX.
Sarah441
New Member
6 0 0

I'm not sure how this helps me. I do not understand coding. 

ilanadavis
Shopify Partner
209 27 71

This is an accepted solution.

Hey @Sarah441 It wasn't meant for you to code something, sorry for my lack of clarity.

 

My comment was that there is nothing you need to do. Your policy pages should not be indexed and Shopify is doing exactly what they should be doing. What's happened here is that you're linking to your policy page from a product page that is indexed. Google will attempt to crawl all links they see. In this case, they "see" the policy page because it's linked to from the product page so Google indexed the policy page. But Shopify is saying "Wait, we don't want this indexed, please don't spend resources crawling this page."

 

So Google will then send you a notice (your screenshot from Search Console) that says we see this policy page because it's linked, but we're not doing anything else with it. If you don't want that, take action.

 

But Google is doing exactly what you want it to do, so there is no action required. You can ignore that particular issue.

 

Additionally, if you are not a developer, I wouldn't go futzing with creating a robots.txt file outside of Shopify's. You'd have to maintain that and any future changes.

 

In summary, leave it as it is and focus on more important areas of growing your business.

Ilana Davis, Shopify Superhero
JSON-LD for SEO, adds structured data to your theme, getting you more organic traffic with Rich Results.
Linking Llama, link complementary, comparable, or related products to improve SEO & UX.

KieranR
Shopify Partner
388 34 118

There are dev docs that show this exact example.

KieranR_0-1718668834580.png

 

From here: https://shopify.dev/docs/storefronts/themes/seo/robots-txt#remove-a-default-rule-from-an-existing-gr... 

 

So basically replace the group.rules for rule in your template with the one from example.

 

But personally, this seems like such a minor issue to even bother fixing. But if you're committed to fixing it, being unsure about Liquid coding means you run the risk of breaking it more than fixing it. A single character wrong could block all your product or collection pages for example...

Part time Shopify SEO, based in NZ. Sometimes freelance outside the 9-5.