Indexed Robot.txt block - Why block the policy pages ?

[Victor_bustos]

Hello, does anyone know why Shopify's robots.txt file blocks the search for the policies on the page? Is there a strategic or legal reason for this?

For instance, is it to prevent future policy changes from being compared? Or is there another reason?

It's worth noting that this behaviour exists in all free themes. I was testing it with Screaming Frog, and indeed, when checking the robots.txt with Google Tools, it indicated blocked policies.

[theycallmemakka · Accepted solution]

Hi @Victor_bustos ,

The decision to block the search for policies in Shopify's robots.txt file likely stems from strategic considerations related to SEO.

By preventing search engine crawlers from indexing or accessing specific pages, such as policy pages, Shopify may aim to avoid these pages appearing in search engine results.

Several reasons could underlie this strategy:

1. Duplicate Content Concerns: Policies on e-commerce websites are often standardized and may resemble each other across different stores. Blocking these pages can help mitigate the risk of duplicate content issues, which can have a detrimental effect on SEO.

2. Prioritizing Product Pages: Shopify may intend to guide search engines to prioritize product pages over policy pages. This emphasis on product visibility in search results can potentially enhance conversion rates.

3. Following SEO Best Practices: It's a common SEO practice to disallow the indexing of pages that are not pivotal for search engine results. Since policies might not be the primary content that merchants wish to highlight in search results, blocking them aligns with SEO best practices.

[Victor_bustos]

Hey thanks so much for the explain 

[StorySEO]

Hey thanks it is more clear now why you do it (Duplicate) becuase when i saw yesterday it weird. One more Q about this - Why you considered them as "Not pivotal to search engines" ? Becuse they very much are. When you block crawlers by deafult to reaching this page but users can reach it via the footer (because in most cases the client will be not aware that this page is blocked to crawlers) it can result in Cloaking. Have you considered this risk?

[DonnaW]

I understand this, but what about when a person needs to have the policies crawled by robots.  For instance, on my store I have the Pinterest app to connect my store to Pinterest.  I have tried for months to get my store to connect to Pinterest, but every time I try Pinterest comes up saying it can't find the Policies on my store and will not connect my store, because it thinks my store is not legitimate or is not complete with what it should have.  I just now in the last few days saw that the Policies are disallowed.  How do I make them allowed?  What about other sites that may need to connect to my store, those also might need to be able to see and know the Policies are there by crawling my store, to show the site is doing the correct things it should since stores should have Policies set up.  That makes no sense to have them disallowed from this perspective.  Something needs to be done about this and fixed, at least give a person the CHOICE of whether they want the Policies allowed or not.

[DonnaW]

FYI, I do have the Policies showing on the footer of my store.

[Monica24]

Did you ever get a resolve to this? I am having the same issue with google indexing my pages.