Have your say in Community Polls: What was/is your greatest motivation to start your own business?

Robots.txt is disallowing multiple pages. How to fix this?

Robots.txt is disallowing multiple pages. How to fix this?

TeodoraA_
Visitor
3 0 0

Hello,

 

After a site audit, I've noticed that Robots.txt is blocking some pages from showing in search engine.

When checking, it looks like it is  "disallowing" several pages of my Shopify store, like:

 

User-agent: *
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/ Disallow: /checkout Disallow: /55629021526/checkouts Disallow: /55629021526/orders Disallow: /carts Disallow: /account Disallow: /collections/*sort_by* Disallow: /*/collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /policies/ Disallow: /*/*?*ls=*&ls=* Disallow: /*/*?*ls%3D*%3Fls%3D* Disallow: /*/*?*ls%3d*%3fls%3d* Disallow: /search Disallow: /apple-app-site-association Disallow: /.well-known/shopify/monorail Disallow: /cdn/wpm/*.js Disallow: /services/login_with_shop Disallow: */collections/*filter*&*filter* Disallow: /password Sitemap: https://diffney.ie/sitemap.xml User-agent: adsbot-google Disallow: /checkouts/ Disallow: /checkout Disallow: /carts Disallow: /orders Disallow: /55629021526/checkouts Disallow: /55629021526/orders Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /cdn/wpm/*.js Disallow: /services/login_with_shop User-agent: Nutch Disallow: / User-agent: AhrefsBot Crawl-delay: 10 Disallow: /a/downloads/-/* Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /55629021526/checkouts Disallow: /55629021526/orders Disallow: /carts Disallow: /account Disallow: /collections/*sort_by* Disallow: /*/collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /policies/ Disallow: /*/*?*ls=*&ls=* Disallow: /*/*?*ls%3D*%3Fls%3D* Disallow: /*/*?*ls%3d*%3fls%3d* Disallow: /search Disallow: /apple-app-site-association Disallow: /.well-known/shopify/monorail Disallow: /cdn/wpm/*.js Disallow: /services/login_with_shop Disallow: */collections/*filter*&*filter* Sitemap: https://diffney.ie/sitemap.xml User-agent: AhrefsSiteAudit Crawl-delay: 10 Disallow: /a/downloads/-/* Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /55629021526/checkouts Disallow: /55629021526/orders Disallow: /carts Disallow: /account Disallow: /collections/*sort_by* Disallow: /*/collections/*sort_by* Disallow: /collections/*+* Disallow: /collections/*%2B* Disallow: /collections/*%2b* Disallow: /*/collections/*+* Disallow: /*/collections/*%2B* Disallow: /*/collections/*%2b* Disallow: /blogs/*+* Disallow: /blogs/*%2B* Disallow: /blogs/*%2b* Disallow: /*/blogs/*+* Disallow: /*/blogs/*%2B* Disallow: /*/blogs/*%2b* Disallow: /*?*oseid=* Disallow: /*preview_theme_id* Disallow: /*preview_script_id* Disallow: /policies/ Disallow: /*/*?*ls=*&ls=* Disallow: /*/*?*ls%3D*%3Fls%3D* Disallow: /*/*?*ls%3d*%3fls%3d* Disallow: /search Disallow: /apple-app-site-association Disallow: /.well-known/shopify/monorail Disallow: /cdn/wpm/*.js Disallow: /services/login_with_shop Disallow: */collections/*filter*&*filter*

 

Also, when I check the code in "Themes" section, I see the following:

 

support.jpg

 

 

 

 

 

 

 

 

 

 

 

 

Please advise, what exactly do I need to replace in the code, in order for these pages to not be disallowed?

 

Thank you,

Teodora

Replies 2 (2)

coolkey
Visitor
1 0 0

Same happens to me, see Shopify`s support reply which I do not think is right, they must allow us to decide what to crawl and what not in our own website. In may case I have 5 pages and Ubersuggest and Google Search Console  are showing that as an error to be fixed, pages are:

/cart 
/policies/contact-information
/policies/privacy-policy
/policies/refund-policy Blocked (Robots.txt)
/policies/terms-of-service

See their reply: 

"Following our recent chat conversation, we discussed the concerns you're facing related to SEO optimizations on your Shopify store. You raised an issue flagged by an SEO tool, Ubersuggest, stating an error with the 'robots.txt' setup. 
Firstly, I want to assure you that we understand the importance of SEO and the impact it can have on your store's visibility and performance. Please know that all your concerns are being treated with due diligence. 
Upon further investigation and in consultation with our technical team, we've reviewed the 'robots.txt' configuration in your store. One thing to bear in mind is that this file is designed to instruct web robots on the pages they should not crawl. The potential errors that Ubersuggest and Google Search Console flagged imply that they can't crawl these pages which, in reality, is the exact function of the 'robots.txt' rules. 
This essentially means that your 'robots.txt' file is working as expected, and those 'errors' can safely be ignored. They don't pose any threat to your website's SEO or visibility. 
If you made any modifications to your 'robots.txt' file. Since the change has already been implemented, the new rules are now in effect on your online store.

* While Shopify does not validate these files, you may use third-party tools for validation purposes. What tool to use? ; a Shopify friendly third party tool?
Thank you for your understanding and patience.

CJ8
Visitor
1 0 0

Did you ever get this fixed? I have the same problem and got zero response from support other than to say "we don't block pages".

I'm trying to fix it now but my site appears to be a mess of duplicate pages that I can't even find to delete! 

Not showing up in google will mean a nightmare in regards to SEO

thanks.