robots.txt

Solved
HunkyBill
Shopify Expert
4509 46 492

Hi,

Just a quick question about robots.txt.

Typical shop looks like this:

# robots.txt file for www.shopify.com e-commerce engine

User-agent: *

Disallow: /admin

Disallow: /cart

Disallow: /orders

Sitemap: http://www.dizzbuzz.com/sitemap.xml

User-agent: Nutch

Disallow: /                             <--------------    why is there a disallow on the root? 

Any insight most appreciated. Thanks

 

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes
spicydesign
Shopify Expert
313 0 7

Hey Dave,

Apache Nutch robot mostly used by scrappers, I guess that's why they block the entire site for "User-agent: Nutch". 

Just my 2 cents.

 

Cheers

Alex

www.spicydesign.ca  http://experts.shopify.com/spicydesign
0 Likes
HunkyBill
Shopify Expert
4509 46 492

Okay thanks.. I get it that by handing off the sitemap.xml there is no need to inform the robot of anything else.. 

 

 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes
lmtwashington
Shopify Partner
1 0 0

Shouldn't the robots.txt file also be keeping the robots from reading */account too? When the user hits the cart?

0 Likes
Tilo_Mildner
New Member
2 0 1

Google send me a Mail with an 100% Error Rate by reading the robots.txt. Google is not crawling the Site:

 

Google said:

"Over the last 24 hours, Googlebot encountered 1 errors while attempting to access your robots.txt. To ensure that we didn't crawl any pages listed in that file, we postponed our crawl. Your site's overall robots.txt error rate is 100.0%. "

 

WHY?!?

0 Likes
Shopify_Craig
Shopify Staff (Retired)
Shopify Staff (Retired)
332 0 26

That is very strange.  All Shopify stores have identical robots.txt files and Google is not have any issues with any other sites.  Perhaps it was an intermitant connection issue.  I wouldn't be worried about it - it should solve itself tomorrow.

Craig

0 Likes
iCollection
New Member
6 0 0

Hi Guys,

We've just an SEO report done on our website and one of the recommendations is to remove the /tag links e.g:

http://www.icollectionshop.com/collections/lifeproof
http://www.icollectionshop.com/collections/lifeproof/tag

Both of these pages link to the same page but we haven't made a /tag page, I'm guessing its something Shopify has already made?

To stop this, the SEO company said we need to block pages with /tag with 'Disallow: /*/tag' in the robot.txt file, we need to access this as Google looks at these pages as duplicate contact.

If anyone can help we would really appreciate it.

Thanks

0 Likes
Jason
Shopify Expert
10360 158 2004

You've got no control over the robots.txt but you can add noindex meta tags to the page or collection header if required. 
https://support.google.com/webmasters/answer/93710?hl=en

I jump on these forums to help and share some insights. Not looking to be hired, and not looking for work.

Don't hand out staff invites or give admin password to forum members unless absolutely needed. In most cases the help you need can be handled without that.


★ http://freakdesign.com.au ★
0 Likes
iCollection
New Member
6 0 0

Hi Jason,

Thanks for getting back to me so promptly.

Does this meta tag '
<meta name="robots" content="noindex">'

need to be put on every page or just the main .index page?

Thanks

0 Likes
Jason
Shopify Expert
10360 158 2004

If you must do this (and I am not convinced you do) you'd need to wrap the metatag in a liquid if statement making sure it ONLY appears when you want to disable indexing. Depending on your theme design it may need to go inside the theme.liquid itself. Don't add anything until you know the if statement is (you don't want to make an error!).

I jump on these forums to help and share some insights. Not looking to be hired, and not looking for work.

Don't hand out staff invites or give admin password to forum members unless absolutely needed. In most cases the help you need can be handled without that.


★ http://freakdesign.com.au ★
0 Likes