Can robot txt be added in Shopify?

Razzl_Dazzl_Hai
New Member
2 0 0

Hello,

We went live a few days ago and our developer forgot to robot txt the old pages and our website has being crawl by google already. What can we go to prevent the old and unwanted pages to be visible?

Like this page - https://razzldazzlhair.com/collections/all

0 Likes
JarvisColtrane
Shopify Expert
267 1 16

Hi,

You can redirect them: Visit yoursite.myshopify.com/admin/redirects and adjust according to your needs.

Jarvis @ Kaleido
- Slow Shopify Store? Install our PageSpeed Magic app.
- Have more questions? Feel free to email me.
- My reply helped? Click Like to show me some love!

0 Likes
rezaul
Tourist
26 0 2

Razzl Dazzl Hair,

Robots.txt is automatically added to your site by Shopfy (eg. https://razzldazzlhair.com/robots.txt)

However, you can't directly edit this file.

If you don't want certain pages to be indexed, you can either unpublish them, or add meta robot tags to their template so its not indexed.

eg. <META NAME="ROBOTS" CONTENT="NOINDEX">

 

Toronto, Canada | e-commerce enthusiast | Just helping the community
0 Likes
Tofit
New Member
2 0 0

I spent like 30 hours this week doing SEO, just to wake up at the end of the week having nightmares about Robot.txt 

The truth is we can't control it and after I read it from googles site I realized that shopify blocks my blog and my category pages from being looked at. I put SEO in my blog and category pages. 

 

0 Likes
Sabine_Mohr
Excursionist
67 0 4

Why is collection and blog blogged for indexing by Shopify??

How can I change this?

 

 

Beemohr
?We love to knit ?
* www.beemohr.com *  www.beemohr.de * 
0 Likes
Jason
Shopify Expert
10360 158 2010

What's a url on your site that you think is being blocked by the robots file?

I jump on these forums to help and share some insights. Not looking to be hired, and not looking for work.

Don't hand out staff invites or give admin password to forum members unless absolutely needed. In most cases the help you need can be handled without that.


★ http://freakdesign.com.au ★
Sabine_Mohr
Excursionist
67 0 4

Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*

extracted from shopify via robot.txt

Beemohr
?We love to knit ?
* www.beemohr.com *  www.beemohr.de * 
tim
Shopify Expert
2926 143 1021

Sabine, it blocks only those pages, which has + in the url, like /collection/baloons/red+big. This only happens if you have filtered your collection or blog with more then one tag (red and big in my example).

I guess you most probably would not want to index those anyway.

Want to hire me to tweak a theme? Mail me at tairli@yahoo.com!
My post solved your problem? Like it!
Sabine_Mohr
Excursionist
67 0 4

o I understand thank you for this explanation Sabine.

 

 

Beemohr
?We love to knit ?
* www.beemohr.com *  www.beemohr.de * 
0 Likes
Tushar_Trivedi
New Member
3 0 0

Tim,

I understand blocking pages with + but what about the other 

Disallow: /collections/*%2B*
Disallow: /collections/*%2b* ???

0 Likes