We went live a few days ago and our developer forgot to robot txt the old pages and our website has being crawl by google already. What can we go to prevent the old and unwanted pages to be visible?
Like this page - https://razzldazzlhair.com/collections/all
Razzl Dazzl Hair,
Robots.txt is automatically added to your site by Shopfy (eg. https://razzldazzlhair.com/robots.txt)
However, you can't directly edit this file.
If you don't want certain pages to be indexed, you can either unpublish them, or add meta robot tags to their template so its not indexed.
eg. <META NAME="ROBOTS" CONTENT="NOINDEX">
I spent like 30 hours this week doing SEO, just to wake up at the end of the week having nightmares about Robot.txt
The truth is we can't control it and after I read it from googles site I realized that shopify blocks my blog and my category pages from being looked at. I put SEO in my blog and category pages.
What's a url on your site that you think is being blocked by the robots file?
extracted from shopify via robot.txt
Sabine, it blocks only those pages, which has + in the url, like /collection/baloons/red+big. This only happens if you have filtered your collection or blog with more then one tag (red and big in my example).
I guess you most probably would not want to index those anyway.