I need to redirect a URL that looks like this:
because robots.txt is blocking Google from searching this, which is a great shame because users see this in Google search:
Aspect Dr - Victorian Cosmetic Institute Online Skincare
A description for this result is not available because of this site's robots.txt – learn more.
and that search result is actually very high for keywords we are interested in.
(this URL is in Google because the landing page is full of URLs made with the URL_for_vendor filter )
robots.txt is blocking it because the space in the vendor name appears to be two tags to the very simple rule in robots.txt, and that's blocked. This robots.txt problem is very annoying because it seems unavoidable if URL_for_vendor is used when combined with vendors which dare to have spaces in their name.
I tried redirecting it to the appropriate custom collection but the redirect is ignored.
but this is not working.
Meanwhile I am trying to get google to crawl to the custom collection page via (a) submittings its URL and (b) making a link in navigation to the custom URL, but nothing is displacing the search result above. So a bit of a dilemma; if I force out the bad URL from google via webmaster tools I don't know what I'll be left with.
As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here.
Due to the age of the topic, I will be locking this thread. If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.