Is there any way to block a specific User-agent from accessing our store? As I understand it, Shopify will not let me touch the robots.txt file which would normally be used for this (ie. throwing in something like a "User-agent: CreepyCrawler" and next line "Disallow: /").
How can I go about fixing this?
Just a quick update about the robot.txt file that was referenced in this post. As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here.
If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.