SEO, AdWords, affiliates, advertising, and promotions
Join us March 21 for an AMA on planning your 2023 marketing budget with 2H Media co-owners, Matt and Aron
I keep having this pop up as an SEO critical error on my blogs. How do I allow robots.txt crawling on selected pages?
Would you share the URL to your store as well as the exact error you are getting that would list the URLs affected? Maybe there is discrepancy between sitemap.xml and the content of robots.txt.
it may be because of the blogs are disallowed to crawl and index. So you can fix it by removing those URLs from the robots. if possible share the domain name so that we can have a look at what exact issue is there.
Just a quick update! As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here.
Due to the age of the topic, I will be locking this thread. If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.
Trevor | Community Moderator @ Shopify
- Was my reply helpful? Click Like to let me know!
- Was your question answered? Mark it as an Accepted Solution
- To learn more visit the Shopify Help Center or the Shopify Blog
Thanks to all Community members that participated in our inaugural 2 week AMA on the new E...By Jacqui Mar 10, 2023
Upskill and stand out with the new Shopify Foundations Certification programBy SarahF_Shopify Mar 6, 2023
One of the key components to running a successful online business is having clear and co...By Ollie Mar 6, 2023