Wondering if someone can help,
190 pages are indexed but blocked by robots.txt. HOWEVER, when using a robot’s txt validator, pages are allowed.
The robots.txt is correct but there has to be something blocking this robots.txt? I am very confused.
Topic summary
A user reports 190 pages showing as “indexed but blocked by robots.txt” in Google Search Console, yet robots.txt validators indicate these pages should be allowed. The robots.txt file appears correctly configured, creating confusion about what might be blocking crawlers.
Current Status:
- Another user requested example URLs to help diagnose the issue
- No resolution has been reached yet
Side Discussion:
- A separate user asked for clarification about robots.txt files in general, mentioning spam marketers claiming their file is broken
- They’re seeking basic guidance on whether fixes are costly or time-consuming, specifically for Shopify platforms
- This question remains unanswered
Can you give some example URLs?
What is this Robot txt thing all about? I have been contacted endlessly from spam marketers telling me mine is also broke, what is this and how do I fix it? Is this going to cost hundreds of dollars or hours and hours to fix? Any help out there for Shopify new to this whole thing, I’m spinning my wheels in the mud right now…
@MatchesSeedsM
Robots.txt is simply a file that tells search engines what pages (URLs) it can and cannot access. By default, Shopify creates this file, and disallows the most sensitive pages, like checkout and customer account pages. Things that shouldn’t be on a search engine. It also disallows filtered results. That way when you search for a company it’s not showing page 258 of some filter as a search result. Here’s an example
There is nothing wrong here. Everything is correct and as it should be. People telling you it’s incorrect or broke are just scammers. That being said, there could be instances where something happened and a collection or page is being disallowed, or God forbid something like account is allowed. It would appear here. That is a rare thing though.
If you have a shopify store, the url is yourstore.com/robots.txt
Hey Hollie, this issue usually happens when Google indexes pages that were previously crawlable before your robots.txt was updated, or if those URLs were discovered through external links. Even if your current robots.txt allows them now, Google may still show the “indexed, though blocked by robots.txt” message until it re-crawls and refreshes the status. You can try submitting the affected URLs again in Google Search Console to request reindexing or temporarily remove them if needed. Make sure your robots.txt is accessible and hasn’t been cached by Google.
So I don’t need an app for my store at $30USD or $200 software to maintain this? Thats what this spammer was trying to sell. Shouldn’t this be built in to Shopify so Google can crawl my site and make it visible? I get that this space is massive now and everyone wants their piece of the cake, but I’d think this would be set us for a shop already?
How do I fix my shop? Is it safe to share my robot.txt list here and someone point out how it works and anything I need to fix?
It’s already built in. Shopify already did the work. No you don’t need to do anything. Yes it’s a safe file to share. Anyone can go to a site and type in .txt at the end to view it. How do you fix your shop? IDK, but it’s not with the robots.txt…
appreciated! Thank you!