For some reason the vast majority of my pages (products, collections etc) have got screwed up on Google search. I noticed that they are being blocked by Shopify's Robots.txt for my site. This only became an issue a few months ago but since then visitor number and sales have gone through the floor. How do I go about fixing this? I've trawled through all the data available on Google and can see nothing wrong but something inside the robots.txt. My sitemaps look okay, but all the pages contained within are again being blocked by the robots.txt file.
I've had a good look around the forum here and it seems pretty clear that Shopify have some sort of issue with allowing their customers to access their own robots.txt file, but if this is still the case, what does Shopify intend to do to solve my issue as it is having a clear impact on my business and if not sorted soon will mean having to jump ship to another ecommerce provider.
What makes you think the pages are being blocked? What's the offending line in the robots file?
If you can give a bit more context the forum members may be able to give better insight. For now, let us start with those questions and go from there.
I did the usual Google search for my own business name and got
So I visited my Google search console to try and work out what the issue was to discover that since the start of August 2094 valid pages that were listed with Google have transformed into error pages. When I looked at why the pages were being marked as errors it clearly states that the submitted urls are being blocked by robots.txt.
The urls include my product pages, collection pages, information pages etc.
I don't think it is unreasonable to make the jump to assuming that my pages are being blocked as that is what it says. As for the offending line in the robots file... I haven't got a clue. Not sure exactly what further context is required, I thought the original post was fairly self explanatory and a look through the forums suggest that I am far from the only one to have this issue, though it seems in their cases too they have either got no assistance at all or got jumped on from the get go.
Just a quick update about the robot.txt file that was referenced in this post. As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here.
If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.