Re: Robots.txt is blocking all blog posts from being crawled

Hi, i also have a question regarding the robots.txt and blog posts, please. the current settings make it completely pointless to invest in blog posts as they can not be indexed, so they can not rank and drive traffic to your website. However now Shopify allows you to amend the file so I can in theory i can remove the block. However, Shopify must have blocked them for a reason? what issues is removing the block going to cause?

Thanks

I think you are reading the robots.txt file wrong. As default blogs are not excluded from crawling.

You can test your robots.txt file here: https://technicalseo.com/tools/robots-txt/

Copy/paste your entire robots.txt file content type in a blog post (or front page of the blog) and click test. In the lower right corner you should see the response"Allowed".

ALSO, keep in mind that robots.txt is only limiting crawling. Not indexing. Crawling and indexing are two different things. In fact, Google have mange million pages indexed that they have not crawled. Off course, in that case they know very little about them and as such they most often do not rank for anything. But, technically it is just important to understand the difference :slightly_smiling_face:

So, yes, your blog should off course be crawled. But check it. I am sure it is not blocked.

Let me know if you need any further help. I will be happy to check your robots.txt file and blogs for you.

1 Like

Hi, thanks for getting back to me, im getting the info from search console, please see the image, also below is the blog

https://fightco.co.uk/blogs/fitness/great-stretches-for-martial-arts

I checked that page. It is marked as noindex in the HTML code.


Yes, that’s a problem. But it should be easy to solve.

I suggest you first to search and analyze all occurrences of “robots” in you theme files and if you don’t find anything could be that the code is managed by some other Apps (SEO) installed in the store.