Why does shopify disallow collections and blogs when i check robots.txt?

Shopify Discussion
1

Shopify dissallows collections and blogs in robots.txt. Won’t this affect the organic searches. Because all the products in my website are part of some collection.And if I try to maintain a blog for maintaining an organic position by maintaining blogs of my products- it is disallowed in robots.txt. Will the defult robots.txt cause problem?

Because I am getting issues from google search console that many of my product pages are not indexed. Can somebody tell me if I am mistaking this for something else?

2

To review what are your store pages that have been indexed (or missing), type in site:yourstoreurl.com into your browser to a get a bird’s eye view of what has been indexed.

For new store pages, Google takes a long time - a few weeks to months to index all the pages, if you only submit your store’s sitemap.

To expedite the process, go to the menu> click on URL inspection and Request Indexing - one URL at a go. There is no option to do it in bulk and there is a limit of 10 URLs every day.

Highly recommend to submit your URLS based on the importance of the pages so your priority pages get indexed more quickly. When I say quickly, I don’t mean immediately.

This URL inspection tool is a function to inform Google there are specific pages that needs to be indexed, rather than Google figuring out all the pages that exist through the sitemap.

3

Hello @tfm_webmaster ,

Gina here from flareAI.

robots.txt is only limiting crawling. Not indexing. Crawling and indexing are two different things. I don’t think Shopify will block your collections and blogs in robots.txt by default. Check your collection and blog pages and see if the below meta tag is present or not.

If this tag is there, the page will be blocked from indexing. See whether any of your installed apps is managing your robots.txt file.

You can get the reasons and fix the needed for the Google Search Console errors from this documentation.

To get better sales and traffic, you have to make sure your products are found on Google. If your store products are not found on Google, customers won’t find the URLs through Google Search. You can try flareAI as a tool to get the basic necessity of getting found ( indexed ) verifiably submitted to Google for consideration. flareAI works daily for you, tapping into the world’s largest free sales channels. flareAI help you to grow your revenue on the world’s best free sales channels.

Hope this helps.

Gina
flareAI: Generate Sales from Google Search, on Autopilot

4

I have been looking for a solution for this, and somebody told that collection are collection of individual products which have their page indexed on google. So to avoid this duplication, shopify does this. Do you think that could be the solution. They tell me this is same aas categories in wordpress. Categories are not indexed in wordpress

5

How do you see if the collections and blogs are disallowed from being indexed by Google?

Are you referring to seeing this in your robots.txt?

Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*/blogs/*+*
Disallow: /*/blogs/*%2B*
Disallow: /*/blogs/*%2b*
6

Yes I am referring to this, and I want to knw if this will effect my organic reach. I am running paid ads but I am curious about my organic reach

7

There seems to be misconceptions about the default Shopify robots.txt file. Blog articles will be crawl-allowed by default.

This part of the robots.txt file seems to cause confusion:

Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*/blogs/*+*
Disallow: /*/blogs/*%2B*
Disallow: /*/blogs/*%2b*

It allows these pages:

https://example.com/blogs/news
https://example.com/blogs/tuts
https://example.com/blogs/docs
https://example.com/blogs/tuts/how-to-setup
https://example.com/blogs/blog/my-article
https://example.com/blogs/blog/meet-jerry
https://example.com/blogs/news/tagged/business-update
https://example.com/blogs/news/tagged/product-launch

It disallows these ones:

https://example.com/blogs/news/tagged/business-update**+**product-launch
https://example.com/blogs/news/tagged/business-update**%2b**product-launch

So what is robots.txt disallowing in this case?

Multi-tag filtered blog list URLs. More info:

https://shopify.dev/themes/architecture/templates/blog#filter-articles-by-tag

What is the %2b% thing?

That is literally just a URL encoded + symbol. Probably because it’s possible that some themes link to the URL encoded versions. They are technically also HTTP200 valid pages in Shopify, though they are just dupe of the ‘+’ URL version.

Why leave the default disallow?

They are harmless, if not good for you under normal use. The blog disallows are likely acting as a “catchall” to both prevent Googlebot getting into recursive multi tag page URL crawl paths. This would be bad for both crawl budget consumption as well as negatively impacting Shopify’s own server resources (at scale x 1m Shopify sites). Also, multi-tagged pages are often thin/dupe content too, so not usually worth ranking in their own right. So it makes sense why it’s there.

8

i asked GPT4 and here is its answer:

The lines you’ve provided from the robots.txt file include disallow rules related to URLs of blogs. Let’s break them down:

  1. Disallow: /blogs/+

    • This disallows any URL that starts with /blogs/ and contains a + character.

  2. Disallow: /blogs/%2B

    • This disallows any URL that starts with /blogs/ and contains the URL encoded version of the + character (%2B).

  3. Disallow: /blogs/%2b

    • This disallows any URL that starts with /blogs/ and contains the URL encoded version of the + character in lowercase (%2b).

  4. Disallow: //blogs/+*

    • This disallows any URL that has /blogs/ anywhere in the path and contains a + character.

  5. Disallow: //blogs/%2B*

    • This disallows any URL that has /blogs/ anywhere in the path and contains the URL encoded version of the + character (%2B).

  6. Disallow: //blogs/%2b*

    • This disallows any URL that has /blogs/ anywhere in the path and contains the URL encoded version of the + character in lowercase (%2b).

From these rules, it does not appear that all blogs are disallowed. Instead, only those blog URLs that contain the + character or its encoded forms are disallowed from being crawled. If your blog URLs don’t contain these specific characters, they should still be allowed by search engine crawlers.

9

Great explainer Kieran, this would be good as a spotlighted post in its own right… I noticed a lot of questions on here with the same concern as this one, but after reading about 30 posts with various responses, this is the first one that actually provided me with a simple explanation. I was getting ready to customise my robots.txt file, but now I see there’s clearly no need! Thanks.