Hello,
I am currently troubleshooting an issue. The vast majority of my pages are not being indexed (42,600), while only 1720 are being indexed (and going down). Is anyone able to examine my robots.txt file and tell me if there is anything that would be causing this?
Thank you!
Hey CaptBret,
This is normal to see a bunch of URLs not being indexed, as each page can have many different URL’s that lead to it. It’s basically just automatically avoiding indexing URL’s that lead to pages which are already indexed.
It’s also worth noting that the robots.txt file is generated automatically (in the same way for all Shopify stores), and cannot be manually edited.
There’s nothing you need to do here (or worry about). As long as you’re seeing a reasonable amount of pages being indexed (ex. the 1720 that you referred to) then you should be good-to-go. Think about how many products/pages/collections/blogs that you actually have on the store – you probably have closer to 1720 in total, than 42,600 … which means the indexing is working correctly.
Cheers,
Stephen
Hello Stephen,
Thank you for your reply, which is similar to responses that I have received elsewhere.
First, I have every reason to be concerned. Prior to March 2023 I was receiving orders. Since March I have received 4 orders. That is very concerning. Catastrophic, in fact.
“Think about how many products/pages/collections/blogs that you actually have on the store – you probably have closer to 1720 in total, than 42,600”
I don’t need to think about it, I know exactly. I have 36629 products, divided into 1775 Collections, and 14 pages. Each product has a unique and accurate title, with matching URL, and SEO Title. The same with the collections.
The majority of products are unique to the internet. Prior to 2023, I could enter a product name into Google, or any search and be first in the SERPS. Now that same product is not indexed. (Referencing one product is just an example, the same has happened to most)
I spent countless hours creating all of the proper SEO for EACH product and collection. It is pounded into us that this is critical. If a product is not going to be indexed, why bother with the SEO on the products in the first place? Are we supposed to just add SEO to all and hope the Google indexes a handful of them and be thankful?
I can point you to an e-commerce store that I do business with. They launched their new site with Shopify about 2-3 years ago. They have thousands more products that I have. If you search any product they will come up 1st or 2nd in the serps every time. The other competing company is also usually always 1st or 2nd but not a SHopify Store. It would appear that in both cases, each of their products are being indexed.
“It’s basically just automatically avoiding indexing URL’s that lead to pages which are already indexed. ”
Most of my products are not being indexed at all. I’m not dropshipping and the vast majority of the products are unique to the e-commerce world and internet.
I previous website that I owned (not Shopify) offered over 1000 t-shirts. I could search any product name and come up first in the SERPS, so each of those products were being indexed.
As far as editing, the robots.txt file, it appears that it can be edited but not something I would attempt.
https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
My initial question was to ask if someone could examine the robots.txt file to determine if there is anything instructing the search engines to not index products.
It is extremely odd to offer a product with a unique name, URL and SEO title, with no other product like it in existence on the internet, and have that product not appear in search.
This product is not being indexed:
https://seakoast.net/products/chickahominy-river-virginia-nautical-chart-puzzle
If you search the product name, the product does not come up, and there are no others like it.
There honestly has to be more than ‘Everything is fine’.
Another note: I updated from the old Debut Theme to the 2.0 Craft theme in Feb/March 2023. While using Debut I was being indexed and receiving orders. Since switching to Craft website traffic and orders have all but dried up.
There must be a solution somewhere.
By the way, the number of indexed pages has been steadily declining. On 8/7/203 There were 3033 indexed pages. As of 11/4/2023 there are 1670 pages being indexed, which is lower than when I first posted this question.
If my issue is not a robots.txt issue, what could it be? Do I have to manually submit each product in Google Search Console?
Something else?
After posting the previous, I see that Google sees the pages but is not indexing them. Can a robots.txt file indicate to google not to index pages? I need to find a solution and make whatever necessary fixes that I can. The process of creating this website has been all consuming and I honestly thought I was following all SEO ‘good practices’
Hey Bret,
Thank you for sharing those extra details.
I didn’t realize that your site had 36k+ products. In theory, you should be seeing more of them indexed, in that case.
Based on the last screenshot you shared, from within Google’s search console, it’s showing the indexing as “not started” … which could mean that you just need to wait for Google to catch up … but it could also mean that Google has decided not to index your pages.
I assume you’ve already submitted your sitemap to Google, based on the above screenshot? If you haven’t then definitely do that as soon as possible (https://help.shopify.com/en/manual/promoting-marketing/seo/find-site-map#submit-your-sitemap-file-to-google-search-console) and then wait a week or so, to see if how those ‘indexed’ numbers improve.
Here’s Google’s official help doc on indexing pages: https://support.google.com/webmasters/answer/7440203?hl=en
It mentions that it can “take weeks” for pages to be indexed once you’ve created them and submitted your sitemap.
I also found this help doc (https://clashpanda.com/crawled-currently-not-indexed/) which explains the “crawled - not currently indexed” label.
“When Google crawls a website’s URLs, it analyzes the content and determines whether it is relevant and useful for search users. If Google determines that the content is not valuable or relevant, it may choose not to index the URL. Alternatively, Google may choose not to index the URL because it has determined that it is a duplicate or very similar to other pages that are already indexed.”
If you’re certain that no one else is selling similar products to yours, and that you don’t have any other websites with these products on them (ie. other ecommerce platforms), then the issue is most likely with Google directly (and not your Shopify store). Their search engine can kind of do whatever it wants, in terms of indexing.
I did search for one of your products at random on Google (“Nautical Chart-11521 Charleston Harbor-approaches - Duvet Cover”) … and there were very similar results being shown (in terms of page names & images).
It could be that your products are too similar to other indexed products/pages, and then Google thinks that they’re the same thing. Also when you compare the product images, they’re very close to the same as the actual map images. Google might not be smart enough to realize you’re selling a “duvet cover” rather than the same map that other stores/sites are selling or showcasing.
One thing you can also try doing is requesting “validation” from Google (see here: https://support.google.com/webmasters/answer/7440203?hl=en#validation) … this functional let’s you request that Google validate your pages for future consideration of indexing.
The only other thing I could suggest is posting to Google’s own help forums/community for help with this issue (https://support.google.com/webmasters/community?hl=en&help_center_link=[7440203,%22Page%20Indexing%20report%22]&sjid=3593174454090175801-NC). There’s also a ‘feedback form’ you can fill out in the bottom right-hand corner of this page, which I believe is about as close to a ‘contact form’ as there is with Google.
I wish I had a more for-sure ‘fix’ for you here. I apologize in advance if you don’t find my responses helpful. It’s worth noting that I don’t work for Shopify, and I’m just trying to help out here on my own time (for free).
I wish you the best of luck with your indexing!
Cheers,
Stephen
Sorry, just wanted to add to my last message:
You provided this product as an example: https://seakoast.net/products/chickahominy-river-virginia-nautical-chart-puzzle
Which has the name “Chickahominy River-virginia Nautical Chart - Puzzle”.
When I search that name on Google, your product is actually the 1st result:
… so in that specific case, it does look like the indexing is working correctly.
Also, you mentioned wanting someone to take a look at your ‘robots’ file (https://seakoast.net/robots.txt) – which is essentially a list of page URLs/pathways that Shopify is requesting not be indexed. If you go to the link above, it shows you in plain-text which URLs/pathways are being “disabled” – they’re all duplicate forms of the regular/main URLs, and none of them reference “products” – so none of your products should be getting blocked on Shopify’s end (which is good news, as far as your indexing concerns go).
Again, I wish you the best of luck with your indexing. Hopefully Google will clue-in and realize that your pages are legit and not just duplicates of existing content elsewhere on the internet! 
Cheers,
Stephen
