Search console canonical and robots.txt issues

[Rene89]

Hello. I recently migrated our website (www.allcam.nl) to Shopify and I'm getting some conflicting data in Search console. We used to have about 800 pages indexed correctly but after migrating to Shopify we have about 4,400. A lot of URLs with parameters (from collections with filters) are indexed even though they have correctly referencing canonical URLs.

When inspecting, Google says these URLs are indexed but blocked by Robots.txt. I'm assuming this means Google also can't read the canonical URL. But we also have thousands of other similar URLs where Google correctly sees the canonical URL.

What is best to do now? I'm assuming we could add a Meta Noindex tag to URLs with parameters, but I couldn't find a simple technical solution for this. Any other ideas? Thanks for any help.

[DaisyVo]

Hi @Rene89 

I completely understand your concern about non-indexed URLs. Let me clarify a bit!

Firstly, it’s actually quite normal for a Shopify store to have more non-indexed URLs than indexed ones. This isn’t necessarily a bad thing because...

Your website contains many URLs, but Google only indexes the important and canonical ones—such as product pages, collections, blogs, and other key content. Meanwhile, non-essential URLs (like login, cart pages, and URLs with filters, parameters, and sorting options) are typically not indexed. This is completely normal and nothing to worry about!

 

Blocked by robots.txt

This means Googlebot is restricted from crawling certain pages due to your site's robots.txt file. In Shopify, URLs like cart and account pages are intentionally blocked to protect your SEO. This is actually a best practice, so there’s no need for concern. You can read more in Shopify’s Robots.txt guide.

Alternate Page with Proper Canonical Tag

This happens when a page is marked as an alternate version of another page (such as an AMP page pointing to its desktop version or a mobile version pointing to a desktop canonical). This setup ensures that the correct version is indexed, and no action is needed on your part. Note that alternate language pages may not be detected by Google Search Console.

For further details, you can check out Google’s official guide here.

Hope this helps! 

Best,

Daisy

[Rene89]

Hi Daisy,

Thank you very much for your reply. To clarify, I'm not talking about the non-indexed URLs but about the indexed ones. What I did in the meantime is to remove the Disallow: */collections/*filter*&*filter* line from our robots.txt to allow Google to crawl duplicate pages to that is can see the correct canonical URL (which it couldn't because of the blocking by robots.txt).

[DaisyVo]

Hi @Rene89 

You're on the right track by removing the *Disallow: /collections/filter&filter line from robots.txt. Now that Google can crawl these URLs, it should start recognizing the canonical tags correctly and gradually de-index those duplicate filtered URLs.

Since these URLs were previously blocked, Google may take some time to recrawl them and process the canonical tags. Here’s what you can do to speed things up:

• Be patient – Google needs time to update its index, and this process might take multiple crawls.
• Manually request reindexing – Use Google Search Console to request reindexing for key pages.
• Monitor "Indexed but blocked by robots.txt" – This number should decrease over time as Google processes the changes.

If everything goes as expected, you should see fewer filtered URLs indexed and more of them correctly canonicalized.

I hope this helps

Best,

Daisy