Google Search Console 5000 pages not indexed – Server error (5xx), with only 170 products in store

[AlphaOmega]

Background - we started from scratch on Shopify about two years ago. Always on the same 2.0 Shopify Theme. Recent activity on our part includes: shortening product meta descriptions, adding tags to some products, adding descriptions to collection pages.

I received the following email from Google

Detailed Index Report

The detailed index report has 8 reasons

1. Alternate page with proper canonical tag
2. Not found (404)
3. Blocked by robots.txt
4. Page with redirect
5. Excluded by ‘noindex’ tag
6. Server error (5xx)
7. Crawled - currently not indexed
8. Duplicate, Google chose different canonical than user

I read from another similar question https://community.shopify.com/post/2453683 a solution that said “as long as Google continues indexing pages it’s supposed to and you keep working on making sure it’s high-quality content you’re putting on the site, you’ll see no problems”. It also said “Provided you do occasional checks to make sure that the right pages are being indexed it’s fine as long as canonical URLs are set up it won’t be a problem”

This sounds great, but should I not address some of these 8 reasons. For example: reason 2: Not found (404). Should these pages be deleted somehow? If so, how? Should I look into any of the 8 reasons? Which ones? How?

The solution also says “Provided you do occasional checks to make sure that the right pages are being indexed it’s fine as long as canonical URLs are set up it won’t be a problem”

Questions:

1. How do I know the “right pages are being indexed”
2. How do I setup canonical URLS? For the right pages?

[KieranR]

1. Right pages generally means:

• All your product pages
• All your collection pages
• Home page
• Your articles and pages if you have any

You want those pages getting regularly crawled and indexed because those are the entry points to your site from Google. I would be very surprised if they are not by indexed because defaults are generally good.

2. Canonical URLs are normally pre-setup in theme code
You probably don’t need to do setup anything if it’s already done by default. Themes nearly all have some default canonical rules built in.

Most Shopify themes will:

• Self-canonicalise ‘root’ product URLs like /product/red-tshirt
• Canonicalise alt product URLs to the ‘root’ product URL eg /collection/tshirts/product/red-tshirt
• Canonicalise product variants ?variant=xyz to the root product page
• Self-canonicalise home
• etc..

Look into each of those GSC problems one by one:

• Do research on each one
• Group by the URL types
• Find the cause
• Research the Shopify SEO context
• Research the SEO best practise
• Decide on a fix

Generally you’d want to try fix those 404’s. That’s a great place to start like you’ve pointed out. The others can be a bit more “it depends”

For fixing 404’s the process is something like this:

• Get list from GSC
• Crawl site using ScreamingFrog or Sitebulb
• For each 404 page, identify source
  • Source being how Google is finding that page, is it coming from sitemap, a broken internal link, a broken backlink etc
• Usually the fix is:
  1. Fix the originating link at source if possible
  2. If not possible, redirect:
     • To an equivalent page, if exists
     • If no equivalent page, to parent category
     • If no parent, to home

[AlphaOmega]

Thank you for the reply, it is greatly appreciated! Would you be so kind to reply to all 8 reasons individually?

Reason 2 Not found (404) – Yesterday I fixed most of these. I redirected them to their appropriate collection page, which was the highest level I was able to redirect them to. There were about 20 urls in this section that have no real association with our site. 5 examples are: /cdn, /wpm, /${e}, /contact, /comments. Should these be redirected to the home page? I’ve read conflicting opinions on the matter, some say fix all, some say having some 404s isn’t a bad thing…

Reason 1 Alternate page with proper canonical tag. I’m not sure if these needs fixing and if so, how. I did review a number of the URLs to see if I understand why, these are listed. It appears that all of the URLS in this section are associated with a new variant app we are using.

Most of these links bring me to their new variant page. Do I want every variant indexed? Some of the URLs in this section give me a 404 Page Not Found error. Is this something that the app developer should fix? If so, what should I tell them, and how do I verify it’s been resolved?

Reason 7 Crawled - currently not indexed. Most of the URLs in this section are also products using our new variant app. Not sure what to do with these as well☹. A few URL’s in this section end in .atom. When I click on them it brings up code. What should I do with these?

Reason 3 Blocked by robots.txt. When I click on most of these, they bring me to pay.shopify.com/pay/…. sites. Some bring me to my search, cart, checkout…pages. I assume these are OK and nothing for me to fix.

Reason 4 Page with redirect. These appear to all be products. Most with a variant, some without. Not sure if there is anything I should do.

Reason 5 Excluded by ‘noindex’ tag. More pay.shopify.com/….URLs. I don’t think I need to fix.

Reason 6 Server error (5xx). More pay.shopify.com/….URLs. I don’t think I need to fix.

Reason 8 Duplicate, Google chose different canonical than user. My password URL. Not sure if I should fix, if so how?

Well that the last of the dreaded 8 plagues sent by Google. I suppose I should be happy there isn’t 10, and it doesn’t end with me and our products buried at the bottom of the Red Sea where no one can find them . Any help is greatly appreciated!

[KieranR]

Really hard to answer without spending some proper time diagnosing then looking at options to fix.

In general, errors from GSC and SEO tools are about as useful as a “check engine” light to a mechanic, needs context and proper diagnosing before jumping to a fix.

But making some shotgun assumptions, I can make some guesses:

Reason 7 crawled - currently not indexed - If they are not unique (h1, title, meta, content) then no point having variants indexed separately. No real harm in the .atoms getting crawled

Reason 3 blocked by robots.txt - they don’t need to be indexed, no harm in seeing those there

Reason 4 Page with redirect - need to look at a few live examples and see the source/destination page and figure out why there might be or might need to be a redirect and if there should be a redirect

Reason 5 - excluded by noindex tag - ignore

Reason 6 - If they are just cart, checkout or payment pages doesn’t matter.

Reason 8 - dunno what’s the password URL? Can’t remember seeing those. But it sounds unimportant.

[AlphaOmega]

I’ve been doing some further research. I’m looking for advise on reason 1 and 7.

• Summary
  • I have a Shopify app that provides variant options (color, size).
  • I looked up two sample products from my Shopify store using this variant feature in my Google Search Console.
  • Both main product page root URLs are properly indexed
  • On my Google Search Console for pages that aren’t indexed: Product A has 12 unique URLs for “Alternate page with proper canonical tag and 18 unique URLS for “Crawled – currently not indexed”. Product B has 8 unique URLs for “Alternate page with proper canonical tag and 7 unique URLS for “Crawled – currently not indexed”.
  • Example B
    • Root URL: /products/yellow-grosbeak-ceramic-figurine
    • Alternate page with proper canonical: /products/yellow-grosbeak-ceramic-figurine?pr_prod_strat=e5_desc&pr_rec_id=6a7e2dd59&pr_rec_pid=7792988881138&pr_ref_pid=7792837263602&pr_seq=uniform
    • Crawled – currently not indexed: /products/yellow-grosbeak-ceramic-figurine?pr_prod_strat=e5_desc&pr_rec_id=9c6cb46be&pr_rec_pid=7792988881138&pr_ref_pid=7792849387762&pr_seq=uniform
  • Questions
    • How do I know if the “Alternate page with proper canonical” URLs are referencing the my indexed main product page URL?
  • Possible solutions – What is best practice?
    • Make these URLs page variants inaccessible to search engines through for example using a “#” in the URL
    • Turn “Crawled – currently not indexed” Urls into “Alternate page with proper canonical” Urls
    • Using my robots.txt file to prevent search engines from crawling only the the “Crawled – currently not indexed” URLs
    • Thanks!

[SEOAnt-Zoey]

The items in your screenshot that are not indexed:
You can check “Not found (404)” and "Excluded by ‘noindex’ tag.
You don’t need to worry about the other items because they are all “duplicate pages”. Just like you said you have 170 products, but it told you that 5,000 pages are not indexed.

Google will basically only index one page it considers to be a canonical page in a group of duplicate pages (referring to different URL names, but the content is exactly the same). Google has its own algorithm to identify canonical pages, which we cannot control. Even if the Shopify page has the code for specifying the canonical page, Google will refer to the one you specify, but the canonical page actually identified by Google will be different from the page you specify.

[Tara-C]

These are noindex urls from Shopify. Google is just reporting it and it does not affect anything. The 5xx server errors are coming from Shop Pay, but there are reported noindex so it’s not for concern, although it sends annoying emails. To disallow the noindex urls you can follow these steps to setup a new robots.txt.liquid template https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt then update with the code below, the request a recrawl in GSC.

# we use Shopify as our ecommerce platform

{%- comment -%}

# Caution! Please read [https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt](https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt) before proceeding to make changes to this file.

{% endcomment %}

{% for group in robots.default_groups %}

  {{- group.user_agent -}}

  {% for rule in group.rules %}

    {{- rule -}}

  {% endfor %}

  {%- if group.sitemap != blank -%}

    {{ group.sitemap }}

  {%- endif -%}

{%- if group.user_agent == '*' -%}

    {{ 'Disallow: /services/login_with_shop/*' }}

  {%- endif -%}

{% endfor %}