Google Search Index ... 90% of all 4,000 pages not indexed. How to correct ?

[Digit_PaxMentis]

New Shopify store for POD (Print-on-Demand) products with 22 collections, 369 main products and 4,160 variants ...

Result by Google Search console: Indexed: 401 ... not indexed: 3.62K = 90%

9 main reasons named by Google:
-         1 page : duplicate with out user-selected canonoical
-         1 page : excluded by "noindex" tag
-         1 page : blocked due to other 4xx issue
-       14 pages: discovered, not indexed
-       25 pages: blocked by robots.txt (see following 3.1 and 3.2)
-       37 pages: re-direct (Rec.: We use a shorten URL app to "re-direct" to very long URLs being used in advertising banners and on social media. Thats fine for us.)
-     109 pages: 404 not found (resolved as we found a broken URL by forwarding a Shorten URL to a deleted product page)
-     567 pages: Crawled - currently not indexed ... Google System
- 2,880 pages: Alternate page with proper canonical tag (see following position 2.)

We have studied all possible materials to get an understanding by our own, e.g. 

1. Basics about Google search indexing here in Shopify Community and external links like this. Yet, we need some guidance for following points 2 and 3.

2. Lets start with the biggest number: about "Alternate page with proper canonical tag" we found here a hint. - Can we assume, that the upper named 2,880 pages are not an issue ???  The high number is worrying, but so far as we understand, "variants" (e.g. different colours, hardware materials or sizes within one printable product refer to the main product page (in total we have 369) so therefore it won't be indexed, right ?

3. When indexing, we have used the "original robot.txt" file (under robots.txt.liquid in the theme code editor) and also offered Google Search console the different XML Sitemap pages (overview here). The number is small, but these pages are highly relevant for a trust-building process with our new customers.

Yet, we have noticed, that very relevant pages are blocked, which has not been initiated by us within the robots.txt , e.g. "policies" pages (shipping, refund, terms of services, data privacy etc. ...) are highly important for the users. - E.g. a potential buyer who sees our advertising on Instagram might first google for our brand name and policies. If not popping up, this user might stop to be further interested.

In Robots.txt we only have added following exclusion:

User-agent: *
Disallow: /checkout
Disallow: /checkout/thank_you
Disallow: /account
Disallow: /account/login
Disallow: /account/register
Disallow: /cart
Disallow: /admin

Yet, if you look at the output, it says very strange things to any search crawler. Here the robots.txt file.

3.2 Similar with with our collection pages being blocked by robots.txt . But it should not be, see 3.1.

We use often the term "new collection" for the info text in our promos on different social media (e.g. x/Twitter, Instagram, interests, reddit/reddit community). If a user might google with the brand name for collections, same here nothing would pop up.

4.) 567 pages: Crawled - currently not indexed ... Google System. What does this mean ??? No idea about at all, see screenshot. 

We'd like to avoid any "self damaging", naturally. As we need traffic and conversions. - For any hint and advice we thank in advance.

[SealSubs-Roan]

Hi @Digit_PaxMentis !

It would be best to reach out to google support as this issue is most likely directly coming from their site. They will be able to give you further insight.

[INA_MSWEB]

Hi @Digit_PaxMentis  Thank you for sharing the detailed information about your Shopify store's indexing issues. I understand the challenges you are facing with getting your pages indexed by Google, which is crucial for driving traffic and conversions. Let's address your concerns and provide actionable solutions. 

1. Understanding the High Number of Alternate Pages with Proper Canonical Tags

Regarding the 2,880 pages marked as "Alternate page with proper canonical tag," it is indeed correct that these are likely your product variants. Variants are not individually indexed but rather refer back to the main product page to avoid duplicate content issues. This is generally not a problem as long as your main product pages are well-optimized and indexed. Ensure that your canonical tags are correctly implemented on all pages.

2. Addressing Robots.txt and Blocked Pages

Your current robots.txt file seems to be blocking essential pages like policy pages and collection pages. These pages are crucial for building trust with your customers and should be accessible to search engines. (Suggested Update to Robots.txt)

User-agent: * Disallow: /checkout Disallow: /checkout/thank_you Disallow: /account Disallow: /account/login Disallow: /account/register Disallow: /cart Disallow: /admin Allow: /policies/ Allow: /collections/

This adjustment ensures that your policy and collection pages are not blocked and can be indexed by search engines.

Collection Pages Not Indexed

Ensure that your collection pages are not disallowed in the robots.txt file. Additionally, check if these pages have any "noindex" tags that might be preventing them from being indexed.

3. Pages Crawled but Not Indexed

The 567 pages that are "Crawled - currently not indexed" by Google's system usually indicate that Google has visited these pages but decided not to index them. This could be due to several reasons, such as:

• Low-Quality Content
• Duplicate Content
• Insufficient Internal Links

Regularly monitor Google Search Console for any changes or updates regarding these pages.

4. Other Issues

• Duplicate without user-selected canonical: Ensure canonical tags are correctly implemented and point to the main version of the page.
• Noindex Tags: Review your pages to ensure no important pages are tagged with "noindex."
• 4xx Errors: Resolve any broken links and ensure all URLs are functioning correctly.

5. Shopify Coupon Discount on Cart App

Implementing the Coupon Discount on Cart app can significantly enhance your sales strategy. Displaying available discounts directly on the cart page encourages customers to complete their purchases. Transparency with discounts and promotions enhances customer satisfaction and trust. Offering tiered discounts or time-limited promotions can incentivize customers to add more items to their carts. The app provides insights into discount usage, helping you optimize your promotional strategies effectively. 

If you need further assistance or have any more questions, feel free to reach out. I am here to help you succeed.

Best regards,

Ina

[EcomSEO]

Hey!

I have gone over your website, It seems as though the robots.txt file has expanded>
Have you solved this issue?

I have a few pointers about your SEO and Navigational structure if your interested