pages not indexed blocked by robots.txt

Shopify Discussion

Topic summary

A Shopify store owner reports 395,000+ pages blocked from Google indexing due to robots.txt issues. Screenshots show Google Search Console errors indicating pages blocked by robots.txt.

Technical Details:

  • The user shared their complete robots.txt file, which appears to be a standard Shopify configuration
  • File includes typical disallow rules for checkout, cart, admin, and filtered collection pages
  • Sitemap is properly declared at the end of the file

Community Response:

  • Another user indicates this is a known, ongoing issue affecting multiple Shopify stores
  • Claims the problem has persisted for over a year without resolution from Shopify
  • Suggests this is a platform-wide concern rather than an individual configuration error

Status: The issue remains unresolved with no clear solution provided in the thread.

Summarized with AI on November 1. AI used: claude-sonnet-4-5-20250929.
1

pages blocked by robots.txt i have over 395k pages not indexed

can anyone help

www.mommyandme.online

WhatsApp Image 2024-12-30 at 2.37.29 PM.jpeg
WhatsApp Image 2024-12-30 at 2.37.29 PM.jpeg720×1280 113 KB

WhatsApp Image 2024-12-30 at 2.38.08 PM.jpeg
WhatsApp Image 2024-12-30 at 2.38.08 PM.jpeg1280×720 100 KB

2

what do you mean by collaboration code

3

we use Shopify as our ecommerce platform

User-agent: *
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /26797867177/checkouts
Disallow: /26797867177/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/sort_by
Disallow: //collections/sort_by
Disallow: /collections/+*
Disallow: /collections/%2B
Disallow: /collections/%2b
Disallow: //collections/+*
Disallow: //collections/%2B*
Disallow: //collections/%2b*
Disallow: /collections/filter&filter
Disallow: /blogs/+*
Disallow: /blogs/%2B
Disallow: /blogs/%2b
Disallow: //blogs/+*
Disallow: //blogs/%2B*
Disallow: //blogs/%2b*
Disallow: /?oseid=
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /policies/
Disallow: //policies/
Disallow: //?ls=&ls=*
Disallow: //?ls%3D%3Fls%3D*
Disallow: //?ls%3d%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/.js
Disallow: /recommendations/products
Disallow: //recommendations/products
Sitemap: https://www.mommyandme.online/sitemap.xml

Google adsbot ignores robots.txt unless specifically named!

User-agent: adsbot-google
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /orders
Disallow: /26797867177/checkouts
Disallow: /26797867177/orders
Disallow: /?oseid=
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /cdn/wpm/.js

User-agent: Nutch
Disallow: /

User-agent: AhrefsBot
Crawl-delay: 10
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /26797867177/checkouts
Disallow: /26797867177/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/sort_by
Disallow: //collections/sort_by
Disallow: /collections/+*
Disallow: /collections/%2B
Disallow: /collections/%2b
Disallow: //collections/+*
Disallow: //collections/%2B*
Disallow: //collections/%2b*
Disallow: /collections/filter&filter
Disallow: /blogs/+*
Disallow: /blogs/%2B
Disallow: /blogs/%2b
Disallow: //blogs/+*
Disallow: //blogs/%2B*
Disallow: //blogs/%2b*
Disallow: /?oseid=
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /policies/
Disallow: //policies/
Disallow: //?ls=&ls=*
Disallow: //?ls%3D%3Fls%3D*
Disallow: //?ls%3d%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/.js
Disallow: /recommendations/products
Disallow: //recommendations/products
Sitemap: https://www.mommyandme.online/sitemap.xml

User-agent: AhrefsSiteAudit
Crawl-delay: 10
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /26797867177/checkouts
Disallow: /26797867177/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/sort_by
Disallow: //collections/sort_by
Disallow: /collections/+*
Disallow: /collections/%2B
Disallow: /collections/%2b
Disallow: //collections/+*
Disallow: //collections/%2B*
Disallow: //collections/%2b*
Disallow: /collections/filter&filter
Disallow: /blogs/+*
Disallow: /blogs/%2B
Disallow: /blogs/%2b
Disallow: //blogs/+*
Disallow: //blogs/%2B*
Disallow: //blogs/%2b*
Disallow: /?oseid=
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /policies/
Disallow: //policies/
Disallow: //?ls=&ls=*
Disallow: //?ls%3D%3Fls%3D*
Disallow: //?ls%3d%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/.js
Disallow: /recommendations/products
Disallow: //recommendations/products
Sitemap: https://www.mommyandme.online/sitemap.xml

User-agent: MJ12bot
Crawl-delay: 10

User-agent: Pinterest
Crawl-delay: 1

4

Welcome to Shopify!

More than a year that this problem exists, @Shopify_77 does nothing while the community keeps asking them to fix this SERIOUS problem .