Good morning all,
The robots.tx should be all disallowed?
Thanks,
Manuel
Topic summary
A user questions whether their robots.txt file should disallow all pages from search engine crawlers. Community members clarify that blocking everything would prevent search engines from indexing the site, which is typically undesirable.
Key guidance provided:
- Only specific pages should be disallowed (admin panels, checkout pages, cart, account areas)
- Critical pages like homepage, collections, and product pages must remain crawlable for SEO
- The user’s current Shopify-generated robots.txt appears standard and appropriate
Current status:
The user shares their existing robots.txt configuration and store URL for review. One responder offers to analyze the specific store setup, while another provides a Google resource explaining robots.txt functionality and basic syntax examples.
The discussion remains open as the user seeks specific recommendations on which rules to modify, though the consensus suggests their current configuration may already be appropriate for a typical e-commerce store.
No, it should not disallowed all pages. It disallowed just pages that should not access by the customer. like the Admin Panel, Online Store Editor etc.
Good morning
Thanks for the quick answer, can you please explain how to change to
allowed and which ones to change
@Coelho63 it depends on your goals, however, setting robots.txt to disallow all will prevent search engine crawlers from indexing your site.
Hi PieLab
Attached are my store robots.txt, i have no knowledge to change them. Can
you please let me know which ones should I allow
we use Shopify as our ecommerce platform
User-agent: *
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /90662699346/checkouts
Disallow: /90662699346/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/sort_by
Disallow: //collections/sort_by
Disallow: /collections/+*
Disallow: /collections/%2B
Disallow: /collections/%2b
Disallow: //collections/+*
Disallow: //collections/%2B*
Disallow: //collections/%2b*
Disallow: /collections/filter&filter
Disallow: /blogs/+*
Disallow: /blogs/%2B
Disallow: /blogs/%2b
Disallow: //blogs/+*
Disallow: //blogs/%2B*
Disallow: //blogs/%2b*
Disallow: /?oseid=
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /policies/
Disallow: //policies/
Disallow: //?ls=&ls=*
Disallow: //?ls%3D%3Fls%3D*
Disallow: //?ls%3d%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/.js
Disallow: /recommendations/products
Disallow: //recommendations/products
Sitemap: https://lojamanuelcoelho.com/sitemap.xml
User-agent: adsbot-google
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /orders
Disallow: /90662699346/checkouts
Disallow: /90662699346/orders
Disallow: /?oseid=
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /cdn/wpm/.js
User-agent: Nutch
Disallow: /
User-agent: AhrefsBot
Crawl-delay: 10
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /90662699346/checkouts
Disallow: /90662699346/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/sort_by
Disallow: //collections/sort_by
Disallow: /collections/+*
Disallow: /collections/%2B
Disallow: /collections/%2b
Disallow: //collections/+*
Disallow: //collections/%2B*
Disallow: //collections/%2b*
Disallow: /collections/filter&filter
Disallow: /blogs/+*
Disallow: /blogs/%2B
Disallow: /blogs/%2b
Disallow: //blogs/+*
Disallow: //blogs/%2B*
Disallow: //blogs/%2b*
Disallow: /?oseid=
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /policies/
Disallow: //policies/
Disallow: //?ls=&ls=*
Disallow: //?ls%3D%3Fls%3D*
Disallow: //?ls%3d%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/*.js
Sitemap: https://lojamanuelcoelho.com/sitemap.xml
User-agent: AhrefsSiteAudit
Crawl-delay: 10
Disallow: /a/downloads/-/*
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkouts/
Disallow: /checkout
Disallow: /90662699346/checkouts
Disallow: /90662699346/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/sort_by
Disallow: //collections/sort_by
Disallow: /collections/+*
Disallow: /collections/%2B
Disallow: /collections/%2b
Disallow: //collections/+*
Disallow: //collections/%2B*
Disallow: //collections/%2b*
Disallow: /collections/filter&filter
Disallow: /blogs/+*
Disallow: /blogs/%2B
Disallow: /blogs/%2b
Disallow: //blogs/+*
Disallow: //blogs/%2B*
Disallow: //blogs/%2b*
Disallow: /?oseid=
Disallow: /preview_theme_id
Disallow: /preview_script_id
Disallow: /policies/
Disallow: //policies/
Disallow: //?ls=&ls=*
Disallow: //?ls%3D%3Fls%3D*
Disallow: //?ls%3d%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association
Disallow: /.well-known/shopify/monorail
Disallow: /cdn/wpm/*.js
Sitemap: https://lojamanuelcoelho.com/sitemap.xml
User-agent: MJ12bot
Crawl-delay: 10
User-agent: Pinterest
Crawl-delay: 1
Hey thanks,
Can you please share your store URL so that I take a look and based on your store I provide you robot.txt
Thanks
Hi @Coelho63
Honestly to say, the robots.txt should not be all disallowed. For some specific pages, like Contacts, admin pages, or Return Policy, they can be set as disallowed. But for some crucial pages like the Homepage, Collection Page, and Product Page, the robots.txt file should be allowed.
If you are not familiar with how robots.txt works, you can check this article about Introduction to robots.txt from Google. When you intend to disallow robots.txt on a specific file or folder, you can add the below codes in robots.txt directly:
User-agent: *
Disallow: /topsy/
Disallow: /crets/
Disallow: /hidden/file.html
I hope my answer can work on your raised question here. Thank you!