robots txt

Shopify Discussion
1

Hello,

Do we need to edit all robots.txt pages from Disallow to Allow for the best website performance on search engines?

thanks

2

Hi @sibelle

Thank you for your question.
In fact, you shouldn’t switch all pages in the robots.txt file from Disallow to Allow.
Some pages contain duplicate or meaningless content, so blocking them is the right decision to prevent Googlebot from wasting time crawling unnecessary pages.

3

This is Richard from PageFly - Shopify Page Builder App
No, you should NOT change all Disallow rules to Allow in your robots.txt file just for better search engine performance. Doing so could lead to indexing issues, duplicate content problems, and even security risks. Instead, you should optimize your robots.txt based on best SEO practices.

:police_car_light: Why You Shouldn’t Allow Everything in robots.txt1. Unnecessary Pages Get Indexed

  • Allowing all pages can lead to search engines indexing low-value pages (e.g., cart pages, checkout, login, admin pages).
  • This can dilute your website’s SEO strength.

  1. Duplicate Content Issues

    • If search engines crawl filter pages, tag pages, or internal search pages, it can create duplicate content, affecting rankings.

  2. Wastes Crawl Budget (For Large Sites)

    • Search engines have a limited crawl budget.
    • If bots waste time crawling unnecessary URLs, important pages may not get indexed efficiently.

  3. Security Risks

    • Some private or sensitive areas (like /admin/ or /cart/) should not be crawled.

Hope my solution will help you resolve the issue.Best regards,
Richard | PageFly