Robot.txt file are blocking my product pages

Technical Q&A
, , , ,
1

Hi Everyone!

I have had a huge drop in traffic to my site. I couldn’t figure out why. I went to Google search console and found out that my product pages and collection pages are being blocked my a robot.txt file. If none of my product pages and collections are able to be crawled. I won’t get any traffic my search engines to those pages.

I am using the Dawn theme. Does anyone have any suggestions on how to fix this?

eleventhirteenindexrobot
eleventhirteenindexrobot1366×688 103 KB

2

It’s not your robots.txt
Something else is going on…Here’s a breakdown

  1. Robots.txt does not have any disallow line items blocking product pages.
    https://eleventhirteenent.com/robots.txt

The only pages being blocked having to do with products is recommendations.

image
image511×640 10.1 KB

  1. I can use my crawling tool and Crawl as Googlebot and the page returns a 200 (Good) Indexable status code.

image
image922×457 18 KB

So there may be something else going on. Try to submit that URL to google again for crawling through search console.

1 Like
3

Okay great! Thank you so much for your thorough reply. I started requesting reindexing and that seems to be working.

4

@costumecavern

Hi! :waving_hand:

You’re absolutely right — if your robots.txt file is blocking Google from crawling your product and collection pages, it can severely affect your SEO and traffic.

Here’s what you can do:

:white_check_mark: Step-by-Step Fix:

  1. Check robots.txt
    Go to: yourstore.com/robots.txt and look for these lines:

    Disallow: /collections/
Disallow: /products/

    If they’re present — that’s the problem.

  2. Customize robots.txt in Shopify
    Since Shopify allows limited customization of the robots.txt file (starting from 2021), follow these steps:

    • From your Shopify admin, go to Online Store > Themes.
    • Click Actions > Edit Code on your live theme.
    • In the Layout or Templates folder, click Add a new file and select robots.txt.liquid if it’s not already there.
    • Customize it by removing the Disallow rules for /products/ and /collections/.

    Example:

    User-agent: *
Allow: /products/
Allow: /collections/

  3. Save and publish the theme changes.

  4. Re-submit to Google Search Console

    • Go to URL Inspection Tool and re-submit a few key product and collection pages for indexing.

Note: Shopify generates the robots.txt automatically, so proceed with care when editing. If you’re unsure, it’s best to work with a developer or SEO expert