Why are there unexpected changes in my website's Robots.txt?

Technical Q&A
1

Hi all - I got an alert that 204 (11 pages) Robots.txt changes were made to my website. We did not make any changes and it’s a bit concerning.

  1. Does Shopify Theme automatically make changes to this?

  2. How do I know if these changes are good? Seems like it is not allowing certain bots, including adsbot-google to crawl my site

  3. Can anyone help?

Thanks in advance!

Allison

2

Did you recently add some product images or install /use an SEO app?

3

Hi @AllisonG ,

Shopify may update the default robots.txt file from time-to-time to make sure that irrelevant content doesn’t get indexed by search engines. This makes sure that only the main website pages (such as your homepage, products, collections, etc) are indexed.

You may notice a section in the robots.txt file that looks something like this:

# Google adsbot ignores robots.txt unless specifically named!
User-agent: adsbot-google
Disallow: /checkouts/
Disallow: /checkout
Disallow: /carts
Disallow: /orders

This is nothing to worry about. This code is simply telling the Google Adsbot not to look at the cart, order and checkout pages. The bot will still look at all your other website pages.

The best way to monitor your website is to use the Google Search Console. This will tell you which pages have been blocked by your robots.txt file. If you ever notice important pages are being blocked, then you should investigate that, otherwise everything is working as it should.

All the best with your store!