Edit Robots.txt to allow index of multiple tags

Technical Q&A
1

Hi, what edits do I need to make to the robots.txt file to allow Google to crawl and index multiple tags?

Example : /collections/demo/tag1+tag2

Currently only /collections/demo/tag1 is allowed, but +tag2 is blocked?

Thanks

2

Hi @MBroadbridge

This is Richard from PageFly - Shopify Page Builder App

You will need to create a robots.txt for your site in the theme under templates structures first

Then you can exclude it by this

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {%- unless rule.value == '/collections/demo/tag2' -%}
      {{ rule }}
    {%- endunless -%}
  {%- endfor -%}

  {%- if group.sitemap != blank -%}
      {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

PageFlyRichard_0-1709689005859.png
PageFlyRichard_0-1709689005859.png1896×1122 234 KB

Hope this can help you solve the issue

Best regards,

Richard | PageFly

3

Thanks Richard, however how do I do this for this example :

/collections/disney/phone-cases indexes

but

/collection/disney/iphone-11+phone-cases does not. I have thousands of collections, with about 80 different phone models to replace the ‘iphone-11’. How do I go about this in a sensible way?

4

With collection, you can do something like this to get all

Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*