Edit Robots.txt to allow index of multiple tags

Edit Robots.txt to allow index of multiple tags

MBroadbridge
Shopify Partner
4 0 0

Hi, what edits do I need to make to the robots.txt file to allow Google to crawl and index multiple tags?

 

Example : /collections/demo/tag1+tag2

Currently only /collections/demo/tag1 is allowed, but +tag2 is blocked?

 

Thanks

Replies 3 (3)

PageFly-Richard
Shopify Partner
4731 1078 1743

 

Hi @MBroadbridge 

 

This is Richard from PageFly - Shopify Page Builder App

 

You will need to create a robots.txt for your site in the theme under templates structures first

 

Then you can exclude it by this

{% for group in robots.default_groups %}
  {{- group.user_agent }}


  {%- for rule in group.rules -%}
    {%- unless rule.value == '/collections/demo/tag2' -%}
      {{ rule }}
    {%- endunless -%}
  {%- endfor -%}

  {%- if group.sitemap != blank -%}
      {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

PageFlyRichard_0-1709689005859.png

 

 

Hope this can help you solve the issue 

 

Best regards,

Richard | PageFly

Please let me know if it works by giving it a Like or marking it as a solution!


➜ Optimize your Shopify store with PageFly Page Builder (Free plan available) 


All features are available from Free plan. Live Chat Support is available 24/7.

MBroadbridge
Shopify Partner
4 0 0

Thanks Richard, however how do I do this for this example : 

 

/collections/disney/phone-cases indexes

but

/collection/disney/iphone-11+phone-cases does not. I have thousands of collections, with about 80 different phone models to replace the 'iphone-11'. How do I go about this in a sensible way?

PageFly-Richard
Shopify Partner
4731 1078 1743

With collection, you can do something like this to get all

Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*

 

Please let me know if it works by giving it a Like or marking it as a solution!


➜ Optimize your Shopify store with PageFly Page Builder (Free plan available) 


All features are available from Free plan. Live Chat Support is available 24/7.