I want to remove the indexed URLs containing “collections/vendors” and “collections/products,” as well as everything that follows these URLs. Additionally, I want to stop the crawlers from crawling my website.
Is the following code correct for achieving this?
{%- if group.user_agent.value == ‘’ -%}
{{ 'Disallow: /collections/product’ }}
{{ ‘Disallow: /collections/products*’ }}
{%- endif -%}
Also, what would the code be if I wanted to allow indexing for a specific URL like “/collections/products/salt-black” or multiple URLs in bulk?