How to correctly modify Robot.txt to block a specific site?

[zerozenecostore]

Hello All

I need a little help before I make a stupid mistake.. 

I have a back link from a dodge site, did not ask for it and wanted to block the site so they remove the link for ages but was unable to remove it because I needed to add the URLmetrics to the robot.txt but I see Shopify now lets us change the robot.txt.. I am not planning to change anything else as I personally think unless you really know what you are doing you should not be playing around with that file/or any.. but I would like to block this site..  

User-agent: URLmetrics
Disallow: /

But unsure where I put it on this page before I save it..  any advice would be amazing 🙂

# we use Shopify as our ecommerce platform
{%- comment -%}
# Caution! Please read https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt before proceeding to make changes to this file.
{% endcomment %}
{% for group in robots.default_groups %}
{{- group.user_agent -}}

{% for rule in group.rules %}
{{- rule -}}
{% endfor %}

{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}

Kind Regards 

[zerozenecostore]

Sry to bump but could no one advise me on this pls? Really do not want to mess things up 

[KieranR]

I think you're confusing two things.

Robots.txt is is used to control crawlers / bots. If they are nice bots they respect your rules and not crawl your site, or the parts of your site you don't want them to.

Disavows are a Google Search Console feature typically used to indicate to Google that "this link is dodgy don't count it". Different SEO's have different opinions on if this is even worth doing.

Also, what makes you think the link is dodgy?

[zerozenecostore]

It has a spam score of 60+ and link checker tools all say this is a bad link. It's a no follow but I still want it removed as I never asked them to list my site. Nor have I paid for any links..  To remove without disavow i need to add text to the robot.txt.. 

urlm.co.uk/remove/

I'm more confused how to add code to the robot txt than I am about what the purpose of a robot txt is used for 🙂

🙂

I think this is one of them bad bots you are talking about that dont respect others..

Thanks 

[KieranR · Accepted solution]

So there's a difference between bad bots and bad links. They are two completely different things. 

But to answer your question, if you want to request that bot not crawl your site. You could add the static disavow rule above the stuff in the Shopify robots.txt.liquid template: 

User-agent: URLmetrics
Disallow: /

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.user_agent.value == '*' -%}
    {{ 'Disallow: /*?q=*' }}
  {%- endif -%}

  {%- if group.sitemap != blank -%}
      {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Edit: ahh right so this is actually to meet the requirements of the form you linked: 

[zerozenecostore]

Success

Website removed from Database

Please allow upto 10 days for Search engines to update their indexes

Thank you it worked, all I was trying to achieve was remove the bad link, unsure how they go my details in the first place because other sites are not listed, this link I believe was not hurting my ranking in anyway but I still wanted to remove just incase it had an effect with future  algorithms updates.

Again thank you