How can I edit robots.txt

Solved
House_Of_Hoye
New Member
1 0 1

I need to edit robots.txt file because, the "https://www.jeremy-hoye.co.uk/collections/unique-engagement-rings"  page not getting link juice value.  How to modify robots file? 

 

Replies 92 (92)
blyons71
New Member
1 0 0

2013 and still nothing from Shopify. This is completely ignoring customers at this point. Why would you remove a URL from robots.txt. Perhaps you have a filtered collection page you want to quickly use for a Google Ad...blocked by robots...Google disapproves. Perhaps you have a Search page you want to quickly use for a Google Ad...blocked by Robots.txt...disapproved by Google. At this point it is just arrogance. Clearly Shopify unlike the other platforms like Magento and Salesforce and even Oracle Commerce do not consult with the SEO community. This is why it is not even a contest when competing against a Shopify website.

 

S3ratin
Tourist
8 0 4

I can't get my blog post to be indexed in google console. Google telling me its blocked by tag "noindex". So my question here is: Why robot.txt blocking all Blog posts with or without tags? 

I contact the Shopify team and they told me it's not by them. So, I install a fresh new Shopify theme and test live URL for blog, guess what...

 

Google Console

URL Inspection: https://pimpyworld.com/blogs/birthday-party-ideas

AvailabilityExcluded by ‘noindex’ tag

Indexing allowed?: No: 'noindex' detected in 'robots' meta tag

 

ss.jpgShopify is telling me again it's not done by them and I understand they blocking tags for being indexing by Google, but they blocked the whole Blog section.

When you talking about this they are all blinde and silent with the same answer as always. Shopify doing best what you need for your bla bla bla.

 

How can make my blog live to google search? Please, anybody... It's obvious they don't want to talk about this and they block it with purpose. Something between Google and Shopify is not right and they hiding it in robot.txt but is it possible to get this done or its best to make a custom page as blog article?

 

Please, anybody, expect Shopify and all people paid by Shopify (you making idiots of us) to write so suggestions or solutions.

 

Thank you,

Jovica

S3ratin
Tourist
8 0 4

I can't get my blog post to be indexed in google console. Google telling me its blocked by tag "noindex". So my question here is: Why robot.txt blocking all Blog posts with or without tags? 

I contact the Shopify team and they told me it's not by them. So, I install a fresh new Shopify theme and test live URL for blog, guess what...

 

Google Console

URL Inspection: https://pimpyworld.com/blogs/birthday-party-ideas

AvailabilityExcluded by ‘noindex’ tag

Indexing allowed?: No: 'noindex' detected in 'robots' meta tag

 

ss.jpgShopify is telling me again it's not done by them and I understand they blocking tags for being indexing by Google, but they blocked the whole Blog section.

When you talking about this they are all blinde and silent with the same answer as always. Shopify doing best what you need for your bla bla bla.

 

How can make my blog live to google search? Please, anybody... It's obvious they don't want to talk about this and they block it with purpose. Something between Google and Shopify is not right and they hiding it in robot.txt but is it possible to get this done or its best to make a custom page as blog article?

 

Please, anybody to write so suggestions or solutions.

 

Thank you,

Jovica

Furqan
Tourist
4 0 2

@S3ratin wrote:

I can't get my blog post to be indexed in google console. Google telling me its blocked by tag "noindex". So my question here is: Why robot.txt blocking all Blog posts with or without tags? 

I contact the Shopify team and they told me it's not by them. So, I install a fresh new Shopify theme and test live URL for blog, guess what...

 

Google Console

URL Inspection: https://pimpyworld.com/blogs/birthday-party-ideas

AvailabilityExcluded by ‘noindex’ tag

Indexing allowed?: No: 'noindex' detected in 'robots' meta tag

 

ss.jpgShopify is telling me again it's not done by them and I understand they blocking tags for being indexing by Google, but they blocked the whole Blog section.

When you talking about this they are all blinde and silent with the same answer as always. Shopify doing best what you need for your bla bla bla.

 

How can make my blog live to google search? Please, anybody... It's obvious they don't want to talk about this and they block it with purpose. Something between Google and Shopify is not right and they hiding it in robot.txt but is it possible to get this done or its best to make a custom page as blog article?

 

Please, anybody to write so suggestions or solutions.

 

Thank you,

Jovica


I can try to fix your issue email me on: crawlerknightim @ gmail.com

Josh_Uebergang
Shopify Expert
853 30 213

@S3ratin That is very unusual. I see no reasons why their robots.txt file would be doing that. Shopify's robots.txt should only be blocking blogs when there's a "+" symbol in the URL as per these three rules in the file:

Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*

I see you've renamed the URL of your blog. Is the issue fixed? Google can report on issues incorrectly at times.

Run Google Shopping ads? Get the free definitive guide to Google Shopping for Shopify (no optin required): https://www.digitaldarts.com.au/google-shopping
Louisdoremon
Tourist
5 0 1

Use these following code:

 

Your site url should not be added in robot.txt

Example:

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkout

Disallow: https://mynstrom.com/

Otherwise google will not index your site...

Remember before adding

nathanhuffhines
New Member
2 0 3

Is there any plan for shopify to allow for robots.txt editing in the near future? I don't think anybody's explained why this very simple feature doesn't exist. The fact that it doesn't exist makes me think this is intentional and you aren't planning to change it.

dikshaseo
New Member
1 0 0

Currently not found any solution for robots.txt. I also want to know about how can i edit robots.txt, I am early waiting to know solution

cannafleur
New Member
1 0 0

Hello there is a possibility to change the robots.txt for https://cannafleur.at/

Josh_Uebergang
Shopify Expert
853 30 213

Guys, you cannot edit the robots.txt file. Plain and simple. I have not heard any plans for this to change.

Run Google Shopping ads? Get the free definitive guide to Google Shopping for Shopify (no optin required): https://www.digitaldarts.com.au/google-shopping
mkuratomi0827
New Member
1 0 0

Were you able to resolve this in webmasters? If so, how did it effect your store? I know you can use URL Parameters. Would be cleaner to include in robots.txt

 

https://www.seerinteractive.com/labs/the-seo-starter-guide-to-google-search-console/using-url-parame...

LeFlo
Tourist
49 0 1

Hello,

 

I would like to prevent Google from crawling pages. I talk about crawling, not indexing (these pages are excluded, but I still have a problem).

 

These pages are 90% the variants created by the "suggested products" of the Venture theme. they should not be crawled, but they are.

 

For the remaining 10%, they are all newest-product type pages.

 

how to do it? do i have to go into the code to put a "disallow" command?

 

LeFlo
Tourist
49 0 1

Hi Don

 

I wish you can help me!

 

I would like to prevent Google from crawling pages. I have created more or less 250 pages, but I have almost 10 000 pages crawled.

 

These pages are 90% the variants created by the "suggested products" of the Venture theme. they should not be crawled, but they are.

It's pages like [URL product](+)?pr_prod_strat=collection_fallback&pr_rec_pid=4429241909294&pr_ref_pid=4405363277870&pr_seq=uniform

Always there are "?pr_prod_strat=collection_fallback&pr_rec_pid=" and "pr_ref_pid=" and "pr_seq=uniform"

 

For the remaining 10%, they are all newest-product type pages, or things like that.

 

It's been several weeks since I used the page you give  by adding lines of code.
I also used URLs parameters on the search console, and to finish always on the search I asked for the removal of all pages that bother me
but nothing to do I'm always getting more and more!!!

 

Maybe I've done mistakes ? here are the lines of code I added (theme liquid of venture):

 

{% if template contains 'search' %}

<meta name="robots" content="noindex">

{% endif %}

 

  {% if template contains 'all' %}

<meta name="robots" content="noindex">

{% endif %}

 

  {% if template = "product" %}

{% assign $productUrl = product.url | within: collection %}

{% if productUrl contains 'collections' %}

<meta name="robots" content="noindex">

{% endif %}

{% endif %}

 

  {% if handle contains 'rec_pid' %}

<meta name="robots" content="noindex">

{% endif %}

 

  {% if handle contains 'pr_prod_strat' %}

<meta name="robots" content="noindex">

{% endif %}

 

 {% if handle contains 'pr_prod_strat=collection' %}

<meta name="robots" content="noindex">

{% endif %}

 

   {% if handle contains 'conditions' %}

<meta name="robots" content="noindex">

{% endif %}

 

    {% if handle contains 'faq' %}

<meta name="robots" content="noindex">

{% endif %}

 

  {% if handle contains 'frontpage' %}

<meta name="robots" content="noindex">

{% endif %}

 

  {% if handle contains 'new' %}

<meta name="robots" content="noindex">

{% endif %}

 

  {% if handle contains 'regles-de-confidentialite' %}

<meta name="robots" content="noindex">

{% endif %}

 

   {% if handle contains 'product-reviews' %}

<meta name="robots" content="noindex">

{% endif %}

 

   {% if handle contains 'variant' %}

<meta name="robots" content="noindex">

{% endif %}

   

   {% if handle contains 'politique-de-remboursement' %}

<meta name="robots" content="noindex">

{% endif %}

 

  {% if handle contains 'frontpage' %}

<meta name="robots" content="noindex">

{% endif %}

sexshopamerica
New Member
4 0 0

Hi Don.

 

And what to do if you want google to be able to find your products and rich snippets?

Can you somewhere allow the robots to crawl these sites?

 

Kind regards

 

Kim

andrefreitas28
New Member
7 0 0

Hello guys, 

My robots.txt has instructions to some bots, such as: adsbot-google, Ahrefsbot, Majestic bot and even Pinterest. However, I use SEMrush to monitor my SEO Performance. How can I add some instructions to SEMRush bot?

My site audit is reporting some errors. In my opinion they're related to the "velocity" which the bot crawls the site. So, I need to give an instruction to SEMrush bot related to Crawl delay. How can I do that?

thank you

masoodjaved88
New Member
1 0 0

did any one found some solution for this ....? why this plat from is headache for SEO optimizer i m seeing google console is also not indexing urls properly,and every time when i request to index one a message pop up your url in google but not in sitemap,when i check my jackethunt.com/sitemap.xml site map url in this list,i did not found how to edit and xml site map and robots.txt for my website jackethunt.com if any one have succeed please share on this palt form  

Danpascu
New Member
2 0 1

Finally which is the solution for edit on robots.txt?

There is any analytic platform which can break this barrier?

David_Peón
New Member
1 0 1

It is incredible... Shopify is hiding and not giving us a solution about this

Too many pages are being crawled : all the search by url, which means a lot of duplicate content

Shopify, this is shameful, nobody is providing a solution

IndieEast
Excursionist
33 0 15

It seems as though Shopify doesn't really practice what they preach.  I was doing some audits of my website and accidentally came across what really surprised and upset me.  This affects everyone who owns a shopify store.   Shopify suppresses the pages of all Shopify store owners by adding by default, the word "disallow" into the robot.txt in the coding to many pages of our stores? This cannot be changed!  What this does is block Google's search engine from finding our pages, which brings the site down in rankings as well, but traffic to the site will be greatly affected. If Shopify really advocates for supporting independent business, which includes black business, what is the purpose of doing the very thing that helps our business succeed?  Looks like they are another billion dollar company that only speaks but does not practice what it preaches. One thing is for sure and that's that Shopify does not want business to succeed.  If you want to check your site then just add /robots.txt to the end of your website address to see the file.   Why would Shopify work against it's paying customers is such a way as to purposefully hinder everyone's success?

snowboardernh
Tourist
9 0 3

Don;

It is shameful that in 2020, that Shopify won't allow customers to decide what content is allowed and not allowed in robots.txt.  It seems we will have to look another platform and publicize this practice.  We have been live for a week and even google is now reporting that they demoted our site due to all of our files being disallowed.  Really ... collections and product pages disallowed!   Shopify needs to grow up up and be a big boy or customers will move on to the next big thing.   Already looking now.