Gaining Control of Robots.txt

Solved
Taylor14
Tourist
3 0 4

Looking for any and all ideas about how to deal with the fact that shopify auto generates the robots.txt file. I’d prefer not to get into the details around why we want to edit and have control over the file, but is there anyone out there that has found a successful work around?

Is there anything at all that we can do to either remove or get control of the robots.txt file?

Will shopify allow you to redirect the robots.txt file?

Can we prevent shopify from creating the file all together?

All the threads so far have not provided any solutions. 

Accepted Solution (1)

Accepted Solutions
Trevor
Community Moderator
Community Moderator
3210 432 739

This is an accepted solution.

Hello!

As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here

Due to the age of the topic, I will be locking this thread and marking it as resolved to help direct anyone that lands on this page. If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.

Trevor | Community Moderator @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

View solution in original post

Replies 144 (144)
Jason
Shopify Expert
10882 190 2178

No. If you are simply wanting to have a bit more granular control over the indexing you can adjust that with meta tags on a page by page basis.

Can't think of too many reasons why you would need to edit that file so you've got me curious.

★ I jump on these forums in my free time to help and share some insights. Not looking to be hired, and not looking for work. http://freakdesign.com.au ★
Taylor14
Tourist
3 0 4

Thanks for the reply Jason, but I should have mentioned that the meta tags on a page by page basis won't really do the trick. 

Out of the box the auto generated robots.txt file and xml sitemaps seem like a good idea and is probably a good solution for the vast majority of sites, but as soon as you start customizing the site and adding various applications you start to run into lots of reasons where you want the ability to edit or manipulate these files.

This is a significant short coming of Shopify in my mind and according to the threads in this forum and across the web I'm not alone in thinking that. At the very least, Shopify should allow customers the flexibility to prevent the auto generated files from being created and give customers the option to create their own robots.txt file and XML sitemaps.

 

Jason
Shopify Expert
10882 190 2178

Out of the box the auto generated robots.txt file and xml sitemaps seem like a good idea and is probably a good solution for the vast majority of sites, but as soon as you start customizing the site and adding various applications you start to run into lots of reasons where you want the ability to edit or manipulate these files.

I get the reason to keep it locked down to save the sanity of general users and support staff, but totally agree edit access would be great for those will the know how to manage it. Here's hoping for an 'enable advanced options' button in the near future that will open these things up.

★ I jump on these forums in my free time to help and share some insights. Not looking to be hired, and not looking for work. http://freakdesign.com.au ★
SachaG
Excursionist
29 0 4

You can actually create your own XML sitemap. Afterall, you decide which sitemap to submit to Webmaster Tools so feel free to create your own sitemap and submit that one.

Taylor14
Tourist
3 0 4

Sacha, thanks for you response. The robots.txt file is automatically generated, which includes a line that references the XML sitemap. So, although you could submit your own XML sitemap through Webmaster Tools, Google and the other engines will find and use the auto generated XML sitemap regardless of whether you officially submit it or not. 

Mauricio_Bonifa
Tourist
8 0 4

I also would like to have control over the subheadings on a google search of my online store, and I have the feeling that my sitemap and the robots have a massive importance on google's choice for those subheadings on google's search results. Am I correct? is there any suggestion about how can I influence the choice of those subheadings?

 

shoppit
Tourist
48 0 1

Agreed. I have no idea why we can't edit our own robots.txt file.. it should be as simple as if you'd like to override the default add a new robots.txt to assets or the like.. i have scenarios with an app which needs one version of a product hidden and the other visible.. I still want to make sure I hide those products from search engines as i don't want users coming in on those pages...

 

Can we request that something be done about this as a feature request...

Jason
Shopify Expert
10882 190 2178

Agreed. I have no idea why we can't edit our own robots.txt file..

You can surely see how it could create a much larger support problem for those that choose to edit it, but do a rubbish job of it. Perfect for you and I to have that control, but it needs to be a well balanced choice so Shopify doesn't end up wasting time supporting endless bad robot files instead of adding a far more important 'advanced user' features. 

For anyone that wants the feature (like me) be sure to do more than just post here - be sure to contact Shopify directly. The more voices the more likely you'll see the feature.

★ I jump on these forums in my free time to help and share some insights. Not looking to be hired, and not looking for work. http://freakdesign.com.au ★
shoppit
Tourist
48 0 1

But surely anyone who's adding robots.txt would be considered a more advanced user and shopify could basically say yes the feature exists but is not officially supported..

Jason
Shopify Expert
10882 190 2178

But surely anyone who's adding robots.txt would be considered a more advanced user

In a perfect world, but users can be crazy folk. I don't disagree though. Here's hoping for my 'enable advanced features' button that would let us tinker with a whole bunch of things at our own risk, and show a whole heap more data in the dashboard...

★ I jump on these forums in my free time to help and share some insights. Not looking to be hired, and not looking for work. http://freakdesign.com.au ★
David_Theodorou
New Member
5 0 0

As of right now, Google is on my ass wanting me to alter my robots.txt file.

I can't seem to get my site properly crawled because it seems to be disallowing it. 

What the smurf?

Here are the official responses from Google.

Hello,

Thank you for your email.
I have reviewed your account ID 9862459 and noticed that your items are under review as you currently have a robots.txt file that is preventing us from crawling the images on your site. In order for us to access these images, we'd like you to modify your robots.txt file to allow user-agent 'googlebot' to crawl your site.
Please add the following lines to the end of your robots.txt file:
User-agent: googlebot
Disallow:
User-agent: googlebot-image
Disallow:

Visit http://www.robotstxt.org/wc/norobots.html for a description of the robots exclusion protocol.
 

Hey mate

Got Shopify as my e-commerce system
They don't allow me to modify my robots.txt. 
 

 

Hello,

Thank you for your email.
Please know that you will have to get in touch with your service provider to make the required changes. As informed in the previous email, in order for us to access these images, we'd like you to modify your robots.txt file to allow user-agent 'googlebot' to crawl your site.

CMA
Excursionist
54 0 6

I also got the same email from Google this morning about altering the robots.txt file. My Google Products Listing is not indexing because they say they can't properly crawl my images.

What is the solution???

John_from_Londo
Shopify Partner
31 0 6

Hi all, 

I too have just got a problem with product feed to google merchant account. Google help have just told me I must update my Robots.txt file.

Below if from the Google team.

In the meantime, I would recommend that you modify your robots.txt file to ensure that it allows crawling of all your products and images: https://support.google.com/merchants/answer/160156?hl=en-GB

Help !!! At the moment we are at a standstill and we have 261 products unlisted on Google Shopping.

 

John
Shopify_Craig
Shopify Staff (Retired)
Shopify Staff (Retired)
359 0 27

Hi,

Editing your robots.txt file to add those lines won't do anything as we don't block Googlebot or Googlebot-image from any images today.  We do serve content from a CDN, exactly like Google suggests.

This sounds like a Google bug so we're escalating it with them.  Editing your robots.txt file won't do anything to solve this.

Craig

John_from_Londo
Shopify Partner
31 0 6

OK thanks for that. Since the last couple of days everything was working fine of the Product Feed to Merchant Account. We haven't changed anything. Can you tell me if anything has changed with the APP to upload our products to Google Shopping ?

We have the same products - nothing should have changed.

Help.... is it Google or you guys ?

Graph shows how in last 2 days everything is up for review.

 

 

John
CMA
Excursionist
54 0 6

Shopify Craig, thanks for the attention to this problem. Please keep us updated on the progress you are making with Google to resolve this issue. Like most of those who use Product Listings, not being able to upload will put a hurt on business. It is a pretty big deal.

Shopify_Craig
Shopify Staff (Retired)
Shopify Staff (Retired)
359 0 27

What day did this happen?

Can you try resubmitting right now and see if there's an issue?

 

Craig

CMA
Excursionist
54 0 6

My submission was made on Sunday.

Because I do have some products running, I am kind of afraid to try again until I have a real idea of resolution. I don't want to lose what I have up now, if you know what I mean.

If anybody has nothing running - all products under review - please try and let us know what happens???

John_from_Londo
Shopify Partner
31 0 6

Everything seemed to be working OK until 29th October for me.I then noticed no hits on my Adword Product Feed campaign yesterday. I then rang google they said to wait 24 hours. Rang again this morning and they are checking into it.

What I am trying to find out is how your Shopify App does its submissions. Do you have any details on this. Does it just update everyday or does it do a total new submission at a certain date ? Does anyone know ?

John