It is time to give us access to the robots.txt, and here is why!

Solved
Henry21
Tourist
5 0 1

I've seen many threads over the months and years about having access to robots.txt and many valid responses back and forth. However, I think it's time for Shopify to allow the users access to this file.

1. Individual Blog posts are disallowed under the current auto-generated file. Everything I've read online says that the best way to market your idea is content marketing, and there is no stronger content generator than your blog. Under the current robots.txt file, only the main blog page is not disallowed. Individual blog posts are not crawled by Google. How does this help our SEO?

2. Many organic search terms yield not only the homepages of retail websites, but also certain categories and departments. For example, if you Google mens shoes, the first page result is Nordstrom men's shoe department. The equivalent of that would be our collections pages. And yet, the robots.txt file disallows all collections pages besides the main one. In fact, the main collections page is the only one in which we have no control over! (myshopify.com/collections/all)

Many Shopify users have spent countless hours and hard earned money to customize their website, but the current robots.txt file punishes our SEO through "hiding" our content generated and our collections pages. It's time to give us access and allow us to take control of what we want Google to crawl.

Please post and let me know what you think, whether I am right or wrong.

0 Likes
Henry21
Tourist
5 0 1

Oh yes, I forgot to mention. If you ever change your url for your products or any page and Google has already crawled your site, you get a handful of 404 error pages. Even when trying to remove your urls using Google Webmasters and redirecting urls, without robots.txt file to block those dead links, your SEO will be severely impacted. 

Let me know what you guys think.

 

0 Likes
Stephen_Parkin1
Tourist
14 0 1

I got booted down the rankings and lost >60% of my traffic from the 4.0 update. Why?...because every  time I used a tag in an article or product it created a new page for that tag and even when I deleted the tag, Google still indexes the page.

So for instance, example.com/blogs/blog1/newpost is exactly the same as example.com/blogs/blog1/tagged/peanuts but even after deleting the tag, the 'peanuts' page is still indexed but the content from /blogs/blog1 is shown on its url.

I can't delete them in WMT's

Henry21
Tourist
5 0 1

Hey Stephen, thanks for your response. We have fallen into the same "trap", Google crawls every page you create pretty quickly. I've had deleted pages I created months ago still show up as dead links. 

Have you tried going into Google Webmasters and Removing those specific URLs? It takes a while for Google to update, but it works. Also I would try to use Shopify's Redirect feature under Navigation and redirect the 404 broken links to the original blog posts.

Let me know if this helps.

Stephen_Parkin1
Tourist
14 0 1

Hi Henry

i have tried deleting the pages in webmaster tools but when I create a removal request, the request vanishes a few hours later and am still left with the urls. I'll try using the redirect option for now and I'll let you know how I get on.

i appreciate the help mate. 

0 Likes
Stephen_Parkin1
Tourist
14 0 1

Hello again

 

So I have around 60 url's which are like this...

/blogs/blog1/tagged/whatever

...and they all show the same content. I have read somewhere that too many url redirects can be bad so I don't wanna try this.

To be honest, I don't wanna use tags on my site so I can get rid of these urls in webmaster tools i think but not before I add this code to the theme.liquid file...

{% if template == 'blog' %}{% if blog.url contains 'tagged' %}
    <meta name="robots" content="noindex,follow" />
  {% endif %}{% endif %}

Do you think this will work?

If so, now I just have to figure out how to do it for my product pages.

0 Likes
Henry21
Tourist
5 0 1

I am no expert, but 301 redirects should not affect your Google Pagerank at all: 

http://searchengineland.com/google-pagerank-dilution-through-a-301-redirect-is-a-myth-149656

The best way to solve your issue is to use 301 redirects on Shopify. I wouldn't touch the code on your website unless if you are or have access to an expert.

Of course, if we were able to access our robots.txt file, this would be more thorough...

0 Likes
Trevor
Community Moderator
Community Moderator
2577 338 524

This is an accepted solution.

Hello!

As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here

Due to the age of the topic, I will be locking this thread and marking it as resolved to help direct anyone that lands on this page. If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.

Trevor | Community Moderator @ Shopify 
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

0 Likes