How can I edit robots.txt

Solved
House_Of_Hoye
New Member
1 0 1

I need to edit robots.txt file because, the "https://www.jeremy-hoye.co.uk/collections/unique-engagement-rings"  page not getting link juice value.  How to modify robots file? 

 

Replies 92 (92)
ColesBalloons
Excursionist
18 1 2

Don | Social Care @ Shopify

you need to check your resources because you can not edit the robot.txt file so not sure how you can add to it ?

 

Jordo-Website
New Member
2 0 3

Yes this is increasingly frustrating for me as well. Every week I get emails from search console telling me my site has errors due to not having access to the robots.txt file so I can get my dang product pages indexed. Why is this even a thing???

IndieEast
Excursionist
33 0 15

Yes, there is no way to delete it, but there are tutorials in the Shopify Community on how to add to it.

IndieEast
Excursionist
33 0 15

Exactly!  So your pages will NEVER be found by Google crawling, which means customers won't find it either.  I get that some pages need this, but I think it should be up to the store owners rather than Shopify declaring it's for our own good why they do this.  If we created our own pages using a developer, we wouldn't have this issue.

PaulNewton
Shopify Partner
4184 302 886

@Jordo-Website wrote:

Yes this is increasingly frustrating for me as well. Every week I get emails from search console telling me my site has errors due to not having access to the robots.txt file so I can get my dang product pages indexed.


/products/ are not in the general robots.txt, not being index is more likely due to bad meta tags or lack of canonical tags in the theme.

Why is this even a thing???

Because most people have massive misconceptions about robots.txt moreso for it's syntax , and would do a lot of damage to themselves which is not a good policy for a SASS platform to have.

If there is an SEO problem seek help in the marketing forum by describing the actual problem https://community.shopify.com/c/Ecommerce-Marketing/bd-p/ecommerce-marketing

 

Save time seeking solutions - Learn To Ask Questions The Smart Way

Answers powered by coffee Buy Paul a Coffee for more answers or donate to eff.org
Problem Solved? ✔Accept and Like solutions to help future merchants
Confused? Busy? Buy a custom solution paull.newton+shopifyforum@gmail.com
PaulNewton
Shopify Partner
4184 302 886

@IndieEast wrote:

Exactly!  So your pages will NEVER be found by Google crawling, which means customers won't find it either. 


If it was true how would customers ever get to /account  or even /checkout? Those endpoints are in the robots.txt.

Actual relevant landing points like the homepage aren't in the robots text and get indexed just fine as to products unless a theme is setup to not do so.

So obviously that misconception is 100% not true, customers can easily get to any links built into a theme structure and navigation specifically not meant for bots to access.

 

This is an example misconception of why editing robots.txt isn't allowed.

If merchants could edit it they would delete|add things they have misconceptions about and never give it another thought until they were out of business because they ran their SEO into the ground editing files without the expertise to realize they don't know what they are doing and the understanding of shopifys url patterns.

 

Save time seeking solutions - Learn To Ask Questions The Smart Way

Answers powered by coffee Buy Paul a Coffee for more answers or donate to eff.org
Problem Solved? ✔Accept and Like solutions to help future merchants
Confused? Busy? Buy a custom solution paull.newton+shopifyforum@gmail.com
IndieEast
Excursionist
33 0 15

As I wrote, some pages do need the robots.txt, but let the shop owner decide.  You think that every single person that uses a web developer to create their own website ends up running it into the ground?  Really?!  So we are all being saved here, lol.  This isn't 1984 where big brother enforces what they think is best for everybody.  There should be a way to edit the robots.txt, IF the site owner wants to do it.  If they run their store into the ground, so be it, lesson learned, but there should be a way to edit this and some information provided by Shopify on the pitfalls instead of the complete inability to do so and no information, which causes misunderstanding.  We do pay for this, right?

TungDao
Shopify Partner
257 10 35

Agree, there's no reasons Shopify should stop users from editing robots.txt file. It's not rocket science.

Avada is an e-commerce solution provider based in the heart of Vietnam where the best developers unite. Founded in 2017 by Mageplaza, Avada is responsible to complete the mission to help 1 million online businesses grow revenues.
ColesBalloons
Excursionist
18 1 2

Exactly as this file is needed for website crawling so why so many restrictions.. And so annoying that they don't allow the policies in the report which you would is something that is is needed due to compliance and legalities. 

PaulNewton
Shopify Partner
4184 302 886

@IndieEast wrote:

We do pay for this, right?


Merchants pay for use of the platform services and that platform's rules, and one of the rules is merchants do not edit robots.txt.

 


@IndieEast wrote:

,there should be a way to edit this 

This is not likely to change, meta tags, canonical linking and properly structuring content suffice , those are the things to focus on.

The only way around this currently is to setup your own servers instead of using the locked down online-store sales channel hosted by shopify.

Configure you own domain rules, a hosting server, and basically embed shopify using it as then ecommerce backend and payment processor.

Then you can edit robots.txt however desired, you can even make whatever url structure is wanted, and all you have to do is support any technical issues by yourself.

Save time seeking solutions - Learn To Ask Questions The Smart Way

Answers powered by coffee Buy Paul a Coffee for more answers or donate to eff.org
Problem Solved? ✔Accept and Like solutions to help future merchants
Confused? Busy? Buy a custom solution paull.newton+shopifyforum@gmail.com
PaulNewton
Shopify Partner
4184 302 886

@TungDao wrote:

Agree, there's no reasons Shopify should stop users from editing robots.txt file. It's not rocket science.


Just because a system seems trivial to you does not mean a platform should in anyway expose it to millions of merchants.

I doesn't have to be rocket science to still be misunderstood, misconfigured, abused and endlessly being a technical support burden.

Have you forgotten Shopify is a platform? Some features have to be locked down by default or outright policy.

 

Save time seeking solutions - Learn To Ask Questions The Smart Way

Answers powered by coffee Buy Paul a Coffee for more answers or donate to eff.org
Problem Solved? ✔Accept and Like solutions to help future merchants
Confused? Busy? Buy a custom solution paull.newton+shopifyforum@gmail.com
IndieEast
Excursionist
33 0 15

Yawn.  Thanks for your detailed information of all I would need to do, but I can easily use another platform that doesn't have this restriction.  Next time I will do that.  

PaulNewton
Shopify Partner
4184 302 886

@ColesBalloons wrote:

Exactly as this file is needed for website crawling so why so many restrictions..

Because shopify is a hosted ecommerce platform and the online-sales channel has it's own prescribed url structure with many caveats that novices would easily misconfigure.

Why would a platform support a feature that can dangerously impact stores with the slightest typo, now scale that problem to millions of stores.


And so annoying that they don't allow the policies in the report which you would is something that is is needed due to compliance and legalities. 

Could you elaborate on what you mean by this "report"? Are you talking about policy pages?

Those endpoints can contain pregenerated boilerplate content they should not be indexed to prevent confusion and duplicate content penalties.

Save time seeking solutions - Learn To Ask Questions The Smart Way

Answers powered by coffee Buy Paul a Coffee for more answers or donate to eff.org
Problem Solved? ✔Accept and Like solutions to help future merchants
Confused? Busy? Buy a custom solution paull.newton+shopifyforum@gmail.com
freesko
New Member
1 0 0

This is stupid, you need to allow advanced users to change the file!

PaulNewton
Shopify Partner
4184 302 886

If you do not understand and appreciate why it's off limits, or the know the workarounds already,  then your not an advanced user.

Use metatags and canonicals as prescribe for normal users.

https://help.shopify.com/en/manual/promoting-marketing/seo/seo-overview

https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines

 

 

Save time seeking solutions - Learn To Ask Questions The Smart Way

Answers powered by coffee Buy Paul a Coffee for more answers or donate to eff.org
Problem Solved? ✔Accept and Like solutions to help future merchants
Confused? Busy? Buy a custom solution paull.newton+shopifyforum@gmail.com
nathanhuffhines
New Member
2 0 3

"If you do not understand and appreciate why it's off limits, or the know the workarounds already, then your not an advanced user."

Really? LOL. OK, what's the workaround for blocking pages from being crawled altogether? I'm worried about crawl budget. Do you know why that matters?

I'm not worried about pages being indexed. I'm not worried about pages being followed. I know I can deal with that with meta robots. But the problem is I don't want certain pages to be crawled in the first place. How would I go about blocking all pages (without having the ridiculous task of setting up my own servers) that end in /all.html or some pages with this parameter prefix ?brand=, etc.?

PaulNewton
Shopify Partner
4184 302 886

nofollow attribute on the relevant links is the quickest internal method, or a custom sitemap.

Crawl budget is primarily about not impacting servers with excess connections,  then index popularity over stale content.

Shopify can handle the connections.

 

It takes a lot of content for crawl budget to start being an actual issue, and then when done how long it took to get around to a page is irrelevant to ranking.

"improve" the crawl budget with optimized content(slow pages == slow UX == slow crawls), fix cloaked content(unclosable popups) , deduping, etc.

Save time seeking solutions - Learn To Ask Questions The Smart Way

Answers powered by coffee Buy Paul a Coffee for more answers or donate to eff.org
Problem Solved? ✔Accept and Like solutions to help future merchants
Confused? Busy? Buy a custom solution paull.newton+shopifyforum@gmail.com
RexMJ
New Member
1 0 1

If I may say so, no matter what the subject matter - There are some very ignorant, high and mighty, I am better than you answers coming from the Shopify experts here.  Get off your high horses and treat people in the way that is expected from a help forum instead of just saying "You can't do it because you're stupid and will f*** it up!!"

Almokinsgov
New Member
1 0 0

@PaulNewton wrote:

Shopify can handle the connections

Ha, don't think so, some of us are system admins and like the option to edit the necessary files. I shouldn't have to shove a whole bunch of code into pages just to add to the darn file. Plus their cereal box servers seem to crash on any major launch 🤷🏽‍♂️.

 

MrCheers
New Member
2 0 5

I'm quite shocked at this thread and the lack of this feature.

I can't see that anybody so far has presented a legitimate technical reason why robots.txt is not editable for those users who have confidence in editing this file for legitimate and valid technical SEO reasons. To suggest it is platform or policy based surely has to be a red herring. It's perfectly viable to allow merchants the option to edit this file with a huge disclaimer before they click the 'unlock your robots.txt editor' button. People just need to be told that it is for advanced users and if they break something, it's entirely their own fault. Shopify shouldn't be any more responsible for my robots.txt file as it is compliance with local tax laws or website terms and conditions. Whilst, I accept that some people may naively tinker and possibly remove their entire site from Google, this is surely their perogative tnd simply about clear communication and suitable presentation of warnings from Shopify.

2. To suggest that there is a suitable workaround here https://help.shopify.com/en/manual/promoting-marketing/seo/hide-a-page-from-search-engines as an alternative to powerful robots.txt directives that might, for example, target masses of legacy URLs containing query strings demonstrates a lack of understanding of advanced level SEO. There are a number of scenarios where directives in robots.txt can support good SEO.

3. Protection of crawl budget is also important and very often best deployed via robots.txt. It's entirely legitimate for SEO experts to want to steer Google crawlers to focuses their attention on pages which they want crawled rather than having Google scour potentially tens of thousands of pages unnecessarily. Crawl budget has in some scenarios where sites are enormous no relationship at all with the capability of servers at either end. With massive websites, Google will very often only crawl or recrawl a certain amount of pages per day. It can therefore take many months for Google to recrawl some pages on huge sites. I know this because I've seen it first hand. In this scenario, if Google is crawling or recrawling pages that we no longer need it to, this can seriously slow down the crawling of pages we actually do want indexed or regularly recrawled. Being able to clearly direct Google crawlers with advanced level directives via robots.txt helps to keep crawling efficient and targeted.

Such housekeeping is good practice. It is appreciated by Google and it's simply good SEO.

Yes, some of this is advanced and yes it is required mostly for edge cases but it certainly doesn't mean it's unimportant or absolutely necessary in some cases.

Shopify's current stance with robots.txt appears to be:-

- We don't want to support merchants in having highly flexible and robust SEO tools and optimisation options for every scenario.
- We don't really understand why this is so important to you because we don't fully understand SEO.
- We don't trust you and can't be bothered to fix the mess of people who don't know any better.

From what I've read here so far, it seems to be all three.

Would be nice if they could clarify though.