I need to edit robots.txt file because, the "https://www.jeremy-hoye.co.uk/collections/unique-engagement-rings" page not getting link juice value. How to modify robots file?
Solved! Go to the solution
Don's reply is unbelievable. Don't listen to him, this guy has literally 0 knowledge when it comes to optimizing for SEO. A sitemap tells Google which pages are important your site. Believe it or not it's very expensive to for Google to crawl every page on the www, so a sitemap helps when it comes to budget crawling with 1000 page websites. A sitemap DOES NOT STOP Google from crawling pages on your site. Eventually, it will still crawl and index pages which you don't want Google to see.
Still an issue, still no solutions or suggestions, still nothing but confrontational responses from Shopify experts, when there is any response at all of course.
This is my website, my half pages have been crawl but not yet indexing. I don't know what is the problem. I tried everything
Submit my website to search engine submission sites
This problem coming since 1+ month
I felt compelled to add an update to this post because it backs up a post I have previously submitted to this thread.
For clarity, I work in ecommerce and specialise in commercial optimisation and growth strategies . I have been an ecommerce consultant for over 25 years and support the delivery of an estimated £100m per annum across a number of verticals. In case you hadn't realised, I think I know a bit about SEO too.
What I am about to share will astound many reading this post. It acutely demonstrates Shopify's ignorance, total lack of understanding and downright negligence when it comes to Advanced SEO provision on their platform.
Yes, I appreciate that is a damning indictment of Shopify, one which may even get this post removed and me banned but as I am about to explain, it's completely justified.
The information I am going to share to back up why Shopify are deserving of such criticism is provided by some little known company that none of us of have probably heard of and should clearly not be taking any advice from when it comes to SEO. Yeah, you guessed right. Google!
Unsurprisingly, the information which highlights how very important robots.txt directives are is nestled in the "Advanced SEO" section of their Search Central Documentation.
Shockingly for a multi-billion dollar company, this is clearly a part of the Internet, Shopify have never properly familiarised themselves with. If they had done, they might have realised the significance of Google and SEO best practice.
So, let's scrutinise exactly what Google expect us to do with a robots.txt file and let's do so by breaking it down in their own words
Block crawling of URLs that shouldn't be indexed. Some pages might be important to users, but shouldn't appear in Search results. For example, infinite scrolling pages that duplicate information on linked pages, or differently sorted versions of the same page. If you can't consolidate them as described in the first bullet, block these unimportant (for search) pages using robots.txt or the URL Parameters tool (for duplicate content reached by URL parameters). Don't use noindex, as Google will still request, but then drop the page when it sees the noindex tag, wasting crawling time. Don't use robots.txt to temporarily free up crawl budget for other pages; use robots.txt to block pages or resources that you think that we shouldn't crawl at all. Google won't shift this freed-up crawl budget to other pages unless Google is already hitting your site's serving limit.
So, let's just step through that again and compare what Google expect us to do for good SEO and what Shopify are clearly advising in this thread which is also supported by the lack of vital SEO support on their platform.
So, here we have it. In a nutshell, utter incompetence on Shopify's part.
The link for reference is here
To be clear, I know this optimisation technique applies to all sites regardless of size. I have proven it to be the case.
It doesn't just work well on massive sites, it's just way more important and effective on them.
This, what we have here is corporate negligence. The lack of feature to edit robots.txt files for merchants is commercially damaging. It will be costing all such merchants lost optimisation, hindered SEO and search visibility and ultimately revenue. Probably lots of it.
By not allowing Google to efficiently utilise crawl budget as recommended above with properly targeted and optimised robots.txt files, Shopify is quite literally holding back the search performance and search visibility of potentially tens of thousands of merchant websites.
What's more scary is that by holding all of these sites back in this way, it's probably costing Shopify revenue by hindering the growth of such merchants. It's madness in my opinion.
If you're an investor in Shopify, your investment is simply not growing at the rate it could until this is fixed.
If you're a developer at Shopify, maybe flag this one to someone higher up the chain. It's kind of important to listen to Google.
If you're a merchant on Shopify, be aware that Shopify don't care what Google advise to make your website rank well.
If you're considering Shopify as a platform, maybe consider another.
If you advise on SEO at Shopify, maybe quit and do something else you're good at.
One last point. For the record, I have actually proven that this is an issue with a real world test scenario. I have a Shopify website. The crawl rate is dreadful. If I could optimise robots.txt as I would like and block some wasteful query parameters, the efficiency would shoot up and my site would be crawled way more efficiently, more frequently and would then rank better. The lack of efficiency in crawl budget is definitely costing me in hindered growth and revenue. It's taking way too long for Google to re-crawl important pages on my site. If Shopify want to reach out for more information and supporting data, I'd be happy to share this.
Rather than dismiss people on this thread and even what Google advocate. It's maybe time for Shopify to demonstrate that you care about merchant websites, optimisation and growth and fix the issue by making robots.txt accessible and editable for advanced users who quite literally need this feature to drive the growth of their websites on your platform.
I would love to hear a solution to this issue. You previously wrote to pay Shopify $50 for premium, but I'm sure that was pure sarcasm. They have no right to block the edits because this goes against what Google says to do. If I could leave this platform I would. Anyways, you also wrote to DM you so please email me at email@example.com. I would really appreciate to know how.
Well looks like you can now - https://shopify.dev/tutorials/customize-theme-customize-robots-txt-liquid
The feature is rolling out, should have generally availability by monday, so far for all plans.
⚠️Notice: This is an unsupported customization . If you break something you're on your own Shopify support will not help you and will have to pay someone to fix it.
Many of the existing directives are there to protect you , be 1000% sure then check again that you understand the syntax of robots.txt before you remove a directive based on a suggestion, especially suggestions that don't understand shopify's url structure and endpoints.
⚠️ Again this is an unsupported customization.
This is not like editing a product template that might mess up the layout display there can be real financial costs with a bad edit of robots.txt.liquid sometimes silently bleeding money.
So do not confuse something like
as a directive blocking collections as that is not what that syntax does, every character matters and the above single example prevents duplicate tag content.
*If the create template method is not working by next week you may need to manually add the file to the theme, also notice this the robots.txt.liquid is in one theme so switching themes may need the contents copied over.
If you know of other robots.txt threads on the forums that need to know let me know the url
Unless someone is an expert, I cannot imagine anyone editing this on their own. I initially hired someone when they told me it's not allowed, which they thought was strange, but clearly any edits done within our back end by us, has always been our responsibility so this is no different. We should be allowed to edit whatever we need to as we are paying for the ability to customize our store the way we need to.
We should be allowed to edit whatever we need to as we are paying for the ability to customize our store the way we need to.
Not sure I agree with this, Shopify is a SaaS platform, so you pay to license the software and for them to host and maintain the technical backend.
If you want control over every aspect of your site then a self hosted and maintained solution rather than SaaS.
Sure ok, host and maintain the backend, not control the backend and tell me what I can and can't do with it; what I can or can't change, especially as they are not taking any responsibility. Other ecomm sites allow editing of the robots.txt, so I don't see what the issue. Telling everyone after the fact that this is an SaaS platform doesn't help. They should have mentioned it up front to say, 'btw, you won't be able to make all the edits you want to your back end because we don't allow it'. But now they do, so the fact that it's an SaaS platform doesn't seem to be an issue anymore.
Shopify don’t refer to themselves as SaaS explicitly but it is and always has been software as a service.
like anything where you lease, rent or license something, as opposed to building yourself or owning something. There will be restrictions as to what you can and can’t do. Just like leasing a retail location.
At least with a lease or license agreement, you have the fine print. Is this written as a part of the Shopify agreement somewhere that I missed? Maybe you can point out where? I'm seriously asking.
Wow, quite heated for the ability or lack of to edit a robots.txt file ha
Just look at the features in each level. The $80 (?) plan has "Ability to edits robots.txt" - so it's missing from the bottom plan.
Would you love to unleash the unbridled power of the Google Shopping Channel into your sho...By Gabe Jan 6, 2023
How can you turn a hobby into a career? That’s what Emmanuel did while working as a wa...By Skye Dec 30, 2022