Robots.txt Blocking Javascript Resources

Solved
Leanne_Orr
Tourist
4 0 1

I have to say I think it ridiculous that Shopify doesn't allow you to edit your own robots.txt file! Surely this is fundamental for any eCommerce platform?

Everywhere I've read thus far the reply has simply been: 'why would you want to edit your robots.txt file?' - despite there obviously being a strong need for it. Pretty much all the forum discussion threads are simply closed without resolution.

Google has recently specified that sites should not block their javascript resources from being crawled: https://developers.google.com/speed/docs/insights/BlockingJS

When I fetch pages on my site using Search Console, these are being fetched & rendered with errors because of the statements Shopify is using in the robots.txt file -

Basically: Javascript elements are being blocked by the statements you're forcing upon my store in the robots file - but I can't change it!

Can someone please look at this? And could someone please also explain why Shopify users still don't have the capacity to edit their own robots.txt files?

Jason
Shopify Expert
10360 158 2004

Basically: Javascript elements are being blocked by the statements you're forcing upon my store in the robots file - but I can't change it!

You've not understood what Render Block JS is. The google page you linked to is about how javascript is loaded into the page, and how that can effect load times. This has nothing to do with the robots file and is 100% a theme related concern.

Blocked Resources on the other hand are something different, and relate to files that Google wants access to so it can see the site in a way similar to the average user. More on that here. The first file in the pic you added is from a totally different website and relates to the improvely app. Your robots file has no control over that.

I'm curious, what's your reasoning for even wanting cart.js to be indexed? It's not meant to be read by humans (and I can't see why you'd want anything related to the cart being indexed). Cart.js looks a lot like this:

{"token":null,"note":null,"attributes":{},"total_price":0,"total_weight":0,"item_count":0,"items":[],"requires_shipping":false}

 

 

I jump on these forums to help and share some insights. Not looking to be hired, and not looking for work.

Don't hand out staff invites or give admin password to forum members unless absolutely needed. In most cases the help you need can be handled without that.


★ http://freakdesign.com.au ★
0 Likes
Leanne_Orr
Tourist
4 0 1

You're right, I didn't link to the right page. Like it says in your link:

Googlebot needs access to many resources on your page in order to render and index the page optimally. For example, JavaScript, CSS, and image files should be available to Googlebot so that it can see pages like an average user.

If a site's robots.txt file disallows crawling these resources, it can affect how well Google renders and indexes the page, which can affect the page's ranking in Google search.

Unblocking cart.js won't make it readable to humans, it will make it readable to Googlebot. There's no need to block the file - a JS file is never going to show up in any search results.

I work in SEO & for a number of our clients, allowing GBot access to all JS & CSS files has dramatically improved their organic performance,

Nevertheless I still can't understand why Shopify users wouldn't be able to access their own robots file?

0 Likes
Jason
Shopify Expert
10360 158 2004

I work in SEO & for a number of our clients, allowing GBot access to all JS & CSS files has dramatically improved their organic performance,

Than I would suggest you gather those improved performance stats and show them to Shopify. If you've got clear evidence that unlocking cart.js will improve performance I am sure they'd love to see it. Sounds cool to me.

I jump on these forums to help and share some insights. Not looking to be hired, and not looking for work.

Don't hand out staff invites or give admin password to forum members unless absolutely needed. In most cases the help you need can be handled without that.


★ http://freakdesign.com.au ★
0 Likes
HunkyBill
Shopify Expert
4509 46 492

I think Shopify does not let merchants edit that file precisely because they do not need to. Shopify provides a feed that points GoogleBot at all the shop resources, with no errors. That is the key here. No errors. Open it up to the merchant to edit, and they will certainly make errors, and suffer. 

As for the argument that since Google recommends you uncork all JS and CSS so the bot sees it as you see it, it would make sense in the context of a site that rigs up resource navigation with JS (and can anyone explain how CSS could affect a bot?)... I am pretty sure most Shopify sites are not rigged to use JS as the rendering engine of their Shop. That would defeat most of the advantages of using Shopify.

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes
Leanne_Orr
Tourist
4 0 1

That's ridiculous - I've already shown you that I do need to edit the file as it's blocking resources it shouldn't! Whether users might make errors or not - as a shop owner I would expect that functionality as standard. I don't think it's for Shopify to decide which elements of their site users are competent enough to edit & which they are not. It's my store & if I get it wrong then I pay the price!

Google needs access to CSS as it can use it to understand page content - position of content on the page, size [and therefore importance] of headings/text, hidden content etc.

I would be very surprised if there are any Shopify sites which don't utilise Javascript resources.

I'm not interested in getting into a debate over this - I'd just like to be able to edit my own robots file, or for someone at Shopify to review the robots template. My experience of Shopify so far has been extremely positive but I must admit to being frustrated at this.

0 Likes
HunkyBill
Shopify Expert
4509 46 492

Thanks for the laughs! 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes
Jason
Shopify Expert
10360 158 2004

The only file in the pic you posted that may be blocked by the robots file is cart.js. Everything else is unrelated.

I jump on these forums to help and share some insights. Not looking to be hired, and not looking for work.

Don't hand out staff invites or give admin password to forum members unless absolutely needed. In most cases the help you need can be handled without that.


★ http://freakdesign.com.au ★
0 Likes
Leanne_Orr
Tourist
4 0 1

Thanks for your contribution Bill. I hope your 3,000 posts on other threads have been more useful than you've been here.

0 Likes
HunkyBill
Shopify Expert
4509 46 492

Sorry. Nothing I, or anyone else in the public can do will illicit changes here. My contribution was only to point out that GoogleBot currently crawls Shopify fine as is. Regardless of JS or CSS. Yak shaving day!
 

Custom Shopify Apps built just for you! hunkybill@gmail.com http://www.resistorsoftware.com
0 Likes