Did robots.txt change recently?

Brian67
Excursionist
30 0 2

Ok I know I can't change my file, but shopify must have changed something...

On 11/3 I had 4 pages blocked by robots (and I have always had 4 or less pages blocked going back to the beginning)

But on 11/10 (and going forward) i have 22,608 blocked by robots...

That's a MASSIVE change obviously.    Now my number of indexed pages has remained somewhat constant around 14,300 and I haven't noticed a significant change in traffic patterns.....

But can someone explain this (and if I should be concerned about it)?

Thanks

0 Likes
Jason
Shopify Expert
10407 163 2046

This is your robots file.

# we use Shopify as our ecommerce platform

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /account
Disallow: /collections/*+*
Disallow: /blogs/*+*
Sitemap: http://organizedprepper.com/sitemap.xml

User-agent: Nutch
Disallow: /

I would assume whatever blocks are in there are to reduce duplicate content smackdowns between the collection/product and product pages. Just an assumption of course.

I jump on these forums to help and share some insights. Not looking to be hired, and not looking for work.

Don't hand out staff invites or give admin password to forum members unless absolutely needed. In most cases the help you need can be handled without that.


★ http://freakdesign.com.au ★
0 Likes
Brian67
Excursionist
30 0 2


Thanks Jason...yes I knew what was in my robots file, but I'm wondering if shopify changed it recently given the massive jump in blocked pages.  

The real question is should I be concerned that suddenly so many pages are being blocked from google?

I'm not an expert in this area, but seems a bit concerning that the blogs and collections would be blocked.    Its been a long time since I looked at my robots file, but I don't think the disallow on collections and blogs used to be there.

0 Likes
Jason
Shopify Expert
10407 163 2046

I don't look at robots.txt much either but I don't recall seeing collections there either - hence my initial comment. Any page that is a sub of collections would have duplicate content from the main product page. A SEO monkey will know more about the logic here, but it does make some sense.

I jump on these forums to help and share some insights. Not looking to be hired, and not looking for work.

Don't hand out staff invites or give admin password to forum members unless absolutely needed. In most cases the help you need can be handled without that.


★ http://freakdesign.com.au ★
0 Likes
Lulu4
New Member
4 0 0

Hi Brian, I noticed the same problem for my site. There's a big jump in the number of blocked URLs in the last couple of days. I don't recall Collections and Blogs were in robots.txt. Does anyone else experience the same problem? Can any SEO expert shed some light here? Is this a Shopify issue? 

0 Likes
Patti
Explorer
161 0 15

I would like to know why collections is being blocked also.

http://www.aromatherapynaturals.com
0 Likes
Derek_Riccardo_
Tourist
13 0 1

I am also wondering if blogs should be blocked? I blog a lot so what does that mean? 

0 Likes
Chris_Miller2
New Member
14 0 0

I am having the same problem... has anybody had an answer for way we are getting blocked URLs from google? 

Thanks. Chris

0 Likes
Party_Fun_Box
Excursionist
82 0 10

I too have the same problem ... I don't understand what is going on ... I had 4 pages or so blocked by robots.txt (cart, admin) and now I have over 1000 blocked pages (blog, collection, etc) It happened on December 15th (according to Webmaster Tools).

I wish I were more knowledgeable on this subject ... but 1000 + blocked pages ... that can't be good can it??

Does anybody out there know why ?

0 Likes
Patti
Explorer
161 0 15

I don't understand why Shopify won't answer these questions. Seems strange.

http://www.aromatherapynaturals.com
0 Likes