Google Merchant Center Support say Shopify robots.txt has wrong format

Solved
ohreallyhmmmm
Tourist
12 2 4

We've been trying to troubleshoot some issues with Google Merchant Center over the past few days (product mobile pages and desktop pages are not crawlable by Googlebot or Googlebot-Image). This problem results in products being disapproved on Google Merchant Center, and we cannot advertise our products. See example screenshot below:

product disapproved 20210216.JPG

 

Google Merchant Center have sent through an email saying that our robots.txt file is in the wrong format, this is why the product is disapproved.

Here is the email that they sent:

 

Google Support robots.txt issues 20210217.JPG

For reference, here are the links they reference in their email:

https://support.google.com/webmasters/answer/48620?hl=en

https://support.google.com/merchants/answer/6098185?hl=en

https://support.google.com/merchants/answer/6101165?hl=en

 

Has anyone else run into this issue?

I know we can't edit the robots.txt file; though, I hope that the internal team at Shopify would keep their file updated in line with Google Merchant Center's guidelines?

Is there anything that we can do on our side to fix this issue? (of mobile pages and desktop pages not being crawlable by Googlebot or Googlebot-Image)?

Accepted Solution (1)
ohreallyhmmmm
Tourist
12 2 4

This is an accepted solution.

In case anyone else runs into this issue (of your website not being crawlable by Google) - this is how we solved the issue:

We created the robots.txt template in our theme.

 

We added the following code:

User-agent: Googlebot
Disallow:

User-agent: Googlebot-image
Disallow:

in line with Google's documentation here and here.

 

This resolved our Merchant Center issues, and also resolved our Search Console issues (our domain was not being crawled by Google - in Google Search Console we had the warning "Indexed, though blocked by robots.txt").

 

View solution in original post

Replies 12 (12)
EmmanuelFlossie
Shopify Partner
2695 195 648

If it's just a few random products, then this is no reason for concern. As it happens to all merchants.

This is caused by failed access attempts to the page. For example, bad DNS, slow website response, Google has issues, etc.

If pages are not crawlable and there are only a few, just let it be, in a few days, it will be active again.

I'm a Google Shopping Specialist and a Google Product Expert Education Award winner, a Google Ads Diamond Product Expert, and also a Google Premier Partner.
Need Data Feed or Google Shopping support? Get in touch.
ohreallyhmmmm
Tourist
12 2 4

It doesn't seem to be active again after a few days. There are a consistent  ~10% of our products in a disapproved state, and this is affecting a substantial amount of volume for our shopping ads campaigns.

disapproved products over time.JPG

Because it is so consistent (and such a big part of our account) - it is very concerning.

Especially when Google Merchant Center explain that Shopify's robots.txt file is not in compliance with their standards.

Is there anything that we can do to solve this issue?

EmmanuelFlossie
Shopify Partner
2695 195 648

I have never seen so many issues regarding crawling issues.

I then recommend you communicate with Shopify support regarding this.

I'm a Google Shopping Specialist and a Google Product Expert Education Award winner, a Google Ads Diamond Product Expert, and also a Google Premier Partner.
Need Data Feed or Google Shopping support? Get in touch.
ohreallyhmmmm
Tourist
12 2 4

Thanks Emmanuel.

I have reached out to Shopify support but after 3 emails, and 2 chat conversations, all they have been able to tell me is that the robots.txt file cannot be edited (which I already knew).

I've just sent them an email explaining the issue of product pages not being crawlable, and asked for the issue to be investigated.

Is there any way you know of to get issues escalated to Tier 2 support at Shopify?

Trevor
Community Moderator
Community Moderator
3350 455 948

Hello!

Just a quick update about the robot.txt file that was referenced in this post. As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here

If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.

Trevor | Community Moderator @ Shopify
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog

ohreallyhmmmm
Tourist
12 2 4

This is an accepted solution.

In case anyone else runs into this issue (of your website not being crawlable by Google) - this is how we solved the issue:

We created the robots.txt template in our theme.

 

We added the following code:

User-agent: Googlebot
Disallow:

User-agent: Googlebot-image
Disallow:

in line with Google's documentation here and here.

 

This resolved our Merchant Center issues, and also resolved our Search Console issues (our domain was not being crawled by Google - in Google Search Console we had the warning "Indexed, though blocked by robots.txt").

 

Healthunbox
New Member
5 0 0

Hello Shopify Community,

I'm reaching out for some advice regarding an issue we're facing with our Shopify store in the Google Merchant Center. We have been notified to update our robots.txt file to allow "Googlebot" and "Googlebot-Image" access for crawling our site. However, upon reviewing our robots.txt file, it doesn't seem like we are blocking these user-agents.

Screenshot_494.jpg

Has anyone else encountered a similar issue? I'm trying to figure out if there's a default setting in Shopify that might be causing this problem or if I'm missing something in the robots.txt configuration. Any insights or experiences you can share would be extremely helpful.

Thank you in advance for your help!

EmmanuelFlossie
Shopify Partner
2695 195 648

Try adding:

 

User-agent: Googlebot
Disallow: /admin/
Allow: /
User-agent: Googlebot-image
Disallow: /admin/
Allow: /
User-agent: Storebot-Google
Disallow: /admin/
Allow: /

 

I'm a Google Shopping Specialist and a Google Product Expert Education Award winner, a Google Ads Diamond Product Expert, and also a Google Premier Partner.
Need Data Feed or Google Shopping support? Get in touch.
Healthunbox
New Member
5 0 0

Hello EmmanuelFlossie,

Thank you for your valuable suggestion. We are keen to implement your recommendation and will inform you about the outcomes once applied.

Healthunbox
New Member
5 0 0

Hello EmmanuelFlossie,

We've received a suggestion from Google Ads to modify our robots.txt for a full-site crawl, as follows:

makefile
Copy code
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:
This differs from your previous advice. Your thoughts on this change would be valuable.

Screenshot_515.jpg

EmmanuelFlossie
Shopify Partner
2695 195 648

Mine blocks the admin URL. If you don't want that, then use your own version and add Storebot-Google

I'm a Google Shopping Specialist and a Google Product Expert Education Award winner, a Google Ads Diamond Product Expert, and also a Google Premier Partner.
Need Data Feed or Google Shopping support? Get in touch.
Healthunbox
New Member
5 0 0

Thank you for your suggestion to block the admin URL. I agree with this approach for enhanced security and will also consider adding Storebot-Google as you recommended. I'll ensure these changes are implemented promptly.