Google Merchant Center Support say Shopify robots.txt has wrong format

[ohreallyhmmmm]

We've been trying to troubleshoot some issues with Google Merchant Center over the past few days (product mobile pages and desktop pages are not crawlable by Googlebot or Googlebot-Image). This problem results in products being disapproved on Google Merchant Center, and we cannot advertise our products. See example screenshot below:

Google Merchant Center have sent through an email saying that our robots.txt file is in the wrong format, this is why the product is disapproved.

Here is the email that they sent:

For reference, here are the links they reference in their email:

https://support.google.com/webmasters/answer/48620?hl=en

https://support.google.com/merchants/answer/6098185?hl=en

https://support.google.com/merchants/answer/6101165?hl=en

Has anyone else run into this issue?

I know we can't edit the robots.txt file; though, I hope that the internal team at Shopify would keep their file updated in line with Google Merchant Center's guidelines?

Is there anything that we can do on our side to fix this issue? (of mobile pages and desktop pages not being crawlable by Googlebot or Googlebot-Image)?

[EmmanuelFlossie]

If it's just a few random products, then this is no reason for concern. As it happens to all merchants.

This is caused by failed access attempts to the page. For example, bad DNS, slow website response, Google has issues, etc.

If pages are not crawlable and there are only a few, just let it be, in a few days, it will be active again.

[ohreallyhmmmm]

It doesn't seem to be active again after a few days. There are a consistent  ~10% of our products in a disapproved state, and this is affecting a substantial amount of volume for our shopping ads campaigns.

Because it is so consistent (and such a big part of our account) - it is very concerning.

Especially when Google Merchant Center explain that Shopify's robots.txt file is not in compliance with their standards.

Is there anything that we can do to solve this issue?

[EmmanuelFlossie]

I have never seen so many issues regarding crawling issues.

I then recommend you communicate with Shopify support regarding this.

[ohreallyhmmmm]

Thanks Emmanuel.

I have reached out to Shopify support but after 3 emails, and 2 chat conversations, all they have been able to tell me is that the robots.txt file cannot be edited (which I already knew).

I've just sent them an email explaining the issue of product pages not being crawlable, and asked for the issue to be investigated.

Is there any way you know of to get issues escalated to Tier 2 support at Shopify?

[Trevor]

Hello!

Just a quick update about the robot.txt file that was referenced in this post. As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here. 

If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.

[ohreallyhmmmm · Accepted solution]

In case anyone else runs into this issue (of your website not being crawlable by Google) - this is how we solved the issue:

We created the robots.txt template in our theme.

We added the following code:

User-agent: Googlebot
Disallow:

User-agent: Googlebot-image
Disallow:

in line with Google's documentation here and here.

This resolved our Merchant Center issues, and also resolved our Search Console issues (our domain was not being crawled by Google - in Google Search Console we had the warning "Indexed, though blocked by robots.txt").

[Healthunbox]

Hello Shopify Community,

I'm reaching out for some advice regarding an issue we're facing with our Shopify store in the Google Merchant Center. We have been notified to update our robots.txt file to allow "Googlebot" and "Googlebot-Image" access for crawling our site. However, upon reviewing our robots.txt file, it doesn't seem like we are blocking these user-agents.

 

Has anyone else encountered a similar issue? I'm trying to figure out if there's a default setting in Shopify that might be causing this problem or if I'm missing something in the robots.txt configuration. Any insights or experiences you can share would be extremely helpful.

Thank you in advance for your help!

[EmmanuelFlossie]

Try adding:

User-agent: Googlebot
Disallow: /admin/
Allow: /
User-agent: Googlebot-image
Disallow: /admin/
Allow: /
User-agent: Storebot-Google
Disallow: /admin/
Allow: /

[Healthunbox]

Hello EmmanuelFlossie,

Thank you for your valuable suggestion. We are keen to implement your recommendation and will inform you about the outcomes once applied.

[Healthunbox]

Hello EmmanuelFlossie,

We've received a suggestion from Google Ads to modify our robots.txt for a full-site crawl, as follows:

makefile
Copy code
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:
This differs from your previous advice. Your thoughts on this change would be valuable.

 

[EmmanuelFlossie]

Mine blocks the admin URL. If you don't want that, then use your own version and add Storebot-Google

[Healthunbox]

Thank you for your suggestion to block the admin URL. I agree with this approach for enhanced security and will also consider adding Storebot-Google as you recommended. I'll ensure these changes are implemented promptly.