Google Merchant Center Support say Shopify robots.txt has wrong format

ohreallyhmmmm
Tourist
5 0 1

We've been trying to troubleshoot some issues with Google Merchant Center over the past few days (product mobile pages and desktop pages are not crawlable by Googlebot or Googlebot-Image). This problem results in products being disapproved on Google Merchant Center, and we cannot advertise our products. See example screenshot below:

product disapproved 20210216.JPG

 

Google Merchant Center have sent through an email saying that our robots.txt file is in the wrong format, this is why the product is disapproved.

Here is the email that they sent:

 

Google Support robots.txt issues 20210217.JPG

For reference, here are the links they reference in their email:

https://support.google.com/webmasters/answer/48620?hl=en

https://support.google.com/merchants/answer/6098185?hl=en

https://support.google.com/merchants/answer/6101165?hl=en

 

Has anyone else run into this issue?

I know we can't edit the robots.txt file; though, I hope that the internal team at Shopify would keep their file updated in line with Google Merchant Center's guidelines?

Is there anything that we can do on our side to fix this issue? (of mobile pages and desktop pages not being crawlable by Googlebot or Googlebot-Image)?

0 Likes
EmmanuelFlossie
Shopify Partner
739 23 162

If it's just a few random products, then this is no reason for concern. As it happens to all merchants.

This is caused by failed access attempts to the page. For example, bad DNS, slow website response, Google has issues, etc.

If pages are not crawlable and there are only a few, just let it be, in a few days, it will be active again.

I'm a Google Shopping Specialist & a Google Ads Platinum Product Expert (Only a hand full of Platinum experts in the world!)
Need Premium Data Feed Management Support? Get in touch.
ohreallyhmmmm
Tourist
5 0 1

It doesn't seem to be active again after a few days. There are a consistent  ~10% of our products in a disapproved state, and this is affecting a substantial amount of volume for our shopping ads campaigns.

disapproved products over time.JPG

Because it is so consistent (and such a big part of our account) - it is very concerning.

Especially when Google Merchant Center explain that Shopify's robots.txt file is not in compliance with their standards.

Is there anything that we can do to solve this issue?

0 Likes
EmmanuelFlossie
Shopify Partner
739 23 162

I have never seen so many issues regarding crawling issues.

I then recommend you communicate with Shopify support regarding this.

I'm a Google Shopping Specialist & a Google Ads Platinum Product Expert (Only a hand full of Platinum experts in the world!)
Need Premium Data Feed Management Support? Get in touch.
ohreallyhmmmm
Tourist
5 0 1

Thanks Emmanuel.

I have reached out to Shopify support but after 3 emails, and 2 chat conversations, all they have been able to tell me is that the robots.txt file cannot be edited (which I already knew).

I've just sent them an email explaining the issue of product pages not being crawlable, and asked for the issue to be investigated.

Is there any way you know of to get issues escalated to Tier 2 support at Shopify?

0 Likes