We've been trying to troubleshoot some issues with Google Merchant Center over the past few days (product mobile pages and desktop pages are not crawlable by Googlebot or Googlebot-Image). This problem results in products being disapproved on Google Merchant Center, and we cannot advertise our products. See example screenshot below:
Google Merchant Center have sent through an email saying that our robots.txt file is in the wrong format, this is why the product is disapproved.
Here is the email that they sent:
For reference, here are the links they reference in their email:
Has anyone else run into this issue?
I know we can't edit the robots.txt file; though, I hope that the internal team at Shopify would keep their file updated in line with Google Merchant Center's guidelines?
Is there anything that we can do on our side to fix this issue? (of mobile pages and desktop pages not being crawlable by Googlebot or Googlebot-Image)?
If it's just a few random products, then this is no reason for concern. As it happens to all merchants.
This is caused by failed access attempts to the page. For example, bad DNS, slow website response, Google has issues, etc.
If pages are not crawlable and there are only a few, just let it be, in a few days, it will be active again.
It doesn't seem to be active again after a few days. There are a consistent ~10% of our products in a disapproved state, and this is affecting a substantial amount of volume for our shopping ads campaigns.
Because it is so consistent (and such a big part of our account) - it is very concerning.
Especially when Google Merchant Center explain that Shopify's robots.txt file is not in compliance with their standards.
Is there anything that we can do to solve this issue?
I have never seen so many issues regarding crawling issues.
I then recommend you communicate with Shopify support regarding this.
I have reached out to Shopify support but after 3 emails, and 2 chat conversations, all they have been able to tell me is that the robots.txt file cannot be edited (which I already knew).
I've just sent them an email explaining the issue of product pages not being crawlable, and asked for the issue to be investigated.
Is there any way you know of to get issues escalated to Tier 2 support at Shopify?