Main issue: Google Merchant Center shows “robots.txt on the server is unreachable” for most products from a Shopify store (via Content API), persisting for ~2 days and worsening.
Context: robots.txt is a site file that tells crawlers what they can access. If unreachable, Google may pause or limit crawling/processing of product pages.
Analysis and perspectives:
Actions suggested:
Current status: No resolution yet. The discussion remains open. Key unanswered questions: What is causing robots.txt to be intermittently unreachable, and why are errors expanding across more products rather than occurring randomly.
Note: A screenshot is referenced to illustrate the growing number of affected items.
I’m getting this error on most of my products
open_in_newDeluxe Cocoa Butter Soap Making Kit - GROW AND MAKE info_outline
shopify_US_5293612353_16349352705
November 9, 2021 11:15:38 PM PST
robots.txt on the server is unreachable
Content API
Anyone have insights? It’s been happening for 2 days.
This can happen at random, and is not really related to any issues on what you have on the website, but for example a temporary DNS issue, or Google has connection issues.
It should auto resolve after a while.
You could however double check your robots.txt file, Shopify has recently allowed merchants to modify it, but from my experience, there is no need.
I don’t understand how it could be a random, since it’s getting worse every day.
This seems like something that is a problem with Shopify?
I recommend you contact Shopify to look into the issue. It should not be all your products. Only random products.
