I am using Claude Sonnet LLM to help with SEO content. But it is blocked by robots.txt

Technical Q&A
1

Our site is zerowastesupply.co.uk and it has a pretty standard robots.txt file.
When I ask Claude to read a page it says it is blocked by robots.txt

I deleted robots.txt and it still cannot read it.

Another site - https://friendlysoap.co.uk/ has an identical robots.txt and it can be read perfectly fine by Claude

Claude could read my pages a few days ago

Has there been a change in CDN, Cloudflare, or settings that would block Claude’s web_fetch tool? I have been trying to figure this out all day.

THanks

2

Hi @RobbyC

Apart from checking CDN, Cloudflare, or the settings you mentioned in the question, you can check via two other aspects if they are okay currently:

  1. Check if your store has verified ownership in Google Search Console.

  2. Find out which rule in robots.txt is blocking your page and fix or remove the related rule.

For more details about how to operate them, you can check this guideline from the Search Console Help directly. Hope it can give you some reminders as well. Thank you!