Hey,
i am trying to crawl my website on screaming frog, but whenever i try to start it, an issue comes up:
-response codes: Internal blocked by Robots.txt
Does anyone know how i can fix this?
thanks!
Hey,
i am trying to crawl my website on screaming frog, but whenever i try to start it, an issue comes up:
-response codes: Internal blocked by Robots.txt
Does anyone know how i can fix this?
thanks!