Top of the morning!
We are having some problems with the web crawler from google. They are telling me they couldn’t see the whole website while we (and our customers) can see it clearly. Google is telling me it has something to do with the web crawler, and they told me to do the following:
Kindly update the robots.txt file on your web server to allow Google’s crawler to fetch the provided landing pages and images. In order for us to access your whole site, ensure that your robots.txt file allows both user-agents Googlebot-image (used for images) and Googlebot (used for web pages) to crawl your site. You can do this by changing your robots.txt file as follows:
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:
So I just copy/pasted the script to the robots file but now i’m wondering if have done this the right way.
If you scroll through the bottom at the following link you can see it’s included now. Can somebody tell me if this is the right way to do it?