My website does not allow google crawl

[tjieko]

Hi, I have been wondering why my Shopify products do not show up in the Google products section. I think I found out that it is because Google cannot crawl on my website. I am using GoDaddy for my website domain. I also cannot submit my sitemap to Google Search Console, which I think is because the Googlebot is not allowed to crawl. How do I allow Googlebot to crawl on my website?

I am also wondering why my website logo does not change. I changed my favicon brand icon a few weeks ago, and it still hasn't updated on Google.

Thanks already

My url is https://vouwfatbike.nl

This is my robots.txt file https://vouwfatbike.nl/robots.txt

[Yvonne_Yoast]

Hi @tjieko, 

I checked the source code of your website https://vouwfatbike.nl/ and I don't see the meta robots tags that will tell search engines if you would like to index or noindex your site in search results. 

The robots.txt settings come from Shopify and handles this. To allow Googlebot to crawl your website, you should check your robots.txt file to ensure that it is not blocking Googlebot. You can use the robots.txt tester to validate the file and make sure it allows Googlebot access. If there are any disallow rules for Googlebot, you should remove them.

If you want to edit your robots.txt file, you can edit the robots.txt liquid file by going to your Online Store admin page. From there, go to the theme section and select Actions -> Edit code. Find the template section and click Add new template. Click the dropdown and select robots.txt at the bottom. Click Create Template, and you can start editing. You can find more information on these helpful Shopify articles:

• https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt
• https://shopify.dev/themes/seo/robots-txt?shpxid=a330256c-0178-47DE-D15B-9A9EADD4F091

In addition, you can read more about optimizing Shopify for search in our post:Shopify SEO: the ultimate guide where you can see what a standard Shopify robots.txt file looks like.

Please keep in mind that there are certain parts of your website that are supposed to be blocked according to Google's guidelines (and Shopify's, which blocks them by default in the robots file). Therefore your entire website should not be caught by Googlebot. Still if there are parts in your robots.txt that you believe are wrongly blocked you can edit the robots.txt based on the Shopify resources I shared.

With that being said, our Yoast for Shopify app - >  Advanced settings allow you to decide what to show to search engines and, by extension, what shows up in the search results. For instance, you can hide the entire page or parts of it, such as the links or images. Moreover, you can instruct search engines not to show a snippet of the page or its cached version. 

Regarding the issue with submitting your sitemap to Google Search Console, ensure that your sitemap is correctly located and accessible. If Google Search Console reports that the sitemap could not be read, try removing any existing sitemaps and resubmitting just the main sitemap_index.xml.

As for the favicon not updating, it might take some time for changes to propagate across Google's cache. Ensure that your favicon is correctly linked in your site's HTML and try clearing your browser cache to see if the changes appear locally. If the issue persists, it might be worth checking if there are any caching plugins or settings on your site that could be affecting this.

I’m excited to let you know that the Yoast SEO App is now available on Shopify! This app can help you optimize your store’s SEO, improve its visibility and Rich Resuls. You can take advantage of a 14-day free trial to explore its features and see how it works for your store. 

[tjieko]

Hi @Ecom_Dev,

Thanks for your reply! I am not sure if the bot is blocked or not, I know that Googlebot is disallowed many things but I don't know which are supposed to be allowed and how to change them. I tried indexing my website through Google Search Console, but it said it can't index it. Could you help me out?

Thanks already!

[tjieko · Accepted solution]

Hi again,

Nevermind... turns out I was trying to index https://www.vouwfatbike.nl instead of https://vouwfatbike.nl. Silly mistake... Thanks for your time anyway. I hope my favicon updates soon!