Merchant Center Error - Page not crawlable due to robots.txt

dmontenegro
Visitor
1 0 0

Hi!

We have a lot of items disapproved on Google Merchant Center.

 

Screenshot 2024-05-31 at 3.06.44 pm.png

 

I saw the solution was paste this in the robot txt.file:

 

User-agent: Googlebot
Disallow:

User-agent: Googlebot-image
Disallow:

 

I did this:

Screenshot 2024-05-31 at 2.50.38 pm.png

 

I would like to know if someone has a similar problem and if it is okay if I paste the code. Otherwise, could someone provide me with more information about how to solve the issue?

 

Thanks heaps

Replies 2 (2)

saim007
Shopify Partner
535 68 89

@dmontenegro First of all you are using the wrong code in your robots.txt file it will stop the whole traffic on your site along with images.

 

First thing you have to find out in the Google Merchant Center which products are disapproved and why disapproved.

Second check the product is indexed or not in the search engine.

Third check the current robots.txt and find out which one is disallowed and which one should disallow and which one have to allow so moto is find the root cause and fix the issue.

Fourth point is if you can't able to fix then you can hire SEO and Shopify expert to fix these issues because it is longer process to find the cause and fix these all.

Saim | Shopify Partner Expert

Was I helpful?

Buy me a Coffee

🙂
Hire me   to unlock the full potential of your e-commerce store 🙂

Shanthi_Vutukur
Shopify Partner
11 2 4

Hi! ‘Page not crawlable due to robots.txt’ is a common error in Google Merchant Center. 

 

But the good part is that it can be resolved easily. 

 

Here's how to fix it:

 

  1. First, you need to update your robots.txt file so that Googlebot and Googlebot-Image can crawl your side. 

 

As you shared yourself in the question, you have to make these changes to your robots.txt file:

 

User-agent: Googlebot

Disallow:

 

User-agent: Googlebot-image 

Disallow:

 

And yes it is okay to paste this code (the answer to your question.)

 

  1. Once you are done making/updating these changes, please submit a new product feed. Or, use the Content API to update your products in Google Merchant Center. 

 

Once you are done with this, as per Google, the issue will be resolved in a matter of a few days, and then your products will start appearing again in shopping ads and listings. 

 

Tip: In case you want to speed up the process, you can increase Google's crawl rate.

 

Hope this helps!

Shanthi Vutukur | Content Writer@ AdNabu


 - Was my reply helpful? Click Like to let me know! 


 - Was your question answered? Mark it as an Accepted Solution


 - Try our Nabu for Google Shopping Feed App from Shopify App Store.