/recommendations/products in Google Search Console

[catalyst13]

I'm getting a warning for the site http://wanderingyogi.com.au/ that /recommendations/products is:

Blocked due to other 4xx issue

These pages aren't indexed or served on Google

I can't see any links to that URL, it's only appearing in the recommendation block as the data-url that I believe is used to call the API? Couple of questions.

1. How is google seeing this URL?

2. Is adding it to robots.txt as blocked the best way to handle it?

3. Any other suggestions to remove the issue?

[PaulNewton · Accepted solution]

Robots.txt is the way to go

You should first figure out on what pages google is finding that path and figure out what is the source.

Check any sitemaps as well not unlikely someone may have added it to a sitemap if you've had any custom SEO done.

Generally that url being crawled should not be happening as the route in most themes is only present javascript and /recommendations/products shouldn't ever be a direct link to that endpoint it would be like cart/clear showing up in results.

Unless now goggle bot et al. are checking html tag data attributes for a "url" keyword to infer link, or crawling ajax request endpoints which would be wild but not surprising. 

[Emm_]

I also have this issue. Google Search Console indicates that https://volupto.com.au/recommendations/products is "Blocked due to other 4xx issue".

One of the referring pages is listed as https://volupto.com.au/products/renoir-stone-bath.

Nothing in the page links to that URL, but I found this in the code:

@PaulNewton would you still recommend adding it to Robots.txt? Or setting up a 301?

Thanks in advance for your help!

Emmanuelle.