Hi am working on client's website from last 1 month and today I have notised that the URL of the product pages are not craled yet..
Then I checked the robots.txt and the sitemap.xml
In robots.txt the product links were disallowed .. and in sitemap.xml no product link is listed.. Please see the attached sitemap.xml which I genrated using some online tool and it contains around 250 links that I need to be craweled by google..
Folllwing are the few links that are not yet crawled by google
Collection urls are not added to the sitemap. The base product urls are.
For example, I see your first product linked above in the sitemap:
<url> <loc> https://golfbasestore.com/products/colorblock-dry-zone-crew </loc> <lastmod>2018-02-15T00:56:19+00:00</lastmod> <changefreq>daily</changefreq> <image:image> <image:loc> https://cdn.shopify.com/s/files/1/2630/3606/products/DynamicImageHandler_78e78efb-de31-4ea8-a888-c82... </image:loc> <image:title>Colorblock Dry Zone Crew</image:title> </image:image> </url>
Yes, this is the issue...
You have added the image links in sitemap but the product links are not in sitemap so I want to list product page URL's in the Sitemap so tht google crawl the same.
Is it possible.. ?
The product links are in the sitemap. The product links that appear in collections do not.
So this appears:
You don't want those collection product urls in there since a product could (in theory) appear in a million collections. Google won't be thrilled about seeing the same product at many different urls.