not able to Edit Sitemap and Robot.txt file

[Reecha_Bhardwaj]

https://golfbasestore.com/sitemap.xml

https://golfbasestore.com/robots.txt

Hi am working on client's website from last 1 month and today I have notised that the URL of the product pages are not craled yet.. 

Then I checked the robots.txt and the sitemap.xml

In robots.txt the product links were disallowed .. and in sitemap.xml no product link is listed.. Please see the attached sitemap.xml which I genrated using some online tool and it contains around 250 links that I need to be craweled by google..

Folllwing are the few links that are not yet crawled by google

https://golfbasestore.com/collections/t-shirt/products/colorblock-dry-zone-crew
https://golfbasestore.com/collections/t-shirt/products/ultra-cotton-t-shirt-6
https://golfbasestore.com/collections/golfbase/products/go1fbase-crewneck-pullover-sweatshirt
https://golfbasestore.com/collections/golfbase/products/golfbase-samsung-galaxy-s7-phone-case-3

Please help 

[Jason]

Collection urls are not added to the sitemap. The base product urls are.

For example, I see your first product linked above in the sitemap:

<url>
<loc>
https://golfbasestore.com/products/colorblock-dry-zone-crew
</loc>
<lastmod>2018-02-15T00:56:19+00:00</lastmod>
<changefreq>daily</changefreq>
<image:image>
<image:loc>
https://cdn.shopify.com/s/files/1/2630/3606/products/DynamicImageHandler_78e78efb-de31-4ea8-a888-c82...
</image:loc>
<image:title>Colorblock Dry Zone Crew</image:title>
</image:image>
</url>

[Reecha_Bhardwaj]

Yes, this is the issue...

https://cdn.shopify.com/s/files/1/2630/3606/products/DynamicImageHandler_78e78efb-de31-4ea8-a888-c82...

You have added the image links in sitemap but the product links are not in sitemap so I want to list product page URL's in the Sitemap so tht google crawl the same.

https://golfbasestore.com/collections/t-shirt/products/colorblock-dry-zone-crew

Is it possible.. ?

[Reecha_Bhardwaj]

As I am doing SEO for this website so I want these links to be crawled by google..

[Jason]

The product links are in the sitemap. The product links that appear in collections do not.

So this appears:

https://golfbasestore.com/products/colorblock-dry-zone-crew

This won't:

https://golfbasestore.com/collections/t-shirt/products/colorblock-dry-zone-crew

You don't want those collection product urls in there since a product could (in theory) appear in a million collections. Google won't be thrilled about seeing the same product at many different urls.

[ozge-bny]

Hello,

I would like to remove some pages from both the sitemap and robots. I have tried some apps but none of them work properly. Could you tell me how the sitemap and robots can be edited without an app? I also work with a web developer so, the solution can contain editing codes.

Thank you,

[HarrySuyal]

I face same issue my website XML sitemap <lastmod>2020-12-22T13:34:33-05:00</lastmod> do not update daily, either mention in sitemap change frequency is daily.

[Trevor]

Hello!

As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here. 

Due to the age of the topic, I will be locking this thread. If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.