FROM CACHE - en_header

not able to Edit Sitemap and Robot.txt file

Reecha_Bhardwaj
New Member
3 0 0

https://golfbasestore.com/sitemap.xml

https://golfbasestore.com/robots.txt

Hi am working on client's website from last 1 month and today I have notised that the URL of the product pages are not craled yet.. 

Then I checked the robots.txt and the sitemap.xml

In robots.txt the product links were disallowed .. and in sitemap.xml no product link is listed.. Please see the attached sitemap.xml which I genrated using some online tool and it contains around 250 links that I need to be craweled by google..

Folllwing are the few links that are not yet crawled by google

https://golfbasestore.com/collections/t-shirt/products/colorblock-dry-zone-crew
https://golfbasestore.com/collections/t-shirt/products/ultra-cotton-t-shirt-6
https://golfbasestore.com/collections/golfbase/products/go1fbase-crewneck-pullover-sweatshirt
https://golfbasestore.com/collections/golfbase/products/golfbase-samsung-galaxy-s7-phone-case-3

Please help 

Replies 7 (7)
Jason
Shopify Expert
10918 193 2191

Collection urls are not added to the sitemap. The base product urls are.

For example, I see your first product linked above in the sitemap:

<url>
<loc>
https://golfbasestore.com/products/colorblock-dry-zone-crew
</loc>
<lastmod>2018-02-15T00:56:19+00:00</lastmod>
<changefreq>daily</changefreq>
<image:image>
<image:loc>
https://cdn.shopify.com/s/files/1/2630/3606/products/DynamicImageHandler_78e78efb-de31-4ea8-a888-c82...
</image:loc>
<image:title>Colorblock Dry Zone Crew</image:title>
</image:image>
</url>

 

★ I jump on these forums in my free time to help and share some insights. Not looking to be hired, and not looking for work. http://freakdesign.com.au ★
Reecha_Bhardwaj
New Member
3 0 0

Yes, this is the issue...

https://cdn.shopify.com/s/files/1/2630/3606/products/DynamicImageHandler_78e78efb-de31-4ea8-a888-c82...

You have added the image links in sitemap but the product links are not in sitemap so I want to list product page URL's in the Sitemap so tht google crawl the same.

https://golfbasestore.com/collections/t-shirt/products/colorblock-dry-zone-crew

Is it possible.. ?




 

Reecha_Bhardwaj
New Member
3 0 0

As I am doing SEO for this website so I want these links to be crawled by google..

Jason
Shopify Expert
10918 193 2191

The product links are in the sitemap. The product links that appear in collections do not.

So this appears:

https://golfbasestore.com/products/colorblock-dry-zone-crew

This won't:

https://golfbasestore.com/collections/t-shirt/products/colorblock-dry-zone-crew

You don't want those collection product urls in there since a product could (in theory) appear in a million collections. Google won't be thrilled about seeing the same product at many different urls.

★ I jump on these forums in my free time to help and share some insights. Not looking to be hired, and not looking for work. http://freakdesign.com.au ★
ozge-bny
New Member
1 0 0

Hello,

 

I would like to remove some pages from both the sitemap and robots. I have tried some apps but none of them work properly. Could you tell me how the sitemap and robots can be edited without an app? I also work with a web developer so, the solution can contain editing codes.

 

Thank you,

HarrySuyal
New Member
2 0 0

I face same issue my website XML sitemap <lastmod>2020-12-22T13:34:33-05:00</lastmod> do not update daily, either mention in sitemap change frequency is daily.

Trevor
Community Moderator
Community Moderator
3223 436 753

Hello!

As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here

Due to the age of the topic, I will be locking this thread. If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.

Trevor | Community Moderator @ Shopify (Out of Office until Sept 7th)
 - Was my reply helpful? Click Like to let me know! 
 - Was your question answered? Mark it as an Accepted Solution
 - To learn more visit the Shopify Help Center or the Shopify Blog