Exclude pages from site map.xml

Solved
phil_CM
Excursionist
81 1 8

Is it possible to exclude pages from the site map? (Without using an App)

I don’t mean how to stop them being indexed as I have already implemented this but obviously now Google webmasters sees these pages as an issue as they are included in the site map. 

many thanks in advance 

Accepted Solution (1)

Accepted Solutions
KieranR
Shopify Partner
310 25 104

This is an accepted solution.

Answer from @g33kgirl is correct but doing so via the Admin API can be technically challenging.

There are two easier ways to manage this other than directly interfacing with the API. 

  1. Use a metafields app. I know off the top of my head that Accentuate Custom Fields has a built in setting to control the seo/hidden interger flag from a control panel
  2. Use the Shopify bulk editor with a metafield URL string like: https://your-store.myshopify.com/admin/bulk?resource_name=Product&edit=metafields.seo.hidden
    • or select your Products, then fire up bulk editor, then add the bolded bit to the bulk edit URL
    • you'll need to set the value 1 for any products you want hidden

This field will do the following: 

  • remove the product URLs from the sitemap.xml structure
  • hide the products from internal site search
  • add NOINDEX,NOFOLLOW tags to the pages
  • leave the pages live

Can I ask why you're trying to do this? Might be a different/better approach to the same end goal.

 

Full time Shopify SEO guy, based in NZ. Sometimes freelance outside the 9-5.

View solution in original post

Replies 13 (13)
g33kgirl
Shopify Partner
331 96 116

@phil_CM, this should help: https://shopify.dev/tutorials/manage-seo-data-with-admin-api#hide-a-resource-from-search-engines-and...

If you found my answer helpful, please LIKE and ACCEPT.
buymeacoffee.com/g33kgirl
If you're not comfortable with code, please modify code files at your own risk.
phil_CM
Excursionist
81 1 8
Click to expand...
 

I’m not sure how to implement admin API. It is specific products I want to exclude from the site map links 

KieranR
Shopify Partner
310 25 104

This is an accepted solution.

Answer from @g33kgirl is correct but doing so via the Admin API can be technically challenging.

There are two easier ways to manage this other than directly interfacing with the API. 

  1. Use a metafields app. I know off the top of my head that Accentuate Custom Fields has a built in setting to control the seo/hidden interger flag from a control panel
  2. Use the Shopify bulk editor with a metafield URL string like: https://your-store.myshopify.com/admin/bulk?resource_name=Product&edit=metafields.seo.hidden
    • or select your Products, then fire up bulk editor, then add the bolded bit to the bulk edit URL
    • you'll need to set the value 1 for any products you want hidden

This field will do the following: 

  • remove the product URLs from the sitemap.xml structure
  • hide the products from internal site search
  • add NOINDEX,NOFOLLOW tags to the pages
  • leave the pages live

Can I ask why you're trying to do this? Might be a different/better approach to the same end goal.

 

Full time Shopify SEO guy, based in NZ. Sometimes freelance outside the 9-5.
phil_CM
Excursionist
81 1 8

I have products I have added noindex to. The reason for this is that they are “dummy products”. They are products of variants of the same product and simply redirect to a main product with that variant selected. So I added noindex to these as otherwise it would appear that I was trying to do something dodgy with the pages other than their real purpose which is to improve user experience.

now that I have these pages set with noindex, they currently still exist in the sitemap so Google sees them as being set to noindex as an error. It’s no great big deal but I would like to wrap up the logic by having them removed from the sitemap.

as I set the noindex using a tag on the product, can I run the admin api on the product template code based on this tag in the same way and auto set the product of accordingly? Is this how you run the admin api as a script in the store code?

KieranR
Shopify Partner
310 25 104

The Admin API can be accessed programmatically via REST/GraphQL requests. It's not too hard to do if you already know what that means.

The suggestion is to set a flag metafields.seo.hidden integer on a product (can't set it per product variant, has to be at the product level). You can do this by API or you can do it by the options (1/2) I gave which IMO are a bit simpler to do.

If Shopify sees metafields.seo.hidden == 1 on a product/collection/blog/blog-article resource then it will inject a NOINDEX,NOFOLLOW meta tag inside the {{ content_for_header }} include in the HTML of that page.

If you also have a custom noindex tag defined within the <head> tags then you'll end up with duplicate noindex tags. You can't access the Admin api via liquid templating alone, there is no way to do HTTPs requests (or any 'proper' server-side logic) within liquid templating.

An alternative approach here would be using custom canonicals, but need more context to know if that would make sense.

 

 

Full time Shopify SEO guy, based in NZ. Sometimes freelance outside the 9-5.
phil_CM
Excursionist
81 1 8

I’ll give this method a go 

  1. Use the Shopify bulk editor with a metafield URL string like: https://your-store.myshopify.com/admin/bulk?resource_name=Product&edit=metafields.seo.hidden
    • or select your Products, then fire up bulk editor, then add the bolded bit to the bulk edit URL
    • you'll need to set the value 1 for any products you want hidden

I assume I don’t need the App to do this and can be done using the admin as it is?

phil_CM
Excursionist
81 1 8

The only thing with doing this is that I currently set noindex, follow rather than no index no follow.

 

phil_CM
Excursionist
81 1 8

Thinking about it I probably don’t want the products removed from the internal site search though. So I may have to give in a leave it where I have it.

 

thanks all for your advice

phil_CM
Excursionist
81 1 8

Hi Kieran,

 

I just got around to trying this and it appears I have the folowing result when I add the value 1 to hidden:

1. Removed from sitemap.

2. Removed from site search.

 

It doest appear to add the robots meta tag whichg suits me as I alreay have the custom tags set up and mine are noindex, follow as apposed to noindex, no follow as you suggested it might be.

 

Does this sound right? I have cleared cache and tried different browser and the source code only seems to have my own meta tag not the Shopify generated one.

 

Ive only done this to one product so far in case thee is some sot of delay in it adding this so that I can test.

KieranR
Shopify Partner
310 25 104
Not what I'd expect. Are you including content _for_header? I think that's
where I've seen noindex be included
Full time Shopify SEO guy, based in NZ. Sometimes freelance outside the 9-5.
phil_CM
Excursionist
81 1 8

yep, content for header is included right at the bottom of the header.

very odd but works for me lol though I did read yesterday a little trick I could have used that reads the content for header and I could specify something to strip for it so in my case if I was duplicating the noindex and putting “noindex, nofollow” in addition to my “noindex, follow” I would probably be able to remove it (hopefully). 

KieranR
Shopify Partner
310 25 104

Huh weird, would be interested to see an example URL and the metafield values set on the back, and the liquid, not working the way I've seen it before but anyway. Yep you can filter replace it if need be, that'll do it. But also Shopify could change their output at any time without notification, so filters could stop working. You should monitor things if you're doing that. 

Full time Shopify SEO guy, based in NZ. Sometimes freelance outside the 9-5.
phil_CM
Excursionist
81 1 8

Good advice. Happy to message you an example url if you like.