Exclude pages from site map.xml

[phil_CM]

Is it possible to exclude pages from the site map? (Without using an App)

I don’t mean how to stop them being indexed as I have already implemented this but obviously now Google webmasters sees these pages as an issue as they are included in the site map. 

many thanks in advance 

[KieranR · Accepted solution]

Answer from @g33kgirl is correct but doing so via the Admin API can be technically challenging.

There are two easier ways to manage this other than directly interfacing with the API. 

1. Use a metafields app. I know off the top of my head that Accentuate Custom Fields has a built in setting to control the seo/hidden interger flag from a control panel
2. Use the Shopify bulk editor with a metafield URL string like: https://your-store.myshopify.com/admin/bulk?resource_name=Product&edit=metafields.seo.hidden
   
   • or select your Products, then fire up bulk editor, then add the bolded bit to the bulk edit URL
   • you'll need to set the value 1 for any products you want hidden

This field will do the following: 

• remove the product URLs from the sitemap.xml structure
• hide the products from internal site search
• add NOINDEX,NOFOLLOW tags to the pages
• leave the pages live

Can I ask why you're trying to do this? Might be a different/better approach to the same end goal.

[g33kgirl]

@phil_CM, this should help: https://shopify.dev/tutorials/manage-seo-data-with-admin-api#hide-a-resource-from-search-engines-and...

[phil_CM]

Click to expand...

I’m not sure how to implement admin API. It is specific products I want to exclude from the site map links 

[KieranR · Accepted solution]

Answer from @g33kgirl is correct but doing so via the Admin API can be technically challenging.

There are two easier ways to manage this other than directly interfacing with the API. 

1. Use a metafields app. I know off the top of my head that Accentuate Custom Fields has a built in setting to control the seo/hidden interger flag from a control panel
2. Use the Shopify bulk editor with a metafield URL string like: https://your-store.myshopify.com/admin/bulk?resource_name=Product&edit=metafields.seo.hidden
   
   • or select your Products, then fire up bulk editor, then add the bolded bit to the bulk edit URL
   • you'll need to set the value 1 for any products you want hidden

This field will do the following: 

• remove the product URLs from the sitemap.xml structure
• hide the products from internal site search
• add NOINDEX,NOFOLLOW tags to the pages
• leave the pages live

Can I ask why you're trying to do this? Might be a different/better approach to the same end goal.

[phil_CM]

I have products I have added noindex to. The reason for this is that they are “dummy products”. They are products of variants of the same product and simply redirect to a main product with that variant selected. So I added noindex to these as otherwise it would appear that I was trying to do something dodgy with the pages other than their real purpose which is to improve user experience.

now that I have these pages set with noindex, they currently still exist in the sitemap so Google sees them as being set to noindex as an error. It’s no great big deal but I would like to wrap up the logic by having them removed from the sitemap.

as I set the noindex using a tag on the product, can I run the admin api on the product template code based on this tag in the same way and auto set the product of accordingly? Is this how you run the admin api as a script in the store code?

[KieranR]

The Admin API can be accessed programmatically via REST/GraphQL requests. It's not too hard to do if you already know what that means.

The suggestion is to set a flag metafields.seo.hidden integer on a product (can't set it per product variant, has to be at the product level). You can do this by API or you can do it by the options (1/2) I gave which IMO are a bit simpler to do.

If Shopify sees metafields.seo.hidden == 1 on a product/collection/blog/blog-article resource then it will inject a NOINDEX,NOFOLLOW meta tag inside the {{ content_for_header }} include in the HTML of that page.

If you also have a custom noindex tag defined within the <head> tags then you'll end up with duplicate noindex tags. You can't access the Admin api via liquid templating alone, there is no way to do HTTPs requests (or any 'proper' server-side logic) within liquid templating.

An alternative approach here would be using custom canonicals, but need more context to know if that would make sense.

[phil_CM]

The only thing with doing this is that I currently set noindex, follow rather than no index no follow.

[phil_CM]

I’ll give this method a go 

1. Use the Shopify bulk editor with a metafield URL string like: https://your-store.myshopify.com/admin/bulk?resource_name=Product&edit=metafields.seo.hidden
   
   • or select your Products, then fire up bulk editor, then add the bolded bit to the bulk edit URL
   • you'll need to set the value 1 for any products you want hidden

I assume I don’t need the App to do this and can be done using the admin as it is?

[phil_CM]

Thinking about it I probably don’t want the products removed from the internal site search though. So I may have to give in a leave it where I have it.

thanks all for your advice

[phil_CM]

Hi Kieran,

I just got around to trying this and it appears I have the folowing result when I add the value 1 to hidden:

1. Removed from sitemap.

2. Removed from site search.

It doest appear to add the robots meta tag whichg suits me as I alreay have the custom tags set up and mine are noindex, follow as apposed to noindex, no follow as you suggested it might be.

Does this sound right? I have cleared cache and tried different browser and the source code only seems to have my own meta tag not the Shopify generated one.

Ive only done this to one product so far in case thee is some sot of delay in it adding this so that I can test.

[KieranR]

Not what I'd expect. Are you including content _for_header? I think that's
where I've seen noindex be included

[phil_CM]

yep, content for header is included right at the bottom of the header.

very odd but works for me lol though I did read yesterday a little trick I could have used that reads the content for header and I could specify something to strip for it so in my case if I was duplicating the noindex and putting “noindex, nofollow” in addition to my “noindex, follow” I would probably be able to remove it (hopefully). 

[KieranR]

Huh weird, would be interested to see an example URL and the metafield values set on the back, and the liquid, not working the way I've seen it before but anyway. Yep you can filter replace it if need be, that'll do it. But also Shopify could change their output at any time without notification, so filters could stop working. You should monitor things if you're doing that. 

[phil_CM]

Good advice. Happy to message you an example url if you like.

[sean4949 · Accepted solution]

I used your method, and it worked before, but now it does not allow me to use an integer anymore but true or false. This is no longer working for me except for the products I have already hidden.

https://bentleyhq.com/shopify/how-to-remove-shopify-pages-and-products-from-an-xml-sitemap/

I used the link above on a new website, and it worked well and was easier to use. I tried it on my old website, and it says the definition exists and I do not know how else to add it to the Product Admin. Thoughts?

Thank you, 

Sean

[KieranR]

Yeah nice! Shopify rebuilt the entire metafield admin interface so agree with @sean4949 that method from bentlyhq is now a much easier way.

[VisibleWebs]

Hey.
What if I want to exclude this product from the sitemap xml, but do not want "noindex, nofollow", because I have rel="canonical" to the page I want. What to do in this case?

[KieranR]

@VisibleWebs I'm sure there are other ways to get to the business end goal, depends on context. But this is a technical ask with narrow scope. You may be banging the wrong nail with the wrong hammer, so to speak. 

But I'll give it a go....

Alternative 1 - Is redirecting not an option? 

Alternative 2 - do nothing, sounds like a minor issue. May not be worth stressing over a technically perfect but overkill fix. 

Option 1 - create a static sitemap .xml structure by file upload (customized to exclude the URL in question) and redirect sitemap.xml to that. You'll need to maintain this sitemap manually or code up an automation to do this automatically.

Option 2 - filter the output of {{ content_for_header }} although this is a hacky, flakey setup. I could break the whole site suddenly if Shopify changes their output. 

{% capture content_for_header_fixed %}
{{ content_for_header }}
{% endcapture %}  

{% if product.metafields.seo.hidden == 1 or collection.metafields.seo.hidden == 1 %}
    {{ content_for_header_fixed | remove: '<meta name="robots" content="noindex,nofollow">' }}
{% else %}
{{ content_for_header }}
{% endif %}

Option 3 - use cloudflare o2o (enterprise ~$500/mo plan reqd) edge scripts to custom either the sitemap or the page HTML output