We were told by the developer of one of the most popular sitemap and noindex manager apps on Shopify that Shopify forces a NOINDEX, NOFOLLOW when you remove a URL from the sitemap via the API which is crazy... And I am trying to determine is this is indeed the case.
Yes, if you NOINDEX a page with a <meta name="robots" /> element, you should always remove it from the sitemap.xml as it will trigger errors with the search engines because it's ambiguous (the meta robots says don't index it while the URL's existence in the sitemap.xml says do index it). However, the converse is not necessarily true. Just because you remove a file from the sitemap.xml does NOT mean you always want it flagged NOINDEX.
But what is FAR worse than forcing a NOINDEX when you remove a URL from the sitemap.xml is to force a NOFOLLOW in the <meta name="robots" /> element. It's VERY detrimental to SEO and the ability of ALL URLs on the site to rank. It creates PageRank/link juice black holes in your site where PageRank/link juice flows into a page but cannot flow back out. This drain your site of link juice and drastically reduces all of your site's URLs' ability to rank.
Is there documentation for the Shopify API that shows ALL metafield keys, ALL values valid for each key, and what the values do as they relate to sitemap, site product search, and meta robots? I haven't seen much in the way of complete documentation, only a couple of examples here https://help.shopify.com/en/api/guides/updating-seo-data.
With regards to the seo "hidden" metafield mentioned in this guide, you're correct in that adding this metafield will add both NOINDEX and NOFOLLOW tag to the resource. This is currently the only metafield that allows configuration of SEO index behaviour.
Note that this method of adding the SEO tags through a metafield has been created to allow a quick and easy way to remove a particular product/resource from search engine index and storefront search results. In cases where something has been published/indexed that a merchant wants removed, this metafield can be added to remove the resource from search engine and storefront search results. This should only be used on individual resources, and not on theme templates. If you're looking to add just the NOINDEX tag to particular a resource, this can be done on theme templates. In case you haven't already seen it, this guide shows how to add a liquid tag that will render a NOINDEX meta tag on the storefront for a particular resource.
Thanks for the reply JB. This is not something that would seem doable at the theme level. I need to selective choose particular collections, products, and pages that I don't want in the sitemap and/or site search and/or with noindex and/or noarchive.
Is there a way, using the API, that I could write an app that would provide functionality for users to independently control at the collection, page, and product level the ability to:
1) remove that URL from the sitemap
2) remove that URL from the on-site search
3) add NOINDEX
4) add NOFOLLOW (really don't even want this, should never be used but would do for completeness)
5) add NOARCHIVE
I would want the app to have the 5 checkboxes above and control each option independently of each other so that users could select them in any combination. For example, there are times that you might want to remove a URL from the sitemap, but leave it in site search and leave it indexable. And while I might include a NOFOLLOW option in the app, I would put a warning when selected that this could be detrimental to the site's visibility in organic search (SEO).
I have always heard that Shopify has challenges when it comes to SEO, and now that I've worked with a couple of clients on Shopify I know this is the biggest SEO flaw I've seen with the platform other than page load times. No knowledgeable SEO would ever suggest adding NOFOLLOW to a meta robots element because it literally siphons off link juice and drains the site of link juice.
Is there a way to JUST remove a URL from the sitemap without triggering a meta robots with NOINDEX, NOFOLLOW?
I've sent up your feedback regarding the addition of NOFOLLOW in the current metafield. It's very possible there are things I'm not aware of that were considered when the decision was made to add NOFOLLOW with the metafield, but I definitely see where you're coming from in this regard. I can't make any promises as to if/when this will change, but your feedback is in the right hands now.
For your questions, #1 and #2 can be achieved with the metafield we discussed. Note that this metafield is currently the only way to remove a URL from the sitemap. #3, 4, and 5 can be achieved by adding code to the theme template. One idea could be to use a code snippet on the collection, page, or product template, containing conditional logic for rendering the HTML for NOINDEX, NOFOLLOW, or NOARCHIVE based on the resource's tags. Your app can inject this snippet on the template, and then add/remove tags based on the user's selection. This is just one idea, there may be other more elegant ways to do this as well. You may also want to explore using metafields instead of tags for the logic.
Thanks so much for the reply JB. I appreciate you passing this on. It would be ideal if they could decouple everything. For example, currently:
removes the URL from the sitemap, removes from site search, and adds noindex, nofollow. You could continue to support this existing treatment when value=1 while decoupling by adding new values for the existing metafield namespace=seo, key=hidden such as:
value=1: Continue doing what you do today
value=2: Remove from sitemap and sitesearch only (and do NOT add a meta robots at all)
value=3: Remove from sitemap only
value=4: Remove from site search only
If we had the above then users can handle generating the meta robots the way they want it. It's the fact that removing a URL from the sitemap with the existing method has three other side effects (remove from search, add noindex, add nofollow) which is problematic.
Again thanks so much for responding and looking into a possible fix/enhancement.
Is there an update on this issue? I am really waiting for the possibility to:
1) remove an URL from the sitemap
2) remove that same URL from the on-site search
3) add NOINDEX
4) add FOLLOW
Now it is only possible to remove an URL from the sitemap when you add NOFOLLOW. You don't want to add NOFOLLOW from a SEO perspective.
Is there any possibility to do this now?
Hope to hear from you.
Crickets... I have heard nothing since posting and them saying they were going to raise the issue to the developers. It's funny as the current way this is implemented is very detrimental from an SEO perspective, and this should be a VERY minor fix (likely one or two dozen lines of strategically placed code in the API that would supply much needed functionality while leaving the existing calls backward compatible for those sites/apps that don't mind their link juice flowing into a black hole. It appears they don't have anyone with a lot of SEO talent. I saw the other day they were advertising for an SEO.