How to index blog tags in search engines?

[michalc]

Hello Shopify community,

I'm facing a problem with shopify store indexing: blog tags aren't indexed. This is a problem due to SEO and organic traffic. Indexing of tags of the entire shopify store is prohibited via robots.txt, which cannot be edited in any way. But I need to categorize content and target keywords with categories. See our blog for yourself:

• https://bravior.at/blogs/magnetfischen
• https://bravior.de/blogs/magnetfischen

I put a lot of work into each content category. every category is an unique content. The articles are precisely categorized, each category has a nice H1 meta tag and contains a short unique text that talks about what the visitor will find in the category. I use this liquid code to display different content on tags:

{% if current_tags contains 'A' %}
A
{% elsif current_tags contains 'B' %}
B
{% elsif current_tags contains 'C' %}
C
{% else %}
Z
{% endif %}

But of course this is not the case. It is important that those categories are not indexed. This is the problem we are trying to solve. How to index these tag categories? Or how to do this so that this content is properly indexed? Thank you.

Best regards

Michal

[KieranR · Accepted solution]

Indexing of tags of the entire shopify store is prohibited via robots.txt

Crawling level 1 tag URLs is allowed. But multiple tags is blocked by robots.txt, for example: 

Allowed: myshop.com/blogs/blog-handle/tagged/tag1
Allowed: myshop.com/blogs/blog-handle/tagged/tag2
Blocked: myshop.com/blogs/blog-handle/tagged/tag2+tag1

You can validate this for yourself with something like technicalseo.com/tools/robots.txt - so I don't think robots.txt is an issue in this case.

But those pages are still not indexed. There are a few things I'd suggest trying, from a birds-eye view without getting too hands on: 

• Just wait, not sure what time-scales you're talking here. But if you have a new site or low authority, then I would only start to be slightly concerned after ~2-4 weeks following going live with those URLs. If you did this more recently, then just be patient. Google seems to be a lot slower recently with adding in newly discovered Shopify sites & pages to the index. 
• Make sure the intro paragraph is unique enough. It looks like only one sentence? Try aim for like 100 words.
• Make sure the title+metas are unique enough for those tag page URLs. Meta descs currently not present.
• Make sure Googlebot can discover the pages through static crawling, check how many internal links those pages are getting. Aim for like 2+ for each tag page ideally. Crawl tools like Sitebulb, SreamingFrog can help to understand (and even visualize) the internal structure here.
• You could try throw up a supplemental sitemap. Get a list, generate a sitemap, upload it to shopify files. Setup a redirect from /tagpages.xml to endpoint of the actual xml. Add /tagpages.xml to Google Search Console

[michalc]

Ok thank you very much for your answer and your advices. We will implement them. Hopefully will all go ok

[PaulNewton]

@michalc , @KieranR 

⚠️Notice:  This is an unsupported customization . If you break something you're on your own Shopify support will not help you and will have to pay someone to fix it.

Editing robots.txt feature is rolling out, should have generally availability by monday, so far for all plans.

Bigger thread https://community.shopify.com/c/Shopify-Discussion/How-can-I-edit-robots-txt/m-p/1212458/highlight/t... 

https://help.shopify.com/en/manual/online-store/themes/theme-support#shopify-design-policy 

Be careful while you can make it so more tags are crawl tags can be applied in any order risking duplicate content SEO penalties.

[Trevor]

Hello!

Just a quick update about the robot.txt file that was referenced in this post. As of today, June 21st, 2021, we have launched the ability to edit the robot.txt file to give merchants more control over the information that is crawled by search engines. You can learn more about how to edit your robot.txt file through our community post here. 

If you have any questions about the new feature, please do not hesitate to create a new post under our "Techincal QA" board.