We noticed that sites on the Shopify platform are generated using scripts, which is why the code lacks text content in HTML format. So we have some questions here:
1. What methods do you have for solving the problem of serving text content to search engine robots in HTML format?
2. Is there any pre-rendering and which search engine used for the bots?
By default, your store's robots.txt file tells bots (usually search engine indexing bots, or "crawlers") which pages or files they should or shouldn't request to view from a website. For example, the shopping cart page is requested not to be viewed by search engines because it's unique to individual customers and it isn't useful to be indexed by a search engine.
All Shopify stores have a preconfigured and optimized robots.txt.liquid template. If you want to edit your robots.txt, then refer to Editing robots.txt.
You can hide pages that are not included in your robots.txt.liquid file by customizing the <head> section of your store's theme.liquid layout file. You need to include some code to noindex particular pages.