Hi there. We don’t have a custom robots.txt file and our cart and checkout pages and are running into issues with crawlability. We received a message from our marketing team:
“We noticed that the site is currently blocking the Google Adsbot from the checkout/cart pages so Google can’t crawl them. Are you able to add the robots.txt file and make sure they aren’t being disallowed for anything on the site? Please refer to the below.”
What’s the process for setting up a robots.txt file on Shopify (1.0 store) and is this a typical request? Why does Shopify block this to begin with?
Thanks!