Hi there. Our marketing team is troubleshooting an issue with Google Merchant Center. According to Google our cart and checkout pages are not working properly (even though we’re not seeing this issue on our end).
The marketing folks sent over an article requesting we allow a full-site crawl by chancing the robots.txt file because we’re using “[link] attribute” for tracking.
Article to reference from Google: https://support.google.com/merchants/answer/6098185?hl=en
We have a custom robots.txt file but Shopify’s is different than the ones we’ve used on WordPress. For example, we can’t touch Shopify’s default robots.txt file, it appears we need to override the current settings. Additionally, the robots file is Shopify’s language, so we’re looking for any advice from the community on how to properly set this up without breaking our search crawlers.
How do I add what Google is recommending when I can’t actually change the robots.txt file?
You can allow a full-site crawl by changing your robots.txt file as follows:
User-agent: Googlebot
Disallow:
User-agent: Googlebot-image
Disallow:
This is the robots.txt file we’ve customized in an attempt to allow GoogleAdsBots.
# we use Shopify as our ecommerce platform
{%- comment -%}
# Caution! Please read https://help.shopify.com/en/manual/promoting-marketing/seo/editing-robots-txt before proceeding to make changes to this file.
{% endcomment %}
{% for group in robots.default_groups %}
{{- group.user_agent -}}
{% for rule in group.rules %}
{{- rule -}}
{% endfor %}
{%- if group.user_agent.value == 'adsbot-google' -%}
{{ 'Allow: /checkouts/' }}
{{ 'Allow: /checkout' }}
{{ 'Allow: /carts' }}
{{ 'Allow: /orders' }}
{{ 'Allow: /27373109384/checkouts' }}
{{ 'Allow: /27373109384/orders' }}
{{ 'Allow: /*?*oseid=*' }}
{{ 'Allow: /*preview_theme_id*' }}
{{ 'Allow: /*preview_script_id*' }}
{%- endif -%}
{%- if group.sitemap != blank -%}
{{ group.sitemap }}
{%- endif -%}
{% endfor %}
Are we doing this wrong? How would I allow for all Google Bots? Do we just say “Allow”? Any advice is appreciated!!
