[TECHNICAL] Shopify bot-detection (complexity-score) permanently blocks Google Search Console sitemap crawler — reproducible with ASN 15169 IP rotation

Technical Q&A
1

Issue since: December 2025 (post-migration from Shopware)
Impact: 0/500+ pages indexed in GSC, sitemap never successfully fetched

Background

After migrating from Shopware to Shopify in December 2025, Google Search Console has reported a persistent Sitemap could not be fetched error for https://www.premium-electronics.eu/sitemap.xml. Every resubmission fails instantly. The sitemap was working correctly on the same domain under Shopware.

I’ve spent considerable time ruling out the obvious causes before concluding this is a Shopify infrastructure issue.

What I ruled out

  • :white_check_mark: Sitemap returns HTTP 200, valid XML, correct Content-Type: application/xml; charset=utf-8

  • :white_check_mark: All 16 sub-sitemaps return HTTP 200

  • :white_check_mark: SSL valid, SANs cover both www and non-www

  • :white_check_mark: robots.txt correctly allows Googlebot (whitespace bug in Liquid template fixed)

  • :white_check_mark: No noindex on sitemap URLs

  • :white_check_mark: Sitemap submitted and resubmitted in GSC multiple times

  • :white_check_mark: Canonical URLs consistent (www. prefix)

  • :white_check_mark: Non-www 301 redirect in place

Root cause hypothesis: shopify-complexity-score on rotating IPs

While building a diagnostic tool, I noticed Shopify returning a non-standard response header:

shopify-complexity-score: 1580

Here is the reproducible pattern from my test suite (5 consecutive requests, same Googlebot UA, 1s interval, from a single external IP):

Request 1:  shopify-complexity-score: 1580  ← CRITICAL

Request 2: shopify-complexity-score: 270

Request 3: shopify-complexity-score: 450

Request 4: shopify-complexity-score: 390

Request 5: shopify-complexity-score: 270

The key insight: The score spikes massively on the first request from any previously-unseen IP. After the IP becomes “known” to Shopify’s scoring system, subsequent requests drop to ~270.

This becomes a critical problem specifically for Google Search Console’s sitemap crawler because:

  1. GSC fetches sitemaps from rotating IP addresses within Google’s infrastructure (ASN 15169)

  2. Every fetch attempt from GSC is effectively a “first request” from a new, unknown IP

  3. First request → complexity-score 1580 → internal throttle or empty response body despite HTTP 200 header

  4. GSC receives no usable sitemap content → reports could not be fetched

  5. This has been happening on every single GSC sitemap check since December 2025

This explains why:

  • The sitemap works fine from my browser and test machines (IPs become “known” after first visit)

  • The sitemap works fine from curl, Chrome UA, AdsBot-Google

  • Only GSC’s datacenter crawler fails — because it always comes from a fresh rotating IP

Test methodology

import requests, time

SITEMAP = "https://www.premium-electronics.eu/sitemap.xml"

UA = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

for i in range(1, 6):

r = requests.get(SITEMAP, headers={"User-Agent": UA}, timeout=15)

score = r.headers.get("shopify-complexity-score", "n/a")

print(f"Request {i}: HTTP {r.status_code} | score: {score} | bytes: {len(r.content)}")

time.sleep(1)

Output (from a fresh IP each test run):

Request 1: HTTP 200 | score: 1580 | bytes: [varies — investigating if content is truncated]

Request 2: HTTP 200 | score: 270 | bytes: [full content]

User-Agent comparison (same IP, single requests)

User-Agent HTTP Status complexity-score
Googlebot/2.1 (first req, new IP) 200 1580
Googlebot-Image/1.0 200 440
Google-InspectionTool/1.0 200 250
AdsBot-Google 200 300
Chrome/120 200 300
(empty UA) 403 -–

Note: Google-InspectionTool (used by GSC’s URL Inspection tool) scores 250 and works — this is why manual URL inspection in GSC shows pages as crawlable, while the automated sitemap fetcher fails. The inspection tool likely hits the same IP repeatedly.

What I’m asking Shopify to do

  1. Check server logs for requests to /sitemap.xml from Google ASN 15169 since December 2025. Are these requests being served with full content, partial content, or empty body despite HTTP 200?

  2. Exempt sitemap endpoints (/sitemap.xml, /sitemap_*.xml) from complexity-score-based throttling. These endpoints have no user-session context by design — they exist exclusively for crawler consumption. Applying session-based bot scoring to them is architecturally counterproductive.

  3. Whitelist Google ASN 15169 for sitemap endpoints, or raise the internal action threshold for these specific paths.

Why this affects every Shopify store to some degree

If my hypothesis is correct, this isn’t unique to my store. Any Shopify store where:

  • GSC has recently resubmitted a sitemap

  • The store is on a less-trafficked domain (fewer cached IPs)

  • Google’s sitemap crawler rotates IPs faster than Shopify’s scoring system “learns” them

…will exhibit the same failure pattern. Larger stores with high traffic may not notice because their IP pool is pre-warmed.

GSC impact

Indexed pages:          94  (expected: 500+)

Not indexed: 457

Crawled, not indexed: 221

Sitemap status: "Could not be fetched" — since December 2025

Five months of essentially zero organic visibility on a store that was performing correctly on the same domain under a different platform.

Has anyone else observed shopify-complexity-score values above 1000 on first requests? I’m particularly interested in whether the response body is actually truncated or empty when the score is that high, as that would be the smoking gun.

Tagging: @Shopify_Support @ShopifyPartners — would appreciate an escalation to someone with server-log access.

Diagnostic report with full HTTP response data, headers, and timing available on request.

FORUM UPDATE:

[UPDATE - Critical Finding: ALL URLs affected, not just /sitemap.xml]

After further investigation, I can now confirm this is significantly worse than initially described.

The complexity-score does not only block /sitemap.xml — it blocks every single URL on the domain when accessed by GSC’s crawler.

Test result from today on /pages/sitemap-static (a regular Shopify page, freshly created):

HTTP: 200

shopify-complexity-score: 4530

Content-Type: text/html

Score 4530 — from an IP that has already made multiple requests to this domain. From a completely fresh IP (as GSC always uses) the score would be even higher.

The consequence:

  • Zero successful crawls since migration in December 2025

  • GSC column “Last read”: empty for every submitted sitemap — not a single successful fetch in 5 months

  • 94 indexed pages out of 500+ are remnants from the previous platform

  • Every new product, page, and article published since December is invisible to Google

This is not a sitemap configuration issue. This is a complete crawl blackout affecting the entire domain.

The Google-InspectionTool user-agent (score ~250) works fine — which is why manual URL inspection in GSC shows pages as crawlable. But the automated GSC sitemap crawler and the standard Googlebot crawler are systematically blocked on first request from every new rotating IP in ASN 15169.

This affects every Shopify merchant with a lower-traffic domain where Google’s IP pool is never “pre-warmed” in Shopify’s scoring system. High-traffic stores may not notice because their IPs are already known.

Demand: Shopify must exempt ASN 15169 (Google LLC) from complexity-score throttling, or at minimum set the action threshold high enough that HTTP 200 responses include full content regardless of score.

Five months of zero organic reach on a correctly configured store is not acceptable.

[UPDATE - Critical Finding: ALL URLs affected, not just /sitemap.xml]

After further investigation, I can now confirm this is significantly worse than initially described.

The complexity-score does not only block /sitemap.xml — it blocks every single URL on the domain when accessed by GSC’s crawler.

Test result from today on /pages/sitemap-static (a regular Shopify page, freshly created):

HTTP: 200

shopify-complexity-score: 4530

Content-Type: text/html

Score 4530 — from an IP that has already made multiple requests to this domain. From a completely fresh IP (as GSC always uses) the score would be even higher.

The consequence:

  • Zero successful crawls since migration in December 2025

  • GSC column “Last read”: empty for every submitted sitemap — not a single successful fetch in 5 months

  • 94 indexed pages out of 500+ are remnants from the previous platform

  • Every new product, page, and article published since December is invisible to Google

This is not a sitemap configuration issue. This is a complete crawl blackout affecting the entire domain.

The Google-InspectionTool user-agent (score ~250) works fine — which is why manual URL inspection in GSC shows pages as crawlable. But the automated GSC sitemap crawler and the standard Googlebot crawler are systematically blocked on first request from every new rotating IP in ASN 15169.

This affects every Shopify merchant with a lower-traffic domain where Google’s IP pool is never “pre-warmed” in Shopify’s scoring system. High-traffic stores may not notice because their IPs are already known.

Demand: Shopify must exempt ASN 15169 (Google LLC) from complexity-score throttling, or at minimum set the action threshold high enough that HTTP 200 responses include full content regardless of score.

Five months of zero organic reach on a correctly configured store is not acceptable.