[TECHNICAL] Shopify bot-detection (complexity-score) permanently blocks Google Search Console sitemap crawler — reproducible with ASN 15169 IP rotation
Issue since: December 2025 (post-migration from Shopware)
Impact: 0/500+ pages indexed in GSC, sitemap never successfully fetched
Background
After migrating from Shopware to Shopify in December 2025, Google Search Console has reported a persistent Sitemap could not be fetched error for /sitemap.xml. Every resubmission fails instantly. The sitemap was working correctly on the same domain under Shopware.
What I ruled out
-
Sitemap returns HTTP 200, valid XML, correct Content-Type: application/xml -
All 16 sub-sitemaps return HTTP 200
-
SSL valid, SANs cover both www and non-www
-
robots.txtcorrectly allows Googlebot (Liquid whitespace bug fixed) -
Sitemap submitted and resubmitted in GSC multiple times
-
Canonical URLs consistent (www. prefix)
Root cause hypothesis: shopify-complexity-score on rotating IPs
While building a diagnostic tool, I noticed Shopify returning this non-standard response header:
shopify-complexity-score: 1580
Reproducible pattern — 5 consecutive requests, Googlebot UA, 1s interval, fresh IP:
Request 1: shopify-complexity-score: 1580 ← CRITICAL
Request 2: shopify-complexity-score: 270
Request 3: shopify-complexity-score: 450
Request 4: shopify-complexity-score: 390
Request 5: shopify-complexity-score: 270
The key insight: Score spikes on the first request from any previously-unseen IP, then drops to ~270 on subsequent requests from the same IP.
This is catastrophic for GSC’s sitemap crawler because:
-
GSC fetches sitemaps from rotating IPs within Google’s infrastructure (ASN 15169)
-
Every GSC fetch is a “first request” from a new, unknown IP
-
Score 1580 → internal throttle or empty response body despite HTTP 200
-
GSC reports
could not be fetched -
Happening on every single GSC sitemap check since December 2025
Test script
import requests, time
SITEMAP_URL = "https://www.premium-electronics.eu/sitemap.xml"
UA = "Mozilla/5.0 (compatible; Googlebot/2.1)"
for i in range(1, 6):
r = requests.get(SITEMAP_URL, headers={"User-Agent": UA}, timeout=15)
score = r.headers.get("shopify-complexity-score", "n/a")
print(f"Request {i}: HTTP {r.status_code} | score: {score} | bytes: {len(r.content)}")
time.sleep(1)
Run this from a fresh IP (VPS, new proxy) — score on Request 1 will be dramatically higher than Request 2+.
User-Agent comparison (same IP, single requests)
| User-Agent | HTTP | complexity-score |
|---|---|---|
| Googlebot/2.1 — first req, new IP | 200 | 1580 |
| Googlebot-Image/1.0 | 200 | 440 |
| Google-InspectionTool/1.0 | 200 | 250 |
| AdsBot-Google | 200 | 300 |
| Chrome/120 | 200 | 300 |
| (empty UA) | 403 | -– |
Critical observation: Google-InspectionTool (GSC URL Inspection) scores 250 and works fine. The automated sitemap fetcher fails. This is why GSC shows pages as crawlable in URL Inspection but the sitemap is never processed.
What I’m asking Shopify to do
-
Check server logs — requests to /sitemap.xml from Google ASN 15169 since December 2025. Full content served, or empty body despite HTTP 200?
-
Exempt /sitemap.xml and
/sitemap_*.xmlfrom complexity-score throttling. These endpoints have no user-session context by design — they exist exclusively for crawler consumption. Session-based bot scoring is architecturally counterproductive here. -
Whitelist Google ASN 15169 for sitemap endpoints.
GSC impact
Indexed: 94 (expected 500+)
Not indexed: 457
Crawled, not indexed: 221
Sitemap: "Could not be fetched" — since December 2025
Has anyone else measured shopify-complexity-score above 1000 on first requests from a fresh IP? Does the response body get truncated at that score level? That would confirm the mechanism.
Tagging @Shopify_Support @ShopifyPartners — escalation to someone with server-log access appreciated.
Full diagnostic JSON report available on request.