Shop Pay link marked 406 by my crawler

Shopify Discussion
, ,

Topic summary

A user encountered a 406 Not Acceptable error when their Siteimprove crawler attempted to access a Shop Pay SDK authorization link on their Shopify store.

Root cause:

  • The Shop Pay endpoint (pay.shopify.com/pay/sdk-authorize) is designed exclusively for in-app payment flows, not for web crawlers
  • Shopify’s server rejects the crawler’s request because the Accept header requests a content type the endpoint doesn’t provide
  • This is intentional security behavior—Shopify blocks unauthorized bots from accessing payment session links

Solution:

  • Exclude the Shop Pay SDK endpoint from the crawl configuration
  • The crawler itself is functioning correctly; it’s simply attempting to access a resource not meant for automated parsing

The issue is resolved with the recommendation to whitelist/exclude this specific URL pattern from future crawls.

Summarized with AI on October 27. AI used: claude-sonnet-4-5-20250929.
1

I am using Siteimprove for my site but this shop pay link is marked 406 Not acceptable everytime by my crawler.Can anyone help me with this.

This is the link: https://pay.shopify.com/pay/sdk-authorize?analytics_context=loginWithShopPrequal&analytics_trace_id=…

2

Hi @nr789 , the 406 Not Acceptable means Shopify’s server is rejecting the request because your crawler’s Accept header is asking for a content type that link doesn’t provide. Shop Pay pages are not meant for crawlers to parse.

Why this is happening? Shop Pay SDK endpoint behavior, it’s meant for in-app payment flows, not consumption by generic bots, so no crawler-friendly content is offered.

TL;DR:

  • The crawler isn’t broken, you’re poking a shop-only payment link.
  • Shopify wisely blocks non‑authorized, non‑session bots with a 406.
  • Just exclude the endpoint from your crawl, problem solved.

That’s all. If you find it helpful, please like and mark Solved.

1 Like