How do I allow a developer crawl my Shopify site?

Technical Q&A
, ,

Topic summary

Allowing a third‑party developer/bot to crawl a Shopify store to collect product information for inclusion on a price comparison website.

  • Issue: The developer’s crawl attempts fail with a reported “43 0” error (exact code/context unclear).
  • Goal: Enable crawler access so the site’s data can be ingested.
  • Context: “Crawl” refers to automated fetching/indexing of site pages by a bot.
  • Missing details: Crawler/user agent used, access method, Shopify settings (e.g., robots.txt, password protection), or IP restrictions not provided.
  • Status: No troubleshooting steps or solutions yet; guidance requested and the question remains unresolved.
Summarized with AI on January 4. AI used: gpt-5.
1

How do I allow a developer to crawl my Shopify site? They are currently trying to do this, but are running into a 430 error.

The reason I want them to access this is so they can add our information to their price comparison website.