Unable to crawl website due to 429 Too many Request

Technical Q&A
1

Hi, we’re trying to crawl our website, but when screaming frog starts going past 2-6 URLs, we start getting Too Many Requests Status codes.

Any solutions to navigate this?

4

It’s probably better to share a link to the original Shopify documentation on crawler authentication:
https://help.shopify.com/en/manual/promoting-marketing/seo/crawling-your-store

5

try using screaming frog, from there you will be able to find the exact issue

6

Shopify is protecting your store (and their infrastructure) so will reject requests if you make too many in a short period of time without giving them a heads up.

You need to create a signature, which is the way to allow those requests through, and use it with Screaming Frog. The link that @tim_tairli provided also includes a link to Screaming Frog docs showing where to add the signature.