Hi, I tried using Screaming Frog SEO Spider (SF) to crawl websites built by Shopify. Some websites were working, but more websites couldn’t be crawled, SF showcase status “Connection Refused” I then tried to contact the team at SF. They recommended an approach of adding my IP address and the ‘Screaming Frog SEO Spider’ user-agent to an allowlist in the website’s CDN security settings to enable crawling. Do you have any ideas for this case? Thank you!!
4 Likes