We do crawl some of Shopify stores (with owners consent) for our business. Recently we noticed that crawling those stores returning HTTP error 430. I wonder if Shopify have started actively blocking crawling attempts.
We are also receving same error (430) in our dialy crawl process. In our product feed, we are getting this error for almost 70% of our product URLs. Is that realted to crawl rate limit? If that is the case, it should be the 429 error code as per Shopify doc.
Any more info on this?
It is indeed related to crawl rate - I've tried this today with Screaming Frog.
Go to configuration > Speed and set the Max Threads to 1.
I tried the Max Threads set to 2, but the errors start populating even at 2 threads.
Enjoy your painfully slow crawl! A giant pain in the ass for an SEO audit.
Got a great explanation about this from Tier 2 tech support at Shopify about the many 430 (server run-time errors) showing in my SEOProfiler.com bot crawl of my shopify store ( SEOProlifer uses a bot which crawls and reports all kind of SEO stuff, very nice tool for a one-person shop).
BACKGROUND: About 2 weeks ago I started getting 'error 430, server error) for my site which had been consistently clean. First I asked SEOProfiler to examine the cause of the 430's as I (nor my coder guy) touched anything on the site in the theme etc, only making a few small HTML changes to products and info pages, nothing else changed during the time the 430's started showing up.
THEN, took the issue to Shopify and had a few back and forths with chat level tech support. After the 3rd round I asked to be bumped up (escalated) to next level of tech support. Within 24 hours here's what they explained to me:
Jun 4, 22:52 EDT
I'm Emily, from Shopify's technical support team. Beth let me know about the problem you're having with 430 errors when you use SEO Profiler. I've looked into this for you and I have some more information about it.
The 430 error is happening because you're getting too much traffic on your site from the same IP, much too quickly, so we're blocking it temporarily for security reasons. It makes a lot of sense that you could see this error when you're using a service like that to check your site, because that is exactly what is happening: a ton of requests are coming through to your site, very quickly, all from the same IP.
The good news is, that error does not indicate there are any SEO problems with your shop. It also does not affect your customers' shopping experience in any way.
To avoid getting that error, you'll need to run the test more slowly.
I hope this helps. Please let me know if you have any other questions or concerns about this.
Tier 2 Support
So, my next step is to share this email with SEOProfiler and hope they can adjust the speed of their crawls. Anyway, since this maddening problem cost me alot of time (but fortunately ended good in that there were no real problems with my site coding) thought I'd lay this out here for others.
Follow up: I changed the delay time in SEOProfiler bot to 30seconds. No 430s after that. Just figure out how many pages total in your site and calculate so that all pages (or whatever limit you have, mine's 500) can be crawled within a 24 hr period. Also, it helps to use the google webmaster feature and you can see google's errors during their routine crawl of your site. a bit redundant but it's good to have the 2 bot results to compare imo.
Hi Yoko - Just wanted to add my thanks to you for resolving this one. I too have had the same issue over the past couple of months' crawls (using two separate tools) but found virtually nothing in the way of an explanation until I saw your post. Thanks again.
I just commented on a separate post with two different solutions to this problem. Here's a direct link to the solution...
Hope this helps!
Hi Where can we change this please? I dont have SEO Profiler
What are you using?
Whatever the tool, you'll want to crawl the shop slower if you're hitting issues. A lot of tools will give you settings but where those settings might be would be down to each tool. If you're using something that doesn't give you control over crawl speed then you may need to look into alternatives.
Howdy SEMrush users... I've been working with the product owner for the SEMrush Site Audit tool to resolve this issue. He's pretty confident that they'll have the crawl rate issue that's causing the false-positive 429 and 430 and “no-code” errors sorted out in the next month or so. Here's where they're at as of the end of July 2019...
We’ve made a lot of improvements in how Site Audit works with Shopify. Right now our bot can identify Shopify and automatically change crawling speed to the best setting in real-time. We see significantly less false positive errors triggered for Shopify websites.
I'll post an update here once he confirms that they've resolved the issue. In the meantime, if you want to ensure a clean audit of your Shopify site, I highly recommend checking out Sitebulb (https://sitebulb.com/). It's $35/mo and a little more technical than SEMrush, but it crawls Shopify sites much better.
Hi @larac, I just reached out to my SEMrush contact again and this is what he said....
We still working on this problem. We saw this drop in September and made a fix.
But we haven't stopped on this. We are still working on a scalable and stable solution. The first improvements will be finished in the next few weeks and solve the problem with speed. We plan to implement all improvements in the next few months one by one. In the end, we will publish a post in our news section.
Personally, I've noticed a decrease in 4xx errors and an increase in pages crawled over the past few months, but it's still not working perfectly. Stay tuned...
Thanks for sharing what the Shopify team has suggested. I am having the same issue, but without using programs such as your SEO profiler.
I am keen to know if you get 430 errors during periods when you aren't using the SEO profiler?
Connect your PayPal account to allow your customers to checkout using the PayPal gateway a...By Ollie Jul 28, 2022