FROM CACHE - en_header

Crawling Shopify stores returns HTTP error 430

baseplan2_Admin
Shopify Partner
1 0 2

Hi, 

We do crawl some of Shopify stores (with owners consent) for our business. Recently we noticed that crawling those stores returning HTTP error 430. I wonder if Shopify have started actively blocking crawling attempts. 

Replies 20 (20)
Janak
New Member
1 0 1

Hi,

We are also receving same error (430) in our dialy crawl process. In our product feed, we are getting this error for almost 70% of our product URLs. Is that realted to crawl rate limit? If that is the case, it should be the 429 error code as per Shopify doc.

Any more info on this?

 

Thanks

 

Kevin_Purchase
New Member
1 0 1

Hi -

It is indeed related to crawl rate - I've tried this today with Screaming Frog.

Go to configuration > Speed and set the Max Threads to 1.

I tried the Max Threads set to 2, but the errors start populating even at 2 threads.

Enjoy your painfully slow crawl! A giant pain in the ass for an SEO audit.

Yoko_Lewis
Tourist
8 1 12

Got a great explanation about this from Tier 2 tech support at Shopify about the many 430 (server run-time errors) showing in my SEOProfiler.com bot crawl of my shopify store ( SEOProlifer uses a bot which crawls and reports all kind of SEO stuff, very nice tool for a one-person shop).

BACKGROUND: About 2 weeks ago I started getting 'error 430, server error) for my site which had been consistently clean. First I asked SEOProfiler to examine the cause of the 430's as I (nor my coder guy) touched anything on the site in the theme etc, only making a few small HTML changes to products and info pages, nothing else changed during the time the 430's started showing up.

THEN, took the issue to Shopify and had a few back and forths with chat level tech support. After the 3rd round I asked to be bumped up (escalated) to next level of tech support.  Within 24 hours here's what they explained to me:

Emily (Shopify)

Jun 4, 22:52 EDT

Hi *******

I'm Emily, from Shopify's technical support team. Beth let me know about the problem you're having with 430 errors when you use SEO Profiler. I've looked into this for you and I have some more information about it.

The 430 error is happening because you're getting too much traffic on your site from the same IP, much too quickly, so we're blocking it temporarily for security reasons. It makes a lot of sense that you could see this error when you're using a service like that to check your site, because that is exactly what is happening: a ton of requests are coming through to your site, very quickly, all from the same IP.

The good news is, that error does not indicate there are any SEO problems with your shop. It also does not affect your customers' shopping experience in any way.

To avoid getting that error, you'll need to run the test more slowly.

I hope this helps. Please let me know if you have any other questions or concerns about this.

Kind regards,

Emily
Tier 2 Support

-----------
So, my next step is to share this email with SEOProfiler and hope they can adjust the speed of their crawls. Anyway, since this maddening problem cost me alot of time (but fortunately ended good in that there were no real problems with my site coding) thought I'd lay this out here for others.

Shell1
New Member
4 0 0

Thankyou !! I needed this too.

Yoko_Lewis
Tourist
8 1 12

Follow up: I changed the delay time in SEOProfiler bot to 30seconds. No 430s after that. Just figure out how many pages total in your site and calculate so that all pages (or whatever limit you have, mine's 500) can be crawled within a 24 hr period. Also, it helps to use the google webmaster feature and you can see google's errors during their routine crawl of your site. a bit redundant but it's good to have the 2 bot results to compare imo.

Tom_M
Excursionist
76 0 11

Hi Yoko - Just wanted to add my thanks to you for resolving this one. I too have had the same issue over the past couple of months' crawls (using two separate tools) but found virtually nothing in the way of an explanation until I saw your post. Thanks again.

Joe_Lau
New Member
51 0 0

Thx Yoko 🙂

Suzane_Campbell
Tourist
6 0 7

Thank you! I couldn't understand that either. 

KevinW
Explorer
55 8 64

I just commented on a separate post with two different solutions to this problem. Here's a direct link to the solution...

 

https://community.shopify.com/c/Ecommerce-Marketing/429-Errors-and-Crawl-Delay/m-p/487388/highlight/...

 

Hope this helps!

Kevin Wallner // SEO & Analytics Expert
- Was my reply helpful? Click Like to let me know!
- Was your question answered? Click Accept as Solution
- Ready to accelerate your SEO growth? Let's chat!
StuartAK
Tourist
6 0 6

Thanks so much.

colin3bears
Tourist
4 0 0

Hi Where can we change this please? I dont have SEO Profiler

 

Jason
Shopify Expert
10918 193 2191

@colin3bears wrote:

Hi Where can we change this please? I dont have SEO Profiler

 


What are you using?

Whatever the tool, you'll want to crawl the shop slower if you're hitting issues. A lot of tools will give you settings but where those settings might be would be down to each tool. If you're using something that doesn't give you control over crawl speed then you may need to look into alternatives.

★ I jump on these forums in my free time to help and share some insights. Not looking to be hired, and not looking for work. http://freakdesign.com.au ★
colin3bears
Tourist
4 0 0
Ok thanks I will
KevinW
Explorer
55 8 64

Howdy SEMrush users... I've been working with the product owner for the SEMrush Site Audit tool to resolve this issue. He's pretty confident that they'll have the crawl rate issue that's causing the false-positive 429 and 430 and “no-code” errors sorted out in the next month or so. Here's where they're at as of the end of July 2019...

 

We’ve made a lot of improvements in how Site Audit works with Shopify. Right now our bot can identify Shopify and automatically change crawling speed to the best setting in real-time. We see significantly less false positive errors triggered for Shopify websites.

I'll post an update here once he confirms that they've resolved the issue. In the meantime, if you want to ensure a clean audit of your Shopify site, I highly recommend checking out Sitebulb (https://sitebulb.com/). It's $35/mo and a little more technical than SEMrush, but it crawls Shopify sites much better. 

Kevin Wallner // SEO & Analytics Expert
- Was my reply helpful? Click Like to let me know!
- Was your question answered? Click Accept as Solution
- Ready to accelerate your SEO growth? Let's chat!
larac
New Member
1 0 0

Anyone know if this issue has been fixed yet?

 

Thanks

KevinW
Explorer
55 8 64

Hi @larac, I just reached out to my SEMrush contact again and this is what he said....

 

We still working on this problem. We saw this drop in September and made a fix.
But we haven't stopped on this. We are still working on a scalable and stable solution. The first improvements will be finished in the next few weeks and solve the problem with speed. We plan to implement all improvements in the next few months one by one. In the end, we will publish a post in our news section.

Personally, I've noticed a decrease in 4xx errors and an increase in pages crawled over the past few months, but it's still not working perfectly. Stay tuned...

Kevin Wallner // SEO & Analytics Expert
- Was my reply helpful? Click Like to let me know!
- Was your question answered? Click Accept as Solution
- Ready to accelerate your SEO growth? Let's chat!
KevinW
Explorer
55 8 64
Hot off the press! This morning SEMrush announced that they’ve updated their Site Audit tool to improve the way they crawl Shopify stores. This is a big win for the Shopify SEO community 🙂

“ We’re happy to inform you that we have completed the final cycle of training the Site Audit algorithm to crawl sites built on the Shopify platform.

So now you can check your Shopify website for over 120 on-page and technical SEO issues without any problems!

You don’t need any specific settings to configure the Site Audit tool for a Shopify website, and the new algorithm already works for all existing projects. Namely, the tool:

-Identifies Shopify websites automatically and crawls the site at the optimal speed.
-In the case of a 429 error, the tool will double-check the page availability a moment later.
-There are no false positive "broken internal links" or “unavailable pages” issues.

If there are any crawling issues or you’d like to share your feedback, feel free to write to us at site-audit-feedback@semrush.com. We are happy to become better for you!”
Kevin Wallner // SEO & Analytics Expert
- Was my reply helpful? Click Like to let me know!
- Was your question answered? Click Accept as Solution
- Ready to accelerate your SEO growth? Let's chat!
zaraLou
Tourist
6 0 0

Thanks for sharing what the Shopify team has suggested. I am having the same issue, but without using programs such as your SEO profiler. 

I am keen to know if you get 430 errors during periods when you aren't using the SEO profiler?

 

Zara

colin3bears
Tourist
4 0 0
Not sure Zara I don't think so it doesn't show in my merchant centre