Crawling Shopify stores returns HTTP error 430

Highlighted
Tourist
5 0 1

Thanks so much.

0 Likes
Highlighted
Tourist
3 0 0

Hi Where can we change this please? I dont have SEO Profiler

 

0 Likes
Highlighted
Shopify Expert
9863 98 1700

@colin3bears wrote:

Hi Where can we change this please? I dont have SEO Profiler

 


What are you using?

Whatever the tool, you'll want to crawl the shop slower if you're hitting issues. A lot of tools will give you settings but where those settings might be would be down to each tool. If you're using something that doesn't give you control over crawl speed then you may need to look into alternatives.

★ Winning Partner of the Build a Business competition. ★ http://freakdesign.com.au
1 Like
Highlighted
Tourist
3 0 0
Ok thanks I will
0 Likes
Highlighted

Howdy SEMrush users... I've been working with the product owner for the SEMrush Site Audit tool to resolve this issue. He's pretty confident that they'll have the crawl rate issue that's causing the false-positive 429 and 430 and “no-code” errors sorted out in the next month or so. Here's where they're at as of the end of July 2019...

 

We’ve made a lot of improvements in how Site Audit works with Shopify. Right now our bot can identify Shopify and automatically change crawling speed to the best setting in real-time. We see significantly less false positive errors triggered for Shopify websites.

I'll post an update here once he confirms that they've resolved the issue. In the meantime, if you want to ensure a clean audit of your Shopify site, I highly recommend checking out Sitebulb (https://sitebulb.com/). It's $35/mo and a little more technical than SEMrush, but it crawls Shopify sites much better. 

0 Likes
New Member
1 0 0

Anyone know if this issue has been fixed yet?

 

Thanks

0 Likes
Highlighted

Hi @larac, I just reached out to my SEMrush contact again and this is what he said....

 

We still working on this problem. We saw this drop in September and made a fix.
But we haven't stopped on this. We are still working on a scalable and stable solution. The first improvements will be finished in the next few weeks and solve the problem with speed. We plan to implement all improvements in the next few months one by one. In the end, we will publish a post in our news section.

Personally, I've noticed a decrease in 4xx errors and an increase in pages crawled over the past few months, but it's still not working perfectly. Stay tuned...

1 Like
Highlighted
Hot off the press! This morning SEMrush announced that they’ve updated their Site Audit tool to improve the way they crawl Shopify stores. This is a big win for the Shopify SEO community :)

“ We’re happy to inform you that we have completed the final cycle of training the Site Audit algorithm to crawl sites built on the Shopify platform.

So now you can check your Shopify website for over 120 on-page and technical SEO issues without any problems!

You don’t need any specific settings to configure the Site Audit tool for a Shopify website, and the new algorithm already works for all existing projects. Namely, the tool:

-Identifies Shopify websites automatically and crawls the site at the optimal speed.
-In the case of a 429 error, the tool will double-check the page availability a moment later.
-There are no false positive "broken internal links" or “unavailable pages” issues.

If there are any crawling issues or you’d like to share your feedback, feel free to write to us at site-audit-feedback@semrush.com. We are happy to become better for you!”
0 Likes