Hi Where can we change this please? I dont have SEO Profiler
What are you using?
Whatever the tool, you'll want to crawl the shop slower if you're hitting issues. A lot of tools will give you settings but where those settings might be would be down to each tool. If you're using something that doesn't give you control over crawl speed then you may need to look into alternatives.
Howdy SEMrush users... I've been working with the product owner for the SEMrush Site Audit tool to resolve this issue. He's pretty confident that they'll have the crawl rate issue that's causing the false-positive 429 and 430 and “no-code” errors sorted out in the next month or so. Here's where they're at as of the end of July 2019...
We’ve made a lot of improvements in how Site Audit works with Shopify. Right now our bot can identify Shopify and automatically change crawling speed to the best setting in real-time. We see significantly less false positive errors triggered for Shopify websites.
I'll post an update here once he confirms that they've resolved the issue. In the meantime, if you want to ensure a clean audit of your Shopify site, I highly recommend checking out Sitebulb (https://sitebulb.com/). It's $35/mo and a little more technical than SEMrush, but it crawls Shopify sites much better.
Hi @larac, I just reached out to my SEMrush contact again and this is what he said....
We still working on this problem. We saw this drop in September and made a fix.
But we haven't stopped on this. We are still working on a scalable and stable solution. The first improvements will be finished in the next few weeks and solve the problem with speed. We plan to implement all improvements in the next few months one by one. In the end, we will publish a post in our news section.
Personally, I've noticed a decrease in 4xx errors and an increase in pages crawled over the past few months, but it's still not working perfectly. Stay tuned...