On my site health report in semrush.com I am getting 429 errors for virtually every nav link on every page of my site. The links are being reported as broken, but all of the links I've tested are working fine.
For example, please see this screenshot: http://www.screencast.com/t/jDmYT9MliNE4
Typically these 429 errors come about when you're going over Shopify's API call limit. This document from our Shopify manual will explain in detail what this means and how to avoid it. It just means that SEMrush is sending Shopify too many requests at one time, and Shopify is sending a message back saying "whoa, that's way too many requests."
If you go into your SEMrush.com settings, you can manually set a delay for the bot in SEMrush. When you do that it reduces the amount of 429 errors that will appear.
Don't worry though, these errors don't indicate that there is anything wrong with your links or your site.
Hey Lisa P, unless I'm missing something in my SEMrush Pro account, the setting you're referring to doesn't exist. That being the case, and since we can't modify robots.txt, we're kinda left stranded by both parties. I've requested that SEMrush add that option, but what can Shopify do for us? In the 429 Errors and Crawl Delay post, someone said "Would be great if Shopify limited their crawl rate in their own robots.txt. Without that setting, we have zero control over whether Shopify blocks a legitimate tool or not, since we often can't set the crawl limit at the robot's side." Have you considered that?
|an hour ago|