Why aren't all pages being crawled by SEMrush and Screaming Frog?

2 0 0

SEMrush and Screaming Frog only managed to crawl a few pages, mainly the homepage, CSS files and XML files. None of the product pages were accessed by the crawlers.

Why is it so? How can we manage those crawlers to crawl all the pages? Couldn't find the robots.txt file in the themes.

Replies 0 (0)

Community Blog Articles

The 9 Questions You Need To Answer When Starting a B2B Store

Starting a B2B store is a big undertaking that requires careful planning and execution. W...

By JasonH Sep 23, 2024

Get ready for peak holiday selling season 2024

By investing 30 minutes of your time, you can unlock the potential for increased sales,...

By Jacqui Sep 11, 2024

Highlights | Top Contributors August '24

We appreciate the diverse ways you participate in and engage with the Shopify Communi...

By JasonH Sep 9, 2024