Why aren't all pages being crawled by SEMrush and Screaming Frog?

2 0 0

SEMrush and Screaming Frog only managed to crawl a few pages, mainly the homepage, CSS files and XML files. None of the product pages were accessed by the crawlers.

Why is it so? How can we manage those crawlers to crawl all the pages? Couldn't find the robots.txt file in the themes.

Replies 0 (0)

Community Blog Articles

Dropshipping Viral Products to Sell in 2025

As 2024 wraps up, the dropshipping landscape is already shifting towards 2025's trends....

By JasonH Nov 27, 2024

Shining a spotlight on our top members of October 2024!

Hey Community! It’s time to share some appreciation and celebrate what we have accomplis...

By JasonH Nov 14, 2024

Tips for Running BFCM Deals and Discounts 2024

In today’s interview, we sat down with @BSS-Commerce to discuss practical strategies f...

By JasonH Nov 13, 2024