New Shopify Certification now available: Liquid Storefronts for Theme Developers

Not crawling all pages

Luxury2020
Visitor
2 0 0

SEMrush and Screaming Frog only managed to crawl a few pages, mainly the homepage, CSS files and XML files. None of the product pages were accessed by the crawlers. 

 

Why is it so? How can we manage those crawlers to crawl all the pages? Couldn't find the robots.txt file in the themes.

Replies 0 (0)