I think the order is wrong here: the problem is more important than the tool. It may sound like a trivial distinction, but I really believe this is important. When you put the tool first, you just look for places where to use it, without thinking about what will be accomplished.
So my main suggestion is to first define the problem you wish to solve, and then think about what tools are appropriate.
I would like to give some examples of problems I tried solving using web scraping:
- when developing an app, we scraped the whole app marketplace to see what are the prices of apps in that category. We wanted to see what are the usual prices for apps.
- when we started writing blogs, we didn't know what our competitors wrote about, so we scraped their blog pages to see the titles of all blog posts they wrote...
- SEO: we wanted to see what domains rank for a given set of keywords (and what is their ranking)
Hi @Octoparse, this is Richard - CRO expert at PageFly.
I'd say that I agree with @adrianmisak reply. However, from my experience, the most common practices of e-commerce scraping tool are to: