Web scraping, the act of extracting information from websites to create data which can then be used to analyze or repurpose content, may be more prevalent than many experts believed. That is one takeaway from The 2016 Economics of Web Scraping report which found that web scrapers can be used for less than $4/hour.
Web scrapers can be anything from competitors trying to get a leg up on a category, fraudsters and hackers, and can even mimic user behaviors. The data, from Distil Networks, shows that just over one-third (38%) of web scraping is done to get new content - research, price comparisons and other data, with real estate websites getting scraped the most.
"If your content can be viewed on the web, it can be scraped," said Rami Essaid, CEO and co-founder of Distil Networks. "Not only does web scraping pose a critical challenge to a website's brand, it can threaten sales and conversions, lower SEO rankings, or undermine the integrity of content that took considerable time and resources to produce. Understanding the pervasive nature of today's web scraping economy not only raises awareness about this growing challenge, it also allows website owners to take action in the protection of their proprietary information."