New Findings Announced in Distil’s 2016 Economics of Web Scraping Report

September 7, 2016 Jennifer Glaeser

Did you know that you can hire agents to scrape any website? An entire website, including product descriptions, competitor pricing, data sheets, contact lists, and any desirable information can be scraped for as little as $3.33 per hour. For less than a cup of coffee, a competitor can take all your prices from your site and use them against you.

This is one of the key findings in Distil Networks’ new report, 2016 Economics of Web Scraping. It illustrates web scraping’s prevalence, sophistication, and industry use cases. Through analysis of top scraping platforms and services, the report discusses how the democratization of web scraping lets users effortlessly steal sensitive digital information. Key findings reveal that scraping tools are readily available in a variety of forms, are inexpensive, and are used by a range of industries.

Scraping is an automated software method used to extract information from websites. It often includes transforming unstructured data into a database for subsequent competitive analysis, or illicitly repurposing content for the scraper’s own business operations. Bad bots, which comprise around 20% of all web traffic, are run by perpetrators to perform scraping at a much faster rate than humans can do so manually.

Diverse actors leverage web scraping bots. They include nefarious competitors, Internet upstarts, hedge funds, fraudsters, hackers, and spammers, who effortlessly steal whatever content they program their bots to find. The bots often mimic legitimate user behavior, making them hard to detect—and even harder to block.  

If your content can be viewed on the web, it can be scraped,” said Rami Essaid, CEO and co-founder of Distil Networks. “Not only does scraping pose a critical challenge to a website’s or company’s brand, it can threaten sales and conversions, lower SEO rankings, or undermine the integrity of content that took considerable time and resources to produce. Site owners should understand the pervasive nature of today’s web scraping economy so as to take action toward protecting their proprietary information.”

To download a copy of the 2016 Economics of Web Scraping report, visit


Previous Article
Can Comic Sans Detect Cyber Attacks?
Can Comic Sans Detect Cyber Attacks?

In our effort against bad bots, we used information theory in a study of comparing fonts installed on 1.2 m...

Next Article
Plagiarism in Politics and Content Scraping
Plagiarism in Politics and Content Scraping

Melania Trump’s speech at the RNC has come under fire for plagiarism. View more on politicians stealing con...