Think of it as “The Imitation Game.” Not the 2014 movie about British mathematicians racing against time to crack Nazi codes during World War II but a battle on the Internet between website operators and others who copy or steal their content.
The practitioners of web scraping use Internet bots to gather data from someone else’s website and then copy the content on their own site or use it for a variety of nefarious purposes, such as undercutting a competitor’s promotional pricing, stealing leads or hijacking marketing campaigns.
You’d think web scraping, which is on the rise, would be against the law. But the legal landscape is rife with inconsistencies and inconclusive cases, and varies from country to country.
Scraping has been around almost as long as the web and, in its good form, it’s actually a key component of how the Internet operates. Search engines like Google use “good bots” to index web content and quickly find, say, the latest Grumpy Cat video.
“Bad bots,” however, are designed to fetch content from a website with the intent of using it for purposes outside the site owner’s control — prices, promotions, offers or information that’s meant to be available only to paid subscribers or authorized business partners.
About the Author
Rami Essaid is the Chief Product and Strategy Officer and Co-founder of Distil Networks, the first easy and accurate way to identify and police malicious website traffic, blocking 99.9% of bad bots without impacting legitimate users. With over 12 years in telecommunications, network security, and cloud infrastructure management, Rami continues to advise enterprise companies around the world, helping them embrace the cloud to improve their scalability and reliability while maintaining a high level of security.Follow on Twitter More Content by Rami Essaid