Do you know which of your website visitors are Human? Can you tell the difference between Good Bots and Bad Bots?
Distil Networks had the opportunity to work with Peter Bernstein, Senior Editor at TMCNet, on a webinar discussing the problems that growing armies of bots and botnets pose for website security. Peter’s extensive experience in networking and communications, combined with Distil Networks’ CEO Rami Essaid’s expertise in the world of bots, made for a fascinating and enlightening hour. You can watch a replay of the video below, but for those of you who don’t have an hour to spare, we’ve summarized the highlights for you here below.
Who do bots and botnets target?
Almost everyone. Unless you don’t have a web presence, or your website is completely disconnected from the rest of your business, you likely are, or have been, a target. It was instructive to see how the webinar attendees answered the simple question “Has your organization been threatened in the last 12 months by a malicious online attack?” Here were the results:
Rami correctly predicted the “not sure” majority response you see above. Why? Because it can be very hard to tell whether your site is being targeted unless you’re (a) extremely familiar with what normal traffic patterns and sources on your site look like or (b) have specific botnet monitoring and protection in place.
Bots and Botnets – cheaper and easier than ever before
Distil Networks often sees that up to 60% of its customers’ website traffic is bot-driven. Cloud computing and virtualization have enabled attackers to launch bot attacks faster and at a lower cost than ever before – so much so that any nefarious actor with a credit card can rent or acquire more than enough bandwidth. Freeware and a rental fee of $12 an hour for 1000 machines easily buys the necessary software and infrastructure.
So what IS a bot?
The answer is not as obvious as it might seem. Like malware, bots can take many different forms and perform many different tasks. Unlike malware, bots can be used for good – for example, search engine bots like Google bots are fundamental to your sites’ search engine rankings. So any technology that’s used to protect against Bad bots must also be able to distinguish the bad from the good – and from the human beings that are legitimately trawling through your site.
Bad Bot Attack Breakdown We see six primary types of bot attacks in our work with clients:
- Data theft bots can do serious damage, especially for online directories, classifieds, real estate listings, and digital publishers, where content and data are their lifeblood. If that data is stolen and published elsewhere, revenue is lost, market presence is eroded, and ultimately the entire business model is at risk.
- Price scraping bots can be business-killers for ecommerce and travel industry websites that use real-time product pricing as a competitive advantage. A bot stealing the price advantage is effectively stealing the entire business.
- Click fraud bots hit marketers where it hurts by falsifying clickthroughs. Up to $6 billion in wasted adspend is happening every year, thanks to bots clicking on paid advertising campaigns and maxing out their daily spends. It also harms the ad networks by distorting the results they provide to their clients.
- Brute force login attacks are the bane of any online service that requires a username and password to access the site. A couple of months ago, a Russian hacker group was able to compromise 30,000 accounts using bots to pound a domain registration site’s servers with email and password lists until they got lucky.
- Application DDoS attacks have increased 20% in the past year, and can cost an impacted organization an average of $882,000 (source: Gartner). A few years ago, an attack that was in the single-digit gigabits/second is now in the hundreds of gigabits/second. When Realtor.com was hit, the site was down for a week and had to spend over a million dollars in offline ads to keep the business going.
- Vulnerability scans look for holes and vulnerabilities by using automated programs to scour your site for entry points. These can easily turn into extortion, and are what was behind this year’s devastating Heartbleed attacks.
It’s clear from just these few examples (and there are many more) that knowing what is transpiring on your website is key to protecting your business. Additionally, the courts are unlikely to bring much comfort, since there is no clear legal definition of a bot. Under European Union law, once information has been published on a website, it is deemed to be in the public domain. In the US, lawyers have attempted to apply various existing statutes concerned with data breach and trespass, with little success – and much expense to corporate clients.
Traditional IP-based approaches to blocking bots no longer work
The root of the dilemma can be summed up by the uncertainty of differentiating good bots from bad bots and from human interactions. The traditional approach to detecting bots has been based around IP addresses – the standard good list/bad list tactic. But since many IP addresses rotate through multiple users, good and bad, this really cannot be a reliable identifier for ill-intentioned bots anymore. This is one instance where, unlike malware, code fingerprints can be a reliable indicator. When fingerprinting is combined with a behavioral analysis of your site’s normal human traffic, a well-designed technology solution can identify bad bots – and let you decide what to do about them. And to separate bots from humans, some fairly sophisticated behavioral detection techniques are required.
How can you tell if you’ve been attacked by bots?
A front end indicator that your site has been targeted by bots is a decrease in page load times on your website. From a business perspective, if you notice an uptick in competitive activity, or increased activity from unexpected IP ranges or foreign user agents, there’s a strong likelihood your website security is under attack. The reality is that most organizations will not notice a botnet attack until the damage has already been done.
Here’s why your Web Application Firewall (WAF) can’t protect you from bots
Your first instinct is to look at your firewall logs, but the limitations of web application firewalls (WAF) in addressing bot attacks will quickly become apparent. Firewalls look at packets, and can only block specific IP addresses; additionally, Layer 7 protection is not a function of most firewall technologies. It is also possible to build some degree of protection using in-house resources, but such approaches are likely to require a considerable degree of hands-on, manual work, which doesn’t make a lot of sense from a business perspective. We’ve already seen that IP blocks are largely ineffective against bots, and CAPTCHA-type filters can be bypassed by most semi-smart bot technologies. To strengthen website security against anything beyond amateur bot deployers, most organizations will need expert assistance and some sort of machine learning that evolves with the bots.
Leveraging the community to repel bot attacks
Having blocked (as of December 2014) over 25 billion Bad Bots, Distil Networks has been able to develop the industry’s leading dedicated bot detection and mitigation solution. With a combination of fingerprinting, machine learning, and our community of clients, we are in the unique position of being able to help any organization understand their website activity landscape – and how to deal with unwelcome visitors. Our community network effect provides each of our customers with a real-time database of known violators drawn from fingerprinting bots across Fortune 500 clients and hundreds of SMBs. This community effect ensures that whenever we identify a new bot, all of our clients are immediately protected – no matter whether those clients use our SaaS or on-premises solution.
Wondering who or what is coming to your website now? Distil Networks is offering a free threat analysis of your website security. Visit Distil Networks and enter the promo code TMCWebinar for your personalized report.
About the AuthorFollow on Twitter More Content by Courtney Brady