The Rise of the Bot

November 4, 2016

Note: This is a guest post from Network Computing, a resource for enterprise product reviews, comparisons, analysis and advice for IT professionals.

Advanced Persistent Bots use deception and evasion to avoid detection. Distil Networks CEO Rami Essaid explains how to protect your web application infrastructure against automated attacks

IT teams have been fighting malevolent actors flooding their IT infrastructure since the beginning of Internet time. There are a number of tools that can help manage these issues and many enterprises employ distributed denial of service (DDoS) mitigation providers, perimeter firewall and Web Application Firewalls (WAFs). Bot mitigation should also be deployed as an integral part of this web application security stack.

With a DDoS attack the website is flooded, preventing access to its services. It's a layer 3 attack and easy to spot; it can flood your upstream infrastructure to the point where the packets never arrive at the web server. For defence DDoS protection solutions use scrubbing centres filled with blunt blocking appliances and large pipes to handle the volume. Alongside this, firewalls use Access Control Lists (ACLs) and rules to stop network layer assaults.

In contrast, bot attacks fly under the radar and aren't limited to volumetric attacks. Instead, they programmatically abuse and misuse websites. When the abuse intensifies, website slowdown or downtime occurs: in other words, an application denial of service. This happens at layer seven, so you won't notice it on your firewall and your load balancer will be just fine. It's the web application and backend that keels over.

For example, if traffic to your homepage triples, you can probably handle it. The same amount of traffic to your shopping cart page will be much more painful as the web application will start sending requests to the components involved in a transaction. Examples include contacting the inventory database, connecting with payment processing and fraud tools, and using analysis tools for cross-sell opportunities, and so it doesn't take much traffic for an Application DoS attack to take hold.

Bots attacking the application layer can result in problems that go beyond slowdowns and downtime. This has caused the application security community OWASP to create a 75-page handbook that identifies 20 common threats observed in the wild.

Bots are now significant culprits behind web scraping, account takeovers, transaction fraud, vulnerability scanning, spam, digital ad fraud, API abuse, skewed analytics and, yes, denial of service. These aren't the simple bots of yore.

Advanced persistent bots (APBs) comprise as much as 88 per cent of all bad bot traffic. APBs have several advanced capabilities that include mimicking human behaviour, browser automation, loading JavaScript and external resources, and cookie support.

These bots are harder to identify and block and their persistency comes from their process for evading detection using such tactics as dynamic user agent and header rotation, Tor networks and peer-to-peer proxies to obfuscate their origins and distribute attacks over hundreds of thousands of IP addresses.

For example, an APB could use 1000 IP addresses, making one request each, instead of one IP address to make 1000 requests. This renders IP-centric defences impotent.

Advanced bot detection and mitigation techniques must be deployed in conjunction with traditional tools like DDoS mitigation and firewalls. To protect against bots consider new approaches that leverage techniques including real-time analysis, device fingerprinting, honeypot injection and machine learning, to identify and block malicious automation.

Sharing data about new attacks is also important to spot and stop bots. Threat intelligence from a community-sourced violator's database that updates in real time will help you stay one step ahead of those who would do you harm. Most importantly, any approach to defending against bots needs to be proactive. If you have to write a rule in a firewall or WAF it's likely to be far too late.

Previous Article
Shortening the Rust Edit/Feedback Cycle
Shortening the Rust Edit/Feedback Cycle

Casey Robinson discusses one of the most common frustrations when working with rust, the speed of the compi...

Next Article
Infographic: The Evolution of Hi-Def Device Fingerprinting
Infographic: The Evolution of Hi-Def Device Fingerprinting

Distil Networks now uses hi-def device fingerprinting to eliminate and identify bad bots across multiple de...