Advanced Persistent Bots use deception and evasion to avoid detection. Distil Networks CEO Rami Essaid explains how to protect your web application infrastructure against automated attacks
IT teams have been fighting malevolent actors flooding their IT infrastructure since the beginning of Internet time. There are a number of tools that can help manage these issues and many enterprises employ distributed denial of service (DDoS) mitigation providers, perimeter firewall and Web Application Firewalls (WAFs). Bot mitigation should also be deployed as an integral part of this web application security stack.
DDOS VERSUS APPLICATION DOS
With a DDoS attack the website is flooded, preventing access to its services. It's a layer 3 attack and easy to spot; it can flood your upstream infrastructure to the point where the packets never arrive at the web server. For defence DDoS protection solutions use scrubbing centres filled with blunt blocking appliances and large pipes to handle the volume. Alongside this, firewalls use Access Control Lists (ACLs) and rules to stop network layer assaults.
In contrast, bot attacks fly under the radar and aren't limited to volumetric attacks. Instead, they programmatically abuse and misuse websites. When the abuse intensifies, website slowdown or downtime occurs: in other words, an application denial of service. This happens at layer seven, so you won't notice it on your firewall and your load balancer will be just fine. It's the web application and backend that keels over.
For example, if traffic to your homepage triples, you can probably handle it. The same amount of traffic to your shopping cart page will be much more painful as the web application will start sending requests to the components involved in a transaction. Examples include contacting the inventory database, connecting with payment processing and fraud tools, and using analysis tools for cross-sell opportunities, and so it doesn't take much traffic for an Application DoS attack to take hold.
Bots attacking the application layer can result in problems that go beyond slowdowns and downtime. This has caused the application security community OWASP to create a 75-page handbook that identifies 20 common threats observed in the wild.
Bots are now significant culprits behind web scraping, account takeovers, transaction fraud, vulnerability scanning, spam, digital ad fraud, API abuse, skewed analytics and, yes, denial of service. These aren't the simple bots of yore.
THE RISE OF ADVANCED PERSISTENT BOTS
These bots are harder to identify and block and their persistency comes from their process for evading detection using such tactics as dynamic user agent and header rotation, Tor networks and peer-to-peer proxies to obfuscate their origins and distribute attacks over hundreds of thousands of IP addresses.
For example, an APB could use 1000 IP addresses, making one request each, instead of one IP address to make 1000 requests. This renders IP-centric defences impotent.
COMPLETING THE WEB APPLICATION SECURITY STACK
Advanced bot detection and mitigation techniques must be deployed in conjunction with traditional tools like DDoS mitigation and firewalls. To protect against bots consider new approaches that leverage techniques including real-time analysis, device fingerprinting, honeypot injection and machine learning, to identify and block malicious automation.
Sharing data about new attacks is also important to spot and stop bots. Threat intelligence from a community-sourced violator's database that updates in real time will help you stay one step ahead of those who would do you harm. Most importantly, any approach to defending against bots needs to be proactive. If you have to write a rule in a firewall or WAF it's likely to be far too late.