There has been much discussion among Bot Defense Council members about the value – or lack thereof – of IP blacklists as part of a sound security strategy. IP blacklists are a staple in the security world, appearing in firewalls, IPS, web application firewalls, fraud prevention, bot mitigation and more. It’s how many organizations start their security efforts. First, extract from your weblogs a list of IP addresses that are performing malicious activity or employing bad bots and upload it to your web application firewall, IPS, Firewall, etc., and even share it across the infosecurity community.
The challenge of IP access control list (ACL) maintenance
Trouble is, how do you keep a list current and manageable? It’s easy to see how a list of suspect IP addresses is going to get very big very quickly. Here are some of the challenges which make ACLs difficult to maintain:
When less than half the traffic that’s coming to your website is human, there are a lot of IP addresses to keep track of. Plus, IP addresses don’t stay under the control of one individual or entity for very long. Organizations go out of business. Companies trade blocks of IP addresses. Additionally anyone can rent space in Amazon or Google cloud and use the IP address blocks that come with that space, which allows hackers and legitimate users to quickly obtain new IP addresses, and use them for short periods of time.
It gets even more complicated if you decide to block IP addresses based on geolocation, especially if you do business on an international scale. Can you really be sure the IP address that was being used by a North Korean hacker yesterday isn’t being used by a genuine customer in South Korea today?
Anonymous Traffic Sources
In a post-Snowden world, the use of traffic origin obfuscation has become more popular than it was in the past. According to a study conducted by the Pew Research Center, roughly 9% of Americans have adopted sophisticated steps to shield their information such as using a Tor network, using a proxy server, or using a VPN to obscure origin IP Addresses. The output of these techniques is an anonymous or altered traffic source, which of course doesn’t lend itself well to ACL use.
The bad guys also employ these obfuscation tactics. They are out there routing traffic through Tor and other darknets, continuously randomizing, picking up and discarding batches of IP addresses. Why? Because they know IP blocking is a frequent choice for a first line of defense. Even IP intelligence solutions can’t agree on who owns what IP addresses at any particular moment in time.
Botnets are becoming more and more prevalent inside home computers and residential networks and ISPs routinely reassign residential IP address blocks to protect their own infrastructures. There’s a VPN called Hola that’s distributed across residential IPs and runs malware on the back end, and it’s not the only one.
And what happens to all those legitimate users who ended up with IP addresses that are on your blacklist? Those people now have no way of getting in touch with you because you’ve banished them from ever making it past your firewall. You don’t get their emails and they can’t access your site.
Given all of these challenges, does that mean ACLs are obsolete? We feel they aren’t. We view them as an important part of a sound security strategy, but one that could use a facelift. To that end, we suggest a couple of slight tweaks.
ACL self-maintenance through deprecation
One way to address the problem of stale or outdated control lists – and we think it makes a lot of sense - is to develop policies that enable access control lists to deprecate over time, based around the idea of "if this violation occurs, then block that address for X amount of time”. Repeat offenders would get hit with longer time-outs since they have repeatedly proven themselves to be untrustworthy.
Dynamic look ups to ensure freshness
We also recommend refining this approach by NOT converting organization or country blocks to IPs and uploading those IP lists. Because of the ever-changing relationship between IP addresses and users, you need to be doing dynamic look-ups – it’s the only way to avoid the problem of North Korean hacker versus South Korean customer noted above. Using this approach, if you add China to your blacklist, every time an IP comes in, we dynamically look it up and ask "is this China? No? You’re admitted. If it is, yes, block." The IP access control list updates dynamically every time this question is asked and answered.
The same principle applies to whitelisting of trusted sources like business partners, clients, and licensees. It’s a much more reliable way to determine who’s trustworthy than trying to maintain accurate status for a list of tens of thousands of IP addresses.
Bear in mind, too, that IP access control lists are just one element of a bot prevention strategy. I’ll be taking a look at other elements like fingerprinting and machine learning, in future posts.
Who’s on your site today?
In the meantime, to get a clearer picture of who’s visiting your website, you can request a no-strings trial of Distil’s multifaceted bot prevention at https://www.distilnetworks.com/trial.
About the Author
Rami Essaid is the Chief Product and Strategy Officer and Co-founder of Distil Networks, the first easy and accurate way to identify and police malicious website traffic, blocking 99.9% of bad bots without impacting legitimate users. With over 12 years in telecommunications, network security, and cloud infrastructure management, Rami continues to advise enterprise companies around the world, helping them embrace the cloud to improve their scalability and reliability while maintaining a high level of security.Follow on Twitter More Content by Rami Essaid