Partnering To Fight Bad Bots
Federal Websites Have The Hardest Time Stopping Advanced Persistent Bots
The internet’s top websites are powerless against advanced persistent bots (APBs). The Online Trust Alliance’s (OTAs) Online Trust Audit on the top 1,000 websites in retail, banking, consumer services, federal government, news media, internet service providers, and OTA members shows that an average of 16 percent of the websites can thwart simple bot attacks. For the second year in a row APBs had an incredible success rate, nearly every website (95%) failed to stop them.
Distil tested each website included in the OTA Online Trust Audit on their ability to defend against bot attacks of different sophistication levels, including:
- Sophisticated Bots – “Low-and-slow” bots coming in from dozens of IP addresses, using browser automation tools that can hold cookies and maintain state
- Moderate Bots – Bots with normal browser user agents and headers, coming in slowly from one IP
- Simple Bots – Non-browser user agents and headers, coming in fast from one IP
- Crude Bots – Basic script that behaves like a bot, coming fast from one IP address
Below is a table showing 2017 bot detection rates by sector. While none of the industries fared particularly well against moderate and sophisticated bots, federal sites were the worst of the bunch. Against moderate bad bots, federal sites only had a 2% detection rate and a 1% detection rate against sophisticated bots.
Federal sites have become well-known bad bot targets. In February of 2016, cyber thieves used personal information obtained elsewhere and a bot to grab electronic filing PINs from the IRS website. The PINs are a way for taxpayers to verify their identities when filing returns online. Cyber thieves need them to file tax returns, under stolen identities, to steal refunds. These criminals were able to collect 101,000 PINs before they were shut down.
According to our recent Bad Bot Report 2017, 75% of bad bots found in the wild are APBs. Our report also showed that one in ten bad bots is mobile. Bad bot operators are turning their attention to mobile (and web) API endpoints. Sounding the alarm bell, OWASP has now added “Underprotected APIs” to the OWASP Top Ten.
Nefarious competitors, hackers, and fraudsters use bots to scrape data, takeover accounts, commit fraud, and deny service to users. The goals of the attackers vary from account takeover, credit card fraud, competitive data mining, click fraud, and sabotage. The impacts on the business range from poor customer experiences, website downtime, SEO penalties, lowered conversion rates, and increased fraud scores.
To minimize the risk APBs pose, enterprises must utilize real time analysis of APBs with fingerprinting, honeypot traps, machine learning, and behavioral modeling to augment their current risk mitigation solutions such as DDoS appliances and Web Application Firewalls (WAFs). A comprehensive APB mitigation solution is critical to assure the highest level of online trust is maintained.
About the Author
Peter Zavlaris weighs in on various topics around bot mitigation, bot defense sharing white papers, videos and other resources on the topic.More Content by Peter Zavlaris