Distil Networks Partners with Online Trust Alliance (OTA) and Finds Advanced Persistent Bots Still Massive Problem for Websites

June 20, 2017 Peter Zavlaris

Partnering To Fight Bad Bots

Federal Websites Have The Hardest Time Stopping Advanced Persistent Bots

The internet’s top websites are powerless against advanced persistent bots (APBs). The Online Trust Alliance’s (OTAs) Online Trust Audit on the top 1,000 websites in retail, banking, consumer services, federal government, news media, internet service providers, and OTA members shows that an average of 16 percent of the websites can thwart simple bot attacks. For the second year in a row APBs had an incredible success rate, nearly every website (95%) failed to stop them.

Distil tested each website included in the OTA Online Trust Audit on their ability to defend against bot attacks of different sophistication levels, including:

  • Sophisticated Bots – “Low-and-slow” bots coming in from dozens of IP addresses, using browser automation tools that can hold cookies and maintain state
  • Moderate Bots – Bots with normal browser user agents and headers, coming in slowly from one IP
  • Simple Bots – Non-browser user agents and headers, coming in fast from one IP
  • Crude Bots – Basic script that behaves like a bot, coming fast from one IP address

Below is a table showing 2017 bot detection rates by sector. While none of the industries fared particularly well against moderate and sophisticated bots, federal sites were the worst of the bunch. Against moderate bad bots, federal sites only had a 2% detection rate and a 1% detection rate against sophisticated bots.

Federal sites have become well-known bad bot targets. In February of 2016, cyber thieves used personal information obtained elsewhere and a bot to grab electronic filing PINs from the IRS website. The PINs are a way for taxpayers to verify their identities when filing returns online. Cyber thieves need them to file tax returns, under stolen identities, to steal refunds. These criminals were able to collect 101,000 PINs before they were shut down.

According to our recent Bad Bot Report 2017, 75% of bad bots found in the wild are APBs. Our report also showed that one in ten bad bots is mobile. Bad bot operators are turning their attention to mobile (and web) API endpoints. Sounding the alarm bell, OWASP has now added “Underprotected APIs” to the OWASP Top Ten.

APBs can mimic human behavior, load JavaScript and external assets, tamper with cookies, perform browser automation, and spoof IP addresses and user agents. They evade detection with tactics like dynamic IP rotation from huge pools of IP addresses, the use of Tor networks and peer to peer proxies to obfuscate their origins, and implement distributed attacks over hundreds of thousands of IP addresses.

Nefarious competitors, hackers, and fraudsters use bots to scrape data, takeover accounts, commit fraud, and deny service to users. The goals of the attackers vary from account takeover, credit card fraud, competitive data mining, click fraud, and sabotage. The impacts on the business range from poor customer experiences, website downtime, SEO penalties, lowered conversion rates, and increased fraud scores.

To minimize the risk APBs pose, enterprises must utilize real time analysis of APBs with fingerprinting, honeypot traps, machine learning, and behavioral modeling to augment their current risk mitigation solutions such as DDoS appliances and Web Application Firewalls (WAFs). A comprehensive APB mitigation solution is critical to assure the highest level of online trust is maintained.


About the Author

Peter Zavlaris

Peter Zavlaris weighs in on various topics around bot mitigation, bot defense sharing white papers, videos and other resources on the topic.

More Content by Peter Zavlaris
Previous Article
Rami Essaid CEO of Distil Networks Interviewed on BizTalkRadio’s “The Big Biz Show”
Rami Essaid CEO of Distil Networks Interviewed on BizTalkRadio’s “The Big Biz Show”

Rami Essaid CEO of Distil Networks Interviewed on BizTalkRadio’s “The Big Biz Show”.

Next Article
Web Scraping Bots Flood Real Estate Sites
Web Scraping Bots Flood Real Estate Sites

Get an inside look at just how much the real estate industry deals with bots web scraping their sites. Web ...