Bad Bot Report 2019: The Bot Arms Race Continues

April 16, 2019 Edward Roberts

Today, Distil Networks released its sixth annual report into the trends in the bad bot landscape. Titled the “Bad Bot Report 2019: The Bot Arms Race Continues”, this publication is the industry's leading research into bad bots and their effect on businesses.  

Key Findings from the 2019 Bad Bot Report:

  • In 2018, bad bots accounted for 1 in 5 website requests (20.4 percent of web traffic). Good bots decreased slightly to make up 17.5 percent of traffic.
  • 73.6 percent of bad bots are classified as Advanced Persistent Bots (APBs), which are characterized by their ability to cycle through random IP addresses, enter through anonymous proxies, change their identities, and mimic human behavior.
  • Nearly 50 percent (49.9 percent) of bad bots report their user agent as Chrome. Mobile browsers, such as Safari Mobile, Android and Opera increased from 10.4 percent last year to 13.9 percent.
  • Amazon is the leading ISP for originating bad bot traffic. In 2018, 18 percent of bad bot traffic originated from Amazon compared with 10.62 percent the previous year.
  • Despite the fact that 53.4 percent of bot traffic originates from the United States, Russia and Ukraine combined make up nearly half (48.2 percent) of country-specific IP block requests.

Increasingly, the bot problem is an arms race. Bots are mimicking real human workflows across web applications to “behave” like real users. They are obfuscating their activity by reverse engineering detection systems. Advanced attackers now show definitive behavior that they know about the technology they’re trying to defeat, and they’re continuously learning how to adapt their tactics.

Targeted advanced persistent bots (APBs)include:

  • Dedicated scraping tools targeted at avoiding digital rights management processes to gain free access to online books
  • Systematic gift card balance checking  
  • Airline market intelligence operators running a dedicated scraping team unscrupulously attacking airlines in high volume and re-selling the gathered data.
  • Airlines see dedicated bots rotating from attacking the website to mobile app to API and back ad nauseum to avoid any blocks impeding their goal.

This year’s report provides a comprehensive breakdown of some of the top industries impacted by bots and the specific challenges they face. Key findings include:

Industry Percentage of Bad Bot Traffic Type of Attacks
Financial Services 42.2 percent Credential stuffing to access or take over user accounts
Ticketing 39.3 percent Scalping bots, seat inventory checkers, credential stuffing
Education 37.9 percent Scraping for research papers, class inventory and user account access
Government 29.9 percent Voter registration account interference, scraping business registration listings
Gambling and Gaming 25.9 percent Scraping ever-changing betting lines, account takeover seeking loyalty points
Airlines 25.9 percent Scraping pricing information, account takeover to empty airline mile balances
Ecommerce 18 percent Price scraping, content scraping, account takeovers, credit card fraud and gift card abuse

About the Author

Edward Roberts

Edward Roberts leads Product Marketing and has over twenty years experience in technology marketing. Previously he worked for Juniper Networks, heading up Product Marketing for the Counter Security team. Before that he ran marketing for Mykonos Software, a web security company.

More Content by Edward Roberts
Previous Article
Gartner Market Guide for Online Fraud Detection
Gartner Market Guide for Online Fraud Detection

Gartner used to publish a Gartner Magic Quadrant Fraud, specifically for web fraud detection, but the onlin...

Next Article
Distil Networks - The Number 1 Bot Manager
Distil Networks - The Number 1 Bot Manager

In Forrester’s evaluation of the emerging bot management market, they identified the 12 most significant pr...