Last week I was at a marketing conference in San Francisco, where I met Elias Terman, VP of Marketing at Distil Networks. He had a booth set up outside the conference hall and I got to chatting with him about his service. I know, it sounds like a pretty standard conference conversation, but it got interesting when he started talking about some of the research his bot mitigation company had done about web scraping.
It’s one thing Priceline, Amazon, Expedia and a ton of other massive retail websites deal with: bot traffic scraping their sites to pull data. And it affects a large number of other sites too. Why this happens and what bots do with the data is the story though.
I sat down with him to get some more information, and go into the rabbit hole a bit more.
Clayton: So, how much of the traffic to people’s sites is bot traffic?
Elias: According to our Bad Bot Report, 20% of all web traffic is bad bots, but the amount of bad bot traffic on your site will vary, and can be much higher if your website has features like a login page, payment processor, web forms, or proprietary content or pricing information. And even good bots, another 19% of overall web traffic, skews website analytics.
Clayton: How does it affect businesses?
Elias: Bots rarely make the headlines, but they’re the key culprits behind web scraping, competitive data mining, account takeovers, transaction fraud, unauthorized vulnerability scans, spam, click fraud, denial of service, skewed analytics, and API abuse.