Human traffic on the Web dropped from 55% to 41% over the past year, mostly due to "good bots" reflecting advanced marketing services from social networks.
There are good bots and there are bad bots. Should somebody tell you that the percentage of bad bots is on the decline, they won't be lying. The fact is, though, that the number of good bots filling cyberspace are on a rocket-like trajectory. So yes, bad bots account for a smaller share of traffic on the Web, but there are still lots more of them this year than than were last year.
Many more interesting bits of robot reality emerge from the “2015 Bot Landscape Report” published by Distil Networks, which makes its money isolating the good from the bad bots for its blue-chip publisher and advertiser clientele. First a definition: A good bot is one that adds value to a publisher or advertiser and abides by the robot.txt industry opt-in standard. A bad bot is just a bad guy masquerading as a person and scraping data, content, and eventually cash from Internet players. An ad targeting tool is a good bot; fake publishers loading up a hundred screens behind that of an actual publisher and stealing ads is a bad bot.
Distil Networks' analysis, which covers hundreds of customers,17 data centers, and some 23 billion bad bot threats finds that good bots nearly doubled their Web presence over the past year. Bad bots' share of traffic dipped a single point to 23%, but good bots added 15 points to rise to 36% of traffic. The sheer magnitude of this shift is best borne out by the fact that most of that share transfer came from human traffic, which declined from 55% to 41%.
“The reason that good bots increased so much is that social medial companies have doubled the number of bots they used in the past year,” says Distil Networks CEO Rami Essaid. “Google's rate of crawling went up 30% in the past two weeks because of the change to its algorithm.”
As for bad bots, more of them are working their way into the Web through cable and wireless networks. Comcast and Time Warner Cable are both among the Top 10 companies serving bad bot traffic, and together account for more than 11% of it. T-Mobile became the first wireless provider to crack the Top 20 at 1.22%. “We've been noticing a trend over the past three years where people are putting one and one together that, if it's coming from a data center, it might be fraudulent,” Essaid says. “So, the bad guys are reacting and that's why we're seeing more bad bots from cable networks. On wireless, they're pretending to be mobile browsers.”