Web Bots: What Every Website Owner Needs to Know

August 7, 2014 Courtney Brady

If you or your company operates a website or utilizes PPC or other methods of online advertising, then web bots and spiders are a cause of serious concern. Malicious bots can threaten the viability of your ad campaigns, affect your KPI tracking and analytics, and negatively impact your sales, conversions and overall customer experience. To get an idea of their true impact, let’s first discuss the kinds of web bots to which your site could fall victim.

Theft Bots

Theft bots aim to steal content, pricing, images and other information from sites on the web. Sometimes, they do this to sell the content to rival websites or content farms; at other times, they simply use it to post directly to their own sites. In some cases, a competitor may even use your data to get an inside look at your content and pricing, so they can get a leg up in the marketplace. In addition to scraping your content and data, theft bots can also harvest lists of email addresses for spam purposes.

In addition to these negative effects, theft bots can also cause SEO problems for site owners. If content is stolen and duplicated elsewhere on the web, it severely downgrades the original site’s search engine rankings and could even result in getting a site blacklisted.

A great example of a theft bot can be seen in a recent court case. Last year, the Metropolitan Regional Information Systems Inc., or MRIS, filed suit against the American Home Realty Network for copyright infringement. MRIS alleged that AHRN had used web bots to effectively steal content from various real estate MLS listing sites and then pass the information off as its own. While the case is still pending, MRIS was granted an injunction, which likely means that AHRN will be found guilty.

Fraud Bots

Fraud bots create fraudulent traffic by clicking on paid advertisements and display ads throughout the web, and sometimes they even post spam to websites, blogs and various web forms. These bots can create fraudulent accounts on databases or networks to gain access to directories of customer information. Fraud bots mean bad news all around for your site, your customers and your sales.

The biggest issue with fraud bots is the problem of click fraud. In fact, according to AdWeek, web bots steal an average of $6 billion from advertisers every year through fraudulent traffic and clicks. Furthermore, according to Medialink president Wenda Millard, about 25% of all traffic attracted by online ads is likely fraudulent. That means one out of every four clicks your site gets on its PPC or display ad campaigns could be a fake. When you consider the costs of those fraudulent clicks over a week, a month or even a year, it can result in substantial financial losses.

More on Bad Bots

In addition to the above repercussions, web bots also lower the quality of the user’s experience as a whole. Bots can cause slow moving, poorly performing websites; infect users’ devices, computers and mobile phones; and diminish the overall web-surfing experience. Businesses and website owners may see their bandwidth decreased, their KPIs and analytics become hard to decipher, and their intellectual property, content and data stolen.

The number of both fraud and theft bots has grown exponentially over the last few years. According to Distil Network’s most recent Bad Bot Landscape Report, the number of damaging web bots doubled in 2013. Bad bots accounted for just 12.25% of all web traffic in Q1, but climbed to a whopping 23.6% by the end of Q4. Furthermore, though many bad bots come from overseas, the Landscape Report reveals that more bad bots originate in the United States than any other country in the world.

On May 15, 2014, the Executive Director and President of the Online Trust Alliance (OTA), Craig Spiezle, testified to the U.S. Senate about the risks of malicious ads and web bots. OTA research reports that malicious advertising, or “malvertising,” rose by more than 200% in 2013. In addition to threatening the effectiveness of your website, malicious ads can be used to infect personal computers, steal personal information, or manipulate devices into becoming part of a DDoS attack.

Fortunately, there are technologies and strategies for blocking bots and protecting your website and your ads from fraud and theft. We’ll be discussing these strategies, as well as more on web bots, in an upcoming post, so stay tuned! In the meantime, take your traffic readings and PPC results with a grain of salt—you never know when web bots may strike. 

About the Author

Courtney Brady

Courtney Brady is the Director of Marketing at Distil Networks. She comes to Distil Networks from a variety of start-up companies, routed in SaaS and DaaS solutions. Formerly the global communications manager at multiple companies, Courtney is responsible for developing the company’s marketing strategy and branding campaign.

Follow on Twitter More Content by Courtney Brady
Previous Article
How Web Bots Affect Your KPIs and Your Bottom Line
How Web Bots Affect Your KPIs and Your Bottom Line

As web bots increase in prevalence, the efficacy of various KPIs and analytic points has been affected. Web...

Next Article
Your Options When Dealing with (Startup) Stock Options
Your Options When Dealing with (Startup) Stock Options

First in a series of posts on stock options and how they relate to early startup employees. This post cover...