For The Non-Techy: 3 Signs You Might Have A Bot Problem

August 14, 2014 Courtney Brady

bot
Photo credit 

Let’s say you own a business with a website where you promote your services, write original content about things that matter to your customers and probably also sell your products as well. You’ve spent a lot of time working on that site, getting it perfect and marketing your business to get quality traffic to your site that will bring the money. You also may or may not have heard the term “malicious bot” here and there but have no idea what it means or why it might be important to you. Today I’m going to tell you what those malicious bots are, why they matter to you, and how you can get some bot detection help on your own site.

What are malicious bots?

“Bots” is the term used to refer to a piece of scripted software that hackers write. This software is incredibly cheap to create and deploy, and once sent out into the internet world, that “bot” can do pretty much whatever its creator wants it to do unless a site has built in protection to stop it from entering. Most often bots are sent to gather information, and not just financial information. Many bots will also steal pricing information, product descriptions and SEO keyword placements to aid anybody out there who might want to start a site similar to yours without having to do all the work to make it look great. Bots will also spam your site if you have submission forms for things like newsletters or updates. And lastly, because bots are so prevalent on the internet (sometimes accounting for more than 60% of site traffic) they can actually slow your site down, making the quality of the experience for humans visiting your webpage extremely frustrating and difficult.

Why should I care about bot detection?

Well, apart from the above reasons, maybe a better question to ask yourself is “Do I care whether someone else is capitalizing off of my hard work and stealing away my customers?” because that is what many bots do. In addition, there are bots that will steal financial information from customers, use your site to test customer username and password info, and commit identity theft. It’s a pretty safe bet that any and every website out there is targeted by bots for some reason or another. Websites typically aren’t built to protect themselves from bots, which makes all websites inherently vulnerable. So, if you’re operating an online business you are probably being targeted.

So what are some ways I can tell if I’ve got a bot problem?

If you are a tech savvy person, the first thing you would do for bot detection is check to see is if there is a large disparity between google analytics data and your server access logs. You can also look for things like suspicious IP addresses (i.e. hosting providers like Amazon EC2 & Rackspace including suspicious countries), user agents that don’t match their IP addresses (i.e. Google crawlers that don’t originate from Google IP’s), and outdated user agents (i.e. Internet Explorer version 8 or lower, Firefox 3.X, etc).  If you are not a tech savvy person on the other hand, here is a short list of things that might be a sign of a bot causing problems for your site and business.

1) Your site is going down frequently. If you’ve had frequent issues with your site slowing down or crashing completely, you may have a bot problem. Bots move fast across websites, and do so in hoards. So you get a lot of server requests per second, which can overload the system and cause a major slowdown in loading times. The result is that you end up spending more money on server costs for traffic that doesn’t translate into any benefit for your business. In addition, any humans who try to visit your site or make a purchase at a time when the site slows down will typically leave and take their business somewhere else.

2) When you filter bot traffic on Google Analytics your numbers drop. First it is important to note that Google Analytics only tracks events that load javascript. Since most bots do not load javascript it is impossible to know the full extent of your bot problem from Google Analytics. However, Google Analytics recently introduced a way to filter out all known bot and crawler traffic from your analytics numbers. Simply go to your Google Analytics Admin → View Settings → and check ‘Exclude all hits from known bots and spiders’. This will exclude bot traffic from your future analytics. If you see a noticeable drop, you may have a bot problem. The problem with this method is that you can not analyze bad bot traffic vs real traffic in the same time frame.

3) Your site content is showing up elsewhere on the internet. Occasionally people may receive notifications from third parties that their content has shown up somewhere it isn’t supposed to be on the internet. If you keep any kind of worthwhile content on your site, it’s possible that it’s been copied and posted elsewhere without your permission or knowledge. The site Copyscape can help you to determine whether or not any of your site information has been posted elsewhere on the internet. If you enter in the URL of a page from your website into their search field, they will return any pages that have high percentage matches to the content on the referenced page. While this isn’t a sure fire way of telling whether your content has been copied, it can potentially give you some idea.

If you’re interested in delving further to see exactly how much bot traffic is getting in the way of your online business, Distil Networks offers its bot blocking services as well. Our portal allows you to see exactly how much and what type of bot traffic your site is getting and take any actions to prevent them from causing damage to your business. Most importantly, Distil also allows you to differentiate between the “good” and the “bad” bots; since many search engines use a version of bots to crawl pages for their search results, you want to be sure that you have the ability to let them access your site.

Sign-up here for your free 20 minute private demo of our service and portal features to get an idea of what an in-depth view of your bot traffic can reveal about your site and who’s visiting it. 

About the Author

Courtney Brady

Courtney Brady is the Director of Marketing at Distil Networks. She comes to Distil Networks from a variety of start-up companies, routed in SaaS and DaaS solutions. Formerly the global communications manager at multiple companies, Courtney is responsible for developing the company’s marketing strategy and branding campaign.

Follow on Twitter More Content by Courtney Brady
Previous Article
Distil’s ALS Ice Bucket Challenge
Distil’s ALS Ice Bucket Challenge

Distil Networks accepted Dyn Inc.'s nomination for the ice bucket challenge to support the fight against AL...

Next Article
4 Ways Web Bots Negatively Affect Your KPIs
4 Ways Web Bots Negatively Affect Your KPIs

As web bots increase in prevalence, the efficacy of various KPIs and analytic points has been affected. Bad...