Tracking Good Bots and Crawlers with Distil

May 2, 2016

In this tutorial video, you'll learn how to track good bots or crawlers accessing your site. 

Although your site lies prey to any number of bad bots—ones that attempt scraping, brute force attacks, competitive data mining, brownouts, account hijacking, and more, the good news is that not all bots are bad. 

Good bots include search engine crawlers from Google, Bingbot, Yahoo Slurp, Baidu, Lycos, Yandex, et al. Like those, social media crawlers from Facebook, LinkedIn, Twitter, and Google+ help humans find and access your site.

By using the good bot report, you can understand the traffic patterns and origin sources of the benevolent bots visiting your site. This information may be helpful in optimizing your website to cater to these bots if needed.

Previous Video
Classifying Bots by Organization with Distil
Classifying Bots by Organization with Distil

In this tutorial video, you'll learn how to use Distil's Threats by Organization report. One way to class...

Next Video
Creating Wildcard Domains And Subdomains with Distil
Creating Wildcard Domains And Subdomains with Distil

Learn about Distil Networks' Wildcard and Subdomains. Protect a large volume of domains or subdomains with ...