Online Directory Cuts Pageviews in Half by Blocking Bots | CrunchBase Case Study

September 1, 2015 Distil Networks

“Distil Networks’ service seemed like the stuff of magic. It has proven to be so.”

-- Kurt Freytag, Head of Product, CrunchBase

Overview

CrunchBase is the leading platform to discover innovative companies and the people behind them. It delivers market insights to millions of users and businesses around the world. The CrunchBase Dataset is constantly expanding through contributions from its community of users, investment firms, and network of global partners. CrunchBase accelerates innovation by bringing together data on companies and the people behind them.

After completing the buildout of their 2.0 site, CrunchBase noticed oddly shaped traffic, along with random spikes, all of which were putting a tremendous strain on their servers. Freytag investigated and quickly came to the conclusion that they were getting hit hard by bots crawling their site.

“We looked for what was causing the hurt on our servers and discovered that a relatively small number of entities were aggressively crawling our site in a structured pattern,” said Freytag.

Challenges

  • Random spikes in bot traffic were causing brownouts
  • IT lacked visibility, tools and staff to combat sophisticated bot attacks
  • Top priority was blocking bots without impacting users or ad deliveries
  • Leadership knew content theft could disrupt long term business model

Deployment

Distil Cloud CDN

Results

  • Transformed server load to pure user load -- well shaped and predictable
  • Blocked bot attacks and stabilized web infrastructure with minimal effort
  • Cut pageviews in half without impacting users or ad deliveries
  • Protected user generated content, safeguarding long term business model
  • Generated almost 1,000 prospects for CrunchBase’s commercial content license

About the Author

Distil Networks, the global leader in bot detection and mitigation, is the first easy and accurate way to identify and police malicious website traffic, blocking 99.9% of bad bots without impacting legitimate users. Distil protects against web scraping, competitive data mining, account hijacking, form spam and click fraud while slashing the high tax that bots place on your internal teams and web infrastructure.

Follow on Twitter More Content by Distil Networks
Previous Article
How Publishers Can Protect Themselves from Bots That Steal Content
How Publishers Can Protect Themselves from Bots That Steal Content

Rami Essaid, CEO of Distil Networks, discusses content theft, the digital publishing industry, and how to b...

Next Article
Distil Networks Releases New Data on The State of Digital Advertising Fraud
Distil Networks Releases New Data on The State of Digital Advertising Fraud

Findings show large majority of publishers (75 percent) and advertisers (59 percent) are either unable, or ...