Rami Essaid, Distil’s CEO and Co-founder, was recently invited to speak at the International Classified Media Association (ICMA) conference in Madrid, Spain, about the dangers posed by non-human site traffic to online classified ad sites and marketplaces. What follows is a summary of the presentation, which you can watch online here:
Distil Networks works with some of the world's largest classified ad sites and marketplaces, like Airbnb and StubHub, this gives us great insight into the ways bots threaten marketplaces and helps us help our clients combat those threats.
Who’s really visiting your website?
Our research tells us that only 40% of typical site traffic is human. Another 35% is good bots – Google, Bing, and the like. The remaining 25% is non-human traffic, and they’re hitting your site with a trifecta of damaging activity:
Massive numbers of page requests cause slowdowns, downtime, and poor customer experiences
Content theft and SEO attacks lead to a decline in useful traffic
Resale of stolen leads and data drives lost revenue and an eroding customer base
The most obvious problem – web scraping
Put simply, web scraping is used by malicious actors to duplicate and misuse your proprietary content. The problem is growing because it’s cheaper than ever to get into the game and harder than ever to identify and stop attacks. Thanks to cloud services and P2P networks, hunting down scrapers by blocking specific IP addresses has become an endless game of whack-a-mole.
Bots-as-a-service is now a solution category. There’s a company called Import.io that will do all the heavy lifting on web scraping for you, or you can hire someone for as little as $100 to copy an entire site, including backend databases. For $200, they'll take that data and turn it into a classified site. For $300, they’ll clone an entire online classified site or marketplace. The cost is nothing compared with the value of the data. And your ability to hit back through the courts is limited.
Copying content is just the beginning
In a marketplace, it's not just SEO that drives traffic to your site – people are looking for the lowest price. If your competitors can undercut your pricing by even 1%, or detect when you’re out of inventory on a particular item, they're going to steal those prospects.
Then there’s negative SEO attacks. Some bad guys will take content not for their own site but to duplicate it around the web and dilute your uniqueness. When Google or other search engines index this duplicate content, your website’s page rankings plummet.
Bots will skew your analytics, too. If more than 50% of your traffic is from bots, but you can’t tell which 50%, then how can you tell what the real users are doing? You can’t. While it’s true that blocking non-human traffic like bad bots (and the garbage pageviews they create) will lower your absolute number of pageviews, your conversion rates will go up significantly, because there is an actual potential purchaser behind every hit you’re getting.
And then there’s the security issue
For hackers, bots are game changing; they provide hackers access to much greater scalability and automation for their operations to bypass your web application security. All today’s hacker needs for a brute force account takeover is an army of bots and a list of usernames and passwords – of which there are billions floating around as a result of recent breaches like Ashley Madison and Target. Since most consumers use the same credentials across many websites, all a hacker has to do is take a stolen password list, upload it to a bot, and then try the credentials in rapid succession against many websites until one works – then it’s off to the races.
The same thing happens with credit cards. When credit card information is stolen, bots go carding – testing those different credit cards on a lot of different sites in search of valid cards which then yield opportunities for fraudulent transactions. If you can eliminate these bot transactions, you’ll automatically reduce fraud and boost your bottom line.
The StubHub story
StubHub, an eBay company, is the largest secondary ticket marketplace in the world. They specialize in event ticket resale, which means their traffic volume is huge and transaction speeds are lightning-fast.
StubHub had a big bot problem. Bots were taking over accounts. Competitors were using bots to monitor Stubhub’s pricing and inventory and using the data they garnered to undermine their pricing strategy in real time. These bots were so aggressive that the random spikes in their traffic caused unpredictable bursts in resource utilization. By adopting dedicated bot management strategies, StubHub was able to understand the full gamut of traffic coming to their sites and, most importantly, block any visitor that was neither human nor a good bot. The result was improved SEO, increased human traffic, more robust site security, and no more account fraud.
Want to keep the bad guys out of your online classified site or marketplace?
Get a true picture of your site traffic with a free no-strings trial of Distil Networks’ solution at www.distilnetworks.com/trial
About the Author
Orion Cassetto joined Distil Networks as Director of Product Marketing in 2015, bringing with him nearly a decade of experience in the Cyber Security industry. His strengths include competitive strategy, positioning, and messaging for web application security and SaaS-based security solutions.More Content by Orion Cassetto