How Hayneedle Ended Their Bad Bot Problem with Distil Networks

October 26, 2016 Katherine Oberhofer

Are Bot Operators Eating Your Lunch on Your Ecommerce Site?

That’s the title we came up with for the latest webinar we put together for specialist media company Retail Touchpoints to highlight the particular dangers bots pose to online retailers. Distil CEO and co-founder Rami Essaid was joined in the presentation by, a leading online provider of home furnishings and décor - and a happy Distil Networks customer who’s not sharing his lunch with non-humans any longer!

Etailers are tempting targets

Retail Touchpoints’ Executive Editor Adam Blair began by setting the scene for attendees, noting that 30% of all ecommerce site visitors are estimated to be unsavory hackers and fraudsters, and that online retailers are particularly susceptible. Why? Because there are so many tempting targets and so many ways to go after them – price scraping, product matching, variation tracking, vulnerability targeting are just a few of the tactics employed by less-than-scrupulous competitors. Worse still, breaches like transaction fraud can endanger the overall security of your website, your customers, your brand, even your entire business.

Bots are everywhere …

The webinar kicked off with a quick summary of good (search engine crawling, system checkers) and bad (content theft, vulnerability scanning, fraud) bot activities. APIs are a double-edged sword – they can improve site experience for the customer, but they can also provide easy access to site content for bots.

Bots are no longer impacting only large operators like QVC, who lost $2M in revenue after a major bot attack.  Between 2014 and 2015 alone, small and medium ecommerce sites saw a 100% increase in bad bot traffic. Bottom line – every online business is vulnerable, because every site is a potential gateway to treasure troves of user name/password combinations that in turn can open the door to PII (personally identifiable information), credit card and banking information, and more. It’s a game anyone can get into in a few hours, thanks to cheap or free virtual servers, almost infinite bandwidth, and easy-to-use tools.

… And they’re getting smarter every day

88% of today’s bots are what we call Advanced Persistent Bots – bots that not only dynamically rotate through IP addresses (now 73% of all bots) and hide behind proxies but can also load JavaScript and other external resources. 39% can even convincingly mimic human behavior online by randomly pausing and moving around on a site – they’re so realistic, they don’t need to hide their identities any longer.

It’s this dedication to avoiding detection and evading traditional defenses like Web Application Firewalls that makes it so tough to keep bad bots out of a site. This was the challenge faced by Brian Gress, Director of IT Governance at Hayneedle.

Playing IP Whack-a-Mole No Longer Worked at Hayneedle

Brian’s security team had been able to more-or-less keep the bot problem under control until early 2015, when it was quickly becoming clear that IP blocking was no longer a viable strategy. Platforms like Amazon EC2 have millions of IP addresses available directly to them, and there’s no way for traditional tools to keep track of which ones are legitimate users, so simply blocking them is not a viable solution.

How bots can eat your lunch

The threats to Hayneedle’s business were multifaceted and very real – and all of them had the potential to significantly impact the company’s bottom line:

  • Decreased customer loyalty as a result of accidental blocking and site brownouts caused by application layer attacks
  • Lowered search engine rankings caused by stolen content (once duplicated by competitors, it’s no longer unique and search engine rankings go down)
  • Lost cross-sell and upsell opportunities as competitors undercut prices and take advantage of stock outages to steal customers
  • Skewed analytics make it impossible to conduct valid A/B testing and determine effective marketing tactics
  • Decreased competitive opportunities as IT focuses on bot fighting instead of business building – Brian’s “battle of the bots” was taking 20% of his time for only 30% effectiveness at best
  • Increased fraud through credit card abuse from CVV guessing games and the billion-plus user name/password combinations in the wild
  • Increased cost of security resources and fraud protection

But it’s not just competitors who are out to get you. Common penetration testing tactics can be turned against you through automated network mapping and vulnerability scanning, turning security into a game of cat and mouse to see who can find vulnerabilities first. Credit card fraud and malware delivered via comment spam can result in your company being blamed for problems that emanated from a completely different site!

Time to take back control

Things came to a head for Brian’s team when they were on call every weekend to tweak scripts and juggle firewall settings. The fight had become a major time sink that was not their core business and was not delivering any positive benefits to customers or the business – just a never-ending battle to counteract the negatives.

That’s when he set out to find a third-party solution that met his criteria:

  • An automated solution to free up the team’s time for business-positive activities
  • No negative impact on human visitors – no false positives
  • Self-tuning to defend against emerging threats
  • Crowd-sourced threat intelligence for optimum broad and deep defense
  • Seamless coexistence with Hayneedle’s complex technology stack
  • Ability to see and learn through a deeper understanding of traffic patterns – no “black boxes”

Fortunately, Distil was able to meet all of these criteria – and then some. In fact, the value of the last one was well and truly brought home to Brian when Hayneedle experienced a 10x traffic spike on August 7th of this year.

Fast, accurate, and informative protection

Because Distil sits at the edge of the stack, it was able to filter out the bot traffic right away, preventing an inevitable brownout. CAPTCHA serving to weed out the bots represented a whopping 73% of Hayneedle’s traffic that day – out of 17 million served, only 78 were solved, a tiny impact on legitimate customers and an acceptable trade-off against the protecting the site from an all-out bot attack.

What’s more, the visibility Brian gained into the traffic behavior on his site proved invaluable. It’s essential for him to understand not only what all his traffic is (regardless of what it says it is), but how his defenses impact legitimate customers and their shopping experience. That way, he can focus on making that experience better instead of preventing it from getting worse.

And the icing on that particular cake was much lower charges from the CDN provider than might otherwise have been the case, since CDNs charge by the amount of traffic served.

Lessons learned – and a happy customer

Brian’s advice for any company adopting bot protection? Start out slow. Understand what your traffic should look like, so you can work on strategies to deal with anomalies. Before you start CAPTCHAing, monitor your traffic to understand why your solution identifies particular traffic as a bot (is it not loading JavaScript? Spending too long in one session? Behaving in too organized – or too randomized – a fashion?). Review threats by organization so you can build a meaningful whitelist – and block data centers that have no legitimate relationship with your business. Apply defense measures appropriate to the threat – if a CVV test is occurring, just block it – there’s no need to use a CAPTCHA.

Distil has become a key factor in Hayneedle’s uptime strategy, clean analytics, fraud detection and prevention, and ability to focus on servicing customers, not bots. The detailed reporting even enabled Hayneedle to identify and resolve a problem with their CDN provider. But the best result of all was giving the team their lives back. Now they can go to baseball games with their kids – and leave their phones at home!

You can read more about Hayneedle’s experience with Distil Networks’ solution in this case study.

Concerned about your site’s vulnerability?

Request a free assessment and analysis at

Read the Article

Previous Flipbook
Pet and Hobby Network integrates Distil with Verizon Edgecast and Eliminates Price Scraping | Case Study
Pet and Hobby Network integrates Distil with Verizon Edgecast and Eliminates Price Scraping | Case Study

Pet and Hobby Network was falling victim to price scraping across its many domains, leading to many lost sa...

Next Video
What is Web Scraping, Denial of Service & Skewing?
What is Web Scraping, Denial of Service & Skewing?

Distil Networks blocks every OWASP Automated Threat, including web scraping, denial of service, skewing, an...