How Bots and Scraping Affect Your Online Travel Site

December 28, 2015 Orion Cassetto

Online Travel Fraud

A couple of weeks ago, Travel Brand’s Digital Marketing Officer Rob Gennaro joined Orion Cassetto, Distil Networks’ Director of Product Marketing, on stage for a joint presentation at the 2015 PhocusWright Conference. The goal of the presentation was to bring the industry audience up to speed on today’s sophisticated bots and the damage they can inflict on online travel businesses.

Starting at the ‘Bot’﹘tom

So, what is a bot? A bot is essentially an automated program that runs on the Internet. It could be a simple command line script, a browser plug-in, or a browser automation tool that accurately mimics human behavior.  Depending on their intent, bots can be either benevolent or malicious. Good bots like Google, Yahoo, Baidu, and others perform helpful functions such as aiding visitors in finding your website, while bad bots perform malicious activities such as scanning for security vulnerabilities, stealing unique content, hijacking inventory and upsell opportunities, and committing fraud.

According to the 2015 Bad Bot Landscape Report by Distil Networks,  only around 40% of the visitors to the average travel website are human. The remaining 60% breaks down into 36% good bots and 23% bad.  The biggest problem with bots is that they’re enabling interaction with websites ﹘ for good or bad ﹘ on a massive scale.  When they throw massive numbers of requests at your site, they cause slowdowns and downtime that create a negative user experience. Once they start stealing content and duplicating it elsewhere, SEO rankings are impacted (search engines penalize duplicate content). Then your human traffic numbers begin to decline, taking your revenues with them.

Almost anyone can get into the web scraping game

It’s incredibly cheap and easy to get into web scraping especially in online travel. Apps are available online for little or no investment, and it’s simple to use cloud computing to spin up multiple instances of these apps, each of which becomes another bot.

Lazy would be attackers are in luck as there is an even easier way to get into the scraping game; botnets. Instead of needing to manage Amazon instances and install software, attackers can simply rent a botnet for a few hours off of the the dark web and point it at a target online travel website.  These turn key systems will then start scraping content – that’s all it takes.  

High return on investment motivates would be attackers

The logic is simple. You could spend years of effort and millions of dollars building a content-rich online travel website,  or you could steal it.

From airline tickets and hotel rooms, to user generated reviews and unique editorial content, regardless of the nature of a travel site, any unique content on a website could be stolen by bots. If a website is not specifically protected against web scraping, anyone can duplicate that content for next to nothing – no investment in research, infrastructure, personnel, or anything else necessary. That content can then be sold to a competitor,  or even used against you steal your organic search traffic.

When presented with these options, why would attackers pay to license an API or to legitimately acquire content when they can bypass the authorization process entirely?

Price scraping hijacks sales and damages your brand

Price scraping is particularly egregious in the travel world, where even a few dollars’ difference in pricing can determine who gets a sale. Once a sale has been hijacked, the hijacker can pick up all upselling and cross selling opportunities – hotel rooms, rental cars, insurance, etc. – which can represent up to $40 of additional revenue per transaction.  This represents a big chunk of the planned profit margins for many travel businesses.

Unfortunately, these unwanted middlemen also frequently cause friction with customers which affects their purchasing experience and overall satisfaction. One well publicized example is that of RyanAir, who has been in the news for its legal battles against aggregators like eDreams that are hooking into the company’s sales process without authorization.  In the case with eDreams, Ryanair claims that the eDreams published low price, non-existence fares with Ryanair’s name.

This was allegedly done by putting Ryanair’s name in the URL of these fares to attract customers to the eDreams site, even though these tickets were not actually available from Ryanair.  Ryanair argued that this impacted its customers’ perception of their brand because 82% of adults surveyed believed this bait-and-switch was Ryanair’s doing.

In the past, Ryanair has also complained of other poor user experiences caused by unauthorized aggregators who failed to accurately pass on important travel information such as special needs requests, web check-in information, flight updates and other essential information. When this information isn’t accurately relayed, miscommunications with customers such as missed flights ensue.

Online fraud targets travel loyalty programs

Loyalty programs, too, are a major target - mileage points quickly be converted to cash or used to purchase resaleable items like flights, cameras, even designer clothing. These loyalty programs are being attacked by bots performing what is known as a brute force attack, where an attacker uses a bot to systematically try millions of username/password combinations at victim sites until they get lucky and find one that works. As we all know, many people use the same credentials for multiple sites, so when a site like Ashley Madison gets hacked and 32 million credentials are suddenly available – well, you get the picture. To make matters worse, older mileage programs that use only a four-digit PIN are particularly vulnerable – that’s only 10,000 possible combinations, and a botnet can crack that in less than a minute.

Bots hit core business drivers

Like many online businesses, online travel sites are driven by customer loyalty that’s continually reinforced by predictable and high-quality customer experiences, and by customer satisfaction. Successful online business are also adept at capturing the opportunity  to insert relevant upsell opportunities at the time customers are ready to buy. Between these business drivers as well as fraud, negative SEO, brand impact, loss of customer loyalty, and the cost of the web infrastructure that’s serving only these bots,  it’s clear that these automated miscreants chip away at profit margins and resources of online travel sites.

Fighting back against the bots

Rob Gennaro at Red Tag has been on the receiving end of every possible bad bot experience. He’s seen slowdowns lasting up to an hour a day during peak times, while visitors got more and more frustrated, unable to ask questions or complete transactions. He’s seen countless instances of online fraud, and bots that can rip off his entire 300,000 page website in 15 minutes.

Red Tag’s engineers developed applications to block bots at the IP level, but the bots were cycling through IP addresses faster than the software could detect and block them, and the customer experience got worse and worse. Other home-grown solutions simply could not scale to Red Tag’s traffic levels, and caused as many site slowdowns as the bots themselves. It was only when Rob connected with Distil Networks that he saw how a new thought process and self-optimizing technology built on community-based intelligence, could really make a difference.

EasyJet also had many run-ins with eDreams over ticket price fee loading of up to 60% that impacted more than 300,000 customers over a six-month period; they too chose to resolve the situation with Distil Networks technology rather than follow RyanAir through the courts.

Join the conversation on Quora

In addition to writing this blog post, I’ve also answered a related question on Quora.  Feel free to read my answer there and join in the conversation. I’d be happy to answer any specific questions you have!

Want to dig deeper?

Check out this webinar we recently hosted with Skift for more in-depth info on bots and the travel industry; you can also learn more about how Red Tag fought back against bots in this case study.

To test out Distil Networks’ solution on your travel website - no strings, go to www.distilnetworks.com/trial.

About the Author

Orion Cassetto

Orion Cassetto joined Distil Networks as Director of Product Marketing in 2015, bringing with him nearly a decade of experience in the Cyber Security industry. His strengths include competitive strategy, positioning, and messaging for web application security and SaaS-based security solutions.

More Content by Orion Cassetto
Previous Article
A Great Offense Begins with an Exceptional Bot Defense Council
A Great Offense Begins with an Exceptional Bot Defense Council

The Bot Defense Council (BDC) is a group of IT practitioners and technically adept business leaders committ...

Next Article
The Best Piece of Security Advice for CISOs in 2016
The Best Piece of Security Advice for CISOs in 2016

Advice & Biggest Cybersecurity Threats for CISOs in 2016. Fourteen Industry experts weigh-in on how to miti...