Would you be willing to contribute a percentage of your website resources to help your competition? Does that sound crazy? Distil Networks discovered that a good portion of traffic to travel-related websites is from automated agents, screen scrapers, bots, or crawlers. These bots steal and use your resources against you… and this is a big problem for travel websites.
Don’t believe us, ask easyJet who has been out front on this issue. Earlier this month, Travolution published an article that discussed how EasyJet was under attack from screen-scrapers. In fact, prices presented to customers were as high as 60% more. This is not a small problem, affecting as many as 1,500 passengers per day, 7 days a week… well over half a million passengers per year. That’s a big problem and this is an example from only one airline.
How is this possible? A quick search of developer job postings for web scrapers, bot builders, etc will generate a long list of results. There are indeed many job seekers and freelancers boasting of skills in automating the completion of forms (form SPAM) and web scraping (content/data theft). When you search online for “web scrapers” you will find many solutions. Some software for scraping is sold preconfigured to crawl directories or other specific websites. Also there are Scraping as a Service providers.
Within the article it vaguely discusses the next steps that EasyJet will take – the problem is that they have to take all of these steps retroactively. Sure, it’s possible to take web scrapers to court – but the lengthy trial and legal fees for litigation are definitely going to add up. In addition, taking them to court does NOT stop all of the other possible hundreds of scrapers out there who will continuously try to steal their data – it’s just a vicious cycle unless the data owners (EasyJet) are able to stop the problem altogether.
Distil Networks’ specialty is in separating human web traffic from the bots and web scrapers – ultimately blocking those malicious website scrapers that are stealing your prices, images, and travel listings. The old adage is certainly true here: “An ounce of prevention is worth a pound of cure”. More often than not, our customers come to us after they’ve experienced a problem. And yes, we solve their problems moving forward, but it is far more cost effective to protect your websites before the damage occurs.
Upon seeing this article on easyJet, our team took a good look at our travel customers and that industry in general – and we saw some alarming numbers, like the number of uniquely identifiable “fingerprints” we found for web scrapers on travel web sites. We ultimately created an original infographic depicting our findings based on data from our own customer base. The infographic will be coming out next week, but I thought I’d leave you with a few teasers on some the things we were able to find out:
- How the industry has changed with new website aggregators (or meta searches)
- Just how much traffic across the travel industry is from bad bots
- Where the bots (scrapers) are coming from
- And much more
Feel free to contact us if you’d like to be included on the distribution list for the infographic.
About the AuthorFollow on Twitter More Content by Courtney Brady