What You Need to Know About Web Scraping: How to Understand, Identify, and Sometimes Stop

August 26, 2013 Courtney Brady

NB: This is a guest article by Rami Essaid, co-founder and CEO of Distil Networks on the Tnooz blog.

Here’s the thing about web scraping in the travel industry: everyone knows it exists, but few know the details.

Details like how does web scraping happen and how will I know? Is web scraping just part of doing business online, or can it be stopped? And lastly, if web scraping can be stopped, should it always be stopped?   

These questions and the challenge of web scraping are relevant to every player in the travel industry. Travel suppliers, OTAs and meta search sites are all being scraped. We have the data to prove it; over 30% of travel industry website visitors are web scrapers.

Google Analytics, and most other analytics tools do not automatically remove web scraper traffic, also called “bot” traffic, from your reports – so how would you know this non-human and potentially harmful traffic exists? You have to look for it.

This is a good time to note that I am CEO of a bot-blocking company called Distil Networks, and we serve the travel industry as well as digital publishers and eCommerce sites to protect against web scraping and data theft – we’re on a mission to make the web more secure.

So I am admittedly biased, but will do my best to provide an educational account of what we’ve learned to be true about web scraping in travel – and why this is an issue every travel company should at the very least be knowledgeable about.

Overall, I see an alarming lack of awareness around the prevalence of web scraping and bots in travel, and I see confusion around what to do about it. As we talk this through I’ll explain what these “bots” are, how to find them and how to manage them to better protect and leverage your travel business.

Read the full post at Tnooz.  

About the Author

Courtney Brady

Courtney Brady is the Director of Marketing at Distil Networks. She comes to Distil Networks from a variety of start-up companies, routed in SaaS and DaaS solutions. Formerly the global communications manager at multiple companies, Courtney is responsible for developing the company’s marketing strategy and branding campaign.

Follow on Twitter More Content by Courtney Brady
Previous Article
Is Pushdo Screwing You? Details of the Botnet
Is Pushdo Screwing You? Details of the Botnet

This is a PSA for anyone seeing Pushdo traffic across their website. The virus is more sophisticated and cy...

Next Article
WordPress Development with Vagrant
WordPress Development with Vagrant

One of the reasons why I dread working with WordPress is the process involved to copy an existing site to a...