What You Need to Know About Web Scraping: How to Understand, Identify, and Sometimes Stop

August 26, 2013 Courtney Brady

NB: This is a guest article by Rami Essaid, co-founder and CEO of Distil Networks on the Tnooz blog.

Here’s the thing about web scraping in the travel industry: everyone knows it exists, but few know the details.

Details like how does web scraping happen and how will I know? Is web scraping just part of doing business online, or can it be stopped? And lastly, if web scraping can be stopped, should it always be stopped?   

These questions and the challenge of web scraping are relevant to every player in the travel industry. Travel suppliers, OTAs and meta search sites are all being scraped. We have the data to prove it; over 30% of travel industry website visitors are web scrapers.

Google Analytics, and most other analytics tools do not automatically remove web scraper traffic, also called “bot” traffic, from your reports – so how would you know this non-human and potentially harmful traffic exists? You have to look for it.

This is a good time to note that I am CEO of a bot-blocking company called Distil Networks, and we serve the travel industry as well as digital publishers and eCommerce sites to protect against web scraping and data theft – we’re on a mission to make the web more secure.

So I am admittedly biased, but will do my best to provide an educational account of what we’ve learned to be true about web scraping in travel – and why this is an issue every travel company should at the very least be knowledgeable about.

Overall, I see an alarming lack of awareness around the prevalence of web scraping and bots in travel, and I see confusion around what to do about it. As we talk this through I’ll explain what these “bots” are, how to find them and how to manage them to better protect and leverage your travel business.

Read the full post at Tnooz.  

Read the Article

About the Author

Courtney Brady

Courtney Brady is the Director of Marketing at Distil Networks. She comes to Distil Networks from a variety of start-up companies, routed in SaaS and DaaS solutions. Formerly the global communications manager at multiple companies, Courtney is responsible for developing the company’s marketing strategy and branding campaign.

Follow on Twitter More Content by Courtney Brady
Previous Article
Building A Better Mouse Trap: How We Detect and Block Bot Traffic
Building A Better Mouse Trap: How We Detect and Block Bot Traffic

Bots are one of the most vexing technical problems web applications must deal with today. They tax server r...

Next Article
One Thing Google and Content Thieves Have in Common
One Thing Google and Content Thieves Have in Common

Both Google and content thieves know how valuable unique content is. Content scrapers are websites that st...