What You Need to Know About Web Scraping: How to Understand, Identify, and Sometimes Stop

March 12, 2015 Courtney Brady

NB: This is a guest article by Rami Essaid, co-founder and CEO of Distil Networks on the Tnooz blog.

Here’s the thing about web scraping in the travel industry: everyone knows it exists, but few know the details.

Details like how does web scraping happen and how will I know? Is web scraping just part of doing business online, or can it be stopped? And lastly, if web scraping can be stopped, should it always be stopped?   

These questions and the challenge of web scraping are relevant to every player in the travel industry. Travel suppliers, OTAs and meta search sites are all being scraped. We have the data to prove it; over 30% of travel industry website visitors are web scrapers.

Google Analytics, and most other analytics tools do not automatically remove web scraper traffic, also called “bot” traffic, from your reports – so how would you know this non-human and potentially harmful traffic exists? You have to look for it.

This is a good time to note that I am CEO of a bot-blocking company called Distil Networks, and we serve the travel industry as well as digital publishers and eCommerce sites to protect against web scraping and data theft – we’re on a mission to make the web more secure.

So I am admittedly biased, but will do my best to provide an educational account of what we’ve learned to be true about web scraping in travel – and why this is an issue every travel company should at the very least be knowledgeable about.

Overall, I see an alarming lack of awareness around the prevalence of web scraping and bots in travel, and I see confusion around what to do about it. As we talk this through I’ll explain what these “bots” are, how to find them and how to manage them to better protect and leverage your travel business.

Read the full post at Tnooz.  

Read the Article

About the Author

Courtney Brady

Courtney Brady is the Director of Marketing at Distil Networks. She comes to Distil Networks from a variety of start-up companies, routed in SaaS and DaaS solutions. Formerly the global communications manager at multiple companies, Courtney is responsible for developing the company’s marketing strategy and branding campaign.

Follow on Twitter More Content by Courtney Brady
Previous Article
The Search Is On: Competitive Price Scraping
The Search Is On: Competitive Price Scraping

Have you ever searched Google, Bing or Yahoo for “competitive price scraping?" If you work in the IT or sec...

Next Article
Is Web Scraping Illegal? Depends on What the Meaning of the Word Is Is.
Is Web Scraping Illegal? Depends on What the Meaning of the Word Is Is.

Depending on who you ask, web scraping can be loved or hated. Startups love it because it’s a cheap and pow...