Have you ever searched Google, Bing or Yahoo for “competitive price scraping”??? If you work in the IT or security department of an eCommerce or travel website, I bet you have. Otherwise, give it a whirl! You’d be shocked to see what you find.
Before joining Distil Networks, I had never given it much thought – I’m not in IT per se – I’m the “Sales guy” at an IT organization. I know just enough technical information to be dangerous. Personally, I was blown away to scroll through the results. Outside of a few articles on the subject, there they were: companies jockeying for SEO privileges to be the target of your price and content scraping needs and desires. All of them asked the same thing. How many pages do you need to crawl? How many times per day/week/month? How many different competitor sites? How automated do you want the data? All of that before you even clicked on the ad to get to their website.
What do these scraping ‘companies’ mean for businesses? Quite frankly it destroys competitive advantage. If you have unique or valuable content and it is not protected– then it’s possible it won’t be so niche or creative for very long. Think of the hours invested, creative resources used, and the value of your “proprietary” pricing. Do you really want this in the hands of your competitors? End result: they steal your traffic and customers. Thank you for the added revenue! Of course, don’t expect to get a heartfelt thank you card or check in the mail.
Companies now specialize in aggregating and automating website data to a point where you can sit back, relax and let the money-making engine run. Whether you call them bots, scrapers, crawlers, spiders, aggregators, or insignificant nuisances – they are out there at massive scale, disguising themselves as humans, undermining your strategic goals.
Price scraping bots are in everyday traffic. I have seen bots equate to 60-70% of overall traffic for well-branded sites! This doesn’t include the traffic from dreaded botnet attacks that can be as high as 99%! In fact the ‘brown out’ from a bot can be more damaging and difficult to defend. If you don’t see value in protecting your competitive edge, think about what these bots do to your bandwidth and infrastructure resources.
In a day where Internet Security is in your face, there are many options out there to choose from. Just be weary. Sophisticated bots of today no longer originate from the same IP’s or operate rudimentary software. Playing an ongoing game of whack-a-mole through IP blocking and rate limiting is a losing strategy. Price scraping bots are not run by a couple of guys hanging out in their mom’s basement anymore. They are your competitor, affiliate, and newcomer to the market! Do your research! It’s time to slam the door shut on bots before they get to your website.
Bots don’t buy, they steal! As we approach the holiday season, scraping will only continue to rise and competitors will try to edge others out the best they can. Ensure that your prices and products are protected – block those bots and let your real customers shop.
About the Author
Ron Abisi is the VP of Sales focused on direct sales for enterprises and high-growth Internet properties. He has over 15 years of sales leadership and technology experience. Prior to Distil, he worked for Dyn managing some of the largest and most well-known Fortune 500 and Alexa top 500 accounts.More Content by Ron Abisi