Web Scraping Bots Flood Real Estate Sites

June 16, 2017 Peter Zavlaris

Nearly Three Times More Bots Than Humans Visit Real Estate Websites

Did you know that real estate websites are infested with bots? Even more disconcerting, in 2016 bad bots accounted for nearly half (43%) of all real estate site visits. Bad bots scrape data (e.g., pricing, inventory levels) from sites without permission in order to reuse it and gain a competitive edge. They take the form of scripts, programs, networks of workstations or PCs having installed malware (i.e., botnets), and fake browser plugins able to perform man-in-the-browser attacks. Truly nefarious ones undertake criminal activities, such as fraud and outright theft.

Good bots, such as search engine crawlers, accounted for only 25.6% of real estate site visitors in 2016—this means a total of 68.6% of visitors were bots of one form or another. Humans accounted for only 31.4% of visits. Distil’s 2017 Bad Bot Report reports that 97% of all websites have problems with scrapers.

Real Estate Good vs. Bad Bots

Web Scraping MLS Data

Bad bots are most often deployed against real estate sites to scrape home listings. Here, an interloper creates a real estate site, but instead of paying for listing data from a multiple listing service (MLS), they steal it from other real estate sites.

Once installed on a site, the bot does the work of a search engine. A potential homebuyer searches for homes, while the bot responds by returning results from scraped sites. The end user doesn’t know the data comes from another site, nor do they care. They give their business to whichever site provides them the listing information they seek.

Using this approach isn’t illegal and avoids fees charged by MLS services. The only group that feels any repercussion are the owners of sites being scraped. They’re charged MLS search costs while serving bots—not potential home buyers—valid listing information. This results in lower conversion rates and fewer leads to sell to Realtors.

Spamming

Another bad bot attack against real estate sites takes the form of “Contact Agent” form spamming. Provided for potential home buyers to complete, bad bots populate such forms with false information. Their goal is usually to earn ad revenue by stuffing links to digital ads in the comments section, such ads sometimes being for competitors’ real estate sites.  

If you’re thinking your contact forms are safe behind a login page where only authenticated users have access, this is incorrect. Spammers are very adept at getting their bots past logins.

If miscreants sign up for accounts themselves and are denied, they break into existing accounts by using bad bots to guess usernames and passwords. Spammers also use sophisticated bad bots that masquerade as humans to sign up for multiple accounts at once, providing them with an endless supply of new accounts to churn through.

96% of login pages are hit by bad bots, while 90% of sites requiring login credentials (authentication) to gain access already have bad bots operating on the other side of the login page.

While authentication and encryption does protect against man-in-the-middle attacks, it doesn’t stop bad bots. If you’re blindly trusting anything with a valid username and password as it seeks entry, you’re asking for trouble. Sophisticated bad bots attack your business logic in an effort to reap rewards from the data and intellectual property you deliver to your end users.

The fight against bad bots requires constant vigilance as their operators are able to change tactics in an instant in response to defensive measures put in their path.

To stop them you need a blend of technologies: JavaScript injection, fingerprinting, and token management. Couple these with behavioral analysis that uses machine learning to analyze log information to detect anomalies, along with automated response capabilities such as access blocking, CAPTCHA serving, rate limiting, and monitoring.

 
 
 

About the Author

Peter Zavlaris

Peter Zavlaris weighs in on various topics around bot mitigation, bot defense sharing white papers, videos and other resources on the topic.

More Content by Peter Zavlaris
Previous Article
Distil Networks Partners with Online Trust Alliance (OTA) and Finds Advanced Persistent Bots Still Massive Problem for Websites
Distil Networks Partners with Online Trust Alliance (OTA) and Finds Advanced Persistent Bots Still Massive Problem for Websites

Distil Networks fights bad bots with a new partnership with Online Trust Alliance (OTA) because advanced pe...

Next Article
Life at Distil: Welcome to Our House
Life at Distil: Welcome to Our House

Have you ever wondered what it’s like to work at Distil? We’re more than just your average tech company. Ch...