Why Web Scraping is Costly for Real Estate Agents, Brokers and Portals

February 23, 2015 Charlie Minesinger

Though web scraping is a threat to any website, sites that distribute information regarding online real estate listings are particularly vulnerable. Property data and photos are highly valuable and relatively easy to steal using malicious web scraping bots. For this reason, an effective bot detection solution is crucial for any real estate website or portal with an inventory of listing data.  

Bots in real estate

Malicious bots can steal your listings and negatively impact your business by:

Scraping listings from your portal

Aggressive web scraping bots steal listing data for distribution on competing sites, pulling customers away from your site. Web scraping is not syndication. When bots steal your data, photos and copyrights, your portal loses traffic while your competitor’s portal gains.

Diminishing the value of your SEO

When bots steal and distribute listing data widely, your competitors will attract website traffic and leads, gain authority and improve their pagerank. The value of your SEO decreases as your content is replicated across the web. In turn, costs for real estate agents go up as they are forced to spend more on qualified leads.

Presenting inaccurate listing data

Once your listing data has been scraped, you lose control over the presentation of the property, the quality of the data, and the ads and agents that are displayed adjacent to the listing. Rogue websites do not update listing data frequently, and potential buyers may call the agent or broker to inquire about a listing which is either inaccurate or out-of-date. The resulting frustration creates a lack of trust, reflecting poorly on your brand.

Stealing information on agents who advertise on your portal

Automated web scraping bots steal contact information for real estate agents and brokers advertising on portal sites, and this data is used by your competitors to steal the advertising dollars those agents spend on your portal.  Your site can become a lead generator to sell competing ad space to your realtor and broker clients.

Committing form spam and click fraud

Malicious bots can fill in fraudulent registration forms and click on ads, mimicking human behavior.  Agents, lenders, developers and other portal advertisers see the ROI of their advertising spend decrease as ads are clicked by bots.  Advertisers want, and will demand, certified human traffic, not bots.

Putting downward pressure on your margins

Real estate data is gathered tediously from multiple sources and historical files which often require fees. When web scraping bots steal your listing data, they’re essentially stealing the value of your investment in collecting and compiling such data, and this affects your margins.

Distil Networks has found that up to 40% of traffic on real estate websites is generated from malicious web scraping bots.  While in-house bot detection is often a first step in shutting down bots, these techniques use up engineering resources quickly and rarely catch more than 20% of bad bots. Blocking IPs and using rate limits to protect listing data is obsolete when a high percentage of bot traffic originates from IP addresses that belong to consumer ISPs, such as Comcast or Verizon FiOS. A purpose-built solution such as Distil Networks will accurately catch 99.9% of bad bots without impacting consumers or relying on IP addresses.

Distil has signed up more than 15 individual MLSs and has also had major wins with real estate platforms, such as FlexMLS. Distil’s FlexMLS implementation protects 130 MLS customers and more than 10,000 agent and broker websites using Internet data exchange (IDX) search.  In 2014, IRES LLC (ColoProperty®), the regional MLS for Northern Colorado, implemented Distil Networks’ technology to protect online MLS information. The Distil solution identifies and blocks scrapers independently from the legitimate traffic to MLS portals.

If you’re in the real estate business and you want to ensure that web scraping bots can’t steal your listings and drive up costs for your member agents, then consider Distil Networks. Our proprietary bot detection solution will help protect your site from bots, scrapers, and other malicious attacks. We’ve helped others just like you, including OntheHouse and the Honolulu Board of Realtors

About the Author

Charlie Minesinger

Charlie Minesinger is the Director of Sales at Distil Networks focused on strategic accounts and channels. Charlie has over a decade of experience selling into enterprises and brand accounts, including Sprint, Disney, Nortel, Amazon, Cisco, AIG, Mattel. Charlie brings with him experience in start-ups and selling into new markets.

Follow on Twitter More Content by Charlie Minesinger
Previous Article
How Google and Bing Offer Click Fraud Protection
How Google and Bing Offer Click Fraud Protection

Though Google and Bing offer some click fraud protection, it’s often not enough to protect your website and...

Next Article
How Online Travel Site Red Label Vacations Stopped Web Scraping Bots in their Tracks
How Online Travel Site Red Label Vacations Stopped Web Scraping Bots in their Tracks

On average, 30% of travel site traffic is generated by web scraping bots. Learn how Red Label Vacations sto...