Earlier this year, engineers at Wal-Mart Stores who track rivals' prices online got a rude surprise: the technology they were using to check Amazon.com several million times a day suddenly stopped working.
Losing access to Amazon.com's data was no small matter. Like most big retailers, Wal-Mart relies on computer programs that scan prices on competitors' websites so it can adjust its listings accordingly. A difference of even 50 cents can mean losing a sale.
But a new tactic by Amazon to block these programs — known commonly as robots or bots — thwarted the Bentonville, Arkansas-based retailer.
Tests conducted in recent weeks for Reuters show that among major U.S. retail chains, Amazon had by far the most sophisticated bot detection in place, both for its home page and for two popular items selected by Reuters because they change price frequently — a De'Longhi coffee maker and a Logitech webcam.
The tests were run by San Francisco-based Distil Networks, which sells anti-bot tools. In one of the tests, Distil programmed bots to hit each retailer's website 3,000 times, but slowly enough to mimic a person clicking through listings. This tricked most retail behemoths, but not Amazon.