Prediction Markets

A small circle of scientists and practitioners started to suspect that virtual stock markets could be designed specifically to elicit and aggregate more reliable information and to uncover personal expectations hidden and distributed in many people’s minds.

In 1997, a researcher at the University of Technology in Vienna, Gerhard Ortner, ventured a practical experiment which became famous as the first industrial application of this newly coined “prediction market” method. Ortner set up a stock exchange game for a big software development project of Siemens Austria. Traders traded two shares on this exchange. One share represented the likelihood of making the project's completion date, scheduled for October 1997; it would pay one dollar if the project completed by its deadline and zero dollars otherwise. The other share represented the likely delay, it would pay a number of dollars equivalent to the number of days by which the completion date was late.

All project members were invited to trade. Immediately upon opening the market, share prices started to drop, indicating a yet undetected problem in the project plan. After two weeks, the shares stabilized at a predicted delay of two weeks. The likelihood of meeting the deadline traded at a price of only 40%.

This begged the question if these market prices were to be taken seriously. After all, the German software giant puts significant effort into its state-of-the-art project management techniques and has vast experience in managing large scale IT projects. The exercise was obviously just a game, and the participants were not professional stock traders but computer programmers, requirement analysts and testers. Like in Iowa, trading was visibly not perfect, and mistakes were made. Accordingly, management all but dismissed the new method’s indications.

Two thirds into the project, trading activity all but stopped for several days (see Figure2). First generation prediction markets needed to match a buy and a sell order to close a trade but there were no more buyers, just sellers. One month before the official deadline, the price of the likelihood share all but collapsed. Siemens’ project management, which still used the original project plan, realized that virtually nobody on the team believed in it anymore. When the project was finally completed in November, it clocked in a two week delay just as anticipated by the prediction market consensus.

Since this first demonstrable success, prediction markets have been applied to an ever wider range of commercial forecasting tasks and decision problems. Hewlett-Packard ran an in-house prediction market to forecast printer sales, and found that the prediction market beat HP’s sophisticated in-house forecasting process three times out of four (Plott, 2002). In 2005 Arcelor, the global steel company, started to apply prediction markets for steel demand and price forecasting. Collectively, the 30-40 employees who played the market have accurately predicted volumes for the flat product division (Llera, 2006). Another prediction market predicted the first day closing price of the Google IPO better than traditional investment banking processes. Had Google set its IPO price at the prediction market forecast level, IPO proceeds would have been $379 million higher (Berg, 2007), arguably offering a worthwhile ROI on the predictive market’s project cost.

However, early systems were unwieldy. They required a significant investment in technology, bespoke development, implementation effort, and special training of participants. Second generation prediction markets now deliver vastly improved technology as a cloud service. Scaled and discrete prediction questions instead of binary yes/no shares facilitate sophisticated research designs. Action standards, benchmarks, and proven test patterns replace the laissez-faire 'crystal ball' approach of early experiments; the best systems even provide volumetric forecasts instead of just probabilistic predictions. Social conversations and text analysis back all quantitative foresights with qualitative insights.

This article will (1) analyze some major problems of the traditional questionnaire method when applied to future or intent questions, (2) summarize key principles and capabilities of modern prediction markets, and (3) describe practical applications such as to identify winning product ideas, find the best communication concept, pre-test new brands, packages and adverts, or anticipate industry trends and competitive moves.

Next: Are crowds intelligent?