Predicting the GOP Primaries An examination of crowd wisdom

Transcription

1 Predicting the GOP Primaries An examination of crowd wisdom You Will Learn: 1. Can a cross-sectional prediction market produce the same, real-time predictions as a longitudinal prediction market? 2. Will Traders who invest virtual currency, rather than their own money, assign accurate prices to specific outcomes if their winning virtual shares can be exchanged for real value? 3. Can a properly designed sample of the general population produce the same accuracy as targeted polling methods?

2 For over a century, prediction markets have been used to make accurate forecasts and projections of the future. In January of 2012, Infosurv and GMI decided to put this technique to the test by attempting to predict the winner of the GOP primary contests in Iowa, South Carolina, and Florida. In this report, we ll tell you how our markets performed and how we re applying these results to consumer research. Today, prediction markets are used to predict a wide variety of outcomes such as movie box office receipts, sporting events, pop culture news, and weather trends. In the business world, prediction markets are being used to predict things like project delays, expected sales, and the success of new product concepts. However, betting on future outcomes is not a recent phenomenon. Prediction markets were first publically employed during the late 19 th century among traders on Wall Street. In what has become known as the Curb Exchange, the American public gathered to place bets on who would become the next President. The markets became so popular that during the 1916 Presidential contest between Woodrow Wilson and Charles Evans Hughes, roughly $160M was wagered (in current dollars) on the election outcome. These betting markets consistently made headlines. Not only were they extremely accurate, they were happening during a period when targeted, quantitative polling was not possible. As communication and information technology evolved, and quantitative research methods became the norm, predictions from the Curb were replaced by targeted polling methods, political experts, and pundits in the national media. Today, prediction markets still play a role in the political arena. The Iowa Electronic Markets, InTrade, and other organizations continue to buy and sell contracts tied to political outcomes. These organizations offer 24/7, continuous double auction markets, which allow traders to buy or sell shares in specific outcomes as new information becomes available. In these markets, contract prices are equated to the probability of the expected outcome. For example, in early March 2012 the bid for a contract stating that Barack Obama will win the general election was $60.40 (see Figure A). This tells us that the market believes there was a 60.4% chance that Barack Obama would actually win in November. Figure A

3 As we now know, Obama won a 2 nd term and InTrade share prices had predicted this outcome since January Even as national polls showed a tight race, or even a Romney lead in the days and weeks before the election, InTrade s prediction market consistenly forecast an Obama victory. While these markets provide very accurate information regarding the future, they require an active community of traders who are constantly acting on new information. These traders must also feel very confident in their knowledge and predictions. After all, they must put their own cash at stake in order to participate. Leveraging this model, Infosurv and GMI set out to test a few theories, applying of our years of experience in quantitative survey research and research oriented prediction markets. Namely: 1. Can a cross-sectional prediction market produce the same, real-time predictions as a longitudinal prediction market? 2. Will Traders who invest virtual currency, rather than their own money, assign accurate prices to specific outcomes if their winning virtual shares can be exchanged for real value? 3. Can a properly designed sample of the general population produce the same accuracy as targeted polling methods? Design To begin the experiment, we adapted Infosurv s patented prediction market model, the Infosurv Concept Exchange (or ice), to offer shares in active GOP candidates for each primary. Each prediction market exercise consisted of trader recruitment, a brief introduction to trading, and a trading period. Trader recruitment: Infosurv partnered with GMI to provide a diverse sample of the general population. Traders were presented with multiple prediction market topics for evaluation. Traders who chose to evaluate U.S. Politics or Current Events were invited to participate in the trading exercise. Introduction to Trading: Once qualified traders were identified, we provided them with an introduction to the exercise, and a brief overview of the active candidates. Traders were informed that the virtual shares they purchased in the winning candidate would be exchanged for real currency once a winner was determined.

4 The Prediction Market: Traders were asked, Which of these candidates will win the [state] Republican Caucus/Primary?. Traders then estimated the probability of each candidate s victory, effectively investing their virtual currency into shares in those outcomes. Each trader participated only once. Results Iowa Caucuses As we now know, the Iowa Caucus was a difficult contest to call. Just a week before the caucus date, national polls were projecting a host of outcomes including outright wins by Romney and Paul, and even a tie between Romney, Paul and Gingrich. Seven days ahead of the caucus, there was no clear indication of Santorum s eventual victory. To properly interpret the results of our study you must first realize that prediction markets are not intended to make absolute predictions (e.g. Romney will win ). Rather, prediction market data can be used to understand the current probability of explicit outcomes. In our market, traders predicted a 32% chance of a Romney victory, followed by Paul at 19%, Gingrich at 17%, and Perry at 10%. Both Santorum and Bachmann were estimated to have a 7% chance of victory, and Huntsman rounded out the field with a 6% chance. In our view, this was Romney s race to lose. However, as we now know, Santorum gained significant momentum over the final seven days, and eventually won the contest by a mere 34 votes. Following the prediction market, we compared our prediction to publicly available polls of likely voters in order to assess the perceived accuracy of our result. Six of the seven polls confirmed our prediction; Romney was expected to win or share the lead in Iowa. Only the PPP poll projected a different winner (Paul), and no national poll was projecting Santorum at the time of our prediction market. Finally, we compared our data to publicly available, on-going prediction markets like the Iowa Electronic Markets (IEM) and InTrade. The IEM prediction market opened in August 2011 with seven contracts. These included Michelle Bachmann, Herman Cain, New Gingrich, Rick Perry, Ron Paul, Mitt Romney, and a Rest of Field contract assigned to candidates not named in other contracts. Each contracted paid $1 if the candidate received the first or second most votes in the Iowa Republican caucus, and $0 otherwise.

5 Polling + Caucus Results: Iowa Dates N= Romney Paul Gingrich Perry Santorum Bachman n Huntsman (Projected) Winner PPP CNN ARG We Ask America Insider Advantage 12/26-12/27 12/21-12/27 12/26-28/ Paul Romney Romney 12/29/ Romney 12/28/ Tie Rasmussen 12/28/ Romney 12/ Romney NBC/Marist 28/11 Actual Caucus Results Iowa Caucus 1/3/12 NA Santorum Infosurv/GMI 12/22-12/27 Probability of Victory Romney IEM 12/27 NA Tie InTrade 12/27 NA Paul On 12/27/11, the IEM data estimated that both Ron Paul and Mitt Romney had a 87% chance of placing in the top two on caucus day. The rest of field contract, which included Santorum, finished the day trading at.054. This implied a 5.4% chance of Santorum or any other dark horse candidate winning the caucus. InTrade s prediction market painted a slightly different picture. On 12/27/11, Ron Paul led their Iowa caucus market with a closing contract price of $49. Romney followed at $34, then Gingrich at $11, and Santorum at $4. South Carolina Primary Like Iowa, the South Carolina Primary was highly contested and difficult to project. As shown below, Romney held the lead in all major polls going into the debate on Thursday, 1/19/12. However, Gingrich s positive performance in the debate and an endorsement for Gingrich from Rick Perry after he dropped out of the race, changed the game. Two days later, South Carolina voters expressed their approval by awarding Gingrich 40% of the popular vote; he defeated Romney by 12 percentage points.

6 Polling + Primary Results: South Carolina Dates N= Romney Gingrich Santorum Paul Perry Implied Winner PPP 1/11/ Romney 1/13- CNN 1/ Romney 1/11- ARG 1/ Romney We Ask America 1/15/ Romney Insider Advantage 1/15/ Romney Rasmussen 1/12/ Romney 1/12- Monmouth/SurveyUSA 1/ Romney Actual Primary Results SC Primary 1/21/12 NA Gingrich Infosurv/GMI 1/18-1/19 Probability of Victory Romney InTrade 1/19 NA Romney The Infosurv/GMI prediction market was conducted from 1/18-19, 2012, in advance of the pivotal South Carolina debate. At the time, our traders predicted Romney s probability of victory to be 52%. Gingrich was the projected runner-up with a 16% chance of earning the outright victory. As in Iowa, our findings were comparable to all major polls conducted during the same time period. The InTrade prediction market, however, began to illustrate a difference between real money traders and virtual currency traders. While both the Infosurv/GMI market and InTrade projected a win by Romney, the real money traders at InTrade were hesitant to bet against the consensus, and more likely to bet heavily on the front-runner. Florida Primary The Florida Primary proved to be the least surprising of the three contests we examined. Romney led in all major public polls leading up to the primary, and no other candidate was able to sway the public opinion. He won 46% of the popular vote, besting Gingrich by a full 14 percentage points. Once again, our prediction market echoed the results of nationally recognized polls, which are conducted among likely votes in the contest state. Our traders predicted a 48% chance of a Romney victory. The probability of a Gingrich victory was estimated to be 28%.

7 Polling + Primary Results: Florida Dates N= Mitt Romney Ron Paul Newt Gingrich Rick Santorum Implied Winner 1/29- ARG 1/ Romney 1/24- Mason-Dixon 1/ Romney 1/27- ARG 1/ Romney 1/28- PPP (D) 1/ Romney War Room Logistics 1/27/ Romney HuffPost Pollster 1/30/ Romney Rasmussen Romney 1/25- NBC News/Marist College 1/ Romney 1/26- Ipsos/Reuters 1/ Romney 1/27- Quinnipiac 1/ Romney 1/28- Suffolk University 1/ Romney 1/27- SurveyUSA 1/ Romney We Ask America 1/29/ Romney NewsMax/InsiderAdvantage 1/29/ Romney Actual Primary Results Floriday Primary 1/31/12 NA Romney Probability of Victory 1/ Romney Infosurv GMI 1/30 InTrade 1/30 NA Romney Compared to InTrade, we again observed the polarization that real currency brings to prediction markets. At close on 1/30/12, InTrade was projecting the probability of Romney winning Florida to be 96%; the Infosurv/GMI prediction market pegged his chances at 48%. Both markets projected the correct winner with varying degrees of confidence. Conclusions So, what did we prove? Let s begin with our first hypothesis: A cross-sectional prediction market will produce the same real-time predictions as a longitudinal prediction market. In the South Carolina and Florida primaries, our cross-sectional prediction markets projected the same front-runner as InTrade, the most popular online trading exchange for predicting future events. In Iowa, our prediction market

8 produced estimates consistent with the Iowa Electronic Markets, and more representative of the final outcome than the markets at InTrade. Notably, the Infosurv/GMI prediction market produced conservative probabilities for likely outcomes and more liberal estimates for unlikely outcomes when compared to InTrade. In our opinion, the difference in these projections can be primarily attributed to one thing: the risk of losing money. In the markets at InTrade, investors are required to use their own money to invest in contracts. Losses and gains are both realized, and as one may expect, this can encourage conservative trading behavior. The price of unlikely outcomes is quickly driven down to zero, while the most likely outcomes tend to receive a significant portion of the betting action. In the Infosurv/GMI market, traders are provided with a virtual currency that may be redeemed for cash value at the end of the market. This virtual currency can be used to purchase shares in any outcome, but like the InTrade markets, only shares in the correct outcome are awarded. The difference is, traders using virtual currency have nothing to lose, only to gain. This model encourages traders to hedge their bets by investing in multiple candidates, and to feel more comfortable predicting an outcome that goes against the current consensus. Next, we set out to test the hypothesis that traders who invest virtual currency, rather than their own money, will assign accurate prices to specific outcomes if their winning virtual shares can be exchanged for real value. In these markets, the degree of accuracy in regards to current probabilities is debatable and difficult to prove. However, as discussed previously, we did observe that virtual currency traders invested directionally the same as realcurrency traders in the IEM and InTrade markets. These predictions were also in-line with nearly all nationally recognized polls, implying that they were a sound measure of current public opinion. Lastly, we examine the hypothesis, Can a properly designed sample of the general population produce the same accuracy as targeted polling methods? Our cross-sectional market produced the same front-runner as 14 of 14 nationally recognized polls in Florida, 7 of 7 in South Carolina, and 6 of 7 in Iowa. These predictions were made by a random sample of the U.S. general population who expressed an interest in predicting outcomes regarding U.S. Politics or Current Events. The polls used for comparison were conducted among likely voters in the contest state.

9 Lessons for Market Research 1. When trying to predict the future behavior of a targeted group of individuals, it s not always necessary to ask that group, What will you do?. A group of general observers may be able to provide you with the same (or better) answer. 2. Cross-sectional prediction markets can be effective predictors of consumer behavior, and require fewer resources than longitudinal designs. 3. By providing respondents with an incentive to be correct, you will encourage honest, independent, and carefully considered feedback. About Infosurv Infosurv is a full-service survey research firm based in Atlanta, Georgia, leading the Market Research industry in the development and implementation of prediction markets and crowd wisdom applications. Founded in 1998, Infosurv has grown to be a trusted research partner for hundreds of organizations around the world, including Fortune 500 Companies, small businesses, and government agencies. For more information, visit us online at About GMI GMI (Global Market Insite, Inc.) provides access to the right people at the right time to deliver the right results, empowering researchers and marketers worldwide to generate reliable, consistent and actionable information that enhances their decision-making capability. Founded in 1999 with global headquarters in Bellevue, WA, GMI has operations in America, Europe and Asia Pacific.

10 Data & References Closing prices for shares in Obama victory. Source: InTrade Closing prices for shares in Romney victory. Source: InTrade

11 Price comparison of InTrade data over time. Source: InTrade Price comparison of Iowa Electronic Markets data over time (will win Popular vote). Source: IEM