CrowdWorx Berlin Munich Boston Poznan http://www.crowdworx.com White Paper Series Social Forecasting vs. Market Research When to use a survey, when to use Social Forecasting? Abstract Social Forecasting has achieved impressive results in market research. But it is often not clear to market researchers when to use a Social Forecasting and what the advantages of conventional methods versus Social Forecasting are. This paper reviews both approaches by presenting some results from market research projects where both approaches have been compared with each other. Social Forecasting outperforms surveys and conjoint analyses but is are less powerful when it comes to diagnostics and gathering descriptive data.
2 1 Introduction Market research has traditional surveys, conjoint analyses and other well established methods. To understand when to use traditional market research methods and when Social Forecasting one has to make an important first distinction: One must know whether the goal of a market research project is gathering opinions or data about the status you, e.g. age, gender, and preference distributions in a population, or getting forecasts, e.g. about the sales potential of new or existing products. This distinction has a direct impact on the structure of the participants in both methods. When it the goal is to gathering opinions or descriptive data the participant group should be a representative sample in both approaches. When the goal is to get a forecast, the two approach differ significantly. First, Social Forecasting is by the far the better forecasting tool compared surveys and conjoint analysis (see extensive examples in chapter 3). Second, Social Forecasting does not require a representative sample, which brings down cost to less than half of what traditional market research methods cost. Most market researchers cannot believe that any method can work if there was no representative sample. But for a forecast representativeness is irrelevant! What matters is that the participants have relevant knowledge about the forecasted topic. You do not need a representative crowd, you need a wise crowd. 2 Comparison of traditional market research methods and Social Forecasting After the preliminary discussion of the goals of market research, we will now take a look at the main differences of traditional market research approaches and Social Forecasting (SF). While both methods are designed to aggregate inputs from multiple human participants, they differ in the way they incentivize participants to provide a truthful response and in the information which participants receive about the results of the market research. Surveys or polls incentivize participants with a small payout for participating. The payout does not incentivize for truthtelling or deep thinking but only for answering the questions. In contrast, Social Forecasting work with performance based rewards. The payout for a participant is usually tied to the accuracy of its answer. In market research it is often hard to find a measurable outcome to which the accuracy of a participants answer can be compared. In these cases SF uses proven incentive designs, which preserve the incentives of SF. One such widely used approach is to pay participants by the average outcome of the PM. In fact, this has been proven to provide very good truth-telling incentives (see Skiera et al., 2009). As for the information which participants see about the results: Surveys are designed in such a way that each participant gives an independent opinion without knowing the other answers. Social Forecasting takes the opposite approach and shows the participants the current crowd average at all times. Conventional market research Social Forecasting (SF) Rationale Time from request to result 2-3 weeks 1-2 weeks SF requires less manual analytics and is exclusively conducted online Costs 15-70 keur 10-40 keur When the goal is forecasting SF does not require representative samples Stability of results vs. design errors and small samples low high SF has won every comparison against market research in numerous projects Diagnostic ability high medium SF collects less data about participants Table 1: Comparison of market research methods and Social Forecasting from a business point of view.
3 This seems to contradict all conventional rules about survey design but then again, Social Forecasting is not a survey. The advantage of the SF approach is that participants can receive signals from the crowd in order to reconsider their answers. To avoid that this results in herding or a so called Keynesian Beauty contest, the aforementioned incentives of SF are key. While professional market researchers think this approach cannot work, the examples in ch. 3 prove: It works because incentives keep herding in check so that SF can reap the benefits of information exchange without its downsides (i.e. correlated responses). surveys with 1,000 participants each. These results indicate that the incentives and information design of Social Forecasting are a powerful lever to increase the validity results in market research. Since the participants are the main cost driver in market research the decreased number of participants shows that Social Forecasting can deliver results at much lower cost than surveys and polls. While these methodological comparisons are interesting, the ultimate goal of any method will be to provide accurate and timely results at the lowest possible cost. Table 1 is a highlevel comparison of traditional market research methods and Social Forecasting from a business point of view. The comparison clearly shows that Social Forecasting excels in forecasting. 3 Results from market research with Social Forecasting Predictive opinion polls Opinion polls are the daily bread and butter business of market researchers. Most polls are done online now which has brought the cost of surveys down and closer to the cost of Social Forecasting. Despite these advances in traditional polling accuracy of surveys has not managed to catch up with that of Social Forecasting. As shown by a comprehensive 10 year study by Rietz et al. in 2003 Social Forecasting regular by far outperforms polls when it comes to predictive ability (see Table 2). 1992 1996 2000 Total Number of elections 151 157 229 537 Elections where poll was more accurate Social Forecasting more accurate 28% 13% 24% 22% 72% 87% 76% 78% Table 2: Comparison of accuracy of polls and Social Forecasting in U.S. election forecasting. (Source: Rietz et al, 2003) But predictive accuracy is not the only advantage of Social Forecasting versus surveys. Figure 1 shows results of a study conducted by CrowdWorx in 2009. The predictive accuracy of Social Forecasting with less than 100 participants was the same as the one of traditional 89 Partici- 1,000 1,000 89 1,000 1,000 1,000 1,000 1,000 pants Figure 1: Forecasting accuracy and number of participants of a Social Forecast versus polls in the German Federal election 2009 (Source: CrowdWorx, 2009) A last aspect which is important in practical market research is the availability of results. This has two facets: Cost of updated results after a significant change, e.g. new information becomes public, and convergence of research results to the actual outcome as the date of the outcome comes closer. Figure 2 has insights on these two facets. First, we can see that Social Forecasting updates its data constantly. As soon as new information comes in, the SF participants have an incentive to be the first to adjust their opinion, which updates the Social Forecast quickly. Surveys, in contrast, need to go and ask another 1,000 people before they can provide an updated result. Second, the chart shows clearly that most surveys are getting closer to the actual outcome with time but the Social Forecast has been closer to the actual outcome already 6 months before the actual outcome occurred. This makes SF a much more useful forecasting tool than a poll. While the SF results for predictive opinion polls are impressive, it should not be forgotten that surveys have their own advantages, especially with respect to diagnostic ability. For instance, a survey can explain which age groups, gender, or income groups are voting for a certain party. In principle, this can be done with SF too, but surveys are a more straightforward way for such descriptive analytics. This is also why sometimes it is a good idea to use a combination of Social Forecasting and surveys: When a market research project requires both forecasts and descriptive data SF and surveys should be used in conjunction.
4 40% 30% 20% 10% 0% Harris poll Hotline polls ABC poll CBS poll Social forecast Figure 2: 1996 U.S. Presidential Election Clinton winning margin (dashed line) as predicted by polls (dots) and Social Forecast (solid line). (Source: Berg et al., 2000) New products and consumer preferences CNN/Princeton Time poll Gallup poll Actual outcome -10% Feb-96 Mar-96 Apr-96 May-96 Jun-96 Jul-96 Sep-96 Oct-96 Another very important area of application for market research are new products and consumer preferences. A major part of Social Forecasting with consumers is focused on this area. The rationale behind this interest is two-fold. First, market research with traditional methods, e.g. conjoint analysis, is very costly and time consuming, and second, the results of complex market research methods are often not stable, i.e. have a high variance when a small part of the experimental design is changed or even the same experiment is repeated with another group. This causes executives to wonder as to how much they can trust such results. Soukhoroukova and Spann (2005) have used Social Forecasting and Conjoint Analysis to predict the market share of 4 different MP3 player product concepts. Using four different conjoint aggregation methods (First Choice, Average Prob. Model, Logit Model) on the same data set resulted in an extremely high variation of results (see Table 3). When using conjoint it is unclear which aggregation method to use, hence one can never know which conjoint result is the right one! In contrast, the predicted market share for the same four MP3 player product concepts as computed by four different SF aggregation methods resulted in very stable results. Table 3 compares the two experiments: Social Forecasting outperforms conjoint analysis on several dimensions: The results of SF are more stable, less costly (less participants), and available faster than those of conjoint analyses. Variance of predicted market share using different aggregation methods Effort: number of required participants Conjoint Social Forecasting 67,76% 3,29% 307 20 Time to result* 4 weeks 1 week Table 3: Comparison of stability, effort and time for estimating the market share of four new MP3 player concepts (Source: Soukhoroukova and Spann, 2005). *Estimate from CrowdWorx client projects. Sprenger et al. (2007) pitted Social Forecasting against focus groups in predicting market shares of five mobile phone concepts. The correlation with the actual market share was 83% for the Social Forecast and only 64% for the focus groups. Descriptive data and diagnostics Often a prediction is not the only goal of a market research project. Understanding the reasons behind a result is key, too. In this area surveys are more powerful than Social Forecasting. Surveys can easily get socio-demographic data and reveal underlying relations between answers and individual attributes with simple cross tabulations or conditional distributions. This is why we sometimes combine Social Forecasting with traditional surveys to get the best of both worlds: predictive accuracy and a good diagnostic ability. Predicting sales of new products for European retailer Tchibo has given us similar results. The CrowdWorx Social Forecasting system achieved the same forecast accuracy (over 80% accuracy) as the combination of test sales and consumer surveys run by Tchibo. However, the cost of using Social Forecasting at Tchibo was 10 times lower than the cost of the test sales and consumer surveys (Source: CrowdWorx, 2008).
5 4 Conclusions It is important what the ultimate goal of one s market research project is. The two main purposes of market research are gathering opinions or data about the status quo, e.g. age, gender, and preference distributions in a population or getting forecasts, e.g. sales potential of new or existing products. Conventional market research methods are strong when it comes to gathering opinions and descriptive data. Social Forecasting excels when forecasting is the ultimate goal of the research. Regarding cost and effort Social Forecasting and traditional market research methods differ a lot. A representative sample is the largest cost driver but SF does not require this when the research goal is forecasting. As shown in the previous chapter, Social Forecasting by far outperforms surveys and conjoint in terms of forecasting accuracy and validity while at the same time saving 90% of the cost vs. those traditional approaches. This greatly reduces the cost of SFbased market research. To get the best of both worlds, Social Forecasting and surveys have proven to be a powerful combination. CrowdWorx is the first, and to date, the only Social Forecasting software which combines Social Forecasting and surveys to deliver both predictive accuracy and a good diagnostic ability in a seamlessly integrated environment. References CrowdWorx (2008): Forecasting new products at European retailer Tchibo, CrowdWorx client case study. CrowdWorx (2009): Prediction Markets versus polls in the German Federal election 2009, internal research project. Berg et al. (2000): Results from a Dozen Years of Election Futures Markets Research, University of Iowa, Working Paper Rietz et al. (2003): Accuracy and Forecast Standard Error of Prediction Markets, University of Iowa, Working Paper Skiera, Bernd, C. Slamka, W. Jank (2009): Second-Generation Prediction Markets for Information Aggregation: A Comparison of Payoff Mechanisms, Available at SSRN: http://ssrn.com/abstract=1435316 Soukhoroukova, Arina; Spann, Martin (2005): New Product Development with Internet Based Information Markets: Theory and Empirical Application, ECIS 2005 Proceedings, Paper 133 Sprenger et al. (2007): Conditional Prediction Markets as Corporate Decision Support Systems, Journal of Prediction Markets, Vol. 1, No. 3, Buckingham University Press, Buckingham
6 Contact You can find the CrowdWorx white papers series in the CrowdWorx resources section on www.crowdworx.com. CrowdWorx Social is a global Forecasting provider, headquartered in Poznan, West Poland. We serve clients in Europe and North America with the full range of Social Decision Support services based on Collective Intelligence and State-of-the-Art Enterprise 2.0 methods. For more information on CrowdWorx please write to team@crowdworx.com or call an office near you. Berlin, Germany Rotherstr. 18 10245 Berlin Germany Tel: +49-30-51300-211 Fax: +49-30-41721-281 Munich, Germany Isartalstraße 30 82008 Munich (Unterhaching) Germany Tel: +49-89-33982-510 Fax: +49-89-61-066-793 Boston, USA 17 Stonecleve Rd. Wellesley (Boston), MA 02482 Tel: +1-617-763-8082 Fax: +1-781-489-5595 Poznan, Poland ul. Fredry 1 61-701 Poznan Poland Tel: +48-61-88577-00 Fax: +48-61-88577-15 www.facebook.com/crowdworx www.twitter.com/crowdworx