Securities Trading of Concepts (STOC)

Transcription

1 Securities Trading of Concepts (STOC) [Running title: SECURITIES TRADING OF CONCEPTS (STOC)] Ely Dahan *, Andrew W. Lo **, Tomaso Poggio ***, Nicholas Chan and Adlar Kim May 6, 2007 * ** (corresponding author) Assistant Professor of Marketing, B-514, UCLA Anderson School, 110 Westwood Plaza, Los Angeles, CA , edahan@ucla.edu Harris & Harris Group Professor of Finance; Director, Laboratory for Financial Engineering, MIT Sloan School of Management, Cambridge, MA *** Eugene McDermott Professor, McGovern Institute, Computer Science and Artificial Intelligence Lab, Brain and Cognitive Sciences Department, MIT, Cambridge, MA Managing Director, AlphaSimplex Group, One Cambridge Center, Cambridge, MA PhD Candidate, MIT Department of Electrical Engineering, MIT, Cambridge, MA This report describes research done within the Center for Biological and Computational Learning in the Department of Brain and Cognitive Sciences, the Artificial Intelligence Laboratory, and the Laboratory for Financial Engineering at the Massachusetts Institute of Technology. This research was sponsored by grants from: Office of Naval Research under contract No. N , Office of Naval Research (DARPA) under contract No. N , National Science Foundation (ITR) under contract No. IIS , National Science Foundation (KDI) under contract No. DMS , and National Science Foundation under contract No. IIS This research was partially funded by the Center for e-business (MIT), and the MIT Laboratory for Financial Engineering. Additional support was provided by Central Research Institute of Electric Power Industry, Eastman Kodak Company, DaimlerChrysler AG, Compaq, Honda R&D Co., Ltd., Komatsu Ltd., Merrill Lynch, NEC Fund, Nippon Telegraph & Telephone, Siemens Corporate Research, Inc., and The Whitaker Foundation. The authors also wish to thank Rob Hardy and Leonard Lee of MIT, and Limor Weisberg for their efforts in programming and designing many of the web sites that comprise this research. 1

2 Securities Trading of Concepts (STOC) Abstract Market prices are well known to efficiently collect and aggregate diverse information regarding the economic value of goods and services, particularly for financial securities. We propose a novel application of the price discovery mechanism in the context of marketing research: to use pseudo-securities markets to measure consumer preferences over new product concepts. A securities-trading approach may yield significant advantages over traditional methods for measuring consumer preferences such as surveys, focus groups, concept tests, and conjoint studies, which are costly to implement, time-consuming, and often biased. Our approach differs from prior research on simulated markets and experimental economics in that we do not require any exogenous, objective truth such as election outcomes or box office receipts on which to base our securities market. Our trading experiments show that the market prices of securities designed to represent product attributes and features are remarkably efficient and accurate measures of preferences, even with relatively few traders in the market. The STOC method may offer a particularly efficient screening mechanism for firms developing new products and services, and deciding where to invest additional product-development dollars. 2

3 1 Introduction Markets are well-known to be an efficient tool for collecting and aggregating diverse information regarding the value of commodities and assets (Hayek 1945). They have been particularly successful in the domain of financial securities. In this paper, we explore a novel application of the price-discovery mechanism of financial markets to marketing research: using securities trading of concepts (STOC) to collect consumer preferences on product concepts. This application is motivated by the need for reliable, accurate, and economical means to gauge consumer preferences, and the belief that markets are efficient in aggregating information. It also exploits the incentive-compatible nature of markets and most participants preference for game-playing over responding to direct surveys. In particular, we present results for multiple market experiments with live subjects in which participants express their preferences over new product concepts by trading virtual securities. The resulting securities prices are compared against preference orderings derived from separate studies of the same products using direct surveys (rank order choice, for example) and the virtual concept testing (VCT) methodology developed by Dahan and Srinivasan (2000). We find that results across different market experiments are, for the most part, highly consistent with each other and are highly correlated with those from more traditional market research methods. Moreover, we consider six stock-market metrics that summarize group preferences and identify the best performers among these metrics. To gain a better understanding of how STOC may be achieving these results, we relate our market experiments with some classic examples from the experimental economics literature. 3

4 The essence of the STOC methodology centers around the establishment of virtual stock markets that trade virtual securities, each associated with an underlying product or service. These products or services could be a concept or prototype under evaluation or an existing product that anchors the market to the real world. Upon entering a concept market, each participant receives an initial portfolio of cash (virtual or real) and virtual stocks. Participants are also provided with detailed information on the products (stocks) that includes specifications, images, and multimedia illustrations. A typical objective of the STOC game might be for each participant to maximize the value of his or her portfolio, evaluated at the last price prior to the closing of the market, and markets are typically open for 20 to 30 minutes. If participants play with real money, they will have the opportunity to profit from trading and bear the risk of losing money. The financial stakes in the game provide incentives to reveal true preferences, process information and conduct research. If fictitious money is used, prizes can be awarded according to individuals performance. One can also reward all participants just for their service. As in real financial markets, the stock prices are determined by the demand and supply in the market, which depend on the participants evaluation of their own and others preferences for the underlying products. Thus, at the market equilibrium, prices should fully reflect all participants aggregate preference of the products. Traders make trading decisions just as they would in a financial stock market: they assess the values of the stocks, sell overvalued one and buy undervalued ones, essentially voting on the worth of the underlying products. In this way, a stock s price becomes a convenient index of a product s consumer value. There are, of course, several well-established methods for estimating consumer preferences, e.g., surveys (cf. Burchill and Brodie 1997), voice-of the-customer methods (cf. Griffin and Hauser 1993), conjoint analysis (cf. Srinivasan and Shocker 1973, Green and Wind 4

5 1981, Green and Srinivasan 1990) and concept tests (cf. Urban, Hauser and Roberts 1990, Dahan and Srinivasan 2001, Dahan and Hauser 2002), and focus groups (cf. Mahajan and Wind 1992, Calder 1977, Fern 1982). However, concept markets may be a useful alternative to these methods for several reasons: 1. Accuracy: In order to win the game, participants have the incentive to trade according to the best, most up-to-date knowledge because of their financial stake in the market. STOC also captures, continuously, the ever changing pulse of the market for all participants who can express their opinions multiple times during the course of the market rather than responding only once to a survey question. 2. Interactive Learning: A STOC market participant not only evaluates concepts on his or her own behalf, but also considers the opinions of the public at large. Furthermore, participants can observe others valuations of the virtual products and update and adjust their own valuations dynamically in the market environment. In short, learning is a crucial element in these markets. 3. Scalability: Unlike surveys, markets are intrinsically scalable. In fact, the efficiency of the market, and therefore the quality of data collected, improves with the number of participants. This extends to the number of product concepts that may be evaluated since there is no requirement that each participant trade every security, the bounded rationality of the traders does not limit the number of concepts that can be evaluated in a STOC market. 4. Unarticulated Needs: The market method is particularly useful over survey methods when a product cannot be naturally described or represented by a set of attributes (for example, a movie script, fashion item, car body style or piece of art). Market participants evaluate the concepts directly and market prices effectively reflect the overall viability of the concepts, including the ability of a concept to fulfill unarticulated needs. All that is required is a thorough description (and visualization) of each concept. Of course, market-based methods for eliciting information also have certain limitations. Unlike typical marketing research techniques in which information is collected from individuals and aggregated in subsequent analysis, the market method focuses on aggregate beliefs and preferences and neglects those of individuals. Virtual concepts markets are vulnerable to price manipulations and speculative bubbles because the values of virtual securities hinge on the aggregate beliefs, which are endogenously determined within the same market. Traders may form false beliefs that could cause prices to deviate from their fundamentals. And all of the 5

6 behavioral critiques that have been leveled against the Efficient Markets Hypothesis in the financial economics literature (see, for example, Shefrin, 2005) apply to concepts markets as well. For these reasons, the market method must be applied with cautions and consistency of the results must be checked. In Section 2 we provide a literature review of related approaches to estimating consumer preferences, a description of a concept testing project that this study is based on, and relevant research in experimental economics. Section 3 presents the designs of the securities and markets. In Section 4 we present several conjectures on how the survey markets work by considering an equilibrium model and simulations with artificial agents. Section 5 presents results from several market experiments and, we conclude in Section 6 by describing several extensions and open questions for the STOC method. 2 Background A number of practical issues arise in attempting to infer consumer preferences via the STOC method. What are the requirements for the group of traders, e.g., how many are needed, do they be experts at securities trading, how long do they need to trade to collect useful data, how knowledgeable does each participant need to be of the product category being studied, and what strategy do traders adopt in order to win the game? Beyond the structure of the game itself, questions arise regarding the description of the concepts and the objectives and incentives involved. More specifically, what exactly is being measured by STOC? Is it an aggregation of diverse, independent opinions or a negotiation process in which participants learn from and are influenced by each other, ultimately achieving consensus? What matters most in the game, the underlying fundamentals of each security, based on some external truth, or the (potentially 6

7 biased) perceptions of those truths by the actual traders playing the game? And how can the data collected during the STOC game be best summarized are closing prices the ultimate measure of the market consensus, or should metrics based on all of the data collected be employed? To address these issues, we provide an overview of other market-based mechanisms for gathering information in Section 2.1 such as the Iowa Electronic Market; these examples shed considerable light on the practical use of markets for information aggregation. In Section 2.2, we present further background for STOC by reviewing the concept of rational expectations and the related literature on experimental economics. And in Section 2.3, we give a summary of the virtual concept testing approach of Dahan and Srinivasan (2000), which is the primary benchmark by which we will gauge the efficacy of the STOC methodology. 2.1 Opinion-Collecting Electronic Markets The application of the market mechanism is not restricted to the pricing of assets in financial markets. Different non-financial markets have been established for opinion polling, forecasts and predictions. The Iowa Electronic Markets (IEM) 1 from the University of Iowa is one of the pioneers of non-financial markets in the polling of opinions (Forsythe, Nelson, Neumann & Wright 1993). The IEM was founded for research and educational purposes. Trading profits from the market provide incentives for traders to collect and process information of relevant future events. The IEM features real-money futures markets in which contract payoffs depend on the outcome of future political and economic events. Examples of these markets are the U.S. Presidential Election Market and the Computer Industry Returns Markets. In the U.S. Presidential Election Vote Share Market, for example, the contract RepVS would pay $1.00 times the vote share (percentage of popular vote) received by the Republican Party 1 The Iowa Electronic Markets, 7

8 nominee, George W. Bush, in the November 2000 election. Traders at IEM invest their own funds, buy and sell listed contracts according their own judgment of the likelihood of the underlying events, which is equivalent to the values of the corresponding contracts. On the election day, the contract RepVS (Bush) was liquidated at $0.497 while the contract DemVS (the Democratic Party nominee, Al Gore) was liquidated at $0.499, indicating that the overall market "thought" that Bush and Gore would receive 49.7% and 49.9% of the popular votes, respectively. 2 IEM predicted the voting results, in terms of popular votes, of the past two Presidential elections within two-tenths of a percentage point, outperforming most national polls. 3 The Gallup Poll s predictions, for example, deviated from election results by 1.9% and 5.7% for the Democratic candidates in years 1992 and 1996 respectively. 4 The Hollywood Stock Exchange or HSX 5 establishes virtual markets trading movie stocks and star bonds. Each share of a movie stock pays a percentage of a particular movie s U.S. box office total; a star bond is priced based on a movie star s performance at the box office of his or her recently released movies. Prices of these stocks and bonds are determined by the demand and supply in the market, which in turn depend on players consensus. Players trade with fictitious money called Hollywood dollar. Market prices serve as predictions on earnings of movies and consensus of movie stars popularity. Box office forecast is an invaluable service to film makers and Hollywood marketers. But traditional marketing research techniques have been found notoriously inaccurate and unreliable. 6 HSX markets provide an alternative means of obtaining such forecasts in an arbitrarily large scale inexpensively. 2 Bush and Gore received 47.87% and 48.38% of the popular vote, respectively. 3 BusinessWeek, 11/11/ The Hollywood Stock Exchange, 6 The Wall Street Journal, July 21, Box Office Muddle: How Will Firms Fare From Week to Week? 8

9 And, finally, the Foresight Exchange (FX) 7, applies the market mechanism to predictions of the probability of future events occurring such as changes in the environment, scientific breakthroughs, the collapse of companies, or political and news outcomes. The three markets described above share, with STOC, the benefits of information aggregation, the joy of competitive play, the ability to learn from others and incentive compatibility. The STOC method differs from the IEM, HSX and FX stock markets in several important respects: (1) STOC games last between 10 and 60 minutes typically, while the other three are ongoing and are subject to new information arriving to update players beliefs, (2) STOC securities can describe actual product concepts or purely virtual ones (even ones that are not actually buildable), while the other three focus on observable outcomes in the real world, (3) the incentive when playing STOC is to win a prize and recognition in a one-shot, short term game, while the incentive in the other markets is less obvious, possibly deriving from self-satisfaction in the main, and (4) while IEM, HSX and FX are geared toward predicting events and outcomes, STOC holds the distinction of being primarily aimed at predicting preferences. Thus, traders are more able to rely on their personal preference and the expectation of others preferences as a primary source of information in STOC than in the other three markets. 2.2 Rational Expectations Models and Experimental Markets Our trading experiments are closely related to the literatures in rational expectations (RE) models with asymmetric information and experimental markets. In a standard asymmetric information RE model (Grossman, 1981), heterogeneous agents with diverse information trade with each other and, under certain conditions, the market will converge to an equilibrium in which prices fully reveal all relevant information. The most important criterion for convergence 7 The Foresight Exchange, 9

10 is that agents condition their beliefs on market information. In particular, agents make inferences from market prices and quantities about other agents private information. The RE model has received considerable attention in the study of experimental markets (Plott and Sunder, 1982, 1988; Forsythe and Lundholm, 1990; Davis and Holt, 1993). Studies of the informational efficiency of a market relative to the RE benchmark fall into two categories: markets with fully informed agents ( insiders ) and uninformed agents, and markets with many partially informed agents. In various experimental markets with human subjects, the results for both market structures are the same: markets eventually converge to the RE equilibrium, i.e., information aggregation and dissemination occur successfully. Our trading experiments share some common characteristics with such experimental markets, and information aggregation and dissemination provide compelling motivation for the success of our STOC market. For example, traders who possess superior information about the products or have high confidence on their beliefs can be considered insiders. On the other hand, traders who have little knowledge or opinion of the products can be regarded as the uninformed. The interaction between the insider and uninformed constitutes information dissemination. What is intriguing about this scenario is that even when a subset of traders ignore the underlying product information and only focus on market information, the market still converges to efficient prices that impound all the relevant information and beliefs. Alternatively, individual traders may form their own beliefs about the products, acknowledging that market prices will depend on aggregate beliefs. This is similar to the information aggregation scenario in which there are no insiders, but where all traders are partially informed. Even in this case, where no single trader has full information, an RE 10

11 equilibrium will be reached under very general conditions (Grossman, 1981; Davis and Holt, 1993, Chapter 7). However, there is one important differences between our STOC market and those in the experimental markets literature. In a typical experimental market, subjects preferences and their information set are fixed and assigned by the researchers. Therefore, even before trading begins, theoretical equilibrium prices can be calculated. In contrast, in a STOC market, neither the subjects preferences nor their information sets are known in fact, these are what STOC market trading experiments are meant to discover. This suggests an important practical consideration in implementing STOC markets: the composition of traders should match the population of target consumers as closely as possible. For example, if the target population of a particular product is teenage female consumers, a STOC market consisting of middle-age males will not yield particularly useful preference rankings for that product. However, if the cross section of traders in a STOC market is representative of the target population, the force of market rationality will ensure that the price-discovery mechanism will provide an accurate measure of aggregate preferences. 2.3 Virtual Concept Testing Our virtual markets are set up to collect consumer preferences to facilitate product concept testing, a critical step in new product development. Concept testing is a procedure to narrow down multiple design concepts to the optimal design according to the responses collected from potential users of the product or service being studied. Dahan and Srinivasan (2000) present a virtual concept testing (VCT) methodology that conducts product concept testing over the Internet using virtual prototypes in place of real, physical ones. Dahan and Hauser (2002) extend this work and add new web-based market research methods to the mix. 11

12 The authors consider the World Wide Web as an attractive environment for conducting marketing research because of its interactive nature, instantaneous access to respondents, and availability of new technologies to deliver rich multimedia contents. The authors claim that the use of virtual product reduces costs in new product development so that a larger number of concepts can be explored. In VCT, virtual prototypes are presented in the form of visual static illustrations and animations, plus a virtual shopping experience. Through an interactive Web page, respondents are able to rank different products by specifying the prices they are willing pay for individual products. After data are collected from respondents, conjoint analysis is conducted to obtain market share predictions of the concept products. Conjoint analysis is a technique used to decompose respondents preference on individual attributes of a product based on their overall preferences. Green & Wind (1981) provide a tutorial of the technique. The goal of the Dahan and Srinivasan (2000) study is to choose the best design of a bicycle bump among nine concept products depicted in Figure 1, and two commercially available products. The authors find that the virtual prototype tests produce market share predictions that closely resemble to those given by tests in which real physical prototypes are used. In Dahan and Hauser (2002), the VCT method is applied to the eight existing and yet-to-be-released crossover vehicles depicted in Figure 2. Following these successful applications of virtual concept testing, we adopt the identical virtual illustrations and study the same market research problems using STOC in place of VCT. 12

13 3 Design of Markets and Securities The market method is applied to the same product concept testing problems presented in Dahan and Srinivasan (2000) and Dahan and Hauser (2002). Multiple trading experiments were conducted to predict the market share of nine concept bike pumps and eight crossover vehicles (part SUV, part minivan, part luxury car). In the case of bike pumps, a market is set up with eleven securities - nine concept products as depicted in Figure 1, and two commercially available products. Figure 1: Bike Pump Product Concepts Underlying the STOC Securities Worst Best The eight crossover vehicles depicted in Figure 2 consist of three existing products (Lexus, Mercedes and BMW) and five yet-to-be-released (at the time of the STOC tests) vehicles (Pontiac, Acura, Buick, Audi and Toyota). 13

14 Figure 2: (8) Eight Crossover Vehicles Each of the securities is the stock of the virtual company that manufactures and sells a particular pump or vehicle as its only product. These companies will go public, and their initial public offering (IPO) prices are to be determined in our virtual market. All companies will offer the same number of shares of common stock to the public. The objective of the game is to maximize the value of one s portfolio at market close. The value of a portfolio is calculated as the sum of the cash and total worth of the stocks, which are valued at the closing market price. Participants strive to maximize their profits by trading the stocks using their personal valuation of the companies, their perceptions of others valuations, and any information they can observe from the trading dynamics. Fictitious money was used in the markets, but we rewarded top players with prizes. This provides the participants an incentive to perform in the experiments. It is assumed that all of the companies have identical production cost structures, manufacturing capacity, distribution channels, financial structures, management expertise and all other factors that may affect their profitability. In other words, all factors other than the quality and desirability of the products can be ignored in valuing the stocks. 14

15 The eleven bike pump companies are Cyclone, AirStik, Soliboc, Gearhead, Silver Bullet, TRS, Gecko, Epic, Skitzo, RimGripper and 2wister. To anchor the value of the fictitious currency, one of the eleven securities---cyclone---has its price fixed at $10 and is not traded. Cyclone is served as a reference price or numeraire security. For example, if a trader thinks that the company TRS is worth twice as much as Cyclone, he or she would pay up to $20 for one share of TRS. The stocks of the ten freely traded companies may be priced at any level, depending on the demand and supply in the market. Figure 3: Typical Product Information for Bike Pumps and Crossover Vehicles A typical trading experiment is conducted in the following way. Detailed product information for the bike pumps are given in the form of static illustrations and animations (see Figure 3). These visual depictions show participants the appearance of the pumps as well as how they work. In addition, each bike pump is rated in terms of four attributes: the speed with which a pump inflates a tire, compactness, the ease of operation and durability. The participants are 15

16 presented with a single Web page with the visual depictions and profiles of the bike pumps, trading instructions, and the objective of the trading game. They have ten minutes to study the material before trading begins. Figure 4: Ratings System for Bike Pumps and Crossover Vehicles Black R ed All participants are provided with an identical portfolio that consists of $10,000 of cash and 100 shares for each of the securities. No borrowing or short-selling is allowed in the market. At the end of the experiments, we reward the top three players with highest portfolio values with Amazon.com gift certificates of $50, $30 and $20 respectively. Participants first log in with selfchosen user-names and passwords (see Figure 5), then trade the securities through a graphical user interface (see Figure 6). 16

17 Figure 5: Figure 6: STOC Trading User Interface 17

18 Market information available to the traders includes the last transaction price and size, current bid/ask prices and sizes, and a historical price and volume chart for each security. A trader can submit either a limit or market order to trade, or cancel an outstanding order that has not been executed. The markets are typical double auction markets with no market-makers. A transaction occurs when a market or limit order matches with a limit order on the opposite side of the market. All prices are specified in one-sixteenth of a dollar. 4 Possible Trading Strategies Our market experiments serve to aggregate diverse preferences or beliefs from all participants. One s beliefs may consist of three independent elements: 1. Product Information. This is what a participant knows about the underlying products. All participants are provided with the same facts and specifications of the products, but they may have obtained extra product information from their personal experience outside the experiments. 2. Personal Preferences. This is what surveys and polls try to collect. Although the aggregate preferences of the whole market is the object of interest, one s personal view should contribute to his or her trading decisions. 3. Assessments of Others Preferences. A participant may form opinions of what others think so as to make profitable trading decisions. How are beliefs or preferences aggregated in these markets with virtual securities? Not only should the traders form their own assessment of the stocks, but they should also infer the stocks potential market value from the market. In a typical market in experimental economics, both the preferences of the traders and the state of nature (for example, probability distribution of a security payoff) are known to the researchers (Plott & Sunder 1982, Plott & Sunder 1988, Forsythe & Lundholm 1990, O Brien & Srivastava 1991). Traders are assigned preferences that specify securities payoffs in various possible states. The theoretical equilibrium (rational 18

19 expectations equilibrium) prices can be derived given full information of the markets. The main focus of these experiments is whether and under what conditions rational expectations equilibria can be attained in double auction markets. Some attempts have been made to understand the convergence of prices and how learning occurs in the market as a whole. But it is unclear how individual human traders learn and react to the market. Attempts to model the trading strategies of individual traders from the market data may be overly ambitious. Here we try to shed some light on some possible strategies. The objective of the trading game is to predict the final prices of the securities. A trader may form an assessment of the fair values of the securities before trading starts. This opinion may take into account her own preference on the underlying products, and perhaps more importantly what she perceives as the preferences of the whole group. The trader may then make trading decisions based on her belief: she buys undervalued stocks and sells over-valued ones. During the course of the market, the trader may either maintain her valuations or update her beliefs in real time conditioning on her observation of the market dynamics. Learning is taking place if the latter approach is taken. But learning is a rather complex process because one s expectations of prices affect prices, prices are used to infer others assessments, and the inference of others assessments in turn affects both prices and expectations of prices. Some traders may take a dramatically different approach by largely ignoring all fundamental information about the underlying products and focusing on market information only. These traders play the roles of speculators or market-makers who try to gain from the market by taking advantage of price volatility, providing liquidity, or looking for arbitrage opportunities. Their presence may introduce mixed effects to the market. While they could 19

20 enhance liquidity on one hand, they may also introduce speculative bubbles and excess volatility into the market. In summary, STOC markets are meant to be microcosms of the general population, hence some combination of naïve traders, long-term investors, and predatory arbitrageurs will be present in a broad group of participants. To the extent that a particular target population is different from the general population, the STOC market participants should be similarly biased. The dynamics of the interactions between different groups within a given population is quite complex (Farmer & Lo 1999, Farmer 2002), and are beyond the scope of our study, but the principal of information revelation via the price-discovery process is the key to the STOC market s ability to infer aggregate preferences for products and concepts. 5 Experimental Results and Comparison with Survey Study Two trading experiments were conducted from September 1999 to April Subjects in the trading experiments are MBA students from MIT Sloan School of Management. 8 Table 1: Data Describing two STOC Trading Experiments for Bike Pumps Experiment Number of Duration Volume Volume Participants (Minutes) (Shares) (Trades) Venue , Trading Room , Internet Experiment 1 was conducted at a centralized location, the Sloan school trading laboratory and was not timed---we closed the market when trading activities died down. It lasted 10 minutes. Experiment 2, on the other hand, was conducted over the Internet---participants joined the market remotely from arbitrary locations. The market was open for trading for one hour, during which participants might enter or leave the market as they wished. The longer trading 8 Three groups of students were recruited from Prof. Ely Dahan s class New Product Development in fall 1999; two other groups were recruited from Prof. Andrew W. Lo s class Investment in Spring

21 time was aimed to offer more flexibility to the participants and investigate whether time constraints affect the trading activities of a market. Table 1 describes the details of the experiments. Experiment 2 has the higher volume (in share and trade) despite a smaller number of participants. This seems to suggest that a longer duration does generate more trading interest. Figure 7 presents the sample price and volume history of a virtual stock---airstik---in Experiments 1 and 2. The prices close around $25. 21

22 Figure 7: Price and Volume History of AirStik Experiment 1 Experiment 2 For each of the market experiments, trade and quote data is collected. For analysis, we focused on the trade data, which consists of the time series of trading prices and quantities ( p 1, i, q1, i),( p2, i, q2, i),...,( pt, i, qt, i) i i, where i is the index for the i th product and T i is the total number of the cleared trades for the i th product. Our hypothesis is that prices reveal market consensus of the profitability of the bike pumps. To provide an analogous study to that by 22

23 Dahan and Srinivasan (2000), we focused on the potential market share of the products. In particular, we propose that a product s market share can be predicted by its relative market capitalization. The market capitalization of a company, or the total value of its stocks, equals to the product of market price and the number of outstanding shares. The relative market capitalization is defined as the ratio of a company s market capitalization to the capitalization of the entire market (all the companies). Since all the companies have the same number of outstanding shares, market capitalization is proportional to the market prices. The market closing price is a natural candidate for the valuation of the companies. However, it is observed that the closing price is not a particular robust measure for stock valuation. Because the traders portfolios are valued at the closing prices, prices tend to become more volatile towards market close. This is especially true for the low-volume stocks. Hence, in addition to the closing price, we consider other price statistics that take into account all transactions during the session: the high, low, mean, median and volume weighted average prices. The high, low, mean and median prices are calculated from the time series of trade prices p 1, i p2, i,..., pt i, i, ; the volume-weighted average price (VWAP) is computed as follows: VWAP i = T i t = 1 T p i t = 1 t, i q q t, i t, i The mean, high and low prices are sensitive to outliers---a small number of transactions that occur at extreme prices. All but VWAP ignore the volume in a transaction and treat all transactions equally. Volume can be regarded as a measure of the amount of information in a transaction. A trade with higher volume may well be more informative than one with lower volume, since traders are risking more when they trade larger quantities of a stock. In our 23

24 concept markets, volume is also related to how confident the traders are at the corresponding transaction price. VWAP effectively summarizes the prices by considering the amount of information and confidence behind the trades. In practice, VWAP has been a widely accepted benchmark price in financial markets. It is a commonly used metric for the evaluation of trade executions. Now given a price statistic p~ i, which can be the high, low, closing, mean, median or volume weighted average prices, we can arbitrarily compute pred icted market share as the relative market capitalization, ~ pin ~ pi MS i = =, ~ N ~ p N p j = jn 1 j = 1 j where N is the number of securities comprising the market (in this case we compare each concept against two real world products so that N = 3) and n is the total number of shares for a security. Among the four price statistics, we expect the median price and VWAP to be particularly robust against potential price volatility. To relate these metrics to those in the computer simulations, it is important to point out that the mean price in Appendix A is equivalent to the VWAP because all transactions are restricted to one share. To verify the validity of the market method, we ask two questions: (1) whether the results from the market method are consistent across different experiments, and (2) how close the results from the markets are to those from the independent survey study. We focus on the market share predictions derived from the four types of price statistics and those from the survey. 24

25 Skitzo Table 2: Market share predictions for Virtual Concept Test (VCT) and STOC Silver Bullet Epic Rim Gripper Gear-head TRS 2wister Gecko Cyclone Correl Physical Physical 31.4% 30.4% 9.8% 4.9% 2.0% 2.0% 1.0% 0.0% 0.0% Web 27.6% 32.2% 11.5% 2.3% 1.1% 1.1% 0.0% 2.3% 0.0% Experiment % 37.9% 30.5% 16.7% 21.1% 20.6% 18.7% 29.9% 19.3% Experiment % 32.9% 31.1% 25.3% 26.8% 25.1% 26.1% 25.1% 23.3% Correl. Web Physical r2 Web r2 Table 2 presents the predicted market share based on VWAP for the two experiments. We find that the top three products (Skitzo, Silver Bullet and Epic), in terms of predicted market share, are the same in the two experiments, as well as from the survey study. Furthermore, the rankings among the three are exactly the same across different experiments. In a typical concept testing process, it is more important to be able to identify the best designs because they are more likely to materialize. Figure 8: Comparison of STOC Results vs. Physical and Web-based Concept Tests STOC Outcomes vs. Market Research Based on Median STOC prices Market Share Potential 40% 30% 20% 10% 0% Web r 2 = 0.76 Web, 0.64 Physical r 2 = 0.86 Web, 0.83 Physical Experiment 2 Experiment 1 Skitzo SilverBullet Epic RimGrip Gearhead TRS 2wister (9) Bike Pumps Gecko Cyclone Physical For consistency across experiments, we calculate the pair-wise sample correlation between market share predictions based on each price statistic from individual experiments. For 25

26 example, the association between experiments a and b is quantified by the sample correlation a a a between MS,MS,...,MS ) ( 1 2 N b b b and MS,MS,...,MS ). For comparison with the survey ( 1 2 N method, we calculate the sample r-squared s between the market share predicted by the survey study and those derived from individual experiments. These r-squared coefficients are presented in Table 3. The results from the two experiments show significant correlation. The r-squared s for VWAP, Median and Mean security prices demonstrate good prediction of survey results based on STOC trading and a high degree of consistency between independent STOC tests. As we expected, the closing price is too noisy to give conclusive results. Table 3: Price Based R 2 : STOC Experiments vs. Web-Based Virtual Concept Test (VCT) VWAP VCT Web Experiment 2 Experiment Experiment Median VCT Web Experiment 2 Experiment Experiment Mean VCT Web Experiment 2 Experiment Experiment Close VCT Web Experiment 2 Experiment Experiment To ensure robustness of the results, we present another measure of association between different market share prediction results: Spearman s rank correlation. We transform all the market share data into ranks, R = rank( MS ), and calculate the sample correlation (see Table 4). i i 26

27 A similar conclusion is reached: the results from the two experiments are significantly correlated among themselves and with the survey data. Table 4: Rank Order Correlations: STOC Experiments vs. Web-Based Virtual Concept Test Rank Correlation VWAP VCT Web Experiment 2 Experiment Experiment Median VCT Web Experiment 2 Experiment Experiment Mean VCT Web Experiment 2 Experiment Experiment Close VCT Web Experiment 2 Experiment Experiment It is also interesting to quantify the association between experiments and the survey by noting their average absolute difference in the predicted market share. The difference between experiments a and b is computed as follows: MS = N i= 1 MS a i MS b i The mean absolute error (MAE) based on VWAP is shown in Table 5. Experiments 1 and 2 show differences between each other similar to those in the literature. 27

28 Table 5: Mean Absolute Error (MAE) in market share between VCT and STOC Experiment 1 Experiment 2 VCT 5.5% 6.8% Experiment 1 1.8% Table 6: Mean Absolute Error (MAE) in market share between VCT and STOC Experiment 1 Experiment 2 VCT 5.5% 6.8% Experiment 1 1.8% The results from our bike pump experiments show a remarkable agreement with those from the survey study despite many fundamental differences between the two methods. These differences include the differences in the data collection mechanism (a virtual security market versus a virtual shopping experience), the modeling of the predicted market share (the use of relative market capitalization of the virtual securities versus conjoint analysis), the questions asked (what you prefer versus what the group prefers), and lastly the subject population (MIT students versus Stanford students). The crossover vehicle results shown in Figure 9 are also quite informative about preferences over the eight vehicles being tested. Figure 9: Comparison of STOC Results vs. Choice Survey for Crossover Vehicles STOC Median Price 40% 30% 20% 10% 0% STOC Median Price vs. Top 3 Choices (8) Crossover Vehciles - SDM October 20, 2000) BMW Mercedes Audi Lexus Pontiac Buick Toyota Acura 0% 10% 20% 30% 40% Top 3 Choice STOC VWAP STOC Vol.-Weighted Avg. Price vs. 1st Choice (8) Crossover Vehicles - SDM's Oct. 20, % 30% 20% 10% 0% Audi Lexus BMW Buick Pontiac Mercedes Acura 0% 10% 20% 30% 40% 1st Choice Toyota 28

29 STOC Price vs. Top-3 Choices March 20, 2001 MBA Test 1 25% 20% % "Share" 15% 10% 5% 0% Acura Toyota Mercedes Lexus Audi BMW Pontiac Buick Median Price % r 2 2 = 0.60 VWAP % r 2 2 = 0.63 MBA Test 1 Top 3 (8) Crossover Vehicles Figure 10: Six STOC Metrics Correlations Between STOC and 1st Choice (Max - Average - Min for four independent trials) Correlation Median VWAP Mean Close Min Max STOC Metric 29

30 Rank Difference (Buyer - Seller) Figure 11: How Do Preferences Affect Individual Trades? STOC Traded Vehicles Are Close in Rank (n = 165 transaction pairings) % of Trades 30% 25% 20% 15% 10% 5% 0% % of trades are in the "right" direction STOC Random Figure 12: What determines Trading Success? Session r 2 = Session r 2 = 0.31 Trader Rank Using Actuals Trader Rank Using Actuals Trader Rank Using Market Trader Rank Using Market 6 Conclusions In this paper we study a novel application of the market mechanism: the use of experimental securities markets to aggregate and infer diverse consumer preferences. We 30

31 implement this idea in the specific context of a product-concept testing study that aims to predict potential market share for several product prototypes. The results from three market experiments show remarkably high consistency among themselves, and significant correlation with an independent survey study. The sefficacy of STOC markets may not be particularly surprising to economists. After all, Keynes (1958) commented on the similarities between stock selection and a beauty contest over a half-century ago:...professional investment may be likened to those newspaper competitions in which the competitors have to pick out the six prettiest faces from a hundred photographs, the prize being awarded to the competitor whose choice most nearly corresponds to the average preferences of the competitors as a whole... The analogy is perhaps more accurate for describing what happens in the stock market in the short run. After all, over the long run stock prices depend not only on investors subjective beliefs or expectations, but also on other objective information such as companies earning potential and valuations of assets. On the other hand, the trading experiments presented in this paper are precisely beauty contests, since values of the virtual securities are derived endogenously from the expectations of the market participants, which are largely subjective. To improve the reliability of these virtual markets, one may need to anchor the values of the securities to some objective fundamental variables of the corresponding products. To test predictions of market shares, for example, one could compare security values with the realized market shares of the corresponding products, or, barring the existence of real market share data, 31

32 with the outcomes of actual customer choice surveys. We hope to refine STOC market methods along these lines in future research. 32

33 7 References Box Office Muddle: How Will Films Fare From Week to Week, (1996) Wall Street Journal, July 21, Business Week, November 11. Burchill, G. W. and C. H. Brodie (1997). Voices into Choices. (Cambridge, MA: Center for Quality Management. Calder, B. J. (1977). Focus Groups and the Nature of Qualitative Marketing Research. Journal of Marketing Research, 14, Chan, N. (2001) Ph.D. Thesis: Artificial Markets and Intelligent Agents, February. Chan, N., E. Dahan, A. Lo and T. Poggio (2001). Experimental Markets for Product Concepts, CBCL Working Paper #200/AI Memo # , Massachusetts Institute of Technology, Cambridge, MA, July. Chen, K. and C. R. Plott (2002), Information Aggregation Mechanisms: Concept, Design and Implementation for a Sales Forecasting Problem, California Institute of Technology. Dahan, E. and J. R. Hauser (2002), The Virtual Customer, Journal of Product Innovation Management, 19 (5), and V. Srinivasan (2000), The predictive power of internet-based product concept testing using depiction and animation, Journal of Product Innovation Management 17(2), Davis, D. and C. Holt (1993). Experimental Economics. Princeton, NJ: Princeton University Press. Fama, E. F. (1970), Efficient Capital Markets: A Review of Theory and Empirical Work, Journal of Finance, 25, (1991), Efficient Capital Markets: II, Journal of Finance, 46 (5), Farmer, D. (2002), Market force, ecology and evolution, Industrial and Corporate Change 11, Farmer, D. and A. Lo (1999), Frontiers of finance: Evolution and efficient markets, Proceedings of the National Academy of Sciences 96, Fern, E. F. (1982). The Use of Focus Groups for Idea Generation: The Effects of Group Size, Acquaintanceship, and Moderator on Response Quantity and Quality. Journal of Marketing Research, 9, Forsythe, R. and R. Lundholm (1990), Information aggregation in an experimental market, Econometrica, 58,

34 , F. Nelson, G. Neumann, and J. Wright (1993), The Iowa Presidential Stock Market: A Field Experiment, Research in Experimental Economics, 1-43., T. Palfrey, and C. Plott (1982), Asset Valuation in an Experimental Market, Econometrica, 50 (3), , T. Rietz, and T. Ross (1999), Wishes, Expectations and Actions: A Survey on Price Formation in Election Stock Markets, Journal of Economic Behavior & Organization, 39, Green, P. E. and Y. Wind (1981), New way to measure consumers judgment, in Y. Wind, V. Mahajan & R. N. Cardozo eds., New Product Forecasting, D.C. Heath and Company, Lexington, MA, and V. Srinivasan (1990), Conjoint Analysis in Marketing: New Developments With Implications for Research and Practice, Journal of Marketing, Griffin, A. J. and J. R. Hauser (1993), The Voice of the Customer, Marketing Science, Winter, Grossman, S. J. (1981), An introduction to the theory of rational expectations under asymmetric information,. Review of Economic Studies, 48, Hanson, R., R. Oprea, and D. Porter (2005), Information Aggregation and Manipulation in an Experimental Market, Journal of Economic Behavior and Organization, 60 (4), Hayek, F. (1945), The Use of Knowledge in Society, American Economic Review, XXXV, No. 4; September, O Brien, J. and S. Srivastava (1991), Dynamic stock markets with multiple assets, Journal of Finance, 46, Pennock, D. M. and M. P. Wellman (1997), Representing Aggregate Belief through the Competitive Equilibrium of a Security Market, in Proceedings of the Thirteenth Conference on Uncertainty in Artificial Intelligence, Pennock, D. M., S. Lawrence, C. Giles, and F. Nielsen (2001), The real power of artificial markets, Science, 291, Plott, C. R. and S. Sunder (1982), Efficiency of experimental security markets with insider information: An application of rational-expectations models, Journal of Political Economy, 90, and (1988), Rational Expectations and the aggregation of diverse information in laboratory security markets, Econometrica, 56, Servan-Schreiber, E., J. Wolfers, D. M. Pennock, and B. Galebach (2004), Prediction Markets: Does Money Matter?, Electronic Markets, 14 (3),