Performance Assessment of the Lead User Idea Generation Process for New Product Development

Transcription

1 Performance Assessment of the Lead User Idea Generation Process for New Product Development By Gary Lilien, Penn State Pamela D. Morrison, University of New South Wales Kathleen Searls, ASI Associate Mary Sonnack, 3M Eric von Hippel, MIT ISBM Report Institute for the Study of Business Markets The Pennsylvania State University 402 Business Administration Building University Park, PA (814) or (814) Fax

2 Performance Assessment of the Lead User Idea Generation Process for New Product Development Gary L. Lilien*, Penn State Pamela D. Morrison, University of New South Wales Kathleen Searls, ASI Associates Mary Sonnack, 3M Eric von Hippel, MIT March 2001 Acknowledgements: The authors gratefully acknowledge comments and suggestions by John Roberts and Christophe Van den Bulte and appreciate the support of Penn State s Institute for the Study of Business Markets and MIT s Innovation Lab. *Corresponding author: 402 BAB, Penn State, University Park, PA 16802; , G5L@psu.edu.

3 Performance Assessment of the Lead User Idea Generation Process for New Product Development Abstract Traditional idea generation techniques based on customer input usually collect information on new product needs from a random or typical set of customers. The lead user process takes a different approach, collecting information about both needs and solutions from users at the leading edges of the target market, as well as from those in other markets that face similar problems in a more extreme form. This paper describes the unique characteristics of the process and report on a natural experiment conducted within the 3M Company on the effect of the lead user idea generation process relative to more traditional methods. 3M is known for its innovation capabilities and we find that the lead user process improves upon those capabilities. Annual sales of lead user product ideas generated for the average lead user project at 3M are conservatively projected to be $146 million after 5 years - more than eight times higher than sales for the average contemporaneouslyconducted traditional project. Each funded lead user project created a major product line for a 3M division. As a direct result, divisions funding lead user project ideas experienced their highest rate of major product line generation in the past 50 years. Keywords: new product development, lead users, idea generation IGLU - 03/09/01 1

4 Section 1: Introduction Many firms use customer-focused new product development processes to generate new product ideas based on information collected from current or potential users. What distinguishes such processes across companies is the kind of information they collect and the respondents from whom they collect it. Most traditional market research techniques collect information from users at the center of the target market. The lead user (LU) process takes a different approach, collecting information about both needs and solutions from the leading edges of the target market and from markets that face similar problems in a more extreme form. Our key research question is: how does the LU process influence the quality of the idea generation stage of new product and service development? More specifically, we investigate the outcomes of that process, and the individual/team characteristics that may influence the effectiveness of that process relative to more traditional idea generation approaches. In this paper we report on results of the IGLU (Idea Generation with Lead Users) research program. The first phase of the program was a qualitative examination of new product development-related attitudes and actions observed among informants working at 3M, a corporation known for its cultural emphasis on innovation. In this phase we studied language use, shared assumptions and culturally valued frameworks for understanding innovation-oriented activities. The second phase of the program is a natural experiment conducted within the several 3M divisions that used both the LU methods and more conventional approaches for new product idea generation. We collected outcome data (new strategies being implemented, new markets identified, new product and service concepts) as well as pre- and post-project team process skills IGLU - 03/09/01 1

5 inventories that allowed us to evaluate changes in team member skills due to LU project participation. Our findings show that the LU method leads to a higher rate of breakthrough new product generation at 3M than traditional methods produce: compared with traditional projects, funded ideas emerging from LU studies had higher novelty (usually being judged new to the world ), and their estimated sales in year 5 were more than eight times higher than those of the traditional contemporaneously-funded projects an average $146 million in forecast annual sales for LU projects versus an average $18 million for the other projects (after adjustment for expected forecast bias). Funded ideas from a total of only 5 LU idea generation projects are conservatively projected to yield $730m in incremental annual sales for 3M. We proceed as follows: In section 2 we review relevant literature and motivate and describe our research hypotheses. In section 3, we describe our research methods. In section 4 we present our findings. Section 5 discusses the implications of the study, and section 6 offers suggestions for further research. Section 2: Research Context, Research Site and Hypothesis Generation Despite the strategic importance of effective new product development as a source of competitive advantage, most new product development activities fail to achieve their anticipated level of market success. Eliashberg et al. (1997) report on a survey of 154 senior marketing officers of US corporations, 79% of whom believe their new product development process could be significantly improved. A major complaint that that study identified (see Booz, Allen and Hamilton, 1982, as well) was that most of the resulting products tended to be marginal contributors to the firm s portfolio, rarely resulting in radical or breakthrough ideas. IGLU - 03/09/01 2

6 To assess the range of market research methods that might be used to help enhance the output of new product development processes, Eliashberg et al. (1997) classify those methods along two dimensions: type of product sought (radically new vs. marginally new) and nature of the research method (traditional versus nontraditional) (Table 1). As they point out, most traditional market research methods that address the idea generation stage of the process use respondents at the center of the market whose thinking is limited by their current experience base and environment. [Insert Table 1 here] As Eliashberg et al. (1997) also point out, the traditional methods in box 1 in Figure 1 have generally failed to produce the kind of radical new product breakthrough that developers had hoped, leading to interest in nontraditional methods. Some successes have been reported for information acceleration (Urban, Weinberg, and Hauser1996) and virtual reality based research (Pels and Merrick, 1993 and Crispell, 1993), methods aimed at the assessment and testing stages of the new product development process. The only non-traditional idea generation method they identified was the lead user method. In their study of 127 successful and unsuccessful new consumer products Goldenberg et al (2001) point out that the marketing literature has paid little attention to the way an idea is generated as a possible predictor of it success. (p 71). Their analysis supports the results of Finke et al (1992), showing that ideas based on solution spotting (where the solution either precedes traditional market research or where the solution and the need are spotted simultaneously) are superior to those associated with prior need assessment (need spotting). The lead user process emerged from two research findings about the sources of innovation. First, users rather than manufacturers are often the initial developers of what IGLU - 03/09/01 3

7 later become commercially significant new products and processes (Enos 1962, Knight 1963, Freeman 1968, Lionetta 1977, von Hippel 1976, 1977, 1988, VanderWerf 1990, Shaw 1985, Shah 1999). Second, innovation by users tends to be concentrated among lead users of those products and processes (Urban and von Hippel 1988, Morrison, Roberts and von Hippel 2000, Shah 1999, Luthje 2000). Lead users are defined as users of a given product or service type who combine two characteristics: (1) they expect attractive innovation-related benefits from a solution to their needs and so are motivated to innovate, and (2) they experience needs for a given innovation earlier than the majority of the target market (von Hippel 1986). Research on innovation-related incentives and capabilities provides a theoretical basis for these empirical observations on innovation by users in general, and innovation by lead users in particular. From the perspective of innovation as an economically motivated activity (Schmookler 1966, Mansfield 1968), those users expecting significantly higher economic or personal benefit from developing an innovation one of the two characteristics of lead users have a higher incentive and are more likely to innovate. Also, given that lead users experience needs in advance of the bulk of a target market, the nature, risks, and eventual size of that target market are often not clear to manufacturers. This lack of clarity can reduce manufacturers incentives to innovate, and increase the likelihood that lead users will be the first to develop their own innovative solutions for leading-edge needs including those that later prove to represent mainstream market demand (Gans and Stern 1999). Customer driven new product idea generation processes have yet to incorporate these ideas, with all but the lead user process focusing on new product need data from average or major customers within target markets.. Griffin (1997) found that the three IGLU - 03/09/01 4

8 most important market research tools for new product development were: Voice of the Customer, Customer Site Visits, and Beta Site Testing, all of which focus on existing customers. Other tools used to support ideation include brainstorming (Rossiter and Lilien, 1994) focus groups, customer visits (McQuarrie, 1998), conjoint analysis and some promising new work on channeled ideation (Goldenberg et al, 1999) Griffin reports that solution generation then follows, typically executed by internal corporate personnel without input from external sources. In other words, traditional idea generation processes and tools are designed to collect new product need and solution data primarily from sources other than lead users. Urban and von Hippel, (1988) evaluated the LU concept in a laboratory setting, and showed that new product concepts generated from lead user data (in the area of computer-aided systems for the design of printed circuit boards PC-CAD) were strongly preferred to those generated from more mainstream users by a sample of PC-CAD users. That encouraging evaluation, along with the above observations, suggest that a carefully monitored field assessment of the effectiveness of an implementation of the procedure would be of value. Von Hippel, Churchill and Sonnack (2001) have developed an idea generation process designed to collect both need and solution data from lead users. This process involves four major steps and usually takes four to six months to complete, as follows: Step 1: Goal Generation and Team Formation. A firm provides an ideationgoal for the project and forms a team. Teams typically are composed of 3-5 experienced people from marketing and technical departments, with one member serving as project leader. Team members typically spend 12 to 20 hours per week on a project for its duration. During the initial step, the LU project team is formed and works with relevant IGLU - 03/09/01 5

9 key company stakeholders to select the general market it will target and the type and level of innovation desired. For instance, does the company seek a breakthrough product or does it wish to merely extend current product or service lines? At the same time, the team identifies key business constraints. Step 2: Trend Research. During step 2, LU teams focus on identifying and deeply understanding important market and technical trends in the field being explored. Team members begin by reviewing conventional information sources. Next, they systematically identify and interview leading experts in the marketplace that they are exploring - people who have a broad view of emerging technologies and leading edge applications in that field or fields. They then select an important trend or trends as the central focus of further project work. Step 3: LU Pyramid Networking. In step 3, unique to the LU process, the project team engages in a Pyramid networking exercise to identify and learn from users at the leading edge of the important trends selected for as a study focus. They identify lead users both in the target market and in other markets that face similar needs in an even more extreme form. Then, the team learns from those lead users about needs and solutions they are encountering at the leading edge, seeking to identify information and ideas that might hold the seeds of a breakthrough. The Pyramid Networking technique is a modified version of the snowballing technique used by researchers to identify sample members having rare characteristics in common. Snowballing utilizes the notion that people frequently know people like themselves, and accumulates a sample by asking people with the desired, rare characteristics to identify other people like themselves. In contrast, the Pyramid technique relies on the fact that people with a strong interest in a topic or field tend to know people more expert than themselves: for example, good computer science professionals tend to know the identity of computer scientists who are IGLU - 03/09/01 6

10 even more skilled than they are. In the Pyramid technique, LU project teams use telephone interviewing to network their way up pyramids of expertise (see Figure 1, which also includes the percentage of contacts of each type identified in the field research reported on later) to identify increasingly expert lead users in a field of interest. Team members begin with relatively expert users identified from articles and other sources. Then, they ask each user contacted to identify users who are even more advanced, seeking users at the leading edge of the target market, and then to identify lead users in other markets facing similar challenges in an even more advanced or extreme form ( advanced analog markets ). Team members conclude this step by beginning to shape the information they have collected into preliminary product, service and strategy ideas, and to assess the business potential of these for their firm. [Insert Figure 1 here] Step 4: LU Workshop and Idea Improvement. In the fourth and final step of a LU project, activity centers around a lead user workshop in which invited lead users work with company personnel to improve the preliminary concepts generated in step 3 and sometimes to generate entirely new concepts (An economic rationale for users to freely share their ideas in such a forum is developed in Harhoff et al., 2000). Typically, 10 to 15 people attend this workshop, of which a third may come from the firm sponsoring the study. Participants first work in small groups and then work as a whole to design final concepts that fit the company's needs. Thereafter, the entire group evaluates the concepts in terms of technical feasibility, market appeal, and management priorities. After the workshop, the organization conducts more traditional market research to test the concepts against the needs of the target market and also tests the concepts with key stakeholders in IGLU - 03/09/01 7

11 the firm. The team then makes any necessary refinements and presents its recommendations to senior management. Our focus in this paper is to compare the effectiveness of the LU procedure described above in a field setting against procedures used by non LU teams in the same organization. Following Cook and Campbell (1979) we sought a field situation that could closely approximate a pre-post/test-control situation, with at least quasi-random assignments to treatment cells. Such a situation requires a participating organizational field site that satisfies three conditions. First, the site should be one for which innovation has historically been strategically important, to ensure that the LU intervention would not signal a change (increase) in the strategic incentive to innovate. (That is, we sought a challenging test environment, one in which the culture for and incentives to innovate were historically strong). Second, the site must be using more traditional methods, which could act as a control within the same organizational units in parallel with the LU method. Third, the site must have data on a number of projects utilizing both LU and traditional idea generation processes for study, so that we can statistically distinguish the effect of the methods from other firm or industry-specific characteristics. At the time of our study, we found that 3M satisfied these requirements, as follows: 1. 3M has historically been known for and has always placed major emphasis on innovation. To illustrate, its 1999 Annual Report is titled The Elements of Innovation, and its letter to shareholders states, For nearly 100 years, 3M has grown by pioneering innovative technologies and products. Innovation remains the driving force of 3M culture and growth. (3M, 1999 Annual Report, p. 12.) 2. 3M began formally training staff in one division with the LU method in 1996 and by May 2000, LU training had been carried out in several others, each of which was at IGLU - 03/09/01 8

12 the same time employing more traditional idea generation methods. These divisions could then play the role of experimental units. 3. By May 2000, 3M had completed 7 LU projects and had funded further development of the ideas generated by 5 of these. The divisions carrying out LU projects also had 42 contemporaneous funded projects (that had utilized a range of traditional idea generation processes) available for study. In contrast to the LU teams, non LU teams collected market information only from target markets. Data collection methods varied from project to project and included market data collected by outside organizations, focus groups with major customers, customer panels, customer requests relayed through sales personnel and advances introduced by competitors. In all cases, idea generation was carried out by 3M product developers and sales personnel with product designs generated in 3M labs. All LU and non LU product developers were charged to seek important innovation breakthroughs. As existing research does not report specific, in-company experience with LU idea generation methods, we conducted an inductive, qualitative research phase at 3M to generate more specific research hypotheses and associated measurements. Our approach utilizes a research tradition informed by both ethnographic (Douglas 1986; Willis 1981; Lave 1988, Van Maanen 1988; and Workman 1993) and Grounded Theory (Glaser and Strauss, 1967) perspectives, incorporating three goals: 1 Allow data and the field to drive theory creation. Product developers throughout the firm shared their time with us and showed considerable patience early on as they explained what to look for, where to look and why we should expect to find (or not) certain themes, as we sought to understand how innovative product concept generation is understood, communicated and practiced at 3M. IGLU - 03/09/01 9

13 These same professionals then led us through their professional networks and their communities of practice (Wenger 1998), to collect deeper insights prior to formulating our hypotheses. Our approach also provided us the opportunity to gain insight into the language and terminology used by 3M new product developers. 2 Develop trust-based relationships between 3M new product development professionals and researchers to ensure researchers were properly focused. To implement this goal, we began our grounded research activities only after obtaining strong top management support and permissions for access to confidential company data. These permissions enabled unguarded and informal relationships to develop between the research team and 3M product development team, increasing the willingness of product developers to shape, direct and redirect researcher attention. 3M informants tested and critiqued our interview agendas, data collection instruments, survey schedules and our early interpretations of our observations. At a few important junctures, informants simply told us You re asking the wrong questions, and improved our ability to see and understand what the practitioners were actually doing in the course of their work. 3 Use fieldwork to drive measurement development. To implement this goal, we studied the variables and scales associated with 3M s own internal new product development measurement, prioritization and resourcing decisions. We were then in a position to integrate these measures and scales with those gleaned from the academic literature. IGLU - 03/09/01 10

14 In the course of our grounded research at 3M, we specifically sought balance between LU and non LU participants. We interviewed more than 20 managers and related new product development professionals in several 3M divisions. These internal experts represented two broad categories: the technical/scientific side of new product development (the professional group historically most credited as being the source of the company s innovative product offerings) and the business side (including marketing professionals). Research team members also observed and participated in LU training sessions and studied relevant internal company documentation. We learned a number of things from the grounded theory process that shaped our research. All respondents agreed that they were strongly motivated to unearth breakthrough ideas, but LU team members seemed to feel that their approach might provide potential for qualitatively different types of ideas to emerge. However, some LU participants expressed concern that these breakthrough ideas might be less compatible with the 3M technical, production and market environment, and hence, less likely to be judged worthy of funding by 3M management. If this concern proved correct, it obviously would represent a major barrier to gaining actual marketplace benefit from breakthrough ideas generated by the lead user method. Also, managers expressed a concern that a process built upon distilling new ideas from lead user needs and prototype solutions would not generate ideas that could be effectively patented by 3M. Since patents are an important source of competitive advantage for 3M, this would represent a major drawback to the method if true. Finally, many LU participants expressed concern about the time and effort involved in the LU method relative to alternative approaches. We used these findings in conjunction with our review of the literature to develop both hypotheses and measurement instruments. (We will use the term non LU methods IGLU - 03/09/01 11

15 to refer to the range of more mainstream methods 3M has been traditionally employing for idea generation.) With respect to project outcomes our hypotheses are: H 1 : Lead user methods will generate opportunities with greater commercial potential than will non LU methods (i.e., the market potential per funded idea will be greater for LU output than for non LU output). H 2 : Lead user methods will generate breakthrough ideas that result in major product lines at a greater rate than will non LU methods (i.e., the LU method will increase the overall rate at which the organization generates major new product lines). H 3 : Projects from LU methods will exhibit a lower level of compatibility with organizational selection criteria than will non LU projects. H 4 : Ideas generated by LU methods will be less protectable by patent than will ideas generated by non LU methods. H 5 : Projects from LU methods will take more time to develop and cost more than will non LU projects. With respect to process outcomes our hypothesis is: H 6 : After participating in LU training, team members will have higher capability on key activities regarded as an integral part of the LU method than before training. Section 3: Research Samples and Methods Samples Our study analyzes two samples drawn from the 3M divisions that had completed lead user projects at the time of our quantitative data collection. The first is a sample of IGLU - 03/09/01 12

16 funded ideas. 3M has a formal set of stages through which ideas generated by any method must pass. The earliest stage involving critical management review is the initial request for funding. A funded idea, therefore, has received careful review and positive evaluations from 3M divisional managers and has received, at a minimum, initial funding within the 3M product development system. Our sample of funded ideas was a complete census of all ideas receiving funding in the 3M divisions that also funded LU project ideas during our data collection period of February, 1999 to May, We identified these ideas via divisional records of funded projects. The sample consisted of 5 funded ideas developed by LU studies ( LU ideas ) and 42 funded ideas developed by other, more traditional divisional processes ( non LU ideas ). In our qualitative work, we could identify no differences between the pressures for breakthroughs placed on LU and non LU team members. As we will show later, the backgrounds and capabilities of team members were similar as well. Hence, while not satisfying the random assignment criterion for experimental design, these samples appear to satisfy the rough equivalence criteria in test and control conditions associated with natural or quasi-experimentation. In order to incorporate a pre-post dimension in our research, we developed a second sample, consisting of all major product lines introduced to the marketplace between by the 3M divisions that executed LU studies. Major product lines such as the line of Scotch transparent tapes and the line of Post-It memo pads - are vital to the financial success of 3M. Following the advice of 3M controllers (and in line with Stalk, Pecaut and Burnett, 1996) we operationalize the concept of major product lines as those that were separately reported upon in divisional financial statements. (Divisional financial statements broke overall sales into several major product lines plus, typically, a IGLU - 03/09/01 13

17 residual category grouping minor products of a range of types.) In 1999 in the 3M divisions we studied, sales of individual major product lines ranged from 7% to 73% of total divisional sales Our major product line sample contains 21 major product lines developed by non LU methods that met our criteria and that had been developed and funded by the divisions executing LU studies during the period. It also contains 5 funded ideas developed by LU methods. The characteristics and projected sales of these LU ideas - conservatively deflated in line with historical forecast accuracy (see Section 4) all met our criteria for a major product line. Data Collection To address the what (characteristics of the ideas) and the how (characteristics of the process) questions, we selected appropriate respondents within each division from LU and non LU teams. We used multiple approaches to increase our response rate, including: pre-survey, face-to-face meetings conducted with each division contact (typically, but not always, the lead user team leader) to explain the data collection process and to gain trust and cooperation. The actual data collection (outcome and process measures) was done via with Microsoft Word / Form attachments sent to the individual respondents, and a week after delivery of the instrument an individualized follow-up phone call and e- mail (the communication means uniformly used within 3M) was sent if the Form(s) hadn t been returned. In addition to the ideas themselves as units of analysis, we are interested in the impact of the LU process on attitudes, behaviors and skills. Hence, we developed instruments to measure the performance of the various idea generation methods both IGLU - 03/09/01 14

18 through outcome data (new strategies, new markets identified, new product and service concepts) as well as through pre and post-project team process skills inventories. At a face-to-face meeting with LU team leaders we asked them to: (a) identify and recruit the members of their team as respondents, (b) identify and recruit appropriate respondents within the same division to provide non LU idea method data, and (c) describe the New Product Development goals and outputs from the LU project (see Tables 2b and 2c), provide organizational information about their division, and provide process information (Table 2c). [Insert Tables 2a,b,c here] After the meeting with the team leader, we contacted the respondents identified in (a) and (b) above. The respondents from non LU idea generation methods provided detailed information both on funded ideas (as per Tables 2a, b), as well as process data (as per Table 2c). The LU team member respondents provided process data only (as per Table 2c), as we had already collected the funded idea data from team leaders. For lead user teams that started their LU training during the study time period, we took process measures twice once before LU training, and, again, at the conclusion of the project. In order to increase the size of our LU process measure sample (from n = 7 to n = 47 individuals) we included LU projects that were either completed or underway when we began data collection, and for these teams, we could collect only post-training process measures. We took process measures once for the non LU team members (n = 32) at the conclusion of their projects. Table 3 outlines the measurement flow process. [Insert Table 3 here] IGLU - 03/09/01 15

19 We collected data on major new product ideas developed early in the period through in-depth interviews conducted with long-tenure employees in each of the LU study-divisions who had a good knowledge of their division s and product line histories. We converted all historical dollar figures to 1999 dollar equivalents using U.S. consumer price index data (Council of Economic Advisors 2000). Measurement Instrument Development. We developed separate measurement instruments to collect project outcomes and process activities. Outcomes Instrument. We developed the New Product Idea Description Form (Table 2b), from measures used in previous academic research: novelty of ideas, originality/newness of customer needs, and potential for proliferating into an entire product line from the New Product Creativity scale (Moorman and Miner, 1997); company sales/ market share from the new idea in year 5, and probability of business success from NPD Success Criteria (Griffin and Page, 1996); and global market potential for all competitors from Cooper, (1993). We supplemented these measures with items revealed during the exploratory phase of the research. These latter items included several that were in general use within 3M as part of their internal new product idea assessment procedure. We began with long batteries of scales and then we carefully pre-tested all items and refined them during preliminary instrument development meetings with participating staff at 3M. We tested if the scales had meaning to the respondents in their specific organizational setting and retained those that respondents said did have meaning. We collected completed outcome instruments (Table 2b) for all 47 funded ideas (5 LU major product lines, 1 non LU major product line, and 41 non LU incremental ideas) that were current in the period February 1999 to May 2000; that is, our database represents a IGLU - 03/09/01 16

20 census. We also collected responses for the same items for the additional 21 historical major product lines that we will be using for comparison, collected from key informants via in-depth personal interviews. Process Instrument. We developed the process instrument to measure individual skills in idea generation activities, personality traits, and individual characteristics such as job level, time in the job, and R&D/ marketing/ accounting/ manufacturing expertise. We developed items for the individual skills identified in the literature to be associated with generating new product ideas (Thomas, 1993) and supplemented those items with skills identified from interviews with lead user team leaders and their primary LU trainer. We measured personality traits using a web-based version of Myers-Brigg s Personality Inventory ( Forty-seven of the 50 (94%) members of the completed LU project teams and 32 (out of 37 contacted, 86%) members of the non LU teams provided the post-process measures (Table 2). In addition, the 17 members of those LU teams that started training during the data collection phase completed a preprocess measurement instrument which had the same process activity items as the postmeasure instrument. Of these 17 respondents who completed a pre-process measurement instrument, 7 had completed the lead user training and thus also provided post-process measures at the time of this data analysis. (We collected these latter data to view changes in perceived skill level associated with the lead user training via a within-subject design). Note that since 100% of the Idea Description Forms for LU projects and 79% for non LU projects were returned and 94% of the Process Measure Forms for LU projects and 86% for non LU projects were returned, the response rates were sufficiently high to suggest that validity checks for non-response bias are not needed: (Malhotra, 1996). IGLU - 03/09/01 17

21 Section 4: Analysis and Results In this section, we will analyze the three sources of data noted above to study what happened (outcome analysis) as well as how it happened (process analysis). In each case we will match, as best we can within a natural (quasi) experimental framework, the test condition (LU) versus the control situation (non LU). For the outcome analysis, we have two subsets of analysis: first, a cross-sectional analysis of the census of all contemporaneous funded ideas (that is, currently funded ideas in the data collection period February 1999 May 2000), and second, an intertemporal analysis of major product lines, for which we will compare the LU results with a 50 year history of major product line development at 3M. We will use both of these sets of data to test our hypothesis separately in the sections that follow and will use differences between treatment groups to test our hypotheses. In several of these analyses, we will be comparing forecast data (projected sales for a line of products not yet launched) against actual, historical sales. Armstrong (2001) reviews some literature on forecast bias for new product introduction that supports the notion that forecasts for new product sales are generally optimistic, but that that upward bias decreases as the level of the sales forecast increases. Indeed, Tull s (1967) model calculate $15 million in revenue as a level above which forecasts actually become pessimistic on average. In order to account for possible biases in the relationship between actual and forecast/estimated sales, we collected data from five division controllers responsible for authorizing investment expenditure for developing new ideas into products and also obtained data from a 1995 internal study that compared such forecasts with actual sales. By combining this information, we developed a distribution of forecast errors for a IGLU - 03/09/01 18

22 number of 3M divisions, plus overall forecast errors across the full corporation. Those errors range from forecast/actual of +30% (overforecast) to 13% (underforecast). Based on this information, and in consultation with 3M management, we use a 25% salesforecast deflator and apply that adjustment factor to all projected sales data in the analysis that follows. Based on 3M historical experience, and consistent with Tull s (1967) findings, that level of forecast deflation should provide conservative sales projection figures. Outcome Analysis 1: Comparison of Lead User and Non LU Funded Ideas We began our empirical analyses with a comparison of all funded ideas that were generated by LU and non LU methods during the time period of our data collection (February 1999 to May 2000). Table 4 provides a census of all funded ideas during the noted period in the 5 divisions that funded LU ideas. During that time, five ideas generated by lead user projects were being funded along with 42 ideas generated by non LU idea generation methods. [Insert Table 4 here] From Table 4, we see that LU ideas are significantly more novel than are ideas generated by non LU methods, address more original/newer customer needs, have significantly higher market share, have greater potential to develop into an entire product line, and are more strategically important. We also find that the LU ideas have projected annual sales in year 5 that are eight times higher than those of ideas generated by non LU methods an average of $146 million versus an average of $18 million in forecast annual sales. Thus, we find support for H 1 (p<0.005). The size of this difference is striking and we investigated further. We found that LU ideas differed in kind from ideas produced by non LU methods. Non LU methods IGLU - 03/09/01 19

23 were producing mainly funded ideas for product improvements and extensions to existing product lines. In contrast, the LU method was producing funded ideas that clearly fit 3M divisional criteria for major product lines. Those ideas were clearly different from existing product lines, and the projected sales of each fell well within (and sometimes exceeded) the proportion of divisional sales accounted for by existing individual divisional major product lines: projected sales five years after introduction for funded LU ideas, conservatively deflated as discussed above, ranged from 25% to over 300% of current total divisional sales. Table 5 shows the qualitative difference in the type of product (incremental vs. major product line/breakthrough) that the LU process has been generating. The probability of this outcome occurring by chance is p < 0.005, providing support for H 2. [Insert Table 5 here] To give a flavor of the type of innovations that LU process teams generated at 3M, here are descriptions of four: 1. A new approach to the prevention of infections associated with surgical operations. The new approach replaces the traditional one size fits all approach to infection prevention with a portfolio of patient-specific measures based upon each patient s individual biological susceptibilities. This innovation involved new product lines plus related business and strategy innovations made by the team to bring this new approach to market successfully and profitably. 2. Electronic test and communication equipment that, for the first time, enables physically isolated workers such as telecommunication equipment repair people to carry out their problem-solving work as a team. Linked workgroup members can contribute to the solution of a problem being worked upon by a single, physically isolated worker in real time. 3. A new approach, implemented via novel equipment, to the application of commercial graphics films that cuts the time of application from 48 hours to less than 1 hour. (Commercial graphics films are used, for example, to cover entire truck trailers, buses and other vehicles with advertising or decorative graphics.) The LU team ideas involved needed technical innovations plus related channel and business model changes to help diffuse the innovation rapidly. IGLU - 03/09/01 20

24 4. A new approach to packaging fragile items in shipping cartons to replace current packaging materials such as foamed plastic peanuts. The new product lines implementing the approach are more environmentally friendly and much faster and more convenient for both shippers and package recipients than are present products and methods. H 3 addresses the concern that the apparently more ambitious outcomes from the LU process would be less compatible with key organization-fit criteria than would non LU ideas. The last three items in Table 4 address this issue: we find no statistical difference in quality of fit of LU and non-lu ideas with respect to existing divisional distribution channels, manufacturing capabilities or divisional strategic plans. Hence, we cannot reject the null hypothesis H 3 that LU and non LU major product lines are equal on most dimensions of compatibility (p>.10). H 4 was that ideas generated by the lead user method would be less protectable by patents than would ideas generated by other methods. A single item in Table 4 tests this hypothesis and shows no statistical difference between intellectual property protection for LU vs non LU ideas. Apparently the novel synthesis of a number of lead user ideas that is carried out during the LU process enables 3M to obtain as effective intellectual property protection for LU process ideas as for non-lu ideas generated internally by the firm. Thus we cannot reject the null hypothesis H 4 that LU and non LU major product lines are equal on intellectual property protection (p>.10). H 5 addresses the issue of cost differences per idea between LU and non LU methods. An audit of idea development time shows that the generation of a funded LU idea consumed 154 (sd = 82) person days on average. In contrast, generation of an average non LU idea consumed an average 60 (sd = 43) person days. The hypothesis of equality of these two means is rejected at the p <0.05 level. Using internal 3M data on IGLU - 03/09/01 21

25 average costs of professional development personnel, we obtained a total cost of approximately $100,000 per funded LU project (including additional costs associated with the LU training) versus $30,000 for non LU projects. After accounting for the different probability of success for LU projects (Table 5: 80% for LU projects vs. 66% for the census of non LU projects), we determined that 3M pays about $80,000 more, on average, for a successful LU project than it does for a successful non LU project. These results provide support for H 5. Outcome Analysis 2: Major Product Lines. Following our finding that the LU method tends to produce ideas for major product lines, we compare the major product line ideas generated by the LU method with those that had been generated by the 3M divisions in our study via other methods. This comparison provides alternative means of testing H 1 and H 2. To generate a sufficient sample for statistical analysis, we went back to 1950 (as far back as we could go and still find company employees who could provide some data about the products) to capture major product line data in the divisions of interest. The major product lines tended to be long lived. (We were able to identify only two, relatively small ones that had been discontinued). Each major product line had also been broadened over time, resulting in general patterns of strong growth and good margins for the 3M divisions. Examples of historical major product lines in our sample include: 1. Scotch tape: A line of transparent mending tapes that was a major success in many household and commercial applications. 2. Disposable patient drapes for operating room use: A pioneering line of disposable products for the medical field now sold in many variations. 3. Box sealing tapes: The first type of tape strong enough to reliably seal corrugated shipping boxes, it replaced stapling in most corrugated shipper applications. IGLU - 03/09/01 22

26 4. Commercial graphics films: Plastic films capable of withstanding outdoor environments that could be printed upon and adhered to vehicles. This product line changed the entire approach to outdoor signage. 5. Insulation displacement-type connectors: This type of connector makes reliable electrical connections to telecommunication wires by displacing insulation and contacting the metal wire underneath. This technology represents a significant improvement over previous technologies, and has become the standard type of wire connector in the telecommunications field. During the period, the 3M divisions studied produced 21 major product lines. During the period, they produced 5 of those major product lines using LU methods and two using non LU methods. It is useful to look at the entire 4-year period during which the LU process was implemented at 3M and compare it (on a rolling basis) with all other four-year periods during The average rate of major product line development during this half-century for the divisions studied was 1.64 per 4-year period. As Figure 2 shows, the highest rate achieved over a 4-year period was 7 major product lines and this result was achieved during the period during which the LU process was implemented by those divisions. During this period, 5 of the 7 major product lines developed were generated by LU methods. Non LU methods generated 5 or more major product lines in two of the 47 4 year rolling periods. [Insert Figure 2 here] To revisit H 2, we look at the three comparisons: (1) LU output vs. pre-1997 output; (2) LU output vs. non LU output, ; and (3) Pre-1997 output vs. all post output in the divisions studied. For comparison (1), assume that 3M generates major product lines at a (constant) rate that represents the null hypothesis of no difference in rate of major product line generation over time. Using the four year data in Figure 2, that rate is 1.64 major product IGLU - 03/09/01 23

27 lines per four year period for non LU methods. Then, a formal test of our hypothesis is What is the likelihood that a process that generates λ = 1.64 events on average per study period could generate 5 (or more) in a period? ( for LU). Using an assumption of a Poisson generation process (Grassman, 1981), we calculate that P(5 or more λ=1.64) = 0.025, providing support for H 2 (rejecting the null hypothesis at the 0.05 level). For comparison (2), if we compare the period (5 major product lines for LU methods vs 2 major product lines for non LU methods), we get P(5 or more λ=4 x 0.5=2.0) = 0.052, again providing support for H 2. For comparison (3), if we consider the LU method an organizational intervention (Mills, 1999), we can compare the post-1997 period for 3M (blending both LU and non LU ideas) with the results of the previous regime. These results give 7 major product lines (5 LUs and 2 non LUs) in the 4 year period, versus the historical average (adjusted to account for four fewer intervals and 2 fewer major product lines) of 1.79, giving P(7 or more λ=1.79) = Thus, all three approaches provide further support for H 2. H 1, perhaps our most important hypothesis, also addresses the economic returns associated with the LU idea-generation investment. The previous subsection looked at a comparison of contemporaneous lead-user vs. all nonlead-user ideas and found support for this hypothesis. Here we consider an alternative to H 1, call it H 1A : Major product lines that emerge from LU methods generate greater commercial potential than do non LU methods. IGLU - 03/09/01 24

28 As major product lines occur relatively infrequently and because the major product lines that have emerged from lead user methods have yet to reach market maturity, we can only provide a suggestive test of this hypothesis with our data. To proceed, we make the following two assumptions: A1. Returns from all major product lines that have emerged at 3M using methods other than the LU method can be considered as draws from the same probability distribution. A2. Revenue projections for LU major product lines are biased to the same degree on average as historical projections have been for other major product lines. A1 allows us to compare the major product lines from the LU method with all major product lines in these divisions during the prior four decades. A2 allows us to use historical 3M figures on major product line forecast error to adjust the forecasts from the LU data. Following A1 and A2, for major product lines introduced to market in 1994 or earlier we used as a reference actual sales 5 years after introduction (including loss/gain from sales of related products in the division). 3M management maintains such records and the 5-year sales goal is part of 3M s project justification process. To provide comparable data, we translated all sales data into 1999 dollars. Following A2 for the major product line ideas generated from the LU process (and for two major product line ideas recently generated by non LU processes) we have forecasts of 5-year sales vs. actual figures for the historical major product lines. After deflation of the forecasts as discussed earlier, we find that the average sales in year 5 for LU major product lines (n=5), is $146m, while the similar figure for major IGLU - 03/09/01 25

29 product lines generated by non LU methods (n=16, as we were able to obtain detailed data on only 16 of the 21 non LU major product lines) is $62m. (Table 6). We address H 1A by testing if the mean LU major product line yields higher sales than those derived from non LU sources. At the p =.05 level we reject the hypothesis that these values are equivalent (Table 7), providing support for H 1A, conditional on assumptions A1 and A2. [Insert Tables 6 and 7 here] As was the case in our analyses of current projects, we investigate how major product line ideas generated by the lead user method differ from those generated by non LU methods in terms of dimensions of compatibility (H 3, redux). Table 7 provides profiles of the 5 LU major product lines and the 16 non LU major product lines for which we were able to collect data.. Ideas produced by the LU method tend to be more novel or breakthrough (although not statistically significantly so) than are the major product lines produced from conventional methods. As with our previous analysis (Table 4), we reject the hypothesis (H 3 ) of difference between LU and non LU major product lines for most dimensions of compatibility, at the p<0.10 level; the only difference statistically significant at that level is estimated 5-year sales. Process Activity Analysis. To investigate H 6, we contrast the characteristics, skills and capabilities of LU participants with those in non LU groups. The individual respondent measures we collected included a range of process activities as well as personality measures and selfperceived expertise in key activities (see Table 2c). Because we collected our data in a natural experimental setting, we could not assume that measured differences on the postprocess measures between LU and non LU team members were not due to differences in the respondent traits, rather than due to the LU training. As Table 8 shows, while the IGLU - 03/09/01 26