Four Other Studies Prove That People Use Police Objectives to Generate Alternative Data

Transcription

1 This article was downloaded by: [ ] On: 01 July 2016, At: 10:19 Publisher: Institute for Operations Research and the Management Sciences (INFORMS) INFORMS is located in Maryland, USA Operations Research Publication details, including instructions for authors and subscription information: Creating More and Better Alternatives for Decisions Using Objectives Johannes Siebert, Ralph L. Keeney To cite this article: Johannes Siebert, Ralph L. Keeney (2015) Creating More and Better Alternatives for Decisions Using Objectives. Operations Research 63(5): Full terms and conditions of use: This article may be used only for the purposes of research, teaching, and/or private study. Commercial use or systematic downloading (by robots or other automatic processes) is prohibited without explicit Publisher approval, unless otherwise noted. For more information, contact The Publisher does not warrant or guarantee the article s accuracy, completeness, merchantability, fitness for a particular purpose, or non-infringement. Descriptions of, or references to, products or publications, or inclusion of an advertisement in this article, neither constitutes nor implies a guarantee, endorsement, or support of claims made of that product, publication, or service. Copyright 2015, INFORMS Please scroll down for article it is on subsequent pages INFORMS is the largest professional society in the world for professionals in the fields of operations research, management science, and analytics. For more information on INFORMS, its publications, membership, or meetings visit

2 OPERATIONS RESEARCH Vol. 63, No. 5, September October 2015, pp ISSN X (print) ISSN (online) INFORMS Creating More and Better Alternatives for Decisions Using Objectives Johannes Siebert Faculty of Law, Business and Economics, University of Bayreuth, D Bayreuth, Germany, Ralph L. Keeney Fuqua School of Business, Duke University, Durham, North Carolina 27708, The quality of alternatives is crucial for making good decisions. This research, based on five empirical studies of important personally relevant decisions, examines the ability of decision makers to create alternatives for their important decisions and the effectiveness of different stimuli for improving this ability. For decisions for which the full set of potentially desirable alternatives is not readily apparent, our first study indicates that decision makers identify less than half of their alternatives and that the average quality of the overlooked alternatives is the same as those identified. Four other studies provide insight about how to use objectives to stimulate the alternative-creation process of decision makers and confirm with high significance that such use enhances both the number and quality of created alternatives. Using results of the studies, practical guidelines to create alternatives for important decisions are presented. Keywords: decision analysis: creating alternatives. Subject classifications: decision analysis: applications, theory. Area of review: Decision Analysis. History: Received February 2014; revisions received July 2014, November 2014, April 2015; accepted June Published online in Articles in Advance September 10, Introduction Creating quality alternatives has a crucial role in operations research. It is relevant to many important decisions: those without an obviously complete set of alternatives and others without a set of desirable alternatives. Yet research on the potential usefulness of creating alternatives and how to effectively create alternatives is sparse. Even for decisions where dedicated effort to create alternatives may be useful, decision makers tend to identify alternatives that readily come to mind and, as shown in this article, miss many alternatives that are better than any in the self-generated set. Most of the decision-making effort is spent evaluating the identified alternatives. For some decisions, relatively little effort spent on creating alternatives could result in better achieving the decision maker s objectives. This paper investigates how such effort might be well spent. It has been proposed that using the objectives of a decision to stimulate thoughts about potential alternatives should result in more and better alternatives (e.g., Keeney 1992). However, whether this proposition is correct has not been systematically tested with empirical studies and not at all with experiments using subjects who are decision makers facing personally relevant decisions. In this context, our studies address five fundamental questions: (a) are decision makers able to identify alternatives for their own decisions? (b) how good are alternatives that decision makers overlook? (c) can using objectives stimulate the creation of more and better alternatives? (d) how can using objectives most effectively stimulate the creation of alternatives? and (e) is it the additional time used to create alternatives or the use of objectives during that additional time that produces a higher quality set of alternatives? To our knowledge, these questions have only been cursorily studied in previous research, which is summarized in 2. Section 3 provides an overview of our research approach. Sections 4 through 8 present five studies, each involving actual decisions of substantial importance to the participants involved. Section 9 uses the empirical results of our studies to answer the five questions above. Section 10 combines the insights of our results into practical guidance for creating alternatives for individual and organizational decisions. 2. Relevant Research on Creating Alternatives Two lines of research on creating alternatives are relevant to our research. One is methodological research that examines foundational concepts for creating alternatives and develops procedures to implement those concepts. The other is empirical research that tests various procedures to generate alternatives in a variety of situations. 1144

3 Operations Research 63(5), pp , 2015 INFORMS Methodological Research on Creating Alternatives Keller and Ho (1988) characterize five approaches for the creation of alternatives. Three approaches use attributes, states of nature, or both and generate alternatives by focusing on different attributes, states, or combinations to stimulate the creation of alternatives. Improving existing alternatives also can lead to the creation of different alternatives. In addition to these structured techniques, Keller and Ho mention general creativity techniques such as brainstorming (see, for example, Ackoff 1978, Adams 1979). Another framework for creating alternatives uses valuefocused thinking (Keeney 1992). The foundation of this method is values, defined as what one hopes to achieve by making a given decision. These values, which can be stated as objectives, provide the basis for interest in a decision problem, so they may be useful in guiding effort to solve it. For creating alternatives, value-focused thinking uses objectives to stimulate thinking about new alternatives. The morphological box (Zwicky 1967) and the strategy table (Howard 1988) are similar techniques to identify a set of complex alternatives or comprehensive strategies (Tani and Parnell 2013). Zwicky decomposes a system into its functional subsystems, creates elements of an alternative that achieve the function of each subsystem, and creates an alternative by combining an element for each subsystem. Howard decomposes a strategy decision into a set of strategy areas and creates possible alternatives for each. A strategy alternative is a combination of one alternative from each area. These procedures systematically identify a comprehensive set of feasible alternatives, even though many of the combinations may be unappealing. The ideas in this paper, which would mainly create alternatives for functional subsystems and strategy areas, complement the use of the morphological box and the strategy table Empirical Research on Creating Alternatives Pitz et al. (1980) tested seven different conditions to stimulate the creation of alternatives for five personal decisions (e.g., a university student uncomfortable with her roommate smoking marijuana). Three conditions involved showing either one objective or two objectives at a time or the whole set of objectives at once; two conditions presented examples of four alternatives in a single list and categorized into two types of alternatives. There were two control conditions, one stating that objectives were important to consider and the other with no special instructions. Participants recruited from an introductory psychology course were allowed 12 minutes to create alternatives for each decision. Participants stimulated by one objective at a time created 6.47 alternatives, significantly more than participants in any of the other conditions. Collectively, participants generated between 18 and 26 distinctive realistic alternatives for each of the five decision situations. Jungermann et al. (1983) analyzed the effect of goal explicitness in creating alternatives. In an experiment on alternatives for a vacation, the detail of presented goal hierarchies was varied. More than 130 students, nurses, secretaries, or post office workers were stimulated by seeing only the top-level objective: I want to have a nice vacation; the top-level objective and the second-level objectives: to recover and to engage in some activity; or the complete hierarchal structure including the third-level objectives: recovering physically, relaxing mentally, doing some sports, pursuing hobbies, improving my education, and hiking and sightseeing. The number of alternatives generated increased with the degree of goal explicitness. Jungermann et al. concluded that greater goal explicitness enhances the creative search process of the individuals. Gettys et al. (1987) analyzed the ability of individuals to create alternatives to solve two decision problems. Because the results were similar, we summarize one experiment, where 60 introductory psychology students were instructed to take all the time needed to create as many alternatives as possible to solve the parking problem at their university. Their self-generated alternatives were compared to a pooled list of distinct alternatives for solving the parking problem developed by the experimenters. Subjects on average generated 11.2 alternatives, but the pooled list contained 128 alternatives. Butler and Scherer (1997) examined the effects of presented objectives on quantity and quality of created alternatives. They instructed 129 undergraduate and graduate students enrolled in psychology courses to create as many alternatives as they could for a sexual harassment problem and for an employee compensation problem. The experimenters created two broad objectives for each problem. Participants were stimulated by one objective at a time or by both objectives simultaneously or were not explicitly stimulated with objectives. The quality of the alternatives was evaluated by two experimenters who analyzed the extent to which an alternative resolved conflicting facets of the problem. For the sexual harassment problem, stimulation by one objective produced the most alternatives but did not enhance the quality of alternatives, whereas for the compensation problem this stimulation enhanced the quality but not the number of alternatives. Selart and Johansen (2011) conducted an experiment using 70 human resource employees from a resource management department of a large organization in Sweden. The employees were asked to suggest how the organization could save a minor amount of money (around 3,500 dollars) during the coming budget year. One group of participants was stimulated with a set of seven goals, whereas a second group was not. Participants stimulated with goals created better alternatives, as evaluated by the researchers. The participants not stimulated with goals created more alternatives. The authors explained this result by suggesting that the participants stimulated with goals had no experience with using goals to create alternatives.

4 1146 Operations Research 63(5), pp , 2015 INFORMS 3. Overview of Our Research Studies Previous empirical research on creating alternatives mainly consists of individual experiments with a descriptive focus on whether different procedures to generate alternatives affect the number or quality of alternatives generated. Participants were asked to create alternatives for realistic decisions, but not decisions that they were personally facing. Our five studies have a prescriptive focus on what individual decision makers could do to create more and better alternatives. In addition, the participants in the first four studies were personally facing the decision that had potentially significant consequences. Our fifth study concerns a decision that affects the participants daily lives. Prescriptive experimental studies require one feature that is inappropriate in descriptive studies of decision making. In descriptive studies, subjects should not be influenced by the experimenter, in order to observe what decisions they make on their own. In prescriptive studies, to learn what guidance to give decision makers to help them make better choices, our experiments must include interventions that guide subjects. Comparing what subjects do when influenced by different interventions, operations researchers and decision analysts can evaluate the usefulness of such influences. A succinct summary of the results of our five studies, presented in the next five sections, is the following. The first three studies indicate that for decisions where creating alternatives is worthwhile, decision makers asked to list their possible alternatives identify about one third of their relevant alternatives and those that they miss are of the same quality as those that they identify. Studies II and III demonstrate that using the objectives of the decision maker to stimulate the creation of alternatives increases the number created and using those objectives one at a time roughly doubles the initial number of alternatives. Our last two studies focus on the overall desirability of an entire set of created alternatives. Use of objectives to stimulate the creation Table 1. of alternatives increases both the quality and the quantity of a set of created alternatives, and this influence is much greater than using the same amount of time to create alternatives without the stimulus of objectives. 4. Study I: Are Decision Makers Aware of Their Alternatives? This study examines whether individuals can generate their better alternatives for a significant personal decision Methods Decision Problem and Participants. Most German university students in business studies face the decision of how to benefit as much as possible from an internship at least once during their studies. An internship provides practical experience so the student learns about both a particular job and the reality of working full time after graduation. These internships typically happen in businesses, nonprofits, or government organizations. Companies use internships as a recruiting tool because they get a thorough impression of the interns. The participants were 201 full-time bachelor s or master s degree students enrolled in business related courses such as Business Administration, Media and Business, and Industrial Engineering with Business Studies in a German university. Overall, 188 participants completed the study in an average of approximately 25 minutes. Responses of 13 participants, who spent less than eight minutes on the study, are not included in the results. The 100 male and 88 female students spread evenly across the courses of study and degrees. Incentives for the participants were insight about an important personal decision and a three Euro voucher for a coffee Material and Procedure. Before conducting this study, experimenters created the master list of alternatives given in Table 1. Each alternative in this table could Master list of alternatives for enhancing benefits from an internship (translated from German). A Use skills developed in college Q Pursue options to enhance skills (workshops, training) B Take responsibility for interesting and responsible tasks R Ask colleagues and superiors for feedback C Ask superior for a project or a task S Search out relevant information for your professional future D Suggest own field of activity T Gain experience in multiple corporate divisions E Request and readily accept targeted support U Switch to a better internship F Demonstrate a quality work ethic and flexibility V Critical appraise your own performance G Initiate and deepen contact to superiors W Ask questions to clarify your understanding H Participate with colleagues in leisure activities X Participate in meetings, negotiations, conferences I Contribute positively to team projects Y Work effectively and efficiently J Contribute to a good working atmosphere Z Improve existing language skills K Make suggestions for improvement AA Be willing to work overtime L Assume management responsibility in a project BB Discover culture, region, and city M Provide constructive feedback on occasion CC Request an internship certificate N Search for and tackle challenges DD Gain experience in a particular division O Make yourself aware of job openings EE Gain experience in everyday working life P Take advantage of opportunities to demonstrate your abilities

5 Operations Research 63(5), pp , 2015 INFORMS 1147 be thought of as a narrowly defined category of alternatives. The intent of the master list was to include all of the alternatives that might be relevant to participants in this study. Two sources were used to create the master list. First, 20 students, who were not study participants, each listed their alternatives for an internship. Second, we adapted the objectives for choosing an MBA internship in Bond et al. (2008) to the present decision situation and created at least one alternative to achieve each objective. The lists of alternatives were aggregated, redundancies were eliminated, and similar alternatives combined into a more broadly defined alternative. The resulting draft master list of alternatives was tested with 20 students. Participant feedback was used to compile the final master list of 31 alternatives in Table 1. The study utilized a questionnaire administered electronically in German with Qualtrics that measures the time spent by participants on single tasks. To avoid external influences, the study took place in reserved computer rooms supervised by trained operators. The participants were instructed not to communicate with others during or after the study. The study consisted of four steps similar to those used in Bond et al. (2008), who investigated whether individuals are aware of their objectives in important decision situations. In step 1, participants were asked to assume that they had just accepted an offer for an internship. Each participant was then instructed to list all alternatives that he or she could think of to benefit from this internship. The screen provided 20 blank lines for responses. Participants were given the master list of alternatives in step 2 and instructed to check the box next to each alternative that they considered relevant for their personal decision. In step 3, participants were asked to match each of their personally generated alternatives to a single equivalent alternative on the master list if possible. For each match, participants wrote the letter of that equivalent alternative next to the corresponding alternative that was personally generated. In step 4, the participants evaluated the quality of each personally generated or checked alternative. Four separate indicators provided a comprehensive evaluation. The suitability of an alternative to achieve one s objectives was evaluated using a 9-point Likert scale (extremely bad = 1 to extremely good = 9). The creativeness of an alternative and the effort to implement an alternative were evaluated using 9-point Likert scales (extremely low = 1 to extremely high = 9). A ranking of the alternatives indicated the relative likelihood that participants would use them in their next internship. To reduce potential biases in the evaluation, participants were presented a randomized list of all of their alternatives without indicating whether an alternative was generated in the first step or only recognized in the second step. The ranking was elicited using a drag and drop procedure that allowed participants to drag each alternative from the left to a ranking on the right and easily modify the ranking. Table 2. Average self-generated and recognized alternatives. Alternatives listed in step Redundant alternatives 0 78 Self-generated alternatives 6 66 Alternatives with no match on 1.26 the master list Alternatives with a match on 5.40 the master list Alternatives matched to checked 4.14 alternatives in step 2 Alternatives matched to alternatives 1.26 not checked in step 2 Alternatives checked in step Alternatives matched to checked 4 14 alternatives in step 2 Recognized alternatives Total relevant alternatives, self-generated and recognized 4.2. Results Was the Self-Generation of Alternatives Complete? The results are summarized in Table 2. In step 1, participants listed 7.44 alternatives on average; the maximum number was 20. In step 2, alternatives on average were checked on the master list as relevant. Participants assigned an average of 0.78 self-generated alternatives to the same alternative on the master list. These alternatives were deemed redundant, so participants self-generated on average 6.66 distinct alternatives. Of these, in step 3, 1.26 were not matched with alternatives on the master list. The remaining 5.40 were matched to alternatives on the master list, 4.14 to alternatives checked in step 2, and 1.26 to alternatives that had not been checked in step 2. The latter could occur because participants in step 3 viewed the whole master list, not only the alternatives they had checked in step 2. Subtracting the 4.14 alternatives matched to listed alternatives from the listed alternatives indicates that on average alternatives were only recognized as being relevant to the decision maker from the master list and not self-generated in step 1. Results clearly indicate that participants failed to generate a comprehensive list of alternatives on their own. Compared to 0, the mean number of additionally recognized alternatives is highly significant (T = 25 03, p = 0). The abundance of alternatives that were only recognized by using the master list indicates the potential usefulness of procedures to stimulate the creation of alternatives. To better interpret these results, it is useful to examine the completeness of the master list of alternatives. Collectively, participants self-generated 1,251 distinct alternatives. They did not match about one fifth of these to an alternative on the master list. We analyzed these 236 alternatives by grouping them into three classes. Class 1 consisted of 78 alternatives that were not meaningful for the given decision situation, including alternatives like meet friends or

6 1148 Operations Research 63(5), pp , 2015 INFORMS earn additional money. Class 2 consisted of 59 alternatives that clearly matched an alternative on the master listbut that match was not noted by the participants. Class 3 contains 99 meaningful alternatives that were not included on the master list. Two alternatives each mentioned about a dozen times were talk with other interns about the internship or balance work and life (to remain being efficient). This classification is strongly supported by the average suitability of alternatives. The average suitability of the alternatives classified as not meaningful was 4.21, which is significantly lower than the average suitability of the alternatives classified as meaningful in the second (6.69) and the third classes (6.33) (T = 5 61, p 0; T = 5 59, p 0). As participants matched 1,015 self-generated alternatives to the master list, and missed only 59, this 94.5% (=1 015/1 074) accuracy rate indicates a high level of thoroughness and attentiveness. Also, the master list included 1,074 alternatives identified as relevant to participants and did not include 99. Thus, the master list was reasonably complete since it included 91.6% (=1 074/1 173) of meaningful self-generated alternatives What was the Quality of the Alternatives? The average suitability of alternatives was 7.00 for selfgenerated alternatives and 7.01 for recognized alternatives. Correspondingly, the effort to implement alternatives was on average 4.89 versus 4.97, the creativeness of alternatives was 4.94 versus The mean value analyses using a two-sided t-test reveals that none of these three quality measures for the self-generated alternatives differs significantly from that for the recognized alternatives. Of the alternatives ranked 1 to 5 by each participant, 3.14 on average were recognized alternatives. Broadening the set to those ranked 1 to 10, an average of 6.5 was recognized. Therefore, a majority of the higher-ranked alternatives were not self-generated and only identified by using the master list. Figure 1 presents the proportion of participants who either (1) generated all or (2) failed to generate even one of their top-ranked alternatives. Note that, 56% of participants did not generate their highest-ranked alternative Is it Worthwhile to Spend Time on Generating Alternatives? On average, the participants spent about Figure 1. Proportion of participants (%) Participants self-generating top-ranked alternatives in study I Set of ranked alternatives No self-generated alternatives (only recognized alternatives) All self-generated alternatives four minutes generating alternatives in the first step. The time spent on generating alternatives and the number generated is highly correlated (r = 0 56, n = 188, T = 9 18, p 0), so the more time participants spent on generating alternatives, the more alternatives they generated. Of the 44% who did self-generate their highest-ranked alternative, only 60% generated this alternative within their first five alternatives mentioned. Analysis of the self-generated alternatives reveals that the more suitable alternatives tend to be generated earlier. However, the average suitability of alternatives decreases only slightly as the order in which the alternative was generated increases. For example, on the 9-point Likert scale, the average suitability of the first, fifth, and ninth selfgenerated alternatives are 7.28, 6.98, and 6.69, respectively. We computed correlations between the time spent and different quality measures of the self-generated alternatives. None of the correlations were significant with correlation coefficients in the range of ±0 1. For example, the correlation between time spent in the first step and the suitability of the best ranked self-generated alternative was The correlation between time spent in the second step and the number of alternatives selected on the master list was Discussion Study I provides insights regarding the capability of individuals to generate alternatives for a personally significant decision. Even though participants were able to produce a reasonable number of alternatives without external help, they were not aware of a majority of their relevant alternatives. More than half of the participants failed to selfgenerate the alternative they later identified as the most likely to implement. One may conjecture that participants devoted inadequate effort in the first step and therefore did not generate many later recognized alternatives. However, two arguments appear to contradict this account. First, if inadequate effort during the generation task was driving the difference in self-generated and recognized alternatives, participants who listed the fewest alternatives at step 1 should have recognized the most new alternatives in step 2, implying a negative correlation between the quantity of selfgenerated and recognized alternatives. Analysis revealed this correlation to be slightly, but not significantly, negative (r = 0 07, n = 188, T = 0 96, p = 0 17). Second, the ranking data reveal that almost all participants generated some alternatives that they did not consider very important to the decision (i.e., alternatives not included in their top 10). Hence, most participants took the generation task seriously enough to list both highly important and less important alternatives. This suggests that much of the failure to generate many of the better alternatives was not caused by inadequate effort during the time spent in the generation process but rather other factors attributable to the complexity of the task such as an incomplete cognitive representation of the decision.

7 Operations Research 63(5), pp , 2015 INFORMS 1149 Study I strongly suggests that it is worthwhile to dedicate effort to generate alternatives for three reasons. First, many of the highest ranked alternatives were not found in the first phase of self-generating alternatives. Second, the results indicate that the time spent to generate alternatives and the number of alternatives generated is highly correlated. Third, the data show that, on average, there are no significant differences regarding the quality of the selfgenerated and recognized alternatives. Since many more alternatives were recognized after seeing the master list of objectives than were self-generated, there are many good alternatives to be created. 5. Study II: Do Objectives Help to Create More and Better Alternatives? Study II is our first test of whether and how using objectives stimulates the creation of more and better alternatives Methods Decision Problem and Participants. All German university students face an important decision situation about how to maximize the benefits of their higher education. The decisions available in this situation concern (1) how to focus one s classroom education, including what courses to take and the effort to exert in those courses, and (2) what extracurricular activities to pursue. This decision situation is the basis for our second study. The participants were 313 full-time bachelor s or master s degree students mainly enrolled in business related Figure 2. Means-ends objectives network of educational and extracurricular objectives of German university students for their higher education. courses like Business Administration, Media and Business, and Industrial Engineering with Business Studies at a German university. Responses of 24 participants who did not complete the study are not included in the results. Of the 289 participants who completed the study, 146 registered for a bachelor s and 143 for a master s degree. There were 143 male and 146 female participants. The setting of Study II was the same as in Study I, conducted in German, and lasting on average approximately 20 minutes Material and Procedure. Prior to the study, a master list of objectives and a means-ends objectives network for this decision situation were created by the experimenters for use in the study. First, 190 objectives were elicited from current students for this phase of education. One experimenter and four advanced graduate students eliminated redundant objectives and, after some iteration, produced a final master list of 27 distinct objectives. Next, each of the five created a means-ends objectives network for these 27 objectives. The group reviewed these and created a means-ends objectives network that included the central elements of each member s network. In-depth interviews with pre-test students about the meaningfulness of the resulting means-ends objectives network led to small adaptations. Figure 2 presents an English translation of the final means-ends objectives network. The first column presents the four fundamental objectives. The second column contains means objectives that directly influence the achievement of the fundamental objectives. The third column contains means objectives that mainly contribute to the objectives in the second column.

8 1150 Operations Research 63(5), pp , 2015 INFORMS Study II consisted of three steps. In the first step, the participants were divided randomly into two groups. Participants in group 1 were first asked to write down their objectives to enhance the benefit of their higher education. Then they were asked to list as many alternatives as they could that would help to achieve their objectives. Participants in group 2 were asked to list all sources of information they consider when making decisions. Responses included talking to friends or family members, searching the internet, reading the newspaper, etc. This task was used to engage the participants of group 2 roughly as long as the participants of group 1 spent listing their objectives. Next, those in group 2 were asked to list as many alternatives as they could that would enhance the benefits of their higher education. In the second step, participants were randomly assigned to five groups. All groups were informed that recent studies confirm that participants usually list less than half of their relevant alternatives. All but the fifth group were given different information from the objectives in Table 2 and asked to use it to generate additional alternatives. Group A was given a list of the four fundamental objectives: lay foundation for future success, develop personality, gain knowledge, and enjoy life. Group B was given a list of four means objectives: meet the demand of labor market, be committed to social activities, gain professional knowledge, and participate actively in student life. These means objectives collectively contribute to all four fundamental objectives, and thus provide the same range of objectives for groups A and B. Group C was provided the master list of all 27 objectives and group D was provided the meansends objectives network. Group E was asked to generate more alternatives without being explicitly stimulated with objectives. In the third step, participants evaluated their alternatives with three of the measures previously described in Study I, the suitability to achieve one s objectives, creativeness, and the effort to implement it Results Are Objectives an Effective Stimulus to Create More Alternatives? Table 3 presents the results of Study II. In the first step, the mean number of alternatives generated by the participants of group 1 was 5.79 with a standard deviation of The corresponding data for group 2 are 6.71 alternatives with a standard deviation of The difference between groups is significant (T = 2 11, p = 0 018). An analysis of the data from the second step indicates the following: The participants stimulated with objectives (groups A, B, C, D) created significantly more alternatives than did the participants of the control group E (T = 3 07, p = ). The participants stimulated by the master list of objectives (groups C and D) created significantly more alternatives than did participants in groups A and B, Table 3. Average number of alternatives created and their quality in Study II. Average Average Average Average Average Average Step 1 Stimulus Participants alternatives Participants alternatives Participants alternatives suitability effort creativeness Group 1 Objectives Group 2 Sources of information All Individuals in Individuals in Step 2 Group 1 Group 2 All All All All Group A Fundamental objectives Group B Means objectives Group C Master list of objectives Group D Means-ends objectives network Group E More time, no objectives All

9 Operations Research 63(5), pp , 2015 INFORMS 1151 stimulated by only four objectives (T = 2 29, p = 0 012). There is no statistically significant difference between the number of created alternatives of the participants stimulated by fundamental objectives (group A) and by means objectives (group B) (T = 1 33, p = 0 093). There is no statistically significant difference between the number of created alternatives of the participants stimulated by the master list of objectives (group C) and by the means-ends objectives network (group D) (T = 0 95, p = 0 17). Most alternatives generated in steps 1 and 2, were distinct, such as Carrying out an internship, Studying abroad, or Working as a student assistant. However, participants sometimes listed conceptually redundant alternatives such as Carrying out an internship for car manufacturer A and Carrying out an internship for car manufacturer B or Studying in the United States and Studying in Canada, i.e., Studying abroad. From the set of all alternatives identified by participants, experimenters created a master list of 50 distinct alternatives and treated redundant alternatives as a single alternative generated by a participant. On average, in the first step, participants of group 1 generated 0.90 redundant alternatives compared to 1.34 redundant alternatives generated by group 2, a difference that is statistically significant (T = 1 98, p = 0 024). The intervention in the first step had no impact on the average number of alternatives created in the second step, 6.50 for participants who were in group 1 and 6.52 for those in group 2. Also, participants in group E were of average productivity in step 1, creating 6.13 alternatives on average compared to the overall average in group 1 of This suggests that group E participants put the same level of thought and effort into the experiment as others Does the Use of Objectives Lead to Better Alternatives? In the first step, participants of group 1 evaluated the suitability of their alternatives as 7.18 on average, whereas participants of group 2 evaluated the average suitability as 6.95, a statistically significant difference (T = 2 47, p = ). On average, the creativeness of the alternatives generated by participants stimulated by objectives was 5.52 and is significantly higher than the creativeness of the alternatives generated by participants not stimulated with objectives that was 5.33 (T = 1 69 p = 0 046). The difference in the effort required to implement an alternative between groups 1 and 2 was not significant. In the second step, the participants stimulated with objectives (groups A, B, C, D) evaluated the suitability of their alternatives significantly higher than the participants of the control group E (T = 3 28, p = ) and the effort to implement an alternative as significantly higher than the control group (T = 3 16, p = ). The difference in creativeness between those participants who were stimulated with objectives and the control group is not significant. Furthermore, the average effort to implement was 4.13 in the first step and is significantly lower than the effort to implement alternatives generated by participants stimulated with experimenter-created objectives that was 4.83 (T = 8 71, p 0) Discussion Participants who listed their own objectives in the first step produced 0.92 fewer alternatives on average than those participants who listed sources of information considered when making a decision. Adjusting for redundant alternatives produced by the two groups, the difference in distinct alternatives generated reduced to 0.48 on average. In the second step, when the master list of 27 experimentercreated objectives for the decision was given to participants in groups C and D, they generated 2.47 more alternatives on average than group E did without objectives. When just four of the 27 objectives were provided to groups A and B, they generated 1.20 more alternatives on average than group E did. There are different possible explanations for why selfgenerated objectives were less effective in generating alternatives in the first step of the study. First, the participants in group 1 were not told to use objectives to create alternatives. For creating alternatives, being aware of one s objectives and being told to use them may be different. Second, individuals initial lists of their objectives for important decisions are not very complete. The usefulness of an incomplete set of objectives to stimulate the creation of alternatives would tend to be less than a better set of personally relevant objectives. Some support for this notion is provided by the results in step 2. In all four groups stimulated by objectives, the average number of additional alternatives created using these objectives was the same as or more than the average number of alternatives created initially by group 1. Third, when participants used objectives to create alternatives, they may have looked for alternatives that satisfy all of the objectives, which could lead to fewer alternatives. The higher quality of the alternatives supports this explanation. Fourth, in the first step participants generate the alternatives that are easy to identify and implement regardless of the stimulation. This produces the so-called low hanging fruit, so the impact of a stimulus, such as using objectives, may not be great. In the second step, creating alternatives becomes more difficult and the objectives are useful for identifying the higher hanging fruit which require more effort to implement. Study II confirms two important results of Study I. Spending the additional time in step 2 to generate additional alternatives produces many more alternatives and many of these are of high quality. Also, using objectives to guide the creation of alternatives developed more and higher-quality alternatives than using no stimulation. It is worth noting that constructing a means-ends objectives network, our most complete stimulus, requires

10 1152 Operations Research 63(5), pp , 2015 INFORMS considerably more analyst skill and effort than just providing a master list of objectives. Relative to using the master list of objectives, the use of a means-ends objectives network to stimulate additional alternatives led only to a 12% (=7 93/7 08) increase in alternatives created. 6. Study III: How Should Objectives Be Used to Create More Alternatives? The previous study demonstrates that individual decision makers can use objectives to stimulate the creation of additional high-quality alternatives for important personally relevant decisions. Study III examines and compares two different ways to use the objectives for this purpose Methods Decision Problem and Participants. Almost all MBA students in the United States have an internship during the summer between their two years of courses. Choosing an internship is an important decision because the intern can explore a new area of business, receive or enhance the chance of a full-time job offer upon graduation, and expand contacts and professional relationships. Study III concerns decisions to enhance the benefits of an MBA internship. Participants were 295 students in the first year of an MBA program at a large eastern university in the United States. It was part of a course assignment questionnaire using Qualtrics that also included several questions on other matters Material and Procedure. This study had three steps embedded in the questionnaire. In the first step, participants were asked to list as many alternatives as they could pursue before or during their internship that would enhance their benefits from the internship. In the second step, participants were asked to list all objectives of their internship and then to check all objectives on a master list generated by the experimenters that they considered relevant in choosing an internship. For the third step, conducted after several additional questions not relevant to this study, participants were divided into two groups. Participants were provided with a list of the objectives that they had listed and/or checked and asked to use them to generate any additional alternatives that they could pursue before or during their internship to get more benefit from the experience. The participants in group 1 were told to use each of their objectives separately to create additional alternatives. Participants in group 2 were told to use their objectives to create additional alternatives without the additional guidance to use the objectives separately. Group 1 had 130 participants and group 2 had Results How should objectives be used to create more alternatives? Prior to specifying any objectives, the average number of alternatives created in the first step was Participants who subsequently were in groups 1 or 2 generated an average of 5.48 and 5.27 alternatives, respectively, in step 1 suggesting that there was no difference between groups (T = 0 68, p = 0 50). In the third step, participants of group 1 generated on average 5.69 additional alternatives compared to 3.73 additional alternatives by participants of group 2. The difference is highly significant (T = 6 04, p 0). Relative to simply guiding individuals to use their list of objectives to create alternatives, the additional guidance to use the objectives one at a time led to a 53% (=5 69/3 73) increase in alternatives created Discussion Study III indicates that individuals using objectives to create alternatives generate more alternatives if they sequentially use each of their objectives. This finding for personally relevant decisions is consistent with results of Pitz et al. (1980), Butler and Scherer (1997) and Selart and Johansen (2011) for subjects responding to realistic hypothetical decisions. It may also lend some insight about our result in Study II, where participants stimulated by their self-developed objectives created one less alternative on average than did participants stimulated by self-developed sources of information for their decision. This result can be explained by the competing implications of the stimulating and constraining effects of a set of objectives in creation of alternatives. On the one hand, a set of objectives inspires an individual to think how these objectives could be achieved, which supports the creation of alternatives. On the other hand, when a potential alternative does not help achieve some of the stated objectives, an individual may feel that it is not an appropriate alternative or not even think of it. Thus, when an individual considers multiple objectives simultaneously, fewer alternatives may be generated. 7. Study IV: Do Objectives Help to Create a Better Set of Alternatives? The overall quality of a set of alternatives is a useful indicator that combines aspects of quantity as well as quality of alternatives. In Study IV, we analyze whether using objectives improves the quality of the set of alternatives generated Methods Decision Problem and Participants. This study took place during a two-week international summer program on multicriteria decision making. Almost all of the 45 participants were Ph.D. students or post doctoral researchers coming from 25 different countries. There were 21 female and 24 male participants with an average age of

11 Operations Research 63(5), pp , 2015 INFORMS years. Most of the program participants were, or soon would be, highly involved in searching for an attractive faculty position at a university. A specific decision they each faced was to identify alternatives that they could pursue to enhance the likelihood of obtaining such a position Material and Procedure. Study IV was conducted in English, performed with paper and pencil, and consisted of two parts. The first part was embedded in lectures on problem structuring and value-focused thinking and lasted approximately 40 minutes. The second part occurred two days later prior to a lecture and lasted approximately 10 minutes. In the first part, participants were randomly divided into two groups, 22 participants in group 1 and 23 in group 2. Group 1 was stimulated by a master list of objectives, which had been created by the experimenters using a process similar to the one described in Study II. The participants were not guided about how to use objectives; they were simply asked to list all alternatives that could help achieve their objectives. In contrast, group 2 created alternatives for the decision situation without being explicitly stimulated. In the second part of this study, conducted two days later, participants holistically evaluated the relative quality of pairs of sets of alternatives generated in the first part. In each comparison, one set of alternatives was created by an individual in group 1 and the other by an individual in group 2, so there were (23 22 = 506) possible pairs of sets of alternatives. Since we wanted participants to carefully evaluate the relative quality of these pairs, each participant was only given approximately nine randomly generated pairs from the potential set. Participants could not relate the sets of alternatives to the experimental condition in which they were created and no pair was compared more than once Results Participants of group 1, which had been stimulated with objectives, created between 5 and 19 alternatives with an average of alternatives and a standard deviation Participants of group 2 created between 3 and 14 alternatives with an average of 9.44 and a standard deviation The difference between the numbers of alternatives generated by the groups is not significant. This lack of difference was desired, which was why the participants of group 1 received no guidance about how to use the objectives to create alternatives. Hence, the subsequent comparison of the quality of the sets of alternatives generated by the two processes would be more insightful. Are the sets of alternatives created with objectives better? The participants collectively evaluated 415 distinct pairs of sets of alternatives. In 17 cases, preferences were not clearly indicated. In the remaining 398 pairwise comparisons, the set of alternatives created with objectives was preferred 230 times and the set of alternatives created without objectives 168 times. If the sets of alternatives were equivalent in quality, the probability for choosing either set would be 0.5. Hence, using a binomial distribution with n 1 = 230 and n 2 = 168, one obtains that the sets of alternatives created with objectives are better with a significance level of Discussion Study IV confirms that using objectives enhances the overall quality of a set of alternatives. This finding might mainly be due to the fact that the quality of a set of alternatives depends on the number of alternatives since there is a strong correlation between the number of alternatives in a set of alternatives and the percentage of time in which that set of alternatives was preferred in a pairwise comparison (r = 0 63, n = 45, T = 5 32, p 0). Could that additional one alternative created by the group members stimulated by objectives cause the difference in the evaluation of the sets of alternatives? To examine this, the experimenters made the average number of alternatives of both groups approximately identical by deleting the data from the two participants in group 1 who generated 18 and 19 alternatives, by far the most alternatives. These sets of alternatives were preferred in 29 of 38 comparisons. Deleting this data and using the binomial distribution with n 1 = 201, n 2 = 159, the sets of alternatives created with objectives are still significantly better with a significance level of Participants in this experiment each possessed a great amount of knowledge about theories and procedures of decision making relative to participants in our other studies. Even with such specialized and detailed knowledge, these participants created more and better alternatives by using objectives. 8. Study V: Does a Systematic Use of Objectives Improve a Set of Alternatives? Results of Studies I and II indicate that the time spent generating alternatives and the number of generated alternatives are highly correlated. Study IV found a high correlation between the number of alternatives in a set of alternatives and the quality of the set. Study V is designed to gain more insight about these findings. We investigated the separate effects of using objectives to create alternatives and using more time by controlling the time spent on the creation task. To compare whether using objectives leads to higher quality sets of alternatives, distinct from the quantity of alternatives in the sets, we evaluate only the set of the best five alternatives created with different procedures.

12 1154 Operations Research 63(5), pp , 2015 INFORMS 8.1. Methods Decision Problem and Participants. The parking situation at a university in Germany affects most of the students daily. How to improve this situation is the decision problem in Study V. We adapted the parking problem from Gettys et al. (1987) with appropriate adjustments for the number of parking lots, faculty members, and students. The study was divided into two tasks: creating sets of alternatives and evaluating the sets. All participants in the study were students at this university in Germany. In the creation of alternatives, 66 full-time master s degree students, 32 male and 34 female, participated during a course on operations research. Printed copies of a questionnaire were filled out in three eight-minute segments with one minute between segments. All participants completed the questionnaire. In the evaluation of alternatives, 122 full-time students, 66 males and 56 females, participated, including about 44 of the participants involved in the creation task. Overall, 101 participants completed a questionnaire administered electronically with Qualtrics and took on average approximately 18 minutes. Both tasks were conducted in German Material and Procedure. Prior to conducting this study, experimenters identified sets of three means objectives and four fundamental objectives for the parking decision. The fundamental objectives concerned cost efficiency, sustainability, environmental friendliness, and stakeholder satisfaction. The means objectives were use available parking lots more efficiently, increase available area for parking, and reduce demand for parking. Creation of alternatives. The participants were divided randomly into three groups. Two groups were stimulated differently with objectives and the third was the control group. All participants read the case description of the parking problem at the university and had the chance to ask the experimenter questions. The experiment was divided into three phases. The time for each phase was set at eight minutes, as most participants in previous studies stopped the creation of alternatives within eight minutes. During each phase, the remaining time was displayed on the computer screen. In the first phase, the participants had to create alternatives and select the best five alternatives. In the second and third phases, the participants were instructed to create new alternatives and/or improve current alternatives and then select the best five alternatives. The participants could transfer their alternatives from earlier phases with arrows to save time. The questionnaire was presented on a double-sided landscape sheet so the alternatives in the three phases could be arranged side by side. The sheet was folded such that participants could not see the instructions of later phases. Group A, the control group, was not stimulated with objectives in any phase. Group B was stimulated with all seven means and fundamental objectives in all phases. In the first phase, participants were instructed to create alternatives that achieve each objective separately. In the second phase, participants were instructed to consider several specific pairs of two objectives and create new alternatives or modify existing alternatives from the first phase. In the third phase, participants were told to consider all of the objectives simultaneously and create new or modify existing alternatives. Group C was also stimulated with objectives. In the first phase, the participants were presented the three means objectives and instructed to consider one objective at a time. In the second phase, participants were stimulated by the fundamental and means objectives and instructed to create or modify alternatives using pairs of objectives. In the third phase, the participants were instructed to consider as many objectives as possible to create or modify alternatives. Evaluation of alternatives. Each phase of the creation task produced a set of the five alternatives selected as best by their creator. Participants in the evaluation task were asked to rank 12 combinations of three sets, two for each of six situations. Three situations involved intrapersonal rankings comparing sets of alternatives of randomly selected participants in the three phases, two rankings for each of groups A, B, and C. The other three situations involved interpersonal rankings comparing randomly selected sets of alternatives from the same phase, one chosen from each group, with two rankings for each phase. As about one third (i.e., 44/122) of the evaluators also created alternatives in this experiment, and each evaluator had a 1/11 chance of evaluating his or her own sets of alternatives over the phases; approximately 3% of the evaluations were of self-generated sets of alternatives. This situation would not meaningfully distort the results below Results There were no significant differences in the results of groups B and C, which were each stimulated with objectives. Therefore, in the following we combine groups B and C into a joint group BC to compare the results of using objectives to the control group A. Does using objectives produce a better set of alternatives? In the creation task, almost all participants (63 out of 66) made changes in their attempt to improve their set of best five alternatives in the second phase. The participants stimulated with objectives (group BC) made on average 2.80 changes, which are significantly more than the 2.23 changes of the control group (p 0 047). In going from the second to the third phase, only 11 out of 22 participants of group A made changes to their set of best five alternatives in contrast to 44 out of 44 participants in group BC. Those in group BC made on average 2.61 changes, significantly more than the 1.14 changes of the control group (p 0).

13 Operations Research 63(5), pp , 2015 INFORMS 1155 Does using objectives improve alternatives beyond simply using more time to create alternatives? The rankings provide pairwise comparisons, so the binomial distribution was used to test for significant differences. In the first phase, the best five alternatives created by participants of group A and group BC were preferred 185 times and 219 times, respectively. There is no significant difference regarding the quality of the best five alternatives at this phase. However, in the second and third phases, the alternatives created by group BC were preferred 233 times versus 171 and 244 versus 160 to those of the control group A. These differences are significant at a level. Does additional time to create alternatives improve the set of alternatives? The only resource that group A participants had to improve alternatives in phases 2 and 3 was more time. These participants did not improve their sets of best five alternatives with the additional time in phases 2 and 3. In 202 comparisons of phases 1 and 2, the set of alternatives from the first phase was preferred 109 times and the set from the second phase 99 times. Comparing the sets of alternatives generated in second and third phases, each set was preferred 48 times over the other. The lower absolute number of pairwise comparisons is based on the fact that only 11 of 22 participants in group A made any changes in the third phase. In contrast, group BC improved the quality of their best five alternatives significantly over time. The sets of alternatives of the second phase were preferred 231 to 173 times to those of the first phase and the alternatives of the third phase were preferred 249 to 149 times to those of the second phase. These differences are significant on a level Discussion The systematic use of objectives, specifically using the objectives first one at a time, then in pairs, and then the entire set, is an effective way to stimulate the creation of alternatives. Studies II and III indicate this is better than providing a list of objectives with no guidance for how to use them. This study indicates that it is much better than not providing objectives. For the control group, the quality of the best five sets of alternatives did not improve between the first and the third phase. This strongly suggests that without a stimulus, spending more time on creating/improving alternatives is relatively ineffective to improve the quality of an initial set of alternatives. In contrast, the participants stimulated with and instructed how to use objectives significantly increase the quality of their sets of best five alternatives in both the second and third phase. 9. Discussion and Summary Our studies provide useful responses to the five questions posed in 1. Are decision makers able to identify the alternatives for their own decisions? When asked to identify alternatives, decision makers identify less than half of the alternatives useful to consider. In Study I, participants identified on average 37% of the set of alternatives that each individual later identified as relevant from a master list of alternatives. In Studies II and III, participants first listed alternatives for their respective decisions given little or no stimulation to create alternatives. Subsequently, using self-generated and/or experimenter-provided objectives in various ways to stimulate the creation of alternatives, participants more than doubled the number of relevant alternatives. How good are the alternatives that decision makers overlook? On average, the sets of identified and recognized alternatives were essentially equivalent in Study I in terms of three indicators of quality: the suitability to which the alternatives would achieve the decision maker s objectives, the effort necessary to implement the alternatives, and the creativeness of the alternatives. Only 44% of the participants identified their highest-ranked alternative, 10% of participants identified their three highest-ranked alternatives, and merely 1% identified all of their top five alternatives. Can using objectives stimulate the creation of more and better alternatives? The results of Studies II through V indicate that using objectives creates more and better alternatives. In Studies II and III, participants using objectives slightly more than doubled the number of alternatives relative to those initially generated without objectives. Study IV had two groups of participants create sets of alternatives with and without a set of objectives as a stimulus. Comparing pairs of sets of alternatives created with and without objectives, participants evaluated the set created with objectives as better in 230 of 398 cases. In Study V, there were three sequential eight-minute phases to create a set of the best five alternatives. Some participants used objectives as a stimulus in each phase and others were given only time without any stimulus. In independent pairwise comparisons, the set of alternatives created using objectives was preferred 54% of the time in the first phase, 58% of the time in the second phase, and 60% of the time in the third phase. How can using objectives most effectively stimulate the creation of alternatives? Study II compared four different representations of objectives to create alternatives. Using more objectives produces more alternatives. Two groups received all of the objectives, one as a master list and the other with the list structured as a means-ends objectives network. The means-ends stimulus, which requires much more analyst skill and time to produce than the master list, resulted in the creation of 12% more alternatives than just using the master list. However, Study III indicates that using a master list with the additional guidance to use the objectives one at a time to create alternatives results in a 53% increase in alternatives, and this stimulus requires essentially no analyst skill or time to implement. Study V

14 1156 Operations Research 63(5), pp , 2015 INFORMS indicates that using objectives as a stimulus to create alternatives first one at a time, then two at a time, and finally using larger sets can lead to improvements in each step. Is it the additional time used to create alternatives or the use of objectives during that additional time that produces a higher quality set of alternatives? Using objectives to stimulate the creation of alternatives is more effective than just using an equivalent amount of time without guidance. In Study V, there were three sequential eightminute phases to create alternatives and select the set of the best five alternatives. For participants not stimulated by objectives, self-appraisal indicated some improvements in the set of the best five alternatives between phases 1 and 2 and few improvements between phases 2 and 3. For the participants stimulated by objectives and guided on how to use them, self-appraisal recognized many improvements between each successive phase. When a large number of different individuals separately evaluated the sets of best five alternatives, they judged that there was essentially no improvement from phase 1 to phase 2 or from phase 2 to phase 3 for participants without objectives. In contrast, they felt that there were many improvements for the analogous two comparisons for the group stimulated by objectives. 10. Practical Guidelines to Create Alternatives for Decisions On decisions where the generation of alternatives is important, decision makers should benefit from dedicating some time to create desirable alternatives. Practical advice to do this follows from the results of our five studies. It is presented in three steps for an analyst working on a decision with the decision makers and other participants in the decision-making process. Step 1. Make sure that decision makers and participants involved in creating alternatives agree on a statement of the decision problem. Participants should be informed that many good alternatives have not yet been discovered and that their thoughtful participation will likely create some of those desirable alternatives. Step 2. Create a master list of objectives for the decision. Since individuals, including the authors, can rarely create a relatively complete list, it is useful to generate objectives with input from several individuals. Either an analyst or the decision maker can elicit help from several appropriate individuals (i.e., coworkers, colleagues, friends). Experience with creating master lists objectives for decisions indicates that 10 individuals, if motivated to think hard about objectives for a decision, will collectively create about 80% of the objectives on a master list created by input from 100 or more individuals. Even five individuals will usually produce at least half of a very thorough master list and many more than any individual. Suggestions to enhance an individual s ability to create objectives are found in Keeney (1992) and Bond et al. (2010). The analyst or the decision maker can then eliminate redundancies and combine objectives that are nearly the same. The master list is essentially the union of the distinct objectives identified. The analyst or decision maker should then put each of these objectives into a common format, such as verb-noun (e.g., minimize accidents, enhance convenience, reduce environmental impact) to make the list more meaningful. Step 3. Create alternatives using the objectives. Initially, have each participant in the decision-making process, and perhaps a few other knowledgeable individuals, work alone to obtain ideas from everyone and reduce anchoring on comments made by others. The individuals should first use one objective at a time, then pairs of objectives, and finally larger sets of objectives to create new alternatives or improve alternatives that they have already created. The collective list of all different alternatives identified in step 3 is the creative material for developing a final set of distinct alternatives. In this process, the decision makers and analysts may initially find it useful to try to generate better alternatives by combining desirable features and reducing undesirable features of different alternatives on the collective list. Depending on the decision context and the alternative creation process, at this stage there will either be distinct alternatives, one of which could be chosen, or elements of alternatives, several of which should be combined into a distinct alternative for possible choice. This combining process could use tools such as the strategy table (Howard 1988) or the morphological box (Zwicky 1967). Two practical issues about the process above are worth mentioning. First, participants involved in creating objectives can do better if they are able to devote full attention to the task. In some decision contexts, a group of knowledgeable individuals brought together to create objectives may each have previously given serious thought to how to solve the problem. In such cases, prior to creating objectives in step 2, it may be helpful to acknowledge that the participants certainly have good ideas for alternatives already and ask them to write them down for subsequent discussion. This may allow them to clear their minds and create objectives with fewer distractions concerned with moving on to make the decision. Second, the process outlined to create alternatives can be done by any operations researcher or decision analyst by motivating participants to think hard and creatively. The difficult aspect of creating a reasonably thorough list of objectives is done by the collection of participants as described in our experiments. The second author has more than 25 years experience in facilitating the identification of objectives and created the master list for Study III. The first author had little experience creating objectives prior to guiding the creation of master lists for Studies I, II, IV, and V. This is a task that any analyst or decision maker can quickly learn to do well. Once you have done this a few times you will have improved and have more experience than most others.

15 Operations Research 63(5), pp , 2015 INFORMS A Policy Application A process similar to that described above was used on a policy decision (Keeney 2012), and the results suggested aspects of the five studies described in this paper. After the 2001 terrorist attacks on the World Trade Center, the National Institute of Standards and Technology (2005) recommended that Building evacuation should be improved to include system designs that facilitate safe and rapid egress, methods for ensuring clear and timely emergency communications to occupants, better occupant preparedness regarding their roles and duties for evacuation during emergencies, and incorporation of appropriate egress technologies. In 2008, a 2.5-day invitation-only workshop was held to create alternatives that might improve evacuation of large buildings. In this workshop the second author acted as a facilitator. The participants had experiences covering many relevant fields including firefighting, building codes, communications, construction, human behavior, and emergency management. Because all participants in the workshop had previously thought about building evacuation, each was initially asked to list any alternatives that he or she thought would enhance evacuation. This was partly to assure each participant that his or her previously developed ideas for alternatives would not be lost in the creative process to stimulate new thoughts for evacuation. The 30 individuals collectively listed 221 alternatives. These individuals then generated objectives, which were aggregated by the facilitator into 19 main objectives, such as facilitate responder access, isolate the fire, enhance communication, minimize evacuation time, and save lives. Using these objectives, 21 participants created 81 more alternatives considering single objectives, and then 18 participants created 48 more alternatives considering pairs of objectives. For a group of very knowledgeable professionals, who had created an average of 7.4 alternatives on their own, to be able to create an average of 6.5 additional alternatives when stimulated by objectives seems noteworthy. The alternatives created in this workshop were elements of alternatives. These elements were organized into 18 categories, such as sprinklers/active suppression systems, building material changes, communications, stairwell evacuation, and event detection. Combinations of elements from each category would constitute a distinct alternative Ideas for Future Research Results in this article suggest several potentially useful experiments about creating alternatives for decisions where the subjects are a decision maker. Such experiments could address, for example, questions like the following: 1. What are the pros and cons of having decision makers create objectives first and then create alternatives compared to having the decision maker create alternatives first, then create objectives, and then use the objectives to create more alternatives? 2. What are the pros and cons of having the decision maker create his or her own objectives versus receiving an externally generated set of objectives (e.g., by a decision analyst) prior to using objectives to create alternatives? 3. Does the use of objectives to create alternatives affect different types of decision makers (e.g., experts versus novices) differently? 4. How effective are other techniques, such as specific incentives and challenges, to stimulate the generation of alternatives, used either alone or in conjunction with the use of objectives? 5. When a single decision maker must create alternatives for a decision without any external guidance or assistance, what process might best help that individual create better alternatives? Acknowledgments The authors thank the area editor, associate editor, three anonymous referees, Jörg Schlüchtermann, and Bernd R. L. Siebert for their valuable comments and Constanze Kopp, Matthias Molitor, and Alexander Alt for their support in executing the experiments. References Ackoff RL (1978) The Art of Problem Solving (John Wiley & Sons, New York). Adams JL (1979) Conceptual Blockbusting: A Guide to Better Ideas, 2nd ed. (Norton, New York). Bond SD, Carlson KA, Keeney RL (2008) Generating objectives: Can decision makers articulate what they want? Management Sci. 54(1): Bond SD, Carlson KA, Keeney RL (2010) Improving the generation of decision objectives. Decision Anal. 7(3): Butler AB, Scherer LL (1997) The effects of elicitation aids, knowledge, and problem content on option quantity and quality. Organ. Behav. Human Decision Processes 72(2): Gettys CF, Pliske RM, Manning C, Casey JT (1987) An evaluation of human act generation performance. Organ. Behav. Human Decision Processes 39(1): Howard RA (1988) Decision analysis: Practice and promise. Management Sci. 34(6): Jungermann H, von Ulardt I, Hausmann L (1983) The role of the goal for generating actions. Humphreys P, Svenson O, Vari A, eds. Analyzing and Aiding Decision Processes (North-Holland, Amsterdam), Keeney RL (1992) Value-Focused Thinking A Path to Creative Decisionmaking (Harvard University Press, Cambridge, MA). Keeney RL (2012) Value-focused brainstorming. Decision Anal. 9(4): Keller LR, Ho JL (1988) Decision problem structuring: Generating options. IEEE Trans. Systems 18(5): National Institute of Standards and Technology (2005) Federal building and fire safety investigation of the World Trade Center disaster: Final report of the National Construction Safety Team on the collapses of the World Trade Center Towers (NIST NCSTAR 1). Report, U.S. Department of Commerce, Washington, DC, customcf/get_pdf.cfm?pub_id= Pitz G, Sachs N, Heerboth T (1980) Procedures for eliciting choices in the analysis of individual decisions. Organ. Behav. Human Performance 26(3): Selart M, Johansen ST (2011) Understanding the role of value-focused thinking in idea management. Creativity Innovation Management 20(3): Tani SN, Parnell GS (2013) Design creative alternatives. Parnell GS, Bresnick TA, Tani SN, Johnson ER, eds. Handbook of Decision Analysis, 1st ed. (John Wiley & Sons, Hoboken, NJ),

16 1158 Operations Research 63(5), pp , 2015 INFORMS Zwicky F (1967) The morphological approach to discovery, invention, research and construction. Zwicky F, Wilson AG, eds. New Methods of Thought and Procedure. Contributions to the Symposium on Methodologies (Springer, Berlin), Johannes Siebert is Habilitand and Akademischer Rat at the Faculty of Law, Business and Economics of the University of Bayreuth. His research objective is to contribute to better informed decision making and thereby to making better decisions, for individuals as well as for organization using problem structuring, value-focused thinking, and decision analysis. He pursues his research objective with three pillars: empirical studies, developing methods, and real problems applying consulting. Ralph L. Keeney is Research Professor Emeritus at the Fuqua School of Business, Duke University. His professional interest is to help organizations and individuals make better decisions by identifying, structuring, and analyzing their decisions using valuefocused thinking and decision analysis. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. You are free to download this work and share with others, but cannot change in any way or use commercially without permission, and you must attribute this work as Operations Research. Copyright 2015 INFORMS. used under a Creative Commons Attribution License: