Stochastic simulation of risk factor potential e ects for software development risk management

Transcription

1 The Journal of Systems and Software ) 247±257 Stochastic simulation of risk factor potential e ects for software development risk management Dan X. Houston a, *, Gerald T. Mackulak b,1, James S. Collofello c,2 a Honeywell Inc., 817 W. Northview Ave., Phoenix, AZ 85021, USA b Industrial Engineering Department, Arizona State University, Tempe, AZ , USA c Computer Science & Engineering Department, Arizona State University, Tempe, AZ , USA Received 12 July 2000; received in revised form 1 December 2000; accepted 22 March 2001 Abstract One of the proposed purposes for software process simulation is the management of software development risks, usually discussed within the category of project planning/management. However, modeling and simulation primarily for the purpose of software development risk management has been quite limited. This paper describes an approach to modeling risk factors and simulating their e ects as a means of supporting certain software development risk management activities. The e ects of six common and signi cant software development risk factors were studied. A base model was then produced for stochastically simulating the e ects of the selected factors. This simulator is a tool designed speci cally for the risk management activities of assessment, mitigation, contingency planning, and intervention. Ó 2001 Elsevier Science Inc. All rights reserved. Keywords: Software development risk; Risk factors; Software risk management; Stochastic simulation; Software process simulation 1. Introduction * Corresponding author. Tel.: addresses: dxlsh@uswest.net D.X. Houston), G.T. Mackulak), collofello@asu.edu J.S. Collofello). 1 Tel.: Tel.: One of the proposed purposes for software process simulation is the management of software development risks, usually discussed within the category of project planning/management Kellner et al., 1999). However, modeling and simulation primarily for the purpose of software development risk management has been quite limited. A notable exception is Madachy's model Madachy, 1994), designed partially for the purpose of risk assessment. It uses an expert system front-end to provide two simulation inputs that represented combinations of COCOMO drivers heuristically graded for degrees of risk. This paper describes a di erent approach to simulation for managing software development risks, that of researching common and signi cant software development risk factors and their e ects, then adapting a base model for stochastically simulating the e ects of the selected factors. This simulator is a tool designed speci cally for the risk management activities of assessment, mitigation, contingency planning, and intervention. This paper is organized to provide an overview of a project in software development risk factor SDRF) research, modeling, and simulation. The basis for the risk factor approach to modeling is discussed rst Section 2), followed by results of a literature search that identi ed software development risk factors Section 3). Modeling SDRFs required new research into their potential e ects Section 4), the results of which were used to extend a base model Section 5) with modeling constructs for SDRFs Section 6). After discussing model validation Section 7), a risk management framework is described as a context for model usage Section 8). Results from risk assessment runs compare the e ects of selected SDRFs Section 9). 2. Assessing uncertainty through risk factors Modeling for risk management requires that one both understand risk and de ne a means of characterizing it. This section discusses the nature of risk using categories and examples, then discusses the approach of using risk factors in system modeling /01/$ - see front matter Ó 2001 Elsevier Science Inc. All rights reserved. PII: S )

2 248 D.X. Houston et al. / The Journal of Systems and Software ) 247± Describing uncertainty and risk Uncertainty gives rise to risk, the potential of loss. Whether they think of themselves as risk managers or not, those who manage projects are managing or failing to manage) the many uncertainties attendant to a project. The categories of Gemmer 1997) Table 1) are helpful for thinking about the many and various uncertainties in managing software projects. Project managers confront all three of these types of uncertainty. For example, a manager may not know when his lead developer will leave the project uncertainty in time); he may exhibit a lack of in uence over a peer review process uncertainty in control); or, he may lack accurate information about customer expectations for product performance uncertainty in information). One means of managing the risks arising from uncertainty is to characterize risky scenarios and identify the factors in those scenarios. Each scenario can then be associated with a probability of occurrence, a potential cost, and its value utility) to the stakeholders. However, analysis of scenarios begins with identi cation and quantitative description of the factors composing scenarios. These risk factors can then be arranged in various scenarios and, using a vehicle for propagating their uncertainties, can be related to system outcomes A generic process for identifying and assessing system risk When risk factors and their associated probabilities of adversely a ecting an outcome can be speci ed, then a variety of methods can be employed to synthesize the factors in a model for estimating the system risk. These methods can be generally characterized by a generic process for estimating the inherent uncertainty for each risk factor, then propagating the uncertainties from the factor level to the system level. This process is composed of ve steps that are aligned with the de nition of risk. Identify the risk factors. A system must be analyzed for the purpose of identifying the signi cant risk factors. Model the system to incorporate the risk factors. A model of the system describes the relationships between the signi cant risk factors. Table 1 Managerial categories for uncertainty Type of De nition uncertainty In time Uncertainty about when certain events may occur or the ability to react to them In control Inadequate authority to make or in uence decisions or inconsistency in processes In information Inadequate or inaccurate information on which to base decisions Quantify risk factor uncertainties. The uncertainty characterizing the risk factors is described by assigning each a probability of occurrence. Propagate the uncertainties. In this step the model is exercised to generate a probability of system success/failure. Sensitivity analysis. Once the probability of system success/failure is calculated, the model can be used to study alternatives that decrease the probability of system failure. This step, or the whole process, can be iterated until a solution is achieved. This generic process for identifying and assessing system risk can be implemented in a variety of methods. Three of these are regression analysis, expert systems, and stochastic modeling. Software cost estimation tools are typically developed by single regression across job size and multiple regression across in uential, or cost driver, variables that can be treated as risk factors. Input variances are propagated arithmetically. These tools can be used for rudimentary risk assessment through sensitivity analysis, particularly when measures of statistical uncertainty for the inputs are provided. Madachy's model, mentioned in Section 1, provides a good example of an expert system for assessment of scenarios composed of risk factors in the form of combinations of COCOMO cost drivers cf. Madachy, 1997). This expert system produces a constant that represents the software development risk. The risk rather than the uncertainty) is propagated by using the constant as a simulation input. PERT Project Evaluation and Review Technique) charts can be modeled stochastically by assigning a probability distribution usually triangular) to each task duration, thereby treating the task durations as risk factors. The uncertainty is propagated by calculating a project schedule for a sample from each distribution. A su cient number of these calculations provides a distribution describing the project schedule. This paper demonstrates dynamic stochastic simulation as a exible vehicle for e ectively propagating risk factor uncertainties of all types Table 1). A simulation model supports risk management to the extent that it supports the generic process described above. 3. Software development risk factors SDRFs) The software engineering literature on risk has focused on three topics: a) the appropriation of techniques from other disciplines; b) the development of risk management approaches for software development; and c) the identi cation of software development risk factors and their relationship to project outcomes. Although the second topic, risk management, has received the most attention, considerable attention has been and is being given to understanding the sources of risk in

3 D.X. Houston et al. / The Journal of Systems and Software ) 247± Table 2 Most important SDRFs Creeping user requirements Unavailability of key sta Reliance on a few key personnel Project manager unavailable Instability and lack of continuity in project sta ng Lack of sta commitment, low morale Excessive paperwork Large and complex project Lack of experience with the software product type Inexperience with project's platform/environment/ Methods Lack of senior management commitment Lack of client support Lack of quantitative historical data Lack of contact person's competence Inaccurate metrics Lack of organizational maturity Complex application Incapable project management Inaccurate cost estimation Excessive schedule pressure Unnecessary features Inadequate con guration control Lack of user support Inexperience with the user environment/operations Excessive reliance on a single development improvement Large number of complex external interfaces Immature technology Low productivity Unreliable subproject delivery software development, particularly through identi cation of risk factors. A variety of approaches have been used to investigate SDRFs. From them have emerged collections of risk factors, including prioritized lists, taxonomies, questionnaires, and matrices for assessing software development risk. Some investigators have produced SDRF lists numbering on the order of 150 or more factors. In a study of this literature, Houston 2000) found twentynine 29) of these factors were cited most often in studies intended to identify the most important SRDFs. This set of common and important SDRFs, listed in Table 2, was used to study the potential e ects of SDRFs. 4. Identifying risk factor potential e ects The identi cation of SDRFs provides a starting point for modeling them in a software process simulator. However, their potential e ects must also be known and the probabilities of occurrence understood. While a small but important body of work identifying SDRFs is accumulating, little work has been undertaken on the potential e ects of these factors. To address this knowledge gap, a study was performed in two stages to identify and quantify the potential e ects. In the rst stage, a qualitative survey based on causal mapping, a technique commonly employed in system dynamics modeling Richardson and Pugh, 1981), was used to identify potential e ects. To facilitate the survey process, a set of preliminary e ects diagrams was drawn using software development problem e ects suggested in the literature, particularly in Jones 1994). Software project managers were invited to review these diagrams and revise them, adding missing e ects, deleting insigni cant or nonexistent e ects, and indicating the relative importance of the e ects. Thirty-six respondents, representing 22 software organizations, each reviewed and marked diagrams for approximately a third of the factors. Since the quality of the survey results depended on the knowledge of the reviewers, the results were validated by gathering data on the reviewers, using years of experience as a surrogate variable for knowledge of software development risk factor e ects. Reviewers were asked for their years of experience in each software development role, for the major software domains in which they had experience, and for the project sizes with which they had experience. Statistics calculated for the respondents to each diagram veri ed that distribution of the survey packages was unbiased relative to reviewer experience. Tables 3 and 4 provide summary statistics on the experience of the 36 respondents with regard to roles and domains. All ranges of project sizes were also represented in the experience of these reviewers. Both the depth and breadth of the reviewers' experience were viewed as supporting the assumption that they constituted a knowledgeable group for this survey. The results were compiled by producing a merged set of diagrams that showed added elements, deleted elements, and degree of agreement on existing elements. A third set of diagrams was then produced by extracting from the second set only those elements upon which respondents generally agreed. For example, an element Table 3 Reviewers' average years of experience by software development role Role Average years of experience Individual contributor 10.1 Project manager 9.6 Other roles 7.0 Total experience in all roles 21.0 Table 4 Number of reviewers experienced in each software domain Software domain Number of reviewers Management information systems 21 Systems software for control of physical 27 devices) Commercial 14 Military/DoD 19 Other 8

4 250 D.X. Houston et al. / The Journal of Systems and Software ) 247±257 new to a preliminary diagram was used only when it was independently added by two or more respondents. One of the nal diagrams for which agreement was particularly strong, ``excessive schedule pressure'', Fig. 1 3 )is shown here with the description provided in the survey instrument. Excessive schedule pressure is de ned by multiple direct e ects on loss of process discipline, productivity, morale, and exhaustion. The survey results showed that participants usually had little di culty agreeing to the description provided with each SDRP, but o ered considerable modi cations to the e ects diagram for each factor. In the second stage of the study, six of the 29 SDRFs were selected for further investigation, based on their importance both the frequency with which they were cited as problematic and their potential for adversely a ecting projects) and the interrelationships found in the rst survey. Table 5 lists the selected factors. 4 Based on the results of the rst survey, these six factors were hypothesized to form a network of in uence as shown in Fig. 2. One of the reasons for selecting these six factors for further study and modeling is that the network in Fig. 2 was at the center of a larger network of factors, so inclusion of these six factors can serve as a basis for later modeling e orts. A quantitative survey was developed to measure the occurrence and e ects of each factor. This survey asked project managers to estimate the degree to which each factor was present as a problem on their most recently completed project, and the degree to which each of the hypothesized e ects occurred. The survey was announced through and hosted on a website. Over an eighteen-day period, 458 people participated. Regarding the project roles of the survey respondents, 89% were either the project manager 73%) or the team leader 16%) of the project about which they reported. Most of the respondents were experienced as both an individual developer and as a team leader/project manager. Table 6 pro les the experience of the respondent population in each of these two role categories. The respondents worked mostly on custom software projects, either for clients internal or external to their 3 Regarding notation, positive and negative signs indicate the direction, same +) or opposite )), of a relationship between a cause and its e ect. A positive sign indicates that the e ect increases as the cause increases; a negative sign indicates that the e ect decreases as the cause increases. Multiple signs indicate a stronger causal in uence. Also, in an enhancement to the usual method of causal diagramming, it was necessary to introduce conditions as quali ers for some of the e ects. E ects in bold are also risk factors. 4 These six factors exhibit all three managerial categories in Table 1. Requirements creep and sta ng instability manifests uncertainty in time. Inaccurate cost estimation is a result of information uncertainty. The other three factors in Table 3 are conditions that produce uncertainty in control. companies Table 7), and the projects ranged from very small 64 person-months of e ort) to very large >8000 person-months), but a third of them were of intermediate size 13±70 person-months). The survey results validated the common occurrence of the six problems, as well as their potential impacts. Table 8 shows the percentage of surveyed projects that exhibited each problem. Analysis of this data was used to validate or deny the results of the rst survey. Most e ects were validated, but a few were not. For example, ``Lack of senior management commitment'' was expected to produce an unwillingness to extend a project schedule as new work was discovered and requirements changed. However, only a minor statistical relationship was found between these two variables. On the other hand, senior management commitment was found to condition the e ects of other factors, such as changes in morale due to excessive schedule pressure. The data was also analyzed to produce distributions for random variables that described the e ects of each of the six factors. For example, the percentage of additional work due to requirements creep and the percentage of rework due to requirements creep were both modeled as exponential distributions. 5. A base model Software development projects have been modeled in a number of simulators, but usually these have not been designed speci cally for risk management activities. However, due to their scopes, they can be considered, individually or in combination, as a basis for a model to which SDRFs can be added. Thus, the modeling e ort described here proceeds with the notion of a base model extended with modeling of risk factors and signi cant e ects. From among four published, system dynamics models of software projects, two ± Abdel-Hamid and Madnick 1991) A-H&M) and Tvedt 1996) ± were selected for constructing a base model. A-H&M was selected for its comprehensively modeled scope of software project management and development. It also models the widely used waterfall process. 5 However, it is limited by its reliance on COCOMO for sta ng and scheduling; by its de nition of experience as project-speci c; and by its aggregation of defect management. Also, its treatment of reviews has been outdated by advances in work product inspection techniques and practice, and research has not supported its biological modeling of er- 5 In the quantitative survey conducted for this research, 29% of respondents reported using a waterfall process. Other processes reported used were incremental 23%), prototype-driven 20%), code and x 16%), spiral 7%), and combination or other 5%).

5 D.X. Houston et al. / The Journal of Systems and Software ) 247± Risk Factor: Excessive schedule pressure Description: Medium size and larger projects are susceptible to the expectation that deliverables can be completed in shorter durations than are technically feasible. The source may be internal to the project (for example,project leadership), internal to the organization (for example, senior management or a marketing group), or external (for example, a client or a user group). Fig. 1. Description and e ects diagram for ``Excessive schedule pressure''. Table 5 Six SDRFs selected for quantitative study Creeping user requirements Lack of sta commitment, low morale Inaccurate cost estimation Instability and lack of continuity in project sta ng Excessive schedule pressure Lack of senior management commitment Table 7 Types of software projects about which respondents reported Project type Percent of respondents Custom, internal 28% Custom, external 27% Systems 4% Commercial 16% Military/DoD 4% Outsourced or third party 11% Other 10% Fig. 2. Hypothesized in uence diagram of six selected SDRFs. Table 6 Experience pro les of quantitative survey respondents by software development roles Experience years) Percent of respondents Individual developer role 0 21% 5% 1±5 28% 43% 6±10 27% 31% 11±15 15% 14% 16±20 5% 5% >20 4% 2% Team leader/project manager roles Table 8 Occurrence of six problems studied Inaccurate cost estimation a 57% Requirements creep b 60% Sta ng attrition and turnover c 58% Lack of sta commitment, low morale d 50% Excessive schedule pressure e 58% Lack of senior management commitment f 38% a Inaccurate cost estimation was considered a problem when the actual e ort expended on the project, after subtracting for requirements creep, varied from the project's estimated e ort by more then 20% more than 20% higher or less than 20% lower than the estimated e ort for the project). b Requirements creep was considered a problem when requirements additions and changes either added more than 10% to the product size, or caused more than 10% of the developed work product to be reworked. c Sta ng attrition was de ned as unplanned departures. In other words, sta ng attrition and turnover was considered a problem if any of the project sta members left the project prior to the time agreed upon when they joined the project. d Lack of commitment, low morale was considered a problem when a survey respondent reported the sta 's lowest morale during the project was lower than a neutral point described as ``Neither `up' nor `down'.'' e Excessive schedule pressure was considered a problem when a survey respondent described the highest schedule pressure during the project as greater than or equal to ``High pressure 1.3)'', where 1.3 is the ratio of ``the e ort required for the remaining work'' to ``the e ort available for the remaining work''. f Lack of senior management commitment was considered a problem when a survey respondent disagreed with or was neutral regarding the statement, ``Senior management was committed to the success of this project and supported it with adequate resources sta, tools, time, etc.) and interest''.

6 252 D.X. Houston et al. / The Journal of Systems and Software ) 247±257 rors May et al., 1990). Tvedt was selected because it reused many of the modeling constructs employed in A- H&M and addressed these limitations. In addition to combining features of these two models, the base model was extended to include a rework, or test-and- x, cycle. Composing a base model from existing models has several advantages. Although a composition from existing models must be validated, it does reuse concepts and constructs of validated models. To the extent that the selected models are the same, modeling is facilitated. When di erent constructs are used to model the same phenomenon, the more attractive one can usually be selected. And, the purpose of the new model may be partially addressed. In this case, some risk factors are already partially incorporated by both of the source models simply due to their scope. The largest gain in this reuse e ort was the modeling of schedule pressure. Excessive schedule pressure and two of its e ects, on productivity and error generation, were already modeled by A-H&M and reused in Tvedt. The organization of the base model is similar to that of its predecessors. It has sectors for Planned sta ng, Actual sta ng, E ort allocation, Project planning, Project control, Adjustment of jobe ort, Productivity, Work ow, and Quality management. A brief description of each sector follows: Planned sta ng. The base model borrows Tvedt's sta ng pro les, one for each combination of experienced/new and developer/tester. The pro les are used 1) for summing cumulative planned manpower, 2) for calculating planned daily manpower, and 3) for calculating remaining planned manpower as it is used. Actual sta ng. A-H&M's personnel ow into and out of a project was adapted to provide two de nitions of experience: experienced in the project A-H&M) and experienced in platform and processes of the software development organization Tvedt). E ort allocation. This sector controls the priority of e ort usage as in A-H&M, with the following modi cations: 1) e ort allocated for reviews is based on nominal e ort required rather than a fraction of total available e ort; 2) review backlog tolerated, depending on schedule pressure, is added; 3) e ort allocations for development and testing are treated independently since they are sta ed separately; 4) an allocation for rework after testing is added because this model represents a project in which testing rework is done by developers; 5) desired rework delay is a function of time, per Tvedt, so that the delay is shortened as the project completion date approaches; 6) the testing e ort allocation divides the total testing e ort between planned testing and testing of rework new in this model), such that the planned testing is all done rst. Project planning. This sector, taken from A-H&M, adjusts the schedule according to what is needed to nish the project, subject to the policies for adding sta. 1) The workforce_level_needed was modi ed such that it cannot be less than the planned sta ; 2) a switch that turns o the willingness to add personnel was added so that usage of a contractor pool can be selected/deselected. Productivity. Adapted from A-H&M, this sector contains all the calculations for modifying sta productivity, both developers and testers. The modi ers are workforce experience mix, learning as the project progresses, communication overhead, motivation morale), and the work rate. Motivation is modi ed by attrition and by schedule pressure. The work rate is modi ed by exhaustion and the ability to increase the work rate. Project control. This sector, taken from A-H&M, is the tracking and reporting sector. It tracks productivity, perceived completeness, e ort still needed, and available e ort remaining. This sector also has calculations for the start of testing, schedule pressure, and ending the simulation. Adjustment of job e ort. This sector updates the planned project e ort as new work is discovered. A- H&M had this function as part of the Control sector, but Tvedt broke it out into a separate sector. Two e ort amounts are updated: total jobe ort and e ort for testing. Total jobe ort is updated by three ows: increases in development work due to additional work, increases in testing work due to additional work, and increases due to underestimating the job. E ort for testing is updated with increases in testing work due to additional work and, once testing has started, with the increases due to jobunderestimation. Work ow. This sector combines modeling ideas from A-H&M with those of Tvedt and then extends them. This sector is based on the ow of work products during a project, from high level design through rework for defects and retesting. Work can enter the project four ways: as originally estimated work, as undiscovered work due to underestimation, as new work due to requirements creep, and as rework due to requirements changes. Although a single new requirement may produce both new work and rework of existing products, these are modeled as two distinct e ects and the inputs are taken as percentages of the initial perceived jobsize. Following both A-H&M and Tvedt, the generation of the software product is highly aggregated: design and coding are not distinguished. The remainder of the work ow is adapted from Tvedt's model and extended to include a rework phase. Quality management. This sector is based on the ow of errors and defects. Errors are generated during product generation. They may be detected during peer reviews and then reworked. A bad x rate is associated with this rework because some of the errors may not be xed, or they may have undesirable side e ects. Un-

7 D.X. Houston et al. / The Journal of Systems and Software ) 247± detected errors become defects and may be found in testing. Those defects not found in testing are presumed to escape in the product as released beyond the scope of the project. Defects detected in testing can be reworked, with an associated bad x rate. The explicit modeling of peer reviews and lack of attention to unit testing re ects an industrial environment in which peer reviews are emphasized organizationally and unit testing is left to the discretion of the individual developer. The type of testing is not speci ed in the model, except that it is not unit testing. It is performed by an independent testing sta, so may include product testing, integration testing, and system testing ± the scope of the testing represented by the model depends on the scope of the project simulated and how the model is calibrated. 6. Modeling SDRFs Though the base model is deterministic, it provides a sound vehicle for propagating the uncertainties of risk factor e ects characterized by random variates. The project outcomes, re ecting the variation of the random distributions, can be estimated as con dence intervals rather than as point estimates. Specifying outcomes in terms of con dence intervals, calculated speci cally from selected risk factors, provides a strong statistical tool for supporting risk management. Risk factor e ects in this model have been implemented stochastically, with the exception of e ects already modeled. When e ects of a risk factor were incorporated in the base model, the existing deterministic modeling construct for the e ect was retained so as to limit the expense of data collection. For new e ects modeled, analysis of the survey data generally provided both the probability of each e ect's occurrence and the distribution of a random variable for the degree of each e ect. Table 9 lists the potential e ects modeled for each risk factor and the variables used to model each e ect. Each random variable is implemented with a distinct random distribution. Some distributions in the model are sampled once per run and others are sampled continuously throughout each run. For example, requirements creep is calculated for each run, so the distributions for its vari- Table 9 Six SDRFs selected for simulation Risk factor Potential e ects modeled Random variables Requirements creep Increased jobsize Percent new work due to requirements creep Time of maximum requirements creep Percent of requirements creep after the maximum Rework Percent rework due to requirements creep Relative rework productivity Inaccurate cost estimation Excessive schedule pressure Lack of sta Commitment, low Morale Instability and lack of continuity in sta ng Actual jobe ort more or less than e ort provided by sta ng and schedule Fluctuating productivity, exhaustion, and higher error generation Morale change Weaker reviews Lower productivity Increase in error generation Attrition Attrition Morale change Lower productivity due to loss of expertise Size estimation inaccuracy a In existing models) Change to morale for schedule pressure In existing models) Review backlog tolerated) Multiplier to productivity for morale Multiplier to error generation for morale Multiplier to attrition for morale Attrition Replacement delay Change to morale for attrition In existing models) Lack of senior Management Initial schedule compression Multiplier to schedule Commitment Understa ng Multiplier to sta ng Lower morale after periods of excessive schedule pressure and attrition Change to morale for schedule pressure Change to morale for attrition Willingness to extend schedule Maximum schedule extension a Data were collected for calculating inaccuracy of both size and e ort estimates. Analysis revealed that 1) the distribution of cost estimation inaccuracy was the same for projects that estimated size as for those that did not; and 2) the distributions for size estimation inaccuracy and cost estimation inaccuracy were nearly the same. For these reasons and because the model production stream is driven by job size, size estimation inaccuracy is used as a surrogate for cost estimation inaccuracy.

8 254 D.X. Houston et al. / The Journal of Systems and Software ) 247±257 ables are sampled at the beginning of each run. The e ect of low morale on productivity can vary during a project and it can be continuously recalculated, so the distributions for its variables are sampled continuously throughout each run Stochastic modeling construct examples An explanation of every stochastic modeling construct employed in the model is beyond the scope of this paper, but two examples, requirements creep and the e ect of low morale on productivity, are presented here. These two examples were chosen because they demonstrate two di erent kinds of stochastic modeling, one for a risk modeled as characteristic of an entire project requirements creep), and the other modeled as a risk that may vary throughout a project e ect of low morale on productivity) Requirements creep Requirements creep was modeled as a continuous ow of requirements additions and changes into a project. The ow increases linearly until it reaches a maximum, then decreases linearly Fig. 3). To parameterize this construct for new work due to requirements creep, the survey asked for the amount of this work as a percentage of the estimated jobsize, for t max, and for the percentage of the requirements creep that occurred after t max. Analysis of the data produced exponential distributions for the new work as a percentage of the estimated jobsize and for the percentage of requirements creep after the maximum; the time of maximum requirements creep is approximated by a Weibull distribution. These distributions are sampled once at the beginning of each run E ect of low morale on productivity Morale is modeled as stock having an initial level set by the user. This starting level, a value between 1 lowest morale) and 11 highest morale), typically re ects good morale, for example 7 slightly ``up'') or 8 fairly satis- ed team). The morale level can be a ected by high schedule pressure and by attrition. It may, in turn, produce e ects on productivity, error generation, and attrition. Fig. 3. A continuous model of requirements creep. Fig. 4. Logic for modeling the e ect of low morale on productivity. With regard to the e ect of sta morale on productivity, the survey results showed that a) when low morale a ected productivity, productivity decreased, b) the probability of an e ect increased from 0.46 to 0.95 as morale decreased from ``slightly down'' to ``lowest: open rebellion'', and c) the degree of decreased productivity can be characterized by a di erent random distributions dependent on the level low morale. Consequently, the productivity decrease due to low morale is modeled conditionally, as shown in the listing of Fig. 4. In Fig. 4, U 0; 1 is a random number generated from a uniform distribution 0; 1Š to represent the probability of the productivity e ect at each level of low morale. lognormal_distribution and Weibull_distribution are inverse transforms for random distributions of the fractional decrease in productivity. These distributions are sampled continuously during each run, producing stochastic productivity decreases throughout the periods when morale is low. 7. Model veri cation and validation The completed model, including both the base model and risk factors, was veri ed and validated using the framework proposed by Richardson and Pugh 1981). This framework allocates con dencebuilding activities into a three-by-two matrix, having veri cation, validation, and evaluation activities on the one axis, and structure and behavior on the other. These activities were performed by three individuals, the rst author and two experts in software process simulation who both have software development experience. The two experts reviewed the model for structural adequacy, structural sensitivity, face validity, parameter validity, and the appropriateness of model characteristics for the intended audience. The remaining exercises were performed by the rst author. The most interesting of the exercises was a case study in which the model's behavior was validated by using it to replicate an actual project. In the actual project, an average full-time equivalent sta of slightly less than

9 D.X. Houston et al. / The Journal of Systems and Software ) 247± Table 10 Actual project replication results Cost person-days) Duration days) Defect density defects/kloc) Size KLOC) Actual Simulated % Di erence )0.9 0 four people produced a forms processing application in 10 months. The application had been underestimated about 18% at 26 KLOC. The project experienced a nominal amount of requirements creep 5% of the estimated product size). The project sta had very high morale and experienced no attrition, despite very high schedule pressure in the later stages. Senior management was very committed to the project. The simulation model was calibrated to the actual project using values for 37 variables, including the amounts of requirements creep and estimation inaccuracy. The actual project had no risk exposure, aside from these two factors. Consequently, no variation is re ected in the simulation results Table 10). The simulation results closely approximated the actual project outcomes, cost and schedule being higher in the simulation 6.3% and 5.0%, respectively) than for the actual project. 8. Model usage Software project risk management is usually described in terms of six activities in two major categories, risk assessment and risk control. Boehm's framework Fig. 5) is illustrative. Within this framework, the present model, called Software Project Actualized Risk Simulator SPARS), is designed to support risk analysis and risk management planning for the six risk factors studied. Two types of simulation runs can be used in analyzing risks: 1) running the simulator without treating any of the risks as actualized provides a baseline for the project outcomes; and 2) running the simulator with risk factor s) actualized provides potential outcome measures for comparison with the baseline. Fig. 5. Boehm's risk management framework. SPARS is also designed to support three kinds of risk management planning activities: risk mitigation, risk contingency planning, and risk intervention. For risk mitigation study, various risk reduction measures may be simulated by setting inputs at the outset of a run. For risk contingency planning, triggers have been added to the model so as to simulate invocation of a mitigation activity upon the occurrence of a predetermined condition. For risk intervention, outputs provide the user with simulated project states so that a ``project'' may be paused and project parameters may be manipulated, thereby simulating an unplanned change for reducing the e ects of an actualizing risk. 9. Model results To illustrate some results of SPARS, it was rst parameterized similarly to A-H&M's EXAMPLE project 1991), their prototype project for experimentation. SPARS could not be calibrated exactly to imitate EX- AMPLE due to di erent means of inputting sta ng and schedule, calculating attrition, and processing defects found in testing.) The EXAMPLE project plan was to produce thousand lines of code KLOC) in 296 days with an average sta ng level of eight full-timeequivalent personnel, for an estimated cost of 2359 person-days. Due to underestimated product size, EX- AMPLE produced 64 KLOC in 430 days at a cost of 3795 person-days. The SPARS base case had an average full-time-equivalent sta of 10 people scheduled for 348 days. It produced 64 KLOC in 383 days at a cost of 3606 person-days. Although the cost of SPARS baseline case was within 5% of EXAMPLE, the SPARS base case includes the cost of xing defects found in testing and testing the xes, a cost not included in EXAMPLE. The duration of the SPARS case was 11% shorter than EXAMPLE because EXAMPLE uses a COCOMOcalculated sta ng rampup and rampdown, whereas the SPARS base case was fully sta ed at the start with constant sta ng levels, did not sta from a contractor pool, and had no attrition. Lack of rampup and rampdown, as well as separation of developer and test personnel, account for the higher average sta ng level in SPARS. The lower cost of the SPARS base case despite a higher average sta ng level is also explained by the fact that SPARS sums the applied costs rather than the cost of available sta.

10 256 D.X. Houston et al. / The Journal of Systems and Software ) 247±257 After establishing baseline outcomes, the risk factors were actualized. Figs. 6 and 7, which illustrate results of risk assessment using survey data with the SPARS base case, display six sets of results, the baseline run and a 90% con dence interval on the mean calculated from 100 runs) for each of ve actualized risk factors. Fig. 6 displays the results for e ort consumed and Fig. 7 displays the results for project duration. These con dence intervals indicate, with 90% con dence, that a mean project outcome falls within the respective interval. The con dence intervals ordinarily indicate variance above the baseline case outcomes, but the results for inaccurate estimation are an exception because overestimation can occur. Usually the presence of a risk is indicative of probable higher project cost and duration. Simulation results re ect this tendency by being very positively skewed. Consequently, the con dence intervals were not calculated in the usual manner using a t- statistic; instead, a distribution, such as a Weibull, was t to the outcome data and an inverse transform of the tted distribution was used to obtain the 1 a values for a ˆ 0:05; 0:95. The model and distributions derived from the survey show that requirements creep is the most signi cant risk factor modeled. Its con dence intervals are the longest, indicating a very high degree of unpredictability in outcomes for projects subject to this risk. Its con dence intervals also achieve the highest magnitude, indicating the largest impact on project outcomes. Inaccurate estimation also has long con dence intervals with high magnitudes, making it the second most signi cant risk of this group. Lack of senior management commitment is also a major risk, more to project duration than to project cost because one of its e ects can be cutting sta. The potential e ects of attrition and to low morale are relatively unimportant next to the previous three factors. Fig. 8 displays 90% con dence intervals on the mean project duration for each degree of senior management commitment without the e ect of cutting sta ). Recall that the nature of this factor's in uence is di erent from others: aside from sometimes in uencing project sta ng and schedule, it does not have direct e ects. Rather it tends to act in concert with other risk factors. This factor is a subtle one and its in uence on project outcomes exhibits an unusual pattern. It demonstrates a tendency to have less impact but be more unpredictable with increasing commitment. Low morale can also occur at varying levels and the model provides for setting the starting level for morale. Fig. 9 shows the project cost in response to the starting level of sta morale. These results are shown as points because the runs showed little variation within each starting level. The interesting trend in this graph is that project cost is a ected measurably as morale decreases, but the cost increases dramatically when the sta morale decreases from low to very unhappy. In addition to studying the e ects of individual risk factors, the model o ers the bene t of simulating any combination of the modeled risk factors, as well as a wide variety of risk management plans. Furthermore, the distributions in the model are based on industry data, but may be treated as a starting point for risk assessment because they can be re ned to re ect the experience of a particular organization. This usage was demonstrated by producing an extensive ctional risk management scenario in which the mitigated e ects of the risk factors were simulated in a series of twelve steps that produced a) a list of 11 items for a risk management plan, and b) numbers that illustrated the project planning trade-o s between released defect density, duration, and cost Houston, 2000). Fig. 6. Con dence intervals on mean cost for actualized risk factors. Fig. 8. Con dence interval on mean project duration for each degree of senior management commitment. Fig. 7. Con dence intervals on mean duration for actualized risk factors. Fig. 9. Variation in project cost due to starting level of low sta morale.

11 D.X. Houston et al. / The Journal of Systems and Software ) 247± Summary and conclusions The study of SDRFs was undertaken to identify common and signi cant risk factors for modeling in simulation. Qualitative and quantitative surveys were used to study the factors and their potential e ects. These survey results provided the basis for developing stochastic modeling constructs and statistical relationships incorporated into a software process simulator. This simulator, with its ability to model development project outcomes probabilistically, demonstrates that explicit modeling of risk factors and their potential effects provides an additional tool for software development risk management, particularly in the assessment and planning phases. Software project managers can use such a model to study the potential impact of various combinations of risk factors on project outcomes, then run simulations for risk mitigation plans, risk contingency plans, and interventions, all as means of elucidating their experience and supporting project management decisions. 11. Further research Research into SDRFs is ongoing and research into their potential e ects has begun. Continued research in this area is expected to provide further insights into and data for the development of models for software project risk management. Modeling constructs for risk factors are also evolving. Experimentation with risk-oriented simulators may provide insights into the relative in uence of various risk factors and into best practices for project risk management. Though some SDRFs are quite common, some appear to be speci c to certain software development domains Jones, 1994), which suggests domain-speci c simulation models for risk management. Finally, this research was oriented toward collection, analysis, and use of industrial data from a wide range of software projects. While use of such data provides a good starting point for risk management in organizations lacking such data, results can be expected to improve by re ning probability distributions using local data. References Abdel-Hamid, T., Madnick, S., Software Project Dynamics: An Integrated Approach. Prentice-Hall, Englewood Cli s, NJ. Gemmer, A., Risk management: moving beyond process. IEEE Software 14 3), 33±43. Houston, D., A software project simulation model for risk management. Ph.D. Dissertation, Arizona State University, Tempe, AZ. Jones, C., Assessment and Control of Software Risks. Prentice- Hall, Englewood Cli s, NJ. Kellner, M.I., Madachy, R.J., Ra o, D.M., Software process simulation modeling: why? what? how? Journal of Systems and Software 46, 91±105. Madachy, R.J., A software project dynamics model for process cost, schedule, and risk assessment. Ph.D. Dissertation, University of Southern California. Madachy, R.J., Heuristic risk assessment using cost factors. IEEE Software 14 3), 51±59. May, R.G., Jones, C.L., Hathaway, G.J., Studinski, D.P., Experiences with defect prevention. IBM Systems Journal 29 1), 4±32. Richardson, G.P., Pugh III, A.L., Introduction to System Dynamics Modeling with DYNAMO. MIT Press, Cambridge, MA. Tvedt, J., An extensible model for evaluating the impact of process improvements on software development cycle time. Ph.D. Dissertation, Arizona State University, Tempe, AZ. Dan Houston received his Ph.D. and M.S. degrees in Industrial Engineering at Arizona State University, Master of Divinity at St. Thomas University Houston, Texas), and B.S. Mechanical Engineering at the University of Texas at Austin. He leads a global software development team at Honeywell. His research interests include software project management and economics, software risk and quality management, and statistical modeling of software processes. He is a member of the IEEE Computer Society, the Association for Computing Machinery, and the American Society for Quality. Gerald T. Mackulak received his B.Sc., M.Sc., and Ph.D. degrees from the Department of Industrial Engineering at Purdue University. He is currently the Executive Associate Chair Undergraduate Studies in the Department of Industrial Engineering at Arizona State University. Dr. Mackulak has held positions with US Steel, Burroughs Corporation, and Pritsker and Associates as well as consulting for over 75 national companies. His primary areas of research interest are in the modeling of manufacturing systems particularly semiconductor AMHS), simulation methodology and production control. James S. Collofello is a professor in the Department of Computer Science and Engineering at Arizona State University. He received his Doctor of Philosophy degree in Computer Science from Northwestern University, Master of Science in Mathematics and Computer Science from Northern Illinois University, and Bachelors of Science in Mathematics and Computer Science from Northern Illinois University. He is a member of the IEEE Computer Society and Association for Computing Machinery. His research and teaching interests are in the software engineering area with an emphasis on software project management, software process improvement and software quality assurance.