Software project management using PROMPT: A hybrid metrics, modeling and utility framework

Transcription

1 Information and Software Technology 47 (2005) Abstract Software project management using PROMPT: A hybrid metrics, modeling and utility framework David M. Raffo * School of Business Administration, Portland State University, Portland, OR, USA Available online 17 November 2005 In this paper, we present a forward-looking decision support framework that integrates up-to-date metrics data with simulation models of the software development process in order to support the software project management control function. This forward-looking approach (called the PROMPT method) provides predictions of project performance and the impact of various management decisions. Tradeoffs among performance measures are accomplished using outcome based control limits (OBCLs) and are augmented using multi-criteria utility functions and financial measures of performance to evaluate various process alternatives. The decision support framework enables the program manager to plan, manage and track current software development activities in the short term and to take corrective action as necessary to bring the project back on track. The model provides insight on potential performance impacts of the proposed corrective actions. A real world example utilizing a software process simulation model is presented. q 2005 Elsevier B.V. All rights reserved. Keywords: Software process modeling; Project management; Simulation; Software measurement repositories; Multi-criteria decision making; Control limits 1. Introduction Making decisions about software processes is challenging. Each member of a project may have a different opinion as to the best way to improve the development process for their project even if project members are aware of best practices. In recent work, Raffo developed the process tradeoff analysis (PTA) method. This method builds on previous work by Kellner et al. at the Software Engineering Institute (SEI) [9 11] by developing a quantitative approach for evaluating potential process changes in terms of development cost, product quality, and project schedule [13,14]. This work has predominately been applied to the software project management planning function [13,15] which is one of six major applications of process simulation as described in Kellner, Madachy and Raffo [12]. The goal of this current research is to develop a forwardlooking approach that supports the software project management control function. We call it the PROMPT method (PROject Management of Process Tradeoffs). PROMPT integrates timely metrics data with rapidly deployable simulation models of the software development * Tel.: C ; fax: C address: raffod@pdx.edu /$ - see front matter q 2005 Elsevier B.V. All rights reserved. doi: /j.infsof process. By utilizing up-to-date project data, the simulation model becomes a project management tool that can predict likely project outcomes with greater and greater certainty as the project progresses. The concept for this framework was first reported in [16] and focused on the novel role played by the flexible metrics repository. In this paper, we focus on the overall decision framework using outcome based control limits (OBCLs), utility functions and financial performance measures (e.g. return on investment (ROI), net present value (NPV), break even point (BEP) and others) which is new. In addition, in the previous work, the method had not been as well developed or generalized and a full numerical example was not shown. Using PROMPT, tradeoffs among performance measures (e.g. cost, quality, and schedule) are accomplished using outcome based control limits (OBCLs) augmented by multicriteria utility functions and financial measures of performance (e.g. net present value, payback period, etc.) of various process alternatives as compared with a baseline. By combining metrics and predictive models, a more comprehensive performance picture of the project is achieved than by using metrics alone. Moreover, the predictive models can support managers as they attempt to re-plan to bring a project back on track. In this paper, the focus is to illustrate how process tradeoffs are made in the context of the PROMPT framework. To do this, we provide an overview of PROMPT

2 1010 D.M. Raffo / Information and Software Technology 47 (2005) by setting it in the context of Demming s PLAN-DO-STUDY- ACT cycle for process improvement [5,18]. We then present a real-world example showing the use of PROMPT. 2. Overview of the PROMPT framework The PROMPT framework is designed to be an iterative ongoing process improvement framework set in the context of Demming s PLAN-DO-STUDY-ACT cycle. PROMPT augments Demming s work by utilizing in-process data and quantitative models to support the planning, studying and action phases of the Demming PDSA cycle. Specifically by using models that predict process performance we augment: PLANNING: The model supports the planning phase by guiding the selection of process actions and decisions. By predicting the likely performance of each decision alternative, the model can be used to evaluate the expected utility or expected financial value of each choice so that sound tradeoffs can be made. STUDYING: By using timely metrics information to up date model parameters, the model can be used to study the progress of the plan. Using the up-to-date information, the model provides a more accurate prediction of current project trajectory and the likely outcome of the project. In addition, for models that provide distributions of project performance measures presented using either probability distribution functions (PDFs), cumulative distribution functions (CDFs) or confidence intervals, a quantitative assessment of the risk or uncertainty associated with the performance prediction is provided. The tradeoff in the STUDYING phase is whether to continue along the current project trajectory or to take corrective action. ACTION: When corrective action is deemed necessary, the model is used to identify corrective actions that can be deployed to bring the project back on track if the selected process plan is not yielding satisfactory benefits. The tradeoff in this step is to identify the best corrective action from a set of possible choices. Hence, the model supports testing process-based, risk mitigation strategies. As mentioned in [16], the above activities were supported by combining a universal metrics repository with a stochastic software process simulation model. In addition, this work also supports and operationalizes the experience factory concept by using collected metrics from past projects in a repository to be used to predict time, cost and performance for future projects. This work compliments the operational concepts put forward by Basili et al. [2]. For the general PROMPT approach, using a stochastic software process simulation model is not essential. Any model that fits the following criteria may be used: 1. Predicts one or more performance measures that are of interest to managers. The model must predict one or more performance measures for which managers can set performance targets and acceptable ranges. This enables managers to track process performance using outcome based control limits (OBCLS) on a short term or long term time horizon. To utilize the multi-dimensional utility function aspects of the PROMPT framework, the model or collection of models must be able to predict multiple performance measures for the same project. For the example presented in this paper, we utilize a discrete event process simulation model that predicts multiple performance measures development cost, project schedule and delivered product defects. However, a number of different model types such as cost estimation models, regression models, and so forth would be able to fulfill the above requirements. 2. Predicts process performance as point estimates or as stochastic distributions. OBCLs can be used with models that predict process performance as point values or as distributions (with a mean and standard deviation). If the models predict performance measures as point values, the results obtained from using OBCLs are very clear. Model predictions are either within the OBCLs, which means that the project is on track, or the are out of the OBCLs indicating that the project may be in trouble. There is no gray area because these point estimates do not communicate the inherent uncertainty associated with software development activities (or model predictions). When model predictions are in the form of distributions, they indicate the probability that the process will achieve the levels specified by the OBCLs and thereby provide a quantitative assessment of risk. The probability distributions can be presented in a variety of fashions using probability density functions (PDFs), cumulative distribution functions (CDFs) (which look similar to project management S-curves), confidence intervals, and so forth. Many models used in practice including cost estimation models such as COCOMO [3] provide point estimates. Some simulation paradigms (such as system dynamics [1]) also provide point estimates of project performance measures. Simulation paradigms such as discrete event, state-based, and newer applications of system dynamics models provide distributions for process performance measures. In the example provided in this paper, we show how OBCLs work when the model predicts performance measures in the form of PDFs that are graphed as box-and-whisker plots. CDFs which would look similar to S-curves could have equivalently been drawn to present the results. 3. Captures process level issues. In order to evaluate process decision alternatives, the model utilized in the decision framework must capture a sufficient level of process detail. Process Simulation models (discrete event, state-based and system dynamics simulation paradigms) capture process issues at a more detailed level than most other modeling techniques. As a result, these types of models are recommended when trying to support process level decisions. Moreover, to provide both short-term predictions as well as long-term predictions, models providing process level details are necessary. Models capturing process level issues are also needed to assess process-based risk mitigation strategies on a project. However, the PROMPT framework is

3 D.M. Raffo / Information and Software Technology 47 (2005) general and can be applied broadly with many different kinds of models provided they fit these criteria. For the example presented in this paper, a simulation model using the discrete event modeling paradigm was used. 4. Can utilize up-to-date metrics to refine predictions. In order to support project management and control decisions, the model used must be capable of being updated by data from an on-going project. If model parameters cannot be updated by timely, in-process data, the model will not likely be suitable to support project management activities. For example, cost estimation models use high-level summary inputs. If these inputs and cost drivers can only be derived from completed projects, then they would not be able to utilize data from on-going projects. As a result, these models would neither be suitable for project management and control activities nor for addressing many process level issues. Process simulation models contain parameters for each process step. As a result, these models can incorporate up-to-date metrics data very well and model predictions can be refined and improved when using this data. 3. The scenario under consideration In this section, we present the process example that will be used throughout the paper. The scenario described below was encountered through our work with a leading software development firm in the course of applying our process simulation models to their organization. However, the size of the proposed project and other sensitive data in the model such as productivity, defect injection and detection rates, among others were changed in order to maintain company confidentiality requirements. As a result, the performance predictions presented in this paper are different from those observed by the client organization Project scope The software development firm has been contracted to provide a major enhancement (approximately 32 KLOC) to an existing system. The enhancement consists of modifying six computer software configuration items (CSCIs) with modifications to each CSCI ranging between 4 and 8 thousand lines of additional code (KLOC). A software process model has been used to estimate the three performance measures of cost (project effort), schedule (duration) and quality (delivered defects) for the proposed project using past project data augmented with updated estimates for productivity, earned value, defect injection and detection, and project size among other parameters for the proposed project. The estimates were used in the bid process. After some negotiation, the client organization accepted the bid. The resulting estimates have now become the contracted project performance targets by which the project will be evaluated. Typically, calibration between the bid and the model is required. The model used in this example captures project effort for development, inspection and test activities. The model does not include overhead activities or other costs often included in contract bids The software development process used For this project, the software development firm plans to use an incremental software development process with the life cycle phases of requirements, preliminary design, detailed design, coding, unit test, integration test, and system test. inspections of the requirements, preliminary design, detailed design, and coding phases were conducted. 4. Overview of the simulation model A discrete event simulation model was developed of the software development process of Section 3.2. The model was written using the Extend simulation language by ImagineThat Inc. [6]. The model predicts process performance in terms of development cost, product quality and project schedule for the overall project. In addition, the model can easily record these performance measures for any of the individual process steps as desired. Some of the inputs to the simulation model include: Productivity rates for various processes Volume of work (i.e. KLOC) Defect detection and injection rates for all phases Effort allocation percentages across all phases of the project Rework costs across all phases Parameters for process overlap Amount/effect of training provided Resource constraints. Actual project data were used for model parameters where possible. For example, inspection data was collected from individual inspection forms for the past three releases of the project. Distributions for defect detection rates and inspection effectiveness where developed from these individual inspection reports by integrated project teams (IPTs). Effort and schedule data were collected from the corporate project management tracking system. Senior developers and project managers were surveyed and interviewed to obtain values for other project parameters when hard data were not available. Models were developed from this data using multiple regression to predict defect rates and task effort (for insights into how to develop these models, the reader is referred to the empirical software engineering literature, in addition, we offer [17]). These distributions and models were integrated into one model that predicted the three main performance measures of cost, quality, and schedule at each process step. These results were then summed as appropriate to develop overall project performance measures as can be seen in the following model equations: Total Effort Z SSeffort ij for all i and j (1) Effort i Z Sf ðproductivity i ; earned value i ; size j ; defects j Þ (2)

4 1012 D.M. Raffo / Information and Software Technology 47 (2005) Schedule i Z last_finish ij Kearliest_start ij for all j (3) Corrected_defects ij Z ðescaped ik1;j Cinjected i;j Þ!corr_rate i (4) Where: Total Effort is the sum of effort utilized over all process steps i for all work products j Effort i is the effort for an individual process step i summed over all work products j. Effort i is a function of productivity i, earned value i, work product size j, number of defects j detected or corrected (if an inspection/test or rework step, respectively) summed over all j. Schedule i is the duration for process step i. It is the difference between the latest finish time of all work products flowing through process step i less the earliest start time of all work products flowing through process step i. Corrected_defects ij is the number of defects that have been corrected in work product j in process step i. The number of corrected defects is the sum of the escaped defects from the previous process step ik1 for work product j plus the number of defects that are injected during process step i for work product j multiplied by the correction rate (corr_rate) for process step i. In the simulation model used for this paper, productivity, defect injection rates and defect correction rates were stochastic with distributions being derived from actual project data. The baseline simulation model of the lifecycle development process was validated in a number of ways. The most important of which were: Face validity Process diagrams, model inputs, and model parameters were reviewed by members of the software engineering process group as well as senior developers and managers for their fidelity to the actual. Output validity The model was used to accurately predict the performance of several past releases of the project. This included using data from different integrated project teams (IPTs) and using these parameters to predict individual CSCI performance as well as combining the teams CSCI performances to predict overall project performance. Scenario validity Two special cases regarding CSCI complexity and test performance occurring on the project were predicted accurately by the model. The model provides distributions for the values of each performance measure. These distributions are denoted by the expected values (mean) and standard deviations for the baseline process and were verified to be normally distributed. These are shown in Table 1. Again, due to company Table 1 Model estimates of project performance Mean STD Total effort (cost) in person months Project duration (schedule) in months Number of remaining defects (quality) confidentiality requirements, the numbers shown in Table 1 are based upon the modified data set that was used for this example and are not the actual numbers obtained by the development organization. Since, making tradeoffs using the PROMPT framework is the focus of this paper and since, any model that meets the criteria listed in Section 2 may be used, we shall not describe the simulation model further. Suffice to say that any model that fits the criteria such that it 1. Predicts one or more performance measures that are of interest to managers 2. Predicts process performance as point estimates or as stochastic distributions 3. Captures process level issues 4. Can utilize up-to-date metrics to refine predictions and is validated would be suitable. 5. Outcome based control limits Outcome based control limits (OBCLs) are acceptable ranges or control limits for project performance that are set by management. OBCLs are distinctly different than traditional statistical process control (SPC) limits in what they represent and how they are set although they are used in a similar manner. With traditional SPC models, control limits are derived based upon past process performance. As a result, control limits are more a measure of production consistency than an indication that the process is achieving management s desired goals. Because software projects can have high degrees of variability, managers need to get involved at the point when the project is going off track regardless of whether the project is performing consistently. Outcome based control limits identify the targets for project performance and the acceptable ranges of performance for the overall project. The simulation model is used to map current performance (which is reflected in the timely metrics that are collected by the metrics repository) to probable project outcomes. If the predicted performance deviates too much from the desired project outcomes, corrective action may be taken. This is further illustrated in the following sections. In some commercial or military settings, the customer/client evaluates the software firm on a number of measures such as cost, schedule, quality, productivity and so forth using a color rating system to indicate the level of accomplishment made toward achieving the planned performance targets. Different products addressing different markets require different levels of performance to be successful. Using outcome-based control limits, target values and control limits are determined based upon outcomes that management chooses to achieve. Hence, OBCLs can be determined based upon management s strategic or financial objectives, contract requirements or other concerns. The result is that this approach is very flexible and adaptable to the needs of a firm.

5 D.M. Raffo / Information and Software Technology 47 (2005) Table 2 Project performance targets and limits Total effort (cost) in person months Project duration (schedule) in months Number of remaining defects (quality) Target (from Table 1) Blue limits Green limits Yellow limits Red zone Upper (C5%) Lower (K5%) Upper (C15%) Lower (K15%) Upper (C30%) Lower (K30%) Above (OC30%) Below (!K30%) We will now continue with the numerical example. As the project begins, tight limits have been set around each of the performance measures. We use four color distinctions where blue is the highest rating and deemed to be excellent. It indicates that the performance measure of interest is within 5% of the target value. Green is the second highest and deemed to be good performance. It indicates that the measure of interest deviates more than 5% but remains within 15% of the target value. Yellow is marginal performance and indicates that the measure of interest deviates more than 15% but remains within 30% of the target value. Finally, Red indicates poor or unacceptable performance the measure of interest deviates more than 30% from the target value. Other color coding schemes and control limits can be used. The performance limits for the project are shown in Table Using the model and metrics to quantitatively manage the process In this section, the model is used to predict project performance for each performance measure (1) prior to beginning the project and (2) after the preliminary design phase has been completed. The model also predicts the risk or variability associated with the expected values (as measured by the standard deviation). The predicted distributions are compared to the OBCLs to determine the probability that the project will perform within the outcome based control limits. The metrics repository is used to provide timely updates to model parameters. The updated model parameters are used to reassess expected project performance at each successive project milestone Determining the performance of the baseline process After the outcome based control limits are set (Table 2), the next step is to evaluate the baseline performance of the project. To do this, we evaluate the estimated project performance in light of the OBCLs. As can be seen in Fig. 1 and Table 3, prior to project start, predictions from the model indicate that the project has a 99.9% chance of achieving a blue or outstanding rating for effort or cost performance; a 74.1% chance of achieving a blue rating for task duration or schedule performance; and a 70.7% chance of achieving a blue rating for target defect level or quality performance. Furthermore, we are able to see that the model predicts that the project has a 99.9% chance of achieving a blue or green rating for effort or cost performance; a 97.4% chance of achieving a blue or green rating for task duration or schedule performance; and a 99.8% chance of achieving a blue or green rating for target defect level or quality performance. (Calculations for the numbers presented in Table 3 are shown in Appendix A.) The blue rating is deemed to be outstanding. Developers and project managers receive a bonus for the project performing to this level. Achieving a level of green or blue is considered to be good performance by the client. For this project, management decided that it would be acceptable if all three performance measures had a 90% or better chance of being within 15% of the target (i.e. achieving a blue or green level of performance). Fig. 1. Graphs showing bar and whisker charts of each performance measures with the appropriate OBCL.

6 1014 D.M. Raffo / Information and Software Technology 47 (2005) Table 3 Probability of achieving blue/green performance for the baseline process Prob. of blue rating a If the baseline process had not obtained a high enough level of certainty for achieving the desired outcomes, then changes would need to be made to create a process or assign staff that could enable the desired targets to be attained. As a result, the OBCL approach can be used for up-front planning as well Coordinating the model and metrics to assess project trajectory Prob. of blue/ green rating b Total effort (cost) in person months 99.9% 99.9% Project duration (schedule) months c 74.1% 97.4% Remaining defects (quality) 70.7% 99.8% a Probability performance is within 5% of target. b Probability performance is within 15% of target. c Since, there are multiple CSCIs being modified of various durations, the chance of the schedule reducing below the limits was assessed to be negligible. A flexible, extensible repository that can answer a number of ad hoc queries is essential to this effort. The repository provides the critical link between raw project metrics and model parameters. Since, the simulation model needs to provide a timely view of the project at all times, the repository must facilitate the collection of data on a real-time basis. The repository that was used [7] is based upon a transformation view of the software development process that allows flexibility in both information storage as well as the types of queries it can accommodate. The transformation model considers artifacts such as specifications, designs and code to be transformed by the application of a transformation process into a new artifact. For instance, a design artifact may be transformed into a code artifact by the application of a programming transformation. Continuing with the example, as the project begins to execute, the repository is automatically updated. For instance, if CSCI a is modified in response to an enhancement request, this event is reflected in the repository by adding an entity occurrence for CSCI a 0 (the new version of CSCI a) as well as an enhancements transformation link that associates CSCI a with CSCI a 0 and the effort consumed in the process. At the completion of each major life cycle phase (i.e. requirements, preliminary design, detailed design and so forth), a current snapshot of the project is taken from the metrics repository by aggregating data from all of the current transformations and artifacts as well as links to the defect tracking systems. Updated parameters for the model are generated using the new data. These parameters include: detected defects, estimated size of the project, productivity, effort and duration expended, among others. To incorporate the updated project information into the model, we replace previously estimated parameters in the model with actuals from the lifecycle phases of the project that have been completed. This improves the accuracy of the model and reduces variability of the estimated project outcomes because instead of using stochastic parameters for the phases of the project that have been completed, we now have actual observations. In this example, we observed that instead of expecting 67 defects to be detected during preliminary design, we detected 78 (a 15.5% increase). Upon further investigation, it was found that although the project was staffed by developers with more than five years experience with the firm, over half of them are new to developing internet applications and so the higher than expected defect levels are likely to continue. In a real situation, we expect that differences in productivity, effort expended and other parameters would be observed as well giving further evidence of a problem. For the purposes of this example, we have intentionally limited the observed differences only to increased defects during preliminary design. After investigating the causes of the increased defects, management and the developers decided that they should expect the number of defects to increase during the remainder of the project. As a result, the distribution of defects injected into the model was increased by 10%. The variability of the number of defects injected into the product was increased proportionally as well Assessing implications and developing action plans The predicted outcomes for the parameter changes described in the previous paragraph are shown in Table 4. Depending upon how close the predicted outcomes are to the target, management will decide if corrective action needs to be taken in order to bring the project back into a satisfactory performance range. For this example, target values are shown in Table 4 along with predicted performance levels obtained from the model based upon updated parameter values. The probability of achieving blue or green performance for all performance measures is also shown. As can be seen, the new performance levels are not acceptable given management s goal of having at least a 90% probability of achieving the blue or green performance for all performance measures. Consequently, action will need to be taken. 7. Assessing alternatives using multi-attribute utility functions Once it is decided that action needs to be taken, a variety of potential process changes can be explored using the simulation model. For this example, some possible corrective actions would be bringing on expert software engineers to help with development; providing additional testing; or other options. The results of these corrective actions are compared to the OBCLs to determine if one or more of the alternatives enables the project to achieve the desired level of performance. If multiple alternatives achieve the level of desired performance, then a tradeoff must be made in order to select the best alternative. At this point, additional factors must be taken into account such as implementation costs as well as staffing and schedule constraints. A multi-criteria utility function can be used to tradeoff among the variety of performance measures

7 D.M. Raffo / Information and Software Technology 47 (2005) Table 4 Project performance using observed defect levels for requirements and preliminary design Target values Model predictions Prob. of blue rating a Prob. of blue/green rating b Mean STD Total effort (cost) in % 99.9% person months Project duration % 69.8% (schedule) in months c Remaining defects (quality) % 71.7% a Probability performance is within 5% of target. b Probability performance is within 15% of target. c Since, there are multiple CSCIs being modified of various durations, the chance of the schedule reducing below the limits was assessed to be negligible. that may be in force. For a good reference on multi-criteria utility functions, see Ref. [8]. Care must be taken in developing the utility function. Changing stakeholder interests and preferences can mean that different choices will be made under different circumstances. For this project, we developed a three-tiered linear utility function that made different tradeoffs among the main performance measures of cost, quality and schedule depending upon the magnitude of difference between the process configuration under consideration and the baseline. Moreover, it was observed that management s preferences changed depending upon the point at which the project was. For instance, managers expressed different tradeoffs between effort and quality at the beginning of the project compared to just prior to release. Since, management opinion may change frequently, it is highly desirable to find a utility function whose weights fluctuate less and whose values are widely accepted and understood within an organization. Financial measures of performance such as net present value (NPV), return on investment (ROI) and payback period (PB) are special cases of general multi-attribute utility functions and are highly desirable to use if the measures of performance are amenable. Even if some measures of performance are not amenable, sometimes a financial measure of performance can be applied to two or more of the performance measures under consideration. This can simplify the decision in two respects: (1) Combining two or more performance measures can provide a partial solution that substantially simplifies the problem, and (2) It can be easier for managers to tradeoff the remaining performance measure(s) against a monetary amount than an abstract number (i.e. utils) In this example, effort, implementation costs and remaining defects (which can be reduced to effort) can be converted into meaningful cash equivalents, thus reducing four dimensions down to two. Moreover, decision makers are left with a tradeoff that they clearly understand cost vs schedule. At this point, management selects and implements the best corrective action alternative (least costly, highest benefit) and monitors the process to see that improvement is being made. The next project snapshot will be taken at the end of the detailed design phase. At this point, model parameters will again be updated with actual observations from detailed design. The feedback cycle would then be repeated to see if the project was performing to the desired level. 8. Conclusions Software development is a challenging and complex endeavor. When developing a product, many decisions need to be made while managing the development of a project. In this paper, we present an application of the PROject Management Process Tradeoff (PROMPT) method. The metrics are collected using a flexible metrics repository. The process is modeled at a detailed level using discrete event simulation and evaluations are made using outcome based control limits, multi-criteria utility functions as well as financial measures of performance. The modeling and metrics framework has been applied at a leading software development firm. The flexible metrics repository provides feedback that is used to generate updated simulation model parameters at predefined project milestones. Model predictions using updated parameters and current project data are compared to outcome based control limits (OBCLs) defined for the project. If the expected performance is outside of the OBCLs, management can be alerted to a potential problem and take corrective action. This is the goal of methods that are based on statistical process control (SPC) but it cannot be achieved using individual metrics alone. The predictive model is then used to evaluate the outcomes of potential management decisions to bring the project back in control and help the project meet its pre-determined goals. This approach directly supports the ability to quantitatively monitor and assess software projects. As a result, this approach supports the quantitative project management (QPM), and organizational process performance (OPP) Level 4 process areas (PAs) of the capability maturity model integrated (CMMI) [4]. This also addresses organizational innovation and deployment (OID) and causal analysis and resolution (CAR) Level 5 PAs of the CMMI. Acknowledgements This work has benefited from modeling support and participation from Siri-on Setamanit. Her contributions are

8 1016 D.M. Raffo / Information and Software Technology 47 (2005) gratefully acknowledged. In addition, the authors are grateful to the Software Engineering Research Center, a National Science Foundation Industry/University Collaborative Research Center and Northrop Grumman Corporation for supporting this research effort. Appendix A. Example for calculating the probability of achieving OBCL limits To compute the probability of achieving an outcome based control limit, we use the distribution of the selected performance measure. We denote the mean and standard deviation of this distribution as m and s. We denote the upper and lower outcome based control limits as UCL and LCL. We compute the following: PROB Z N m;s ðuclþkn m;s ðlclþ Where: (A1) PROB is the probability that the observed value of the selected performance measure will be within the OBCLs Z U is the number of standard deviations the UCL is away from the mean of the performance measure Z L is the number of standard deviations the LCL is away from the mean of the performance measure N m,s represents the area under the standard normal curve, normalized using m and s,respectively. This standardization can be done using Eqs. (A2) and (A3). This yields Eq. (A4), which would be used in conjunction with a standard normal look-up table. Z U Z UCLKm (A2) s Z L Z mklcl s N 0;1 ðz U ÞKN 0;1 ðz L Þ Where: (A3) (A4) Z U and Z L are defined the same as above N 0,1 represents the area under the standard normal curve from KN to Z. Once the Z scores are determined we use a table that maps Z scores to normal distribution probabilities. For instance: From Table 1, let the distribution for the number of remaining defects (NRDs) m Z 77:4 defects s Z 3:68 From Table 2, we get that the upper and lower OBCLs for blue performance are: UCL Z 81:3 LCL Z 73:5 Then, Z U Z 81:3K77:4 Z 1:059 3:68 Z L Z 77:4K73:5 3:68 Z 1:059 PROB Z Nð1:059Þ CNð1:059Þ Z 35:35% C35:35% Z 70:7% as shown in row 3 of Table 3. The method presented in this appendix assumes: (1) Performance measure predictions are normally distributed. This assumption usually holds when using discrete event simulation because the simulation results are based upon the summation of many random variables. (2) The predicted mean of the performance measure is within the OBCLs. This assumption should hold. If it does not, then the probability of the project achieving the OBCLs is less that 50%. In the interest of space, we refer the reader to Summers [20] for further details regarding how to calculate these probabilities. References [1] T. Abdel-Hamid, S. Madnick, Software Project Dynamics: An Integrated Approach, Prentice-Hall Software Series, 1991, ISBN: [2] Basili, Lindval, Costa, Implementing the experience factory concepts as a set of experience bases, Proceedings of the International Conference on Software Engineering and Knowledge Engineering, held in Buenos Aires, Argentina, June 13 15, [3] B. Boehm, Software Engineering Economics, Prentice-Hall, 1981, ISBN: [4] CMMI Product Team, CMMI SM for Software Engineering, Version 1.1, Staged Representation (CMMI-SW, V1.1, Staged), Software Engineering Institute, Carnegie Mellon University, 2002, cmmi/cmmi.html [5] Edwards, Deming, Out of the Crisis, MIT Press, Cambridge, MA, [6] Extend Simulation Software, Imagine That, Inc., [7] W. Harrison., A universal metrics repository, Proceedings of the Pacific Northwest Software Quality Conference, Portland, Oregon, October 18 19, [8] R.L. Keeney, H. Raiffa, Decisions with Multiple Objectives: Preferences and Value Tradeoffs, Wiley, New York, [9] Kellner M., software process modeling: value and experience, SEI Technical Review, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA, [10] M. Kellner., Software process modeling experience, presented at The 11th International Conference on Software Engineering, Pittsburgh, Pennsylvania, USA, [11] M. I. Kellner., Experience with enactable software process models, Presented at The Fifth International Conference on The Software Process, Kennebunkport, Maine, USA, 1990.

9 D.M. Raffo / Information and Software Technology 47 (2005) [12] M.I. Kellner, R.J. Madachy, D.M. Raffo, Software process simulation modeling: What? Why? How?, Journal of Systems and Software 46 (1999) [13] D.M. Raffo, Modeling Software Processes Quantitatively and Assessing the Impact of Potential Process Changes on Process Performance, Graduate School of Industrial Administration, Carnegie Mellon University, Pittsburgh, PA, UMI Dissertation Services # [14] D. M. Raffo, M.I. Kellner., Field study results using the process tradeoff analysis method, Proceedings of the 1996 Software Engineering Process Group Conference, Held in Atlantic City, New Jersey, May 20 23, [15] D.M. Raffo, J.V. Vandeville, R. Martin, Software process simulation to achieve higher CMM levels, Journal of Systems and Software 46 (2/3) (1999) 15 April. [16] D.M. Raffo, W. Harrison, J.V. Vandeville, Coordinating models and metrics to manage software projects, International Journal of Software Process Improvement and Practice 5 (2/3) (2000) [17] D.M. Raffo, M.I. Kellner, Empirical analysis in software process simulation modeling, Journal of Systems and Software 47 (9) (2000) [18] D.C.S. Summers, Quality, second ed., Prentice-Hall, Dr David Raffo is an Associate Professor in the School of Business Administration and the Maseeh College of Engineering and Computer Science at Portland State University. Raffo is also a Visiting Scientist at the Software Engineering Institute at Carnegie Mellon University and is also Editor-in-Chief of the international journal of Softwhere Process: Improvement and Practice Raffo s research interests include: software process modeling and simulation, strategic software engineering, and process improvement. He has over thirty-five refereed publications in the field of software engineering and has received research grants from the National Science Foundation, the Software Engineering Research Center (SERC), NASA, IBM, Tektronix, Bosch and Northrop-Grumman. Prior professional experience includes programming as well as managing software development projects. He has also taught in Purdue University s Software Engineering Retraining Program (SERT) through the Department of Computer Science. Dr Raffo received his PhD at Carnegie Mellon University.