Knowledge-Based Systems Engineering Risk Assessment Raymond Madachy, Ricardo Valerdi University of Southern California - Center for Systems and Software Engineering Massachusetts Institute of Technology Systems Engineering Advancement Research Initiative madachy@usc.edu, rvalerdi@mit.edu Abstract. A knowledge-based method for systems engineering risk assessment has been automated in an expert system tool. Expert COSYSMO performs systems engineering risk assessment in conjunction with cost estimation using the Constructive Systems Engineering Cost Model (COSYSMO). The technique is an extension of COSYSMO which supports project planning by identifying, categorizing, quantifying, and prioritizing system-level risks. Workshops and surveys with seasoned systems engineering practitioners are used to identify and quantify risks, and the expert assessment has been implemented in an Internet-based tool. The tool is being refined for sustained usage on projects by providing risk control advice, updating the rule base and being integrated into a more comprehensive risk management framework. Introduction Approaches for identifying systems engineering risks are usually separate from cost estimation. However, risk management practice can be improved by leveraging on existing knowledge and expertise during cost estimation activities through the use of cost factors to detect patterns of project risk. For this, we have implemented an automated technique that identifies system engineering risks in conjunction with cost estimation. This information helps users determine and rank associated sources of project risk for mitigation plans. Expert COSYSMO is an expert system tool for systems engineering risk assessment that uses factors in the COSYSMO cost model (Valerdi 2005). It runs on the Internet at http://csse.usc.edu/tools/expertcosysmo.php. The tool automatically identifies project risks in conjunction with cost estimation similar to Expert COCOMO (Madachy 1997). It currently covers 98 risk conditions which are being further discretized into about 600 finer-level risk conditions using the same inputs. The usage of Expert COSYSMO supports project planning by identifying, categorizing, quantifying, and prioritizing system-level risks. Risk situations are characterized by combinations of cost driver values indicating increased effort with a potential for more problems. It simultaneously calculates cost to enable tradeoffs with risk, as it analyzes patterns of cost driver ratings submitted for a COSYSMO (Valerdi 2005) cost estimate. In practice, risks must be identified as specific instances to be manageable. The method identifies individual risks that an experienced systems engineering manager might recognize but often fails to take into account. It also helps calibrate and rank collections of risks, a process which many managers wouldn t do otherwise. This information is then used to develop and execute project risk management plans. With these risks, mitigation plans can be created based on the relative risk severities and provided advice. Cost estimation and risk management are strongly connected since cost estimates are used to
evaluate risk and perform risk trade-offs; risk methods such as Monte Carlo simulation can be applied to cost models; and the likelihood of meeting cost estimates depends on risk management (Madachy 1997). The same cost inputs can also be used to assess risk using sensitivity analysis or Monte Carlo simulation such as the COSYSMO-R approach (Valerdi, Gaffney 2007), but the approach described here uses them to infer specific risk situations. Expert COSYSMO has been developed with collaboration between the USC Center for Systems and Software Engineering (CSSE), its industrial affiliates and MIT. It has been specifically supported with focused COSYSMO workshops conducted with systems engineering practitioners from the CSSE affiliates. These seasoned professionals serve as experts providing a collective knowledge base for the method. Method The tool analyzes patterns of cost driver ratings submitted for a COSYSMO cost estimate against pre-determined risk rules. COSYSMO predicts the systems engineering effort per the following equation, where PM stands for Person-Months (PM) of effort. 14 PM NS = A ( we, kφ e, k + wn, kφ n, k + wd, kφ d, k ) EM j k j= 1 Where: PM NS = effort in Person Months (Nominal Schedule) A = calibration constant derived from historical project data k = {REQ, IF, ALG, SCN} w x = weight for easy, nominal, or difficult size driver Φ x, k = quantity of k size driver at weight x E = represents diseconomy of scale (currently equals 1) EM = effort multiplier for the jth cost driver. The geometric product results in an overall effort adjustment factor to the nominal effort. The cost drivers represented in the effort multiplier term EM are used for expressing the risk rules. Predetermined combinations of driver ratings provide red flags of possible risks as the project progresses along its life cycle. For example, if the architecture understanding cost driver is rated Very Low and the level of service requirements is Very High then this indicates a potential risk in the project given that systems with high service requirements are more difficult to implement especially when the architecture is not well understood. These scenarios are predetermined and configured into the model as a set of rules that can automate and improve the risk management process. This method is derived from the Expert COCOMO model for heuristic risk assessment with cost factors (Madachy 1997). The knowledge representation scheme and risk quantification algorithm are similar, however a larger set of experts has been invoked for the knowledge base. Elicitation of knowledge came from systems engineering domain experts in CSSE-sponsored workshops. A survey was used to identify and quantify risks. Figure 1 shows these risk conditions identified from the iterated workshop survey as implemented in the model. The matrix shows the interactions of the cost factors, and the corresponding risk conditions classified into high, medium and low risks per the inputs of 19 systems engineering experts familiar with the cost estimation concepts in COSYSMO. E
SIZE RQMT ARCH LSVC MIGR TRSK DOCU INST RECU TEAM PCAP PEXP PROC SITE TOOL SIZE (REQ + INTF + ALG + OPSC) 21 21 9 12 5 4 7 10 8 9 11 7 6 7 Requirements Understanding 17 9 7 8 3 5 9 5 10 8 5 4 1 Architecture Understanding 9 10 12 3 7 11 6 11 11 5 6 4 Level of Service Requirements (the ilities) 5 7 4 5 3 6 4 4 2 3 2 Migration Complexity (legacy system considerations) 8 1 10 1 4 7 7 3 5 4 Technology Risk (maturity of technology) 2 8 6 4 9 5 3 3 5 Documentation match to life cycle needs 2 3 4 4 2 6 2 3 Number and Diversity of Installations or Platforms 4 3 5 6 4 8 5 Number of Recursive Levels in the Design 4 8 7 7 2 5 Stakeholder Team Cohesion 7 9 3 8 3 Personnel/team capability 12 9 8 5 Personnel Experience and Continuity 10 8 3 Process Capability 5 8 Multisite Coordination 8 Tool Support high risk small x = 0.5; big X = 1 medium risk n = 19 low risk Figure 1. Risk conditions from survey. The values contained in each cell represent the number of votes obtained by each driver combination. Since participants were not limited on the total number of votes allowed, the values in each cell are absolute indicators of perceived risk. However, their relative values are not informative since respondents were not asked to rank the driver combinations to each other. In order to subdivide the risks into meaningful categories, the cells that received more than 10 votes from the participants were labelled as high risk. Since there were 19 participants, 10 votes indicated that more than half of the respondents felt that these combinations were of significant risk. Subsequently, cells receiving between 5 and 9 votes were identified as medium risk and the rest were low risk. The breakdown of high/medium/low risks in Figure 1 is as follows: High risk items = 10/105 = 9% Medium risk items = 43/105 = 41% Low risk items = 52/105 From a risk management standpoint, this is a reasonable distribution of different risk categories. In other words, the most critical risk items are approximately 10% of the possible driver combinations. The risk taxonomy and risk quantification algorithms are shown in Figure 2. Risk impact, or risk exposure, is defined as the probability of loss multiplied by the cost of the loss. The quantitative risk weighting scheme accounts for the nonlinearity of the assigned risk levels and cost multiplier data to compute overall risks for each category and for the entire project. The risk level corresponds to the nonlinear relative probability of the risk occurring, and the effort multiplier product represents the cost consequence of the risk. The product involves those effort multipliers involved in the risk situation. Each of the risk categories includes rules that include relevant cost factors. For example, process risks would include conditions that involve
the factors for process capability, multi-site coordination and tool support. Project Risk Product risk Process risk Personnel risk Platform risk # categories # category risks Project Risk = risk level i, j * effort mu ltiplier p roduct i, j j = 1 i = 1 where risk level = 1 moderate 2 high 4 very high effort multiplier product= (driver #1 effort multiplier) * (driver #2 effort multiplier)... * (driver #n effort multiplier). Figure 2. Risk taxonomy and weighting. Figure 3 shows the Expert COSYSMO interface. The size and cost driver inputs are provided by the user, and the effort and risk outputs are shown underneath.
Figure 3. Expert COSYSMO interface. The risks as identified in Figure 1 are currently being further elaborated into more detailed conditions requiring no further inputs from the user. Figure 4 shows how the coarse risk conditions are being decomposed into finer grained conditions with more precision. The magnitudes of the coarse risks represented in Figure 1 will be used to set the more detailed ranges. This will multiply the number of risk conditions from 98 into approximately 600.
Figure 4. Assignment of detailed risk levels Current and Future Work The risk levels are being calibrated for usage, and we are making the outputs more actionable so that explicit risk management steps can be undertaken by users. Specific tasks being completed include scaling the risk summary outputs for each category and defining ranges for low, medium and high risks; adding more explanation to the summary outputs to rationalize the risk quantities; and furthering the capability for automated risk mitigation recommendations. We are creating more granular risk quantification rules. The initial risk assessment scheme is being elaborated into a finer grained risk assessment, and the survey is being continued. We are currently documenting expert risk mitigation advice for each risk condition, and providing that automated guidance to users to help develop their own mitigation actions. The researchers and workshop participants have identified opportunity trees for making associations with relevant risk mitigation actions. An example tree from is shown in Figure 5 from (Madachy 2007).
Figure 5. Opportunity tree for risk mitigations. The opportunity trees from (Madachy 2007) are being tailored for systems engineering usage as structures for representing the risk mitigation knowledge. The workshop participants will be involved in scoring the opportunities for specific risks. The tool will also be expanded to detect COSYSMO input anomalies. It will capture inconsistent inputs and flag those to the user. For example, if size indicates a project years
longer than any previously undertaken, requirements understanding is very low, and architecture understanding is rated high then there is a conflict in the inputs. Systems engineering risk data from industrial projects from over 60 projects is being analyzed to enhance and refine the technique. With this the method will be better supported with statistical validation tests. Domain experts from industry and government will continue to provide feedback and clarification. Future work will also involve exploration of alternate risk and uncertainty approaches including COSYSMO-R to integrate multiple risk management viewpoints into a more complete risk management framework. References Madachy R.J., Heuristic Risk Assessment Using Cost Factors, IEEE Software, May 1997. Madachy R.J., Software Process Dynamics, IEEE-Wiley, Hoboken, NJ, 2007. Valerdi, R., The Constructive Systems Engineering Cost Model (COSYSMO), PhD Dissertation, University of Southern California, Los Angeles, CA, May 2005. Valerdi, R., Gaffney, J., Reducing Risk and Uncertainty in COSYSMO Size and Cost Drivers: Some Techniques for Enhancing Accuracy, 5th Conference on Systems Engineering Research, Hoboken, NJ, March 2007. Biographies Raymond Madachy is a Research Assistant Professor in the USC Industrial and Systems Engineering Department and a Principal of the USC Center for Systems and Software Engineering. He is currently serving as Interim Director of the Systems Architecting and Engineering Program. He has 25 years of management and technical experience in industry including Chief Science Officer at the Cost Xpert Group, Chief Scientist at C-bridge Institute, and Manager of the Software Engineering Process Group at Litton Guidance and Control Systems. His research interests include modeling and simulation of processes for architecting and engineering of complex software-intensive systems; economic analysis and value-based engineering of software-intensive systems; systems and software measurement, process improvement, and quality; quantitative methods for systems risk management; integrating systems engineering and software engineering disciplines; and integrating empirical-based research with process simulation. Ricardo Valerdi is a Research Associate in the Lean Aerospace Initiative and a Lecturer in the Engineering Systems Division at MIT. He is also the co-founder of the Systems Engineering Advancement Research Initiative (SEAri) which was launched in 2007, and a Visiting Associate at the Center for Systems & Software Engineering at USC. He previously worked as a systems engineer at Motorola, and has been affiliated with the Aerospace Corporation's Economic and Market Analysis Center as a Member of the Technical Staff since 2003. His current research interests include systems engineering cost estimation, system level metrics and models, dynamics in large-scale government system acquisition, and system-of-systems ontologies.