Risk Analysis: a Key Success Factor for Complex System Development

Risk Analysis: a Key Success Factor for Complex System Development MÁRCIO DE O. BARROS CLÁUDIA M. L. WERNER GUILHERME H. TRAVASSOS COPPE / UFRJ Computer Science Department Caixa Postal: 68511 - CEP 21945-970 - Rio de Janeiro RJ Fax / Voice: 5521 590-2552 {marcio, werner, ght}@cos.ufrj.br Abstract Complex software development is a risky job. The number of unsuccessful projects largely surpasses the number of successful developments. Many studies relate this situation to non-technical problems, especially to inadequate project management. Risk analysis is a management technique that has shown a high correlation to successful projects. This paper describes the software risk management approach of a reuse infrastructure based on domain models, named Odyssey. Within this process, risks are identified and documented by risk patterns, a standard information structure that describes a potential recurring problem to the software development process or product. A risk pattern includes context information upon the forces that enable or inhibit the problem in a software project, presenting standard solutions to solve the problem or to minimize the effects of its occurrence. Dynamic models of the software development process are used to support risk analysis and evaluation. These models use mathematical and logical formulations to integrate and formally describe the policies governing the software development process. Dynamic models are also used for decision analysis, project control and risk identification. Keywords: Risk management, project management, software process modeling

1 Motivation The state of practice and contemporary literature prove the persistence of a recurrent problem in software development: most projects use more resources than planned, take more time to be concluded, have less functionality and less quality than expected. Software unsuccessful stories can be found in several case studies and experiments documented over the last decades [14] [9] [7]. Two different philosophical paradigms relate factors responsible for the continuing software faults in industry. The first one relates software development problems to technological problems, augmented by the growing complexity of current software projects [2]. The second, transfers the responsibility to management problems, bad communication and difficulties in handling the uncertainties that are present in innovative, complex or large software projects [5]. The techniques currently applied to software project management are based on a set of fundamental assumptions, necessary for their accuracy [7]. Some of these assume that projects have clear and delimited objectives in order to be viable 1, development time and resources can be precisely stated before the project start, the operational environment is well known and defined, and that measures of project success can be quantified. Due to many successes that were achieved by using these techniques in large projects, project managers tend to passively take their underpinning assumptions for granted in every project [7]. This misconception is pretty common in software project management. The development of projects for innovative application domains with high volatility of requirements, the need for domain integration to achieve several objectives, ambiguity, complexity, discontinuity, diseconomy of scale, nonlinearities, and complicated feedback loops are characteristics of complex projects. These characteristics undermine the assumptions of traditional techniques. The invalidation of these assumptions, and consequently of these techniques, creates a demand for new technology and new management paradigms. Risk management is one of these paradigms. In the software development context, risk is defined as the probability of a project failing to achieve particular cost, performance and schedule objectives, and the consequence of failing to achieve these objectives [8]. A risk management process transforms the set of existing uncertainties and doubts in a software development project to acceptable risks, according to the existing knowledge about the project. Risk management techniques were introduced in the software development context by the end of the 1980 s, with the definition of the spiral model [3]. The work on software risk management has been directed, since the last decade, to distinct directions: new qualitative risk management processes [17] [18] [16], quantitative processes [10] [15], knowledge based processes [19], and risk information databases [6] [13]. This paper focus on risk management as the central element of the project management paradigm of a reuse oriented software development infrastructure based on domain models, named Odyssey [4]. We have developed a risk management process, which is organized into two sub-processes, and conveys domain risk analysis and reuse of risk information in projects developed within specific domains. The process focuses on qualitative risk identification and quantitative risk impact and resolution strategies evaluation. In this paper, we briefly describe the process for the development of domain models. This paper is organized on five sections. The first one is this motivation. Section 2 presents the roles of risk management in the Odyssey infrastructure. Section 3 presents an overview of the risk management process for the development of domain models. Section 4 presents some related works. Section 5 presents the conclusions of this paper. 1 That is, at least one solution to the problem at hand exists.

2 Risk Management in the Odyssey Infrastructure Odyssey is a reuse oriented software development infrastructure based on the development of reusable domain models and components, through a domain engineering 2 process. The main objective of Odyssey s domain engineering process is the development of reusable components for a particular application domain, attending the needs of application developers for that domain at every required abstraction level [4]. Within the Odyssey infrastructure, risk is defined as the possibility of an event to occur causing prejudices to the software development process. Odyssey has one risk management process, divided into two risk sub-processes: one for domain risk management, other for application risk management. The main objective of the risk management process for domain models is to identify the most common risks that occur in projects developed within a specific application domain, organizing this information and allowing its reuse in risk management processes for specific software projects. This process has its origin at the development of the domain model, and is reviewed along the development of specific applications for the domain. The main objective of the risk management process for application development is to identify the risks of a project developed within a specific domain, reusing the domain risks and defining risks that are specific to the project. This process occurs in parallel with the development life cycle of an application, monitoring the evolution of the identified risks for that project. Odyssey risk management processes integrate qualitative and quantitative risk analysis. Risks are documented by risk patterns (see section 2.1). These risk patterns convey information for qualitative risk identification and dynamic models of the risk and resolution strategies impact. These models are aggregated with a dynamic model of the development process 3, allowing the project manager to quantitatively evaluate the risks and to monitor the project evolution. Section 2.2 describes the use of dynamic models within the Odyssey infrastructure. 2.1 Risk Patterns A risk pattern is a description of a potential recurring problem, which can cause loss to a software product or development process. It includes a description of the context where the problem can occur and provides solution strategies that can be applied before and after the problem occurrence. The definition of a risk pattern is based on the concept of a design pattern, as defined by Gamma et al. [12]. A design pattern describes an object oriented structure, composed of interacting objects and classes, that can be used as a solution to a generic problem in a software project. Each design pattern identifies, documents and evaluates a design problem recurrent along a series of projects. The design pattern captures the experience of developing the solution, allowing this experience to be reused by other projects facing the same problem. Like design patterns, risk patterns identify, document and evaluate potential events that can negatively affect software development projects. The experience of forecasting, identifying and resolving the risk is documented by the risk pattern, which can be effectively reused in future projects. Risk patterns determine the information that must be organized with the potential problem description, presenting its characteristics in a structured and standard form. The use of a standard representation promotes risk communication, understanding, and a clear distinction among several risk events. The structured form allows consistent risk documentation, so that a risk pattern conveys all the information needed for future risk analysis. Information completeness is very important to reduce psychological 2 Domain Engineering is a set of activities that allows the creation of abstract models (i.e., domain models) that can be reused in the development of a family of applications. 3 A software development process model formalizes and integrates the laws and procedures that govern software development through a set of mathematical and logical formulations.

barriers to risk analysis: risk is an abstract concept [18] and an incomplete description of the risks to which a project is subjected inhibits the motivation for risk management. Examples of risk patterns are schedule and cost overruns, the need for rework activities, quality and functionality losses, unplanned development tasks, requirement changes, validation and verification problems. The standard structure that represents a risk pattern is composed of five blocks, described on the following paragraphs. Pattern identification: this block describes the potential problem represented by the risk pattern. The pattern name, a list of aliases, a textual description of the potential problem and its effects, and a dynamical model of the risk impact compose this block. The dynamic model, which is called risk impact model, is aggregated to project models, integrating the risk impact with other project variables. This integration will be described in section 2.2; Identification mechanisms: this block describes the mechanisms that are used to identify this risk in a particular software project. A risk occurrence context textual description, a risk identification checklist, and a known occurrence cases list compose this block. The checklist conveys several questions, presented to the project manager during risk identification assessment for a specific project. The answers to the checklist will drive risk partial ordering, risk prioritization, and the selection of contention strategies. The known occurrence cases list provides references to projects where the risk pattern has been previously identified. During risk identification, the project manager can navigate through these references, analyzing the previous occurrences of a particular risk. Contention Strategies: this block describes possible resolution strategies to inhibit or eliminate the potential problem described by the risk pattern before the risk occurs in a project. It is composed by a textual description of the contention plan and its effects, the conditions of its application, and a dynamic model of its impact. As it happens with the risk impact model, the contention plan impact model will be integrated and evaluated within the process model. The contention plan applicability conditions are based on specific answers to the identification checklist, since its questions refer to the basic uncertainties underpinning the risk that will be attacked by the contention plan; Contingency Strategies: this block describes possible resolution strategies to reduce the impact of the potential problem described by the risk pattern after its occurrence in a project. A textual description of the contingency plan and its effects, and a dynamic model of its impact compose this block. Again, the contingency plan impact model will be integrated and evaluated within the process model; Related Patterns: this block describes risk patterns that occur in similar situations or risk patterns that can replace the current risk pattern. A list of induced risks and a list of similar risks compose this block. Every time a risk pattern is identified in a project, its induced patterns are automatically identified for that project. The similar risk patterns are a list of references to other risk patterns that document similar problems to the one described by the current risk pattern. During risk identification, the project manager can navigate through similar risk patterns, determining the risk that best fits the project characteristics. It is expected that the creation of a new risk pattern enriches the development team vocabulary. Like it occurs with design patterns, the name of the risk pattern is used as an alias to the recurrent problem represented by it. The citation of the risk pattern name tends to be associated to its full description, bringing forth all the information conveyed by the pattern. The extension of the team vocabulary, with terms referring to risk identification, promotes the risk culture in the software development organization. 2.2 Dynamic Models of Software Development Processes A formal model describes the relationships among real world elements through mathematical or logical postulations. A software development process model formalizes and

integrates the laws and procedures that govern software development through a set of mathematical or logical formulations. System dynamics [11] is a modeling technique and language to describe complex systems, focusing on these systems structural aspects. This technique identifies and models cause-effect relationships and feedback loops. Abdel-Hamid and Madnick model [1] is a system dynamic model that formalizes the effects of management policies and actions taken during a software development project. The model is divided into four major sections, which cover human resource management, software production, project planning, and control. Odyssey uses process models to track project evolution, and to identify, evaluate and monitor risks, through scenario analysis and simulation. Odyssey process models are based on Abdel-Hamid and Madnick model [1]. This model was modified to allow the representation of the uncertainties associated with its parameters. This feature allows the accomplishment of Monte Carlo simulation [20] and semi-automatic risk scenarios identification, through sensitivity analysis. Within Odyssey risk management processes, risk and resolution strategies impact models are integrated with the software development process model. This integration allows the quantification of risk impact upon several process variables, such as cost, schedule, staff productivity and size. The impact of several risks and resolution strategies can be combined and integrated with the process model, allowing the project manager to design complex scenarios involving risk occurrences and resolution plans. Nonlinear relations among risk impact and the variables describing the development process, coupled with the effects generated by the feedback loops, are explicitly represented in the model and analyzed using simulations. These simulations numerically evaluate selected resolution strategies, presenting their advantages and problems. Scenario analysis and stress testing allow, respectively, top-down and bottom-up risk analysis. The first allows the definition of a scenario, the analysis of the scenario impact on model parameters and the influence of the parameter changes on model results. The later specifies scenarios directly based on model parameters, allowing an analysis of these parameters impact on model results. These model results are expressed as probability distributions functions, reflecting the uncertainties described in the model. Other extensions were proposed to the original Abdel-Hamid and Madnick model [1] to add and enhance the following characteristics: Extendibility: models must be extensible, allowing the specialization of their parameters and the relation among their variables for each domain. Model interfaces must be clearly defined, so that a large model can be developed from the connection of smaller models. This feature is necessary to allow the aggregation of risk and resolution strategies impact models within a major development process model; Readability: the laws expressed by the model must be easily readable from that model. This feature promotes model fundamentals learning, enhancing its extension and training capabilities; Decomposition: management models must handle a large number of variables. To enhance model readability, it must be decomposable into minor and less complex models, handling a much lower number of variables that can be studied in isolation. This characteristic improves model understandability, model modification and domain specialization of models. 3 Risk Management in Domain Models Development An application domain contains risks that can occur in several projects developed within the domain. For instance, some domains are subjected to higher requirement volatility, strict performance or reliability goals. Moreover, those restrictions can be found in specific concepts, subsystems or application categories.

Development teams should reuse knowledge of risks that occur in applications developed for a domain. Risk reuse has been previously reported [18] [13]. The formalization of these risks, during domain modeling, reduces the effort spent by every project in risk identification to find the same doubts and uncertainties. The formalization of these risks also prevents them from passing unnoticed through risk identification activities. The objectives of a risk management process for domain model development are the identification of software risks that can occur in applications developed within the domain, specification of the conditions when those risks are relevant to a particular project, definition of mechanisms for risk identification in projects, and guidelines for contention and contingency plans. This process aims to: Prevent relevant risks from passing unnoticed, annotating the risks that can occur in several projects developed within the domain prior to a project risk identification activity; Enhance risk management staff productivity, through domain specific risk reuse in the development of new applications; Support the preparation of risk management sessions, where domain risks serve as a basis for the discussion of a project risks and critical points; Breaking risk analysis paralysis, that can happen during risk management activities for a project, when there are no guidelines to start the identification process. The risk management process for domain model development defines these guidelines and offers a list of domain related potential problems. The risk management process for domain model development is composed by the following activities: risk pattern identification, risk pattern checklist identification, contention plans formulation, contingency plans formulation, identification of induced risks, and process model specialization. The first five activities produce domain specific risk patterns. The process model specialization activity refines a generic process model, aggregating domain specific information and developing a domain process model, which will be used by applications developed within this domain. Given the incremental and iterative nature of the risk management process, its activities are modeled as the spiral software life cycle. The risk management spiral is orthogonal to the domain engineering process spiral. For each domain engineering iteration, several iterations of the risk management spiral occur. The spiral radius reduces gradually along the domain engineering iterations, reflecting the reduction of the number of risks to be described. The risk management process for application development reuses the risk patterns that are created during the risk management process for domain model development. This application oriented process uses the risk patterns checklist to identify the risks that can occur in the project, qualitatively ordering these risks by their occurrence probabilities. The domain process model is specialized for the project, based on specific project information, such as initial staff, network of tasks, expected conclusion date, and so on. 4 Related Works Odyssey s approach to risk management has several innovations when compared to software risk management processes described in the literature [6] [10] [15] [17] [18] [8] [16]. First, risk patterns offer a structured way to document risk information. This structured documentation helps to break psychological barriers to risk analysis, due to incomplete information about risks. This also contributes to create a risk vocabulary, relating the pattern names to the whole information presented in the risk pattern. The vocabulary promotes discussion about project risks, helping to create a risk culture in the software development organization. Garvey [13] presents a standard information structure to document risks, but this information structure does not allow risk impact simulation within a

specific project. Hall [16] and Conrow [8] do not define a strict risk information structure in their processes. Although, they propose several risk forms or databases to store static risk information. This information does not allow an integrated analysis of risk impact. Risk patterns also allow navigation through projects that have faced the same problem before, helping the risk manager to evaluate the efficacy and the impact of resolution strategies previously applied. Induced risks capture risk coupling. Navigation through induced risks helps the risk manager to understand the impact that risks create upon other risks. Garvey s [13] risk information structure allows navigation among similar risks and through projects that had faced some risks. However, there is no information about risk coupling, which we capture in the induced risk list. Next, the use of dynamic models of software development process allows the risk manager to investigate his/her project, searching for risk and opportunities. The model can be studied via simulations, scenario analysis and parameter sensitivity. Project status can be reflected in the model, allowing the manager to evaluate risks throughout the life cycle. Grey s [15] risk evaluation model, based on PERT diagrams, allows the analysis of cost and schedule risks, but the model does not consider other variables than the time to conclude each project task and the topological order of those tasks. Odyssey s system dynamic models consider aspects such as team exhaustion, communication overhead, staff size, testing approach, soft variables, and feedback loops, among others. Fairley s [10] approach applies Monte Carlo simulation to a small set of project variables, organized in a simple model, that also does not capture the above dynamics. Risk impact, contention and contingency plan effects can be aggregated to process models so that they can be evaluated. The evaluation captures the nonlinear impact of the risk and plan over the process. For instance, consider a risk whose outcome is to reduce a project team for a predefined amount of developers. The effect of this team reduction can be directly evaluated on the process model, capturing all the relationships with the project current status and with other model variables. Kontio [18] quantifies the effect of risk scenarios by using utility functions. These utility functions become very complicated when several process aspects are taken into analysis, due to the nonlinear relationships among those aspects and feedback loops inherent to the software development process. Odyssey s approach of simulating risk impact models, coupled with process models, offers a better way to understand the cascade of impacts imposed by the risk. Risks are identified through a checklist, as in [6] and [17]. The probability of this risk is qualitatively evaluated, as in [18], while its impact is quantitatively evaluated within the process model. The qualitative method, together with the risk documentation, helps the identification and prioritization of risks. The quantitative impact models promote risk aggregation and process-variable oriented impact assessment. 5 Conclusions This paper described a software risk management process defined to be the central element of the project management paradigm of a reuse infrastructure based on domain models, named Odyssey. The process, which is organized in two sub-processes, conveys domain risk analysis and reuse of risk information in projects developed within specific domains. The main contributions of this paper are the integration of risk management and domain engineering, the definition of a standard risk information structure that captures all risk information, including risk coupling, allowing navigation through similar risks and projects that had faced the same risks before, and using dynamic models to evaluate risk and resolution strategies impact.

Acknowledgements The authors would like to thank CNPq and CAPES, for their financial investment in this work. References [1] Abdel-Hamid, T.; Madnick, S.E., Software Project Dynamics: an Integrated Approach, Englewood Cliffs, New Jersey: Prentice-Hall Software Series, 1991 [2] Augustine, N.R., Augustine s Laws, American Institute Of Aeronautics and Astronautics, New York, 1982 [3] Boehm, B.W., A Spiral Model of Software Development and Enhancement, IEEE Computer, May 1988 [4] Braga, R.M.; Werner, C.M.L.; Mattoso, M., "Odyssey: A Reuse Environment Based on Domain Models"; 2 nd IEEE Symposium on Application Specific Systems and Software Engineering Technology, ASSET 99, Richardson, USA, March 1999. [5] Brown, N., Industrial-Strength Management Strategies, IEEE Software, July 1996 [6] Carr, M.J. et al., Taxonomy-Based Risk Identification, SEI Technical Report, 93-TR- 006, Software Engineering Institute, Pittsburgh, 1993 [7] Charette, Robert N., Large-scale Project Management is Risk Management, IEEE Software, July 1996 [8] Conrow, E.H.; Shishido, P.S. Implementing Risk Management on Software Intensive Projects, IEEE Software, May/June 1997 [9] Davis, A.M., Software Requirements Analysis & Specification, Englewood Cliffs, New Jersey: Prentice-Hall Software Series, 1990 [10] Fairley, R., Risk Management for Software Projects, IEEE Software, May 1994 [11] Forrester, J.W., Industrial Dynamics, Cambridge, MA: The M.I.T. Press, 1961 [12] Gamma, E. et al., Design Patterns: Elements of Reusable Object-Oriented Software, Addison-Wesley Publishing Company, 1994 [13] Garvey, P.R., Phair, D.J.; Wilson, J.A., An Information Architecture for Risk Assessment and Management, IEEE Software, May/June 1997 [14] Gilb, T., Principles of Software Engineering Management, Addison Wesley Publishing Company, 1988 [15] Grey, S., Practical Risk Assessment for Project Management, Wiley Series in Software Engineering Practice, New York, NY: John Wiley & Sons Inc, 1995 [16] Hall, E.M., Managing Risk: Methods for Software Systems Development, SEI Series in Software Engineering, Addison Wesley Longman Inc., 1998 [17] Karolak, D.W., Software Engineering Risk Management, IEEE Computer Society Press, 1996 [18] Kontio, J., Basili, V.R., Risk Knowledge Capture in the Riskit Method, SEW Proceedings, SEL-96-002, University of Maryland, 1996 [19] Madachy, R.J., Heuristic Risk Assessment Using Cost Factors, IEEE Software, May/June 1997 [20] Vose, D., Quantitative Risk Analysis: A Guide to Monte Carlo Simulation Modeling, New York, NY: John Wiley & Sons, Inc, 1996