DIFFERENT TECHNIQUES FOR RISK MANAGEMENT IN SOFTWARE ENGINEERING: A REVIEW

ASAC 2006 Banff, Alberta Subhas C. Misra Vinod Kumar Uma Kumar Eric Sprott School of Business Carleton University DIFFERENT TECHNIQUES FOR RISK MANAGEMENT IN SOFTWARE ENGINEERING: A REVIEW In this article, we introduce the principles of risk management, elaborate on the wellknown risk management approaches, and summarize some of the important concepts of each approach. Almost all of them encompass the basic steps of risk management, i.e., identifying the risks, analyzing them, planning mitigation strategies, and controlling them. Introduction The last few decades especially the end of 20 th century, and the beginning of 21 st century has shown an increase in the interest in automation of different activities. Automation is dependent in its core on sound functional software. The complexity of software development has increased significantly over the years. Articles showing the failure of projects in the software industry are not surprising. Standish Group reports (Standish Group, 1994) show that about 53% of projects get completed, but they do not meet the cost, and schedule requirements, and about 31% are canceled before the completion of the projects. These failure reports are significantly alarming. McManus (2004) identified that 65% of the project failures are accounted by management issues, and 35% by technical issues. Managerial issues include problems with project structure, project resources, planning methodologies, customer buy-in, and inadequate risk management. Technical issues include poor software design, non-adherence to software requirements, improper technical reviews, and incorrect development, and testing methodologies (McManus, 2004). It is conjectured that the management of risks can lead to the success of projects. Risk management has been popular in non-software domains for several decades. However, it is primarily in the last few years that risk management in software domains has become popular. However, at present risk management in software is a developing discipline it is poorly understood, and practiced. Compared to the risk management literature available in other disciplines (e.g., insurance, and manufacturing), the volume of risk management literature available in software is scarce. In this article, we attempt to review the fundamentals of software risk management, the different popular risk management process models, and the recent research trends in the area of software engineering risk management. Software engineering risk management activities are conducted at the project level, process level, and product level. However, in this article, we focus primarily on project-level, and process-level risk management. Practitioners and researchers poorly understand the area of software engineering risk management, even though the management of risks in software projects is an important issue, which can save millions of dollars in the projects. We also realized the scarcity of review articles in this area. This prompted us to exhaustively review the work done in this area, and prepare this article describing the important fundamental concepts in the area, the different important process models for risk management 196

that have become popular in the last 1.5 decades, and the recent research advances that have taken place in the last few years. This article should help the academicians, researchers, and practitioners interested in the area of risk management in software engineering to gain an overall understanding of the area. This article should be of immense help to the software engineering community, because to the best of our knowledge there is no such recent article that reviews the above. WHAT IS RISK? Principles The dictionary meaning of risk is the possibility of loss or injury (Source: Merriam-Webster Dictionary). The term risk has its Etymological origin in the Latin word resceare, which means to cutoff. It has evolved since then as the French word risqué, and the Italian word risco. The term risk is used universally in different contextual domains. For example, it is used in the financial sector to mean the possibility of incurring financial loss, and in the medical sector to mean the possibility of physiological loss to life. In the software world, risk is an important issue often referring to the sources of danger to software development, acquisition, procurement, or maintenance. One of the important considerations challenging any risk management researcher is the definition of risk. In other words, before proposing any risk management framework one needs to specify/quantify the dimensions of risks. This is because it is a challenge to unanimously agree on the definition of risk. There are several formal definitions of risk available in literature, few of which are presented below. A possible future event that, if it occurs, will lead to an undesirable outcome (Leishman and VanBuren, 2003). Risk is a combination of an abnormal event or failure, and the consequences of that event or failure to a system s operators, users, or environment. A risk can range from catastrophic (loss of an entire system, loss of life, or permanent disability) to negligible (no system damage or injury) (Glutch, 1994). Risk refers to a possibility of loss, the loss itself, or any characteristic, object, or action that is associated with that possibility (Kontio, 2001). WHAT IS RISK MANAGEMENT? Simply put, risk management is a way to manage risks. In other words, it concerns all activities that are performed to reduce the uncertainties associated with certain tasks, or events. In the context of projects, risk management reduces the impacts of undesirable events on a project. Risk management in any project requires undertaking decision-making activities. ORIGIN OF RISK MANAGEMENT Risk management has its roots in probability theory, and decision making under uncertainty. Three well-known theories in these areas expected utility theory (Bernoulli, 1954; Hogarth 1987), theory of bounded rationality (Simon, 1979), and prospect theory (Kahneman and Tversky, 1973; Kaheman et al., 1982) were of the greatest influence. These theories may be considered as disciplines by themselves. Therefore, to put our discussions on risk management in context, we briefly state below only what each of these theories propose. 197

In brief, the expected utility theory discusses how people make choices from different alternatives, based on their expected utility. The theory of bounded rationality states that for real life events, the outcomes, and their associated probabilities are very limitedly understood by people to make the required decisions to maximize their expected utility. Therefore, people have a tendency to set up targets of aspiration in life by eliminating alternatives from the different options they have. This theory is useful for modeling the behavior of project management personnel in charge of risk management. Prospect theory, which has its origin in Psychology, helps to model how the perceptions of human beings influence their choices from the given options. It, thus, helps for understanding, and estimating the utility losses of different alternatives while analyzing risks in risk management. PURPOSE OF SOFTWARE ENGINEERING RISK MANAGEMENT Risk management in software projects has different uses. It helps to save projects from failing due to different factors such as non-completion of projects within the specified schedule, and budget constraints, and not meeting customer expectations. Risk management looks at projects from different perspectives to ensure that the threats to the projects are identified, and analyzed, and appropriate strategies are undertaken to mitigate, and control risks. The mitigation strategies may not necessarily mean the cancellation of tasks that involve risks. Many tasks are undertaken in the software industries even after knowing that undertaking them involves taking high risks. The high-risk tasks are sometimes important to provide the industries a leading edge over their competitors. The main purpose of risk management is to know all possible risks to a project, assess their severity, and consequence, and then determine resolution steps depending on the nature of the risks. The idea is to minimize any unforeseen, and unexpected issues arising during the course of the project by properly planning for eventualities. Proper planning leads to minimizing uncertainties, which might lead to a turbulent completion, or a complete cancellation of the projects. Software engineering risk management takes a preventative approach leading to completion of projects within predictable time, and money. In fact, risk-managed projects have the ability to reduce project costs, and time of completion, and increase the overall quality of the project deliverables. Without these, projects could risk loss of revenue, and customer trust in an average case, or a complete bankruptcy of the participating organizations in the worst. RISK MANAGEMENT IN SOFTWARE ENGINEERING Risk management in software has been in existence for many decades. However, as mentioned earlier, it is only in the last decade, and a half or so that it has gained widespread importance in the software community. The software development projects in the early years of the last century conducted risk management using different ad hoc approaches, without following any systematic methodologies. However, with the increasing complexity of software development, industries have realized the importance of risk management, because it helps in reducing the uncertainties involved in developing software, and decreasing the chances of project failures. Before applying any risk management process, the project team members should be clear about the following dimensions of risks in their projects (Smith and Pichler, 2005): The nature of uncertainty involved, and the likelihood with which the risk will occur. The loss that will be incurred if the risk occurs. Loss in software projects can take many forms including loss of revenue, loss of market share, and loss of customer goodwill. The severity of the loss. The duration of the risks. 198

Research on Risk Management in Software Engineering POPULAR SOFTWARE RISK MANAGEMENT MODELS Several software risk management approaches have been proposed in the past, most of which assess risks during all the phases of software development, by integrating risk management practices along with the software development process. As a result, in these approaches, the risk management models follow a disciplined process. These approaches are listed below. Boehm s Risk Management Model (Win-Win) (Boehm, 1988; Boehm and Ross, 1989; Boehm and Bose, 1994; Boehm et al., 1998) SEI s Software Risk Management Model (SRE Version 2.0) (Williams et al., 1999) Hall s Risk Management Model (P 2 I 2 ) (Hall, 1998) Karolak s Risk Management Model (Just-In-Time Software) (Karolak, 1998) Kontio s Riskit Methodology (Kontio, 1997; Kontio, 2001) These approaches are summarized below. A horizontal comparison of all of these approaches may not be fair because, although each of them address risk management, they were developed under different circumstances for solving may be related but different issues. For example, Hall s P 2 I 2 was developed from a risk management capability modeling perspective. On the other hand, Boehm s Win- Win model was developed primarily as a novel software development process model ( spiral development) taking a risk-based approach. We provide below a high-level overview of these approaches. Boehm s Foundational Contributions: Boehm proposed a software development model that was riskdriven. The strength of his model, referred to as the Original Spiral Model (Boehm, 1988), eliminates risks from the early stages of software development, instead of encountering project barriers at the later stages. Boehm extended his Original Spiral Model using the Theory W (Win-Win) Model (Boehm and Ross, 1988; Boehm and Bose, 1994), which aims at satisfying the objectives, and concerns of the stakeholders. The Win-Win Model also supports risk identification, resolution, and continuous monitoring of risks. Although the strategy taken by Win-Win may not always be attainable in practice, it is an important contribution towards engaging stakeholders in the risk management process. Boehm (1991) also proposed a risk management framework, which helps to identify the primary sources of risk, analyze, and resolve them. This risk management framework can be integrated into the Original Spiral, or the Win-Win Model. SEI s Software Risk Management Approach: SEI provided a comprehensive risk management framework comprising of the following three groups of practices: Software Risk Evaluation, Continuous Risk Management, and Team Risk Management. The Software Risk Evaluation approach concerns the identification, analysis, communication, and mitigation strategies for software risk management. The approach depends on, amongst other elements, the risk taxonomy, which consists of constructs used for organizing risk information. The taxonomy helps in providing with an instrument (questionnaire) to elicit different classes of risks. The entire taxonomy of risks can be found in (Higuera and Haimes, 1996), and is omitted from here. The taxonomy has classification of risks into categories such as Requirements risks, Design risks, Coding and testing risks, Contract risks, Resource risks, and so on. Continuous risk management takes a principle-based approach for providing processes, methods, and tools for continuously managing risks during all the phases of software lifecycle. 199

Team risk management, on the other hand, is also a principle-based practice, but concerns the development of methodologies, processes, and tools for developing working relationships between the customers, and suppliers. All these three groups of practices are helpful to each other. For instance, taking a team-oriented approach for risk management helps in continuously managing risks. Hall s P 2 I 2 Approach: Hall (1998) approached risk management by identifying four different factors that have the potential to alter the expected results in any project. These factors are People, Process, Infrastructure, and Implementation. The People factor is concerned with human resource aspects for risk management. This is important because the success of any risk management activities is dependent on the successful communication of different issues arising while conducting risk management activities. The Process factor defines the processes that should be taken to manage risks for minimizing uncertainties involved in the project. The Infrastructure factor defines the requirements, resources, and results required to perform risk management activities in an organization. The Implementation factor concerns the actual implementation of risk management activities such as, establishing the initiatives for risk management, developing the plan, customizing the standard processes to meet the requirements of the project, assessing risks, and controlling risks. Karolak s Approach: Karolak (1998) took a Just-In-Time approach for risk management in software engineering. The Just-In-Time approach attempts to minimize the amount of risks involved, while optimizing the contingency strategies for problematic situations. It takes a risk-driven approach, and advocates the principle of managing risk during the early phases of software development lifecycle to reduce project cost, and time, and improving customer expectations. In his approach, he first identifies a set of high-level risk categories. Then he associates these risk categories to risk factors, risk metrics, and questions to be asked to project stakeholders. These questions are useful as checklists for identifying different classes of risks. Kontio s Riskit Approach: Kontio (2001) proposed the Riskit methodology, which provides a complete conceptual framework for risk management using a goal-, and stakeholder-oriented approach. It attempts to manage risks by capturing the intentions of stakeholders in the risk management process. The implementation of the Riskit methodology helps project managers with the accurate and timely dissemination of project information, opportunity, and risk to different stakeholders, thereby enabling to make critical decisions for the overall success of the project. Riskit also helps for systematically managing the project starting from identification, and analysis of risk to the monitoring, and control of them. At the heart of the Riskit approach is the designing of the Riskit Analysis Graph for analyzing risk factors, risk events, risk outcomes, risk counter-actions, risk effects, and utility loss that would occur due to a risk event. Kontio also proposed a risk management process improvement framework using concepts from Victor Basili s Experience Factory (Basili, 1993). Therefore, the actual understanding of the Riskit 200

Process Management Improvement (PMI) Framework requires an understanding of Experience Repository. Without getting into the details of the Experience Repository, we mention that the essential idea underlying Kontio s Riskit PMI Framework is utilizing experience, and information from previous software development projects for managing risks in the current project. RECENT ADVANCES After discussing the important software risk management process models, we discuss below five recent contributions in the area. They primarily propose risk analysis methodologies, and not a complete risk management framework, unlike the approaches we have seen in Section 3.1. Foo and Murugananthan s Approach: Foo and Murugananthan (2000) proposed a questionnaire-based approach for analyzing risks to provide their quantitative assessment. Their approach can be used to quantify risk elements, and use them to estimate a normalized value of the overall project risk. Their model, called the software risk assessment model (SRAM), is based on the use of situational factors to predict project risks. In other words, risk assessment in this model is dependent on the nature of the project, and the situations facing it. Their model is questionnaire based, and analyzes risks to provide their quantitative assessment. In their model, they consider nine critical risk elements: software complexity, project staff, targeted reliability, product requirements, estimation methodology, monitoring methodology, development process adopted, usability of development software, and tools. Thereafter, they frame a list of questions for the risk assessor, by providing three choices for each of the above critical risk elements. The answers of the assessors are assessed, and sorted according to their increasing risk-levels. Deursen and Kuipers Approach: Deursen and Kuipers (2003) proposed a novel risk assessment methodology by identifying the different primary facts, and secondary facts in a project. The primary facts are obtained by analyzing the system, and the secondary facts are obtained by interviewing different stakeholders, reviewing contract documents, project plans, requirements specifications, and design documents. Finally, the primary facts, and the secondary facts are taken in tandem, and compared to observe whether the risks perceived from both the angles are consistent with each other. This software risk assessment methodology is different from the traditional extremes of product risk management, and process risk management. The advantage of it is that it builds on the advantages of both the above-mentioned extremes to resolve risks having conflicting viewpoints amongst stakeholders. Roy s Approach: Roy (2004) developed the ProRisk management framework by extending the AS/NZS 4350 standard. It categorizes the risk management activities into the business domain, and the operational domain. It performs different activities such as, identifying the stakeholders, identifying the risks factors, constructing a risk-free model, calibrating the risk-free model, estimating the probabilities of risk events, evaluating the combined values of risk, developing action plans, and monitoring the progress. The ProRisk framework identifies two important focal points for risk management in projects: Business domain: Focuses on the organization and the project domain perspectives. It identifies the business parameters of the environment in which the project is conducted. Operational domain: It focuses on the formal modeling of different aspects of risk management in the project. Typical activities constituting the operational domain are measurement of risk values, 201

performing risk assessments, identifying, and proposing action plans for mitigating risks, implementing them, and continuously managing them. Tiwana and Keil s Approach: Recently, Tiwana, and Keil (2004) developed a handy software development risk assessment tool (methodology) that project managers could use to quickly assess some of the important project risks, and their effects. This tool, and the questions in it were developed as a result of risk management data collected from the IT managers of 60 companies. The important achievement of this tool is that it can help in quickly assessing the important risks threatening a project, instead of deploying a full-fledged, time, and budget consuming risk management methodology. Misra et al s Approach: Misra et al. (2005) have also proposed an approach for software engineering risk management. This approach could be used by project managers to model, and control risks in software projects. There are no similar approaches on modeling software project risks in the existing pieces of literature. The approach is, thus, novel to the area of software risk management. The approach is helpful to project managers for performing means-end analysis, thereby uncovering the structural origin of risks in a project, and how the root-causes of such risks can be controlled from the early stages of the projects. Though some attempt has been made to model risk management in enterprise information systems using conventional modeling techniques, like data flow diagrams, and UML, the previous works have analyzed, and modeled the same just by addressing what a process is like. However, they do not address why the process is the way it is. Our proposed approach addresses this limitation of the existing software project risk management models by exploring the strategic dependencies between the actors of a project, and analyzing the motivations, intents, and rationales behind the different entities, and activities in a project. The concept of strategic dependencies between the actors of a project is not new. A good review of the concept can be found in Chung et al. (2000). This approach is restricted to providing a methodology that one could use in the existing risk management lifecycle models to analyze, and uncover the structural origin of the risks, and control the risks from the early phases of a project. OTHER RELATED WORKS The approaches described above do not help in predicting the reliability of the final products. Another class of researchers approached risk management from a product reliability viewpoint. They took a probabilistic approach for assessing software risk by measuring reliability of the products. Specifically, they do not solve the problem of managing risks in project failure due to its inability to complete within the required budget, and schedule. Therefore, as mentioned before, in this article we intentionally omit discussing such approaches here, and limit the scope of this article to the discussions of project, and process risk management methodologies. However, for the sake of completeness, we mention that some of the excellent works on product risk management (specifically, software reliability) can be found in Karunanithi and Whitley (1992), Lanning (1995), Lyu (1995), and Musa (1998). Conclusions In this article, we attempted to present an overview of the fundamental concepts on risk management in software engineering and important research conducted in the area, and then evaluated them. Although the work in risk management in other disciplines started more than a century ago, most of the important works in the area of risk management in software engineering were published in the last 15 years. Still there is a lack of understanding of the area amongst the software engineering practitioners. This review article should help the academicians, researchers, and practitioners. It should help the software project managers to quickly grasp the fundamentals in the area, and quickly decide which methodology they should use in their organization for specific projects. However, it should be noted that the article is aimed to provide only an overview of the different concepts, and methodologies. For 202

obtaining a detailed understanding of how each of the methodologies works, one needs to refer to the corresponding articles that published the methodology. The graduate students in advanced software engineering, project management, information systems courses, and other researchers in these areas will find this article invaluable, to learn about the different concepts in software engineering risk management. Many of the approaches discussed in this article are limited by the lack of empirical evidences supporting them. This is an important area in which future work should be targeted. Focus should be made on comparing the competing approaches with respect to a predefined set of evaluation criteria. For most of the proposed approaches we need controlled case studies, and actual field trials for assessing their effectiveness, and applicability under modern contexts, and shifting paradigms. References Basili, V.R. 1993. The Experience Factory and its Relationship to Other Improvement Paradigms. Proceedings of the 1993 European Software Engineering Conference, Springer-Verlag. Bernoulli, D. 1954. Exposition of New Theory on the Measurement of Risk, Econometrica, 22: 23-36. Boehm, B. W. 1988. A Spiral Model of Software Development and Enhancement, Computer, May, 61-72. Boehm, B. W. 1991. Software Risk Management: Principles and Practices, IEEE Software, 8(1):32-41. Boehm, B. W. et al. 1988. Using the Win-Win Spiral Model: A Case Study, IEEE Computer, 31(7), 33-44. Boehm, B. W. and Bose, P. 1994. A Collaborative Spiral Software Process Model Based on Theory W, Proceedings of the 1994 International Conference on Software Process, IEEE Computer Society, Washington. Boehm, B.W. and Ross, R. 1988. Theory-W Software Project Management: A Case Study, Proceedings of the 1988 International Conference on Software Engineering, Singapore, 30-40. Boehm, B.W.and Ross,R. 1989. Theory W Software Project Management: Principles and Examples. IEEE Transactions on Software Engineering. 15(7):902-916. Chung, L, Nixon, B.A.,Yu, E. and Mylopoulos J. 2000. Non-Functional Requirements in Software Engineering. Kluwer Academic Publishers. Deursen, A. v. and Kuipers, T. 2003. Source-Based Software Risk Assessment. Proceedings of the 2003 International Conference on Software Maintenance, Amsterdam, Netherlands. Donzelli, P. 2002. Agents, Goals, and Quality in a Structured Requirements Engineering Framework - A Case Study. Proceedings of 2002 CAiSE'02, Toronto, Ontario. Dorofee, A. J. et al. 1996. Continuous Risk Management Guide Book, SEI, Carnegie Mellon University, Pittsburgh, PA, USA. 203

Foo, S. -W. and Muruganantham, A. 2000. Software Risk Assessment Model. Proc. of the 2000 IEEE International Conference on Management of Innovation and Technology, 2, 536-544. Giunchiglia, F., Mylopoulos, J., and Perini, A. 2002. The Tropos Software Development Methodology: Processes, Models and Diagrams. Proceedings of AOSE-2002, Bologna, Italy. Glutch, D. P. 1994. A Construct for Describing Software Risks, Technical Report, Report # CMU/SEI-94- TR-14, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA, USA. Hall, E. M. 1998. Managing Risk: Methods for Software Systems Development, Addison-Wesley, Reading, U.K. Hogarth, R. M. 1987. Judgment and Choice, John Wiley & Sons, New York, USA. Higuera R. P. and Haimes, Y. Y. 1996. Software Risk Management, Technical Report, Report # CMU/SEI-96-TR-012, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA, USA. Kahneman D., and Tversky A. 1973. On the Psychology of Prediction, Psychology Review, 80, 237-251. Kahneman, D., Slovic, P., and Tversky, A. 1982. Judgment under Uncertainty: Heuristics and Biases, Cambridge University Press, New York. Karolak, D. 1996. Software Engineering Risk Management, IEEE Computer Society Press, Los Alamitos, CA, USA. Karolak, D. 1998. Software Engineering Risk and Just-In-Time Development, International Journal of Computer Science and Information Management, 1 (4). Karunanithi, N., and Whitley, D. 1992. Using Neural Networks in Reliability Prediction, IEEE Software. Kontio, J. 1997. The Riskit Method for Software Risk Management. Version 1.00, Technical Report, CS- TR-3782/UMIACS-TR-97-38, University of Maryland, Computer Science, College Park, MD, USA. Kontio, J. 2001. Software Engineering Risk Management: A Method, Improvement Framework, and Empirical Evaluation. Ph.D. Thesis, Department of Computer Science and Engineering, Hensinki University of Technology, Finland, 2001. Lanning, D. L. 1995. A Neural Network Approach for Early Detection of Program Modules Having High Risk in the Maintenance Phase, Journal of Systems and Software, 29. Leishman, T. R., and VanBuren, J. 2003. The Risk of Not Being Risk Conscious: Software Risk Management Basics, STSC Seminar Series, Hill AFB, UT. Lyu, M. 1995. Software Reliability Engineering, IEEE Computer Society Press. McManus J. 2004. Risk Management in Software Development Projects, Elsevier. Misra, S. C., Kumar V., and Kumar, U. 2005. Modeling Strategic Actor Relationships to Support Risk Analysis and Control in Software Projects. Proceedings of the 2005 International Conference on Enterprise Information Systems, Miami, Florida, USA, May 25-28, 288-293. 204

Musa, J. 1998. Software Reliability Engineering: More Reliable Software, Faster Development and Testing, McGraw Hill. Roy, G.G. 2004. A Risk Management Framework for Software Engineering Practice, Proceedings of the 2004 Australian Software Engineering Conference (AAWEC 04), IEEE Computer Society, April, Melbourne, Australia. Simon, H. A. 1979. Rational Decision Making in Business Organizations. The American Economic Review, 69 (4), 493-513. Smith P. G., and Pichler R. 2005. Agile Risks/Agile Rewards, Software Development Magazine, 50-53. Standards Australia. 1999. AS/NZS 4360: 1999, Risk Management. Standish Group. 1994. The Chaos Report, http://www.standishgroup.com/sample_research/chaos_1994_1.php Tiwana, A., and Keil. M. 2004. The One-Minute Risk Assessment Tool. Communications of the ACM, 47 (11), 73-77. Williams, R.C, et al. 1999. Software Risk Evaluation (SRE) Method Description (Version 2.0), Technical Report, Report # CMU/SEI-99-TR-029, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA, USA. 205