Evaluation and Integration of Risk Management in CMMI and ISO/IEC 15504

Transcription

1 Evaluation and Integration of Risk Management in CMMI and ISO/IEC Dipak Surie, ens03dse@cs.umu.se Computing Science Department Umea University, Umea, Sweden Abstract. During software development, software companies are exposed to risks (unwanted events) which lead to economic and competitive loss. Risk management technique can be used to minimise the loss in software projects. Some software process improvement models include risk management among their guidelines. The goal of this paper is to evaluate and analyse how risk management is handled by Capability Maturity Model Integration (CMMI) and International Organization for Standardization/International Electrotechnical Commision (ISO/IEC) A risk management framework is used in this evaluation study. The results of this study show that risk management in CMMI is more comprehensive than ISO/IEC 15504, but both models handle risk factors to appreciable extent. 1 Introduction Most of the software development projects are performed in unpredictable environments with many risks resulting in unsuccessful completion of the projects. The following research results give an apprehensive picture of the amount of risks involved in software development. According to the Standish Group Report [1], 31.1% of projects are cancelled before completion and only 16.2% of the projects are completed on-time, within budget and meet requirements. Due to the unsatisfactory results of many of the software projects, there arouse a need for improvement in software quality and including the concept of risk management in software development. Risk management techniques can be used to mitigate the probability of risk occurrence which minimises the loss in software projects and invariably improve software quality [2]. Many organisations have realised the importance of software processes in quality improvement which has resulted in the evolution of Software Process Improvement (SPI) models [3]. These models introduce innovative and improved concepts to software development with new tools, techniques and practices [4],[5]. Some SPI models include risk management among their guidelines, one of these is the CMMI which has one process area on risk management. The aim of this paper is to analyse the risks involved in software development with a risk management framework that is described in this paper. The analysis is further extended to check how risk management is taken care in SPI models

2 2 Dipak Surie, ens03dse@cs.umu.se like CMMI and ISO/IEC A few popular risk management models like Boehm Software Risk Management, Software Risk Evaluation (SRE), Continuous Risk Management (CRM) and Team Risk Management (TRM) of Software Engineering Institute (SEI) are used in the risk management framework along with risk factors obtained from various empirical studies and research papers. The paper is organised as follows. First, the frame work for risk management is discussed, followed by an introduction to SPI models. Then, risk management concepts in CMMI and ISO/IEC are discussed. Finally, an evaluation with respect to risk management framework is performed and derived results are discussed. 2 A Framework for Risk Management In this paper a framework for risk management is used in evaluating CMMI and ISO/IEC This framework includes Risk management model by Boehm, Software Risk Evaluation model, Team Risk Management model and Continuous Risk Management model proposed by Software Engineering Institute and also the risk factors obtained from various research papers. 2.1 Related Concepts Software risks impose a threat to the successful completion of a software development project and hence a collection of methods to either avoid their occurrence or to mitigate their effect is needed [6]. This resulted in the evolution of risk management concepts which are very important and indispensable part of project management. Charette quotes that, Risk Management does not deal with future decisions, but with the future of present decisions [7]. A continuous risk management is essential throughout the development life cycle as problems addressed late in the cycle results in changes that are more expensive and sometimes ineffective [8]. Any risk management program includes risk identification as its first step, followed by risk analysis, strategies to either avoid or mitigate risks and finally monitor the risk management tactics. 2.2 Existing Models There are many software risk models like the Boehm Software Risk Management model, Software Risk Evaluation, Continuous Risk Management and Team Risk Management that are comprehensive for risk management. These models are based on risk avoidance when compared to risk correction. The basic techniques that they use are avoidance, control questionnaires and expert opinion. These methods can be used in all phases of software development. Boehm Software Risk Management Model The Boehm Software Risk Management model is based on the concept of risk exposure which is dependent on probability of an unsatisfactory outcome and the loss that occurs due

3 Risk Management in CMMI and ISO/IEC to unsatisfactory outcome [9]. This model uses decision trees and top-ten risk identification check list. According to Boehm, accurate estimation of probability or loss associated with an unsatisfactory outcome is in itself a major risk due to the uncertain nature of the estimation. The top ten risks proposed by Boehm are included in the risk factors subsection. Software Risk Management Model by Software Engineering Institute The Multi-dimensional Holistic vision of Software Risk Management includes practices like Software Risk Evaluation (SRE), Continuous Risk Management (CRM) and Team Risk Management (TRM). The frame-work is based on three basic constructs (Risk Management Paradigm, Risk Taxonomy and Risk Clinic) and risk dimensions including temporal, methodological and human [10]. Communication of risk is included in its risk management processes along with basic processes like identification, analysis, planning, tracking and controlling. The Figure 1 describes the commonly performed risk management practices in SRE, TRM and CRM. It also describes how communication is the centre focus around which the various practices are built. Fig. 1. Risk Management Practices [10] The Figure 2 describes the Team Risk Management concept wherein both the customer and the supplier have their own risk managment program but form a team with open communication and take decisions together. Fig. 2. Team Risk Management [10] The SRE method involves risk taxonomy, that is based on questionnaire for risk assessment. SRE is a complex and well defined risk management approach that takes care of initial risk evaluation as well as continuous risk management.

4 4 Dipak Surie, ens03dse@cs.umu.se The continuous risk management is comprised of comprehensive process, guidelines and techniques for structuring information, documenting risks, analyzing and prioritizing risks. Team risk management is integrated into continuous risk management with principles like shared product vision, open communication and effective teamwork through a defined process. Overall, Software Engineering Institutes Software Risk Management is a very comprehensive risk management program and covers a wide range of constraints. Fig. 3. Continuous Risk Management [10] The figure 3 describes the Continuous Risk Management model with three layers. The innermost layer 1 represents core principle, the subsequent layer 2 represents sustaining principle and the outermost layer 3 represents defining principle. This model is also built around the concept of open communication. Risk Factors There are various risk factors that have to be considered in a risk management program. The top ten risk factors by Boehm is an extensive list covering various possible sources of risk. It is obtained by empirical study of the commonly occurring risk in many large software projects [9]. The risk factors addressed by Ropponen and Lyytinen are an extension of the work done by Boehm with additional risks factors [2]. Addison and Vallabh have proposed risk factors that are based on comprehensive empirical studies [6]. Hence in this paper the risk items from all the three sources are considered in evolving a risk factor framework and in evaluating the risk management concepts addressed in CMMI and ISO/IEC Software development risks can be classified into the following eight risk areas. All the risk factors that are acknowledged by the above three sources could be fit into one of these areas.

5 Risk Management in CMMI and ISO/IEC Scheduling and Timing risks Problems in timetable Budgets estimated incorrectly Wrong size estimates Estimates for personnel need Managing project complexity Changes in timetable Development time estimated incorrectly Lack of senior management commitment 2. System functionality risks Estimates for personnel need Wrong software functions development Developing wrong user interface Wrong estimation of hardware and software capabilities Development of unneeded software functions 3. Subcontracting risks Estimates for personnel need Shortfalls in externally furnished components shortfalls in externally performed tasks 4. Requirements management risks Gold plating Requirements changes Changes in timetable Unclear or misunderstood scope/objectives Failure to gain user involvement Incomplete and infeasible requirements 5. Resource usage and performance risks Resource usage and deadline Managing project complexity Evaluation of performance requirements Estimation of hardware and software capabilities Real-time performance shortfalls Lack of technical solutions 6. Personnel Management risks Personnel shortfalls Insufficient expertise Unrealistic expectation of personnels abilities Evaluation of performance requirements Unclear or misunderstood scope/objectives Lack of senior management commitment 7. Project Management risks Lack of effective project management methodology Personnel management risks 8. Technology risks Introduction of new technology Personnel shortfalls

6 6 Dipak Surie, ens03dse@cs.umu.se 3 SPI Models 3.1 Introduction The quality of software process plays a vital role for organisations in delivering quality software within budget and schedule. This resulted in the development of software process improvement initiatives. There are numerous case studies conducted at various organisations that indicate the benefits obtained by applying software process improvement activities. But software process improvement programs are not intended to show overnight results and it is a continuous process. Software quality and process improvement initiatives began in the mid 1980s with many improvement themes like Total Quality Management, Process Definition, Quality System Definition/Improvement and Process Improvement Models. The Software Process Improvement models focus on processes to achieve quality software. The key point is in actual implementation of these models. The question of what to implement is addressed in a comprehensive way but less efforts are taken in answering how to implement them [11]. According to Herbsleb and Goldenson, 67% of SPI managers need assistance on how to implement SPI processes instead of what to implement [12]. Successful implementation of SPI involves strong commitment from the senior management [13]. The two SPI models evaluated in this paper are CMMI and ISO/IEC CMMI The Capability Maturity Model (CMM) is a process improvement model developed by Software Engineering Institute (SEI) with evolutionary maturity levels embedded in it. It follows the basis principle of close correlation between quality of software product and quality of the software development process. The Capability Maturity Model Integration (CMMI) was released in 2002 with a purpose of integrating the Software CMM and the Systems CMM [14]. The need for this integration is to reduce the complexity of performing multiple process improvement and assessments using multiple CMM models. CMMI projects benefit from common, integrated vision of improvement for all organizational elements [14]. Another goal for the development of CMMI is to make CMM compatible and conformant with emerging ISO standard for software process assessment, the ISO/IEC CMMI has two representations with the continuous version having similar process areas to ISO/IEC and helps in efficient comparison process. In this paper we focus on the continuous CMMI, which uses capability levels from 0 to 5 for each process area. Each capability level has its own generic goals, specific goals, generic practices and specific practices [14]. Capability levels are determined by analysing how well the specific and general practices are implemented and the extent to which the general and specific goals are achieved in an organisation. Generic goals and practices are applied to multiple process areas whereas specific goals and practices are applied only to individual process areas.

7 Risk Management in CMMI and ISO/IEC Fig. 4. Components of CMMI [16] The capability level starts with Incomplete process, followed by Performed, Managed, Defined, Quantitatively managed and Optimized. The organisation processes are mapped to the CMMI process areas, and its process is rated according to the capability level. A particular capability level to a process area implies that it satisfies the goals and practices of not just its capability level but also the lower capability levels. 3.3 ISO/IEC The Software Process Improvement and Capability Determination (SPICE) ISO/ IEC is an emerging international standard for software process improvement. The ISO/IEC type 2 technical reports were released in 1998 and it gives a hint of becoming the international standard for software assessment. There are many process improvement models like the Bootstrap, Trillium, CMMI, ISO 9000 etc. and hence a need for international standard arose which was the main goal for the development of ISO/IEC The main focus of this standard is on process assessment and process improvement. This standard includes SPICE process model which is used as the basis on which software process assessment can be executed. The reference model has two dimensions with process dimension describing the processes to be assessed and the capability dimension describing the measure on which to assess [15]. There are six capability levels with rating from 0 to 5 as in CMMI. The capability level begins with incomplete process, followed by performed, managed, established, predictable and optimizing. There are five categories of processes defined in SPICE standard including Engineering Processes, Customer Supplier Processes, Management Processes, Support Processes and Organization Process. Risk management is included in the Management Processes category and its purpose is to identify and mitigate risks throughout the lifecycle of software development. One of the main advantage of this standard is that the organisation can concentrate on the required process areas for improvement according to its business needs when compared to a step-by-step evolutionary approach.

8 8 Dipak Surie, ens03dse@cs.umu.se 4 Risk Management in CMMI Managing risk is one of the important features of CMMI. It uses the concepts of defining a risk management strategy, identify and analyse risks and finally handling the identified risks [16]. Simple risk identification is taken care of in the project planning and project monitoring and control process areas. Risk management process area is a comprehensive structure that defines a systematic plan and the means to anticipate and mitigate risks. Structured decision making for the identified risks is taken care in decision analysis and resolution process area. Fig. 5. Risk Management process area [16]

9 Risk Management in CMMI and ISO/IEC The three specific goals in risk management process area are prepare for risk management, identify and analyse risks and finally mitigate risks. The risk sources are identified with well-defined risk parameters to establish a risk management strategy. Risk categories include technology, engineering, manufacturing and design. A risk taxonomy framework is used which is very similar to risk management constructs of SEI. The concept of probability of risk occurrence and their impact is used along with defining thresholds for each risk category. The scope of risk management and methods/tools used for risk management processes like identification, analysis, mitigation etc. are included in the risk management strategy. The figure 5 describes the components of risk management process area [16]. The risk mitigation techniques used include prototyping, simulation, alternative designs and evolutionary development. The human dimensions to risk management including training, communication and commitment and are well addressed in CMMI. Organisational training process area identifies the training needs and provides a structured training program. CMMI includes training people as one of its general practices in institutionalize a managed process. The problem of stakeholders involvement and their commitment is addressed in project monitoring and control process area. Risk management in CMMI is a continuous process that is executed throughout the software development life cycle [16]. 5 Risk Management in ISO/IEC Risk management is addressed in the risk management process, which is a part of organisational life cycles. This risk management process takes care of the basic risk management steps such as risk identification and mitigation. This process extends to the entire duration of the project life cycle. The two dimensions wherein risks are monitored are at the project level and the organisational level. The scope and strategy for risk management are firstly defined and then implemented. The concept of risk metrics and risk priority are used wherein the former measures the changes in risk states while the latter is used in risk monitoring. This risk management is not as comprehensive as the one supported by CMMI. The risk management is spread across more than one process categories [17]. The human dimension to risk management is addressed in the human resource management process with concepts such as training, team building, effective interaction, performance etc.. Continuous monitoring and performance verification are dealt with in the quality management process. Performance monitoring is comprehensive for software process performance with very less information on its influence on risk management. The Organisational process category ensures that the standard is in compliance with continuous risk management with concepts such as open communication and shared product vision [17]. The software process improvement steps include cultural issues wherein the commitment and responsibility of management is given the principal focus. Other software process

10 10 Dipak Surie, ens03dse@cs.umu.se Fig. 6. ISO/IEC process categories [17] improvement steps include education and training, management, customer satisfaction, planning for process improvement etc. Customer satisfaction is dealt very effectively with a separate process category customer-supplier. This takes care of some of the risk factors like new customer requirements, gold plating, unclear or misunderstood scope/objectives etc.. Even though risk management is not a comprehensive part of ISO/IEC 15504, the basic risk management concepts are well applied [17]. 6 Analysis and Evaluation with respect to Risk Management Framework 6.1 CMMI CMMI and Boehm Risk Management Risk management proposed by Boehm includes the concepts of top ten risk identification checklist and decision trees. This risk model is fully satisfied in CMMI. The active process areas associated with this risk model are risk management, project planning and decision analysis and resolution process areas. Risk identification is supported with specific goals identify and analyse risks and develop a project plan. The supportive practices are to identify risks. Decision trees concept is implemented by establishing the specific goal evaluate alternatives which uses the practices like selecting decision making technique, identify and evaluate alternatives etc..[9],[16]. CMMI and Continuous Risk Management CMMI is based on continuous risk management approach which uses the principles of open communication,

11 Risk Management in CMMI and ISO/IEC continuous process, integrated management, team work, forward looking view, global perspective and shared product vision. The important process areas involved in this approach include risk management, requirements management, project planning and decision analysis and resolution. Team work is supported in CMMI with generic practices that identify and involve relevant stakeholders, assign responsibilities and continuous status review with higher level management. The practices like ensure continuous process improvement and monitor/control the process ensures that the continuous process principle is satisfied. Forward looking view involves decision making techniques as well as means to identify, evaluate, classify and prioritise risks. These practices are clearly described in decision analysis and resolution and risk management process areas. The other principles are also satisfied to appreciable extent [10], [16]. CMMI and Team Risk Management Team risk management is built on the customer-supplier team oriented activities wherein the decisions are taken together. The principal process areas in CMMI that take care of TRM are project monitoring and control, supplier agreement management, requirements management and risk management. Specific goals such as establish supplier agreement and satisfy supplier agreements make sure that customer-supplier activities are on track with synchronisation. The general idea in this model that is satisfied by CMMI is to involve all the relevant stake holders in a continuous monitoring environment. Requirements engineering process area has the specific goal obtain understanding of requirements where in practices like reaching understanding between requirements provider and project participants and also in obtaining commitment to requirements are performed [10], [16]. CMMI and Software Risk Evaluation CMMI performs functions including risk detection, specification, assessment, consolidation, planning and coordination, verification and validation, training and communication. CMMI also satisfies the Software risk evaluation model. The SEI provides a comprehensive risk taxonomy which is followed in the CMMI approach in risk management process area. The Specific practice determine risk sources and categories is based on risk taxonomy. The compliance of CMMI to popular risk management models is a clear indication that it is of comprehensive nature with respect to risk management. But one can argue the fact that both CMMI and popular risk management models like SRE, TRM and CRM are developed by the same institute, Software Engineering Institute [10], [16]. 6.2 ISO/IEC ISO/IEC and Boehm Risk Management This process improvement model is more concerned with process improvement and capability determination with a comprehensive assessment approach. But risk management is an important part of any process and quality improvement approach. The Software risk management of Boehm includes decision tree approach but ISO/IEC does

12 12 Dipak Surie, ens03dse@cs.umu.se not support this approach to satisfactory level. The priority list for risks is a concept that is in compliance with top-ten risk identification checklist proposed by Boehm and mechanism to obtain importance of a risk [9], [17]. ISO/IEC and Team Risk Management Team risk management includes customer-supplier activities that are dealt with a separate process category customer-supplier as we have seen earlier. The various processes in this category include acquisition process that prepares for acquisition, supplier selection, monitoring and also customer acceptance. The other processes are supply process, requirements elicitation process and operation process [10], [17]. ISO/IEC and Software Risk Evaluation ISO/IEC includes documentation process that takes care of specification function in software risk evaluation. But there is no explicit indication of risk documentation. The support life cycle processes include verification and validation process that hardly describes about risk management. Training and communication is dealt at appreciable levels in human resource management process. But training personnel to perform risk management is not directly supported as in CMMI. This process includes effective interaction between individuals and groups, team building and performance feedback for both individuals and the group. Overall not all the software risk evaluation principles are fully satisfied [10], [17]. ISO/IEC and Continuous Risk Management Continuous risk management includes principles like team work, open communication, shared product vision etc. that are well supported in ISO/IEC Organisational process category includes organisational alignment process that addresses shared product vision and open communication. This ensures that everyone in the organisation understand their roles and share common vision and goals [10], [17]. 6.3 Comparative Study with Risk Factors All the risk factors that are included in the eight risk areas are used in evaluating CMMI and ISO/IEC There are many risk factors in the requirements management risk area including requirements changes, gold plating, unclear or misunderstood scope/ objectives etc. These risks are handled in the requirements engineering process area for CMMI and customer-supplier process category in ISO/IEC Requirements elicitation process in ISO/IEC handles new customer requirements, continuous monitoring of customer needs, change in technology and customer needs etc. [9], [2], [6], [16], [17]. The risk factors under subcontracting risk area are handled in the supplieragreement management process area for CMMI and customer-supplier process category for ISO/IEC The risk factor estimates for personnel need is not handled by both CMMI and ISO/IEC System functionality risks include wrong software functions development, estimates for personnel need, developing

13 Risk Management in CMMI and ISO/IEC wrong user interface etc.. CMMI addresses them in technical solution process area where as ISO/IEC takes care of them in the engineering process category. Risks related to scheduling and timing is well addressed in both SPI models. Since CMMI is a more comprehensive effort, it handles these risks better compared to ISO/IEC [9], [2],[6], [16],[17]. Resource usage and performance risks include managing project complexity, real-time performance shortfalls, lack of technical solution etc.. The risk factor resource usage and deadline is handled in infrastructure process of ISO/IEC The verification and organisational process performance process areas are responsible for real-time performance shortfalls in CMMI whereas quality management process take care of them in ISO/IEC Technical solution process area handles the risk lack of technical solutions in CMMI. Managing project complexity is a risk factor that is handled very efficiently in CMMI but not so well in ISO/IEC [9], [2],[6], [16],[17]. Personnel management risks include unrealistic expectation of personnels abilities, evaluation of performance requirements, personnel shortfalls, insufficient training etc.. Lack of senior management commitment is a very important risk factor that is handled by both SPI models efficiently. Organisational training process area is responsible to address personnel shortfalls and insufficient expertise in CMMI, whereas human resource management process takes care of them in ISO/IEC Since feedback of individuals performance concept is used in ISO/IEC 15504, it deals with unrealistic expectation of personnels abilities effectively. But CMMI handles them less comprehensively. Both project management risks and technology risks are well managed in CMMI as well as in ISO/IEC Hence both models cover most of the commonly occurring risk factors [9], [2],[6], [16],[17]. 7 Conclusion Risk management is a very important aspect of any software development. It determines to a very large extend the success or failure of a project. CMMI has a very comprehensive concept of risk management and is in compliance with popular risk models like the risk management model of Boehm and the SRE, TRM and CRM of SEI. The above risk management framework contains many commonly existing risk factors. Both CMMI and ISO/IEC handle these risk factors efficiently. In ISO/IEC 15504, risk management concept is spread across many process categories. ISO/IEC is not as comprehensive as CMMI in risk management but covers all the basic risk management requisites. Hence the study of how risk management is supported in CMMI and ISO/IEC References 1. The Standish Group Report, Available at beau/pm/standish-report.html

14 14 Dipak Surie, ens03dse@cs.umu.se 2. J.Ropponen, K.Lyytinen, Components of software Development Risk: How to Address Them? A Project Manager Survey, IEEE Transaction on Software Engineering, February B.Pitterman, Telcordia technologies:journey to high maturity, IEEE Software, July-August W.S.Humphrey, Managing the Software Process, Pittsburgh, PA, Published by Addison-Wesley, B. McFeeley, IDEAL: A Users Guide for Software Process Improvement, CMU/SEI-96-HB-001, Pittsburgh, PA, Software Engineering Institute, T. Addison, S.Vallabh, Controlling Software Project Risks-An Empirical Study of Methods used by Experienced Project Managers, Proceedings of SAICSIT, R.N.Charette, Information Technology Risk Engineering, SEI/NSIA Workshop on Software Risk, Pittsburgh, PA, Software Engineering Institute, B.W.Boehm, A Spiral model of software development and enhancement, IEEE Computer, May B.W.Boehm, Industrial Software Metrics top 10 list, IEEE Software, R.P.Higuera, Y.Y.Haimes, Software Risk Management, CMU/SEI-96-TR- 012, ESC-TR , Software Engineering Institute, M.Niazi, D.Wilson, D.zowghi, A maturity model for the implementation of software process improvement: an empirical study, The journal of Systems and Software, J.D.Herbsleb, D.R.Goldenson, A Systematic survey of CMM experience and results, 18th International Conference on Software Engineering (ICSE 18), Germany, D.R.Goldenson, J.D.Herbsleb, After the Appraisal : A Systematic Survey of Process Improvement, its Benefits and Factors that influence success. CMU/SEI-95-TR-009, Software Engineering Institute, CMMI Product Team, CMMI for Software Engineering,Version 1.1, Continuous Representation, CMU/SEI-2002-TR-028, Software Engineering Institute, K.E.Emam, J.N.Drouin, W.Melo, SPICE The theory and Practices of Software process improvement and capability determination, IEEE Computer Society Press, Los Alamitos, California, CMMI Product Team, CMMI for Systems Engineering/Software Engineering, Version 1.02, CMU/SEI-2000-TR-019 and ESC-TR , Software Engineering Institute, ISO/IEC TR :1998(E), Information technology- Software process assessment-part 2: A reference model for processes and process capability, Available at /SPICE2003/tr pdf