Capability Maturity Model Software Development Using Cleanroom Software Engineering Principles - Results of an Industry Project

Transcription

1 Capability Maturity Model Software Development Using Cleanroom Software Engineering Principles - Results of an Industry Project Robert S. Oshana Member Group Technical Staff Raytheon Systems Company oshana@ti.com Richard C. Linger Software Engineering Institute Carnegie Mellon University Pittsburgh, PA rlinger@sei.cmu.edu Abstract The Capability Maturity Model (CMM) for Software sm is a development framework that describes the key elements for an effective software process. Cleanroom software engineering (CSE) is a managerial and engineering process for the development of high quality software with certified reliability. The combination of CMM management and organizational capabilities and the judicious application of Cleanroom technical practices represents a powerful process improvement paradigm. Cleanroom principles are also compatible with the Systems Engineering Maturity model. This paper describes the results of a Cleanroom technology insertion effort into an industrial environment with a mature CMM process in place. 1. Introduction Recently, a software development group at Raytheon Systems Company, operating at SEI Level 4/5, was confronted with a major challenge: significantly reduce the estimated delivery time and cost for an embedded systems software project without sacrificing software quality or reliability. Furthermore, the project was not be put at risk by any major changes or technology insertion. At a less mature level, Level 1 or 2, the project would certainly not have been equipped to tackle such a challenge (although the siren song of a Silver Bullet might have been proved tempting). However, a solid understanding of current software process combined with measurement data tracking similar projects, made it possible to assess where improvement could be made to increase sm Capability Maturity Model and CMM are service marks of Carnegie Mellon University competitiveness by reducing cost and compressing the schedule while maintaining quality. Any process improvement effort should be constructed to support the overall business goals of the organization. The purpose of this process improvement effort was to gain competitive advantage by improving the effectiveness and efficiency of the processes used to develop software. This paper will present the results of an actual industrial approach to using Cleanroom software engineering within the framework of the Capability Maturity Model. 2. Project environment The project environment consists of a strong process oriented development strategy typical in many DoD environments. The project has been chosen to pilot the process level environment of CMM level 4 (managing) and 5 (optimizing) maturity levels. The respective key process areas (KPA) which include components of continuous process improvement, process optimization, and process measurement and analysis are being implemented. The project is a real-time embedded software application using a heterogeneous computer architecture consisting of two fundamental computing environments; Digital signal processors (DSP) using the C programming language for a primarily signal processing algorithm based application. There are significant real-time constraints within the signal processing application. PowerPC single board computer (SBC) using Ada for embedded command and control, The project follows a tailored version of DoD-STD- 2167A and MIL-STD-498 documentation standards. 1

2 The project is built upon an Integrated Product Team (IPT) structure. There is a fully defined software development process as well as system engineering and hardware development processes. The software effort for the project is divided into a number of Computer Software Configuration Items (CSCI), with a software team of two to sixteen software engineers per CSCI. A CSCI is a complete set, or any of the individual items of the set, of computer programs, procedures, and associated documentation and data designated for delivery to a customer or end user. 3. The Capability Maturity Model The Capability Maturity Model for Software (CMM) is a framework that describes the key elements of an effective software process. The CMM describes an evolutionary improvement path from an ad hoc, immature process to a mature, disciplined process. The CMM covers practices for planning, engineering, and managing software development and maintenance. When followed, these key practices improve the ability of organizations to meet goals for cost, schedule, functionality, and product quality. The CMM establishes a yardstick against which it is possible to judge, in a repeatable way, the maturity of an organization s software process and compare it to the state of the practice of the industry. The CMM can also be used by an organization to plan improvements to its software process. As organizations establish and improve the software processes by which they develop and maintain their software work products, they progress through levels of maturity. Each maturity level provides a layer in the foundation for continuous process improvement. Achieving each level of the maturity model institutionalizes a different aspect in the software process, resulting in an overall increase in the process capability of the organization [1]: Level 1: Ad hoc software development and an unstable development environment. Level 2: Basic software management controls and disciplined process adherence. Level 3: Standard and consistent development process. Level 4: Predictable and well measured process. Level 5: Continuously improving process. 4.1 The Key Process Areas of the CMM Each CMM level is composed of a set of Key Process Areas (KPAs) as shown in Table 1. Each KPA identifies a cluster of related activities that, when performed collectively, achieve a set of goals considered important for enhancing process capability. The KPAs are building blocks that indicate the areas an organization should focus on to improve its software process. KPAs identify the issues that must be addressed to achieve a maturity level. Each KPA, when satisfied, stabilizes an important aspect of the software process. Table 1. The Key Process Areas by Maturity Level Maturity Level Key Process Areas Initial (Level 1) None Repeatable (Level 2) Software Configuration Management (SCM) Software Quality Assurance (SQA) Software subcontract management (SSM) Software project tracking and oversight (SPTO) Software project planning (SPP) Requirements management (RM) Defined (Level 3) Peer reviews (PR) Inter-group communication (IC) Software product engineering (SPE) Integrated software management (ISM) Training program (TP) Organization process definition (OPD) Organization process focus. (OPF) Managed (4) Software quality management (SQM) Quantitative process management (QPM) Optimizing (5) Process change management (PCM) Technology change management (TCM) Defect prevention (DP) 2

3 4. Cleanroom software engineering Cleanroom software engineering (CSE) is a managerial and engineering process for the development of high quality software with certified reliability [2]. Cleanroom software engineering is a collection of principles, practices, and processes. The overriding goal of CSE is development of software that exhibits zero failures in the field. The principles form the theoretical base, and constitute the core of CSE. The practices constitute the implementation of CSE. The process descriptions help engineers and managers organize and control the engineering activity to reach specific project goals. Cleanroom software engineering consists of a body of practical and theoretically sound engineering principles applied to the activity of software engineering. Cleanroom consists of a thorough specification phase which culminates with a precise description of the software part of a system. Software development proceeds from this specification phase using a step-wise refinement procedure using boxstructured design concepts. This process focuses on defect prevention, effectively eliminating costly error removal phases (i.e., debugging) and produces verifiably correct software parts. Development of software proceeds in parallel with a usage specification of the software. This usage profile becomes the basis for a statistical test of the software, resulting in a scientific certification of the quality of the software part of the system. The Cleanroom process spans the software life cycle. The technology provides engineering disciplines with which software teams can plan, measure, specify, design, verify, code, test, and certify software. Management of the Cleanroom process is based on an incremental life cycle in which development and certification are conducted in a pipeline of userfunction increments. The Cleanroom Software development process is shown in Figure 1. This process decomposes the system level black box which defines overall system behavior into a series of increments which accumulate into the full system. The increments are defined and ordered so that each accumulation including the first can be executed by user supplied stimuli and the results checked by user observable responses. Functional Specification Box structure Specification and Design Source Code Improvement and Feedback Specification Function Usage Increment Planning Incremental Development Plan Statistical Testing Quality Certification Model Usage Specification Usage Modeling and Test Case Generation Failure Data Measurements of Operational Performance Test Cases Figure 1. The Cleanroom Software Engineering Process The core principles of Cleanroom are as follows: 1. Incremental development under Statistical Quality Control (SQC). Incremental development provides a basis for statistical quality control of the development process. Each increment is actually a complete iteration of the process. Each increment is measured and compared with pre-established standards to determine whether or not the process is "in control." If the process is in control, work on the next increment continues. If quality standards are not met, testing of the increment stops and the increment is returned to the developers to redesign. Performance is typically assessed during increment testing in measures such as rate of growth in MTTF, or number of consecutive errorfree random test cases. Results from each increment are used for management and process improvement. The team examines all results, identifies problem areas, and re-plans if necessary. 2. Software development based on mathematical principles. A principal concept in Cleanroom software engineering is that a computer program is a rule for a mathematical function. The domain of the function is the set of all possible input histories, the co-domain of the function is the set of all possible outputs, and the range of the function is the set of all correct outputs. The Box Structure Method is used for specification and design. 3

4 3. Software testing based on statistical principles. The statistical testing approach to software treats the software like a statistical experiment. A statistical subset of all possible software uses is first generated. Performance on this subset is used to form conclusions about operational performance based on the usage model developed. The expected operational use is represented in a usage model of the software. Test cases are then randomly generated from the usage model. These tests are executed in an operational environment and failures are interpreted according to mathematical and statistical models. 5. The CMM reference model for Cleanroom software engineering The SEI has recognized that the CMM for software and the Cleanroom engineering process share a common concern with aspects of software quality and software development. The main focus of the CMM is on process management and the principles and practices associated with software process maturity. The main focus of Cleanroom is engineering practices using theory based engineering processes for the development and certification of software under statistical quality control. The SEI has developed a Cleanroom software engineering Reference Model (CRM) that provides a framework for developing a project or organization level Cleanroom process [3]. The CRM addresses 14 Cleanroom processes and 20 work products. Each of the Cleanroom principal functions has one or more Cleanroom processes associated with it. Each of these Cleanroom processes has one or more work products associated with it. The CRM is a guide to help Cleanroom project management and performance. The CRM is also useful for project performance assessment and improvement, and for technology transfer. Some Cleanroom processes do not have matching KPAs. Since the CMM does not address specific issues on matters relating to testing, for example, the Cleanroom processes that are associated with testing (i.e. Statistical test cases and usage models) are not addressed by any KPA. The SEI has also produced a Cleanroom implementation document which provides a more detailed mapping of the 14 defined Cleanroom processes to the 18 CMM key process areas [4]. Cleanroom software engineering implements a majority of the CMM for software development. The CRM exists as a high level template. Therefore, the Cleanroom processes and work products should be tailored for use by the specific project and/or organization. The actual implementation details and procedures are left to the individual project, and should be defined in the Cleanroom Engineering Guide or other equivalent document. The CMM mapping of the reference model provides a link to the specific KPA areas. Using a Cleanroom software engineering process in a CMM framework provides what Linger calls an enriched CMM and an enriched Cleanroom process. Linger, et al. state that the CMM is fundamentally about management, and Cleanroom is fundamentally about rigorous engineering. There is considerable overlap between the scopes of the two, and there are areas of each that are not addressed by the other [5]. The CMM, for example, has Key Process Areas (KPAs) at Level 2 (Repeatable) that are outside the scope of Cleanroom. Configuration Management and Subcontractor Management, for example, are important management issues not addressed by Cleanroom ideas. Cleanroom, on the other hand, enforces the mathematical basis of software development and the statistical basis of software testing, while the CMM is silent on the merits of various methods. In general, the CMM and Cleanroom are compatible and complementary. The combination of CMM management and organizational capabilities and Cleanroom technical practices represents a powerful process improvement paradigm. The correspondence between each of the KPAs and Cleanroom was performed using the following definitions; ; The CMM KPA is consistent with Cleanroom processes and implementation using Cleanroom processes is high. ; The CMM KPA is consistent with Cleanroom processes and implementation using Cleanroom processes is partial. Low; The CMM KPA is consistent with Cleanroom processes and implementation using Cleanroom processes is low. Consistent; The CMM KPA is consistent with Cleanroom processes but is not implemented in Cleanroom processes, or is implemented in an indirect way. Alternative; The CMM KPA is inconsistent with Cleanroom processes and an alternative method is defined by Cleanroom processes. The results from Linger et al. are shown in Table 2. 4

5 Table 2. KPA to Cleanroom Mapping KPA Level 2 KPAs (Repeatable) Software Configuration Management (SCM) Software Quality Assurance (SQA) Software subcontract management (SSM) Software project tracking and oversight (SPTO) Software project planning (SPP) Requirements management (RM) Level 3 KPAs (Defined) Peer reviews (PR) Inter-group communication (IC) Software product engineering (SPE) Integrated software management (ISM) Training program (TP) Organization process definition (OPD) Organization process focus. (OPF) Level 4 KPAs (Managed) Software quality management (SQM) Quantitative process management (QPM) Level 5 KPAs (Optimizing) Process change management (PCM) Technology change management (TCM) Defect prevention (DP) Consistent Consistent Cleanroom Consistency 6. An industrial application of Cleanroom and the CMM The CRM and the Cleanroom Implementation document provide guidelines for incorporating Cleanroom software engineering principles into organizations that are also using the CMM for software development. These guidelines have been successfully tailored in projects using the CMM and CSE [6,7]. The Cleanroom Reference Model describes fourteen Cleanroom processes with a set of proposed documentation artifacts (Table 3). This list of artifacts was tailored by the project organization by mapping them into a set of documents and artifacts consistent with the existing software development structure of the organization. Overall, the set of Cleanroom processes described in Table 3 was managed and executed successfully within our existing CMM software development approach. Overall, application of CSE within a mature CMM based framework resulted in synergism and few changes or tailoring of the existing process, as depicted in Table 4. The main Cleanroom practices were also successfully incorporated into the existing CMM based framework, as shown in Table 5. 5

6 Table 3. The 14 defined Cleanroom processes and the proposed and actual artifacts. Cleanroom Process Proposed CRM Artifacts Actual Project Artifacts Project Planning process Cleanroom Engineering Guide - Cleanroom process documentation. Software Standards and Procedures Manual - Cleanroom process documented as part of Software Software Development Plan - Plans for the project. Standards and Procedures Manual (SSPM) Software Development Plan - Contained plans for the use of Cleanroom for the project. Project Management process Project Record - Documentation of actions, reviews, other events in a project lifecycle. Performance Improvement process Performance Improvement plan - Documentation of plans to refine the Cleanroom process or introduce new tools. Engineering Change process Engineering Change log - Record of change proposals, evaluations, etc. Electronic Software Development Files (SDF) was the repository for all project artifacts. Lessons learned and causal analysis meetings and minutes stored in the SDF. History of all requirements changes, trade studies, etc. stored electronically in configured requirements database. Requirements analysis process Software requirements Software Requirements Specification. Interface Requirements Specification. Function Specification process Function Specification - Complete representation of the external view of the system, mapping all possible stimuli to responses. Usage Specification process Usage Specification - Description of expected users, uses, and environments. Architecture Specification process Software Architecture - Overall structure of the software and design strategies for development. Increment Planning process Increment construction plan - Specifies the number of increments for the software product, the functions in each increment, etc. Level 1 (CSCI level) Black box specifications, including sequence based enumeration and mapping tables stored in files on the SDF. Boundary diagram created and stored in Teamwork tool. Software Test Plan - contained information on users, uses, environments as well as other test planning information. Next lower level Computer Software Unit (CSU) of black box decomposition, including black box, mapping tables, and state box. Black boxes mapped to software tasks. Increment plan required to be presented at Software Requirements Review (SSR), Preliminary Design Review (PDR), and Critical Design Review (CDR). Contains information on functions for each increment, number of increments, schedule, etc. Requirements database links requirements to increments. Increment metric used to track requirements mapping to increment 6

7 Table 3. The 14 defined Cleanroom processes and the proposed and actual artifacts (continued). Cleanroom Process Proposed CRM Artifacts Actual Project Artifacts Software Re-engineering process Re-engineering plan - Technical plan for evaluation and possible reengineering of reused software. Software reuse plan - plans for reuse of software. Re-engineered software - Reused software reengineered as necessary. Increment Design process Increment design - Complete specification, design, code for a software increment. Correctness Verification process Increment verification report - Record of correctness verification reviews. Usage Modeling and Test planning process Statistical Testing and Certification process Usage Models - Models used to generate test cases. Increment Test plan - Information concerning statistical testing, schedule, models, etc. Statistical test cases - Randomly selected test cases from the usage models. Executable system - the load modules for testing. Statistical testing report - the record of the statistical tests run. Increment certification report - Record of measures of product quality and process control. Complete set of black box, state box, and clear box representations of the software stored in the SDF. Final clear box is the actual software. Separate schedule and work package maintained for each increment. Peer reviews of all correctness verification reviews stored on SDF. Software Test Description - details concerning test cases and test classes, test environment, usage models, schedule, etc. Software Test Procedure - step by step sequence of test execution steps and details of the test scripts generated from the usage models. Test cases included both random and hand crafted tests. Executable system. Test report - results of the testing process including both statistical and hand crafted test cases results. Reliability measures optional but not required. Table 4. Comparison of Cleanroom and an SEI Level 4/5 Process. Features of CSE requiring us to change or modify our SEI Level 4/5 process. Features of Cleanroom consistent with our SEI Level 4/5 Process. Inchstone-based progress management supported by Cleanroom Multiple increments supports causal analysis & process improvement Metrics support quantitative decision making. Disciplined software development approach Separate development and certification teams Incremental development (more and smaller increments). Peer reviews. (more frequent with less up front preparation). Main areas where Cleanroom was tailored. Algorithms/Signal Processing Software. Unit and structural testing part of the testing process Development team responsible for a clean compile People may transition between development and certification teams 7

8 Table 5. Cleanroom Practices/Implementation at the Introductory Level Cleanroom Practice Program Implementation Management & Team Operations Document Cleanroom Process Process documented (see Table 3) Set up teams Software work team responsible for CSCI Provide training Training plan developed and executed Cleanroom consultant regularly on-site Customer Interaction Shift to customer driven development Integrated Product Team organization with customer on teams Develop usage specifications with Usage model developed from customer documents and data customer Incremental Development Shift to incremental development CSCI development based on multiple internal increments Increment plan defined and adjusted during execution System Specification and Design Develop precise work specifications that Cleanroom artifacts developed and maintained during remain current throughout process development Design via stepwise refinement of Box structured decomposition incorporated into development specifications process Conduct frequent team reviews Peer review process shifted from a few Fagan style reviews to more frequent reviews using verification methods Correctness Verification Shift from unit testing to correctness verification Unit testing replaced with correctness verification. Tool support being developed. Statistical Testing and Reliability Certification Shift from coverage testing to usage testing Separate design from testing phase CSCI I&T performed using usage based statistical testing Use biased sample set of covering test cases initially 7. Lessons learned and project results Based on our experience to date, we believe Cleanroom is consistent with and supportive of an SEI level 4/5 software process. The evolution of our existing process and careful tailoring of Cleanroom practices resulted in a process that addressed stakeholder concerns. In particular, the Cleanroom concept of incremental development provided the software development teams the opportunity to improve the software development process at periodic intervals. After each increment, the software teams assess the current process, develop a list of lessons learned, determine the root cause of any process variances using causal analysis, and modify the process prior to the development of the next increment. This approach is consistent with the CMM KPA of continuous process improvement and was manifest in the improvement in the integration time of each increment (Figure 2). Man Months to Integrate Integration Time for Software Increments Increment Number Man Months to Integrate Figure 2. Improvement in the integration time for each increment for one software team 8

9 As mentioned previously, CSE does not include an independent Software Quality Assurance group since this function is largely performed by the CSE project teams themselves. CSE does incorporate to a very high degree the checks and reports required by the SQA KPA in the form of team reviews. In the implementation of CSE in our project, the responsibility for product quality shifted from the SQA group to the developers. SQA now supplies the development group with feedback on how the process is working. This information is developed by collecting metrics and performing audits of the software. This information is supplied to the development group throughout the development cycle. This process flow is more along the lines of continuous process improvement. The new role of SQA is also similar to the concept of statistical process control in many manufacturing operations. The customer/supplier relationship is different for the process (Figure 3). The Cleanroom process facilitates this new relationship. Code Walkthroughs Software Testing Verification Peer Reviews (SSPM peer review instructions) Unit Tests SQA Fix Software Configure Software Traditional Software Quality Engineering Process Metrics Process Monitoring Process Improvement Certification Fix Software SQA Configure Software Cleanroom Software Quality Engineering Process Qualification Test Qualification Test Figure 3. Traditional versus Cleanroom SQA process By focusing on one major technology insertion strategy at a time, we were able to assess its effects, and continue to do so as the project continues in operation. The challenge was to incorporate an established process such as Cleanroom into an already well-defined software process. Some of the lessons learned were: Just-in-time training of staff in new technology helped maintain momentum. Consultants kept the group focused and provided useful feedback. A careful assessment of the impact of the new technology on our existing process was required. A studied understanding of the new technology was required to adapt it to our own project effort. A mature process and team consciousness facilitated the new technology insertion. Tailoring the process, where appropriate, and using judicious application of new technology concepts led to overall acceptance with the practitioners as well as the customer. 8. Summary and future work The CMM is a development framework that describes the key elements for an effective software process. The CMM describes an evolutionary improvement path from an ad hoc, immature process to a mature, disciplined process. Cleanroom software engineering (CSE) is a managerial and engineering process for the development of high quality software with certified reliability. Cleanroom software engineering is a collection of principles, practices, and processes. The combination of CMM management and organizational capabilities and the judicious application of Cleanroom technical practices represents a powerful process improvement paradigm. These same principles; quality improvement, risk reduction, cost reduction, predictable process, and statistical quality control are also goals for many organizational system engineering processes. Our organization has aligned itself closely with the Systems Engineering Capability Maturity Model (also developed by the SEI) for development of high quality systems. The SEI developed the SE-CMM, recognizing the need for corporations to efficiently translate customers needs into a product that meets those needs. The SE-CMM provides a mechanism to measure and enhance the performance in systems engineering, a key activity in translating customers needs into products. The main idea behind this approach is the quality of a product is a direct function of the process and technology used to develop the product, and the people performing the work on the project [8]. The systems engineering process group has developed a system development process that defines the interacting tasks that must be performed during product architecture, development, and production (Figure 4). This model has several consistencies with the Cleanroom methodology, in particular the increment planning approach. In the systems engineering process in Figure 4, the system is decomposed into a set of system development builds that include design, production, and integration. The incremental development performed in CSE performs the same three tasks, with the final increment being the entire integrated product. Future work in the industrial 9

10 application of Cleanroom will attempt to integrate Cleanroom best practices into the system engineering development process. [8] Bate, Roger, et al, A Systems Engineering Capability Maturity Model, Version 1.0, Software Engineering Institute, December 1994 Train and Develop People and Teams Define and execute Plan Plan system builds Manage system builds and manage risk business strategy Program and plan risk Capture Design Mature and validate requirements program system Develop core products and technology Develop supplier capability Develop prototype build Design components and process Produce Design components and process Produce Design components and process Produce Produce production components Integrate Integrate Integrate Integrate System development builds Produce Build and Delivery Product Architecure Product Development Production Figure 4. Systems Engineering Development Process References [1] Paulk, Mark C., Bill Curtis, Mary Beth Chrissis, Charles V. Weber, Capability Maturity Model for Software, v1.1, 1993 [2] Becker, Shirley A., James A. Whittaker, Cleanroom Software Engineering Practices, Idea Group Publishing, copyright [3] Linger, R.C. and Carmen J. Trammell, The SEI Cleanroom Software Engineering Reference Model and its Implementation for the CMM for Software, Proceedings of the 3rd Annual International Conference on Cleanroom Software Engineering Practices, October [4] Linger, R.C., Mark C. Paulk, and Carmen J. Trammell, Cleanroom Software Engineering Implementation of the Capability Maturity Model for Software, Software Engineering Institute, December [5] Linger, R.C. and Carmen J. Trammell, Cleanroom Software Engineering Reference Model, Version 1.0, Software Engineering Institute, November [6] Kelly, D.P, and Robert S. Oshana. Integrating Cleanroom Software Methods into an SEI Level 4-5 Program, Crosstalk, November [7] Sherer, S. Wayne, Paul G. Arnold, Ara Kouchakdjian, Successful Process Improvement Effort Using Cleanroom Software Engineering, Proceedings of the 6th Software Technology Conference, May