Enabling Economics-Driven System Development through Return-on-Investment Analysis of Software Defect Prevention

Transcription

1 47th AIAA Aerospace Sciences Meeting Including The New Horizons Forum and Aerospace Exposition 5-8 January 29, Orlando, Florida AIAA Enabling Economics-Driven System Development through Return-on-Investment Analysis of Software Defect Prevention Richard W. Selby 1 Head of Software Products, Northrop Grumman Space Technology, Redondo Beach, California, 9278 Rick.Selby@NGC.com, Adjunct Professor, Computer Science Department, University of Southern California Economics-driven system development utilizes strategic frameworks, models, and data to identify options, evaluate benefits, understand tradeoffs, and execute decisions to help enable successful system development. Return-on-investment analysis methods provide one perspective for quantifying the economic value from system development techniques. This research investigates the usefulness of return-on-investment analysis methods for software engineering and applies them to evaluate the benefits from software peer reviews for defect prevention. This work also defines the disciplined process steps that constitute the particular software peer review technique used in the environment that provides data for this study. 1. INTRODUCTION System developers continue to pursue improvements in development methods, and their understanding of these methods can benefit by quantifying the economic value that results from their application [Boe81]. Return-on-investment analysis techniques provide a structured approach for using measurement-driven methods to understand, model, and evaluate the relationships between software development methods and economic benefits. This research investigates the usefulness of return-on-investment analysis methods for software engineering and applies them in an example empirical analysis to evaluate the benefits from software peer reviews for defect prevention [Bas84] [Kit2]. 2. SOFTWARE PEER REVIEW AND SOFTWARE REWORK REDUCTION PROCESS This study investigates the effectiveness of software defect detection and prevention using peer reviews in a development environment for large-scale systems [Tic98]. These system development projects embrace a software rework reduction best practice that incorporates peer reviews as well as two related process elements to decrease software defects and consequently reduce rework: root cause analysis and closed loop feedback (see Figure 1). This best practice stitches together many process steps in order to emphasize software rework reduction, and it is complementary with CMMI as well as other process improvement initiatives such as Six Sigma [Fag76] [Hum88] [Myr5]. Appendix 1 defines common definitions for terminology used in this software rework reduction process. The key principles in this software rework reduction practice are: Adopt standard peer review process with moderator and reviewer roles and responsibilities defined. Apply objective peer review checklists for each lifecycle phase. Use automated tool infrastructure for defect data capture. Analyze trends, evaluate predictions, and control using quantitative management. Conduct project- and lifecycle-specific root cause analysis and continuous improvement to decrease defects and reduce rework. Identify project milestones and control limits for defect reduction within the project lifecycle. Figure 1 defines the overall process flow and combines the following process elements that, among other benefits, have synergies with CMMI: 1. Define overall method for rework reduction (synergy with CMMI Level 2 Project Planning). 2. Plan peer review (synergy with CMMI Level 3 Verification). 3. Conduct peer review (synergy with CMMI Level 3 Verification). 4. Conduct root cause analysis (synergy with CMMI Level 5 Causal Analysis and Resolution). 1 AIAA Member 1 Copyright 29 by Copyright 28 by Richard W. Selby. All rights reserved. Published by the, Inc., with permission.

2 5. Conduct closed loop feedback (synergy with CMMI Level 5 Organizational Innovation and Deployment). 6. Ensure compliance and institutionalization (synergy with CMMI Level 4 Quantitative Project Management and Level 4 Organizational Process Performance). Start 1. Define overall method for rework reduction For each software delivery/phase For each peer review 2. Plan peer review Feedback 4. Conduct root cause analysis No 3. Conduct peer review More peer reviews? Yes 5. Conduct closed loop feedback 6. Ensure compliance and institutionalization No More software deliveries/phases? Yes End Figure 1. Software peer review and software rework reduction process. 3. DETAILED STEPS FOR PEER REVIEW AND REWORK REDUCTION PROCESS The detailed steps for the software peer review and software rework reduction process are outlined in the following subsections. 3.1 Define Overall Method for Rework Reduction Establish and maintain a process/plan for rework reduction including peer reviews and defect prevention activities, including any improvements from feedback resulting from previous peer reviews and defect prevention activities. Assign responsibility and authority for managing the peer review and defect prevention process. Provide resources for the peer reviews and defect prevention activities. Train the personnel on the peer review and defect prevention process. Identify work products that are subject to peer reviews, and identify when in the lifecycle the work products are to be reviewed. Define the procedure to be used for each peer review. Define evaluation criteria for each work product using checklists. Define criteria for when repeating the peer review is required such as based on the number of defects and/or the amount the product needs to be reworked. Prepare tools such as process flow tools to monitor progress, action item databases to track closure status, and static analyzers to evaluate standards conformance. 3.2 Plan Peer Review Identify work product to be peer reviewed and the scope of the review, including any improvements from feedback resulting from previous peer reviews. 2

3 Assign lead reviewer, supporting reviewers, moderator, observers, and author. Identify and involve relevant stakeholders, including Software Quality Assurance (SQA). SQA does not attend every review. Prepare work products to be peer reviewed and any supporting materials. Ensure products are compliant with standards prior to the review. Prepare logistics such as review location, schedule, and duration. Announce peer review and distribute review materials at least one to two weeks in advance. Direct reviewers to evaluate the review materials in advance of the peer review with a minimum individual preparation time, such as one to four hours. The level of preparation time depends on the work product type and size. Manage versions and configurations of peer review materials. 3.3 Conduct Peer Review Moderator starts the review by collecting all action items identified by individual reviewers prior to the peer review. Lead reviewer coordinates discussion of the work product and elicits feedback from reviewers through structured discussions. Reviewers ensure the work product is compliant with process and product specifications and standards using checklists that are specific to work product types such as software requirements, designs, code, tests, or development plans. Moderator records action items, including defects and non-defects such as enhancements. At the conclusion of the review, determine whether a re-review is necessary using the criteria established in the peer review plan. The moderator distributes minutes to attendees and other relevant stakeholders including review scope, date, attendees, decisions/conclusions, issues, and action items including severity. After the peer review, the moderator ensures that the action items are tracked to closure. The moderator collects metrics, such as reviewer preparation time, actual meeting duration, number of action items, time spent closing action items, etc. 3.4 Conduct Root Cause Analysis Before completing each software delivery, conduct root cause analysis for high severity defects and out-of-phase defects. Analyze the root cause of high severity defects. For out-of-phase defects, determine why the defects were not detected earlier. For high severity or out-of-phase defects, determine how the defects could have been prevented. Investigate whether other projects or organizations have experienced similar defects. Analyze metrics from the peer review preparation, execution, and closure activities as well as the defects detected, determine methods for making the peer reviews more effective/efficient, and compare metrics to appropriate project and organizational baselines. 3.5 Conduct Closed Loop Feedback Implement defect prevention countermeasures to facilitate long-term learning. Based on the root cause analysis, implement the steps that would have prevented the defects from occurring such as adding/changing process steps, improving static analysis tools, or updating checklists. Revise the peer review processes and plans as needed, with a focus on benefiting the current project as well as concurrent and future projects. Revise organizational policies and processes as needed based on the improvements resulting from the peer review analysis. Add lessons learned to knowledge base and incorporate them into training materials. 3.6 Ensure Compliance and Institutionalization Monitor and control the peer review process performance using statistical process control techniques such as control charts. Recalculate project and organizational baselines such as upper and lower control limits as needed. Conduct objective audits for process and product compliance. Review process status and improvement activities with higher level management. 3

4 Collect improvement suggestions from peer review attendees and other relevant stakeholders. Revise the peer review processes and plans as well as organizational policies and processes as needed to incorporate improvements. 4. ROLES AND RESPONSIBILITIES The key roles and responsibilities for the software peer review and software rework reduction process are as follows. Process owner -- Establishes and maintains rework reduction and peer review policies, processes, and supporting artifacts including checklists. Rework reduction planner -- Develops software rework reduction plan that defines processes, activities, and tools to be used to decrease defects, foster defect prevention, and reduce rework. Work product author -- Creates work products and supporting materials to be peer reviewed. Lead reviewer -- Provides technical leadership of the peer review and has overall responsibility for the process and technical integrity of the peer review. Reviews and evaluates the work products and generates action items before and during peer review. Moderator -- Provides administrative leadership of the peer review. Distributes announcement and review materials, records and distributes minutes. Functions as a recorder of action items. Supporting reviewers -- Reviews and evaluates the work products and generate action items before and during peer review. Observers -- Attends peer review to learn about reviews and product development, but are not expected to review the work products or generate action items. Software Quality Assurance -- Audits peer reviews to ensure compliance to process and product specifications and standards. Root cause analyst -- Conducts root cause analysis of defects in order to understand causes and defines corrective actions and preventative actions. Quantitative analyst -- Conducts quantitative analysis of metrics in order to identify statistical trends and outliers using methods such as control charts. Lessons learned coordinator -- Coordinates with stakeholders and captures lessons learned from rework reduction activities in order to ensure institutional learning such as through improved work methods, tools, documentation, web sites, and training. 5. RETURN-ON-INVESTMENT ANALYSIS This study investigates the effectiveness of software defect detection and prevention using peer reviews across 12 system development phases on nine large-scale systems. The systems are large-scale mission-critical embedded software systems that implement functionality for a wide variety of electronics, flight control, and ground control systems. Experienced developers apply domain-specific architectures, modular components, and mature development processes to design, develop, and enhance these systems in incremental software lifecycles. System sizes range from 75, to over 2,, source lines, and teams include between 1 to over 2 developers. This investigation analyzes 2621 defects from 257 peer reviews and benchmarks the defect injection and detection performance across the 12 system development phases (see Figure 2). Return-on-investment analysis techniques calculate favorable ratios of at least 9:1 in terms of net cost avoidance relative to non-recurring costs for software defect prevention (see Figure 3). On some projects, the return-on-investment ratios exceeded 3:1 because high severity defects were detected very early in the projects (see Figure 4). 4

5 # Reviews # Defects # Reviews # Defects CTIS Figure 2. Software peer reviews conducted and defects detected. E2 C Hawkeye JSF DAS LAIRCM NMS GFC/C B-2 Support JSTARS ASU NPOESS # Defects ROI # Defects LAIRCM B-2 Support CTIS Figure 3. Return-on-investment (ROI) calculations for software peer reviews in terms of net cost avoidance relative to non-recurring costs. JSTARS ASU ROI 5

6 4,5 4, 3,5 3, ROI 2,5 2, 1,5 1, 5 % 2% 4% 6% 8% 1% Requirements+Design Defects Figure 4. Return-on-investment (ROI) versus percentage defects detected early in the lifecycle during requirements and design. 6. CONCLUSIONS Economics-driven system development utilizes strategic frameworks, models, and data to identify options, evaluate benefits, understand tradeoffs, and execute decisions to facilitate successful system development. To help enable economicsdriven system development, this study examines return-on-investment analysis methods for quantifying the economic value from system development techniques. For software peer reviews, calculations of simple ratios of net cost avoidance relative to non-recurring costs showed favorable ratios of at least 9:1. Future research applies these return-on-investment analysis methods to broader aspects of system development lifecycles and techniques in order to understand and characterize the engineering tradeoffs in large-scale systems. APPENDIX 1. TERMINOLOGY DEFINITIONS FOR SOFTWARE REWORK REDUCTION PROCESS Defect -- A non-conformance between a work product and its process or product specification or standard. Peer review -- A peer review is a meeting whose purpose is to detect defects in a work product. The meeting assembles experienced peers of a work product author, and they follow a structured objective process using checklists and group discussions. A peer review typically involves about six to eight attendees and lasts two to four hours excluding time for preparation and action item closure. Root cause analysis -- An analytic activity that investigates the cause of a defect so that the underlying cause is well enough understood that actions can be defined to prevent the defect from occurring in the future. Closed loop feedback -- The implementation of defect prevention actions resulting from root cause analysis of defects. Work product -- A work product is the artifact being evaluated during a peer review, and it can be comprised of software requirements, designs, code, tests, or development plans. EKSLOC -- EKSLOC is equivalent 1s of source lines of code, and this metric measures the quantity of work product. This metric counts logical lines, not physical lines, of source code. When the work product being peer reviewed is not source code (such as in a requirements peer review), the metric may use the corresponding equivalent source lines of code or a phase-specific metric such as number of requirement shalls. Development phase -- A development phase is a separately identifiable segment within an overall software project. A simple set of development phases may include just the following four phases: requirements, design, code, and test. A more complete set of development phases may include the following 12 phases: proposal, external (non-software) requirement source, requirements, preliminary design, detailed design, code, unit test, integration test, verification, 6

7 support to system I&T, maintenance, and operations. Detection phase -- The development phase in which a defect is detected. Injection phase -- The development phase in which a defect is injected into (i.e., originates in) a work product. In-phase defect -- A defect whose injection phase and detection phase are the same. Out-of-phase defect -- A defect whose injection phase and detection phase are different. Defect severity -- Defect severity is based on a severity classification scheme that uses a differentiated scale to capture the overall impact of a defect. A simple severity scale uses two levels called major and minor where major means that a defect limits user-visible software functionality. Another example severity scale uses four levels: 1 means that the software is inoperable, 2 means that there are functionality limitations without workaround, 3 means that there are functionality limitations with workaround, and 4 means cosmetic problems. Another example severity scale uses five levels based on the project impact if the defect had not been detected: 1 means more than six weeks impact, 2 means four-six weeks impact, 3 means two-four weeks impact, 4 means one-two weeks impact, and 5 means less than one week impact (this impact is the schedule impact to the software project not to the overall program in which the software is just one element). High severity defect -- Defect that has major severity or 1, 2, or 3 severity (from either the four-level or five-level scale). Defect density -- Defect density is a ratio of the number of defects to the quantity of work product (commonly expressed as defects per EKSLOC). Control chart -- A graphical metric display that uses time on the horizontal axis and a product quality metric (such as defect density) or a process quality metric (such as defect closure cycletime) on the vertical axis. The display includes an upper control limit and a lower control limit that define statistical boundaries for the data points based on plus or minus two to three standard deviations from the mean. Corrective action -- The action taken to correct a defect in a work product. Preventative action -- The action taken to prevent a defect from occurring in future work products. Stakeholder -- A stakeholder is anyone who has a vested interest in the successful outcome of a work product. Stakeholders can include customers, users, system engineers, requirements analysts, software developers, hardware developers, testers, operations personnel, quality assurance personnel, process owners, and managers. Peer review announcements typically include stakeholders, and they may serve as reviewers or observers or simply track progress to stay informed. REFERENCES [Bas84] V. R. Basili and D. M. Weiss, A Methodology for Collecting Valid Software Engineering Data, IEEE Transactions on Software Engineering, Vol. SE-1, No. 6, November 1984, pp [Boe81] B. W. Boehm, Software Engineering Economics, Prentice-Hall, Englewood Cliffs, NJ, [Fag76] M. E. Fagan, "Design and Code Inspections to Reduce Errors in Program Development," IBM Systems Journal, No. 3, 1976, pp [Hum88] W. S. Humphrey, Characterizing the Software Process: A Maturity Framework, IEEE Software, Vol. 5, No. 2, March 1988, pp [Kit2] B. A. Kitchenham, S. L. Pfleeger, L. M. Pickard, P. W. Jones, D. C. Hoaglin, K. E. Emam, and J. Rosenberg, "Preliminary Guidelines for Empirical Research in Software Engineering," IEEE Transactions on Software Engineering, Vol. SE-28, No. 8, 22. [Myr5] I. Myrtveit, E. Stensrud, and M. Shepperd, "Reliability and Validity in Comparative Studies of Software Prediction Models," IEEE Transactions on Software Engineering, Vol. SE-31, No. 5, May 25, pp [Tic98] W. F. Tichy, "Should Computer Scientists Experiment More?," IEEE Computer, Vol. 31, No. 5, 1998, pp BIOGRAPHY Richard W. Selby is the Head of Software Products at Northrop Grumman Space Technology in Redondo Beach, CA. He leads a 25-person software organization and has served in this position since 21. Previously, he was the Chief Technology Officer and Senior Vice President at Pacific Investment Management Company (PIMCO) in Newport Beach, CA where he managed a 15-person organization for three years. From , he was a Full Professor of Information and Computer Science (with tenure) at the University of California in Irvine, CA (UC Irvine). Since 24, he has been an Adjunct Full Professor of Computer Science at the University of Southern California (USC) in Los Angeles, CA. In 1993, he held visiting faculty positions at the Massachusetts Institute of Technology (MIT) Laboratory for Computer Science and Sloan School of Management in Cambridge, MA, and in 1992, he held a visiting faculty position at the Osaka University Department of Computer Science in Osaka, Japan. His research focuses on development, management, and economics of large-scale mission-critical systems, software, and processes. He has authored over 115 refereed publications, authored or edited four books, and given over 245 presentations at professional meetings. He co-authored an international best-selling book that analyzed Microsoft s technology, strategy, 7

8 and management that was entitled Microsoft Secrets: How the World s Most Powerful Software Company Creates Technology, Shapes Markets, and Manages People. The book, written with Michael Cusumano, has been translated into 12 languages, has 15, copies in print, and was ranked as a #6 best-seller in Business Week. He also edited the book Software Engineering: Barry W. Boehm s Lifetime Contributions to Software Development, Management, and Research. At Northrop and PIMCO, he was responsible for the successful development and management of over 55 products that had over 33, system requirements and over 11,, source-lines-of-code. At Northrop, he led the $3 billion company to a successful enterprise-wide rating of Capability Maturity Model Integration (CMMI) Level 5 for Software. He served as the Software Integrated Project Team (IPT) Manager and/or Chief Software Engineer for the NASA Prometheus spacecraft to Jupiter, NASA Crew Exploration Vehicle (CEV) replacement for the Space Shuttle, and Air Force Advanced Extremely High Frequency (AEHF) satellite communications payload system. At PIMCO, he led the $1 billion company to be ranked as the fourth most innovative technology organization in financial services, according to Wall Street & Technology. He received his Ph.D. and M.S. degrees in Computer Science from the University of Maryland, College Park, MD in 1985 and 1983, respectively. He received his B.A. degree in Mathematics from St. Olaf College, Northfield, MN in