Praise for The Economics of Software Quality

Transcription

1

2 Praise for The Economics of Software Quality This book provides the best treatment on the subject of economics of software quality that I ve seen. Peppered with valuable industry data, in-depth analysis, empirical methods for quality improvement, and economic analysis of quality, this book is a must-read for anyone who is interested in this subject. With the many real-life and up-to-date examples and stories linking software quality to daily-life activities, readers will find this book an enjoyable read. Stephen H. Kan, Senior Technical Staff Member and Program Manager, Software Quality IBM Systems and Technology Group, and author of Metrics and Models in Software Quality Engineering Finally, a book that defines the cost and economics of software quality and their relationship to business value. Facts such as the inability of testing alone to produce quality software, the value of engineering-in quality, and the positive ROI are illustrated in compelling ways. Additionally, this book is a mustread for understanding, managing, and eliminating technical debt from software systems. Dan Galorath, CEO, Galorath Incorporated & SEER by Galorath Congrats to Capers and Olivier as they release their relevant, extensive, and timely research on the costs of defects in today s software industry. The authors don t stop with the causes of defects; they explore injection points, removal, and prevention approaches to avoid the technical mortgage associated with defective software products. In today s quick-to-market world, an emphasis on strengthening the engineering in software engineering is refreshing. If you re a software developer, manager, student, or user, this book will challenge your perspective on software quality. Many thanks! Joe Schofield, Sandia National Laboratories; Vice President, IFPUG; CQA, CFPS, CSMS, LSS BB, SEI-certified instructor Whether consulting, working on projects, or teaching, whenever I need credible, detailed, relevant metrics and insights into the current capabilities and performance of the software engineering profession, I always turn to Capers Jones s work first. In this important new book, he and Olivier Bonsignour make

3 the hard-headed, bottom-line, economic case, with facts and data, about why software quality is so important. I know I ll turn to this excellent reference again and again. Rex Black, President, RBCS ( and author of seven books on software quality and testing, including Managing the Testing Process, Third Edition This masterpiece of a book will empower those who invest in software and the businesses and products that depend on it to do so wisely. It is a groundbreaking work that rigorously applies principles of finance, economics, management, quality, and productivity to scrutinize holistically the value propositions and myths underlying the vast sums invested in software. A mustread if you want to get your money s worth from your software investments. Leon A. Kappelman, Professor of Information Systems, College of Business, University of North Texas Capers Jones is the foremost leader in the software industry today for software metrics. The Economics of Software Quality is a comprehensive, data-rich study of challenges of quality software across the many application domains. It is an essential read for software quality professionals who wish to better understand the challenges they face and the cost and effectiveness of potential solutions. It is clear that much research and thought has been put into this. Maysa-Maria Peterson Lach, Senior Principal Software Engineer, Raytheon Missile Systems In no other walk of life do we resist the necessity and validity of precise, rigorous measurement, as software practitioners have so vigorously resisted for more than fifty years. Capers Jones took up the challenge of bringing sanity and predictability to software production more than three decades ago, and now with Olivier Bonsignour, he brings forth his latest invaluable expression of confidence in applying standard engineering and economic discipline to what too often remains the Wild, Wild West of software development. Douglas Brindley, President & CEO, Software Productivity Research, LLC

4 The Economics of Software Quality

5 This page intentionally left blank

6 The Economics of Software Quality Capers Jones Olivier Bonsignour Upper Saddle River, NJ Boston Indianapolis San Francisco New York Toronto Montreal London Munich Paris Madrid Capetown Sydney Tokyo Singapore Mexico City

7 Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have been printed with initial capital letters or in all capitals. The authors and publisher have taken care in the preparation of this book, but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential damages in connection with or arising out of the use of the information or programs contained herein. The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales, which may include electronic versions and/or custom covers and content particular to your business, training goals, marketing focus, and branding interests. For more information, please contact: U.S. Corporate and Government Sales (800) corpsales@pearsontechgroup.com For sales outside the United States, please contact: International Sales international@pearson.com Visit us on the Web: informit.com/aw Library of Congress Cataloging-in-Publication Data Jones, Capers. The economics of software quality / Capers Jones, Olivier Bonsignour p. cm. ISBN (hardcover: alk. Paper) 1. Computer software Quality control Economic aspects. 2. Software maintenance Economic aspects. 3. Computer software Validation. 4. Computer software verification. I. Subramanyam, Jitendra. II. Title. QA76.76.Q35J dc Publisher Paul Boger Acquisitions Editor Bernard Goodwin Managing Editor John Fuller Full-Service Production Manager Julie B. Nahil Copy Editor Christal White Indexer Infodex Indexing Services Proofreader Christine Clark Editorial Assistant Michelle Housley Cover Designer Nicolas Herlem Compositor LaurelTech Copyright 2012 Pearson Education, Inc. All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permission must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For information regarding permissions, write to: Pearson Education, Inc. Rights and Contracts Department 501 Boylston Street, Suite 900 Boston, MA Fax: (617) ISBN-13: ISBN-10: Text printed in the United States on recycled paper at Courier in Westford, Massachusetts. First printing, July 2011

8 This book is dedicated to Watts Humphrey and Allan Albrecht. Watts was a tireless champion of software quality. Allan developed the most effective metric for studying software quality economics.

9 This page intentionally left blank

10 Contents Foreword xix Preface xxi Acknowledgments xxvii About the Authors xxxi Chapter 1: Defining Software Quality and Economic Value 1 Introduction Why Is Software Quality Important? Defining Software Quality Defining Economic Value and Defining the Value of Software Quality The Economic Value of Software and Quality to Enterprises that Build Internal Software for Their Own Use The Economic Value of Software and Quality to Internal Software Users The Economic Value of Software and Quality to Commercial Software Vendors The Economic Value of Software and Quality to COTS Users and Customers The Economic Value of Software and Quality to Embedded Software Companies The Economic Value of Software and Quality to Embedded Equipment Users The Economic Value of Software and Software Quality to Other Business Sectors Multiple Roles Occurring Simultaneously Summary and Conclusions Chapter 2: Estimating and Measuring Software Quality 35 Introduction Using Function Point Metrics for Defect Potentials Software Defect Potentials ix

11 x Contents The Special Case of Software Requirements The Origins of Software Requirements The Size, Structure, and Completeness of Software Requirements Minimizing Software Requirements Defects Conclusions about Software Requirements Defects The Special Case of Coding Defects Estimating Software Defect Prevention Estimating Software Defect Detection and Defect Removal Measuring Application Structural Quality Measuring Reliability Measuring Performance Efficiency Measuring Security Measuring Maintainability Measuring Size Summary of Application Structural Quality Measurement Attributes Examples of Structural Quality Assessments Bypassing the Architecture Failure to Control Processing Volumes Application Resource Imbalances Security Weaknesses Lack of Defensive Mechanisms Desiderata for Systems Evaluating Structural Quality Three Problems That Distort Software Economic Analysis Leakage from Software Historical Data Economic Problems with Lines of Code (LOC) Metrics Economic Problems with Cost-per-Defect Metrics Case A: Poor Quality Case B: Good Quality Case C: Zero Defects Useful Rules of Thumb for Predicting Software Defect Potentials Summary and Conclusions on Software Quality Estimation and Measurement

12 Contents xi Chapter 3: Software Defect Prevention 119 Introduction The Early History of Defect Prevention Studies in the 1970s at IBM Synergistic Combinations of Defect Prevention Methods Defect Potentials and Defect Origins Defect Prevention, Patterns, and Certified Reusable Materials Software Defect Prevention and Application Size Analysis of Defect Prevention Results Agile Embedded Users Automated Quality Predictions Benchmarks of Software Quality Data Capability Maturity Model Integrated (CMMI) Certification Programs Cost-per-Defect Measures Cost of Quality (COQ) Cyclomatic Complexity Measures (and Related Complexity Measures) Defect Measurements and Defect Tracking Formal Inspections Function Point Quality Measures ISO Quality Standards, IEEE Quality Standards, and Other Industry Standards Quality Function Deployment (QFD) Risk Analysis Six Sigma Static Analysis Summary and Conclusions of Software Defect Prevention Chapter 4: Pretest Defect Removal 191 Introduction Small Project Pretest Defect Removal Large System Pretest Defect Removal Analysis of Pretest Defect Removal Activities Personal Desk Checking Informal Peer Reviews

13 xii Contents Automated Text Checking for Documents Proofs of Correctness Scrum Sessions Poka Yoke Kaizen Pair Programming Client Reviews of Specifications Independent Verification and Validation (IV&V) Software Quality Assurance (SQA) Reviews Phase Reviews Inspections (Requirements, Architecture, Design, Code, and Other Deliverables) User Documentation Editing and Proofreading Automated Static Analysis of Source Code Summary and Conclusions about Pretest Defect Removal Chapter 5: Software Testing 279 Introduction Black Box and White Box Testing Functional and Nonfunctional Testing Automated and Manual Testing Discussion of the General Forms of Software Testing Subroutine Testing PSP/TSP Unit Testing Extreme Programming (XP) Unit Testing Unit Testing New Function Testing Regression Testing Integration Testing System Testing The Specialized Forms of Software Testing Stress or Capacity Testing Performance Testing Viral Protection Testing Penetration Testing Security Testing Platform Testing Supply Chain Testing Clean Room Testing

14 Contents xiii Litigation Testing Cloud Testing Service Oriented Architecture (SOA) Testing Independent Testing Nationalization Testing Case Study Testing The Forms of Testing Involving Users or Clients Agile Testing Usability Testing Field Beta Testing Lab Testing Customer Acceptance Testing Test Planning Test Case Design Methods Errors or Bugs in Test Cases Numbers of Testing Stages for Software Projects Testing Pattern Variations by Industry and Type of Software Testing Pattern Variations by Size of Application Testing Stages Noted in Lawsuits Alleging Poor Quality Using Function Points to Estimate Test Case Volumes Using Function Points to Estimate the Numbers of Test Personnel Using Function Points to Estimate Testing Effort and Costs Testing by Developers or by Professional Test Personnel Summary and Conclusions on Software Testing Chapter 6: Post-Release Defect Removal 347 Introduction Post-Release Defect Severity Levels Severity Levels from a Structural Quality Perspective Maintainability of Software Defect Discovery Rates by Software Application Users Invalid Defect Reports

15 xiv Contents Abeyant Defects That Occur Under Unique Conditions Duplicate Defects Reported by Many Customers First-Year Defect Discovery Rates Measuring Defect Detection Efficiency (DDE) and Defect Removal Efficiency (DRE) Variations in Post-Release Defect Reports Variations in Methods of Reporting Software Defects Who Repairs Defects after They Are Reported? Case Study 1: Development Personnel Tasked with Maintenance Defect Repairs Case Study 2: Maintenance Specialists Handle Defect Repairs Comparing the Case Studies Litigation Due to Poor Quality Cost Patterns of Post-Release Defect Repairs Software Occupation Groups Involved with Defect Repairs Examining the Independent Variables of Post-Release Defect Repairs The Size of the Application in Function Points Error-Prone Modules in Software Applications User and Industry Costs from Post-Release Defects Impact of Security Flaws on Corporations and Government Agencies Customer Logistics for Defect Reports and Repair Installation Case Study 1: A Small Application by a Small Company Case Study 2: A Large Application by a Large Company Measurement Issues in Maintenance and Post-Release Defect Repairs Summary and Conclusions on Post-Release Defects Chapter 7: Analyzing the Economics of Software Quality 433 Introduction The Economic Value of Software

16 Contents xv Methods of Measuring Value Funding Approval and Application Size The Impact of Software Construction Difficulties on Software Quality Revenue Generation from Software Difference Between Software and Other Industries Cost Reduction from Software Economic Impact of Low-Quality and High-Quality Software Software Development and Maintenance Software as a Marketed Commodity Software as a Method of Human Effort Reduction Software and Innovative New Kinds of Products Technical Debt A Measure of the Effect of Software Quality on Software Costs A Framework for Quantifying Business Value Moving Beyond Functional Quality The Impact of Software Structure on Quality The Impact of Staff Training on Quality The Impact of Professional Certification on Quality The Impact of Technology Investment on Quality The Impact of Project Management on Quality The Impact of Quality-Control Methodologies and Tools on Quality The Impact of High and Low Quality on Software Schedules The Impact of High and Low Quality on Software Staffing The Impact of High and Low Quality on Software Development Effort The Impact of High and Low Quality on Development Productivity Rates The Impact of High and Low Quality on Software Development Costs The Impact of High and Low Quality on Development Cost per Function Point The Impact of High and Low Quality on Project Cancellation Rates

17 xvi Contents The Impact of High and Low Quality on the Timing of Cancelled Projects The Impact of High and Low Quality on Cancelled Project Effort The Impact of High and Low Quality on Effort Compared to Average Projects The Impact of High and Low Quality on Software Test Stages The Impact of High and Low Quality on Testing as a Percent of Development The Impact of High and Low Quality on Test Cases per Function Point The Impact of High and Low Quality on Numbers of Test Cases Created The Impact of High and Low Quality on Test Coverage The Impact of Professional Testers on High and Low Quality The Impact of High and Low Quality on Software Defect Potentials The Impact of High and Low Quality on Total Software Defects The Impact of High and Low Quality on Defect Detection Efficiency (DDE) The Impact of High Quality and Low Quality on Defect Removal Efficiency (DRE) The Impact of High and Low Quality on Total Defect Removal The Impact of High and Low Quality on Defects Delivered to Customers The Impact of High and Low Quality on Delivered Defects per Function Point Impact of High and Low Quality on Delivered Defect Severity Levels The Impact of High and Low Quality on Severe Defects per Function Point The Impact of High and Low Quality on Software Reliability

18 Contents xvii The Impact of High and Low Quality on Maintenance and Support The Impact of High and Low Quality on Maintenance and Support Costs The Impact of High and Low Quality on Maintenance Defect Volumes The Impact of High and Low Quality on Software Enhancements The Impact of High and Low Quality on Enhancement Costs The Impact of High and Low Software Quality on Maintenance and Enhancement Staffing The Impact of High and Low Quality on Total Effort for Five Years The Impact of High and Low Quality on Total Cost of Ownership (TCO) The Impact of High and Low Quality on Cost of Quality (COQ) The Impact of High and Low Quality on TCO and COQ per Function Point The Impact of High and Low Quality on the Useful Life of Applications The Impact of High and Low Quality on Software Application Tangible Value The Impact of High and Low Quality on Return on Investment (ROI) The Impact of High and Low Quality on the Costs of Cancelled Projects The Impact of High and Low Quality on Cancellation Cost Differentials The Distribution of High-, Average-, and Low-Quality Software Projects Summary and Conclusions on the Economics of Software Quality High-Quality Results for 10,000 Function Points Low-Quality Results for 10,000 Function Points References and Readings 545 Index 561

19 This page intentionally left blank

20 Foreword As a major telecommunications company, our business consists of a complex mix of products and services. Some are decades old, and some are only emerging. In just one part of our business, customers now access sophisticated business processes via myriad mobile devices operating on multiple platforms, technologies, and standards. The mobile access revolution is only one example of the continual change we must master. In several of our new markets, some of our competitors were not even on the radar ten years ago. The IT systems that service our customers have been built over decades of changing regulatory frameworks, intense competition, and M&A activity; these systems provide mission-critical network management, billing, and customer service infrastructure for our existing and emerging products. We simply don t have the luxury of crafting Greenfield solutions in response to pressing business needs. Despite the complex nature of IT, our shareholders expect nothing less than continuous improvement in service quality with simultaneous cost reductions. This has been the case in our market for quite some time and a major operational focus for my organization. One area on which we have focused in addressing this challenge is measuring software development productivity and quality. As the CIO, I oversee the company s internal information technology organization and infrastructure, as well as all evolving software applications. When you get down to it, the core expertise of our business is encoded in the software that automates our mission-critical processes. It is that software layer of our IT stack that fundamentally drives our time to market, our risk profile, and our cost structure. We measure software productivity to allocate resources and make informed tradeoffs in our investments. We measure software quality at a structural level, in addition to the functional level through testing, to make the right trade-offs between delivery speed, business risk, and technical debt the longer-term costs of maintaining and enhancing the delivered solutions. For several years now, we have been successfully measuring the quality of our development projects and including these metrics in some of our Service Level Agreements. We are now starting to put productivity measurements across our portfolio side-by-side with quality measurements to get a truer picture of where we are trading present delivery agility for future business agility. xix

21 xx Foreword The Economics of Software Quality is a landmark for three reasons. It is practical, it is data-driven, and it goes beyond the traditional treatments of quality to demonstrate how to manage structural quality an important element of software quality for our business. Just as we invest in our enterprise architecture to actively manage the evolution of our core application software, we are putting a strong focus on the analysis and measurement of these applications at the structural level. These measures enable my organization to take a proactive stance to building a better future for our business and for me to closely manage the economic fundamentals in meeting and exceeding shareholder expectations. As we look forward to an exciting period of rapid growth in fixed-line, mobile, data, and on-demand products and services, I can assure you that this is one book my management team and I will keep close at hand. Thaddeus Arroyo Chief Information Officer, AT&T Services, Inc. F. Thaddeus Arroyo, Chief Information Officer, is responsible for AT&T s information technology. He was appointed to his current position in January 2007, following the close of the merger between AT&T, BellSouth, and Cingular. In his role, he is responsible for directing the company s internal information technology organization and infrastructure, including Internet and intranet capabilities, developing applications systems across the consumer and mobility markets, enterprise business segments, and AT&T s corporate systems. He also oversees AT&T s enterprise data centers.

22 Preface This book is aimed at software managers, executives, and quality assurance personnel who are involved in planning, estimating, executing, and maintaining software. Managers and stakeholders need to understand the economics of software quality when planning and developing new applications and enhancing or maintaining existing ones. The goal of this book is to quantify the factors that influence software quality and provide readers with enough information for them to predict and measure quality levels of their projects and applications. To serve this goal, we consolidate an expansive body of software quality data data on software structural quality, software assurance processes and techniques, and the marginal costs and benefits of improving software quality. The book provides quantitative data on how high and low quality affect software project schedules, staffing, development costs, and maintenance costs. This information should enable software managers to set and track progress toward quality targets and to make the right trade-offs between speed to market and business risk. We quantify the positive economic value of software quality and the high costs of poor software quality using software quality data from large organizations in the private and public sectors. This is not a how to do it book there are many good how-to books on processes and techniques for testing, inspections, static analysis, and other quality topics. We hope to have added a substantial amount of software quality data from real-world applications to complement those how-to books and enable IT managers to quantify the relative efficacy and economic value of these techniques. In small projects, individual human skills and experience play a major role in successful outcomes. Quality is important, but individual skill tends to be the dominant driver of high quality. But as projects grow larger, with development teams from 20 on up to more than 1,000 personnel, individual skills tend to regress to the mean. Quality becomes progressively more important because, historically, the costs of finding and fixing bugs have been the largest known expense for large software applications. This is true of both new development as well as enhancement and maintenance. Most discussions of software quality focus almost exclusively on functional quality. In this book, we expand our treatment beyond functional quality to xxi

23 xxii Preface cover nonfunctional and structural quality. Measuring structural quality requires going beyond the quality of individual components to the quality of the application as a whole. We show how to clearly define and repeatably measure nonfunctional and structural quality. Reliable measurements of all three kinds of quality structural, nonfunctional, and functional are essential for a complete treatment of the economics of software quality. We use these quality metrics to compare a number of quality improvement techniques at each stage of the software development life cycle and quantify their efficacy using data from real-world applications. To achieve high-quality levels for large systems, a synergistic set of methods is needed. These include defect prevention methods, which can reduce defect levels; pretest defect removal methods such as inspections and static analysis; and more than 40 kinds of testing. Several newer kinds of development methods also have beneficial impacts on software quality compared to traditional waterfall development. These include Agile development, Crystal development, Extreme Programming (XP), Personal Software Process (PSP), the Rational Unified Process (RUP), the Team Software Process (TSP), and several others. The generally poor measurement practices of the software industry have blurred understanding of software quality economics. Many executives and even some quality personnel tend to regard software quality as an expense. They also tend to regard quality as a topic that lengthens schedules and raises development costs. However, from an analysis of about 13,000 software projects between 1973 and today, it is gratifying to observe that high quality levels are invariably associated with shorter-than-average development schedules and lower-thanaverage development costs. The reason for this is that most projects that run late and exceed their budgets show no overt sign of distress until testing begins. When testing begins, a deluge of high-severity defects tends to stretch out testing intervals and cause massive bursts of overtime. In general, testing schedules for low-quality, large software projects are two to three times longer and more than twice as costly as testing for high-quality projects. If defects remain undetected and unremoved until testing starts, it is too late to bring a software project back under control. It is much more cost-effective to prevent defects or to remove them prior to testing. Another poor measurement practice that has concealed the economic value of software quality is the usage of the cost-per-defect metric. It has become an urban legend that it costs 100 times as much to fix a bug after delivery as during development. Unfortunately, the cost-per-defect metric actually penalizes quality and achieves its lowest values for the buggiest software. As quality

24 Preface xxiii improves, cost per defect rises until a level of zero defects is reached, where the cost-per-defect metric cannot be used at all. The real economic value of high quality is only partially related to defect repair costs. It is true that high quality leads to fewer defects and therefore to lower defect repair costs. But its major economic benefits are due to the fact that high quality Reduces the odds of large-system cancellations Reduces the odds of litigation for outsourced projects Shortens development schedules Lowers development costs Lowers maintenance costs Reduces warranty costs Increases customer satisfaction This book contains seven chapters. The Introduction in Chapter 1 discusses the fact that software has become one of the most widely used products in human history. As this book is written, a majority of all business activities are driven by software. A majority of government operations are controlled by software, such as civilian taxes, military and defense systems, and both state and local government organizations. Because software is so pervasive, high and low quality levels affect every citizen in significant ways. Chapter 1 defines software quality, considering the topic of quality is ambiguous both for software itself and for other manufactured products. There are many diverse views of what quality actually means. Chapter 1 examines all of the common views and concludes that effective definitions for quality need to be predictable in advance and measurable when they occur. Because this book deals with quantification and economic topics, there is emphasis on quality factors that can be measured precisely, such as defects and defect removal efficiency. In addition to these well-defined metrics, we show how to precisely measure software structural quality. Other definitions of quality, such as fitness, use, or aesthetic factors, are important but not always relevant to economic analysis. Chapter 2 is about estimating and measuring software quality. It is important for executives, clients, stakeholders, venture capitalists, and others with a financial interest in software to understand how quality can be predicted before projects start and measured during development and after release. Because software quality involves requirements, architecture, design, and many

25 xxiv Preface other noncode artifacts, the traditional lines of code metric is inadequate. This book uses function point metrics and structural quality metrics for quantifying quality. The function point metric is independent of code and therefore can deal with noncoding defects such as toxic requirements. Structural quality metrics get to the root causes of application quality and serve as foundational measures of software costs and business risks. Chapters 3 deals with the important topic of defect prevention. The set of methods that reduce defect potentials and minimize errors are difficult to study because they cannot be studied in isolation, but need numerous cases where a specific method was used and similar cases where the method was not used. Examples of methods that have demonstrated success in terms of defect prevention include Six Sigma, quality function deployment (QFD), test-driven development (TDD), and formal inspections. The kaizen and poka yoke inspections from Japan are also defect prevention methods. Some of these, such as inspections, happen to be effective as both defect prevention and defect removal methods. Chapter 4 deals with pretest defect removal methods in use today. The term pretest refers to quality and defect removal methods that occur prior to the start of testing. Among these methods are peer reviews, formal inspections, and static analysis. Although the literature on pretest defect removal is sparse compared to the literature on testing, these methods are important and have great value. Effective pretest methods such as inspections and static analysis shorten test schedules and raise testing efficiency. Twenty-five different kinds of pretest defect removal are discussed. Chapter 5 deals with testing, which is the traditional quality control technique for software projects. Although there is an extensive literature on testing, there is a surprising lack of quantified data on topics such as defect detection efficiency (DDE) and defect removal efficiency (DRE). If testing is performed without effective defect prevention methods and without pretest defect removal, most forms of testing are usually less than 35% efficient in finding bugs and quite expensive as well. A synergistic combination of defect prevention, pretest removal, and formal well-planned testing can raise test removal efficiency substantially. The goal of effective quality control is to approach 99% in terms of cumulative defect removal efficiency. Forty kinds of testing stages are discussed in Chapter 5. Chapter 6 deals with post-release defect removal, which is an unfortunate fact of life for software applications. Cumulative defect removal efficiency in the United States is only about 85%, so all software applications are delivered with latent defects. As a result, customers will always find bugs, and software organizations will always need customer support and maintenance personnel

26 Preface xxv available to repair the bugs. However, state-of-the-art combinations of defect prevention, pretest removal, and testing can top 96% in terms of defect removal efficiency on average and even achieve 99% in a few cases. Chapter 7 consolidates all of the authors data and shows side-by-side results for low-quality, average-quality, and high-quality software projects. Both the methods used to achieve high quality and the quantitative results of achieving high quality are discussed. Using structural quality data from 295 applications from 75 organizations worldwide, we define and quantify the notion of technical debt the cost of fixing problems in working software that, if left unfixed, will likely cause severe business disruption. We juxtapose this with a framework for quantifying the loss of business value due to poor quality. Together with technical debt, this business value framework provides a platform for future software economics research.

27 This page intentionally left blank

28 Acknowledgments By Capers Jones There are two authors for this book, and we each want to acknowledge those who helped in its creation. As always, thanks to my wife Eileen for her support of the many months of time spent in writing 16 books over a 25-year period. While this book was in process, two friends and colleagues passed away. Special thanks should go to both Al Albrecht and Watts Humphrey. Allan J. Albrecht was one of the original creators of function point metrics, without which this book would not be possible. Al and I first met in 1978 when he gave a talk on function points at the joint IBM/SHARE/GUIDE conference in Monterey, California. Although we both worked for IBM, Al was located in White Plains, New York, and I was located in San Jose, California, so we had not met until the conference. Al s talk and the function point metric made economic analysis of software feasible and provided insights that older metrics such as lines of code and cost per defect could not replicate. Al Albrecht, IBM, and the conference management kindly gave permission to publish Al s paper in my second book, Programming Productivity: Issues for the Eighties through the IEEE Press in From this point on, all of my technical books have used function points for quantitative information about software quality, productivity, and economic topics. After Al retired from IBM, we both worked together for about five years in the area of expanding the usage of function point metrics. Al created the first certification exam for function points and taught the metric to many of our colleagues. Al was an electrical engineer by training and envisioned function point metrics as providing a firm basis for both quality and productivity studies for all kinds of software applications. Today, in 2011, function points are the most widely used software metric and almost the only metric that has substantial volumes of benchmark information available. About two weeks before Al Albrecht passed away, the software industry also lost Watts Humphrey. Watts, too, was a colleague at IBM. Watts was an inventor and a prolific writer of excellent books, as well as an excellent public speaker and often keynoted software conferences. xxvii

29 xxviii Acknowledgments After retiring from IBM, Watts started a second career at the Software Engineering Institute (SEI) where he pioneered the development of the original version of the capability maturity model (CMM). Watts was one of the first software researchers to recognize that quality is the driving force for effective software development methods. It would be pointless to improve productivity unless quality improved faster and further because otherwise higher productivity would only create more defects. At both IBM and the SEI, Watts supported many quality initiatives, such as formal inspections, formal testing, and complete defect measurements, from the start of software projects through their whole useful lives. Watts also created both the Personal Software Process (PSP) and the Team Software Process (TSP), which are among the most effective methods for combining high quality and high performance. Watts s work in software process improvement was recognized by his receipt of the National Medal of Technology from President George Bush in In recent years, Watts took part in a number of seminars and conferences, so we were able to meet face-to-face several times a year, usually in cities where software conferences were being held. In this book, the importance of quality as being on the critical path to successful software development is an idea that Watts long championed. And the ability to measure quality, productivity, and other economic factors would not be possible without the function point metric developed by Al Albrecht. Many other people contributed to this book, but the pioneering work of Al and Watts were the key factors that made the book possible. By Olivier Bonsignour First and foremost, I would like to thank Capers Jones. It has been a pleasure working with him on this book. I owe a debt to my colleagues Lev Lesokhin and Bill Curtis at CAST. Lev and Bill were the first ones to suggest this project and have been exceptional sounding boards throughout. Their imprint on the ideas, organization, and content is so extensive that they should be considered coauthors of this book. I ve borrowed from the work of other colleagues at CAST. First of all, Jitendra Subramanyam, who has done a tremendous job helping me elaborate the content of this book. Also, my work with Bill Curtis and Vincent Delaroche on the distinction between software structural quality at the application level, as opposed to quality at the component level appears in Chapter 2. This attribute of software quality that the whole is greater than the sum of its parts is critical to the analysis and measurement of software quality. The definition of software structural quality metrics in that chapter is based on work I did with Bill and with Vincent. The framework in Chapter 7 for calculating the

30 Acknowledgments xxix business loss caused by poor structural quality is also based on Bill s work. Jay Sappidi did the groundbreaking work of collecting and analyzing our first batch of structural quality data and crafting a definition of Technical Debt. Much of the structural quality analysis in Chapters 6 and 7 is based on Jay s work. The product engineering team at CAST Razak Ellafi, Philippe-Emmanuel Douziech, and their fellow engineers continue to create a magnificent product that admirably serves the needs of hundreds of organizations worldwide. The CAST Application Intelligence Platform is not only a piece of fine engineering, it is also the generator of all the structural quality data in this book.

31 This page intentionally left blank

32 About the Authors Capers Jones is currently the President and CEO of Capers Jones & Associates LLC. He is also the founder and former chairman of Software Productivity Research LLC (SPR). He holds the title of Chief Scientist Emeritus at SPR. Capers Jones founded SPR in Before founding SPR, Capers was Assistant Director of Programming Technology for the ITT Corporation at the Programming Technology Center in Stratford, Connecticut. He was also a manager and researcher at IBM in California. Capers is a well-known author and international public speaker. Some of his books have been translated into six languages. All of his books have been translated into Japanese, and his newest books are available in Chinese editions as well. Among his book titles are Patterns of Software Systems Failure and Success (Prentice Hall 1994), Applied Software Measurement, Third Edition (McGraw- Hill, 2008), Software Quality: Analysis and Guidelines for Success ( International Thomson, 1997), Estimating Software Costs, Second Edition (McGraw-Hill, 2007), and Software Assessments, Benchmarks, and Best Practices (Addison-Wesley, 2000). The third edition of his book Applied Software Measurement was published in the spring of His book entitled Software Engineering Best Practices was published by McGraw-Hill in October His current book is The Economics of Software Quality, with Olivier Bonsignour as coauthor. Capers and his colleagues have collected historical data from more than 600 corporations and more than 30 government organizations. This historical data is a key resource for judging the effectiveness of software process improvement methods. More than 13,000 projects have been reviewed. In addition to his technical books, Mr. Jones has also received recognition as an historian after the publication of The History and Future of Narragansett Bay in 2006 by Universal Publishers. His research studies include quality estimation, quality measurement, software cost and schedule estimation, software metrics, and risk analysis. xxxi

33 xxxii About the Authors Mr. Jones has consulted at more than 150 large corporations and also at a number of government organizations such as NASA, the U.S. Air Force, the U.S. Navy, the Internal Revenue Service, and the U.S. Courts. He has also worked with several state governments. Olivier Bonsignour is responsible for Research & Development and Product Management in a continual effort to build the world s most advanced Application Intelligence technology. Prior to joining CAST, Mr. Bonsignour was the CIO for DGA, the advanced research division of the French Ministry of Defense. Prior to that role, also at DGA, he was in charge of application development and a project director working on IT systems that support operations. A pioneer in the development of distributed systems and object oriented development, he joined CAST after having been an early adopter of CAST technology in Mr. Bonsignour holds a graduate degree in engineering and computer science from the National Institute of Applied Sciences (INSA), Lyon, and a master s degree in management from the executive program at IAE Aix-en-Provence. In his free time, Mr. Bonsignour enjoys swimming, cycling, and skiing, as well as sailing his boat off the coast of France.

34 Chapter 1 Defining Software Quality and Economic Value Introduction This book deals with two topics that have been ambiguous and difficult to pin down for many years: software quality and economic value. The reason for the ambiguity, as noted in the Preface, is that there are many different points of view, and each point of view has a different interpretation of the terms. For example, software quality does not mean the same thing to a customer as it does to a developer. Economic value has a different meaning to vendors than it has to consumers. For vendors, revenue is the key element of value, and for consumers, operational factors represent primary value. Both of these are discussed later in the book. By examining a wide spectrum of views and extracting the essential points from each view, the authors hope that workable definitions can be established that are comparatively unambiguous. Software quality, as covered in this book, goes well beyond functional quality (the sort of thing to which customers might react to in addition to usability and reliable performance). Quality certainly covers these aspects but extends further to nonfunctional quality (how well the software does what it is meant to do) and to structural quality (how well it can continue to serve business needs as they evolve and change as business conditions do). Why Is Software Quality Important? Computer usage in industrial countries starts at or before age 6, and by age 16 almost 60% of young people in the United States have at least a working 1