The Economics of Software Reliability

Transcription

1 The Economics of Software Reliability Barry Boehm, ISSRE 2003 Keynote Address November 19, 2003 ( 11/19/03 -CSE 1 Outline The business case for software reliability Example: net value of test aids Value depends on stakeholder value propositions What are stakeholders really relying on? Safety, accuracy, response time, Attributes often conflict Software dependability in a competitive world Conclusions 11/19/03 -CSE 2 1

2 Software Reliability Business Case Software reliability investments compete for resources With investments in functionality, response time, adaptability, speed of development, Each investment option generates curves of net value and ROI Net Value NV = PV(benefit flows-cost flows) Present Value PV = Flows at future times discounted Return on Investment ROI = NV/PV(cost flows) Value of benefits varies by stakeholder and role 11/19/03 -CSE 3 Software Testing Business Case Vendor proposition Our test data generator will cut your test costs in half We ll provide it to you for 30% of your test costs After you run all your tests for 50% of your original costs, you re 20% ahead Any concerns with vendor proposition? 11/19/03 -CSE 4 2

3 Software Testing Business Case Vendor proposition Our test data generator will cut your test costs in half We ll provide it to you for 30% of your test costs After you run all your tests for 50% of your original costs, you re 20% ahead Any concerns with vendor proposition? Test data generator is value-neutral* Every test case, defect is equally important Usually, 20% of test cases cover 80% of business value * As are most current software engineering techniques 11/19/03 -CSE 5 20% of Features Provide 80% of Value: Focus Testing on These (Bullock, 2000) % of Value for Correct Customer Billing Automated test generation tool - all tests have equal value Customer Type 11/19/03 -CSE 6 3

4 Value-Based Testing Provides More Net Value 60 (30, 58) Value-Based Testing Net Value NV (100, 20) Test Data Generator % Tests Percent of tests run Test Data Generator Cost Value NV Value-Based Testing Cost Value NV /19/03 -CSE 7 There is No Universal Dependability-Value Metric Different stakeholders rely on different value attributes Protection: safety, security, privacy Robustness: reliability, availability, survivability Quality of Service: performance, accuracy, ease of use Adaptability: evolvability, interoperability Affordability: cost, schedule, reusability Value attributes continue to tier down Performance: response time, resource consumption (CPU, memory, comm.) Value attributes are scenario-dependent 5 seconds normal response time; 2 seconds in crisis Value attributes often conflict Most often with performance and affordability 11/19/03 -CSE 8 4

5 Major Information System Dependability Stakeholders Dependents - passengers, patients Information Suppliers -citizens, companies Information System Information Brokers -financial services, news media Information Consumers -decisions, education, entertainment Mission Controllers -pilots, distribution controllers Developers, Acquirers, Administrators 11/19/03 -CSE 9 Overview of Stakeholder/Value Dependencies Strength of direct dependency on value attribute **- Critical ; *-Significant; blank-insignificant or indirect Attributes Stakeholders Info. Suppliers, Dependents Info. Brokers Info. Consumers Mission Controllers, Administrators Developers, Acquirers ** ** ** ** * * Protection Robustness * * ** ** ** * * ** * ** ** Quality of Service Adaptability Affordability * * ** 11/19/03 -CSE 10 5

6 Elaboration of Stakeholder/Value Dependencies Stakeholders Info Suppliers Dependents D-Attribute s Mission - Prote ction crit uncrit Sa fe ty ** Se curity * ** ** Privacy ** * Robustross Re liability ** ** ** ** Ava ilability * ** ** ** ** ** Surviva bility ** ** ** ** Quality of Service Pe rforma nce ** ** ** ** *-** Accuracy, Consiste ncy ** ** ** * ** * Acce ssibility, e ase of use ; ** ** ** ** ** ** difficulty of misuse Evolvability ** ** * * ** ** ** Interope rability ** ** ** Cost * ** Sche dule * ** ** Reusa bility ** * * 11/19/03 -CSE 11 Info Brokers Consumers Info Mission Controllers Developers, Maintenance Administrators Acquirers Implications for Dependability Engineering There is no universal dependability metric to optimize Need to identify system s success-critical stakeholders And their dependability priorities Need to balance satisfaction of stakeholder dependencies Stakeholder win-win negotiation Dependability attribute tradeoff analysis Need value-of-dependability models, methods, and tools 11/19/03 -CSE 12 6

7 Outline The business case for software reliability What are stakeholders really relying on? Software dependability in a competitive world Is market success a monotone function of dependability? Is quality really free? Is faster, better, cheaper really achievable? Value-based vs. value-neutral methods Conclusions 11/19/03 -CSE 13 Competing on Cost and Quality - adapted from Michael Porter, Harvard ROI Too cheap (low Q) Cost leader (acceptable Q) Stuck in the middle Quality leader (acceptable C) Overpriced Price or Cost Point 11/19/03 -CSE 14 7

8 Competing on Schedule and Quality - A risk analysis approach Risk Exposure RE = Prob (Loss) * Size (Loss) Loss financial; reputation; future prospects, For multiple sources of loss: RE = Σ [Prob (Loss) * Size (Loss)] source sources 11/19/03 -CSE 15 Example RE Profile: Time to Ship - Loss due to unacceptable dependability Many defects: high P(L) Critical defects: high S(L) RE = P(L) * S(L) Few defects: low P(L) Minor defects: low S(L) Time to Ship (amount of testing) 11/19/03 -CSE 16 8

9 Example RE Profile: Time to Ship - Loss due to unacceptable dependability - Loss due to market share erosion Many defects: high P(L) Critical defects: high S(L) Many rivals: high P(L) Strong rivals: high S(L) RE = P(L) * S(L) Few rivals: low P(L) Weak rivals: low S(L) Few defects: low P(L) Minor defects: low S(L) Time to Ship (amount of testing) 11/19/03 -CSE 17 Example RE Profile: Time to Ship - Sum of Risk Exposures Many defects: high P(L) Critical defects: high S(L) Many rivals: high P(L) Strong rivals: high S(L) RE = P(L) * S(L) Few rivals: low P(L) Weak rivals: low S(L) Sweet Spot Few defects: low P(L) Minor defects: low S(L) Time to Ship (amount of testing) 11/19/03 -CSE 18 9

10 Comparative RE Profile: Safety-Critical System RE = P(L) * S(L) er S(L): defects -Q Sweet Spot Mainstream Sweet Spot Time to Ship (amount of testing) 11/19/03 -CSE 19 Comparative RE Profile: Internet Startup Low-TTM Sweet Spot er S(L): delays RE = P(L) * S(L) Mainstream Sweet Spot TTM: Time to Market Time to Ship (amount of testing) 11/19/03 -CSE 20 10

11 Interim Conclusions Unwise to try to compete on both cost/schedule and quality Some exceptions: major technology or marketplace edge There are no one-size-fits-all cost/schedule/quality strategies Risk analysis helps determine how much testing (prototyping, formal verification, etc.) is enough Buying information to reduce risk Often difficult to determine parameter values Some COCOMO II values discussed next 11/19/03 -CSE 21 RELY Rating Software Development Cost/Quality Tradeoff Defect Risk - COCOMO II calibration to 161 projects Rough MTBF(mean time between failures) Loss of Human Life 100 years Financial Loss 2 years Nominal Moderate recoverable loss 1 month 1.0 In-house support software Low Low, easily recoverable loss 1 day Low Slight inconvenience 1 hour Relative Cost/Source Instruction 11/19/03 -CSE 22 11

12 RELY Rating Software Development Cost/Quality Tradeoff Defect Risk - COCOMO II calibration to 161 projects Rough MTBF(mean time between failures) Loss of Human Life 100 years Financial Loss 2 years 1.10 Commercial quality leader Nominal Moderate recoverable loss 1 month 1.0 In-house support software Low Low, easily recoverable loss 1 day Commercial cost leader 0.92 Low Slight inconvenience 1 hour Relative Cost/Source Instruction 11/19/03 -CSE 23 RELY Rating Software Development Cost/Quality Tradeoff Defect Risk - COCOMO II calibration to 161 projects Rough MTBF(mean time between failures) Loss of Human Life 100 years 1.26 Safety-critical Financial Loss 2 years Commercial quality leader 1.10 Nominal Moderate recoverable loss 1 month 1.0 In-house support software Low Low, easily recoverable loss 1 day Commercial cost leader 0.92 Low Slight inconvenience (1 hour) Startup demo Relative Cost/Source Instruction 11/19/03 -CSE 24 12

13 Quality is Free Did Philip Crosby s book get it all wrong? Investments in dependable systems Cost extra for simple, short-life systems Pay off for high-value, long-life systems 11/19/03 -CSE 25 Software Life-Cycle Cost vs. Dependability Relative Cost to Develop Low Low Nominal COCOMO II RELY Rating 11/19/03 -CSE 26 13

14 Software Life-Cycle Cost vs. Dependability Relative Cost to Develop, Maintain Low Low Nominal COCOMO II RELY Rating 11/19/03 -CSE 27 Software Life-Cycle Cost vs. Dependability Relative Cost to Develop, Maintain % Maint Low Low Nominal COCOMO II RELY Rating 11/19/03 -CSE 28 14

15 Software Ownership Cost vs. Dependability VL = 2.55 L = 1.52 Operational-defect cost at Nominal dependability = Software life cycle cost 1.26 Relative Cost to Develop, Maintain, Own and Operate Operational - defect cost = % Maint Low Low Nominal COCOMO II RELY Rating 11/19/03 -CSE 29 Outline The business case for software reliability What are stakeholders really relying on? Software dependability in a competitive world Is market success a monotone function of dependability? Is quality really free? Is faster, better, cheaper really achievable? Value-based vs. value-neutral methods Conclusions 11/19/03 -CSE 30 15

16 Cost, Schedule, Quality: Pick any Two? C S Q 11/19/03 -CSE 31 Cost, Schedule, Quality: Pick any Two? Consider C, S, Q as Independent Variable Feature Set as Dependent Variable C C S Q S Q 11/19/03 -CSE 32 16

17 C, S, Q as Independent Variable Determine Desired Delivered Defect Density (D4) Or a value-based equivalent Prioritize desired features Via QFD, IPT, stakeholder win-win Determine Core Capability 90% confidence of D4 within cost and schedule Balance parametric models and expert judgment Architect for ease of adding next-priority features Hide sources of change within modules (Parnas) Develop core capability to D4 quality level Usually in less than available cost and schedule Add next priority features as resources permit Versions used successfully on 32 of 34 digital library projects 11/19/03 -CSE 33 Value-Based vs. Value Neutral Methods Value-based defect reduction Constructive Quality Model (COQUALMO) Information Dependability Attribute Value Estimation (idave) model 11/19/03 -CSE 34 17

18 Value-Based Defect Reduction Example: Goal-Question-Metric (GQM) Approach Goal: Our supply chain software packages have too many defects. We need to get their defect rates down Question:? 11/19/03 -CSE 35 Value-Based GQM Approach I Q: How do software defects affect system value goals? ask why initiative is needed - Order processing - Too much downtime on operations critical path - Too many defects in operational plans - Too many new-release operational problems G: New system-level goal: Decrease software-defect-related losses in operational effectiveness - With high-leverage problem areas above as specific subgoals New Q:? 11/19/03 -CSE 36 18

19 Value-Based GQM Approach II New Q: Perform system problem-area root cause analysis: ask why problems are happening via models Example: Downtime on critical path Order items Validate order Validate items in stock Schedule packaging, delivery Produce status reports Prepare delivery packages Deliver order Where are primary software-defect-related delays? Where are biggest improvement-leverage areas? Reducing software defects in Scheduling module Reducing non-software order-validation delays Taking Status Reporting off the critical path Downstream, getting a new Web-based order entry system Ask why not? as well as why? 11/19/03 -CSE 37 Value-Based GQM Results Defect tracking weighted by system-value priority Focuses defect removal on highest-value effort Significantly higher effect on bottom-line business value And on customer satisfaction levels Engages software engineers in system issues Fits increasing system-criticality of software Strategies often helped by quantitative models COQUALMO, idave 11/19/03 -CSE 38 19

20 Current COQUALMO System Software Size Estimate Software platform, Project, product and personnel attributes Defect removal profile levels Automation, Reviews, Testing COCOMO II COQUALMO Defect Introduction Model Defect Removal Model Software development effort, cost and schedule estimate Number of residual defects Defect density per unit of size 11/19/03 -CSE 39 Defect Removal Rating Scales COCOMO II p.263 Low Low Nominal Extra Automated Analysis Simple compiler syntax checking Basic compiler capabilities Compiler extension Basic req. and design consistency More elaborate req./design Basic distprocessing Intermediatelevel module Simple req./design Formalized specification, verification. Advanced distprocessing Peer Reviews No peer review Ad-hoc informal walkthrough Well-defined preparation, review, minimal follow-up Formal review roles and Welltrained people and basic checklist Root cause analysis, formal follow Using historical data Extensive review checklist Statistical control Execution Testing and Tools No testing Ad-hoc test and debug Basic test Test criteria based on checklist Well-defined test seq. and basic test coverage tool system More advance test tools, preparation. Distmonitoring ly advanced tools, modelbased test 11/19/03 -CSE 40 20

21 Defect Removal Estimates - Nominal Defect Introduction Rates Delivered Defects / KSLOC VL Low Nom VH XH Composite Defect Removal Rating 11/19/03 -CSE 41 idave Model 11/19/03 -CSE 42 21

22 ROI Analysis Results Comparison - Business Application vs. Space Application R O I N- >H H- >VH VH- >XH Dependability Investment Levels Sierra Order Processing Planetary Rover 11/19/03 -CSE 43 Conclusions: Software Reliability (SWR) There is no universal SWR metric to optimize Need to balance stakeholder SWR value propositions Increasing need for value-based approaches to SWR Methods and models emerging to address needs Faster, better, cheaper is feasible If feature content can be a dependent variable Quality is free for stable, high-value, long-life systems But not for dynamic, lower-value, short life systems Future trends intensify SWR needs and challenges Criticality, complexity, decreased control, faster change 11/19/03 -CSE 44 22