The Economics of Software Quality

Size: px
Start display at page:

Download "The Economics of Software Quality"

Transcription

1 The Economics of Software Quality Barry Boehm, USC Trustworthy Software Process Presentation June 3, 2009 ( 06/03/09 USC-CSSE 1

2 Outline The business case for software quality Example: net value of test aids Value depends on stakeholder value propositions What qualities do stakeholders really value? Safety, accuracy, response time, Attributes often conflict Software quality in a competitive world Conclusions 06/03/09 USC-CSSE 2

3 Software Quality Business Case Software quality investments compete for resources With investments in functionality, response time, adaptability, speed of development, Each investment option generates curves of net value and ROI Net Value NV = PV(benefit flows-cost flows) Present Value PV = Flows at future times discounted Return on Investment ROI = NV/PV(cost flows) Value of benefits varies by stakeholder and role 06/03/09 USC-CSSE 3

4 Software Testing Business Case Vendor proposition Our test data generator will cut your test costs in half We ll provide it to you for 30% of your test costs After you run all your tests for 50% of your original costs, you re 20% ahead Any concerns with vendor proposition? 06/03/09 USC-CSSE 4

5 Software Testing Business Case Vendor proposition Our test data generator will cut your test costs in half We ll provide it to you for 30% of your test costs After you run all your tests for 50% of your original costs, you re 20% ahead Any concerns with vendor proposition? Test data generator is value-neutral* Every test case, defect is equally important Usually, 20% of test cases cover 80% of business value * As are most current software engineering techniques 06/03/09 USC-CSSE 5

6 20% of Features Provide 80% of Value: Focus Testing on These (Bullock, 2000) % of Value for Correct Customer Billing Automated test generation tool - all tests have equal value Customer Type 06/03/09 USC-CSSE 6

7 Value-Based Testing Provides More Net Value 60 (30, 58) Value-Based Testing Net Value NV (100, 20) Test Data Generator Percent of tests run % Tests Test Data Generator Value-Based Testing Cost Value NV Cost Value NV /03/09 USC-CSSE 7

8 There is No Universal Quality-Value Metric Different stakeholders rely on different value attributes Protection: safety, security, privacy Robustness: reliability, availability, survivability Quality of Service: performance, accuracy, ease of use Adaptability: evolvability, interoperability Affordability: cost, schedule, reusability Value attributes continue to tier down Performance: response time, resource consumption (CPU, memory, comm.) Value attributes are scenario-dependent 5 seconds normal response time; 2 seconds in crisis Value attributes often conflict Most often with performance and affordability 06/03/09 USC-CSSE 8

9 Major Information System Quality Stakeholders Dependents - passengers, patients Information Suppliers -citizens, companies Information System Information Brokers -financial services, news media Information Consumers -decisions, education, entertainment Mission Controllers -pilots, distribution controllers Developers, Acquirers, Administrators 06/03/09 USC-CSSE 9

10 Overview of Stakeholder/Value Dependencies Strength of direct dependency on value attribute **- Critical ; *-Significant; blank-insignificant or indirect Stakeholders Attributes Info. Suppliers, Dependents Info. Brokers Info. Consumers Mission Controllers, Administrators Developers, Acquirers Protection Robustness Quality of Service ** * * ** ** ** ** * * ** * * * ** ** ** * * ** ** Adaptability Affordability 06/03/09 USC-CSSE 10

11 Elaboration of Stakeholder/Value Dependencies Stakeholders Info Suppliers Dependents Q-Attributes Mission - Protection crit uncrit Safety ** Security * ** ** Privacy ** * Robustross Reliability ** ** ** ** Availability * ** ** ** ** ** Survivability ** ** ** ** Quality of Service Performance ** ** ** ** *-** Accuracy, Consistency ** ** ** * ** * Accessibility, ease of use; ** ** ** ** ** ** difficulty of misuse Evolvability ** ** * * ** ** ** Interoperability ** ** ** Info Brokers Consumers Info Mission Controllers Developers, Maintenance Administrators Acquirers Cost * ** Schedule * ** ** Reusability ** * * 06/03/09 USC-CSSE 11

12 Implications for Quality Engineering There is no universal quality metric to optimize Need to identify system s success-critical stakeholders And their quality priorities Need to balance satisfaction of stakeholder dependencies Stakeholder win-win negotiation Quality attribute tradeoff analysis Need value-of-quality models, methods, and tools 06/03/09 USC-CSSE 12

13 Outline The business case for software quality What qualities do stakeholders really value? Software quality in a competitive world Is market success a monotone function of quality? Is quality really free? Is faster, better, cheaper really achievable? Value-based vs. value-neutral methods Conclusions 06/03/09 USC-CSSE 13

14 Competing on Cost and Quality - adapted from Michael Porter, Harvard ROI Too cheap (low Q) Cost leader (acceptable Q) Stuck in the middle Quality leader (acceptable C) Overpriced Price or Cost Point 06/03/09 USC-CSSE 14

15 Competing on Schedule and Quality - A risk analysis approach Risk Exposure RE = Prob (Loss) * Size (Loss) Loss financial; reputation; future prospects, For multiple sources of loss: RE = Σ [Prob (Loss) * Size (Loss)] source sources 06/03/09 USC-CSSE 15

16 Example RE Profile: Time to Ship - Loss due to unacceptable quality Many defects: high P(L) Critical defects: high S(L) RE = P(L) * S(L) Few defects: low P(L) Minor defects: low S(L) Time to Ship (amount of testing) 06/03/09 USC-CSSE 16

17 Example RE Profile: Time to Ship - Loss due to unacceptable quality - Loss due to market share erosion Many defects: high P(L) Critical defects: high S(L) Many rivals: high P(L) Strong rivals: high S(L) RE = P(L) * S(L) Few rivals: low P(L) Weak rivals: low S(L) Few defects: low P(L) Minor defects: low S(L) Time to Ship (amount of testing) 06/03/09 USC-CSSE 17

18 Example RE Profile: Time to Ship - Sum of Risk Exposures Many defects: high P(L) Critical defects: high S(L) Many rivals: high P(L) Strong rivals: high S(L) RE = P(L) * S(L) Few rivals: low P(L) Weak rivals: low S(L) Sweet Spot Few defects: low P(L) Minor defects: low S(L) Time to Ship (amount of testing) 06/03/09 USC-CSSE 18

19 Comparative RE Profile: Safety-Critical System RE = P(L) * S(L) Higher S(L): defects High-Q Sweet Spot Mainstream Sweet Spot Time to Ship (amount of testing) 06/03/09 USC-CSSE 19

20 Comparative RE Profile: Internet Startup Low-TTM Sweet Spot Higher S(L): delays RE = P(L) * S(L) Mainstream Sweet Spot TTM: Time to Market Time to Ship (amount of testing) 06/03/09 USC-CSSE 20

21 How much Architecting is Enough? A COCOMOII Analysis Percent of Time Added to Overall Schedule KSLOC 100 KSLOC 10 KSLOC Sweet Spot Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution Added Schedule Devoted to Rework (COCOMO II RESL factor) Total % Added Schedule Sweet Spot Drivers: Rapid Change: leftward 10 High Assurance: rightward Percent of Time Added for Architecture and Risk Resolution 06/03/09 USC-CSSE 21

22 Interim Conclusions Unwise to try to compete on both cost/schedule and quality Some exceptions: major technology or marketplace edge There are no one-size-fits-all cost/schedule/quality strategies Risk analysis helps determine how much testing (prototyping, formal verification, etc.) is enough Buying information to reduce risk Often difficult to determine parameter values Some COCOMO II values discussed next 06/03/09 USC-CSSE 22

23 RELY Rating Software Development Cost/Quality Tradeoff Defect Risk - COCOMO II calibration to 161 projects Rough MTBF(mean time between failures) Very High Loss of Human Life 100 years High High Financial Loss 2 years Nominal Moderate recoverable loss 1 month 1.0 In-house support software Low Low, easily recoverable loss 1 day Very Low Slight inconvenience 1 hour Relative Cost/Source Instruction 06/03/09 USC-CSSE 23

24 RELY Rating Software Development Cost/Quality Tradeoff Defect Risk - COCOMO II calibration to 161 projects Rough MTBF(mean time between failures) Very High Loss of Human Life 100 years High High Financial Loss 2 years 1.10 Commercial quality leader Nominal Moderate recoverable loss 1 month 1.0 In-house support software Low Low, easily recoverable loss 1 day 0.92 Commercial cost leader Very Low Slight inconvenience 1 hour Relative Cost/Source Instruction 06/03/09 USC-CSSE 24

25 RELY Rating Software Development Cost/Quality Tradeoff Defect Risk - COCOMO II calibration to 161 projects Rough MTBF(mean time between failures) Very High Loss of Human Life 100 years 1.26 Safety-critical High High Financial Loss 2 years 1.10 Commercial quality leader Nominal Moderate recoverable loss 1 month 1.0 In-house support software Low Low, easily recoverable loss 1 day 0.92 Commercial cost leader Very Low Slight inconvenience (1 hour) 0.82 Startup demo Relative Cost/Source Instruction 06/03/09 USC-CSSE 25

26 Quality is Free Did Philip Crosby s book get it all wrong? Investments in dependable systems Cost extra for simple, short-life systems Pay off for high-value, long-life systems 06/03/09 USC-CSSE 26

27 Software Life-Cycle Cost vs. Quality Relative Cost to Develop Very Low Low Nominal High COCOMO II RELY Rating Very High 06/03/09 USC-CSSE 27

28 Software Life-Cycle Cost vs. Quality Relative Cost to Develop, Maintain Very Low Low Nominal High COCOMO II RELY Rating Very High 06/03/09 USC-CSSE 28

29 Software Life-Cycle Cost vs. Quality Relative Cost to Develop, Maintain % Maint Very Low Low Nominal High COCOMO II RELY Rating Very High 06/03/09 USC-CSSE 29

30 Software Ownership Cost vs. Quality VL = 2.55 L = 1.52 Operational-defect cost at Nominal dependability = Software life cycle cost 1.26 Relative Cost to Develop, Maintain, Own and Operate Operational - defect cost = % Maint Very Low Low Nominal High COCOMO II RELY Rating Very High 06/03/09 USC-CSSE 30

31 Outline The business case for software quality What qualities do stakeholders really value? Software quality in a competitive world Is market success a monotone function of quality? Is quality really free? Is faster, better, cheaper really achievable? Value-based vs. value-neutral methods Conclusions 06/03/09 USC-CSSE 31

32 Cost, Schedule, Quality: Pick any Two? C S Q 06/03/09 USC-CSSE 32

33 Cost, Schedule, Quality: Pick any Two? Consider C, S, Q as Independent Variable Feature Set as Dependent Variable C C S Q S Q 06/03/09 USC-CSSE 33

34 Tradeoffs Among Cost, Schedule, and Reliability: COCOMO II Cost ($M) For 100-KSLOC set of features Can pick all three with 77-KSLOC set of features Development Time (Months) (RELY, MTBF (hours)) (VL, 1) (L, 10) (N, 300) (H, 10K) (VH, 300K) -- Cost/Schedule/RELY: pick any two points 06/03/09 USC-CSSE 34

35 C, S, Q as Independent Variable Determine Desired Delivered Defect Density (D4) Or a value-based equivalent Prioritize desired features Via QFD, IPT, stakeholder win-win Determine Core Capability 90% confidence of D4 within cost and schedule Balance parametric models and expert judgment Architect for ease of adding next-priority features Hide sources of change within modules (Parnas) Develop core capability to D4 quality level Usually in less than available cost and schedule Add next priority features as resources permit Versions used successfully on 32 of 34 USC digital library projects 06/03/09 USC-CSSE 35

36 Value-Based vs. Value Neutral Methods Value-based defect reduction Constructive Quality Model (COQUALMO) Information Dependability Attribute Value Estimation (idave) model 06/03/09 USC-CSSE 36

37 Value-Based Defect Reduction Example: Goal-Question-Metric (GQM) Approach Goal:Our supply chain software packages have too many defects. We need to get their defect rates down Question:? 06/03/09 USC-CSSE 37

38 Value-Based GQM Approach I Q: How do software defects affect system value goals? ask why initiative is needed - Order processing - Too much downtime on operations critical path - Too many defects in operational plans - Too many new-release operational problems G: New system-level goal: Decrease software-defect-related losses in operational effectiveness New Q:? - With high-leverage problem areas above as specific subgoals 06/03/09 USC-CSSE 38

39 Value-Based GQM Approach II New Q: Perform system problem-area root cause analysis: ask why problems are happening via models Example: Downtime on critical path Order items Validate order Validate items in stock Schedule packaging, delivery Produce status reports Prepare delivery packages Deliver order Where are primary software-defect-related delays? Where are biggest improvement-leverage areas? Reducing software defects in Scheduling module Reducing non-software order-validation delays Taking Status Reporting off the critical path Downstream, getting a new Web-based order entry system Ask why not? as well as why? 06/03/09 USC-CSSE 39

40 Value-Based GQM Results Defect tracking weighted by system-value priority Focuses defect removal on highest-value effort Significantly higher effect on bottom-line business value And on customer satisfaction levels Engages software engineers in system issues Fits increasing system-criticality of software Strategies often helped by quantitative models COQUALMO, idave 06/03/09 USC-CSSE 40

41 Current COQUALMO System Software Size Estimate Software platform, Project, product and personnel attributes COCOMO II COQUALMO Defect Introduction Model Software development effort, cost and schedule estimate Number of residual defects Defect density per unit of size Defect removal profile levels Automation, Reviews, Testing Defect Removal Model 06/03/09 USC-CSSE 41

42 Defect Removal Rating Scales COCOMO II p.263 Very Low Low Nominal High Very High Extra High Automated Analysis Simple compiler syntax checking Basic compiler capabilities Compiler extension Basic req. and design consistency More elaborate req./design Basic distprocessing Intermediatelevel module Simple req./design Formalized specification, verification. Advanced distprocessing Peer Reviews No peer review Ad-hoc informal walk-through Well-defined preparation, review, minimal follow-up Formal review roles and Welltrained people and basic checklist Root cause analysis, formal follow Using historical data Extensive review checklist Statistical control Execution Testing and Tools No testing Ad-hoc test and debug Basic test Test criteria based on checklist Well-defined test seq. and basic test coverage tool system More advance test tools, preparation. Dist-monitoring Highly advanced tools, modelbased test 06/03/09 USC-CSSE 42

43 Defect Removal Estimates - Nominal Defect Introduction Rates Delivered Defects / KSLOC VL Low Nom High VH XH Composite Defect Removal Rating 06/03/09 USC-CSSE 43

44 idave Model 06/03/09 USC-CSSE 44

45 ROI Analysis Results Comparison - Business Application vs. Space Application ROI N- >H H- >VH VH- >XH Dependability Investment Levels Sierra Order Processing Planetary Rover 06/03/09 USC-CSSE 45

46 Conclusions: Software Quality (SWQ) There is no universal SWQ metric to optimize Need to balance stakeholder SWQ value propositions Increasing need for value-based approaches to SWQ Methods and models emerging to address needs Faster, better, cheaper is feasible If feature content can be a dependent variable Quality is free for stable, high-value, long-life systems But not for dynamic, lower-value, short life systems Future trends intensify SWQ needs and challenges Criticality, complexity, decreased control, faster change 06/03/09 USC-CSSE 46

The Economics of Software Reliability

The Economics of Software Reliability The Economics of Software Reliability Barry Boehm, ISSRE 2003 Keynote Address November 19, 2003 (boehm@sunset.usc.edu) (http://sunset.usc.edu) 11/19/03 -CSE 1 Outline The business case for software reliability

More information

Assessing Quality Processes with ODC COQUALMO

Assessing Quality Processes with ODC COQUALMO Assessing Quality Processes with ODC COQUALMO Ray Madachy, Barry Boehm USC {madachy, boehm}@usc.edu 2008 International Conference on Software Process May 10, 2008 USC-CSSE 1 Introduction Cost, schedule

More information

Some Critical Success Factors for Industrial/Academic Collaboration in Empirical Software Engineering

Some Critical Success Factors for Industrial/Academic Collaboration in Empirical Software Engineering Some Critical Success Factors for Industrial/Academic Collaboration in Empirical Software Engineering Barry Boehm, USC (in collaboration with Vic Basili) EASE Project Workshop November 7, 2003 11/7/03

More information

The ROI of Systems Engineering: Some Quantitative Results

The ROI of Systems Engineering: Some Quantitative Results The ROI of Systems Engineering: Some Quantitative Results Barry Boehm Center for Systems and Software Engineering University of Southern California boehm@usc.edu Ricardo Valerdi Lean Aerospace Initiative,

More information

Integrated Modeling of Business Value and Software Processes

Integrated Modeling of Business Value and Software Processes Integrated Modeling of Business Value and Software Processes Raymond Madachy, USC Center for Software Engineering Department of Computer Science, SAL 8 University of Southern California Los Angeles, CA

More information

Software Economics: A Roadmap

Software Economics: A Roadmap Software Economics: A Roadmap Barry Boehm, Kevin Sullivan, UVirginia ICSE 2000 June 8, 2000 Software Economics: A Roadmap Where have we been? Where are we now? Where do we need to go? How can we get there?

More information

The Dynamics of Software Testing

The Dynamics of Software Testing The Dynamics of Software Testing Anas TAWILEH, Steve McINTOSH, Brent WORK and Wendy IVINS Cardiff University School of Computer Science Queen s Buildings, 5 The Parade, Cardiff, CF24 3AA, Wales +44(0)29

More information

Simulation for Business Value and Software Process/Product Tradeoff Decisions

Simulation for Business Value and Software Process/Product Tradeoff Decisions Simulation for Business Value and Software Process/Product Tradeoff Decisions Raymond Madachy USC Center for Software Engineering Dept. of Computer Science, SAL 8 Los Angeles, CA 90089-078 740 570 madachy@usc.edu

More information

Value-Based Feedback in Software/IT Systems

Value-Based Feedback in Software/IT Systems Value-Based Feedback in Software/IT Systems Barry Boehm, USC FEAST/ProSim/SOCE 2000 Keynote July 12, 2000 boehm@sunset.usc.edu http://sunset.usc.edu Outline COCOMO II 4-Cycle feedback model Value-based

More information

Bridge the Gap between Software Test Process and Business Value: A Case Study

Bridge the Gap between Software Test Process and Business Value: A Case Study Bridge the Gap between Software Test Process and Business Value: A Case Study Qi Li 1, Mingshu Li 2, Ye Yang 2, Qing Wang 2, Thomas Tan 1, Barry Boehm 1, and Chenyong Hu 2 1 Universy of Southern California

More information

An Introduction to. Metrics. used during. Software Development

An Introduction to. Metrics. used during. Software Development An Introduction to Metrics used during Software Development Life Cycle www.softwaretestinggenius.com Page 1 of 10 Define the Metric Objectives You can t control what you can t measure. This is a quote

More information

Cost/Benefit-Aspects of Software Quality Assurance

Cost/Benefit-Aspects of Software Quality Assurance Cost/Benefit-Aspects of Software Quality Assurance Master Seminar Software Quality Marc Giombetti Institut für Informatik Technische Universität München Boltzmannstr. 3, 85748 Garching b. München, Germany

More information

CDC UNIFIED PROCESS JOB AID

CDC UNIFIED PROCESS JOB AID CDC UNIFIED PROCESS JOB AID Independent Verification & Validation Activities Document Purpose This Job Aid is a brief document listing the items to be noted, checked, remembered, and delivered when completing

More information

Balancing Plan-Driven and Agile Methods in Software Engineering Project Courses

Balancing Plan-Driven and Agile Methods in Software Engineering Project Courses Computer Science Education 0899-3408/02/1203-187$16.00 2002, Vol. 12, No. 3, pp. 187±195 # Swets & Zeitlinger Balancing Plan-Driven and Agile Methods in Software Engineering Project Courses Barry Boehm,

More information

The Real Estate Philosopher Porter s Five Forces in the Real Estate World

The Real Estate Philosopher Porter s Five Forces in the Real Estate World The Real Estate Philosopher Porter s Five Forces in the Real Estate World Michael Porter is a professor at Harvard Business School. He has spent his long career analyzing strategy and competition. His

More information

The use of Trade-offs in the development of Web Applications

The use of Trade-offs in the development of Web Applications The use of Trade-offs in the development of Web Applications Sven Ziemer and Tor Stålhane Department of Computer and Information Science Norwegian University of Technology and Science {svenz, stalhane}@idi.ntnu.no

More information

Top 10 Do s and Don ts in Selecting Campus Recreation Software

Top 10 Do s and Don ts in Selecting Campus Recreation Software Top 10 Do s and Don ts in Selecting Campus Recreation Software Discussion Paper Investing in technology can make your Campus Recreation department more efficient, increase revenues, reduce costs, and improve

More information

Improving Software Development Economics Part II: Reducing Software Product Complexity and Improving Software Processes

Improving Software Development Economics Part II: Reducing Software Product Complexity and Improving Software Processes Improving Software Development Economics Part II: Reducing Software Product Complexity and Improving Software Processes by Walker Royce Vice President and General Manager Strategic Services Rational Software

More information

Measuring ROI of Agile Transformation

Measuring ROI of Agile Transformation Measuring ROI of Agile Transformation Title of the Paper: Measuring Return on Investment (ROI) of Agile Transformation Theme: Strategic & Innovative Practices Portfolio, Programs & Project (PPP) Management

More information

RFID Journal Live! 2006 May 1-3, 2006 MGM Grand Conference Center Las Vegas, Nevada

RFID Journal Live! 2006 May 1-3, 2006 MGM Grand Conference Center Las Vegas, Nevada RFID Journal Live! 2006 May 1-3, 2006 MGM Grand Conference Center Las Vegas, Nevada Ingenuity in Manufacturing John Rommel, Senior Manager RFID Channel Development, Symbol Technologies Toby Rush, President,

More information

SoftwareCostEstimation. Spring,2012

SoftwareCostEstimation. Spring,2012 SoftwareCostEstimation Spring,2012 Chapter 3 SOFTWARE COST ESTIMATION DB Liu Software Cost Estimation INTRODUCTION Estimating the cost of a software product is one of the most difficult and error-prone

More information

Future of CMM and Quality Improvement. Roy Ko Hong Kong Productivity Council

Future of CMM and Quality Improvement. Roy Ko Hong Kong Productivity Council Future of CMM and Quality Improvement Roy Ko Hong Kong Productivity Council 1 Agenda Future Development of CMMI CMMI and Small Organizations CMMI and Agile Development Good Enough Quality CMMI and Other

More information

Mitigating Risk through OEM Partnerships. Leveraging OEM to Drive the Bottom Line

Mitigating Risk through OEM Partnerships. Leveraging OEM to Drive the Bottom Line Mitigating Risk through OEM Partnerships Leveraging OEM to Drive the Bottom Line Executive Summary Incorporating third-party technology to add new capability to existing applications is not a new concept

More information

Software Process Improvement Software Business. Casper Lassenius

Software Process Improvement Software Business. Casper Lassenius Software Process Improvement Software Business Casper Lassenius Topics covered ² The process process ² Process measurement ² Process analysis ² Process change ² The CMMI process framework 2 Process ² Many

More information

case study Coverity Maintains Software Integrity of Sun Microsystems Award-Winning Storage Products

case study Coverity Maintains Software Integrity of Sun Microsystems Award-Winning Storage Products case study Coverity Maintains Software Integrity of Sun Microsystems Award-Winning Storage Products 25% of the defects identified by Coverity would have stalled a product launch and delayed time to market

More information

EXHIBIT L. Application Development Processes

EXHIBIT L. Application Development Processes EXHIBIT L Application Development Processes Optum Development Methodology Development Overview Figure 1: Development process flow The Development phase consists of activities that include the building,

More information

ni.com/services NI Services

ni.com/services NI Services ni.com/services NI Services Service You Can Trust Success You Can Measure NI delivers services as part of its commitment to your success in efficiently designing, developing, and maintaining high-quality

More information

Current and Future Challenges for Systems and Software Cost Estimation

Current and Future Challenges for Systems and Software Cost Estimation Current and Future Challenges for Systems and Software Cost Estimation Barry Boehm, USC-CSSE 29 th COCOMO-SSCM Forum October 21, 2014 Summary Current and future trends create challenges for systems and

More information

The purpose of Capacity and Availability Management (CAM) is to plan and monitor the effective provision of resources to support service requirements.

The purpose of Capacity and Availability Management (CAM) is to plan and monitor the effective provision of resources to support service requirements. CAPACITY AND AVAILABILITY MANAGEMENT A Project Management Process Area at Maturity Level 3 Purpose The purpose of Capacity and Availability Management (CAM) is to plan and monitor the effective provision

More information

Extending CMMI Level 4/5 Organizational Metrics Beyond Software Development

Extending CMMI Level 4/5 Organizational Metrics Beyond Software Development Extending CMMI Level 4/5 Organizational Metrics Beyond Software Development CMMI Technology Conference and User Group Denver, Colorado 14-17 November 2005 Linda Brooks Northrop Grumman Corporation Topics

More information

Practical Metrics for Managing and Improving Software Testing

Practical Metrics for Managing and Improving Software Testing Practical Metrics for Managing and Improving Software Testing Presented By: Shaun Bradshaw shaun.bradshaw@zenergytechnologies.com Slide 1 Part 1 Test Metrics Ten key metrics testers should track One bonus

More information

Measurement Strategies in the CMMI

Measurement Strategies in the CMMI Measurement Strategies in the CMMI International Software Measurement & Analysis Conference 9-14 September 2007 Rick Hefner, Ph.D. Director, Process Management Northrop Grumman Corporation One Space Park,

More information

Cost Estimation Driven Software Development Process

Cost Estimation Driven Software Development Process Cost Estimation Driven Software Development Process Orsolya Dobán, András Pataricza Budapest University of Technology and Economics Department of Measurement and Information Systems Pázmány P sétány 1/D

More information

Current and Future Challenges for Software Cost Estimation and Data Collection

Current and Future Challenges for Software Cost Estimation and Data Collection Current and Future Challenges for Software Cost Estimation and Data Collection Barry Boehm, USC-CSSE GSAW 2010 Cost Data Workshop March 3, 2010 Summary Current and future trends create challenges for DoD

More information

Automation can dramatically increase product quality, leading to lower field service, product support and

Automation can dramatically increase product quality, leading to lower field service, product support and QA Automation for Testing Medical Device Software Benefits, Myths and Requirements Automation can dramatically increase product quality, leading to lower field service, product support and liability cost.

More information

Wait-Time Analysis Method: New Best Practice for Performance Management

Wait-Time Analysis Method: New Best Practice for Performance Management WHITE PAPER Wait-Time Analysis Method: New Best Practice for Performance Management September 2006 Confio Software www.confio.com +1-303-938-8282 SUMMARY: Wait-Time analysis allows IT to ALWAYS find the

More information

MTAT.03.244 Software Economics. Lecture 5: Software Cost Estimation

MTAT.03.244 Software Economics. Lecture 5: Software Cost Estimation MTAT.03.244 Software Economics Lecture 5: Software Cost Estimation Marlon Dumas marlon.dumas ät ut. ee Outline Estimating Software Size Estimating Effort Estimating Duration 2 For Discussion It is hopeless

More information

Agility, Uncertainty, and Software Project Estimation Todd Little, Landmark Graphics

Agility, Uncertainty, and Software Project Estimation Todd Little, Landmark Graphics Agility, Uncertainty, and Software Project Estimation Todd Little, Landmark Graphics Summary Prior studies in software development project estimation have demonstrated large variations in the estimated

More information

Optimizing Rewards and Employee Engagement

Optimizing Rewards and Employee Engagement Optimizing Rewards and Employee Engagement Improving employee motivation and engagement, and identifying the right total rewards strategy to influence workforce effectiveness. Kevin Aselstine, Towers Perrin

More information

Skating to Where the Puck Is Going:!

Skating to Where the Puck Is Going:! Skating to Where the Puck Is Going: Anticipating Change via Empirical Methods Barry Boehm, USC-CSSE http://csse.usc.edu Rombach 60 Colloquium June 7, 2013 1 Motivation What helped me most in becoming a

More information

CHAPTER 1: INTRODUCTION TO RAPID APPLICATION DEVELOPMENT (RAD)

CHAPTER 1: INTRODUCTION TO RAPID APPLICATION DEVELOPMENT (RAD) CHAPTER 1: INTRODUCTION TO RAPID APPLICATION DEVELOPMENT (RAD) 1. INTRODUCTIONS RAD refers to a development life cycle designed Compare to traditional life cycle it is Faster development with higher quality

More information

Exhibit F. VA-130620-CAI - Staff Aug Job Titles and Descriptions Effective 2015

Exhibit F. VA-130620-CAI - Staff Aug Job Titles and Descriptions Effective 2015 Applications... 3 1. Programmer Analyst... 3 2. Programmer... 5 3. Software Test Analyst... 6 4. Technical Writer... 9 5. Business Analyst... 10 6. System Analyst... 12 7. Software Solutions Architect...

More information

A Guide to Open Source Transformation Services. How and Why Organizations are Making the Move to Open Source

A Guide to Open Source Transformation Services. How and Why Organizations are Making the Move to Open Source A Guide to Open Source Transformation Services How and Why Organizations are Making the Move to Open Source A fter decades of relying on commercial off-the-shelf software (COTS), thousands are moving to

More information

Management. Project. Software. Ashfaque Ahmed. A Process-Driven Approach. CRC Press. Taylor Si Francis Group Boca Raton London New York

Management. Project. Software. Ashfaque Ahmed. A Process-Driven Approach. CRC Press. Taylor Si Francis Group Boca Raton London New York Software Project Management A Process-Driven Approach Ashfaque Ahmed CRC Press Taylor Si Francis Group Boca Raton London New York CRC Press is an imprint of the Taylor St Francis Croup, an Informa business

More information

Value-Based Software Engineering Concepts: Overview and Introduction

Value-Based Software Engineering Concepts: Overview and Introduction Value-Based Software Engineering Concepts: Overview and Introduction Stefan Biffl Stefan.Biffl@tuwien.ac.at http://qse.ifs.tuwien.ac.at/~biffl Motivation Software Engineering aims at creating high-quality

More information

Latest Trends in Testing. Ajay K Chhokra

Latest Trends in Testing. Ajay K Chhokra Latest Trends in Testing Ajay K Chhokra Introduction Software Testing is the last phase in software development lifecycle which has high impact on the quality of the final product delivered to the customer.

More information

Points of Defect Creation

Points of Defect Creation WHITE PAPER Points of Defect Creation by Shankar Krishnamoorthy Software Product development methodologies try to improve quality by promoting the tactic of testing "early and often." When a defect is

More information

Measuring Success Service Desk Evaluation Guide for the Midsized Business: How to Choose the Right Service Desk Solution and Improve Your ROI

Measuring Success Service Desk Evaluation Guide for the Midsized Business: How to Choose the Right Service Desk Solution and Improve Your ROI BEST PRACTICES WHITE PAPER Measuring Success Service Desk Evaluation Guide for the Midsized Business: How to Choose the Right Service Desk Solution and Improve Your ROI Table of Contents INTRODUCTION...1

More information

Software Engineering Compiled By: Roshani Ghimire Page 1

Software Engineering Compiled By: Roshani Ghimire Page 1 Unit 7: Metric for Process and Product 7.1 Software Measurement Measurement is the process by which numbers or symbols are assigned to the attributes of entities in the real world in such a way as to define

More information

TEST MANAGEMENT SOLUTION Buyer s Guide WHITEPAPER. Real-Time Test Management

TEST MANAGEMENT SOLUTION Buyer s Guide WHITEPAPER. Real-Time Test Management TEST MANAGEMENT SOLUTION Buyer s Guide WHITEPAPER Real-Time Test Management How to Select the Best Test Management Vendor? The implementation of a Test Management system to automate business processes

More information

Integrated methodology for testing and quality management.

Integrated methodology for testing and quality management. Integrated methodology for testing and quality management. MindTest overview MindTest is an integrated testing methodology that meshes all the components of a testing engagement, manages the quality of

More information

Strategic Solutions that Make Your Work Easier. Projects Made Easier Decisions Made Easier Business Made Easier

Strategic Solutions that Make Your Work Easier. Projects Made Easier Decisions Made Easier Business Made Easier Strategic Solutions that Make Your Work Easier Projects Made Easier Decisions Made Easier Business Made Easier Have You Outgrown Your Systems? Buyers Say the Partner and the Product are More Important

More information

3 rd Party Vendor Risk Management

3 rd Party Vendor Risk Management 3 rd Party Vendor Risk Management Session 402 Tuesday, June 9, 2015 (11 to 12pm) Session Objectives The need for enhanced reporting on vendor risk management Current outsourcing environment Key risks faced

More information

Treasure Trove The Rising Role of Treasury in Accounts Payable

Treasure Trove The Rising Role of Treasury in Accounts Payable Treasury and Trade Solutions North America July 30, 2015 Treasure Trove The Rising Role of Treasury in Accounts Payable 2015 Citibank, N.A. All rights reserved Today s Speakers Andrew Bartolini Chief Research

More information

Module J Trade Study Process

Module J Trade Study Process Module J Trade Study Process CC04264140.ppt Trade Study Objectives Support the decision needs of the system engineering process Evaluate alternatives (requirements, functions, configurations) Integrate

More information

Creating Velocity in Data Center Migrations to AWS. Presented by Harald Viehweger Tuesday 30 June 2015

Creating Velocity in Data Center Migrations to AWS. Presented by Harald Viehweger Tuesday 30 June 2015 Berlin Creating Velocity in Data Center Migrations to AWS Presented by Harald Viehweger Tuesday 30 June 2015 Success in AWS In with the new, out with the old Success in AWS We are Cloud pioneers Using

More information

Project Management. Lecture 3. Software Engineering CUGS. Spring 2012 (slides made by David Broman) Kristian Sandahl

Project Management. Lecture 3. Software Engineering CUGS. Spring 2012 (slides made by David Broman) Kristian Sandahl Project Lecture 3 Software Engineering CUGS Spring 2012 (slides made by David Broman) Kristian Sandahl Department of Computer and Information Science Linköping University, Sweden A Software Life-cycle

More information

VENDOR SELECTION: WHERE TO BEGIN?

VENDOR SELECTION: WHERE TO BEGIN? VENDOR SELECTION: WHERE TO BEGIN? INTRODUCTION Selecting the right software for your organization, regardless if it s a best-of breed HR or Sales application or a full-fledged ERP system, can be a daunting

More information

ITSM Maturity Model. 1- Ad Hoc 2 - Repeatable 3 - Defined 4 - Managed 5 - Optimizing No standardized incident management process exists

ITSM Maturity Model. 1- Ad Hoc 2 - Repeatable 3 - Defined 4 - Managed 5 - Optimizing No standardized incident management process exists Incident ITSM Maturity Model 1- Ad Hoc 2 - Repeatable 3 - Defined 4 - Managed 5 - Optimizing No standardized incident process exists Incident policies governing incident Incident urgency, impact and priority

More information

Project Management. Lecture 3. Software Engineering CUGS. Spring 2011 (slides made by David Broman)

Project Management. Lecture 3. Software Engineering CUGS. Spring 2011 (slides made by David Broman) Lecture 3 Software Engineering CUGS Spring 2011 (slides made by David Broman) Kristian Sandahl Department of Computer and Information Science Linköping University, Sweden A Software Life-cycle Model Which

More information

CLOUD MIGRATION STRATEGIES

CLOUD MIGRATION STRATEGIES CLOUD MIGRATION STRATEGIES Faculty Contributor: Dr. Rahul De Student Contributors: Mayur Agrawal, Sudheender S Abstract This article identifies the common challenges that typical IT managers face while

More information

How To Write Software

How To Write Software Overview of Software Engineering Principles 1 Software Engineering in a Nutshell Development of software systems whose size/ complexity warrants a team or teams of engineers multi-person construction of

More information

Test Automation. Full service delivery for faster testing at optimum cost

Test Automation. Full service delivery for faster testing at optimum cost Test Automation Full service delivery for faster testing at optimum cost To safeguard their competitive edge in today s information economy, organizations must constantly improve their products and services.

More information

Systems Engineering Complexity & Project Management

Systems Engineering Complexity & Project Management Systems Engineering Complexity & Project Management Bob Ferguson, PMP NDIA: CMMI Technology Conference November 2007 Outline A conversation Defining complexity and its effects on projects Research into

More information

Software Process Engineering & Management Models

Software Process Engineering & Management Models Software Process Engineering & Management Models Paul Grünbacher Institute for Systems Engineering & Automation Johannes Kepler University Linz Christian Doppler Laboratory for Automated Software Engineering

More information

Windows Server 2003 migration: Your three-phase action plan to reach the finish line

Windows Server 2003 migration: Your three-phase action plan to reach the finish line WHITE PAPER Windows Server 2003 migration: Your three-phase action plan to reach the finish line Table of contents Executive summary...2 Windows Server 2003 and the big migration question...3 If only migration

More information

Following is a discussion of the Hub s role within the health insurance exchanges, the results of our review, and concluding observations.

Following is a discussion of the Hub s role within the health insurance exchanges, the results of our review, and concluding observations. Testimony of: Kay Daly Assistant Inspector General for Audit Services Office of Inspector General, U.S. Department of Health and Human Services Hearing Title: The Threat to Americans Personal Information:

More information

Competency Requirements for Executive Director Candidates

Competency Requirements for Executive Director Candidates Competency Requirements for Executive Director Candidates There are nine (9) domains of competency for association executives, based on research conducted by the American Society for Association Executives

More information

Aspire's Approach to Test Automation

Aspire's Approach to Test Automation WHITE PAPER Aspire's Approach to Test Automation by Ujjawal Bagaria, Aspire Systems Automation has been seen as the long term solution for cost reduction of manual testing across the globe. A successfully

More information

Benefits of Test Automation for Agile Testing

Benefits of Test Automation for Agile Testing Benefits of Test Automation for Agile Testing Manu GV 1, Namratha M 2, Pradeep 3 1 Technical Lead-Testing Calsoft Labs, Bangalore, India 2 Assistant Professor, BMSCE, Bangalore, India 3 Software Engineer,

More information

Establishing your Automation Development Lifecycle

Establishing your Automation Development Lifecycle Establishing your Automation Development Lifecycle Frequently I engage clients in assessing and improving their automation efforts. The discussion normally starts from a position of frustration We ve invested

More information

Fundamentals of Measurements

Fundamentals of Measurements Objective Software Project Measurements Slide 1 Fundamentals of Measurements Educational Objective: To review the fundamentals of software measurement, to illustrate that measurement plays a central role

More information

Major Project Governance Assessment Toolkit

Major Project Governance Assessment Toolkit Major Project Governance Assessment Toolkit Mark Ritchie, University of Edinburgh Pauline Woods-Wilson, Lancaster University Project and Change Management Group Project and Change Management Group Established

More information

Competitive Advantage

Competitive Advantage Competitive Advantage When a firm sustains profits that exceed the average for its industry, the firm is said to possess a competitive advantage over its rivals. The goal of much of business strategy is

More information

HOW TO IMPROVE QUALITY AND EFFICIENCY USING TEST DATA ANALYTICS

HOW TO IMPROVE QUALITY AND EFFICIENCY USING TEST DATA ANALYTICS HOW TO IMPROVE QUALITY AND EFFICIENCY USING TEST DATA ANALYTICS Discover 8 ways in our guide for advanced manufacturers Do you perform advanced manufacturing in an industry such as aerospace, automotive,

More information

Software Quality Management

Software Quality Management Software Lecture 9 Software Engineering CUGS Spring 2011 Kristian Sandahl Department of Computer and Information Science Linköping University, Sweden A Software Life-cycle Model Which part will we talk

More information

MKS Integrity & CMMI. July, 2007

MKS Integrity & CMMI. July, 2007 & CMMI July, 2007 Why the drive for CMMI? Missed commitments Spiralling costs Late delivery to the market Last minute crunches Inadequate management visibility Too many surprises Quality problems Customer

More information

Market Research. Market Research: Part II: How To Get Started With Market Research For Your Organization. What is Market Research?

Market Research. Market Research: Part II: How To Get Started With Market Research For Your Organization. What is Market Research? Market Research: Part II: How To Get Started With Market Research For Your Organization Written by: Kristina McMillan, Sr. Project Manager, SalesRamp Scope: This white paper discusses market research on

More information

Developing and Implementing a Balanced Scorecard: A Practical Approach

Developing and Implementing a Balanced Scorecard: A Practical Approach RL Consulting Developing and Implementing a Balanced Scorecard: A Practical Approach White Paper Prepared by: Rick Leopoldi March 31, 2004 Copyright 2004. All rights reserved. Duplication of this document

More information

Ten steps to better requirements management.

Ten steps to better requirements management. White paper June 2009 Ten steps to better requirements management. Dominic Tavassoli, IBM Actionable enterprise architecture management Page 2 Contents 2 Introduction 2 Defining a good requirement 3 Ten

More information

ENTERPRISE ARCHITECTUE OFFICE

ENTERPRISE ARCHITECTUE OFFICE ENTERPRISE ARCHITECTUE OFFICE Date: 12/8/2010 Enterprise Architecture Guiding Principles 1 Global Architecture Principles 1.1 GA1: Statewide Focus 1.1.1 Principle Architecture decisions will be made based

More information

SAMPLE INVITATION TO TENDER ADVERTISEMENT (CONTRACT)

SAMPLE INVITATION TO TENDER ADVERTISEMENT (CONTRACT) SAMPLE INVITATION TO TENDER ADVERTISEMENT (CONTRACT) Invitation to Tender [Insert brief description of project/consultancy E.g. provision of legal services for X native title claim ]. [Name of Representative

More information

The ROI of Systems Engineering: Some Quantitative Results for Software-Intensive Systems

The ROI of Systems Engineering: Some Quantitative Results for Software-Intensive Systems The ROI of Systems Engineering: Some Quantitative Results for Software-Intensive Systems Barry Boehm, 1 Ricardo Valerdi, 2, * and Eric Honour 3 Regular Paper 1 Center for Systems & Software Engineering,

More information

PROJECT MANAGEMENT PLAN CHECKLIST

PROJECT MANAGEMENT PLAN CHECKLIST PROJECT MANAGEMENT PLAN CHECKLIST The project management plan is a comprehensive document that defines each area of your project. The final document will contain all the required plans you need to manage,

More information

Application Security in the Software Development Lifecycle

Application Security in the Software Development Lifecycle Application Security in the Software Development Lifecycle Issues, Challenges and Solutions www.quotium.com 1/15 Table of Contents EXECUTIVE SUMMARY... 3 INTRODUCTION... 4 IMPACT OF SECURITY BREACHES TO

More information

Reaching CMM Levels 2 and 3 with the Rational Unified Process

Reaching CMM Levels 2 and 3 with the Rational Unified Process Reaching CMM Levels 2 and 3 with the Rational Unified Process Rational Software White Paper TP174 Table of Contents INTRODUCTION... 1 LEVEL-2, REPEATABLE... 3 Requirements Management... 3 Software Project

More information

Selecting Enterprise Software

Selecting Enterprise Software Selecting Enterprise Software Introduction The path to selecting enterprise software is riddled with potential pitfalls and the decision can make or break project success, so it s worth the time and effort

More information

Building Relationships by Leveraging your Supply Chain. An Oracle White Paper December 2001

Building Relationships by Leveraging your Supply Chain. An Oracle White Paper December 2001 Building Relationships by Leveraging your Supply Chain An Oracle White Paper December 2001 Building Relationships by Leveraging your Supply Chain EXECUTIVE OVERVIEW This white paper illustrates why a fusion

More information

Lean operations: measurable results

Lean operations: measurable results Lean operations: measurable results Philips Industry Consulting 2 Philips Industry Consulting Your expert guide on the lean journey When you choose to bring lean to your operations, you are embarking on

More information

Project Knowledge Areas

Project Knowledge Areas From Houston S: The Project Manager s Guide to Health Information Technology Implementation. Chicago: HIMSS; 2011; pp 27 39. This book is available on the HIMSS online bookstore at www. himss.org/store.

More information

Agile project portfolio manageme nt

Agile project portfolio manageme nt Agile project portfolio manageme nt Agile project & portfolio summit at Harrisburg University May 9, 2016 Agile project portfolio management Agenda Portfolio management challenges Traditional portfolio

More information

CSTE Mock Test - Part III Questions Along with Answers

CSTE Mock Test - Part III Questions Along with Answers Note: This material is for Evaluators reference only. Caters to answers of CSTE Mock Test - Part III paper. 1. Independence is important in testing is mostly due to the fact that (Ans: C) a. Developers

More information

Open Source CRM Benefits and Case Studies in the Public Sector Richard Baldwin, Pacific Northwest Regional VP

Open Source CRM Benefits and Case Studies in the Public Sector Richard Baldwin, Pacific Northwest Regional VP Open Source CRM Benefits and Case Studies in the Public Sector Richard Baldwin, Pacific Northwest Regional VP 2007 SugarCRM Inc. All rights reserved. Agenda Government Adoption of Open Source Software

More information

Guideline to purchase a CRM Solution

Guideline to purchase a CRM Solution Guideline to purchase a CRM Solution esphere Whitepaper Content list Introduction... 3 Challenges... 3 Overview... 4 Define Your CRM Requirements and Business Objectives and start gather information...

More information

Using Measurement to translate Business Vision into Operational Software Strategies

Using Measurement to translate Business Vision into Operational Software Strategies Using Measurement to translate Business Vision into Operational Software Strategies Victor R. Basili University of Maryland and Fraunhofer Center - Maryland BUSINESS NEEDS Any successful business requires:

More information

Relationship Manager (Banking) Assessment Plan

Relationship Manager (Banking) Assessment Plan Relationship Manager (Banking) Assessment Plan ST0184/AP03 1. Introduction and Overview The Relationship Manager (Banking) is an apprenticeship that takes 3-4 years to complete and is at a Level 6. It

More information

ORACLE TELESALES ORACLE DATA SHEET KEY FEATURES

ORACLE TELESALES ORACLE DATA SHEET KEY FEATURES ORACLE TELESALES KEY FEATURES Maintain customer and contact information View of business activities across operating units Lead and opportunity management Quote and order management Forecast management

More information

Take value-add on a test drive. Explore smarter ways to evaluate phone data providers.

Take value-add on a test drive. Explore smarter ways to evaluate phone data providers. White Paper Take value-add on a test drive. Explore smarter ways to evaluate phone data providers. Employing an effective debt-collection strategy with the right information solutions provider helps increase

More information

ASAE s Job Task Analysis Strategic Level Competencies

ASAE s Job Task Analysis Strategic Level Competencies ASAE s Job Task Analysis Strategic Level Competencies During 2013, ASAE funded an extensive, psychometrically valid study to document the competencies essential to the practice of association management

More information