Value-Based Software Engineering Concepts: Overview and Introduction

Transcription

1 Value-Based Software Engineering Concepts: Overview and Introduction Stefan Biffl

2 Motivation Software Engineering aims at creating high-quality (i.e., value adding) products by effectively using scarce resources. These goals influence the selection and tailoring of processes, methods, and tools. Surprisingly, added value is not explicitly addressed in SE standards like the IEEE Software Engineering Body of Knowledge, ISO 900x oder CMM-I. Success-critical project participants (stakeholders) need to understand and align their win conditions to create products, services, and processes, which consistently provide added value. This presentation introduces concepts of Value-Based Software Engineering to elicit sources of value, reconcile value conflicts, and organize software engineering activities according to the value framework in the project (e.g., identify most worthwhile test cases). The value framework provides a foundation for flexibility in project management. Practical examples include Linking software project deliverables and stakeholder win conditions based on benefits realization analysis; Value-based quality assurance planning (e.g., tests, reviews). Recommended reading: books Software Ecosystem (MIT Press, 2003) and Value-Based Software Engineering (Springer, 2005).

3 Value-Based Software Engineering Value in Software Engineering 1. Link between Business Case and Software Requirements. 2. Stakeholder win conditions and risks. Risks are present in all projects and need to be valued and managed. Value-Based with respect to value considerations of success-critical stakeholders. Goals Integration of success-critical project participants over the full life cycle. Make stakeholders value propositions (SVPs) explicit to allow tuning decisions on all levels of SE to these value propositions, e.g.: Do project deliverables actually address stakeholder win conditions? Which requirements should be part of the next release? Which test cases are particularly relevant for the project? Risks Missed/forgotten stakeholders. Hidden/implicit value propositions. Conflicts between stakeholder value propositions.

4 Sources of Value Value Chain Examples Software Product Creation (e.g., classic shrink-wrap software development) Integration of Software Products (e.g., SAP application) Software-based Services (e.g., weather forecast, financial services) Software Process Improvement (e.g., CMM-I, Spice capabilities) Aspects of value [Messerschmitt and Szyperski, 2003] Productivity, better cater to needs Functionality and performance/quality Network Effects, broader usage Operational cost, learnability, usability, security Key non-financial corporate value drivers (Forbes.com with Wharton and Ernst & Young) Innovation Ability to attract talented employees, Alliances, Quality of major processes, products, or services, Environmental performance. Challenge: How to transform understanding about external sources of value into internal software engineering decision making.

5 Example: Medical Logistics Dispatcher System Material Inventory System Client Dispatcher Accounting System Dispatcher Cockpit Sales & Invoicing Truck Assignment Tender Transporter Fleet base Locatn/Status Tracking Dispatching RFID Partner Systems Secure communication channels Institut für Softwaretechnik Pervasive/Mobile und Interaktive Application Systeme

6 System-Level Value Function Typical Marketplace Competition Value Estimating Relationships Internet Services, Wireless Infrastructure: Value Loss vs. System Delivery Time Fixed-schedule Event Support: Value of On-time System Delivery Market Share Loss VL(T d ) Critical Region Market Share Loss VL(T d ) T event System Delivery Time T d System Delivery Time T d Off-line Data Processing: Value Loss vs. System Delivery User Value Loss VL(T d ) System Delivery Time T d [Huang and Boehm, 2005]

7 Risks from Inadequate Requirements Project Challenged Factors (% of Responses) 1. Lack of User Input 12.8% 2. Incomplete Requirements & Specifications 12.3% 3. Changing Requirements & Specifications 11.8% Reasons for project interruption - survey including 365 industrial responses (8.380 applications) [Chaos Report, 1994]: 1. Incomplete requirements (13.1%) 2. Lack of User Involvement (12.4%) 6. Changing Requirements and Specifications (8.7%) Requirements transport value idea into the project. Make sure that requirements actually fit stakeholder value propositions. [Grünbacher, 2006] There are 2 main risks from risk management point of view: a) Incomplete and unclear requirements b) Changing requirements

8 Value Considerations are Success Critical in Software Engineering Success depends on key stakeholder values E.g., it is risky to exclude key stakeholders. Value is not an absolute thing Stakeholders may have different or even conflicting stakeholder value propositions (SVPs) These stakeholder value propositions must be made explicit so decisions at all levels in the project can be optimized to meet or reconcile these SVPs. Win conditions may link to project/process (what/how to make), product (concrete asset features and qualities), and service (use of assets) issues. Software requirements are typically good to capture and track product requirements, other win conditions may be scattered over many documents and changes may not be tracked good enough.

9 Risk Exposure Definition Risk exposure (RE) is defined as: RE = Prob (Loss) * Size (Loss) RISK 1. The probability that an event will become a problem. 2. The expected size of loss as consequence of the problem. Is characterized by Probabilty Is characterized by Size of Loss Is defined by Goals/Expectation Size of loss depends on project goals and expectations. Goals depend on stakeholders. Stakeholder Proposed by

10 Software Engineering Planning: How much Quality Assurance is Enough? Market Share vs. Product Quality (Time to Ship) Risk Exposure Loss financial; reputation; future prospects, For multiple sources of loss: RE = Σ [Prob (Loss) * Size (Loss)] source sources Many defects: high P(L) Critical defects: high S(L) Many rivals: high P(L) Strong rivals: high S(L) RE = P(L) * S(L) Sweet Spot Few rivals: low P(L) Weak rivals: low S(L) Few defects: low P(L) Minor defects: low S(L) [Huang and Boehm, 2005] Time to Ship (amount of testing)

11 VBSE Theory 4+1 Dependency theory How do dependencies affect value realization? What values are important? How is success assured? Theory W: SCS Win-Win Utility theory How important are the values? How do values determine decision choices? Control theory How to adapt to change and value realization? Decision theory Reference: Biffl, S.; Aurum, A.; Boehm, B.; Erdogmus, H.; Grünbacher, P. (Eds.); Value-Based Software Engineering ; 2005, 388 p., Springer-Verlag, ISBN:

12 Value-Based SE Key Process Steps Identification of success-critical project participants (Stakeholder) Benefits Realization Analysis (Analysis of organizational/technology dependencies) Elicit and validate stakeholder value propositions Utility theory: model clash spider web diagram, discounted Cash Flows, Decision trees Negotiate value propositions Decision theory: EasyWinWin process and tool support Win Win Condition involves Issue covers addresses Agreement adopts Option Monitoring und control Reports wrt. to value propositions Change management; risk management (e.g., RiskIT) [Biffl et al., 2005]

13 Key VBSE Process Step 1/4: Identification of Success-Critical Stakeholders Input: Goals of a protagonist (motivator for performing a project) Output: list of success-critical stakeholders Dependency Theory: Provides insights into various organizational, project, process, and product interdependencies What is necessary, what fits, what does not fit, how they are connected. Spans socio-political, environment, and technical dimensions Dependency theory provides the following approaches: Benefits Realization Analysis, Contingency Theory, Model clashes, Critical Path, System Architecture Collaborations are based on organizational/business dependencies Technology dependencies, emerging system properties from subsystem integration Motivation to use: Probability of misfits is usually high. Size of loss due to misfits can be very high (COTS vs. Waterfall).

14 Benefits Realization Analysis Example Step 1: Characterize key deliverables and dependencies between deliverables. Step 2: Define stakeholder groups and their key value propositions. Step 3: Identify supporting (+) & conflicting (-) dependencies between value propositions. Step 4: Link key deliverables to stakeholder value propositions (VBSE book, chapter 6). Do deliverables directly contribute to stakeholder benefits? If not, add success-critical initiatives and assumptions between deliverables and stakeholder benefits (e.g., risk countermeasures). Explore ways to make the project less risky and/or add extra benefits.

15 Value-Based Project Plan (Example)

16 Key VBSE step 2/4: Elicit and validate stakeholder value propositions Goal: Identify stakeholder value propositions (SVPs) and conflicts between SVPs Input: List of success-critical stakeholders (SCSs) Output: value propositions of SCSs and conflicts Utility theory provides the following methods: model clash spider web diagram, discounted cash flows, decision trees Helps identify conflicting stakeholder value-propositions Very useful for expectations management and upfront planning Examples: Maintainers ease of maintenance vs. Developers freedom of choice Users many features vs. Acquirers cost effectiveness Acquirers rigorous contract vs. Developers flexible contract [Biffl et al., 2005]

17 Key VBSE step 3/4: Negotiate value propositions Goal: reach a win-win state Input: set of value propositions and conflicts Output: negotiated and agreed set of SVPs, and risks from open issues. Decision theory provides the following methods: EasyWinWin process and tool support Different stakeholders may have different/conflicting value propositions We need a way to: Identify all success-critical stakeholders Clarify conflicting value propositions in order to avoid the risks of Hidden/implicit win conditions Customer dissatisfaction Waste of money and resources Value-Based Software Engineering is the discipline that deals with value considerations and resolution of conflict.

18 Easy Win Win Negotiation Model Win Condition: objective which makes a stakeholder feel like a winner Issue: conflict or constraint on a win condition Option: a way of overcoming an issue Agreement: mutual commitment to an option or win condition WinWin Equilibrium State - All Win Conditions covered by Agreements - No outstanding Issues Open issues are risks that may prevent the project or need risk management. [Biffl et al., 2005]

19 Easy Win Win Process Identify Success-critical Stakeholders Review and expand negotiation topics Brainstorm stakeholder interests Converge on Win Conditions Capture a glossary of Terms Prioritize Win Conditions Reveal Constraints and Assumptions Identify Issues and Options Negotiate Agreements

20 Requirements Elicitation (1/2) Many stakeholders Hundreds of win conditions Detailed analysis of priorities and conflicts Tool support essential for high-volume stakeholder value proposition collection and negotiation [Grünbacher, 2006]

21 Requirements Elicitation (2/2) [Grünbacher, 2006]

22 Transformation of Value Propositions with QFD Stakeholder value propositions (SVPs) can range from concrete implementation to strategic win conditions. To make project-external SVPs operational, transformation may be needed to clarify objectives for internal roles (e.g., PM, QA) and their decision making. Project-level win conditions can typically be well aligned to product features and qualities, people skills, organizational processes, and project objectives. More strategic win conditions may need further analysis to link them to sufficiently concrete change goals. Quality Function Deployment (QFD) steps can be applied to VBSE for transformation of strategic external SVPs to measurable internal contributions. Definition of scope (e.g., market & product). QFD templates can be downloaded from Start with strategic SVPs. Identify organization-level capabilities that seem most relevant to address the SVPs. Select most relevant and changeable capabilities. Identify most relevant change initiatives to form a change project and derive measurable change targets.

23 Key VBSE step 4/4: Monitoring and Control Goal: Controlling progress toward stakeholder win-win realization Track real progress (i.e., results), not just budget and time used for activities. Input: Stakeholder requirements & project status Output: Value and Risk analysis Control theory provides the following methods: value-based monitoring and control, risk-and opportunity management (e.g., RiskIt) Needs good requirements management (requirements tracing and change management) Foundations for flexible and manageable software projects Automated Build LifeCycle for frequent building of testable product versions. Automated Quality Assurance for Build process and to measure product version quality (level of features and qualities reached). Strong tracking of current SVPs and requirements to project artifacts and product status for value-based reporting.

24 Requirements Tracing Support for VBSE Integrated tool support for IDEs Allows efficient tracking of dependencies between (changing) requirements and software artifacts (e.g., components, test cases). Developed in cooperation with and applied at Siemens Austria (IT Solutions and Services). Benefits Integrated traceability support for requirements managers and developers in their native tool environments (no new tool). Tracing effort reduction compared to external tracing (spreadsheet). Continuous status reporting for project managers allows valuebased project reporting.

25 Defect Reduction Heuristics Boehm and Basili; Software defect reduction report [Boehm and Basili, 2001] Finding and fixing a software problem after delivery is often 100 times mores expensive than finding and fixing it during the requirements and design phase. Current software projects spend about 40 to 50% of their effort on avoidable rework. About 80% of avoidable rework comes from 20% of the defects. About 80% of the defects com from 20% of the modules, and about half of the modules are defect free. About 90% of the downtime comes from, at most, 10% of the defects. Peer reviews (i.e., inspections) catch 60% of the defects. Perspective-based reviews catch 35% more defects than non-directed review. Disciplined personal practices can reduce defect introduction rates by up to 75%. Challenge: How to link these research results to VBSE in practice.

26 Value-Based Approaches for QA/Testing Stakeholder value and risk: Focusing resource allocation with value-based software engineering (VBSE). Challenge: Balancing External and Internal Dimensions of QA External dimension: Market view of quality Stakeholder win conditions define project scope and constraints. Stakeholder value and risk attitude Internal dimension: scope of QA manager how to organize SE & QA. Translate QA goals into planning QA activities. Effective from customer and user points of view Efficient resource usage from project point of view

27 Test Methods Determine a test strategy Goal: find the most important defects early with lowest cost Efficient use of available resources Variation of testing intensity by product component and quality attribute Strategy may change by testing level and flexibly if test budget gets cut. Test case specification Definition of test cases according to test strategy before testing. Checklists Test of quality criteria without executing the program Review-Techniques Check documents and source code in a team Consider Test-Driven Development for early feedback. Better understanding of expected results before implementation. Goes well with build-and-test automation approaches for quality measurement. Better-informed QA and PM from continuous building and testing. [Frast, 2004; Biffl and Schatten, 2007]

28 Value of Priorities The Vital Few and the Useful Many Is every test case, defect, requirement equally important? In the example: 15 different customer billing types Type #1 accounts for about 50% of the transactions Type #1 to #3 together for about 80% Pareto principle: 20% of clients account for 80% of the profits [Ramler, 2006]

29 Best Practices for Value-based Software Testing Requirements-based testing What do the stakeholder value? Risk-based testing What puts the value realization in danger? Test case selection techniques What are your best test cases? TCS Risks [Ramler, 2006] Requirements

30 3 Suitability Accuracy Interoperability Compliance Security Maturiy Fault Tolerance Recoverability Understandability Learnability Operability Attractiveness Time Behavior Resource Utilization View events View events View events View events View events View events View events Requirements- & Risk-based QA Focus Example How to organize QA to focus on the most worthwhile tasks first? Focus QA on: High-value scenarios and system parts; major defects and risks. Allow flexible reaction, if the QA budget gets cut back. Usage Scenarios login <<extend>> view details User view events <<extend>> upload new document attach document <<extend>> Major quality Factors Functionality Registered User add event edit event delete event subscribe to event <<extend>> <<extend>> <<extend>> i nvite a tte nd ees <<i nclude>> set security properties select exisitng document Quality x Feature Matrix Security aspect Feature View events View details Add event Edit event Attach document Set properties Delete event General identity B B C C C C C Quality Aspects Reliability Message content authenticity B B C C C C C Message content origin C C B B B A B Usability Integrity C C A A A A Efficiency Secrecy and privacy B B C B C C C Accountability B B B B B [Ramler et al., 2005; Ramler, 2006]

31 Risk-based Testing Example Level of Detail in Testing Quality Level Basic Operation Detail Feature View events View details Add event Edit event Attach document Set properties Delete event Risk Profile [Ramler, 2006]

32 Test Case Selection Research Studies Random (un-prioritized) Coverage (measured in terms of total number of functions) Change (measured in terms of additional modified functions covered) Optimal (upper bound on prioritization effectiveness) Rothermel G. and S. Elbaum Putting Your Best Tests Forward IEEE Software Aug./Sept [Ramler, 2006]

33 Value-Based QA Patterns in Overview Review of mission-critical test cases Developers make test cases for key requirements; review of these test cases against requirements show issues early (e.g., requirements that are hard to test; different points of view). Test-Driven Development, Test automation Test cases are made before implementation and show continuously in a quantitative way the current state of quality of the system under development. Value-based Testing Requirements- & Risk-based Testing: Tune priorities for test objects and intensity of tests with changing resources (e.g., budget cut for testing). Value-Based Reviewing Reading techniques exist for effectively finding defects in requirements, documents and models (e.g., with scenarios or perspectives). Pair Development & systematic QA Fitting of agile development practices with light-weight repeatable QA methods. Tracing of requirements (and value) to development artifacts Tool support, dependency analysis

34 Summary VBSE in Practice Introduction to Value-Based Software Engineering (VBSE) Value Considerations Value in Software Engineering Planning and Management Value and Risk Assessment Examples in Software Engineering Planning and Management Benefits realization analysis: Linking deliverables to stakeholder win conditions EasyWinWin, RiskIT: Processes with tool support for requirements and risk analysis, evaluation, and management Value-Based QA: Testing, Reviews; test case selection A value-based approach for software project planning helps to make the value propositions of all relevant stakeholders explicit organize engineering activities according to the value framework in the project Benefits Solid foundation for project management options and their evaluation More effective use of available resources Early identification of fundamental conflicts and risks

35 References Biffl, S.: Software Inspection Techniques to support Project and Quality Management, Shaker Verlag, Biffl S., Aurum A., Boehm B., Erdogmus H., Grünbacher P. (eds.); Value-Based Software Engineering, Springer Biffl S., Schatten A. Best-Practice Software Engineering Komplexe Systeme unter Kontrolle halten, Workshop-Unterlagen (in German); Boehm B., Turner R.; Balancing Agility and Discipline, Addison-Wesley, Frast D., Software Testen, lecture notes (in German) TU Wien, Grünbacher Paul Requirements Analysis and Specification, lecture notes, TU Wien, Huang and Boehm, Determining How Much Software Assurance Is Enough ; Presentation at ISERN Meeting ; Lee K., Boehm B. Empirical Results from an Experiment on Value-Based Review (VBR) Processes ; Proc. of Int. Symposium on Empirical Software Engineering; IEEE; Nov Messerschmitt D., Szyperski C.; Software Ecosystems Understanding an Indispensable Technology and Industry, MIT Press, Ramler R., Biffl S., Grünbacher P. "Value-Based Management of Software Testing", in: Biffl S., Aurum A., Boehm B., Erdogmus H., Grünbacher P. (eds.) "Value-Based Software Engineering", Springer Verlag, p ; Ramler R., Value-Based Software Test Management ; Presentation Management Process Summit; November Shaw, M., Arora A., Butler S., Poladian V., Scaffidi C., First Steps Toward a Unified Theory for Early Prediction of Software Value, Proc. Equity, Amsterdam, Software Engineering Best Practices: