How CMMI contributes to Software Testing Dr. Uwe Hehn method park Software AG Uwe.Hehn@methodpark.de
Contents 1. Motivation for S/W Quality Models 2. Why Testers should have some knowledge of Quality Models 3. CMMI: Some Basics 4. CMMI and S/W Test: Finding test relevant material 5. A tester s view at the CMMI Level 2 process areas 6. To what extent do we find Best Test Practises in CMMI? 7. Conclusion EuroSTAR 2003 CMMI+Test / 2
Motivation for S/W Quality Models 1 EuroSTAR 2003 CMMI+Test / 3
Motivation for S/W Quality Models orders software Customer requirements delivers s/w product S/W Manufacturer develops s/w delivers s/w EuroSTAR 2003 CMMI+Test / 4
Motivation for S/W Quality Models orders software Customer requirements delivers s/w product S/W Manufacturer develops s/w delivers s/w delivery on time? at planned costs? quality as expected? delivery on time? at planned costs? quality as expected? EuroSTAR 2003 CMMI+Test / 5
Motivation for S/W Quality Models Customer's Requirements Software development...... software development Product EuroSTAR 2003 CMMI+Test / 6
Motivation for S/W Quality Models Customer's Requirements Software development Project management Quality management Engineering...... Test Quality models for software development make the development process clear by partitioning it into several processes with specific purposes and describing the interdependence of these processes Product EuroSTAR 2003 CMMI+Test / 7
Why Testers should have some knowledge of Quality Models 2 a. We focus mainly on Assessment and Maturity Models b. Testers? Test managers, Test leaders EuroSTAR 2003 CMMI+Test / 8
Why Testers should have some knowledge of Quality Models Testing in real life Quality models EuroSTAR 2003 CMMI+Test / 9
Why Testers should have some knowledge of Quality Models Testing in real life Quality models Q T Hostility? EuroSTAR 2003 CMMI+Test / 10
Why Testers should have some knowledge of Quality Models Testing in real life Quality models Q or Partnership? T EuroSTAR 2003 CMMI+Test / 11
Why Testers should have some knowledge of Quality Models What Quality Models are good for in general Description of whole software development process Assessment according to some goal (eg Level ) Improvement of processes - by learning from Best Practises Software development Project management Quality management Engineering...... Test EuroSTAR 2003 CMMI+Test / 12
Why Testers should have some knowledge of Quality Models and what Testers can expect from Quality Models Understanding how the Test Process is related to and embedded into the overall development process Be aware of the mutual requirements of the test process and other processes Learning from Best Practises Tester s View Software development Project management Quality management Engineering...... Test EuroSTAR 2003 CMMI+Test / 13
Why Testers should have some knowledge of Quality Models Quality models are not only theory: A real-world example The leading German automotive manufacturers have founded a common interest group HIS - the Manufacturer Initiative - Software HIS has worked out concrete requirements for several software development processes - software test is one of the major topics Every contractor should successfully pass assessments with regard to the HIS selection of processes Otherwise the contractor status could be lost EuroSTAR 2003 CMMI+Test / 14
CMMI: Some Basics 3 EuroSTAR 2003 CMMI+Test / 15
CMMI: Some Basics EuroSTAR 2003 CMMI+Test / 16
CMMI: Some Basics CMMI = successor of SW-CMM (and other CMM derivates) CMMI claims to provide guidance for improving your organization s processes It can be considered as a general assessment and improvement model particularly suited for the development of software CMMI (SE/SW) consists of 25 so-called process areas which are grouped to 5 levels (staged representation of the model, level 2 up to level 5) EuroSTAR 2003 CMMI+Test / 17
CMMI: Some Basics CMMI Level 2 Process Areas Necessary basis of s/w development (focus on projects) Contains mostly organisational and supporting process areas Requirements Management (REQM) Project Planning (PP) Project Monitoring and Control (PMC) Supplier Agreement Management (SAM) Measurement and Analysis (MA) Process and Product Quality Assurance (PPQA) Configuration Management (CM) We will show a tester s view of these process areas EuroSTAR 2003 CMMI+Test / 18
CMMI: Some Basics CMMI Level 3 Process Areas The Engineering process areas including Verification and Validation are part of CMMI Level 3 (focus on standard processes) Requirements Development (RD) Technical Solution (TS) Product Integration (PI) Verification (VER) Validation (VAL) plus 9 more non-engineering process areas EuroSTAR 2003 CMMI+Test / 19
CMMI: Some Basics CMMI Specific Goals and Practises focus of study CMMI process area required, but very abstract specific goal specific goal generic goal specific practise specific practise Additional: notes, typical work products, examples, suggestions for implementation examples for implementation EuroSTAR 2003 CMMI+Test / 20
CMMI and S/W Test: Finding test relevant material 4 Characterisation of CMMI process areas regarding relevance to test: A proposal. EuroSTAR 2003 CMMI+Test / 21
CMMI and S/W Test: Finding test relevant material Significant parameters of a CMMI process area Number s of specific practises relevant to test (compared to total number n of specific practises (s/n) Extent of material relevant to test (number p of pages in CMMI specification) (a rough estimation of this number will be sufficient) Notation: PA (s/n, p) Example: Requirements Management (2/5, 2) By these two parameters the process areas most relevant for s/w testing are easily identified and ordered EuroSTAR 2003 CMMI+Test / 22
CMMI and S/W Test: Finding test relevant material Project Planning 2 0,55 Project Monitoring and Control 3 0,10 Supplier Agreement Manag. 4 1,29 Measurement and 20,00 Analysis 5 0,06 Proc. and Prod Quality 18,00Assur. 6 0,00 Configuration Management 16,00 7 2,00 Requirements Development 14,00 8 1,65 Technical Solution 12,00 9 2,86 Product Integration 10,00 10 2,33 Verification 8,00 11 17,50 Validation 6,00 12 12,00 4,00 2,00 0,00 Coverage Test-relevant CMMI processes Requirements Management Project Planning Project Monitoring and Control Supplier Agreement Manag. Measurement and Analysis Proc. and Prod Quality Assur. Product Integration Configuration Management Requirements Development Technical Solution CMMI process areas Verification Validation EuroSTAR 2003 CMMI+Test / 26
CMMI and S/W Test: Finding test relevant material Masking out Verification and Validation Project Planning 2 0,55 Project Monitoring and Control 3 0,10 Supplier Agreement Manag. 4 1,29 Measurement and Analysis 3,50 5 0,06 Proc. and Prod Quality 3,00Assur. 6 0,00 Configuration Management 7 2,00 Requirements Development 2,50 8 1,65 Technical Solution2,00 9 2,86 Product Integration 1,50 10 2,33 Coverage Test-relevant CMMI processes (without VER/VAL) 1,00 Test 0,50 0,00 Requirements Management Project Planning Project Monitoring and Control Configuration Management Supplier Agreement Manag. Measurement and Analysis Proc. and Prod Quality Assur. Requirements Development CMMI process areas Technical Solution Product Integration EuroSTAR 2003 CMMI+Test / 27
A tester s view at the CMMI Level 2 process areas Software development 5 CMMI 5 CMMI 4 CMMI 3 CMMI 2 Project management Quality management Engineering...... Test We will have a look at test-relevant specific practises of selected Level 2 process areas EuroSTAR 2003 CMMI+Test / 28
A tester s view at the CMMI Level 2 process areas Level 2: Requirements Management SP1.1 Obtain an Understanding of Requirements Develop an understanding with the requirements providers on the meaning of the requirements 2: Establish objective criteria for the acceptance of requirements Lack of acceptance criteria often results in inadequate verification, costly rework, or customer injection Examples of acceptance criteria include Clearly and properly stated Complete Consistent with each other Uniquely identified Verifiable (testable) EuroSTAR 2003 CMMI+Test / 29
A tester s view at the CMMI Level 2 process areas Level 2: Measurement and Analysis SP 1.2 Specify Measures Specify measures to address the measurement objectives. Examples of commonly used base measures include the following: Quality measures (e.g., number of defects, number of defects by severity) Examples of commonly used derived measures include the following: Defect density Peer review coverage Test or verification coverage Quality measures (e.g., number of defects by severity/total number of defects) EuroSTAR 2003 CMMI+Test / 33
To what extent do we find Best Test Practises in CMMI? Categories according to ISEB / ISTQB certified tester syllabus 6 Basics of Software Testing Testing in the Software Life Cycle Static Test / Dynamic Test Test management CMMI 5 CMMI 4 CMMI 3 CMMI 2 Checklists We inspect Level 2 process areas plus the Engineering process areas of Level 3 EuroSTAR 2003 CMMI+Test / 35
To what extent do we find Best Test Practises in CMMI? a. Basics of Software Testing EuroSTAR 2003 CMMI+Test / 36
Basics of Software Testing Test and Requirements Testers should be considered as stakeholders for Developing Customer Requirements [RD SG1+GP 2.7] Requirements on Requirements (Examples) [RD SP 2.1 SubP1] Involvement in definition of product or product component requirements to ensure that these requirements are verifiable [VER SP 1.1 + SubP4+5] EuroSTAR 2003 CMMI+Test / 37
Basics of Software Testing Procedure of Verification or Validation Perform Peer Reviews on selected work products [VER SG2] Prepare for Peer Reviews [VER SP 2.1] Conduct Peer Reviews [VER SP 2.2] Analyze Peer Review Data [VER SP 2.3] Verification of selected work products [VER SG3] Perform Verification [VER SP 2.3] Analyze Results [VER SP3.2] Validation of selected work products [VAL SG2] Perform Verification [VAL SP2.1] Analyze Results [VAL SP2.2] EuroSTAR 2003 CMMI+Test / 38
To what extent do we find Best Test Practises in CMMI? b. Testing in the Software Life Cycle EuroSTAR 2003 CMMI+Test / 39
Testing in the Software Life Cycle Unit tests and Peer Reviews [TS SP3.1 (Implement the Design) + SubP 3+4] Integration Integration sequence [PI SP1.1, Determine Integration Sequences ] Integration Procedures [PI SP1.3, Establish Product Integration Procedures and Criteria, SubP3 (++)] Interfaces [PI SP2.2, Manage Interfaces ] Integration Test (but: only mentioned once, at PI SP1.3) EuroSTAR 2003 CMMI+Test / 40
Testing in the Software Life Cycle System Integration Test [PI SP3.3, Evaluate Assembled Product Components ] System-/Acceptance Test Preparation [VAL SP1.1, Select Products for Validation ; 1.2, Establish the Validation Environment, 1.3, Establish Validation Procedures and Criteria ] Performance [VAL SP2.1, Perform Validation; 2.2, Analyze Validation Results ] Acceptance Reviews and Test [SAM SP 1.3, Establish Supplier Agreements ; 2.2, Execute the Supplier Agreement ] EuroSTAR 2003 CMMI+Test / 41
To what extent do we find Best Test Practises in CMMI? c1. Static Test EuroSTAR 2003 CMMI+Test / 42
Static Test Peer Reviews of product components [TS SP3.1 SubP3, Implement the Design ] Reviews of Work Products [TS GP2.9 Elaboration, Objectively Evaluate Adherence ] Perform Peer Reviews [VER SG2] Prepare [VER SP2.1, Prepare for Peer Reviews ] Conduct [VER SP2.2, Conduct Peer Reviews ] Analyze Data [VER SP2.3, Analyze Peer Review Data ] EuroSTAR 2003 CMMI+Test / 43
To what extent do we find Best Test Practises in CMMI? c2. Dynamic Test EuroSTAR 2003 CMMI+Test / 44
Dynamic Test Coverage testing, boundary value testing, special value testing [TS SP3.1, Examples, Implement the Design ] Path coverage testing, Decision-table-based testing, Functional decomposition-based testing [VER SP1.1, Examples for SWE, Select Work Products for Verification ] Load, stress and performance testing, [VER SP1.1, Examples for SWE, Select Work Products for Verification ] EuroSTAR 2003 CMMI+Test / 45
To what extent do we find Best Test Practises in CMMI? d. Test management EuroSTAR 2003 CMMI+Test / 46
Test management Identify verification criteria/procedure for the product integration environment [PI SP1.2 SubP2, Establish the Product Integration Environment ] Develop an integration environment if a suitable environment cannot be acquired [PI SP1.2 SubP4, Establish the Product Integration Environment ] Degree of simulation permitted for a product component to pass a test, constraints of the environment to be used for the integration test [PI SP1.3, Establish Product Integration Procedures and Criteria ] Establish verification environment (incl. equipment and tools) [VER SP1.2, Establish the Verification Environment, SubP1-4] EuroSTAR 2003 CMMI+Test / 47
Test management Kind of testing is work product-specific [VER SP1.1, Typical Work Products 1+2, Select Work Products for Verification ] Definition of test methods and criteria [VER SP1.3, Establish Verification Procedures and Criteria ] EuroSTAR 2003 CMMI+Test / 48
Conclusion 7 EuroSTAR 2003 CMMI+Test / 49
Conclusion: Some observations Of course Verification and Validation belong to the most important process areas relating to s/w test But there are some more very important process areas related to s/w test, e.g. Requirements Management Requirements Development Technical Solution Product Integration Supplier Agreement Management Note: the basic Non-Engineering process areas of Level 2 contain a lot of test-relevant information EuroSTAR 2003 CMMI+Test / 50
Conclusion: Some impressions Peer Reviews are considered important in a general sense Test is considered to be one of several methods of verification or validation Verification/validation issues are woven into the CMMI model no straightforward mapping of the tester s view in CMMI onto the usual models of testing EuroSTAR 2003 CMMI+Test / 51
Conclusion: How CMMI contributes to S/W testing? Summary 1 CMMI gives valuable support for test leaders and test managers ( checklist for setting up the process ) Even if only aiming at CMMI level 2: there are important recommendations for test leaders and test managers And: There are a lot of concrete examples regarding s/w testing and related areas EuroSTAR 2003 CMMI+Test / 52
Conclusion: How CMMI contributes to S/W testing? Summary 2 Compared with SW-CMM CMMI cares much more for test aspects, but - of course - CMMI is not a dedicated Test Improvement Model Verification and validation is seen as an integral part of the overall development process EuroSTAR 2003 CMMI+Test / 53
Conclusion EuroSTAR 2003 CMMI+Test / 54
Conclusion: Questions?! Slides? Uwe.Hehn@methodpark.de EuroSTAR 2003 CMMI+Test / 55