Measuring the benefits of verification. Jan Jaap Cannegieter. SYSQA B.V. Almere

Transcription

1 Measuring the benefits of verification Jan Jaap Cannegieter SYSQA B.V. Almere Almere Quality Assurance in ICT / 1 Agenda Measuring the benefits of SPI Reasons for implementing reviews / inspections Measuring the benefits of verification in theory and in practice Three cases Almere Quality Assurance in ICT / 2 Pagina 1

2 Why measure the benefits of SPI? Justify investment in SPI / CMMI etc. It s not core business Provide insight in the performance of processes Measure the capabilities of employees Almere Quality Assurance in ICT / 3 Typical ROI measurements Productivity X hour per FP/LOC Quality Number of defects per FP/LOC Costs Costs per FP/LOC compared to peer Time Planning accuracy Almere Quality Assurance in ICT / 4 Pagina 2

3 Why is it so hard? No historical data Quality of data disputed Good data demands mature processes Good data demands time Almere Quality Assurance in ICT / 5 Almere Quality Assurance in ICT / 6 Pagina 3

4 The quickest and easiest win Reviews and inspections Almere Quality Assurance in ICT / 7 Examples of reviews and inspections Peer review Expert review Management review Structured walkthrough Inspection Audit Almere Quality Assurance in ICT / 8 Pagina 4

5 Advantages of reviews / inspections Easy to implement Little resistance Gives much insight in processes Helps to create the need for a quality system Benefits easy to measure Almere Quality Assurance in ICT / 9 Theoretical basis of measuring benefits Costs of rework REQ Design Build ST AT Production Project phase Almere Quality Assurance in ICT / 10 Pagina 5

6 Assumptions All defects are found in testing Capers Jones: 85%-95% are found in testing Finding 1 mayor defect takes 1 hour Advantages simplified Phase Requirements Functional design Technical design Build System test Acceptance test Production Ratio Benefit Almere Quality Assurance in ICT / 11 Products of a review / inspection Defects Data Time spend # defect Example: Inspection of a functional design Time: 50 hours 24 mayor defects ROI: (24*16)/50 = 7.68 Almere Quality Assurance in ICT / 12 Pagina 6

7 Case 1 Semi state controlled organization CMM-assessment Time driven project management Testing as primary defect finding activity Improvement project No CMM! Reviews / inspections Quality system Almere Quality Assurance in ICT / 13 Case 1 - results Implementation time: four months ROI first quarter: 7.92 ROI second quarter: 6.26 Investment in SPI earned back within 6 months Quality increased from 42% to 75% Almere Quality Assurance in ICT / 14 Pagina 7

8 Case 2 Insurance company CMMI-assessment No management of requirements No coordination between the teams Testing only quality measure Improvement project CMMI continuous Requirements management Verification Process Definition Almere Quality Assurance in ICT / 15 Case 2 - results ROI first quarter: 7.3 ROI second quarter: 6.9 Savings on not accepting ambiguous requirements: Creation of the quality system Almere Quality Assurance in ICT / 16 Pagina 8

9 Case 3 System development outsourced QA done by SYSQA Inspecting work products ROI: 20 Supplier realized he had to deliver quality Almere Quality Assurance in ICT / 17 Lessons learned Short term measurement of ROI isn t difficult ROI-figures convince organizations This is no long term measurement! Review data provides insight in processes Employees accept the calculation method Almere Quality Assurance in ICT / 18 Pagina 9

10 Thank you and lots of success Literature: Peer reviews in software Kwaliteitszorg in ICT-projecten (PROQA) Almere Quality Assurance in ICT / 19 Pagina 10