Measuring the benefits of verification Jan Jaap Cannegieter SYSQA B.V. Almere Almere Quality Assurance in ICT / 1 Agenda Measuring the benefits of SPI Reasons for implementing reviews / inspections Measuring the benefits of verification in theory and in practice Three cases Almere Quality Assurance in ICT / 2 Pagina 1
Why measure the benefits of SPI? Justify investment in SPI / CMMI etc. It s not core business Provide insight in the performance of processes Measure the capabilities of employees Almere Quality Assurance in ICT / 3 Typical ROI measurements Productivity X hour per FP/LOC Quality Number of defects per FP/LOC Costs Costs per FP/LOC compared to peer Time Planning accuracy Almere Quality Assurance in ICT / 4 Pagina 2
Why is it so hard? No historical data Quality of data disputed Good data demands mature processes Good data demands time Almere Quality Assurance in ICT / 5 Almere Quality Assurance in ICT / 6 Pagina 3
The quickest and easiest win Reviews and inspections Almere Quality Assurance in ICT / 7 Examples of reviews and inspections Peer review Expert review Management review Structured walkthrough Inspection Audit Almere Quality Assurance in ICT / 8 Pagina 4
Advantages of reviews / inspections Easy to implement Little resistance Gives much insight in processes Helps to create the need for a quality system Benefits easy to measure Almere Quality Assurance in ICT / 9 Theoretical basis of measuring benefits Costs of rework 10.000 9.000 8.000 7.000 6.000 5.000 4.000 3.000 2.000 1.000 0 REQ Design Build ST AT Production Project phase Almere Quality Assurance in ICT / 10 Pagina 5
Assumptions All defects are found in testing Capers Jones: 85%-95% are found in testing Finding 1 mayor defect takes 1 hour Advantages simplified Phase Requirements Functional design Technical design Build System test Acceptance test Production Ratio 1 2 4 8 16 32 64 Benefit 64 32 16 8 4 2 1 Almere Quality Assurance in ICT / 11 Products of a review / inspection Defects Data Time spend # defect Example: Inspection of a functional design Time: 50 hours 24 mayor defects ROI: (24*16)/50 = 7.68 Almere Quality Assurance in ICT / 12 Pagina 6
Case 1 Semi state controlled organization CMM-assessment Time driven project management Testing as primary defect finding activity Improvement project No CMM! Reviews / inspections Quality system Almere Quality Assurance in ICT / 13 Case 1 - results Implementation time: four months ROI first quarter: 7.92 ROI second quarter: 6.26 Investment in SPI earned back within 6 months Quality increased from 42% to 75% Almere Quality Assurance in ICT / 14 Pagina 7
Case 2 Insurance company CMMI-assessment No management of requirements No coordination between the teams Testing only quality measure Improvement project CMMI continuous Requirements management Verification Process Definition Almere Quality Assurance in ICT / 15 Case 2 - results ROI first quarter: 7.3 ROI second quarter: 6.9 Savings on not accepting ambiguous requirements: 50.000 Creation of the quality system Almere Quality Assurance in ICT / 16 Pagina 8
Case 3 System development outsourced QA done by SYSQA Inspecting work products ROI: 20 Supplier realized he had to deliver quality Almere Quality Assurance in ICT / 17 Lessons learned Short term measurement of ROI isn t difficult ROI-figures convince organizations This is no long term measurement! Review data provides insight in processes Employees accept the calculation method Almere Quality Assurance in ICT / 18 Pagina 9
Thank you and lots of success Literature: Peer reviews in software 0201734850 Kwaliteitszorg in ICT-projecten (PROQA) 9044003690 Almere Quality Assurance in ICT / 19 Pagina 10