SLA For Testing Approach to Quantify Quality Thomas Rumi Imbus AG 1
SLA for Testing Thomas Rumi, imbus AG EuroStar 2005, Copenhagen 2005 Slide 2 2
About imbus Founded in 1992 in Germany Specialist for Software Quality Assurance and Test ~100 employees at three locations Headquarter in Möhrendorf (near Nuremberg) Munich Hofheim (near Frankfurt/Main) 2005 Slide 3
Services and Solutions Softwaretest Services Consulting SW QA SW Test Process Test Outsourcing SLA for Testing Training ISTQB Certified Tester Trainings worldwide Products & Solutions imbus TestBench 2005 Slide 4
SLA for Testing - Agenda Service Level Agreement (SLA) Examples from other domains Characteristics of SLAs SLA for Testing Challenges Approach Example - How to find KPIs for testing? Summary and Outlook 2005 Slide 5
Example - SLAs in a DSL offer Service Level name Key Performance Indicators (KPIs) Price Optional conditions DSL offer from Nefkom (http://www.maxidsl.de) 2005 Slide 6
1st level support Example - SLAs in Help desk services Phone 2nd level support End users E-mail Web portal Expert Expert Expert Key performance indicators (KPIs) Service hours Timeframe for trouble ticket confirmation 2nd level support within guaranteed time Customer Satisfaction Surveys 2005 Slide 7
Characteristics of SLAs Contract between a customer and a service provider Formalized service quality by service levels Service level definition based on clearly defined and measurable KPIs Guarantee of the service level Monitoring of service level based on KPIs Direct mapping of service level and price 2005 Slide 8
SLA Process Customer Service Provider Service Level Requirements Service Level Agreement (Quality guarantees and prices) Service execution (including SLA monitoring) Payment with bonus and remedies SLA assessments / SLA improvement process 2005 Slide 9
Benefits of SLAs Customer Service Provider Get the appropriate level of quality Prevent too much or too little service Guarantee that service quality is good enough Optimal quality / cost ratio Get satisfied customers Visualize service ability and quality Be comparable against competitors Optimize resource planning 2005 Slide 10
SLA for Testing - Agenda Service Level Agreement (SLA) Examples from other domains Characteristics of SLAs SLA for Testing Challenges Approach Example - How to find KPIs for testing? Summary and Outlook 2005 Slide 11
SLA for Testing - Roles Customer Service Provider Product Development Test Outsourcing Provider 2005 Slide 12
SLA for Testing - Challenges SLAs are new on the testing market Adaptable Service Level for testing currently not available Find clearly defined and measurable KPIs Develop new Service Levels Restrict additional monitoring overhead 2005 Slide 13
SLA for Testing - Approach Step 1: Service Level Requirements (SLR) Step 2: Identify the KPIs Step 3: Define Service Levels Step 4: Assign prices to Service Levels Step 5: Improve and maintain Service Levels SLA Framework 2005 Slide 14
Step 1: SLR - Example: Test Management Costs Productivity Quality Minimal costs Fast execution Run the right tests Quick startup of tests Be flexible on release delays Provide clear test status report Appropriate reporting intervals 2005 Slide 15
Step 1: SLR - Example: Bug Reporting Quality Report only essential bugs Prevent misleading or duplicated bugs Precise and clear bug reports Bug reproducible by development team Point to root cause 2005 Slide 16
Step 2: Identify the KPIs - Example: Bug Reporting Measure test effectiveness DDP Bug rates against prev. projects Control QA activities for bug reports Review Reproduce bug Pre-analyze bug Measure the ratio of bad bug report # (Duplicates + Closed Bugs) = # (All found Bugs) Evaluate feedback for bug report from Product management Development 2005 Slide 17
Step 3: Define Service Levels - Example: Bug Reporting and Reporting Intervals Service Levels Category Basic Advanced Expert QA activities for bug reports Critical / Serious Medium Reproduce Reproduce Review Reproduction not guaranteed Reproduce Reproduce Review Root cause analysis Reproduce Review Low No bug report Reproduction not guaranteed Reproduce Reporting interval Test status reports monthly weekly daily Escalation reports weekly weekly immediately 2005 Slide 18
SLA for Testing - Agenda Service Level Agreement (SLA) Examples from other domains Characteristics of SLAs SLA for Testing Challenges Approach Example - How to find KPIs for testing? Summary and Outlook 2005 Slide 19
Summary SLA are useful for testing service, but testing services are complex Main challenge is to identify essential KPIs imbus has launched test outsourcing services with SLAs First customer project with SLAs is up and running 2005 Slide 20
Outlook Our strategy - SLA for Testing Extend SLA offer for all test services Improve current KPI approach within SLA improvement process 2005 Slide 21
Space shuttle Atlantis landing at Edwards Air Force Base, CA. (32K) Image: NASA Interested in further information? WWW: http://www.imbus.de/engl/testlab E-Mail: Thomas.Rumi@imbus.de 2005 Slide 22