ISTQB Expert level Improving the testing process First practical experience www.ip-consult.be Isabelle Robrechts Version 1.1 Agenda Training content Practical experience Conclusion 2 2014 Improve Quality Services BV 01 01 Course Introduction 1
ISTQB Structure Bron : www.istqb.org IP-Consult bvba 3 Goal ISTQB Expert ITP Coaching/Leading test process improvement tracks in own organisation Goal part 1 : Assessing the test processes Improvement models Linking and situate Goal part 2 : Improving the test processes Change Management Organisation and human factors 4 2014 Improve Quality Services BV 01 01 Course Introduction 2
IDEAL Part 1: Assessing Part 2: Implementing 2014 Improve Quality Services B.V. 5 Overview Improvement Approaches Approaches Model based Analytical Hybrid / Mixed Other IT/SW Testing GQM Measures Metrics Indicators Analytical & Models Analytical & Content People Skills Standards & Regulations Infrastructure & Tools CMMI IEEE15504 Process models Content models Causal Analysis Evaluations TMMi TPI Next CTP STEP (TMap) 2014 Improve Quality Services B.V. 6 2014 Improve Quality Services BV 01 01 Course Introduction 3
Agenda Training content Practical experience Conclusion 7 Practical experience: Overview Environment Business goals Retrospective Starting point Transfer to improvement: Causal analysis? exercise Improvement proposition Short term Long term 8 2014 Improve Quality Services BV 01 01 Course Introduction 4
Environment Financial institution SDLC : Combi V-model / Agile (inhouse invention) IS : tester is not a profession but a role (<-> Bsn) 9 Business goals To go quickly to the market with new products Risk-based testing! Regression testing! Good quality of the requested products Fit for purpose Passive testing! Less nuisance for the operations department 10 2014 Improve Quality Services BV 01 01 Course Introduction 5
Retrospective Assignment 2009 : IS Set-up : test organization test methodology test culture testing Representing the department in an organizational testing competence center Representing the IS testing at business meetings Report to the department management just CTAL 11 Retrospective : where do I start? What is present? Test work products? Test Policy Test Strategy (Master)Test Plan Test Cases Test culture Test process? Test Tooling Test Management Defect Logging 12 2014 Improve Quality Services BV 01 01 Course Introduction 6
Retrospective : where do I start? Knowledge through measuring is power... But what are you going to measure? Inspired by the circumstances? Inspired by others (lecture, management,...) Inspired by own interests 13 Retrospective : Data Collect Defects : # incidents in production # scope changes per phase # defects per phase (incl.priority) Test cases : # written test cases per phase (incl.priority) # executed, (in-)correct test cases Budget : Planned & Actual test budget Planned & Actual total budget Planned & Actual Realization (Analysis & Development) Planned & Actual repair budget 14 2014 Improve Quality Services BV 01 01 Course Introduction 7
Retrospective : first metrics... Can we obtain information from this data? Test Effectiveness How many defects are found per test case? How many defects are found per test phase? HOW EFFECTIVE ARE OUR TESTS? Test Efficiency What s the price of a defect found during the ST/SIT phase vs a defect found during the UAT phase? 15 Retrospective : Test Effectiveness How effective are our tests?? Beware of the numbers! 16 2014 Improve Quality Services BV 01 01 Course Introduction 8
Retrospective : Test Efficiency 17 Transfer to improvement : Causal Analysis? Pre-condition : an effective defect and/or incident management process 4 steps : 1. Select the items for causal analysis 2. Collect and organize the information 3. Identify any root causes 4. Draw conclusions 18 2014 Improve Quality Services BV 01 01 Course Introduction 9
1. Select the items for Causal Analysis Defects/incidents Irregularities reported by stakeholders Problems found during the analysis Aids: The pareto principle : 80/20 Categorization of defects Analysis of statistics Results of retrospectives 19 Analysis of statistics Reality: incidents doubled the last two years normal requests (no projects) decreased Peaks are indicating the close-out of a project. 20 2014 Improve Quality Services BV 01 01 Course Introduction 10
Analysis of statistics 21 2. Collect and organize the information Let someone take the lead Aid while structuring the data : Ishikawa 22 2014 Improve Quality Services BV 01 01 Course Introduction 11
2. Collect and organize the information Planning Infrastructure Management Unrealistic Project planning Unavailability of the test environment is 1 on 5 Tests are delayed by strict authorizations Minimal budgets for resources Lack of van PM capacities Monthly refresh of test data Testing is a role, no profession No money for trainings No budget for resources Bsn is often seen as the enemy by IS Missing milestines is easily Accepted. No commitment to any test work # Incidents doubles in the last 2 years No transparancy in the IS tests IS : why loose time testing now if it is thoroughly tested later? No professional IS testers Defects are found much too late Motivation/work spirit often very low Communication Culture Human 23 3. Identificy the root causes Identify the most-likely causes Work top-down through the causes Add (if necessary) extra information to the analysis Identify any Tipping Points or Broken windows Maintain brainstorm rules 24 2014 Improve Quality Services BV 01 01 Course Introduction 12
4. Draw conclusions Are there any conclusions? Be Aware for single events? Do we find any problems in the test process which can be avoided in the future? Are there any other processes involved? Mutual causes? 25 Improvement proposal Short term : some quick-wins are defined: More transparancy between IS and Bsn Visualisation of the businessplan with all milestones Weekly reporting on planning at the Division Management Definition of quality gates and its controles Better communication (1N) Usage of 1 Test Management Tool (creates responsibility) 26 2014 Improve Quality Services BV 01 01 Course Introduction 13
Improvement Proposal LongTerm : elaborate an improvement track based on TPI Next, with possiblity to expand to CMMi Why TPI Next? And not TMMi? Test process exists (TMAP) Test process has already a certain maturity Emotional sensitive to improvement Clear visibility of the impact on the business goals Link with CMMi exists Fit in the model... How does it look dd 2014Q1? TPI model in oktober Where do we want to go? 27 Improvement Proposal : Long Term TPI Next : Test Maturity Matrix (dd 2014Q1) TPI Next : Test Maturity Matrix (dd 2014Q4) 28 2014 Improve Quality Services BV 01 01 Course Introduction 14
Improvement Proposal : Long Term TPI Next : Test Maturity Matrix (2016) Maturity Level : - Controlled on all key areas - Evolution from Efficient on the business critical key areas 29 Don t forget the soft skills! IP-Consult bvba 30 2014 Improve Quality Services BV 01 01 Course Introduction 15
Agenda Training content Practical experience Conclusion 31 Conclusion Models and causale analyse are complement Models for Long Term visions Causale analyse for selection and short term activities In the training links between techniques are shown Assignments connect the theory with own practical experiences Don t forget the soft skills 32 2014 Improve Quality Services BV 01 01 Course Introduction 16
33 2014 Improve Quality Services BV 01 01 Course Introduction 17