Risk-Based Testing Paul Gerrard Technical Director, Systeme Evolutif Limited Systeme Evolutif Limited 3 rd Floor, 9 Cavendish Place London W1M 9DL, UK Tel: +44 (0)20 7636 6060 Fax: +44 (0)20 7636 6072 email: paulg@evolutif.co.uk http://www.evolutif.co.uk/ 2002 Systeme Evolutif Ltd Slide 1 Agenda I Why Risk-Based Testing? II Introduction to Risk-Management III Risk and Test Objectives IV Designing the Test Process V Illustrative Example from an RBT Prototype V1 Closing Comments, Q&A Here s the commercial bit: This material is based on: Risk-Based E-Business Testing, Gerrard and Thompson, Artech House, 2002 Visit www.riskbasedtesting.com for more information. 2002 Systeme Evolutif Ltd Slide 2 Page 1
Paul Gerrard Systeme Evolutif are a software testing consultancy specialising in E -Business testing, RAD, test process improvement and the selection and implementation of CAST Tools. Evolutif are founder members of the DSDM (Dynamic Systems Development Method) consortium, which was set up to develop a non-proprietary Rapid Application Development method. DSDM has been taken up across the industry by many forward-looking organisations. Paul is the Technical Director and a principal consultant for Systeme Evolutif. He has conducted consultancy and training assignments in all aspects of Software Testing and Quality Assurance. Previously, he has worked as a developer, designer, project manager and consultant for small and large developments. Paul has engineering degrees from the Universities of Oxford and London, is Co-Programme Chair for the BCS SIG in Software Testing, a member of the BCS Software Component Test Standard Committee and Former Chair of the IS Examination Board (ISEB) Certification Board for a Tester Qualification whose aim is to establish a certification scheme for testing professionals and training organisations. He is a regular speaker at seminars and conferences in Europe and the US, and won the Best Presentation award at EuroSTAR 95. 2002 Systeme Evolutif Ltd Slide 3 Why Risk Based Testing? 2002 Systeme Evolutif Ltd Slide 4 Page 2
V-Model Requirements Functional Specification User Acceptance Test Is there ever a one-to-one relationship between baseline documents and testing? System Test Physical Design Program Specification Unit Test Integration Test Where is the static testing (reviews, inspections, static analysis etc.)? 2002 Systeme Evolutif Ltd Slide 5 Traditional approach Methodology Test stage Test stage Test stage Test stage Consider Schedule, Environments, Timescales etc. Unit Test System Test Acceptance Build and Execute tests Not Done Not focused Not again! Stakeholder Involvement We have to Trust them Too detailed To understand Are these faults really severe? 2002 Systeme Evolutif Ltd Slide 6 Page 3
Problems with tradition Sequence of decisions Stages responsibility capability objectives Guidance to developers and testers None, except generic, text book mantras demonstrate software meets requirements Input of stakeholders Only when system/acceptance tests reveal problems Far too late! Decision making Timescale driven in early stages Crisis driven towards the end Unsatisfactory all round. 2002 Systeme Evolutif Ltd Slide 7 W-Model Write Requirements Test the Requirements Install System Acceptance Test Specify System Test the Specification Build System System Test Design System Test the Design Build Software Integration Test Write Code Unit Test 2002 Systeme Evolutif Ltd Slide 8 Page 4
Risk-based testing Plan assess product risks define test objectives test techniques, products to test Stakeholder Involvement Schedule responsibility estimation process Decide assess product risks risk-based test reporting Implement focused test design and execution 2002 Systeme Evolutif Ltd Slide 9 Risk-based test planning If every test aims to address a risk, tests can be prioritised by risk It s always going to take too long so Some tests are going to be dropped Some risks are going to be taken Proposal: The tester is responsible for making the project aware of the risks being taken Only if these risks are VISIBLE, will management ever reconsider. 2002 Systeme Evolutif Ltd Slide 10 Page 5
How much testing is enough? Enough testing has been planned when the stakeholders (user/customer, project manager, support, developers) approve: TESTS IN SCOPE They address risks of concern and/or give confidence THE TESTS THAT ARE OUT OF SCOPE Risk is low OR these tests would not give confidence The amount and rigour of testing is determined by CONSENSUS. 2002 Systeme Evolutif Ltd Slide 11 Introduction to Risk Management 2002 Systeme Evolutif Ltd Slide 12 Page 6
Some general statements about risk Risks only exist where there is uncertainty If the probability of a risk is zero or 100%, it is not a risk Unless there is the potential for loss, there is no risk ( nothing ventured, nothing gained ) There are risks associated with every project Software development is inherently risky. 2002 Systeme Evolutif Ltd Slide 13 Cardinal Objectives The fundamental objectives of the system to be built Benefits of undertaking the project Payoff(s) that underpin and justify the project Software risks are those that threaten the Cardinal Objectives of a project. 2002 Systeme Evolutif Ltd Slide 14 Page 7
Three types of software risk Project Risk resource constraints, external interfaces, supplier relationships, contract restrictions Primarily a management responsibility Process Risk variances in planning and estimation, shortfalls in staffing, failure to track progress, lack of quality assurance and configuration management Planning and the development process are the main issues here. Testers are mainly concerned with Product Risk Product Risk lack of requirements stability, complexity, design quality, coding quality, non-functional issues, test specifications. Requirements risks are the most significant risks reported in risk assessments. 2002 Systeme Evolutif Ltd Slide 15 Process Risk identification what are the risks to be addressed? Risk analysis nature, probability, consequences, exposure Risk response planning pre-emptive or reactive risk reduction measures Risk resolution and monitoring Stakeholders should be involved at all stages. 2002 Systeme Evolutif Ltd Slide 16 Page 8
Assessing consequences (loss) Severity Description Score Critical business objective cannot be accomplished 5 High business objective undermined 4 Moderate business objectives are affected 3 Low slight effect on business 2 Negligible No noticeable effect 1 2002 Systeme Evolutif Ltd Slide 17 Assessing probability (likelihood) Probability Description Score >80% almost certainly, highly likely 5 61-80% probable, likely, we believe 4 41-60% we doubt, improbable, better than even 3 21-40% unlikely, probably not 2 1-20% highly unlikely, chances are slight 1 2002 Systeme Evolutif Ltd Slide 18 Page 9
Risk exposure Risks with the highest exposure are those of most concern Worst case scenarios drive concerns Risk EXPOSURE is calculated as the product of the PROBABILITY and CONSEQUENCE of the risk A simple notation is L 2 where L 2 = LIKELIHOOD x LOSS. 2002 Systeme Evolutif Ltd Slide 19 What do the numbers mean? Sometimes you can use numeric assessments We may have experience that tells us» Likelihood is high (it always seems to happen)» Loss is 50,000 (that s what it cost us last time) But often, we are guessing Use of categories help us to compare risks Subjective perceptions (never the same) E.g. Developers may not agree with users on probability! Maybe you can only assign risk RAG numbers RED, AMBER, GREEN The ability to compare is what is most important. 2002 Systeme Evolutif Ltd Slide 20 Page 10
The danger slope Critical High Moderate Low Negligible Highly Unlikely Unlikely Improbable Likely Very Likely Where we want to move all risks 2002 Systeme Evolutif Ltd Slide 21 Risk and Test Objectives 2002 Systeme Evolutif Ltd Slide 22 Page 11
Why use risks to define test objectives? If we focus on risks, we know that bugs relating to the selected mode of failure are bound to be important. If we focus on particular bug types, we will probably be more effective at finding those bugs If testers provide evidence that certain failure modes do not occur in a range of test scenarios, we will become more confident that the system will work in production. 2002 Systeme Evolutif Ltd Slide 23 Defining a test objective from risk We turn around the failure mode or risk Risk: a BAD thing happens and that s a problem for us Test objective: demonstrate using a test that the system works without the BAD thing happening The test: execute important user tasks and verify the BAD things don t happen in a range of scenarios. 2002 Systeme Evolutif Ltd Slide 24 Page 12
Risks and test objectives - examples Risk Test Objective The web site fails to function To demonstrate that the application functions correctly on the user s client correctly on selected combinations of operating system and browser operating systems and browser version configuration. combinations. Bank statement details presented in the client browser do not match records in the back-end legacy banking systems. Vulnerabilities that hackers could exploit exist in the web site networking infrastructure. To demonstrate that statement details presented in the client browser reconcile with back-end legacy systems. To demonstrate through audit, scanning and ethical hacking that there are no security vulnerabilities in the web site networking infrastructure. 2002 Systeme Evolutif Ltd Slide 25 Risk-based test objectives are usually not enough Other test objectives relate to broader issues contractual obligations acceptability of a system to its users demonstrating that all or specified functional or non-functional requirements are met non-negotiable test objectives might relate to mandatory rules imposed by an industry regulatory authority and so on Risk assessment might miss something, or de-scope something important Generic test objectives catch all measure e.g. all requirements coverage complete the definition of your test stages. 2002 Systeme Evolutif Ltd Slide 26 Page 13
Generic test objectives Test Objective Demonstrate component meets requirements Demonstrate component is ready for reuse in larger sub-system Demonstrate integrated components correctly assembled/combined and collaborate Demonstrate system meets functional requirements Typical Test Stage Component Testing Component Testing Integration testing Functional System Testing Demonstrate system meets non-functional requirements Non-Functional System Demonstrate system meets industry regulation requirements Demonstrate supplier meets contractual obligations Validate system meets business or user requirements Demonstrate system, processes and people meet business requirements Testing System or Acceptance Testing (Contract) Acceptance Testing (User) Acceptance Testing (User) Acceptance Testing 2002 Systeme Evolutif Ltd Slide 27 Tests as demonstrations Demonstrate is most often used in test objectives Better than Prove which implies mathematical certainty (which is impossible) But is the word demonstrate too weak? it represents exactly what we will do we provide evidence for others to make a decision we can only run a tiny fraction of tests compared to what is possible so we really are only doing a demonstration of a small, sample number of tests. 2002 Systeme Evolutif Ltd Slide 28 Page 14
But tests should aim to locate faults, shouldn't they? The tester s goal: to locate faults We use boundary tests, extreme values, invalid data, exceptional conditions etc. to expose faults: if we find faults these are fixed and re-tested we are left with tests that were designed to detect faults, some did detect faults, but do so no longer We are left with evidence that the feature works correctly and our test objective is met No conflict between: strategic risk-based test objectives and tactical goal of locating faults. 2002 Systeme Evolutif Ltd Slide 29 Testing and meeting requirements Risk-based test objectives do not change the methods of test design much Functional requirements We use formal or informal test design techniques as normal Non-functional requirements Test objectives are often detailed enough to derive specific tests. 2002 Systeme Evolutif Ltd Slide 30 Page 15
Designing the Test Process 2002 Systeme Evolutif Ltd Slide 31 Master Test Planning process Tester Activity Risk Identification Consult business, technical staff Prepare a draft register of risks Workshop Risk Analysis Discuss risks Assign probability and consequence scores Calculate exposure Tester Activity Review and Decision Risk Response Test Scoping Formulate test objectives, select test technique Document dependencies, requirements, costs, timescales for testing Assign Test Effectiveness score Nominate responsibilities Agree scope of risks to be addressed by testing Agree responsibilities and budgets Tester Activity Test Process Definition Draft the test process from the Test Process Worksheet Complete test stage definitions 2002 Systeme Evolutif Ltd Slide 32 Page 16
Test process worksheet Failure Mode or Objective Probability Consequence Test Effectiveness RISK Number Prototyping Infrastructure Sub- System Application System Non- Functional Tests User Acceptance Operational Acceptance (BTS) Live Confidence Customer Live Trial Test Technique Client Platform 1 Which browsers, versions and O/S platforms will be supported, includes non-fr ames, non - graphic browsers etc.)? 2 New platforms: Web TV, Mobile Phones, Palm Pilots etc. 3 Connection through commercial services e.g. MSN, Compuserve, AOL 4 Browser HTML Syntax Checking SS SS SS 5 Browser compatibility HTML Checking SS 6 Client configuration e.g. unusable, local character sets being re jected by database etc. 7 Client configuration: Client turns off graphics, rejects cookies, Cookies time out, Client doesn t have required plug-ins etc. 8 Minimum suppo rted client platform to be determined/validated SS SS SS Component Functionality 9 Client component functionality SS 10 Client web-page object loading SS 11 Custom-built infrastructure component functionality 12 COTS component functionality SS SS 13 HTML page content checking - spelling, HTML val idation System/Application functionality SS 14 End -to-end system functionality CC 15 Loss of context/persistence between transactions SS CC 2002 Systeme Evolutif Ltd Slide 33 Using the worksheet - risks Failure Mode or Objective column failures/risks requirements for demonstrations mandatory/regulatory/imposed requirements Probability of the problem occurring Consequence of failure Test Effectiveness - if we test, how likely would the problem be detected? 2002 Systeme Evolutif Ltd Slide 34 Page 17
Creating the worksheet Create a template sheet with initial risks and objectives based on experience/checklists Cross-functional brainstorming stakeholders or technically qualified nominees might take all day, but worth completing in one session to retain momentum If you can t get a meeting, use the specs, then get individuals to review. 2002 Systeme Evolutif Ltd Slide 35 Illustrative Example from an RBT Prototype 2002 Systeme Evolutif Ltd Slide 36 Page 18
Residual Risks Test products through the lifecycle initial risk assessment test objectives master test planning test stages start today Planne d end test process definition test specification test plan/ procedures Progress through the test plan release risk assessment test results analysis test log test execution 2002 Systeme Evolutif Ltd Slide 37 Risk detail This is the risk detail it is one row of the test process worksheet An example screen from the RBT prototype application. 2002 Systeme Evolutif Ltd Slide 38 Page 19
From risks to test objectives initial risk assessment test objectives 2002 Systeme Evolutif Ltd Slide 39 2002 Systeme Evolutif Ltd Slide 40 Page 20
Test stage - key attributes Test Objectives Component(s) under Test Baseline Responsibility Environment Entry Criteria Exit Criteria Techniques/tools Deliverables The objectives of this stage of testing based on the risks to be addressed and the generic objectives for the test stage. The architectural components, documents, business processes to be subjected to the test. Document(s) defining the requirements to be met for the components under test (to predict expected results). Groups responsible for e.g. preparing tests, executing tests and performing analysis of test results. Environment in which the test(s) will be performed. Criteria that must be met before test execution may start. Criteria to be met for the test stage to end. Special techniques, methods to be adopted; test harnesses, drivers or automated test tools to be used. Inventory of deliverables from the test stage. 2002 Systeme Evolutif Ltd Slide 41 Test stage definition initial risk assessment test objectives test stages 2002 Systeme Evolutif Ltd Slide 42 Page 21
Test stage definition 2 2002 Systeme Evolutif Ltd Slide 43 Test stage definition 3 2002 Systeme Evolutif Ltd Slide 44 Page 22
Test stage definition 4 Test stage definition includes the required items to create an IEEE 829 format test plan. 2002 Systeme Evolutif Ltd Slide 45 Master Test Planning initial risk assessment test objectives master test planning test stages test process definition 2002 Systeme Evolutif Ltd Slide 46 Page 23
Master Test Plan glues it all together 2002 Systeme Evolutif Ltd Slide 47 Example test stage Test stage definition example stage objectives technique organisation object under test environment etc. etc. Project: Sample IT Project Test Stage: LSI Large Scale Integration Description Features To Be Tested The System -Tested application will be tested in conjunction with associated systems that it must integrate with. The Sample IT Application will be tested in conjunction with the Banking Interface and the Electronic Mail Interface. The following features are in scope: - payment processing - payments exceptions - electronic mail notifications (all types). Features Not To Be Tested Integration with legacy Financial Accounting system FTBS is out of scope for this project. Item Pass Fail Criteria: Suspend/Resume Criteria Test Deliverables: Object Under test Test Objectives: Technique: Risk: R07 Organisation: Environment: Name: Estimate: Dependencies: Timescale: Messages are triggered as predicted. Data transfer performed in a synchonised way. Data reconciles across integrated systems. Testing will be suspended if it is not possible to perform the basic data transfers: - payments - exceptions - electronic mail notifications. LSI Test Plan LSI Test Procedures LSI End of Phase Report Sample IT A pplication Banking Interface Electronic Mail interface. System Integration Test Demonstrate system and banking systems integrate. Joint Customer and Supplier Activity ACCTEST Evolutif Test 120 days Assume functional system testing is complete to the degree that the key billing functions operate with stubbed -out interfaces. 40 days 2002 Systeme Evolutif Ltd Slide 48 Page 24
Residual Risks Ongoing risk assessment initial risk assessment test objectives master test planning test stages start today Planne d end test process definition test specification test plan/ procedures Progress through the test plan release risk assessment test results analysis test log test execution 2002 Systeme Evolutif Ltd Slide 49 Risk-based test reporting start today Planned end Residual Risks all risks open at the start residual risks of releasing TODAY Progress through the test plan 2002 Systeme Evolutif Ltd Slide 50 Page 25
Benefits of risk-based test reporting Risk of release is known: On the day you start and throughout the test phase On the day before testing is squeezed Progress through the test plan brings positive results risks are checked off, benefits available Pressure: to eliminate risks and for testers to provide evidence that risks are gone We assume the system does not work until we have evidence guilty until proven innocent Reporting is in the language that management and stakeholders understand. 2002 Systeme Evolutif Ltd Slide 51 Benefit & objectives based test reporting Benefit Benefit Benefit Benefit Benefit Benefit Objective Objective Objective Objective Risks Objective Open Closed Open Closed Closed Open Closed Open 2002 Systeme Evolutif Ltd Slide 52 Benefits available for release Page 26
Benefits of benefit-based test reporting Risk(s) that block every benefit are known: On the day you start and throughout the test phase Before testing is squeezed Progress through the test plan brings positive results benefits are delivered Pressure: to eliminate risks and for testers to provide evidence that benefits are delivered We assume that the system has no benefits to deliver until we have evidence Reporting is in the language that management and stakeholders understand. 2002 Systeme Evolutif Ltd Slide 53 How good is our testing? Our testing is good if it provides: Evidence of the benefits delivered Evidence of the CURRENT risk of release At an acceptable cost In an acceptable timeframe Good testing is: Knowing the status of benefits with confidence Knowing the risk of release with confidence. 2002 Systeme Evolutif Ltd Slide 54 Page 27
Closing Comments 2002 Systeme Evolutif Ltd Slide 55 Risk-based test approach: planning RBT approach helps stakeholders: They get more involved and buy-in The have better visibility of the test process RBT approach helps testers Approval to test against risks in scope Approval to not test against risks out of scope Clearer test objectives upon which to design tests RBT approach helps developers Specifies their responsibility for testing in detail No hiding place. 2002 Systeme Evolutif Ltd Slide 56 Page 28
Risk-based test approach: execution and reporting RBT approach helps stakeholders: They have better visibility of the benefits available and the risks that block benefits RBT approach helps management: To see progress in terms of risks addressed and benefits that are available for delivery To manage the risks that block acceptance To better make the release decision. 2002 Systeme Evolutif Ltd Slide 57 Risk-Based Testing Close Any Questions? document templates can be found at http://www.riskbasedtesting.com/ 2002 Systeme Evolutif Ltd Slide 58 Page 29