RBT Framework Coupled with Keyword-driven Test Design Approach

Transcription

1 RBT Framework Coupled with Keyword-driven Test Design Approach Authors : Arindam Bhattacharyya Test Manager Zeba Khan Technical Consultant Dr. Anil Kumar Thacker Manager Technology Solution

2 ABSTRACT Testing is potentially a risk-based activity. The fact is, one can t test everything equally, and even if one could, it would be too expensive. The Risk Based Testing (RBT) approach helps in identification of risk factors associated with the testing. Various RBT models and approaches available today talk about assessing and calculating risks associated with the test scenarios for a system under test. These studies have described risk calculation for test cases by considering various factors from both business and technical perspectives. Once identified, the risks are prioritized based on their impact and likelihood of occurrence and relevant test cases are designed and executed accordingly. This paper presents an approach for risk analysis, prioritization, and combines keyword driven test scenario design using an extended RBT approach. This approach enables risk monitoring and continuous adjustment of testing effort and resources. The paper also briefly reviews various RBT approaches, discusses associated challenges, and describes the benefits of the extended RBT approach, and possible future enhancements. KEYWORDS Risk based testing, Keyword driven approach, Testing framework, Test scenario design INTRODUCTION Testing is a time and effort intensive activity. Studies indicate that about 40 percent of the effort is spent during pre-testing phase, 40 percent in testing phase, and about 20 percent of the effort is consumed in post-testing phase. In many organizations, software testing accounts for 30 to 50 percent of software development costs. Yet most people believe that software is not well tested before it is delivered 1 According to Boehm about 80 percent of the defects come from 20 percent of the modules, and about half the modules are defect free. 2 Therefore, in order for a project to be successful, testers must decide which parts of the system need to be thoroughly tested, and this requires careful analysis of the risks involved. 1 2 Heumann Jim, Generating Test Cases from Use Cases, 2001, _cases_jh.html Boehm B.W., and V.R. Basili (2001, January). Software Defect Reduction Top 10 List. Computer, 34, Software testing is a quality control and engineering activity to ensure the correctness, completeness and reliability of any program or software application. The main purpose of testing is to evaluate the attributes and capabilities of a program, or a software application or a system and determine that it meets expected results, and user requirements. 3 However, often testing is carried out late in the development lifecycle, and under constrained time and budget. As a result, everything planned cannot be tested. In this context, it becomes important to determine the amount of testing to be performed based on the risks associated with the application components. This requires careful analysis of the risks involved. These risks can arise from the newness and reliability of the technology being used, the nature of the application, or from the priority of the business functions under test. The amount of testing that should be performed is directly related to the amount of risk involved. Understanding the risk-based nature of testing is also the key to dealing with chronic problem of inadequate test resources. Risk must be used as the basis for allocating the test time that is available, and for helping to make the selection of what to test and how to allocate resources. 4 Many researchers are investigating Risk Based Testing approach and techniques. Their research spans a wide variety of topics. There are several ways to carryout Risk Based Testing. Some of the prominent RBT approaches or models are discussed below. RBT consists of a set of activities regarding risk factors identification related to software requirements and testing. Once identified, the risks are prioritized according to their likelihood and impact. The test cases are designed based on the strategies for treatment of the identified risk factors. The testing efforts are continuously adjusted according to risk monitoring. 5 Software testing consultant Stale Amland in a case study on risk based testing described: The probability of fault for each function P(f) were estimated. This depended on whether the code was changed, if there was new functionality, the design Hetzel, William C., The Complete Guide to Software Testing, 2nd ed. Publication info: Wellesley, Mass.: QED Information Sciences, ISBN: Physical description: ix, 280 p. Guide CSTE Common Body Of Knowledge, V6.1 Ellen Souza, Cristine Gusmao, Julio Venancio, "Risk-Based Testing: A Case Study," pp , 2010 Seventh International Conference on Information Technology, 2010

3 quality, the size and the complexity. These factors were given weights from one to five, where the higher are the most important indicator of a function with poor quality. Then, for each function of the system the probability for all indicators is then considered given values from one to three. The weights and values for each function are multiplied. The probability is calculated as the Weighted Average of a weighted function divided by the highest Weighted Average of all functions. This gave a probability numbers between zero and one. The risk exposure for each function RE(f) was calculated by multiplying the cost of fault and the probability using the formula: RE(f) = P(f) * ( C(c) + C(v) ) / 2 The testing resources where focused on the functions with the highest risk exposures. 6 Once the Risk Exposure is computed, testing risks are prioritized based on RE value. Then steps to control the risk are planned. Software testing guru James Bach in his seminal work Heuristic Risk Based Testing, stresses on human experience to find the most risky areas. Bach presents two approaches to heuristic Risk-Based testing, inside-out and outside-in. In inside-out the tester studies the product together with the developer and asks questions such as What can go wrong here? For each part of the product the tester will look for vulnerabilities, threats and victims. Outside-in begins with a set of potential risks and matches them to the details of the situation. A list of predefined risks is consulted and used to determine whether they apply here and now. Finally, Bach prescribes three ways to organize testing risk, viz., risk watch list (lists of quality criteria categories), risk task list (generic risk list), and component risk metrics (risk catalogues). He concluded in his study report: Management may assume that you did a sloppy job, and they may not be impressed with the standard argument that all testing is incomplete. That s when it s nice to have that risk list or matrix. With risk-based testing, you can show management that you strive to make the best use of the resources they invest. They ll respect you for that. 7 Stephen Besson, Research and Development Director of Karat Software (and now Freeborders) describes how to carry out Risk-Based testing by prioritizing the most vital functions. The method starts by identifying vital functions that may have faults of high severity. The severity is defined by measuring the negative impact a fault has on the business, high, medium or low. If the user cannot work with the software and wastes time and money because of the failure, the severity is considered to be high. Besson says, However, if you see testing as an investment, the strategy becomes: What is the minimum testing effort that I should invest in order to maximize risk reduction? 8 Chen a computer science graduate from School of Information Technology and Engineering, Canada and software engineering scholar at IBM has presented a specification-based (black box) regression test selection technique. This technique is customeroriented and also risk-based. It provides methods to obtain both Targeted Tests and Safety Tests. The case study indicates that the technique is effective in finding defects. Regression testing is terminated whenever we run out of time and resources. The final selected regression test suite is the union of the Targeted Tests and Safety Test 9 Other scholarly studies by Garrard talks about the RBT for testing of e-business apps and Schaefer has given approach for prioritizing tests against deadlines. The studies reviewed above uses risk priority to identify critical tests. Most of them use similar methods as described by Amland, where the factors that can increase the cost and consequences of failure for some parts of the product are found. Cost in respect of maintenance, legal issues, and reputation for the vendor and loss of customer is considered. Schaefer and Besson look at what functions have the highest importance and cause bigger inconveniences of failure for the user. Schaefer used defect generators to predict the most defective parts of the system and applied risk based testing. Besson verified the degree of failure that could prevent the user from using the system and frequencies of use to prioritize what to test. Chen used similar ideas as Amland and focused on test cases and not functions of the applications. The risk analysis helps decide what functions to put extra effort for testing, but could also be used to decide what functions not to test. This paper gives an overview of the different Risk Based Testing approaches currently present. 6 7 Stale Amland, Risk Based Testing and Metrics, November 1999, James Bach, Heuristic Risk-Based Testing, Software Testing and Quality Engineering Magazine, 11/99, Besson Stephen, A Strategy for Risk-Based Testing, StickyMinds.com, 2013 Chen Y, Probert R L, Sims D P, Specificationbased Regression Test Selection with Risk Analysis, Tech Journal IBM Canada, 2002.

4 CURRENT RBT APPROACH Most of the RBT approaches available today emphasize on assessing and calculating the risks associated with the test scenarios for a SUT (System under test). Various factors are considered from both business and technical aspects to prioritize test scenarios. These approaches have described Risk factor calculation for test cases in scope. This paper presents a unified approach of how RBT can be implemented in collaboration with business, technical and quality personnel in an organization. It also shows the value-ads in terms of earned confidence of the SUT at each milestone of the execution cycle. RBT in conjunction with Keyword- Driven Test Scenario design approach helps in improvement of review efficiency and reusability. This approach can be adopted by an organization irrespective of domain and technology. The following sections in this paper explain the approach with an example for an Airlines Reservation System. KEYWORD-DRIVEN Coupled with RISK BASED TESTING APPROACH This paper presents a concept of an extended approach to implementing a risk-based testing approach around following areas; a) Test Scenario Design using Keyword driven approach b) Risk Analysis c) Risk-based test prioritization d) Resource management e) System Confidence level at each interval What is a Risk? The possibility of an unfavorable event occurring is defined as a Risk. It can be defined in terms of two components: the probability that this situation may occur and its impact. Why Risk Based Testing? Risk based testing approach assesses all the test scenarios in scope for the risk associated with these scenarios not being tested. High priority scenarios can be tested earlier for an increased confidence in the system leading to possible early go-live targets. Risk Based Testing Assumptions Not all the tests have the same importance o Some tests significantly reduce the risks associated with going live o Other tests provide little value Confidence in the system is driven by executing the important tests and removing defects, not by running thousands of low value tests successfully Projects can go live quicker and at a lower cost if focus (i.e. time, priority, resources) is spent on highest risk areas, and not wasted on areas where failures are low impact and least probable. a) Keyword Driven Test Scenario Design Approach This approach provides a simple and effective way to design test cases for a given testing project based on the key attributes defining the scope of the project. All these attributes are then listed down along with their possible values. A combination of these values would define a high-level test scenario. An example of this approach is as stated below: Online Airline Reservation Application Key-word options Benefits of using Keyword Driven approach for test scenario design: Figure 2 Figure 1 Less time consuming and easy to understand approach Easy to review Provides a good coverage of all the possible test scenarios Easy to maintain Once the High Level Test Scenario design is complete, we would move to our next step of conducting a workshop with the Business Analysts/Business Architects and Technical Architects. b) Workshop on Risk Based Testing with BAs and TAs

5 This workshop should be conducted with the Business Analysts and the Technical Architects to discuss the overall Risk Based Testing Approach and their role in achieving accurate risk evaluation on the High-level Test Scenarios. Business Analysts should define the Impact of Failure for the high-level test scenarios under evaluation. Following factors are to be considered for defining the Impact: Halt - when there is no workaround for the business process in case of any failure. Visible when the failure of a particular business would be very visible. However, there can be a workaround for the same. Minor when the failure of a business case does not directly impact the customer. Technical Architects should define the Likelihood of Failure for the high-level test scenarios under evaluation. Following factors are to be considered for defining the Likelihood: Maturity of Technology, Integration of Technology, Complexity of Solution, Areas of heavy load, Complexity of code, High number of Interfaces. Once this evaluation is complete, the output should be fed into the next step, Build Matrix to analyze the Risk based on BA/TA evaluation. c) Risk Analysis Risk Analysis is at the heart of risk-based testing. This is performed using a 3x3 grid to distinguish high value tests (top right) from low value tests (bottom left). Each area of functionality or non-functionality is assessed in terms of likelihood of failure (If not tested) and impact of failure (as perceived by the end customer). The Analysis is to be performed on highlevel scenarios. i. Step (b) should be repeated with BAs and TAs ii. Once revised evaluation is received, Step (c) should be repeated. iii. Project Scope/Requirements should be assessed for a need to be tested. d) Prioritize Test Scenarios Scope Identification Based on the Analysis done in the previous step, the High-level Test Scenarios are then prioritized considering the factors defined in the grid explained above (Figure 3). Following grid (Figure 4) extends the analysis by prioritizing the high-level scenarios. Prioritization will depend on the project. For example, when a project uses commonly used code/components, Highly Probable-Visible scenarios may have higher priority over Probable-Halt scenarios. In general, top-right is more important than bottom-left where 1 being the highest on priority and 5 being the lowest. The most important tests should be tested thoroughly, perhaps with additional ad hoc tests. Priority 3s may have normal testing, while priority 4 tests could be simplified to touch or identification tests. Priority 5 tests may be de-scoped. This MUST be agreed between the Project and Business, who must accept the trade-off between an earlier go-live versus 100% testing. Figure 4 Figure 3 All the High-level Scenarios are categorized in each of these cells (in Figure 3) based on the evaluation from previous step. After Categorization, the number of scenarios in each of the cells is to be assessed. Most of the Scenarios should fall in top right half of the grid. In case an inverted trend is observed, following steps should be followed: e) Build Test Schedule After a thorough analysis on the risk associated with all the scenarios and their prioritization, a test schedule is prepared planning for the high-priority scenarios during the initial phases of testing and then moving on the low-priority test cases providing an efficient coverage of test scope. f) Resource Planning The Resource allocation should be aligned with the Test Scenario Prioritization Grid (Figure 5). The most experienced resources should be aligned with the high priority test scenarios, with new/less experienced resources being their backup. However, less experienced resources should be aligned with the lower priority scenarios and more experienced

6 resources being their back up. The following grid provides a pictorial view for Resource Allocation. Placing your best people (whether designers, developers or testers) on the most important tests can improve the chances that 1) designs will be through, complete and unambiguous, 2) code will be accurate and c) tests will be effective. g) Checkpoints Figure 5 Checkpoints should be established to assess the test coverage vs. the earned confidence. The test coverage would give the number of test cases executed, whereas, the earned confidence would give the confidence level in the system based on successful execution of high priority scenarios earlier. The graph (Figure 6) gives some more detail on the earned confidence vs. test completion. As the graph suggests, higher level of confidence can be achieved with lesser test coverage to enable incremental and early delivery of the well-tested system. Future Enhancements Risk-based testing is mistakenly thought of as an execution prioritization technique. In fact, it can provide a range of benefits across all phases of a project. It is never too late to perform a risk analysis, to prioritize work, allocate resources, expedite or report confidence in the system. RBT is SDLC Agnostic The Risk-based testing is consistent with prioritybased methodologies including Agile, Waterfall etc.: The emphasis is on supplying working software The approach encourages the business and technical communities to collaborate and share a collective view of risks, impacts and priorities. Risk based analysis provides a prioritized list of test scenarios (not a Test Plan) that can drive iteration priorities, resource allocation and level of testing conducted. The approach ensures that the most important tests are conducted first. Confidence in the system is gained fast. Figure 6

7 Although the case results are encouraging, implementing this approach may require fine-tuning for production fixes and minor enhancements. CONCLUSION Figure 7 This paper has introduced a new testing risk prediction technique that works at granularity level in combination with the key word driven testing. The metrics collected after implementing Keyword Driven approach reveals about 60% reduction in the review effort, and 48% improvement in reusability. The data collected during execution cycle by implementing RBT shows that 99% confidence was achieved by executing 65% of the test cases. Additionally, this work provides an evaluation of the relative contributions of various RBT approaches. ACKNOWLEDGEMENT Authors would like to acknowledge and extend heartfelt gratitude to all the Business partners, Technical architects and the Testing team for implementing this approach. And lastly to our family and friends who made this possible. About HTC HTC Global Services Inc., is an IT solutions provider, focusing on providing business value to our clients through information Technology (IT) and Business Process Outsourcing (BPO) services and solutions. An Inc 500 Hall of Fame Company and Certified Minority-Owned Business, HTC delivers business solutions to customers through combination of process excellence, quality frameworks, and service delivery innovation.