From Assurance to Confirmation A Quantitative Test Coverage Solution Using Open Source Tools Software Testing Conference, 2013

Transcription

1 From Assurance to Confirmation A Quantitative Test Coverage Solution Using Open Source Tools Software Testing Conference, 2013 Authors: Himanshu Dhingra - Overall 9+ years IT extensive experience years on Automation testing - Expertise on Automation Testing tools QTP, SilkTest, FitNesse, Selenium - Name & Address of Organization: Deloitte Consulting, Salarpuria Supreme, Near Innovative Multiplex, Marathalli Tushar Roy - Overall 7+ years IT extensive experience years on Automation testing - Expertise on Automation Testing tools QTP, TestPartner, Selenium - Name & Address of Organization: Deloitte Consulting, Salarpuria Supreme, Near Innovative Multiplex, Marathalli

2 Abstract Testing as a competency faces a lot of challenges of its own and one of the major challenges is giving the confirmation of quality of the end-product. With growing client sdemand for quality confirmation, it has become almost critical for Testing Competency to start moving from the stereotype Assurance mindset towards Confirmation of quality. Confirmation of quality in a way can be achieved when we say all the branches of development code have been covered while testing activity and there are 0 defects. This would ensure that 100% of the development code has been touched during test execution and with zero defects would mean the product is meeting the desired quality. In other words, we have arrived at a quantitative test coverage which can be measured to arrive at conformity. Our proposed solution is based on addressing this question on how to fill up the gap between Assurance to Confirmation of the quality of product. The solution not only establishes the quantitative test coverage and gives the confirmation of product quality but also covers the complete Test Cycle from an end to end Test Traceability where there would be concrete linkage between Requirement-Test Case-Defects-Code Coverage%- Requirement With the usage of only open-source tools, we could also save lot of client s money, gain the trust and value from the client. From Assurance to Confirmation A Quantitative Test Coverage Solution Using Open Source Tools With growing client s business and their needs for cutting-edge solutions, the demand for top-most quality of product is also in demand these days. Clients no longer ask for mere assurance of the quality of their product. Instead, they started asking for the confirmation of the quality of the product. To fill the gap between Assurance and Confirmation of quality, the traditional and stereotyped testing approach is not sufficient. With mere testing, we can only concludethat zero defectsare present in the product but there is no guarantee if all the development code branches have been covered during the testing activity. (As depicted in Figure 1 below)

3 Figure 1: Traditional Testing Approach To track the coverage of all branches of development code, we need to have a mechanism of Code Coverage. But just code coverage will not tell us if the product in free from defects or not.(as depicted in Figure 2 below) Figure 2: Code Coverage Mechanism But if we integrate the traditional testing activity with the code coverage mechanism, we can arrive to a point where we would have the snapshot of defects present in the product and the coverage % of the development code branches that were covered during the testing activity. Thus, Confirmation of quality in a way can be achieved when we say all the branches of development code have been covered while testing activity and there are 0 defects. This would ensure that 100% of the development code branches have been touched during test execution and with zero defects would mean the product is meeting the desired quality. In other words, we have arrived at a quantitative test coverage which can be measured to arrive at conformity.

4 Figure 3: Integration of Testing Activity with Code Coverage Mechanism With the metrics of Quantitative Test Coverage as an outcome of the integration of Testing activity and Code Coverage mechanism, we can fill the gap between Assurance to Confirmation of product quality and meet client s demand. Further integration of above 2 processes viz., the Testing activity and Code Coverage Mechanism with business requirements and various test artifacts such as Test Cases, Defect Status Reports, Defects metrics we can establish an end-to-end traceability. This could provide One-Stop centralized location to find all the key reports. By implementing end-to-endtraceability workbench effectively, one can get an overall visibility of the software implementation process, gauge the various parameters which has potential risk from project delivery perspective. It ensures that all requirements have been tested. Once the test cases are linked to the defects, it helps to find out how many requirements have failed test cases linked to them, the number of defects for each requirement, the severity and priority of defects. Further integration of Test Case Execution with Code Coverage mechanism will result in the capture of code coverage %. As a result, it helps to see the status of each requirement at any point of time in the Software Life Cycle. A number of testing tools are available in market to establish traceability. Rising software costs motivates to find open source solutions which are industry based and offer all the required functionality. Integration Flow of Open-Source Tools for Test Coverage and E2E Test Traceability Our proposed solution is based on the usage of open-source tools. Below are the details and description of the tools we have used to establish the Test Coverage.

5 Figure 4: Snapshot of Integration Approach TestLink: Requirement Tracking and Test Management tool. TestLink enables to input requirements and create a requirement specification. It further enables you to create test cases from Requirements and hence establish traceability between Requirements and test cases. The greatest advantage of establishing link between requirements and test cases in TestLink is to check the progress of testing in a particular release. JIRA: Defect Management tool. JIRA enables you to log and track defects. TestLink can be integrated with JIRA and hence you can link each failing test case with a defect in JIRA. TestLink enables you to access JIRA from TestLink and log a defect and create a link to the defect in TestLink. This basically helps in creating the end to end traceability from Requirements to Test Cases to Defects. At any point of time in a release, we can check the progress with respect to requirements and find out which requirements have failed test cases associated with it and how many numbers of defects are linked to each requirement. Selenium: Automation Testing tool. Selenium enables to create automated test cases for web based application. It allows executing test cases on multiple browsers. Selenium can be integrated with TestLink and the result of execution of Selenium test cases can be stored in TestLink. Hence, TestLink becomes a single repository to store requirements, test cases and result of execution of manual and automation test cases. EMMA: EMMA is an open-source toolkit for measuring and reporting Java code coverage. It supports the coverage types at the following level: class, method, line, basic block, package level. It is capable of detecting when a single source code line is covered only partially. The following diagram shows the overall integration flow of various open-source tools that can be used to establish the Test Coverage and end-to-end traceability.

6 Recommendations and Conclusion With the integration of Traditional Testing activity with Code Coverage Mechanism, we can have better a better picture of every requirement s defect status and the effectiveness of test coverage measured during the test execution. With the achievement of zero defect count and 100% coverage of development code branches through the test execution phase, we can give the confirmation on the quality of th/e product. With the adoption of all open-source tools, we can help the client to save lots of money which might have been otherwise invested in procuring expensive licensed tools. With the metrics of Quantitative Test Coverage as an outcome of the integration of Testing activity and Code Coverage mechanism, we can fill the gap between Assurance to Confirmation of product quality and meet client s demand. The outcome of this end-to-end linkage of Requirement-Test Case-Defects-Code Coverage%- Requirement is a one-stop dashboard.

7 References: 1. EMMA Code Coverage: 2. Selenium: