Software Testing. Testing types. Software Testing

Transcription

1 1 Software Testing Testing types Software Testing

2 2 References Software Testing IBM testing course available through the Academic Initiative: Principles of Software Testing for Testers

3 3 Software Testing Course objectives Present different testing terms distinctions Differentiate between black box testing and white box testing, structural and behavioral testing, etc Test cases and test plans High level description of the most important testing techniques Relate testing techniques to product methodology

4 4 Software Testing History of testing 1950: Fix bugs: what programmers did to find bugs in their programs 1960: Exhaustive testing : Prove it works (acceptance testing) : Prove it does not work (defect detection) : Defect prevention, Test automation 1990: Early test design (V Model) 2000: Agile testing (exploratory, risk-based)

5 5 Software Testing Traditional Distinctions Black box versus glass box (white box) versus gray box Behavioral versus structural Functional versus parafunctional (nonfunctional) Unit versus integration versus system versus acceptance Independent versus internal Manual versus automated Static versus dynamic Alpha versus beta

6 6 Software Testing Black box testing Black Box Testing is based on externally observed characteristics of the program we are testing. The tests are designed without regard to how the program works internally; Tests are performed on developed prototypes and versions; Sample techniques: decision tables, equivalence partitioning, range testing, boundary value testing, database integrity testing, exception testing, random testing, etc; Tests are driven by product requirements and specifications: mostly used on system testing and acceptance testing; Such tests may not reveal side effects of the user actions like unwanted mail messages being sent, database updates, etc.

7 7 Software Testing White box testing Glass Box Testing (a.k.a. White Box) draws on knowledge of the internal characteristics of the program; The tests are designed to exercise specific parts of the code using that knowledge on paths of logic and internal data; The requirements and specifications are not considered; Sample techniques: basis path analysis, statement coverage, branch/condition coverage, etc; Used for unit testing and integration testing; It doesn t detect missing paths and data sensitive errors; It is extremely hard to check all the possible logic paths.

8 8 Software Testing Gray box testing Combination of black box and white box testing; The tester communicates with the developer on the way the specifications were implemented; Helpful when specifications are ambiguous; May reduce the number of the required tests;

9 9 Software Testing Other distinctions Structural testing is similar with the white box testing; Behavioral testing is similar with the black box testing; Functional testing is a subset of the black box testing focusing on the inputs given to the program and the received answers; Non-functional testing focuses on non-functional specifications like performance, reliability, availability, etc Alpha testing refers to the evaluation of the alpha milestone (software is buggy but usable) Beta testing refers to the evaluation of the beta milestone (software is almost complete)

10 10 Software Testing Acceptance testing Testing based on customer requirements: validates the product; Performed at the end of the product development life cycle; Covers only the essential tests of the program; Often performed by an external organization than the one building the product or by some end users; A black box testing type; May involve non-functional testing; Performed regardless of the product lifecycle.

11 11 Software Testing System testing Testing based on product specifications; Does not validate requirements; A black box type of testing; Covers functional and non-functional aspects; Performed on an already completed product; Some methodologies do not include it (XP);

12 12 Software Testing Integration testing Testing of few parts of code together; Discovers interface errors and protocol errors; Integration testing starts with few modules and eventually cover all the application; Combined with module testing provide for a divide and impera approach on evaluating the software; often called incremental testing; Often performed by developers; Bottom-up approach.

13 13 Software Testing Unit testing Testing of individual modules of code before being integrated into the product; Performed by developers; Involves writing other code calling and making assertions on the tested code; Very good support for regression testing; XP methodology combines it with the acceptance testing to cover all the testing effort;

14 Typical Testing Life Cycle Requirements Analysis Test Planning Test Case Development Environment Setup Test Execution Test Cycle Closure

15 Requirements Analysis Study of the requirements in order to identify the testable requirements. Activities Identify types of tests to be performed. Gather details about testing priorities and focus. Prepare Requirement Traceability Matrix (RTM). Identify test environment details where testing is supposed to be carried out. Automation feasibility analysis (if required). Deliverables Requirements Traceability Matrix Automation feasibility report. (if applicable)

16 Test Planning The QA manager will determine effort and cost estimates for the testing and would prepare and finalize the Test Plan. Activities Preparation of test plan/strategy document for various types of testing Test tool selection Test effort estimation Resource planning and determining roles and responsibilities. Training requirement Deliverables Test plan /strategy document. Effort estimation document.

17 Test plan IEEE STD : A document describing the scope, approach, resources and schedule of intended testing activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning.

18 Test plan Objectives (Kaner chapter 12) Facilitates the technical task of testing Improve testing coverage Produces useful artifacts: lists, tables, charts, etc Provide focus Facilitates communication about testing tasks and process Communicate testing strategies Communicate the size of the testing job Divide the work Provides structure for organizing, scheduling and managing the testing project Reach agreement on testing tasks Allocate testing tasks Measure project status Improve individual accountability/responsibility

19 Test Plan Structure (IEEE 829) Test Plan Identifier References Introduction Test Items Software Risk Issues Features to be Tested Features not to be Tested Approach Item Pass/Fail Criteria Suspension Criteria and Resumption Requirements Test Deliverables Remaining Test Tasks Environmental Needs Staffing and Training Needs Responsibilities Schedule Planning Risks and Contingencies Approvals Glossary

20 Test Case Development Creation, verification and rework of test cases & test scripts. Test data is identified/created and is reviewed and then reworked Activities Create test cases, automation scripts (if applicable) Review and baseline test cases and scripts Create test data (If Test Environment is available) Deliverables Test cases/scripts Test data

21 Test Case Definition: a set of conditions or variables under which a tester will determine whether a system under test satisfies requirements or works correctly Elements of the test case Test Id Summary: objective for the test case Related requirement: the Id of the requirement this test case traces to Prerequisites: preconditions that must be fulfilled prior to executing the test Dependencies: test cases this test depends on Procedure: step-by-step procedure to execute the test Test data: the test data, or links to the test data, that are to be used while conducting the test Expected result: the expected behavior Observations Created by: the test creator Date of creation

22 Sample Test Case Test Case ID: TC001 Test Case Summary: To verify that clicking the Generate Coin button generates coins. Related Requirement: RS001 Prerequisites User is authorized. Coin balance is available. Test Procedure Select the coin denomination in the Denomination field. Enter the number of coins in the Quantity field. Click Generate Coin. Test Data Denominations: 0.05, 0.10, 0.25, 0.50, 1, 2, 5 Quantities: 0, 1, 5, 10, 20 Expected Result Coin of the specified denomination should be produced if the specified Quantity is valid (1, 5) A message Please enter a valid quantity between 1 and 10 should be displayed if the specified quantity is invalid. Remarks: This is a sample test case. Created By: John Doe Date of Creation: 01/14/2020

23 Test Case Best Practice Consider positive and negative testing Simple and transparent: author!= tester Create tests cases with reasonable probability to catch an error Avoid test case repetition Include only the needed steps Trace to requirements Reusability Ensure 100% coverage Peer review test cases

24 Test Cases Suite Collection of test cases used to test some aspect of the product under test. Also help prioritization Elements: Test suite id Product name Product version Objective Environment List of referenced test cases Observations

25 Testing Data Input given to the software product Positive testing data Negative testing data Related with the test case: created before the execution Sources: Direct examination of code Customer Production environments Automated generation

26 Test Environment Setup Decides the software and hardware conditions under which a work product is tested. Activities Understand the required architecture, environment set-up and prepare hardware and software requirement list for the Test Environment. Setup test Environment and test data Perform smoke test on the build Deliverables Environment ready with test data set up Smoke Test Results.

27 Test Execution Execution of the test plans and the prepared test cases. Bugs will be reported back to the development team for correction and retesting will be performed. Activities Execute tests as per plan Document test results, and log defects for failed cases Map defects to test cases in RTM Retest the defect fixes Track the defects to closure Deliverables Completed RTM with execution status Test cases updated with results Defect reports

28 Executed test case Test Case Id: the id of the test Actual result: the obtained behavior after running the test Status: pass or fail Executed by Date of execution Test environment Observation

29 Sample Executed Test Case Test Id: TC001 Actual Result If the specified quantity is valid, the result is as expected. If the specified quantity is invalid, nothing happens; the expected message is not displayed Status: Fail Executed By: Jane Roe Date of Execution: 02/16/2020 Test Environment OS: Windows Y Browser: Chrome N Observation: More denominations are likely to be considered for addition

30 Test Cycle Closure Discussing and analyzing testing artifacts to identify strategies that have to be implemented in future, taking lessons from the current test cycle. Activities Evaluate cycle completion criteria based on Time, Test coverage, Cost, Software, Critical Business Objectives, Quality Prepare test metrics based on the above parameters. Document the learning out of the project Prepare Test closure report Qualitative and quantitative reporting of quality of the work product to the customer. Test result analysis to find out the defect distribution by type and severity. Deliverables Test Closure report Test metrics