Software Testing Definition: Testing is a process of executing a program with data, with the sole intention of finding errors in the program. Testing can only reveal the presence of errors and not the absence of errors. A successful test is a one which discovers more, yet-undiscovered, errors An ideal testing process never ends. Definition: Quality assurance is a planned and systematic pattern of all actions necessary to provide adequate confidence that the item conforms to established requirements. -- IEEE 1
Objectives of software testing To find more errors in the software design and implementation To confirm that the product meets (satisfies) the requirements, including non-functional requirements Testing is one type of validation Validation is a process that answers to the question, Are we developing the right product? Testing and Debugging Definitions Testing is a process of finding the presence of errors. Debugging is a process of finding and correcting errors. Debugging requires testing efforts to find the errors first, and then retest the program after correcting the errors. (Thus, testing occurs before and after debugging.) Testing is usually performed manually. Debugging is often performed with the help of tool support. 2
Test Case Definition A test case is a description of input/output relationship of one particular testing activity - a set of inputs and expected output(s). Example: Let y = f(x) be the function to be tested; a test case will indicate for what range of values of (input) x the function will return (output) and for what input values the function will not return an output Sample Test Cases The following are some of the test cases for the factorial function int factorial (int x) Input x Expected output Actual output x = 0 1 x = 1 1 x >= 0 x! x < 0 error 3
Testing principles All tests should be traceable to their source (requirements and/or code). Testing activity should be planned (test cases prepared) well before testing process is exercised. Exhaustive testing is not possible. To be most effective, testing should be conducted by a third party not by the developers. Quality Assurance team Test case design methods black-box testing The software is assumed to be a black box (with unknown code). Test cases are derived from problem descriptions, requirements, design documents, etc. white-box testing Test cases are derived by examining the source code. Both black-box and white-box testing are required for a complete testing process. 4
Black-box and White-box testing Test cases can be derived before the program is written. The objective is to ensure that the program meets (satisfies) the requirements. Knowledge of the application domain is necessary but not about the program or the programming language. Test cases can be derived only after the program is complete. The objective is to ensure that all parts of the logic of the program are correct. Knowledge of the programming language is needed but possibly not the application domain. White-box testing Also called glass-box testing and structural testing path coverage statement coverage Derivation of test cases from the program s structure. What we would like to do... exercise every possible execution path What we are able to do... exercise every possible statement at least once 5
Example computing factorial int factorial (int n) { int i, ret = 0; if (n < 0) return ret; i = n; ret = 1; while (i > 0) { ret = ret * i; i = i - 1; } return ret; } Input n n < 0 Test cases Expected output Error n==0 1 n > 0 n! Flow graph notation Also called program graph Abstractly describes the control flow of the program Consists of nodes and arrows Each node represents a contiguous segment of code (statement(s) or expression(s)). Arrows represent control flow transitions. 6
sequence if if-else case while for repeatuntil Flow graph for the factorial example int i, ret=0; int factorial (int n) { int i, ret = 0; if (n < 0) return ret; i = n; ret = 1; while (i > 0) { ret = ret * i; i = i - 1; } return ret; } ret = 0; (n < 0) return ret; i = n; ret = 1; (i > 0) ret = ret * i; i = i - 1; return ret; 7
For path coverage we must ensure that every node is executed by at least one test case. 1) number nodes 2) check test cases 1 int i, ret=0; Test cases Case statements n < 0 1,2,3,8 n==0 1,2,3,4,5,7 n > 0 1,2,3,4,5,6,7 2 3 4 5 6 ret = 0; (n < 0) return ret; i = n; ret = 1; (i > 0) ret = ret * i; i = i - 1; 8 7 return ret; Another example Remove an integer from an array and shift the array, adding 0 at the end. void squeeze (int k, int[] arr) { boolean flag = false; int index = 0, j; while (index < arr.length &&!flag) { if (arr[index] == k) { for (j = index; j < arr.length; j++) arr[j] = arr[j+1]; arr[n] = 0; flag = true; } else index = index + 1; } } Draw the flow graph and generate the test cases. 8
What about this one? public char reportgrade (int[] exams) { char ch = ; int s,i; s = sum (exams) / exams.length; if (s >= 90) ch = 'A'; else if (s < 90 && s >= 75) ch = 'B'; else if (s < 75 && s >= 60) ch = 'C'; else if (s < 60 && s >= 50) ch = 'D'; else } ch = 'F'; return ch; public int sum (int[] exams) { int n, total = 0; for (n = 0; n < exams.length; n++) total = total + exams[n]; return total; } White-box Testing Issues When is statement coverage impossible? When is statement coverage impractical? Does statement coverage ensure correctness? 9
Black-box testing Black-box testing is a testing approach in which test cases are generated without knowledge of the program (source code). also called functional testing or behavioral testing Test cases can be derived even before coding begins. Example: Automated teller machine (ATM) Consider the functional requirement for a withdraw operation. Test cases include... withdraw an amount less than the balance withdraw an amount equal to the balance withdraw an amount more than the balance withdraw an amount beyond a maximum withdraw a negative amount (robustness testing) 10
Objectives of black-box testing Identify improper implementation of functionalities. Identify missing functionalities. Identify errors in interfaces among the components of the software.... including external software such as a database. Identify performance (and other non-functional) errors. Identify initialization and/or termination errors. Equivalence partitioning A black-box testing method Partition the input domain (the set of potential input values) into mutually disjoint subsets, called equivalence classes. Every element of a subset shares one characteristic with all other elements in the same subset. As a consequence, it is sufficient to select one element from each subset, there by reduce the time for testing. 11
Input domain Software system Equivalence partitioning does not impose any constraints on the output domain. Output domain Equivalence partitioning example Factorial example Three subsets of input (the parameter n ) are negative integers zero positive integers Equivalence classes: negative integers - invalid factorial input zero - base case for recursive definition of factorial positive integers - factorial defined recursively 12
More examples A function that accepts a 2-digit positive integer as input All integers less than 10 Integers between 10 and 99 both inclusive All integers greater than 99 Calculate income tax based on net income Income between $0 through $33000 Income between $33001 through $150000 Income above $150000 Limitations of Equivalence Partitioning User is responsible for ensuring the mutual exclusiveness of the partitions. Too difficult to determine equivalence classes Too many equivalence classes 13
Boundary Value Analysis Boundary value analysis is a technique that complements the equivalence partitioning approach. Boundary value analysis can help to partition the input domain when the input lies within a range(s) of values. Boundary value analysis works best when parameters (if more than one) are fairly independent of each other. Guidelines for boundary value analysis Select the members at the boundary of each equivalence class. Select a middle value. Also select those values which are adjacent to the boundaries. Include out of range values for robustness. Example: Suppose an equivalence class contains values from 120 through 175. Test case values: 119, 120, 121, 149, 174, 175, 176 14
More examples Calculate property tax based on the value of the property. Single variable function Calculate the score of a competition based on the number of events participated and the score given for each such event. Two-variable function Boundary value analysis for a two-variable function - f (x,y) Y max Y min X min X max 15
Boundary values for different data types Data type Integer, Double (Real) String Array File Boundary cases Range of values given in the problem Empty string, String with one char, String of maximum length, String of length greater than the maximum Similar to String Empty file, File with one record, file of max. size, file overflow Testing Phases Unit testing Test individual components in isolation. Often performed by the developer. Module testing Test a group of components (units) that interact with each other or a group that implements a particular functionality. May be a group of procedures for the imperative paradigm; a group of methods or classes in the O-O paradigm. Interfaces among these components must also be tested. 16
Testing Phases(continued) Subsystem testing Test a group of modules and their interfaces. A subsystem may be a higher-level group of modules in the procedural paradigm or a package in the O-O paradigm. Interfaces between modules, particularly access restrictions across modules, must be tested along with functionalities Integration testing This is a combination of module and subsystem testing. Testing Phases(continued) Acceptance testing Test the entire system as one single product. GUI testing falls into this category. Also requires testing of interfaces between external components such as a database or a device driver. System testing Testing to conform that the software product meets the standards and/or specific criteria set by the customers/users. 17
Classifications of Testing Black-box testing and White-box testing classify testing methods processes to generate test cases and test data. Stages of testing classify the portion of the software product that must be tested at any one time. Combining these two... A unit can be tested both by black-box testing and by white-box testing. A module may undergo black-box testing and white-box testing. Conclusion: Each classification defines a different perspective of testing Further classifications of testing Bottom-up testing Test the units first, then the modules, then the subsystems and so on until the entire product is tested Top-down testing Test the entire software product as one unit, then test its individual subsystems giving emphasis to their interfaces, then test the modules and so on until the components are broken down at the unit level. May require stubs. 18
Stubs - example public static void main(string[] args) { int[] arr = new int[20]; read(arr); sort(arr); if (search(arr, 23)) System.out.println( 23 is in the array ); else System.out.println( 23 NOT in the array ); } public void read(int[] array) { //STUB System.out.println ( read to be implemented ); } public void sort(int[] array) { //STUB System.out.println ( sort to be implemented ); } public boolean search(int[] array, int k) { //STUB return true; } A few more testing classifications Desk Checks Developers often perform walk-throughs of their own, or colleague s code. Alpha testing Testing performed at the developer s site. Generally performed by the QA team, not by the development team. Purpose: to check and correct errors before the release of the actual product Beta testing Testing performed at the customer s site. Generally conducted by the end users (may be the customer) Purpose: to report back to the developers any errors before product release 19
Regression testing Regression testing is the act of repeating a suite of test cases after a significant change to a system. Regression testing is an important part of software maintenance. Documentation of test cases is essential to successful regression testing. Dependency graphs can help to determine which components require regression testing. Object-Oriented Software Testing Research on O-O testing reports that testing methods proposed for imperative approach are not adequate for O-O approach. O-O poses additional problems due to the distinguishing characteristics such as encapsulation, inheritance and polymorphism. 20
Class testing Class testing are modules. Methods are units. Interactions within a class are module interfaces. Interactions between different classes are subsystem interfaces. Testing O-O Encapsulation Within O-O design encapsulation specifies access restrictions to the structural and behavioral members of a class. Such encapsulation should be tested. Check whether all the public members are accessible from everywhere. Check whether all the private members are hidden outside the class. If language supports other access mechanisms, such as protected or class-specific scope, those features should be tested for both accessibility and non-accessibility. 21
Testing Inheritance Subclasses introduce new features (attributes and methods) and can modify inherited behaviors. Modified behaviors must be tested in the context where they are used. Polymorphism adds complexity (e.g., co-variance or contra-variance?) Multiple inheritance adds complexity (What if two methods with the same signature conflict? i.e., both are inherited from different superclasses.) Any change to a superclass requires regression testing of all subclasses. 22