Test Adequacy Assessment using Control Flow and Data Flow

What does coverage cover?
What is the coverage of decisions in respect to the statement , bock , and decision coverage criteria?
What is the term for Decision Coverage?

Transcription

1 CSE592 Advanced Topics in Computer Science Software Engineering Test Adequacy Assessment using Contro Fow and Data Fow Lecture 14 Conditions Any expression that evauates to true or fase constitutes a condition. Such an expression is aso known as a predicate. Exampes Given that A and B are Booean variabes, and x and y are integers, A, x > y, A OR B, A AND (x<y), (A AND B), are a sampe conditions. Note that in programming anguage C, x and x+y are vaid conditions, and the constants 1 and 0 correspond to, respectivey, true and fase. Sources: Foundations of Software Testing (textbook) Dr. Ichu Yoon (icyoon@sunykorea.ac.kr) Simpe and Compound Conditions A simpe condition No use of any Booean operators except for the not operator. Made up of variabes and at most one reationa operator from the set {<, >, ³, ==, ¹ }. Referred to as atomic or eementary conditions Conditions as Decisions Condition can serve as a decision in an appropriate context within a program. Exampe: if, whie, and switch statements to serve as contexts for decisions. A compound condition Made up of two or more simpe conditions joined by one or more Booean operators.

2 Outcomes of a Decision A decision can have three possibe outcomes, true, fase, and undefined. In some cases the evauation of a condition might fai in which case the corresponding decision's outcome is undefined. Couped Conditions Q: How many simpe conditions are there in the compound condition? Cond = (A AND B) OR (C AND A)? The first occurrence of A is said to be couped to its second occurrence. Q: Does Cond contain three or four simpe conditions? Conditions within Assignments Stricty speaking, a condition becomes a decision ony when it is used in the appropriate context such as within an if statement. Decision Coverage

3 Decision coverage A decision is considered covered if the fow of contro has been diverted to a possibe destinations that correspond to this decision i.e., a outcomes of the decision have been taken. For if or a whie statement, expressions must be evauated to true and fase by (mutipe) execution(s). For the switch statement, the fow of contro must be diverted to a possibe destinations in a set of executions. Covering a decision within a program might revea an error that is not reveaed by covering a statements and a bocks. Decision Coverage: Exampe Requirements Receive input (an integer x), and if x is negative, transforms it into a positive vaue before invoking foo-1 to compute the output z. If the input x ³ 0, the program is supposed to compute z using foo-2 Obviousy, the program has errors. Decision Coverage: Exampe (cont.) Assume a test set T={t1:<x=-5>}. It is adequate with respect to statement and bock coverage criteria, but does not revea the error. Another test set T'={t1:<x=-5> t2:<x=3>} does revea the error. It covers the decision whereas T does not. Decision Coverage: Definition Coverage domain D e is the set of decisions in the program Decision coverage of T with respect to (P, R) = D c is the set of decisions covered D i is the set of infeasibe decisions T is considered adequate with respect to the decision coverage criterion if the decision coverage of T with respect to (P, R) is 1.

4 Condition Coverage A decision can be a simpe condition or a compound condition. x<0 (( x<0 AND y<0 ) OR ( p³q )). Condition Coverage In condition coverage, we consider that A simpe condition is covered if it evauates to true and fase in 1 + executions of the program. A compound condition is considered covered if each simpe condition it is comprised of is covered. 50 Decision vs. Condition Coverage Decision coverage does not care the outcome of each condition. Exampe There exist mutipe ways in which the decision in ine 1 is evauated to true or fase! The condition evauates to fase when x³0, regardess of the vaue of y. Another condition, such as x<0 OR y<0, evauates to true regardess of the vaue of y, when x<0. 51 Condition Coverage Coverage domain C e is the set of simpe conditions in the program Condition coverage of T with respect to (P, R) = C c is the set of simpe conditions covered C i is the set of infeasibe simpe conditions T is considered adequate with respect to the condition coverage criterion if the condition coverage of T with respect to (P, R) is 1. 52

5 Condition Coverage: Aternate Formua An aternate formua = where ( ) each simpe condition contributes 2, 1, or 0 to C c depending on whether it is covered, partiay covered, or not covered, respectivey. Condition Coverage: Exampe Requirements for computing z For exampe, if x < y evauates to true but never to fase, then it is considered partiay covered, and contribute a 1 to C c. Consider a test set T is adequate with respect to the statement, bock, and decision coverage criteria and the program behaves correcty against t1 and t Condition Coverage: Exampe (cont.) Truth tabe for computing z: Condition/Decision Coverage Condition Coverage C c =1, C e =2, C i =0. Hence condition coverage for T=0.5. Add the foowing test case t3: <x=3, y=4>to T The enhanced test set T is adequate with respect to the condition coverage criterion and possiby reveas an error in the program. 55

6 Condition/Decision Coverage When a decision is composed of a compound condition, decision coverage does not impy that each simpe condition within a compound condition has taken both vaues true and fase. Condition coverage ensures that each component simpe condition within a condition has taken both vaues true and fase. However, condition coverage does not require each decision to have taken both outcomes. Condition/decision coverage is aso known as branch condition coverage. 57 Condition/Decision Coverage: Exampe Consider the foowing program and two test sets. In cass exercise: Confirm that T1 is adequate with respect to decision coverage but not condition coverage. In cass exercise: Confirm that T2 is adequate with respect to condition coverage but not decision coverage. 58 Condition/Decision Coverage: Definition The condition/decision coverage of T with respect to (P, R) is computed as = ( ( ) where C c is the set of simpe conditions covered D c is the set of decisions covered C e and D e are the set of simpe conditions and decisions respectivey C i and D i are the set of infeasibe simpe conditions and decisions, respectivey. T is considered adequate with respect to the mutipe condition coverage criterion if the condition coverage of T with respect to (P, R) is Condition/Decision Coverage: Exampe adequate with respect to decision coverage adequate with respect to condition coverage In cass exercise: Check that the foowing test set is adequate with respect to the condition/decision coverage criterion. 60

7 Mutipe Condition Coverage Mutipe Condition Coverage Consider a compound condition with two or more simpe conditions. Using condition coverage on some compound condition C impies that each simpe condition within C has been evauated to true and fase. However, does it impy that a combinations of the vaues of the individua simpe conditions in C have been exercised? Mutipe condition coverage is known as branch condition combination coverage. 62 Mutipe Condition Coverage: Exampe Consider a condition D=(A<B) OR (A>C) Consists of 2 simpe conditions A< B and A> C. There are 4 possibe combinations of the outcomes of these two simpe conditions. Consider a test suite T Mutipe Condition Coverage: Definition Assumption Program under test contains a tota of n decisions. Each decision contains k 1, k 2,, k n simpe conditions. Each decision has severa combinations of vaues of its constituent simpe conditions. Check: Does T cover a four combinations? The tota number of combinations to be covered is: n å i=1 2 ki

8 Mutipe Condition Coverage: Definition Mutipe Condition Coverage: Exampe Coverage domain C e is the set of combinations in the program n å i=1 2 ki Consider the foowing program with specifications in the tabe. Requirements for computing S Condition coverage of T with respect to (P, R) = C c is the set of combinations covered C i is the set of infeasibe combinations T is considered adequate with respect to the mutipe condition coverage criterion if the mutipe condition coverage of T with respect to (P, R) is There is an obvious error in the program. Is T adequate with respect to Decision coverage? Condition coverage? Mutipe condition coverage? Does it revea the error? 66 MCC: Exampe (cont.) To improve decision coverage, add t3 to T MCC: Exampe (cont.) In cass exercise: Construct a tabe showing the condition combinations covered by T. Some combinations of simpe conditions remain uncovered. Does T revea the error? Now add a test to T to cover the uncovered combinations. Does your test revea the error? If yes, then under what conditions? 67 68

9 MCC: Exampe (cont.) Condition coverage tabe Linear Code Sequence and Jump Coverage 69 Linear Code Sequence and Jump (LCSAJ) Consider the execution of sequentia programs as a chain of (bock, jump) pairs, where Bock: a sequence of statements, executed one after the other Jump: a pointer to the next such pair. A Linear Code Sequence and Jump (LCSAJ) A program unit comprised of a textua code sequence that terminates in a jump to the beginning of another code sequence and jump. Represent as a tripe (X, Y, Z) where X and Y are, respectivey, ocations of the first and the ast statements Z is the ocation to which the statement at Y jumps. 71 LCSAJ: Exampe1 Consider this program. The ast statement in an LCSAJ (X, Y, Z) is a jump and Z may be program exit. When contro arrives at statement X, foows through to statement Y, and then jumps to statement Z, we say that the LCSAJ (X, Y, Z) is traversed or covered or exercised. 72

10 LCSAJ: Exampe1 (cont.) Consider a program and test suite T. LCSAJ: Exampe2 Consider a program. t1 covers (1, 6, exit), t2 covers (1,4,7) and (7, 8, exit). In cass exercise: Find a LCSAJs. T covers a three LCSAJs. Consider a test set. Does this set cover a LCSAJs? LCSAJ: Definition The LCSAJ coverage of a test set T with respect to (P, R) is computed as: T is considered adequate with respect to the LCSAJ coverage criterion if the LCSAJ coverage of T with respect to (P, R) is. Modified Condition/Decision Coverage (MC/DC) 75

11 Probem of MCC Mutipe condition coverage can be very expensive. Assume that there are many embedded simpe conditions. When a compound condition C contains n simpe conditions, the maximum number of tests required to cover C is 2 n. Compound Conditions and MC/DC MC/DC coverage requires that Every compound condition in a program must be tested by demonstrating that each simpe condition within the compound condition has an independent effect on its outcome. Thus MC/DC coverage is a weaker criterion than the mutipe condition coverage criterion MC/DC: Brute-Force Way Consider a condition C = (C1 and C2) or C3 Fix 2 variabes and varies the vaue of the remaining one Redundant test cases Seected tests MC/DC: Tests for Compound Condition Required tests for a compound condition with 2 simpe conditions fase true Minima tests 79 80

12 MC/DC: Tests for Compound Condition Required tests for a compound condition with 3 simpe conditions MC/DC: Procedure to Generate Tests for Compound Conditions Let C=C1 C2 C3. Generate test for C by extending simper compound condition. Create a tabe with five coumns and four rows. Labe the coumns as Test, C1, C2, C3 and C, from eft to right. The coumn abeed Test contains rows abeed by test case numbers t1 through t4. The remaining entries are empty MC/DC: Procedure to Generate Tests for Compound Conditions (cont.) Copy a entries in coumns C1, C2, and C from the tabe for simpe conditions into coumns C2, C3, and C of the empty tabe. MC/DC: Procedure to Generate Tests for Compound Conditions (cont.) Fi the first three rows in the coumn marked C1 with true and the ast row with fase

13 MC/DC: Procedure to Generate Tests for Compound Conditions (cont.) Fi the ast row under coumns abeed C2, C3, and C with true, true, and fase, respectivey. MC/DC: Definition Adequacy of a test set with respect to the MC/DC A test set T for program P written to meet requirements R, is considered adequate with respect to the MC/DC coverage criterion if upon the execution of P on each test in T, the foowing requirements are met. We now have a tabe containing MC/DC adequate tests for C=(C1 C2 C3) derived from tests for C=(C1 C2). 85 Grows ineary! Each bock in P has been covered. Each simpe condition in P has taken both true and fase vaues. Each decision in P has taken a possibe outcomes. Each simpe condition within a compound condition C in P has been shown to independenty effect the outcome of C. (MC Coverage) 86 MC/DC: Anaysis The first three requirements above correspond to the bock, condition, and decision coverage, respectivey. The 4 th requirement is for the MC coverage. i.e., the MC/DC coverage criterion is a mix of four coverage criteria based on the fow of contro. Conditions that are not part of a decision, such as the one in the foowing statement A= (p<q) (x>y) are aso incuded in the set of conditions to be covered. For a couped condition such as (A B) (C A), poses a probem. ony demonstrate the independent effect of any one occurrence of the couped condition MC/DC: Adequacy Coverage domain (for MC coverage) Let C 1, C 2,.., C N be the conditions in P. n i denote the number of simpe conditions in C i e i is the number of simpe conditions shown to have independent affect on the outcome of C i f i the number of infeasibe simpe conditions in C i. MC coverage of T with respect to (P, R) = ( ) T is considered adequate with respect to the MC coverage if MC c of T is