Achieving Test Case Efficiency By Combining Use Case Scenarios And Orthogonal Arrays

Transcription

1 Achieving Test Case Efficiency By Combining Use Case Scenarios And Orthogonal Arrays Th15 Randall Rice, Randall Rice Consulting Group, USA

2 Achieving Test Case Efficiency by Combining Use Case Scenarios and Orthogonal Arrays Randall W. Rice, CTFL Rice Consulting Services, Inc , Rice Consulting Services, Inc.

3 Agenda Objectives and Benefits Use Cases Test Scenarios Test Conditions Test Case Matrix Orthogonal Arrays Caveats Final Thoughts 3

4 Objectives and Benefits Provide a consistent process to create test cases from use cases Reduce the number of test cases via optimization techniques Leverage traditional test design techniques, such as boundary value analysis & equivalence classes 4

5 Tasks in the Process 1. Identify the Use Cases 2. Identify the Test Scenarios 3. Develop a Test Scenario Matrix 4. Identify the Driving Test Conditions 5. F ind the B est F it O rthogonal A rray 6. Map the Conditions to the Array 5 7. Document the Test Cases

6 Use Cases Describe a sequence of interactions between systems and users Comprised of following components: Brief Description Participating Actors Use Case Diagram Main Course (MC) Alternate Courses (AC) Rules 6

7 Use Case to Test Case Process Flow Project 1 1..n Capability 1 1..n 1 Use Case n 1..n Main Course Alternate Course Test Scenario n 7 Test Case

8 Use Case Diagram ATM1 ^ ATM System 8 ^ Customer ATM Withdrawal ^ Customer DB ^ Savings ^ System DDA System

9 Test Scenarios Begin Use Case Main Course Alternate Course 3 Alternate Course 1 Alternate Course 4 Alternate Course 2 End Use Case End Use Case End Use Case 9

10 Courses 10 Main Successful Withdrawal (MC) Alternate Fast Cash (AC1) Multiple Transactions (AC2) Cancelled Transaction (AC3) Exceptional Login Failure (EX1) Over Limit (EX2) Insufficient Balance (EX3) Invalid Account Status (EX4) Card Reported Stolen (EX5)

11 Concerns About Use Cases as a Basis for Test Cases Use Cases may contain gaps and inaccuracies. Reviews are needed prior to test scenario design. Use Cases often convey conditions to be perform ed, but seldom convey the edges and negatives. Use boundary-value analysis and other test case design techniques to complete the picture. 11

12 12 Test Scenario Matrix

13 Test Conditions System response to user based on procedural rules. Field validation and business rules e.g. Withdrawal Amount is required and is restricted to values in multiples of 20. e.g. Receipt Required is not required and can be indicated by pressing either the Y es or N o key on the A T M. T he default is N o. e.g. Daily Limit is a pre-defined value of $500 per day per customer. 13

14 Test Conditions Bank Customer Y or N (2) Daily Limit Reached Y or N (2) NSF Balance Y or N (2) Login Limit Reached Y or N (2) Receipt Required Y or N (2) Account Types (6) Student, Senior, Regular, Premier, Free, Ultimate Invalid Amount Multiple of 20 or not (2) 14

15 Total Number of Test Cases Conditions (Factors) Values (Levels) Bank Customer 2 Daily Limit Reached 2 NSF Balance 2 Login Limit Reached 2 Receipt Required 2 Account Type 6 Invalid Amount 2 15

16 Test Scenario Matrix with Conditions Example (Optional) 16 Test Conditions Added

17 The Test Conditions Drive the Use Case Scenarios.

18 So, What is a Workable Test Approach? Test everything? Take a sample? Perform exploratory testing? Automate? 18

19 W hat s the N eed? One of the biggest problems for testers is how to deal with software complexity and multiple interactions. It would be great to be able to test all com binations of variables, but w e just don t have the resources, even with automated test tools. Orthogonal arrays give us the next best thing the ability to test pairs of parameters. 19

20 Orthogonality A n orthogonal array is a tw o- dimensional array of numbers that has this interesting property choose any two columns in the array, all the combinations will occur in every colum n pair. Lee Copeland, A Practitioner s Guide to Software Test Design 20

21 The Fault Model Behind Orthogonal Array Testing Integration and interaction between software units is a major source of defects. 21

22 Double-mode Defects Each of these conditions may work correctly in isolation. Condition A Condition B Condition C 22 But when combined with another value may be more likely to reveal a defect.

23 What Research Indicates There have been three studies done where defects were researched after the fact. Research has shown that the highest likelihood of an integration defect relates to interaction between pairs of items or parameters. 23

24 Example of Research Findings [Evaluating FDA recall class failures in medical devices we established that] [...] out of the 109 reports that [were] detailed [enough], 98% showed that the problem could have been detected by testing the device with all pairs of parameter settings. D.R. Wallace and D.R. Kuhn,

25 The Likelihood of Triple-mode or Higher Defects Condition A Condition B Condition C Condition D Condition E 25 Double-mode defects are m ore likely than Triple-mode or higher defects.

26 What Common Sense Indicates It s im possible to test all com binations in applications of any level of complexity. Random performance of conditions is insufficient and gives no level of confidence in the application. 26

27 The Value of Designing Tests with Orthogonal Arrays You know for certain all pairwise conditions have been identified. You get the greatest test coverage with a minimal number of test conditions. You get an even distribution of pairwise combinations. The arrays already exist. All you have to do is find the right one and plug in the values. 27

28 Exact Fit Orthogonal Array MA.XX M A stands for M ixed A rray XX Runs (Test Cases) 1 Factor has 6 Levels 6 Factors have 2 levels However, in actual practice, we seldom get an exact fit. 28

29 Likely Candidates If w e go to N eil Sloan s w eb site We find some arrays that are close: OA MA MA We want the one with the fewest rows. 29

30 Best Fit Orthogonal Array MA Runs (Test Cases) 1 Factor (condition) has 6 Levels (values) 6 Factors have 3 levels This is larger than what we actually need. But, we have reduced 384 test cases to

31 The Actual Array Looks Like This 31

32 Array Mapping 32 Account Type 0 = Student 1 = Senior Citizen 2 = Regular 3 = Premier 4 = Free 5 = Ultimate Bank Customer 0 = No 1 = Yes Daily Limit Reached 0 = No 1 = Yes NSF Balance 0 = No 1 = Yes Login Limit Reached 0 = No 1 = Yes Receipt Required 0 = No 1 = Yes Amount 0 = Not Multiple of 20 1 = Multiple of 20

33 Orthogonal Array with Assignments G aps can be filled with any value. 33

34 T hen W e N eed to A ssociate Each Row With a Test Scenario 34

35 Finally, We Fill The Gaps Each row represents a test case with a combination of conditions. 35

36 AllPairs Output 384 cases reduced to

37 Caveats About Pairwise Testing Pairwise testing is simply a technique built on findings from initial research that doublemode failures are much more likely than triple-mode (or higher) failures. However, there is no rule or guarantee that your applications will not experience a triplemode failure. Therefore, if you have good cause to test more than pairwise cases, by all means do so. 37

38 Caveats About Pairwise Testing (2) The conditions placed in the array must be logically possible to perform with each other. Mutually exclusive conditions will not work in creating test cases unless it is a test to prevent such a condition. 38

39 Caveats About Pairwise Testing (3) When dealing with test scenarios that span applications with diverse types of input, you will need to limit the array to contain related types of input. A E B C D F G 39

40 Final Thoughts This process gives: A way to view pairwise testing at a user scenario level Reduction of test cases via optimization techniques This can provide a minimal number of test cases for regression testing. It is a starting point, not the complete set of cases you will need for high confidence. The ability to incorporate techniques such as boundary value analysis and equivalence classes 40

41 Resources for Orthogonal Arrays and Pairwise Testing A Practitioner s G uide to Softw are Test Design by Lee Copeland This is an excellent resource for many types of test planning as well. Introducing Software Testing by Louise Tamres Also discusses orthogonal arrays Contains many resources for pairwise testing and orthogonal arrays. 41

42 Resources for Orthogonal Arrays and Pairwise Testing (2) Sources of pre-identified orthogonal arrays Quality Engineering Using Robust Design by Madhav S. Phadke N eil Sloan s collection at SAS page on orthogonal arrays managed by Warren F. Kuhfeld Orthogonal Array Testing Strategy (OATS) Technique. 42

43 Resources for Orthogonal Arrays and Pairwise Testing (3) All Pairs Tool James Bach has written a tool called ALLPAIRS, a command-line PERL script that will create all-pairs tables. James' site is 43

44 Finally Thanks to Ron Rissel and Mike Stevens at Vanguard Mutual Funds for their research and trials of this approach. To hear a replay of this session: 44

45 Bio - Randall W. Rice 45 Over 30 years experience in building and testing information systems in a variety of industries and technical environments Certified Software Quality Analyst Certified Software Tester ASTQB Certified Tester Foundation level Director on the American Software Testing Qualification Board (ASTQB) Chairperson, Q A I s annual softw are testing conference Co-author with William E.Perry, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems Principal Consultant and Trainer, Rice Consulting Services, Inc.

46 Contact Information Randall W. Rice, CTFL Rice Consulting Services, Inc. P.O. Box 6127 Moore, OK Ph: Fax: Web site:

47