Achieving Test Case Efficiency By Combining Use Case Scenarios And Orthogonal Arrays Th15 Randall Rice, Randall Rice Consulting Group, USA
Achieving Test Case Efficiency by Combining Use Case Scenarios and Orthogonal Arrays Randall W. Rice, CTFL Rice Consulting Services, Inc. www.riceconsulting.com 2006, Rice Consulting Services, Inc.
Agenda Objectives and Benefits Use Cases Test Scenarios Test Conditions Test Case Matrix Orthogonal Arrays Caveats Final Thoughts 3
Objectives and Benefits Provide a consistent process to create test cases from use cases Reduce the number of test cases via optimization techniques Leverage traditional test design techniques, such as boundary value analysis & equivalence classes 4
Tasks in the Process 1. Identify the Use Cases 2. Identify the Test Scenarios 3. Develop a Test Scenario Matrix 4. Identify the Driving Test Conditions 5. F ind the B est F it O rthogonal A rray 6. Map the Conditions to the Array 5 7. Document the Test Cases
Use Cases Describe a sequence of interactions between systems and users Comprised of following components: Brief Description Participating Actors Use Case Diagram Main Course (MC) Alternate Courses (AC) Rules 6
Use Case to Test Case Process Flow Project 1 1..n Capability 1 1..n 1 Use Case 1 1 1..n 1..n Main Course Alternate Course Test Scenario 1 1 1 1 1..n 7 Test Case
Use Case Diagram ATM1 ^ ATM System 8 ^ Customer ATM Withdrawal ^ Customer DB ^ Savings ^ System DDA System
Test Scenarios Begin Use Case Main Course Alternate Course 3 Alternate Course 1 Alternate Course 4 Alternate Course 2 End Use Case End Use Case End Use Case 9
Courses 10 Main Successful Withdrawal (MC) Alternate Fast Cash (AC1) Multiple Transactions (AC2) Cancelled Transaction (AC3) Exceptional Login Failure (EX1) Over Limit (EX2) Insufficient Balance (EX3) Invalid Account Status (EX4) Card Reported Stolen (EX5)
Concerns About Use Cases as a Basis for Test Cases Use Cases may contain gaps and inaccuracies. Reviews are needed prior to test scenario design. Use Cases often convey conditions to be perform ed, but seldom convey the edges and negatives. Use boundary-value analysis and other test case design techniques to complete the picture. 11
12 Test Scenario Matrix
Test Conditions System response to user based on procedural rules. Field validation and business rules e.g. Withdrawal Amount is required and is restricted to values in multiples of 20. e.g. Receipt Required is not required and can be indicated by pressing either the Y es or N o key on the A T M. T he default is N o. e.g. Daily Limit is a pre-defined value of $500 per day per customer. 13
Test Conditions Bank Customer Y or N (2) Daily Limit Reached Y or N (2) NSF Balance Y or N (2) Login Limit Reached Y or N (2) Receipt Required Y or N (2) Account Types (6) Student, Senior, Regular, Premier, Free, Ultimate Invalid Amount Multiple of 20 or not (2) 14
Total Number of Test Cases Conditions (Factors) Values (Levels) Bank Customer 2 Daily Limit Reached 2 NSF Balance 2 Login Limit Reached 2 Receipt Required 2 Account Type 6 Invalid Amount 2 15
Test Scenario Matrix with Conditions Example (Optional) 16 Test Conditions Added
The Test Conditions Drive the Use Case Scenarios.
So, What is a Workable Test Approach? Test everything? Take a sample? Perform exploratory testing? Automate? 18
W hat s the N eed? One of the biggest problems for testers is how to deal with software complexity and multiple interactions. It would be great to be able to test all com binations of variables, but w e just don t have the resources, even with automated test tools. Orthogonal arrays give us the next best thing the ability to test pairs of parameters. 19
Orthogonality A n orthogonal array is a tw o- dimensional array of numbers that has this interesting property choose any two columns in the array, all the combinations will occur in every colum n pair. Lee Copeland, A Practitioner s Guide to Software Test Design 20
The Fault Model Behind Orthogonal Array Testing Integration and interaction between software units is a major source of defects. 21
Double-mode Defects Each of these conditions may work correctly in isolation. Condition A Condition B Condition C 22 But when combined with another value may be more likely to reveal a defect.
What Research Indicates There have been three studies done where defects were researched after the fact. Research has shown that the highest likelihood of an integration defect relates to interaction between pairs of items or parameters. www.pairwise.org 23
Example of Research Findings [Evaluating FDA recall class failures in medical devices we established that] [...] out of the 109 reports that [were] detailed [enough], 98% showed that the problem could have been detected by testing the device with all pairs of parameter settings. D.R. Wallace and D.R. Kuhn, 2001 http://csrc.ncsl.nist.gov/staff/kuhn/final-rqse.pdf 24
The Likelihood of Triple-mode or Higher Defects Condition A Condition B Condition C Condition D Condition E 25 Double-mode defects are m ore likely than Triple-mode or higher defects.
What Common Sense Indicates It s im possible to test all com binations in applications of any level of complexity. Random performance of conditions is insufficient and gives no level of confidence in the application. 26
The Value of Designing Tests with Orthogonal Arrays You know for certain all pairwise conditions have been identified. You get the greatest test coverage with a minimal number of test conditions. You get an even distribution of pairwise combinations. The arrays already exist. All you have to do is find the right one and plug in the values. 27
Exact Fit Orthogonal Array MA.XX.6.1.2.6 M A stands for M ixed A rray XX Runs (Test Cases) 1 Factor has 6 Levels 6 Factors have 2 levels However, in actual practice, we seldom get an exact fit. 28
Likely Candidates If w e go to N eil Sloan s w eb site http://www.research.att.com/~njas/oadir/ We find some arrays that are close: OA.49.8.7.2 MA.36.6.1.3.6.2.3 MA.18.3.6.6.1 We want the one with the fewest rows. 29
Best Fit Orthogonal Array MA.18.3.6.6.1 18 Runs (Test Cases) 1 Factor (condition) has 6 Levels (values) 6 Factors have 3 levels This is larger than what we actually need. But, we have reduced 384 test cases to 18. 30
The Actual Array Looks Like This 31
Array Mapping 32 Account Type 0 = Student 1 = Senior Citizen 2 = Regular 3 = Premier 4 = Free 5 = Ultimate Bank Customer 0 = No 1 = Yes Daily Limit Reached 0 = No 1 = Yes NSF Balance 0 = No 1 = Yes Login Limit Reached 0 = No 1 = Yes Receipt Required 0 = No 1 = Yes Amount 0 = Not Multiple of 20 1 = Multiple of 20
Orthogonal Array with Assignments G aps can be filled with any value. 33
T hen W e N eed to A ssociate Each Row With a Test Scenario 34
Finally, We Fill The Gaps Each row represents a test case with a combination of conditions. 35
AllPairs Output 384 cases reduced to 12. 36
Caveats About Pairwise Testing Pairwise testing is simply a technique built on findings from initial research that doublemode failures are much more likely than triple-mode (or higher) failures. However, there is no rule or guarantee that your applications will not experience a triplemode failure. Therefore, if you have good cause to test more than pairwise cases, by all means do so. 37
Caveats About Pairwise Testing (2) The conditions placed in the array must be logically possible to perform with each other. Mutually exclusive conditions will not work in creating test cases unless it is a test to prevent such a condition. 38
Caveats About Pairwise Testing (3) When dealing with test scenarios that span applications with diverse types of input, you will need to limit the array to contain related types of input. A E B C D F G 39
Final Thoughts This process gives: A way to view pairwise testing at a user scenario level Reduction of test cases via optimization techniques This can provide a minimal number of test cases for regression testing. It is a starting point, not the complete set of cases you will need for high confidence. The ability to incorporate techniques such as boundary value analysis and equivalence classes 40
Resources for Orthogonal Arrays and Pairwise Testing A Practitioner s G uide to Softw are Test Design by Lee Copeland This is an excellent resource for many types of test planning as well. Introducing Software Testing by Louise Tamres Also discusses orthogonal arrays www.pairwise.org Contains many resources for pairwise testing and orthogonal arrays. 41
Resources for Orthogonal Arrays and Pairwise Testing (2) Sources of pre-identified orthogonal arrays Quality Engineering Using Robust Design by Madhav S. Phadke N eil Sloan s collection at www.research.att.com/~njas/oadir/index.html SAS page on orthogonal arrays managed by Warren F. Kuhfeld http://support.sas.com/techsup/technote/ts723.html Orthogonal Array Testing Strategy (OATS) Technique. http://www.seilevel.com/oats.html 42
Resources for Orthogonal Arrays and Pairwise Testing (3) All Pairs Tool James Bach has written a tool called ALLPAIRS, a command-line PERL script that will create all-pairs tables. James' site is http://www.satisfice.com/. 43
Finally Thanks to Ron Rissel and Mike Stevens at Vanguard Mutual Funds for their research and trials of this approach. To hear a replay of this session: http://www.riceconsulting.com/pres/index.htm 44
Bio - Randall W. Rice 45 Over 30 years experience in building and testing information systems in a variety of industries and technical environments Certified Software Quality Analyst Certified Software Tester ASTQB Certified Tester Foundation level Director on the American Software Testing Qualification Board (ASTQB) Chairperson, 1995-2000 Q A I s annual softw are testing conference Co-author with William E.Perry, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems Principal Consultant and Trainer, Rice Consulting Services, Inc.
Contact Information Randall W. Rice, CTFL Rice Consulting Services, Inc. P.O. Box 6127 Moore, OK 73153 Ph: 405-691-8075 Fax: 405-691-1441 Web site: www.riceconsulting.com e-mail: rrice@riceconsulting.com 46