Random Testing: The Best Coverage Technique - An Empirical Proof

Transcription

1 , pp Random Testing: The Best Coverage Technique - An Empirical Proof K Koteswara Rao 1 and Prof GSVP Raju 2 1 Asst prof, CSE Department, GMRIT Rajam 2 Department of CS&ST, Andhra University, VIZAG, AP, INDIA Abstract Soft testing is one way of assuring quality of the software. There are many testing techniques to test the software, however random testing is one among the best alternatives, which will generate the test inputs based on some dissemination. It is evaluated theoretically and concluded that the performance of random testing is very impressive when it is compared with partition testing even though it gives good results for testing. Here in this paper we addressed the difference in coverage between random testing and partition testing, here we have taken one example program and written the test cases for that. To test the program we have used automated tools JUnit for partition testing and Randoop for random testing. Once it is over using EclEmma the coverage of test cases is evaluated and proved RT is best. Keywords: Random testing, Partition Testing, Randoop, Eclemma, Junit, Coverage 1. Introduction Software testing [1] is essential in the development of software life cycle, to uncover the bugs in the software. Testing is essential to uncover the bugs in the software, testing is the tedious job, there are many testing techniques but the exhaustive testing is impossible for every software, we can say testing a software is nothing but checking the each and every line of code if we are able to get maximum coverage then that testing technique will reveals more number of bugs. For testing the software one best alternative approach is that dividing the test input into partitions this type testing is called partition testing if the input parameters are many then partition testing will very difficult to perform and the coverage of the partition testing is very poor if the input domain consists of too many parameters. If the input domain is too large then Random testing [2] is a useful testing technique in finding the low frequency bugs with maximum coverage rate, since the low frequency bugs may cause huge disaster like Ariane-5 which is crashed due to a small exception in conversion of 64-bit number to 16-bit, the cost of this bug is over 1 billion dollars to cover this type of mint errors random testing is best approach [4], [6]. 2. Partition Testing Partition testing [10] was first introduced by Weyuker and Ostrand. The input domain is divided into subsets, the system under test is executed and the results are compared with system requirements [3], Here a simple example Sample Grade.java To show the test case coverage we used EclEmma, it is a code coverage tool which is available with eclipse. EclEmma is a coverage tool; it is an Eclipse plug-in that generates code coverage reports and provides simple trace information about test cases. Coverage is defined as a measure of the completeness of the set of test cases. In other words your test ISSN: IJSEIA Copyright c 2015 SERSC

2 cases are better when they cover more source code [7]. Basically there are four types of coverage s cases? cases? Method coverage: how many of your methods have been called by your test Statement coverage: how many of your statements have been run by your test Decision/Branch coverage: how many decision points (if statements) have evaluated both true and false? Condition coverage: how many Boolean sub-expressions have been evaluated both true and false at a decision point? We can easily calculate coverage for smaller programs, however, for larger programs, it is much more difficult. Coverage tools, like EclEmma, help by automating the coverage process and providing readable reports. EclEmma can be used with JUnit. EclEmma runs with the JUnit test cases and generates the coverage report from the execution. EclEmma provides overall, package, and file level information on statement. EclEmma can be downloaded for a manual install from its Source Forge Project Page. Additionally, we can also install EclEmma with eclipse. For more information on installing eclipse plugins, consult the tutorial on the subject. EclEmma's update site is Sample Grade.java Public static char getlettergrade(int mark) { // assume that mark is between 0 and 100 (inclusive) assert (mark >= 0 && mark <= 100) : "mark is out-of-range: " + mark; if (mark >= 75) { return'a'; elseif (mark >= 60) { return'b'; elseif (mark >= 50) { // an logical error here return'c'; elseif (mark < 0&& mark >100){ return'n'; //not valid else { return'f'; For this simple program the partition testing is performed as follows, as we can see in the code the how the input is divided into partitions. Here we are dividing the test input into four subsets 1. Test with the typical value in the partition 2. Test at the lower and higher boundary values 116 Copyright c 2015 SERSC

3 3. Test at exact boundaries 4. Test with out of bound values Test cases developed for Grade.java Public void testtypical() { // test a typical(in between values) value in partitions assertequals("wrong grade", 'A', Grade.getLetterGrade(95)); assertequals("wrong grade", 'B', Grade.getLetterGrade(72)); assertequals("wrong grade", 'C', Grade.getLetterGrade(55)); assertequals("wrong grade", 'F', Grade.getLetterGrade(30)); Public void testgreatestboundaries() { // test the boundaries of the partitions assertequals("wrong grade", 'A', Grade.getLetterGrade(100)); assertequals("wrong grade", 'B', Grade.getLetterGrade(60)); assertequals("wrong grade", 'B', Grade.getLetterGrade(74)); assertequals("wrong grade", 'C', Grade.getLetterGrade(58)); assertequals("wrong grade", 'C', Grade.getLetterGrade(59)); assertequals("wrong grade", 'F', Grade.getLetterGrade(0)); assertequals("wrong grade", 'F', Grade.getLetterGrade(49)); Public void testboundariesatequal() { // test the equal boundaries of the partitions assertequals("wrong grade", 'A', Grade.getLetterGrade(75)); assertequals("wrong grade", 'B', Grade.getLetterGrade(60)); assertequals("wrong grade", 'C', Grade.getLetterGrade(50)); assertequals("wrong grade", 'F', Grade.getLetterGrade(49)); Public void testlowest() { // test with out of bound values assertequals("wrong grade", 'F', Grade.getLetterGrade(-1)); assertequals("wrong grade", 'F', Grade.getLetterGrade(101)); assertequals("wrong grade", 'F', Grade.getLetterGrade(-100)); assertequals("wrong grade", 'F', Grade.getLetterGrade(-75)); The detailed procedure of partition testing with automated JUnit tool [8] can be viewed in following steps Step1: Create a JUnit test case for Grade.java; you can see the following figure for creating test case for Grade.java Copyright c 2015 SERSC 117

4 Figure 2.1. Creating JUnit Test Cases Step2: Whenever we click on JUnit test case a new dialog box will be appeared to select the testing frame work you would like to test (i.e., JUnit 3 or JUnit 4) here we used JUnit 3. Figure 2.2. Selecting Framework Step 3: After selecting testing frame work it will be displayed as you can see in the following figure the JUnit 3 framework displayed for new test case 118 Copyright c 2015 SERSC

5 Figure 2.3. Showing Selected Framework Step 4: Write the manual test cases for all the input partitions that we are divided the input domain (here for Grade.java we divided the input domain into 4 subsets). Figure 2.4. Displaying Test Cases for the Grade Class Copyright c 2015 SERSC 119

6 Step 4: Once if we write test cases for all partitions run the test class and you can see passed and failure test cases after that click on EclEmma code coverage icon it will display the coverage as you can see in the following figure(coverage, Covered instructions, Missed Instructions, total instructions). The coverage for Grade.java with partition testing is 61.5% over all coverage for entire source code with all test cases is 64.3%. 3. Random Testing Figure 2.5. Showing Partition Testing Code Coverage Random testing is a test technique [4-5] where test cases are chosen at random from the input domain of the program. As explained above, random testing uses a minimum of knowledge about the SUT and is therefore considered to be a black-box testing technique. Sometimes an unsystematic approach to selecting test data is appealing. It could be the case that a systematic plan is hard to develop or too expensive to perform functional or structural testing and therefore a random test approach is preferred. To increase its relevancy, Random Testing may choose the input according to an operational profile [6]. Random testing with Randoop Randoop [9] takes the input as set of classes under test,seed value, maximum test run length, and contract checkers. Randoop outputs two types of test suites (failure test suite and successful test suite). Here for the same example(grade.java) program and test program considering for the random testing The procedure for random testing with Randoop as follows: 1. Install Randoop plug-in as available with eclipse 120 Copyright c 2015 SERSC

7 2. Consider the same test project which is considered for partition testing 3. Run the test program with randoop 4. Provide the input as the seed value(here we taken as 0), test input length(here we taken as 100), maximum test run length(here we taken as 10000) Once if we run the test cases Randoop [9] test inputs it will write failure and passed test suites, after that click on EclEmma code coverage icon it will display the coverage as you can see in the following figure(coverage, Covered instructions, Missed Instructions, total instructions). The coverage for Grade.java with Random testing is 86.5% over all coverage for entire source code with all test cases is 99.6%. 4. Results Figure 3.1. Displaying Random Testing Coverage The following results shows that the coverage of partition testing and random testing. It is clearly saying that for the same example and same number of test cases with the help of random inputs random testing gives better results than partition testing. Copyright c 2015 SERSC 121

8 120 Test Coverage Test Coverage Partition Testing Random Testing Figure 4.1. Code Coverage of Partition Testing and Random Testing 5. Conclusion Random testing is very much useful technique in generating random inputs with these random inputs the test cases will cover more number of lines in the source code. The test cases with more coverage processes better test results due the unawareness on random testing it is not used in many applications testing. References [1] R. S. Pressman, Software Engineering a practitioners approach, 7 th edition, ISBN-10: [2] K. K. Rao and GSVP Raju, Optimizing the Software Testing Efficiency by Using a Genetic Algorithm - A Design Methodology, ACM SIGSOFT Software Engineering Notes, vol. 38, no. 3, (2013) May. [3] M. Grindal, J. Offutt and S. Andler, Combination testing strategies: a survey, Software Testing, Verification and Reliability, vol. 15, no. 3, (2005) March, pp [4] K. K. Rao and GSVP Raju, Theoretical Investigations to Random Testing Variants and its Implications, International Journal of Software Engineering and Its Applications, vol. 9, no. 5, (2015), pp [5] A. Arcuri and L. Briand, Random Testing: Theoretical Results and Practical Implications, IEEE transactions on Software Engineering, vol. 38, no. 2, (2012) April. [6] Developing Optimal Directed Random Testing Technique to Reduce Interactive Faults-Systematic Literature and Design Methodology, Indian Journal of Science and Technology, ISSN , vol. 8, no. 8, (2015) April, pp [7] [8] [9] randoop.googlecode.com/hg-history/v1.3.3/doc/index.html [10] P. Ammann and J. Offutt, Introduction to Software Testing, ISBN Copyright c 2015 SERSC