AUTOMATED UNIT TEST GENERATION DURING SOFTWARE DEVELOPMENT A Controlled Experiment and Think-aloud Observations

Transcription

1 AUTOMATED UNIT TEST GENERATION DURING SOFTWARE DEVELOPMENT A Controlled Experiment and Think-aloud Observations ISSTA 2015 José Miguel Rojas [email protected] Joint work with Gordon Fraser and Andrea Arcuri

2 Testing is a widespread validation approach in industry, but it is still largely ad hoc, expensive, and unpredictably effective. Software Testing Research: Achievements, Challenges, Dreams, A. Bertolino. Future of Software Engineering. IEEE

3 Testing is a widespread validation approach in industry, but it is still largely ad hoc, expensive, and unpredictably effective. Software Testing Research: Achievements, Challenges, Dreams, A. Bertolino. Future of Software Engineering. IEEE Test case generation has a strong impact on the effectiveness and efficiency of testing. one of the most active research topics in software testing for several decades, resulting in many different approaches and tools. An orchestrated survey of methodologies for automated software test case generation, S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M.B. Cohen, W. Grieskamp, M. Harman, M.J. Harrold, P. McMinn. J. Systems and Software. Elsevier

4

5 BACK IN ISSTA 2013 Does automated white-box test generation really help software testers?, G. Fraser, M. Staats, P. McMinn, A. Arcuri and F. Padberg

6 BACK IN ISSTA 2013 ARE UNIT TEST GENERATION TOOLS HELPFUL TO DEVELOPERS WHILE THEY ARE CODING? Does automated white-box test generation really help software testers?, G. Fraser, M. Staats, P. McMinn, A. Arcuri and F. Padberg

7

8 CODE COVERAGE

9 CODE COVERAGE TIME SPENT ON TESTING

10 CODE COVERAGE TIME SPENT ON TESTING IMPLEMENTATION QUALITY

11 CONTROLLED EXPERIMENT

12 CONTROLLED EXPERIMENT Golden Implementation

13 CONTROLLED EXPERIMENT Golden Implementation Class Template

14 CONTROLLED EXPERIMENT Golden Implementation Class Template Implementation

15 CONTROLLED EXPERIMENT Golden Implementation Class Template Implementation

16 CONTROLLED EXPERIMENT Manual Golden Implementation Class Template Implementation

17 CONTROLLED EXPERIMENT Manual Golden Implementation Class Template 1 hour Implementation

18 CONTROLLED EXPERIMENT Manual Golden Implementation Class Template 1 hour Implementation 41

19 CONTROLLED EXPERIMENT Manual Golden Implementation Class Template 1 hour Implementation 41

20 CONTROLLED EXPERIMENT Manual Golden Implementation Class Template 1 hour Implementation 41 2

21 CONTROLLED EXPERIMENT Manual Golden Implementation Class Template 1 hour Implementation

22 DOES USING EVOSUITE DURING SOFTWARE DEVELOPMENT LEAD TO TEST SUITES WITH HIGHER CODE COVERAGE? RQ 1

23 CODE COVERAGE participants test suites run on their own implementations 100% Assisted Manual 80% 83% Branch Coverage 60% 40% 63% 57% 39% 41% 38% 50% 20% 26% 0% FilterIterator FixedOrderComparator ListPopulation PredicatedMap

24 CODE COVERAGE Times coverage was checked Times Coverage was checked Assisted Manual FilterIterator FixedOrderComparator ListPopulation PredicatedMap Category Axis

25 CODE COVERAGE participant s test suites run on their own implementations Branch Coverage (%) 100% 75% 50% 25% 0% ListPopulation Manual EvoSuite Assisted Time (min)

26 CODE COVERAGE participant s test suites run on their own implementations Branch Coverage (%) 100% 75% 50% 25% 0% ListPopulation Manual EvoSuite Assisted Time (min)

27 CODE COVERAGE participant s test suites run on their own implementations Branch Coverage (%) 100% 75% 50% 25% 0% ListPopulation Manual EvoSuite Assisted Time (min)

28 CODE COVERAGE participant s test suites run on their own implementations Branch Coverage (%) 100% 75% 50% 25% 0% ListPopulation Manual EvoSuite Assisted Time (min)

29 CODE COVERAGE participants test suites run on golden implementations 100% Assisted Manual 80% Branch Coverage 60% 40% 20% 41% 30% 35% 21% 37% 28% 42% 50% 0% FilterIterator FixedOrderComparator ListPopulation PredicatedMap

30 CODE COVERAGE participant s test suites run on golden implementations, over time Branch Coverage Branch Coverage 100% 75% 50% 25% 0% 100% 75% 50% 25% 0% FilterIterator FixedOrderComparator Time (min) Assisted Manual EvoSuite-generated ListPopulation PredicatedMap Time (min) Branch Coverage Branch Coverage 100% 75% 50% 25% 0% Time (min) 100% 75% 50% 25% 0% Time (min)

31 CODE COVERAGE participant s test suites run on golden implementations, over time Branch Coverage Branch Coverage 100% 75% 50% 25% Assisted Manual EvoSuite-generated FilterIterator FixedOrderComparator 0% depending on how the generated 0% tests are used Time (min) Time (min) ListPopulation PredicatedMap 100% 75% 50% 25% 0% Time (min) Branch Coverage Branch Coverage 100% 75% Coverage can be higher when using EvoSuite, 50% 25% 100% 75% 50% 25% 0% Time (min)

32 DOES USING EVOSUITE DURING SOFTWARE DEVELOPMENT LEAD TO DEVELOPERS SPENDING MORE OR LESS TIME ON TESTING? RQ 2

33 TESTING EFFORT Number of test runs Assisted Manual FilterIterator FixedOrderComparator ListPopulation PredicatedMap

34 TESTING EFFORT Minutes spent on testing Assisted Manual FilterIterator FixedOrderComparator ListPopulation PredicatedMap

35 TESTING EFFORT Minutes spent on testing Assisted Manual Using EvoSuite reduces the time spent on testing FilterIterator FixedOrderComparator ListPopulation PredicatedMap

36 DOES USING EVOSUITE DURING SOFTWARE DEVELOPMENT LEAD TO SOFTWARE WITH FEWER BUGS? RQ 3

37 IMPLEMENTATION QUALITY Golden test suites run on participants implementations Number of Failures+Errors Assisted Manual FilterIterator FixedOrderComparator ListPopulation PredicatedMap

38 IMPLEMENTATION QUALITY Golden test suites run on participants implementations Number of Failures+Errors Assisted Manual Using EvoSuite during development did not lead to to better implementations. FilterIterator FixedOrderComparator ListPopulation PredicatedMap

39 DOES SPENDING MORE TIME WITH EVOSUITE AND ITS TESTS LEAD TO BETTER IMPLEMENTATIONS? RQ 4

40 PRODUCTIVITY Time spent with EvoSuite Correlation with number of failures plus errors Number of runs Time spent on tests FilterIterator FixedOrderComparator ListPopulation PredicatedMap

41 PRODUCTIVITY Time spent with EvoSuite Correlation with number of failures plus errors Number of runs Time spent on tests 0.35 Implementation quality improves the more time developers spend with EvoSuite-generated tests FilterIterator FixedOrderComparator ListPopulation PredicatedMap

42 Using automated unit test generation does impact developers productivity,

43 Using automated unit test generation does impact developers productivity, but

44 Using automated unit test generation does impact developers productivity, but how to make the most out of unit test generation tools?

45 THINK ALOUD OBSERVATIONS K. A. Ericsson and H. A. Simon, Protocol Analysis: Verbal Reports as Data (revised edition). MIT Press, J. Hughes and S. Parkes, Trends in the use of verbal protocol analysis in software engineering research, Behaviour and Information Technology, vol. 22, no. 2, pp , 2003.

46 THINK ALOUD OBSERVATIONS Subject K. A. Ericsson and H. A. Simon, Protocol Analysis: Verbal Reports as Data (revised edition). MIT Press, J. Hughes and S. Parkes, Trends in the use of verbal protocol analysis in software engineering research, Behaviour and Information Technology, vol. 22, no. 2, pp , 2003.

47 THINK ALOUD OBSERVATIONS Observer Subject K. A. Ericsson and H. A. Simon, Protocol Analysis: Verbal Reports as Data (revised edition). MIT Press, J. Hughes and S. Parkes, Trends in the use of verbal protocol analysis in software engineering research, Behaviour and Information Technology, vol. 22, no. 2, pp , 2003.

48 THINK ALOUD OBSERVATIONS Golden Implementation Class Template 2 hours Implementation 5 1 4

49 THINK ALOUD OBSERVATIONS Golden Implementation Class Template 2 hours Implementation 5 1 4

50 THINK ALOUD OBSERVATIONS Golden Implementation Class Template 2 hours Implementation 5 1 4

51 THINK ALOUD OBSERVATIONS Golden Implementation Class Template 2 hours Implementation 5 1 4

52 THINK ALOUD OBSERVATIONS Golden Implementation Class Template 2 hours Implementation 5 1 4

53 THINK ALOUD OBSERVATIONS Golden Implementation Class Template 2 hours Implementation 5 1 4

54 THINK ALOUD OBSERVATIONS Golden Implementation Class Template 2 hours Implementation 5 1 4

55 THINK ALOUD OBSERVATIONS Golden Implementation Class Template 2 hours Implementation 5 1 4

56 THINK ALOUD OBSERVATIONS Golden Implementation Class Template 2 hours Implementation 5 1 4

57 THINK ALOUD OBSERVATIONS Golden Implementation Class Template 2 hours Implementation 5 1 4

58 RESULTS FilterIterator FixedOrderComparator ListPopulation PredicatedMap PredicatedMap % 92% 97% 72% 94% 95% 100% 94% 100% 83% Images:

59 RESULTS FilterIterator FixedOrderComparator ListPopulation PredicatedMap PredicatedMap % 97% 94% 95% 100% 100% Images:

60 RESULTS FilterIterator FixedOrderComparator ListPopulation PredicatedMap PredicatedMap % 97% 72% 94% 100% 100% Images:

61 LESSONS LEARNED

62 LESSONS LEARNED There are different approaches to testing and test generation tools should be adaptable to them

63 LESSONS LEARNED There are different approaches to testing and test generation tools should be adaptable to them Developers behaviour is often not driven by code coverage

64 LESSONS LEARNED There are different approaches to testing and test generation tools should be adaptable to them Developers behaviour is often not driven by code coverage Readability of generated unit tests is paramount

65 LESSONS LEARNED There are different approaches to testing and test generation tools should be adaptable to them Developers behaviour is often not driven by code coverage Readability of generated unit tests is paramount Integration into development environments must be improved

66 LESSONS LEARNED There are different approaches to testing and test generation tools should be adaptable to them Developers behaviour is often not driven by code coverage Readability of generated unit tests is paramount Integration into development environments must be improved Education/Best practices: Developers do not know how to best use automated test generation tools!

67

68

69

70

71 Coverage is easy to assess because it is a number, while readability is a very nontangible property

72 Coverage is easy to assess because it is a number, while readability is a very nontangible property What is readable to me may not be readable to you. It is readable to me just because I spent the last hour and a half doing this.

73 Coverage is easy to assess because it is a number, while readability is a very nontangible property What is readable to me may not be readable to you. It is readable to me just because I spent the last hour and a half doing this. Participant 5

74

75