Code Change Impact Analysis for Testing Configurable Software Systems. Mithun Acharya ABB Corporate Research Raleigh NC USA

Transcription

1 Code Change Impact Analysis for Testing Configurable Software Systems Mithun Acharya ABB Corporate Research Raleigh NC USA

2 ABB: A power and automation company >125 years, >100 nations, ~150,000 employees Power products and electronics, Control Systems, Robotics, Smart Grid, Renewable Energy, 2

3 ABB Corporate Research Industrial Software Systems (ISS) research group USA Germany Switzerland Poland Sweden India China Raleigh, NC 7 research centers worldwide 3

4 Software in ABB Hardware with software inside Software with few hardware components Pure Software 4

5 Software Evolution: A CSS constantly changes 5 Hundreds of such changes committed daily

6 Change Control Board meetings Change impact visualizations for managers for decision making Change impact at the code level for developers Unit/module-level change impact for testers 6

7 Imp: Code change impact analysis for C/C++ programs IMP BUILD SERVER Version control SERVER CHECKOUT NIGHTLY BUILD Impact of C23567 Impact of C23586 Impact of C23712 Impact of changes since last nightly build C23586 Impact database 7

8 Quantifiable risk/cost analysis of changes to CSS Automated Dependency Analysis Will changes to foo.c, affect Bob s module? Dependency analysis Automated Risk/Cost Analysis Automated What-If Analysis 3 days to release!!! Should I implement this feature or bug fix? What is the best way to fix this bug or implement that new feature? Overlay change impact with risky areas in code Automated Regression Testing Should I re-run ALL of my test suite for this change? New tests required? 3941 lines vs. 6 lines 8 Test suite

9 Program and System Dependence Graphs for Slicing Program Dependence Graph (PDG) for main void main() { int i = 1; int sum = 0; while (i<11) { sum = add(sum, i); i = add(i, 1); } printf("sum = %d\n", sum); printf("i = %d\n", i); } static int add(int a, int b) { return(a+b); } System Dependence Graph (SDG) PDG for add 9 Code/Image Source: GrammaTech

10 Making impact analysis practical and useful 10

11 Making impact analysis practical and useful Mithun Acharya, Brian Robinson. Practical Change Impact Analysis based on Static Program Slicing for Industrial Software Systems. ICSE 2011 SEiP, FSE 2012 (tool demo) Mithun Acharya, Xiao Qu, Brian Robinson. Cross-System Change Impact Analysis Using Test Cases. Under submission. Scaling beyond million lines 10

12 Making impact analysis practical and useful Mithun Acharya, Brian Robinson. Practical Change Impact Analysis based on Static Program Slicing for Industrial Software Systems. ICSE 2011 SEiP, FSE 2012 (tool demo) Mithun Acharya, Xiao Qu, Brian Robinson. Cross-System Change Impact Analysis Using Test Cases. Under submission. Scaling beyond million lines Xiao Qu, Mithun Acharya, Brian Robinson. Configuration Selection Using Code Change Impact Analysis for Regression Testing. ICSM 2012 Xiao Qu, Mithun Acharya, Brian Robinson. Impact Analysis of Configuration Changes for Test Case Selection. ISSRE 2011 Testing configurable systems 10

13 Making impact analysis practical and useful Mithun Acharya, Brian Robinson. Practical Change Impact Analysis based on Static Program Slicing for Industrial Software Systems. ICSE 2011 SEiP, FSE 2012 (tool demo) Mithun Acharya, Xiao Qu, Brian Robinson. Cross-System Change Impact Analysis Using Test Cases. Under submission. Scaling beyond million lines Xiao Qu, Mithun Acharya, Brian Robinson. Configuration Selection Using Code Change Impact Analysis for Regression Testing. ICSM 2012 Xiao Qu, Mithun Acharya, Brian Robinson. Impact Analysis of Configuration Changes for Test Case Selection. ISSRE 2011 Testing configurable systems Tingting Yu, Xiao Qu, Mithun Achayra, Gregg Rothermel. Oracle-Based Regression Test Selection. Under submission. Regression test selection 10

14 What configurations should we select for retesting? Mithun Acharya, Brian Robinson. Practical Change Impact Analysis based on Static Program Slicing for Industrial Software Systems. ICSE 2011 SEiP, FSE 2012 (tool demo) Mithun Acharya, Xiao Qu, Brian Robinson. Cross-System Change Impact Analysis Using Test Cases. Under submission. Scaling beyond million lines Xiao Qu, Mithun Acharya, Brian Robinson. Configuration Selection Using Code Change Impact Analysis for Regression Testing. ICSM 2012 Xiao Qu, Mithun Acharya, Brian Robinson. Impact Analysis of Configuration Changes for Test Case Selection. ISSRE 2011 Testing configurable systems Tingting Yu, Xiao Qu, Mithun Achayra, Gregg Rothermel. Oracle-Based Regression Test Selection. Under submission. Regression test selection 11

15 Regression testing of CSS with code change impact analysis Automated Dependency Analysis Will changes to foo.c, affect Bob s module? Dependency analysis Automated Risk/Cost Analysis Automated What-If Analysis 3 days to release!!! Should I implement this feature or bug fix? What is the best way to fix this bug or implement that new feature? Overlay change impact with risky areas in code Automated Regression Testing Should I re-run ALL of my test suite for this change? New tests required? 3941 lines vs. 6 lines 12 Test suite

16 Outline Motivation Approach Implementation Empirical Evaluation Conclusions 16

17 Configurable Software Systems Software that can be customized through a set of options Example: Internet Explorer 17

18 Configurable Software Systems Software that can be customized through a set of options Example: Internet Explorer Configurable option: Pop-up Blocker 18 Values: ON, OFF

19 Internet Explorer Configurations C 1 C 2 Pop Up Blocker ON ON Google Toolbar Disabled Enabled Do Not Track Yes Yes 19

20 Internet Explorer Configurations C 1 C 2 Pop Up Blocker ON ON Option Google Toolbar Disabled Enabled Do Not Track Yes Yes 20

21 Internet Explorer Configurations Value C 1 C 2 Pop Up Blocker ON ON Option Google Toolbar Disabled Enabled Do Not Track Yes Yes 21

22 Internet Explorer Configurations Value C 1 C 2 Pop Up Blocker ON ON Option Google Toolbar Disabled Enabled Do Not Track Yes Yes 22 Configuration instance C 1 = {ON, Disabled, Yes, }

23 Internet Explorer Configurations Value C 1 C 2 Pop Up Blocker ON ON Option Google Toolbar Disabled Enabled Do Not Track Yes Yes 23 Configuration instance C 1 = {ON, Disabled, Yes, } Configuration C 1

24 Impact of Configurations on System Behavior Faulty System Behavior under Certain Configurations I have discovered that using the newly released IE 7 with Google Toolbar = Enabled can cause the right-click menu to lose the Open In New Tab option 24

25 Impact of Configuration on System Behavior To fix this, open IE7, click Tools > Manage Add-ons > Disable Google Toolbar 25

26 Test Case: Open IE7, Right Click a link on webpage C 1 C 2 Pop Up Blocker ON ON Google Toolbar Disabled Enabled Do Not Track Yes Yes 26

27 Test Case: Open IE7, Right Click a link on webpage Test: PASS C 1 C 2 Pop Up Blocker ON ON Google Toolbar Disabled Enabled Do Not Track Yes Yes 27

28 Test Case: Open IE7, Right Click a link on webpage Test: PASS Test: FAIL No Open in New Tab C 1 C 2 Pop Up Blocker ON ON Google Toolbar Disabled Enabled Do Not Track Yes Yes 28

29 Test Case: Open IE7, Right Click a link on webpage Test: PASS Test: FAIL No Open in New Tab C 1 C 2 Pop Up Blocker ON ON Google Toolbar Disabled Enabled Do Not Track Yes Yes 29 A test case that passes with one configuration may fail with another

30 Configurations control system execution IE7 IE7 30

31 Configurations control system execution IE7 IE7 C 1 = {ON, disabled, Yes, } 31

32 Configurations control system execution IE7 Test: PASS IE7 C 1 = {ON, disabled, Yes, } 32

33 Configurations control system execution IE7 Test: PASS IE7 C 1 = {ON, disabled, Yes, } C 2 = {ON, enabled, Yes, } 33

34 Configurations control system execution IE7 Test: PASS IE7 Test: FAIL C 1 = {ON, disabled, Yes, } C 2 = {ON, enabled, Yes, } 34

35 Configurations control system execution IE7 Test: PASS IE7 Test: FAIL C 1 = {ON, disabled, Yes, } C 2 = {ON, enabled, Yes, } Can we statically approximate how configurations (options) control system execution? 35

36 Challenges for testing configurable systems 36

37 Challenges for testing configurable systems n options 2 n configurations 37

38 Challenges for testing configurable systems n options 2 n configurations IE5 Product evolves IE7 38

39 Challenges for testing configurable systems n options 2 n configurations IE5 Product evolves IE7 T = {t 1, t 2, t 3, t 4, t 5 } T = {t 2, t 5 } Test case selection 39

40 Challenges for testing configurable systems n options 2 n configurations IE5 Product evolves IE7 T = {t 1, t 2, t 3, t 4, t 5 } T = {t 2, t 5 } Test case selection {C 1, C 2, C 3, C 4, C 5 } {C 1, C 5 } Configuration selection 40

41 Configuration Sampling Reducing the exponential number of configurations to a manageable size 41

42 Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 {C 1, C 2, C 3, C 4, C 5, } Exponentially large set T = {t 1, t 2, t 3, t 4, t 5 } 42

43 Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 IE7 {C 1, C 2, C 3, C 4, C 5, } Sampling {C 1, C 3, C 4 } Exponentially large set Manageable size set T = {t 1, t 2, t 3, t 4, t 5 } 43

44 Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 IE7 {C 1, C 2, C 3, C 4, C 5, } Sampling {C 1, C 3, C 4 } Exponentially large set Manageable size set T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } 44

45 Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 IE7 {C 1, C 2, C 3, C 4, C 5, } Sampling {C 1, C 3, C 4 } Exponentially large set Manageable size set T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } Example: Configuration Interaction Testing (CIT) 45

46 Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 IE7 {C 1, C 2, C 3, C 4, C 5, } Sampling {C 1, C 3, C 4 } Exponentially large set Manageable size set T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } Example: Configuration Interaction Testing (CIT) We choose to test IE7 only under sampled configurations C 1, C 3, and C 4 and for each configurations we test IE7 with all tests {t 1, t 2, t 3, t 4, t 5 } 46

47 vim: A configurable system 47

48 vim: A configurable system 2 90 configurations 48

49 vim: A configurable system 2 90 configurations sampling CIT selects 60 configurations 49

50 vim: A configurable system 2 90 configurations sampling CIT selects 60 configurations Rerun the full test suite on each 60 configurations 50

51 vim: A configurable system 2 90 configurations sampling CIT selects 60 configurations Rerun the full test suite on each 60 configurations 7 hours to execute the full test suite Takes 7*60 = 420 hours (~2.5 weeks) to run all test cases under each configuration 51

52 vim: A configurable system 2 90 configurations sampling CIT selects 60 configurations Rerun the full test suite on each 60 configurations 7 hours to execute the full test suite Takes 7*60 = 420 hours (~2.5 weeks) to run all test cases under each configuration Do we have to run all tests under each configuration? 52

53 Test case selection when configuration under test changes* 53 * Qu, Acharya, Robinson, Impact analysis of configuration changes for test case selection, ISSRE 2011

54 Test case selection when configuration under test changes* IE7 54 * Qu, Acharya, Robinson, Impact analysis of configuration changes for test case selection, ISSRE 2011

55 Test case selection when configuration under test changes* Source code DOES NOT change IE7 IE7 Configuration under test changes C 1 = {ON, Disabled, Yes, } C 2 = {ON, Enabled, Yes, } 55 * Qu, Acharya, Robinson, Impact analysis of configuration changes for test case selection, ISSRE 2011

56 Test case selection when configuration under test changes* Source code DOES NOT change IE7 IE7 Configuration under test changes C 1 = {ON, Disabled, Yes, } C 2 = {ON, Enabled, Yes, } What test cases should I re-run for the new configuration? T = {t 1, t 2, t 3, t 4, t 5 } 56 * Qu, Acharya, Robinson, Impact analysis of configuration changes for test case selection, ISSRE 2011

57 Test case selection when configuration under test changes* Source code DOES NOT change IE7 IE7 Configuration under test changes C 1 = {ON, Disabled, Yes, } C 2 = {ON, Enabled, Yes, } What test cases should I re-run for the new configuration? T = {t 1, t 2, t 3, t 4, t 5 } T = {t 2, t 5 } 57 * Qu, Acharya, Robinson, Impact analysis of configuration changes for test case selection, ISSRE 2011

58 Test case selection when configuration under test changes* Source code DOES NOT change IE7 IE7 Configuration under test changes C 1 = {ON, Disabled, Yes, } C 2 = {ON, Enabled, Yes, } What test cases should I re-run for the new configuration? T = {t 1, t 2, t 3, t 4, t 5 } T = {t 2, t 5 } For the ABB system analyzed, only about 20% of the tests had to be re-run for a configuration change 58 * Qu, Acharya, Robinson, Impact analysis of configuration changes for test case selection, ISSRE 2011

59 Product Evolution Internet Explorer IE1.0 & 2.0 IE7 IE6 IE5 59

60 Configuration prioritization for regression testing* *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA

61 Configuration prioritization for regression testing* IE5 {C 1, C 2, C 3, C 4, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } 61 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008

62 Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } 62 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008

63 Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Prioritization {C 1, C 4, C 3. } T = {t 1, t 2, t 3, t 4, t 5 } 63 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008

64 Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Prioritization {C 1, C 4, C 3. } Reorder and run as many as you can T = {t 1, t 2, t 3, t 4, t 5 } 64 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008

65 Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Prioritization {C 1, C 4, C 3. } Reorder and run as many as you can T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } 65 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008

66 Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Prioritization {C 1, C 4, C 3. } Reorder and run as many as you can T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } Increases rate of fault detection. But 66 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008

67 Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Prioritization {C 1, C 4, C 3. } Reorder and run as many as you can T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } Increases rate of fault detection. But Does not eliminate redundancy. Does not detect all faults. Not safe. 67 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008

68 Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Prioritization {C 1, C 4, C 3. } Reorder and run as many as you can T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } Increases rate of fault detection. But Does not eliminate redundancy. Does not detect all faults. Not safe. Can we select a subset of {C 1, C 2, C 3, C 4, C 5 } that is both non-redundant and safe? 68 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008

69 Configuration selection for regression testing (Focus of this talk) 69

70 Configuration selection for regression testing (Focus of this talk) IE5 {C 1, C 2, C 3, C 4, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } 70

71 Configuration selection for regression testing (Focus of this talk) Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } 71

72 Configuration selection for regression testing (Focus of this talk) Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } 72

73 Configuration selection for regression testing (Focus of this talk) Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } 73

74 Configuration selection for regression testing (Focus of this talk) Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } {C 1, C 5 } is both safe (wrt retest-all configurations) and non redundant 74

75 State of the Art in Configurable System Testing Focus of this talk 75

76 State of the Art in Configurable System Testing Configuration sampling Single version No test case selection Example, CIT Focus of this talk 76

77 State of the Art in Configurable System Testing Configuration sampling Single version No test case selection Example, CIT Focus of this talk 77 Test case selection [ISSRE 11] Single version Configuration under test changes Non-redundant Safe

78 State of the Art in Configurable System Testing Configuration sampling Single version No test case selection Example, CIT Focus of this talk 78 Test case selection [ISSRE 11] Single version Configuration under test changes Non-redundant Safe Configuration prioritization [ISSTA 08] Source code changes Regression Testing No test case selection Redundant Not safe

79 State of the Art in Configurable System Testing Configuration sampling Single version No test case selection Example, CIT Focus of this talk Configuration selection [ICSM 12] Source code changes Regression testing Non-redundant Safe No test case selection 79 Test case selection [ISSRE 11] Single version Configuration under test changes Non-redundant Safe Configuration prioritization [ISSTA 08] Source code changes Regression Testing No test case selection Redundant Not safe

80 Outline Motivation Approach Implementation Empirical Evaluation Conclusions 80

81 Key Idea: Map configuration options to code 81

82 Key Idea: Map configuration options to code Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE5 82

83 Key Idea: Map configuration options to code Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE5 IE7 change 83

84 Key Idea: Map configuration options to code Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE5 IE7 change 84

85 Key Idea: Map configuration options to code Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE5 IE7 change For ABB systems, configurable options (stored in a DB) maps to variables in the source code 85

86 Key Idea: statically compute configuration option impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 86

87 Key Idea: statically compute configuration option impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 87

88 Key Idea: statically compute configuration option impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 88

89 Key Idea: statically compute configuration option impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 89

90 Key Idea: statically compute impact of the changes Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 90

91 Key Idea: Intersect configuration impact with change impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 91

92 Key Idea: Intersect configuration impact with change impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 92 Select configuration option Google Toolbar for regression testing Safely discard pop-up blocker and Do Not Track

93 Example Program Mapping configurable options to source code Configurable Options: {P 1, P 2, P 3 } 93

94 Example Program Mapping configurable options to source code Configurable Options: {P 1, P 2, P 3 } 94

95 Example Program Mapping configurable options to source code Configurable Options: {P 1, P 2, P 3 } 95

96 Example Program Mapping configurable options to source code Configurable Options: {P 1, P 2, P 3 } 96

97 Example Program Mapping configurable options to source code Configurable Options: {P 1, P 2, P 3 } Function f 1 changes 97

98 Example 98

99 Example Configurable Options Options Values P 1 True False P 2 True False P 3 True False 99

100 Example Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 100

101 Example Simplified dependency graph Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 101

102 Impact of configuration option P 1 f 1, f 2, f 6, f 7, and f 8 Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 102

103 Impact of configuration option P 2 f 7 and f 8 Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 103

104 Impact of configuration option P 3 f 4 Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 104

105 Impact of changed function f 1 f 1, f 2, and f 6 Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 105

106 Select option P 1 and safely discard P 2 and P 3 Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 106

107 Select option P 1 and safely discard P 2 and P 3 Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 107

108 Outline Motivation Configuration Selection Approach Implementation Empirical Evaluation Conclusions 108

109 Research Questions How does our selection compare to retest-all, in terms of fault detection? What percentage of configurations is discarded as redundant by our selection? How much regression time can our selection save? Effectiveness Efficiency 109

110 Research Questions How does our selection compare to retest-all, in terms of fault detection? What percentage of configurations is discarded as redundant by our selection? How much regression time can our selection save? Effectiveness Efficiency 110

111 Subjects Make (Software Infrastructure Repository) V3.77 to v LOC: 15k LOC Code changes: selects 60 from 869 Seeded 15 faults Configurable options: 11 (binary) 7 configurations Grep V1.0 to V2.0 LOC: 8k LOC Code changes: 15 Seeded 15 faults Configurable options: 14 (binary) 7 configurations 111

112 Results Fault Detection Ability Make Retest-all 8/15 6/15 Our selection 8/15 6/15 Random selection 3/15 5/15 Grep 112

113 Results Fault Detection Ability Make Retest-all 8/15 6/15 Our selection 8/15 6/15 Random selection 3/15 5/15 Grep Our approach is safe wrt retest-all configurations 113

114 Research Questions How does our selection compare to retest-all, in terms of fault detection? What percentage of configurations is discarded as redundant by our selection? How much regression time can our selection save? Effectiveness Efficiency 114

115 Subject ABB1 LOC: 1.18 MLOC Number of Functions: 20,432 functions Code changes: 203 Configurable options: 545 (number of values range from 2 to 9) 159 configurations Among the 203 changes, we selected three sets of 30 changes for analysis 115

116 Results Percentage of configurations selected NUMBER OF CONFIGURABLE OPTIONS SELECTED Change set 1 Change set 2 Change set 3 Average Retest-all 545 Selected reduction 69% 70% 70% 70% NUMBER OF CONFIGURATIONS SELECTED Change set 1 Change set 2 Change set 3 Average Retest-all 159 Selected reduction 25% 25% 25% 25% 116

117 Research Questions How does our selection compare to retest-all, in terms of fault detection? What percentage of configurations is discarded as redundant by our selection? How much regression time can our selection save? Effectiveness Efficiency 117

118 Results Testing time savings Testing time Overhead of selection Time savings grep make ABB1 Retest-all 70m 700m 795h Our approach 60m 300m 600h 5.2m 13m 28h 5m 387m 167h 50% 55% 21% 118

119 Results Testing time savings Testing time Overhead of selection Time savings grep make ABB1 Retest-all 70m 700m 795h Our approach 60m 300m 600h 5.2m 13m 28h 5m 387m 167h 50% 55% 21% Our configuration selection approach saves about 20-55% of testing time wrt retest-all configurations 119

120 Better than random, safe wrt retest-all How does our selection compare to retest-all and random in terms of fault detection? safe What percentage of configurations is discarded as redundant by our selection? 15 60% How much regression time can our selection save? 20 55% Effectiveness Efficiency 120

121 Outline Motivation Configuration Selection Approach Implementation Empirical Evaluation Conclusions 121

122 First configuration selection approach for regression testing 122

123 First configuration selection approach for regression testing Source code CHANGES IE5 IE7 123

124 First configuration selection approach for regression testing Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } 124

125 First configuration selection approach for regression testing Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } 125

126 First configuration selection approach for regression testing Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } {C 1, C 5 } is both safe (wrt retest-all configurations) and non redundant 126

127 First configuration selection approach for regression testing Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } {C 1, C 5 } is both safe (wrt retest-all configurations) and non redundant 127 In our experiments, 15-60% of configurations were discarded as redundant saving 20-55% of regression testing time

128 128