Code Change Impact Analysis for Testing Configurable Software Systems Mithun Acharya ABB Corporate Research Raleigh NC USA
ABB: A power and automation company >125 years, >100 nations, ~150,000 employees Power products and electronics, Control Systems, Robotics, Smart Grid, Renewable Energy, 2
ABB Corporate Research Industrial Software Systems (ISS) research group USA Germany Switzerland Poland Sweden India China Raleigh, NC 7 research centers worldwide 3
Software in ABB Hardware with software inside Software with few hardware components Pure Software 4
Software Evolution: A CSS constantly changes 5 Hundreds of such changes committed daily
Change Control Board meetings Change impact visualizations for managers for decision making Change impact at the code level for developers Unit/module-level change impact for testers 6
Imp: Code change impact analysis for C/C++ programs IMP BUILD SERVER Version control SERVER CHECKOUT NIGHTLY BUILD Impact of C23567 Impact of C23586 Impact of C23712 Impact of changes since last nightly build C23586 Impact database 7
Quantifiable risk/cost analysis of changes to CSS Automated Dependency Analysis Will changes to foo.c, affect Bob s module? Dependency analysis Automated Risk/Cost Analysis Automated What-If Analysis 3 days to release!!! Should I implement this feature or bug fix? What is the best way to fix this bug or implement that new feature? Overlay change impact with risky areas in code Automated Regression Testing Should I re-run ALL of my test suite for this change? New tests required? 3941 lines vs. 6 lines 8 Test suite
Program and System Dependence Graphs for Slicing Program Dependence Graph (PDG) for main void main() { int i = 1; int sum = 0; while (i<11) { sum = add(sum, i); i = add(i, 1); } printf("sum = %d\n", sum); printf("i = %d\n", i); } static int add(int a, int b) { return(a+b); } System Dependence Graph (SDG) PDG for add 9 Code/Image Source: GrammaTech
Making impact analysis practical and useful 10
Making impact analysis practical and useful Mithun Acharya, Brian Robinson. Practical Change Impact Analysis based on Static Program Slicing for Industrial Software Systems. ICSE 2011 SEiP, FSE 2012 (tool demo) Mithun Acharya, Xiao Qu, Brian Robinson. Cross-System Change Impact Analysis Using Test Cases. Under submission. Scaling beyond million lines 10
Making impact analysis practical and useful Mithun Acharya, Brian Robinson. Practical Change Impact Analysis based on Static Program Slicing for Industrial Software Systems. ICSE 2011 SEiP, FSE 2012 (tool demo) Mithun Acharya, Xiao Qu, Brian Robinson. Cross-System Change Impact Analysis Using Test Cases. Under submission. Scaling beyond million lines Xiao Qu, Mithun Acharya, Brian Robinson. Configuration Selection Using Code Change Impact Analysis for Regression Testing. ICSM 2012 Xiao Qu, Mithun Acharya, Brian Robinson. Impact Analysis of Configuration Changes for Test Case Selection. ISSRE 2011 Testing configurable systems 10
Making impact analysis practical and useful Mithun Acharya, Brian Robinson. Practical Change Impact Analysis based on Static Program Slicing for Industrial Software Systems. ICSE 2011 SEiP, FSE 2012 (tool demo) Mithun Acharya, Xiao Qu, Brian Robinson. Cross-System Change Impact Analysis Using Test Cases. Under submission. Scaling beyond million lines Xiao Qu, Mithun Acharya, Brian Robinson. Configuration Selection Using Code Change Impact Analysis for Regression Testing. ICSM 2012 Xiao Qu, Mithun Acharya, Brian Robinson. Impact Analysis of Configuration Changes for Test Case Selection. ISSRE 2011 Testing configurable systems Tingting Yu, Xiao Qu, Mithun Achayra, Gregg Rothermel. Oracle-Based Regression Test Selection. Under submission. Regression test selection 10
What configurations should we select for retesting? Mithun Acharya, Brian Robinson. Practical Change Impact Analysis based on Static Program Slicing for Industrial Software Systems. ICSE 2011 SEiP, FSE 2012 (tool demo) Mithun Acharya, Xiao Qu, Brian Robinson. Cross-System Change Impact Analysis Using Test Cases. Under submission. Scaling beyond million lines Xiao Qu, Mithun Acharya, Brian Robinson. Configuration Selection Using Code Change Impact Analysis for Regression Testing. ICSM 2012 Xiao Qu, Mithun Acharya, Brian Robinson. Impact Analysis of Configuration Changes for Test Case Selection. ISSRE 2011 Testing configurable systems Tingting Yu, Xiao Qu, Mithun Achayra, Gregg Rothermel. Oracle-Based Regression Test Selection. Under submission. Regression test selection 11
Regression testing of CSS with code change impact analysis Automated Dependency Analysis Will changes to foo.c, affect Bob s module? Dependency analysis Automated Risk/Cost Analysis Automated What-If Analysis 3 days to release!!! Should I implement this feature or bug fix? What is the best way to fix this bug or implement that new feature? Overlay change impact with risky areas in code Automated Regression Testing Should I re-run ALL of my test suite for this change? New tests required? 3941 lines vs. 6 lines 12 Test suite
Outline Motivation Approach Implementation Empirical Evaluation Conclusions 16
Configurable Software Systems Software that can be customized through a set of options Example: Internet Explorer 17
Configurable Software Systems Software that can be customized through a set of options Example: Internet Explorer Configurable option: Pop-up Blocker 18 Values: ON, OFF
Internet Explorer Configurations C 1 C 2 Pop Up Blocker ON ON Google Toolbar Disabled Enabled Do Not Track Yes Yes 19
Internet Explorer Configurations C 1 C 2 Pop Up Blocker ON ON Option Google Toolbar Disabled Enabled Do Not Track Yes Yes 20
Internet Explorer Configurations Value C 1 C 2 Pop Up Blocker ON ON Option Google Toolbar Disabled Enabled Do Not Track Yes Yes 21
Internet Explorer Configurations Value C 1 C 2 Pop Up Blocker ON ON Option Google Toolbar Disabled Enabled Do Not Track Yes Yes 22 Configuration instance C 1 = {ON, Disabled, Yes, }
Internet Explorer Configurations Value C 1 C 2 Pop Up Blocker ON ON Option Google Toolbar Disabled Enabled Do Not Track Yes Yes 23 Configuration instance C 1 = {ON, Disabled, Yes, } Configuration C 1
Impact of Configurations on System Behavior Faulty System Behavior under Certain Configurations I have discovered that using the newly released IE 7 with Google Toolbar = Enabled can cause the right-click menu to lose the Open In New Tab option 24
Impact of Configuration on System Behavior To fix this, open IE7, click Tools > Manage Add-ons > Disable Google Toolbar 25
Test Case: Open IE7, Right Click a link on webpage C 1 C 2 Pop Up Blocker ON ON Google Toolbar Disabled Enabled Do Not Track Yes Yes 26
Test Case: Open IE7, Right Click a link on webpage Test: PASS C 1 C 2 Pop Up Blocker ON ON Google Toolbar Disabled Enabled Do Not Track Yes Yes 27
Test Case: Open IE7, Right Click a link on webpage Test: PASS Test: FAIL No Open in New Tab C 1 C 2 Pop Up Blocker ON ON Google Toolbar Disabled Enabled Do Not Track Yes Yes 28
Test Case: Open IE7, Right Click a link on webpage Test: PASS Test: FAIL No Open in New Tab C 1 C 2 Pop Up Blocker ON ON Google Toolbar Disabled Enabled Do Not Track Yes Yes 29 A test case that passes with one configuration may fail with another
Configurations control system execution IE7 IE7 30
Configurations control system execution IE7 IE7 C 1 = {ON, disabled, Yes, } 31
Configurations control system execution IE7 Test: PASS IE7 C 1 = {ON, disabled, Yes, } 32
Configurations control system execution IE7 Test: PASS IE7 C 1 = {ON, disabled, Yes, } C 2 = {ON, enabled, Yes, } 33
Configurations control system execution IE7 Test: PASS IE7 Test: FAIL C 1 = {ON, disabled, Yes, } C 2 = {ON, enabled, Yes, } 34
Configurations control system execution IE7 Test: PASS IE7 Test: FAIL C 1 = {ON, disabled, Yes, } C 2 = {ON, enabled, Yes, } Can we statically approximate how configurations (options) control system execution? 35
Challenges for testing configurable systems 36
Challenges for testing configurable systems n options 2 n configurations 37
Challenges for testing configurable systems n options 2 n configurations IE5 Product evolves IE7 38
Challenges for testing configurable systems n options 2 n configurations IE5 Product evolves IE7 T = {t 1, t 2, t 3, t 4, t 5 } T = {t 2, t 5 } Test case selection 39
Challenges for testing configurable systems n options 2 n configurations IE5 Product evolves IE7 T = {t 1, t 2, t 3, t 4, t 5 } T = {t 2, t 5 } Test case selection {C 1, C 2, C 3, C 4, C 5 } {C 1, C 5 } Configuration selection 40
Configuration Sampling Reducing the exponential number of configurations to a manageable size 41
Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 {C 1, C 2, C 3, C 4, C 5, } Exponentially large set T = {t 1, t 2, t 3, t 4, t 5 } 42
Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 IE7 {C 1, C 2, C 3, C 4, C 5, } Sampling {C 1, C 3, C 4 } Exponentially large set Manageable size set T = {t 1, t 2, t 3, t 4, t 5 } 43
Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 IE7 {C 1, C 2, C 3, C 4, C 5, } Sampling {C 1, C 3, C 4 } Exponentially large set Manageable size set T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } 44
Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 IE7 {C 1, C 2, C 3, C 4, C 5, } Sampling {C 1, C 3, C 4 } Exponentially large set Manageable size set T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } Example: Configuration Interaction Testing (CIT) 45
Configuration Sampling Reducing the exponential number of configurations to a manageable size IE7 IE7 {C 1, C 2, C 3, C 4, C 5, } Sampling {C 1, C 3, C 4 } Exponentially large set Manageable size set T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } Example: Configuration Interaction Testing (CIT) We choose to test IE7 only under sampled configurations C 1, C 3, and C 4 and for each configurations we test IE7 with all tests {t 1, t 2, t 3, t 4, t 5 } 46
vim: A configurable system 47
vim: A configurable system 2 90 configurations 48
vim: A configurable system 2 90 configurations sampling CIT selects 60 configurations 49
vim: A configurable system 2 90 configurations sampling CIT selects 60 configurations Rerun the full test suite on each 60 configurations 50
vim: A configurable system 2 90 configurations sampling CIT selects 60 configurations Rerun the full test suite on each 60 configurations 7 hours to execute the full test suite Takes 7*60 = 420 hours (~2.5 weeks) to run all test cases under each configuration 51
vim: A configurable system 2 90 configurations sampling CIT selects 60 configurations Rerun the full test suite on each 60 configurations 7 hours to execute the full test suite Takes 7*60 = 420 hours (~2.5 weeks) to run all test cases under each configuration Do we have to run all tests under each configuration? 52
Test case selection when configuration under test changes* 53 * Qu, Acharya, Robinson, Impact analysis of configuration changes for test case selection, ISSRE 2011
Test case selection when configuration under test changes* IE7 54 * Qu, Acharya, Robinson, Impact analysis of configuration changes for test case selection, ISSRE 2011
Test case selection when configuration under test changes* Source code DOES NOT change IE7 IE7 Configuration under test changes C 1 = {ON, Disabled, Yes, } C 2 = {ON, Enabled, Yes, } 55 * Qu, Acharya, Robinson, Impact analysis of configuration changes for test case selection, ISSRE 2011
Test case selection when configuration under test changes* Source code DOES NOT change IE7 IE7 Configuration under test changes C 1 = {ON, Disabled, Yes, } C 2 = {ON, Enabled, Yes, } What test cases should I re-run for the new configuration? T = {t 1, t 2, t 3, t 4, t 5 } 56 * Qu, Acharya, Robinson, Impact analysis of configuration changes for test case selection, ISSRE 2011
Test case selection when configuration under test changes* Source code DOES NOT change IE7 IE7 Configuration under test changes C 1 = {ON, Disabled, Yes, } C 2 = {ON, Enabled, Yes, } What test cases should I re-run for the new configuration? T = {t 1, t 2, t 3, t 4, t 5 } T = {t 2, t 5 } 57 * Qu, Acharya, Robinson, Impact analysis of configuration changes for test case selection, ISSRE 2011
Test case selection when configuration under test changes* Source code DOES NOT change IE7 IE7 Configuration under test changes C 1 = {ON, Disabled, Yes, } C 2 = {ON, Enabled, Yes, } What test cases should I re-run for the new configuration? T = {t 1, t 2, t 3, t 4, t 5 } T = {t 2, t 5 } For the ABB system analyzed, only about 20% of the tests had to be re-run for a configuration change 58 * Qu, Acharya, Robinson, Impact analysis of configuration changes for test case selection, ISSRE 2011
Product Evolution Internet Explorer IE1.0 & 2.0 IE7 IE6 IE5 59
Configuration prioritization for regression testing* *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008 60
Configuration prioritization for regression testing* IE5 {C 1, C 2, C 3, C 4, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } 61 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } 62 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Prioritization {C 1, C 4, C 3. } T = {t 1, t 2, t 3, t 4, t 5 } 63 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Prioritization {C 1, C 4, C 3. } Reorder and run as many as you can T = {t 1, t 2, t 3, t 4, t 5 } 64 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Prioritization {C 1, C 4, C 3. } Reorder and run as many as you can T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } 65 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Prioritization {C 1, C 4, C 3. } Reorder and run as many as you can T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } Increases rate of fault detection. But 66 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Prioritization {C 1, C 4, C 3. } Reorder and run as many as you can T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } Increases rate of fault detection. But Does not eliminate redundancy. Does not detect all faults. Not safe. 67 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008
Configuration prioritization for regression testing* Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Prioritization {C 1, C 4, C 3. } Reorder and run as many as you can T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } Increases rate of fault detection. But Does not eliminate redundancy. Does not detect all faults. Not safe. Can we select a subset of {C 1, C 2, C 3, C 4, C 5 } that is both non-redundant and safe? 68 *Qu, Cohen, Rothermel, Configuration-aware regression testing: An empirical study of sampling and prioritization, ISSTA 2008
Configuration selection for regression testing (Focus of this talk) 69
Configuration selection for regression testing (Focus of this talk) IE5 {C 1, C 2, C 3, C 4, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } 70
Configuration selection for regression testing (Focus of this talk) Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } 71
Configuration selection for regression testing (Focus of this talk) Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } 72
Configuration selection for regression testing (Focus of this talk) Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } 73
Configuration selection for regression testing (Focus of this talk) Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } {C 1, C 5 } is both safe (wrt retest-all configurations) and non redundant 74
State of the Art in Configurable System Testing Focus of this talk 75
State of the Art in Configurable System Testing Configuration sampling Single version No test case selection Example, CIT Focus of this talk 76
State of the Art in Configurable System Testing Configuration sampling Single version No test case selection Example, CIT Focus of this talk 77 Test case selection [ISSRE 11] Single version Configuration under test changes Non-redundant Safe
State of the Art in Configurable System Testing Configuration sampling Single version No test case selection Example, CIT Focus of this talk 78 Test case selection [ISSRE 11] Single version Configuration under test changes Non-redundant Safe Configuration prioritization [ISSTA 08] Source code changes Regression Testing No test case selection Redundant Not safe
State of the Art in Configurable System Testing Configuration sampling Single version No test case selection Example, CIT Focus of this talk Configuration selection [ICSM 12] Source code changes Regression testing Non-redundant Safe No test case selection 79 Test case selection [ISSRE 11] Single version Configuration under test changes Non-redundant Safe Configuration prioritization [ISSTA 08] Source code changes Regression Testing No test case selection Redundant Not safe
Outline Motivation Approach Implementation Empirical Evaluation Conclusions 80
Key Idea: Map configuration options to code 81
Key Idea: Map configuration options to code Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE5 82
Key Idea: Map configuration options to code Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE5 IE7 change 83
Key Idea: Map configuration options to code Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE5 IE7 change 84
Key Idea: Map configuration options to code Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE5 IE7 change For ABB systems, configurable options (stored in a DB) maps to variables in the source code 85
Key Idea: statically compute configuration option impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 86
Key Idea: statically compute configuration option impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 87
Key Idea: statically compute configuration option impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 88
Key Idea: statically compute configuration option impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 89
Key Idea: statically compute impact of the changes Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 90
Key Idea: Intersect configuration impact with change impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 91
Key Idea: Intersect configuration impact with change impact Configuration options: {pop-up-blocker, Google Toolbar, Do Not Track} IE7 92 Select configuration option Google Toolbar for regression testing Safely discard pop-up blocker and Do Not Track
Example Program Mapping configurable options to source code Configurable Options: {P 1, P 2, P 3 } 93
Example Program Mapping configurable options to source code Configurable Options: {P 1, P 2, P 3 } 94
Example Program Mapping configurable options to source code Configurable Options: {P 1, P 2, P 3 } 95
Example Program Mapping configurable options to source code Configurable Options: {P 1, P 2, P 3 } 96
Example Program Mapping configurable options to source code Configurable Options: {P 1, P 2, P 3 } Function f 1 changes 97
Example 98
Example Configurable Options Options Values P 1 True False P 2 True False P 3 True False 99
Example Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 100
Example Simplified dependency graph Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 101
Impact of configuration option P 1 f 1, f 2, f 6, f 7, and f 8 Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 102
Impact of configuration option P 2 f 7 and f 8 Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 103
Impact of configuration option P 3 f 4 Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 104
Impact of changed function f 1 f 1, f 2, and f 6 Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 105
Select option P 1 and safely discard P 2 and P 3 Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 106
Select option P 1 and safely discard P 2 and P 3 Configurable Options Options Values P 1 True False P 2 True False P 3 True False Configurations by pair-wise CIT P 1 P 2 P 3 C 1 True True True C 2 True False False C 3 False True False C 4 False False True 107
Outline Motivation Configuration Selection Approach Implementation Empirical Evaluation Conclusions 108
Research Questions 1 2 3 How does our selection compare to retest-all, in terms of fault detection? What percentage of configurations is discarded as redundant by our selection? How much regression time can our selection save? Effectiveness Efficiency 109
Research Questions 1 2 3 How does our selection compare to retest-all, in terms of fault detection? What percentage of configurations is discarded as redundant by our selection? How much regression time can our selection save? Effectiveness Efficiency 110
Subjects Make (Software Infrastructure Repository) V3.77 to v3.78.1 LOC: 15k LOC Code changes: selects 60 from 869 Seeded 15 faults Configurable options: 11 (binary) 7 configurations Grep V1.0 to V2.0 LOC: 8k LOC Code changes: 15 Seeded 15 faults Configurable options: 14 (binary) 7 configurations 111
Results Fault Detection Ability Make Retest-all 8/15 6/15 Our selection 8/15 6/15 Random selection 3/15 5/15 Grep 112
Results Fault Detection Ability Make Retest-all 8/15 6/15 Our selection 8/15 6/15 Random selection 3/15 5/15 Grep Our approach is safe wrt retest-all configurations 113
Research Questions 1 2 3 How does our selection compare to retest-all, in terms of fault detection? What percentage of configurations is discarded as redundant by our selection? How much regression time can our selection save? Effectiveness Efficiency 114
Subject ABB1 LOC: 1.18 MLOC Number of Functions: 20,432 functions Code changes: 203 Configurable options: 545 (number of values range from 2 to 9) 159 configurations Among the 203 changes, we selected three sets of 30 changes for analysis 115
Results Percentage of configurations selected NUMBER OF CONFIGURABLE OPTIONS SELECTED Change set 1 Change set 2 Change set 3 Average Retest-all 545 Selected 167 161 161 163 reduction 69% 70% 70% 70% NUMBER OF CONFIGURATIONS SELECTED Change set 1 Change set 2 Change set 3 Average Retest-all 159 Selected 120 120 120 120 reduction 25% 25% 25% 25% 116
Research Questions 1 2 3 How does our selection compare to retest-all, in terms of fault detection? What percentage of configurations is discarded as redundant by our selection? How much regression time can our selection save? Effectiveness Efficiency 117
Results Testing time savings Testing time Overhead of selection Time savings grep make ABB1 Retest-all 70m 700m 795h Our approach 60m 300m 600h 5.2m 13m 28h 5m 387m 167h 50% 55% 21% 118
Results Testing time savings Testing time Overhead of selection Time savings grep make ABB1 Retest-all 70m 700m 795h Our approach 60m 300m 600h 5.2m 13m 28h 5m 387m 167h 50% 55% 21% Our configuration selection approach saves about 20-55% of testing time wrt retest-all configurations 119
Better than random, safe wrt retest-all 1 2 3 How does our selection compare to retest-all and random in terms of fault detection? safe What percentage of configurations is discarded as redundant by our selection? 15 60% How much regression time can our selection save? 20 55% Effectiveness Efficiency 120
Outline Motivation Configuration Selection Approach Implementation Empirical Evaluation Conclusions 121
First configuration selection approach for regression testing 122
First configuration selection approach for regression testing Source code CHANGES IE5 IE7 123
First configuration selection approach for regression testing Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } 124
First configuration selection approach for regression testing Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } 125
First configuration selection approach for regression testing Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } {C 1, C 5 } is both safe (wrt retest-all configurations) and non redundant 126
First configuration selection approach for regression testing Source code CHANGES IE5 IE7 {C 1, C 2, C 3, C 4, C 5 } Selection {C 1, C 5 } T = {t 1, t 2, t 3, t 4, t 5 } No test case selection T = {t 1, t 2, t 3, t 4, t 5 } {C 1, C 5 } is both safe (wrt retest-all configurations) and non redundant 127 In our experiments, 15-60% of configurations were discarded as redundant saving 20-55% of regression testing time
128