Model-based Test Case Generation for (Dynamic) Software Product Lines The 6th Meeting on Feature-oriented Software Development 2014, May 04-07 Schloss Dagstuhl, Germany DFG Priority Programme 1593 Design for Future Managed Software Evolution Johannes Bürdek johannes.buerdek@es.tu-darmstadt.de Tel.+49 6151 16 76089 ES Real-Time Systems Lab Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of Computer Science (adjunct Professor) www.es.tu-darmstadt.de author(s) of these slides 2008 including research results of the research network ES and TU Darmstadt otherwise as specified at the respective slide 08.05.2014
Test Suites Test Models Products Testing Software Product Lines P1 = {f1} P2 = {f2} P3 = {f1,f3} S5 2 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines [Cichos et al. 2011] ES Real-Time Systems Lab
Test Suites Test Models Products Testing Software Product Lines P1 = {f1} P2 = {f2} P3 = {f1,f3} S5 Generate test cases for every product individually is inefficient! 3 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines [Cichos et al. 2011] ES Real-Time Systems Lab
Test Suites 150% Test Model Test Models Products Testing Software Product Lines P1 = {f1} P2 = {f2} P3 = {f1,f3} S5 f2 f1 & f3 S5 f1 4 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines [Cichos et al. 2011] ES Real-Time Systems Lab
Test Suites 150% Test Model Test Models Products Testing Software Product Lines P1 = {f1} P2 = {f2} P3 = {f1,f3} S5 f1 & f3 S5 f2 f1 ISO 9001, ISO 26262, ISO 13485 require certain model coverage for test case generation 5 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines [Cichos et al. 2011] ES Real-Time Systems Lab
SPL Test Suite Generation Feature Model r 150% Test Model f1 f2 exclude f3 f1 & f3 S5 f2 f1 SPL Test Suite (State Coverage) Config. / TC -- -- ---S5 (r, f1) X (r, f2) X (r, f1, f3) X X [Cichos et al. 2011], [Cichos et al. 2012], [Lochau et al. 2014] 6 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab
Using Symbolic Model Checking for Test Case Generation f1 f2 S5 S6 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); Generate C-Code from Model and enrich it with more C-Code 7 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab
Using Symbolic Model Checking for Test Case Generation f1 f2 S5 S6 f1 root f2 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); Impl. Variability Model-Checker CPAchecker Achieve state coverage for test model 8 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab
Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 Assignment Assumption 9 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines 1 2 3 11 4 5 6 9 Path Condition Path Condition Path Condition [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 7 Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2
Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 10 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines 1 2 3 11 assign(x = y + 10) assume(x 15) assume(x < 15) assume(f1) 4 5 6 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2
Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 SMT Solver: y = 5 x = 15 f1 = f2 = 11 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines 1 2 3 11 true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) 4 5 6 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2
Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 SMT Solver: y = 5 x = 15 f1 = f2 = 12 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines 1 2 3 11 true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) 4 5 6 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2
Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 13 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines 1 2 3 11 true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) 4 5 6 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2
Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 14 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines 1 2 3 11 true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) 4 SMT Solver: y = 1 5 6 x = 11 f1 = false f2 = true 7 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2
Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 15 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines 1 2 3 11 true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) (f1 f2) 4 5 6 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2
Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 16 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines 1 2 SMT Solver: y = 1 3 x = 11 f1 = true f2 = false 11 true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) (f1 f2) 4 5 6 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2
Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 A BDD analysis checks 11 if a location ((f1...) may be reached by a path with another feature configuration. 17 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines 1 2 3 true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 assume(f1) (f1 f2) 4 5 6 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) S5 may assume(f2) also be covered in the 7 variant (r,f1) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2
Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 (( f1 f2) (f1 f2)) 18 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines 1 2 3 11 true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) (f1 f2) 4 5 6 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2
Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 (( f1 f2) (f1 f2)) 19 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines 1 2 3 11 true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) (f1 f2) 4 5 6 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2
Testing Dynamic Software Product Lines - Testing SPLs: test that every derivable variant is correctly implemented - Testing DSPLs: test that all possible reconfigurations are correct Environment Program Reconfiguration 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); C1 C3 C2 Non-deterministic behavior Non-deterministic influence on program variables Non-deterministic influence on feature variables 20 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab
Testing Dynamic Software Product Lines - Testing SPLs: test that every derivable variant is correctly implemented - Testing DSPLs: test that all possible reconfigurations are correct Environment Program Reconfiguration 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); C1 C3 C2 Non-deterministic behavior - CPAchecker offers techniques to handle non-deterministic behavior 21 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab
Testing Dynamic Software Product Lines - Testing SPLs: test that every derivable variant is correctly implemented - Testing DSPLs: test that all possible reconfigurations are correct Environment Program 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); Reconfiguration Non-deterministic behavior - CPAchecker offers techniques to handle non-deterministic behavior C1 C3 C2 1. How to test reconfigurations efficiently? 2. What is a suitable coverage criteria? 3. How to ensure liveness of the reconfiguration life cycle? 22 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab
List of Literature [Cichos et al. 2011] H. Cichos, S. Oster, M. Lochau, A. Schürr : Model-Based Coverage- Driven Test Suite Generation for Software Product Lines. MoDELS 2011: 425-439 [Cichos et al. 2012] H. Cichos, M. Lochau, S. Oster, A. Schürr : Reduktion von Testsuiten für Software-Produktlinien. Software Engineering 2012: 143-154 [Lochau et al. 2014] M. Lochau, J. Bürdek, S. Lity, M. Hagner, C. Legat, U. Goltz, A. Schürr : Applying Model-based Software Product Line Testing Approaches to the Automation Engineering Domain. AT Journal 2014 (submitted) [Holzer et al. 2011] A. Holzer, M. Tautschnig, C. Schallhart, H. Veith : An Introduction to Test Specification in FQL. In S. Barner, I. Harris, D. Kroening, and O. Raz, editors, Hardware and Software: Verification and Testing, volume 6504 of Lecture Notes in Computer Science, pages 9 22. Springer Berlin Heidelberg, 2011. [Beyer et al. 2011] D. Beyer and M. E. Keremoglu : CPAchecker: A Tool for Configurable Software Verification. In Proc. CAV, pages 184-190, Springer, 2011. [Holzmann 1997] G. J. Holzmann : The Model Checker Spin, IEEE Trans. on Software Engineering, Vol. 23, No. 5, May 1997, pp. 279-295. 23 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab
List of Literature (2/2) [Beyer et al. 2004] D. Beyer, A. J. Chlipala, T. A. Henzinger, R. Jhala, and R. Majumdar. Generating Test from Counterexamples. In Proc. of the 26th ICSE, pages 326 335, 2004. [Rhein et al. 2011] A. von Rhein, S. Apel, F. Raimondi : Introducing Binary Decision Diagrams in the Explicit-State Verification of Java Code. Java Pathfinder Workshop (co-located with ASE'11), 2011. [Apel et al. 2013] S. Apel, A. von Rhein, P. Wendler, A. Größlinger, D. Beyer : Strategies for Product-Line Verification: Case Studies and Experiments. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pages 482 491. IEEE Computer Society, 2013. 24 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab