Model-based Test Case Generation for (Dynamic) Software Product Lines

Transcription

1 Model-based Test Case Generation for (Dynamic) Software Product Lines The 6th Meeting on Feature-oriented Software Development 2014, May Schloss Dagstuhl, Germany DFG Priority Programme 1593 Design for Future Managed Software Evolution Johannes Bürdek Tel ES Real-Time Systems Lab Prof. Dr. rer. nat. Andy Schürr Dept. of Electrical Engineering and Information Technology Dept. of Computer Science (adjunct Professor) author(s) of these slides 2008 including research results of the research network ES and TU Darmstadt otherwise as specified at the respective slide

2 Test Suites Test Models Products Testing Software Product Lines P1 = {f1} P2 = {f2} P3 = {f1,f3} S5 2 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines [Cichos et al. 2011] ES Real-Time Systems Lab

3 Test Suites Test Models Products Testing Software Product Lines P1 = {f1} P2 = {f2} P3 = {f1,f3} S5 Generate test cases for every product individually is inefficient! 3 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines [Cichos et al. 2011] ES Real-Time Systems Lab

4 Test Suites 150% Test Model Test Models Products Testing Software Product Lines P1 = {f1} P2 = {f2} P3 = {f1,f3} S5 f2 f1 & f3 S5 f1 4 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines [Cichos et al. 2011] ES Real-Time Systems Lab

5 Test Suites 150% Test Model Test Models Products Testing Software Product Lines P1 = {f1} P2 = {f2} P3 = {f1,f3} S5 f1 & f3 S5 f2 f1 ISO 9001, ISO 26262, ISO require certain model coverage for test case generation 5 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines [Cichos et al. 2011] ES Real-Time Systems Lab

6 SPL Test Suite Generation Feature Model r 150% Test Model f1 f2 exclude f3 f1 & f3 S5 f2 f1 SPL Test Suite (State Coverage) Config. / TC S5 (r, f1) X (r, f2) X (r, f1, f3) X X [Cichos et al. 2011], [Cichos et al. 2012], [Lochau et al. 2014] 6 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab

7 Using Symbolic Model Checking for Test Case Generation f1 f2 S5 S6 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); Generate C-Code from Model and enrich it with more C-Code 7 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab

8 Using Symbolic Model Checking for Test Case Generation f1 f2 S5 S6 f1 root f2 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); Impl. Variability Model-Checker CPAchecker Achieve state coverage for test model 8 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab

9 Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 Assignment Assumption 9 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines Path Condition Path Condition Path Condition [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 7 Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2

10 Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 10 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines assign(x = y + 10) assume(x 15) assume(x < 15) assume(f1) [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2

11 Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 SMT Solver: y = 5 x = 15 f1 = f2 = 11 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2

12 Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 SMT Solver: y = 5 x = 15 f1 = f2 = 12 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2

13 Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 13 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2

14 Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 14 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) 4 SMT Solver: y = x = 11 f1 = false f2 = true 7 [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2

15 Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 15 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) (f1 f2) [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2

16 Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 16 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines 1 2 SMT Solver: y = 1 3 x = 11 f1 = true f2 = false 11 true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) (f1 f2) [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2

17 Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 A BDD analysis checks 11 if a location ((f1...) may be reached by a path with another feature configuration. 17 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 assume(f1) (f1 f2) [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) S5 may assume(f2) also be covered in the 7 variant (r,f1) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2

18 Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 (( f1 f2) (f1 f2)) 18 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) (f1 f2) [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2

19 Test Case Derivation from Counterexamples 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); (r,f1) (r,f2) TC1 TC1 TC1 TC1 TC3 TC2 S5 TC3 TC2 S6 TC1 TC1 (( f1 f2) (f1 f2)) 19 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines true ((f1 f2) ( f1 f2)) assign(x = y + 10) assume(x 15) assume(x < 15) x = y + 10 x 15 ((f1...) assume(f1) (f1 f2) [Beyer et al. 2004], [Rhein et al. 2011], [Apel et al. 2013] 9 (x = y + 10) (x < 15) ((f1...) assume( f1) 7 assume(f2) Test Cases TC1 y=5, x=15 TC2 y=11, x=1 f1 f2 TC3 ES Real-Time y=11, x=1 Systems f1 Lab f2

20 Testing Dynamic Software Product Lines - Testing SPLs: test that every derivable variant is correctly implemented - Testing DSPLs: test that all possible reconfigurations are correct Environment Program Reconfiguration 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); C1 C3 C2 Non-deterministic behavior Non-deterministic influence on program variables Non-deterministic influence on feature variables 20 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab

21 Testing Dynamic Software Product Lines - Testing SPLs: test that every derivable variant is correctly implemented - Testing DSPLs: test that all possible reconfigurations are correct Environment Program Reconfiguration 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); C1 C3 C2 Non-deterministic behavior - CPAchecker offers techniques to handle non-deterministic behavior 21 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab

22 Testing Dynamic Software Product Lines - Testing SPLs: test that every derivable variant is correctly implemented - Testing DSPLs: test that all possible reconfigurations are correct Environment Program 1. : int x = y + 10; 3. : if (y < 15) { 5. : dosth1(); 7. : dosth2(); Reconfiguration Non-deterministic behavior - CPAchecker offers techniques to handle non-deterministic behavior C1 C3 C2 1. How to test reconfigurations efficiently? 2. What is a suitable coverage criteria? 3. How to ensure liveness of the reconfiguration life cycle? 22 5/8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab

23 List of Literature [Cichos et al. 2011] H. Cichos, S. Oster, M. Lochau, A. Schürr : Model-Based Coverage- Driven Test Suite Generation for Software Product Lines. MoDELS 2011: [Cichos et al. 2012] H. Cichos, M. Lochau, S. Oster, A. Schürr : Reduktion von Testsuiten für Software-Produktlinien. Software Engineering 2012: [Lochau et al. 2014] M. Lochau, J. Bürdek, S. Lity, M. Hagner, C. Legat, U. Goltz, A. Schürr : Applying Model-based Software Product Line Testing Approaches to the Automation Engineering Domain. AT Journal 2014 (submitted) [Holzer et al. 2011] A. Holzer, M. Tautschnig, C. Schallhart, H. Veith : An Introduction to Test Specification in FQL. In S. Barner, I. Harris, D. Kroening, and O. Raz, editors, Hardware and Software: Verification and Testing, volume 6504 of Lecture Notes in Computer Science, pages Springer Berlin Heidelberg, [Beyer et al. 2011] D. Beyer and M. E. Keremoglu : CPAchecker: A Tool for Configurable Software Verification. In Proc. CAV, pages , Springer, [Holzmann 1997] G. J. Holzmann : The Model Checker Spin, IEEE Trans. on Software Engineering, Vol. 23, No. 5, May 1997, pp /8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab

24 List of Literature (2/2) [Beyer et al. 2004] D. Beyer, A. J. Chlipala, T. A. Henzinger, R. Jhala, and R. Majumdar. Generating Test from Counterexamples. In Proc. of the 26th ICSE, pages , [Rhein et al. 2011] A. von Rhein, S. Apel, F. Raimondi : Introducing Binary Decision Diagrams in the Explicit-State Verification of Java Code. Java Pathfinder Workshop (co-located with ASE'11), [Apel et al. 2013] S. Apel, A. von Rhein, P. Wendler, A. Größlinger, D. Beyer : Strategies for Product-Line Verification: Case Studies and Experiments. In Proceedings of the IEEE/ACM International Conference on Software Engineering (ICSE), pages IEEE Computer Society, /8/2014 Model-based Test Case Generation for (Dynamic) Software Product Lines ES Real-Time Systems Lab