Incremental Model-based Testing of Delta-oriented Software Product Lines

Transcription

1 Incremental Model-based Testing of Delta-oriented Software Product Lines Ina Schaefer (joint work with Malte Lochau and Sascha Lity) IFIP WG 2.11 Meeting, Halmstad, 26 June 2012

2 Software Product Line Engineering Reuse Approaches in SPL Engineering: 1. Annotation: artifact selection from 150% core product 2. Composition: combine core artifacts 3. Transformation: apply deltas to core product 1) 2) 3) Φ 1 (f) p 1 Δ1 Φ 2 (f) p 2 Δ2 Ina Schaefer Incremental SPL Testing 2

3 Software Product Line Engineering Reuse Approaches in SPL Engineering: 1. Annotation: artifact selection from 150% core product 2. Composition: combine core artifacts 3. Transformation: apply deltas to core product 1) 2) 3) Φ 1 (f) p 1 Δ1 Φ 2 (f) p 2 Δ2 ñ But how to reuse test artifacts for SPL quality assurance? Ina Schaefer Incremental SPL Testing 2

4 Agenda Model-based Testing Model-based SPL Testing Incremental Model-based SPL Testing Delta-oriented Test Models Delta-oriented Test Regression Ina Schaefer Incremental SPL Testing 3

5 Model-based Testing Test Model conforms? i k,..., i 2, i 1 o 1, o 2,..., o k IUT Black-box assumption for implementation-under-test (IUT) Automated derivation and application of test suites from a behavioral test model [UL07] Ina Schaefer Incremental SPL Testing 4

6 State Machine Test Model t0 : π 1 /π 2 s0 s1 t1 : π 1 /π 4 t2 : π 3 /π 5 t3 : π 3 /π 2 State machines as test models S is a finite set of states s 0 P S is an initial state L Ď Π I ˆ Π O is a set of transition labels T Ď S ˆ L ˆ S is a transition relation Ina Schaefer Incremental SPL Testing 5

7 State Machine Test Case t0 : π 1 /π 2 s0 s1 t1 : π 1 /π 4 t2 : π 3 /π 5 t3 : π 3 /π 2 State machine test artifacts: Test case: tc pt 0, t 1,..., t k q P T is a sequence of k transitions Test run: execptc, tmq pl 0, l 1,..., l k q P L Test result: iut passes tc :ô execptc, iutq «te execptc, tmq Ina Schaefer Incremental SPL Testing 6

8 State Machine Test Suite t0 : π 1 /π 2 s0 s1 t1 : π 1 /π 4 t2 : π 3 /π 5 t3 : π 3 /π 2 Infinite set of all valid test cases: TCptmq Ď T Derive finite test suites: ts Ď TCptmq Coverage criteria C: finite sets tg Cptmq of test goals such P tg : Dtc P ts : coversptc, gq Ina Schaefer Incremental SPL Testing 7

9 Model-Based Testing of Software Product Lines Reusable Test Model conforms? ik,..., i2, i1 o1, o2,..., ok Product Variants under Test IUT p1q IUT p2q IUT pnq TM p1q TM p2q TM pnq Testing a family of similar software product variants 1. Reusable SPL test model [Oli08] 2. Selection of representative products-under-test [OWES10] Ina Schaefer Incremental SPL Testing 8

10 Model-Based Testing of Software Product Lines Reusable Test Model conforms? ik,..., i2, i1 o1, o2,..., ok Product Variants under Test IUT p1q IUT p2q IUT pnq TM p1q TM p2q TM pnq Testing a family of similar software product variants 1. Reusable SPL test model [Oli08] 2. Selection of representative products-under-test [OWES10] ñ Incremental test artifact reuse between products-under-test Ina Schaefer Incremental SPL Testing 8

11 Delta-Oriented SPL Engineering [CHS11] Feature Configuration Core Product t1: e1/e2 s1 t2: e3/ t3: e2/ t4: e5/e6 t5: e4/e6 s3 Product Variant t7: e1/ s4 t6: e3/ Feature Model Application Condition Delta 1 Add + s1 + s4 + t7: e1/ t6: e3/ Delta 2 Rem s1 - t1: e1/e2 + + Core + Delta 1 Core + Delta 1 + Delta 2 t1: e1/e2 s1 t2: e3/ t3: e2/ t4: e5/e6 s3 t5: e4/e6 t7: e1/ s4 t6: e3/ s1 t2: e3/ t3: e2/ t4: e5/e6 s3 t5: e4/e6 t7: e1/ s4 t6: e3/ s3 Delta 3 Mod * t5: e5/e2;e3;e4 + Core + Delta 1 + Delta 2 + Delta 3 s1 t2: e3/ t3: e2/ t4: e5/e6 s3 t5: e5/e2;e3;e4 Ina Schaefer Incremental SPL Testing 9

12 State Machine Delta Example t0 : π 1 /π 2 s0 s1 t3 : π3 /π 2 t1 : π 1 /π 4 t2 : π 3 /π 5 δ tm = {rem t1, add t5} δ tm = {rem t1, rem t3, add t4, add t5} t0 : π 1 /π 2 s0 s1 t4 : π 3 /π 2 t2 : π 3 /π 5 t5 : π 1 /π 4 t0 : π 1 /π 2 s0 s1 t3 : π 3 /π 2 t5 : π 1 /π 4 t2 : π 3 /π 5 Ina Schaefer Incremental SPL Testing 10

13 1 Incremental SPL Testing tmcore C tm i δ tm i,tm i+1 tmi+1 tm n valid tgcore tg i δ tg i,tg i+1 tgi+1 tg n tpcore retest covers tp i δ tp i,tp i+1 tpi+1 tp n tscore ts i δ ts i,ts i+1 tsi+1 ts n Starting with test artifacts of the core product ta core ptm core, tg core, ts core, tp core q Incremental evolution of product test artifacts ta p ptm p, tg p, ts p, tp p q δ ta,ta 1 ÝÝÝÑ ta p 1 ptm p 1, tg p 1, ts p 1, tp p 1q Ina Schaefer Incremental SPL Testing 11

14 Test Model Regression Delta State machine regression delta δ tm,tm 1 Ď Op captures the difference between test model variant tm and tm 1. t0 : π 1/π 2 s0 s1 t1 : π 1/π 4 t3 : π3/π 2 t2 : π3/π 5 δtm 1 = {add t1, add t3, rem t4, rem t5} δtm = {rem t1, add t5} s0 t4 : π 3/π 2 t0 : π 1/π 2 s1 t2 : π 3/π 5 δtm = {rem t1, rem t3, add t4, add t5} δtm,tm = {add t3, rem t4} s0 t3 : π 3/π 2 t0 : π 1/π 2 s1 t2 : π 3/π 5 t5 : π 1/π 4 t5 : π 1/π 4 Ina Schaefer Incremental SPL Testing 12

15 Regression Delta Construction Proposition (State Machine Regression Delta Construction) For two state machine deltas δ tm Ď Op and δ tm 1 Ď Op, the regression delta is given as δ tm,tm 1 pδ tm zδ tm 1q 1 Y pδ tm 1zδ tm q Invert delta operations exclusive to tm, Keep delta operations shared with tm 1 Add delta operations exclusive to tm 1 ñ Basis for test artifact evolution Ina Schaefer Incremental SPL Testing 13

16 Test Goal Delta Test goal regression delta δ tg,tg 1 for each trem eu Ď δ tm,tm 1 : e P tg ñ trem eu Ď δ tg,tg 1 for each tadd eu Ď δ tm,tm 1 : e P Cptm 1 q ñ tadd eu Ď δ tg,tg 1 ñ Limited to C0/C1-like coverage criteria Ina Schaefer Incremental SPL Testing 14

17 Test Suite Delta Test suite regression delta δ ts,ts 1 Obsolete test cases ts O tszts R Reusable test cases ts R ts X ts 1 New test cases ts N ts 1 zts ts ts O ts R ts N ts Ina Schaefer Incremental SPL Testing 15

18 Test Plan Delta Test plan: tp p ts RT Y ts N Regression Test Selection for ts RT : Retest-all, retest-random, test model slicing,... ts ts O ts RT ts R ts N ts Ina Schaefer Incremental SPL Testing 16

19 Implementation and Sample Tool Chain EMF-based Eclipse plug-in for modeling delta-oriented state machines Domain feature model for product configuration and derivation Test model import into IBM Rational Rhapsody Add-on ATG for automated test case generation and execution Ina Schaefer Incremental SPL Testing 17

20 Evaluation Body Comfort System Human Machine Interface Door System Security Power Window Exterior Mirror Alarm System Central Locking System require Remote Control Key Status LED Manual Power Window Automatic Power Window Finger Protection Electric Heatable Interior Monitoring Automatic Locking Control Alarm System Safety Function Control Automatic Power Window exclude require require LED Alarm System LED Finger Protection LED Central Locking System LED Power Window LED Exterior Mirror LED Heatable Keys require Mandatory Optional Alternative Or exclude require require Automotive Case Study: Body Comfort System 26 features, product variants Delta test model with core model and 40 delta modules, 105 states and 107 transitions Ina Schaefer Incremental SPL Testing 18

21 Evaluation Results New Selection MoSo-PoLiTe P0 P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 P16 P17 Subset selection of 17 representative products-under-test for pairwise feature interaction coverage Product-by-product testing: 64 test cases generated and executed per product on average Incremental testing: 10 test cases generated per product, 9 selected for execution on average Ina Schaefer Incremental SPL Testing 19

22 Conclusions Efficient incremental model-based SPL testing approach based on regression delta analysis. Future Work: Test goal evolution for path-oriented coverage criteria, e.g., MC/DC Perform industrial case studies Scalability: combination with delta-oriented architecture models Ina Schaefer Incremental SPL Testing 20

23 References I Gilles Bernot. Testing against formal specifications: A theoretical view. In S. Abramsky and T. Maibaum, editors, TAPSOFT 91, volume 494 of Lecture Notes in Computer Science, pages Springer Berlin, Heidelberg, / D. Clarke, M. Helvensteijn, and I. Schaefer. Abstract Delta Modeling. Mathematical Structures in Computer Science, (to appear). R. DeNicola and M. C. B. Hennessy. Testing equivalences for processes. Theoretical Computer Science, pages , E. M. Olimpiew. Model-Based Testing for Software Product Lines. PhD thesis, George Mason University, Ina Schaefer Incremental SPL Testing 21

24 References II S. Oster, A. Wübbeke, G. Engels, and A. Schürr. Model-Based Software Product Lines Testing Survey. In Model-based Testing for Embedded Systems to appear. S. Oster, I. Zorcic, F. Markert, and M. Lochau. MoSo-PoLiTe - Tool Support for Pairwise and Model-Based Software Product Line Testing. In VAMOS 11, M. Utting and B. Legeard. Practical Model-Based Testing. A Tools Approach. M. Kaufmann, Ina Schaefer Incremental SPL Testing 22