Barriers to Systematic Model Transformation Testing

Transcription

1 Barriers to Systematic Model Transformation Testing Benoit Baudry, Sudipto Ghosh, Franck Fleurey, Robert France, Yves Le Traon, Jean-Marie Mottu Soroosh Nalchigar October 2012

2

3 Model Transformations Model transformation Automatic One or multiple input models One or multiple output models Transformation definition Set of transformation rules Transformation rule Describe how one or more construct in the source language can be transformed Kleppe et al. (2003), Mens & Gorp (2006)

4 Model Transformations Vertical versus horizontal Abstraction level Endogenous versus exogenous The language in which the source and target models are expressed. Mens & Gorp (2006)

5 Amrani et al. (2012) So why? Different intentions of doing model transformations: Refinement Abstraction Synthesis (e.g., code generation) Reverse engineering Refactoring Synchronization

6

7 Example of model refactoring Before: LeagueOwner + address Advertiser + address Player + address After: What kind of transformation is this? User + address LeagueOwner Advertiser Player

8 Example of code generation User - string +get ():string +set (e:string) +notify(msg:string) LeagueOwner -maxnumleagues:int +getmaxnumleagues():int +setmaxnumleagues(n:int) public class User { private String ; public String get () { return ; } public void set (string value){ = value; } public void notify(string msg) { //... } } public class LeagueOwner extends User { private int maxnumleagues; public int getmaxnumleagues() { return maxnumleagues; } public void setmaxnumleagues (int value) { maxnumleagues = value; } } What kind of transformation is this?

9 What is this paper about? Why it is difficult to, systematically, test model transformations? What are possible ways to overcome these barriers?

10 A small example!

11 A small example! Flattening Before Hierarchical state machine After Flattened state machine

12 How to test model transformations? Step 1: Generate test data Generate input models (test models) Question: What other input models would you suggest?

13 How to test model transformations? Step 2: Define test adequacy criteria It is not possible to test a transformation with all possible input models E.g. instantiate all the meta-classes Step 3: Construct an oracle Determines if the result of a test case is correct Could be partial and check expected properties of the output model

14 Where we are? Why it is difficult to, systematically, test model transformations? What are possible ways to overcome these barriers?

15 Complex input and output data Complexity of data manipulated by transformations: Large input and output models Multiple views Large meta-models OCL constraints This complexity affects generation of test models: Manual: error-prone Automatic: complex constraint solving problem

16 Complex input and output data Lack of historical data on typical errors: Difficult to determine effectiveness of test adequacy criteria Difficult to determine the fault models they can target Complexity of output data complicates oracle problem: To build the expected results Even if the output model is available? Wikipedia (2012)

17 Possible Solutions Automatic test model generation Definition of Oracle Use of SAT Solvers Constructive approach Test the output model directly Partial oracles Work on requirements definition

18 Possible Solutions Use of SAT Solvers It has the limitations of current automatic constraint solvers Use of heuristics and meta-heuristics? Constructive approach Brottier et al. (2006) Test the output model directly Partial Oracles Checks only specific properties of the output instead of checking the result completely Develop techniques for the precise definition of requirements for model transformations Extract logical formulae from natural languages?

19 Model Management Environments MDE is in the research stage, support is needed for: Building Editing Visualizing, and Analyzing models. Manual definition of test models is tedious and error-prone. Construction of models involve either: Writing a program: error-prone Using model editors generated from a meta-model: Lack of language-specific icons Lack of dialog boxes for setting attribute values Lack of assistance for checking the completeness

20 Model Management Environments Regression testing: Need for sophisticated model comparison tools Visualizing output models is difficult Lack of adequate support for layout of diagrams

21 Possible Solutions Ongoing works could be adapted or integrated for testing model transformations. Model differencing: EMFCompare Tool: detects matches and differences between two models based on similarity in their type, name, and values. Versioning of models: Used to define oracles that compares models and detect conflicts Detection of test models that are still relevant and those need to evolve.

22 Heterogeneity of Transformation Languages and Techniques Large number of model transformation languages and techniques Strong impact on the definition of effective white-box test adequacy criteria It is impossible to know if any one of the many techniques will fit all the needs for model transformations. We cannot choose one language as a reference and develop test criteria that are based on language elements.

23 Possible Solutions Black-box techniques: Fleurey et al. (2007) proposed a set of rules and a framework to assess the quality of test models. Assumed that all meta-models are modeled in EMOF. Evaluate the quality based on coverage requirements: Class coverage Attribute coverage Association coverage Benefit: any transformation language. Should consider intent of transformation.

24 Possible Solutions White-box techniques: More detailed and effective than black box, capture mechanics of transformation. Must cover the individual transformation steps. The tester should generate tests models that exercise each steps. Küster et al. (2006) defined a template language to generate test models based on the structure of the transformation rules.

25 Discussion 37 citations from 2010 Does the intent of transformation affects difficulty of tests? What about MMTF? A suggestion for the course My be a basic paper with 468 citations?

26 References [1] B. Baudry, S. Ghosh, F. Fleurey, R. France, Y. Le Traon, J.-M. Mottu. Barriers to Systematic Model Transformation Testing. Communications of the ACM 53(6), pp , June [2] M. Amrani, J. Dingel, L. Lambers, L. L Atucio, R. Salay, G. Selim, E. Syriani, and M. Wimmer. Towards a model transformation intent catalog. 1st Workshop on the Analysis of Model Transformations (AMT), [3] A. Kleppe, J. Warmer, W. Bast MDA Explained, The Model-Driven Architecture: Practice and Promise Addison Wesley (2003) [4] T. Mens and P. V. Gorp. A taxonomy of model transformation. Electronic Notes in Theoretical. Computer Science, 152(0):125 { 142, [5] Fleurey, F., B. Baudry, P.-A. Muller, and Y. Le Traon, Towards Dependable Model Transformations: Qualifying Input Test Data. Software and Systems Modeling, [6] Küster, J.M. and M. Abd-El-Razik. Validation of Model Transformations - First Experiences using a White Box Approach. in MoDeVa'06 (Model Design and Validation Workshop associated to MoDELS'06) Genova, Italy.