Juhan Ernits Inst. of Cybernetics / Dept. of Comp Sci. Tallinn University of Technology Tallinn, Estonia

Transcription

1 Model Based Testing with NModel Juhan Ernits Inst. of Cybernetics / Dept. of Comp Sci. Tallinn University of Technology Tallinn, Estonia

2 Who is this guy? Occupation: a researcher at the Institute of Cybernetics and the Dept. of Computer Science of Tallinn University of Technology Relation to NModel: helped to develop a fragment during an internship at MSR. Currently building several case studies

3 Plan Introduction Model based testing at Microsoft Model programs and NModel Model programs Actions State Abstract data structures Objects Model exploration Composition of model programs Scenarios Building system adapters Offline test generation Online testing

4 Model Based Testing at Microsoft MBT has been used for years at Microsoft SpecExplorer, Model specifications as model programs in ASML or Spec# Explore the models using the exploration engine Unwind the model programs into FSM s Generate tests There is an in house derivative of the tool used in Windows

5 AsmL Abstract State Machine Language Executable specification language based on Gurevich s theory of Abstract State Machines Does not contain loop constructs, instead uses comprehensions

6 Spec# There are two flavours of Spec#: Spec# used in SpecExplorer for modelling specifications was based on AsmL compiler (currently not actively developed) Spec# used for assertion checking in object oriented programs, similar to Java + JML (by Rustan Leino et al., currently actively developed)

7 The Other Spec# (for Assertion Checking) The Spec# programming language: an extension of C# with non null types, checked exceptions and throws clauses, method contracts and object invariants. The Spec# compiler: a compiler that statically enforces non null types, emits run time checks for method contracts and invariants, and records the contracts as metadata for consumption by downstream tools. The Spec# static program verifier: a component (named Boogie) that generates logical verification conditions from a Spec# program. Internally, it uses an automatic theorem prover that analyzes the verification conditions to prove the correctness of the program or find errors in it.

8 NModel NModel is built on the experiences with SpecExplorer Modelling language is C# using artefacts from the modelling library NModel is free to use and develop further There is a book available:

9 SpecExplorer vs NModel NModel uses an industry standard language (C#) and compiler. Modelling artefacts are implemented in a library. Spec# is a separate language that needs a separate compiler. Models built with NModel will be usable as long as there is a compiler supporting.net 2.0

10 Highlights of NModel Allows modelling using abstract data structures (sets, maps, bags) and objects Supports composition of model programs. Models can be split up into features that can be composed into a single model Supports exploration of model programs Supports state graph isomorphism based symmetry reduction

11 Model Program A model program P has the following components: A finite vocabulary X of state variables The state variables are ordered, e.g. by name A finite vocabulary Σ of action symbols An initial state s 0 given by A reset action symbol Reset A family (ϕ f, p f ) f Σ of guarded update rules The arity of f is the number of input parameters of p f The arity of Reset is 0 and Reset takes the model to s 0

12 Model Program In other words There are atomic actions There are action guards that do not have side effects and return a Boolean value There are context variables that can take values of literals (ints, bools, strings,...) or abstract data structures.

13 Modelling: Abstraction It is important to figure out what the goal of modelling is, i.e. what is important from the point of view of what we try to model. The goal sets the level of abstraction. Our goal is to do functional model based testing of the System Under Test: What are the actions in the SUT to test? What variables/data structures capture the state of the SUT?

14 Modelling: Coding To code each action identify When is it enabled? Write an appropriate action guard. What part of the state does it update? Is the action observable or controllable? Does the action return something? It is useful to think of some relevant scenarios

15 Actions Actions are units of behaviour Actions are atomic Actions can have zero or more arguments Actions are enabled if action guards evaluate to true If several actions are enabled the choice is nondeterministic If there are several possible values of the arguments, choice is nondeterministic

16 Example Look at the functionality of the model program viewer Web testing

17 Composition NModel facilitates composing the model of smaller pieces. The pieces, called features, represent certain functionality or requirement. You can select the set of features that are composed for some particular test

18 Scenarios Testers often have some particular scenarios in mind when modelling a system. It is possible to define scenarios as scenario model programs, either as Finite state machines C# model programs A scenario typically is a sequence of actions that lead to some particular state.

19 Another Example Look at an example of a web application It is possible to Log in Log out Save an integer value Save a string value MBT discovers that the saved values do not persist between logins Example available from: nmodel aalborg08/

20 System Adapter The example shows an example how to build a system adapter in NModel The stepper is responsible for executing the model in lock step with the messages arriving from and sent to the SUT Building the system adapter is often quite time consuming

21 Offline test generation NModel contains a tool called otg.exe which can be used for generating offline tests otg produces test suites that contain test cases in term format. The test cases can be replayed using ct.exe and /testsuite argument. For help look at the output of ct.exe /help and otg.exe /help

22 Online Testing NModel comes with ct.exe (ct stands for conformance checker). ct.exe can run previously generated test suites or traverse the model online either using random strategy or some strategy provided by the test engineer (a strategy needs to implement the ISrategy interface) Look at NModel web site and help for more information

23 Summary of NModel NModel is an open source toolkit for modelbased testing If it does not contain the feature you need, you can add it NModel is quite light weight but at the same time powerful and is definitely worth looking at if you are interested in model based testing

24 Tänan tähelepanu eest!