1 Testing Real-Time Systems Using UPPAAL Anders Hessel, Kim G. Larsen, Marius Mikucionis, Brian Nielsen, Paul Pettersson, and Arne Skou. 2008 Selected Topics Software Technology 2 Jorge Santos Simón u www.tugraz.at
2 Contents Introduction: why to model time? (ultra) Short introduction to Timed Automata (ultra) Short introduction to UPPAAL Model-based testing of Timed Automata Offline testing with UPPAAL Online testing with UPPAAL Conclusions
3 Why to bother about time? For many systems, particularly real-time systems, when things happen is as important as what happens We must then be able to incorporate time pass in our models But this makes testing more challenging Tester must carefully consider when to issue commands to the system Verdicts must take into account when the system responds Test execution must also run in real-time!
4 Modeling of real-time systems Embedded systems interacts closely with their environments, so we must model both Both models will run in parallel exchanging input/output signals RealENV: physical environment IUT: System (Implementation) Under Test input/output from IUT point of view!
5 Modeling of real-time systems Advantages of separated models for IUT and environment: We can create tailored environments for particular test purposes Avoid IUT model changes when porting to different scenarios and use-cases
6 Contents Introduction: why time matters? (ultra) Short introduction to Timed Automata (ultra) Short introduction to UPPAAL Model-based testing of Timed Automata Offline testing with UPPAAL Online testing with UPPAAL Conclusions
7 TIOTS We can model real-time systems through Timed Input / Output Transition Systems (TIOTS) TIOTS are similar to Labeled Transition Systems (LTS) but... Actions are divided on input and output Contains delay labels to model time pass
8 Timed Automata Formalism of TIOTS for modeling real-time system An extension of FSM but... Include a set of real-valued clock variables Transitions can be enabled/disabled according to clock values Locations can have invariants according to clock values A timed automaton can progress either: Executing an edge (as long as the guards are satisfied) Remaining in a location and letting time pass (as long as the invariants are satisfied)
9 Contents Introduction: why time matters? (ultra) Short introduction to Timed Automata (ultra) Short introduction to UPPAAL Model-based testing of Timed Automata Offline testing with UPPAAL Online testing with UPPAAL Conclusions
10 UPPAAL Graphic tool for creating, simulating and checking models for real-time systems Developed at the UPPsala University and AALborg University First release in 1995 Newest version is 4.0 Academic tool but... it works!!!
11 UPPAAL Systems can be modeled as parallel running timed automata They communicate sharing discrete and clock variables Can synchronize through complementary input (?) and output (!) actions Initial locations are marked by a double circle Edges can be labeled with guards, an action and assignments, all of them optional If the action label is absent, we call it an internal, unobservable action There are commited (marked with C) and urgent locations (marked with U) Locations can have invariants over clock or integer variables
12 UPPAAL example : a light controller
13 UPPAAL example : possible environments User can touch as fast as Treact More realistic user that waits Tpause in between
14 Contents Introduction: why time matters? (ultra) Short introduction to Timed Automata (ultra) Short introduction to UPPAAL Model-based testing of Timed Automata Offline testing with UPPAAL Online testing with UPPAAL Conclusions
15 Testing Timed Automata with UPPAAL We need a conformance relation that includes time Relativized timed conformance: rtioco Derived from ioco, but taking time and environment constraints into account: The IUT cannot produce an output when it is not allowed by the specification The IUT cannot omit an output required by the specification Intuitively, this boils down into a trace-inclusion relation We can use different environments to target particular behaviors of our IUT
16 Offline vs. online testing Offline Testcases are easier, cheaper and faster to execute
17 Offline vs. online testing Online Testcases can run indefinitely, improving coverage Prevent state-space explosion by computing only the immediate actions Works well with non-deterministic IUTs
18 Contents Introduction: why time matters? (ultra) Short introduction to Timed Automata (ultra) Short introduction to UPPAAL Model-based testing of Timed Automata Offline testing with UPPAAL Online testing with UPPAAL Conclusions
19 Offline Test Generation We assume that our automaton is... Deterministic (always behaves equally with a given input at a given location) Weakly input enabled (cannot reject inputs) Output urgent (do not wait if it can provide an output) With isolated outputs (provides one output at maximum at a time) We can generate testcases out of Test purpose: a particular property of the IUT we want to test, for example that a location can be reached Coverage criteria, for example edge coverage, location coverage, definition-use pair coverage
20 Offline Test Generation Testcases are generated by reformulating test purposes and coverage criteria as reachability problems solvable by a model-checker, as UPPAAL For this, the model must be annotated with variables The model-checker produces as output a diagnostic trace that the IUT must conform to: That is, a sequence of actions and time delays We generate testcases by adding verdicts (PASS/FAIL locations) to these traces
21 Offline Test Generation Example testcase A complete execution terminates in FAIL if the IUT cannot perform the diagnostic trace Otherwise, the execution terminates in PASS
22 Contents Introduction: why time matters? (ultra) Short introduction to Timed Automata (ultra) Short introduction to UPPAAL Model-based testing of Timed Automata Offline testing with UPPAAL Online testing with UPPAAL Conclusions
23 Online Testing Allows testing non-deterministic Timed Automata Possible sources of non-determinism: Abstraction of very complex behavior Timing uncertainty Implementation intrinsically non-deterministic Optional (but correct) behavior Simple example:
24 A Real-Time Online Testing Algorithm Takes as input two weakly input enabled, non-blocking Automata, modeling the IUT and environment Under these assumptions, the algorithm is both complete and sound, given enough time Keeps track of currently reachable states If these are exauxted, the IUT cannot be rtioco conforming At every step, the algorithm can do one of: Send an input to the IUT Wait from an input from the IUT Restart testing IUT outputs and delays are confronted to the reachable states sets to verify its validity
25 Online Testing with TRON The algorithm has been implemented as an UPPAAL extension, TRON UPPAAL extended for Testing Real-time systems ONline Requires a IUT-specific test adapter that: Transforms abstract input actions into physical IUT stimuli Transforms physical IUT outputs into abstract output actions TRON emulates the environment generating appropriate inputs and at the same time monitors IUT outputs, ensuring conformance
26 Online Testing with TRON TRON has been successfully used in several industrial usecases Basic setup:
27 Conclusions Model-Based techniques are mature enough for testing Real-Time systems UPPAAL provides a solid foundation for current and further developments Given the non-deterministic nature of most systems, online testing is a better suited technique That said...