Runtime Verification - Monitor-oriented Programming - Monitor-based Runtime Reflection

Transcription

1 Runtime Verification - Monitor-oriented Programming - Monitor-based Runtime Reflection Martin Leucker Technische Universität München (joint work with Andreas Bauer, Christian Schallhart et. al) FLACOS October 10, 2007 Lehrstuhl IV: Software & Systems Engineering 1

2 Contents Runtime Verification description, applications theoretical background Monitor-oriented Programming description, applications theoretical background Monitor-based Runtime Reflection description, applications theoretical background Lehrstuhl IV: Software & Systems Engineering 2

3 Part I Runtime Verification Lehrstuhl IV: Software & Systems Engineering 3

4 Runtime Verification's Rationale Never trust a running system Lehrstuhl IV: Software & Systems Engineering 4

5 Definition Verification Verification comprises all techniques for showing that a system satisfies its specification. Runtime Verification Runtime verification deals with those techniques that allow checking whether a system under scrutiny satisfies or violates a given safety property. Lehrstuhl IV: Software & Systems Engineering 5

6 Characterization It aims to be lightweight verification technique complementing verification techniques such as model checking testing Lehrstuhl IV: Software & Systems Engineering 6

7 Formally Let be a correctness property Let denote the set of words conforming to Let be an execution a word In its mathematical essence, runtime verification is answering whether thus the word problem. Lehrstuhl IV: Software & Systems Engineering 7

8 How it works Online Monitoring P 1? M 1 M 3 P 3? Four monitors, M 1, M 2, M 3, M 4, S 1 S 3 checking independently, properties P 1, P 2, P 3, P 4 Hierarchical System S Components, Procedures,... S 2 S 4 Properties typically given in some variant of Lineartime Temporal Logic (LTL) P 2? M 2 M 4 P 4? Lehrstuhl IV: Software & Systems Engineering 8

9 How it works Offline Monitoring P 1? M 1 S 1 S 3 Hierarchical System Logging on Disk Analysis by Monitor offline S 2 S 4 Lehrstuhl IV: Software & Systems Engineering 9

10 Model Checking Formally, model checking asks which is the language inclusion problem and often of much higher complexity Model checking consider (usually) infinite traces, runtime verification necessarily finite traces finite but continuously longer traces, asks for monitors working in an incremental fashion Lehrstuhl IV: Software & Systems Engineering 10

11 Model Checking II LTL semantics for finite traces needed Complexity of monitor (automata) generation not important, but complexity of monitor Model checking only applicable to white-box systems Runtime verification ideal also for black-box systems Lehrstuhl IV: Software & Systems Engineering 11

12 Testing (I) Testing is usually incomplete like runtime verification Test case: finite sequence of input/output actions Test suite: finite collection of test cases Test execution: check whether output is as expected when input sequence is given to the system Lehrstuhl IV: Software & Systems Engineering 12

13 Testing II Test case: only (finite) sequence of input actions Test suite: finite collection of test cases Test oracle: monitor checking behavior of the system Test execution: give test cases (input sequences) to system, check whether monitor complains sounds like runtime verification! Lehrstuhl IV: Software & Systems Engineering 13

14 Testing II (cont.) Oracle defined manual Monitor derived from formal specification (LTL formula) Research issue in testing: How to find good test suites? Research issue in runtime verification: How to derive good monitors? Lehrstuhl IV: Software & Systems Engineering 14

15 Runtime verification for LTL Syntax of LTL Abbreviations: Some examples: Interpretation of over sequence of system events (behaviours), M 1 Automatic monitor generation: Inspired by translation of LTL to Büchi-automata p q p q Lehrstuhl IV: Software & Systems Engineering 15

16 LTL over finite traces Introduce 3-valued semantics for LTL (resembling the 2-valued one): The goal finite-state machines for detecting minimal bad prefixes: inconclusive l true true false i j k inconclusive Lehrstuhl IV: Software & Systems Engineering 16

17 How it actually works Use emptiness-per-state as alternative acceptance condition: M 1 Emptiness only says whether language is non-empty, not whether all future traces are accepting or not! Now we have a formal decision procedure: Lehrstuhl IV: Software & Systems Engineering 17

18 Complexity Lehrstuhl IV: Software & Systems Engineering 18

19 But optimal solution... Consider this: FSMs can be minimised (Myhill-Nerode equivalence) As such, any smaller automaton does not accept the same language => Monitor generation optimal Moreover, procedure detects all minimal bad prefixes by defintion of 3-valued semantics: Lehrstuhl IV: Software & Systems Engineering 19

20 Static initialisation order fiasco Lehrstuhl IV: Software & Systems Engineering 20

21 Monitors revisited The general structure Bad prefixes/good prefixes/ugly prefixes Monitorable (after u) Safety/co-safety properties Lehrstuhl IV: Software & Systems Engineering 21

22 Works also for timed systems! Principle stays the same, but we use Timed words TLTL aka state-clock logic for LTL (FO fragment of MSO interpreted over timed words) for each symbol we have two clock variables: Useful for time-bounded response properties! a recording clock, and a predicting clock for translation timed (event-clock) automata with runs of the form: Obviously, this execution scheme poses a problem! Lehrstuhl IV: Software & Systems Engineering 22

23 Observing Not Acting! Runtime verification does not influence/modify the running system at least tries to it might consume resources of the monitored system Lehrstuhl IV: Software & Systems Engineering 23

24 Part II Monitor-oriented Programming (Grigore Rosu et. al. - thanks for Slides) Lehrstuhl IV: Software & Systems Engineering 24

25 But first A commercial... Lehrstuhl IV: Software & Systems Engineering 25

26 Main Ideas Use monitors to check behavior of program Influence behavior of program when monitor either confirms that a correctness property is satisfied or falsified annotate code (e.g. Java code) by monitor description, and code to be triggered by monitor Lehrstuhl IV: Software & Systems Engineering 26

27 JavaMOP example Enforce authentication before use class Resource { /*@ scope = class logic = PTLTL { Event authenticate: end(exec(* authenticate())); Event use: begin(exec(* access())); Formula : use -> <*> authenticate } violation Handler void authenticate() {...} void access() {...}... } Lehrstuhl IV: Software & Systems Engineering 27

28 Example Programming methodology some ideas similar as in aspect-oriented programming Lehrstuhl IV: Software & Systems Engineering 28

29 MOP key features Extensible logic framework Plug and use user-defined logics using logic plugins Supported logics: ERE (Extended Regular Expression), PtLTL (Past Time Linear Temporal Logic), FtLTL (Future Time Linear Temporal Logic), ATL (Allen Temporal Logic) Generic and efficient support for universal parameters Allow monitor instances per groups of objects Faster than logic-specific monitoring techniques, e.g., Tracematches and PQL. [OOPSLA 07] Configurable monitors Running mode: inline/outline, online/offline Working scope: method, class, global, User-provided actions for violations and validations Lehrstuhl IV: Software & Systems Engineering 29

30 Part III Monitor-oriented Runtime Reflection Lehrstuhl IV: Software & Systems Engineering 30

31 Runtime reflection An architecture for model-based runtime analysis: Control commands Reconfiguration Distributed real-time system Diagnosis Monitoring Logging System events Lehrstuhl IV: Software & Systems Engineering 31

32 How it works A functional overview P 1? M 1 M 3 P 3? S 1 S 3 Distributed real-time system, S Diagnosis {S 1,S 2,S 3,S 4 } S 2 S 4 P 2? M 2 M 4 P 4? Lehrstuhl IV: Software & Systems Engineering 32

33 Part - Monitoring Reconfiguration Distributed real-time system Diagnosis As done in Runtime Verification Monitoring Logging Lehrstuhl IV: Software & Systems Engineering 33

34 Part - Diagnosis Reconfiguration Diagnosis Distributed real-time system Monitoring Logging Lehrstuhl IV: Software & Systems Engineering 34

35 Part Diagnosis Distributed real-time system System model: set of components,, system description,, specified in terms of first-order sentences, e.g., Reconfiguration Diagnosis Monitoring Logging Diagnostic task: given a set of observations,, find all minimal diagnoses s.t. is explicable, i.e., is consistent S1 S2 MBD-Engine Lehrstuhl IV: Software & Systems Engineering 35 S3 S4

36 But what about... Distributed real-time system Reconfiguration Diagnosis Monitoring Logging But, first-order diagnosis is hard! Automatic conflict set determination may not always be possible (e.g., non-termination, HS-problem is NP-hard) Although tailored for it, introduction of unknowns may result in exponential blowup of conflict search space Not laid out for real-time / event-triggered systems Ultimately, hard to think of doing all this continuously at runtime! Lehrstuhl IV: Software & Systems Engineering 36

37 Combined analysis Diagnosis-on-demand, using the automatically generated monitors as inputs Distributed real-time system Reconfiguration Diagnosis Monitoring Logging M 1 false. true,?.. true,? MBD-Engine M n true,? Propositional system model, SD: ok(m1), ok(m2),... Problem still in NP, but powerful algorithms for computing conflicts Lehrstuhl IV: Software & Systems Engineering 37

38 Example: where is the fault?!! i1 i2 S 1 m1? S 3 o1!! i3 i4 S 2 m2? S 4 o2 Lehrstuhl IV: Software & Systems Engineering 38

39 Let's compare! The three dimensions of runtime analysis (computational) complexity So, how is our combined approach doing now? precision communication Lehrstuhl IV: Software & Systems Engineering 39

40 Problems and strengths of methods (computational) complexity usually no diagnosis (and if, at a high comm. cost) Model-based Diagnosis Runtime reflection verification usually static, very complex, no detection mechanisms, not suited for real-time precision communication High precision with minimal communication overhead (at runtime, for real-time and distributed systems) Lehrstuhl IV: Software & Systems Engineering 40

41 Towards conclusion Let's wrap up! Lehrstuhl IV: Software & Systems Engineering 41

42 Runtime verification Control commands Reconfiguration Distributed real-time system Diagnosis Monitoring Logging System events Lehrstuhl IV: Software & Systems Engineering 42

43 Monitor-oriented programming Combination of Runtime Verification with Diagnosis Control commands Reconfiguration Distributed real-time system Diagnosis Monitoring Logging System events Lehrstuhl IV: Software & Systems Engineering 43

44 Runtime reflection Combination of Runtime Verification with Diagnosis Control commands Reconfiguration Distributed real-time system Diagnosis Monitoring Logging System events Lehrstuhl IV: Software & Systems Engineering 44

45 Thank you! Lehrstuhl IV: Software & Systems Engineering 45