Quality Assurance in Software Development

Transcription

1 Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institute for Software Technology Graz University of Technology Austria Summer Term / 19

2 Agenda 1 Organisation 2 Contents 3 Quality of SW 2 / 19

3 Quality Assurance in Summer Term 2013 Lecture: Tue 11:15 12:00 (i12) Exercise: Tue 12:00 12:45 (i12) Optional: Fri 14:15 15:00 (i12) Registration via tugonline strict! Two student assistants: Sandra Fruhmann Martin Tappler 3 / 19

4 Additional Resources WWW: Newsgroup: tu-graz.lv.qs Subject: [QS13] 4 / 19

5 Marking Marking iff registered on (Dreamspark licence) Written exam (Klausur) : (50 %). Three practical tasks (groups of 3): Task 1: publication (10 %) Task 2: publication (10 %) Task 3: publication (15 %) Task 4: publication (15 %) Oral interviews about tasks: after Task 3 Exam and exercises count each 50% Positive if > 50% total points 5 / 19

6 Marking: Nachklausur Date: Beginning of July After excused absence at exam e.g. due to illness with sick certificate (ärztlicher Bestätigung) work is no excuse (take leave!) For all negative! 6 / 19

7 What we will not cover? Quality management: Process management, e.g. Scrum Process improvement, e.g. ISO 9000 Test management, e.g. IBM Rational Quality Manager Version control, e.g. svn, git... 7 / 19

8 General Aims of the Course Getting familiar with quality assurance techniques Raising the awareness for quality of software Practising fault-based thinking Being able to distinguish process-oriented and technical quality assurance Challenging common quality criteria Realising the relations between the foundations of software engineering and quality assurance. 8 / 19

9 Specific Learning Targets and Skills Getting familiar with a professional IDE Writing good tests :( Generating good tests :) Specifying test oracles Using and understanding state-of-the-art test case generators Test coverage metrics "Don t write test cases, generate them!" (John Hughes) 9 / 19

10 Course Contents Introduction and motivation Challenges of software testing Coverage: Measuring the quality of test cases Testing with Concolic Execution Contracts as test oracles Model-based Testing Proof-based Development Tools (exercise): MS Visual Studio Ultimate 2010 MS Pex.NET Code Contracts MS Spec Explorer You will get a DreamSpark licence (registration)! 10 / 19

11 Quality? Today, often a sufficient level of correctness efficiency costs cannot be guaranteed. Vision: Development methods for SW with warranty. 11 / 19

12 Bugs Part of engineering jargon for many decades: Moth trapped in relay of Mark II (Hopper 1946) Little faults and difficulties (Edison 1878): Software bugs Relay #70 Panel F (moth) in relay. First actual case of bug being found. 12 / 19

13 Bugs Part of engineering jargon for many decades: Moth trapped in relay of Mark II (Hopper 1946) Little faults and difficulties (Edison 1878): Software bugs Definition A software bug is the common term used to describe an error, flaw, mistake, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways. (Wikipedia 2012) 12 / 19

14 Some Bugs Become Famous! Ariane 5 test flight (1996) out of control due to software failure controlled destruction! Loss of money and time satellites research (TU Graz) Dijkstra (EWD 1036): call it error, not bug a programmer created it 13 / 19

15 Binary search bug in Java Some Bugs Hide for a Long Time! JDK 1.5 library (2006) out of boundary access of large arrays due to integer overflow 9 years undetected Algorithm was proven correct! Programming Pearls [Bentley86, Bentley00] assuming infinite integers :( 1 public static 2 int binarysearch(int[] a,int key) 3 { 4 int low = 0; 5 int high = a.length - 1; 6 7 while (low <= high) { 8 int mid = (low + high) / 2; 9 int midval = a[mid]; if (midval < key) 12 low = mid + 1; 13 else if (midval > key) 14 high = mid - 1; 15 else 16 return mid; // key found 17 } 18 return -(low + 1); // key not found 19 } Beware of bugs in the above code; I have only proved it correct, not tried it. [Knuth77] 14 / 19

16 Binary search bug in Java Some Bugs Hide for a Long Time! JDK 1.5 library (2006) out of boundary access of large arrays due to integer overflow 9 years undetected Algorithm was proven correct! Programming Pearls [Bentley86, Bentley00] assuming infinite integers :( 1 public static 2 int binarysearch(int[] a,int key) 3 { 4 int low = 0; 5 int high = a.length - 1; 6 7 while (low <= high) { 8 int mid = (low + high) >>> 1; 9 int midval = a[mid]; if (midval < key) 12 low = mid + 1; 13 else if (midval > key) 14 high = mid - 1; 15 else 16 return mid; // key found 17 } 18 return -(low + 1); // key not found 19 } Beware of bugs in the above code; I have only proved it correct, not tried it. [Knuth77] 14 / 19

17 Risks for the Public due to SW The Risks Digest 15 / 19

18 Edsger W. Dijkstra In academia, in industry, and in the commercial world, there is a widespread belief that computing science as such has been all but completed and that, consequently, computing has matured from a theoretical topic for the scientists to a practical issue for the engineers, the managers, and the entrepreneurs. [...]... then he characterises the software crisis and concludes: I would therefore like to posit that computing s central challenge, "How not to make a mess of it," has not been met. 16 / 19

19 Edsger W. Dijkstra (cont.) On the contrary, most of our systems are much more complicated than can be considered healthy, and are too messy and chaotic to be used in comfort and confidence. The average customer of the computing industry has been served so poorly that he expects his system to crash all the time, and we witness a massive worldwide distribution of bug-ridden software for which we should be deeply ashamed. (Communications of the ACM, Mar 2001) 17 / 19

20 Why? A possible explanation: Quality assurance requires a certain degree of (healthy) redundancy extra costs! (appr. 50% for critical systems) Examples of redundancy: process management documentation pair programming modelling test cases redundant components in fault-tolerant systems (accepted) 18 / 19

21 Demo: Triangle Example Writing unit tests in VS 2010 Code coverage in VS 2010 Automatic test case generation in Pex How good are the tests? 19 / 19

23 Agenda 1 The Challenge of Software Testing 2 Definitions 3 Functional testing strategies Boundary Value Testing Random Testing Equivalence Class Testing 2 / 47

24 The Challenge of Software Testing Correctness Requirements Development Process 3 / 47

25 A little joke A mathematician, a physicist, and an engineer are told: All odd numbers are prime. The mathematician says, That s silly, nine is a non-prime odd number. The physicist says, Let s see, 3 is prime, 5 is prime, 7 is prime looks like it s true. The engineer says, Let s see, 3 is prime, 5 is prime, 7 is prime, 9 is prime, 11 is prime looks like it s true. 4 / 47

26 Becoming more serious The little joke illustrates us the following challenges in testing: oracle problem: How to predict the outcome of a test-case? incompleteness: exhaustive testing is usually infeasable. How much testing is needed? test case selection: Are the chosen test cases adequate? correctness of test cases: Are the test cases correctly designed? 5 / 47

27 Testing and Correctness Testing can only demonstrate the presence of errors, not their absence (E. Dijkstra, 1972) Reason: discrete nature of software Testing of all possible inputs would be necessary Exhaustive testing is infeasible Representative test cases have to be selected (testing strategy) 6 / 47

28 Testing and Requirements Quality of testing strongly depends on the quality of requirement documents Qualit"at des Testens h"angt stark von Qualit"at der Anforderungsdokumente ab. Requiremens that are not documented cannot be tested systematically 60 % of all bugs in the requirements phase are due to missing requirements Requirements written in natural language are often incomplete ambiguous inconsistent Formal specifications and formal models help! 7 / 47

29 Testing in the Development Process In the classical waterfall model testing is the last phase. Danger: if project is delayed, testing gets cut Dramatic: most of the large SW projects overrun their deadline Consequence: the test cases have to be designed as early a possible, as soon as the SW requirements are known. this lead to agile software processes High testing costs! Test automation is needed! 1/3 1/2 of the resources in big projects 8 / 47

30 Self test Write a set of test cases that you feel would adequately test this program (Myers, The Art of Software Testing): The program reads three integer values from an input stream. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral. 9 / 47

31 Self test: questions Do you have... 1 a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid. 2 a test case that represents a valid equilateral triangle? 3 a test case that represents a valid isosceles triangle? 4 at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))? 5 a test case in which one side has a zero value? 6 a test case in which one side has a negative value? 7 a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third? 10 / 47

38 Self test: questions (cont.) 8 at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))? 9 a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4)) 10 at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))? 11 a test case in which all sides are zero? 12 at least one test case specifying noninteger values? 13 at least one test case specifying the wrong number of values? 14 for each test case the expected outputs specified? 11 / 47

45 Naming the Wrong IEEE Standard Glossary of Software Engineering Terminology: Mistake: a human action that produces an incorrect result. Example: an incorrect action taken by the programmer. Fault: an incorrect step, process, or data definition in a computer progra (also defect, bug). Error: the difference between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition. Failure: the inability of a system or component to fulfill its required functions within specified performance requirements. Dijkstra was against calling it a bug. 12 / 47

46 Naming the Wrong: Example Consider a statement x = y + x; By a mistake a programmer changes it to x = y x, thus characterising a fault. Executing the fault with x = 0 does not lead to an error, and consequently not to an observable failure. 13 / 47

47 Testing, Test, Test Case testing: the act of designing, debugging, and executing test cases.. test: A test is the act of exercising software with test cases. A test has two distinctive goals: to find failures or to demonstrate correct execution. test case: A test case has an identity and is associated with program behaviour. A test case also has a set of inputs and a list of expected outputs. The output portion of a test case is frequently overlooked, which is unfortunate, because it is often the hard part. 14 / 47

48 Black & White Two fundamental approaches are used to identify test cases: functional testing: functional testing is based on the view that any program can be considered to be a function that maps values from its input domain to values in its output range (black box testing). Test cases are identified using the function s specification. structural testing: based on how the function is actually implemented (white box testing). 15 / 47

49 Why We Test validation: the process of evaluating an object to demonstrate that it meets the user requirements. verification: in the ANSI/IEEE Std three possible definitions of verification are given: 1 the process of reviewing, inspecting, testing, etc., if objects, processes, services, documents satisfy the specified requirements. 2 the process of evaluating if an object in a given phase in the software lifecycle, meets the requirements established in the previous phase. 3 formal correctness proofs of programs. falsification: the process of evaluating an object to demonstrate that it does not meet requirements. 16 / 47

50 When We Test The following kinds of tests can be distinguished during sw-lifecycle: unit tests: testing of units: modules, classes, components integration tests: tests that explore the interaction and consistency of successfully tested components. system tests: testing of the whole system in order to explore behaviors that can t be done by unit or integration testing: performance, data integrity, storage management, security, reliability. acceptance tests: does the system satisfy the requirements? Part of the contract. regression tests: tests after changes to prevent unintended changes. 17 / 47

51 The V-model 18 / 47

52 Functional Testing: Programs as Functions In functional testing, a program is considered as a function p : Input Output mapping values from its domain (Input) to values in its range (Output). Types permit the definitions of domain and range. Preconditions restrict the domain further. Postconditions restrict the range. 19 / 47

53 Boundary Value Analysis Focuses on the boundary of input space (domain) to identify test cases. Rationale: errors tend to occur near the extreme values of an input variable. Strategy: input values at 1 minimum 2 just above minimum 3 a nominal value 4 just below maximum 5 maximum 20 / 47

54 Boundary Value Analysis: 2 Input Variables Boundary value test cases for a function of two variables i1 and i2: i2 d c a b i1 Single fault assumption! (no relation between parameters) 21 / 47

55 Generalizing Boundary Value Analysis Number of test cases: 4n+ 1 for a function with n input variables. Different ranges: Triangle problem: e.g. min = 1, max = MAXINT Date input: date corresponds to 3 input variables; e.g. min = , max = Boolean input: does not make sense! Limitations: only good when program is a function of several independent variables that represent bounded physical variables. good: temperature, coordinates bad: date, pin code, telephone number 22 / 47

56 Robustness Testing Exceeding the limits slightly: i2 d c a b i1 Most interesting with expected outputs not inputs! 23 / 47

57 Robustness Testing Examples Robustness testing forces attention on exception handling: Exceeding angle of attack of a plane: Will it stall? Exceeding load capacity of an elevator: Hopefully only a warning!? With strongly typed languages robustness testing may be very awkward: C leads to run-time errors. Exception handling (C#, Java) mandates robustness testing. Specifications: test cases such that pre(in) holds. 24 / 47

58 Worst-Case Testing Rejecting the single-fault assumption: i2 d c a Number of test cases: 5 n, for n input variables! b i1 25 / 47

59 Special Value Testing tester uses his domain knowledge, experience with similar programs, and information about soft spots to devise test cases. most widely practiced form. also called ad hoc testing. very dependent on the abilities of the tester! 26 / 47

60 Example: Commission Rifle 1 salesman sales rifle locks 2, stocks 3, and barrels 4. Locks costs $45, stocks $30, barrels $25. Has to sell at least one complete rifle per month. Has to sell at most 70 locks, 80 stocks, and 90 barrels per month. The gunsmith computes the salesman s commission: 10% on sales up to (and including) $1000, 15% on the next $800, and 20% on any sales in excess of $ Gewehr 2 Verschluss 3 Schaft 4 Lauf 27 / 47

61 Commission: Output Boundary Analysis Case Locks Stocks Barrels Sales ($) Comm ($) Comment output minimum output minimum output minimum output minimum midpoint border point border point border point border point border point border point border point midpoint 28 / 47

62 Commission: Output Boundary Analysis (cont.) Case Locks Stocks Barrels Sales ($) Comm ($) Comment border point border point border point border point border point border point border point midpoint output maximum output maximum output maximum output maximum 29 / 47

63 Random Testing Idea: Random number generator to pick test case values. Motivation: avoiding a form of bias in testing. Question: How many random test cases are sufficient? Answer by 1 coverage criteria 2 reliability models Pro: fully automatic test case generation. Con: more test cases needed to reach structural coverage. At least 2 decades of literature in academia (statistical testing methods). 30 / 47

64 Random Testing of Commission Test Cases 10% Commission 15% Commission 20% Commission avg. 1.01% 3.62% 95.37% 31 / 47

65 Random Testing of Triangle Test Cases NoTriangle Scalene Isosceles Equilateral avg % 47.87% 2.29% 0.01% Program generates random test cases until at least one of each output occurs. Here an upper limit of 200 has been chosen. 32 / 47

66 Software Reliability Engineering reliability = 1 Decide target reliability. number of failures logical unit 2 Operational profile: What use cases (functions) are often executed? 3 More tests for highly frequent use cases. 4 Additional tests for high-risk use cases. 5 Statistical failure models, and the failures detected provide answers, when the target reliability is reached. Book: John Musa, Software Reliability Engineering, 2nd. edition, / 47

67 Equivalence Class Testing Main idea: : Test hypothesis: classes of inputs show equal behaviour Select one test case from each equivalence class. Main motivation: adequate set of test cases (having a sense of completeness) avoiding redundancy! We distinguish: weak / strong equivalence class testing (single vs. multiple faults) normal / robust equivalence class testing 34 / 47

68 Equivalence Classes form a set partition. Definition: (Partition) Given a set A and a set of n subsets A 1, A 2,...,A n of A, the subsets form a partition of A iff and A 1 A 2 A n = A i, j {1,...,n} i j A i A j = {} 1st property provides a form of completeness, 2nd property ensures a form of nonredundancy. 35 / 47

69 Choosing Equivalence Classes Key to Strategy: Choice of equivalence relation. Deduced from the problem specification Triangle: one test case for equilateral (5, 5, 5), we don t expect much more from test cases, like (6, 6, 6), (7, 7, 7). often by guessing a likely implementation Formal interface specifications (contracts): choice of equivalence classes can be automated. 36 / 47

70 Weak Normal Equivalence Class Testing i2 d c a b i1 37 / 47

71 Strong Normal Equivalence Class Testing i2 d c a b i1 38 / 47

72 Weak Robust Equivalence Class Testing i2 d c a b i1 39 / 47

73 Strong Robust Equivalence Class Testing i2 d c a b i1 40 / 47

74 Triangle Example Note that four possible outpus can occur. We can use these to identify output (range) equivalence classes: 1 {(a, b, c) istriangle(a, b, c) = NoTriangle} 2 {(a, b, c) istriangle(a, b, c) = Scalene} 3 {(a, b, c) istriangle(a, b, c) = Isosceles} 4 {(a, b, c) istriangle(a, b, c) = Equilateral} 41 / 47

75 Triangle: Normal Equivalence Class Testing The four weak normal equivalence class test cases are: Test Case a b c Expected Output WN NoTriangle WN Scalene WN Isosceles WN Equilateral No valid subintervals of variables a, b, and c exists strong normal equivalence testing does not add test cases. 42 / 47

76 Triangle: Robust Equivalence Class Testing Let s assume the valid input data has been restricted to [0, 200], and error messages indicate if input is out of its range, then the additional weak robust equivalence class test cases are: Test Case a b c Expected Output WR Value a out of range! WR Value b out of range! WR Value c out of range! WR Value a out of range! WR Value b out of range! WR Value c out of range! 43 / 47

77 Triangle: Robust Equivalence Class Testing We show one corner of the cube of strong robust equivalence class test cases: Test Case a b c Expected Output WS Value a out of range! WS Value b out of range! WS Value c out of range! WS Value a, b out of range! WS Value b, c out of range! WS Value a, c out of range! WS Value a, b, c out of range! 44 / 47

78 Triangle: Input Equivalence Class Equivalence class testing is sensitive to the equivalence relation chosen. Equivalence classes based on the input domain: 1 {(a, b, c) a = b b = c} 2 {(a, b, c) a = b a c} 3 {(a, b, c) a b a = c} 4 {(a, b, c) b = c a b} 5 {(a, b, c) a b a c b c} 45 / 47

79 Triangle: Input Equivalence Class (cont.) Furthermore, we can apply the triangle property to see if they consitute a triange: 6 {(a, b, c) a b + c} 7 {(a, b, c) b a+c} 8 {(a, b, c) c a+b} 46 / 47

80 Triangle: Input Equivalence Class (cont.) We could be even more thorough and distinguish between x = y and x > y in the properties above: 6 {(a, b, c) a = b + c} 7 {(a, b, c) a > b + c} 8 {(a, b, c) b = a+c} 9 {(a, b, c) b > a+c} 10 {(a, b, c) c = a+b} 11 {(a, b, c) c > a+b} 47 / 47

82 Agenda 1 Functional testing strategies (cont.) Decision Tables 2 Structural testing strategies 2 / 18

83 Decision Tables Stub Rule 1 Rule 2 Rule 3,4 Rule 5 Rule 6 Rule 7, 8 c1 T T T F F F c2 T T F T T F c3 T F T F a1 X X X a2 X X a3 X X a4 X X 3 / 18

84 Decision Table for Triangle c1: a, b, c form a triangle F T T T T T T T T c2: a = b T T T T F F F F c3: a = c T T F F T T F F c4: b = c T F T F T F T F a1: NoTriangle X a2: Scalene X a3: Isosceles X X X a4: Equilateral X a5: Impossible X X X 4 / 18

85 Refined Decision Table for Triangle c1: a < b + c F T T T T T T T T T T c2: b < a + c F T T T T T T T T T c3: c < a + b F T T T T T T T T c4: a = b T T T T F F F F c5: a = c T T F F T T F F c6: b = c T F T F T F T F a1: NoTriangle X X X a2: Scalene X a3: Isosceles X X X a4: Equilateral X a5: Impossible X X X 5 / 18

86 Number of Rules Binary entries: 2 n rules (columns), n being the number of conditions c i don t care entries ( ) reduce the number of rules a rule counter can be established: Rules without don t care entry count as one rule Each don t care in a rule doubles the counter in this rule The rule counter for one table shall be 2 n 6 / 18

87 Decision Table with Rule Counter c1: a < b + c F T T T T T T T T T T c2: b < a + c F T T T T T T T T T c3: c < a + b F T T T T T T T T c4: a = b T T T T F F F F c5: a = c T T F F T T F F c6: b = c T F T F T F T F Rule Count: a1: NoTriangle X X X a2: Scalene X a3: Isosceles X X X a4: Equilateral X a5: Impossible X X X 7 / 18

88 Redundant Decision Table Rule 1 4 Rule 5 Rule 6 Rule 7 Rule 8 Rule 9 c1 T F F F F T c2 T T F F F c3 T F T F F a1 X X X X a2 X X X a3 X X X X X 8 / 18

89 Inconsistent Decision Table Rule 1 4 Rule 5 Rule 6 Rule 7 Rule 8 Rule 9 c1 T F F F F T c2 T T F F F c3 T F T F F a1 X X X a2 X X X X a3 X X X X Regel 1-4 und 9 sind inkonsistent. Die Entscheidungstabelle ist nichtdeterministisch. 9 / 18

90 Guidelines 1 Decision table technique is for applications with prominent if-then-else-logic logical relations between input variables cause-effect relations between input and output 2 Decision tables get quickly inconvenient if logic is complex 3 Several refinement iterations help: Start with simple conditions and then refine them! 10 / 18

91 Structural Testing Strategies A software testing technique whereby explicit knowledge of the internal workings of the item being tested are used to select the test data Based on source code. White-box testing Technology available since the mid-1970s. Most techniques use program graphs. Pro: Amenable to rigorous definitions, mathematical analysis, and precise measurement. Con: Cannot test missing parts, e.g. a missing branch. 11 / 18

92 Program Graph Definition: Given a program written in an imperative programming language, its program graph is a directed graph in which nodes are statement fragments, and edges represent flow of control. 12 / 18

93 Program Graph Example 1 public class Program2 : Program // not very elegant implementation of Triangle 2 { 3 new public static string Triangle(int a, int b, int c) 4 { 5 bool istriangle = false; 6 string msg = ""; 7 8 if (a <= c - b a <= b - c b <= a - c) 9 istriangle = false; 10 else 11 istriangle = true; if (istriangle) 14 { 15 if (a == b && b == c) 16 msg = "equilateral"; 17 else 18 if (a == b b == c a == c) 19 msg = "isosceles"; 20 else 21 msg = "scalene"; 22 } 23 else 24 msg = "no triangle"; return msg; 27 } 28 } 13 / 18

94 Program Graph Example (cont.) Node labels denote line numbers. 14 / 18

95 DD-Paths Definition: A chain is a path in a graph, in which the initial and terminal nodes are distinct, and every interior node has indegree 1 and outdegree 1. Definition: A DD-Path (decision-to-decision) is a chain in a program graph such that one of the following cases hold: 1 it consists of a single node with indeg = 0 2 it consists of a single node with outdeg = 0 3 it consists of a single node with indeg 2 or outdeg 2 4 it consists of a single node with indeg = 1 and outdeg = 1 5 it is a maximal chain of length 1 15 / 18

96 DD-Path Graph Definition: Given a program written in an imperative language, its DD-Path Graph is the directed graph in which nodes are DD-Paths of its program graph, and edges represent control flow between successor DD-Paths. Today it is known as control flow graph 16 / 18

97 Control Flow Graph / 18

98 Test Coverage Metrics Metric C 0 C 1 C 2 C d C MCC C i k C inf Description of Coverage Every statement Every DD-Path (predicate outcome) C 1 coverage + loop coverage C 1 coverage + Every dependent pair of DD-Paths Multiple condition coverage Every program path that contains up to k repetitions of a loop (usually k = 2) All possible execution paths 18 / 18

99 TU Graz Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institut für Softwaretechnologie(IST) TU Graz Summer Term /44

100 TU Graz Agenda 1 Symbolic Execution 2 Concolic Execution 2/44

101 TU Graz Literature: Symbolic Execution James C. King. Symbolic Execution and Program Testing, Communications of the ACM, Volume 19, Issue 7. July 1976, Pages /44

102 TU Graz Symbolic Execution Instead of normal program inputs one supplies symbols representing arbitrary values. Execution proceeds like normal execution, except that values may be symbolic formulas over the input values. Interesting: symbolic execution of branching statements(e.g. if-statements) A technique between testing and formal proofs. Testing:executionwithsomeconcreteinputvalues Proofs:noexecution 4/44

103 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { 3 int x,y,z; 4 x = a + b; 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 5/44

104 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) a=α a,b = α b,c = α c 2 { 3 int x,y,z; 4 x = a + b; 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 6/44

105 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) a=α a,b = α b,c = α c 2 { 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 7/44

106 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) a=α a,b = α b,c = α c 2 { 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = α a +α b,y =0,z =0 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 8/44

107 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) a=α a,b = α b,c = α c 2 { 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = α a +α b,y =0,z =0 5 y = b + c; x = α a +α b,y = α b +α c,z =0 6 z = x + y b; 7 return z; 8 } 9/44

108 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) a=α a,b = α b,c = α c 2 { 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = α a +α b,y =0,z =0 5 y = b + c; x = α a +α b,y = α b +α c,z =0 6 z = x + y b; x = α a +α b,y = α b +α c,z = α a +α b +α c 7 return z; 8 } 10/44

109 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) a=α a,b = α b,c = α c 2 { 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = α a +α b,y =0,z =0 5 y = b + c; x = α a +α b,y = α b +α c,z =0 6 z = x + y b; x = α a +α b,y = α b +α c,z = α a +α b +α c 7 return z; return α a +α b +α c 8 } 11/44

110 TU Graz Bsp. Min 1 int min(int x, int y) 2 { 3 int z; 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 12/44

111 TU Graz Path Condition Symbolic execution of if-statement requires path condition(pc). pc is a Boolean expression over symbolic inputs. pc never contains program variables! pc is a conjunction of branching conditions. 13/44

112 TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 14/44

113 TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; x = α x,y = α y,z =0,pc =true 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 15/44

114 TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; x = α x,y = α y,z =0,pc =true 4 if(x<y) Case1:x = α x,y = α y,z =0,pc = α x < α y 5 z=x; 6 else 7 z=y; 8 return z; 9 } 16/44

115 TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; x = α x,y = α y,z =0,pc =true 4 if(x<y) Case1:x = α x,y = α y,z =0,pc = α x < α y 5 z=x; x = α x,y = α y,z = α x,pc = α x < α y 6 else 7 z=y; 8 return z; 9 } 17/44

116 TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; x = α x,y = α y,z =0,pc =true 4 if(x<y) Case1:x = α x,y = α y,z =0,pc = α x < α y 5 z=x; x = α x,y = α y,z = α x,pc = α x < α y 6 else Case2:x = α x,y = α y,z =0,pc = α x < α y 7 z=y; 8 return z; 9 } 18/44

117 TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; x = α x,y = α y,z =0,pc =true 4 if(x<y) Case1:x = α x,y = α y,z =0,pc = α x < α y 5 z=x; x = α x,y = α y,z = α x,pc = α x < α y 6 else Case2:x = α x,y = α y,z =0,pc = α x < α y 7 z=y; x = α x,y = α y,z = α y,pc = α x < α y 8 return z; 9 } 19/44

118 TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; x = α x,y = α y,z =0,pc =true 4 if(x<y) Case1:x = α x,y = α y,z =0,pc = α x < α y 5 z=x; x = α x,y = α y,z = α x,pc = α x < α y 6 else Case2:x = α x,y = α y,z =0,pc = α x < α y 7 z=y; x = α x,y = α y,z = α y,pc = α x < α y 8 return z; return(z = α x α x < α y ) (z = α y α x < α y ) 9 } 20/44

119 TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { 3 return 4 min( 5 min(a,b), 6 c 7 ); 8 } 21/44

120 TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), 6 c 7 ); 8 } 22/44

121 TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), (= α a pc = α a < α b ) (= α b pc = α a α b ) 6 c 7 ); 8 } 23/44

122 TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), (= α a pc = α a < α b ) (= α b pc = α a α b ) 6 c c = α c 7 ); 8 } 24/44

123 TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), (= α a α a < α b ) (= α b α a α b ) 6 c c = α c 7 ); (= α a pc = α a < α b α a < α c ) 8 } 25/44

124 TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), (= α a α a < α b ) (= α b α a α b ) 6 c c = α c 7 ); (= α a pc = α a < α b α a < α c ) 8 } (= α b pc = α a α b α b < α c ) 26/44

125 TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), (= α a α a < α b ) (= α b α a α b ) 6 c c = α c 7 ); (= α a pc = α a < α b α a < α c ) 8 } (= α b pc = α a α b α b < α c ) 9 (= α c pc = α a < α b α a α c ) 27/44

126 TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), (= α a α a < α b ) (= α b α a α b ) 6 c c = α c 7 ); (= α a pc = α a < α b α a < α c ) 8 } (= α b pc = α a α b α b < α c ) 9 (= α c pc = α a < α b α a α c ) 10 (= α c pc = α a α b α b α c ) 28/44

127 TU Graz Symbolic Execution Tree Symbolic execution forcs at each if-statement: α a < α b T F α a < α c α b < α c T F T F min = α a min = α c min = α b min = α c 29/44

128 TU Graz From Path Conditions to Concrete Test Cases Calculation:pc 2 =true α a < α b (α a < α c ) Path condition represents equivalence class for all concrete values taking a path. Concrete test cases: find concrete values satisfying pc e.g.forpc2:a =0,b =1,c =0 Result: path coverage 30/44

129 TU Graz Limitations infinite number of paths(loops) infeasible paths(pc = false) limitations of solvers and theorem provers all code must be accessible(white-box) only for single-threaded programs 31/44

130 TU Graz Concolic Execution Concolic = Concrete + Symbolic also called Dynamic Symbolic Execution(Microsoft) Concrete and Symbolic Execution in parallel Due to concrete execution: Noinfeasablepaths! Integrationofblack-boxcomponentspossible. 32/44

131 TU Graz Concolic Execution Loop Algorithm 1 Covered := {} coveredsetofinputs 2 selectnewinputi Covered stopifnoonefound 3 execute Program(i) and record path condition C C(i) holds 4 Covered :=Covered {i C(i)} 5 goto2 Newinputi Coveredafterniterations: solve C i1 C in Stopwhen C i1 C in =false 33/44

132 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { 3 int x,y,z; 4 x = a + b; 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 34/44

133 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { a=(α a,1),b = (α b,2),c = (α c,3) 3 int x,y,z; 4 x = a + b; 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 35/44

134 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { a=(α a,1),b = (α b,2),c = (α c,3) 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 36/44

135 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { a=(α a,1),b = (α b,2),c = (α c,3) 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = (α a +α b,3),y = (0,0),z = (0,0) 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 37/44

136 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { a=(α a,1),b = (α b,2),c = (α c,3) 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = (α a +α b,3),y = (0,0),z = (0,0) 5 y = b + c; x = (α a +α b,3),y = (α b +α c,5),z = (0,0) 6 z = x + y b; 7 return z; 8 } 38/44

137 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { a=(α a,1),b = (α b,2),c = (α c,3) 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = (α a +α b,3),y = (0,0),z = (0,0) 5 y = b + c; x = (α a +α b,3),y = (α b +α c,5),z = (0,0) 6 z = x + y b; x = (α a +α b,3),y = (α b +α c,5), 7 z = (α a +α b +α c,6) 8 return z; 9 } 39/44

138 TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { a=(α a,1),b = (α b,2),c = (α c,3) 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = (α a +α b,3),y = (0,0),z = (0,0) 5 y = b + c; x = (α a +α b,3),y = (α b +α c,5),z = (0,0) 6 z = x + y b; x = (α a +α b,3),y = (α b +α c,5), 7 z = (α a +α b +α c,6) 8 return z; return(α a +α b +α c,6) 9 } 40/44

139 TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 41/44

140 TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; x = (α x,0),y = (α y,0),z = (0,0),pc 1 =true 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 41/44

141 TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; x = (α x,0),y = (α y,0),z = (0,0),pc 1 =true 4 if(x<y) (0 <0) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 41/44

142 TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; x = (α x,0),y = (α y,0),z = (0,0),pc 1 =true 4 if(x<y) (0 <0) 5 z=x; 6 else x = (α x,0),y = (α y,0),z = (0,0),pc 1 = (α x < α y ) 7 z=y; 8 return z; 9 } 41/44

143 TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; x = (α x,0),y = (α y,0),z = (0,0),pc 1 =true 4 if(x<y) (0 <0) 5 z=x; 6 else x = (α x,0),y = (α y,0),z = (0,0),pc 1 = (α x < α y ) 7 z=y; x = (α x,0),y = (α y,0),z = (α y,0),pc 1 = (α x < α y ) 8 return z; 9 } 41/44

144 TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; x = (α x,0),y = (α y,0),z = (0,0),pc 1 =true 4 if(x<y) (0 <0) 5 z=x; 6 else x = (α x,0),y = (α y,0),z = (0,0),pc 1 = (α x < α y ) 7 z=y; x = (α x,0),y = (α y,0),z = (α y,0),pc 1 = (α x < α y ) 8 return z; returnz= (α y,0),pc 1 = (α x < α y ) 9 } 41/44

145 TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; x = (α x,0),y = (α y,0),z = (0,0),pc 1 =true 4 if(x<y) (0 <0) 5 z=x; 6 else x = (α x,0),y = (α y,0),z = (0,0),pc 1 = (α x < α y ) 7 z=y; x = (α x,0),y = (α y,0),z = (α y,0),pc 1 = (α x < α y ) 8 return z; returnz= (α y,0),pc 1 = (α x < α y ) 9 } FindnewinputvaluessuchthatC = pc 1 = α x < α y holdsand execute again, e.g. min(0, 1) 41/44

146 TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 42/44

147 TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; x = (α x,0),y = (α y,1),z = (0,0),pc 2 =true 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 42/44

148 TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; x = (α x,0),y = (α y,1),z = (0,0),pc 2 =true 4 if(x<y) (0 <1): 5 z=x; 6 else 7 z=y; 8 return z; 9 } 42/44

149 TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; x = (α x,0),y = (α y,1),z = (0,0),pc 2 =true 4 if(x<y) (0 <1): pc 2 = α x < α y 5 z=x; 6 else 7 z=y; 8 return z; 9 } 42/44

150 TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; x = (α x,0),y = (α y,1),z = (0,0),pc 2 =true 4 if(x<y) (0 <1): pc 2 = α x < α y 5 z=x; x = (α x,0),y = (α y,1),z = (α x,0),pc 2 = α x < α y 6 else 7 z=y; 8 return z; 9 } 42/44

151 TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; x = (α x,0),y = (α y,1),z = (0,0),pc 2 =true 4 if(x<y) (0 <1): pc 2 = α x < α y 5 z=x; x = (α x,0),y = (α y,1),z = (α x,0),pc 2 = α x < α y 6 else 7 z=y; 8 return z; returnz= (α x,0),pc 2 = α x < α y 9 } 42/44

152 TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; x = (α x,0),y = (α y,1),z = (0,0),pc 2 =true 4 if(x<y) (0 <1): pc 2 = α x < α y 5 z=x; x = (α x,0),y = (α y,1),z = (α x,0),pc 2 = α x < α y 6 else 7 z=y; 8 return z; returnz= (α x,0),pc 2 = α x < α y 9 } FindingnewinputvaluessuchthatC = pc 1 pc 2 =falseisnot possible, hence terminate. 42/44

153 TU Graz Limitations infinite number of paths: upper bounds limitations of solvers and theorem provers for single-threaded programs 43/44

154 TU Graz Tools Cute, jcute Pex(see exercise) PathCrawler 44/44

156 Agenda 1 Spezifikationen Beispiel Eigenschaften von Spezifikationen Contracts 2 Invarianten 2 / 23

157 Oracle Definition: any means used to predict the outcome of a test. Kind of oracles: specification existing test cases (regression testing) previous version of the program prototype domain expert actual program (dangerous!) 3 / 23

158 Specifications fix the criteria to accept an implementation. are a basis for a contract between user and supplier. define what should be achieved, but not how. greatest possible freedom for developer. programs are not suitable as specifications! 4 / 23

159 Specification vs. Implementation Specification vs. Implementations abstract concrete what how 5 / 23

160 Specification Implementation 1 The specification serves as starting point for the design of the implementation. 2 The specification is the bases for any verification: 1 for proving that an implementation satisfies a specification 2 for the generation of test cases 3 as a test oracle 6 / 23

161 Programs as Specifications? public static double P (double tfactorial) { double x1=0; double x0=tfactorial/2; double a=tfactorial; boolean finished=false; while (finished==false) { x1=(x0+(a/x0))/2; if (x1>x0){ if ((x1-x0)< ){ finished=true;} } else if (x0>x1){ if ((x0-x1)< ){ finished=true;} } x0=x1; } return x1; } 7 / 23

162 Spezifikation Vers. 1 Example (Postcondition) Postcondition (Nachbedingung): post P (in, out) = df out 2 = in 8 / 23

163 Spezifikation Vers. 2 Not all inputs are valid! Example (Pre-Postcondition) Postcondition: post P (in, out) = df out 2 = in Precondition (Vorbedingung): pre P (in) = df in 0 9 / 23

164 Spezifikation Vers. 3 We are only interested in positive results! Example (Pre-Postcondition) Postcondition: post P (in, out) = df out 2 = in out 0 Precondition: pre P (in) = df in 0 10 / 23

165 Spezifikation Vers. 4 In practice, the result is an approximations! Example (Pre-Postcondition) Postcondition: post P (in, out) = df abs(in out 2 ) ǫ in out 0 Precondition: ǫ... error constant abs... absolute value pre P (in) = df in 0 11 / 23

166 Satisfiability In order to be a meaningful specification, one solution must exist. Thus, one program/implementation must exist that satisfies the specification. We also say that there is a model of the specification. Or, the specification is realisable (implementable). 12 / 23

167 Underspecification Specifications need not to determine unique results. Examples are flight reservation systems, compilers. Underspecification is a form of abstraction. 13 / 23

168 Preciseness Specifications must be precise. That means that the set of solutions (models) must be defined. Underspecified does not mean imprecise. Distinguish between abstract and imprecise! 14 / 23

169 Design by Contract A method for software development Pre- and postconditions als contracts between classes and clients Pre- and postconditions are executable. History: Hoare-Logik (1969), VDM (70-ies), Eiffel (1992), JML (1999), C# Code Contracts (2008) 15 / 23

170 C# Code Contracts Example (sqrt) public static double sqrt2(double x) { Contract.Requires(x >= 0); Contract.Ensures( Math.Abs(x - (Contract.Result<double>() * Contract.Result<double>())) < ); return Math.Sqrt(x); } Vorbedingung: Requires (normale) Nachbedingung: Ensures 16 / 23

171 Java Modeling Language (JML) Example (sqrt) public final static double eps = ; //@ requires x >= 0.0; //@ ensures JMLDouble.approximatelyEqualTo(x, \result * \result, eps); public static double sqrt(double x) { return Math.sqrt(x); } Vorbedingung: requires (normale) Nachbedingung: ensures 17 / 23

172 Vorteile von Contracts zur Dokumentation (Spezifikation) von Klassen und Interfaces Kontrakt abstrakter als Code (kein Algorithmus!) Kontrakt kann maschinell geprüft werden (im Gegensatz zu informalen Kommentaren) Fehlersuche (debugging) Wartung Schuldzuweisung bei Fehlern: Client vs. Class Effizienz: Keine ineffizienten defensiven Überprüfungen (checks) im Code 18 / 23

173 Beispiel Effizienz Example (Binärsuche) public static int search(int[] a, int x) { Contract.Requires(a!= null); Contract.Requires(Contract.ForAll(1, a.length, i => a[i-1] <= a[i])); return Array.BinarySearch(a, x); } All-quantor ( ): ForAll Endliche Bereichsangabe f"ur i macht den Ausdruck ausf"uhrbar. Eine alternative Prüfschleife im Endprogramm wäre ineffizient: Laufzeit: logarithmisch linear 19 / 23

174 Modularität des Programmverstehens Wie kann man solchen Code verstehen? Example (Typischer OO Code)... source.close(); target.close(); getfile.setlastmodified( loc.modtime().gettime()); return modtime(); Möglichkeit 1: Lesen des Codes aller Methoden, lesen des Codes aller Methoden in den Methoden, lesen des... Erschwernis: Subtypen polymorphismus (dynamische Bindung) Möglichkeit 2: Lesen der Kontrakte aller Methoden 20 / 23

175 Invarianten Definition (Klasseninvariante, allgemein) Eine Klasseninvariante ist eine Eigenschaft des Klassenzustandes (Variablen, Felder), die immer nach Ausführen eines jeden Konstruktors, sowie vor und nach Ausführen einer jeden Methode gelten mu s. auch Dateninvariante oder kurz Invariante genannt. 21 / 23

176 Rolle von Invarianten Typeninvariante: schränkt einen Datentyp ein (z.b. Natürliche Zahlen) Repräsentationsinvariante: dokumentiert Designentscheidung der Datenrepräsentation (z.b. Sortierung einer Liste) Dokumentieren Beziehungen zwischen Variablen (Feldern). Invarianten werden durch Kapselung geschützt. (Repräsentations-)invarianten sind oft für die Korrektheit eines Algorithmus notwendig. 22 / 23

177 C# Beispiel Purse: Translated from JML [Burdy et al. 2005] 23 / 23

179 Agenda 1 Model-based Specifications Example: Alarm System 2 / 10

180 Model-based Specifications Motivation: Observation: pre- and postconditions are closely coupled to the inner data representation of a class. Advantage: Pecise description of the inner behaviour of the implementation. Precise documentation of the representation invariant. Disadvantage: When changing a variable of a class, also the specifications must be changed. The variables of a class must be declared before the specification. Not suitable for specifying abstract classes and interfaces! Conclusion We need abstractions of data representations models 3 / 10

181 Model-based Specification in C# Some restrictions apply: Unlike JML, the Code Contracts framework does not know model variables. Model variables are abstractions of one or more concrete variables Invariants cannot be checked on interfaces In C# we can use generic abstract classes for modelling the abstract state with (collections of) interfaces Using sets (ISet), sequences (IList), maps (IDictionary) to abstract from efficient implementation representations. 4 / 10

182 Anforderungen für ein Alarmsystem Eine chemische Fabrik ist mit einer Anzahl von Überwachungseinrichtungen ausgerüstet, welche abhängig vom Zustand der Fabrik Alarme auslösen können. Wird ein Alarm ausgelöst, mu s ein Experte angefordert werden. Experten haben verschiedene Qualifikationen um mit verschiedenen Alarmen umgehen zu können. 5 / 10

183 Anforderungen für ein Alarmsystem (cont.) R1 Es ist ein computerbasiertes System zu entwickeln, welches die Alarme in dieser Fabrik managed. R2 Vier Arten von Qualifikationen werden gebraucht um mit den Alarmen umzugehen. Diese sind elektrische, mechanische, biologische und chemische Qualifikationen. R3 Es müssen während allen festgelegten Zeitperioden Experten abrufbereit sein. R4 Jeder Experte hat eine Liste von Qualifikationen. 6 / 10

184 Anforderungen für ein Alarmsystem (cont.) R5 Jeder Alarm, der an das System gemeldet wird hat eine zugeordnete Qualifikation und eine Beschreibung des Alarms, die ein Experte verstehen kann. R6 Wann immer ein Alarm vom System empfangen wird, soll ein Experte mit der richtigen Qualifikation gefunden werden, so da s er angepaged werden kann. R7 Die Experten sollen die Systemdatenbank abfragen können, um zu überprüfen wann sie abrufbereit sein werden/müssen. R8 Es mu s möglich sein die Anzahl der abrufbaren Experten abzufragen. 7 / 10

185 Mögliche Klassen und Methoden Klassen (Typen) IAlarmSystem IQualification IAlarm IPeriod IExpert (Description) Methoden ExpertToPage ExpertIsOnDuty NumberOfExperts 8 / 10

186 Deklaration der Methoden R6: public int NumberOfExperts(IPeriod p); R7: public IPeriod[] ExpertIsOnDuty(IExpert e); R8: public IExpert ExpertToPage(IAlarm a, IPeriod p); 9 / 10

187 C# Example AlarmSystem.cs 10 / 10

189 Agenda 1 Äquivalenzklassen aus Disjunktiver Normalform 2 Bildung von Testfallfolgen 3 Testabdeckungen Testabdeckung von Finiten Zustandsmaschinen 2 / 27

190 DNF-Partitionierung Gegeben eine formale Spezifikation, z.b. pre(in) post(in, out) Rekursives Umschreiben durch Anwenden der Regel: a b (a b) ( a b) (a b) das ergibt eine Disjunktive Normalform (DNF). Logische Äquivalenzen ergeben neue Disjunktionen: a b a b Spezialwerte v eines Typs T können auch behandelt werden : x : T (x = v x v) x : T 3 / 27

191 DNF-Partitionierung (Forts.) Normale Äquivalenzklassen durch umschreiben von pre(in) post(in, out) Robustheits-Äquivalenzklassen durch umschreiben von pre(in) post(in, out) Achtung! Es wird die In-Output-Relation partitioniert, nicht nur der Eingabebereich. 4 / 27

192 Maximum Beispiel Ein Programm soll das Maximum von zwei ganzen Zahlen ausgeben. Wir können in Pseudonotation spezifizieren: max: Int Int Int max(x,y) as r post (r = x r = y) r x r y 5 / 27

193 Maximum Beispiel (Forts.) Umschreiben der Disjunktion ergibt die Postcondition: post ((r = x r = y) (r x r = y) (r = x r y)) r x r y 6 / 27

194 Maximum Beispiel (Forts.) Distributivgesetz auf zweimal ergibt post (r = x r = y r x r y) (r x r = y r x r y ) (r = x r y r x r y ) 7 / 27

195 Maximum Beispiel (Forts.) Nach Vereinfachung erhalten wir: post (r = x r = y) (r > x r = y) (r = x r > y) Das repräsentiert drei Äquivalenzklassen von Testfallen: 1 {(x, y, r) r = x r = y} 2 {(x, y, r) r > x r = y} 3 {(x, y, r) r = x r > y} 8 / 27

196 Testfallfolgen Für komplexe zustandsbasierte Systeme (OO), Eingabe-Ausgabestrategien sind nicht ausreichend: interner Zustand kann nicht direkt als Eingabe gesetzt werden (Kapselung), sondern nur durch Interaktionen. der Tester muss eine adäquate Folge von Aktionen finden, die den internen Zustand setzen, von dem eine Methode (Funktion) getestet werden soll. jede Funktion mag einfach sein, aber ihre Interaktionen sind es nicht. Typische Anwendungen: Unit-Testen von Objekt-Orientierten Systemen. Systemtesten von interaktiven Systemen. 9 / 27

197 Formale Technik 1 Wähle ein Modell für den internen Zustand. 2 Spezifikation jeder Methode durch Vor-Nachbedingungen 3 DNF Partitionierung jeder Methode 4 Konstruktion eines endlichen Zustandsautomaten (finite state machine): Extrahiere pro Methode zwei Constraints: Zustand davor und danach. Partitionierung des Zustands Transitionen sind partitionierte Sub-Methoden 5 Wähle Folgen, die den Zustandsautomaten abdecken 10 / 27

198 Scheduler Beispiel Ein Prozessscheduler behandelt Prozesse, die in drei möglichen Zuständen sind: ready um am Prozessor zu laufen, waiting um ready zu werden, optional, einen einzigen Prozess der active ist. Diese Prozesse werden durch eine eindeutige Pid gekennzeichnet. Ein Prozess kann nicht gleichzeitig ready und waiting sein. Der aktive Prozess ist weder ready noch waiting Es muss einen active Prozess geben, wann immer es einen ready Prozess gibt, der laufen kann. 11 / 27

199 Zustandsmodell scheme scheduler = class type Pid = Nat value nil: Nat variable active : Pid, ready : Pid-set, waiting: Pid-set 12 / 27

200 Zustandsinvariante axiom ready waiting = {}, active (ready waiting), nil (ready waiting), active = nil ready = {} 13 / 27

201 Schedule value schedule: Pid-set Pid schedule(ps) as p post p ps pre ps {}, 14 / 27

202 Init Init: Unit write any Unit Init() post active = nil ready waiting = {}, 15 / 27

203 New New: Pid write waiting Unit New(p) post waiting = waiting old {p} pre p active p (ready waiting), 16 / 27

204 Ready Ready: Pid write active, waiting, ready Unit Ready(q) post waiting = waiting old \ {q} active old = nil (ready = ready old active = q) active old nil (ready = ready old {q} active = active old ) pre q waiting, 17 / 27

205 Swap Swap: Unit write any Unit Swap() post waiting = waiting old {active old } ready old = {} (active = nil ready = {}) ready old {} (active = schedule(ready old ) ready = ready old \ {active}) pre active nil, 18 / 27

206 Konstruktion des Automaten 1 Analysiere die Äquivalenzklassen an allen Methoden (Funktionen) und leite Sub-methoden anhand dieser Klassen ab. Hier unterscheiden wir ob active = nil und ob ready oder waiting leer sind. 2 Von jeder partitionierten Methode erhält man die Vorher- Nachherzustände durch existientielles Quantifizieren der Variablen, die nicht in Betracht gezogen werden. 3 DNF Partitionierung der Zustände aus Schritt 2. Das ergibt die Zustände des Automaten. 4 Konstruiere den Automaten durch Auflösen der Transitionen gegen die Zustände. 19 / 27

207 Partitioned New New1: Pid write waiting Unit New1(p) post waiting = waiting old {p} pre active = nil ready = {} p waiting, New2: Pid write waiting Unit New2(p) post waiting = waiting old {p} pre p active active nil p (ready waiting), 20 / 27

208 Partitioned Ready Ready1: Pid write active, waiting, ready Unit Ready1(q) post waiting = waiting old \ {q} ready = ready old active = q pre q waiting active = nil, Ready2: Pid write active, waiting, ready Unit Ready2(q) post waiting = waiting old \ {q} ready = ready old {q} active = active old pre q waiting active nil, 21 / 27

209 Partitioned Swap Swap1: Unit write any Unit Swap1() post waiting = waiting old {active old } active = nil ready = {} pre active nil ready = {}, Swap2: Unit write any Unit Swap2() post waiting = waiting old {active old } active ready old ready = ready old \ {active} pre active nil ready {}, 22 / 27

210 Berechnung der Zustände DNF Analyse der Invariante Existentielles Quantifizieren (hiding) ergibt den 1 Vorherzustand: variable 1,...,variable n pre post 2 Nachherzustand: variable1 old,...,variablen old pre post Danach DNF-Analyse aller Zustände ergibt Äquivalenzklassen der Zustände. 23 / 27

211 Zustände des Automaten Variables State 1 State 2 State 3 State 4 State 5 State 6 active = nil = nil nil nil nil nil ready = {} = {} = {} = {} = {} = {} waiting = {} = {} = {} = {} = {} = {} 24 / 27

212 Konstruktion der Transitionen Aus den Zuständen und partitionierten Methoden, kann man die Transitionen konstruieren. Eine Transition existiert von Zustand St1 nach St2 genau dann, wenn die folgende Bedingung wahr ist: s, s old, in, out : St1(s old ) pre i (s old, in) post i (s old, in, s, out) St2(s) wobei die Zustandsvariablen s und s old den gesamten Variablenvektor repr asentieren. s entspricht \old(s) in JML 25 / 27

213 Der endliche Automat für den Scheduler New1 Swap1 New2 Swap2 Swap2 Init Swap2 1 2 Ready1 3 4 Ready2 5 6 New1 New2 New2 Swap1 Swap2 Ready2 Ready1 Ready2 New2 26 / 27

214 Testabdeckung des Zustandsautomaten Einige mögliche Abdeckungen: 1 Alle Zustände abdecken. 2 Alle Methoden abdecken. 3 Alle Transitionen abdecken. 4 Alle aufeinanderfolgenden Paare von Transitionen. 5 Alle mögliche Pfade mit n Wiederholungen. 27 / 27

216 Agenda 1 Conformance Testing Properties 2 Labelled Transition Systems Equivalence Preorder ioco Examples 3 Test generation 2 / 31

217 Input/Output Conformance Relation Jan Tretmans Based on IO-Labeled Transition Systems 3 / 31

218 Input/Output Conformance Relation Jan Tretmans Based on IO-Labeled Transition Systems 3 / 31

219 Conformance Testing 4 / 31

220 Conformance Testing - Soundness Test suite is sound: conformance all tests pass 5 / 31

221 Conformance Testing - Soundness Test suite is sound: conformance all tests pass 5 / 31

222 Conformance Testing - Exhaustiveness Test suite is exhaustive: conformance all tests pass 6 / 31

223 Conformance Testing - Exhaustiveness Test suite is exhaustive: conformance all tests pass 6 / 31

224 Conformance Testing - Completeness Test suite is complete: conformance all tests pass 7 / 31

225 Conformance Testing - Completeness Test suite is complete: conformance all tests pass 7 / 31

226 Conformance Testing with ioco System (Implementation) is modeled as IOTS weakly input enabled Specification is an IOLTS possibly incomplete possible non-deterministic τ!g1 τ!g1?g2?g2!g3!g3!g1 8 / 31

227 Conformance Testing with ioco System (Implementation) is modeled as IOTS weakly input enabled Specification is an IOLTS possibly incomplete possible non-deterministic τ!g1 τ!g1?g2?g2!g3!g3!g1 8 / 31

228 Input Output Labeled Transition Systems Input Output Labeled Transition System An IOLTS is an LTS M = (Q M, A M, M, q0 M) with Q M a finite set of states A M = A M I A M O {τ} where A M I and A M O are input and output alphabets τ A M I A M O is an unobservable, internal action M Q M A M Q M is the transition relation q0 M QM is the initial state. QUESTION? What means: The implementation conforms-to the specification? 9 / 31

229 Input Output Labeled Transition Systems Input Output Labeled Transition System An IOLTS is an LTS M = (Q M, A M, M, q0 M) with Q M a finite set of states A M = A M I A M O {τ} where A M I and A M O are input and output alphabets τ A M I A M O is an unobservable, internal action M Q M A M Q M is the transition relation q0 M QM is the initial state. QUESTION? What means: The implementation conforms-to the specification? 9 / 31

230 How to relate 2 LTSs? Equivalence Relations (=) Bisimulation Trace Equivalence Testing Equivalence... Preorder Relations ( ) Trace Preorder Testing Preorder... Input-Output Relations... ioconf ioco / 31

231 (Weak) Bisimulation Two states are bisimilar iff they simulate each other and go to states which are bisimilar Bisimulation is not suited for testing! 11 / 31

232 Trace Equivalence A trace is an observable sequence of actions Two states are trace equivalent iff they have the same traces Trace equivalence is the weakest notion of conformance 12 / 31

233 Equivalence vs. Preorder Relations Equivalence Relation (R) reflexive (srs) symmetric: irs sri transitive: irs srt irt Preorder Relations ( ) NOT necessarily antisymmetric: irs i s s i simplifies testing e.g.: Trace Preorder i tr s traces(i) traces(s) 13 / 31

234 Some Notations: Transitions q a M q = df (q, a, q ) M q ǫ q = df (q = q ) (q τ M q 1 q n 1 τ M q ) q a q = df q 1, q 2 : q ǫ M q 1 a M q 2 ǫ M q 14 / 31

235 Some Notations: Quiescence δ is used to represent quiescence q δ q = df q is a quiescent state. Quiescent state = no edge labeled with an output or an internal action 15 / 31

236 Some Notations: Quiescence δ is used to represent quiescence q δ q = df q is a quiescent state. Quiescent state = no edge labeled with an output or an internal action 15 / 31

237 Some Notations: Suspension Automaton (M) = (Q M, A (M), (M), q M 0 ) where: A (M) = A M {δ} with δ A (M) O (M) is obtained from M by adding loops q δ q for each quiescent state 16 / 31

238 Some Notations: After q after M σ = df {q q σ M q } Q after M σ = df q Q (q after M σ). 17 / 31

239 Some Notations: Out Out M (q) = df {a A M O q a M } Out M (Q) = df q Q (Out M(q)) 18 / 31

240 ioco Definition: ioco Let IUT = (Q IUT, A IUT, IUT, q0 IUT ) be weakly input enabled with A IUT = A IUT I A IUT O {τ} and S = ( Q S, A S, S, q0) S be strongly responsive with A S = A S I A S O {τ}. Then: IUT ioco S = df σ traces( (S)) : Out IUT ( (IUT) after IUT σ) Out S ( (S) after S σ). IUT ioco S iff outputs (and quiescences) of the IUT are possible in S after an arbitrary suspension trace of S. 19 / 31

241 ioco Definition: ioco Let IUT = (Q IUT, A IUT, IUT, q0 IUT ) be weakly input enabled with A IUT = A IUT I A IUT O {τ} and S = ( Q S, A S, S, q0) S be strongly responsive with A S = A S I A S O {τ}. Then: IUT ioco S = df σ traces( (S)) : Out IUT ( (IUT) after IUT σ) Out S ( (S) after S σ). IUT ioco S iff outputs (and quiescences) of the IUT are possible in S after an arbitrary suspension trace of S. 19 / 31

242 P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 20 / 31