Quality Assurance in Software Development

Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institute for Software Technology Graz University of Technology Austria Summer Term 2013 1 / 19

Agenda 1 Organisation 2 Contents 3 Quality of SW 2 / 19

Quality Assurance in Summer Term 2013 Lecture: Tue 11:15 12:00 (i12) Exercise: Tue 12:00 12:45 (i12) Optional: Fri 14:15 15:00 (i12) Registration via tugonline 14.3. strict! Two student assistants: Sandra Fruhmann Martin Tappler 3 / 19

Additional Resources WWW: http://www.ist.tugraz.at/qs.html Newsgroup: tu-graz.lv.qs Email: Subject: [QS13] qs@ist.tugraz.at 4 / 19

Marking Marking iff registered on 15.3. (Dreamspark licence) Written exam (Klausur) : 25.6. (50 %). Three practical tasks (groups of 3): Task 1: publication 19.3. (10 %) Task 2: publication 16.4. (10 %) Task 3: publication 30.4. (15 %) Task 4: publication 14.5. (15 %) Oral interviews about tasks: after Task 3 Exam and exercises count each 50% Positive if > 50% total points 5 / 19

Marking: Nachklausur Date: Beginning of July After excused absence at exam e.g. due to illness with sick certificate (ärztlicher Bestätigung) work is no excuse (take leave!) For all negative! 6 / 19

What we will not cover? Quality management: Process management, e.g. Scrum Process improvement, e.g. ISO 9000 Test management, e.g. IBM Rational Quality Manager Version control, e.g. svn, git... 7 / 19

General Aims of the Course Getting familiar with quality assurance techniques Raising the awareness for quality of software Practising fault-based thinking Being able to distinguish process-oriented and technical quality assurance Challenging common quality criteria Realising the relations between the foundations of software engineering and quality assurance. 8 / 19

Specific Learning Targets and Skills Getting familiar with a professional IDE Writing good tests :( Generating good tests :) Specifying test oracles Using and understanding state-of-the-art test case generators Test coverage metrics "Don t write test cases, generate them!" (John Hughes) 9 / 19

Course Contents Introduction and motivation Challenges of software testing Coverage: Measuring the quality of test cases Testing with Concolic Execution Contracts as test oracles Model-based Testing Proof-based Development Tools (exercise): MS Visual Studio Ultimate 2010 MS Pex.NET Code Contracts MS Spec Explorer You will get a DreamSpark licence (registration)! 10 / 19

Quality? Today, often a sufficient level of correctness efficiency costs cannot be guaranteed. Vision: Development methods for SW with warranty. 11 / 19

Bugs Part of engineering jargon for many decades: Moth trapped in relay of Mark II (Hopper 1946) Little faults and difficulties (Edison 1878): Software bugs Relay #70 Panel F (moth) in relay. First actual case of bug being found. 12 / 19

Bugs Part of engineering jargon for many decades: Moth trapped in relay of Mark II (Hopper 1946) Little faults and difficulties (Edison 1878): Software bugs Definition A software bug is the common term used to describe an error, flaw, mistake, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways. (Wikipedia 2012) 12 / 19

Some Bugs Become Famous! Ariane 5 test flight (1996) out of control due to software failure controlled destruction! Loss of money and time satellites research (TU Graz) Dijkstra (EWD 1036): call it error, not bug a programmer created it 13 / 19

Binary search bug in Java Some Bugs Hide for a Long Time! JDK 1.5 library (2006) out of boundary access of large arrays due to integer overflow 9 years undetected Algorithm was proven correct! Programming Pearls [Bentley86, Bentley00] assuming infinite integers :( 1 public static 2 int binarysearch(int[] a,int key) 3 { 4 int low = 0; 5 int high = a.length - 1; 6 7 while (low <= high) { 8 int mid = (low + high) / 2; 9 int midval = a[mid]; 10 11 if (midval < key) 12 low = mid + 1; 13 else if (midval > key) 14 high = mid - 1; 15 else 16 return mid; // key found 17 } 18 return -(low + 1); // key not found 19 } Beware of bugs in the above code; I have only proved it correct, not tried it. [Knuth77] 14 / 19

Binary search bug in Java Some Bugs Hide for a Long Time! JDK 1.5 library (2006) out of boundary access of large arrays due to integer overflow 9 years undetected Algorithm was proven correct! Programming Pearls [Bentley86, Bentley00] assuming infinite integers :( 1 public static 2 int binarysearch(int[] a,int key) 3 { 4 int low = 0; 5 int high = a.length - 1; 6 7 while (low <= high) { 8 int mid = (low + high) >>> 1; 9 int midval = a[mid]; 10 11 if (midval < key) 12 low = mid + 1; 13 else if (midval > key) 14 high = mid - 1; 15 else 16 return mid; // key found 17 } 18 return -(low + 1); // key not found 19 } Beware of bugs in the above code; I have only proved it correct, not tried it. [Knuth77] 14 / 19

Risks for the Public due to SW The Risks Digest http://catless.ncl.ac.uk/risks/ 15 / 19

Edsger W. Dijkstra In academia, in industry, and in the commercial world, there is a widespread belief that computing science as such has been all but completed and that, consequently, computing has matured from a theoretical topic for the scientists to a practical issue for the engineers, the managers, and the entrepreneurs. [...]... then he characterises the software crisis and concludes: I would therefore like to posit that computing s central challenge, "How not to make a mess of it," has not been met. 16 / 19

Edsger W. Dijkstra (cont.) On the contrary, most of our systems are much more complicated than can be considered healthy, and are too messy and chaotic to be used in comfort and confidence. The average customer of the computing industry has been served so poorly that he expects his system to crash all the time, and we witness a massive worldwide distribution of bug-ridden software for which we should be deeply ashamed. (Communications of the ACM, Mar 2001) 17 / 19

Why? A possible explanation: Quality assurance requires a certain degree of (healthy) redundancy extra costs! (appr. 50% for critical systems) Examples of redundancy: process management documentation pair programming modelling test cases redundant components in fault-tolerant systems (accepted) 18 / 19

Demo: Triangle Example Writing unit tests in VS 2010 Code coverage in VS 2010 Automatic test case generation in Pex How good are the tests? 19 / 19

Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institute for Software Technology Graz University of Technology Austria Summer Term 2013 1 / 47

Agenda 1 The Challenge of Software Testing 2 Definitions 3 Functional testing strategies Boundary Value Testing Random Testing Equivalence Class Testing 2 / 47

The Challenge of Software Testing Correctness Requirements Development Process 3 / 47

A little joke A mathematician, a physicist, and an engineer are told: All odd numbers are prime. The mathematician says, That s silly, nine is a non-prime odd number. The physicist says, Let s see, 3 is prime, 5 is prime, 7 is prime looks like it s true. The engineer says, Let s see, 3 is prime, 5 is prime, 7 is prime, 9 is prime, 11 is prime looks like it s true. 4 / 47

Becoming more serious The little joke illustrates us the following challenges in testing: oracle problem: How to predict the outcome of a test-case? incompleteness: exhaustive testing is usually infeasable. How much testing is needed? test case selection: Are the chosen test cases adequate? correctness of test cases: Are the test cases correctly designed? 5 / 47

Testing and Correctness Testing can only demonstrate the presence of errors, not their absence (E. Dijkstra, 1972) Reason: discrete nature of software Testing of all possible inputs would be necessary Exhaustive testing is infeasible Representative test cases have to be selected (testing strategy) 6 / 47

Testing and Requirements Quality of testing strongly depends on the quality of requirement documents Qualit"at des Testens h"angt stark von Qualit"at der Anforderungsdokumente ab. Requiremens that are not documented cannot be tested systematically 60 % of all bugs in the requirements phase are due to missing requirements Requirements written in natural language are often incomplete ambiguous inconsistent Formal specifications and formal models help! 7 / 47

Testing in the Development Process In the classical waterfall model testing is the last phase. Danger: if project is delayed, testing gets cut Dramatic: most of the large SW projects overrun their deadline Consequence: the test cases have to be designed as early a possible, as soon as the SW requirements are known. this lead to agile software processes High testing costs! Test automation is needed! 1/3 1/2 of the resources in big projects 8 / 47

Self test Write a set of test cases that you feel would adequately test this program (Myers, The Art of Software Testing): The program reads three integer values from an input stream. The three values are interpreted as representing the lengths of the sides of a triangle. The program prints a message that states whether the triangle is scalene, isosceles, or equilateral. 9 / 47

Self test: questions Do you have... 1 a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid. 2 a test case that represents a valid equilateral triangle? 3 a test case that represents a valid isosceles triangle? 4 at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))? 5 a test case in which one side has a zero value? 6 a test case in which one side has a negative value? 7 a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third? 10 / 47

Self test: questions Do you have... 1 a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid. 2 a test case that represents a valid equilateral triangle? 3 a test case that represents a valid isosceles triangle? 4 at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))? 5 a test case in which one side has a zero value? 6 a test case in which one side has a negative value? 7 a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third? 10 / 47

Self test: questions Do you have... 1 a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid. 2 a test case that represents a valid equilateral triangle? 3 a test case that represents a valid isosceles triangle? 4 at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))? 5 a test case in which one side has a zero value? 6 a test case in which one side has a negative value? 7 a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third? 10 / 47

Self test: questions Do you have... 1 a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid. 2 a test case that represents a valid equilateral triangle? 3 a test case that represents a valid isosceles triangle? 4 at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))? 5 a test case in which one side has a zero value? 6 a test case in which one side has a negative value? 7 a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third? 10 / 47

Self test: questions Do you have... 1 a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid. 2 a test case that represents a valid equilateral triangle? 3 a test case that represents a valid isosceles triangle? 4 at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))? 5 a test case in which one side has a zero value? 6 a test case in which one side has a negative value? 7 a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third? 10 / 47

Self test: questions Do you have... 1 a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid. 2 a test case that represents a valid equilateral triangle? 3 a test case that represents a valid isosceles triangle? 4 at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))? 5 a test case in which one side has a zero value? 6 a test case in which one side has a negative value? 7 a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third? 10 / 47

Self test: questions Do you have... 1 a test case that represents a valid scalene triangle? Test cases (1, 2, 3) and (2, 5, 10) are not valid. 2 a test case that represents a valid equilateral triangle? 3 a test case that represents a valid isosceles triangle? 4 at least three test cases that represent valid isosceles triangles such that you have tried all three permutations of two equal sides (e.g., (3, 3, 4), (3, 4, 3), and (4, 3, 3))? 5 a test case in which one side has a zero value? 6 a test case in which one side has a negative value? 7 a test case with three integers greater than zero such that the sum of two of the numbers is equal to the third? 10 / 47

Self test: questions (cont.) 8 at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))? 9 a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4)) 10 at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))? 11 a test case in which all sides are zero? 12 at least one test case specifying noninteger values? 13 at least one test case specifying the wrong number of values? 14 for each test case the expected outputs specified? 11 / 47

Self test: questions (cont.) 8 at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))? 9 a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4)) 10 at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))? 11 a test case in which all sides are zero? 12 at least one test case specifying noninteger values? 13 at least one test case specifying the wrong number of values? 14 for each test case the expected outputs specified? 11 / 47

Self test: questions (cont.) 8 at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))? 9 a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4)) 10 at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))? 11 a test case in which all sides are zero? 12 at least one test case specifying noninteger values? 13 at least one test case specifying the wrong number of values? 14 for each test case the expected outputs specified? 11 / 47

Self test: questions (cont.) 8 at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))? 9 a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4)) 10 at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))? 11 a test case in which all sides are zero? 12 at least one test case specifying noninteger values? 13 at least one test case specifying the wrong number of values? 14 for each test case the expected outputs specified? 11 / 47

Self test: questions (cont.) 8 at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))? 9 a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4)) 10 at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))? 11 a test case in which all sides are zero? 12 at least one test case specifying noninteger values? 13 at least one test case specifying the wrong number of values? 14 for each test case the expected outputs specified? 11 / 47

Self test: questions (cont.) 8 at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))? 9 a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4)) 10 at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))? 11 a test case in which all sides are zero? 12 at least one test case specifying noninteger values? 13 at least one test case specifying the wrong number of values? 14 for each test case the expected outputs specified? 11 / 47

Self test: questions (cont.) 8 at least three test cases in Category 7 such that you have tried all three permutations where the length of one side is equal to the sum of the lengths of the other two sides (e.g., (1, 2, 3), (1, 3, 2), and (3, 1, 2))? 9 a test case with three integers greater than zero such that the sum of two of the numbers is less than the third (e.g., (1, 2, 4)) 10 at least three test cases in Category 9 such that you have tried all three permutations (e.g., (1, 2, 4), (1, 4, 2), and (4, 1, 2))? 11 a test case in which all sides are zero? 12 at least one test case specifying noninteger values? 13 at least one test case specifying the wrong number of values? 14 for each test case the expected outputs specified? 11 / 47

Naming the Wrong IEEE Standard Glossary of Software Engineering Terminology: Mistake: a human action that produces an incorrect result. Example: an incorrect action taken by the programmer. Fault: an incorrect step, process, or data definition in a computer progra (also defect, bug). Error: the difference between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition. Failure: the inability of a system or component to fulfill its required functions within specified performance requirements. Dijkstra was against calling it a bug. 12 / 47

Naming the Wrong: Example Consider a statement x = y + x; By a mistake a programmer changes it to x = y x, thus characterising a fault. Executing the fault with x = 0 does not lead to an error, and consequently not to an observable failure. 13 / 47

Testing, Test, Test Case testing: the act of designing, debugging, and executing test cases.. test: A test is the act of exercising software with test cases. A test has two distinctive goals: to find failures or to demonstrate correct execution. test case: A test case has an identity and is associated with program behaviour. A test case also has a set of inputs and a list of expected outputs. The output portion of a test case is frequently overlooked, which is unfortunate, because it is often the hard part. 14 / 47

Black & White Two fundamental approaches are used to identify test cases: functional testing: functional testing is based on the view that any program can be considered to be a function that maps values from its input domain to values in its output range (black box testing). Test cases are identified using the function s specification. structural testing: based on how the function is actually implemented (white box testing). 15 / 47

Why We Test validation: the process of evaluating an object to demonstrate that it meets the user requirements. verification: in the ANSI/IEEE Std 729-1983 three possible definitions of verification are given: 1 the process of reviewing, inspecting, testing, etc., if objects, processes, services, documents satisfy the specified requirements. 2 the process of evaluating if an object in a given phase in the software lifecycle, meets the requirements established in the previous phase. 3 formal correctness proofs of programs. falsification: the process of evaluating an object to demonstrate that it does not meet requirements. 16 / 47

When We Test The following kinds of tests can be distinguished during sw-lifecycle: unit tests: testing of units: modules, classes, components integration tests: tests that explore the interaction and consistency of successfully tested components. system tests: testing of the whole system in order to explore behaviors that can t be done by unit or integration testing: performance, data integrity, storage management, security, reliability. acceptance tests: does the system satisfy the requirements? Part of the contract. regression tests: tests after changes to prevent unintended changes. 17 / 47

The V-model 18 / 47

Functional Testing: Programs as Functions In functional testing, a program is considered as a function p : Input Output mapping values from its domain (Input) to values in its range (Output). Types permit the definitions of domain and range. Preconditions restrict the domain further. Postconditions restrict the range. 19 / 47

Boundary Value Analysis Focuses on the boundary of input space (domain) to identify test cases. Rationale: errors tend to occur near the extreme values of an input variable. Strategy: input values at 1 minimum 2 just above minimum 3 a nominal value 4 just below maximum 5 maximum 20 / 47

Boundary Value Analysis: 2 Input Variables Boundary value test cases for a function of two variables i1 and i2: i2 d c a b i1 Single fault assumption! (no relation between parameters) 21 / 47

Generalizing Boundary Value Analysis Number of test cases: 4n+ 1 for a function with n input variables. Different ranges: Triangle problem: e.g. min = 1, max = MAXINT Date input: date corresponds to 3 input variables; e.g. min = 1.3.2003, max = 31.3.2003 Boolean input: does not make sense! Limitations: only good when program is a function of several independent variables that represent bounded physical variables. good: temperature, coordinates bad: date, pin code, telephone number 22 / 47

Robustness Testing Exceeding the limits slightly: i2 d c a b i1 Most interesting with expected outputs not inputs! 23 / 47

Robustness Testing Examples Robustness testing forces attention on exception handling: Exceeding angle of attack of a plane: Will it stall? Exceeding load capacity of an elevator: Hopefully only a warning!? With strongly typed languages robustness testing may be very awkward: C leads to run-time errors. Exception handling (C#, Java) mandates robustness testing. Specifications: test cases such that pre(in) holds. 24 / 47

Worst-Case Testing Rejecting the single-fault assumption: i2 d c a Number of test cases: 5 n, for n input variables! b i1 25 / 47

Special Value Testing tester uses his domain knowledge, experience with similar programs, and information about soft spots to devise test cases. most widely practiced form. also called ad hoc testing. very dependent on the abilities of the tester! 26 / 47

Example: Commission Rifle 1 salesman sales rifle locks 2, stocks 3, and barrels 4. Locks costs $45, stocks $30, barrels $25. Has to sell at least one complete rifle per month. Has to sell at most 70 locks, 80 stocks, and 90 barrels per month. The gunsmith computes the salesman s commission: 10% on sales up to (and including) $1000, 15% on the next $800, and 20% on any sales in excess of $1800. 1 Gewehr 2 Verschluss 3 Schaft 4 Lauf 27 / 47

Commission: Output Boundary Analysis Case Locks Stocks Barrels Sales ($) Comm ($) Comment 1 1 1 1 100 10 output minimum 2 1 1 2 125 12.5 output minimum + 3 1 2 1 130 13 output minimum + 4 2 1 1 145 14.5 output minimum + 5 5 5 5 500 50 midpoint 6 10 10 9 975 97.5 border point - 7 10 9 10 970 97 border point - 8 9 10 10 955 95.5 border point - 9 10 10 10 1000 100 border point 10 10 10 11 1025 103.75 border point + 11 10 11 10 1030 104.5 border point + 12 11 10 10 1045 106.75 border point + 13 14 14 14 1400 160 midpoint 28 / 47

Commission: Output Boundary Analysis (cont.) Case Locks Stocks Barrels Sales ($) Comm ($) Comment 14 18 18 17 1775 216.75 border point - 15 18 17 18 1770 215.5 border point - 16 17 18 18 1755 213.25 border point - 17 18 18 18 1800 220 border point 18 18 18 19 1825 225 border point + 19 18 19 18 1830 226 border point + 20 19 18 18 1845 229 border point + 21 48 48 48 4800 820 midpoint 22 70 80 89 7775 1415 output maximum - 23 70 79 90 7770 1414 output maximum - 24 69 80 90 7755 1411 output maximum - 25 70 80 90 7800 1420 output maximum 29 / 47

Random Testing Idea: Random number generator to pick test case values. Motivation: avoiding a form of bias in testing. Question: How many random test cases are sufficient? Answer by 1 coverage criteria 2 reliability models Pro: fully automatic test case generation. Con: more test cases needed to reach structural coverage. At least 2 decades of literature in academia (statistical testing methods). 30 / 47

Random Testing of Commission Test Cases 10% Commission 15% Commission 20% Commission 91 1 6 84 27 1 1 25 72 1 1 70 176 1 6 169 48 1 1 46 152 1 6 145 125 1 4 120 avg. 1.01% 3.62% 95.37% 31 / 47

Random Testing of Triangle Test Cases NoTriangle Scalene Isosceles Equilateral 1289 663 593 32 1 15436 7696 7372 367 1 17091 8556 8164 367 1 2603 1284 1252 66 1 6475 3197 3122 155 1 5978 2998 2850 129 1 9008 4447 4353 207 1 avg. 49.83% 47.87% 2.29% 0.01% Program generates random test cases until at least one of each output occurs. Here an upper limit of 200 has been chosen. 32 / 47

Software Reliability Engineering reliability = 1 Decide target reliability. number of failures logical unit 2 Operational profile: What use cases (functions) are often executed? 3 More tests for highly frequent use cases. 4 Additional tests for high-risk use cases. 5 Statistical failure models, and the failures detected provide answers, when the target reliability is reached. Book: John Musa, Software Reliability Engineering, 2nd. edition, 1998. 33 / 47

Equivalence Class Testing Main idea: : Test hypothesis: classes of inputs show equal behaviour Select one test case from each equivalence class. Main motivation: adequate set of test cases (having a sense of completeness) avoiding redundancy! We distinguish: weak / strong equivalence class testing (single vs. multiple faults) normal / robust equivalence class testing 34 / 47

Equivalence Classes form a set partition. Definition: (Partition) Given a set A and a set of n subsets A 1, A 2,...,A n of A, the subsets form a partition of A iff and A 1 A 2 A n = A i, j {1,...,n} i j A i A j = {} 1st property provides a form of completeness, 2nd property ensures a form of nonredundancy. 35 / 47

Choosing Equivalence Classes Key to Strategy: Choice of equivalence relation. Deduced from the problem specification Triangle: one test case for equilateral (5, 5, 5), we don t expect much more from test cases, like (6, 6, 6), (7, 7, 7). often by guessing a likely implementation Formal interface specifications (contracts): choice of equivalence classes can be automated. 36 / 47

Weak Normal Equivalence Class Testing i2 d c a b i1 37 / 47

Strong Normal Equivalence Class Testing i2 d c a b i1 38 / 47

Weak Robust Equivalence Class Testing i2 d c a b i1 39 / 47

Strong Robust Equivalence Class Testing i2 d c a b i1 40 / 47

Triangle Example Note that four possible outpus can occur. We can use these to identify output (range) equivalence classes: 1 {(a, b, c) istriangle(a, b, c) = NoTriangle} 2 {(a, b, c) istriangle(a, b, c) = Scalene} 3 {(a, b, c) istriangle(a, b, c) = Isosceles} 4 {(a, b, c) istriangle(a, b, c) = Equilateral} 41 / 47

Triangle: Normal Equivalence Class Testing The four weak normal equivalence class test cases are: Test Case a b c Expected Output WN1 4 1 2 NoTriangle WN2 3 4 5 Scalene WN3 2 2 3 Isosceles WN4 5 5 5 Equilateral No valid subintervals of variables a, b, and c exists strong normal equivalence testing does not add test cases. 42 / 47

Triangle: Robust Equivalence Class Testing Let s assume the valid input data has been restricted to [0, 200], and error messages indicate if input is out of its range, then the additional weak robust equivalence class test cases are: Test Case a b c Expected Output WR1-1 5 5 Value a out of range! WR2 5-1 5 Value b out of range! WR3 5 5-1 Value c out of range! WR4 201 5 5 Value a out of range! WR5 5 201 5 Value b out of range! WR6 5 5 201 Value c out of range! 43 / 47

Triangle: Robust Equivalence Class Testing We show one corner of the cube of strong robust equivalence class test cases: Test Case a b c Expected Output WS1-1 5 5 Value a out of range! WS2 5-1 -5 Value b out of range! WS3 5 5-1 Value c out of range! WS4-1 -1 5 Value a, b out of range! WS5 5-1 -1 Value b, c out of range! WS6-1 5-1 Value a, c out of range! WS7-1 -1-1 Value a, b, c out of range! 44 / 47

Triangle: Input Equivalence Class Equivalence class testing is sensitive to the equivalence relation chosen. Equivalence classes based on the input domain: 1 {(a, b, c) a = b b = c} 2 {(a, b, c) a = b a c} 3 {(a, b, c) a b a = c} 4 {(a, b, c) b = c a b} 5 {(a, b, c) a b a c b c} 45 / 47

Triangle: Input Equivalence Class (cont.) Furthermore, we can apply the triangle property to see if they consitute a triange: 6 {(a, b, c) a b + c} 7 {(a, b, c) b a+c} 8 {(a, b, c) c a+b} 46 / 47

Triangle: Input Equivalence Class (cont.) We could be even more thorough and distinguish between x = y and x > y in the properties above: 6 {(a, b, c) a = b + c} 7 {(a, b, c) a > b + c} 8 {(a, b, c) b = a+c} 9 {(a, b, c) b > a+c} 10 {(a, b, c) c = a+b} 11 {(a, b, c) c > a+b} 47 / 47

Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institute for Software Technology Graz University of Technology Austria Summer Term 2013 1 / 18

Agenda 1 Functional testing strategies (cont.) Decision Tables 2 Structural testing strategies 2 / 18

Decision Tables Stub Rule 1 Rule 2 Rule 3,4 Rule 5 Rule 6 Rule 7, 8 c1 T T T F F F c2 T T F T T F c3 T F T F a1 X X X a2 X X a3 X X a4 X X 3 / 18

Decision Table for Triangle c1: a, b, c form a triangle F T T T T T T T T c2: a = b T T T T F F F F c3: a = c T T F F T T F F c4: b = c T F T F T F T F a1: NoTriangle X a2: Scalene X a3: Isosceles X X X a4: Equilateral X a5: Impossible X X X 4 / 18

Refined Decision Table for Triangle c1: a < b + c F T T T T T T T T T T c2: b < a + c F T T T T T T T T T c3: c < a + b F T T T T T T T T c4: a = b T T T T F F F F c5: a = c T T F F T T F F c6: b = c T F T F T F T F a1: NoTriangle X X X a2: Scalene X a3: Isosceles X X X a4: Equilateral X a5: Impossible X X X 5 / 18

Number of Rules Binary entries: 2 n rules (columns), n being the number of conditions c i don t care entries ( ) reduce the number of rules a rule counter can be established: Rules without don t care entry count as one rule Each don t care in a rule doubles the counter in this rule The rule counter for one table shall be 2 n 6 / 18

Decision Table with Rule Counter c1: a < b + c F T T T T T T T T T T c2: b < a + c F T T T T T T T T T c3: c < a + b F T T T T T T T T c4: a = b T T T T F F F F c5: a = c T T F F T T F F c6: b = c T F T F T F T F Rule Count: 32 16 8 1 1 1 1 1 1 1 1 a1: NoTriangle X X X a2: Scalene X a3: Isosceles X X X a4: Equilateral X a5: Impossible X X X 7 / 18

Redundant Decision Table Rule 1 4 Rule 5 Rule 6 Rule 7 Rule 8 Rule 9 c1 T F F F F T c2 T T F F F c3 T F T F F a1 X X X X a2 X X X a3 X X X X X 8 / 18

Inconsistent Decision Table Rule 1 4 Rule 5 Rule 6 Rule 7 Rule 8 Rule 9 c1 T F F F F T c2 T T F F F c3 T F T F F a1 X X X a2 X X X X a3 X X X X Regel 1-4 und 9 sind inkonsistent. Die Entscheidungstabelle ist nichtdeterministisch. 9 / 18

Guidelines 1 Decision table technique is for applications with prominent if-then-else-logic logical relations between input variables cause-effect relations between input and output 2 Decision tables get quickly inconvenient if logic is complex 3 Several refinement iterations help: Start with simple conditions and then refine them! 10 / 18

Structural Testing Strategies A software testing technique whereby explicit knowledge of the internal workings of the item being tested are used to select the test data Based on source code. White-box testing Technology available since the mid-1970s. Most techniques use program graphs. Pro: Amenable to rigorous definitions, mathematical analysis, and precise measurement. Con: Cannot test missing parts, e.g. a missing branch. 11 / 18

Program Graph Definition: Given a program written in an imperative programming language, its program graph is a directed graph in which nodes are statement fragments, and edges represent flow of control. 12 / 18

Program Graph Example 1 public class Program2 : Program // not very elegant implementation of Triangle 2 { 3 new public static string Triangle(int a, int b, int c) 4 { 5 bool istriangle = false; 6 string msg = ""; 7 8 if (a <= c - b a <= b - c b <= a - c) 9 istriangle = false; 10 else 11 istriangle = true; 12 13 if (istriangle) 14 { 15 if (a == b && b == c) 16 msg = "equilateral"; 17 else 18 if (a == b b == c a == c) 19 msg = "isosceles"; 20 else 21 msg = "scalene"; 22 } 23 else 24 msg = "no triangle"; 25 26 return msg; 27 } 28 } 13 / 18

Program Graph Example (cont.) 16 9 15 19 5 6 8 13 18 11 24 21 26 Node labels denote line numbers. 14 / 18

DD-Paths Definition: A chain is a path in a graph, in which the initial and terminal nodes are distinct, and every interior node has indegree 1 and outdegree 1. Definition: A DD-Path (decision-to-decision) is a chain in a program graph such that one of the following cases hold: 1 it consists of a single node with indeg = 0 2 it consists of a single node with outdeg = 0 3 it consists of a single node with indeg 2 or outdeg 2 4 it consists of a single node with indeg = 1 and outdeg = 1 5 it is a maximal chain of length 1 15 / 18

DD-Path Graph Definition: Given a program written in an imperative language, its DD-Path Graph is the directed graph in which nodes are DD-Paths of its program graph, and edges represent control flow between successor DD-Paths. Today it is known as control flow graph 16 / 18

Control Flow Graph 16 9 15 19 5-6 8 13 18 11 24 21 26 17 / 18

Test Coverage Metrics Metric C 0 C 1 C 2 C d C MCC C i k C inf Description of Coverage Every statement Every DD-Path (predicate outcome) C 1 coverage + loop coverage C 1 coverage + Every dependent pair of DD-Paths Multiple condition coverage Every program path that contains up to k repetitions of a loop (usually k = 2) All possible execution paths 18 / 18

TU Graz Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institut für Softwaretechnologie(IST) TU Graz Summer Term 2013 1/44

TU Graz Agenda 1 Symbolic Execution 2 Concolic Execution 2/44

TU Graz Literature: Symbolic Execution James C. King. Symbolic Execution and Program Testing, Communications of the ACM, Volume 19, Issue 7. July 1976, Pages 385 394. 3/44

TU Graz Symbolic Execution Instead of normal program inputs one supplies symbols representing arbitrary values. Execution proceeds like normal execution, except that values may be symbolic formulas over the input values. Interesting: symbolic execution of branching statements(e.g. if-statements) A technique between testing and formal proofs. Testing:executionwithsomeconcreteinputvalues Proofs:noexecution 4/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { 3 int x,y,z; 4 x = a + b; 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 5/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) a=α a,b = α b,c = α c 2 { 3 int x,y,z; 4 x = a + b; 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 6/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) a=α a,b = α b,c = α c 2 { 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 7/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) a=α a,b = α b,c = α c 2 { 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = α a +α b,y =0,z =0 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 8/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) a=α a,b = α b,c = α c 2 { 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = α a +α b,y =0,z =0 5 y = b + c; x = α a +α b,y = α b +α c,z =0 6 z = x + y b; 7 return z; 8 } 9/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) a=α a,b = α b,c = α c 2 { 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = α a +α b,y =0,z =0 5 y = b + c; x = α a +α b,y = α b +α c,z =0 6 z = x + y b; x = α a +α b,y = α b +α c,z = α a +α b +α c 7 return z; 8 } 10/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) a=α a,b = α b,c = α c 2 { 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = α a +α b,y =0,z =0 5 y = b + c; x = α a +α b,y = α b +α c,z =0 6 z = x + y b; x = α a +α b,y = α b +α c,z = α a +α b +α c 7 return z; return α a +α b +α c 8 } 11/44

TU Graz Bsp. Min 1 int min(int x, int y) 2 { 3 int z; 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 12/44

TU Graz Path Condition Symbolic execution of if-statement requires path condition(pc). pc is a Boolean expression over symbolic inputs. pc never contains program variables! pc is a conjunction of branching conditions. 13/44

TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 14/44

TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; x = α x,y = α y,z =0,pc =true 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 15/44

TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; x = α x,y = α y,z =0,pc =true 4 if(x<y) Case1:x = α x,y = α y,z =0,pc = α x < α y 5 z=x; 6 else 7 z=y; 8 return z; 9 } 16/44

TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; x = α x,y = α y,z =0,pc =true 4 if(x<y) Case1:x = α x,y = α y,z =0,pc = α x < α y 5 z=x; x = α x,y = α y,z = α x,pc = α x < α y 6 else 7 z=y; 8 return z; 9 } 17/44

TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; x = α x,y = α y,z =0,pc =true 4 if(x<y) Case1:x = α x,y = α y,z =0,pc = α x < α y 5 z=x; x = α x,y = α y,z = α x,pc = α x < α y 6 else Case2:x = α x,y = α y,z =0,pc = α x < α y 7 z=y; 8 return z; 9 } 18/44

TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; x = α x,y = α y,z =0,pc =true 4 if(x<y) Case1:x = α x,y = α y,z =0,pc = α x < α y 5 z=x; x = α x,y = α y,z = α x,pc = α x < α y 6 else Case2:x = α x,y = α y,z =0,pc = α x < α y 7 z=y; x = α x,y = α y,z = α y,pc = α x < α y 8 return z; 9 } 19/44

TU Graz Bsp. Min 1 int min(int x, int y) x = α x,y = α y,pc =true 2 { 3 int z; x = α x,y = α y,z =0,pc =true 4 if(x<y) Case1:x = α x,y = α y,z =0,pc = α x < α y 5 z=x; x = α x,y = α y,z = α x,pc = α x < α y 6 else Case2:x = α x,y = α y,z =0,pc = α x < α y 7 z=y; x = α x,y = α y,z = α y,pc = α x < α y 8 return z; return(z = α x α x < α y ) (z = α y α x < α y ) 9 } 20/44

TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { 3 return 4 min( 5 min(a,b), 6 c 7 ); 8 } 21/44

TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), 6 c 7 ); 8 } 22/44

TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), (= α a pc = α a < α b ) (= α b pc = α a α b ) 6 c 7 ); 8 } 23/44

TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), (= α a pc = α a < α b ) (= α b pc = α a α b ) 6 c c = α c 7 ); 8 } 24/44

TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), (= α a α a < α b ) (= α b α a α b ) 6 c c = α c 7 ); (= α a pc = α a < α b α a < α c ) 8 } 25/44

TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), (= α a α a < α b ) (= α b α a α b ) 6 c c = α c 7 ); (= α a pc = α a < α b α a < α c ) 8 } (= α b pc = α a α b α b < α c ) 26/44

TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), (= α a α a < α b ) (= α b α a α b ) 6 c c = α c 7 ); (= α a pc = α a < α b α a < α c ) 8 } (= α b pc = α a α b α b < α c ) 9 (= α c pc = α a < α b α a α c ) 27/44

TU Graz Bsp. Min(a,b,c) 1 int min(int a, int b, int c) 2 { a=α a,b = α b,c = α c,pc =true 3 return 4 min( 5 min(a,b), (= α a α a < α b ) (= α b α a α b ) 6 c c = α c 7 ); (= α a pc = α a < α b α a < α c ) 8 } (= α b pc = α a α b α b < α c ) 9 (= α c pc = α a < α b α a α c ) 10 (= α c pc = α a α b α b α c ) 28/44

TU Graz Symbolic Execution Tree Symbolic execution forcs at each if-statement: α a < α b T F α a < α c α b < α c T F T F min = α a min = α c min = α b min = α c 29/44

TU Graz From Path Conditions to Concrete Test Cases Calculation:pc 2 =true α a < α b (α a < α c ) Path condition represents equivalence class for all concrete values taking a path. Concrete test cases: find concrete values satisfying pc e.g.forpc2:a =0,b =1,c =0 Result: path coverage 30/44

TU Graz Limitations infinite number of paths(loops) infeasible paths(pc = false) limitations of solvers and theorem provers all code must be accessible(white-box) only for single-threaded programs 31/44

TU Graz Concolic Execution Concolic = Concrete + Symbolic also called Dynamic Symbolic Execution(Microsoft) Concrete and Symbolic Execution in parallel Due to concrete execution: Noinfeasablepaths! Integrationofblack-boxcomponentspossible. 32/44

TU Graz Concolic Execution Loop Algorithm 1 Covered := {} coveredsetofinputs 2 selectnewinputi Covered stopifnoonefound 3 execute Program(i) and record path condition C C(i) holds 4 Covered :=Covered {i C(i)} 5 goto2 Newinputi Coveredafterniterations: solve C i1 C in Stopwhen C i1 C in =false 33/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { 3 int x,y,z; 4 x = a + b; 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 34/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { a=(α a,1),b = (α b,2),c = (α c,3) 3 int x,y,z; 4 x = a + b; 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 35/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { a=(α a,1),b = (α b,2),c = (α c,3) 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 36/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { a=(α a,1),b = (α b,2),c = (α c,3) 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = (α a +α b,3),y = (0,0),z = (0,0) 5 y = b + c; 6 z = x + y b; 7 return z; 8 } 37/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { a=(α a,1),b = (α b,2),c = (α c,3) 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = (α a +α b,3),y = (0,0),z = (0,0) 5 y = b + c; x = (α a +α b,3),y = (α b +α c,5),z = (0,0) 6 z = x + y b; 7 return z; 8 } 38/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { a=(α a,1),b = (α b,2),c = (α c,3) 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = (α a +α b,3),y = (0,0),z = (0,0) 5 y = b + c; x = (α a +α b,3),y = (α b +α c,5),z = (0,0) 6 z = x + y b; x = (α a +α b,3),y = (α b +α c,5), 7 z = (α a +α b +α c,6) 8 return z; 9 } 39/44

TU Graz Bsp. Sum 1 int sum(int a, int b, int c) 2 { a=(α a,1),b = (α b,2),c = (α c,3) 3 int x,y,z; x =0,y =0,z =0 4 x = a + b; x = (α a +α b,3),y = (0,0),z = (0,0) 5 y = b + c; x = (α a +α b,3),y = (α b +α c,5),z = (0,0) 6 z = x + y b; x = (α a +α b,3),y = (α b +α c,5), 7 z = (α a +α b +α c,6) 8 return z; return(α a +α b +α c,6) 9 } 40/44

TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 41/44

TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; x = (α x,0),y = (α y,0),z = (0,0),pc 1 =true 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 41/44

TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; x = (α x,0),y = (α y,0),z = (0,0),pc 1 =true 4 if(x<y) (0 <0) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 41/44

TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; x = (α x,0),y = (α y,0),z = (0,0),pc 1 =true 4 if(x<y) (0 <0) 5 z=x; 6 else x = (α x,0),y = (α y,0),z = (0,0),pc 1 = (α x < α y ) 7 z=y; 8 return z; 9 } 41/44

TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; x = (α x,0),y = (α y,0),z = (0,0),pc 1 =true 4 if(x<y) (0 <0) 5 z=x; 6 else x = (α x,0),y = (α y,0),z = (0,0),pc 1 = (α x < α y ) 7 z=y; x = (α x,0),y = (α y,0),z = (α y,0),pc 1 = (α x < α y ) 8 return z; 9 } 41/44

TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; x = (α x,0),y = (α y,0),z = (0,0),pc 1 =true 4 if(x<y) (0 <0) 5 z=x; 6 else x = (α x,0),y = (α y,0),z = (0,0),pc 1 = (α x < α y ) 7 z=y; x = (α x,0),y = (α y,0),z = (α y,0),pc 1 = (α x < α y ) 8 return z; returnz= (α y,0),pc 1 = (α x < α y ) 9 } 41/44

TU Graz Bsp. Min: Iteration 1 1 int min(int x,int y) x = (α x,0),y = (α y,0),pc 1 =true 2 { 3 int z; x = (α x,0),y = (α y,0),z = (0,0),pc 1 =true 4 if(x<y) (0 <0) 5 z=x; 6 else x = (α x,0),y = (α y,0),z = (0,0),pc 1 = (α x < α y ) 7 z=y; x = (α x,0),y = (α y,0),z = (α y,0),pc 1 = (α x < α y ) 8 return z; returnz= (α y,0),pc 1 = (α x < α y ) 9 } FindnewinputvaluessuchthatC = pc 1 = α x < α y holdsand execute again, e.g. min(0, 1) 41/44

TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 42/44

TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; x = (α x,0),y = (α y,1),z = (0,0),pc 2 =true 4 if(x<y) 5 z=x; 6 else 7 z=y; 8 return z; 9 } 42/44

TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; x = (α x,0),y = (α y,1),z = (0,0),pc 2 =true 4 if(x<y) (0 <1): 5 z=x; 6 else 7 z=y; 8 return z; 9 } 42/44

TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; x = (α x,0),y = (α y,1),z = (0,0),pc 2 =true 4 if(x<y) (0 <1): pc 2 = α x < α y 5 z=x; 6 else 7 z=y; 8 return z; 9 } 42/44

TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; x = (α x,0),y = (α y,1),z = (0,0),pc 2 =true 4 if(x<y) (0 <1): pc 2 = α x < α y 5 z=x; x = (α x,0),y = (α y,1),z = (α x,0),pc 2 = α x < α y 6 else 7 z=y; 8 return z; 9 } 42/44

TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; x = (α x,0),y = (α y,1),z = (0,0),pc 2 =true 4 if(x<y) (0 <1): pc 2 = α x < α y 5 z=x; x = (α x,0),y = (α y,1),z = (α x,0),pc 2 = α x < α y 6 else 7 z=y; 8 return z; returnz= (α x,0),pc 2 = α x < α y 9 } 42/44

TU Graz Bsp. Min: Iteration 2 1 int min(int x,int y) x = (α x,0),y = (α y,1),pc 2 =true 2 { 3 int z; x = (α x,0),y = (α y,1),z = (0,0),pc 2 =true 4 if(x<y) (0 <1): pc 2 = α x < α y 5 z=x; x = (α x,0),y = (α y,1),z = (α x,0),pc 2 = α x < α y 6 else 7 z=y; 8 return z; returnz= (α x,0),pc 2 = α x < α y 9 } FindingnewinputvaluessuchthatC = pc 1 pc 2 =falseisnot possible, hence terminate. 42/44

TU Graz Limitations infinite number of paths: upper bounds limitations of solvers and theorem provers for single-threaded programs 43/44

TU Graz Tools Cute, jcute Pex(see exercise) PathCrawler 44/44

Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institute for Software Technology Graz University of Technology Austria Summer Term 2013 1 / 23

Agenda 1 Spezifikationen Beispiel Eigenschaften von Spezifikationen Contracts 2 Invarianten 2 / 23

Oracle Definition: any means used to predict the outcome of a test. Kind of oracles: specification existing test cases (regression testing) previous version of the program prototype domain expert actual program (dangerous!) 3 / 23

Specifications fix the criteria to accept an implementation. are a basis for a contract between user and supplier. define what should be achieved, but not how. greatest possible freedom for developer. programs are not suitable as specifications! 4 / 23

Specification vs. Implementation Specification vs. Implementations abstract concrete what how 5 / 23

Specification Implementation 1 The specification serves as starting point for the design of the implementation. 2 The specification is the bases for any verification: 1 for proving that an implementation satisfies a specification 2 for the generation of test cases 3 as a test oracle 6 / 23

Programs as Specifications? public static double P (double tfactorial) { double x1=0; double x0=tfactorial/2; double a=tfactorial; boolean finished=false; while (finished==false) { x1=(x0+(a/x0))/2; if (x1>x0){ if ((x1-x0)<0.00005){ finished=true;} } else if (x0>x1){ if ((x0-x1)<0.00005){ finished=true;} } x0=x1; } return x1; } 7 / 23

Spezifikation Vers. 1 Example (Postcondition) Postcondition (Nachbedingung): post P (in, out) = df out 2 = in 8 / 23

Spezifikation Vers. 2 Not all inputs are valid! Example (Pre-Postcondition) Postcondition: post P (in, out) = df out 2 = in Precondition (Vorbedingung): pre P (in) = df in 0 9 / 23

Spezifikation Vers. 3 We are only interested in positive results! Example (Pre-Postcondition) Postcondition: post P (in, out) = df out 2 = in out 0 Precondition: pre P (in) = df in 0 10 / 23

Spezifikation Vers. 4 In practice, the result is an approximations! Example (Pre-Postcondition) Postcondition: post P (in, out) = df abs(in out 2 ) ǫ in out 0 Precondition: ǫ... error constant abs... absolute value pre P (in) = df in 0 11 / 23

Satisfiability In order to be a meaningful specification, one solution must exist. Thus, one program/implementation must exist that satisfies the specification. We also say that there is a model of the specification. Or, the specification is realisable (implementable). 12 / 23

Underspecification Specifications need not to determine unique results. Examples are flight reservation systems, compilers. Underspecification is a form of abstraction. 13 / 23

Preciseness Specifications must be precise. That means that the set of solutions (models) must be defined. Underspecified does not mean imprecise. Distinguish between abstract and imprecise! 14 / 23

Design by Contract A method for software development Pre- and postconditions als contracts between classes and clients Pre- and postconditions are executable. History: Hoare-Logik (1969), VDM (70-ies), Eiffel (1992), JML (1999), C# Code Contracts (2008) 15 / 23

C# Code Contracts Example (sqrt) public static double sqrt2(double x) { Contract.Requires(x >= 0); Contract.Ensures( Math.Abs(x - (Contract.Result<double>() * Contract.Result<double>())) < 0.0001 ); return Math.Sqrt(x); } Vorbedingung: Requires (normale) Nachbedingung: Ensures 16 / 23

Java Modeling Language (JML) Example (sqrt) public final static double eps = 0.0001; //@ requires x >= 0.0; //@ ensures JMLDouble.approximatelyEqualTo(x, \result * \result, eps); public static double sqrt(double x) { return Math.sqrt(x); } Vorbedingung: requires (normale) Nachbedingung: ensures 17 / 23

Vorteile von Contracts zur Dokumentation (Spezifikation) von Klassen und Interfaces Kontrakt abstrakter als Code (kein Algorithmus!) Kontrakt kann maschinell geprüft werden (im Gegensatz zu informalen Kommentaren) Fehlersuche (debugging) Wartung Schuldzuweisung bei Fehlern: Client vs. Class Effizienz: Keine ineffizienten defensiven Überprüfungen (checks) im Code 18 / 23

Beispiel Effizienz Example (Binärsuche) public static int search(int[] a, int x) { Contract.Requires(a!= null); Contract.Requires(Contract.ForAll(1, a.length, i => a[i-1] <= a[i])); return Array.BinarySearch(a, x); } All-quantor ( ): ForAll Endliche Bereichsangabe f"ur i macht den Ausdruck ausf"uhrbar. Eine alternative Prüfschleife im Endprogramm wäre ineffizient: Laufzeit: logarithmisch linear 19 / 23

Modularität des Programmverstehens Wie kann man solchen Code verstehen? Example (Typischer OO Code)... source.close(); target.close(); getfile.setlastmodified( loc.modtime().gettime()); return modtime(); Möglichkeit 1: Lesen des Codes aller Methoden, lesen des Codes aller Methoden in den Methoden, lesen des... Erschwernis: Subtypen polymorphismus (dynamische Bindung) Möglichkeit 2: Lesen der Kontrakte aller Methoden 20 / 23

Invarianten Definition (Klasseninvariante, allgemein) Eine Klasseninvariante ist eine Eigenschaft des Klassenzustandes (Variablen, Felder), die immer nach Ausführen eines jeden Konstruktors, sowie vor und nach Ausführen einer jeden Methode gelten mu s. auch Dateninvariante oder kurz Invariante genannt. 21 / 23

Rolle von Invarianten Typeninvariante: schränkt einen Datentyp ein (z.b. Natürliche Zahlen) Repräsentationsinvariante: dokumentiert Designentscheidung der Datenrepräsentation (z.b. Sortierung einer Liste) Dokumentieren Beziehungen zwischen Variablen (Feldern). Invarianten werden durch Kapselung geschützt. (Repräsentations-)invarianten sind oft für die Korrektheit eines Algorithmus notwendig. 22 / 23

C# Beispiel Purse: Translated from JML [Burdy et al. 2005] 23 / 23

Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institute for Software Technology Graz University of Technology Austria Summer Term 2013 1 / 10

Agenda 1 Model-based Specifications Example: Alarm System 2 / 10

Model-based Specifications Motivation: Observation: pre- and postconditions are closely coupled to the inner data representation of a class. Advantage: Pecise description of the inner behaviour of the implementation. Precise documentation of the representation invariant. Disadvantage: When changing a variable of a class, also the specifications must be changed. The variables of a class must be declared before the specification. Not suitable for specifying abstract classes and interfaces! Conclusion We need abstractions of data representations models 3 / 10

Model-based Specification in C# Some restrictions apply: Unlike JML, the Code Contracts framework does not know model variables. Model variables are abstractions of one or more concrete variables Invariants cannot be checked on interfaces In C# we can use generic abstract classes for modelling the abstract state with (collections of) interfaces Using sets (ISet), sequences (IList), maps (IDictionary) to abstract from efficient implementation representations. 4 / 10

Anforderungen für ein Alarmsystem Eine chemische Fabrik ist mit einer Anzahl von Überwachungseinrichtungen ausgerüstet, welche abhängig vom Zustand der Fabrik Alarme auslösen können. Wird ein Alarm ausgelöst, mu s ein Experte angefordert werden. Experten haben verschiedene Qualifikationen um mit verschiedenen Alarmen umgehen zu können. 5 / 10

Anforderungen für ein Alarmsystem (cont.) R1 Es ist ein computerbasiertes System zu entwickeln, welches die Alarme in dieser Fabrik managed. R2 Vier Arten von Qualifikationen werden gebraucht um mit den Alarmen umzugehen. Diese sind elektrische, mechanische, biologische und chemische Qualifikationen. R3 Es müssen während allen festgelegten Zeitperioden Experten abrufbereit sein. R4 Jeder Experte hat eine Liste von Qualifikationen. 6 / 10

Anforderungen für ein Alarmsystem (cont.) R5 Jeder Alarm, der an das System gemeldet wird hat eine zugeordnete Qualifikation und eine Beschreibung des Alarms, die ein Experte verstehen kann. R6 Wann immer ein Alarm vom System empfangen wird, soll ein Experte mit der richtigen Qualifikation gefunden werden, so da s er angepaged werden kann. R7 Die Experten sollen die Systemdatenbank abfragen können, um zu überprüfen wann sie abrufbereit sein werden/müssen. R8 Es mu s möglich sein die Anzahl der abrufbaren Experten abzufragen. 7 / 10

Mögliche Klassen und Methoden Klassen (Typen) IAlarmSystem IQualification IAlarm IPeriod IExpert (Description) Methoden ExpertToPage ExpertIsOnDuty NumberOfExperts 8 / 10

Deklaration der Methoden R6: public int NumberOfExperts(IPeriod p); R7: public IPeriod[] ExpertIsOnDuty(IExpert e); R8: public IExpert ExpertToPage(IAlarm a, IPeriod p); 9 / 10

C# Example AlarmSystem.cs 10 / 10

Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institute for Software Technology Graz University of Technology Austria Summer Term 2013 1 / 27

Agenda 1 Äquivalenzklassen aus Disjunktiver Normalform 2 Bildung von Testfallfolgen 3 Testabdeckungen Testabdeckung von Finiten Zustandsmaschinen 2 / 27

DNF-Partitionierung Gegeben eine formale Spezifikation, z.b. pre(in) post(in, out) Rekursives Umschreiben durch Anwenden der Regel: a b (a b) ( a b) (a b) das ergibt eine Disjunktive Normalform (DNF). Logische Äquivalenzen ergeben neue Disjunktionen: a b a b Spezialwerte v eines Typs T können auch behandelt werden : x : T (x = v x v) x : T 3 / 27

DNF-Partitionierung (Forts.) Normale Äquivalenzklassen durch umschreiben von pre(in) post(in, out) Robustheits-Äquivalenzklassen durch umschreiben von pre(in) post(in, out) Achtung! Es wird die In-Output-Relation partitioniert, nicht nur der Eingabebereich. 4 / 27

Maximum Beispiel Ein Programm soll das Maximum von zwei ganzen Zahlen ausgeben. Wir können in Pseudonotation spezifizieren: max: Int Int Int max(x,y) as r post (r = x r = y) r x r y 5 / 27

Maximum Beispiel (Forts.) Umschreiben der Disjunktion ergibt die Postcondition: post ((r = x r = y) (r x r = y) (r = x r y)) r x r y 6 / 27

Maximum Beispiel (Forts.) Distributivgesetz auf zweimal ergibt post (r = x r = y r x r y) (r x r = y r x r y ) (r = x r y r x r y ) 7 / 27

Maximum Beispiel (Forts.) Nach Vereinfachung erhalten wir: post (r = x r = y) (r > x r = y) (r = x r > y) Das repräsentiert drei Äquivalenzklassen von Testfallen: 1 {(x, y, r) r = x r = y} 2 {(x, y, r) r > x r = y} 3 {(x, y, r) r = x r > y} 8 / 27

Testfallfolgen Für komplexe zustandsbasierte Systeme (OO), Eingabe-Ausgabestrategien sind nicht ausreichend: interner Zustand kann nicht direkt als Eingabe gesetzt werden (Kapselung), sondern nur durch Interaktionen. der Tester muss eine adäquate Folge von Aktionen finden, die den internen Zustand setzen, von dem eine Methode (Funktion) getestet werden soll. jede Funktion mag einfach sein, aber ihre Interaktionen sind es nicht. Typische Anwendungen: Unit-Testen von Objekt-Orientierten Systemen. Systemtesten von interaktiven Systemen. 9 / 27

Formale Technik 1 Wähle ein Modell für den internen Zustand. 2 Spezifikation jeder Methode durch Vor-Nachbedingungen 3 DNF Partitionierung jeder Methode 4 Konstruktion eines endlichen Zustandsautomaten (finite state machine): Extrahiere pro Methode zwei Constraints: Zustand davor und danach. Partitionierung des Zustands Transitionen sind partitionierte Sub-Methoden 5 Wähle Folgen, die den Zustandsautomaten abdecken 10 / 27

Scheduler Beispiel Ein Prozessscheduler behandelt Prozesse, die in drei möglichen Zuständen sind: ready um am Prozessor zu laufen, waiting um ready zu werden, optional, einen einzigen Prozess der active ist. Diese Prozesse werden durch eine eindeutige Pid gekennzeichnet. Ein Prozess kann nicht gleichzeitig ready und waiting sein. Der aktive Prozess ist weder ready noch waiting Es muss einen active Prozess geben, wann immer es einen ready Prozess gibt, der laufen kann. 11 / 27

Zustandsmodell scheme scheduler = class type Pid = Nat value nil: Nat variable active : Pid, ready : Pid-set, waiting: Pid-set 12 / 27

Zustandsinvariante axiom ready waiting = {}, active (ready waiting), nil (ready waiting), active = nil ready = {} 13 / 27

Schedule value schedule: Pid-set Pid schedule(ps) as p post p ps pre ps {}, 14 / 27

Init Init: Unit write any Unit Init() post active = nil ready waiting = {}, 15 / 27

New New: Pid write waiting Unit New(p) post waiting = waiting old {p} pre p active p (ready waiting), 16 / 27

Ready Ready: Pid write active, waiting, ready Unit Ready(q) post waiting = waiting old \ {q} active old = nil (ready = ready old active = q) active old nil (ready = ready old {q} active = active old ) pre q waiting, 17 / 27

Swap Swap: Unit write any Unit Swap() post waiting = waiting old {active old } ready old = {} (active = nil ready = {}) ready old {} (active = schedule(ready old ) ready = ready old \ {active}) pre active nil, 18 / 27

Konstruktion des Automaten 1 Analysiere die Äquivalenzklassen an allen Methoden (Funktionen) und leite Sub-methoden anhand dieser Klassen ab. Hier unterscheiden wir ob active = nil und ob ready oder waiting leer sind. 2 Von jeder partitionierten Methode erhält man die Vorher- Nachherzustände durch existientielles Quantifizieren der Variablen, die nicht in Betracht gezogen werden. 3 DNF Partitionierung der Zustände aus Schritt 2. Das ergibt die Zustände des Automaten. 4 Konstruiere den Automaten durch Auflösen der Transitionen gegen die Zustände. 19 / 27

Partitioned New New1: Pid write waiting Unit New1(p) post waiting = waiting old {p} pre active = nil ready = {} p waiting, New2: Pid write waiting Unit New2(p) post waiting = waiting old {p} pre p active active nil p (ready waiting), 20 / 27

Partitioned Ready Ready1: Pid write active, waiting, ready Unit Ready1(q) post waiting = waiting old \ {q} ready = ready old active = q pre q waiting active = nil, Ready2: Pid write active, waiting, ready Unit Ready2(q) post waiting = waiting old \ {q} ready = ready old {q} active = active old pre q waiting active nil, 21 / 27

Partitioned Swap Swap1: Unit write any Unit Swap1() post waiting = waiting old {active old } active = nil ready = {} pre active nil ready = {}, Swap2: Unit write any Unit Swap2() post waiting = waiting old {active old } active ready old ready = ready old \ {active} pre active nil ready {}, 22 / 27

Berechnung der Zustände DNF Analyse der Invariante Existentielles Quantifizieren (hiding) ergibt den 1 Vorherzustand: variable 1,...,variable n pre post 2 Nachherzustand: variable1 old,...,variablen old pre post Danach DNF-Analyse aller Zustände ergibt Äquivalenzklassen der Zustände. 23 / 27

Zustände des Automaten Variables State 1 State 2 State 3 State 4 State 5 State 6 active = nil = nil nil nil nil nil ready = {} = {} = {} = {} = {} = {} waiting = {} = {} = {} = {} = {} = {} 24 / 27

Konstruktion der Transitionen Aus den Zuständen und partitionierten Methoden, kann man die Transitionen konstruieren. Eine Transition existiert von Zustand St1 nach St2 genau dann, wenn die folgende Bedingung wahr ist: s, s old, in, out : St1(s old ) pre i (s old, in) post i (s old, in, s, out) St2(s) wobei die Zustandsvariablen s und s old den gesamten Variablenvektor repr asentieren. s entspricht \old(s) in JML 25 / 27

Der endliche Automat für den Scheduler New1 Swap1 New2 Swap2 Swap2 Init Swap2 1 2 Ready1 3 4 Ready2 5 6 New1 New2 New2 Swap1 Swap2 Ready2 Ready1 Ready2 New2 26 / 27

Testabdeckung des Zustandsautomaten Einige mögliche Abdeckungen: 1 Alle Zustände abdecken. 2 Alle Methoden abdecken. 3 Alle Transitionen abdecken. 4 Alle aufeinanderfolgenden Paare von Transitionen. 5 Alle mögliche Pfade mit n Wiederholungen. 27 / 27

Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institute for Software Technology Graz University of Technology Austria Summer Term 2013 1 / 31

Agenda 1 Conformance Testing Properties 2 Labelled Transition Systems Equivalence Preorder ioco Examples 3 Test generation 2 / 31

Input/Output Conformance Relation Jan Tretmans - 1996 Based on IO-Labeled Transition Systems 3 / 31

Input/Output Conformance Relation Jan Tretmans - 1996 Based on IO-Labeled Transition Systems 3 / 31

Conformance Testing 4 / 31

Conformance Testing - Soundness Test suite is sound: conformance all tests pass 5 / 31

Conformance Testing - Soundness Test suite is sound: conformance all tests pass 5 / 31

Conformance Testing - Exhaustiveness Test suite is exhaustive: conformance all tests pass 6 / 31

Conformance Testing - Exhaustiveness Test suite is exhaustive: conformance all tests pass 6 / 31

Conformance Testing - Completeness Test suite is complete: conformance all tests pass 7 / 31

Conformance Testing - Completeness Test suite is complete: conformance all tests pass 7 / 31

Conformance Testing with ioco System (Implementation) is modeled as IOTS weakly input enabled Specification is an IOLTS possibly incomplete possible non-deterministic τ!g1 τ!g1?g2?g2!g3!g3!g1 8 / 31

Conformance Testing with ioco System (Implementation) is modeled as IOTS weakly input enabled Specification is an IOLTS possibly incomplete possible non-deterministic τ!g1 τ!g1?g2?g2!g3!g3!g1 8 / 31

Input Output Labeled Transition Systems Input Output Labeled Transition System An IOLTS is an LTS M = (Q M, A M, M, q0 M) with Q M a finite set of states A M = A M I A M O {τ} where A M I and A M O are input and output alphabets τ A M I A M O is an unobservable, internal action M Q M A M Q M is the transition relation q0 M QM is the initial state. QUESTION? What means: The implementation conforms-to the specification? 9 / 31

Input Output Labeled Transition Systems Input Output Labeled Transition System An IOLTS is an LTS M = (Q M, A M, M, q0 M) with Q M a finite set of states A M = A M I A M O {τ} where A M I and A M O are input and output alphabets τ A M I A M O is an unobservable, internal action M Q M A M Q M is the transition relation q0 M QM is the initial state. QUESTION? What means: The implementation conforms-to the specification? 9 / 31

How to relate 2 LTSs? Equivalence Relations (=) Bisimulation Trace Equivalence Testing Equivalence... Preorder Relations ( ) Trace Preorder Testing Preorder... Input-Output Relations... ioconf ioco... 10 / 31

(Weak) Bisimulation Two states are bisimilar iff they simulate each other and go to states which are bisimilar Bisimulation is not suited for testing! 11 / 31

Trace Equivalence A trace is an observable sequence of actions Two states are trace equivalent iff they have the same traces Trace equivalence is the weakest notion of conformance 12 / 31

Equivalence vs. Preorder Relations Equivalence Relation (R) reflexive (srs) symmetric: irs sri transitive: irs srt irt Preorder Relations ( ) NOT necessarily antisymmetric: irs i s s i simplifies testing e.g.: Trace Preorder i tr s traces(i) traces(s) 13 / 31

Some Notations: Transitions q a M q = df (q, a, q ) M q ǫ q = df (q = q ) (q τ M q 1 q n 1 τ M q ) q a q = df q 1, q 2 : q ǫ M q 1 a M q 2 ǫ M q 14 / 31

Some Notations: Quiescence δ is used to represent quiescence q δ q = df q is a quiescent state. Quiescent state = no edge labeled with an output or an internal action 15 / 31

Some Notations: Quiescence δ is used to represent quiescence q δ q = df q is a quiescent state. Quiescent state = no edge labeled with an output or an internal action 15 / 31

Some Notations: Suspension Automaton (M) = (Q M, A (M), (M), q M 0 ) where: A (M) = A M {δ} with δ A (M) O (M) is obtained from M by adding loops q δ q for each quiescent state 16 / 31

Some Notations: After q after M σ = df {q q σ M q } Q after M σ = df q Q (q after M σ). 17 / 31

Some Notations: Out Out M (q) = df {a A M O q a M } Out M (Q) = df q Q (Out M(q)) 18 / 31

ioco Definition: ioco Let IUT = (Q IUT, A IUT, IUT, q0 IUT ) be weakly input enabled with A IUT = A IUT I A IUT O {τ} and S = ( Q S, A S, S, q0) S be strongly responsive with A S = A S I A S O {τ}. Then: IUT ioco S = df σ traces( (S)) : Out IUT ( (IUT) after IUT σ) Out S ( (S) after S σ). IUT ioco S iff outputs (and quiescences) of the IUT are possible in S after an arbitrary suspension trace of S. 19 / 31

ioco Definition: ioco Let IUT = (Q IUT, A IUT, IUT, q0 IUT ) be weakly input enabled with A IUT = A IUT I A IUT O {τ} and S = ( Q S, A S, S, q0) S be strongly responsive with A S = A S I A S O {τ}. Then: IUT ioco S = df σ traces( (S)) : Out IUT ( (IUT) after IUT σ) Out S ( (S) after S σ). IUT ioco S iff outputs (and quiescences) of the IUT are possible in S after an arbitrary suspension trace of S. 19 / 31

P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 20 / 31

P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 20 / 31

P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 21 / 31

P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 21 / 31

P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 22 / 31

P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 22 / 31

P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 23 / 31

P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 23 / 31

P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 24 / 31

P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 24 / 31

P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 25 / 31

P ioco S? P ioco S = df σ traces( (S)) : Out IUT ( (P) after IUT σ) Out S ( (S) after S σ). 25 / 31

Test Cases A test case is an IOLTS Inputs = Outputs IUT, Outputs = Inputs IUT Equipped with verdict states (pass, fail) In each state (except Pass, Fail): Single output and all inputs All inputs and θ 26 / 31

Formal Test Execution 27 / 31

Formal Test Execution 27 / 31

Formal Test Execution 27 / 31

Formal Test Execution 27 / 31

Formal Test Execution 27 / 31

Formal Test Execution 27 / 31

Formal Test Execution 27 / 31

Formal Test Execution 27 / 31

Formal Test Execution 27 / 31

Formal Test Execution 27 / 31

Formal Test Execution 27 / 31

A Complete Test Generation Algorithm Given the suspension automaton of a specification as an LTS S = (Q S, A S, S, q S 0 ) 1 Initially compute K = q S 0 after S ǫ 2 Do non-deterministically, either: Stop test case with verdict pass Let the test case produce an output (!a) with K = K after S?a. Also accept all inputs at the same time and add fail states for unexpected results. Accept all inputs (and quiescence) and add fail states for unexpected results. Compute new K for valid inputs. 3 Repeat Step 2 with new set K. 28 / 31

A Complete Test Generation Algorithm δ δ 29 / 31

A Complete Test Generation Algorithm δ δ 29 / 31

A Complete Test Generation Algorithm δ δ 29 / 31

A Complete Test Generation Algorithm δ δ 29 / 31

A Complete Test Generation Algorithm δ δ 29 / 31

A Complete Test Generation Algorithm δ δ 29 / 31

A Complete Test Generation Algorithm δ δ 29 / 31

Tools TGV: offline testing tool jtorx: online testing MoMut: offline model-based mutation testing tool, AIT and TU Graz SpecExplorer: uses Alternating Simulation (equivalent to ioco for deterministic LTS) 30 / 31

References Martin Weiglhofer, Bernhard Aichernig, and Franz Wotawa. Fault-based conformance testing in practice. International Journal of Software and Informatics, 3(2-3):375 411, June/September 2009. Copyright by Institute for Software, Chinese Academy of Science. 31 / 31

Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institut für Softwaretechnologie (IST) TU Graz Summer Term 2013 1 / 19

Agenda Mutation Testing Model-based Mutation Testing Test-Case Generation with ioco checking 2 / 19

Binary search bug in Java Some Bugs Hide for a Long Time! JDK 1.5 library (2006) out of boundary access of large arrays due to integer overflow 9 years undetected Algorithm was proven correct! Programming Pearls [Bentley86, Bentley00] assuming infinite integers :( 1 public static 2 int binarysearch(int[] a,int key) 3 { 4 int low = 0; 5 int high = a.length - 1; 6 7 while (low <= high) { 8 int mid = (low + high) / 2; 9 int midval = a[mid]; 10 11 if (midval < key) 12 low = mid + 1; 13 else if (midval > key) 14 high = mid - 1; 15 else 16 return mid; // key found 17 } 18 return -(low + 1); // key not found 19 } Beware of bugs in the above code; I have only proved it correct, not tried it. [Knuth77] 3 / 19

Binary search bug in Java Some Bugs Hide for a Long Time! JDK 1.5 library (2006) out of boundary access of large arrays due to integer overflow 9 years undetected Algorithm was proven correct! Programming Pearls [Bentley86, Bentley00] assuming infinite integers :( 1 public static 2 int binarysearch(int[] a,int key) 3 { 4 int low = 0; 5 int high = a.length - 1; 6 7 while (low <= high) { 8 int mid = (low + high) >>> 1; 9 int midval = a[mid]; 10 11 if (midval < key) 12 low = mid + 1; 13 else if (midval > key) 14 high = mid - 1; 15 else 16 return mid; // key found 17 } 18 return -(low + 1); // key not found 19 } Beware of bugs in the above code; I have only proved it correct, not tried it. [Knuth77] 3 / 19

Observations Verification failed (wrong assumption) Established testing strategies failed: statement coverage branch coverage fails multiple condition coverage MC/DC: standard in avionics [DO-178B/ED109] Long array needed: int[] a = new int[integer.max_value/2+2] Lesson Concentrate on possible faults, not on structure. Generate test cases covering these faults Mutation Testing [Lipton71, Hamlet77, DeMillo et al.78] 4 / 19

What Is Mutation Testing? Originally: Technique to verify the quality of test cases There is a pressing need to address the, currently unresolved, problem of test case generation. [Jia&Harman11] 5 / 19

How Does It Work? Step 1: Create mutants Mutation Process Source Code Mutant Mutation Operator 6 / 19

Example: Scala Program Kind of triangles: equilateral isosceles scalene Create mutants mutation operator == >= creates 5 mutants 1 object triangle { 2 3 def tritype(a : Int, b : Int, c: Int) = 4 (a,b,c) match { 5 case _ if (a <= c-b) => "no triangle" 6 case _ if (a <= b-c) => "no triangle" 7 case _ if (b <= a-c) => "no triangle" 8 case _ if (a == b && b == c) => 9 " equilateral" 10 case _ if (a == b) => "isosceles" 11 case _ if (b == c) => "isosceles" 12 case _ if (a == c) => "isosceles" 13 case _ => " scalene" 14 } 15 } Source code in Scala 7 / 19

Example: Scala Program Kind of triangles: equilateral isosceles scalene Create mutants mutation operator == >= creates 5 mutants 1 object triangle { 2 3 def tritype(a : Int, b : Int, c: Int) = 4 (a,b,c) match { 5 case _ if (a <= c-b) => "no triangle" 6 case _ if (a <= b-c) => "no triangle" 7 case _ if (b <= a-c) => "no triangle" 8 case _ if (a >= b && b == c) => 9 " equilateral" 10 case _ if (a == b) => "isosceles" 11 case _ if (b == c) => "isosceles" 12 case _ if (a == c) => "isosceles" 13 case _ => " scalene" 14 } 15 } Mutant 7 / 19

Example: UML Model Car Alarm System event-based controllable events observable events Open OpenAndUnlocked Close ClosedAndUnlocked AlarmSystem_StateMachine Lock Unlock OpenAndLocked Unlock Alarm Activate Alarms /entry Deactivate Alarms /exit Mutate the model mutation operator Unlock 20 Lock Close ClosedAndLocked Open FlashAndSound 30 / Deactivate Sound Flash 17 mutants Unlock Armed Show Armed /entry Show Unarmed /exit Close SilentAndOpen Open 300 State machine model in UML 8 / 19

Example: UML Model Car Alarm System event-based controllable events observable events Mutate the model mutation operator 17 mutants Mutated UML model 8 / 19

How Does It Work? Step 2: Try to kill mutants A test case kills a mutant if its run shows different behaviour. 9 / 19

Example: Scala Program Mutant survives path coverage (MC/DC): tritype(0,1,1) tritype(1,0,1) tritype(1,1,0) tritype(1,1,1) tritype(2,3,3) tritype(3,2,3) tritype(3,3,2) tritype(2,3,4) Mutant killed by tritype(3,2,2) 1 object triangle { 2 3 def tritype(a : Int, b : Int, c: Int) = 4 (a,b,c) match { 5 case _ if (a <= c-b) => "no triangle" 6 case _ if (a <= b-c) => "no triangle" 7 case _ if (b <= a-c) => "no triangle" 8 case _ if (a >= b && b == c) => 9 " equilateral" 10 case _ if (a == b) => "isosceles" 11 case _ if (b == c) => "isosceles" 12 case _ if (a == c) => "isosceles" 13 case _ => " scalene" 14 } 15 } Mutant 10 / 19

Example: UML Model Mutant survives function coverage state coverage transition coverage Killed by Lock(); Close(); After(20); Mutated UML model 11 / 19

From Analysis to Synthesis State of art: Analysis of test cases How many mutants killed by test cases? mutation score = #killed mutants #mutants Problem: equivalent mutants Solution: review of surviving mutants Research: Synthesis of test cases Find test cases that maximise mutation score. Idea: Check equivalence between original and mutant Use counter-example as test case. Problem: equivalence checking is hard (undecidable in general) Solution: generate from models (abstraction) model-based mutation testing 12 / 19

Model-Based Mutation Testing then conforms Model Mutation Tool Model Mutant if if conforms Test Case Generator Abstract Test Case if conforms SUT Test Driver then pass then pass/fail / 13 / 19

Reactive Systems React to the environment Do not terminate Servers and Controllers Events: controllable and observable communication events Test cases: sequences of events Unlock AlarmSystem_StateMachine Unlock OpenAndUnlocked Open Close Lock Unlock ClosedAndUnlocked OpenAndLocked Unlock Lock Close Open ClosedAndLocked 20 Close Armed SilentAndOpen Show Armed /entry Show Unarmed /exit Open Alarm Activate Alarms /entry Deactivate Alarms /exit FlashAndSound 30 / Deactivate Sound Flash 300 17 obs pass obs AlarmArmed_SetOff 11 ctr Unlock 16 obs AlarmArmed_SetOn 15 ctr Close 14 obs OpticalAlarm_SetOff 13 obs AcousticAlarm_SetOff 12 obs after(270) 10 obs AcousticAlarm_SetOff 9 obs after(30) 8 obs AcousticAlarm_SetOn 7 obs OpticalAlarm_SetOn 6 obs AlarmArmed_SetOff 5 ctr Open 4 obs AlarmArmed_SetOn 3 obs after(20) 2 ctr Lock 1 Adaptive test cases: trees branching at non-deterministic observations ctr Close 0 14 / 19

Semantics Operational semantics e.g. Labelled Transition Systems Input-output conformance (ioco) [Tretmans96] 10 ctr Unlock 11 obs AcousticAlarm_SetOff ctr Unlock 2 obs after (c_waittime: 30 ) 8 obs after (c_waittime: 270 ) obs AcousticAlarm_SetOn 7 obs OpticalAlarm_SetOn 4 SUT ioco Model = df 15 σ traces(model) : obs AcousticAlarm_SetOff obs AlarmArmed_SetOff 14 obs AcousticAlarm_SetOff out(sut after σ) out(model after σ) ctr Open 13 1 out... outputs + quiescence after... reachable states after trace obs AlarmArmed_SetOn ctr Unlock 12 5 obs after (c_waittime: 20 ) obs AlarmArmed_SetOff 16 ctr Lock ctr Close ctr Open ctr Unlock obs OpticalAlarm_SetOff ctr Close 9 Model: 6 ctr Open obs OpticalAlarm_SetOff!soundOn 15 / 19 ctr Close 0 ctr Lock ctr Unlock 17 ctr Unlock 3

Explicit Conformance Checking Model and Mutant LTS Determinisation Model:!soundOn!flashOn!soundOn!flashOn Mutant:!soundOff!flashOn?unlock Build synchronous product modulo ioco If mutant has additional!output: fail sink state?input: pass sink state Model ioco Mutant:!soundOn!flashOn?unlock!soundOn pass!soundoff pass pass fail Extract test case covering fail state 16 / 19

Applications of Explicit Conformance Checking HTTP Server (LOTOS) SIP Server (LOTOS) Controllers (UML) Hybrid Systems (Action System) Scalability: abstractions for data-intensive models Bernhard K. Aichernig and Corrales Delgado. From Faults via Test Purposes to Test Cases: On the Fault-Based Testing of Concurrent Systems, FASE 2006. Martin Weiglhofer, Bernhard K. Aichernig, and Franz Wotawa. Fault-based conformance testing in practice. International Journal of Software and Informatics, 3(2-3):375-411, 2009. Chinese Academy of Science. Bernhard K. Aichernig, Harald Brandl, Elisabeth Jöbstl, and Willibald Krenn. Efficient mutation killers in action, ICST 2011. Harald Brandl, Martin Weiglhofer, and Bernhard K. Aichernig. Automated conformance verification of hybrid systems, QSIC 2010. 17 / 19

Agile Development 6$1*7+")& 345%8&!"#$%& 345%$4$(+&,$-+&.*-$-& '$($)*+$&&,$-+&.*-$-& /$)012&,$-+&.*-$-& Model-driven development Model-based test case generation Formal verification Test-driven development 18 / 19

Summary Model-based Testing + Mutation Testing Test case generation via ioco check Industrial applications in EU projects MOGENTES, MBAT, CRYSTAL Testing cannot show the absence of bugs [Dijkstra72]. Testing can show the absence of specific bugs [Aichernig12]. 19 / 19

Quality Assurance in Software Development Qualitätssicherung in der Softwareentwicklung A.o.Univ.-Prof. Dipl.-Ing. Dr. Bernhard Aichernig Institute for Software Technology Graz University of Technology Austria Summer Term 2013 1/23

Übersicht der Vorlesung 1 Reviews Formale Design Reviews Peer Reviews 2/23

Review-Prozeß Definition(IEEE(1990)) Aprocessormeetingduringwhichaworkproduct,orsetofwork products, is presented to project personnel, managers, users, customers, or other interested parties for comment or approval. Methoden: Formale Design Reviews Peer Reviews: Inspektionen Walkthroughs 3/23

Direkte Ziele Auffinden von Analyse- und Entwurfsfehlern, als auch von Bereichen wo Korrekturen und Vervollständigungen bzgl. der Spezifikationen notwendig sind. Identifizieren von neuen Risiken, bzgl. der Fertigstellung des Produkts. Lokalisieren von Abweichungen von Konventionen um größere Uniformität zu erreichen. Bestätigen des Analyse- oder Entwurfsdokuments um fortfahren zu können. 4/23

Indirekte Ziele Bieten eines informellen Forums zum Austausch von Expertenwissen(Entwicklungsmethoden, Werkzeuge, Techniken). Aufzeichnen von Analyse und Designfehlern als Basis für zukünftige Verbesserungen des Entwicklungsprozesses und des Produkts. Stimulation des Entwicklungsteams ihr Bestes zu geben. Die Review-Methoden unterscheiden sich durch Betonung der verschiedenen Ziele. 5/23

(Formale) Design Reviews auch Formale Technische Reviews genannt Besonderheit: Notwendig um das Entwurfsprodukt zu bestätigen undindienächstephasezugehen. können zu jedem Meilenstein stattfinden Example Development Plan R., Software Requirement Specification R., Preliminary Design R., Detailed Design R., Data Base Design R., Test Plan R., Software Test Procedure R., Version Description R., Operator Manual R., Support Manual R., Product Release R. 6/23

Teilnehmer Review-Leiter: WissenundErfahrunginProjektendiesesTyps. Dienstalter(Stellung)ähnlich,wennnichthöherals,Projektleiter. GuteBeziehungzumProjektleiterundseinemTeam. EinePositionexternzumProjekt. Kandidaten:ManagerderEntwicklungsabteilung, Chef-Softwareingeniere, Projektleiter anderer Projekte, Leiter der Qualitätssicherung, Chef-Softwareingenieur des Kunden, Konsulenten. Review-Team Senior-MembersdesProjektsu.andererProjekte, Kundenrepräsentanten, Konsulenten. 3-5Mitglieder(Mehrheitextern!) 7/23

Vorbereitungen Review-Leiter BerufenderMitglieder ZeitlichesFestlegenderReview-Sessions. VerteilendesEntwurfsdokumentesunterdenMitgliedern. Review-Team ReviewderDesign-Dokumente(kannauchaufgeteiltwerden), Auflisten der Kommentare VerwendungvonChecklisten Entwicklungsteam VorbereitungeinerkurzenPräsentationdesEntwurfdokumentes, mitfokusaufdaswichtigste,dasbestätigungbedarf. Achtung!ZulangePräsentationensabotierenReview-Sessions. 8/23

Session-Agenda Dauer: 2h 1 Kurze Präsentation des Entwurfsdokumentes 2 Kommentare des Review-Teams 3 Verifikation und Validierung in der jedes der Kommentare diskutiert wird um die notwendigen Aktionen(Korrekturen, Änderungen, Hinzufügungen) festzulegen. 4 Entscheidung bzgl. des Dokumentes: VolleBestätigung:sofortindienächstePhasedesProjektes,nur kleine Änderungen können gefordert werden. PartielleBestätigung:mitTeilendesProjektesinnächstePhase, mit notwendigen Aktionen für den Rest. Bestätigung nach Änderung durch einen Reviewer oder einer speziellen Session. VerweigerungderBestätigung:Reviewmusswiederholtwerden;bei mehreren schweren bzw. kritischen Fehlern. 9/23

Review-Report Schnelle Herausgabe mit folgenden Sektionen: Zusammenfassung der Review-Diskussionen Entscheidung Eine vollständige Liste aller Aktionen, mit Fertigstellungsdatum und Verantwortlichen. Die Namen der Reviewers, welche die Performance der Änderungen überwachen sollen. Kennzeichen schlechter Review-Reports: Extrem kurzer Report, der nur die Bestätigung ohne Fehler dokumentiert. Ein Report, der Aktionen versch. Gewichtung listet, aber keinen Hinweis auf Follow-up-Aktionen(Zeitplan) beinhaltet. 10/23

Peer Reviews Unterschied zu Design-Reviews liegt in Teilnehmern und deren Autorität: DR-Teilnehmer:überdemProjektleiter,Kundenrepräsentanten PR-Teilnehmer:aufgleicherStufemitProjektleiter,ausseiner Abteilung ZielistdasFindenvonFehlernunddasAbweichenvonStandards. Methoden: Inspektionen:BetonungaufkorrektivenAktionenbzgl.des Entwicklungsprozesses(formeller),[Fagan 1976] Walkthroughs:KommentierungdesDokumentes[Yourdon1979]. Kontroverse um bessere Methode und Definition. 11/23

Inspektionen Checklisten für jeden Typ von Designdokumenten und Programmiersprachen und Werkzeuge, die periodisch upgedated werden. Tabellen mit typischen Häufigkeiten von Fehlertypen, basierend auf vergangenen Inspektionen, um die Inspektoren zu den Problemzonen zu leiten(vorbedingungen!). Ausbildungen zum Software-Inspektor. Periodische Analyse der Effektivität von vergangenen Inspektionen um die Inspektionsmethode zu verbessern. Einführung von Inspektionen in den Zeitplan mit notwendigen Resourcen. 12/23

Teilnehmer bei Inspektionen 3-5 Teilnehmer: Autor und Ebenbürtige(Peers). Moderator: ist der Leiter. Designer: Systemanalyst des Projekts Kodierer: Bevorzugterweise der Leiter des Kodierungsteams Tester: Bevorzugterweise der Leiter des Testteams. Rollen: Präsentator:vomModeratorgewählt,nichtderAutordes Dokumentes, oft der Kodierer, da er am besten die Implikationen auf die Implementierung versteht; zudem ist er neutral und beeinflußt das Urteil der Reviewer weniger. Protokollführer:oftderModerator 13/23

Teilnehmer bei Walkthroughs 3-5 Teilnehmer: Autor und Ebenbürtige(Peers). Koordinator: ist der Leiter. Standards Enforcer: Lokalisieren von Abweichungen von Entwicklungsstandards. Wartungsexperte: Wartbarkeit, Flexibilität, Testbarkeit Kundenrepräsentant: interner bzw. externer Benützer zur Validierung. Rollen: Präsentator:derAutor,daamvertrautestenmitDokument. Protokollführer:oftderModerator 14/23

Vorbereitungen Leiter: FestlegenderTeile,diezuReviewensind(mitdemAutor): komplexe, kritische oder bekannt fehleranfällige AuswahlderMitglieder Zeitl.FestsetzenderSessions <2h,biszweiSessionsproTag VerteilendesDokumentsandieTeilnehmervorderReview-Session. Teilnehmer an Inspektionen: Überblickstreffen: Autor gibt Hintergrundinformationen zum Projekt, Schnittstellen, Logik, Prozeß etc.(kann entfallen). LesenderDokumentsektionenundAuflistenderKommentare. Teilnehmer an Walkthroughs: KurzesDurchlesendesDokumentsumÜberblickzubekommen. Kommentarewerdennichtvorbereitet. 15/23

Inspection-Session Präsentator liest einen Abschnitt des Dokuments vor und kommentiert Schwierigkeiten Teilnehmer geben ihre Kommentare: Diskussionen sollten sich auf Fehler nicht auf Lösungen konzentrieren. Protokollführer dokumentiert Fehler mit Ort, Beschreibung, Typ und Charakter(inkorrekt, fehlender Teil, Extrateile) und Schwere des Fehlers. 16/23

Walkthrough-Session Autor gibt kurze Präsentation zum Hintergrund und Überblick des Projekts. Autor liest einen Abschnitt des Dokuments vor und kommentiert Schwierigkeiten. Teilnehmer geben ihre Kommentare: Diskussionen sollten sich auf Fehler nicht auf Lösungen konzentrieren. Protokollführer dokumentiert Fehler mit Ort, Beschreibung, Typ und Charakter(inkorrekt, fehlender Teil, Extrateile). 17/23

Fehlerklassifizierung bei Inspektionen Nach MIL-STD-498[DOD 1994]: 1 (geringfügig), jeder andere Effekt. 2 Anwender/Operator-Unbequemlichkeit ohne die essentiellen Fähigkeiten zu beeinträchtigen. Unbequemlichkeit für die Entwickler oder das Wartungspersonal, aber verhindert nicht die Realisierung deren Verantwortlichkeit. 3 wie 4, es ist aber eine work-arround-lösung bekannt. 4 Behindert das Erreichen essentieller Fähigkeiten, wo keine work-arround-lösung bekannt ist. Betrifft technische, Kosten-, oder Zeitplan-Risiken, oder Wartbarkeit wo keine work-arround-lösung bekannt ist. 5 (kritisch) Verhindert Erreichen essentieller Fähigkeiten. Beeinträchtigt Safety, Security oder andere kritische Anforderungen. 18/23

Reports bei Reviews Inspection Session Findings Report: Wird sofort nach Beendigung der Session vervollständigt und verteilt. Zweck ist die Dokumentation der Fehler und Follow-up. Inspection Session Summary Report: Nach den Sessions, die ein Dokument betreffen. Zusammenfassung des Gefundenen und der eingesetzten Resourcen. Präsentiert grundlegende Qualitäts- und Effizienzmetriken. Walkthrough Session Findings Report: die Fehlerdokumentation geht an das Entwicklerteam. 19/23

Inspection Session Summary Report Daten Resourcen: Stunden gearbeitet pro Teammitglied Fehlerzusammenfassung: Anzahl, Art und Schwere der Fehler. Fehleranzahl wird standardisiert: Multiplikator je nach schwere des Fehlers(z.B.5 16,4 8,3 4,2 2,1 1) Defektfindungsmetriken: DurchschnittlicheFehlerproSeite(LOC) DuchschnittlichestandardisierteFehlerproSeite(LOC) Defektfindungseffizienz:Stundenpro(standardisierten)Defekt. 20/23

Daten: National Software Quality Experiment 1992 27 Inspektionslaboratorien, 90 925 LOC Ges. Anzahl gefundener Fehler: 1849 Anzahl schwerer Fehler: 242 Ges. Vorbereitungszeit: 22828 Minuten Durchschn. Vorbereitungszeit pro Fehler: 12.3 Minuten Durchschn. Vorbereitungszeit pro schweren Fehler: 94.3 Minuten Gesamte Fehlerdichte(Fehler pro KLOC): 20.30 Schwere Fehlerdichte(Fehler pro KLOC): 2.66 21/23

Daten: Litton Projekt 1998 150Inspektionenvon276422LOC Ges. Anzahl gefundener Fehler: 7165 Anzahl schwerer Fehler: 772 Ges. Vorbereitungszeit: 4612 Stunden Durchschn. Vorbereitungszeit pro Fehler: 1.5 Stunden Durchschn. Vorbereitungszeit pro schweren Fehler: 9.3 Stunden Gesamte Fehlerdichte(Fehler pro KLOC): 25.90 Schwere Fehlerdichte(Fehler pro KLOC): 2.80 22/23

Peer-Review Abdeckung Nur 5-15% werden reviewed: Komplizierte Logik Kritische Abschnitte Abschnitte die neue Umgebungen behandeln Abschnitte von neuen oder unerfahrenen Mitgliedern 23/23