Ingegneria del Software II academic year: Course Web-site: [

Transcription

1 Course: Ingegneria del Software II academic year: Course Web-site: [ An Introduction to Testing Lecturer: Henry Muccini and Vittorio Cortellessa Computer Science Department University of L'Aquila -Italy [ [

2 Copyright Notice» The material in these slides may be freely reproduced and distributed, partially or totally, as far as an explicit reference or acknowledge to the material author is preserved. Henry Muccini 2

3 Acknowledgment» With very special thanks to Antonia Bertolino and Debra J. Richardson which collaborated in previous versions of this lecture note 3

4 Agenda» Testing Basics - Definition of Software Testing» Testing Glossary» Testing Approaches 4

5 Testing basics» Definition of swtesting» Fault vs. failure» Test selection - Coveragetesting - Reliability testing - Specification-basedtesting» Traceability and test execution 5» Testing from LTS

6 Testing» Testingis a word with manyinterpretations - Inspection/review/analysis - Debugging - Conformance testing - Usability testing - Performance testing -. 6

7 What testing is not (citationfromhamlet, 1994):» I've searched hard for defects in this program, found a lot of them, and repaired them. I can't find any more, so I'm confident there aren't any. The fishing analogy:» I've caught a lot of fish in this lake, but I fished all day today without a bite, so there aren't any more.» NB: Quantitatively, a day's fishing probes far more of a large lake than a year's testing does of the input domain of even a trivial program. 7

8 An all-inclusive definition: Software testing consists of:» the dynamic verification of the behavior of a program» on a finite set of test cases» suitably selected from the (in practice infinite) input domain» against the specified expected behavior 8

9 Testing vs. Other Approaches» Where isthe boundarybetween testing and other (analysis/verification) methods? What distinguishessoftware testing from other approachesis execution. Thisrequiresthe abilityto: - launch the tests on some executable version (traceability problem), and - analyse (observe and evaluate) the results(not granted for embedded systems)» I am not sayingtestingissuperior and we should disregard other approaches: on the contrary, testingand other methodsshould complement/support eachother 9

10 Test purpose» The software is tested in orderto: - Find bugs: debug testing, e.g., by: > Conformancetesting > Robustness testing > Coverage testing - Measure its reliability:by statistical inference - Measure itsperformance - Release it tothe customer(acceptancetest) - not mutuallyexclusive goals, butrather complementary 10» Of coursedifferent approaches are taken

11 The high cost of testing» Anysoftware test campaign involves a trade-off between - Limiting resource utilization(time,effort) as few testsaspossible - Augmentingourconfidence as many testsaspossible» Two research challenges: - Determininga feasibleand meaningful stoppingrule - Evaluatingtest effectiveness (reliability, coveragenotion,...verytoughproblem 11

12 Testing Glossary 12

13 Failures, Errors, Faults» Failure: incorrect/unexpected behavior or output - incident is the symptoms revealed by execution - failures are usually classified» Potential Failure: incorrect internal state - sometimes also called an error, state error, or internal error» Fault: anomaly in source code - may or may not produce a failure - a bug» Error: inappropriate development action 13 - action that introduced a fault

14 THE FAULT-FAILURE MODEL Fault Error Failure affects the delivered service must be activated sw internal state is corrupted (latent or manifest) A fault will manifest itself as a failure if all of the 3following conditions hold: 1) EXECUTION: the faulty piece of code is executed 2) INFECTION: the internal state of the program is corrupted: error 3) PROPAGATION: the error propagates to program output 14 PIE MODEL (Voas, IEEE TSE Aug. 1992)

15 The ambiguous notion of a fault» What a tester or a user observes is a failure. What is then a fault? - It is a useful and intuitive notion, but very difficult to formalize.» In practice we often characterise a fault with the modifications made to fix it: e.g. Wrong operator. In such a way, - the concept of a fault becomes meaningful only after it has beenfixed. - Moreover, different people may react to a same failure with different fixes (and clever programmer find minimum fixes)» A solution is to avoid the ambiguous notion of a fault and reason instead in terms of failure regions, i.e. collections of input points that give rise to failures. - A failure region is characterisedby a size and a pattern 15

16 Fundamental Testing Questions» Test Case: How is test described / recorded?» Test Criteria: What should we test?» Test Oracle: Is the test correct?» Test Adequacy: How much is enough? 16» Test Process: Is testing complete and effective? How to make the most of limited resources?

17 Test Case» Specification of - identifier - test items - input and output specs - environmental requirements - procedural requirements» Augmented with history of - actual results - evaluation status 17 - contribution to coverage

18 Test Criterion» Test Criterion provides the guidelines, rules, strategy by which test cases are selected requirements on test data -> conditions on test data -> actual test data» Equivalence partitioning is hoped for - a test of any value in a given class is equivalent to a test of any other value in that class - if a test case in a class reveals a failure, then any other test case in that class should reveal the failure 18 - some approaches limit conclusions to some chosen class of faults and/or failures

19 Theory of Testing Let P S D T C =program under test =specification of P =input domain of S and P = subset of D, used as test set for P = test criterion» P is incorrect if it is inconsistent with S on some element of D» T is unsuccessful if there exists an element of T on which P is incorrect 19

20 Subdomain-based Test Criterion» A test criterion C is subdomain-based ifit induces one or more subsets, or subdomains, of the input domain» A subdomain-based criterion typically does not partition the input domain (into a set of non-overlapping subdomains whose union is the entire domain) 20

21 Test Oracle 21» A test oracle is a mechanism for verifying the behavior of test execution - extremely costly and error prone to verify - oracle design is a critical part of test planning» Sources of oracles - input/outcome oracle - tester decision - regression test suites - standardized test suites and oracles - gold or existing program - formal specification

22 The oracle problem The Expected Output is f*(d) YES, given input d f(d) = f*(d)» In some cases easier (e.g., an existing version, existing formal specification), but generally very difficult (e.g., operational testing) 22» Not enough emphasized research problem?

23 Test Adequacy» Theoretical notions of test adequacy are usually defined in terms of adequacy criteria - Coverage metrics (sufficient percentage of the program structure has been exercised) - Empirical assurance (failures/test curve flatten out) - Error seeding (percentage of seeded faults found is proportional to the percentage of real faults found) - Independent testing (faults found in common are representative of total population of faults) 23» Adequacy criteria are evaluated with respect to a test suite and a program under test

24 Testing Approaches 24

25 Unit, Integration and System Testing [BE90, ch1]» Unit Testing: - A unit is the smallest piece of software > A unit may be a procedure, a piece of code, one component (in object oriented systems), one system as a whole. - The Unit test purpose is to ensure that the uniti satisfies its functional specification and/or that its implemented structure matches the intended design structure - Unit tests can also be applied for test interface or local data structure. 25

26 Integration Testing» Integration testing is specifically aimed at exposing the problems that arise from the combination of components - Assuming that components have been unit tested - Especially communicating interfaces among integrated components need to be tested to avoid communication errors.» Different approaches: - non-incremental: big-bang - incremental: top-down, bottom-up, mixed - In Object-Oriented distributed systems: new techniques, based on dependence analysis or dynamic relations of inheritance and polymorphism. 26

27 System Testing» Involves the whole system embedded in its actual hardware environment» It attempts to reveal bugs which depends on the environment (bugs that cannot be attributed to components)» Recovery testing, security testing, stress testing and performance testing 27

28 Integration Testing» Stress testing: designed to test the software with abnormal situations. - Stress testing attempts to find the limits at which the system will fail through abnormal quantity or frequency of inputs. - The test is expected to succeed when the system is stressed with higher rates of inputs, maximum use of memory or system resources.» Performance testing is usually applied to real-time, embedded systems in which low performances may have serious impact on the normal execution. - Performance testing checks the run-time performance of the system and may be coupled with stress testing. - Performance is not strictly related to functional requirements: functional tests may fail, while performance ones may succeed. 28

29 Henry Muccini s methafor» The metaphor I like to use to explain how unit, integration and system testing differ considers - a unit as a musician, playing his/her instrument. Unit esting consists in verifying that the musician produces high quality music. - Putting together musicians each one playing good music is not enough to guarantee that the group (i.e., the integration of units) produces good music. They may play different pieces, or the same piece with different rhythms. - The group may then play in different environments (e.g., theater, stadium, arena) with different results (i.e., system testing). 29

30 Testing Planning throughout Process levels of abstraction User Requirements Software Requirements Specification plan & validate /verify Acceptance Testing System Testing Architecture Design Specification Integration Testing design & analyze Component Specifications Component Testing integrate & test 30 Unit Implementations Unit Testing time

31 Operational Testing» A completely different approach: - We cannot realistically presume to find and remove the last failure. - Then, we should invest test resources to find out the biggest ones.» Goodness here means to minimise the user perceived faults, i.e. - Try to find earlier those bugs that are more frequently encountered (neglect small bugs) 31 - Not suitable for safety critical applications

32 Structural and Functional Testing» Structural Testing - Test cases selected based on structure of code - Views program /component as white box (also called glass box testing)» Functional Testing - Test cases selected based on specification - Views program/component as black box 32 Can do black-box testing of program by white-box testing of specification

33 Black Box vs. White Box Testing SELECTED INPUTS RESULTANT OUTPUTS DESIRED OUTPUT BLACK BOX TESTING SELECTED INPUTS RESULTANT OUTPUTS INTERNAL BEHAVIOR DESIRED OUTPUT SOFTWARE DESIGN 33 WHITE BOX TESTING

34 Structural (White-Box) Test Criteria» Criteria based on - control flow - data flow - expected faults» Defined formally in terms of flow graphs» Metric: percentage of coverage achieved» Adequacy based on metric requirements for criteria 34 Objective: Cover the software structure

35 Flow Graphs» Control Flow - The partial order of statement execution, as defined by the semantics of the language» Data Flow - The flow of values from definitions of a variable to its uses Graph representation of control flow and data flow relationships 35

36 A Sample Program to Test function P return INTEGER is begin X, Y: INTEGER; READ(X); READ(Y); while (X > 10) loop X := X 10; exit when X = 10; end loop; if (Y < 20 and then X mod 2 = 0) then Y := Y + 20; else Y := Y 20; end if; return 2*X + Y; end P; 36

37 P s Control Flow Graph (CFG) ,3,4 5 T F T F 9 T 9a 9 9b T 14 F F 12 37

38 All-Statements Coverage of P At least 2 test cases needed ,3,4 5 T F F 9 T T 14 Example all-statements-adequate test set: (X = 20, Y = 10) <2,3,4,5,6,7,9,10,14> (X = 20, Y = 30) <2,3,4,5,6,7,9,12,14> F 12 38

39 All-Edges Coverage of P At least 3 test cases needed ,3,4 5 T F F 9a T T 9b T Example all-edges-adequate test set: (X = 20, Y = 10) <2,3,4,5,6,7,9a,9b,10,14> (X = 5, Y = 30) <2,3,4,5,9a,12,14> (X = 21, Y = 10) <2,3,4,5,6,7,5,6,7,5,9a,9b,12,14> F F 12

40 P s Control and Data Flow Graph X Y X X X 2,3,4 5 T 6 F F X 9a T 7 T X 9b T 10 Y 14 X Y X F F 12 Y Y X 40

41 Functional Testing» The internal structure and behavior of the program is not considered» What is assumed is to have the (formal or informal) specification of the system under test and the objective is to analyze if the system correctly behaves with respect to the specifications» This analysis technique is particularly useful when the source code is not available 41» A functional test consists of analyzing how the system reacts to external inputs

42 References» [BE90] B. Beizer. Software Testing Techniques. Van Nostrand Reinhold, 2nd. Edition, 1990.» [M03] E. Marchetti. Software Testing in the XXI century: Methods, Tools and New Approaches to Manage, Control and Evaluate This Critical Phase. Ph.D. Thesis, Dipartimento di Informatica, Universita di Pisa, year 2003, 42