Software Engineering. How does software fail? Terminology CS / COE 1530

Size: px

Start display at page:

Download "Software Engineering. How does software fail? Terminology CS / COE 1530"

Oswin Logan
8 years ago
Views:

1 Software Engineering CS / COE 1530 Testing How does software fail? Wrong requirement: not what the customer wants Missing requirement Requirement impossible to implement Faulty design Faulty code Improperly implemented design Terminology Fault identification: what fault caused the failure? Fault correction / removal: change the system to take out the fault 1

to implement Faulty design Faulty code Improperly implemented design Terminology Fault

2 Types of faults Algorithmic fault Syntax fault ν syntax error Computation and precision fault Documentation fault Stress or overload fault Capacity or boundary fault Timing or coordination fault Throughput or performance fault Recovery fault Hardware and system software fault Standards and procedures fault Chillarege et al. s Orthogonal Defect Classification Orthogonal each classified item belongs to exaclty one class Fault of omission Fault of commission Table 8.1. IBM orthogonal defect classification. Fault type Function Interface Checking Assignment Timing/serialization Build/package/merge Documentation Algorithm Meaning Fault that affects capability, end-user interfaces, product interfaces, interface with hardware architecture, or global data structure Fault in interacting with other components or drivers via calls, macros, control blocks or parameter lists Fault in program logic that fails to validate data and values properly before they are used Fault in data structure or code block initialization. Fault that involves timing of shared and real-time resources Fault that occurs because of problems in repositories, management changes, or version control Fault that affects publications and maintenance notes Fault involving efficiency or correctness of algorithm or data structure but not design 2

s Orthogonal Defect Classification Orthogonal each classified item belongs to exaclty one class Fault of omission Fault of commission Table 8.1. IBM orthogonal defect classification.

3 HP Fault Classification Table 8.2. Typical inspection preparation and meeting times. Development artifact Preparation time Meeting time Requirements document 25 pages per hour 12 pages per hour Functional specification 45 pages per hour 15 pages per hour Logic specification 50 pages per hour 20 pages per hour Source code 150 lines of code per hour 75 lines of code per hour User documents 35 pages per hour 20 pages per hour Table 8.3. Faults found during discovery activities. Discovery activity Faults found per thousand lines of code Requirements review 2.5 Design review 5.0 Code inspection 10.0 Integration test 3.0 Acceptance test 2.0 Test Organization Test stages Unit testing (aka. module or component testing) Integration testing components work together as described in system and program design specification Function testing functions described in requirements specification are performed 3

per hour 20 pages per hour Source code 150 lines of code per hour 75 lines of code per hour User documents 35 pages per hour 20 pages per hour Table 8.3. Faults found during discovery activities.

4 Test Organization (2) Performance testing with respect to specified hardware and software environment Validation testing test in customer s actual working environment validated system if successful Acceptance test system is checked against customer s requirements at site followed by final installation run Test Organization Black Box vs. White Box Testing Black box tester does not know anything about the test object internals White box tester knows about internals and exploits that knowledge to write good test cases Example: know logic structure can ensure branch coverage in test cases 4

installation run Test Organization Black Box vs.

5 How to Find Faults? Reviews Proof techniques static analysis techniques Testing Dynamic Analysis tools such as prefix, ESP, valgrind Code Review Walkthrough developer presents code and doucumentation and leads review Inspection review team checks code & documentation against a list of concerns overview meeting + individual inspection + inspection meeting Jones showed that code review is most effective technique for finding faults Proving code correct Formal proof techniques done by hand Symbolic execution Recently successful: symbolic model checking build model and check if no fault the original system is correct Automated theorem-proving works well in specialized domains but hard in general 5

doucumentation and leads review Inspection review team checks code & documentation against a list of concerns overview meeting + individual inspection + inspection meeting

6 Unit Testing Testing can only show the presence of faults not their absence Successful testing requires good test cases test case (point) = particular choice of input data used to test test =finite collection of test cases Ideally, want to to test all possible behaviors in practice: state space explosion Test thoroughness Statement testing Branch testing Path testing Definition-use testing All-uses testing All-predicate-uses/some-computational-uses testing All-computational-uses/some-predicate-uses testing 6

possible behaviors in practice: state space explosion Test thoroughness Statement testing Branch testing Path testing

7 Comparing techniques Table 8.5. Fault discovery percentages by fault origin. Discovery technique Requirements Design Coding Documentation Prototyping Requirements review Design review Code inspection Unit testing Table 8.6. Effectiveness of fault discovery techniques. (Jones 1991) Requirements faults Design faults Code faults Documentation faults Reviews Fair Excellent Excellent Good Prototypes Good Fair Fair Not applicable Testing Poor Poor Good Fair Correctness proofs Poor Poor Fair Fair Test Strategy Strength Integration testing Bottom-up Top-down Big-bang Sandwich testing Modified top-down Modified sandwich 7

Unit testing 1 5 20 0 Table 8.6. Effectiveness of fault discovery techniques.

8 Example Hierarchy Bottom-up Integration Testing Uses Component Drivers Top-down Integration Testing Uses component stubs 8

9 Modified Top-down Big Bang Integration Testing Not advised: need many drivers need many stubs hard to isolate faults Table 8.7. Comparison of integration strategies. Bottom-up Top-down Modified top-down Big-bang Sandwich Modified sandwich Integration Early Early Early Late Early Early Time to Late Early Early Late Early Early basic working program Component Yes No Yes Yes Yes Yes drivers needed Stubs No Yes Yes Yes Yes Yes needed Work Medium Low Medium High Medium High parallelism at beginning Ability to Easy Hard Easy Easy Medium Easy test particular paths Ability to plan and control sequence Easy Hard Hard Easy Hard Hard 9

Bottom-up Top-down Modified top-down Big-bang Sandwich Modified sandwich Integration Early Early Early Late Early Early Time to Late Early Early Late Early

10 Microsoft Synch-and-Stabilize Traditional vs. OO-Testing Test planning Establish test objectives Design test cases Write test cases Test test cases Execute tests Evaluate test results 10

11 The Test Plan Describes how software will be demonstrated to be correct have to know requirements, specifications, & design Plan contents: series of tests (at unit, integration, functional, acceptance & installation level) how tests will be run and criteria when test is complete and satisfactory 11

of tests (at unit, integration, functional, acceptance & installation

Software testing. Objectives

Software testing. Objectives Software testing cmsc435-1 Objectives To discuss the distinctions between validation testing and defect testing To describe the principles of system and component testing To describe strategies for generating