Traditional Coverage Metrics: How Meaningful Are They?

Transcription

1 Traditional Coverage Metrics: How Meaningful Are They? Ian Gilchrist International Conference On Software Testing, Analysis & Review November Stockholm, Sweden P r e s e n t a t i o n W8 Wednesday 21st November, 2001

2 Wednesday 21 November 2001 W8 Traditional Coverage Metrics: How Meaningful Are They? Ian Gilchrist Ian Gilchrist graduated in Philosophy and Psychology from Oxford University. He started his career in software in 1983, beginning with assembly language (Z80) programming of real-time systems. He was involved in projects for industrial control, and railway-signalling and avionics systems, adding experience with other languages such as Fortran, C, ladder-logic and Ada. Project roles ranged from programmer to project manager. For the last ten years he has been involved in the development and marketing of IPL s software testing tools, Cantata/++ and AdaTEST. Contact details: IPL, Eveleigh House, Grove St, BATH BA1 5LR, UK. Tel (+44)-(0) Ian.Gilchrist@iplbath.com.

3 Traditional Coverage Metrics: how meaningful are they? Ian Gilchrist, IPL, Bath, UK Silke Kuball, University of Bristol, UK 03/11/2001 1

4 Scope of talk Research conducted in early 2001, looking at relationships between: Traditional coverage metrics Software safety standards Statistical Software Testing (SST) Software dependability/reliability Context is software from the protection system of a (hypothetical) heavy water nuclear reactor. 03/11/2001 2

5 What you might learn Traditional (structural) coverage metrics should be enhanced by other techniques if they are to be used as a guide to the reliability of the software-under-test. There are other techniques (SST) which are potentially more rigorous at finding faults, and can additionally help quantify reliability. 03/11/2001 3

6 Definitions of terms Outline of aims and method Description of the software under test The test environment Results Conclusions Contents 03/11/2001 4

7 Definitions of Terms Outline of aims and method Description of the software under test The test environment Results Conclusions 03/11/2001 5

8 Definition - Statistical Software Testing SST is the statistical selection of test-case data, such that the Scenarios chosen reflect in a probabilistic fashion the expected operational use of that software. SST is currently the only recognised way to quantify dependability (apart from formal techniques). A set of Scenarios of Operation reflecting the entire input-space is used to aid modelling this probabilistic process. Input Bins (= Equivalence Partitions). Test cases are chosen from Scenarios (Bins) based on their (estimated) probability of occurrence in real use. Operational Input Distribution (OID) 03/11/2001 6

9 Definition Scenarios of Operation Specification for SuT Code etc Identification of Scenarios of Operation Initial Conditions Input Categories Expected Outputs Input-Space Partition Initial Conditions = Starting State (e.g. Power-Up ) Input Categories = Input types (e.g. Error Type 1 ) 03/11/2001 7

10 Definition Operational Input Distribution Example OID with Probabilities Input Type 1 Input Type 2 Input Type State State /11/2001 8

11 Definition Operational Input Distribution Example of OID with numbers of test cases, for total 50 Input Type 1 Input Type 2 Input Type 3 State State /11/2001 9

12 Definition Traditional Coverage Various measures looking at the proportion of code structures exercised during the running of a specified test set: % Statements (SC) % Decisions (DC) % Boolean Expressions (BC), etc To achieve 100% for these metrics in any testing regime is considered good. 03/11/

13 Example Coverage Requirements Example Structural Coverage requirements, from DO-178B: SIL Level (SIL Category) Statements Decisions Modified Boolean A Catastrophic 100% 100% 100% (Safety-Critical) B Hazardous 100% 100% - (Safety-related) C Major failure 100% - - (Mission-Critical) 03/11/

14 Definition software dependability A piece of software is considered dependable if: it is demonstrably free of faults AND the test set used to demonstrate that is judged good enough. This paper is really a discussion of how to decide what is good enough. Framework for dependability assessment from SST exists. 03/11/

15 Definitions of terms Outline of Aims and Method Description of the software under test The test environment Results Conclusions 03/11/

16 Aims of the study To compare results achieved when adopting both coverage-based approaches and SST to test case generation. What does this tell us about the usefulness of traditional coverage metrics? What could we gain by incorporating SST into our testing regime? 03/11/

17 The Study in four parts 1. Identify an initial set of Scenarios of Operation for the Software Under Test, and then have ONE test case from each bin/scenario. 2. Add more test cases from the refined set of scenarios to get full coverage. 3. Apply full (automated) SST. 4. Can we engineer a minimal test set to give full coverage? 03/11/

18 Definitions of terms Outline of aims and method Description of the Software under Test The test environment Results Conclusions 03/11/

19 Software under Test Part of (hypothetical) reactor protection system. Module (approx 280 lines of C code) is called Format_Check : Message is read, and its format is checked Message Status is set Message contains data on: Message Header Neutron Power Steam Pressure Water Level Message Trailer Message is 76-bytes long, mainly in Hex: \n02 0 B9F FFF FFFC55 C55 C55 EA6\r 03/11/

20 Structure of Software Under Test Level 0 Format_Check Calls (Level 1) Check_Validity Check_Format Check_Seq Check_Reset Calls (Level 2) strlen char_to_int isxdigit are_hexa are_spaces char_to_int next_seq_number is_reset 03/11/

21 State Data for Software under Test Variable Running_Mode Mssg_Counter Plant_Status_Trip_ Values Plant_Status_Data_ Statues Plant_Status_Msg_ Status Current message number. Description Current Plant Status 1 = Initialising 2 = Running_Safe 3 = Error 4 = Tripped >= 0 Values T, W, or N 0,1,2, or 3 0,1,2, or 3 03/11/

22 Definitions of terms Outline of aims and method Description of the software under test The Test Environment Results Conclusions 03/11/

23 Basic tools used Testing infrastructure provided by Cantata : Script generation from TCD file data Coverage analysis, including: Statements Decisions Modified Booleans Purpose-built TCDGenerator Samples input data from the modelled OID automatically Finds expected result in accordance with requirements specification Creates TCD file for input to Cantata 03/11/

24 Testing Process Schematic Spec. Code Domain Knowledge CANTATA Test Results Scenario of Ops and OID TCD Gen. 03/11/

25 Initial Scenario of Operations There are 9 possible message format errors: First char /= linefeed Separator /= space Data-byte /= hexadecimal etc There are 4 possible Plant Statuses: Safe, Power-Up, Trip, Error Expected Output is Message_Status: 0,1,2,or 3 03/11/

26 Initial Scenario of Operations (Partial) Scenario of Operations, in tabular form, with expected Output: Ij Ci 1. No error/ No reset 2. Reset 7. Cksum /= Hex 8. Cksum Not Correct 9. Seq No. Wrong 1.Safe Power-Up Trip Error /11/

27 Definitions of terms Outline of aims and method Description of the software under test The test environment Results Conclusions 03/11/

28 Results Part 1 1. Identify an initial set of Scenarios of Operation for the Software Under Test, and then have ONE test case from each bin/scenario. What coverage is achieved? Have we modelled the software input-space appropriately? We can hypothesise that we should have full coverage of all Level 1 functions, since the initial set of Scenarios was built based on the requirements specification. 03/11/

29 Results Part 1 Actual coverages achieved from initial test data set: Function in SUT Statements Decisions Booleans Modified Booleans Format_Check 100% Check_Validity 100% 90% 66% 33% Check_Format 100% 100% 83% 60% Check_Sequence 100% 100% 60% 50% Check_Reset 100% 100% 100% 100% 03/11/

30 Results Part 1 These coverage results are contrary to our hypothesis, so what is the explanation? The programmer expanded on the specification, so the set of Scenarios of Operation needed to build an appropriate input-space presentation is actually bigger than we thought! How to react? Expand the Scenarios of Operation to cover these aspects of system complexity. 03/11/

31 2. Add more test cases from the refined set of scenarios to get full coverage. Have we found bugs? Results Part 2 Revise Scenario of Operations to have 13 message error types, and 6 plant states, giving total of 78 bins. Repeat test, with ONE test case from each bin. No faults reported. 100% coverage of all metrics for all functions achieved. 03/11/

32 Results Part 2 By traditional coverage standards (e.g. DO-178B) we can stop testing: There are no evident faults. We have achieved full structural coverage as required for even Safety-Critical applications. We can easily argue that we have fully covered all software Requirements. BUT! 03/11/

33 3. Apply full (automated) SST. Have we found bugs? Results Part 3 Automated generation of 450 test cases according to OID. Assigned representational probabilities to each of: Input Categories (format error types) Initial Conditions (plant statuses) 03/11/

34 Results Part 3 Expanded (partial) OID Probabilities: Ci 1. Seq-no. 2. Checksum 11. Char Reset 13. No format Ij incorrect Incorrect /= LF Error/ No Reset 1. Safe Power-Up Mssg_Counter = I/F failure, Status = /11/

35 Results Part 3 (Partial) OID numbers of test cases, for total 450 : Ci 1. Seq-no. 2. Checksum 11. Char Reset 13. No format Ij incorrect Incorrect /= LF Error/ No Reset 1. Safe Power-Up Mssg_Counter = I/F failure, Status = /11/

36 Results Part 3 At this point testing with SST exposed two faults in the SuT! Message byte 5 (reset) is not being checked for Hexadecimal. Could lead to a hazardous reset. Message byte 23 was not being checked for hexadecimal. It could also now be used to give a reliability estimate for this unit. 03/11/

37 Results Part 4 4. Can we engineer a minimal test set to give full coverage? What bugs are found? It was found possible to achieve 100% code coverage with only 15 test cases. No bugs were found. 03/11/

38 Definitions of terms Outline of aims and method Description of the software under test The test environment Results Conclusions 03/11/

39 Conclusion 1 Reliance on traditional structural coverage (even at 100%) is no guarantee of finding bugs. As such it seems that their use as mainstays of the software safety standards relating to quality of testing may be misguided. Coverage does serve as a useful cross-check on understanding of requirements and the development of a functional test set. Beware of implementation details. Beware of possibility of unwanted code! 03/11/

40 Conclusion 2 SST is demonstrably more successful at finding faults than simpler methods. It was the only technique used in this study which successfully enabled two small (but potentially critical) faults to be located. It is also the only technique which offers the basis for making calculations on the dependability of software units. 03/11/

41 Final Observations In our study we looked at various techniques and have noticed that some are more successful than others at exposing faults in the SuT. However, it is generally recognised that no amount of software testing can prove the absence of bugs. Can t realistically test all combinations of all inputs. At best we can achieve a measure of dependability. To move further towards the goal of fault-free software it will be necessary to look to other techniques: Formal methods? 03/11/

42 More Reading Scenario-Based Unit Testing for Reliability Kuball and Gilchrist, RAMS, 2001 Reliability Estimation May, Hughes and Lunn, SEJ, 1995 System failure rate estimator Kuball, May and Hughes, ISSRE, /11/

43 Ian Gilchrist, IPL, Bath, UK Contact us: If you want to know more about Cantata, or the traditional Software Standards Tel: Silke Kuball, University of Bristol, UK If you want to know more about SST and Software Dependability measurement. Tel: /11/