Introduction to Software Testing Chapter 8.1 Building Testing Tools Instrumentation. Chapter 8 Outline

Size: px

Start display at page:

Download "Introduction to Software Testing Chapter 8.1 Building Testing Tools Instrumentation. Chapter 8 Outline"

Alexandrina Powers
10 years ago
Views:

1 Introduction to Software Testing Chapter 8. Building Testing Tools Instrumentation Paul Ammann & Jeff Offutt Chapter 8 Outline. Instrumentation for Graph and Logical Expression Criteria. Building Mutation Testing Tools Introduction to Software Testing (Ch 8.), Ammann & Offutt

introsoftwaretesting.com Chapter 8 Outline.

2 Categories of Test Tools Research Private non-commercial Commercial Test management Simple coverage Metrics Code independent Moderate techniques Acceptors Mostly complete Some code-based Often special purpose Advanced techniques Generators Very powerful Often code-based Usually incomplete General purpose Introduction to Software Testing (Ch 8.), Ammann & Offutt Types of Test Tools. Coverage analyzers (Acceptors) What coverage? (most common is node) What level? (sometimes object code). Flow graph generators. Metrics. Instrumentation support. Test execution automation Runs scripts and reports results JUnit, HttpUnit,. Capture / replay Useful for GUIs 7. Stubs and Drivers 8. Test data generators Graph path based Data flow Logic based Functional Mutation Random Only random test data generators exist Introduction to Software Testing (Ch 8.), Ammann & Offutt

com Ammann & Offutt Types of Test Tools. Coverage analyzers (Acceptors) What coverage? (most common is node) What level? (sometimes object code). Flow graph generators. Metrics.

3 Tools Instrumentation The key technology behind many test tools is instrumentation Tools Instrumentation Coverage analysis is measured with instrumentation Instrument : One or more statements inserted into the program to monitor some aspect of the program Must not affect the behavior May affect timing Source level or object code level public int min (int A, B) int m = A; if (A > B) m = B; return (m); Mark: if body is reached Introduction to Software Testing (Ch 8.), Ammann & Offutt

program Must not affect the behavior May affect timing Source level or object code level public int min (int A, B) int m = A; if

4 Instrumenting for Statement Coverage. Each node is given a unique id # Node # or statement #. Create an array indexed by id #s nodecover [ ]. Insert an instrument at each node nodecover [ i ] ++;. Save nodecover [ ] after each execution Must accumulate results across multiple test cases Introduction to Software Testing (Ch 8.), Ammann & Offutt 7 Statement Coverage Example int nodecover[] = 0,0,0,0,0,0,0,0,0,0 Read (nodecover []) nodecover [] ++ nc []++ nc []++ nc []++ nc []++ nc [7]++ After running a sequence of tests, any node for which nodecover[node]==0 has not been covered. nc [] nc [9]++ nc [8]++ 0 nc [0]++ Introduction to Software Testing (Ch 8.), Ammann & Offutt 8

5 Edge Coverage Instrumentation ec []++ int edgecover[] = 0,0,0,0,0,0,0,0,0,0,0,0,0 Read (edgecover []) ec []++ For each edge e, put edgecover[e]++ on the edge. ec [] If edgecover[e] == 0, e has not been covered. 9 0 Note that the arrays could be boolean Introduction to Software Testing (Ch 8.), Ammann & Offutt 9 CACC Coverage Instrumentation CACC_Mark_ (A, B) if (A && B) if (C && (D E)) CACC_Mark_ (C, D, E) A && B F t T t t F t T CACCCover [ ] CACC_Mark_ (A, B) if (!A) // Don t allow short circuit if (B) CACCCover[]++; else if (A) if (B) CACCCover []++; else CACCCover[]++ Introduction to Software Testing (Ch 8.), Ammann & Offutt 0

com Ammann & Offutt 9 CACC Coverage Instrumentation CACC_Mark_ (A, B) if (A && B) if (C && (D E)) CACC_Mark_ (C, D, E) A && B F t T t t F t T CACCCover [ ] CACC_Mark_ (A, B)

6 CACC Coverage Instrumentation () C && (D E) CC [ ] F t t T t f f t t F f t T f 7 t f F t f T 8 CACC_Mark_ (C, D, E) if (! C) if (D E) CACCCover []++; else if (! D) if (! E) CACCCover []++; else CACCCover []++; CACCCover [8]++; else // C and D if (! E) CACCCover []++; CACCCover [7]++; else CACCCover []++; Introduction to Software Testing (Ch 8.), Ammann & Offutt All-Uses Coverage Instrumentation defcover [y] = def (y) def (x, y) defcover [x] = defcover [y] = For each track of location. variable, keep its current def use (x) use (x) usecover [, x, defcover[x]] ++ usecover [, x, defcover[x]] ++ use (y) At each use, increment a counter for the def that usecover [, y, defcover[y]] ++ reached it. Introduction to Software Testing (Ch 8.), Ammann & Offutt

introsoftwaretesting.com Ammann & Offutt All-Uses Coverage Instrumentation defcover [y] = def (y) def (x, y) defcover [x] = defcover [y] = For each track of location.

7 Instrumentation Summary Instrumentation can be added in multiple copies of the program Source code Java byte code (or other intermediate code) Executable Instrumentation must not change or delete functionality Only add new functionality Instrumentation may affect timing behavior Requires the program to be parsed Once parsed, inserting instruments is straightforward Most challenging part is the user interface Present reports as dumps of instrument arrays? Present tables? Program source with colored annotations? Graph of program with colored annotations? Introduction to Software Testing (Ch 8.), Ammann & Offutt 7

Once parsed, inserting instruments is straightforward Most challenging part is the user interface Present reports as dumps of instrument arrays? Present tables?

Software Testing. Definition: Testing is a process of executing a program with data, with the sole intention of finding errors in the program.

Software Testing. Definition: Testing is a process of executing a program with data, with the sole intention of finding errors in the program. Software Testing Definition: Testing is a process of executing a program with data, with the sole intention of finding errors in the program. Testing can only reveal the presence of errors and not the