Software Testing Strategies. Brian Nielsen. Center of Embedded Software Systems Aalborg University, Denmark CSS

Transcription

1 Software Testing Strategies Brian Nielsen Center of Embedded Software Systems Aalborg University, Denmark CSS

2 Test strategies Overview of testing types Test levels Unit Integration Top-down, buttom up, sandwich, System Acceptance Embedded Systems test environments Gradual refinement towards real system Test Plans When is what to be tested by whom how thoroughly???

3 Types of Testing system Level integration unit robustness performance white box black box Accessibility usability reliability functional behaviour Aspect

4 Types of Testing system Level integration unit robustness performance white box black box Accessibility usability reliability functional behaviour Aspect

5 Quality-Characteristics (ISO-9126) Functionality Suitability, accuracy, security, compliance, interoperability Reliability maturity, fault tolerance, recoverability Usability understandability, learnability, operability Efficiency time behaviour, resource utilization Maintainability Analysability, changeability, stability, testability Portability Adaptability, intallability, conformance, replaceability

6 Quality-Characteristics (ISO-9126) Functionality functional testing Suitability, accuracy, security, compliance, interoperability Reliability reliability testing maturity, fault tolerance, recoverability Usability usability testing understandability, learnability, operability Efficiency performance testing time behaviour, resource utilization Maintainability maintainability testing?? Analysability, changeability, stability, testability Portability portability testing? Adaptability, intallability, conformance, replaceability

7 requirements V - Model acceptance test spec acceptance test specification system test spec system test architecture spec integration test spec integration test detailed design module test spec module test implementation code unit test spec unit-test

8 Unit-testing Goal: Ensure that module functions correctly according to module spec Module interfaces are tested for proper information flow. Local data are examined to ensure that integrity is maintained. Boundary conditions are tested. Basis path testing should be used. All return values All error handling paths should be tested. Drivers and/or stubs need to be developed to test incomplete software. Mostly white-box techniques.

9 river: a code fragment that nvokes the U-ut Simulate Supplies input data Invokes Unit Collects and compares results ~xunit testing frame work Function call Method invocation Message passing Shared variables test driver Unit under test Unit test E Component Hierarchy F A B C D C-function Abstract Data Type Object Process / thread G test stub test stub Stub: a code fragment that simulates a missing unit Simulate Pre-programmed return values Simplified impl.

10 Integration Testing Goal: Systematic construction of program structure and testing for interfacing errors Loss of data/information across interfaces Side-effects (shared data-structures) Accumulation of errors (imprecision) Incremental Consider required cost and thoroughness Mostly blackbox, supplemented with whitebox tests to ensure coverage

11 Big-bang integration. Test all components in isolation. Integrate all in one step Component Hierarchy A B C D Integration Order Test B Test A Test E E F G - Stubs and drivers needed! - Difficult to localize cause of errors - Difficult to distinguish interface faults Test C Test F Test A,B,C,D E,F,G Test D Test G

12 Buttom-Up integration. Test lower levels first (utility routines). Implement driver (easy) to invoke module. Use tested modules as stubs Component Hierarchy A B C D Integration Order E F G Test E Test F Test B,E,F Test C Test A,B,C,D E,F,G +No stubs needed! -Mundane low-level utilities tested first, but most important top level modules tested late Critical errors found late Main design problems are at higherlevels Timing determined by high-level modules Test G Test D,G

13 Top-Down integration Component Hierarchy. Test higher levels first (controlling modules) A. Implement stubs to deliver responses. Use tested modules as drivers B C D Integration Order E F G Test A Test A,B,C,D Test A,B,C,D E,F,G +No drivers needed! -Many stubs may be needeed -Stubs may be problematic to write Must provide values as expected by tester May require frequent alteration

14 Sandwich integration. Combination of B-up and T-Down. Divide system into upper, middle (target), and lower levels. B-up in low-levels. T-Down in upper levels Integration Order E Component Hierarchy A B C D F G Test E Test F Test B,E,F + combines B-up and T-Dn + Integration possible early -Individual components not tested thoroughly before integration, especially middle level Test G Test D,G Test A,B,C,D E,F,G Test

15 Testability = how easy can a program be tested operability - it operates cleanly observability the results are easy to see distinct output is generated for each input incorrect output is easily identified controllability processing can be controlled tests can be automated & reproduced decomposability - software modules can be tested independently simplicity - no complex architecture and logic stability - few changes are requested during testing understandability - program is easy to understand

16 Verification & Validation

17 Verification & Validation Verification :

18 Verification & Validation Verification : Software correctly implements a specific function

19 Verification & Validation Verification : Software correctly implements a specific function building the product right (Boehm81) concerns development process intermediate products e.g. checking the testability of the specification

20 Verification & Validation Verification : Software correctly implements a specific function building the product right (Boehm81) concerns development process intermediate products e.g. checking the testability of the specification Validation :

21 Verification & Validation Verification : Software correctly implements a specific function building the product right (Boehm81) concerns development process intermediate products e.g. checking the testability of the specification Validation : Tracable to user requirements

22 Verification & Validation Verification : Software correctly implements a specific function building the product right (Boehm81) concerns development process intermediate products e.g. checking the testability of the specification Validation : Tracable to user requirements building the right product (Boehm81) whether final product meets requirements and needs

23 System Test (Pfleger) Unit & integration: code implements design correctly System testing: system does what customers expect Constructed software tested against established requirements during requirements analysis Validation criteria satisfied? Are all user-visible functional, behavioral, performance requirements satisfied? tegrated odules System functional requirements Function test Requirements For other Quality aspects Non-functional test Customer Requirements Specification Acceptance test User Environment Installation/ Production test System In use functioning system Performance Stress / load Reliability, recoverability Security verified system accepted, validated system

24 Acceptance Test (Pfleger) By customers, User s understanding of requirements Customer writes, conducts, and evaluates acceptance tests Functional Acceptance Test (FAT) Production Acceptance Test (PAT) Techniques Benchmark tests Pilot tests ~ experimental, limited edition Alpha test: in-house, controlled environment, developer supervised Beta test: live test at customer sites, unsupervised Parallel test: run concurrently with old system

25 System test 2*CRTG (4 channels) 2 * 200 k

26 Test Equipment Complete Type Approval Test System (3 M )

27 Generic Embedded System Embedded system User interface Specific interface Power supply System environment Application Software Digital I/O A/D+ D/A OS Plant I/O-drivers Processing Unit Actuators Sensors Interfaces to other systems

28 Multiple V-model Model Prototype(s) Development Iteration(s) Final Product design design design test test test build build build Different objects, objectives, techniques in different stages Complexity of ES: functional decomposition and stepwise integration, but tight coordination (e.g independent development of HW and SW) Cost: Detect and correct errors earlier => Test earlier

29 Embedded System Test Environments Gradual replacement of simulated components by real-ones Test-Level Feed-backsimulation One-waysimulation Rapid-prototype Embedded-SW Simulated Experimental Simulated Platform (OS/CPU) - - Experimental Rest of Embedded System - - Experimental Plant - Simulated SW/U SW/I (1) Experimental Host Simulated Simulated SW/U SW/I (1) Target Emulator Simulated Simulated SW/HW/I Target Target Experimental Simulated System Integration Target Target Prototype Simulated Environmental Target Target Real Simulated Pre-Production Target Target Real Real

30 Techniques Simulation of (FSM-) model model test, single-stepping / animation simulation: one-way feed-back Rapid prototyping Objectives validate requirements proof-of-concept support design design optimization Simulation Stage Signal Generation One-way simulation Simulation model Embedded System Feed-back simulation Plant Simulation model Embedded System Rapid Prototype Simulation model Signal monitoring Plant Embedded System

31 Prototyping / Development 1. SW/U, SW/I (1): Software unit and software integration on host Objectives: verify functionality/behavior Techniques: host-compilation/execution 2. SW/U, SW/I (2): Software unit and software integration on emulator/target Objectives: verify correct execution on target processor Techniques: processor-emulator or cross-compile/download 3. HW+SW/I: Hardware Software/Integration Objectives: Verify execution on target processor and cooperation with surrounding HW Techniques: Experimental HW (evaluation-board), HW-in-the-loop 4. System Integration Objectives: Verify correctness of complete embedded system Techniques: Prototype HW (PCB), HW-in-the-loop 5. Environmental Test Objectives: Check conformance to environmental requirements (eg. EMI/EMC) Techniques: EMI/EMC measurements, vibration, shock, humidityemulation

32 Pre-Production acceptance Objectives: final proof of correctness demonstrate conformity to regulations etc create demo / test-samples for customers prepare production and maintenance facilities Requires test-bench hardware Techniques real-life test (real test data) field test customer tests

33 Post-Development Objectives: check quality of produced articles test production facilities product instance test first-article-inspection product-maintenance tests Techniques:

34 Test-environments Realism Actual environment behavior Target system SW completeness Test ease controllability observability speed Simulation stage Prototype stage Pre-product stage Postdevelopment stage Time Gradual refinement towards deployed system Different aspects tested

35 Master Test Plan Functionality Connectivity Reliability Recoverability Performance Suitability Relative Importance (%) Unit Test SW integration test HW/SW integration System test Acceptance + ++ Field test aspect predominant + aspect is relevant (empty) aspect insignificant What to be tested, when, and how thoroughly

36 Risk Based Test Strategy Make best possible use of resources by identifying and prioritizing quality aspects and subsystems Higher risk more testing No risk no testing Risk = chance of failure damage Use frequency Chance of error being present Complexity New tools/techniques Inexperienced developers Cost of repair Loss of market share Legal claim

37 Relative importance subsystems Risk Matrix (per test level) Relative importance Subsystem A Subsystem B Subsystem C Subsystem D Total system (%) Functionality Performance Reliability Suitability Relative importance of quality characteristic ++ aspect predominant + aspect is relevant (empty) aspect insignificant Select appropriate testing techniques and effort per sub system

38 Who Should Test?

39 Who Should Test? Developer Understands the system But, will test gently And, is driven by deadlines

40 Who Should Test? Developer Understands the system But, will test gently And, is driven by deadlines

41 Who Should Test? Developer Understands the system But, will test gently And, is driven by deadlines Independent tester Must learn system But, will attempt to break it And, is driven by quality

42 Who Should Test? Developer Understands the system But, will test gently And, is driven by deadlines Independent tester Must learn system But, will attempt to break it And, is driven by quality

43 Lessons Testing starts when project starts Have a development approach How to develop (waterfall, cyclic, etc) : roles and activities Clear expectations between testers and developers Clarity about phases and tasks Easier planning Test execution takes 3 steps Test, fix, re-test Let the programmers find bugs 80 % of bugs can be found by programmers (but they don t) Every morning discuss the issues found Testers create test report at end of the day Next morning discuss which issues have to be solved Programmer and tester fix issue together

44 Life Cycle : Testing as a Process

45 Life Cycle : Testing as a Process Testing as phase of development process vs.

46 Life Cycle : Testing as a Process Testing as phase of development process vs. Testing as a process itself with its own phases in parallel with development process

47 Life Cycle : Testing as a Process Testing as phase of development process vs. Testing as a process itself with its own phases in parallel with development process planning & control preparation test generation test execution completion

48 Life Cycle : Testing as a Process Testing as phase of development process vs. Testing as a process itself with its own phases in parallel with development process requirements specification implementation planning & control preparation test generation test execution completion quality repo

49 Life Cycle : Testing as a Process T L I O L : Life-cycle for testing O : Organization Testing as phase of development process vs. Testing as a process itself with its own phases in parallel with development process I : Infrastructure and tools T : Techniques requirements specification implementation planning & control preparation test generation test execution completion quality repo

50 END