Test-Driven Development Educator s Session November 13, 2012
Outline Software Quality Overview of Testing Automated Testing Tools Test-Driven Development Educator's Session 2
SOFTWARE QUALITY Educator's Session 3
Multiple Definitions Software Quality Developer s View vs. User s View Developers = Users = Educator's Session 4
Definitions Software must do the right things Perform the right functions Often referred to as Software must do things right Perform intended functions without problems Often referred to as Educator's Session 5
Failure Quality Definitions: Defects Fault Error Educator's Session 6
Quality Definitions: Defects Educator's Session 7
Customers/Users Quality Focus Developers Educator's Session 8
Defect Prevention QA Activities: Types Defect Reduction Defect Containment Educator's Session 9
Pre-release QA Activities: Timing Defect Prevention & Reduction Remove as many defects as possible before release Post-release Repair failures Reduces defects Significantly more expensive than pre-release Bad PR Contain defects Minimize impact Expensive, so limited in application Educator's Session 10
Defect Prevention: Introduction Reduce the chance of fault injection Approach depends on source There may be specific tools or technologies that can also help Important to establish the correct root-cause Educator's Session 11
Defect Prevention: Education and Training People are most important factor in quality and success Education and Training can improve the quality of the work done by practitioners Elimination of misconceptions will reduce the probability of defect injection Educator's Session 12
Defect Prevention: Education and Training Product and Domain Specific Knowledge Software Development Expertise Knowledge about tools, methods, techniques Development Process Knowledge Educator's Session 13
Defect Prevention: Formal Methods Help eliminate error sources and verify the absence of faults Formal Methods Formal Specification Formal Verification Educator's Session 14
Formal Methods: Axiomatic Approach Code Conditions Describing Program State Before Execution Axiom 1 Axiom 2 Axiom 3 Program Meaning Conditions Describing Program State After Execution Biggest obstacle to use of formal methods: Educator's Session 15
Defect Prevention: Other Techniques Use of additional software development methodologies (besides Formal Methods) Prevent extra functionality Reduce complexity Better management Concrete process definition Enforcement of standards Use of specific tools Enforce coding standards Educator's Session 16
QA Activities: Defect Prevention Educator's Session 17
Defect Reduction: Introduction Unrealistic to expect Defect Prevention step to stop all defects Different approaches Educator's Session 18
Defect Reduction: Testing Execution of software and checking results Locates failures Isolate and fix the fault(s) that led to the failure When to test Need some executable Unit tests of components through acceptance test of entire system Can also use prototypes Educator's Session 19
Defect Reduction: What to Test Functional (black box) Structural (white box) Educator's Session 20
Defect Reduction: When to Stop Testing Can use coverage criteria Assumption: Reliability goals Educator's Session 21
Defect Reduction: Observations Many other techniques available In-field measurement and repair not normally considered part of QA Important to determine risky components Educator's Session 22
QA Activities: Defect Reduction Educator's Session 23
Defect Containment: Introduction Important for systems where the impact of failures is substantial Not all faults can be eliminated (cost, time) Educator's Session 24
Defect Containment: Fault Tolerance Different from manufacturing Approaches Does not focus on identifying and removing the faults that cause the failures Educator's Session 25
Safety-critical systems: Defect Containment: Safety Assurance Address even low probability failures Safety Assurance techniques Educator's Session 26
QA Activities: Defect Containment Educator's Session 27
INTRODUCTION TO TESTING Educator's Session 28
Types of Testing Educator's Session 29
Unit Testing What? Who? What is the focus? What type of testing techniques do we use? Educator's Session 30
Component Testing What? Who? What type of testing techniques do we use? COTS and CBSD Educator's Session 31
Integration Testing What? Who? What is the focus? What type of testing techniques do we use? Merged with System Testing? Educator's Session 32
System Testing What? Who? What is the focus? What type of testing techniques do we use? Embedded systems? Educator's Session 33
Acceptance Testing What? Who? What is the focus? What type of testing techniques do we use? Educator's Session 34
Functional vs. Structural Educator's Session 35
Individual elements Functional vs. Structural Interactions of elements Inputs and outputs functional Educator's Session 36
Black Box (Functional) Testing: Overview Educator's Session 37
Planning Black Box (Functional) Testing: Process Execution Analysis Educator's Session 38
White Box (Structural) Testing: Overview Educator's Session 39
Stopping Criteria: Coverage-Based Ensures some item has been covered Assumes that Approaches Educator's Session 40
Define the model Coverage Based Testing: Process Check the model elements Define the coverage criteria Derive the test cases Educator's Session 41
Test Activities Test Planning Test Execution Analysis and Follow-up Educator's Session 42
High level goal: Test Planning Make the following decisions Have to account for personnel Educator's Session 43
What is needed? Test Planning: Test Case Creation How are they generated? Educator's Session 44
What is a test suite? Test Planning: Test Suite Preparation How are they created? Expensive should be maintained for future use Educator's Session 45
Test Planning: Preparation of Procedure Ordering of test cases One test case should leave the system ready to execute the next Assignment of personnel Educator's Session 46
Major steps: Test Execution: Overview Prevent failed test cases from halting execution Environment Educator's Session 47
Specific Approaches to Testing Control Flow Testing Partition-based Testing Usage-based Testing Data-flow Testing Educator's Session 48
Systematic Testing Drawbacks to ad hoc testing One way to structure is to build a checklist Educator's Session 49
Control Flow Testing: Overview Model of the software is a graph first A B C D E Use F G last Educator's Session 50
Control Flow Testing: Model Construction Educator's Session 51
Control Flow Testing: Model Construction Can also be done with black box testing Elements Educator's Session 52
Control Flow Testing: Path Selection Educator's Session 53
Control Flow Testing: Path Selection Educator's Session 54
Control Flow Testing: Model Construction L1: input(a,b,c) L2: d b*b 4*a*c L3: If (d>0) then L4: r 2 L5: else_if (d=0) L6: r 1 L7: else_if (d<0) L8: r 0 L9: output (r) Educator's Session 55
Control Flow Testing: Creating Test Cases If each decision is based on an independent variable, then just choose appropriate values Can use the idea of equivalence classes If decisions are not independent, some branches may be eliminated as infeasible Some decisions may be based on processing between decisions nodes may be hard to develop test cases Educator's Session 56
Loops Loops complicate the CFT idea. Why? Parts of a loop: Educator's Session 57
Loops While (C) do {B} For (I; C; U) do {B} Educator's Session 58
Loop Testing: Difficulties When loops are nested, number of paths quickly grows unmanageable Complete path coverage not possible, have to be selective Where do most problems occur? What types of test cases do we need and why? Educator's Session 59
Loop Testing: Difficulties Concatenation/Nesting of loops How can we reduce number of test cases? Educator's Session 60
Specific Approaches to Testing Control Flow Testing Partition-based Testing Usage-based Testing Data-flow Testing Educator's Session 61
Partition Based Testing Benefits Increased coverage Reduced overlap Examples: Solve for root of ax 2 + bx + c = 0 Thermostat Educator's Session 62
Partition Testing: Theory A set S contains a list of unique elements Partition of S creates subsets G 1, G 2, G n such that Sets are mutually exclusive Sets are collectively exhaustive G 1..G n are equivalence classes if created based on some definition of equality Properties Symmetric Transitive Reflexive Educator's Session 63
AUTOMATED TESTING TOOLS Educator's Session 64
Automated Testing Frameworks Enable set of tests to be executed repeatedly Family of tools junit cunit (xunit) Demo Educator's Session 65
cunit http://cunit.sourceforge.net Framework to create and execute tests Assertions CU_ASSERT CU_ASSERT_TRUE CU_ASSERT_FALSE CU_ASSERT_EQUAL CU_ASSERT_NOT_EQUAL Educator's Session 66
cunit: Test Registry Repository of test suites and tests Using the test registry Create Clean up Adding tests Create a test suite Add tests to the test suite Educator's Session 67
Can run: All tests Individual suites Individual tests cunit: Running Tests Modes Automated non-automated / XML output Basic non-automated / stdout output Console interactive console under user control Educator's Session 68
TEST-DRIVEN DEVELOPMENT Educator's Session 69
Test-Driven Development: Introduction & Background Basic idea: Part of the agile software development approach Pair programming Educator's Session 70
Test-Driven Development: Overview Focus on unit tests Often require Can be automated or manual Can be performed by developers or testers Educator's Session 71
Test-Driven Development: Motivation Programming practice that instructs developers to: Test-Driven Development Educator's Session 72
Test-Driven Development: Definition From the Agile Alliance Test-driven development (TDD) is the craft of producing automated tests for production code, and using that process to drive design and programming. For every tiny bit of functionality in the production code, you first develop a test that specifies and validates what the code will do. You then produce exactly as much code as will enable that test to pass. Then you refactor (simplify and clarify) both the production code and the test code. Educator's Session 73
Test-Driven Development: Additional Thoughts Refactoring Not a software development methodology Provides automated test Educator's Session 74
Test-Driven Development: Process Educator's Session 75
Test-Driven Development: Example Story Educator's Session 76
Test-Driven Development: Automated Testing TDD assumes the presence of an automated testing framework Test Harnesses xunit Lets users write tests to initialize, execute, and make assertions about code being tested Tests can serve as documentation Educator's Session 77
Test-Driven Development: Evaluation Performed in Industry and Academia Industrial studies 4 studies in small companies Measured defect density Results Programmers using TDD produced code that passed 18% - 50% more tests TDD programmers spent less time debugging TDD decreased productivity but they wrote more test cases Educator's Session 78
Test-Driven Development: Challenges to Adoption Educator's Session 79
Test-Driven Development: Example Design a system to perform financial transactions with money that may be in different currencies For example If the exchange rate from Swiss Francs to US Dollars is 2 to 1, then we can calculate 5 USD + 10 CHF = 10 USD 5 USD + 10 CHF = 20 CHF Educator's Session 80
References Slides adapted from materials in: Roger Pressman. Software Engineering: A Practitioner s Approach. 7th edition. Jeff Tian. Software Quality Engineering: Testing, Quality Assurance and Quantifiable Improvement. Jansen, D. and Saiedian, H. Test-Driven Development: Concepts, Taxonomy, and Future Direction. Computer. Sept. 2005. p. 43-50 Erdogmus, H., Morisio, M., and Torchiano, M. On the Effectiveness of the Test-First Approach to Programming. IEEE Transactions on Software Engineering. 31(5): 226-237. March 2006. Example taken from Kenneth Anderson, Univ. of Colorado, Boulder Educator's Session 81
Materials Updated slides and handouts available on the web http://carver.cs.ua.edu/_tutorial/ Educator's Session 82
HANDS-ON TIME Educator's Session 83