Test-Driven Development. SC12 Educator s Session November 13, 2012

Transcription

1 Test-Driven Development Educator s Session November 13, 2012

2 Outline Software Quality Overview of Testing Automated Testing Tools Test-Driven Development Educator's Session 2

3 SOFTWARE QUALITY Educator's Session 3

4 Multiple Definitions Software Quality Developer s View vs. User s View Developers = Users = Educator's Session 4

5 Definitions Software must do the right things Perform the right functions Often referred to as Software must do things right Perform intended functions without problems Often referred to as Educator's Session 5

6 Failure Quality Definitions: Defects Fault Error Educator's Session 6

7 Quality Definitions: Defects Educator's Session 7

8 Customers/Users Quality Focus Developers Educator's Session 8

9 Defect Prevention QA Activities: Types Defect Reduction Defect Containment Educator's Session 9

10 Pre-release QA Activities: Timing Defect Prevention & Reduction Remove as many defects as possible before release Post-release Repair failures Reduces defects Significantly more expensive than pre-release Bad PR Contain defects Minimize impact Expensive, so limited in application Educator's Session 10

11 Defect Prevention: Introduction Reduce the chance of fault injection Approach depends on source There may be specific tools or technologies that can also help Important to establish the correct root-cause Educator's Session 11

12 Defect Prevention: Education and Training People are most important factor in quality and success Education and Training can improve the quality of the work done by practitioners Elimination of misconceptions will reduce the probability of defect injection Educator's Session 12

13 Defect Prevention: Education and Training Product and Domain Specific Knowledge Software Development Expertise Knowledge about tools, methods, techniques Development Process Knowledge Educator's Session 13

14 Defect Prevention: Formal Methods Help eliminate error sources and verify the absence of faults Formal Methods Formal Specification Formal Verification Educator's Session 14

15 Formal Methods: Axiomatic Approach Code Conditions Describing Program State Before Execution Axiom 1 Axiom 2 Axiom 3 Program Meaning Conditions Describing Program State After Execution Biggest obstacle to use of formal methods: Educator's Session 15

16 Defect Prevention: Other Techniques Use of additional software development methodologies (besides Formal Methods) Prevent extra functionality Reduce complexity Better management Concrete process definition Enforcement of standards Use of specific tools Enforce coding standards Educator's Session 16

17 QA Activities: Defect Prevention Educator's Session 17

18 Defect Reduction: Introduction Unrealistic to expect Defect Prevention step to stop all defects Different approaches Educator's Session 18

19 Defect Reduction: Testing Execution of software and checking results Locates failures Isolate and fix the fault(s) that led to the failure When to test Need some executable Unit tests of components through acceptance test of entire system Can also use prototypes Educator's Session 19

20 Defect Reduction: What to Test Functional (black box) Structural (white box) Educator's Session 20

21 Defect Reduction: When to Stop Testing Can use coverage criteria Assumption: Reliability goals Educator's Session 21

22 Defect Reduction: Observations Many other techniques available In-field measurement and repair not normally considered part of QA Important to determine risky components Educator's Session 22

23 QA Activities: Defect Reduction Educator's Session 23

24 Defect Containment: Introduction Important for systems where the impact of failures is substantial Not all faults can be eliminated (cost, time) Educator's Session 24

25 Defect Containment: Fault Tolerance Different from manufacturing Approaches Does not focus on identifying and removing the faults that cause the failures Educator's Session 25

26 Safety-critical systems: Defect Containment: Safety Assurance Address even low probability failures Safety Assurance techniques Educator's Session 26

27 QA Activities: Defect Containment Educator's Session 27

28 INTRODUCTION TO TESTING Educator's Session 28

29 Types of Testing Educator's Session 29

30 Unit Testing What? Who? What is the focus? What type of testing techniques do we use? Educator's Session 30

31 Component Testing What? Who? What type of testing techniques do we use? COTS and CBSD Educator's Session 31

32 Integration Testing What? Who? What is the focus? What type of testing techniques do we use? Merged with System Testing? Educator's Session 32

33 System Testing What? Who? What is the focus? What type of testing techniques do we use? Embedded systems? Educator's Session 33

34 Acceptance Testing What? Who? What is the focus? What type of testing techniques do we use? Educator's Session 34

35 Functional vs. Structural Educator's Session 35

36 Individual elements Functional vs. Structural Interactions of elements Inputs and outputs functional Educator's Session 36

37 Black Box (Functional) Testing: Overview Educator's Session 37

38 Planning Black Box (Functional) Testing: Process Execution Analysis Educator's Session 38

39 White Box (Structural) Testing: Overview Educator's Session 39

40 Stopping Criteria: Coverage-Based Ensures some item has been covered Assumes that Approaches Educator's Session 40

41 Define the model Coverage Based Testing: Process Check the model elements Define the coverage criteria Derive the test cases Educator's Session 41

42 Test Activities Test Planning Test Execution Analysis and Follow-up Educator's Session 42

43 High level goal: Test Planning Make the following decisions Have to account for personnel Educator's Session 43

44 What is needed? Test Planning: Test Case Creation How are they generated? Educator's Session 44

45 What is a test suite? Test Planning: Test Suite Preparation How are they created? Expensive should be maintained for future use Educator's Session 45

46 Test Planning: Preparation of Procedure Ordering of test cases One test case should leave the system ready to execute the next Assignment of personnel Educator's Session 46

47 Major steps: Test Execution: Overview Prevent failed test cases from halting execution Environment Educator's Session 47

48 Specific Approaches to Testing Control Flow Testing Partition-based Testing Usage-based Testing Data-flow Testing Educator's Session 48

49 Systematic Testing Drawbacks to ad hoc testing One way to structure is to build a checklist Educator's Session 49

50 Control Flow Testing: Overview Model of the software is a graph first A B C D E Use F G last Educator's Session 50

51 Control Flow Testing: Model Construction Educator's Session 51

52 Control Flow Testing: Model Construction Can also be done with black box testing Elements Educator's Session 52

53 Control Flow Testing: Path Selection Educator's Session 53

54 Control Flow Testing: Path Selection Educator's Session 54

55 Control Flow Testing: Model Construction L1: input(a,b,c) L2: d b*b 4*a*c L3: If (d>0) then L4: r 2 L5: else_if (d=0) L6: r 1 L7: else_if (d<0) L8: r 0 L9: output (r) Educator's Session 55

56 Control Flow Testing: Creating Test Cases If each decision is based on an independent variable, then just choose appropriate values Can use the idea of equivalence classes If decisions are not independent, some branches may be eliminated as infeasible Some decisions may be based on processing between decisions nodes may be hard to develop test cases Educator's Session 56

57 Loops Loops complicate the CFT idea. Why? Parts of a loop: Educator's Session 57

58 Loops While (C) do {B} For (I; C; U) do {B} Educator's Session 58

59 Loop Testing: Difficulties When loops are nested, number of paths quickly grows unmanageable Complete path coverage not possible, have to be selective Where do most problems occur? What types of test cases do we need and why? Educator's Session 59

60 Loop Testing: Difficulties Concatenation/Nesting of loops How can we reduce number of test cases? Educator's Session 60

61 Specific Approaches to Testing Control Flow Testing Partition-based Testing Usage-based Testing Data-flow Testing Educator's Session 61

62 Partition Based Testing Benefits Increased coverage Reduced overlap Examples: Solve for root of ax 2 + bx + c = 0 Thermostat Educator's Session 62

63 Partition Testing: Theory A set S contains a list of unique elements Partition of S creates subsets G 1, G 2, G n such that Sets are mutually exclusive Sets are collectively exhaustive G 1..G n are equivalence classes if created based on some definition of equality Properties Symmetric Transitive Reflexive Educator's Session 63

64 AUTOMATED TESTING TOOLS Educator's Session 64

65 Automated Testing Frameworks Enable set of tests to be executed repeatedly Family of tools junit cunit (xunit) Demo Educator's Session 65

66 cunit Framework to create and execute tests Assertions CU_ASSERT CU_ASSERT_TRUE CU_ASSERT_FALSE CU_ASSERT_EQUAL CU_ASSERT_NOT_EQUAL Educator's Session 66

67 cunit: Test Registry Repository of test suites and tests Using the test registry Create Clean up Adding tests Create a test suite Add tests to the test suite Educator's Session 67

68 Can run: All tests Individual suites Individual tests cunit: Running Tests Modes Automated non-automated / XML output Basic non-automated / stdout output Console interactive console under user control Educator's Session 68

69 TEST-DRIVEN DEVELOPMENT Educator's Session 69

70 Test-Driven Development: Introduction & Background Basic idea: Part of the agile software development approach Pair programming Educator's Session 70

71 Test-Driven Development: Overview Focus on unit tests Often require Can be automated or manual Can be performed by developers or testers Educator's Session 71

72 Test-Driven Development: Motivation Programming practice that instructs developers to: Test-Driven Development Educator's Session 72

73 Test-Driven Development: Definition From the Agile Alliance Test-driven development (TDD) is the craft of producing automated tests for production code, and using that process to drive design and programming. For every tiny bit of functionality in the production code, you first develop a test that specifies and validates what the code will do. You then produce exactly as much code as will enable that test to pass. Then you refactor (simplify and clarify) both the production code and the test code. Educator's Session 73

74 Test-Driven Development: Additional Thoughts Refactoring Not a software development methodology Provides automated test Educator's Session 74

75 Test-Driven Development: Process Educator's Session 75

76 Test-Driven Development: Example Story Educator's Session 76

77 Test-Driven Development: Automated Testing TDD assumes the presence of an automated testing framework Test Harnesses xunit Lets users write tests to initialize, execute, and make assertions about code being tested Tests can serve as documentation Educator's Session 77

78 Test-Driven Development: Evaluation Performed in Industry and Academia Industrial studies 4 studies in small companies Measured defect density Results Programmers using TDD produced code that passed 18% - 50% more tests TDD programmers spent less time debugging TDD decreased productivity but they wrote more test cases Educator's Session 78

79 Test-Driven Development: Challenges to Adoption Educator's Session 79

80 Test-Driven Development: Example Design a system to perform financial transactions with money that may be in different currencies For example If the exchange rate from Swiss Francs to US Dollars is 2 to 1, then we can calculate 5 USD + 10 CHF = 10 USD 5 USD + 10 CHF = 20 CHF Educator's Session 80

81 References Slides adapted from materials in: Roger Pressman. Software Engineering: A Practitioner s Approach. 7th edition. Jeff Tian. Software Quality Engineering: Testing, Quality Assurance and Quantifiable Improvement. Jansen, D. and Saiedian, H. Test-Driven Development: Concepts, Taxonomy, and Future Direction. Computer. Sept p Erdogmus, H., Morisio, M., and Torchiano, M. On the Effectiveness of the Test-First Approach to Programming. IEEE Transactions on Software Engineering. 31(5): March Example taken from Kenneth Anderson, Univ. of Colorado, Boulder Educator's Session 81

82 Materials Updated slides and handouts available on the web Educator's Session 82

83 HANDS-ON TIME Educator's Session 83