Agile Testing Workshop

Transcription

1 Agile Testing Workshop Paul C. Jorgensen, Ph.D., Professor School of Computing and Information Systems Grand Valley State University Allendale, Michigan 49491

2 Overview/Übersicht The Agile Development Paradigm Flavors of Agile Testing Context-Driven Testing Good Enough Testing Exploratory Testing Test-Driven Development (TDD) Comparison of MDD and TDD Open Questions and Conclusions

3 The Agile Development Paradigm the Agile Manifesto more than just Something for lazy Americans... Customer driven (via User Stories) Bottom-up rather than traditional Top-down Agile or Fragile?

4 The Agile Manifesto We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. (signed by 17 people;

5 Principles behind the Agile Manifesto Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. Business people and developers must work together daily throughout the project.

6 Principles behind the Agile Manifesto (continued) Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. Working software is the primary measure of progress. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

7 Principles behind the Agile Manifesto (concluded) Continuous attention to technical excellence and good design enhances agility. Simplicity--the art of maximizing the amount of work not done--is essential. The best architectures, requirements, and designs emerge from self-organizing teams. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

8 BUT what Happened to Testing? Mike Beedle (one of the group that signed the Manifesto) suggested in 2004 that another value should be added: we have come to value... Tests over requirements definition and traceability (Brian Marick s Blog,

9 Development with User Stories User Stories Increment Plan Test Cases Increment Coding System Test Small Release Unit Test

10 Nun singen wir, alle zusammen (Melodie von Josef Haydn, 1797) Agile, agile, über alles, Über alles in der Welt. Agile, agile, über alles, Über alles in der Welt. Kein mehr Wasserfall, kein mehr Prozess, Kein mehr Requirements und Design. Agile, agile, über alles, Über alles in der Welt.

11 Testing in an Agile Development Context-Driven Testing Good Enough Testing Exploratory Testing Test-Driven Development

12 Context-Driven Testing (seven basic principles) 1. The value of any practice depends on its context. 2. There are good practices in context, but there are no best practices. 3. People, working together, are the most important part of any project's context. 4. Projects unfold over time in ways that are often not predictable. 5. The product is a solution. If the problem isn't solved, the product doesn't work. 6. Good software testing is a challenging intellectual process. 7. Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

13 Observations and Conclusions Principles 1 and 2 apply to almost any endeavor. Principle 3 applies to all forms of teamwork. Agile development is a good response to Principle 4 Principles 5, 6, and 7 apply to software testing. These principles are so general, they give little direction about HOW to do context-driven testing. Context-driven testing depends primarily on: Skill knowledge Ingenuity Motivation Craftsmanship!

14 Good Enough Testing I want to distinguish the Good Enough approach from the approach that says that we should keep testing until Management pries the product from my cold, dead fingers. - -James Bach Good Enough testing is the process of developing a sufficient assessment of quality, at a reasonable cost, to enable wise and timely decisions to be made concerning the product.

15 Bach s Principles of Good Enough Testing Assess product quality How accurate/complete is the testing? Evaluate the cost of testing Within budget? Any redundancy? Unnecessary testing? Determine whether testing supports necessary decision making Risk? Are testing results timely? Point of diminishing returns.

16 Observations and Conclusions Good Enough Testing can be applied to any software development. It is not necessarily Agile. Very sensible, but it requires effort from both testers and management. No guidelines just generalities.

17 Exploratory Testing Explore: to investigate the unknown The opposite of scripted testing James Bach More than ad hoc testing Shares some pros and cons with Special Value Testing Depends on the abilities of the exploratory tester Curiosity Ingenuity Time and tools A discovery process A learning process in which the results of one test suggest additional tests We will explore Exploratory Testing with an example.

18 James Bach s Exploration Mindset I run a test. I MAKE AN OBSERVATION... - I experience surprise associated with a pattern within an observation. That triggers REFLECTION about PLAUSIBILITY... - The pattern seems implausible relative to my current model of the phenomenon. That triggers REFLECTION about RISK... - I can bring to mind a risk associated with that pattern. That triggers REFLECTION on MAGNITUDE OF RISK... - The risk seems important. That triggers TEST REDESIGN...

19 Exploratory Testing Dynamics Created by James and Jonathan Bach v1.2 Copyright 2005, Satisfice, Inc Chartering. Making your own decisions about what you will work on and how you will work. Manipulating. Making and managing contact with the object of your study; configuring and interacting with it. Designing experiments and establishing lab procedures. Observing. Gathering empirical data about the object of your study; collecting different kinds of data, or data about different aspects of the object. Designing experiments and establishing lab procedures. Modeling. Composing, describing, and working with mental models of the things you are exploring. Identifying relevant dimensions, variables, and dynamics. A good mental model may manifest itself as having a feel for the product; intuitively grasping how it works. Conjecturing. Considering possibilities and probabilities. Considering multiple, incompatible explanations that account for the same facts.

20 Exploratory Testing Dynamics (continued) Created by James and Jonathan Bach v1.2 Copyright 2005, Satisfice, Inc Questioning. Identifying missing information, conceiving of questions, and asking questions in a way that elicits the information that you seek. Recording. Preserving information about your process, progress, and findings. Reporting. Making a credible, professional report of your work to your clients in oral and written form. Pairing. Working and thinking with another person on the same problem. Resourcing. Obtaining tools and information to support your effort. Exploring sources of such tools and information. Getting people to help you. Refocusing. Managing the scope and depth of your attention. Looking at different things, looking for different things, in different ways. Branching/Backtracking. Allowing yourself to be productively distracted from one course of action in order to explore an unanticipated new idea. Generating/Elaborating. Working quickly in a manner good enough for the circumstances. Revisiting the solution later to extend, refine, refactor or correct it. Alternating. Switching among different activities or perspectives to create or relieve productive tension and make faster progress.

21 Exploring the Commission Problem Lock, Stock, and Barrel In the Commission Problem, a rifle salesman in the old American West sells rifles, either as a whole, or in the three parts, the lock, the stock, and the barrel. Prices are: Locks: $45 Stocks: $30 Barrels: $25 Complete Rifle: $100 The salesman s commission is computed on a sliding scale: 10% on the first $1000 of total sales 15% on sales from $1001 to $ % on sales over $1800

22 Recognition of a Fault In the first few months, the salesperson never exceeds $1000 in sales, and the commission is correct. When sales finally reach the 15% level, however, the salesperson s commission is less than expected. And one month, when sales reached $1900, the commission was slightly more than expected. If the salesman had a spreadsheet, he might have tried out some possible sales

23 The First Exploration case # locks stocks barrels sales Expected commission Computed commission Pass? Expected less Computed $100 $10 $10 pass $ $800 $80 $80 pass $ $1,000 $100 $100 pass $ $1,100 $115 $100 fail $ $1,700 $205 $190 fail $ $1,800 $220 $205 fail $ $1,900 $240 $260 fail ($20)

24 The Exploration Begins The rifle salesman knows the computation involves four business rules: (1) sales = $45*locks + $30*stocks + $25*barrels (2) commission = 0.10*sales (for $0 <= sales <= $1000) (3) commission = $ *(sales - $1000) (for $1000 < sales <= $1800) (4) commission = $ *(sales - $1800) (for sales > $1800) The sales calculation is correct, but could there be offsetting errors in the coefficients?

25 The Second Exploration Try calculating with one variable at a time. case # locks stocks barrels sales Expected commission Computed commission Pass Fail? Expected less Computed $ $45.00 $45.00 pass $ $ $30.00 $30.00 pass $ $ $25.00 $25.00 pass $0.00 Looks OK. No problem with the coefficients, so Equation (1) is being computed correctly. The salesman already knows that equation (2) is also being computed correctly. Time to explore equation (3).

26 The Third Exploration Check Equation (3). Try to devise sales near the $1000 level. case # locks stocks barrels sales Expected commission Computed commission Pass/ Fail? Expected less Computed $995. $99.50 $99.50 pass $ $1,000. $ $ pass $ $1,005. $ $85.75 fail $15.00 Aha! (Ein Licht geht auf!) the only thing wrong with equation (3) must be the amount subtracted from sales. Solving equation (3) with the computed and expected commission: (3 expected) $ = $ ($ $1000) (3 computed) $85.25 = $ ($1005 x), x = $1100

27 Not Finished Yet There is a remaining fault. Look at test case 7, in which the salesman receives too much commission. You can try exploratory testing with the explorecommission.xls spreadsheet.

28 Observations and Conclusions Exploratory testing requires discipline Not an ad hoc activity Documentation, especially of intermediate results, is valued Allows a tester to focus on deep faults Can be extremely effective Can be extremely efficient Very dependent on the tester

29 Test-Driven Development The essence of agility Also known as Test-First Development Requires automation A bottom-up process Presumes refactoring Excellent for fault isolation Test granularity is an issue Big Question: does TDD result in a good design?

30 Test-Driven Development (TDD) Life Cycle Start Create a test Run the full test set Write code so test passes Refactor As needed

31 TDD Example: a Boolean Function to Determine Leap Years Definition: A year is a leap year if it is a multiple of 4, but century years are leap years only if they are multiples of 400. Test-Driven Development would break this into small, individual user stories (also called tasks). Coded here in a pseudo-code (a lingua franca) that resembles Visual Basic. Try it yourself (with no looking ahead) in your favorite programming language.

32 My isleap Code

33 User Story 1: A year divisible by 4 is a leap year Test Case 1 Input: 2004 Expected Output: True (existing) Pseudo-Code in normal font Function isleap(year) As Boolean End isleap Running Test Case 1 on this code fails. Add just enough code to make the test pass.

34 User Story 1: A year divisible by 4 is a leap year Test Case 1 Input: 2004 Expected Output: True (updated) Pseudo-Code in bold face font Function isleap(year) As Boolean dim year AS Integer isleap = False 'MOD is the modulo arithmetic built-in operator in most languages If (( year MOD 4) = 0) Then IsLeap = True EndIf End isleap Test Case 1 passes. Now do User Story 2.

35 User Story 2: A year not divisible by 4 is a common year Test Case 1 Input: 2004 Expected Output: True Test Case 2 Input: 2007 Expected Output: False (existing) Pseudo-Code in normal font Function isleap(year) As Boolean dim year AS Integer isleap = False If (( year MOD 4) = 0) Then IsLeap = True EndIf End isleap Test Case 1 passes. Test Case 2 fails. Now add just enough code so that Test Case 2 passes.

36 User Story 2: A year not divisible by 4 is a common year Test Case 1 Input: 2004 Expected Output: True Test Case 2 Input: 2007 Expected Output: False (updated) Pseudo-Code In bold face font Function isleap(year) As Boolean dim year AS Integer isleap = False If (( year MOD 4) = 0) Then IsLeap = True Else IsLeap = True EndIf End isleap Test Cases 1 and 2 pass. Now do User Story 3

37 User Story 3: A century year not divisible by 400 is a common year Test Case 1 Test Case 2 Test Case 3 Input: 2004, Expected Output: True Input: 2007, Expected Output: False Input: 1900, Expected Output: False (existing) Pseudo-Code In normal font Function isleap(year) As Boolean dim year AS Integer isleap = False If (( year MOD 4) = 0) Then IsLeap = True Else IsLeap = True EndIf End isleap Test Cases 1 and 2 pass. Test Case 3 fails. Now add just enough code so that Test Case 3 passes.

38 User Story 3: A century year not divisible by 400 is a common year Test Case 1 Test Case 2 Test Case 3 Input: 2004, Expected Output: True Input: 2007, Expected Output: False Input: 1900, Expected Output: False (updated) Pseudo-Code In bold face font Function isleap(year) As Boolean dim year AS Integer isleap = False If ((( year MOD 4) = 0) AND NOT((year MOD 100) = 0))) Then IsLeap = True Else IsLeap = True EndIf End isleap Test Cases 1, 2 and 3 pass. Now do User Story 4

39 User Story 4: A century year divisible by 400 is a leap year Test Case 1 Test Case 2 Test Case 3 Test Case 4 Input: 2004, Expected Output: True Input: 2007, Expected Output: False Input: 1900, Expected Output: False Input: 2000, Expected Output: True (existing) Pseudo-Code In normal font Function isleap(year) As Boolean dim year AS Integer isleap = False If ((( year MOD 4) = 0) AND NOT((year MOD 100) = 0))) Then IsLeap = True Else IsLeap = True EndIf End isleap Test Cases 1, 2 and 3 pass. Test case 4 fails. Now add just enough code so that test case 4 passes.

40 User Story 4: A century year divisible by 400 is a leap year Test Case 1 Test Case 2 Test Case 3 Test Case 4 Input: 2004, Expected Output: True Input: 2007, Expected Output: False Input: 1900, Expected Output: False Input: 2000, Expected Output: True (updated) Pseudo-Code In bold face font Function isleap(year) As Boolean dim year AS Integer isleap = False If ((( year MOD 4) = 0) AND NOT((year MOD 100) = 0))) OR ((year MOD 400 = 0)) Then IsLeap = True Else IsLeap = True EndIf End isleap Test Cases 1, 2, 3 and 4 pass. Done with function isleap.

41 Advantages of Test Driven Development In this example, the steps are deliberately small. Customer and developer can (should!) jointly determine granularity of user stories. Fault isolation is greatly simplified (in fact, trivial). If a test case fails, the fault must be in the most recently added code. Once a new test case passes, a working (subset) of the desired software can always be delivered. Something always works!

42 Disadvantages of Test Driven Development Useful granularity is an issue. There is no guarantee that user stories arrive in a sensible order. There is no guarantee that user stories are the same size (or require similar effort) Bottom-up coding often results in poorly structured code, making refactoring necessary. Tool support (zb junit) is essential.

43 Comparison of Model Driven Development (MDD) and Test Driven Development (TDD) First American s view of Eagles and Mice Eagles have the big picture Mice focus on the details (both views are important!) MDD is a rigorous, top-down approach. TDD is a bottom-up approach.

44 TDD isleap in Visual Basic (refactored) Public Function isleap(year) As Boolean Dim year As Integer Dim c1, c2, c3 As Boolean 1. c1 = (year Mod 4 = 0) 2. c2 = (year Mod 100 = 0) 3. c3 = (year Mod 400 = 0) 4, isleap = False 5. If ( (c1 AND NOT(c2)) OR (c3)) Then 6. IsLeap = True 7. Else 8. IsLeap = False 9. EndIf End Function

45 Decision Table Model of isleap Conditions r1 r2 r3 r4 r5 r6 r7 r8 C1. year is a multiple of 4 T T T T F F F F C2. year is a century year T T F F T T F F C3. year is a multiple of 400 T F T F T F T F Actions (logically impossible) X X X X A1. year is a common year X x A2. year is a leap year X X test case: year =

46 MDD isleap in Visual Basic 1 Public Function isleap(year) As Boolean Dim year As Integer Dim c1, c2, c3 As Boolean 1. c1 = (year Mod 4 = 0) 2. c2 = (year Mod 100 = 0) 3. c3 = (year Mod 400 = 0) 4, isleap = False 5. If c1 Then 6. If c2 Then 7. If c3 Then 8. isleap = True rule r1 9. Else 10. isleap = False rule r2 11. End If 12. Else 13. isleap = True rule r4 14. End If 15. Else 16. isleap = False rule r8 17. End If End Function

47 Observations The TDD version is less complex (really?) Cyclomatic complexity of the DD version is 2 Cyclomatic complexity of the MDD version is 4 The TDD version gradually built up to a compound condition (that might be hard to understand, and to modify). The decision table model assures Completeness Consistency Non-redundancy Both versions require 4 test cases

48 Open Questions Does Agile (bottom up) Development actually result in better designs? Can any form of Agile Testing reveal deep faults? zb: faults revealed only by data flow (define/use) testing computational faults time-dependent faults What about maintenance? Agile code has well-named variables and components (hopefully) been refactored carefully no comments test cases are the specification

49 Conclusions The Agile community is VERY passionate about their methods. Agile Development clearly works well for SOME applications time critical uncertain customer (I ll know what I want when I see it!) Some agile methods apply to non-agile projects Context-Driven Testing Good Enough Testing Exploratory Testing Test Driven Development is excellent (ausgezeichnet!) for fault isolation. Try it yourself with tddtriangle.xls

50 Also geht alles zu Ende allhier: Feder, Tinte, Tobak auch wir. Zum letzten Mal wird eingetunkt, dann kommt der große schwarze W. Busch