Making Test Automation Work in Agile Projects

Transcription

1 Making Test Automation Work in Agile Projects StarWest 2009 Lisa Crispin With Material from Janet Gregory 1

2 Topics Making Automation Work Learn ways to: Overcome common obstacles Use automation to keep up Succeed with the whole team approach Use automation to enhance ET Evaluate and implement tools Use ROI to help succeed 2

3 Why Automate? Free up time for most important work Repeatable Safety net Quick feedback Help drive coding Manual tests are error prone Tests provide documentation You know what your system does 3

4 Barriers to Test Automation What s holding you back? Skills Fear????? Changing code Not designed For testability 4

5 Pain and Fear Programmers don t feel manual test pain Testers treated as safety net Fear Programmers lack testing skills Testers lack programming skills 5

6 It s Worth It ROI explain to management Present value of automated tests Acknowledge hump of pain 6

7 Initial Investment Hump of pain Legacy code, changing code Tools, infrastructure, time Effort Time 7

8 What Affects ROI Good test practices increase ROI Simple, well-designed, refactored tests Test resources improve over time Poor test practices reduce ROI Tests are hard to understand Tests are hard to maintain Copyright 2008: Janet Gregory, DragonFire Inc. 8

9 Questions on Why Automate? 9

10 Small Group Exercise Write obstacles hindering your team on small Post-Its.. Stick them on the wall or flip chart. Group similar ones. What patterns emerge? Are there any unexpected barriers? Share patterns, surprising outcomes with the class. 10

11 Getting Over the Hump The test automation pyramid The agile testing quadrants What should be automated What shouldn't Difficult areas 11

12 Test Automation Pyramid 12

13 Agile Testing Quadrants 13

14 What Should We Automate Quadrant 1 Tests Unit, Component, TDD (all) Quadrant 2 Tests Behind GUI, API, web services, GUI Quadrant 3 Tests Leverage automation where useful Setting up tests, tools, data generation Quadrant 4 Tests Load, performance, stress 14

15 What Else Can We Automate? Automate tedious or repetitive tasks Testing-related or otherwise Continuous integration, builds Deployments Checking for updates Parsing, comparing output Automating tasks for business 15

16 What We Shouldn t Automate Wizard of Oz tests Paper prototyping Usability tests / look and feel Customer adhoc testing Exploratory testing Tests that never fail??? One-off tests Tests covered elsewhere 16

17 Where Should We Be Careful GUI tests Need to test the GUI Can do ATDD - Watch ROI Automating end to end tests Don't go overboard, automating every path Aim for minimum feature coverage Remember the pyramid Push testing down as low as you can Robust lower-level tests = better ROI 17

18 Hard to Automate? Legacy code Hard to automate, or just lack of skill? Working Effectively with Legacy Code Feathers Strangling Fowler, Thomas Mike Thomas article: Software_Development 18

19 Discussion Where should you start? Or do next? Is there low-hanging fruit? What s the best ROI? 19

20 Agile Automation Strategy What hurts the most Layered approach Applying agile principles Choosing the right tools 20

21 What Hurts the Most Keep an impediment backlog What s biggest obstacle? Time Tools Code design Testability 21

22 Multi-Layered Approach Example: TDD delivers unit, component tests ATDD delivers API level, GUI tests GUI smoke tests as regression safety net For new or legacy code 22

23 Specialists Needed? May bring in shared or outside resources Performance, load Security User experience design Toolsmiths Dev team still responsible for completing all tests Multiple viewpoints provide synergy 23

24 Discussion What skills would help your team overcome automation barriers? What skills are missing from your team? Where could you get them? 24

25 Applying Agile Principles Whole team approach Learn by doing Simplicity Iterative feedback Small chunks

26 Whole-Team Approach Team responsible for testing, quality Team responsible for testing activities Includes automation Whole team has all the skills needed Team designs for ease of test automation 26

27 Learn by Doing Courage don t be afraid to fail Use agile coding practices for automation Object-oriented, libraries, modules Simple design Pairing Refactoring Test-first if scripts have logic 27

28 Simplicity Address one or two needs at a time Understand the problem first Try simplest approach first Work in small chunks, thin slices Incremental & iterative 28

29 Iterative Feedback Commit Try a new tool/framework for n iterations Plan automation tasks for each iteration Use retrospective to evaluate 29

30 Automate a Steel Thread at a Time: Example 4-step UI to upload item information from csv file into inventory system Thread 1: All four pages, user can navigate Thread 2: Select age category, enter info on page 1, display on page 2, browse and upload file on page 2 Thread 3: Validate data in file, display on page 3 Thread 4: Persist data, display success message on page 4 30

31 Steel Thread Example A steel thread diagram, with passes numbered 1, 2. 3 & 4. 31

32 Small Group Exercise You re a development team at Agile Testing Toys. The story on the next slide is your next priority. Break the story down into thin slices or threads whose tests can be automated. Prioritize them. How could the functionality be delivered incrementally? 32

33 Story As an Agile Testing Toys shopper, I want to select a shipping type for my items during checkout and see the shipping cost. Assumptions/ Conditions of Satisfaction User has already entered shipping address. User can choose different options for different items. The options are Ground, 2 day and Overnight. Items > 20 lbs are Ground only API to cost calculator is available, takes shipping type, postal code and weight, returns shipping cost. 33

34 Questions on Creating an Automation Strategy? 34

35 Choosing Your Tools One size does not fit all!

36 Find Time for Evaluating 1. Get commitment for time to research 2. Budget time Individual or group 3. Determine your requirements 4. Do some basic research 5. Compile a list of tools to consider 6. Try the ones which meet your criteria 36

37 Understand the Purpose What s being automated? e.g. Ajax, SSL support; load; embedded s/w Speed up exploratory testing, build test data Existing tools, environment e.g.. integration with build process Reporting needs Who s writing, maintaining the tests? Who s using the tests? What for? What are your constraints? 37

38 What Fits Your Situation Existing skills on team Language of application under test Collaboration needs Utilities (automating tedious tasks) vs. testing Life span, future use of tests 38

39 Vendor Tools - Pros Existing expertise Some built on open-source libraries Fast ramp-up for non-programmers Perceived as safe choice Training, support Part of existing tool set May have robust features 39

40 Vendor Tools - Cons Tend to be heavyweight Tend to be programmer-unfriendly Scripts may be brittle, high-maintenance Capture-playback problematic Not designed for long-term maintainability Can be pricey 40

41 Open-Source Tools - Pros Designed by test-infected programmers Designed for agile environments Designed for maintainability Programmer-friendly May have excellent support, tutorials, doc Easily customized Low up-front cost 41

42 Open-Source Tools - Cons May be difficult for non-programmers Depends on the tool/framework Future enhancements may be uncertain Training, support can be an issue Be sure to look for active development community 42

43 Home-Brewed - Pros Programmer-friendly Integration with app, IDEs Development framework may support Rails, Ruby, Groovy Can build on top of existing framework Fit, Slim, Watir, RSpec Specifically tailored to needs Someone s there to address problems 43

44 Home-Brewed - Cons Team needs enough bandwidth, expertise Reporting framework Allow test specification by nonprogrammers Could be harder sell to management 44

45 Example: A Java Team s Choices IntelliJ Idea, Eclipse for IDEs CruiseControl, Hudson for CI JUnit for TDD, unit, component FitNesse for functional, behind GUI Canoo WebTest for GUI regression smoke tests Watir to aid exploratory testing JMeter for load, performance testing Perl scripts for comparing files, making files human-readable Java programs for concatenating forms, cover letters 45

46 Example: UPM Team s Choices Visual Studio, Resharper MBUnit Hudson for CI FitNesse/SWAT, DBFit for GUI, functional tests ncover for code coverage Confluence wiki for documentation 46

47 Discussion Any tool selection success stories to share? 47

48 Questions about choosing tools? 48

49 Making Test Automation Work Time to learn, implement, design Testable architecture Test data Managing tests 49

50 Time To Do It Right Limit scope Write automation task cards Quality must be goal Long-term, will let you go faster 50

51 Testable Architecture Layered architecture e.g.. UI, business logic, data access Ports and Adapters pattern App can work without UI or database Ports accept outside events Adapters convert for human or automated users 51

52 Test Data Avoid database access when possible Setup/Teardown Independent, rerunnable tests Canonical data Refresh before each test run Customizable data for ET Production-like data Get customers to provide example data 52

53 Managing Automated Tests Documenting tests Tests as documentation Finding what you need Reporting results Analyze failures Let CI do the work Test coverage 53

54 Example: My Team s Tool Choices 54

55 Test Management Tools Tool may include management framework FitNesse Wiki documentation + executable tests Hierarchy Rasta spreadsheets to organize tests Twist uses Eclipse IDE What problem are you trying to solve? Simplest approach Use SCCS for tests 55

56 Small Group Exercise Look at the barriers from the first exercise. What s one thing you can do when you get back to the office to overcome one of your team s barriers? Write it on a post-it and put it next to the barrier on the flip chart paper. Share your first action item with the class 56

57 Make Automation Work for You Don t try to find one perfect solution Start simple Budget time to evaluate, implement Use feedback, retrospectives Apply agile principles, practices Budget time to refactor, try new alternatives 57

58 Remember It s a team problem! Build foundation of core agile practices Experiment Baby steps 58

59 Questions? Aha Moments? 59

60 Some Agile Testing Tool Resources awta.wikispaces.com/2009toolslist softwareqatest.com/qattls1.html opensourcetesting.org nunit.org/index.php webtest.canoo.com testingfaqs.org junit.org wtr.rubyforge.org fit.c2.com gojko.net/fitnesse/dbfit fitnesse.org selenium.openqa.org jameleon.sourceforge.net artofunittesting.com/chapters/tools_and_frameworks 60

61 Now Available Agile Testing: A Practical Guide for Testers and Agile Teams By Lisa Crispin and Janet Gregory 61

62 Test Patterns Xunit Test Patterns: Refactoring Test Code By Gerard Meszaros 62

63 Bridging the Communication Gap Specification By Example and Acceptance Testing Gojko Adzic Copyright 2008 Janet Gregory, DragonFire 63