Test Automation in Agile. Archana Ramesh Senior Manager QA Ariba Technologies, Bangalore Jan 2008

Transcription

1 Test Automation in Agile Archana Ramesh Senior Manager QA Ariba Technologies, Bangalore Jan

2 What the customer got... 2

3 What the customer got... 3

4 Agenda Agile SCRUM Testing in SCRUM Challenges faced by the Testing team in a Agile / SCRUM project Automation Strategic Thinking in Test Automation Test Infrastructure Test Automation Tools and current trends Setting up an ideal Agile Test team Best practices Take Aways 4

5 Agile 5

6 Today s S/W development scenario Today s markets move at blinding speed. There is pressure introduced into the software industry requiring them to be as adaptable as the markets. There is a tension between quality, cost, and time impacting software development processes. Complexity is on the increase. Last simple project was in

7 Think out of the Box We also need to allow for innovation 7

8 Forrester Research Poll Around 65% of IT projects fail Most likely root cause for project failures Poorly scoped projects (57%) Technical or integration issues (44%) Poor implementation staff or services (36%) Buggy software (35%) Unattainable business requirements (30%) Loss of executive sponsorship (27%) Traditional development processes such as Waterfall fail to deliver 8

9 Agile Process philosophy Agile Software development philosophy has it roots in the reality of today s markets. The emergence of agile software processes attempt to deal with the issues introduced by rapidly changing and unpredictable markets. Agile methods attempt to fix this by focusing on Business value Early return on investment S/W development thru incremental and iterative delivery steps 9

10 Agile 30 seconds Elevator talk Pay Attention and Adapt Empirical management & control process inspect and adapt feedback loops Used to manage complex projects since 1990 Delivers business functionality in 30 days Scalable to distributed, large, and long projects CMM Level/3 and ISO 9001 compliant Extremely simple 10

11 Agile Manifesto Utah, 2001 Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan While there is value in the items on the about right, the we Replanning value!!! items on the left more Note that iterative and incremental is not enough to be agile it s all 11

12 Agile Appeal is a reaction to the bureaucracy of the engineering methodologies. compromise between no process and too much process Agile methods are adaptive rather than predictive Agile methods are people-oriented rather than process-oriented. 12

13 Agile Appeal Embody and encourage the values is a reaction to the bureaucracy of the of engineering methodologies. Communication compromise between Simplicity no process and too Feedback - Deliver much process Early and regularly Courage Agile methods are adaptive rather than predictive Agile methods are people-oriented rather than process-oriented. 13

14 Agile(XP) Process Flow Gather requirements as User Stories Release Planning Meeting Acceptance Tests Architect Code Unit test Daily Stand- Up meeting Ite Make small product releases Visualize Iteration Planning Meeting 14

15 Agile practices Crank up all the knobs to 10 on the things that Prioritize are essential and leave out everything else. Re-factor mercilessly Test-Driven Development Pair Programming Leave optimization to the last Stand Up meetings everyday Deliverable software at the end of each iteration is the primary measure of success 15

16 Testing in Agile QA is an essential part of Agile All code must have Unit tests Write your Unit tests first, then code Build a Unit Test Framework early Universal Test Suite Test Often When a bug is found, add a test for it Acceptance Tests Must Are Results run PASS often published 16

17 Scrum 17

18 Scrum Origin and Usage - Jeff Sutherland and Ken Schwaber FDA-approved, Independent Software life-critical Vendors software (ISVs) for x- rays Fortune and 100 MRIs companies Enterprise Small startups Financial, workflow systems Biotech Internal development Call Contract center development systems Laser subsystems for fiber optic networks 24x7 with % uptime requirements Multi-terabyte database applications 18

19 Scrum Characteristics One of the agile methodologies Self-organizing teams Product progresses in a series of month-long Sprints No specific engineering practices prescribed Wraps existing engineering practices, including XP Delivers business functionality in 30 days Provides a Project Management Framework over Agile Processes 19

20 Scrum Characteristics One of the agile methodologies Self-organizing teams Product progresses in a series of month-long Sprints No specific engineering practices prescribed Wraps existing engineering practices, including XP Delivers business functionality in 30 days Provides a Project Management Framework over Agile Processes 20

21 Scrum Characteristics One of the agile methodologies Self-organizing teams Product progresses in a series of month-long Sprints On Time Deliverable software at the end of each Sprint is the primary measure No specific engineering practices prescribed Wraps of success existing!! engineering practices, including XP Delivers business functionality in 30 days Provides a Project Management Framework over Agile Processes 21

22 Scrum Flow Defining Done Daily Scrum Meeting, Sprint Burndown Chart Backlog broken down to tasks in Scrum tracking sheet Test based on User Stories Potentially Shippable Product increment Sprint Planning Meeting Demo and feedback User Stories from PO and Design Partner Sprint Review and re-planning 22

23 Scrum Flow Defining Done Daily Scrum Meeting, Sprint Burndown Chart Backlog broken down to tasks in Scrum tracking sheet Sprint Planning Meeting Test based on User Stories Potentially Shippable Product Scrum is a way for everyone to feel good about their job, their contributions, and that they have done the very best and thus maximize productivity!! increment Demo and feedback User Stories from PO and Design Partner Sprint Review and re-planning 23

24 Scrum s Roles Product Vision, ROI, Release The Product Owner The Scrum Master The Team Everyone else is not part of Scrum The breakfast commitment Someone who is only interested in the project. Process owner, Implementation Owner, Coach, Remove impediments, Lead, Deliver 7+-2, X-FT, Convert backlog items in to functionality increments, Self- Organizing, Self - Motivating, Takes full ownership 24

25 The breakfast commitment 25

26 Daily Scrum: The inner iteration Socialization and Synchronization Time boxed to 15 minutes Daily Stand up meeting Not for problem solving. Resolutions are offline. Three questions: 1. What did you do yesterday 2. What will you do today? 3. What obstacles are in your way? Outsiders are invited Help avoid other unnecessary meetings Only team members can talk 26

27 Sprint: The external iteration Time box of 30 sequential calendar days Team works to turn the sprint backlog into an increment of potentially shippable functionality that meets the sprint goal. Self organization of the team Activities are visible through the Sprint Backlog and Sprint Burndown Charts Tasks not completed in current Sprint moved to the next Sprint backlog 27

28 Defining Done Potentially shippable product increment At the end of every Sprint Customer should be able to use the increment Checked in Thoroughly tested Built into an executable implementable Fully documented Slice of a product which contains all aspects of the final product 28

29 Sprint Review Meeting End of Sprint Time boxed to 4 hrs Informal Team presents (Demo) What was accomplished during the current sprint To product owner, the team and anybody who is interested in the sprint outcome Input to the next sprint planning 29

30 The Product Backlog 30

31 The Sprint Backlog 31

32 Sprint Burndown Chart 32

33 Scrum Task Board 33

34 The Product Release Cycle Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 QA Sprint Release Final Sprint 34

35 The Feature team (Implementation) FT Manager FT Lead Dev Engineers (2-3) QA Engineers (1-2) Doc Writer(1) Product Manager Release Engineer 35

36 Question Who is responsible for delivery in a Scrum team? The whole team! 36

37 Testing in Scrum 37

38 Testing in a Scrum Team Members of the Scrum Team Mantra - Test early Test Often Challenge of testing small increments of product functionalities Test Cases from User Stories, Architect Unit Tests Automate as much as possible Build regression suite as you go Automate build qualification Tight feedback loop with Developers Weekly Group Tests 38

39 Test Orientation in a Scrum Team 39

40 Test team s Value-Add Prioritized list of project requirements QA should stand in for customers QA Engineers select as much product backlog Own the they Quality believe of they increment can test of the sprint functionality during Sprint!! Skills Needed Domain Knowledge - QA should understand The business need Customer problems QA team should understand the technology 40

41 Sprint flow from the Test Team s perspective Test Plan & Estimates Acceptance Testcases Test Execution Test Reports Test Metrics 41

42 The Product Release Cycle Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 QA Sprint Release Final Sprint 42

43 The Product Release Cycle Last 2 sprints are largely the Test Team s responsibility to pull the project thru Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 QA Sprint Release Final Sprint 43

44 QA Sprints Once all the developments are Done Sprints focused on QA Beta/ Level 3 Test Execution Sprint Integrated System testing Defect Fixes All Blocking defects/ Priority 0 fixed within 24 Hrs All Priority 1 defects fixed before exiting the QA Sprint Bug Councils and controlled checkins Final Test / Gold Label test 44

45 Question How do you measure the productivity of a team in Scrum? By its Velocity number of Backlog items converted per Sprint! 45

46 Question How do you measure the productivity of a team member in Scrum? You don t! 46

47 Challenges faced by Testing team in Scrum/Agile 47

48 Task Challenges Testing of product increment (PI) to be completed within the same Sprint Testing of Legacy product features for regression to be completed within the same Sprint Fast enough so as to help fix defects within the same Sprint Cannot neglect System testing till end of release - needs to be incremental too Estimations could go awry, meaning Testing time is squeezed 48

49 PI Challenges PI not ready for end to end testing until late in the Sprint By then, there is very less time to do a full verification and validation Not all PI s have easy testability Often times, complete impact of PI on the rest of the system, not well-understood within the Sprint Little or no time for end to end systemlevel regression and integration testing 49

50 Resource Challenges Testing effort growth as you move forward sprint after sprint Resource planning, therefore, not predictable Slows down testing turnaround time Continuous Sprints leads to QA Engineer burn up Dev QA tight co-ordination is challenging to create and maintain 50

51 Challenges of On-demand software testing vs. Licensed Software testing 51

52 Some differences in On-Demand S/W testing Much shorter release cycles, less time to Beta test, Bake No time to release Patches before customers go Live Less customization options hence functionality needs to be very close to reality. Performance and Scalability are BIG 52

53 Some differences in On-Demand S/W testing Much shorter release cycles, less time to Beta test, Bake No time to release Patches before Quality bar is customers go Live very high Less customization options hence functionality needs to be very close to reality. Performance and Scalability are BIG 53

54 Challenges The most important thing we can do to break the compromises we impose on customers is to move testing forward and put it in-line with (or prior to) coding. Build unit and acceptance tests, integrate code frequently, repeat the tests as often as possible. In other words, find and fix the defects before they even count as defects. 54

55 Challenges The most important thing we can do to break the compromises we impose on customers is to move testing forward and put it in-line But with how (or?? prior to) coding. Build unit and acceptance tests, integrate code frequently, repeat the tests as often as possible. In other words, find and fix the defects before they even count as defects. 55

56 Automation is the answer 56

57 We have to Automate There is no other way you can manage continuous growth in scope without compromising on Turnaround time Automation is not just a Time Saver It s purpose is to provide early and frequent feedback. The mental trap is thinking that automation just compresses the time spent testing at the end of an iteration. 57

58 We have to Automate There is no other way you can manage continuous Test growth Automation in scope is without compromising on Turnaround time about Velocity not just Time!! Automation is not just a Time Saver It s purpose is to provide early and frequent feedback. The mental trap is thinking that automation just compresses the time spent testing at the end of an iteration. 58

59 We have to Automate If we can achieve a good level of test automation then we don t just save time doing traditional test cycles we can move the system test cycles inline with frequent builds, and provide feedback in hours, not days. Will let us address issues while they re cheapest to fix free us to focus on testing small, frequent PIs while it s still fresh in the developer s mind. Its a great way to make system testing a part of the daily workflow, rather than something that happens at the end of an iteration. 59

60 We have to Automate If we can achieve a good level of test automation then we don t just save time doing traditional test cycles we can move the system test cycles inline with We can change the way frequent builds, and provide feedback in hours, not days. we do system / regression testing!! Will let us address issues while they re cheapest to fix free us to focus on testing small, frequent PIs while it s still fresh in the developer s mind. Its a great way to make system testing a part of the daily workflow, rather than something that happens at the end of an iteration. 60

61 Strategic Thinking in Test Automation 61

62 Recipe for failure "You start writing code and I'll go find out what it is supposed to build." Abraca dabra 62

63 Think strategically Don the Product Manager s cap Envision Roadmap / strategy for automation based on existing Engineering Practices nature of product quality focus Do not just look for increasing automation percentage Think beyond just saving time by automating traditional test approaches Think Agile Iterative and incremental in test automation 63

64 Think strategically Don the Product Manager s cap Envision Roadmap / strategy for automation based on Strive for a Co-operative existing Engineering Practices environment that thinks nature automation of productalong with quality product focus development! Do not just look for increasing automation percentage Think beyond just saving time by automating traditional test approaches Think Agile Iterative and incremental in test automation 64

65 Think Strategically Build Testability in to every Manager s objectives. Fund Automation adequately as a separate Budget Line Item And then, plan for ongoing investment Choose the right Tools, Framework based on technology, product and team requirements The right strategy brings higher velocity for the whole project, not just a shorter test cycle 65

66 Think Strategically Build Testability in to every Manager s objectives. Fund Automation adequately as a separate Budget Its like accelerating Line Item the And then, plan whole for ongoing car instead investment of Choose the right just Tools, bringing Framework the based on technology, destination product and a little team requirements closer!! The right strategy brings higher velocity for the whole project, not just a shorter test cycle 66

67 Test Infrastructure 67

68 Test Infrastructure Without up-front planning and infrastructure design, automated tests can quickly become difficult to create and maintain, and soon a dead investment A well planned test infrastructure sets the right tone for phenomenal benefits from automation Do not expect a set of randomly spawned tools to come together as the infrastructure Invest time in designing an Infrastructure with long term Automation goals in mind 68

69 Test Infrastructure Without up-front planning and infrastructure design, automated tests can quickly become difficult to create and maintain, and soon a dead investment At this point, Focus on the A well planned methodology, test infrastructure not the sets the right tone for phenomenal benefits tool!! from automation Do not expect a set of randomly spawned tools to come together as the infrastructure Invest time in designing an Infrastructure with long term Automation goals in mind 69

70 Test Infrastructure Automation library and infrastructure is the code base on which individual test scripts are built. It is literally the foundation of the automation effort. Create a Test Framework such that everybody in the Engineering team can use and contribute to the automated tests Design criteria Nature of product Technology Quality requirements Stability Scalability Adaptability Reusability Ease of use Maintainability 70

71 Test Infrastructure Automation library and infrastructure is the code base on which individual test scripts are built. It is literally the foundation of the automation effort. Create a Test Framework such that everybody in the Engineering team can use and contribute to the automated tests Design criteria Nature of product Technology Quality requirements Stability Scalability Adaptability Reusability Ease of use Maintainability Push for Automatability for all PIs 71

72 Test Infrastructure Planning Elements Framework Design Software Tools Standards and Processes - Who should automate - What should be automated - Coding standards Configuration Management How should scripts be run Hardware / Test Boxes Help on using the tools Robust usage practices 72

73 Test Infrastructure Planning Elements Framework Design Software Tools Standards and Processes In - Who other should words, automate treat it like developing a - What should be automated complimentary product - Coding standards to the one you are Configuration Management testing!! How should scripts be run Hardware / Test Boxes Help on using the tools Robust usage practices 73

74 Question How would you define Test Automation? Consider thinking of Test Automation as any use of tool to support testing! 74

75 Test Automation Tools and current trends 75

76 Tools Bazaar There is an endless barrage of tool vendors trying to hawk their wares. Exceed expectations Synergize Test Reduce And All with Case at 100% the test the generation touch click on-time dynamics of and of a a at increase the button delivery of blink your mouse productivity! and process! of an! zero eye! defects! What the vendors don t tell you is that 80% of all automation efforts fail. Nor do they mention that there is no one product that fits all Buying a tool is not the Cure all for all automation problems 76

77 Tool Selection criteria Tools that recognize your product s controls and can integrate with it well Good recovery and error logging Extensibility and maintainability A robust and flexible programming language support One that allows data-driven testing The operating systems your application supports The development environments used to create your application Third party software with which your application integrates Simple is Great 77

78 Agile / Current trends Picking methods and tools that are simple and slivers of tasks that can be delivered in short durations (very Agile) Activity driven by Testers Increasing usage seen of : Scripting languages (Perl, Python) Programming languages (Java, C++) Open source tools Unit test tools (junit, httpunit) Functional test Tools (Selenium, Canoo Webtest, AutoIt, Fitnesse) Code Coverage Tools (Emma, jcoverage) 78

79 Setting up an ideal Agile Test Team 79

80 Setting up the team Team members : Hire Test Engineers with strong Functional Testing experience and some Programming background Get experienced Developer(s) on board (see next slide) Mentoring the team : Groom to work closely with Developers and Architects Train on technology the product is being built on Train on Application domain Train every one to walk, talk and breathe Agile QA Practices Consider setting up a small team of Toolsmiths to move Automation along 80

81 Introducing the Toolsmith Test automation progresses when supported by dedicated programmers (toolsmiths). Toolsmiths are directed by testers. This is how it works. 81

82 Toolsmith(s) responsibilities Gather and apply a wide variety of tools to support testing. Advocate for testability features and produce tools that exploit those features. Respond rapidly to requests for assistance from testers. Seek out test productivity problems. Apply technology to improve the test process. Advocate for specific testability features in products. Research available tools and learn how to use them. Gather tools that developers or testers produce. Review upcoming product plans to assess automation possibilities. 82

83 Question How many toolsmiths do you need in an average Product team of about 50 Engineers? Atleast 1!! Co-located with the Testers. 83

84 A Day in the life of a Tester in a Scrum team 84

85 9.00 am Attend SCRUM meeting Get yesterday s blockers attended to. J says no easy fix, will give a lengthy workaround. Need to get infra to test my PI for this week. Who can help? Am writing test plans for next week s PI. Will update Test Metrics post that. What? PM called me? Oh, that s for the User Stories. Going now. Guess what Monday s check-in broke Reporting. Team says need to cover more end-to-end if possible and plan on Regression testing each time. Hey, Dev checked in her piece she was supposed to 2 days back. Got to make the build and deploy it. Need a test box for the next 3 hours. Which one? Verifying fixes before they hit mainline. One of them needs unit tests cant test via UI. Writing them now. Don t know how I can get that large data set though!! What? The design of that feature just got changed? Can someone tell me why the build is not deploying? 6.00 pm Well I just managed to unit test the new pieces. 85

86 Best Practices 86

87 Best Practices Plan required infrastructure before Sprints start Develop in an Agile and Iterative fashion, revisiting ROI and Replanning for max. benefit. Set Smart Sprint goals for Automation Build in Test development tasks as Sprint tasks Test Data creation Test Procedure development Early automation of new acceptance tests 87

88 Best Practices Have QA members attend Customer Demos Factors to be considered at tests selection: ROI Time to automate vs Time to Test(manual) No of times the test will be executed Importance of the Test Procedure (Highly used scenario??) Prioritize by applying 80/20 Rule to testing 88

89 Best Practices Besides test cases, automate.. Builds Deployments Test Data Load Build qualification automation suite execution Set up a continuous tools evaluation/selection process a killer Automation team 89

90 Best Practices Besides test cases, automate.. Builds Deployments Evangelize the All Test Data Load existing scripts must pass before check-in Build qualification automation suite execution rule!! Set up a continuous tools evaluation/selection process a killer Automation team 90

91 What if we had this in place Fresh automated test scripts, Test Data - added continuously to Framework Perl, /java / junit, WR Build Qual Robot Build, Deploy and run Sanity tests All checkins must have corresponding Unit Tests All tests must pass at each run. Daily Local Branch Labels Build breaks or Test fails Send mail on results Fix and Check-in The one who breaks it, fixes it Freeze Test Hardware upfront Add Acc tests as you build to Framework. Early!! End-to-end system test part of AllTests Source Control Fix and Check-in Regular Performance test cycles along with Sprints Measure Code Coverage often Weekly Mainline Labels Send mail on results Build breaks or Test fails Build Qual Robot Build, Deploy and run Sanity tests Build, Deploy and run AllSuite Build, Deploy and run AllSuite Build, Deploy and run AllSuite Build, Deploy and run All tests Multiple O/S, DB, AS, WS, Datasets, Configs Test Metrics Auto sent to alias Perl, /java / junit, WR 91

92 A new look at Day in the life of a Tester in a Scrum team 92

93 9.00 am Analyze last night s test reports 9.30 am Attend SCRUM meeting. Get last night s build qual breakers assigned to appropriate owners for checkin by EOD. Get workaround for blocker applied to Test server. Am writing test plans for next week s PI. Test Metrics? In your Inbox. What? PM called me? Oh, that s for the User Stories. Going now. Guess what Last night s check-in broke Reporting. Caught at BuildQual. Plan to kick off All tests on Test Server. Dev checked in her piece she was supposed to 2 days back. Will be available on Test Server. Will test first thing tomorrow morning. Today, I am wrapping all of yesterday s checkins. Verifying fixes before they hit mainline. One of them needs unit tests cant test via UI. Writing them now. M has given me a tool to create that large data set. What? The design of that feature just got changed? Robot halted at Deploy. Assigned to K as P pm Well I managed to unit test the new pieces. And guess what new label creation just started. 93

94 9.00 am Analyze last night s test reports 9.30 am Attend SCRUM meeting. Get last night s build qual breakers assigned to appropriate owners for checkin by EOD. Get workaround for blocker applied to Test server. Am writing test plans for next week s PI. Test Metrics? In your Inbox. What? PM called me? Oh, that s for the User Stories. Going now. Lets go for lunch Guess what Last night s check-in broke Reporting. Caught at BuildQual. Plan to kick off All tests on Test Server. Dev checked in her piece she was supposed to 2 days back. Will be available on Test Server. Will test first thing tomorrow morning. Today, I am wrapping all of yesterday s checkins. Lets set up some time to train Tech Support! Verifying fixes before they hit mainline. One of them needs unit tests cant test via UI. Writing them now. M has given me a tool to create that large data set. What? The design of that feature just got changed? Impact? Regression? All tests Results will be out in 3 hours. For review at the SCRUM meeting tomorrow. See you at 9. Robot halted at Deploy. Assigned to K as P pm Well I managed to unit test the new pieces. And guess what new label creation just started. 94

95 Take Aways 95

96 Take Aways Test Automation is key to the success of software development projects using Agile methodology. Scalability and effectiveness of Agile Test teams depend on the level and maturity of Test automation available and the ongoing investment the team makes into this area. Strategic thinking towards Test infrastructure development, choice of tools and increase in Test Coverage, is the need of the hour. 96

97 References Books Agile Software Development with Scrum Ken Schwaber and Mike Beedle Agile Project Management with Scrum Ken Schwaber and Mike Beedle Agile Estimating and Planning Mike Cohn 97

98 References URLs velopment

99 Points to ponder over Data to prove efficiency? Statistics to prove better delivery? 99

100 100