Workshop on Agile Test Strategies and Experiences W8 Fran O'Hara, Insight Test Services, Ireland
Workshop Agile Test Strategies and Experiences Version 1.0 Fran O Hara Fran.ohara@insight-test.com Europe s leading independent specialist software QA, Test & Compliance full services provider www.insight-test.com Copyright Insight Test Services 2007 2
We help ensure compliance (e.g.business needs, Best Practice in Software Quality and Test, CMMI/TMMi, SOX, FDA, etc.) In a practical way we help Over 75 Organisations and growing using 90+ software QA & Test experts with Business and Regulatory expertise Test Control (Best Practice methodology) Full Life Cycle capability Business and technology expertise Financial Services Public Sector Information & Comms. Technology Life Sciences Transport/utilities Copyright Insight Test Services 2007 3
Helping organisations achieve their software quality, compliance and delivery goals. Consulting Process audits/assessments Process improvement Regulatory Compliance (SOX, Mifid, FDA, etc.) Test (automation) strategy TestControl our risk-based test methodology Vendor quality management Services Training & Competence Development - extensive range of topics including : CMMI, TMM(i), TPI ISEB / ISTQB Risk-based testing, UAT, etc. Managed Test Services On-site, near shore, far shore System (non-)funct., UAT Test automation services Specialist Resourcing Specialist skills, short/long term & Supplementary resources Copyright Insight Test Services 2007 4
Copyright Insight Test Services 2007 5
Workshop Objectives Identify typical challenges in agile development Present/discuss test strategies in agile environments Discuss approaches and experiences in resolving challenges and issues identified Copyright Insight Test Services 2007 6
Your Challenges Copyright Insight Test Services 2007 7
Some Typical Challenges 1/2 Testing issues around iterative/incremental development versus traditional development Hybrid approaches Role of the tester Levels/phases of test what does acceptance testing mean? Effectiveness of automated tests Copyright Insight Test Services 2007 8
Some typical challenges 2/2 System Testing When is it done? Who does it? What s delivered? Project Management Estimate, resources, costs, time, etc Change Management Documentation e.g. testing without detailed requirements Scale Multi-site teams and tester integration Copyright Insight Test Services 2007 9
Agenda Items Agile test overview Different Test Phases, agile terminology Where the different types of testing fit into an agile development lifecycle Role of the tester Test Driven Development (TDD), Automated Unit Testing Acceptance testing, testing and the customer/user Exploratory testing Copyright Insight Test Services 2007 10
Agile Testing - overview Copyright Insight Test Services 2007 11
Agile Values We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more. Copyright Insight Test Services 2007 12
Agile Methodologies extreme Programming (XP) SCRUM DSDM Crystal Lean Development (Manufacturing) Adaptive Software Development Feature Driven Development Context Driven Development Copyright Insight Test Services 2007 13
Remember there are no best practices context is key Copyright Insight Test Services 2007 14
Agile View of Traditional Development Cost Time Requirements Analysis Design Development System Test UAT Support Post Project Delivery of System - Stage at which defects cause most problems Copyright Insight Test Services 2007 15
Traditional Testing Static Requirements & Design Reviews Code Reviews Static Analysis of Code Dynamic E.g. Unit, Unit Integration, System, System Integration, UAT. Functional Non-functional Copyright Insight Test Services 2007 16
Early test design V-Model Requirements Acceptance test Functional Spec. System test Reviews Hi level design Integration test Static Analysis Static Testing Lo level design Code Unit test Dynamic Testing Copyright Insight Test Services 2007 17
Agile Development Cost Time A D Dev AT A D Dev AT A D Dev AT Deliver something of value to the customer Note the tradeoff (with iteration): Features versus time, not quality (and testing) versus time. Copyright Insight Test Services 2007 18
Focus Of Agile Testing High Value Features First Test Driven Development Automation of Unit Testing Continuous Integration Pair Programming Independent Reviews Independent Testing Automated Acceptance Testing Constant regression Exploratory Testing Multiple Builds Delivering Value Copyright Insight Test Services 2007 19
Common Issues in Agile Testing Attempting to substitute unit tests for acceptance tests Not automating the acceptance tests Thinking the automated tests are sufficient Forgetting about non-functional tests. Quality characteristics such as performance, reliability, usability scalability, memory usage, etc. See www.testdriven.com Copyright Insight Test Services 2007 20
Role of the Tester Iteration and Variable scope Have a huge impact on a test department! If development is iterative When do you test? What do you test? How do you test? If scope is variable What do you test? When do you write the tests? Copyright Insight Test Services 2007 21
Role of tester The nature of the tester's role changes in iterative projects. We are no longer the high-profile victims, we are no longer the lonely advocates of quality, we are merely (!) competent service providers, collaborating with a group that wants to achieve high quality. Cem Kaner Dedicated testers bring two benefits: Focus on customer usage over technical implementation Focus on uncovering flaws over confirming completeness (Bret Pettichord) Copyright Insight Test Services 2007 22
Role of tester Fully integrated into team Involved continuously from start e.g. Facilitate communication between the technical & business stakeholders Support early validation of requirements Help the business stakeholders define acceptance criteria Create automated acceptance tests Expand scope of acceptance tests Advise the team about overall risks and trends Perform manual/exploratory tests on early-stage code May need to develop scripting skills Copyright Insight Test Services 2007 23
Test Driven Development Never write a single line of code unless you have a failing automated test Eliminate duplication Kent Beck Write the test Write the code Refactor Copyright Insight Test Services 2007 24
Test Driven Development Can be applied at all levels of test e.g. Unit and Acceptance Preventative/early testing not new but many benefits Testing takes on a specification role not a verification role? A feature is not specified Until it s acceptance test is written. A feature is not done Until all it s acceptance tests pass. Acceptance and Unit tests become key requirements/feature and design artefacts Copyright Insight Test Services 2007 25
Unit level automation Automation Of Unit Test More likely to be done As development is done Structural coverage measurement Daily unit test regression providing stability with high level of change/iteration Need to engineer unit test code with same discipline as application code Coverage complacency & Happy path testing Copyright Insight Test Services 2007 26
Acceptance Testing Automated Acceptance Testing Design and Code for automation Automation Frameworks e.g. Exactor, Brian Swan & Sean Hanly http://www.exoftware.com/xp_tools.htm FIT, Ward Cunningham, http://fit.c2.com/ AXE, Odin Technology, http://www.odin.co.uk/ Issues Can be too low level or not business focused enough Thin UI layer issue Copyright Insight Test Services 2007 27
Exactor Example # Test zero initially StartCalculator CheckResultEquals 0 # Test addition Add 4 5 1 CheckResultEquals 10 # Test subtraction Subtract6 CheckResultEquals 4 Subtract 1 2 CheckResultEquals 1 # more tests... Based on Junit and JWebUnit Freely Available Tester defines Tests using Keywords Automation Eng writes code behind Keywords Copyright Insight Test Services 2007 28
FIT Example Executes Scripts scripted as tables in a html document Will interpret html differently depending on Fixture used See also FitNesse and FitLibrary Copyright Insight Test Services 2007 29
From: Home Brew test automation Bret Pettichord Copyright Insight Test Services 2007 30
Acceptance Testing is it enough? May not be context/risk/strategy issue May not be fully automated partial regression strategy needed Expand to fuller system tests Functional testing Non-functional testing performance, usability, etc. May still need end-to-end business scenario focused User Acceptance test System integration testing issues Etc. Strategy and scheduling issue Adaptive, risk-driven Copyright Insight Test Services 2007 31
Iteration testing From Bret Petticord challenges of agile testing Copyright Insight Test Services 2007 32
Exploratory testing Simultaneous test design, execution and learning Exploratory Testing Experienced-based technique Usually mixed with and complements documented test cases Creative Process Copyright Insight Test Services 2007 33
Exploratory testing agile programs are more subject to unintended consequences of choices simply because choices happen so much faster. This is where exploratory testing saves the day. Because the program always runs, it is always ready to be explored." Ward Cunningham on Why should agile teams do exploratory testing?: Copyright Insight Test Services 2007 34
Exploratory testing Can Use test automation to support early exploratory testing. Traditional test wisdom says we can t start testing a feature until it s accessible from an external interface (like a GUI). But we don t have to wait. Test automation can facilitate manual exploration. Copyright Insight Test Services 2007 35
Discussion & Questions Copyright Insight Test Services 2007 36