ICAgile Learning Roadmap Agile Testing Track

International Consortium for Agile ICAgile Learning Roadmap Agile Testing Track Learning Objectives

Licensing Information The work in this document was facilitated by the International Consortium for Agile (ICAgile) and done by the contribution of various Agile Experts and Practitioners. These learning objectives are intended to help the growing Agile community worldwide and as such this work is licensed under the following license. Creative Commons Attribution 4.0 International License http://creativecommons.org/licenses/by/4.0/legalcode YOU ARE FREE TO: Share copy and redistribute the material in any medium or format Adapt remix, transform, and build upon the material for any purpose. UNDER THE FOLLOWING TERMS: Attribution You must give appropriate credit to The International Consortium for Agile (ICAgile), provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. NOTICES: No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.

Special Thanks For the Agile Testing Track, ICAgile would like to give special thanks to the following track contributors. Janet Gregory Jeff Payne Sharon Robson

Learning Objectives 1. Agile Testing Mindset 1.1. Overview of Agile Testing 1.1.1. Origins of Agile Testing Many people who hear about Agile testing for the first time assume that it was created as part of the Agile movement. In actuality, much like Agile itself, many of the Agile testing techniques where espoused well before the Agile Manifesto was created. The purpose of this LO is to anchor the ideas of Agile testing in earlier work, giving the learners continuity from the past to the present. Debunk the myth that there is no testing in agile teams, as well as clarify that the testing is not "just" unit and customer testing. Short discussion on the good practices of development including testing - not excluding it. 1.1.2. Agile Testing vs. Traditional Approaches Agile testing is much different from testing performed during traditional software development approaches. The purpose of this LO is to provide insight into the major differences between Agile testing and testing performed as part of traditional (phased-based) software development approaches in which testing is primarily performed by software testers who are often in their own organization and sometimes only involved late in the development lifecycle. To be acceptable this learning objective must include a short group discussion detailing the benefits of clear acceptance criteria, frequent feedback loops, early testing. The value of a cross functional team must also be discussed. The group should be aware of the different communication models used in agile and the value of collaboration over documentation. 1.2. Mindset & Culture 1.2.1. Agile Testing Principles The Principles behind the Agile Manifesto establish guiding principles for not only the Agile movement but Agile testing as a discipline. The purpose of this LO is to help learners understand how the Agile Manifesto is realized within an Agile testing process and approach. To be acceptable, you only need to relate some of the principles to testing, not all 12. Specically, feedback and simplicity and collaboration 1.2.2. Whole Team Approach Quality is not owned by a particular role in Agile. It is a property of software that the entire team must make sure is present before software is released to customers. The purpose of this LO is to introduce the learner to the concept that quality is everyone s responsibility during Agile projects and everyone is involved in software testing. Testers are often ideally suited to guide the team toward achieving its quality goals and its definition of done based on team definitions of the quality attributes for the product. The role each team member typically plays for quality will be discussed. To be acceptable, this should include a discussion around whole-team quality concepts plus how "done" is typically defined for stories, iterations, releases. discussed quality roles across team members. 2

1.2.3. Building Quality In The role of a tester shifts in Agile from that of quality gate keeper to a facilitator who supports the team through asking questions to help clarify understanding, as well as testing and critiquing the product. The purpose of this LO is to shift the mindset of testers from that of an independent group responsible for gating the development process to a collaborative team member focused on improving the product and releasing value to the customer. To be acceptable, this should include a discussion of how testers collaborate with other team members through the entire software development life-cycle (planning, implementation, release, maintenance). 1.2.4. Continuous Improvement and Feedback Agile testing provides critical insights and feedback into the software process that can be used to drive team and quality improvements and assist the organization in making informed business decisions regarding software release. The purpose of this LO is to emphasize that Agile testing is a critical feedback component when seeking to improve operational effectiveness. To be acceptable, this LO should include the concept that the automation of testing is a critical component in providing real-time feedback. Other examples for providing feedback should be included such as retrospectives, pairing. 1.2.5. Ingraining The Agile Testing Mindset (Hands-on Exercise) Practicing behaviors can solidify the mindset and culture of Agile testing. The purpose of this LO is to have the learner experience situations in which the Agile testing mindset is likely to be different, so the learner can internalize the difference experientially, not just in concept. To be acceptable, this LO should provide exercises that give the learner a hands-on experience to the agile testing mindset. 2. Testing Techniques 2.1. Categories of Testing 2.1.1. Agile Testing Quadrants or Categories Testing activities can be broken into various categories of testing based upon their purpose and value. Types of testing are often broken into categories that include: testing that supports project team development efforts, testing that looks at quality from a business perspective, testing that critiques the product, and testing that exercises the relationship between software and its deployment platform. The purpose of this LO is to provide the learner with a sound understanding of the purpose of various testing techniques so they can be applied appropriately and at the right time within an Agile environment. To be acceptable, this LO should provide the learner not only with a model to categorize the testing but also with an understanding of how to map the model (quadrants or categories) to their real-life tests. 2.1.2. Automation Pyramid - Introduction 3

Automated testing can be performed at various levels within a software application. An automation pyramid or structure describes these various levels and discusses the approach and likelihood of automating tests within each of them. The purpose of this LO is to provide the learner with a basic understanding of the various types of testing that can be automated and how decisions get made regarding what to automate during an Agile project. To be acceptable, this LO should include TDD's role in providing quick feedback cycles at the unit level, testing below the UI at the service / API level, and testing well as testing workflows through the UI, and thier roles in regression testing. All 3 levels should be addressed. 2.1.3. Testing Techniques Test design techniques are necessary in an agile environment, they need to incorporate existing techniques and extend, and apply them to the collaborative methods. Introduce some simple ideas about approaching test design; For example mind-mapping, context diagrams; collaborative methods that may work in a collaborative environment. To be acceptable, this LO should include a hands on exercise using one of these practices to demonstrate how collaboration can enhance test knowledge. 2.2. Collaborating with Developers 2.2.1. Unit and Component Testing Developer testing of individual software units and associated components is critical to detecting implementation defects within software. Unit and component tests are leveraged within TDD as well. The purpose of this LO is to thoroughly understand the purpose and approach to successfully implementing unit and component testing on Agile projects and discussing how testers support developer testing during development cycles. 2.2.2. Pairing between the Developer and Tester Additional testing techniques beyond unit/component testing are often necessary to test beneath the UI level. The purpose of this LO is to provide the learner with a sound understanding how developers and testers pair during fixture / test method development and API-level testing. To be acceptable, this LO should discuss how testers effectively support test fixture / method development performed by a developer. 2.3. Example Driven Development 2.3.1. Acceptance Test-driven Development (ATDD) ATDD is a common technique for assuring that Stories are implemented in a manner that satisfies the exit criteria defined for Story completion. It is often used as a technique to test Stories but in actuality includes the testing of key business processes and non-functional requirements as well. The purpose of this LO is to thoroughly understand the purpose and approach to successfully implementing Acceptance Test-driven Development (ATDD) on Agile projects. 4

To be acceptable, this LO should include hands-on exercises that enable deep understanding of ATDD at the product level (holistic), feature level, and at the story level. The LO should discuss when and how to generate and record the acceptance criteria through the life-cycle. 2.3.2. Behavior-Driven Development (BDD) BDD is an alternative approach to ATDD that is sometimes used to test Stories, Business Process, and non-functional Requirements based upon an understanding of user behavior. The purpose of this LO is to thoroughly understand the purpose and approach to successfully implementing Behavior-driven Development on Agile projects. To be acceptable, this LO should include hands-on exercises using the "given/when/then" that enable deep understanding of BDD providing main and alternate flows and applying test design techniques to expand existing criteria to include other behavioral traits of the product not previously considered. 2.3.3. Spec by Example Specification by example uses the 3 amigos idea in a workshop environment to express examples. To introduce the terminology and how it is different / same as ATDD and BDD. To be acceptable this LO should relate Specification by Example to both ATDD and BDD highlighting the value of real data-sets, real examples and tangible outcomes in story definitions. 2.4. Feature and Story Testing 2.4.1. User Story Testing Testing of User Stories is critical to successful development of software within an Agile project. This testing is often performed using the techniques above but can be done in other ways as is appropriate or necessary. The purpose of this LO is to thoroughly understand how User Stories are tested during software development; this is an extension to ATDD to include boundary conditions, and other types of testing such as exploratory testing. To be acceptable, this LO should include discussion about user stories and what makes testing complete. It should be emphasized that story testing is fundamental in all testing aspects. 2.4.2. Feature Testing While the above techniques are the most common, there are a variety of other testing techniques that can be applied to test software features. The purpose of this LO is to make sure the learner sees the bigger picture beyond a user story (work-flows, end-to-end scenarios). To be acceptable, this LO should include a discussion and example that enable deep understanding of other feature testing techniques. 2.4.3. Exploratory Testing Exploratory testing provides a mechanism for additional testing to be performed on Stories or Business Processes based upon a tester s intuition and knowledge about the product. 5

The purpose of this LO is to provide the learner with a sound understanding of Exploratory Testing techniques and approaches and how they are best applied to an Agile project. Introduce the idea that automation can support ET. To be acceptable, this LO should include hands-on exercise or a discussion around a specific exploratory technique. 2.4.4. Non-functional testing Non-functional, or Quadrant 4 tests are sometimes ignored by customers. There is no clear understanding of who is responsible for it. The purpose of this LO is to demonstrate how to include testing these quality (non-functional) attributes in our testing. To be acceptable, this LO should include a discussion and examples about how to include constraints in discussions about story and feature acceptance. 3. Agile Testing Process 3.1. Roles and Responsibilities 3.1.1. Team-Based Testing Approach Testing during an Agile project is team-oriented wherein it is common for every member of the team to provide some level of testing support. that within an Agile project, the entire project team is responsible for test plans, test design, test cases, test automation, and test reporting. To be acceptable a discussion on large organizations and dependencies between teams should be included 3.1.2. Typical Business Representative Role in Testing Business Representatives typically provide guidance on acceptability and provide examples of what Stories are intended to accomplish. of the common test activities that a business representative is involved with during an Agile project. 3.1.3. Typical Programmer Role in Testing Software programmers typically build, automate, and run a variety of tests at a variety of levels as part of their development process. TDD and ATDD leverages this testing to improve design and development. of the role software programmers play in agile testing. 3.1.4. Typical Tester Role in Testing Software testers typically work hand-in-hand with the product owner and programmers to plan, execute, and report on the testing that is performed at all levels. of the role software testers play in Agile testing. Skills should include technical awareness; T-shaped skill-set. To be acceptable, this LO should cover the concept that Testers often are responsible for helping create User Story and business process tests and performing exploratory testing; and that Testers participate in and may develop automated tests along with programmers or a dedicated test automation team. 6

3.1.5. Role of Test Managers in Agile When an organization has decided to organize around products or projects, test managers are often left to wonder what their role within this new structure will be. of the new role that a test manager plays, and the reason behind the shift. For example, mentoring test communities of practice and helping coordinate post-development testing activities. 3.2. Test Strategy and Planning 3.2.1. Different strategies based on levels of precision Lightweight planning is typically part of the release planning done prior to associated iterations. of how lightweight test strategy and planning is performed during release planning and how decisions are made regarding what type of test documentation is needed and how much is enough. To be acceptable, this LO should include a discussion and optionally a hands-on exercise that enable deep understanding of the release planning done prior to associated iterations. 3.2.2. During Iteration Planning/Kickoff Test planning at iteration kickoff focuses on detailing acceptance criteria and examples for Stories. of how tests are developed prior to implementation. To be acceptable, this LO should include hands-on exercises that enable deep understanding of the test planning done at iteration kickoff when exercises are required. 3.2.3. Lightweight Test Plan Documentation Test planning in Agile is different from traditional development approaches as the goal is to provide the least amount of documentation needed to get the job done. of how to determine the amount of test documentation necessary for a given environment or situation. 3.2.4. Defect Tracking and Management The amount of defect tracking that is performed during an Agile project depends upon what works best for the team. The purpose of this LO is to describe the key trade-offs for determining which defects to track and which to rely upon team communication to correct without tracking. To be acceptable, this LO should include a discussion about how defects are managed in an agile project. 3.2.5. Results Reporting Test reporting during Agile projects depends upon what works best for the team. The purpose of this LO is to describe the key trade-offs between documented test results and team communication of those results. 7

3.2.6. Test Metrics Metrics collected to support test completeness and release readiness decisions. Example: determining whether a story, feature or/ iteration is "Done". The purpose of this LO is to inform the learner as to which metrics make sense to collect and report on for both test completeness and release readiness within an Agile project. 3.2.7. Regression Tests Automated regression tests are essential to reducing the cost of change and providing real-time feedback during the development process. of how to best leverage tests that have been automated during development within future iterations and releases. To be acceptable, this LO should have a discussion about why most regressions tests should be automated, and what the trade-off are between automated and manual regression tests. 3.3. Successful Delivery 3.3.1. Time-boxed Delivery Time-boxed delivery means that at the end of every iteration, there is a potentially shippable product, including testing. The purpose of this LO is to emphasize that all work on a story needs to be completed within the iteration and time must be allocated by the team to allow that to happen. Testing and coding should be planned for concurrency not sequential work. Testing and coding are to be considered one process not two steps. To be acceptable, this LO should cover the avoidance of mini-waterfalls inside and outside of the iteration. The definition of done must be wholly completed for a unit of work to be completed, including all aspects of the work and verification of the work that was defined as in scope by the team for the story level verification. 3.3.2. Continuous Delivery Continuous delivery means that every build has the potential to be deployed to production. The purpose of the LO is to understand what this means to testing. Inclusion of automation, responsiveness to defects and change. To understand how aspects of testing that enable continuous delivery in a team are captured and incorporated into the work flow. 3.3.3. Post-development test cycles In agile, testing is as early as possible for fast feedback, but during the endgame, there is often final testing that needs to happen. The purpose of this LO is to describe the interaction of testers in postdevelopment test cycles. This includes integration test teams that take advantage of economies of scale for security testing, browser compatibility, interoperability, etc. in an integrated environment. It also includes testing in the End Game for final UAT, or final performance testing.. To be acceptable, this LO should include a discussion about what testing happens in the deployment or "end-game" stages of the work. Should include environments, test types, and test teams. 3.3.4. Iteration Wrap-Up 8

Wrap-up activities during an iteration include a product demo, retrospective, and sometimes a User Acceptance Test. of the role that software testers play during iteration wrap-up activities. To be acceptable, this LO should include a discussion of how to bring up testing problems in a retrospective. Also this LO should provide a list of common iteration problems and issues from a testers perspective. We would like the learner to experience examples of how to articulate and solve the problems testing problems with the team. 3.3.5. Definition of a Release/End Game A release process (aka end game ) is performed whenever a decision has been made by the business to release software to customer(s). of how a release decision is made and what testing activities are typically part of the release process. To be acceptable, this LO should include a discussion about the difference between a project retrospective and an iteration retrospective. Include a typical Release check-list of activities. The end game is not a stabilization activity it is a concluding activity. 3.3.6. User Acceptance Testing (UAT) User Acceptance Testing is used within Agile to gain customer feedback on a working piece of software before its release. The purpose of this LO is to provide the learner with a sound understanding of user acceptance testing (UAT) techniques and approaches and how they are best applied to an Agile project. To be acceptable, this LO should include a discussion that this is to be done as early and often as possible to enable short feedback loops. This should also be seen as on opportunity to capture change and feedback for future work. 3.3.7. System-wide and Cross-team Testing The focus needs to be on products and solutions not on projects. Therefore testing needs to maintain the big picture that spans beyond the individual project and crosses over to multiple teams or projects to a more system-wide view of testing of how to coordinate testing across agile projects and associated testing activities when the project must interact/interface with other projects and IT technology release schedules. 3.3.8. Post-Release Testing Testing after software release typically consists of testing hot fixes for critical defects identified in the field and on-going testing of bug fixes not fixed prior to release. of the types of testing that is performed post release and how continuous testing supports a continuous delivery process. 3.3.9. Documentation for Regulatory Requirements Regulatory compliance often demands particular documentation be developed as part of any software development process. 9

of how regulatory requirements can still be fulfilled when following an Agile development process. 3.4. Test Environments and Infrastructure 3.4.1. Typical Environments for Test Multiple environments are often necessary to support testing activities during iterations and the release process. of the typical test environments that must be set up and maintained to support testing activities during iterations and releases. This is to include information about the tool and data set ups as well. To be acceptable, this LO should include discussion and typical definitions of stable test environments with a known configurations. Each type of testing needs a defined environment to match the purpose of the tests being run. 3.4.2. Build Pipeline A build (or deployment) pipeline is a way to deal with automated tests, by breaking the build into stages. Each stage provides increasing level of confidence with different sets of tests running at each stage. The purpose of this LO is to understand how the build pipeline works to help determine which testing is appropriate for which test environment. What needs to happen before the build is deployed to the next stage. To be acceptable, this LO should discuss the product maturity as it moves through the levels of automation that can be applied in the automation of the testing. 3.4.3. Automated Builds Incremental automated builds are essential for reducing the cost of change and providing rapid feedback on quality to the development team. of how automated builds are set up to support a continuous testing process 3.4.4. Testing the Proper Build As builds are constantly being generated during an Agile process, testing the proper build is critical to an effective testing process. The purpose of this LO is to discuss the best practices associated with choosing a build for test and keeping development and testing in sync during the process. 3.4.5. Test Data Management Effective test data management is essential to all aspects of Agile testing as the ability to select appropriate test data, set this data up, perform testing upon it, and reset any resulting changes is critical to an effective testing process. The purpose of this LO is to discuss the effective ways to generate and manage test data during an Agile process. To be acceptable, this LO is to discuss traceability and techniques for building the relationship between tests, data, environments and features. 3.5. Working on Distributed Teams 3.5.1. Distributed Team Communication 10

Distributed teams are a fact of life in most organizations and must be dealt with to make Agile testing initiatives successful. of how communication can be most effective on distributed teams. To be acceptable, this LO needs to discuss common problems with distance based team. Examples of how tests can become the common language of the team, bridging ambiguity and confirming understanding no matter location or time zone. 3.5.2. Distributed Team Coordination Distributed teams are a fact of life in most organizations and must be dealt with to make Agile testing initiative successful. of how testing activities can be coordinated when the team is distributed. To be acceptable, this LO needs to discuss coordination, liaison overheads, hand over methods for the engagement of the testing activities within the iteration. 4. Test Automation 4.1. Test Automation Strategy 4.1.1. Automation Pyramid Automated testing can be performed at various levels within a software application. An automation pyramid or structure describes these various levels and discusses the approach and likelihood of automating tests within each of them. The purpose of this LO is to provide the learner with a comprehensive understanding of the various types of testing that can be automated and how decisions get made regarding what to automate during an Agile project. 4.1.2. Planning for Automation Defining the approach, tools and timings for automation through the project. The purpose of this LO is to provide the learner with general knowledge regarding how to plan out an Agile test automation effort for each release as part of the test strategy. It also includes how to isolate parts of their system to be able to automate effectively. To be acceptable, this LO should include an exercise on how to illustrate a system, and identify how to isolate modules for automation (i.e.. mocking). 4.1.3. Automation Frameworks Frameworks provide test infrastructure for automating various types and levels of tests. The purpose of this LO is to provide the learner with general knowledge regarding various types of test automation frameworks so they can effectively choose which frameworks make sense for their particular application based on testing requirements and time-lines. 4.1.4. Selecting Tests for Automation It is typically infeasible because of cost and time to automate all tests that are created and/or run. 11

of how to decide which tests that get created and/or run during an Agile project should be automated vs. tested manually. 4.1.5. Supporting Process Test automation is performed at various points during Agile project iterations and release cycles. When test automation is performed and for what purpose must be understood. The purpose of this LO is to the learner with an understanding of when it makes sense to automate tests during development iterations and release cycles. 4.2. Testing and Continuous Integration 4.2.1. Automated Test Cycles (Continuous Testing) Integrating automated testing into a build environment assures that software changes are tested early and often during the development process. The purpose of this LO is to provide the learner with tips and techniques for integrating automated tests into an incremental build process such that software is validated during the entire development process. 4.2.2. Code Analysis/Metrics Code analysis and quality metrics can provide additional insights into an applications quality and release readiness of the software. of code analysis and code metrics for measuring the quality of software applications. 4.3. Automating Story and Feature Testing 4.3.1. Mapping Tests to Automation When automating tests, the bigger picture is often lost because we concentrate of various levels of testing. We need to take the tests needed for all levels and map them to the overall automation strategy. The purpose of this LO is to understand how to map feature and story tests to all levels of automation. To be acceptable this LO should include an exercise on mapping feature and story tests to all levels of automation. 4.3.2. ATDD and BDD Testing Frameworks Frameworks exist that provide support for developing and running automated story and other types of feature tests during software iterations. The purpose of this LO is to give the learner hands-on experience with a commonly used story or feature test framework to further their understanding of how such frameworks are used to support development and testing activities. To be acceptable, this LO should include hands-on exercises (pseudocode or scripting) that enable deep understanding of a commonly used frameworks at the story, API, or service level. 4.3.3. UI Testing Frameworks Tools exist for exercising software through its user interface to test features and combinations of features. The purpose of this LO is to introduce the learner to commonly used UI test tool to further their understanding of how testing can be performed through software s user interface. 12

To be acceptable, this LO should include either a hands-on experience or a demo to illustrate how and when to use testing through the user interface. 4.4. Automation Support for Integration and System Testing 4.4.1. Data Setup and Tear down Effective test automation often includes automating the manual processes associated with setting up and resetting test data. of how test automation can be used to automate the setup and tear down of test data sets. To be acceptable, this LO should introduce multiple ways of how to setup and tear-down data (like using a database, using flat files, setting up data within the tests themselves, etc.) 4.4.2. Data within Automation To have stable test automation, you need controlled data. The purpose of this LO is to give learners insights into different ways of controlling the data so that the automation results are consistent. 4.4.3. Tools to Support Exploratory Testing While Exploratory Testing is inherently a manual testing process, tools can be leveraged to assist in the testing process. of how tools can assist an Exploratory Testing process. To acceptable, the LO should demonstrate tools for recording of results as well as (re)introduce tools to help with exploratory testing such as using Log files to see error messages, automation for data set up. 4.4.4. Tools for Performing Non-Functional Testing Automation tools exist to support non-functional testing like load testing, performance testing, and security testing. of tools that are available for testing non-functional requirements including load, performance, and security. 4.4.5. Virtualization Virtualization provides a mechanism (often automated) to support effective test environment setup, test execution, and test environment tear down during a testing process. The purpose of this LO is to describe how virtualization can support an automated, effective testing process. 13