Setting up a successful test automation project

Setting up a successful test automation project White Paper Author: Antony Edwards

Contents 1 Introduction 2 Project management 2.1 Project objectives 2.2 Project plan 3 Preparation tasks 3.1 Set up your test environment 3.2 Define your test architecture, conventions, and framework 3.3 Set up your relationships with other teams 3.4 Set up your test management 3.5 Set up your automated test execution 3.6 Set up your configuration management 3.7 Train the team 4 On-going activities 5 Appendix: Checklist 6 Appendix: Tools selection Page 2

1 Introduction Test automation can deliver huge benefits in terms of time-to-market, quality, productivity, and auditability to almost any team creating or deploying software. For example, TestPlant has worked with a leading global news publisher to reduce their app update cycle from three weeks to two days, a major UK bank to reduce post-release defects by 65%, and one of the world s top five retailers to double the number of apps they are delivering without increasing the size of their test team. These stories are not uncommon test automation really has delivered amazing benefits to lots of companies. But, there are also a large number of teams which have been trying to rollout test automation for years and not achieved any tangible benefits, often having invested significant effort. Why? TestPlant has worked closely with hundreds of companies to successfully deploy test automation; and has analysed many failed test automation efforts. Based on this experience we have identified two critical success factors that are common in successful efforts and missing in failed efforts. Successful test automation deployments are properly project managed. include day-to-day someone who has successfully deployed test automation before. These critical success factors are obvious when you read them, but the fact is that many (and possibly most) new test automation deployment efforts going on in the world today fail both these criteria, and will most likely not achieve their objectives as a result. This guide is all about helping you setup a successful test automation project. It is a guide, a template, and a checklist for defining a high-quality project plan for the deployment of test automation. Note that this guide assumes that a high-level decision has already been taken to deploy test automation has already, i.e. the project does not have to create a business case. It also assumes that a test automation tool has been selected. A future version of this guide may include these elements if requested so please provide feedback. Page 3

2 Project management Test automation deployments must be properly project managed. Any reasonable project management methodology can be used (e.g. Agile, PRINCE2, CCPM), as long as it is applied diligently. The following project management elements are essential: A Project Manager. A Project Manager with sufficient time to dedicate to the project, an understanding of test automation, an understanding of the product being tested, and enough technical knowledge to understand test environment issues. Objectives. Clearly defined outcome and output objectives (see below). A Project Plan. A clear and maintained project plan. Section 3 defines a set of standard preparation tasks that should be included in the project plan and Section 4 defines the standard on-going activities that should be included in the project plan. Regular Reviews. Regular and frequent project reviews. 2.1 Project objectives What do you want to accomplish with test automation? This sounds like a simple question, but is often difficult to quantify, yet is critical to a successful project. Test automation projects should have both output objectives and outcome objectives. Output objectives are the tangible outputs and activities of the project, e.g. a set of automated test scripts for a product s smoke tests. Outcome objectives are the value that these outputs will provide, e.g. a reduction of a test cycle from one week to two days. Output objectives are key to keeping a project focussed and ensuring delivery; outcome objectives are key to ensuring the project delivers value to the team. The following list presents best-practice for defining project objectives: Test automation can deliver huge benefits, but like any capability you need to build up, and you should avoid being too ambitious in your first project. It is strongly recommended that you run a pilot project with short timescales and valuable but realistic objectives, before diving into a larger project. Set your goals accordingly. Many new test automation deployment projects focus entirely on automating the existing manual test process. Manual test processes typically leave out effective test activities (e.g. regression testing or compatibility testing) because they are simply infeasible in a manual testing approach, but these activities are where test automation can deliver significant benefits. Conversely, there are some test activities where automation will not deliver significant benefits. So when you are defining your output objectives, consider which testing activities will contribute towards your outcome objectives, and then which activities could benefit from automation. 2.2 Project plan Regardless of the project management methodology the plan should include the project objectives, the scope of the project, the timeline and milestones, a list of resources (especially people) available to the project, and a list of tasks to deliver the scope. Page 4

3 Preparation tasks This section lists a set of critical preparation tasks that should be included in your project plan. 3.1 Set up your test environment An inadequate test environment is possibly the most common cause of wasted time and unreliable test results. It is essential that you design and set up a reliable test environment before you start significant test script creation and execution. Note that setting up a reliable test environment is not difficult if planned and resourced properly, but if it is done in an ad-hoc manner it usually causes problems. You should consider the following when setting up your test environment: Ensure all the necessary test systems are included in the environment. If you are testing mobile there are many references available on the Internet to help you define a representative set of devices for your target audience. Ensure there are enough test systems so that no-one is ever waiting. Consider manual testers, people creating and debugging automated test scripts, and automated test execution. It is very common to see manual testers accidentally break an automated test run by trying to use the test system at the same time. Ensure all testers will have access to the test environment. It is very common for a test environment not to be accessible from some test team locations; this wastes time as testers must travel and work away from their normal environment. Ensure all dependencies are in-place and reliable. For example, if you are testing a mobile application that connects to a server, you must have a reliable server to test against. Note that this includes all test data (e.g. test user accounts). Ensure the test environment is fully reliable before you start using it. It is extremely frustrating to investigate a failed test script only to find out that it is an environment issue. Finally ensure the test environment is fully change controlled. This is potentially the most important item because without change control even a perfectly setup test environment will most likely create problems as the project progresses. Without change control unexpected versions of the application-under-test will be installed on test systems, configurations will be changed, test systems will be removed, browser versions will be upgraded, OS versions will be upgraded, network cables will be unplugged and so on. Page 5

3.2 Define your test architecture, conventions, and framework Before you start creating automated test scripts for your test-cases you should define your test architecture, conventions, and framework (referred to simply as framework from here). The key purpose of the framework is to facilitate: Modularization and re-use of common functions (e.g. login). Testers being able to easily execute, review, and maintain tests written by other testers. Consistency of results reporting. There is a temptation at this point to try and define the set of common functions that will be needed and pre-implement these functions before testers start working on implementing scripts for test-cases. We recommend not to do this; our experience shows that many of the functions implemented are never used, many of the functions that are used have to be significantly re-factored to be used effectively from scripts implementing test-cases, and there is a de-motivational effect of putting in a lot of scripting effort and not having any automated test-cases to show for it. Instead we recommend that a clear approach to modularization is defined up-front and then this repository is built-up as needed, e.g. the first person to implement a test script for a test-case that requires a log in step should realise that this is likely to be a common function and add their implementation to the repository. 3.3 Set up your relationships with other teams A test team does not work in isolation. They work with the development team to get new versions of the application-undertest and raise defects. They most likely work with other teams such as the marketing team to understand requirements or the IT team to set up their test environment. It is important to set up your relationships with these other teams, meet the key people you will be working with, understand how they work, explain how you intend to work, understand the infrastructure each other are using, and discuss how to make the interfaces productive. For example, some questions you would ask the development team: Do they have one person who reviews and triages all defects or do defects go directly to the developer who worked on a feature? Do they have a release manager who is responsible for communicating all releases? What information do they need in a defect report? When a new defect is raised typically how long will it be until someone looks at it? If something isn t working in the product and it s really blocking the test team who is the best person to speak to? Page 6

3.4 Set up your test management If you want to use a test management tool such as qtest, Zephyr, TestLink, or HP ALM; this tool should be setup and configured before you start. 3.5 Set up your automated test execution You will almost certainly want to be executing test runs automatically over night or on check-in; and able to manually start a test run. You may do this via a test management tool, a tool focussed on test execution such as eggplant Manager, or a continuous integration tool such as Jira. Again this should be fully setup and configured before you start. This step may again seem obvious, but we have seen many projects waste a lot of time due to badly setup test execution. For example, test execution frameworks that are error-prone to configure and so end up executing tests against the wrong test devices, frameworks that do not save the logs needed to debug errors, frameworks that stop working at 2am due to back-ups starting, and frameworks that have no mechanism for a tester to manually start a test run and so changes can only be verified overnight. Note many people use test automation tools (e.g. eggplant Functional and eggplant Performance) for monitoring of live systems; i.e. they use the test scripts written during development and then use them for application-level monitoring of the live system. In such scenarios this step is even more important and the automated test execution must be carefully tested. 3.6 Set up your configuration management You should setup a configuration management tool (SCM) such as Git or Subversion to store your test scripts; and automated test execution should always use a defined set of scripts from the SCM. Using an SCM facilitates sharing, ensures all script changes are logged, and ensures all scripts are frequently backed up. A common scenario on test automation projects is for a test script that has been passing to suddenly start failing, everyone to say that they have not changed the script at all, but after several wasted hours to discover that someone clearly has made a quick unverified change to a script. An SCM means you can resolve such issues in minutes rather than hours. 3.7 Train the team Ensure all testers have been properly trained on the testing tools being used, the test environment, configuration management, the test architecture, and so on. Successful test automation teams usually also setup mechanisms for team members to share best-practice, useful resources (e.g. links to videos, example scripts), and other useful information. This is typically a mix of on-line mechanisms (e.g. a Wiki) and regular meetings. Page 7

4 On-going activities Section 3 listed a set of preparation tasks that should be included at the start of your project plan. This section lists the key on-going activities for the main implementation stage of your test automation project. The most important point of this section is that test automation projects are not just about writing new test scripts! There are several other crucial activities that must be resourced and managed properly. A common cause of project failure is to only allocate enough people to do the actual initial test script creation; this means that the other activities are done in a low quality way (or not at all), which can quickly undermine the whole project. So your project plan must consider all the on-going activities listed below. Test-case definition. Test-case definition and creating an automated test script to execute the test-case are separate tasks and should be managed separately. You should think about test-case definition as design and creating automated test-scripts as implementation. If you try to design and implement at the same time you are likely to waste time. The most successful test automation projects always define and review a test-case before a test script is created. Framework development. Your test framework needs to be maintained for the benefits of modularization and standardization to be realised. If you are using a mature test automation tool and setup a solid framework at the start of the project this should not be a lot of effort, but it needs to be done regularly. Test script development (first platform). The majority of the effort on the project should be dedicated to creating automated test scripts. Most automated test scripts these days must work across multiple platforms. Tools such as eggplant Functional provide a lot of support for easily porting automated test scripts across platforms, but some work is usually required, and this porting work is often not considered in the project plan. This is why we distinguish between first platform, i.e. creating the original automated test script for a test case for one platform, and other platforms (below) which is about ensuring this script works on other platforms. Test script development (other platforms). See above bullet point. Test execution, reviewing results, and raising defects. Someone in the team needs to ensure that overnight test runs (or whatever approach you have chosen) are setup correctly and they need to review the results. Reviewing results can be significant effort for applications-under-test that are changing frequently or low quality; or when a large number of new automated test scripts are being added to the environment. But diligent, methodical, review of test results is the key to achieving reliable test automation and getting value from the results. Test script maintenance. Existing test scripts must be updated from time-to-time as the underlying applicationunder-test changes, as the target test systems change (e.g. a new mobile device is added to the test environment), and as defects are found in test scripts. The level of test script maintenance effort required can be very different between different test automation tools and should be a key part of your tools choice evaluation. Environment management. Your test environment must be actively maintained; i.e. someone must ensure the environment remains clean, ensure enough test systems are available, and approve changes to the environment (e.g. new versions of the application-under-test). Project management. Too many test automation projects stop doing project management once the project is running. You must continue to monitor progress, resolve issues, mitigate risks, and ensure the project is going to meet its objectives. Page 8

Your project plan needs to ensure that all these activities are included and resourced. The effort required for each will depend on your objectives, application-under-test, existing assets, etc; but they all need to be considered and covered. Note: when reviewing this document TestPlant s Customer Success Managers all agreed that reviewing results (and acting on them) is the on-going activity most projects get wrong. If you aren t carefully reviewing your test results and acting on them then you will not get value out of test automation and you will not have a robust set of tests. As one example, a typical resource allocation for a project team is: Test-case definition 2 people (this assumes test-cases don t already exist). Test script development and test script maintenance 4-5 people. - 55% Test script development (first platform). - 15% Test script development (other platforms). - 20% Test script maintenance. - 5% framework development. Test execution and environment management 1 person. Project management 0.5 people. Page 9

5 Appendix: Checklist The following checklist should be used at the start of the implementation stage of a test automation deployment project to ensure it has been setup correctly. Planning: - Project manager assigned. - Objectives defined (business and technical). - Baseline project plan created. - Regular project review mechanisms defined and scheduled. Preparation tasks: - Test environment set up and tested. - Relationships with other teams established. - Test architecture, conventions, and framework defined. - Test management tool set up and configured. - Automated test execution framework set up. - Configuration management defined and infrastructure set up. - Team trained. On-going activities: - All on-going activities included in the baseline project plan test-case definition, framework development, test script development (first platform), test script development (other platforms), test execution and review, test script management, and environment management. Page 10

6 Appendix: Tools selection This appendix presents high-level guidance for selecting a test automation tool. Beware of being tool led. Define your objectives and needs and then select a tool. Technical Fit. Can the tool test all (or the vast majority) of your scenarios and environments (mobile, desktop, server); and is it future-proof? Productivity. Can your current team quickly and easily achieve automation and get benefits with the tool? Automation projects can quickly become long change projects. Ease-of-maintenance. Are scripts robust and quick to maintain? Integration. Can the tool plug-in to your other testing and development tools? Beware of showroom appeal. Many features look great in a demo (record-and-playback), but simply don t work in real environments. Page 11