The Changing World of Test,
the User Is Automated With increased use of automation in ALM, the role of the tester is changing. In this series of tips, we look at how automation is changing the landscape of software test. This e-book will look at types of test automation including testdriven development and automated GUI testing. Not only has test changed, but the tester s role changes as organizations invest in a test automation strategy. Managers will get advice on what to look for when selecting automated test tools and will learn more about how they may want to invest their testing dollars in automation. for Agile Development By Nari Kannan, SearchSoftwareQuality.com Contributor When using Agile development methodologies, rapid iterations typically leave less time for testing, so automated testing tools take on a lot more importance. Agile methodologies also emphasize an iterative approach to arriving at precise requirements and design. Consequently, functionality and user interface designs may change rapidly. These two factors not only make automated testing tools absolutely necessary, but also require some very special capabilities that non-agile automated test tools need not have. Here are six tips for selecting automated test tools for Agile development: 1. Ensure support for evolving requirements Unlike non-agile methodologies, requirements themselves may be subject to change in Agile iterations. Automated test tools normally expect that requirements and designs are all frozen so that automated tests can be prepared for blackbox testing with specific inputs and expected outputs. When requirements themselves change, automated test tools need to support test scripts at a requirements level rather than at an individual input-output combination. Stories provide a convenient unit for testing at a requirements level. If some requirements are removed and other new ones Page 2 of 16
added, automated test tools need to have the capability of dealing with automated tests at the individual requirement or story level. the User Is Automated 2. Support evolving design An evolving design between two Agile iterations may completely change the nature of traditional automated test tools. A single dialog box in one iteration may be split into two related dialog boxes in the next. Automated test tools need to have support for reconfiguring tests designed for the first iteration into the ones needed for the second iteration rapidly, preferably automatically, or at least in a guided advisory mode. They should be capable of automatically flagging the first test as no longer valid and suggesting new tests to the script developer as much as possible. 3. Allow for rapid test script development Traditional automated testing tools are usually targeted for use at the end of a single, defined, lengthy coding phase in waterfall/sdlc development methodologies. So you have time to develop automated test scripts before they are used. However, in Agile methodologies, the time between iterations is short and time available for developing automated test scripts is usually short. So you need support for rapid development of test scripts in automated test tools for Agile development. 4. Consider modification speed of test scripts How quickly can you reconfigure automated test scripts from one iteration to the next? This becomes critical since Agile methodologies offer less time between iterations for reconfiguring automated test scripts as compared to non-agile ones. There could be a lag between iterations and when automated tests for that iteration are run. Automated tests for Iteration 2 may be run when Iteration 4 is ready, but that may defeat the whole purpose of iterations. The speed of reconfiguration of automated test scripts between iterations is of critical importance in Agile development. 5. Look for tools that support testing scenarios rather than specific inputs Scenarios are more akin to use cases than requirements. Support for automated testing scenarios may be much more useful when using Agile Page 3 of 16
the User Is Automated development methodologies rather than support for specified inputs and expected outputs as in traditional automated testing tools. This is because of both the potential for rapid change in designs between iterations as well as the time available for detailed scripting of automated tests between iterations. Many more automated tests for scenarios become reusable between iterations, so re-scripting between iterations becomes easier. 6. Include script versioning capabilities Script versioning capabilities are important in automated test tools for Agile development. Script developers may need to regress to an older iteration because stakeholders change their mind about a requirement or a design and decide to go back to a previous iteration. Script versioning capabilities may also come in handy to go back and reuse with some modifications or some of the older scripts if they are similar to new features being developed. Conclusion Agile software development poses evolving requirements and designs in successive iterations. Script development time between iterations in Agile development is short compared to other methodologies. Automated test tools become even more necessary, and they need to have many new and critical features to be useful when using Agile methodologies. Raising your awareness of these features helps you make the right choices to maximize the benefits of these tools. About the Author: Nari Kannan is the chief delivery officer of V-Soft Consulting Inc. During his 20-year career, Kannan has served as vice president of engineering or chief technology officer of six Silicon Valley startups, working in business process improvement, IT consulting, automotive claims processing, human resources and logistics applications. Page 4 of 16
By David W. Johnson, SearchSoftwareQuality.com Contributor the User Is Automated Test automation has always been an attractive alternative to expensive, time consuming and inconsistent manual testing. The most common questions when considering a test automation program are the following: When to invest in test automation? How to implement or proceed with a test automation program? How much of your manual testing effort should shift to test automation? We will address these questions in the context of the most common overall program factors. Using the information and considerations described, you ll have the data to help you best determine how to implement test automation in your organization. Key Program Factors The questions of when, how, and how much are always dependent on the context of the overall program What is the test automation target? Key program factors include: the development paradigm, the quality objectives, and your deployment velocity. When, how and how much test automation to apply against a program is dependent on these factors the return on investment must align with these factors; otherwise, the long-term success of the test automation effort will be in jeopardy and almost certainly fail. The most common key program factors are: Development paradigm (Agile, non-agile, instrumented, noninstrumented) Quality objectives (defect escape velocity) Target deployment velocity (volume of new/enhanced functionality per release) The focus of this article is on the implementation of a test automation program. It must be noted that test automation is not a silver bullet treating it as a cure-all will certainly lead to disappointment. Test automation is an Page 5 of 16
the User Is Automated enabler, so if you have an effective (though overburdened) testing process then an appropriate test automation program will yield significant returns for your organization. When to Automate A test automation program should be considered when key program factors indicate the overall development program is not meeting expectations and there are no cost-effective alternatives to test automation. The key indicators from a program objective would be the following: Quality objectives are not being met Defect escape velocity into production is deemed unacceptable Target deployment velocity is not being met because testing is perceived as a bottleneck is not being completed within the assigned timeframe These factors are often experienced by the project team at the same time basically the test group has reached its testing capacity. When this happens, the leadership team has two alternatives: grow the manual testing team or launch a test automation proof of concept to determine if automation can address the testing deficit. It should be noted that testing is often not the true bottleneck, but as long as it is perceived as the bottleneck other systemic quality issues will not be addressed. Test automation is one way to remove the perception of a testing bottleneck empowering the testing team while allowing the program to focus on other systemic quality issues for example, the initial quality of the code. How to Implement a Test Automation Program How to implement a test automation program breaks down into people, process and technology. From a people perspective, the testing group will need both training and internal expertise (consultant or new hire) in the art of test automation. Your people will need these things: Training on the test automation paradigm and tool being used Page 6 of 16
Internal expertise (consultant or new hire) to ensure the appropriate application of automation the User Is Automated From a process perspective, the testing group and the program as a whole will need to determine how test automation will fit into the system development process the initial focus should be on high-yield automation activities that free up manual testers (full-time equivalents). Once a sufficient number of resources are available, the focus can turn to high-risk areas of the application space that do not meet the overall quality objectives. Your process will need to do the following: Integrate test automation into the overall development process Focus on high-yield automation activities Test data creation Smoke test Regression test How and what test automation technology to apply is dependent on other key program factors, specifically the development paradigm being deployed and the target deployment velocity. If the development paradigm includes effective code instrumentation (self-testing code) or effective test-driven development, then a lighter technology that addresses the testing gaps from an outside-in perspective would probably be an appropriate fit. If the development approach does not include sufficient quality checks, then a heavier (tier 1 commercial) toolset would be a more appropriate fit you will need both the organizational and outside-in automation (GUI-based). In either case, if the desired deployment velocity is fast enough, a tier 1 commercial toolset will be required simply to avoid the framework investment required to enable freeware, open source, or tier 2 tool sets to reach the desired testing velocity. Your technology will need to fill the following functions: Address the quality gaps inherent in the development paradigm Lightweight test automation for self-testing development paradigms Page 7 of 16
the User Is Automated Heavyweight test automation for non-self-testing development paradigms Address the deployment velocity challenge For high deployment velocities heavyweight test automation tool sets For lower deployment velocities lightweight test automation tool set may fit Any test automation program should start with an honest assessment of the people, process and available technologies. Based on this assessment, and if justified, based on the responses to a request for proposal (RFP), one or more proof-of-concept initiatives should be launched to determine which process and technology best fits your short-term and long-term needs. Your team will need to do the following: Assess the people, process, technology and success of the current testing process Identify systemic root-cause issues preventing testing success Asses available processes and technologies (RFP) Launch one or more POC initiatives to verify appropriate fit How to Automate How much of your application space should be automated is dependent on the key program factors we have previously discussed. There are some basic guidelines that should always be considered: Never automate the entire testing process skilled exploratory testing will always yield defects that will not be detected through test automation. Test automation is a tool not a solution if it is not applied with intelligence and skill, the net result will be really fast ineffective testing. Test automation should always yield a return-on-investment of at least 4 to 1 based on the consumption of testing resources (fulltime equivalents). The weight of ongoing maintenance will decrease the return on investment over time unless the application becomes static. Page 8 of 16
the User Is Automated Moving beyond these basic guidelines, how much of the application space to automate is dependent on which key program factors are important to your organization. If the goal is to significantly increase deployment velocity or ensure quality objectives are being met then a significant percentage of existing functionality should be automated. On the other hand, if the objective is to reduce the weight of the overall testing effort on the test group, the focus should be on high-yield automation activities: Test data creation Smoke test Regression test In closing, test automation is not a silver-bullet -- it is a powerful test enabler when used appropriately. Test automation is not free. There is a significant cultural and monetary investment that needs to be made to ensure your test automation program is successful -- focus on those areas of the application that will yield the highest and most visible return. About the Author: David W. Johnson is a senior test architect with more than 25 years of experience in IT, He has played key roles in business analysis, software design, software development, testing, disaster recovery and postimplementation support. the User By Chris McMahon, SearchSoftwareQuality.com Contributor While software testers do a wide variety of work in their jobs, they are typically responsible for two critical functions: designing and implementing user interface (UI) level test automation and performing manual exploratory testing. Each activity informs the other, and when done well, each provides critical information to the whole software development team. Page 9 of 16
the User Is Automated Designing Automation There are typically three levels of test automation on a well-maintained software project. Unit tests cover the behavior of the smallest bits of code in order to enforce good design. Integration tests validate correct business function and business logic. UI tests ensure that the user is presented with the proper controls for the application. What this means is that automated UI tests should be very shallow. Such tests should not be concerned so much with business logic or the design of the system, but instead should validate that the interface presented to the user continues to contain the correct information and continues to function in the correct way. UI tests should check two things: First, UI tests should check that the information on the page is correct. These tests are of two forms. One sort of test says, "message X exists under these conditions." The other sort of test says, "message Y does not exist under these conditions." Automated tests like these are important because they confirm that the user continues to have access to information required to manipulate the application correctly. The negative case is even more important: exposing information in the application that should not be exposed could be a security issue, or even cause legal trouble. Second, UI tests should check that elements in the UI that cause a change of state continue to function. If the UI has a clickable link, then that link should take the user to the proper destination. If the UI has a Save button, then information entered in the UI should be saved properly. It is important when designing a UI test that the test takes the most efficient path possible through the UI in order to validate the existence or nonexistence of page elements, and to validate the correct changes of state in the application. UI tests are necessarily slow, which makes them expensive to run. Efficiency is key in a well-designed UI test suite. Page 10 of 16
the User Is Automated Minimize the extent to which the tests exercise individual UI elements as much as possible. When a fault in a single UI element causes many tests to fail, analyzing the failure becomes a painful process. Implementing Automation The needs of each individual team vary widely, but in general, a UI test suite should be run as often as possible, and results of the tests should be visible upon the conclusion of each test run. Any failures in the UI suite should be analyzed and addressed immediately. Anecdotal evidence suggests that many UI test efforts quickly become unmanageable due to bad design, or neglect over time, or both. Implementing well-designed UI tests such that they run as frequently as possible prevents this. Maintaining the UI tests with a minimum number of failures at any given time is a critical activity. Exploratory The great value of having a well-designed and well-implemented automated UI test suite is that it allows testers the freedom to spend more time examining the most interesting aspects of the application. The newest features and the newest code and the most complex areas of the application are the most likely to have issues for exploratory testers to discover. Having an automated UI test suite in place gives testers a significant amount of confidence that the parts of the application that they examined in the past continue to function properly. Being able to devote the energy of the test team solely to the ongoing work brings an enormous advantage in terms of improving quality. But there is still a close relationship between UI test automation and exploratory testing. That relationship is manifested in two ways: For one thing, exploratory testing informs the design of the automated tests. Without intimate firsthand knowledge of the operation of the UI, it is impossible to design automated tests well for the UI. So in the course of building automated tests, exploratory testing is required to view messages in the UI, to exercise elements Page 11 of 16
the User Is Automated in the UI, and to determine the most efficient paths for the automated tests to navigate. Also, every failing automated test should be a charter for some exploratory testing. Automated UI tests are only a very thin layer over a complex application. If an automated UI test fails for some reason, it is often an indication that a deeper problem exists that may only be discovered by exploratory testing from an actual person who knows and understands how the application should behave. Automated and Manual Together Having excellent UI test automation in place allows for excellent exploratory testing by intelligent testers. The automation is keeping an eye on the work already done while the testers are keeping an eye on the work in progress. And when an automated UI test fails, an intelligent tester is on the case immediately, analyzing the failure, investigating the possibility that the failed test indicates some deeper problem in the application. This is the best of all possible situations for a mature UI test effort, and it is worth the time and effort invested. About the Author: Chris McMahon is a software tester and former professional bass player. He has been part of the software testing community since 2004, writing about the industry and contributing to open source projects like Watir, Selenium and FreeBSD. Is Automated By David W. Johnson, SearchSoftwareQuality.com Contributor With the advent of mature test automation tools, both commercial and open source, there is a growing perception that traditional testers are no longer required by IT organizations. Given the correct set of circumstances, I would certainly agree that the role of testers is being transformed, and the number of testing resources required to address any given testing challenge can be decreased. There are several forces responsible for the changing role of testers, at least for on-shore testing resources. These include: Page 12 of 16
the User Is Automated Agile development techniques that integrate test automation into the development process. Mature test automation tools that simplify test creation, automation and test execution. In all cases the need for testing has not decreased, but the responsibility has shifted to either development or functional test automation. In all cases the test resource model for traditional testers is significantly reduced, if not completely removed. Test-Driven Development (TDD) Agile development processes that embrace self-testing code (instrumentation) significantly decrease the need for traditional testers. The need for testing does not go away, but as code instrumentation moves beyond simple unit testing into functional and even system testing, the need for additional testers is decreased. I would always recommend that a separate testing function test the application but the weight of this testing can be reduced, sometimes to the point of a simple user acceptance test. Under these circumstances, where do traditional testers fit into the organization? It really depends on the current (historical) role of the tester and the degree of code instrumentation. Let us look at the case where the code is fully instrumented and only user acceptance testing remains. Under these circumstances the role of manual tester usually disappears, as does traditional test automation. A typically experienced tester who has been fulfilling the role of informal business analyst or SME (system matter expert) moves into the role of exploratory tester within the Agile team basically ensuring the business need has been met and that the instrumentation fully tests the application space. Alternately, this person may take on the role of tester as business analyst or business user. This often occurs because the business is not willing, or not able, to participate as full members of the Agile team leaving a gap in the Agile process that can be filled by an appropriate testing resource. When the code is not fully instrumented, then the role of the tester becomes a balance between the traditional tester role and any new Agile Page 13 of 16
the User Is Automated development role. This can often increase the need for testers, obviously decreasing the value proposition of the Agile development process. The most effective way of mitigating this resource cost is to apply mature test automation tools and techniques to decrease the overall testing investment while increasing test coverage and overall testing velocity. Mature Test Automation Mature test automation solutions that include a fully integrated testing framework work as a force multiplier that decreases the need for traditional testers while increasing overall test velocity. This assumes the testing resources know how to effectively leverage the testing framework to harvest the maximum return. How and what test automation framework to apply is dependent on the development paradigm being deployed and the target deployment velocity. If the development paradigm includes effective code instrumentation or effective test-driven development (TDD), then a lighter testing framework that addresses the testing gaps from an outside-in perspective would probably be an appropriate fit as we stated earlier, if TDD is fulfilling the need for unit, functional and system test, then no additional testing framework is required. If the development approach does not include sufficient quality checks, then a heavier (tier 1) framework would be a more appropriate fit. In either case, the number of testers will decrease and the skills required to be a tester will change. The tester now becomes a test designer, a test automation engineer and a tool-smith (light-weight developer). Is Automation Replacing the Need for Testers? Automation is certainly changing both the role and skills required to fulfill the quality verification task. In some cases the role has moved to the development team (TDD) while in other cases the focus is on mechanizing the testing process (test automation frameworks). Regardless, testing is moving beyond the creation and execution of manual test cases to: A fully integrated testing solution within the development process developer as tester. Page 14 of 16
As separate automation implementation program tester as test designer, test engineer, test manager and as toolsmith. So, does automation replace the need for testers? No. Does automation change the role and skills required to fulfill the quality verification task? Yes. the User Is Automated Page 15 of 16
the User Is Automated Free resources for technology professionals TechTarget publishes targeted technology media that address your need for information and resources for researching products, developing strategy and making cost-effective purchase decisions. Our network of technology-specific Web sites gives you access to industry experts, independent content and analysis and the Web s largest library of vendor-provided white papers, webcasts, podcasts, videos, virtual trade shows, research reports and more drawing on the rich R&D resources of technology providers to address market trends, challenges and solutions. Our live events and virtual seminars give you access to vendor neutral, expert commentary and advice on the issues and challenges you face daily. Our social community IT Knowledge Exchange allows you to share real world information in real time with peers and experts. What makes TechTarget unique? TechTarget is squarely focused on the enterprise IT space. Our team of editors and network of industry experts provide the richest, most relevant content to IT professionals and management. We leverage the immediacy of the Web, the networking and face-to-face opportunities of events and virtual events, and the ability to interact with peers all to create compelling and actionable information for enterprise IT professionals across all industries and markets. Related TechTarget Websites Page 16 of 16