MERCURY CUSTOMER PERSPECTIVE WHITE PAPER: HOW TO AUTOMATE TESTING WITHOUT AGGRAVATING TESTERS
ABOUT THE AUTHOR Michael Anthony is manager of Corporate Systems Quality Assurance for Iron Mountain, a leading provider of outsourced records and information management services. He and his team of analysts at the company s Boston offices provide QA services for the company s corporate-level products, including Oracle ebusiness Suite (HR, real estate, etc.) and Siebel (sales and marketing), and are also responsible for automating proprietary systems for the company s Boston offices. He has been with Iron Mountain for 10 years. TABLE OF CONTENTS About the Author 2 The Right Tools for the Task: Mercury Quality Center 9 Introduction 3 Summary 11 You Know it s Time to Automate Testing When 3 About Iron Mountain 11 Integrating Automation into Your Testing Process: Recommendations and Advice 5 Roles, Responsibilities, and Training 5 Integrating Test Automation within Product Lines 7 Proving the Value to Internal Customers, Application Users, and Management 8
Introduction Your company has made the decision to automate software testing. Now it s up to you to make sure it s done right. That means you ve got to get your team to accept it. To embrace it. To excel at it. To integrate it into their normal routine. To make sure it delivers all the expected benefits: time savings, cost savings, higher efficiency, better quality software, etc. So what s your plan? Better start with the accept it part. Automation is all about speed, but unless your team is behind the move to automated testing, you won t be getting anywhere fast. This paper gives you some practical advice, based on my own experiences at Iron Mountain and lessons learned by my colleagues in other companies, about how to integrate automation into your testing processes without alienating your testing team. We ll examine how to divide up the tasks and responsibilities, how to get training and assistance from vendors, how to integrate automation testing within product lines, and how to prove the benefits of automation not only to your team but to internal IT customers, application users, and management. With the right tools, the right training, and the right attitude, your staff will quickly see the value of automation and transform the hypothetical advantages of automation into bottom-line results for your business. Now that s something management can relate to. And that s good for everyone s outlook. You Know it s Time to Automate Testing When In many cases the decision about whether or not to automate is easy management has already decided to do it, and left it up to you to determine how best to go about it. In other cases, the QA team may have input into the decision. Either way, it s a good idea to understand what test automation can and can t do for your organization. Most often, companies decide to automate because manual testing simply takes too long. Testers and developers have to tediously document each step of a test case and then manually execute each test as well as reproduce defects, eating up time and resources. WWW.MERCURY.COM 3
But that s only part of the story. Today you also have to deal with coverage creep. The proliferation of applications, operating systems, client devices, business processes, and data sets wreaks havoc on manual testing processes. There s an explosion in the numbers of test cases that need to be executed to verify the functionality of an application. You just can t test 100 percent of anything anymore. And that means there s also higher risk. People get tired; they make mistakes entering data; they don t always code the test correctly; and they don t always test everything that should be tested. Will automation solve everything? Of course not. But it can deliver some key advantages. The most obvious benefit: Computers are just one heck of a lot faster than humans at executing test scripts. And unlike humans, they can work 24 hours a day. They don t get bored. And they don t make assumptions about what works and what doesn t. In addition, automated testing is completely flexible and repeatable. Most automated testing products out there can emulate any mix of transactions and any user workload very quickly, and they can give developers an easier way to replicate and document software defects. And by introducing automated testing, you force the test team to formalize its processes, so you get higher consistency and better documentation. It s actually pretty easy to determine whether your company should automate its testing processes. Consider automation if your company has: Applications that require multiple builds/patches/fixes. Applications that need to be tested on multiple hardware and software configurations. Applications that support multiple concurrent users. A high percentage of development costs allocated to finding and fixing bugs. Problems with applications that result in constant development patching. Frequent rollbacks of applications from the production environment. High levels of frustration/complaints from application end users or customers. Inconsistent use of tools and processes across organizations, leading to inconsistent quality levels in production applications. WWW.MERCURY.COM 4
Integrating Automation into Your Testing Process: Recommendations and Advice The single biggest challenge in making automated testing a routine part of life is inertia. It s new and different; it will require some training and some getting used to. The test team may see the long-term value, but may view it as one more thing that will eat up time in the short run. End users of your applications may worry that they ll be adversely impacted somehow. Management may be concerned about the ramp-up time and cost. This section offers a bit of guidance for overcoming the inertia and transforming the angst into dare I say it? enthusiasm. Roles, Responsibilities, and Training The first step we took at Iron Mountain was to define what we were going to automate, and analyze what the roles and responsibilities would be once we applied automated processes. That in turn helped us determine our training requirements. In general, it makes sense to focus your automation efforts first on critical business processes, complex applications, and the use cases that comprise them (as opposed to lower-level tasks such as system-level verifications). Discussions with management and application end users will help guide those decisions. You may also need to take into account the automation tools that management has authorized you to purchase. Most companies today have a mix of web-based and non-web-based applications, and some tools are better at automating one type of application or the other. Once you ve made some preliminary decisions about what to automate and in what priority, you can see how that will impact current roles and responsibilities. You ll need to document how your work is currently divided up: By application? By function? By application vendor (i.e., Oracle, Siebel, etc.)? By business process (i.e., accounts receivable, accounts payable)? Then you can compare your current organization with the workloads you ll have with automation, and you can clearly identify how individual roles and responsibilities will change, and what training will be required of whom. WWW.MERCURY.COM 5
This is where it pays for you to know your team. Some of your QA analysts will have an excellent attitude and aptitude for learning how to automate; others will see it as an imposition; still others may even see it as a personal affront or an indictment of the way they ve always done things. There are several things you can do to smooth any ruffled feathers. First, be clear upfront about why your organization is making this transition. Emphasize the opportunities the change creates as opposed to the problems it solves both for the business and for the team. For example, stress the fact that automation means less time doing tedious, repetitive tasks, and more time focusing on other aspects of delivering high-quality software. Also, highlight the fact that the move to automated process can be useful for career advancement. It broadens their skill set, thereby putting them in line for better internal reviews and, potentially, higher pay or benefits. As an added incentive, you may want to cultivate one or more members of your team as an in-house expert or guru, further increasing the benefits to his/her career and stature. Team members could specialize in specific tools, such as Mercury WinRunner, Mercury LoadRunner, or Mercury QuickTest Professional, and work closely with the vendor to answer staff questions and resolve issues. Inevitably, certain individuals may lack the aptitude or the desire to learn automated processes. Some companies take a learn or leave attitude in these cases; I prefer the approach of reassigning the person to another area of responsibility. There s still plenty of other work to do and people do tend to come around over time. In terms of training, two things are critically important: First, make sure to get help from your vendor in developing the training plan and ramping up the training sessions. They re very experienced and they will know what works and what doesn t. In our case, we also supplemented the training provided by Mercury with additional training from a consultant who helped us define a method for transferring from manual processes to automated and who even provided an entire suite of standardized, canned code for basic functions. The payback was several times what we invested. Second, make sure to involve the staff in training as a group. This will help engender problem-sharing and collaboration, and will help boost morale by creating a sense that we re all in this together. WWW.MERCURY.COM 6
Integrating Test Automation within Product Lines Once you ve restructured your roles and responsibilities and executed your training plan, there s the small matter of actually implementing the new automated processes with your various applications. Again, documenting your current practices and requirements is an excellent first step. You need to start with company requirements for system upgrades; for example, you upgrade Oracle ebusiness Suite once a year, you upgrade Siebel quarterly, and so on. You can then identify the regression test requirements that go with each upgrade. In our case, we used to upgrade Oracle twice a year. Ninety percent of that is standard execution of standard test scripts, and with our previous manual processes all of that had to be done by the testing staff or even the accounting staff. There are more than 500 scripts for the Oracle processing alone; so the upgrade took two to three weeks, using four to five people, twice each year. With our automated processes, we now have that down to two people over eight working days, and none of the user staff is involved anymore. To identify the regression test requirements that come with the upgrades, we work closely with our developers. They present us with a list of what s new and different, and we work with them to understand the impact on existing test scripts or what s required in terms of writing new test scripts. QA then can do the automation prep, and once they run the automation tests we can often see quickly what development may have forgotten to tell us (functionality may have been left off, for example). We also work together very closely to understand the effect of production bug fixes through both scheduled weekly meetings or through non-scheduled/emergency meetings. The regular, recurring meeting to review production changes is extremely important. For example, if we made a bug fix to Oracle A/P last week, we need to ensure that we reviewed it to see if the fix would drive a change to an existing script, deletion of a test script, or creation of a new script. Then we add the new/revised test scripts to the master suite of regression tests for each application or product undergoing QA review. WWW.MERCURY.COM 7
It may also be a good idea to start small and roll out your automation strategy over time, proving the value on selected projects first and building the momentum as the value is proved to development, QA, end users, and management. In our case, we proved the value of automation with Oracle, then brought the same processes to Siebel, and after that we didn t have any more convincing to do. Proving the Value to Internal Customers, Application Users, and Management Demonstrating the success of automation is important to the QA team, to end users, and to management, and it s something that you should do early and often. At Iron Mountain, the proof takes many forms. Here are a few suggestions that may be useful at your company: Document the human hours saved. Labor costs are often the highest costs of testing much more than the cost of the tools so keep track of how long it takes to run scripts before and after automation. The savings here take two forms: first, the actual time savings due to automation multiplied by the approximate hourly cost of labor; but you should also consider the opportunity costs that are avoided through automation. For example, the accounting staff no longer has to participate in testing, so they can do other things. In our case, we achieved a 60-percent reduction in the use of full-time employees for testing. We found that a manual test script that one AP staff person executed in four hours could be executed under automated processes in 30 minutes (and that 30 minutes could happen at night, on weekends, or during holidays). Explain the value of consistency across applications and platforms during testing cycles. For example, after automation you may be able to demonstrate that you went from 40 bugs to 10 bugs during release. From the end-user perspective, that means higher application availability and better quality software. Show how costs can be capitalized. A wide range of development costs including new tools and testing labor can be capitalized when you make the transition from manual processes to automation, saving tax dollars. WWW.MERCURY.COM 8
Demonstrate higher efficiency. When you re spending less time on what was manual testing, you re free to do more in the same testing timeframe. So you can expand your testing coverage, improve the efficacy of your testing process, and get to market faster with better software. Show them the audit trail. With Mercury TestDirector, every test that s run automatically produces a log and audit trail, so you ve got good documentation every step of the way: which test was run, which day, what time, did it succeed or fail, where did it fail, etc. Mercury TestDirector also provides reporting, so if you re audited or you need to report to non-technical management you can provide documentation that s clear and easy to understand. The detailed, fine-grained information Mercury TestDirector provides also takes a lot of the subjectivity out of the equation. Now QA can work with developers and show them exactly what s not working, on what platform, and under what conditions. That helps developers understand what needs to be fixed, and it helps engender a more collaborative working relationship. It helps bring down the wall that can form between development and QA. The Right Tools for the Task: Mercury Quality Center When Iron Mountain decided to automate its testing processes, we turned to Mercury Quality Center, a web-based system for automated software quality testing across a wide range of IT and application environments. Mercury Quality Center is designed to optimize and automate key quality activities, including requirements, test and defects management, functional testing, and business process testing. At all points in the process, you have visibility into where your quality project stands, so you can optimize software quality while managing and controlling risk as you develop and test an application. Mercury Quality Center includes industry-leading products such as Mercury TestDirector, Mercury QuickTest Professional, Mercury WinRunner, and the new Mercury Business Process Testing. WWW.MERCURY.COM 9
MERCURY QUALITY CENTER TM QUALITY CENTER DASHBOARD Requirements Management TestDirector Test Lab Test Plan Defect Management Functional Testing QuickTest Professional WinRunner Business Process Testing APPLICATION DELIVERY FOUNDATION Shared Data Repository Central Administration Workflow Open APIs MERCURY SERVICES Managed Consulting Education Support Mercury Quality Center At Iron Mountain, our current mix of Mercury products includes over 500 scripts in Mercury WinRunner and Mercury LoadRunner. We use these to manage regression testing for our Oracle e-business Suite, including financials, HR, ilearning, and property management modules. With an average of two Oracle upgrades a year, these scripts save us at least a month s manual regression test time. We are also expanding our use of Mercury LoadRunner to accommodate specific customer requirements in our proprietary inventory systems. We recently completed training our staff on Mercury QuickTest Professional, and have created test scripts for our Siebel CRM product as well as a companion third-party vendor sales solution. We re about to automate an additional 200 to 400 manual scripts for our proprietary system. This will reduce test staff during upgrades from 15 analysts to five, and from four weeks testing time to two weeks. We use Mercury TestDirector/Mercury Quality Center to manage all our test scripts, provide audit reports as required, and to match project requirement documents to matching test requirements and test scripts. WWW.MERCURY.COM 10
Summary Integrating test automation into your QA processes does not need to be a daunting proposition. At Iron Mountain, we ve found that our development and QA teams can increase the speed, accuracy, and consistency of the testing process, and the IT department can achieve a higher ROI from software projects while cutting the risk of errors or oversights. At the same time, automation provides new opportunities for career advancement through the development of new skill sets. I encourage you to talk to your colleagues at companies that have made the move to automation and learn more. About Iron Mountain Iron Mountain Incorporated (NYSE:IRM) is the world s trusted partner for outsourced records and information management services. Founded in 1951, the company has grown to service more than 235,000 customer accounts throughout the United States, Canada, Europe, and Latin America. Iron Mountain offers records management services for both physical and digital media, disaster recovery support services, and consulting services that help businesses save money and manage risks associated with legal and regulatory compliance, protection of vital information, and business continuity challenges. WWW.MERCURY.COM 11
Mercury is the global leader in business technology optimization (BTO). We are committed to helping customers optimize the business value of IT. WWW.MERCURY.COM This document is protected by U.S. and international copyright laws, and may be used only with Mercury Interactive Corporation permission. Features of the Mercury Interactive Corporation software, and of other products and services of Mercury Interactive Corporation, may be covered by one or more issued or pending United States or foreign patents. Mercury, Mercury Interactive, the Mercury logo, Mercury Quality Center, Mercury Business Process Testing, Mercury LoadRunner, Mercury QuickTest Professional, Mercury TestDirector, and Mercury WinRunner are trademarks of Mercury Interactive Corporation and may be registered in certain jurisdictions. The absence of a trademark from this list does not constitute a waiver of Mercury s intellectual property rights concerning that trademark. All other company, brand and product names are registered trademarks or trademarks of their respective holders. Mercury Interactive Corporation disclaims any responsibility for specifying which marks are owned by which companies or which organizations. This document and its content are provided for informational purposes only. The opinions expressed in this document are those of the author(s)/contributor(s) and do not necessarily represent those of Mercury Interactive Corporation, its affiliates or its subsidiaries ( Mercury ). This document does not create a contract between Mercury and any person or entity ( You ). Mercury shall not be directly or indirectly liable to You for the document s content including, without limitation, any claims arising from 1) errors in the content, or 2) your use of or reliance on the content. Mercury shall not be liable for any damages or costs of any type arising out of or in any way associated with Your document use. Mercury retains all rights, title and interest in the document. The document and its content are presented AS IS with no warranty. 2005 Mercury Interactive Corporation. All rights reserved. www.mercury.com. WP-1517-0805