www.pwc.com/fsi Need a reliable way to prevent costly and potentially catastrophic software errors? Try automating functional testing.
The heart of the matter Software failures can devastate financial institutions and with growing pressure from customers and regulators for frequent software updates, the risk of failure is rising. We explore how automated functional testing of software can help minimize the odds of failure, and share how to avoid some common pitfalls along way. A close call with World War III. On September 26, 1983, the Soviet Union s nuclear early warning system signaled that the United States had launched five ballistic missiles that were headed for the USSR. Fortunately, Lieutenant Colonel Stanislav Petrov, who was responsible for monitoring the system and alerting Soviet leaders of impending attacks, believed the signal was a false alarm he reasoned that a US assault would involve far more than a handful of missiles. The warning signal was indeed a false alarm the result of a flaw in the system s software. Lt. Col. Petrov s reasoned response likely averted a nuclear catastrophe. The failure of the Soviet system in 1983 highlights in dramatic fashion the need for rigorous testing of software functionality. For all practical purposes, financial institutions are now technology companies that specialize in finance, and a software glitch may be capable of bringing down even the most established firms. What s more, with the spiraling volume of software that must be tested across a growing number of platforms, applications, and scenarios, there simply aren t enough people like Lt. Col. Petrov to recognize when software isn t doing what it s supposed to. A defect identified after software has gone live can cost 100 times more to fix than if it were found in earlier stages. 1 $14, 500 hourly cost to a mid-sized bank of having their ATM systems down. 2 of defects are not identified until after software goes live. 3 industry target for percentage of testing that should be automated. 4 average percentage of automated tests for major financial institutions. 5 1 2 3 4 5 HCL Technologies, A stitch in time saves nine Economics of defect prevention, March 18, 2011. Ibid. Ibid. Jennifer Lent, FAQ: Automated software testing basics, www.searchsoftwarequality.techtarget.com, accessed July 30, 2014. PwC analysis. Why financial institutions need automated testing 1
Driven to automate. A number of factors are driving the need for financial institutions to implement automated functional testing or increase the level of testing performed. We have identified three key drivers: 1. Pressure to improve the customer experience and reduce lead times. Increased pressure from customers and competitors is driving financial institutions to update software more often and release new products to market more rapidly. With each new product release or application enhancement, software functionality must be tested quickly without impacting the core functionality of the application/solution. In the case of mobile apps, functionality must be tested across multiple platforms, devices, and operating systems, adding to the complexity. Manual testing cannot address these challenges. Manual tests are timeconsuming (think shovel vs. snow plow), error-prone, costly, and may result in overlooked defects and delayed time-to-market, degrading the quality of the customer experience. Functional vs. performance testing The terms functional testing and performance testing are often used interchangeably, but the concepts differ: Performance testing is designed to assess how an application works under stress. The goal is to confirm that the application meets or exceeds requirements for stability, reliability, scalability, and responsiveness when a large number of customers use the application simultaneously. The objective of functional testing is to confirm that the software performs as intended for instance, when users click through the app, all the links work and transactions can be completed without errors. The case for automating functional tests. Automation is widely used for performance testing, while most functional tests are performed manually. But financial institutions can reap significant benefits by automating functional tests especially regression tests, which are designed to confirm that software updates or enhancements don t disrupt existing functionality. In an environment in which mobile apps and other software-based solutions are updated frequently, automating functional regression tests can boost speed and efficiency while reducing the potential for software defects. Figure 1: Implementing automated testing can reduce total testing time. 6 Automated functional tests are faster, more accurate, and can be performed repeatedly (see figure 1). Automation enables financial institutions to deliver software enhancements more quickly, from fixing bugs to pushing out new functionality, without the need for dedicated resources..... 6 PwC analysis based on the following: Fiserv, Automation Testing in Mobile Applications, December 5, 2012. Why financial institutions need automated testing 2
2. Risk aversion to post-production defects. When software fails, negative outcomes can include substantial costs from lost sales and time wasted as a result of duplicated efforts. The later in the development cycle defects are found, the more expensive they are to fix. Studies show that the formal testing phase of most development efforts detects only about half of all defects (refer to figure 2). Automated functional testing can be used to identify errors earlier in the development cycle by increasing testing coverage and verifying that system enhancements don t disrupt existing functionality. 3. Evolving regulatory requirements. A rapidly changing, increasingly complex regulatory environment requires financial institutions to have the flexibility to make application and system changes. Many regulatory projects don t always offer the time to fully design, develop, and test system solutions as organizations normally would. To cope with the rapid pace of change, many IT groups are adopting Agile development principles, which include shorter, more iterative cycles and are designed to deliver highquality software more rapidly. We ve seen that automated testing is better suited to the iterative nature of Agile than traditional testing methods, because it helps teams quickly confirm that applications are still working as the code evolves. Figure 2: The formal testing phase detects only about half of all defects. 7.... 7 HCL Technologies, A stitch in time saves nine Economics of defect prevention, March 18, 2011. Why financial institutions need automated testing 3
An in-depth discussion How financial institutions approach automated testing, and why they struggle. As financial institutions begin to recognize the need to improve how they deliver and update softwarebased products and services in a competitive, fast-paced environment, the adoption of automated functional testing has grown. Automated testing supports the trend toward Agile development. Most large financial institutions are pushing automated functional testing to reduce test costs over the long term and improve their ability to deliver quality with agility. However, we ve observed that financial institutions face four key challenges as they strive to implement automated functional testing: 2. Misconception that automated testing is a quick way to cut costs. There s a widespread belief that automating functional testing will lead to immediate cost reductions because fewer resources are required to perform testing. In our experience, costs typically increase initially because employees need to be trained and test scripts need to be built. In most cases, the number of resources needed to maintain the testing infrastructure then declines over time as additional tests are automated (see figure 3). Figure 3: While automation s s immediate benefit is improved software quality, financial institutions will also realize lower costs over the long-term. 1. Lack of long-term planning and a roadmap. Too often, financial institutions implement automated testing on an ad hoc basis, without the long-term plan and roadmap required to succeed on a larger scale. Typically, an institution will launch a pilot project to automate the testing of one or more critical apps. The project may be launched with great fanfare, substantial budgets, and senior management support. But without a plan for automating additional apps, momentum stalls after the project ends. Why financial institutions need automated testing 4
Automation can eventually generate sustainable cost savings, but in our view, cost alone should not be the main factor when considering whether to automate testing. The benefits of automation far outweigh the drawbacks of manual testing including potentially substantial financial losses and damage to the organization s reputation resulting from defective software released to the market. 3. Project vs. process. Another common misconception is that automated functional testing is a once and done exercise. In reality, it should be considered an ongoing process. For example, test scripts should be maintained as features are upgraded or rolled out to new platforms. Test data which test scripts use to run different testing scenarios also needs to be managed. In fact, automated testing magnifies the importance of test data because you need the right data ranges and selections to adequately test multiple device types, browsers, operating systems, and applications. 4. Fragmented approach to implementation. Many financial institutions have fragmented approaches to implementing automated functional testing. For instance, a large percentage of our clients especially large organizations outsource testing to third-party vendors. Large financial institutions likely use several vendors, often hired by different lines of business, resulting in multiple contracts, service-level agreements, and pricing structures. In addition, each vendor may use different tools and implementation frameworks, as well as apply different training and test procedures. This fragmented, inconsistent approach leads to duplicated efforts, decreased efficiency and effectiveness, and increased, often excessive, costs. Unless there is a clear, long-term plan in place to implement automation, organizations can quickly become overwhelmed and abandon their efforts. This is especially true of large institutions, which may have many hundreds of applications that are good candidates for automated testing. These challenges help explain why few financial institutions have come close to reaching the industry target of automating 70% of application testing. 8 Even among large financial institutions with more mature testing capabilities, we have observed closer to 15% to 20% of applications using automated functional testing..... 8 Jennifer Lent, FAQ: Automated software testing basics, www.searchsoftwarequality.techtarget.com, accessed July 30, 2014. Why financial institutions need automated testing 5
Boosting the odds of success: our approach to automation. We ve identified three key steps that help clients boost the odds of success when implementing an automated functional testing program. Step 1: Develop a long-term plan and a roadmap. Identify and prioritize test candidates. High-priority candidates for automated functional testing should include frequently updated applications and any customer-facing apps. Beyond these general guidelines, the selection of test candidates should be a collaborative effort that involves business and IT stakeholders with input from the testing team. The testing team should provide insight into the time, effort, and expense of automating specific apps, so that the larger team can weigh the benefits of testing against the costs. Decide whether automated testing will be implemented inhouse or outsourced. Before embarking on an initiative to automate functional testing, institutions will need to decide whether to implement testing inhouse or contract with a vendor to perform the work. The in-house approach requires a big investment upfront, as institutions generally build their testing capability over the course of 18 to 21 months. For institutions that choose to outsource, there are two options. One option is to own the required testing tools but have a third party perform the actual testing. The other option, which is growing in popularity, is testing as a service (TaaS). In our It s important to emphasize that automated functional testing is not a one-time, short-term initiative to cut costs; rather, it s an ongoing process that will boost efficiency and effectiveness over time. view, this second approach has more potential to lower overall costs: institutions do not have to incur the expense of purchasing testing tools and maintaining an in-house testing infrastructure, and they pay only for the services they use. In addition, TaaS can improve accuracy, reduce the risk of defects, and, in some cases, accelerate the development process. Develop a communication plan. Effective communication with business and IT stakeholders is essential to success. Management should clearly communicate the benefits of automation and address the implications for all employees involved, including any potential downsizing or re-allocation of resources. Step 2: Develop a testing framework and plan. Create a test team. To support automation across the enterprise, the test team should include personnel with the knowledge and expertise to develop, update, and automate test scripts. In a large institution, there may be an entire department of such experts. A subset of the test team should undergo more limited training so that they can simply run the tests. We also recommend that a dedicated team member, someone with knowledge of the applications being tested, be assigned to analyze test results. Establish collaboration between testers and developers. Test and development teams should partner to create a detailed testing plan that focuses on supporting a rapid software development approach. The test team should perform tests periodically as applications are developed, rather than testing only at the end of the development process (the traditional waterfall model). The key is to test small, workable sets of code before they go into production. This approach makes the workflow more manageable for developers and testers, and it also identifies defects earlier. In an Agile testing environment, for example, teams might decide to develop automated test scripts for every three to four sprints, so that the code has stabilized before testing is automated. Start small, and then expand. We recommend that financial institutions begin with a small pilot of automated test scripts, and then gradually expand the set. It s important to define the range of scenarios and test cases, so that all key functions of the app are tested. In addition, we advise that institutions focus on user interface and API testing as well as unit (testing one unit of a code at a time) and back-end testing to maximize Why financial institutions need automated testing 6
defect identification in a wide range of situations. Test teams should start with the most frequently used regression testing scenarios, such as tests of logging into and out of an application and navigating among tabs. Once the success of these tests has been confirmed, they can move on to test more advanced capabilities, such as viewing a bank account or purchasing a product. We suggest that test teams continue adding test candidates until the ratio of manual to automated testing is approximately 30% to 70%. (Beyond this point, additional automated testing will depend on the organization s objectives and the nature of the application.) Don t forget manual tests. Each time an enhancement is made or a new software version is released, we recommend that ad hoc manual tests be performed to confirm that automated test scripts are working properly. Step 3: Centralize management of service providers, tools, and resources. To avoid the inefficient and costly outcomes that result from the fragmented testing approach we discussed earlier, institutions should follow three guidelines to develop a centralized management structure: Consolidate the number of third-party service providers. Limiting the number of third parties involved can promote more effective vendor management by reducing software licensing fees and improving consistency in the testing approach across locations, products, and business units. Consolidate/rationalize the number of testing tools. Many financial institutions lack standardized testing tools, so a variety of tools may be used by different business units. In addition, it s not uncommon for an organization to have more than one license to use the same testing tool because third-party service providers hired by different business units are using the tool. Centralize testing program management. Centralization eliminates duplicate testing, which increases efficiency, reduces costs, and makes it easier to automate a larger percentage of functional tests. One entity should coordinate testing across all lines of business. Typically, the quality assurance (QA) group is given this responsibility. Why financial institutions need automated testing 7
What this means for your business Reaping the benefits of automation. By enabling the more rapid release of error-free software products and updates, automated functional testing can help financial institutions improve the customer experience and reduce organizational risk related to regulation and compliance. Test automation can also reduce an institution s costs by identifying defects early and eliminating the need for expensive rework late in the development cycle. With an automated approach to testing, data management becomes easier, as well; test data can be updated on a regular basis, with minimal human intervention required. Automation is rapidly becoming an industry standard, and the window of opportunity for financial institutions to seize the benefits of test automation may be closing. Financial institutions that embrace automation now will have an opportunity to increase revenue, speed time-tomarket, boost customer satisfaction, increase the ROI of testing, and free up resources to focus on higher value activities that provide a competitive advantage. Make testing permanent, not a project A common misconception is that automated functional testing should be implemented as a project. In fact, to derive the most benefit from automation, it should be an ongoing process that is embedded into the operations of the business. To make automated functional testing permanent, rather than a project, it s critical to gain the buy-in of senior management. This includes: Making a strong business case: Clearly articulate the benefits of automated testing, which include greater efficiency, less human involvement, and avoidance of headline-making errors that result in financial losses and reputational damage. Financial institutions should calculate the expected ROI as part of the business case and then track ROI as testing progresses. Building automation into the QA process: Set aside the budget and time for automation in future annual plans, so that automation becomes part of the QA operating model. With this approach, test scripts are developed along the way to eliminate backlogs. Creating organizational and training structures: In some cases, institutions need new organizational structures to support ongoing automation. For example, we ve seen clients be successful by putting an oversight committee in place to supervise automation efforts. This committee verifies that testing practices are consistent with automated testing principles and helps address roadblocks that may arise. Why financial institutions need automated testing 8
www.pwc.com/fsi For a deeper conversation, please contact: Julien Courbe julien.courbe@us.pwc.com +1 646 471 4771 www.linkedin.com/in/juliencourbe Michael Savelloni michael.b.savelloni@us.pwc.com 617 530 7722 www.linkedin.com/pub/michael-savelloni/5/239/318 Russ Riggen russell.riggen@us.pwc.com (646) 471 2052 www.linkedin.com/pub/russ-riggen/0/b03/299 Todd Johnston todd.a.johnston@us.pwc.com (203) 539 4273 www.linkedin.com/pub/todd-johnston/10/93b/435 We would like to acknowledge the contributions of Parama Bhattacharya to this publication. Follow us on Twitter @ PwC_US_FinSrvcs About our Financial Services practice PwC s people come together with one purpose: to build trust in society and solve important problems. PwC serves multinational financial institutions across banking and capital markets, insurance, asset management, hedge funds, private equity, payments, and financial technology. As a result, PwC has the extensive experience needed to advise on the portfolio of business issues that affect the industry, and we apply that knowledge to our clients individual circumstances. We help address business issues from client impact to product design, and from go-to-market strategy to human capital, across all dimensions of the organization. PwC US helps organizations and individuals create the value they re looking for. We re a member of the PwC network of firms in 157 countries with more than 184,000 people. We re committed to delivering quality in assurance, tax, and advisory services. Gain customized access to our insights by downloading our thought leadership app: PwC s 365 Advancing business thinking every day. Why financial institutions need automated testing, PwC, November 2014, www.pwc.com/fsi This publication was produced by PwC's Financial Services Institute. Visit us at www.pwc.com/fsi to learn more about us and meet the team. 2014 PricewaterhouseCoopers LLP, a Delaware limited liability partnership. All rights reserved. PwC US refers to the United States member firm, and PwC may refer to either the PwC network of firms or the US member firm. Each member firm is a separate legal entity. Please see www.pwc.com/structure for further details. This content is for general information purposes only, and should not be used as a substitute for consultation with professional advisors.