Offshore Remote Application Testing Can Cut Cost of Quality by Up to 75%

Transcription

1 A p r i l A M R R e s e a r c h R e p o r t Offshore Remote Application Testing Can Cut Cost of Quality by Up to 75% by Lance Travis Remote application testing provides companies with a valuable, lower risk way to establish a relationship with an offshore outsourcing service provider. Benefits include lower testing costs and improved quality. More important, testing services provide a way for companies to mature their offshore outsourcing skills in preparation for a long-term development relationship.

2 Copyright 2005 by AMR Research, Inc. No portion of this report may be reproduced in whole or in part without the prior written permission of AMR Research. Any written materials are protected by United States copyright laws and international treaty provisions. AMR Research offers no specific guarantee regarding the accuracy or completeness of the information presented, but the professional staff of AMR Research makes every reasonable effort to present the most reliable information available to it and to meet or exceed any applicable industry standards. AMR Research is not a registered investment advisor, and it is not the intent of this document to recommend specific companies for investment, acquisition, or other financial considerations.

3 Offshore Remote Application Testing Can Cut Cost of Quality by Up to 75% by Lance Travis Offshore software testing saves companies up to 75% over in-house costs, improves quality, and is a lower risk way to develop mature offshore outsourcing skills. The Bottom Line Software defects continue to proliferate, but the cost of ensuring Executive quality is high. Software testing can account for 30% of development time and many companies staff testing positions with Summary high-cost, on-site independent contractors. Companies must decide if offshore testing provides a solution to their cost of quality problems. After interviewing 14 service providers that offer offshore testing services and reviewing the results of 16 customer engagements, AMR Research concludes the following: Low cost, faster test cycles, and improved quality are benefits from offshore testing. Segment testing reduces the risk of offshore outsourcing. Engagement size and add-on services distinguish the service providers. Software quality issues continue to plague CIOs and software product development organizations Academics report that in 2002, software defects cost $60B to fix, and 50% of software development costs were related to finding and fixing defects. Obviously, many companies aren t getting the job done with in-house resources or on-site contractors. They must decide whether the benefits of using offshore software testing services outweigh the risks. Low cost, faster test cycles, and improved quality are benefits from offshore testing Companies interviewed for this Report consistently reported cost savings, faster test cycles, and improved quality as the benefits they received from offshore testing. AMR Research Report April AMR Research, Inc. 1

4 Many companies report 60% to 75% savings by moving testing offshore More than half of the companies that we talked to realized savings between 60% and 75% over in-house costs. Rather than letting people go, one manufacturing company was able to move the replaced in-house staff to other projects and still save $300K per year. Some companies report that rather than realizing savings they maintained spending at roughly the same levels, they increased the level of testing because of the lower costs associated with offshore resources. Savings depend on the percentage of offshore resources and on the amount of overhead associated with managing the offshore relationship. Some projects, such as quality consulting projects, require a much higher on-site presence and result in smaller savings. Leveraging staffing flexibility is another way companies saved money using offshore testing providers. Rather than keeping underutilized in-house resources, companies ramped up and down outsourced resources, depending on the recurring demands of their development cycle. Faster test cycle times result from offshore testing Companies are able to leverage time zone differences to accelerate test cycles in a follow-thesun philosophy. Daytime testing occurs in India and is then handed over to the United States at the end of the day in order to leverage the 9.5- to 12.5-hour time difference between the two countries. One company reduced its test cycle by half, and then improved quality by adding more test scenarios. When leveraging the 24-hour clock, companies had to make arrangements for offhours support of their test environment. Before it made arrangements, one company lost 48 hours worth of testing because a server failed on a Sunday evening (U.S. time) of a three-day weekend. The Indian test team was unable to work Monday and Tuesday in India because no one was available to fix the problem in the United States until Tuesday morning U.S. time (Tuesday night India time). All of the service providers interviewed for this Report offer services to automate tests. After using the offshore service provider to automate its 3,500 tests, one company reduced its test cycle from 90 days to 29. The cost of automating the tests was much less than the cost of continually running them manually AMR Research, Inc. AMR Research Report April 2005

5 Quality improvement results from additional testing, and eventually root-cause analysis Companies reported immediate quality improvements as a result of test automation and reduced cycle times. Test automation allows them to run more tests during a test cycle. Because they are able to run more tests, they are able to find more problems. One manufacturing company reported that because it never allocated enough resources to in-house testing, it never would have been able to automate and test as thoroughly as the offshore provider. A financial services company reported a dramatic decrease in the number of post-go-live reported defects after the offshore provider automated the company s test scripts. As the offshore testers become more familiar with the applications being tested, they begin to help their customers find the root cause of defects and eliminate future problems. However, this ability doesn t happen right away. Companies reported that it took between 14 months and 3 years before the offshore testers had sufficient familiarity to be effective in finding the root cause of problems. Segment testing reduces the risk of offshore outsourcing Testing work can be outsourced offshore with less risk than application development or maintenance because of the following: The work can be easily segmented into process consulting, test execution, test automation, test development, and quality improvement consulting. Process consulting allows companies to leverage the service provider s best practices to improve how they test. Process consulting, test execution, and test automation require minimal interaction with business users, allowing an IT organization to establish a relationship with an offshore provider without involving the business users. Test development requires working with the business user to understand the purpose of the software to be tested and their acceptance criteria to be understood. Testing doesn t require the offshore resources to touch the application source code. The offshore resources can gain experience with the customer s development process and its applications without running the risk of breaking or corrupting an application. Most companies underinvest in testing. An effective offshore testing program can convince outsourcing naysayers that offshore work can be effective. Testing engagements can slowly ramp up. It is possible to start small and work up to a larger engagement. The number of resources supplied by the outsourcer for the customer engagements studied in for this Report ranged from 6 to 350 people. It is also possible to start with lower value activities such as test execution, and ramp up to higher value activities such as root-cause analysis (see Figure 1). AMR Research Report April AMR Research, Inc. 3

6 Not all companies that we talked with were convinced that testing is a good candidate for a company s first foray into offshore outsourcing. One company had difficulty sending test execution offshore because its development process and documentation were nonexistent. It was only after the offshore company was able to drive process improvements into the development organization that the company was able to outsource test execution. Figure 1: Value increases as relationship matures Savings from Problem Elimination Quality Improvements Savings from Process Automation Savings from Labor Arbitrage Savings from Process Improvement Outsourced Task Quality Process Consulting Test Execution Test Automation Test Development Root Cause Analysis and Quality Improvements Maturing Relationship Source: AMR Research, AMR Research, Inc. AMR Research Report April 2005

7 Separate testing and development organizations important to customers Many customers selected their testing service provider because it offered a testing service separate (reporting to different management) from its development organization. Two of the companies selected one offshore company for development and another for testing. The customers felt that having separate organizations ensured that testing and quality concerns would not get subordinated to development schedule concerns. Having multiple companies involved created a healthy competition that benefited the customer. One company using separate organizations for testing and development felt that it was essential to use in-house resources as the quality advocate. The person in charge of the project had trouble getting the Indian resources to push hard enough for process improvements. Despite the lower risk, strong on-site leadership is still required All but one company interviewed stated that having the outsourcing company supply on-site resources was essential. The on-site resources are necessary to manage the flow of work to and from the offshore locations. In addition to having on-site personnel, companies recommend having strong in-house managers handle the relationship. Companies must establish ground rules for the relationship and cannot just throw things over the wall to the outsourcer and expect success. For test execution projects, companies typically are able to move 70% to 80% of the test resources offshore, but find that keeping the remaining 20% to 30% in-house is essential. Offshore testing does require some infrastructure investment The offshore resources need access to different systems than offshore developers require. For example, one financial services company provided access to the systems that are integrated with the to-be-tested systems to ensure a complete systems test. In this case, the company extended its private network to India. Some companies had problems with sending test data to India. Either government regulations or contracts with their customers prevented them from moving test data offshore. In cases such as these, the offshore providers offered tools to facilitate the creation of unrestricted test data based on live data. AMR Research Report April AMR Research, Inc. 5

8 Engagement size and add-on services distinguish the service providers AMR Research interviewed a broad cross-section of service providers for this Report: Accenture, AppLabs, Cognizant, HCL Technologies, Headstrong, Infosys, ITC Infotech, Keane, Patni, Rapidigm, Sapient, Satyam, Tata Consultancy Services (TCS), and Wipro. All the service providers interviewed offer a suite of test consulting and software lifecycle testing services (e.g., manual testing, functional testing, performance testing, acceptance testing, and test automation). All have provided test services for multiple clients, with AppsLab, Cognizant, Infosys, Keane, Patni, Rapidigm, Satyam, TCS, and Wipro having more than 50 customers. The service providers support multiple engagement models: time and materials, fixedpriced projects, and dedicated test centers. However, a majority of the offshore testing engagements are time and materials contracts. The service providers examined fall into three categories: large offshore firms, smaller offshore firms, and traditional consulting firms. The larger firms support the largest engagements Cognizant, Infosys, TCS, and Wipro are primarily focused on large engagements. Their average engagements number over 50 test resources and last a year or more, and their largest engagements are double to triple the size of the smaller firms. Consider these firms for long-term, large test engagements (see Table 1). You can start small with these firms, but unless you show solid plans to scale up the relationship, you may find that your requirements are mismatched with their delivery capabilities. Firms supporting smaller engagements are fine for short-term staff augmentation and midsize, longer term engagements The average deals of smaller firms AppLabs, HCL Technologies, Headstrong, ITC Infotech, Patni, Rapidigm, and Satyam involve 5 to 10 test resources and last 6 to 9 months. Their largest deals range from a low of 15 resources (Headstrong and Rapidigm) to over 60 (Patni and Satyam). Consider these companies for small staff augmentation projects. Consider the high end of this category (AppLabs, HCL, Patni, and Satyam) for larger, longer term projects. Note that project size does not always determine project complexity. Smaller projects can be quite complex, involving multiple applications, geographies, and quality consulting. Despite the relatively smaller size of their engagements, Patni and Satyam have some of the more complex engagements AMR Research, Inc. AMR Research Report April 2005

9 Traditional consulting firms enter the offshore testing market Accenture, Keane, and Sapient fall into this category. They have recently supplemented their implementation services with stand-alone offshore testing capabilities. Accenture created a dedicated test center of excellence in India in 2004 and now has over 500 test resources. Keane also created a dedicated test center in India in 2004 and has more than 300 test resources. Sapient provides testing services through its development resources. To date, its testing engagements have been small compared to the large offshore firms. Accenture's and Keane s largest engagements involve 30 and 26 test resources, respectively. Use a testing engagement with these providers primarily as an entry point into their application implementation and consulting services. Table 1: Attributes that indicate service maturity Number of test center employees Year service Company started AppLabs Cognizant HCL Technologies Headstrong Infosys ITC Infotech Patni Rapidigm Satyam TCS Wipro Source: AMR Research, 2005 In addition to the standard test consulting and quality offerings, some of the providers offer unique test services that may be of specific interest: Usability testing Cognizant, Patni (in 2005), Rapidigm, Satyam Number of resources in largest deal Compliance testing Accenture, Cognizant (Health Insurance Portability and Accountability Act [HIPAA], CFR 21), Infosys (HIPAA, CFR 21, Sarbanes-Oxley Act [SOX]), Patni (HIPAA), TCS (SOX), Satyam (CFR 21), and Wipro (SOX in 2005) Packaged application testing All of the service providers offer some level of packaged application testing, but those planning SAP and Oracle upgrade testing should consider the service providers with the deepest SAP and Oracle implementation skills: Accenture, Cognizant, HCL, Infosys, Patni, Rapidigm, Satyam, TCS, and Wipro. Note that Headstrong only offers Siebel testing services. Business strategy and test strategy alignment offered by Accenture, Infosys, Keane, Satyam, TCS, and Wipro In addition to consulting about test technology and test processes, they can help you prioritize your test strategy based on the importance of specific applications and application functionality to your business strategy. AMR Research Report April AMR Research, Inc. 7

10 Offshore testing services have lower-than-expected attrition rates The service providers have focused on making testing an attractive career path and minimized attrition within their organizations. The companies offer extensive technical and business training and career advancement opportunities in order to attract and retain quality resources. As a result, attrition is often lower in the test organizations than in the rest of the Indian companies. Infosys, ITC Infotech, Rapidigm, Satyam, and TCS reported the lowest attrition rates within their testing organizations. Longterm customers benefit because of consistency of staff on their projects. Common tools facilitate workflow between on-shore and offshore All of the service providers interviewed support standard third-party test tools from companies such as Compuware, Empirix, IBM Rational, Mercury, and Segue, and open source tools such as JUnit. The providers use standard tools such as Mercury TestDirector, IBM Rational s ClearQuest, and Excel spreadsheets sent via to facilitate the flow of information between locations. Whenever possible, companies should leverage the relationships the service providers have with tool vendors when establishing the tools to be used in the engagement. Closing Comments Testing is a low-risk, high-reward entry into offshore services. Companies that have been struggling to develop an offshore outsourcing strategy should look to a limited test engagement as their first step. Outsourcing test execution and test automation limit dependence on the business units for support, offer significant cost savings and improved quality, and give companies a chance to hone outsourcing skills before moving on to more extensive offshore engagements AMR Research, Inc. AMR Research Report April 2005

11 Accenture AppLabs Cognizant Compuware Empirix HCL Technologies Headstrong IBM Rational Infosys ITC Infotech Keane Mercury Oracle Patni Rapidigm SAP Sapient Satyam Segue Siebel Tata Consultancy Services (TCS) Wipro www-306.ibm.com/software/rational Company List

12 Acronyms and Abbreviations HIPAA Health Insurance Portability and Accountability Act SOX Sarbanes-Oxley Act AMR Research provides world-class research and actionable advice for executives tasked with delivering enhanced business process performance and cost savings with the aid of technology. 5,000 leaders in the Global 1000 put their trust in AMR Research s integrity, depth of industry expertise, and passion for customer service to support their most critical business initiatives, including supply chain transformation, new product introduction, customer profitability, compliance and governance, and IT benefits realization. More information is available at Your comments are welcome. Reprints are available. Send any comments or questions to: AMR Research, Inc. 125 Summer Street Boston, MA Tel: Fax: Montgomery Street Suite 650 San Francisco, CA Tel: Whittaker House Whittaker Avenue Richmond, Surrey TW9 1EH United Kingdom Tel: +44 (0) Fax: +44 (0) AMR-R-18109