TCoE based Approach for building an Independant Test Organization

TCoE based Approach for building an Independant Test Organization Best Practices in Software Test Management Chandrashekar S Independent Validation Solutions Infosys Technologies Limited Bangalore, India Jayasundar R Independent Validation Solutions Infosys Technologies Limited Houston, USA Abstract - In today s dynamic economic environment, the complexity of various software applications that a business deploys is increasing exponentially, making it imperative for the IT Organizations to modernize their quality assurance (QA) practices. IT Organizations need to ensure an early ROI while delivering improved quality at lower risk. Complex business needs, highly diversified technology platforms, multi-country rollout requirements, reduced time to market obligations and increased QA demands have necessitated IT Organizations to implement Independent Test Organizations (ITO). The need of the hour is to establish a proactive, robust and scalable test organization in an organized and structured manner. Different techniques have been tried in the past with varied degree of success. This paper highlights the TCoE approach for setting up of an ITO that has been successfully deployed for a global customer of Infosys having diverse ERP/Non-ERP applications like SAP, IBM Maximo, Oracle Hyperion, Java,.NET, C++,VC++ etc. Sakshi.A Independent Validation Solutions Infosys Technologies Limited Bangalore, India Srinivas.T Independent Validation Solutions Infosys Technologies Limited Bangalore, India establishing a proactive, robust and scalable process for ensuring the success of the underlying Business Objective. Hence the success of the testing team is no longer measured based on only the test management and execution of the testing team but also on the overall success of the program. Hence, there is a need for deploying the test organization in an organized and structured manner to ensure quality assurance for all business applications. Different techniques have been tried in the past with varied degrees of success. This paper highlights the TCOE approach that has been successfully used to build an ITO in one of the key programs delivered by Infosys. PROBLEM STATEMENT n today s dynamic economic environment, the complexity of various software and systems that a business deploys is increasing exponentially making it imperative for the organization to modernize its quality assurance and management practices. IT Organizations are increasingly being asked to ensure an early ROI while delivering improved quality at lower risks for the strategic IT related investment. Complex business needs, highly diversified technology platforms, multi-country rollout requirements, reduced time to market obligations and increased quality assurance demands have necessitated such IT Organizations to adopt the concept of Independent Testing. This is a considerable shift from the traditional approach of development teams performing testing themselves. A. Impact of Offshore based IT Services Model on Testing The Software Development Life Cycle has become more complex with the advent of offshore based IT services model. It has essentially resulted in the distribution of the development and testing teams across geographies and organizations. That has necessitated the breaking down of the testing related activities into multiple smaller units so that each can be assigned to different parties with appropriate accountability. Normally this breakdown is achieved by defining various types of testing based on the combination of the deliverable being tested and the party to whom the same is being assigned. Most frequently used testing types are listed below: 1) Unit testing is a type of testing in which individual programs are tested to determine if they are fit for use. A program is the smallest testable part of an application. In a typical development/package implementation,unit testing would correspond to testing of a screen or a report or a workflow. The Software testing sub function needs to mature itself from being just a process of verification and validation of the deployed software/application /program to ensure that it meets the technical and business requirements of the client to that of 2) Functional testing is a type of black box testing that bases its test cases on the technical specifications of a group of software programs related to each other. Functions are tested by feeding them input and examining the output, and Index Terms Building Testing Team, Benefits of TCoE, Challenges for Independent Testing, Continuous Improvements in TCoE, Testing Centre of Excellence, TCoE Case Study I INTRODUCTION 33

internal program structure is rarely considered. Functional testing in a development/package implementation would correspond to testing of certain functions within a module. 3) System testing of software is testing conducted on a complete, integrated application to evaluate the application s compliance with its specified business requirements. System testing falls within the scope of black box testing, and as such, requires minimal knowledge of the inner design of the code or logic. System testing in a development/package implementation would correspond to testing of all the functions within a given module. 4) System Integration testing (SIT), is a type of testing that exercises a software application s coexistence with others. System integration testing takes multiple integrated applications that have passed system testing as input and tests their required interactions as per the end to end Business Process flows. System Integration testing in a development/package implementation would correspond to testing of a complete cycle like the Procure to Pay, Order to Cash which would involve multiple modules. 5) Smoke testing refers to the first test made after the build or repairs to an application, to provide some assurance that the application under test will not catastrophically fail. Typically Smoke Testing would involve combinations of Unit test cases, Functional test cases, System test cases and System Integration test cases to ascertain that vital portions of the application under test are working. 6) Performance testing is a type of testing that is performed, to determine how fast some aspects of an application perform under a particular workload. It can also serve to validate and verify other quality attributes of the application, such as scalability, reliability and resource usage. This assumes lot of significance in the ERP package life cycle as the number of users is very high and is distributed across the world in various geographies. This breakdown of testing related activities into smaller units with pre-defined accountability can pose various forms of challenges and has thus necessitated the formulation of a structured approach towards engaging a testing team. Various testing team engagement models described below can be deployed depending on the context of the Business/IT landscape of the organization. B. Comparative Analysis of various Testing Team Engagement Models In order to meet the evolving needs of the IT Organization, there could be varied approaches for involving the testing team in a development/package implementation project.the approach taken is dependent upon who needs to take the bottom-line to ensure quality of the developed application. Some of the popularly used approaches include: 1) Embedded testing: Traditional way of testing carried out by the development team themselves envisages testing to start after the Detailed Design and Development phase as depicted in the Fig1. Fig.1 In this model it is the responsibility of the development team to ensure the Quality of the application. 2) Independent testing: This is a new age approach tailored for the onsite-offshore software development life cycle model and involves testing by a dedicated third-party testing team. This enables testing to be conducted much ahead in the software development life cycle (during the Requirement Gathering and Analysis phase) as depicted in the Fig2. 7) User Acceptance testing is a type of testing performed on an application prior to its release for use by all its intended end users. This is normally used as a formal stage to indicate the acceptability of the developed solutions by its end users. 8) Regression testing is any type of software testing that seeks to uncover defects after changes are made to the program or a group of programs. The intent of regression testing is to assure that a change, such as a defect, did not introduce new defects. Regression testing can be used to test the application efficiently by systematically selecting the appropriate minimum number of test cases needed to adequately cover the affected part of the application. Fig 2. In this approach, the bottom-line to ensure the Quality of the 34

application lies with the 3rd party testing team. 3) Need based Independent testing: this involves testing by a dedicated team (which reports back into the development/package implementation team manager) for a limited scope of the overall application. Here also testing is performed after the Detailed Design and Development phase as depicted in the Fig3. Fig 3.In this approach, the bottom-line to ensure the Quality of the application lies with the development team. Owing to the above mentioned benefits/advantages, the Independent Testing Model would prove to be the most beneficial for an organization that has embarked on an initiative to modernize its testing practices to ensure lower Total Cost of Ownership, enhanced quality and higher reusability. However there could be different challenges across various dimensions like people, process, knowledge and tools& infrastructure associated with the setting up of an ITO and thus the right approach needs to be taken to ensure efficient and successful ITO deployment. Let us now have a detailed look at some of these challenges and how the TCoE based approach can resolve each one of them: D. TCoE based approach for setting up an ITO Test Centre of Excellence: Consists of a permanent test organization which enables the IT Organization to offer testing services at higher quality, reduced effort and lesser cost. The robust engagement model with a well-defined governance structure helps in establishing a Test Organization with clear vision, mission, objectives and goals. It brings together a best-in-class of career testers to achieve best-in-industry testing capability to provide expert testing skills across various Business and IT initiatives. This enables enterprises to scale up/scale down the testing team in sync with the various Business and IT initiative. It delivers a mature process oriented approach towards testing ensuring standard test methodology and leveraging best practices to improve test quality and effectiveness over a period of time. E. How can a TCoE based approach address these issues? The Table I depicts the solutions provided by the TCoE approach to address the various challenges: Table I A TCoE based approach for building an ITO addresses the typical challenges faced DIMENSIONS CHALLENGES SOLUTIONS PEOPLE C. Typical challenges faced while setting up of an ITO Ensuring the continuity of the testing team across PLC Centralized Testing team Meeting the varying demands during periods of hyper/lower activity. Build In-depth Cross Functional Competency within testing team. Support testing across multiple functional areas using the same team. Flexible Resourcing Core & Extended Team Continuous capability building programs enabling constant learning Develop testing capability that can be easily deployed across engagements and projects. Ramp up in a new technology and business process area PROCESS Fig 4.Typical challenges faced while setting up an ITO Functional silos due to Limited Standardization of QA processes Standardize Test Processes across IT organization. Lack of Uniform test strategy across different Projects. Establish centralized knowledge repository with contributions from all 35

projects and Implement mandatory test strategy review checkpoint for every project Inadequate Test Requirements Coverage Lack of Metrics based testing organization KNOWLEDGE MANAGEMENT TOOLS & INFRASTRUCTU RE Establish a risk based test framework Establish Process Capability Baseline Implement Continuous Improvement Strategies by incorporating Industry Best Practices No shared knowledge management framework to ensure constant learning. Timely availability of the reusable assets Establish a centralized repository Project based purchase of test related tools lead to higher tool cost. Purchase of Enterprise wide Test Management and test automation tool licenses Quality of the test data in the Test Environment. Structured and periodic refresh of the test environment from the production data Higher Maintenance spend due to defect leakage in production Separate Test Environment for various applications with Strict Configuration Management Publish knowledge alerts regarding the reusable assets available for various aspects of testing. Fig 5.Approaches for building an ITO using a TCoE based method G. A Case Study where Pilot based TCoE approach has been implemented Let us now see how a major old field services company was able to overcome certain issues that it faced while setting up of an ITO for its testing requirements using the Pilot based TCOE approach and how it then realized significant benefits in terms of enhanced productivity, improved cycle time, higher quality and lower TCO owing to the successful implementation of the same. Client Profile PROPOSED SOLUTION F. How to build an ITO using a TCoE based approach? Once an IT Organization has decided to adopt the TCoE based approach for setting up of an ITO, depending upon the current state of testing processes, the investment needed, the business risks involved and the expected duration there could be different approaches for doing the same.these approaches can be broadly categorized into: Pilot: Set up a testing team for a project and use the learning's to finalize the TCoE building blocks Transform: Define the Building Blocks of the TCoE and Implement for one large project program. Big-Bang: Define the Building Blocks of the TCoE and Implement for all projects at the same time. A major oilfield services company operating in multiple countries, providing worldwide oil and natural gas industry products and services for drilling, formation evaluation, completion and production. It also provides reservoir engineering and other consulting services. Background The client had a diverse set of Business Applications using various technologies like SAP, IBM Maximo, Oracle Hyperion, Java,.NET, MOSS, C++, VC++, and DB. A variety of IT systems were implemented simultaneously in order to meet the varying business needs of the organization. This led to the creation of a large IT Organization which had to adopt the recent best practices for testing such a diverse set of IT landscape; one of which was to adopt and implement an ITO. Challenges faced by the Client Engaging robust testing procedures was the need of the hour as having to deliver substandard test releases otherwise could be very unpleasant. The client thus wanted to build an ITO so 36

that it could cater to the following areas of improvement: Implement a Process-Centric approach towards testing to prevent functional silos. Introduce enhanced automated testing capabilities to ensure efficient utilization of Business SME s and project teams. Build a structured test knowledge management framework to ensure planned testing measures and high release-standards. Introduce a centralized testing approach to avoid duplication of testing efforts of the project/divisional teams and ensure effective time optimization. Owing to these, the client envisioned certain issues by not having an ITO System vulnerability due to multiple technologies There were multiple point solutions being implemented by different vendors, with different user interfaces, which could cause incompatibilities with the changes that were made to the various applications. This could stop the working of these point solutions, affecting the business adversely. Disruptions in the on-going implementations of various new packages like SAP ERP, IBM MAXIMO, Oracle HYPERION etc. and the significant planned customized application initiatives due to conflicting timelines and technology risks mentioned above. Owing to high degree of customization in various standard ERP packages that had been implemented and integration with Complex Legacy Solutions, any change during the package maintenance(patch applications, upgrades, new functionality implementations and enhancements) could cause an adverse ripple effect on the functionality of the overall application, unless they were validated end-to-end.thus, ensuring the optimal coverage of testing the customizations in various ERP packages that were implemented was seen as a critical requirement. The constant changes that were being made across various applications needed high availability of the Business SME s as per the project timelines which could be in conflict with their day to day Business operations. Hence there was a need to optimize the utility of Business SME s for various IT initiatives. Selecting the approach for building an ITO Upon analysis of various methods of setting up of an ITO, it was jointly decided by the client and the Infosys team to select the Pilot based TCoE approach. The main reason for selecting the same was an earlier failed attempt of using a BigBang approach. Some of the key reasons for the failure were identified as lack of executive support from business and IT leaders, lack of structured and robust TCoE implementation methodology, lack of standardized testing processes that could be deployed across multiple technologies and business areas, lack of organization change management readiness etc. Hence this time they wanted to pilot it in a large IT project and then rollout the same to other projects. An upcoming large SAP upgrade program was chosen as the pilot project. In order to obtain full-fledged support for the proposed ITO, the client and Infosys conducted various workshops showcasing the NEED-FIT analysis and immediate and future benefits of the TCoE based approach, to the various stakeholders, and obtained sign-offs for the same. Implementation of the TCoE Transition to a TCoE model was meticulously reflected upon prior to implementation, with exhaustive planning based on impact and business risk assessment, application profiling, and evaluation of transition readiness. A comprehensive transition plan was then built by the client and Infosys around the strategic assets and a robust governance model outlined to transform the current Test Organization. So the transformation was achieved across multiple tracks related to testing as described below: a) Test governance: A robust two-tier governance structure that manages resources and monitors the functioning of the TCOE under a single umbrella was established. The executive team comprising of the client s TCoE sponsor and Infosys Delivery Management team would set objectives for the relationship and provide guidance. They would also set long-term and upcoming quarter s goals, set evaluation parameters and review the quarterly performance and overall health of the relationship. The Test Management Office (TMO) involved the client s TCoE champion and Infosys TCoE onsite/offshore leads, with their respective teams, who would work under the guidance of the executive team. The TMO in term would collaborate with various program managers to realize the full potential benefits of a TCoE and report functioning to the executive team. b) Test planning: The team crafted a base of shared test management practices by creating re-usable templates with mandatory workflows and propagating them automatically across projects. This helped in standardizing various details like levels of testing, in scope/out of scope details, testing timelines/milestones and deliverables, responsibility matrix and assumptions/dependencies. Maintaining a repository of templates helped in emulating the real life business processes 37

and provided a framework for future efforts estimation. Another aspect of planning was to proactively identify all the Non-Unicode Compliant interfaces that existed, so that the client could work with the respective vendors to make them code-compliant. This process immediately helped in accelerating roll-outs. BENEFITS DELIVERED c) Defect management: Regulation of the defect tracking mechanism, throughout the defect life cycle, helped the development teams to find, prioritize and resolve defects based on business priority. d) Test data management: The TCoE helped in regulating many decisions like how much test data needs to be made available, on which test environment it needs to be made available, etc. This ensured that greater value could be extracted from the available test data and that validated test data was delivered in an organized, consistent, secure and controlled manner. Very efficient system validation testing, cutting across product lines/divisions, could be carried out due to optimal test data management and automation. e) Test-scope change management: The TCoE had an inbuilt mechanism to handle changes in test scope. The team established adequate guidelines to ensure that scope changes could be identified, tracked and taken to logical closure in a consistent manner. A standardized Business Impact Analysis (BIA) was included before implementing any major change. f) Test automation: Standards were developed for test automation like log file formats, tools, library routines, etc., which ensured increased productivity of test automation development teams. g) Performance benchmarking: As a part of the TCoE, a global performance test centre was established which could be utilized for performance benchmarking of all ERP/Non-ERP applications. By deploying a structured approach like the TCoE, the client s IT organization could setup a high quality, robust, scalable independent test organization. Some of the major benefits delivered are classified as below. 38

Key Metrics: Following scorecard shows the high level metrics for all the projects executed in 2010. centralized HP QC base repository with forward/backward traceability. Backward traceability helps in mapping the test cases to the requirements and forward traceability helps in mapping test cases to test results and defects. This bidirectional traceability will help in ensuring that all requirements are covered by test cases and the test results and defects can be linked to possible issues during requirements definition/gathering. 2. Knowledge Management: Preparation and publication of BOK s for each project and made accessible to all the other team members of TCoE through a centralized knowledge repository. Cross training sessions are held to train the team members across various technologies and business areas. 3. Testing Process standardization that can be applicable across new technology areas, business areas. B. How the TCOE performance is continuously improving its Based on the above mentioned initiatives, the TCoE has been able to lay the foundation for continuously improving the performance to provide significant value to the IT organization across various project dimensions like effort, timelines and cost and thus help in achieving the underlying business objectives. Here is the snapshot of the improvements that have been achieved for the various projects executed in 2009-2010. FUTURE DIRECTION FOR THE TCOE A. Key initiatives: Based on the significant business value delivered by the TCoE team, the client decided to involve it in all its future testing endeavors. In order to ensure that the TCoE team is constantly enabled to support various IT initiatives; certain focused initiatives have been undertaken. Broadly these initiatives are classified as following: 1. Improve the agility to support Business events that happen during a short period of time and ensure lower TCO. Some of the continuous improvement initiatives taken to achieve this are: a) Increase the coverage of automated testing by year marking certain part of the IT Budget towards development and maintenance of automated test scripts. b) Focused attempt towards increasing the reusability of existing test assets: Fig 11. Project Delivery Quality All test assets are developed in a structured form so that they can be reused across multiple releases and are stored in a 39

Fig 12. Cost of Quality REFERENCES AND ACKNOWLEDGMENTS Oracle is a registered trademark of Oracle Corporation Maximo is the registered trademark of International Business Machines IBM is a registered trademark of International Business Machines SAP is a registered trademark of SAP AG Any other trademarks referred in this document are the trademarks of their respective owners BIOGRAPHIES Fig 13. Effort Saving through Test Automation Fig 14. Cost Saving through Test Automation CONCLUSION The Infosys TCoE team has been involved in providing one or more of the following testing services for over 30+ applications cutting across different technologies, business functions and geographies. These applications have been developed by the in-house team of the client, Infosys and/or third-party vendors as well. The services provided by the Infosys TCOE team include Functional and Regression testing Automated regression testing for maintenance releases Performance testing Database / Report testing of large amounts of data for version upgrades Middleware Test Automation Browser compatibility testing for all the applications Providing the testing support for the hot fixes released by the ERP package vendors & as well as the client/infosys developed applications As we can see, over a period of 2+ years since its inception in 2009, the ITO has extended its breadth and depth across the IT landscape of the customer to successfully deliver a high quality, robust, scalable independent test organization. We believe that this has been made possible by the structured and scalable application of a TCoE based approach. Thus we conclude that a TCoE based approach is the most appropriate one for setting up an ITO in this type of large complex diversified IT Organizations. Chandrashekar.S is a Senior Project Manager with the Independent Validation Solutions (IVS) group at Infosys. He has 16+ years of experience in ERP projects and has handled multiple roles like Onsite/Offshore Project Manager, QA Manager for many clients across NA, Europe & Asia. He is a Project Management Professional (PMP) and has presented papers in many conferences including STEPAUTO, Retail Oracle Users Group, Ignite Germany. Jayasundar.R is a Senior Project Manager with the IVS group at Infosys with 14+ years of experience in managing multiple QA projects for multiple industry domains like process control, oil and gas industries for many organizations across NA and Europe. He is a Honeywell six sigma Black Belt certified professional and has presented papers in HP Universe. Sakshi.A is an Associate Consultant with the IVS group at Infosys. She has been involved in ERP implementation/testing and has also worked as a System Engineer/Functional Analyst in which she was involved in testing projects across all the lifecycle stages. Srinivas.T is a Senior Project Manager with the IVS group at Infosys Technologies Ltd. He has 12+ years of experience in ERP projects across the lifecycle stages (implementation, upgrade, support & maintenance). He has handled multiple roles like Onsite Project Manager, Offshore Project Manager in these engagements with many organizations across North America, Europe & Asia. 40