www.hcltech.com QEx WHITEPAPER Increasing Cost Predictability in Performance Testing Services via Unit-Based Pricing Model Business Assurance & Testing Across the spectrum of IT projects, whether it is custom application development, systems integration, package implementation, infrastructure services or technology consulting, the engagement model and monetization of IT services have evolved over the last two decades. From Time & Material (T&M) to Fixed Price to Risk Reward to Outcome/ Value Based Pricing Model, there has been a drastic shift. Of course the right pricing strategy for either party depends on various factors including scope of work, division of labor between client and vendor, relationship history, track record, trust factor etc. Performance Testing activities are a minor part of SDLC in terms of effort, but also is a very critical quality gateway before release to production. By and large, Performance Testing projects have been operated in T&M mode or Fixed Price, if it is a finite term engagement with definite scope and deliverables. However, the market demand is for Performance Testing engagement models to shift to the next gear of maturity and operate on a more predictable model with increased accountability of cost. AuthOr: Vijayanand Chelliahdhas Sr. Solutions Architect at HCL America Inc. currently responsible for solutions and business development for specialized testing services including Performance Engineering and Mobility Testing across the US geography WHITEPAPER July 2015
TABLE OF CONTENT 1. INTRODUCTION 3 2. WHAT IS UPM IN PERFORMANCE TESTING SERVICES? 4 3. HOW THE UPM WORKS? 5 4. MONTHLY BILLING CYCLE PROCESS FLOW 7 5. PRE-REQUISITES FOR IMPLEMENTING UPM 7 6. ADVANTAGES & BENEFITS 8 7. CONCLUSION 8 2015, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 2
1. INTRODUCTION Performance Testing as a discipline has evolved significantly over the last decade, to a point where a large number of clients have integrated Performance Testing as a standard quality gateway in their SDLC. Performance Testing projects come in different sizes and durations. For instance, a long running program of 12 to 18 months duration may require Performance Testing for an elapsed duration of 2 to 6 months only depending on the scope and complexity of work. The scope of work typically involves performance requirements gathering, workload modeling and characterization to identify the performance critical use cases and workload scenarios, developing test assets (scripts/ data/ environment), executing various tests such as Load, Capacity, Endurance etc., conducting results analysis, identifying performance bottlenecks with a significant overhead for coordination and reviews with various stakeholder teams. These set of activities could be executed for a given individual project or could be executed for multiple projects by a group of performance test engineers operating as a Centralized Team/ Shared Services Team/ CoE Model. In the former case, estimation and pricing model is straightforward by opting either Time & Material or Fixed Price Model. However, Shared Services Operating Model brings in certain inherent administrative and operational challenges, which are amplified by the nature of Performance Testing as a discipline. Briefly explaining, not all projects scheduled in the Enterprise Release Management calendar undergo Performance Testing. Hence some of them might be exempted by virtue of low criticality of the enhancement or the high maturity of the application. The ones that enter the performance test environment might run for varied durations, depending on the test preparation required or the performance issues identified, leading to multiple iterations of Performance Testing and tuning. Due to this, there is a high potential for effort overruns or under-utilization of resources depending on the situation. Given such challenges in a Shared Services Model, Fixed Price may not be a good fit since the scope of work cannot be defined certain enough upfront to determine the fixed effort and cost. Time & Material is a better choice, however it brings in tremendous administrative overhead for the client. Constantly estimating the effort for each Performance Testing request, planning the resources and adjusting the team size according to the workload is even more challenging in short Release Cycles such as monthly. Not to mention the knowledge management challenges in such a fluid operating model. While any pricing model can possibly be applied for the individual project engagements, it makes more operational and economic sense to leverage a higher maturity pricing mechanism. For example it is relevant to choose Unit-Based Pricing Model (UPM) for a Shared Services Operating Model due to the fluctuating rate of arrival of project requests, option of distributed locations and close-to-factory like program delivery. UPM breaks down the varying size and complexity of the projects into a uniform granular unit of work. The number of units of work per Monthly Release Cycle will be measured and tracked via a defined process. A pre-determined cost for each unit of work will be agreed upon between the client and vendor. This ensures the cost per Release Cycle is predictable for the client, at the same time contractually obligating the vendor to be accountable for the cost in terms of successful project delivery. 2015, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 3
This white paper explains an actual implementation of UPM for a Shared Services Performance Testing Delivery Engagement, for a large US based Asset Management and Insurance client. The following sections explain: y Constructs of the model y How the model works y What are the tracking and measurement mechanisms to be in place for the model to work y Benefits of implementing such a model for Performance Testing services. 2. WHAT IS UPM IN PERFORMANCE TESTING SERVICES? UPM is a mechanism by which the vendor charges the client for the amount of work done in a stipulated period of time (say 1 month), basis the agreed price for one unit of work successfully completed. In this particular client illustration, the unit of work in Performance Testing services was represented as the number of Performance Testing scripts successfully executed, inclusive of all the performance test cycle activities for that script such as requirements gathering, test planning, test design, test data setup, test script development, test execution and results analysis. The payment would be made only for successfully completed units of work i.e. scripts that were developed and tested successfully. One Unit of Work = One Performance Test Script successfully executed [including requirements gathering, test planning, test design, data setup, script development, test execution and results analysis] The Performance Testing scripts were classified as Simple, Medium and Complex on basis of the following criteria, as agreed with the client. Each type of script complexity was charged at a different price. Script Technology Number of Parameterization Elements Number of Transactions/ Operations Involved in Script Price Low Web Services < = 7 < 10 $ X Medium Web HTTP/ HTML 8 to 15 10 to 20 $ Y High Other than above > 15 > 20 $ Z The cost for a Performance Testing project was calculated as a product of the number of scripts in scope of the project and the cost per script based on complexity type. Cost for one Performance Testing request/ project = Number of scripts * Cost per script based on complexity type 2015, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 4
3. HOW DOES THE UPM WORKS? At this client site, a monthly Release Cycle was in place and the average workload every month used to be anywhere between 10 and 15 projects. Each project had to undergo the complete performance test lifecycle from reviewing the requirements and changes to the application to test execution and closure on time for the release to production. Broadly, the following 5 activities needed to be performed in every Release Cycle to operate and track the UPM. Sizing the Projects Though the Release Cycle was monthly, the performance test requests used to be raised to the Shared Services Team about 6 to 8 weeks in advance. The Performance Testing requests were raised in a pre-defined PT Request Form which consisted of all necessary inputs to arrive at the estimates and develop the test strategy. The assigned performance test engineer would review the PT Request Form and conduct a requirements review meeting with the project team, to confirm the number of business scenarios to be scripted, the number of tests to be done, clarify any understanding gaps and agree on the next steps. Cost Estimates Iteration1 Based on the information gathered on each of the in-scope scenarios, the performance test engineer would classify the scenarios into Simple, Medium and Complex in accordance with the script complexity criteria (defined in the previous section). An automated pricing calculator called Unit Price Tracker was developed which also served as the master of record for all key details about each Performance Testing script. The initial information gathered during requirements review would be input into the Unit Price Tracker to determine the initial cost forecast, which will be published to the project team. Following is a snippet of the Project Forecast sheet of the Unit Price Tracker. Release Month Project/ Application Name Area Projected Total No. of Projected # of Low Projected # of Medium Projected # of High Estimated cost of Low Estimated cost of Medium Estimated cost of High Mar Customer Protection RIT 41 22 14 3 XX YY ZZ Mar Express IRA RIT 2 4 2 3 XX YY ZZ Mar 2013 HAWC March Release RIT 46 25 16 4 XX YY ZZ Workload Measurement and Tracking In the Unit Price Tracker, a detailed spreadsheet called Detailed Report was maintained to collect the attributes of each script for all the projects in each monthly release. The individual performance test engineers were responsible for updating it on a daily basis which would be reviewed by the vendor Technical Manager with the client Manager on a weekly basis. Following is a snippet of the Detailed Report from the Unit Price Tracker. Release Month Mar Mar Mar Project/ Application Name Customer Protection Customer Protection Customer Protection Project ID Script Name (as in PT Dashboard) Script Type Script Information Number of Transactions # Parameterization Elements New/ Modified/ Re-used SAS0928004 User Security RSV1_logins WS 1 7 New Low SAS0928004 SAS0928004 User Security RSV1_logins_ site2user-image-phrase User Security RSV1_logins_ authenticate WS 1 6 New Low WS 1 8 New Low Script 2015, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 5
Cost Estimates Iteration2 As and when the script development was completed, the performance test engineers would update the Detailed Report with the actual values for the various parameters such as number of transactions, number of parameterization elements, etc. based on which the Unit Price Tracker would calculate the revised project pricing automatically in the Project Wise Cost worksheet. Following is a snippet of this worksheet Number of Used Project Cost Release Classification Project/ Application Name Project ID Month Re- New Modified Total Low Medium High Total Low Medium High Total Used Mar Customer Protection SAS0928004 35 0 0 35 30 4 1 35 XX YY ZZ TT Mar Express IRA SPA1008002 2 0 0 2 0 2 0 2 XX YY ZZ TT Mar 2013 HAWC March Release STO1019020 26 31 12 69 51 18 0 69 XX YY ZZ TT Monthly Cost Reconciliation Once the monthly release was completed and all the projects signed-off by the respective stakeholders, the monthly Invoice Amount derived automatically by the Unit Price Tracker, based on the number of Units/ successfully executed (grouped by projects) was published to the Enterprise Release Management team for review. Post review, the monthly cost details were shared with the individual projects for reconciliation with the two sets of estimates provided earlier. Once this was approved, the vendor would raise the invoice for the agreed amount in accordance with the work successfully delivered. Following is the snippet of the Monthly Project Cost Report sheet from the Unit Price Tracker. Monthly Project Cost Report - Actuals January February March April Project Completed 0 0 7 3 Test Run Statistics 0 0 Environment Readiness (Smoke) 0 0 15 15 Load 0 0 6 6 Stress/ Scalability 0 0 0 0 Load/ Longevity 0 0 3 1 Volume 0 0 0 0 Failover 0 0 0 0 Total No. of Test Runs Executed 0 0 24 22 Classification # of Low 0 0 83 56 # of Medium 0 0 58 44 # of High 0 0 3 12 Total No. of Test Used 0 0 144 112 Cost Cost for Low 0 0 XX XX Cost for Medium 0 0 YY YY Cost for High 0 0 XX XX Total Cost 0 0 TT TT 2015, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 6
4. MONTHLY BILLING CYCLE PROCESS FLOW Following is the process flow for the Monthly Billing Cycle as per the Unit-Based Pricing mechanism, adapted for Performance Testing services. Timelines T T + 5 days Sign-off date + 1 day Sign-off date + 3 days Project Team Send the PT Request Form to Client PT Lead Estimation Completed Actual Cost Reconciliated Monthly Costing Completed Client PT Lead Acknowledge the PT Request and assign to Vendor Provide the Cost Estimate to the Project Team Provide the Actual Cost to the Project Team Review and acknowledge the total monthly project cost for Invoicing Vendor PT Lead Nominate the PT resource for the project Project the estimated project cost in Projects Forecast (in Unit Price Tracker) Review scripts details in Detailed Report Obtain the Actual Project Cost from Project Wise Cost sheet Update Test Run details based on data from Metrics DB Obtain monthly total cost from Monthly Project Cost Report Vendor PT Engineer Vendor PT Team Review the PT Request Form and setup Requirements Review Meeting with the Project Team Obtain the number of Scenarios to be Scripted Develop the Performance Test Plan Daily update of script details in Detailed Rep in Unit Price Tracker Develop the Performance Test Secure Performance Test Results Sign-offs Execute Performance Tests Sign-off date + 3 days Key Inference The Risk Assessment exercise illustrated here was conducted by 2 non-functional consultants over a 6 weeks period at the client site. Given the effort it takes to conduct such a comprehensive exercise, it may not be a feasible solution for short-term projects, low complex applications or highly matured systems. However for software product vendors and applications with large scale internet usage with unanticipated workload patterns and business growth scenarios, the qualitative and the quantitative benefits of conducting a Non-Functional Risk Assessment exercise clearly outweigh this modest effort and investment. 5. PRE-REQUISITES FOR IMPLEMENTING UPM Though UPM is applicable in any customer context, it is recommended to ensure the following five key pre-requisites before embarking on such an initiative. It is immensely critical to gain adequate understanding of the customer s ways of working, in terms of the types of performance test requests received, Release Cycle followed, complexity of the applications, performance test methodology adapted, environment and data dependencies, various stakeholders involved, etc., before commencing the implementation of UPM. A comprehensive Performance Testing process should be in place which is repeatable and aligned with the UPM. Unless the customer process is robust and matured, implementing an efficient pricing mechanism such as UPM would be an overhead and may result as counter-productive. 2015, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 7
Even though the working model exists per se, a certain degree of customization is required to ensure a successful implementation. Script complexity criteria should be baselined for any given customer context in accordance with the technology protocol, size of the projects and the tools used or any other factor deemed fit. The billing process flow should be tailored to the client s release management process, including the process on the client side to chargeback their internal customers which should be thought through and developed upfront. The Operations Manager on the client side should be knowledgeable on how UPM functions and how the Unit Price Tracker works. Multiple detailed orientation sessions should be conducted for all the key stakeholders. This is essential to ensure the cost reconciliation process is smooth between the client and the vendor and also between the client s Performance Testing team and their internal customers and stakeholders. 6. BENEFITS Following were the key benefits reaped by the client by engaging the vendor in the UPM for Performance Testing Services: y Complete control over OPEX enabled by the pay per use nature of the UPM y Near accurate cost predictability thus ensuring easy and fast budget approvals y Minimal administrative effort in invoicing thus increasing bandwidth for other governance tasks y Shared accountability of the Unit Price Tracker enhancing the transparency and trust in the vendor 7. CONCLUSION The operating and pricing model is one of the key indicators of the maturity of the client-vendor relationship. It is representative of the deep domain and operational knowledge of the vendor, and the trust developed with the client. Any Performance Testing team providing enterprise-wide Performance Testing services across multiple consecutive releases should definitely explore implementing the UPM. This eliminates significant amount of administrative overhead for the client allowing bandwidth to focus on the technical tasks. At the same time, the vendor management team is able to clearly map the vendor output to the expense incurred. Such predictability and accountability brings confidence to both the client and vendor management that the operating model is scalable. 2015, HCL TECHNOLOGIES. REPRODUCTION PROHIBITED. THIS DOCUMENT IS PROTECTED UNDER COPYRIGHT BY THE AUTHOR, ALL RIGHTS RESERVED. 8
ABOUT HCL About HCL Technologies HCL Technologies is a leading global IT services company working with clients in the areas that impact and redefine the core of their businesses. Since its emergence on the global landscape, and after its IPO in 1999, HCL has focused on transformational outsourcing, underlined by innovation and value creation, offering an integrated portfolio of services including software-led IT solutions, remote infrastructure management, engineering and R&D services and business services. HCL leverages its extensive global offshore infrastructure and network of offices in 31 countries to provide holistic, multi-service delivery in key industry verticals including Financial Services, Manufacturing, Consumer Services, Public Services and Healthcare & Life sciences. HCL takes pride in its philosophy of Employees First, Customers Second which empowers its 104,184 transformers to create real value for customers. HCL Technologies, along with its subsidiaries, had consolidated revenues of US$ 5.8 billion, for the Financial Year ended as on 31st March 2015 (on LTM basis). For more information, please visit www.hcltech.com About HCL Enterprise HCL is a $6.8 billion leading global technology and IT enterprise comprising two companies listed in India HCL Technologies and HCL Infosystems. Founded in 1976, HCL is one of India s original IT garage start-ups. A pioneer of modern computing, HCL is a global transformational enterprise today. Its range of offerings includes product engineering, custom & package applications, BPO, IT infrastructure services, IT hardware, systems integration, and distribution of information and communications technology (ICT) products across a wide range of focused industry verticals. The HCL team consists of over 109,643 professionals of diverse nationalities, who operate from 31 countries including over 505 points of presence in India. HCL has partnerships with several leading global 1000 firms, including leading IT and technology firms. For more information, please visit www.hcl.com Hello there! I am an Ideapreneur. I believe that sustainable business outcomes are driven by relationships nurtured through values like trust, transparency and flexibility. I respect the contract, but believe in going beyond through collaboration, applied innovation and new generation partnership models that put your interest above everything else. Right now 105,000 Ideapreneurs are in a Relationship Beyond the Contract with 500 customers in 31 countries. How can I help you?