Results Based Testing

Transcription

1 Using Gain-Sharing Pricing Model for Testing Ayal Zylberman Summary Results Based Testing (RBT) is a new software testing pricing model that sets forth the expected value to be delivered by the Testing Teams. Imagine your testing provider making a commitment to find 97% of the defects in the system. Can that be measured? What would that be worth? What would be the impact on the customer-provider relationships and accountability? How can you bind that commitment? This white paper will attempt to answer those questions. Introduction Very often the question what budget should I allocate for testing is asked. The answer to this question is somewhere between zero and infinity. There are many elements that define the budget for testing, such as complexity of the system under test, amount of iterations of testing and of software releases, quality of development and of unit testing, efforts spent by development on present release and past releases and more. However, an important factor for defining the testing budget is the level of risk the organization is willing to take. No risk means we need to produce a bug-free system (and therefore the testing budget will be close to infinity). You may spend one day to test a system and if you re a good tester, you will probably find lots of bugs or you could spend 3 months with a group of 50 testers to test the same system. The difference would be with the amount of defects you find and more important the amount of defects you DON T find When to use Results Based Testing? Results Based Testing usually involves three elements: 1. A Scope of work 2. A contractual SLA 3. A pricing mechanism RBT is normally utilized when part or all of the software testing process is outsourced to a third party and a core contractual SLA together with a pricing mechanism sets the exact payment made at each SLA level. The pricing mechanism may be a flexible rate for each SLA level or a Penalty/Reward mechanism, all with the goal of creating an incentive for the Testing Supplier to meet the business targets (results) set forth. However, RBT may (and should) also be used for internal testing teams even though in such cases a penalty/reward mechanism is harder to implement. Another good use of RBT is for establishing the necessary framework for measured continuous improvement in which results from previous periods can serve as a baseline for following period s targets. What results should we measure? When evaluating the level of testing, several Key Process Indicators (KPIs) should be measured. However, in order to keep things simple, the main focus should be on two main questions: 1. What percentage of the defects should be found by testing? P a g e 1

2 2. What is the cost spent in order to achieve the above goal? Although it is clearly important to be able to answer the above questions, most organizations fail to measure these two KPIs and thus, are unable to provide accurate visibility into the quality and efficiency of testing. Below we will present a simple method for measuring test coverage which will enable us to answer the first question and a method for setting a budget for achieving these results, thus answering the second question. There are many other indicators such as staff level expertise, attrition level, reporting to stakeholders, communication with development teams, delays in the process and more. However, all other aspects feed directly into the two main questions above. For example, if the level of staff is not sufficiently high, the percentage of defects found will decrease and/or the cost of testing will increase. How to measure Test Coverage KPI? In order to measure the percentage of defects found by testing (a type of test Coverage KPI vs. the escaped defects KPI) the organization should utilize the following process: 1. Reporting of defects each defect reported by the testing team should be documented in a central defect management system. 2. All issues or support tickets raised by customers/users of the system should be documented in a centralized system. Usually the support or help-desk team has this information. 3. Each ticket should be evaluated by the testing team (sometimes the support team filters the tickets and provides only the tickets that result from a defect). 4. Each ticket related to a defect should have one of the following statuses: Not a defect Known defect Cannot be found by testing/un-reproducible New Defect Only defects in the last status (New Defect) are counted for this metric. The process above, although not being used by most organizations, is extremely important in case the organization is looking to start implementing RBT and to continually improve the efficiency and effectiveness of the testing process. Measuring the test coverage is done by dividing the amount of defects found by the testing provider with the amount of defects found by the users of the system. Since critical defects carry a different importance to the organization versus less severe defects, each defect is multiplied by its severity. For example, assuming a scale of 1-5 is used, a critical defect (severity = 5) will be counted in the same way as 5 minor defects (severity = 1). Only defects found in a certain period after release of the system shall be counted (normally this is defined as 3-6 months). Once the data is available the following formula is used to calculate the KPI value: P a g e 2

3 Testing Budget Calculator The Testing Budget Calculator is a formula used to define the testing budget required for testing a system or a version of a system. There is no one generic formula for calculating the testing budget for all types of systems since the total effort required depends on the organization and the system specific characteristics. First we need to identify the parameters that affect a testing budget. These factors might include: Overall development budget Development effort in man days Type of project which can be either a development project from the ground up or an Implementation of off the shelf business software) Estimated number of product versions to be released Scope of testing (Acceptance Test, Testing automation development, establishing a test Environment etc.) Efforts spent in previous versions Number of interfaces or integrated systems Functional Complexity Duration of testing Existing level of test automation infrastructure out of overall test coverage. When selecting the relevant parameters, the organization should verify it has access to the values of each parameter as any change in the value has implications on the testing effort required. It s nice to have sophisticated parameters like lines of code, but there s no use in using such parameters if there s no access to the actual values added in each version or in case the amount of lines does not change the total effort required for testing. The next step is creating a formula using all relevant parameters. It s important that the formula is as simple as possible allowing for certain assumptions based on past experience. An example of a simple calculator might be a set percentage of the development budget. Please see engagement examples in the examples section below. Results-Based Pricing Model for Testing In traditional software testing outsourcing engagements, the testing provider provides the service and the customer pays for the services, either at a fixed price, on a time-and-materials basis or a cost-plus model. Results Based testing is the only gain-sharing approach in which long term interests are perfectly aligned. This model encourages collaboration and creative problem-solving as both parties work toward common business goals. It also provides the supplier a greater freedom to determine how to better achieve the results. The pricing model The basic pricing model is Managed Testing Services based on a pre-defined rate card with only a certain amount paid (usually 70% of the total amount) while the final calculation of the cost is made once actual results are calculated. Budget Adjustments Example According to the difference between the actual hours multiplied by the rates and the expected budget that had been calculated in the Testing Budget Calculator, the following adjustments can be made: P a g e 3

4 1. If the actual cost of testing in a certain project is BELOW the budget as estimated in the Testing Calculator, the testing provider receives 25% from the difference between the actual cost and the estimated budget 2. If the actual cost of testing in a certain project is ABOVE the budget as estimated in the Testing Calculator, the provider pays a 25% penalty from the difference between the actual cost and the estimated budget Coverage Adjustment Example The following adjustments are made based on the actual test coverage KPI: - If the actual coverage meets the expected results a reward of up to 10% of the total actual cost is given to the provider - If the actual coverage is below the expected results a penalty of up to 30% of the total actual cost is given to the provider For example: assuming the coverage KPI target for all defects is 95%, the adjustment mechanism will be as following: Actual Coverage Penalty/Reward 0%-80% 30% penalty 80%-86% 20% penalty 86%-92% 10% penalty 92%-98% No penalty/reward 98%+ 10% reward Additional Adjustment can be defined based on other agreed and measured KPIs P a g e 4

5 Real Life Engagement Examples Large Bank Tier 1 Defense Contractor Telecom Equipment Manufacturer Size of testing team 120 testers 160 testers 140 testers Coverage 96% Coverage - all defects 97% Coverage 85% Main SLA Average delays in schedule 5% Coverage - critical defects 99% Average time to recruit new staff 21 days System up time 99% Satisfaction surveys 3.8/5 Attrition of key employees 5% Location QualiTest Test Factory QualiTest Test Factory QualiTest Test Factory # of parallel testing projects Average duration of each testing project Testing Budget Calculator Pricing model days 1-2 years 3-4 months Set % of development budget Fixed Prices based on the testing calculator Formula with 6 factors: Development man days, # of iterations, efforts in previous versions, ATP required and more T&M. If the actual cost is above Testing Budget, a 30% discounted rates applies If the actual cost is below budget 20% reward % of development man days T&M. If the actual cost is above Testing Budget, a 30% discounted rates applies Conclusion Results Based Testing enables organizations to: - Provide the testing provider with financial incentives to meet customer s business goals - Provide the testing provider with financial incentives to innovate and improve the process in line with customer s business goals and spreads the financial risk between both parties - Provides a framework for continuous improvement - Measures Test Provider s performance. - Provides the customer flexibility to scale the testing up or down in line with its business needs. P a g e 5