A Forrester Total Economic Impact Study Prepared For Codenomicon The Total Economic Impact Of Codenomicon s Defensics Security Testing Suite

Transcription

1 A Forrester Total Economic Impact Study Prepared For Codenomicon The Total Economic Impact Of Codenomicon s Defensics Security Testing Suite Project Director: Michael Speyer July 2012

2 TABLE OF CONTENTS Executive Summary... 2 Defensics Reduces Software Remediation Expenses... 2 Factors Affecting Benefits And Costs... 3 Disclosures... 3 TEI Framework And Methodology... 5 Analysis... 6 Interview Highlights... 6 Costs... 9 Benefits Flexibility Risk Financial Summary Codenomicon Defensics: Overview Appendix A: Total Economic Impact Overview Appendix B: Glossary Appendix C: Endnotes , Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change. Forrester, Technographics, Forrester Wave, RoleView, TechRadar, and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. For additional information, go to About Forrester Consulting Forrester Consulting provides independent and objective research-based consulting to help leaders succeed in their organizations. Ranging in scope from a short strategy session to custom projects, Forrester s Consulting services connect you directly with research analysts who apply expert insight to your specific business challenges. For more information, visit Page 1

3 Executive Summary In July 2012, Codenomicon commissioned Forrester Consulting to examine the total economic impact and potential return on investment (ROI) enterprises may realize by deploying Defensics. The purpose of this study is to provide readers with a framework to evaluate the potential financial impact of Defensics on their organizations. Defensics Reduces Software Remediation Expenses Our interviews with two representatives from an existing customer (a large network equipment vendor) and subsequent financial analysis found that the organization we interviewed experienced the risk-adjusted ROI, costs, and benefits shown in Table 1. We found that the organization: Recovered its first-year investment and costs with the initial testing of a large application, where 40 defects and potential vulnerabilities were found. Found 110 defects and potential security vulnerabilities in the first year of ownership. Based on its current testing methodology with Defensics, the organization is anticipated to find 250 potential security vulnerabilities over three years. Is able to satisfy the requirement of a key customer segment of providing security validation with Defensics. Table 1 Three-Year Risk-Adjusted ROI 1 ROI Payback period Total benefits (PV) Total costs (PV) Net present value 176% 0.5 $2,583,997 ($937,759) $1,646,237 Benefits. The organization we interviewed experienced the following benefits: o o Remediation costs avoided of $2,157,827. These savings represent the difference between remediating defects that are found during testing and defects that are found after the software is released to customers. The savings shown are the cumulative savings of three separate testing efforts, carried out at different times and in different parts of the organization. Test script development effort avoided of $404,592. This represents that expense that would have been incurred to develop test scripts for unique or edge cases required by the organization s customers. Defensics automated test features allowed the organization to avoid developing custom test scripts. Page 2

4 o Training expense avoided of $21,578. This represents the cost that would have been incurred to train engineers to perform security testing using alternate solutions. Defensics ease of use allowed the organization to avoid incurring training expenses. Costs. The organization we interviewed experienced the following costs: o o o License and subscription fees of $598,553. These are the expenses paid by the organization to Codenomicon for the protocol suites that it uses. Professional services fees of $37,800. Professional services were needed to integrate Defensics into the organization s security testing methodology and to provide initial training for the security test engineers. The professional services were provided by Codenomicon. Labor to perform testing with Defensics of $301,406. This represents the total labor effort that the organization used to perform testing with Defensics over the three-year analysis period. Factors Affecting Benefits And Costs Table 1 illustrates the risk-adjusted financial results that were achieved by the interviewed organization. The riskadjusted values take into account any potential uncertainty or variance that exists in estimating the costs and benefits, which produces more conservative estimates. The following factors may affect the financial results that an organization may experience: All remediation costs avoided will depend on the labor effort needed to remediate defects, the number of defects found, the phase in the security development life cycle where defects are found, and the salaries that engineers are paid. The cost avoided for edge case test script development will depend on the number of edge cases that require testing and the effort required to develop test scripts. The cost avoided for security testing training will depend on the number of employees that will require training, and the number of hours needed to perform on-the-job training. The professional services fees will vary with each organization security testing methodology and amount of initial training required. Labor to perform testing with Defensics will vary with the overall frequency at which testing done. Disclosures The reader should be aware of the following: The study is commissioned by Codenomicon and delivered by the Forrester Consulting group. Page 3

5 Forrester makes no assumptions as to the potential ROI that other organizations will receive. Forrester strongly advises that readers should use their own estimates within the framework provided in the report to determine the appropriateness of an investment in Codenomicon/Defensics. Codenomicon reviewed and provided feedback to Forrester, but Forrester maintains editorial control over the study and its findings and does not accept changes to the study that contradict Forrester s findings or obscure the meaning of the study. The customer names for the interviews were provided by Codenomicon. Page 4

6 TEI Framework And Methodology Introduction From the information provided in the interviews, Forrester has constructed a Total Economic Impact framework for those organizations considering implementing Defensics. The objective of the framework is to identify the cost, benefit, flexibility, and risk factors that affect the investment decision. Approach And Methodology Forrester took a multistep approach to evaluate the impact that Defensics can have on an organization (see Figure 2). Specifically, we: Interviewed Codenomicon marketing and sales personnel and Forrester analysts to gather data relative to Defensics and the marketplace for fuzzy testing technologies. Interviewed one organization currently using Defensics to obtain data with respect to costs, benefits, and risks. Constructed a financial model representative of the interviews using the TEI methodology. The financial model is populated with the cost and benefit data obtained from the interviews as applied to the composite organization. Figure 1 TEI Approach Perform due diligence Conduct customer interviews Construct financial model using TEI framework Write case study Forrester employed four fundamental elements of TEI in modeling Defensics: 1. Costs. 2. Benefits to the entire organization. 3. Flexibility. 4. Risk. Given the increasing sophistication that enterprises have regarding ROI analyses related to IT investments, Forrester s TEI methodology serves the purpose of providing a complete picture of the total economic impact of purchase decisions. Please see Appendix A for additional information on the TEI methodology. Page 5

7 Analysis Interview Highlights For this study, we interviewed two representatives from a large network equipment manufacturer. The first representative worked in product-line engineering. In this capacity, the representative and his team used Defensics to perform post-development testing and to analyze competitor s products. The second representative worked in release testing where Defensics is used for product delivery testing and solutions-based testing for existing customer deployments. Both representatives used Defensics for detecting unknown software vulnerabilities using a black-box testing scenario. The organization uses test equipment from other vendors to perform other types of testing (e.g., regression testing) and in some instances to augment Defensics. The organization had been using Defensics for two years. From the two interviews, we learned that using Defensics allowed the organization to: Reduce its software remediation expense by finding critical defects earlier in the security development life cycle. Defects not detected in post-development or prerelease testing could potentially result in post-release remediation expense. This expense can quickly become large for many reasons including the effort needed to replicate the defect in the lab and the subsequent efforts to develop, test, and release a fix. Reduce software defect remediation expense for customers. For the organization s customers, the cost of remediation includes time spent with the organization s technical support engineers, patch installation, and testing. For telco customers in particular, there is the additional challenge of finding a suitable window to apply the fix, the potential cost of network downtime, and the impact on the telco s brand and customers. Increase agility associated with testing edge cases. To test edge cases (e.g., a rarely used capability that a single customer needs or a completely new feature that has never been tested before) would normally require building a test environment, designing the test, and then writing the test script. With Defensics, the organization simply applied the tool to the edge case, thus avoiding the time and effort needed to design and build out a test environment. Improve the stability of released software. The ability of Defensics to generate almost-unlimited test scenarios allowed for easy identification of malformed packets that could potentially crash a piece of equipment. The ability to identify and remediate these instances contributes to the overall stability of the product under test. This gives the organization an additional sense of security knowing that its equipment will be able to gracefully handle malformed packets that may be encountered in live networks. According to the representative that we interviewed, As soon as you put a product into production in the real world, anything that can happen will happen. And you really, really have to ensure that the device doesn t fall over or crash. Easily share and recreate test findings. Defensics ability to recreate test scenarios and its built-in documentation facilities allows the organization s testers to pass identified problems to other users, who can then easily replicate the test scenario so that it can be analyzed and a fix can be created. According to the representative we interviewed, Once you find the bug, it s easy to fix. Finding it is the hard part. Page 6

8 Use engineers with little security expertise to perform testing. We learned that to be successful at using network testing tools requires a certain level of security expertise. The organization found that Defensics makes it easy for nonsecurity experts (e.g., quality assurance engineers) to run tests. This avoids having to wait for skilled security resources to become available to perform testing. Also, Defensics ease of use allowed the company to avoid recurring training expenses. Respond to customers that require Defensics prevalidation before deploying a new code release. Some of the organization s customers require that new code releases be tested with Defensics before they will deploy it. The organization can address this requirement, thus ensuring smooth deployment of new code releases among its customers. Framework Assumptions Table 2 provides the model assumptions that Forrester used in this analysis. Table 2 Model Assumptions Ref. Metric Calculation Value A1 A2 Number of working hours annually Average developer/engineer annual salary 2,080 $96,000 A3 Benefits above salary 25% A4 Developer/engineer fully loaded salary (US salaries) A2*(1+A3) $120,000 A5 Developer/engineer hourly rate A4/A1 $57.69 A6 Number of labor hours needed to fix a defect 100 The discount rate used in the present value (PV) and net present value (NPV) calculations is 10%, and the time horizon used for the financial modeling is three years. Organizations typically use discount rates between 8% and 16% based on their current environment. Readers are urged to consult with their respective company s finance department to determine the most appropriate discount rate to use within their own organizations. Page 7

9 Relative Cost Of Remediating Software Defects To estimate the value of the benefits experienced by the organization, it was necessary for Forrester to estimate the relative cost of remediating software defects at different stages in the security development life cycle (SDL). For simplicity, we divided the SDL into three stages: coding/development, testing, release/maintenance (see Figure 2). We recognize that other more complex models exist, which include a design phase and multiple test phases. Figure 2 SDL Relative Costs Of Remediating Security Vulnerabilities 2 Coding / Development (1x) Test (5x 7x) Release / Maintenance (10x 15x) For this analysis, we assume that: The coding/development phase is when initial software development takes place. We assume that the cost of remediating security vulnerabilities in this phase is 1x. The test phase is where the organization we interviewed used Defensics for detecting security vulnerabilities and software defects. In practice, the actual testing may have to be done closer to the coding phase or closer to the release/maintenance phase. We assume that the cost of remediating security vulnerabilities in this phase is 5x to 7x greater than in the coding phase. The release/maintenance phase is post-release or after software has been released to customers. The cost of remediating software defects in this phase is assumed to be 10x to 15x greater than in the coding phase. It costs 2x to 3x as much to remediate vulnerabilities in the release/maintenance phase than in the test phase. We have used US labor rates when evaluating the cost of labor. An organization may experience different labor rates if they use engineers in different parts of the world. From the interviews, we learned that the organization s average effort needed to remediate a defect or software bug is highly variable, depending on the nature of the defect and where in the test cycle it s found. For this analysis, we assume that it takes on average 100 labor hours to remediate a defect found in testing. The labor hours come from developers, testers, and build costs. If the defect were found after the software was released to customers, the remediation expense would include additional labor from field engineers, account teams, and technical support engineers. Page 8

10 Costs The costs associated with deploying and using Defensics are: Software license and subscription fees. Professional services fees to integrate Defensics into the security development life cycle and to train testers. Labor to perform tests with Defensics. Software Subscription And License Fees Defensics software is sold in suites, where each suite has the code needed to test different communications protocols. The software is sold as either a perpetual license or an annual subscription fee. The organization uses between 40 and 50 suites, purchased with a mix of perpetual license and annual subscriptions. The annual software expense is $240,687 and represents that interviewed organization s environment (see Table 3). All pricing was supplied to Forrester by Codenomicon. Table 3 Annual Software Subscription Fees Ref. Metric Calculation Initial Year 1 Year 2 Year 3 Total B1 Bt Annual subscription and license fees Annual subscription and license fees $240,687 B1 $0 $240,687 $240,687 $240,687 Spread 0% 100% 100% 100% Bto Total (original) $0 ($240,687) ($240,687) ($240,687) ($722,061) Page 9

11 Professional Services Fees When it first began using Defensics, the organization used professional services from Codenomicon to help integrate testing with Defensics into SDL and to provide training. The one-time professional service fee was $35,000 (see Table 4). Table 4 Professional Services Fee Ref. Metric Calculation Initial Year 1 Year 2 Year 3 Total C1 Test training and SDL integration consulting $35,000 Ct Professional services fees C1 $35,000 $0 $0 $0 Spread 100% 0% 0% 0% Cto Total (original) ($35,000) $0 $0 $0 ($35,000) Page 10

12 Labor To Perform Testing At the time of writing, Defensics was used by two separate teams in different parts of the organization. Testing tasks were shared among multiple engineers, with no single person being assigned to perform testing with Defensics. We learned that Defensics requires relatively little labor to set up, run, and analyze results. To calculate the total labor effort needed to use Defensics, we assumed that each team used the equivalent of 0.5 fulltime equivalents (FTEs) annually to perform testing. Assuming a fully loaded salary of $156,000, the annual labor cost for testing with Defensics is $120,000 (see Table 5). Table 5 Labor Hours Needed To Perform Software Testing With Defensics Ref. Metric Calculation Initial Year 1 Year 2 Year 3 Total D1 D2 Number of FTEs used to perform postdevelopment testing Number of FTEs used to perform prerelease testing D3 FTE fully loaded salary A4 $120,000 Dt Labor to perform testing with Defensics (D1+D2)*D3 $0 $120,000 $120,000 $120,000 Spread 0% 100% 100% 100% Dto Total (original) $0 ($120,000) ($120,000) ($120,000) ($360,000) Page 11

13 Total Costs The total costs for acquiring and operating Defensics is $1,117,061 (see Table 6). Table 6 Total Costs Ref. Cost Category Initial Year 1 Year 2 Year 3 Total Bto Annual subscription and license fees $0 ($240,687) ($240,687) ($240,687) ($722,061) Cto Professional services fees ($35,000) $0 $0 $0 ($35,000) Dto Labor to perform testing with Defensics $0 ($120,000) ($120,000) ($120,000) ($360,000) Total costs (original) ($35,000) ($360,687) ($360,687) ($360,687) ($1,117,061) Benefits The financially quantifiable benefits that the organization experienced when using Defensics fall into the following categories: Security vulnerabilities remediation expenses avoided. This is the difference between costs that would have been incurred to remediate bugs and vulnerabilities after the software was released and the costs needed to remediate bugs that were found during testing. Test engineer training expenses avoided. This is the cost that would have been incurred to train engineers in security testing practices. Test script development expenses avoided. This is the cost that would have been incurred to develop test scripts for unique or edge cases. Based on the interview data, we assume: The labor effort needed to remediate a defect found in testing Defensics is 100 hours. The fully loaded hourly labor cost is $ Remediation Cost Avoided Initial Security Vulnerability Detection We learned that the initial (i.e., first) test using Defensics of a critical application discovered approximately 90 potential vulnerabilities that had not been previously identified. Upon examination, it was determined that a portion of these vulnerabilities were actually intended features and not vulnerabilities or bugs. This initial test run was regarded as Page 12

14 highly valuable because of the number of new potential vulnerabilities discovered. The discovery and subsequent remediation process ultimately led to more robust software. To estimate the value of this benefit, we assumed that 40 of the 90 vulnerabilities needed to be addressed. As noted earlier, the cost difference between remediating vulnerabilities in the test and maintenance phases is 2x. This yields a one-time savings of $576,923 (see Table 7). Table 7 Remediation Cost Avoided Initial Vulnerability Detection Ref. Metric Calculation Initial Year 1 Year 2 Year 3 Total E1 E2 E3 E4 Number of vulnerabilities found during first test run Number of labor hours needed to remediate defect in development Engineer average hourly rate Cost difference between fixing vulnerabilities in testing and post-release 40 A6 100 A5 $ Et Remediation cost avoided initial security vulnerabilities detection E1*E2*E3*E4 $576,923 $576,923 $576,923 Spread 100% 0% 0% Eto Total (original) $576,923 $0 $0 $576,923 Remediation Cost Avoided Vulnerabilities Detected In Prerelease Testing In this instance, Defensics is used by a team of engineers that work in a release testing (analogous to quality assurance) group. In particular, this group does testing on new releases in vertical configurations. The code is loaded onto functional devices that are integrated into a fully meshed network environment that mimics a real-world customer environment. This allows the team to test in a simulated customer environment and to satisfy customers' demand for software testing to be done with Defensics before they will deploy it. We learned that this test team discovers between three and nine defects per quarter. For this analysis, we will assume five defects per quarter are discovered. Assuming 100 labor hours per defect and a labor rate of $57.69 per hour, the annual savings in remediation expense is $865,385 (see Table 8). Page 13

15 Table 8 Remediation Cost Avoided Vulnerabilities Detected In Prerelease Testing Ref. Metric Calculation Initial Year 1 Year 2 Year 3 Total F1 Number of vulnerabilities found during functional testing (5 defects per quarter)*4 20 F2 F3 F4 Number of labor hours needed to remediate defect post-release Engineer average hourly rate Cost difference between fixing vulnerabilities in testing and post-release A6 100 A5 $ Ft Remediation cost avoided defect detected in prerelease testing F1*F2*F3*F4 $288,462 $288,462 $288,462 Spread 100% 100% 100% Fto Total (original) $288,462 $288,462 $288,462 $865,385 Remediation Cost Avoided Vulnerabilities Detected In Post-Development Testing We learned that functional testing was done post-development by engineers in the technical marketing engineering team. The testing task was shared among six engineers. This team typically performed its testing post-development on new code releases (as opposed to patches) and tested a broad range of equipment. The testing done by this team is completely independent of testing done by the release testing team. Because of the team s agility and proximity to the organization s customers, it often performed testing in response to unique customer needs. On average, this test team discovers between two and three vulnerabilities per month. Based on data from the organization, we assumed that on average 2.5 vulnerabilities per month (30 per year) were discovered by the product line engineering team that needed remediation. Assuming 100 labor hours per defect and a labor rate of $57.69/hour, the annual savings in remediation expense is $692,308 (see Table 9). Page 14

16 Table 9 Remediation Cost Avoided Vulnerabilities Detected In Post-Development Testing Ref. Metric Calculation Initial Year 1 Year 2 Year 3 Total G1 Number of vulnerabilities found during postdevelopment testing (2.5 defects per month)*12 30 G2 Number of labor hours needed to remediate defect A6 100 G3 G4 Engineer average hourly rate Cost difference between fixing vulnerabilities in testing and post-release A5 $ Gt Remediation cost avoided vulnerabilities detected in post-development testing G1*G2*G3*G4 $432,692 $432,692 $432, % 100% 100% Gto Total (original) $432,692 $432,692 $432,692 $1,298,077 Edge Case Test Script Development Cost Avoided The organization s customers often have unique or one-of-a-kind requirements. The organization refers to these as edge cases. The types of customers that have edge case requirements are service providers or large financial institutions whose network operations act like service providers. The organization develops software to fulfill these customers edge case requirements. Like all of the organization s software, the edge cases need to be tested for vulnerabilities and stability. We learned that testing an edge case requires a setup that is unique for each case, and part of this setup is the need to develop custom test scripts. Developing these scripts is extremely labor intensive, taking many hours of meetings, planning, and script development. The organization found that when using Defensics, it was able to do away with most of the test script development effort. Defensics allows the organization to automate the test process, without having to write custom scripts. To calculate the value of this benefit, we assume the organization tests 20 edge cases annually, requiring 150 hours on average per case to prepare test scripts. Using Defensics, the test script development effort drops to zero. Assuming a labor rate of $57.69 per hour, the annual savings in developing edge case test scripts is $519,231 (see Table 10). Page 15

17 Table 10 Cost Avoided Edge Case Test Script Development Ref. Metric Calculation Initial Year 1 Year 2 Year 3 Total H1 H2 H3 Number of edge cases that require testing annually Number of hours needed to plan and prepare for an edge test Engineer average hourly rate A5 $57.69 Ht Cost avoided edge case test script development H1*H2*H3 $173,077 $173,077 $173,077 Spread 100% 100% 100% Hto Total (original) $173,077 $173,077 $173,077 $519,231 Test Engineer Training Expense Avoided We learned from the one of the interviewees that testing software usually requires an engineer with knowledge of security testing and communications protocols. At this organization, the job of testing with Defensics is shared among multiple engineers and is done on as-needed basis. This means that all of the engineers in the test team need to know how to use Defensics. Since the engineers rotate through different parts of the organization, the organization needs to constantly train new engineers on security testing and using Defensics. This, in theory, would place a large training burden on the organization. The organization found that Defensics ease of use and feature set allows engineers with little or no security test experience to perform testing without any formal training. This saves the organization a recurring training expense. To evaluate this benefit, we assume that eight new engineers would require training annually. We also assume that all training is done on the job (i.e., no course work). The organization estimates that it would take about 20 hours to learn how to perform testing using alternate methods (not Defensics). Assuming a labor rate of $57.69/hour, the annual savings in developing edge case test scripts is $9,231 (see Table 11). Page 16

18 Table 11 Test Engineer Training Expense Avoided Ref. Metric Calculation Initial Year 1 Year 2 Year 3 Total I1 I2 I3 It Number of engineers requiring training annually Number of hours needed for training Engineer average hourly rate Cost avoided training for security testing 8 20 A5 $57.69 I1*I2*I3 $9,231 $9,231 $9,231 Spread 100% 100% 100% Ito Total (original) $9,231 $9,231 $9,231 $27,692 Page 17

19 Total Benefits The total benefits experienced by the organization are $4,931,538 (see Table 12). Table 12 Total Benefits Ref. Benefit Category Initial Year 1 Year 2 Year 3 Total Eto Fto Gto Hto Remediation cost avoided initial security vulnerabilities detection Remediation cost avoided defect detected in functional testing Remediation cost avoided vulnerabilities detected in prerelease testing Cost avoided edge case test script development 576, , , , , , , , ,692 1,298, , , , ,231 Ito Cost avoided training for security testing 9,231 9,231 9,231 27,692 Total benefits (original) $1,480,385 $903,462 $903,462 $3,287,308 Flexibility Flexibility, as defined by TEI, represents an investment in additional capacity or capability that could be turned into business benefit for some future additional investment. This provides an organization with the right or the ability to engage in future initiatives but not the obligation to do so. There are multiple scenarios in which a customer might choose to implement Defensics and later realize additional uses and business opportunities. Flexibility would also be quantified when evaluated as part of a specific project (described in more detail in Appendix A). At the time of writing, the organization was strongly considering using Defensics in the code development stage. While this would require potential investment in licenses, training, and integration, it would generate considerable future value if defects and bugs were discovered and remediated during the coding phase. Risk Forrester defines two types of risk associated with this analysis: implementation risk and impact risk. Implementation risk is the risk that a proposed investment in Defensics may deviate from the original or expected requirements, resulting in higher costs than anticipated. Impact risk refers to the risk that the business or technology needs of the organization may not be met by the investment in Defensics, resulting in lower overall total benefits. The greater the uncertainty, the wider the potential range of outcomes for cost and benefit estimates. Page 18

20 Quantitatively capturing investment and impact risk by directly adjusting the financial estimates results in more meaningful and accurate estimates and a more accurate projection of the ROI. In general, risks affect costs by raising the original estimates, and they affect benefits by reducing the original estimates. The risk-adjusted numbers should be taken as realistic expectations since they represent the expected values considering risk. The following implementation risks that affect costs are identified as part of this analysis: The professional services fees will vary with each organization security testing methodology and amount of initial training required. Labor to perform testing with Defensics will vary with the overall frequency at which testing is done. The following impact risks that affect benefits are identified as part of the analysis: All remediation costs avoided will depend on the labor effort needed to remediate defects, the number of defects found, and the phase in the SDL where defects are found. The cost avoided for edge case test script development will depend on the number of edge cases that require testing and the effort required to develop test scripts. The cost avoided for security testing training will depend on the number of employees that will require training and the number of hours needed to perform on-the-job training. The following impact risks that affect benefits are identified but are not included in this analysis: Labor rates for remediating defects could be significant lower if non-us labor is used. This will lower the overall value of the benefits received. Table 13 shows the values used to adjust for risk and uncertainty in the cost and benefit estimates. The TEI model uses a triangular distribution method to calculate risk-adjusted values. To construct the distribution, it is necessary to first estimate the low, most likely, and high values that could occur within the current environment. The risk-adjusted value is the mean of the distribution of those points. Readers are urged to apply their own risk ranges based on their own degree of confidence in the cost and benefit estimates. Page 19

21 Table 13 Cost And Benefit Risk Adjustments Costs Low Most likely High Mean Professional services fees 100% 100% 125% 108% Labor to perform testing with Defensics 98% 100% 105% 101% Benefits Low Most likely High Mean Remediation cost avoided initial security vulnerabilities detection Remediation cost avoided defect detected in prerelease testing Remediation cost avoided vulnerabilities detected in postdevelopment testing 50% 100% 120% 90% 80% 100% 103% 94% 80% 100% 103% 94% Cost avoided edge case test script development 80% 100% 103% 94% Cost avoided training for security testing 80% 100% 103% 94% Readers are urged to apply their own risk ranges based on their own degree of confidence in the cost and benefit estimates. Page 20

22 Financial Summary The financial results calculated in the Costs and Benefits sections can be used to determine the ROI, NPV, and payback period for the organization s investment in Defensics. These are shown in Table 14. Table 14 Cash Flow Non-Risk-Adjusted Cash flow original estimates Initial Year 1 Year 2 Year 3 Total Present value Costs ($35,000) ($360,687) ($360,687) ($360,687) ($1,117,061) ($931,975) Benefits $1,480,385 $903,462 $903,462 $3,287,308 $2,771,251 Net benefits ($35,000) $1,119,698 $542,775 $542,775 $2,170,247 $1,839,275 ROI 197% Payback period 0.4 months Table 15 below shows the risk-adjusted ROI, NPV, and payback period values. These values are determined by applying the risk-adjustment values from Table 13 in the Risk section to the cost and benefits numbers in Tables 6 and 12. Table 15 Cash Flow Risk-Adjusted Cash flow risk-adjusted estimates Initial Year 1 Year 2 Year 3 Total Present value Costs ($37,800) ($361,887) ($361,887) ($361,887) ($1,123,461) ($937,759) Benefits $1,368,485 $849,254 $849,254 $3,066,992 $2,583,997 Net benefits ($37,800) $1,006,598 $487,367 $487,367 $1,943,531 $1,646,237 ROI 176% Payback period 0.5 months Page 21

23 Codenomicon Defensics: Overview Defensics is a fully automated fuzzing test solution which systematically sends invalid or unexpected inputs to the system under test in order to expose software defects. Defensics model based fuzz testing modules are available in over 200 standard network protocols. These tests can be further enhanced with Defensics Universal and Traffic Capture fuzzing modules, with which can fuzz any network protocol, service interface or application file format. Defensics allows organizations to: Proactively discover exploitable zero day (0-day) vulnerabilities. Create test cases automatically, obviating the need for manual test case creation. Use its interface to guide users through every step of the testing process. A command-line interface that supports third party tools and scripts is also available. Use the reporting system to share detailed test results, and to identify and recreate specific problems or failed test cases. Appendix A: Total Economic Impact Overview Total Economic Impact is a methodology developed by Forrester Research that enhances a company s technology decision-making processes and assists vendors in communicating the value proposition of their products and services to clients. The TEI methodology helps companies demonstrate, justify, and realize the tangible value of IT initiatives to both senior management and other key business stakeholders. The TEI methodology consists of four components to evaluate investment value: benefits, costs, risks, and flexibility. Benefits Benefits represent the value delivered to the user organization IT and/or business units by the proposed product or project. Often product or project justification exercises focus just on IT cost and cost reduction, leaving little room to analyze the effect of the technology on the entire organization. The TEI methodology and the resulting financial model place equal weight on the measure of benefits and the measure of costs, allowing for a full examination of the effect of the technology on the entire organization. Calculation of benefit estimates involves a clear dialogue with the user organization to understand the specific value that is created. In addition, Forrester also requires that there be a clear line of accountability established between the measurement and justification of benefit estimates after the project has been completed. This ensures that benefit estimates tie back directly to the bottom line. Costs Costs represent the investment necessary to capture the value, or benefits, of the proposed project. IT or the business units may incur costs in the form of fully burdened labor, subcontractors, or materials. Costs consider all the investments and expenses necessary to deliver the proposed value. In addition, the cost category within TEI captures Page 22

24 any incremental costs over the existing environment for ongoing costs associated with the solution. All costs must be tied to the benefits that are created. Risk Risk measures the uncertainty of benefit and cost estimates contained within the investment. Uncertainty is measured in two ways: 1) the likelihood that the cost and benefit estimates will meet the original projections, and 2) the likelihood that the estimates will be measured and tracked over time. TEI applies a probability density function known as triangular distribution to the values entered. At a minimum, three values are calculated to estimate the underlying range around each cost and benefit. Flexibility Within the TEI methodology, direct benefits represent one part of the investment value. While direct benefits can typically be the primary way to justify a project, Forrester believes that organizations should be able to measure the strategic value of an investment. Flexibility represents the value that can be obtained for some future additional investment building on top of the initial investment already made. For instance, an investment in an enterprisewide upgrade of an office productivity suite can potentially increase standardization (to increase efficiency) and reduce licensing costs. However, an embedded collaboration feature may translate to greater worker productivity if activated. The collaboration can only be used with additional investment in training at some future point in time. However, having the ability to capture that benefit has a present value that can be estimated. The flexibility component of TEI captures that value. Appendix B: Glossary Discount rate: The interest rate used in cash flow analysis to take into account the time value of money. Although the Federal Reserve Bank sets a discount rate, companies often set a discount rate based on their business and investment environment. Forrester assumes a yearly discount rate of 10% for this analysis. Organizations typically use discount rates between 8% and 16% based on their current environment. Readers are urged to consult their respective organization to determine the most appropriate discount rate to use in their own environment. Net present value (NPV): The present or current value of (discounted) future net cash flows given an interest rate (the discount rate). A positive project NPV normally indicates that the investment should be made, unless other projects have higher NPVs. Present value (PV): The present or current value of (discounted) cost and benefit estimates given at an interest rate (the discount rate). The PV of costs and benefits feed into the total NPV of cash flows. Payback period: The breakeven point for an investment. This is the point in time at which net benefits (benefits minus costs) equal initial investment or cost. Return on investment (ROI): A measure of a project s expected return in percentage terms. ROI is calculated by dividing net benefits (benefits minus costs) by costs. Page 23

25 A Note On Cash Flow Tables The following is a note on the cash flow tables used in this study (see the example table below). The initial investment column contains costs incurred at time 0 or at the beginning of Year 1. Those costs are not discounted. All other cash flows in Years 1 through 3 are discounted using the discount rate (shown in Framework Assumptions section) at the end of the year. PV calculations are calculated for each total cost and benefit estimate. NPV calculations are not calculated until the summary tables and are the sum of the initial investment and the discounted cash flows in each year. Table [Example] Example Table Ref. Category Calculation Initial cost Year 1 Year 2 Year 3 Total Appendix C: Endnotes 1 Forrester risk-adjusts the summary financial metrics to take into account the potential uncertainty of the cost and benefit estimates. For more information on Risk, please see page The data for calculating the relative costs of remediating software bugs was obtained from The Economic Impacts of Inadequate Infrastructure for Software Testing. Source: The Economic Impacts of Inadequate Infrastructure for Software Testing, National Institute of Standards & Technology, May ( Page 24