Executive Summary Introduction And Survey Methodology For Many, Application Security Is Not Yet A Mature Practice... 5

Transcription

1

2 Executive Summary... 2 Introduction And Survey Methodology... 3 For Many, Application Security Is Not Yet A Mature Practice... 5 From Design To Production, Software Security Practices Need To Improve... 8 There Is A Silver Lining Looming Key Recommendations Appendix A: Methodology Appendix B: Demographics Appendix C: Endnotes , Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change. Forrester, Technographics, Forrester Wave, RoleView, TechRadar, and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. For additional information, go to [1-HMGX0Z] Page 1

3 In November 2010, Microsoft commissioned Forrester Consulting to conduct a survey study of 150 North American software development influencers. The purpose of the study is to understand the current state of application security development practices and identify key trends and market directions for application security. From the late 1920s to his final arrest in 1952, Willie Sutton robbed more than 100 banks and stole more than $2 million. When a reporter asked him why he robbed banks, he responded, "Because that's where the money is." Today, the money is in software applications that s where companies process their most sensitive data from credit card numbers to customer and employee information as well as trade secrets. Identity, financial, and intellectual property theft fuel an underground cyber economy estimated to be in the billions of dollars. Our survey found that while a majority of organizations have implemented some form of application security measures, very few have put in place an end-to-end strategic approach that incorporates security throughout the software development life cycle. In fact, the data suggests that organizations typically choose to transfer risk from development to operations, where the remediation cost for vulnerabilities are the highest. Our respondents also told us that the top driver for application security adoption is currently compliance. Beyond that, the lack of time and lack of management support present major challenges for application security. We also found that when it comes to application security, most organizations employ tactical measures and point technologies; few subscribe to a holistic, prescriptive application security methodology. Similarly, not many organizations employ metrics to track the success of application security initiatives. In fact, 72% of everyone who answered the survey told us that their developers are not measured with security-related metrics, and 57% do not send their security requirements downstream to guide quality and security testing. Because of these reasons, the road to better application security is not an easy one along the way, loss of efficiency is common as is the lack of measurable, positive return on investment (ROI). Looking forward, as companies grapple with a more menacing and capable threat landscape, a growing set of regulations and third-party requirements, and an unprecedented level of IT upheaval, they will have no choice but to improve their application security postures. If organizations do not integrate security and privacy into the development practices from the earliest stages, addressing it later will not only be more expensive but also could be ineffective altogether. In this case, companies may find more things than just their applications at risk. Forrester s study yielded these key findings: Application security is not a mature practice for many. The study revealed that a significant percentage of respondents do not employ a coordinated end-to-end approach to application security throughout their development life cycle. Our data shows that 47% do not perform acceptance tests for third-party code, and Page 2

4 30% do not use static analysis or manual code review during development. Another 27% do not practice secure design, and 19% fail to carry out security requirement definition. Even fewer managed to coordinate their security practices in various stages of the development life cycle. In addition, 46% reported that they follow a homegrown application security methodology instead of those that have been independently validated and accepted by the general community, such as a security development life cycle (SDL) and the capability maturity model (CMM). More specifically, the survey data show that 15% of the respondents use the SDL methodology. 1 Twenty percent follow CMM/CMMI. 2 In general, application security remains a tactical concern versus a strategic initiative. Representing 25% of all of the votes, our respondents picked compliance as the top argument for convincing management to invest in application security. Whenever compliance is the main driver, organizations tend to do the bare minimum needed to become compliant, rather than focusing on best practices and long-term objectives. This also suggests that many fail to see the link between application security and business objectives, such as lower support overheads and reduced late-stage development costs, because compliance is what it took for them to treat application security seriously. Accountability and incentives to promote secure software development are lacking. Sixty-one percent said that their company had no special incentive programs to encourage developers and testers working together. In addition, more than 70% reported that their organizations do not measure developers with security-related metrics. Without proper incentives or performance metrics, it s no surprise that application security practices are immature. Those employing a more coordinated, prescriptive approach to application security saw more positive ROIs. Although the population practicing coordinated application security in our respondent pool was not a majority, it did report more positive ROIs for application security than the rest. Organizations that desire to improve their application security competency should treat application security strategically, not tactically integrating security practices throughout the development life cycle, adopting industry-recognized methodologies, incentivizing and measuring developers for security, and tying security to overall business objectives. Application security refers to the mechanisms and processes that help to identify and remediate security vulnerabilities in software applications. These include, but are not limited to, secure design, code-level analysis, code scanning, fuzzing, and penetration testing. In November 2010, Microsoft commissioned Forrester Consulting to conduct a survey study of North American software development influencers. The purpose of the study is to understand the current state of application security practices and identify key trends and market directions for application security. In this study, we surveyed 150 respondents from the US and Canada. Sixty-four percent are in the high-tech industry. Ten percent are financial services and insurance firms. The rest spread across healthcare, Page 3

5 manufacturing, business services, and the public sector (see Figure 1). Those in the high-tech industry include platform vendors (55%), independent software vendors (13%), original equipment manufacturers (11%), original design manufacturers (9%), and value-added resellers (4%). Which industry vertical are you from?" High tech 64% Financial services and insurance 10% Utilities and telecommunications 8% Healthcare and biotech 7% Other 4% Manufacturing 3% Business services and construction 2% Government, education institution, public sector 2% We drew our survey respondents from companies in many verticals with at least $500 million in annual revenues. All respondents are either technologists or managers who are directly involved with their company s application development processes. Of all respondents, sixty-three percent are software developers, 10% are development managers, and the rest are distributed across architects, quality-assurance (QA) personnel, security testing, project managers, product managers, and company executives (see Figure 2). Readers who are interested in a more detailed description of respondent profiles should refer to Appendix A. We conducted the online survey between November and December 2010, with 26 questions spanning software development processes, security mechanisms, metrics, and organizational structures. Page 4

6 Which of the following most closely reflects your job function in relation to IT/software development?" Company executive or business decision-makers, 5% Project management responsible for software development, quality, and security, 5% Product management responsible for software product development, 1% Security testing, 1% Software quality assurance (including testing), 5% Software architect, 9% Development manager, 10% Software development, 63% The first goal of the study is to understand how mature application security is as a general practice. We define maturity as follows: A security practice is mature when it is well defined and able to respond proactively to emerging threats, with established technologies, well-known best practices, and established metrics to measure and track performance. We asked the respondents which application security mechanisms they have deployed within their development life cycle. The answers suggest that many organizations do not practice security consistently across the life cycle (see Figure 3). As an example, 47% reported that they do not perform security acceptance tests for third-party code, 30% do not conduct static analysis on their code, 27% do not practice threat modeling and usage scenario reviews during application design, and 19% do not gather security requirements based on policies or risk assessment. Page 5

7 Percent of respondents who said they do not currently employ the following measures for improving code security in their organization A prescriptive security incident response plan or operational security plan for production code 3% Developer (tester) training 5% Accountability and incentive structures to promote software security 14% Quality or security gate in testing 17% Risk or policy-based security requirement definition 19% Archive release environments and activities as part of a secure release process 21% Threat modeling and usage scenario review 27% Static analysis 30% Stringent security tests prior to acceptance of third-party code 47% When we asked our respondents what was the most effective argument to convince their management to invest in software security, the top answer was to meet our compliance requirements 25% of our respondents selected this answer (see Figure 4). Regulations like those in the payment card industry (PCI), the Sarbanes Oxley (SOX) Act, and the Health Insurance Portability and Accountability Act (HIPAA) have requirements that either specifically call for the use of software security mechanisms or indirectly do so through the mandate for vulnerability management. It s not surprising to us that compliance is a big driver. But regulations typically lag behind threats, and they therefore rarely represent the best practices. In fact, we often see applications that are deemed to be compliant to various regulations being vulnerable to attacks. As a general rule of thumb, whenever compliance is cited as the top (and sometimes the sole) driver, it means that the industry has not taken on the treatment of the problem in a proactive or strategic manner. In other words, the practices are not yet mature. Page 6

8 What is the most effective argument in convincing your management to invest in software security measures during the development life cycle? To meet our compliance requirements 25% It is cheaper to fix bugs earlier in the development life cycle 21% Economic impact of security breaches 13% A risk-centric argument 12% Customers require us to demonstrate secure development practices 12% We haven t convinced our management to invest in software security measures yet 6% It s a competitive differentiator in the market 6% Don t know We need greater accountability for third-party code and greater transparency for our software supply chain 1% 3% It is therefore not surprising when asked which prescriptive or process-based application security methodology they use in development that our respondents returned answers that point to a decidedly tactical practice. Forty-six percent said that they follow some form of homegrown security methodologies. The CMM and SDL methodologies are ranked a distinct second and third (see Table 1). 3 Fourteen percent reported that they do not follow any prescriptive methodologies. Page 7

9 Which prescriptive or process-based software security methodology do you use?" Software security % of respondents We have developed our own software security methodology 46% CMM or CMMI 20% SDL 15% Other 2% OpenSAMM 1% DISA STIG (for operations) We do not use any such methodology 1% 14% Don t know 16% In addition, when asked whether they use any list-based approaches to achieve software security, nearly 50% said that they use a list of approved development tools and option settings (e.g., compiler options). A secure development guideline was also a popular choice, garnering 42% of the votes. Surprisingly, OWASP top 10 and SANS top 25 did not receive many mentions. This, combined with the fact that 46% use homegrown application security methodologies, suggests that an overwhelming majority of companies treat application security with approaches of varying quality few are tackling the issues systematically and in a measured way. In the study, we asked in-depth questions of individual application security mechanisms, spanning the entire software life cycle, from requirement generation to production. Our intention is to study practitioner priorities Page 8

10 as well as the rationale and motivations behind them. The questions provide rich analysis of an organization s software security efforts, adoption strategies, challenges that they face, and efficacy of current practices. A detailed study of the data reveals that even though some practices are more mature and more widely adopted than others, overall application security remains an uneven field in software development houses. More specifically: Security requirement definition practices are not yet mature. We asked our respondents to rank the effectiveness of various security practices in the requirement definition phase. The majority of them, 54%, believe that having a comprehensive set of security and privacy policies to start the requirement definition is an effective practice for improving code security. Beyond that, however, 50% do not believe in the benefits of using a risk-based approach to drive security requirements. Similarly, 57% do not believe in communicating security requirements downstream to guide testing and QA efforts (See Figure 5). The data directly shows the gap between development and testing as well as the lack of risk-based approaches. For security and privacy requirements definition, please rate how effective the following items are for improving security of the code and reducing downstream work that arises due to security vulnerabilities. Most effective (5) Least effective (1) We do not currently have this 1% Start the requirements definition phase with comprehensive security and privacy policies 27% 27% 22% 13% 9% Derive security and privacy requirements based on the application risk profile, the likelihood of threats, and your risk tolerance level 19% 31% 29% 12% 2% 7% Send the security and privacy requirements downstream for security testing 18% 25% 24% 17% 3% 13% Derive requirements exclusively from industry standards (e.g., OWASP top 10, SANS top 25, BSIMM) 9% 18% 30% 17% 7% 18% 0% 25% 50% 75% 100% The majority of them do not see the merit of security design practices. Earlier data shows that 27% of organizations are not doing threat modeling and usage scenario reviews during design (see Figure 3). When asked whether typical design-phase security measures were effective to promote code security, Page 9

11 52% said that they do not believe in the effectiveness of risk-based architecture views, and 59% do not believe in threat modeling and usage scenario review. This is yet another data point that suggests the lack of risk-based approaches (or at least the lack of those that are being used effectively) in security design. Source code management and manual code review dominate the implementation phase. In the implementation phase, our respondents selected source code management, manual code reviews, and a secure coding guideline as the most effective security measures. In the security community, however, technologies like static analysis and, to some extent, fuzzing performed by developers are widely regarded as far more effective in discovering application vulnerabilities. But our survey data shows that these technologies are not widely deployed (see Figure 6). This result, perhaps, is in part due to the fact that most application security activities are driven by compliance, which may emphasize secure coding guidelines (as in PCI) and source code management. One can also infer from this data that organizations are investing in easy-to-implement application security measures, such as secure coding guidelines and manual reviews, but technologies like static analysis that may require more extensive changes to existing development processes are not being adopted. For security measures taken during implementation, please rate how effective the following items are for improving security of the code and reducing downstream work that arises due to security vulnerabilities. Most effective (5) Least effective (1) We do not currently employ this type of measure Systematic source code management and tracking 20% 32% 29% 12% 3% 3% Manual code reviews 20% 32% 30% 14% 3% 1% A secure coding guideline 19% 29% 29% 1% 13% 9% A library of approved or banned functions 11% 24% 30% 13% 1% 21% Manual penetration testing 11% 28% 29% 14% 6% 12% Developers using penetration testing tools (e.g., black-box scanning) 9% 28% 25% 16% 3% 18% Developers performing fuzzing tests on the code 9% 22% 36% 18% 2% 12% Static analysis technologies 8% 25% 24% 10% 3% 30% Binary code analysis 4% 10% 18% 20% 7% 41% 0% 25% 50% 75% 100% Page 10

12 Organizations are more adept at vulnerability management practices. For those respondents who indicated that they remediate at least certain security flaws prior to release, we asked what process they follow to triage and prioritize vulnerability remediation. Sixty percent of all respondents said that they use a riskbased approach in vulnerability triage and for the prioritization of remediation. This is encouraging, especially considering how little risk-based approaches are used in other parts of the development life cycle. Organizations equate secure release with final penetration tests and a formal sign-off. When asked about their code release practices and relevant security measures, 52% find the use of a formal sign-off step (to indicate that all secure measures have been met) to be most effective, and 49% find the most value in a final penetration test prior to release. Only 40% saw the merit of archiving pertinent information and data for the sake of future incident and bug responses. Typically, one would archive specifications, source code, binary builds, documentation, incident response plans, and licensing and service terms (if third-party software). Archiving the information allows a baseline for effective incident triage and response. Again, this result points to the lack of mature security considerations in the respondents software release plan. Secure operation measures are widely practiced. For servicing software in production, we asked the respondents to rate the effectiveness of various security measures. The majority of them reported that they saw the benefits of an established bug-reporting channel, application monitoring/diagnostics mechanisms, and a regular, well-communicated software patch program. The fact that more organizations are practicing secure operations but substantially fewer of them are incorporating security consistently into other parts of the development life cycle suggests that organizations choose to transfer risk from development to operations, where remediation costs are the highest. 4 We also looked at whether organizations institute special accountability and incentive structures to promote software security initiatives and whether these measures indeed help organizations in achieving their goals. We found that: There exists a great divide between development and test/qa. In our interaction with software development and software security professionals, Forrester sees much inefficiency in the way development and test teams typically interact. Often there is much time wasted on going back and forth between development and testing on whether a test finding is a true software bug. This inefficiency underlines many of the challenges for software security you need to change the existing development processes if you want to produce efficient and effective implementation of security measures. We asked our respondents whether their companies use any special incentive programs to encourage development and testing personnel to work together in a more efficient fashion. Sixty-one percent said that they had no such program. Only 43% reported some form of organizational process to strengthen the working ties between development and testing (see Figure 7). Page 11

13 What incentive mechanism do you use to ensure that your development team and testing teams work closely together?" We have no special incentive mechanism to encourage development and test work together 61% We pair developers and testers and measure the combined unit with a set of quality metrics, including security metrics. This ensures that they work together to achieve overall software quality objectives 23% We have testers rate how testable a developer s code is and reward the developer accordingly 11% We have developers rate the test results (e.g., how many false positives, how easy it is to digest the test outcome) and reward testers accordingly 9% Don t know 5% Other 1% Developers are not incentivized to care about security. When asked whether their developers are measured with security-related metrics, 72% of all respondents and, more specifically, 76% of all developers who answered the survey said no. This is a testament that, at a larger scale, developers are often not given the proper incentives to respect security initiatives and goals. Not every organization tracks and measures security ROI. We asked our respondents whether they use any concrete metrics to calculate ROI for incorporating security measures within software development. Nineteen percent said that they do not use any such metrics. Twenty-six percent reported the use of only software time to market as the ROI metric. Our study uncovered an interesting data point that highlights the different perspectives of (hence attitude thereof) the different stakeholders regarding application security. When asked what the top challenges are in implementing a secure development program, developers, development managers, project managers, and testers all cited lack of time to perform security tasks as a top challenge. However, among the seven executives Page 12

14 who answered our survey, zero selected lack of time as a challenge. Rather, they cited lack of security expertise and lack of funding as top challenges. This data is most telling, as it suggests a potential chicken-and-egg problem: If executives do not realize that lack of time is a real concern, they will continue to set goals for development that are non-commesurate with security, such as emphasizing time to market above all things. As a direct consequence, this will continue to encourage developers to overlook security for other performance metrics. As a result, executives view of developers lacking security expertise might be perpetuated. The data in our study, from various fronts, painted a picture of a software industry that does not yet use mature security practices. Many of the top challenges facing developers and application security professionals, including the lack of time and lack of funding, arise from failure to align application security objectives with business goals. In an effort to understand the business impact of employing application security mechanisms, we asked our respondents what ROI metrics they keep and whether they ve seen positive ROI resulting from application security deployment (see Figure 8). As shown, 26% of those who track the amount of developers time spent in post-development bug chasing and remediation saw that the time decreased as a result of implementing software security practices, while another 24% saw that the metric increased. Similarly, 40% (out of 42 that kept track of testers time spent in regression testing) observed an increase in this metric, while 19% saw that it decreased. Page 13

15 What impact has the implementation of software security during development had on the following metrics? Increased No change Decreased Don t know Opportunity cost for developer time (N = 2) 38% 25% 9% 28% Time to market for the software (N = 36) 53% 25% 14% 8% Amount of human time spent in incident response, patch release, and customer servicing (N = 57) 23% 32% 30% 16% Amount of tester s time spent on regression testing (N = 42) 40% 31% 19% 10% Amount of developer s time in post-development bugchasing and remediation (N = 68) 24% 35% 26% 15% 0% 25% 50% 75% 100% The overall ROI metrics do not show clear evidence that there is positive ROI across the board for incorporating security within development. However, when we took a deeper look at some of the data points with correlated analysis, we saw a few interesting and promising results. Those who practice SDL saw better ROI. We took a look at the set of the population using prescriptive application security methodologies. It turns out that those practicing SDL specifically reported visibly better ROI results than the overall population. Unlike point technologies, SDL advocates a coordinated approach to application security throughout the life cycle, and its emphasis is on a set of processes that supports such coordination. We can potentially extrapolate that a coordinated approach to application security is what drives positive ROI. Table 2 shows the ROI with noticeable differences (the other metrics are similar). Those who tie development and testing together effectively saw better ROI. Two practices that have seen particular success in practice have to do with integrating development and testing processes. The first one includes establishing a common quality and security standard between development and the testing team that stipulates what constitutes a flaw or a security flaw. Such a common standard eliminates Page 14

16 disagreements and wasted efforts back and forth between development and testing. The second practice involves communicating security and privacy requirements downstream directly to testing, so that testing can be more targeted and also work from the same set of expectations as development. When we looked at the ROI results reported by those respondents who have implemented either practices effectively (this is the set that had the practices and also rated the practices being effective ), we see that these ROI results are significantly better than those reported by the overall population (see Tables 3-5). In particular, note that within those who established effective common standards across development and testing, a fairly significant percentage, 46% to be exact, reported seeing a decrease in developer time spent on postdevelopment bug remediation. This is in contrast with the 26% overall that reported a decrease in the same metric. Security ROI comparison between SDL users and the overall population ROI metric SDL users Overall Amount of developer s time in postdevelopment bug-chasing and remediation 4 out of 12 reported decreased 18 out of 68 reported decreased Amount of human time spent in incident response, patch release, and customer servicing 7 out of 13 reported decreased 17 out of 57 reported decreased Page 15

17 ROI metric Those who send security and privacy requirements downstream to testing Overall Amount of developer s time in postdevelopment bug-chasing and remediation 37% reported decreased (N=37) 26% reported decreased (N=68) Amount of human time spent in incident response, patch release, and customer servicing 37% reported decreased (N=30) 30% reported decreased (N=57) ROI metric Those who send security and privacy requirements downstream to testing Overall Amount of developer s time in postdevelopment bug-chasing and remediation 37% reported decreased (N=35) 26% reported decreased (N=68) Amount of human time spent in incident response, patch release, and customer servicing 37% reported decreased (N=30) 30% reported decreased (N=57) Page 16

18 ROI metric Those who send security and privacy requirements downstream to testing Overall Amount of tester s time spent on regression testing 7 out of 24 reported decreased 8 out of 42 reported decreased Those who have implemented effective security measures report more positive ROI. If we only look at the ROI results reported by those who rated their security measures as very effective or effective, we also see better ROI results than those reported by the overall population. Figure 9 and Figure 10 show the comparative results. In both figures, the subset of populations who have implemented effective security measures reported better RoI results than the overall survey population. In Figure 9, notice that a much larger percentage of those who practice effective security maintenance reported seeing a decrease in developers time spent on post-development bug-chasing work than reported by the overall population. Similar results can be seen with human time spent on incident response. Page 17

19 Types of respondents who reported a decrease in the amount of the developer s time in post-development bugchasing and remediation Those who practice effective secure maintenance of production code (N = 40) 38% Those who have an effecitve security and privacy requirement gathering process (N = 38) 34% Those who have effective accountability and incentive structures (N = 33) 33% Those who practice effective secure release (N = 40) 33% Those who have effective secure design practices (N = 40) 33% Those who practice effective secure implementation (N = 47) 32% Those who practice effective security or quality testing (N = 35) 31% Overall (N = 68) 26% Page 18

20 Types of respondents who reported a decrease in the amount of the tester s time spent on regression testing Those who have effective accountability and incentive structures (N = 21) 7 Those who have an effecitve security and privacy requirement gathering process (N = 21) 6 Those who practice effective secure maintenance of production code (N = 27) 7 Those who have effective secure design practices (N = 27) 7 Those who practice effective secure release (N = 24) 6 Those who practice effective secure implementation (N = 31) 6 Those who practice effective security or quality testing (N = 22) 4 Overall (N = 42) 8 The RoI metrics results, as shown above, are one of the first concrete examples that show visible difference between those who practice application security in a coordinated way and as an integral part of software development lifecycle versus those who do not. Some of the data referenced above is not statistically significant. As such, we caution readers against making general statements beyond this study. Nonetheless, they are interesting data points that are worth pondering over. Page 19

21 Page 20

22 In this study, Forrester conducted an online survey of 150 North American software development influencers and decision-makers in the US and Canada. Survey participants included leads in engineering, development, product management, and product strategies. The study began in November 2010 and was completed in December Canada, 19% US, 81% Page 21

23 Which industry vertical are you from?" High tech 64% Financial services and insurance 10% Utilities and telecommunications 8% Healthcare and biotech 7% Other 4% Manufacturing 3% Business services and construction 2% Government, education institution, public sector 2% Page 22

24 Which of the following most closely describes your company s total annual revenue?" $499 million USD or less 0% $500 million to $999 million USD 16% $1 billion to $2.49 billion USD 15% $2.5 billion to $5 billion USD 11% More than $5 billion USD 58% Page 23

25 Does your firm conduct software development to supply either company business needs or customer needs (e.g., selling software products)? Yes, we develop software products or services 91% Yes, we develop the software components for nonsoftware products 36% Yes, we are a software outsourcer or a software platform provider (our customers may be development shops themselves) 23% We rarely do software development in-house 0% No, we don t do any software development 0% Don t know 0% Page 24

26 How are you involved in the software development efforts in your company (or within your department/organization)?" I m directly involved with software development, quality assurance, and/or software security; part of a team delivering systems and products 76% I manage a software development team (or teams) delivering systems and products 11% I m indirectly, but significantly involved, with software development (e.g., reviewing deliverables and influencing software development teams and decisions 8% I am senior management or a decision-maker for an organization that does software development 3% My role is compliance, and I help to set our software development standards and practices based on compliance requirements 2% Page 25

27 How would you characterize your firm s current practices for software security?" We have established extensive software security technologies, processes, and metrics to ensure the production of secure software 57% We care about software security primarily because of compliance reasons; our security measures are driven by compliance 23% We have allocated significant investment for software security tools and technologies but haven t built out all of the relevant organizational processes and metrics Our software security methodologies and strategies for software security are still forming, and our practices at present are on an as-needed basis 10% 9% We care about software security in theory but don t currently use any software security measures or allocate any investment for them 1% 1 Microsoft pioneered the security development life cycle work. The Microsoft SDL documentation and process description can be found on Microsoft s website. Source: Microsoft ( 2 CMM is a process improvement methodology that is broadly applicable to a diverse set of problems. It includes security elements but is not designed to be a secure development methodology. The description of CMM and CMMI can be found at Carnegie Mellon University s website. Source: Carnegie Mellon University ( 3 It should be noted that the CMMI methodology (CMMI now super cedes CMM) is designed to assess the maturity of a development organization from a broad set of aspects. The methodology does include certain elements for software security, but is not designed specifically to be a secure development methodology. As such, it does not include a comprehensive set of secure development practices. Page 26

28 4 A NIST study found that fixing bugs earlier in the development life cycle is significantly cheaper than waiting until later stages in the life cycle. Source: Gregory Tassey, Ph.D. The Economic Impacts of Inadequate Infrastructure for Software Testing, May 2002 ( Page 27