State of Software Security Report

Transcription

1 VOLUME 2 State of Software Security Report The Intractable Problem of Insecure Software September 22, 2010 Software Security Simplified

2 AS EVERY CIO AND CISO IS AWARE, the flood of news generated by attacks against insecure software continues unabated across all industry verticals and market segments. Since the publication of Volume 1 at the beginning of the year, there have been multiple new zero-day vulnerabilities reported in Microsoft Windows, at least six material data breaches, 10K filings from Intel disclosing a breach similar to the Chinese attack on Google, and widely covered security concerns about mobile apps, cloud service providers and SCADA systems. Yet despite this evidence that software security efforts are not keeping pace with attacks there is good news to report. It is heartening to see that CXO software security concerns are beginning to translate into concerted efforts to move from ad-hoc security testing to a new paradigm of application risk governance characterized by standardized processes and operating controls extended uniformly across the enterprise. Given the state of the application threat environment, it is not surprising that over 60% of all of Veracode enterprise customers are launching a formal, comprehensive security program for the very first time. It is this action that has driven the submission of nearly 1,400 new applications representing nearly 200% increase in the use of Veracode s cloud-based assessment service over the past reporting period. This report represents the code-level analysis of 2,922 applications (as compared to 1,591 applications in Volume 1), a sure sign that more development and security teams are taking the security of internally developed and third-party components and applications seriously. The data also shows that once vulnerabilities are detected and remediation advice is provided, developers are quick to achieve an acceptable level of security. And, when a class of software such as financial services applications makes security a priority it does appear that security quality improves, particularly with respect to common vulnerabilities such as Cross-site Scripting. When this evidence of progress is juxtaposed with my conversations with CIOs and CISOs who are awakening to the importance of security accountability across the software supply chain, I see a climate that is conducive to more secure software in the future. For you who are ready to act now, this report comprises security intelligence gleaned from billions of lines of code analyzed by the world s first and only cloud-based application risk management services platform. It is our hope that we can assist you to make and buy more secure software. Best Regards, Matthew Moynahan Chief Executive Officer, Veracode veracode.com/ceo-blog

3 Table of Contents Introduction Executive Summary Software Supply Chain Security of Applications Application Threat Space Trends Addendum Assurance Level Definitions

4 Introduction The State of Software Security is a semi-annual report that draws on continuously updated information in Veracode s cloud-based application risk management services platform. Unlike a survey, the data comes from actual code-level analysis of billions of lines of code and thousands of applications. The resulting security intelligence cannot be found anywhere else. It represents multiple testing methodologies (static binary, dynamic, and manual) on the full spectrum of application types (components, shared libraries, web and non-web applications) and programming languages (including Java, C/C++,.NET, ColdFusion, and PHP) from every part of the software supply chain (Internally Developed, Open Source, Outsourced, Commercial). For those executives, security and development professionals who want to better understand the vulnerabilities that threaten the integrity and performance of software in the software supply chain, this series of reports is essential reading. In Volume 2 of the State of Software Security there are nearly 1,400 more applications than in the inaugural report, reflecting the growing use of independent, cloud-based application risk management services. As before, the report first examines the security quality of applications by type of supplier in the software supply chain and then explores application security by language, industry, and by application type across both web and non-web applications. New in Volume 2 are data from third-party assessments, the first inclusion of PHP and ColdFusion applications, a comparison of static binary, dynamic, and manual testing effectiveness, and additional analytics on Financial industry applications. Veracode welcomes any questions or comments from readers and will continually strive to improve and enrich the quality and detail of our analysis. Additionally, we invite all members of the software supply chain to participate in constructive dialogue on the topic of software security at veracode.com/ceo-blog. 2

5 Executive Summary The following are some of the most significant findings in the State of Software Security Volume 2, representing 2,922 applications assessed in the last 18 months by Veracode SecurityReview, a cloud-based application risk management services platform. 1. More than half of all software failed to meet an acceptable level of security and 8 out of 10 web applications failed to comply with the OWASP Top Cross-site Scripting remains the most prevalent of all vulnerabilities 3. Third-party applications were found to have the lowest security quality 4. Developers repaired security vulnerabilities quickly 5. Suppliers of Cloud/Web applications were the most requested third-party assessments 6. No single method of application security testing is adequate by itself 7. The security quality of applications from Banks, Insurance, and Financial Services industries was not commensurate with their business criticality Key Findings 1. More than half of all software failed to meet an acceptable level of security and 8 out of 10 web applications failed to comply with the OWASP Top 10 57% of all applications were found to have unacceptable application security quality on first submission, even when standards were adjusted for applications considered less business critical (Figure 3). Even more troublesome, more than 80% of internally developed and commercial web applications failed to comply with the OWASP Top 10 (Figure 5), an industry standard list of critical web application errors. The level of risk in terms of repair costs, business continuity, and brand from so many business critical applications failing to meet an acceptable level of security on first submission is staggering. The potential exposure to brand reputation and loss of revenue from interruptions to business operations is significant. Recommendation: Utilize industry standards such as OWASP Top 10 and CWE/SANS Top 25 list of most dangerous software errors as minimum thresholds and compliance policies to which applications need to adhere. 2. Cross-site Scripting remains the most prevalent of all vulnerabilities Cross-site Scripting (XSS) remains the most prevalent vulnerability category, accounting for 51% of all vulnerabilities uncovered by Veracode s combined static binary, dynamic, and manual security testing methods (Figure 13)..NET applications, in particular, exhibited an abnormally high rate of Cross-site Scripting vulnerabilities, resulting from the use of.net controls that do not automatically encode output (Table 4). While not as numerous, Cryptographic Issues a category that includes unencrypted or inadequate encryption of data appeared in the most applications, with 41% of all applications containing one or more vulnerabilities in this category (Figure 14). These statistics underscore the need for developers to become better educated and better equipped to avoid common vulnerabilities. Recommendation: These flaws are easy to fix once found (Figure 4). Focusing on developer education and awareness is a cost-effective way to avoid introducing them. 3

6 3. Third-party applications were found to have the lowest security quality Third-party code is getting more attention since Veracode first highlighted in Volume 1 of this report, that between 30% and 70% of software submitted as internally developed contained identifiable third-party components. Both Safecode.org 1 and a report from the research firm Secunia 2 have recently reinforced the elevated risks associated with third-party software in the supply chain. In Figure 3, Veracode shows that applications from all types of third-party suppliers were less secure than Internally Developed applications on first submission. Third-party suppliers failed to achieve acceptable levels of security 81% of the time. However, in Figure 2 it is also evident that third-party code is an essential part of the every organization s portfolio, comprising 29% of all applications submitted to Veracode. Furthermore, between 20% and 37% of very high or high criticality applications are sourced from third-parties. Recommendation: Both internal and third-party components and applications must be subjected to the same level of security verification to ensure consistent security quality across the application portfolio. Procurement contracts for outsourced or commercial software vendors should insist upon the authority to perform independent security testing and specify minimum security acceptance criteria. 4. Developers repaired security vulnerabilities quickly A common misperception is that it is easy to find defects and difficult to fix them. While this may often be true of functional defects in software it is less true for security defects. Observing a variance from functional specifications is relatively easy but determining the root cause can be hard. Conversely, determining that an application allows someone to do something it was never intended to do is actually quite difficult but relatively easy to fix once known (Figure 4). Among the most encouraging data in this report, the evidence that development teams using Veracode can fully remediate unacceptable levels of security quality in only 16 days and 1.1 resubmissions on average is among the best reasons to equip development teams with effective security testing and training they can and did improve the state of software security quickly when properly informed. Recommendation: Equip development teams with the appropriate application security resources and knowledge and plan for security verification and remediation in the project timeline from the outset. 5. Cloud/Web applications were the most requested third-party assessments Assessments of third-party applications at the request of a purchasing organization have grown linearly over the past 6 quarters, reflecting the increased concern over the security of software in the supply chain and the availability of effective, new technologies such as cloud-based, static binary analysis that make third-party assessments possible without requiring source code or tools. In a new section of the report, Veracode explored the types of applications most often reviewed by request. As Figure 8 shows, suppliers of cloud and web applications made up nearly 60% of all third-party assessments requested, while integrators and commercial software providers made up most of the rest in equal parts. Since cloud-based applications are relatively new, their significant presence indicates the reasonable security concerns they raise and the criticality of the work they perform. Like other third-party software, these assessments resulted in low levels of acceptable security and rapid remediation. Recommendation: Require Third-party Cloud/Web application and service providers to demonstrate verification of application security quality

7 6. No single method of application security testing is adequate by itself Others have reported this year on the inadequacy of web application scanning. 3 Veracode s code-level analysis of vulnerabilities using multiple testing techniques on the same applications confirms that dynamic web application scanning tools are not sufficient as the sole testing method. Similarly, manual penetration testing, while necessary to fully comply with policies such as the OWASP Top 10 and the CWE/SANS Top 25, lacks consistency of coverage and will rarely detect all instances of commonly occurring vulnerabilities. However, while the evidence shows that static binary analysis provides the most consistent breadth and depth of coverage, it is also true that not all design and business logic vulnerabilities are discoverable with static methods alone. Recommendation: CISOs and CIOs should view different testing techniques as operating controls that each play an important role in a comprehensive policy driven program. Multiple testing techniques should be adopted based on application business criticality and type of application. The use of multiple techniques is the only way to comply with industry standard security polices such as the OWASP Top 10 and the CWE/SANS Top 25 Most Dangerous Software Errors. 7. The security quality of applications from Banks, Insurance, and Financial Services industries was not commensurate with their business criticality In a very interesting dichotomy, Financial Industry applications were found to have the best raw code-level security scores of any industry but only average levels of acceptability when the business criticality of an application was considered. This speaks to the high degree of awareness such firms have about code-level threats but also to the inadequate application risk management practices employed relative to the importance of these applications. Financial Services applications in particular demonstrated an exceptionally low prevalence of the most common vulnerabilities less than half the rate of Cross-site Scripting errors as compared to Banks and Insurance (Table 7). The implication is that training, testing, and a high degree of focus on specific types of errors can make a significant difference. The net result is both encouraging because improvement is possible; and sobering because the most critical of applications remain too insecure. Recommendation: Inventory and classify the application inventory based on business criticality. In the absence of this business context, an understanding of the code-level security quality is insufficient. What seems to be good code-level security quality may still not render the application fit for purpose when business criticality is taken into account

8 Software Supply Chain While people tend to think that software is written from scratch, modern economics and productivity imperatives have long since changed the reality. Today software is truly a composition of code originating from multiple sources across the world and most organizations rely on third party software suppliers for critical applications. In this section we examine the security quality of software produced by the software supply chain most often found in organizations: Internally Developed, Commercial, Open Source, and Outsourced. Only by understanding the various degrees of software security quality produced by supply chain participants can we begin to understand the requirements to change policies and processes, properly manage application risk in organizations, and protect critical software infrastructure. For CIOs and CISOs, the evidence continues to point to an increasing percentage of software infrastructure and associated liability coming from unknown and unmanaged third-parties. For CIOs and CISOs, the evidence continues to point to an increasing percentage of software infrastructure and associated liability coming from unknown and unmanaged third-parties. While nearly a third of all applications submitted to Veracode were identified as third-party, code-level analysis reveals that third-party code in the supply chain is significantly understated by most organizations. Veracode sampling found as much as 76% of code submitted as Internally Developed was identifiably from third-parties, most often in the form of Open Source components and Commercial shared libraries and components. Furthermore, there was a nesting effect as third-party components themselves often contained other third-party components. 6

9 Distribution of Application Development by Supplier Type Figure 1 reveals that close to a third of the applications analyzed during the reporting period were identified as thirdparty (Commercial, Open Source and Outsourced vendors). The percentage of outsourced applications represented in the dataset was low at 1%. Part of this is a data labeling issue. Organizations sometimes consider code developed by outsourcers as internally developed. Veracode encountered many instances where flaws in internally developed code were traced back to software supplied by outsourcing partners. Another factor is that outsourcing contracts have been silent on the topic of security testing and remediation. As these contracts renew, Veracode expects to see independent security verification requirements inserted and an increase in the percentage of identifiably outsourced code submitted. Applications by Supplier 71% 22% 6% 1% Internally Developed Commercial Open Source Outsourced Figure 1: Application by Supplier Distribution of Application Business Criticality by Supplier Type We know that not all applications have the same level of criticality to the business. However, it is instructive to examine the sources from which the most business critical applications are derived. Veracode explored the relationship between application supplier type and business criticality. As Figure 2 illustrates, 20% of Very High and 37% of High criticality applications are developed by third-parties. Domain expertise, proven functionality, and time-to-market are all factors in the decision to develop applications internally or procure them from third-parties. The significant presence of third-party applications identified as critical increases the importance of applying uniform application security verification policies across internally developed applications and those procured from third-party suppliers. The significant presence of third-party applications identified as critical increases the importance of applying uniform application security verification policies across all supplier types. 7

10 Application Business Criticality by Supplier Commercial Internally Developed Open Source Outsourced * Very High 17% 80% 2% 1% High 26% 63% 10% 1% Medium 21% 74% 4% 1% Low 27% 71% 2% Very Low 100% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 2: Application Business Criticality by Supplier (* small sample size) Distribution of Application Type and Programming Language by Supplier Type Supplier Application Profiles C/C++ Java.NET Other Web Non-Web Internally Developed 11% 56% 33% 2% 61% 39% Commercial 29% 45% 24% 3% 36% 65% Open Source 51% 45% 4% 0% 29% 71% Outsourced 0% 81% 14% 5% 71% 29% Table 1: Supplier Application Profiles Table 1 illustrates that nearly one third of commercially developed applications and over half of open source applications are written in C/C++ indicating a significant reliance on this language platform by these types of software suppliers. It further indicates that over 65% of the software developed by these same suppliers are non-web applications, while Internally Developed and Outsourced suppliers are relied on for web applications to about the same degree. The language and type of application differences among suppliers allows for policies and acceptance criteria to be tailored to the most prevalent risks and, among other things, clearly indicates the requirement for C/C++ language and non-web application support when choosing security testing approaches to third-party software. Support for C/C++ and non-web applications is required when choosing security testing approaches to third-party software. 8

11 Distribution of Security Quality and Remediation Efforts by Supplier Type The illustration below (Figure 3) depicts Supplier Performance on First Submission as measured by the Veracode risk adjusted verification methodology. When calculated as a percentage of total applications submitted 57% of all applications were deemed to have unacceptable security quality upon first submission. Outsourced vendors achieved the lowest scores followed by Commercial suppliers, Open Source and Internally Developed applications. These poor results were consistent with the Veracode s first State of Software Security report. It remains clear that most organizations do not have developers trained in secure coding principles or have not implemented a secure software development lifecycle. No real change in percentage of applications deemed to have unacceptable security quality upon first submission 58% in Volume 1, 57% in Volume 2. Supplier S Performance on First Submission (Adjusted for Business Criticality) Acceptable Not Acceptable Overall 43% 57% Outsourced * 7% 93% Open Source 42% 58% Internally Developed 46% 54% Commercial 35% 65% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 3: Supplier Performance on First Submission (Adjusted for Business Criticality) Applications that do not achieve an acceptable level of security on first submission are returned to the supplier with potential vulnerabilities identified by location in the code and with remediation instructions. Of those applications that were remediated and resubmitted, Figure 4 shows that most achieved acceptable levels of security within 16 days and in 1.1 builds (i.e. resubmissions following the initial analysis of the application). These encouraging results point to the effects of independent, cloud-based security testing. With a similar approach across supply chain participants, CIOs and CISOs can use this information to quantify application security risk versus the cost to mitigate, better estimate software development project costs and schedules, and control rework charges associated with security vulnerabilities in third-party agreements. 9

12 Remediation Performance by Supplier Internally Developed Commercial Open Source Overall DAYS TO REMEDIATE REMEDIATION SUBMISSION TO PASS Figure 4: Remediation Performance by Supplier Distribution by Supplier s Ability to Meet Security Compliance Policy by Supplier CIOs, CISOs, customers and internal auditors are increasingly enforcing compliance with application security policies. Two independent policy standards, one specifically for web applications from OWASP (OWASP Top 10) and one for applications of any type from the US Government, MITRE and the SANS Institute (CWE/SANS Top 25 Most Dangerous Software Errors) have been adopted by many organizations. An analysis of a supplier s ability to meet these industry standards is useful when determining software acceptance criteria. For software providers, evidence of compliance with these policies, such as the VERAFIED HIGH ASSURANCE 4 marks for OWASP Top 10 and CWE/SANS Top 25, anticipates customer security concerns and can differentiate their products. Adopting OWASP Top 10 or CWE/SANS Top 25 policies promotes uniform verification standards and performance measurement across application inventory. OWASP Top 10 Compliance by Supplier on First Submission Acceptable Not Acceptable Open Source 40% 60% Internally Developed 12% 88% Commercial 7% 93% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 5: OWASP Top 10 Compliance by Supplier on First Submission

13 Figure 5 shows the percentage of web applications that met the OWASP Top 10 (2010) policy by supplier. An application was labeled Not Acceptable if it contained any vulnerabilities defined in the standard lists. The number of Commercial and Internally developed web applications that were not acceptable is staggering at more than 80%. The difference between this extraordinary indicator of insecurity when compared to the bad but much higher acceptable levels of security identified earlier is largely explained by the high number of web applications that were submitted as lower business criticality. Another contributing factor may be due to the increasing number of microsites that are generally developed on behalf of large enterprises to support time-based marketing or commercial initiatives where time-to-market is the most important driver. Given the level of interconnectedness of software in most organizations Veracode observes that low business criticality values for web applications or the temporal nature of their existence probably understates the risk and encourages customer to adopt more stringent policies such as the OWASP Top 10 for all web applications. More than 8 out of 10 commercial and interally developed web applications failed against OWASP Top 10 upon first submission. Figure 6 examines suppliers ability to deliver applications as measured by compliance against the CWE/SANS Top 25 Most Dangerous Software Errors. All applications both web and non-web were included in this analysis. Commercial and Internally developed applications performed the best with about 50% and 52% of applications meeting acceptance respectively. The difference in the ranking of open source applications as worse in the ranking when compared to their performance against OWASP may be due to the fact that most open source applications analyzed in the dataset are non-web applications. CWE/SANS Top 25 Compliance by Supplier on First Submission Acceptable Not Acceptable Outsourced * 20% 80% Open Source 38% 62% Internally Developed 52% 48% Commercial 50% 50% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Figure 6: CWE/SANS Top 25 Compliance by Supplier on First Submission 11

14 Distribution of Most Common Security Vulnerabilities by Supplier The distribution of security vulnerabilities by type of supplier may point to more or less effective practices and help in choosing future suppliers. Table 2 reveals relatively similar results by suppliers in terms of both prevalence and type of vulnerabilities detected. Cross-site scripting and cryptographic issues appear in the top five vulnerabilities across all supplier types. Vulnerability Distribution by Supplier Internally Developed Commercial Open Source Outsourced Cross-site Scripting 49% (XSS) Cross-site Scripting 56% (XSS) CRLF Injection 15% Cross-site Scripting 31% (XSS) CRLF Injection 14% Information Leakage 16% Numeric Errors 14% Directory Traversal 16% Information Leakage 10% CRLF Injection 6% Buffer Mgmt Errors 14% Cryptographic Issues 14% Cryptographic Issues 6% Cryptographic Issues 5% Cross-site Scripting 13% (XSS) Time and State 12% SQL Injection 5% Directory Traversal 4% Cryptographic Issues 12% Information Leakage 9% Directory Traversal 3% SQL Injection 4% Error Handling 9% Credentials Mgmt 8% Buffer Overflow 3% Buffer Overflow 2% Buffer Overflow 9% API Abuse 6% Potential Backdoor 2% Potential Backdoor 2% Time and State 4% CRLF Injection 3% Untrusted Search 2% Path Numeric Errors 2% Directory Traversal 4% SQL Injection 2% Time and State 2% Error Handling 2% Potential Backdoor 1% Insufficient Input <1% Validation Error Handling 1% Time and State 1% SQL Injection 1% Error Handling <1% Encapsulation 1% Credentials Mgmt <1% Information Leakage 1% Session Fixation <1% Credentials Mgmt <1% Buffer Mgmt Errors <1% API Abuse <1% OS Command <1% Injection API Abuse <1% API Abuse <1% Credentials Mgmt <1% Other <1% Insufficient Input <1% Validation OS Command <1% Injection Session Fixation <1% Race Conditions <1% Table 2: Vulnerability Distribution by Supplier Third-Party Risk Assessments New in this volume is an analysis of third-party risk assessments performed against vendors at the request of a buyer of software or software development services. These buyers may be purchasing already developed applications for internal use (e.g. Commercial-off-the-shelf or COTS applications), applications to be developed by someone else, or applications and components to be re-distributed under a re-licensing arrangement. Mergers and acquisitions may also trigger a third-party assessment. Third-party risk assessments are among the fastest growing types of assessments requested of Veracode, with linear growth rates over the last 6 quarters. 12

15 Figure 7 shows the types of enterprises that are requesting third-party assessments. They are predominantly in the Financial (including Banks, Insurance, and Financial Services) or Software/IT Services market categories where this category represents enterprises that are both software producers and providers of IT services and equipment. Requester Distribution by Industry 28% 55% 17% Software/IT Services Financial Other Figure 7: Requester Distribution by Industry One of the most striking themes from these assessments is the implication for cloud-based services. Figure 8 shows that Vendors that provide cloud based services, either in Cloud only or Cloud as an option (Cloud+Deployment) accounted for almost 60% of all reviewed third-party applications. The other Vendor Types for which reviews were requested were general ISV s or companies that specialize in integrating disparate components from several sources all of which are likely participants in cloud-based solutions. Cloud only or Cloud as an option (Cloud + Deployment) accounted for almost 60% of all reviewed third-party applications. Reviewed Application Count by Vendor Type 18% Cloud + Deployed 21% 14% 1% 1% Integration ISV 45% Cloud Consulting Deployed Figure 8: Reviewed Application Count by Vendor Type 13

16 The relative proportion of third-party reviews broken down by application functional area is provided in Figure 9. In this diagram, the categories used for functional area are derived from the Balanced Scorecard model (BSC), a widely-used strategic planning and management system. 5 BSC identifies four functional perspectives by which to view and measure an organization: Financial, Customer, Operations, and Learning and Growth. Any application that deals with day-to-day business activity is included in the Operations category shown in Figure 9. This includes business process management applications, product development, information management utilities, IT management tools, and applications to support all non-financial governance functions such as legal and operational risk management. The Customer category includes all content management, customer relationship management and web-facing services provided to customers. The Learning and Growth category includes applications to support HR, training, and human capital management. Financial applications include traditional accounting and finance functions as well as an important and growing class of application that provides mobile access for banking and other finance related tasks. It is interesting to note that Operations is the leading functional area for third-party assessments which comprises about the same portion of requests as the combination of Finance and Customer applications. This indicates that companies are proactively requiring assessments of applications across a wide variety of internal applications (Operations and Finance) as well as external customer-facing web sites. Companies are proactively requiring assessments of applications across a wide variety of internal applications (Operations and Finance) as well as external customer-facing web sites. Requested Third-party Assessments by Application Purpose 45% 29% 15% 11% Operations Financial Customer Learning Growth Figure 9: Requested Third-party Assessments by Application Purpose Application Type Definitions: Operations category includes applications supporting day-to-day non-financial business activity such as product development, information management utilities, IT management tools etc.; Financial category traditional accounting and finance applications and newer mobile banking applications; Customer category includes customer relationship management and content management applications and web customer support applications; Learning and Growth includes applications to support HR, training and human capital management. 5 The Balanced Scorecard (BSC) was originated in the 1990s by Drs. Robert Kaplan (Harvard Business School) and David Norton as a performance measurement framework to enrich traditional financial performance measures with strategic non-financial performance measures, thereby giving a more balanced view of organizational performance. See for additional information 14

17 Figure 10 reveals that, like third-party supplier code in general, third-party risk assessments result in high rates of unacceptable security on first submission. 4 out of 5 assessments failed to achieve acceptable levels of security on first review. Most third-party assessed suppliers also remediated faster than applications on average, with three-quarters of all applications requiring only 11 days to achieve acceptable levels of security quality. It should be noted that many customers implementing a third-party risk management program High ROI with minimal impact to timeline from third-party risk assessments: Three-quarters required less than 11 days to achieve security quality level required by requesting enterprise. employ a customer success program manger or an internal resource that is tasked with policy creation, coordination of third-parties and program execution. This focus may be contributing to a relatively short amount of time for achieving compliance. The fast turnaround further implies that requiring a third-party assessment does not result in delayed deployment of more than a couple of weeks, making it worth the trade-off. Third-party Assessments: Performance Upon Initial Submission 19% 81% Not Acceptable Acceptable Figure 10: Third-party Assessments: Performance Upon Initial Submission A PROFILE IN VERIFICATION Third-party assessments is one of the fastest growing types of security programs as CIOs and CISOs become aware of the unbounded risk inherent in the software supply chain. At one company, a facilitated engagement with third-parties improved the state of software security for all parties.» Program Time 6 months» Third-Parties Assessed Close to 40 applications from distinct vendors (in excess of 50 million lines of code)» Vulnerabilities Remediated Over 500 Severity 5 and 4 vulnerabilities (over 7000 vulnerabilities in total)» Lessons Learned The impossible is possible. Facilitated independent verification improved security for a large number of third-party applications in a short timeframe.» Next Steps Additional third-parties are proactively pursuing verification and the company is using the intelligence gained so far to revise third-party acceptance policies. 15

18 Security of Applications The previous section presented information from the Software Supplier and Purchaser perspectives in an attempt to help enterprises properly manage application risk in the software supply chain. In this section of the report we explore security risks related to web and non web applications, programming languages, types of vulnerabilities, and industry alignment. New in this report, we further consider the effectiveness of multiple security testing techniques and provide a deeper investigation of application security in Banking, Insurance, and Financial Services companies. As background, software vulnerabilities are the attack points in applications used by hackers to compromise a system. Different types of applications have different attack points. For example, web applications have different attack surfaces than desktop software or databases. Additionally, vulnerabilities can vary significantly by programming language and platforms such as the Windows versus BlackBerry operating systems. It is also possible for applications in different industries to have different vulnerabilities based on the secure coding skills of the engineering population serving those industries (e.g. Financial Services versus Retail) and the sophistication of their software development practices or central security teams. While no software will ever be perfectly secure, understanding what makes applications more or less vulnerable provides the basis for CIOs, CISOs, and software professionals to manage application portfolio risk rather than remain blindly susceptible to catastrophic loss of information, business continuity, and reputation. Distribution of Application by Type All applications analyzed by Veracode are inventoried and classified according to a profile which includes key characteristics such as whether the application is web-facing, its language and platform, and the industry of the organization submitting it. In this reporting period we observed a slight shift in favor of non-web applications. They grew to 44% (from 40% as reported in Volume 1) and web applications were down to 56% (from 60% as reported in Volume 1). This reflects a heightened security awareness for legacy and back-end applications and not just those applications exposed to the web. Non-web applications analyzed grew from 40% in the prior report to 44%, reflecting the expansion of application security efforts beyond web applications to legacy and back-end applications. Web versus Non-Web Applications 56% 44% Web Applications Non-Web Applications Figure 11: Web versus Non-Web Applications 16

19 Distribution of Applications by Language An analysis of the Distribution of Applications by Language is a useful indicator and reasonable proxy for the ever-changing attack surface of the world s software infrastructure. Applications by Language Family Java 50% 29% 19% 1% 1% <.NET C/C++ ColdFusion PHP Figure 12: Applications by Language Family In our last report we showed the relative distributions of three development platforms Java, C/C++, and.net. Java still leads at 50%, up slightly from 47% in our last report. However, C/C++ and.net have swapped positions, and we are now seeing.net applications leading C/C++ by a factor of 3 to 2. New in this report are two new platforms, ColdFusion and PHP, which account for 1.4% and 0.7% of all applications, respectively. These numbers should not be used as a representation of the market share of these two platforms because Veracode only recently developed the capabilities required to analyze them. We expect that over time, these percentages will increase to better approximate the real-world distribution of these platforms in the enterprise. Our data shows that all applications, no matter what language is used, require secure development practices to be secure. To better understand the impact of programming language on application security, Table 3 shows the median flaw density for each. The median flaws per thousand lines of code (KLOC) for Java, C/C++, and.net are similar. Many people ask whether switching languages will improve application security. Our data shows that all applications, no matter what language is used, require secure development practices to be secure. 17

20 ColdFusion was very different with a median of 5.2 security flaws per KLOC. While ColdFusion applications were a small percentage of the total number of applications assessed in this period, more than 35 applications were included. The ColdFusion Markup Language is very compact relative to Java and.net, which may explain most of this difference. The simplicity of CFML also allows less accomplished programmers and even non-programmers to develop web applications, so it stands to reason that developer inexperience is another contributing factor to ColdFusion s flaw density. Flaw Density by Language First Quartile Median Mean Third Quartile C/C ColdFusion Java Net Table 3: Flaw Density by Language Distribution of Applications by Vulnerability Type The charts on the following page depict top vulnerability categories by prevalence, presented from two different perspectives. The first is by vulnerability frequency, which illustrates the percentage of the total vulnerabilities discovered. The second is by affected applications, which shows the percentage of applications containing one or more of the vulnerabilities in each category. Rows highlighted in red are vulnerability categories that also appear in the CWE/SANS Top 25 (2010) or OWASP Top 10 (2010) standards. There is considerable overlap between Veracode findings and these industry standards, further confirming the relevance of these vulnerability categories as top areas of security weakness to focus on for enterprises. Cross-site Scripting (XSS) remains the most prevalent vulnerability category by frequency, accounting for 51% of all vulnerabilities in the data set. The next three categories behind XSS Information Leakage, CRLF Injection, and Cryptographic Issues maintain the same rankings as we observed in our last report. SQL Injection was somewhat more prevalent this time and Potential Backdoors broke into the top ten categories. The rise in Potential Backdoors isn t necessarily cause for alarm. Automated scanning cannot reliably judge intent. In order to distinguish the malicious cases from the legitimate ones, Potential Backdoors should be inspected carefully by someone with an understanding of the application s intended design and function. 18

21 Top Vulnerability Categories (Overall Prevalence) Indicate categories that are in the OWASP Top 10 or CWE/SANS Top 25 Cross-site Scripting (XSS) 51% Information Leakage CRLF Injection 12% 11% Cryptographic Issues 6% SQL Injection Directory Traversal 4% 4% Buffer Overflow Potential Backdoor Time and State Error Handling Credentials Management Numeric Errors Untrusted Search Path API Abuse Encapsulation 2% 2% 2% 1% 1% 1% 1% 1% 1% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% Figure 13: Top Vulnerability Categories (Overall Prevalence) Even though a category may account for a small percentage of the total vulnerabilities, the frequency with which it appears across different applications may be a more illuminating statistic. Viewing the vulnerabilities by affected applications, Cryptographic Issues remains atop the list, with 41% of all applications containing one or more vulnerabilities in this category. This category once again comprised insufficient entropy, plain text storage of sensitive data, use of hardcoded cryptographic keys, and use of algorithms with inadequate encryption strength. The unnerving part of this statistic is that while static analysis can uncover a variety of cryptographic flaws, there are far more complex mistakes that can only be detected by a skilled practitioner, either through design review or a focused code review. If automated scanning is uncovering simple cryptographic mistakes at a rate that affects over 40% of all applications, one has to wonder how many other cryptographic issues are lurking beneath the surface. Once again, this statistic underscores the need for developers to become better educated with this less publicized but still prevalent issue in order to safely incorporate cryptographic mechanisms into their applications. 19

22 Top Vulnerability Categories (Percent of Application Affected) Indicate categories that are in the OWASP Top 10 or CWE/SANS Top 25 Cryptographic Issues Information Leakage 41% 40% Cross-site Scripting (XSS) 34% Directory Traversal CRLF Injection SQL Injection 24% 27% 26% Time and State Credentials Management 18% 20% API Abuse 12% Encapsulation Insufficient Input Validation Error Handling Potential Backdoor Buffer Overflow OS Command Injection 9% 8% 8% 8% 8% 7% 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Figure 14: Top Vulnerability Categories (Percent of Application Affected) Vulnerabilities by Language Distribution The table on the following page presents the most prevalent categories (by share of total vulnerabilities discovered) based on language family..net applications continue to exhibit an abnormally high frequency of Cross-site Scripting vulnerabilities relative to the other enterprise languages. This can be attributed to developer use of.net controls that do not automatically encode output. While we reported flaw density in aggregate for ColdFusion above, vulnerability rankings by type of vulnerability for Cold Fusion and PHP are not displayed due to the small sample size for this reporting period..net applications continue to exhibit an abnormally high frequency of Cross-site Scripting. 20

23 Vulnerability Distribution by Language Java C/C++.NET Cross-site Scripting (XSS) 46% CRLF Injection 17% Information Leakage 16% Cryptographic Issues 7% Directory Traversal 4% SQL Injection 3% Time and State 2% Untrusted Search Path 2% Credentials Mgmt 1% Encapsulation 1% API Abuse 1% Insufficient Input Validation <1% Race Conditions <1% OS Command Injection <1% Dangerous Functions <1% Buffer Overflow 32% Potential Backdoor 21% Error Handling 18% Numeric Errors 13% Buffer Mgmt Errors 7% Cryptographic Issues 3% Directory Traversal 2% Dangerous Functions 1% Time and State <1% Race Conditions <1% API Abuse <1% Format String <1% OS Command Injections <1% Credentials Mgmt <1% Untrusted Search Path <1% Cross-site Scripting (XSS) 66% Cryptographic Issues 13% Directory Traversal 8% CRLF Injection 4% Information Leakage 4% Insufficient Input Validation 2% SQL Injection 1% Credentials Mgmt 1% Potential Backdoor <1% Time and State <1% Error Handling <1% OS Command Injection <1% Buffer Overflow <1% Untrusted Search Path <1% Dangerous Functions <1% Table 4: Vulnerability Distribution by Language Vulnerability Distribution by Analysis Type As the market for application security testing changed from manual penetration testing alone to automated dynamic and static analysis, providers of security services and products tended to gravitate to one of the methods, reflecting their particular expertise. With different providers promoting different methods, the idea that one method was most appropriate for certain types of applications and certain people and another for other types and other people began to take hold in organizations. Web applications were tested by security professionals or QA teams using dynamic web application scanners. Developers were asked to use static code analyzers. Manual penetration testing services were used by external consultants on the most critical applications. Although they were not actually mutually exclusive, these distinctions and the total cost of implementing more than one led many organizations to choose between methods rather than choose the right mix of methods for an application based on its business criticality. 21

24 The problem with over-relying on one method is that there is no single testing methodology that can detect all of the important vulnerability types with sufficient depth and breadth there is no silver bullet. For example, conducting a manual penetration test while ignoring automated testing is not ideal; penetration testing can identify complex security issues but it generally provides spotty application coverage, particularly within a single vulnerability category. Similarly, relying solely on automated static analysis provides excellent coverage but does not account for business logic, design flaws or environment and configuration issues. Meanwhile, dynamic analysis only tests code paths that it can discover externally which, as benchmarking comparisons of dynamic web application scanners have shown, can lead to embarrassing results. 6 While most security experts would agree with these points anecdotally, they generally lacked hard data until now. The following table depicts the top 10 vulnerability categories by prevalence, for each analysis type. This allows us to see what vulnerabilities automated static binary analysis, automated dynamic analysis, and manual penetration testing find most frequently. The first thing to notice is that many of the most common vulnerability types are present in all three tables Cross-site Scripting, SQL Injection, and Information Leakage, to name a few. However, there are vulnerability types that are most frequently detected using one particular method, for example, Buffer Overflows (static), Server Configurations (dynamic) and Authorization Issues (manual). Top 10 Vulnerability by Analysis Type Static Dynamic Manual Cross-site Scripting (XSS) 52% CRLF Injection 11% Information Leakage 11% Cryptographic Issues 6% Directory Traversal 4% SQL Injection 3% Buffer Overflow 3% Potential Backdoor 2% Time and State 2% Error Handling 1% Information Leakage 44% SQL Injection 27% Cross-site Scripting (XSS) 26% Server Configuration 2% OS Command Injection <1% Other <1% Session Fixation <1% Cryptographic Issues <1% Insufficient Input Validation <1% Authentication Issues <1% Cross-site Scripting (XSS) 26% Information Leakage 21% Other 12% Cryptographic Issues 11% SQL Injection 11% Authorization Issues 7% Authentication Issues 5% Insufficient Input Validation 2% Credentials Mgmt 2% Directory Traversal 1% Table 5: Top 10 Vulnerability by Analysis Type

25 The previous table provided a high-level view into the types of vulnerabilities that were most frequently detected by different analysis methods, for all applications in the data set. However, we can learn more by drilling down into the subset of the web applications in the data set that were subjected to both automated static and automated dynamic testing at customer request. These applications represented 21% of the total web applications analyzed in the past 18 months. In these cases, dynamic analysis came up empty for CRLF Injection and nearly empty for Cryptographic Errors. It s not impossible for dynamic testing to find some varieties of CRLF Injection HTTP Response Splitting is one example but the bulk of that category in this application set, which we know because it was found by static binary analysis, was Log Injection vulnerabilities. These simply cannot be detected by any dynamic method except in rare circumstances. Furthermore, it s a striking statistic that static binary analysis detected over 20 times more XSS vulnerabilities and nearly twice as many SQL injection vulnerabilities than dynamic analysis across these applications, on average. What accounts for the disparity between static and dynamic methods, independent of vendor? One major contributing factor is that static analysis provides comprehensive coverage of the application whereas dynamic analysis only tests code paths that it can discover externally. Often, dynamic (and even manual) testing completely overlooks portions of the application that are only reachable under certain circumstances. For example, application functionality may be gated behind a series of forms that trigger different behavior depending on how they are filled out. Also, applications that support different types of users (e.g. view-only, author, editor, administrator, power user, etc.) often restrict the functionality that each user level can access, meaning that the application must be scanned multiple times, iterating over all of the user roles, in order to maximize coverage. However, while coverage may be an issue, dynamic analysis is performed against a live application instance, so a higher percentage of its reported vulnerabilities may be demonstrably exploitable. Sacrificing coverage in order to reduce the vulnerability triage effort is a risky tradeoff but it is a tradeoff that some enterprises may choose to make during the early stages of an application security program. The lesson for CISOs and CIOs is that a robust application security program must incorporate multiple testing methods in order to ensure that applications are assessed with sufficient coverage, measured by both depth and breadth. Becoming overly dependent on too few analysis methodologies guarantees blind spots when assessing overall application risk. 23