The Forrester Wave : Application Security, Q4 2014

Transcription

1 For: Security & Risk Professionals The Forrester Wave : Application Security, Q by Tyler Shields, December 23,2014 Key Takeaways HP, IBM, Veracode, WhiteHat, Contrast Security, Quotium, And Checkmarx Lead Forrester analyzed the application security testing market (static analysis, dynamic analysis, and instrumented/interactive technologies). The results of the analysis find that HP Fortify, IBM, Veracode, WhiteHat Security, Contrast Security, Quotium, and Checkmarx lead the field. Beyond Security, Coverity, Qualys, and Virtual Forge offer competitive options, while Trend Micro lags behind. S&R Pros Look For Solutions That Emphasize Accuracy, Integration, And Scalability The application security testing market is steadily growing because S&R pros increasingly trust application security testing providers to act as strategic partners, advising them on top app security decisions. Solutions with the highest accuracy of results, best integration points, and the capability to grow and scale will continue to find success. Combining Assessment Methods Provides Differentiation In The Market By combining static, dynamic, and instrumented assessment technologies, vendors are creating a platform-based application security assessment model. Each individual assessment technology contains delivery weaknesses when compared with the others. Access The Forrester Wave Model For Deeper Insight Use the detailed Forrester Wave model to view every piece of data used to score participating vendors and create a custom vendor shortlist. Access the report online and download the Excel tool using the link in the right-hand column under Tools & Templates. Alter Forrester s weightings to tailor the Forrester Wave model to your specifications. Forrester Research, Inc., 60 Acorn Park Drive, Cambridge, MA USA Tel: Fax:

2 December 23, 2014 Why Read This Report The Forrester Wave : Application Security, Q Tools And Technology: The Security Architecture And Operations Playbook by Tyler Shields with Stephanie Balaouras and Jennie Duong In Forrester s 82-criteria evaluation of application security vendors, we identified the 12 most significant service providers in the category Beyond Security, Checkmarx, Contrast Security, Coverity, HP Fortify, IBM, Qualys, Quotium, Trend Micro, Veracode, Virtual Forge, and WhiteHat Security and researched, analyzed, and scored them. This report details our findings about how well each vendor fulfills our criteria and where they stand in relation to each other to help security and risk professionals select the right partner for their application security requirements. Table Of Contents 2 S&R Pros Must Build Security Into The Application Layer 2 Selection Criteria Target Functional Security Capabilities And Strategy Vendors Offer Application Assessment Using Multiple Technologies Notes & Resources Forrester conducted product evaluations in April 2014 and interviewed 12 vendor and user companies: Beyond Security, Checkmarx, Contrast Security, Coverity, HP Fortify, IBM, Qualys, Quotium, Trend Micro, Veracode, Virtual Forge, and WhiteHat Security The Application Security Market Uncovered Vendor Profiles Leaders Strong Performers Contenders Supplemental Material Related Research Documents TechRadar : Enterprise Mobile Security, Q November 3, 2014 It s Time To Level Up Your Mobile Application Security Program August 26, 2014 Address The Top 10 Nontechnical Security Issues In Mobile App Development April 30, , Forrester Research, Inc. All rights reserved. Unauthorized reproduction is strictly prohibited. Information is based on best available resources. Opinions reflect judgment at the time and are subject to change. Forrester, Technographics, Forrester Wave, RoleView, TechRadar, and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. To purchase reprints of this document, please [email protected]. For additional information, go to

3 The Forrester Wave : Application Security, Q S&R Pros Must Build Security Into The Application Layer Application security remains a crucial component in keeping enterprise servers and data secure. Many firms have rushed to bring applications online, building out consumer-facing websites, buying commercial off-the-shelf (COTS) products, and developing mobile applications to enable and engage with their customers and partners without thinking about the security of the application itself. As a consequence, businesses are exposing their most sensitive corporate and customer data to possible external threats and breaches. Selection Criteria Target Functional Security Capabilities And Strategy S&R pros are looking to work with vendors to detect vulnerabilities more efficiently and effectively, to avoid exposing business-critical security issues, as well as to improve overall business and development efficiency. The vendors assessed meet the growing demand across all businesses without compromising the security needs of their clients operations. After examining the current trends in the market, user needs assessments, and vendor and expert interviews, we developed a comprehensive set of evaluation criteria for application security. We evaluated vendors against 82 criteria, grouped into three high-level categories: Current offering. The vertical axis of the Forrester Wave graphic reflects the strength of each vendor s product offering, including its capabilities in general features (e.g., deployment model, scalability, targeted scanning), static application security testing features, dynamic applications security testing features, instrumented analysis features, reporting features and workflows (e.g., flaw descriptions, centralized policies, and workflow and remediation tracking), developer education and training, integrations (e.g., IDE integration, API access, patches, WAFs, and MDMs), remediation instructions, and customer references. Strategy. The horizontal axis measures the viability and execution of each vendor s strategy, which includes planned enhancements, key technology partners, product focus, target market, cost, average sales prices, maintenance costs, and pricing structure. Market presence. The size of each vendor s bubble on the Forrester Wave graphic represents each vendor s presence in the application security market, based on its installed customer and product base, revenue, systems integrators, services, number of employees, and key technology partners. Vendors Offer Application Assessment Using Multiple Technologies Forrester included 12 vendors in the assessment: Beyond Security, Checkmarx, Contrast Security, Coverity, HP Fortify, IBM, Qualys, Quotium, Trend Micro, Veracode, Virtual Forge, and WhiteHat Security. Each of these vendors has (see Figure 1):

4 The Forrester Wave : Application Security, Q Mindshare with Forrester s clients. Vendors included are frequently mentioned in Forrester client inquiries and other forms of client engagement relating to application security. Ability to offer SAST, DAST, and/or IAST capabilities. The vendors evaluated offer comprehensive approaches in static analysis (SAST), dynamic analysis (DAST), and instrumented/ interactive technologies (IAST) techniques in order to detect weaknesses and vulnerabilities in general code, web applications, mobile applications, and COTS product offerings. Ability to provide easy deployment and integration for its customers. All vendors evaluated could deploy either a scalable on-premises or cloud-based service to their customers. Vendors were focused on improving the security development life cycle for the enterprise buyer by creating a solution that integrated into existing models of analysis and third-party development tools. Relevance to the application security market. Inclusion in this Forrester Wave means that the vendor actively competes in the application security market, showing up in competitive use cases and discussions among experts and Forrester clients.

5 The Forrester Wave : Application Security, Q Figure 1 Evaluated Vendors: Product Information And Selection Criteria Evaluated vendors Product evaluated Product version Beyond Security AVDS 4.00 build 307 Checkmarx Contrast Security Coverity HP Fortify IBM Qualys Quotium CxSuite Contrast Enterprise* 3.1 Code Advisor 7.5 HP Applications Security Portfolio (includes: HP Fortify Software Security Center, HP Fortify Static Code Analyzer, HP WebInspect, HP Fortify on Demand, and HP Application Defender)* IBM Security AppScan Standard, Enterprise, and Source N/A Qualys Web Application Scanning 3.2 Seeker 3.0 Trend Micro Veracode Deep Security for Web Apps* Veracode Platform* N/A N/A Virtual Forge WhiteHat Security CodeProfiler 3.5 WhiteHat Sentinel WhiteHat Sentinel Source* N/A Vendor selection criteria Mindshare with Forrester s clients. Vendors included are frequently mentioned in Forrester client inquiries and other forms of client engagement relating to application security. Ability to offer SAST, DAST, and/or IAST capabilities. The vendors evaluated, offer comprehensive approaches in SAST, DAST, or IAST techniques in order to detect weakness and vulnerabilities in general code, web applications, mobile applications, and COTS product offerings. Ability to provide easy deployment and integration for its customers. All vendors evaluated could deploy either a scalable on-premises or cloud-based service to their customers. Vendors were focused on improving the security development life cycle for the enterprise buyer by creating a solution that integrated into existing models of analysis and third party development tools. Relevance to the application security market. Inclusion in this Forrester Wave means that the vendor actively competes in the application security market, showing up in competitive use cases and discussions among experts and Forrester clients. Source: Forrester Research, Inc. Unauthorized reproduction or distribution prohibited.

6 The Forrester Wave : Application Security, Q The Application Security Market Uncovered The evaluation uncovered a market in which a platform approach strengthens the limitations of each assessment model, raising the importance of a multitechnology approach. Vendors that built a platform leveraging DAST, SAST, and even IAST saw higher marks than those vendors that focus on one or two technology sets (see Figure 2): HP, IBM, Veracode, WhiteHat, Contrast Security, Quotium, and Checkmarx lead. While each vendor designed and built its original solution from a different technology starting point, the end result for each of these vendors is a cross-technology solution. When sorted by technology cut, each vendor in this section excels at more than one solution, giving it the ability to increase the accuracy of its results. Beyond Security, Coverity, Qualys, and Virtual Forge offer competitive options. Competitive offerings from these vendors tended to focus on one specific area (DAST, SAST, or IAST) of the cross-platform solution. These vendors are working toward building an integrated platform but have yet to improve their nondominant offerings to the strength level that puts them in the upper echelon. These vendors should be chosen specifically if their dominant technology area is of a stronger concern to the buyer than the nondominant technologies. Trend Micro is building up momentum from a late market entry. As a late market entry, Trend Micro is lagging behind the other vendors in the space. Focused predominantly on DAST assessments from the cloud, Trend Micro must expand beyond a single technology-based solution in order to remain relevant to the enterprise buyer. The level of resources available for continued innovation keeps Trend Micro in the picture if it can maintain its current pace of advancement. This evaluation of the application security market is intended to be a starting point only. We encourage clients to view detailed product evaluations and adapt criteria weightings to fit their individual needs through the Forrester Wave Excel-based vendor comparison tool.

7 The Forrester Wave : Application Security, Q Figure 2 Forrester Wave : Application Security, Q Strong Current offering Risky Strong bets Contenders performers Leaders IBM HP Fortify Contrast Security Veracode WhiteHat Security Quotium Checkmarx Virtual Forge Beyond Security Qualys Go to Forrester.com to download the Forrester Wave tool for more detailed product evaluations, feature comparisons, and customizable rankings. Trend Micro Coverity Market presence Weak Weak Strategy Strong Source: Forrester Research, Inc. Unauthorized reproduction or distribution prohibited.

8 The Forrester Wave : Application Security, Q Figure 2 Forrester Wave : Application Security, Q (Cont.) Forrester s weighting Beyond Security Checkmarx Contrast Security Coverity HP Fortify IBM Qualys Quotium Trend Micro Veracode Virtual Forge WhiteHat Security CURRENT OFFERING General features Static analysis features Dynamic analysis features Instrumented analysis features Reporting features and workflow Developer education and training Integrations Remediation instructions Customer references STRATEGY Product strategy Corporate strategy Cost MARKET PRESENCE Installed base Revenue growth quarter over quarter Revenue growth year over year Systems integrators Services Employees Technology partners 50% 21% 22% 22% 5% 10% 5% 5% 10% 0% 50% 50% 50% 0% 0% 40% 0% 0% 15% 15% 15% 15% All scores are based on a scale of 0 (weak) to 5 (strong). Source: Forrester Research, Inc. Unauthorized reproduction or distribution prohibited. Vendor Profiles Leaders HP Fortify offers an extensive application security solution. HP Fortify displayed strong capabilities across the majority of our criteria in both static and dynamic analysis. Its product combines comprehensive static and dynamic testing and management across a multitude of languages and frameworks that allow customers to deploy and scale quickly. Customer

9 The Forrester Wave : Application Security, Q references were satisfied with the vendor s ease of installation, ease of use, configurable scans, and administration; one customer reference was very happy with the SDLC-based dynamic application security program built around HP WebInspect and would recommend it to any other organization, even those with rapid development/agile life cycles. Although HP Fortify has more than 4,000 customers of its application security products (on-premises and cloudbased), we noted that fewer than 500 customers were added in the past year. IBM s focus on the developer integration leads to exceptional results. The IBM product offering provides extensive general features on both on-premises and on-demand application security solutions, depending on customer needs. The solution offers limited static analysis features, for data identification, and runtime data tracking. The DAST offering is wellpositioned for web application discovery, Internet-sourced scanning, internal network scanning, and large-scale assessment. IBM has a long lineage in development and has one of the strongest integrations with other product lines and third-party development tools and services. IBM approaches the security market with a developer-centric message and product strategy focus. Veracode unifies the SAST and DAST platform with accurate and timely results. Veracode offers a unified cloud-based security SAST and DAST platform that includes capabilities for developer workflow integration, central policy management, security analytics and benchmarking, compliance reporting, and workflow management; differentiators include mobile behavioral analysis and third-party assessments. Veracode has the capability to simultaneously scan thousands of websites through its web perimeter monitoring scanning service while providing high scalability for static binary assessments. With strong results in both SAST and DAST segments, Veracode delivers a high level of accuracy in its technical findings while embracing a customer-centric approach to integration into the greater development workflow WhiteHat Security, although smaller than its competitors, innovates well. Although this vendor s dedicated research team is somewhat smaller than those of the other Leaders, the vendor s overall SAST and DAST capabilities, coupled with its unified SaaS platform, has gained a large customer base ranging from startups to large enterprise companies. WhiteHat Security provides efficient deployment and scalability, frequent scanning updates, and configurable rules and scanning capabilities. While WhiteHat Security offers both SAST and DAST services, its DAST capabilities are more comprehensive and can conduct continuous concurrent scanning of tens of thousands of web applications. Contrast Security is a small vendor making big advancements in application security. Although this vendor doesn t neatly fall into either the static or dynamic categories, as outlined in our application security evaluation, Contrast Security provides an up-and-coming offering that makes it a new and innovative contender (since 2012) to the application security space. The Contrast Security solution is instrumented via HTTP requests and responses like a dynamic tool; contains source code and binary code analysis like a static tool; and has an on-server agent like an

10 The Forrester Wave : Application Security, Q interactive tool. The vendor uses a combination of techniques to provide an effective and detailed analysis for its customers. Contrast Security can autodiscover applications located on a specific server and continually assess the security of these applications in real time. Since the technology is young, there are a few minor configuration options that are lacking; however, the assessment technique is unique enough to warrant interest from forward-thinking enterprise buyers. Quotium innovating in the area of instrumented assessment results in a unique approach. Another vendor innovating in the space of application security, Quotium has created a solution that doesn t cleanly fit into the SAST and DAST models of assessment, instead targeting a runtime continuous assessment model by using instrumented analysis. The product s runtime analysis technology hooks into the application processes and monitors all code execution, while simulating user, and hacker, traffic to the application. Quotium is an ideal product for enterprises looking for simultaneous testing across a multitude of users and servers, with a centralized repository in various test environments (e.g., AWS, Microsoft Azure, and Rackspace). Customers looking into this solution should note that while Quotium s Seeker is not a DAST or SAST solution, it does have several unique capabilities that accomplish reasonable dynamic and static results using a unique methodology. Checkmarx delivers SAST directly while offering DAST through partners. Checkmarx s solution has strong functional capabilities in deployment, concurrent use, scanning automation, configurable rules and scans, targeted scanning, and multiple user support. General features that the vendor must continue to improve upon include scalability, false positive elimination, and flexible scanning functionality. The Checkmarx offering has strong static analysis features around source code scanning, varied language and framework support, analysis levels, and custom static analysis rules. However, the solution is limited due to an inability to deliver dynamic assessment directly. Instead, Checkmarx looks to partners to deliver the DAST section of its product suite. Strong Performers Beyond Security offers a competitive hybrid option/deployment for its customers. Customers that require easy deployment across multiple environments should consider looking into Beyond Security as a viable dynamic assessment option. Not only is Beyond Security s solution available as a self-contained appliance and hosted (cloud) solution, but it also has a hybrid offering (on-site scanner managed by a cloud-based management system) for its customers. Beyond Security s solution offers competitive dynamic analysis features that support application discovery, Internet-sourced scanning, and internal scanning with either an appliance or VM. However, the solution does not support custom dynamic analysis rules or private data identification. Beyond Security does not offer SAST capabilities in its product suite.

11 The Forrester Wave : Application Security, Q Coverity has a general platform for SAST code analysis but lacks dynamic capabilities. Coverity s strength lies in its general features including scanning automation, targeted scanning, multiclass administration, and false positive elimination. Coverity static analysis can analyze byte code for data analysis in Java applications but has little to no functionality in binary scanning and static interactive testing (e.g., code behavior testing and runtime data tracking). Coverity does not support any features to run dynamic analysis. Qualys gets aggressive in product strategy and expands into application security market. Qualys has traditionally been considered a strong vulnerability assessment vendor, providing continuous security assessment for network-based attacks. Qualys has augmented its technology by moving up the stack into the application assessment realm. The solution offers dynamic analysis features such as application discovery, Internet-sourced scanning, high scalability, and appliance/virtual machine support. Although Qualys solution does not support static code analysis, its solution is ideal for customers looking to automate continuous dynamic assessments of target environments. Qualys is gaining in market share and was one of the few vendors to show a significant amount of new enterprise customer growth in the past year. Forrester expects that with new enhancements in its security portfolio, Qualys may become a more direct force in the DAST space in Virtual Forge extensively secures SAP-specific content but is limited in other features. Virtual Forge s application security solution contains both a cloud-based and on-premises deployment model that has capabilities including concurrent use and configurable rules for source code scanning. Virtual Forge s solution has limited features available for static analysis testing. The solution analyzes the source code of SAP applications only, limiting the product s marketability. The offering is unable to fully support static binary scanning, business logic flaws, and interactive testing. Virtual Forge s product uses open penetration testing frameworks (Metasploit) to do dynamic analysis and scanning, and SAP-specific content is layered on top of these open source tools. Virtual Forge has the most comprehensive product in the space for securing SAP source code; however, you will have to look elsewhere for other languages and features. Contenders Trend Micro is a new entrant to the market and has some catching up to do. Trend Micro is a new vendor to the application security market. In Forrester s evaluation, Trend Micro was one of a few vendors that did not support any SAST capabilities whatsoever, instead focusing only on DAST support from a cloud-only offering. Trend Micro is still developing its product capabilities and strategies and has a robust team with over 1,000 dedicated researchers (including global application security experts) focused on emerging threats and vulnerabilities. The Trend Micro offering is a cloud-based service that can dynamically scale to meet enterpriselevel demands but requires some more time in the market before it gains significant traction.

12 The Forrester Wave : Application Security, Q Supplemental Material Online Resource The online version of Figure 2 is an Excel-based vendor comparison tool that provides detailed product evaluations and customizable rankings. Data Sources Used In This Forrester Wave Forrester used a combination of three data sources to assess the strengths and weaknesses of each solution: Vendor surveys. Forrester surveyed vendors on their capabilities as they relate to the evaluation criteria. Once we analyzed the completed vendor surveys, we conducted vendor calls where necessary to gather details of vendor qualifications. Product demos. We asked vendors to conduct demonstrations of their product s functionality. We used findings from these product demos to validate details of each vendor s product capabilities. Customer reference calls. To validate product and vendor qualifications, Forrester also conducted reference calls with three of each vendor s current customers. The Forrester Wave Methodology We conduct primary research to develop a list of vendors that meet our criteria to be evaluated in this market. From that initial pool of vendors, we then narrow our final list. We choose these vendors based on: 1) product fit; 2) customer success; and 3) Forrester client demand. We eliminate vendors that have limited customer references and products that don t fit the scope of our evaluation. After examining past research, user need assessments, and vendor and expert interviews, we develop the initial evaluation criteria. To evaluate the vendors and their products against our set of criteria, we gather details of product qualifications through a combination of lab evaluations, questionnaires, demos, and/or discussions with client references. We send evaluations to the vendors for their review, and we adjust the evaluations to provide the most accurate view of vendor offerings and strategies. We set default weightings to reflect our analysis of the needs of large user companies and/or other scenarios as outlined in the Forrester Wave document and then score the vendors based on a clearly defined scale. These default weightings are intended only as a starting point, and we encourage readers to adapt the weightings to fit their individual needs through the Excel-based tool. The final scores generate the graphical depiction of the market based on current offering, strategy, and market presence. Forrester intends to update vendor evaluations regularly as product

13 The Forrester Wave : Application Security, Q capabilities and vendor strategies evolve. For more information on the methodology that every Forrester Wave follows, go to Integrity Policy All of Forrester s research, including Waves, is conducted according to our Integrity Policy. For more information, go to Endnotes 1 Source: Qualys Announces Third Quarter 2014 Financial Results, Qualys press release, November 3, 2014 (

14 About Forrester A global research and advisory firm, Forrester inspires leaders, informs better decisions, and helps the world s top companies turn the complexity of change into business advantage. Our researchbased insight and objective advice enable IT professionals to lead more successfully within IT and extend their impact beyond the traditional IT organization. Tailored to your individual role, our resources allow you to focus on important business issues margin, speed, growth first, technology second. for more information To find out how Forrester Research can help you be successful every day, please contact the office nearest you, or visit us at For a complete list of worldwide locations, visit Client support For information on hard-copy or electronic reprints, please contact Client Support at , , or [email protected]. We offer quantity discounts and special pricing for academic and nonprofit institutions. Forrester Focuses On Security & Risk Professionals To help your firm capitalize on new business opportunities safely, you must ensure proper governance oversight to manage risk while optimizing security processes and technologies for future flexibility. Forrester s subject-matter expertise and deep understanding of your role will help you create forward-thinking strategies; weigh opportunity against risk; justify decisions; and optimize your individual, team, and corporate performance. «Sean Rhodes, client persona representing Security & Risk Professionals Forrester Research (Nasdaq: FORR) is a global research and advisory firm serving professionals in 13 key roles across three distinct client segments. Our clients face progressively complex business and technology decisions every day. To help them understand, strategize, and act upon opportunities brought by change, Forrester provides proprietary research, consumer and business data, custom consulting, events and online communities, and peer-to-peer executive programs. We guide leaders in business technology, marketing and strategy, and the technology industry through independent fact-based insight, ensuring their business success today and tomorrow