NEXT-GENERATION FIREWALL

Transcription

1 NEXT-GENERATION FIREWALL INDIVIDUAL PRODUCT TEST RESULTS Check Point Power METHODOLOGY VERSION: 4.0 FEBRUARY 2011 Independent & unsponsored test report. Reprints Licensed to: Check Point Software Technologies This and other related documents available at: To receive a licensed copy or report misuse, please contact NSS Labs at +1 (512) or advisor@nsslabs.com.

2 2011 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors. Please note that access to or use of this report is conditioned on the following: 1. The information in this report is subject to change by NSS Labs without notice. 2. The information in this report is believed by NSS Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this report are at the reader s sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this report. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY THE NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader s expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners. CONTACT INFORMATION NSS Labs, Inc. P.O. Box Carlsbad, CA USA +1 (512) info@nsslabs.com NSS Labs, Inc. All rights reserved.

3 TABLE OF CONTENTS 1 Introduction What is an NGFW? Summary Results Security Effectiveness Firewall Policy Enforcement Baseline Policy Simple POlicy Complex Policies Static NAT (Network Address Translation) Dynamic/Hide NAT (Network Address Translation) Syn Flood Protection Address Spoofing Session Hijacking Intrusion Prevention Coverage by Attack Vector Coverage by Impact Type Attack Leakage Resistance to Evasion Application Control User/Group ID Aware Policies Performance Connection Dynamics Concurrency and Connection Rates HTTP Connections per Second and Capacity Real-World Traffic Mixes UDP Throughput Total Cost of Ownership Labor per Product (in Hours) Purchase Price and Total Cost of Ownership Value: Cost per Mbps and Exploit Blocked Tuned Policy Detailed Product Scorecard Appendix: Special Thanks NSS Labs, Inc. All rights reserved.

4 TABLE OF FIGURES Figure 1: Coverage by Attack Vector... 7 Figure 2: Product Coverage by Impact... 8 Figure 4: Concurrency and Connection Rates...11 Figure 5: HTTP Connections per Second and Capacity...12 Figure 6: Real-World Traffic Mixes...13 Figure 3: UDP Throughput NSS Labs, Inc. All rights reserved.

5 1 INTRODUCTION The Firewall is increasingly up for renewal in a growing number of organizations. Technology and contract cycles are coinciding with increased throughput rates and driving enterprises to re-evaluate current firewall technology and vendor options. No longer a commodity device, firewalls are being called to fulfill greater mandates in light of web 2.0 and other business drivers. Many enterprise IT managers are now faced with some difficult choices regarding one of information security s oldest technologies. Should I upgrade this model to a faster one from the same vendor, switch vendors, or upgrade to a so-called NGFW? This report analyzes the key technology issues and actual capabilities of the first Next Generation Firewall to be evaluated by NSS Labs. Firewall technology has been around for at least 25 years, and undergone several stages of development; from early packet and circuit firewalls to application layer and dynamic packet firewalls. Across these stages, the goal has continued to be to provide a protective barrier between internal and external networks, while allowing for productive communications to pass from one side to the other. With the emergence of new web applications and security threats, firewalls are again evolving. Whereas in the past we could say with a reasonable degree of certainty that application X runs over TCP port 552, and web traffic (and web traffic alone) runs over TCP port 80, this is no longer true today. Add to that, the rise of Web 2.0 and the proliferation of applications which bypass traditional firewall controls by tunneling over HTTP and HTTPS, and it becomes apparent that additional security controls (based upon the application vs. the port) must be added to firewalls. This means that relying on port and protocol combinations to define network applications is no longer enough. Firewalls need to be capable of performing deep packet inspection of all packets, on all ports and over all protocols in order to determine which applications are running over which ports. NSS Labs research indicates that over the past 18 months, the sophistication and strategic capabilities of cybercriminals has outstripped the pace of advancement within information security products. In addition to traditional remote attacks against servers, cybercriminals are increasingly waging highly targeted campaigns against desktop client applications. These campaigns include the use of encrypted websites (such as Gmail), social networking sites, advertising networks, and a long list of compromised websites. The Wall Street Journal, the New York Times,ESPN, and NASDAQ were all found to have been (inadvertently) dishing up exploits to their clients. As such, users need not venture into a dark corner of the Internet to be exploited. Some high profile examples of desktop clients being the primary attack vector are the Operation Aurora attack against Google and the numerous variants of the Zeus attack against financial institutions. Further, compromised systems often communicate back to command and control servers via ports 80 (HTTP) 443 (HTTPS), or DNS (53) since those ports are most likely not blocked by traditional firewalls, which define security policies in terms of IP Addresses, ports, protocols and services. Correspondingly, vendors have begun to market evolving technologies known as Next Generation Firewalls, based on nomenclature coined by Gartner. As a result, the team at NSS Labs decided to investigate the level to which different vendors are delivering next generation capabilities, and what the trade-offs are. As part of this research, we are conducting a group test to provide the industry with a current scientific baseline of NGFW effectiveness. Check Point was the first vendor to submit their solution for evaluation NSS Labs, Inc. All rights reserved. p. 1

6 1.1 WHAT IS AN NGFW? Firewalls are the cornerstone of network security. As Firewalls which will be deployed at critical chokepoints in the network, the stability and reliability of an NGFW is imperative. Therefore prime directive of any NGFW is that it must be as stable, as reliable, as fast, and as flexible as the existing firewall that it is replacing. In addition, an NGFW must provide granular control based upon applications, not just ports. This capability is needed to re-establish a secure perimeter where unwanted applications are not able to tunnel over HTTP or HTTPS. As such, granular application control is a requirement of NGFW since it enables the administrator to define security policies based upon applications vs. ports. For example, the administrator could block all Skype traffic while allowing Twitter apps. And while Application Control has received a lot of attention recently, research conducted by NSS Labs clearly indicates that enterprises are reluctant to embrace the technology beyond a limited scope. Our research shows that Enterprise Security wants to ensure that users are not bypassing the corporate firewall by tunneling Skype, peer-to-peer, instant messaging, and IRC applications over HTTP, which they view as a security concern. However Enterprise Security is reluctant to incurr additional responsibilities for policing users behavior. Enterprise Security views activity such as playing games on Facebook (i.e. Mafia Wars or Farmville) as something that might legitimately occur during lunch, and therefore corporate ownership should be HR and not Security. We were repeatedly told that the enterprise already had a web proxy/filter which had corporate sponsorship in the HR department and that it was managed by IT, not Security. As such, the policing of users is an HR concern, not a Security concern and belongs within web filtering solutions, not within the corporate firewall. Also important is the ability to identify users and groups and apply security policy based on identity. Where possible, this should be achieved via direct integration with existing enterprise authentication systems (such as Active Directory) without the need for custom server-side software. This allows the administrator to create even more granular policies. For example, it would be possible to restrict the use of Enterprise applications such as Salesforce.com and the Salesforce plug-in for Outlook to the sales department, while prohibiting use elsewhere in the company. Intrusion Prevention Systems (IPS) have become standard security devices in almost all sizes of enterprise. And enterprises are looking to consolidate IPS capabilities within the NGFW. Therefore an NGFW must apply full-strength IPS functionality such that it is as capable of identifying and blocking exploits as the existing IPS that it is replacing. Based on the needs identified in the previous section, the following capabilities are considered essential as part of a NGFW device: Traditional firewall including: o Basic packet filtering o Stateful multi-layer inspection o NAT o VPN o Highly Stable o High Availability Integrated IPS specifically with a strength in client-side protection Application awareness/control User/group control Ability to operate at layer 3 ( traditional ) or layer 2 ( bump in the wire ) 2011 NSS Labs, Inc. All rights reserved. p. 2

7 In this test, our engineers took the same approach that modern cyber criminals or hackers would in trying to breach the firewall. These efforts go far beyond replaying PCAPs or pressing the button on a single test tool. In short, our engineers executed fully weaponized attacks against the device under test. Performance: NGFW devices exhibit an inverse correlation between security effectiveness and performance. The more deep packet inspection is performed, the longer it takes to forward packets. Furthermore, it is important to consider a real-world mix of traffic that a device will encounter. NSS Labs utilizes a range of traffic types and mixes. Tuning: Security engineers tune an IPS to ensure its protection coverage matches the needs of the environment where it is being placed. This strategy works well for datacenters and DMZs. However, protecting desktops is a whole different matter. In surveying enterprises, we found most enterprises do not strictly control the desktop and that in larger enterprises it is safe to assume that pretty much anything can be running. As such, enterprises are expecting IPS and NGFW vendors to provide maximum security for desktop client applications with their recommended policies. Further, research indicates that enterprises are not ready to replace their dedicated IPS solutions in the datacenter. Simple deduction therefore tells us that intrusion prevention functionality within an NGFW needs to protect desktop clients with optimal protection pre-defined via a vendor recommended policy NSS Labs, Inc. All rights reserved. p. 3

8 2 SUMMARY RESULTS During Q4 2010, NSS Labs performed an independent test of the Check Point Power NGFW. The product was subjected to thorough testing at the NSS Labs facility in Austin, Texas, based on methodology v4.0 available on This test was conducted free of charge and NSS Labs did not receive any compensation in return for Check Point s participation. While the upcoming Next-Generation Firewall Group Test Report will provide comparative information about all tested products, this Individual Test Report provides detailed information not available elsewhere. As noted in the introduction to this report, enterprises do not plan on tuning the IPS within their NGFW for a variety of reasons. Therefore, NSS Labs evaluation of NGFW products is configured with the vendor pre-defined or default, out-of-the-box settings, in order to provide readers with relevant security effectiveness and performance dimensions based upon their expected usage. As part of this test, Check Point Software Technologies submitted the Power NSS Labs Rating: Recommend Product Overall Protection Client Protection Throughput Check Point Power % 83.3% 2,607 Mbps Stability & Reliability Firewall Enforcement Application Control Identity Aware Excellent 100% 100% 100% Using the default policy, the Power blocked 83.3% of attacks against client applications and 86.6% overall. In addition, the Check Point Power correctly identified 100% of our evasion attempts without error. The product successfully passed 2.6 Gbps of inspected traffic. NSS Labs rates throughput based upon tuned settings averaging out the results from tests 6.5.1, 6.5.2, and 6.4.2: Real World Protocol Mix (Perimeter), Real World Protocol Mix (Core), and 21 KB HTTP Response respectively. Check Point s management interface was well designed and intuitive. For users of Check Point firewalls, there will not be much of a learning curve. Tuning and maintenance is simple and wellthought out. For multi-gigabit environments looking to upgrade defenses from their current firewall to a NGFW, the Check Point Software Technologies Power provides excellent protection and an outstanding 3- year TCO (including labor) NSS Labs, Inc. All rights reserved. p. 4

9 3 SECURITY EFFECTIVENESS This section verifies that the DUT is capable of enforcing a specified security policy effectively. NSS Labs NGFW testing is conducted by incrementally building upon a baseline configuration (simple routing with no policy restrictions and no content inspection) to a complex real world multiple zone configuration supporting many addressing modes, policies, applications, and inspection engines. At each level of complexity, test traffic is passed across the DUT to ensure that only specified traffic is allowed and the rest is denied, and that appropriate log entries are recorded. The DUT must support stateful firewalling either by managing state tables to prevent traffic leakage or as a stateful proxy. The ability to manage firewall policy across multiple interfaces/zones is a required. At a minimum, the DUT must provide a trusted internal interface, an untrusted external/internet interface, and one or more DMZ interfaces. In addition, a dedicated management interface is preferred. 3.1 FIREWALL POLICY ENFORCEMENT Policies are rules that are configured on a firewall to permit or deny access from one network resource to another based on identifying criteria such as: source address, destination address, and service. A term typically used to define the demarcation point of a network where policy is applied is a demilitarized zone (DMZ). Policies are typically written to permit or deny network traffic from one or more of the following zones: Untrusted This is typically an external network and is considered to be an unknown and non-secure. An example of an untrusted network would be the Internet. DMZ This is a network that is being isolated by the firewall restricting network traffic to and from hosts contained within the isolated network. Untrusted (i.e. Internet) Trusted This is typically an internal network; a network that is considered secure and protected. NSS Labs tests the ability to enforce policy between the following: DMZ (i.e. hosted) Trusted to Untrusted Trusted to DMZ Untrusted to DMZ Untrusted to Trusted BASELINE POLICY Policy management was concise and intuitive. We were able to quickly implement our ANY-ANY baseline policy. Our testing determined that all traffic flowed correctly SIMPLE POLICY Trusted (i.e. protected) Building upon the baseline policy we established a simple policy allowing basic web browsing and access for internal clients heading outbound to the internet. We verified that the device correctly enforced this policy outbound traffic was allowed while inbound traffic was not allowed NSS Labs, Inc. All rights reserved. p. 5

10 3.1.3 COMPLEX POLICIES Next we created a complex policy allowing outbound web browsing and as well as inbound traffic to web, mail, DNS, and other traditional internet services. Our testing determined that the device enforced this policy correctly STATIC NAT (NETWORK ADDRESS TRANSLATION) Our testing determined that device correctly implements one-to-one Static NAT from a public IP Address to an internal RFC-1918 private IP Address DYNAMIC/HIDE NAT (NETWORK ADDRESS TRANSLATION) Our testing determined that the device correctly handles Dynamic/Hide Network Address Translation (NAT) where all range of private (RFC-1918) addresses hides behind a single public IP Address (external) SYN FLOOD PROTECTION Our testing determined that the DUT successfully protected against SYN Floods ADDRESS SPOOFING Our testing determined that the CheckPoint Power-1 firewall correctly prevented IP Address Spoofing. The user interface for defining which IP Address ranges and networks reside behind each of the device s interfaces was intuitive and functioned properly. In addition, CheckPoint provides a wizard to assist in configuration for those administrators with less experience SESSION HIJACKING "Why don't computers realize when a session is hijacked?" The answer lies in the way and the type of information that is transferred by the TCP packets and IP packets. Since TCP is connection-oriented it carries only the Sequence Number and not the IP address. The IP packet on the other hand carries only the IP address and not the Sequence Number. Our testing determined that the CheckPoint Power-1 firewall correctly prevented TCP Session Hijacking by tracking the IP addresses associated with each TCP SQN and verifying the IP Addresses do not change during the session. 3.2 INTRUSION PREVENTION In order to accurately represent the protection that an enterprise is likely to achieve, NSS Labs evaluated the products using the best pre-defined default, out-of-the-box settings that ships with the product. Live Exploit Testing: NSS Labs security effectiveness testing leverages deep expertise of our engineers utilizing multiple commercial, open source and proprietary tools as appropriate. With 1,179 live exploits, this is the industry s most comprehensive test to date. We retired 92 attacks and added 112 new exploits compared to our Q test set of 1,159. Most notable, all of the live exploits and payloads in our test have been validated in our lab such that: a reverse shell is returned a bind shell is opened on the target allowing the attacker to execute arbitrary commands a malicious payload installed a system is rendered unresponsive etc NSS Labs, Inc. All rights reserved. p. 6

11 Configuration Total Number of Exploits Run Total Number Blocked Block Percentage Default Configuration 1,179 1, % COVERAGE BY ATTACK VECTOR Because a failure to block attacks could result in significant compromise and impact to critical business systems, Next-Generation Firewalls should be evaluated against a broad set of exploits. Exploits can be categorized into two groups: attacker-initiated and target initiated. Attacker-initiatied exploits are threats executed remotely against a vulnerable application and/or operating system by an individual while target-initiatied exploits are initiated by the vulnerable target. In target-initatied exploits, the attacker has little or no control as to when the threat is executed. Block Rate 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 90% Attacker Initiated 83% Target Initiated Exploits Attempted/Caught Attempted Caught Coverage 90% 83% Figure 1: Coverage by Attack Vector COVERAGE BY IMPACT TYPE The most serious exploits are those which result in a remote system compromise, providing the attacker with the ability to execute arbitrary system-level commands. Most exploits in this class are weaponized and offer the attacker a fully interactive remote shell on the target client or server. Slightly less serious are attacks that result in an individual service compromise, but not arbitrary system-level command execution. Typical attacks in this category include service-specific attacks such as SQL injection that enable an attacker to execute arbitrary SQL commands within the database service. These attacks are somewhat isolated to the service and do not immediately result in full system-level access to the operating system and all services. However, using additional localized system attacks, it may be possible for the attacker to escalate from the service level to the system level. Finally, there are the attacks (often target initiated) which result in a system or service-level fault that crashes the targeted service or application and requires administrative action to restart the service or reboot the system. These attacks do not enable the attacker to execute arbitrary commands. Still, the resulting impact to the business could be severe, as the attacker could crash a protected system or service NSS Labs, Inc. All rights reserved. p. 7

12 % 90% % % Exploits % 50% 40% Coverage % % % 10% 0 System Exposure Service Exposure System-Service Fault Run Blocked Percentage 86% 94% 88% 0% Figure 2: Product Coverage by Impact ATTACK LEAKAGE Unlike NIPS, a Firewall must never allow traffic to pass without inspection in bypass mode. The Check Point Power will drop new connections when resources (such as state table memory) are low, or when traffic loads exceed the device capacity. This will theoretically block legitimate traffic, but maintain state on existing connections (preventing evasion). This is the correct response and prevents attack leakage. 3.3 RESISTANCE TO EVASION Description IP Packet Fragmentation TCP Stream Segmentation RPC Fragmentation URL Obfuscation HTML Evasion FTP Evasion TOTAL Check Point Power Resistance to known evasion techniques was perfect, with the Check Point Power achieving a 100% score across the board in all related tests. IP fragmentation, TCP stream segmentation, RPC fragmentation, URL obfuscation, HTML Evasion and FTP evasion all failed to trick the product into ignoring valid attacks. Not only were the fragmented and obfuscated attacks blocked successfully, but all of them were also decoded accurately NSS Labs, Inc. All rights reserved. p. 8

13 3.4 APPLICATION CONTROL While Application Control has received a lot of attention recently, research conducted by NSS Labs clearly indicates that enterprises are reluctant to embrace the technology beyond a limited scope. Our research shows that Enterprise Security wants to ensure that users are not bypassing the corporate firewall by tunneling Skype, peer-to-peer, instant messaging, and IRC applications over HTTP, which they view as a security concern. However Enterprise Security is reluctant to incurr additional responsibilities for policing users behavior. Enterprise Security views activity such as playing games on Facebook (i.e. Mafia Wars or Farmville) as something that might legitimately occur during lunch, and therefore corporate ownership should be HR and not Security. We were repeatedly told that the enterprise already had a web proxy/filter which had corporate sponsorship in the HR department and that it was managed by IT, not Security. As such, the policing of users is an HR concern, not a Security concern and belongs within web filtering solutions, not within the corporate firewall. Therefore, NSS Labs test methodology is based upon the narrower definition of application control that we heard espoused by enterprise security. Our testing found that CheckPoint Power correctly enforced complex outbound and inbound policies consisting of many rules, objects and applications. We verified that the device successfully determined the correct application and took the appropriate action based upon the policy. For example, the NGFW allowed instant messaging text communications while blocking IM file transfers. Popular Social Networking Websites (Web Applications) Instant Messaging Skype and other VoIP Torrents However, while CheckPoint was able to correctly identify and control applications, through the course of testing we found that the application identification telemetry was not being provided to the IPS Blade. As such, IPS protection is limited to standard ports (i.e. HTTP = Port 80). Attacks using nonstandard ports are not inspected by the IPS. (i.e. HTTP over port 8327 may contain an exploit against a common web browser, but will not be inspected). This product limitation means that administrators should still create policies limiting outbound access to standard ports such as 80 and 443. Alternatively, there is a check box which allows an administrator to enable application control on every port and another which enables http inspection on every port. These are not enabled by default, and therefore the impact on performance is unknown since the device was tested using the vendor pre-defined / default settings. 3.5 USER/GROUP ID AWARE POLICIES Integrating Check Point Smart Center (management station) with our Active Directory implementation was simple and intuitive. Our testing found that Check Point Power correctly enforced complex outbound and inbound policies consisting of many rules, objects and applications. We verified that the device successfully identified the users and groups and took the appropriate action based upon the firewall policy. For example the firewall allowed users in the IT group to SSH to a server in the DMZ while blocking all other users NSS Labs, Inc. All rights reserved. p. 9

14 The following table illustrates Users & Groups + Firewall and Application Control Policies that were defined and successfully verified. Users David (Sales Person) Jay (DB Administrator) Jeff (Operations) Pam (Controller) Richard (VP of Marketing) Scott (Auditor) Application Salesforce.com MySQL DB + SSH ERP Accounting software ALL Accounting software Groups Accounting Consultant Executive IT Operations Sales Applications Accounting software ERP ALL SSH ERP Salesforce.com 2011 NSS Labs, Inc. All rights reserved. p. 10

15 4 PERFORMANCE There is frequently a trade-off between security effectiveness and performance. Because of this tradeoff, it is important to judge a product s security effectiveness within the context of its performance (and vice versa). This ensures that new security protections do not adversely impact performance and security shortcuts are not taken to maintain or improve performance. 4.1 CONNECTION DYNAMICS CONCURRENCY AND CONNECTION RATES The aim of these tests is to stress the detection engine and determine how the sensor copes with large numbers of TCP connections per second, application layer transactions per second, and concurrent open connections. All packets contain valid payload and address data and these tests provide an excellent representation of a live network at various connection/transaction rates. Note that in all tests, the following critical breaking points where the final measurements are taken are used: Excessive concurrent TCP connections - latency within the firewall is causing unacceptable increase in open connections on the server-side. Excessive response time for HTTP transactions/smtp sessions - latency within the firewall is causing excessive delays and increased response time to the client. Unsuccessful HTTP transactions/smtp sessions normally, there should be zero unsuccessful transactions. Once these appear, it is an indication that excessive latency within the firewall is causing connections to time out. 450, , ,000 70, ,000 60,000 Conncurrent Connections 350, , , , , ,000 50,000 20,000 14,700 58,000 50,000 40,000 30,000 20,000 10,000 Connections/Sec 0 without data with data TCP Connections/Sec 20,000 HTTP Connections/Sec 14,700 HTTP Transactions/Sec 58,000 Concurrent TCP Conns 413, ,000 0 Figure 3: Concurrency and Connection Rates 2011 NSS Labs, Inc. All rights reserved. p. 11

16 4.2 HTTP CONNECTIONS PER SECOND AND CAPACITY These tests aim to stress the HTTP detection engine in order to determine how the sensor copes with detecting and blocking exploits under network loads of varying average packet size and varying connections per second. By creating genuine session-based traffic with varying session lengths, the sensor is forced to track valid TCP sessions, thus ensuring a higher workload than for simple packetbased background traffic. Each transaction consists of a single HTTP GET request and there are no transaction delays (i.e. the web server responds immediately to all requests). All packets contain valid payload (a mix of binary and ASCII objects) and address data. This test provides an excellent representation of a live network (albeit one biased towards HTTP traffic) at various network loads. 3,000 14,000 Megabits per Second 2,500 2,000 1,500 1, ,000 10,000 8,000 6,000 4,000 2,000 Connections / Sec 0 44 KB Response 21 KB Response 10 KB Response 4.5 KB Response 1.7 KB Response 0 CPS 6,250 10,250 11,500 12,800 13,200 Mbps 2,500 2,050 1, Figure 4: HTTP Connections per Second and Capacity 4.3 REAL-WORLD TRAFFIC MIXES The aim of this test is to measure the performance of the device under test in a real world environment by introducing additional protocols and real content, while still maintaining a precisely repeatable and consistent background traffic load. Different protocol mixes are utilized based on the location of the device under test to reflect real use cases. For details about real world traffic protocol types and percentages, see the NSS Labs NGFW Test Methodology, available at NSS Labs, Inc. All rights reserved. p. 12

17 4,000 3,500 3,000 3,800 Mbps 2,500 2,000 1,500 1, ,970 0 Real World Protocol Mix (Perimeter) Real World Protocol Mix (Core) Mbps 3,800 1,970 Figure 5: Real-World Traffic Mixes 4.4 UDP THROUGHPUT The aim of this test is purely to determine the raw packet processing capability of each in-line port pair of the device. It is not real world, and can be misleading. It is included here primarily for legacy purposes. This traffic does not attempt to simulate any form of real-world network condition. No TCP sessions are created during this test, and there is very little for the detection engine to do in the way of protocol analysis (although each vendor will be required to write a signature to detect the test packets to ensure that they are being passed through the detection engine and not fast-tracked from the inbound to outbound port). 14,000 12,000 11,400 12,050 Megabits per Second 10,000 8,000 6,000 4,000 2,000 1,900 3,500 6, Byte Packets 256 Byte Packets 512 Byte Packets 1024 Byte Packets 1514 Byte Packets Mbps 1,900 3,500 6,750 11,400 12,050 Figure 6: UDP Throughput 2011 NSS Labs, Inc. All rights reserved. p. 13

18 5 TOTAL COST OF OWNERSHIP Next-Generation Firewall solutions can be complex projects with several factors affecting the overall cost of deployment, maintenance and upkeep. All of these should be considered over the course of the useful life of the solution. Product Purchase the cost of acquisition. Product Maintenance the fees paid to the vendor. Installation the time required to take the device out of the box, configure it, put it into the network, apply updates and patches, initial tuning, and set up desired logging and reporting. Upkeep the time required to apply periodic updates and patches from vendors, including hardware, software, and protection (signature/filter/rules) updates. Tuning the time required to configure the policy such that the best possible protection is applied while reducing or eliminating false alarms and false positives. NSS Labs assumes enterprises will use pre-defined vendor policies and therefore eliminating tuning. 5.1 LABOR PER PRODUCT (IN HOURS) This table estimates the annual labor required to maintain each device. Since vendors sent their very best engineers to tune, NSS Labs assumptions are based upon the time required by a highly experienced security engineer ($75 per hour fully loaded). This allowed us to hold the talent cost variable constant and measure only the difference in time required to tune. Product Installation (Hrs) Upkeep / Year (Hrs) Tuning / Year (Hrs) Check Point Power PURCHASE PRICE AND TOTAL COST OF OWNERSHIP Each vendor provided pricing information. When possible, we selected the 24/7 maintenance and support option with 24-hour replacement as this is the option most organizations will select. Product Purchase Maintenance / year 1 Year TCO 2 Year TCO 3 Year TCO Check Point Power $60,000 $15,000 $77,475 $94,350 $111,225 Year One TCO was determined by multiplying the Labor Rate ($75 per hour fully loaded) x (Installation + Upkeep + Tuning) and then adding the Purchase Price + Maintenance. Year Two TCO was determined by multiplying the Labor Rate ($75 per hour fully loaded) x (Upkeep + Tuning) and then adding Year One TCO. Year Three TCO was determined by multiplying the Labor Rate ($75per hour fully loaded x (Upkeep + Tuning) and then adding Year Two TCO NSS Labs, Inc. All rights reserved. p. 14

19 5.3 VALUE: COST PER MBPS AND EXPLOIT BLOCKED TUNED POLICY There is a clear difference between price and value. The least expensive product does not necessarily offer the greatest value if it blocks fewer exploits than competitors. The best value is a product with a low TCO and high level of secure throughput (security effectiveness x performance). The following table illustrates the relative cost per unit of work performed: Mbps-Protected Product Protection Throughput 3 Year TCO Price / Mbps-Protected Check Point Power % 2,607 $111,225 $49 Price per Protected Mbps was calculated by taking the Three-Year TCO and dividing it by the product of Protection x Throughput. Three-Year TCO/(Protection x Throughput) = Price/Mbps-Protected NSS Labs, Inc. All rights reserved. p. 15

20 6 DETAILED PRODUCT SCORECARD The following chart depicts the status of each test with quantitative results where applicable. A separate product Exposure Report details specific vulnerabilities that are not protected. Test ID Description Result 3 Security Effectiveness 3.1 Firewall Policy Enforcement Baseline Policy 100% Simple Policy 100% Complex Policy 100% Static NAT 100% Dynamic / Hide NAT 100% Syn Flood Protection 100% Address Spoofing Protection 100% Session Hijacking Protection 100% 3.2 Intrusion Prevention Coverage By Attack Vectors Attacker Initiated 90% Target Initiated 83.3% Combined Total 86.6% Coverage By Impact Type System Exposure 86% Service Exposure 94% System or Service Fault 88% Attack Leakage Coverage by Target Type *See Vulnerability Scope Coverage by Result *See Vulnerability Scope Coverage by Vendor *See Vulnerability Scope 3.3 Evasion 100% Packet Fragmentation 100% Ordered 8 byte fragments 100% Ordered 24 byte fragments 100% Out of order 8 byte fragments 100% Ordered 8 byte fragments, duplicate last packet 100% Out of order 8 byte fragments, duplicate last packet 100% Ordered 8 byte fragments, reorder fragments in reverse 100% Ordered 16 byte frags, fragment overlap (favor new) 100% Ordered 16 byte frags, fragment overlap (favor old) 100% Out of order 8 byte fragments, interleaved duplicate packets scheduled for later delivery 100% Stream Segmentation 100% Ordered 1 byte segments, interleaved duplicate segments with invalid TCP checksums 100% Ordered 1 byte segments, interleaved duplicate segments with null TCP control flags 100% Ordered 1 byte segs, interleaved duplicate segments with requests to resync sequence numbers mid-stream 100% Ordered 1 byte segments, duplicate last packet 100% Ordered 2 byte segments, segment overlap (favor new) 100% Ordered 1 byte segments, interleaved duplicate segments with outof-window sequence numbers 100% Out of order 1 byte segments 100% Out of order 1 byte segments, interleaved duplicate segments with faked retransmits 100% Ordered 1 byte segments, segment overlap (favor new) 100% Out of order 1 byte segs, PAWS elimination (interleaved dup segs with older TCP timestamp options) 100% 2011 NSS Labs, Inc. All rights reserved. p. 16

21 Test ID Description Result Ordered 16 byte segs, seg overlap (favor new (Unix)) 100% RPC Fragmentation 100% One-byte fragmentation (ONC) 100% Two-byte fragmentation (ONC) 100% All fragments, including Last Fragment (LF) will be sent in one TCP segment (ONC) 100% All frags except Last Fragment (LF) will be sent in one TCP segment LF will be sent in separate TCP seg (ONC) 100% One RPC fragment will be sent per TCP segment (ONC) 100% One LF split over more than one TCP segment. In this case no RPC fragmentation is performed (ONC) 100% Canvas Reference Implementation Level 1 (MS) 100% Canvas Reference Implementation Level 2 (MS) 100% Canvas Reference Implementation Level 3 (MS) 100% Canvas Reference Implementation Level 4 (MS) 100% Canvas Reference Implementation Level 5 (MS) 100% Canvas Reference Implementation Level 6 (MS) 100% Canvas Reference Implementation Level 7 (MS) 100% Canvas Reference Implementation Level 8 (MS) 100% Canvas Reference Implementation Level 9 (MS) 100% Canvas Reference Implementation Level 10 (MS) 100% URL Obfuscation 100% URL encoding - Level 1 (minimal) 100% URL encoding - Level 2 100% URL encoding - Level 3 100% URL encoding - Level 4 100% URL encoding - Level 5 100% URL encoding - Level 6 100% URL encoding - Level 7 100% URL encoding - Level 8 (extreme) 100% Premature URL ending 100% Long URL 100% Fake parameter 100% TAB separation 100% Case sensitivity 100% Windows \ delimiter 100% Session splicing 100% HTML Obfuscation 100% UTF-16 character set encoding (big-endian) 100% UTF-16 character set encoding (little-endian) 100% UTF-32 character set encoding (big-endian) 100% UTF-32 character set encoding (little-endian) 100% UTF-7 character set encoding 100% Chunked encoding (random chunk size) 100% Chunked encoding (fixed chunk size) 100% Chunked encoding (chaffing) 100% Compression (Deflate) 100% Compression (Gzip) 100% Base-64 Encoding 100% Base-64 Encoding (shifting 1 bit) 100% Base-64 Encoding (shifting 2 bits) 100% Base-64 Encoding (chaffing) 100% Combination UTF-7 + Gzip 100% FTP Evasion 100% Inserting spaces in FTP command lines 100% Inserting non-text Telnet opcodes - Level 1 (minimal) 100% Inserting non-text Telnet opcodes - Level 2 100% 2011 NSS Labs, Inc. All rights reserved. p. 17

22 Test ID Description Result Inserting non-text Telnet opcodes - Level 3 100% Inserting non-text Telnet opcodes - Level 4 100% Inserting non-text Telnet opcodes - Level 5 100% Inserting non-text Telnet opcodes - Level 6 100% Inserting non-text Telnet opcodes - Level 7 100% Inserting non-text Telnet opcodes - Level 8 (extreme) 100% 3.4 Application Control Block Unwanted Applications 100% Block Specific Action 100% 3.5 User / Group ID Aware Policies Users Defined via NGFW Integration with Active Directory 100% Users Defined in NGFW DB (Alternate to 5.3.1) 100% 4 Performance 4.1 Raw Packet Processing Performance (UDP Traffic) Mbps Byte Packets 1, Byte Packets 3, Byte Packets 6, Byte Packets 11, Byte Packets 12, Maximum Capacity Theoretical Max. Concurrent TCP Connections 413, Theoretical Max. Concurrent TCP Connections w/data 401, Stateful Protection at Max Concurrent Connections PASS Maximum TCP Connections Per Second 20, Maximum HTTP Connections Per Second 14, Maximum HTTP Transactions Per Second 58, HTTP Capacity With No Transaction Delays Connections Per Second 44Kbyte Response 6, ,000 Connections Per Second 21Kbyte Response 10, ,000 Connections Per Second 10Kbyte Response 11, ,000 Connections Per Second 4.5Kbyte Response 12, ,000 Connections Per Second 1.7Kbyte Response 13, Real World Traffic Mbps Real World Protocol Mix (Perimeter) 3, Real World Protocol Mix (Core) 1, Latency - UDP Microseconds Byte Packets Byte Packets Byte Packets Byte Packets Byte Packets Application Average Response Time - HTTP Milliseconds Connections Per Second 44Kbyte Response ,000 Connections Per Second 21Kbyte Response ,000 Connections Per Second 10Kbyte Response ,000 Connections Per Second 4.5Kbyte Response ,000 Connections Per Second 1.7Kbyte Response Behavior Of The State Engine Under Load Attack Detection/Blocking - Normal Load 100% State Preservation - Normal Load 100% Pass Legitimate Traffic - Normal Load 100% State Preservation - Maximum Exceeded 100% Drop Traffic - Maximum Exceeded 100% 5 Stability & Reliability 5.1 Blocking Under Extended Attack Yes 5.2 Passing Legitimate Traffic Under Extended Attack Yes 5.3 Protocol Fuzzing Resilient 2011 NSS Labs, Inc. All rights reserved. p. 18

23 Test ID Description Result 5.4 Protocol Mutation Resilient 5.5 Power Fail Correct 5.6 Redundancy Yes 5.7 Persistence of Data Correct 6 Management & Configuration Costs 6.1 Ease of Use Initial Setup (Hours) Time Required for Upkeep (Hours per Year) Time Required to Tune (Hours per Year) Expected Costs Initial Purchase $60, Ongoing Maintenance & Support (Annual) $15, Installation Labor Cost (@$75/hr) $ Management Labor Cost (per $1, Tuning Labor Cost (per $0 6.3 Total Cost of Ownership Year 1 $77, Year 2 $16, Year 3 $16, Year Total Cost of Ownership $111, NSS Labs, Inc. All rights reserved. p. 19

24 7 APPENDIX: SPECIAL THANKS Special thanks go to our test infrastructure partners who provide much of the equipment, software, and support that make this testing possible: 2011 NSS Labs, Inc. All rights reserved. p. 20