TEST METHODOLOGY. Next Generation Firewall (NGFW) v5.4

TEST METHODOLOGY Next Generation Firewall (NGFW) v5.4

Table of Contents 1 Introduction... 5 1.1 The Need For Next Generation Firewalls (NGFW)... 5 1.2 About This Test Methodology And Report... 5 1.3 Inclusion Criteria... 7 2 Product Guidance... 8 2.1 Recommended... 8 2.2 Neutral... 8 2.3 Caution... 8 3 Security Effectiveness... 9 3.1 Firewall Policy Enforcement... 9 3.1.1 Baseline Policy... 10 3.1.2 Simple Policies... 10 3.1.3 Complex Policies... 10 3.1.4 Static NAT (Network Address Translation)... 10 3.1.5 Dynamic/Hide NAT (Network Address Translation)... 10 3.1.6 SYN Flood Protection... 10 3.1.7 IP Address Spoofing... 10 3.1.8 TCP Split Handshake Spoof... 10 3.2 Application Control... 11 3.2.1 Block... 11 3.2.2 Block Specific Action (Depends on the application)... 11 3.3 User/Group ID Aware Policies... 11 3.3.1 Users Defined Via NGFW Integration With Active Directory... 11 3.3.2 Users Defined In NGFW DB... 11 3.4 Intrusion Prevention Policies... 12 3.4.1 False Positive Testing... 12 3.4.2 Coverage By Attack Vector... 13 3.4.3 Coverage By Impact Type... 13 3.4.4 Coverage By Date... 14 3.4.5 Coverage By Vendor... 15 3.4.6 Coverage By Result... 15 3.4.7 Target Type... 16 3.5 Evasion... 16 3.5.1 IP Packet Fragmentation... 17 3.5.2 Stream Segmentation... 17 3.5.3 RPC Fragmentation... 17 3.5.4 SMB & NetBIOS Evasions... 18 3.5.5 URL Obfuscation... 18 2013 NSS Labs, Inc. All rights reserved. 2

3.5.6 HTML Obfuscation... 18 3.5.7 Payload Encoding... 20 3.5.8 FTP Evasion... 20 3.5.9 Layered Evasions... 20 4 Performance... 21 4.1 Raw Packet Processing Performance (UDP Traffic)... 21 4.1.1 64 Byte Packets... 21 4.1.2 128 Byte Packets... 21 4.1.3 256 Byte Packets... 21 4.1.4 512 Byte Packets... 21 4.1.5 1024 Byte Packets... 21 4.1.6 1514 Byte Packets... 21 4.2 Latency... 22 4.2.1 64 Byte Frames... 22 4.2.2 128 Byte Frames... 22 4.2.3 256 Byte Packets... 22 4.2.4 512 Byte Packets... 22 4.2.5 1024 Byte Packets... 22 4.2.6 1514 Byte Packets... 22 4.3 Maximum Capacity... 22 4.3.1 Theoretical Maximum Concurrent TCP Connections... 23 4.3.2 Theoretical Maximum Concurrent TCP Connections With Data... 23 4.3.3 Maximum TCP Connections Per Second... 23 4.3.4 Maximum HTTP Connections Per Second... 23 4.3.5 Maximum HTTP Transactions Per Second... 23 4.4 HTTP Capacity With No Transaction Delays... 24 4.4.1 44KB HTTP Response Size 2,500 Connections Per Second... 24 4.4.2 21KB HTTP Response Size 5,000 Connections Per Second... 24 4.4.3 10KB HTTP Response Size 10,000 Connections Per Second... 24 4.4.4 4.5KB HTTP Response Size 20,000 Connections Per Second... 24 4.4.5 1.7KB HTTP Response Size 40,000 Connections Per Second... 24 4.5 Application Average Response Time: HTTP... 25 4.6 HTTP Capacity with Transaction Delays... 25 4.6.1 21 KByte HTTP Response With Delay... 25 4.6.2 10 KByte HTTP Response With Delay... 25 4.7 Real- World Traffic... 25 4.7.1 Real- World Protocol Mix (Enterprise Perimeter)... 25 4.7.2 Real- World Protocol Mix (Financial)... 25 4.7.3 Real- World Protocol Mix (Education)... 26 4.7.4 Real- World Protocol Mix (Data Center)... 26 2013 NSS Labs, Inc. All rights reserved. 3

4.7.5 Real- World Protocol Mix (US Mobile Carrier)... 26 4.7.6 Real- World Protocol Mix (European Mobile Carrier)... 26 5 Stability & Reliability... 27 5.1 Blocking Under Extended Attack... 27 5.2 Passing Legitimate Traffic Under Extended Attack... 27 5.3 Behavior Of The State Engine Under Load... 27 5.3.1 Attack Detection/Blocking Normal Load... 28 5.3.2 State Preservation Normal Load... 28 5.3.3 Pass Legitimate Traffic Normal Load... 28 5.3.4 State Preservation Maximum Exceeded... 28 5.3.5 Drop Legitimate Traffic Maximum Exceeded... 28 5.4 Protocol Fuzzing & Mutation... 28 5.5 Power Fail... 28 5.6 Redundancy... 29 5.7 Persistence Of Data... 29 5.8 High Availability (HA) Option... 29 5.8.1 Failover Legitimate Traffic... 29 5.8.2 Failover Malicious Traffic... 29 5.8.3 Time To Failover... 29 5.8.4 Stateful Operation... 29 5.8.5 Active- Active Configuration... 29 6 Management & Configuration... 30 7 Total Cost of Ownership & Value... 31 Appendix A: Test Environment... 32 Appendix B: Change Log... 33 Contact Information... 34 2013 NSS Labs, Inc. All rights reserved. 4

1 Introduction 1.1 The Need For Next Generation Firewalls (NGFW) Firewall technology is one of the largest and most mature security markets. Firewalls have undergone several stages of development, from early packet filtering and circuit relay firewalls to application layer (proxy based) and dynamic packet filtering firewalls. Throughout their history, however, the goal has been to enforce an access control policy between two networks, and they should therefore be viewed as an implementation of policy. A firewall is a mechanism used to protect a trusted network from an untrusted network, while allowing authorized communications to pass from one side to the other, thus facilitating secure business use of the Internet. With the emergence of new web applications and security threats, however, firewalls are evolving further. With the emergence of Web 2.0 trends pushing critical business applications through firewall ports that were previously reserved for a single function, such as HTTP, the legacy firewall technology is effectively blinded. Unable to differentiate between actual HTTP traffic and non- HTTP services tunneling over port 80, such as VoIP or instant messaging, the addition of application level monitoring rather than simply port and destination is essential. As a consequence of this, firewalls are evolving further. This means it is no longer possible to rely on port and protocol combinations alone to define network applications. The NGFW therefore requires the capability to perform deep packet inspection of all packets, on all ports and over all protocols in order to determine which applications are running over which ports and thus secure them effectively. 1.2 About This Test Methodology And Report NSS Labs test reports are designed to address the challenges faced by IT professionals in selecting and managing security products. The scope of this particular report includes: Security effectiveness Performance and stability Management Total cost of ownership (TCO) As firewalls are deployed at critical choke points in the network, the stability and reliability of a NGFW is imperative. Therefore, regardless of any new deep inspection capabilities, the main requirement of any NGFW is that it must be as stable, as reliable, as fast, and as flexible as the existing firewall that it is replacing. 2013 NSS Labs, Inc. All rights reserved. 5

Based on the needs identified in NSS research, the following capabilities are considered essential in any NGFW device: Traditional first generation firewall, including: o o o o o o Basic packet filtering Stateful multi- layer inspection NAT VPN Highly Stable High Availability Application awareness/control User/group control Integrated IPS Ability to operate at layer 3 ( traditional ) External intelligence to enhance blocking decisions (i.e., reputation services ) Note that the ability to handle IPv6 traffic is becoming increasingly important. This methodology will include IPv6 for all tests in Section 4, Performance, as well as a Security Effectiveness section to validate the ability of the appliance to inspect and correctly identify malicious traffic over IPv6. If a vendor does not claim IPv6 support, there will be no penalty; the product will appear with IPv4 results only. Live Exploit Testing: NSS security effectiveness testing leverages the deep expertise of our engineers utilizing multiple commercial, open source, and proprietary tools as appropriate. With over 1,500 live exploits, this is the industry s most comprehensive test to date. Most notable, all of the live exploits and payloads in our test have been validated in our lab such that: a reverse shell is returned a bind shell is opened on the target allowing the attacker to execute arbitrary commands a malicious payload is installed a system is rendered unresponsive etc. This test goes far beyond replaying packet captures or pressing the button on a test tool. In short, our engineers trigger vulnerabilities for the purpose of validating that an exploit was able to pass through the device under test. Performance: NGFW devices exhibit an inverse correlation between security effectiveness and performance. The more deep packet inspection is performed, the longer it takes to forward packets. Furthermore, it is important to consider a real- world mix of traffic that a device will encounter. NSS utilizes a range of traffic types and mixes. Tuning: Security engineers will typically tune an IPS to ensure its protection coverage matches the needs of the environment where it is being placed. Though this strategy works well for data centers and de- militarized zones (DMZ), protecting desktops is a different matter. In surveying enterprises, NSS researchers discovered that many enterprises do not strictly control the desktop, and that in larger enterprises there can be a wide range of applications running on the typical endpoint. As such, enterprises are expecting IPS and NGFW vendors to provide maximum security for desktop client applications with their out- of- the- box recommended or default policies. In addition, research indicates that enterprises are not ready to replace their dedicated IPS solutions with NGFW in the data center. 2013 NSS Labs, Inc. All rights reserved. 6

This leads us to conclude that the intrusion prevention functionality within a NGFW will typically be used to protect desktop clients, with optimal protection pre- defined via a vendor- supplied recommended/default policy, and this is how the devices will be tested. 1.3 Inclusion Criteria In order to encourage the greatest participation, and to allay any potential concerns of bias, NSS invites all firewall vendors claiming NGFW capabilities to submit their products at no cost. Vendors with major market share, as well as challengers with new technology, will be included. The firewall should be supplied as a single appliance, where possible (cluster controller solutions are also acceptable), with the appropriate number of physical interfaces capable of achieving the required level of connectivity and performance (minimum of one in- line port pair per Gigabit of throughput, or one in- line 10Gbps port pair per 10Gbps of throughput). Firewall products should be implemented as in- line Layer 3 (routing) devices. Multiple separate 1Gbps or 10Gbps connections will be made from the external to internal switches via the device under test (DUT), subject to a minimum of one in- line port pair per Gigabit of throughput. Thus, an 8 Gbps device with only four port pairs will be limited to 4 Gbps. The maximum number of port pairs will be connected to determine the overall throughput of the DUT. Once installed in the test lab, the DUT will be configured for the use- case appropriate to the target deployment (corporate network perimeter). The DUT should also be configured to block all traffic when resources are exhausted or when traffic cannot be analyzed for any reason. 2013 NSS Labs, Inc. All rights reserved. 7

2 Product Guidance NSS issues summary product guidance based on evaluation criteria that is important to information security professionals. The evaluation criteria are weighted as follows: Security effectiveness The primary reason for buying a firewall is to separate internal trusted networks from external untrusted networks while allowing select controlled traffic to flow between trusted and untrusted. Resistance to evasion Failure in any evasion class permits attackers to circumvent protection. Stability Long- term stability is particularly important for an in- line device, where failure can produce network outages. Performance Correctly sizing a firewall is essential. Management In particular, how difficult is it to configure the highest degree of protection across multiple devices? Value Customers should seek low TCO and high effectiveness and performance rankings. Products are listed in rank order according to their guidance rating. 2.1 Recommended A Recommended rating from NSS indicates that a product has performed well and deserves strong consideration. Only the top technical products earn a Recommended rating from NSS, regardless of market share, company size, or brand recognition. 2.2 Neutral A Neutral rating from NSS indicates that a product has performed reasonably well and should continue to be used if it is the incumbent within an organization. Products that earn a Neutral rating from NSS deserve consideration during the purchasing process. 2.3 Caution A Caution rating from NSS indicates that a product has performed poorly. Organizations using one of these products should review their security posture and other threat mitigation factors, including possible alternative configurations and replacement. Products that earn a Caution rating from NSS should not be short- listed or renewed. 2013 NSS Labs, Inc. All rights reserved. 8

3 Security Effectiveness This section verifies that the DUT is capable of enforcing a specified security policy effectively. NSS firewall analysis is conducted by incrementally building upon a baseline configuration (simple routing with no policy restrictions and no content inspection) to a complex, real world, multiple- zone configuration supporting many addressing modes, policies, applications, and inspection engines. At each level of complexity, test traffic is passed across the firewall to ensure that only specified traffic is allowed and the rest is denied, and that appropriate log entries are recorded. The NGFW must support stateful firewalling either by managing state tables to prevent traffic leakage or as a stateful proxy. The ability to manage firewall policy across multiple interfaces/zones is a required function. At a minimum, the firewall must provide a trusted internal interface, an untrusted external/internet interface, and (optionally) one or more DMZ interfaces. In addition, a dedicated management interface (virtual or otherwise) is preferred. 3.1 Firewall Policy Enforcement Policies are rules that are configured on a firewall to permit or deny access from one network resource to another based on identifying criteria such as: source, destination, and service. A term typically used to define the demarcation point of a network where policy is applied is a demilitarized zone (DMZ). Policies are typically written to permit or deny network traffic from one or more of the following zones: Untrusted This is typically an external network and is considered to be unknown and non- secure. An example of an untrusted network would be the Internet. DMZ This is a network that is being isolated by the firewall restricting network traffic to and from hosts contained within the isolated network. Trusted This is typically an internal network; a network that is considered secure and protected. The NSS firewall tests verify performance and the ability to enforce policy between the following: Trusted to Untrusted Untrusted to DMZ Trusted to DMZ Note: Firewalls must provide at a minimum one DMZ interface in order to provide a DMZ or transition point between untrusted and trusted networks 2013 NSS Labs, Inc. All rights reserved. 9

3.1.1 Baseline Policy Routed configuration with an allow all policy. 3.1.2 Simple Policies Simple outbound and inbound policies allowing basic browsing and email access for internal clients and no external access 3.1.3 Complex Policies Complex outbound and inbound policies consisting of many rules, objects, and services. 3.1.4 Static NAT (Network Address Translation) Inbound network address translation (NAT) to DMZ using fixed IP address translation with one- to- one mapping. 3.1.5 Dynamic/Hide NAT (Network Address Translation) Outbound network address translation (NAT) (from internal to external), where all outbound traffic hides behind the IP address of the external interface of the firewall utilizing a pool of high ports to manage multiple connections. 3.1.6 SYN Flood Protection The basis of a SYN flood attack is to fail to complete the three- way handshake necessary to establish a legitimate session. The objective of SYN flooding is to disable one side of the TCP connection, which will result in one or more of the following: The server is unable to accept new connections. The server crashes or becomes inoperative. Authorization between servers is impaired. The DUT is expected to protect against SYN floods. 3.1.7 IP Address Spoofing This test attempts to confuse the firewall into allowing traffic to pass from one network segment to another. By forging the IP header to contain a different source address from where the packet was actually transmitted, an attacker can make it appear that the packet was sent from a different (trusted) machine. The endpoint that receives successfully spoofed packets will respond to the forged source address (the attacker). The DUT is expected to protect against IP address spoofing. 3.1.8 TCP Split Handshake Spoof This test attempts to confuse the firewall into allowing traffic to pass from one network segment to another. The TCP split handshake blends features of both the three- way handshake and the simultaneous- open connection. 2013 NSS Labs, Inc. All rights reserved. 10

The result is a TCP spoof attack that allows an attacker to bypass the firewall by instructing the target to initiate the session back to the attacker. Popular TCP/IP networking stacks respect this handshaking method, including Microsoft, Apple, and Linux stacks, with no modification. 1 The DUT is expected to protect against TCP split handshake spoofing. 3.2 Application Control Complex outbound and inbound policies consisting of many rules, objects, and applications, verifying that the DUT is capable of correctly determining the correct application from deep packet inspection (regardless of port/protocol used), and taking the appropriate action. Popular social networking web sites (web applications) Instant messaging (IM) Skype and other VoIP Torrents Other applications TBD For each application, NSS will test the DUT s ability to perform the following functions 3.2.1 Block The DUT should be able to correctly identify the application and block it 3.2.2 Block Specific Action (Depends on the application) For example, with instant messaging, the DUT should allow text communications while blocking file transfers. 3.3 User/Group ID Aware Policies Complex outbound and inbound policies consisting of many rules, objects and applications, verifying that the DUT is capable of correctly determining the correct user/group ID from deep packet inspection, and taking the appropriate action. 3.3.1 Users Defined Via NGFW Integration With Active Directory The DUT should be able to integrate with Microsoft Active Directory (AD) through direct connection to the Active Directory server or a vendor- supplied application installed on the AD Server. This integration should allow import of AD users and groups as well as allow the DUT to log and take action on AD events. 3.3.2 Users Defined In NGFW DB Should integration with Active Directory be unavailable, the firewall is expected to allow local creation of users/groups. In addition, the ability to import thousands of users and dozens of groups is highly desirable. 1 The TCP Split Handshake: Practical Effects on Modern Network Equipment, Tod Alien Beardsley & Jin Qian, http://www.macrothink.org/journal/index.php/npa/article/view/285 2013 NSS Labs, Inc. All rights reserved. 11

The following table is an example of Users & Groups + Firewall and Application Control Policies that will be defined. Users David (Sales Person) Jay (DB Administrator) Jeff (Operations) Pam (Controller) Richard (VP of Marketing) Scott (Auditor) Application Salesforce.com MySQL DB + SSH ERP Accounting software ALL Accounting software (Read Only) Groups Accounting Consultant Executive IT Operations Sales Applications Accounting software ERP ALL SSH ERP Salesforce.com 3.4 Intrusion Prevention Policies Policies consisting of threat protection signatures, verifying that the DUT is capable of correctly blocking malicious traffic based on a comparison of packet/session contents against signatures/filters/protocol decoders. The latest signature pack is acquired from the vendor s support site, and the DUT is deployed with the out- of- the- box recommended or default security policy. No tuning of the product is allowed by the vendor. NSS considers it unacceptable for a product of this nature to be sold without a recommended or default policy. No custom signatures are permitted in the testing. All signatures used must be available to the general public at the time of testing. Although intrusion detection systems operate in detection- only mode, an NGFW is required to block and log exploit attempts and hostile traffic. 3.4.1 False Positive Testing The ability of the DUT to identify and allow legitimate traffic while maintaining protection against threats and exploits is just as great as offering safety against malicious content. This test will include a varied sample of legitimate application traffic which should be identified and allowed or blocked, based on policy rules. 2013 NSS Labs, Inc. All rights reserved. 12

3.4.2 Coverage By Attack Vector Threats and exploits can be initiated by either the target or the attacker targeting either local or remote vulnerabilities. As a result, NSS categorizes threats and exploits into the following matrix: Network Local Attacker RPC Exploit Root Kit Target Browser Exploit Trojan *Example exploits included above for reference purposes. 3.4.2.1 Attacker Initiated Also referred to as server- side exploits, the threat/exploit is executed remotely by the attacker against a vulnerable application and/or operating system. 3.4.2.2 Target Initiated The threat/exploit is initiated by the vulnerable target. The attacker has little or no control as to when the target user or application will execute the threat. Given that NGFW devices are typically deployed to protect end users, this class of exploit is the main focus of the NGFW security effectiveness test. 3.4.2.3 Network Threats/exploits that are initiated as a result of network communication. 3.4.2.4 Local Local execution that requires existing access to the target (not applicable to NGFW). Protective ratings are reported in raw percentages of mitigated attacks and their resulting impact: system, service, fault, reconnaissance. Although a system or service exploit may be partially mitigated by the DUT, the service could have crashed because of the residual communications resulting in a fault impact on the service or operating system. 3.4.3 Coverage By Impact Type The NSS threat and attack suite contains thousands of publically- available exploits (including multiple variants of each exploit) from which groups of exploits are carefully selected to test based on appropriate usage. Each exploit has been validated to impact the target vulnerable host(s). Based on the impact of the threat against the target, the following metrics are reported: 3.4.3.1 System Exposure Attacks resulting in remote system compromise and the ability of the attacker to execute arbitrary system- level commands. Most exploits in this class that are weaponized will provide the attacker with a fully interactive remote shell on the target client or server. 2013 NSS Labs, Inc. All rights reserved. 13

3.4.3.2 Service Exposure Attacks resulting in an individual service compromise but not arbitrary system- level command execution. Typical attacks in this category include service specific attacks such as SQL injection that enable the attacker to execute arbitrary SQL commands within the database service. These attacks are somewhat isolated to the service and do not immediately result in full system- level access to the operating system and all services. However, using additional localized system attacks, it may be possible for the attacker to go from the service level to the system level. 3.4.3.3 System Or Service Fault Attacks resulting in a system or service- level fault that crashes the targeted service or application and requires administrative action to restart the service or reboot the system. These attacks do not enable the attacker to execute arbitrary commands. However, the resulting impact to the business could be severe given that the attacker could crash the protected system or service. 3.4.4 Coverage By Date The typical enterprise will run a mix of both old and new applications, and NSS research shows that crimeware kits will frequently include exploits that date back several years. Therefore, NSS security effectiveness testing will include exploits current at the time of the test, as well as targeting vulnerabilities covering multiple years dating backwards from the time of the test. Results will be reported by year for up to 10 years prior to the year of the test. Results prior to that time period, where applicable, will be aggregated into the oldest "bucket." 2013 NSS Labs, Inc. All rights reserved. 14

3.4.5 Coverage By Vendor NSS live exploit test contains many vendors, including but not limited to, the following list. Protection capabilities are indicated as percentages. 3Com Adobe Alt- N Apache Apple Atrium Avast BEA BitDefender Borland CA Cisco Citrix ClamAV EMC Facebook GNU Google HP IBM IPSwitch ISC Kaspersky LanDesk lighttpd Linux Macromedia MacroVision Mailenable McAfee Mercury Microsoft MIT Mozilla Mplayer Multiple Vendors MySQL NOD32 Novell Nullsoft OpenLDAP OpenOffice OpenSSH OpenSSL Oracle Other Misc Panda RealNetworks Samba SAP Snort Sophos SpamAssassin Squid Sun Microsystems Symantec Trend Micro Trillian UltraVNC Veritas VideoLan VMWare WinAmp WinFTP Winzip Yahoo 3.4.6 Coverage By Result The following results of exploitation are represented in NSS live exploit test. Protection capabilities are indicated as percentages. 3.4.6.1 Arbitrary Code Execution A software bug that allows an attacker to execute any commands of the attacker's choice on a target machine or in a target process 2013 NSS Labs, Inc. All rights reserved. 15

3.4.6.2 Buffer Overflow The exploitation of a software bug due to improperly establishing memory bounds allows an attacker to overwrite adjacent memory and execute a command. 3.4.6.3 Code Injection The exploitation of a software bug that allows the processing of invalid data within a program. Code injection can be used by an attacker to introduce code into a computer program to change the course of execution. 3.4.6.4 Cross- Site Script The exploitation of a web application that enables attackers to insert malicious script into web pages, which can then be executed by other users. 3.4.6.5 Directory Traversal The exploitation of a lack of security in an application (as opposed to exploiting a bug in the code) that allows user- supplied input with characters representing "traverse to parent directory" to be passed through to the file APIs. The goal of this attack is to order an application to access a file or executable that is not intended to be accessible. 3.4.6.6 Privilege Escalation This exploit type allows an attacker to gain access to resources that would not normally have been available. 3.4.7 Target Type The following list of web target types is represented in NSS live exploit test. Protection capabilities are indicated as percentages. Web Server ActiveX Web Browser JavaScript Browser Plug- ins/add- ons 3.5 Evasion Attackers can modify basic attacks to evade detection in a number of ways. If a DUT fails to detect a single form of evasion, any exploit can pass through the device, rendering it ineffective. NSS verifies that the DUT is capable of detecting and blocking basic exploits when subjected to varying common evasion techniques. Wherever possible, the DUT is expected to successfully decode the obfuscated traffic to provide an accurate alert relating to the original exploit, rather than alerting purely on anomalous traffic detected as a result of the evasion technique itself. A number of common exploits are executed across the DUT to ensure that they are detected in their unmodified state. These will be chosen from a suite of older/common basic exploits for which NSS is certain that all vendors will have signatures. None of the exploits that were used in Section 3.4 will be used as evasion baselines. This ensures that vendors are not provided with any information on the content of any part of the main NSS exploit library in advance of the test. 2013 NSS Labs, Inc. All rights reserved. 16

3.5.1 IP Packet Fragmentation These tests determine the effectiveness of the fragment reassembly mechanism of the NGFW. Fragments from 8 32 bytes in size Ordered, out- of- order, or reverse order fragments Fragment overlap, favoring new and favoring old data Interleaved, duplicate, duplicate with or without incrementing DWORD, duplicate packets with random payload, or duplicate packets scheduled for later delivery Any combination of the above methods It is a requirement of the test that the DUT submitted should have all IP fragmentation reassembly options enabled by default in the shipping product. 3.5.2 Stream Segmentation These tests determine the effectiveness of the stream reassembly mechanism of the NGFW. Segments from 1-2048 bytes in size Ordered, reverse ordered, or out- of- order segments, with favor old or favor new Duplicate, duplicate interleaved, duplicate last packet, or overlapping segments Invalid or NULL TCP control flags Sequence resync requests, random initial sequence number, or out- of- window sequence numbers Faked retransmits, PAWS elimination, or segments containing random data Endianness interchanged Any combination of the above methods It is a requirement of the test that the DUT submitted should have all TCP stream reassembly options enabled by default in the shipping product. 3.5.3 RPC Fragmentation Both Sun/ONC RPC and MS- RPC allow the sending application to fragment requests, and all MS- RPC services have a built- in fragmentation reassembly mechanism. An attacker can transmit the BIND followed by a single request fragmented over a hundred actual requests with small fragments of the malicious payload. Alternatively, the attacker could transmit both the BIND and request fragments in one large TCP segment, thus foiling any signatures which use a simple size check. NSS uses test tools that combine large writes with many tiny MS- RPC fragments and provide multiple levels of fragmentation. These tests determine the effectiveness of the RPC reassembly mechanism of the NGFW: Sun/Open Network Computing (ONC) RPC byte fragmentation Fragments sent in one or more TCP segments, with or without last fragment RPC fragmentation may or may not be performed in a single segment MSRPC messages are sent with interchangeable endianness 16-1024 byte MSRPC fragments are sent in the same lower layer message MSRPC requests are fragmented to contain at most 2048 bytes of payload Any combination of the above methods 2013 NSS Labs, Inc. All rights reserved. 17

3.5.4 SMB & NetBIOS Evasions Upper- layer protocols such as SMB and NetBIOS can be manipulated in various ways, including segmentation of SMB read/write operations, or inserting fake SMB requests and chaffed NetBIOS messages: SMB or NetBIOS chaff before real message Chaff may contain MSRPC, HTTP POST, HTTP GET, or random payload SMB filename obfuscation using case changes and dummy paths SMB writes are segmented to contain at most 2048 bytes of payload. 1-1024 bytes of random padding inserted into WriteAndX messages between SMB header and payload Any combination of the above methods 3.5.5 URL Obfuscation Random URL encoding techniques are employed to transform simple URLs that are often used in pattern- matching signatures to apparently meaningless strings of escape sequences and expanded path characters using a combination of the following techniques: Escape encoding (% encoding) Microsoft %u encoding Path character transformations and expansions ( /./, //, \ ) These techniques are combined in various ways for each URL tested, ranging from minimal transformation, to extreme (every character transformed). All transformed URLs are verified to ensure they still function as expected after transformation. URL encoding - Levels 1-8 (minimal to extreme) Premature URL ending Long URL Fake parameter TAB separation Case sensitivity Windows\delimiter Session splicing Any combination of the above methods 3.5.6 HTML Obfuscation Recognizing malicious HTML documents is becoming increasingly important when protecting the enterprise. Malicious HTML documents exploit flaws in common web browsers, browser plug- ins, and add- ons to gain control of the client system and silently install malware such as trojans, rootkits, and key loggers. Therefore, it is becoming increasingly important that security products charged with protecting end systems must correctly interpret HTML documents. Many security products use simple pattern matching systems with very little semantic or syntactic understanding of the data they are analyzing. This leaves them vulnerable to evasion through use of redundant, but equivalent, alternative representations of malicious documents. 2013 NSS Labs, Inc. All rights reserved. 18

This test suite uses a number of malicious HTML documents that are transferred from server to client through the DUT. Each malicious HTML document is served with a different form of obfuscation, as follows: UTF- 16 and UTF- 32 character set encoding (big- endian) UTF- 16 and UTF- 32 character set encoding (little- endian) UTF- 7 character set encoding Chunked encoding (random chunk size, fixed 8 byte chunk size, or chaffing/arbitrary numbers inserted between chunks) Compression (Deflate) Compression (Gzip) Base- 64 character set encoding with or without bit shifting or chaffing Any combination of the above methods The UTF- 16 character set specifies a 2- byte sequence for most characters and a 4- byte sequence for the others (a small percentage). Recoding an HTML document in UTF- 16 significantly changes its appearance. A document that contains just the ASCII subset of characters will appear to have a null byte between every one of the original characters. There are also two different forms of the UTF- 16 encoding depending on whether the null high byte comes first (big- endian) or second (little- endian) this test uses big- endian byte ordering The UTF- 32 character set specifies a 4- byte sequence. Like the UTF- 16 character set encoding, there are two variations big- endian and little- endian and this test case uses big- endian byte ordering. The UTF- 7 character set encodes most ASCII characters as themselves. However, in addition to recoding non- English characters as other encodings do, it also recodes many punctuation symbols, including many of the symbols that are important to the HTML specification. Therefore, recoding an HTML document in UTF- 7 significantly changes its appearance Chunked encoding allows the server to break a document into smaller chunks and transmit them individually. The server needs only to specify the size of each chunk before it is transmitted and then indicate when the last chunk has been transmitted. Since chunked encoding intersperses arbitrary numbers (chunk sizes) with the elements of the original document, it can be used to greatly change the appearance of the original document as observed on the wire. In addition, the server can choose to break the document into chunks at arbitrary points. This makes it difficult for simple pattern matching systems to reliably identify the original HTML document from the raw data on the network. Per RFC 2616, the HTTP protocol allows the client to request and the server to use several compression methods. These compression methods not only improve performance in many circumstances, they completely change the characteristic size and appearance of HTML documents. Furthermore, small changes in the original document can greatly change the final appearance of the compressed document. This property of these algorithms could be used to obfuscate hostile content for the purpose of evading detection. The deflate compression method is a Lempel- Ziv coding (LZ77), specified in RFC 1951. The gzip compression method is specified in RFC 1952. For each of the above, it is verified that a standard web browser (such as Internet Explorer) is capable of rendering the results of the evasion. 2013 NSS Labs, Inc. All rights reserved. 19

3.5.7 Payload Encoding This test attempts to confuse the NGFW into allowing an otherwise blocked exploit to pass using various encoding options that are standard within the Metasploit Framework: x86/call4_dword_xor This encoder implements a Call+4 Dword XOR Encoder x86/countdown This encoder uses the length of the payload as a position- dependent encoder key to produce a small decoder stub. x86/fnstenv_mov This encoder uses a variable- length mov equivalent instruction with fnstenv for getip. x86/jmp_call_additive This encoder implements a Jump/Call XOR Additive Feedback Encoder x86/shikata_ga_nai This encoder implements a polymorphic XOR additive feedback encoder. The decoder stub is generated based on dynamic instruction substitution and dynamic block ordering. Registers are also selected dynamically. 3.5.8 FTP Evasion When attempting FTP exploits, it is possible to evade some products by inserting additional spaces and telnet control sequences in FTP commands. These tests insert a range of valid telnet control sequences that can be parsed and handled by IIS FTP server and wu- ftpd, and which also conform to Section 2.3 of RFC 959. Control opcodes are inserted at random, ranging from minimal insertion (only one pair of opcodes), to extreme (opcodes between every character in the FTP command): Inserting spaces in FTP command lines Inserting non- text Telnet opcodes Levels 1-8 (minimal to extreme) Any combination of the above methods 3.5.9 Layered Evasions This test attempts to bypass the DUT by performing any legitimate combination of the evasion techniques specified in Section 3.5. It will be verified that the target machine s standard network stack is capable of decoding the evasion correctly while maintaining the exploit viability. 2013 NSS Labs, Inc. All rights reserved. 20

4 Performance This section measures the performance of the DUT using various traffic conditions that provide metrics for real- world performance. Individual implementations will vary based on usage, however these quantitative metrics provide a gauge as to whether a particular DUT is appropriate for a given environment. 4.1 Raw Packet Processing Performance (UDP Traffic) This test uses UDP packets of varying sizes generated by traffic generation appliances. A constant stream of the appropriate packet size with variable source and destination IP addresses transmitting from a fixed source port to a fixed destination port is transmitted bi- directionally through each port pair of the DUT. Each packet contains dummy data, and is targeted at a valid port on a valid IP address on the target subnet. The percentage load and frames per second (fps) figures across each in- line port pair are verified by network monitoring tools before each test begins. Multiple tests are run and averages are taken where necessary. This traffic does not attempt to simulate any form of real- world network condition. No TCP sessions are created during this test, and there is very little for the detection engine to do (although each vendor will be required to write a signature to detect the test packets to ensure that they are being passed through the detection engine and not fast- tracked from the inbound to outbound port). The goal of this test is to determine the raw packet processing capability of each in- line port pair of the DUT, and its effectiveness at forwarding packets quickly in order to provide the highest level of network performance and lowest latency. 4.1.1 64 Byte Packets Maximum 1,488,000 frames per second per Gigabit of traffic. This test determines the ability of a device to process packets from the wire under the most challenging packet processing conditions. 4.1.2 128 Byte Packets Maximum 844,000 frames per second per Gigabit of traffic 4.1.3 256 Byte Packets Maximum 452,000 frames per second per Gigabit of traffic. 4.1.4 512 Byte Packets Maximum 234,000 frames per second per Gigabit of traffic. This test provides a reasonable indication of the ability of a device to process packets from the wire on an average network. 4.1.5 1024 Byte Packets Maximum 119,000 frames per second per Gigabit of traffic. 4.1.6 1514 Byte Packets Maximum 81,000 frames per second per Gigabit of traffic. This test has been included to demonstrate how easy it is to achieve good results using large packets. Readers should use caution when taking into consideration those test results that only quote performance figures using similar packet sizes. 2013 NSS Labs, Inc. All rights reserved. 21

4.2 Latency The goal of the latency and user response time tests is to determine the effect the firewall has on the traffic passing through it under various load conditions. Test traffic is passed across the infrastructure switches and through all in- line port pairs of the DUT simultaneously (the latency of the basic infrastructure is known and is constant throughout the tests). The packet loss and average latency (µs) are recorded for each packet size (64, 128, 256, 512, 1024 and 1514 bytes) at a load level of 90% of the maximum throughput with zero packet loss as previously determined in Test 4.1 (Raw Packet Processing Performance (UDP Traffic). 4.2.1 64 Byte Frames Maximum 1,488,000 frames per second per Gigabit of traffic 4.2.2 128 Byte Frames Maximum 844,000 frames per second per Gigabit of traffic 4.2.3 256 Byte Packets Maximum 452,000 frames per second per Gigabit of traffic. 4.2.4 512 Byte Packets Maximum 234,000 frames per second per Gigabit of traffic. 4.2.5 1024 Byte Packets Maximum 119,000 frames per second per Gigabit of traffic. 4.2.6 1514 Byte Packets Maximum 81,000 frames per second per Gigabit of traffic. 4.3 Maximum Capacity The use of traffic generation appliances allows NSS engineers to create true real- world traffic at multi- Gigabit speeds as a background load for the tests. The goal of these tests is to stress the inspection engine and determine how it handles high volumes of TCP connections per second, application layer transactions per second, and concurrent open connections. All packets contain valid payload and address data, and these tests provide an excellent representation of a live network at various connection/transaction rates. Note that in all tests, the following critical breaking points where the final measurements are taken are used: Excessive concurrent TCP connections unacceptable increase in open connections on the server- side Excessive response time for HTTP transactions excessive delays and increased response time to client Unsuccessful HTTP transactions normally, there should be zero unsuccessful transactions. Once these appear, it is an indication that excessive latency is causing connections to time out 2013 NSS Labs, Inc. All rights reserved. 22

4.3.1 Theoretical Maximum Concurrent TCP Connections This test is designed to determine the maximum concurrent TCP connections of the DUT with no data passing across the connections. This type of traffic would not typically be found on a normal network, but it provides the means to determine the maximum possible concurrent connections figure. An increasing number of Layer 4 TCP sessions are opened through the device. Each session is opened normally and then held open for the duration of the test as additional sessions are added up to the maximum possible. Load is increased until no more connections can be established, and this number is recorded. 4.3.2 Theoretical Maximum Concurrent TCP Connections With Data This test is identical to 4.3.1, except that once a connection has been established, 21KB of data is transmitted (in 21KB segments). This ensures that the DUT is capable of passing data across the connections once they have been established. 4.3.3 Maximum TCP Connections Per Second This test is designed to determine the maximum TCP connection rate of the DUT with one byte of data passing across the connections. This type of traffic would not typically be found on a normal network, but it provides the means to determine the maximum possible TCP connection rate. An increasing number of new sessions are established through the DUT, ramped slowly to determine the exact point of failure. Each session is opened normally, one byte of data passed to the host, and then the session is closed immediately. Load is increased until one or more of the breaking points defined earlier is reached. 4.3.4 Maximum HTTP Connections Per Second This test is designed to determine the maximum TCP connection rate of the DUT with a 1 byte HTTP response size. The response size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. A 1 byte response size is designed to provide a theoretical maximum HTTP connections per second rate. Client and server are using HTTP 1.0 without keep- alive, and the client will open a TCP connection, send one HTTP request, and close the connection. This ensures that all TCP connections are closed immediately the request is satisfied, thus any concurrent TCP connections will be caused purely as a result of latency the DUT introduces on the network. Load is increased until one or more of the breaking points defined earlier is reached. 4.3.5 Maximum HTTP Transactions Per Second This test is designed to determine the maximum HTTP transaction rate of the DUT with a 1 byte HTTP response size. The object size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. A 1 byte response size is designed to provide a theoretical maximum connections per second rate. Client and server are using HTTP 1.1 with persistence, and the client will open a TCP connection, send ten HTTP requests, and close the connection. This ensures that TCP connections remain open until all ten HTTP transactions are complete, thus eliminating the maximum connection per second rate as a bottleneck (one TCP connection = 10 HTTP transactions). Load is increased until one or more of the breaking points defined earlier is reached. 2013 NSS Labs, Inc. All rights reserved. 23

4.4 HTTP Capacity With No Transaction Delays The goal of these tests is to stress the HTTP detection engine and determine how the DUT copes with network loads of varying average packet size and varying connections per second. By creating genuine session- based traffic with varying session lengths, the DUT is forced to track valid TCP sessions, thus ensuring a higher workload than for simple packet- based background traffic. This provides a test environment that is as close to real world as it is possible to achieve in a lab environment, while ensuring absolute accuracy and repeatability. Each transaction consists of a single HTTP GET request and there are no transaction delays (i.e., the web server responds immediately to all requests). All packets contain valid payload (a mix of binary and ASCII objects) and address data, and this test provides an excellent representation of a live network (albeit one biased towards HTTP traffic) at various network loads. Connections per Second 44Kbyte Response 21Kbyte Response 10Kbyte Response 4.5Kbyte Response 1.7Kbyte Response CPS 2,500 5,000 10,000 20,000 40,000 Mbps 1,000 1,000 1,000 1,000 1,000 Mbps 4.4.1 44KB HTTP Response Size 2,500 Connections Per Second Max 2,500 new connections per second per Gigabit of traffic with a 44KB HTTP response size average packet size 900 bytes maximum 140,000 packets per second per Gigabit of traffic. With relatively low connection rates and large packet sizes, all hosts should be capable of performing well throughout this test. 4.4.2 21KB HTTP Response Size 5,000 Connections Per Second Max 5,000 new connections per second per Gigabit of traffic with a 21KB HTTP response size average packet size 670 bytes maximum 185,000 packets per second per Gigabit of traffic. With average connection rates and average packet sizes, this is a good approximation of a real- world production network, and all hosts should be capable of performing well throughout this test. 4.4.3 10KB HTTP Response Size 10,000 Connections Per Second Max 10,000 new connections per second per Gigabit of traffic with a 10KB HTTP response size average packet size 550 bytes maximum 225,000 packets per second per Gigabit of traffic. With smaller packet sizes coupled with high connection rates, this represents a very heavily used production network. 4.4.4 4.5KB HTTP Response Size 20,000 Connections Per Second Max 20,000 new connections per second per Gigabit of traffic with a 4.5KB HTTP response size average packet size 420 bytes maximum 300,000 packets per second per Gigabit of traffic. With small packet sizes and extremely high connection rates, this is an extreme test for any host. 4.4.5 1.7KB HTTP Response Size 40,000 Connections Per Second Max 40,000 new connections per second per Gigabit of traffic with a 1.7KB HTTP response size average packet size 270 bytes maximum 445,000 packets per second per Gigabit of traffic. With small packet sizes and extremely high connection rates, this is an extreme test for any host. 2013 NSS Labs, Inc. All rights reserved. 24

4.5 Application Average Response Time: HTTP Test traffic is passed across the infrastructure switches and through all in- line port pairs of the DUT simultaneously (the latency of the basic infrastructure is known and is constant throughout the tests). The results recorded at each response size (44KB, 21KB, 10KB, 4.5KB, and 1.7KB HTTP responses) load level of 90% of the maximum throughput with zero packet loss as previously determined in Section 4.4 (HTTP Capacity With No Transaction Delays). 4.6 HTTP Capacity with Transaction Delays Typical user behavior introduces delays in between requests and reponses, for example, as users read web pages and decide which links to click next. This next set of tests is identical to the previous set except that these include a 5 second delay in the server response for each transaction. This has the effect of maintaining a high number of open connections throughout the test, thus forcing the sensor to utilize additional resources to track those connections. 4.6.1 21 KByte HTTP Response With Delay Max 5,000 new connections per second per Gigabit of traffic with a 21KB HTTP response size average packet size 670 bytes - - maximum 185,000 packets per second per Gigabit of traffic. 5 second transaction delay resulting in an additional 50,000 open connections per Gigabit over the test described in Section 4.4.2. With average connection rates and average packet sizes, this is a good approximation of a real- world production network, and all sensors should be capable of performing well throughout this test. 4.6.2 10 KByte HTTP Response With Delay Max 10,000 new connections per second per Gigabit of traffic with a 10KB HTTP response size average packet size 550 bytes maximum 225,000 packets per second per Gigabit of traffic. Repeated with background traffic loads of 25%, 50%, 75%, and 100% of maximum throughput of NGFW. 5 second transaction delay resulting in an additional 100,000 open connections over the test described in Section 4.4.3. With large average packet sizes coupled with very high connection rates, this represents a very heavily used production network, and is a strenuous test for any sensor. 4.7 Real- World Traffic Where previous tests provide a pure HTTP environment with varying connection rates and average packet sizes, the goal of this test is to simulate a real- world environment by introducing additional protocols and real content while still maintaining a precisely repeatable and consistent background traffic load. The result is a background traffic load that is closer to what may be found on a heavily- utilized normal production network. 4.7.1 Real- World Protocol Mix (Enterprise Perimeter) Traffic is generated across the DUT comprising a protocol mix typically seen in an enterprise perimeter. 4.7.2 Real- World Protocol Mix (Financial) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large financial institution. 2013 NSS Labs, Inc. All rights reserved. 25

4.7.3 Real- World Protocol Mix (Education) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large educational environment. 4.7.4 Real- World Protocol Mix (Data Center) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large data center. 4.7.5 Real- World Protocol Mix (US Mobile Carrier) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large US mobile carrier. 4.7.6 Real- World Protocol Mix (European Mobile Carrier) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a European mobile carrier. 2013 NSS Labs, Inc. All rights reserved. 26

5 Stability & Reliability Long- term stability is particularly important for an in- line device, where failure can produce network outages. These tests verify the stability of the DUT along with its ability to maintain security effectiveness while under normal load and while passing malicious traffic. Products that are not able to sustain legitimate traffic (or crash) while under hostile attack will not pass. The DUT is required to remain operational and stable throughout these tests, and to block 100% of previously blocked traffic, raising an alert for each. If any non- allowed traffic passes successfully, caused by either the volume of traffic or the DUT failing open for any reason, this will result in a FAIL. 5.1 Blocking Under Extended Attack The DUT is exposed to a constant stream of security policy violations over an extended period of time. The device is configured to block and alert, and thus this test provides an indication of the effectiveness of both the blocking and alert handling mechanisms. A continuous stream of security policy violations mixed with legitimate traffic is transmitted through the DUT at a maximum of 100Mbps (max 50,000 packets per second, average packet sizes in the range of 120-350 bytes) for 8 hours with no additional background traffic. This is not intended as a stress test in terms of traffic load (covered in the previous section), it is merely a reliability test in terms of consistency of blocking performance. The DUT is expected to remain operational and stable throughout this test, and to block 100% of recognisable violations, raising an alert for each. If any recognisable policy violations are passed, caused by either the volume of traffic or the DUT failing open for any reason, this will result in a FAIL. 5.2 Passing Legitimate Traffic Under Extended Attack This test is identical to Test 5.1, where the external interface of the DUT is exposed to a constant stream of exploits over an extended period of time. The DUT is expected to remain operational and stable throughout this test, and to pass most/all of the legitimate traffic. If an excessive amount of legitimate traffic is blocked throughout this test, caused by either the volume of traffic or the DUT failing for any reason, this will result in a FAIL. 5.3 Behavior Of The State Engine Under Load This test determines whether the DUT is capable of preserving state across a large number of open connections over an extended time period. At various points throughout the test (including after the maximum has been reached), it is confirmed that the DUT is still capable of inspecting and blocking traffic which is in violation of the currently- applied security policy, whilst confirming that legitimate traffic is not blocked (perhaps as a result of exhaustion of the resources allocated to state tables). The DUT needs to be able to apply policy decisions effectively based on inspected traffic at all load levels. 2013 NSS Labs, Inc. All rights reserved. 27

5.3.1 Attack Detection/Blocking Normal Load This test determines if the DUT is able to detect and block policy violations as the number of open sessions reaches 75% of the maximum determined in Test 4.3.1. 5.3.2 State Preservation Normal Load This test determines if the sensor maintains the state of pre- existing sessions as the number of open sessions reaches 75% of the maximum determined in Test 4.3.1. A legitimate HTTP session is opened and the first packet of a two- packet exploit is transmitted. As the number of open connections approaches the maximum, the initial HTTP session is then completed with the second half of the exploit and the session is closed. If the sensor is still maintaining state on the original session, the exploit will be recorded. If the state tables have been exhausted, the exploit string will be seen as a non- stateful attack, and will thus be ignored. Both halves of the exploit are required to trigger an alert. A product will fail the test if it fails to generate an alert after the second packet is transmitted or if it raises an alert on either half of the exploit on its own. 5.3.3 Pass Legitimate Traffic Normal Load This test ensures that the DUT continues to pass legitimate traffic as the number of open sessions reaches 75% of the maximum determined in Test 4.3.1. 5.3.4 State Preservation Maximum Exceeded This test determines if the DUT maintains the state of pre- existing sessions as the number of open sessions exceed the maximum determined in Test 4.3.1. Method of execution is identical to Test 5.3.2. 5.3.5 Drop Legitimate Traffic Maximum Exceeded This test ensures that the DUT continues to drop all traffic as the number of open sessions exceed the maximum determined in Test 4.3.1. Note: If a DUT allows traffic to leak due to the way it expires old connections, the result will be an automatic fail for the entire test. 5.4 Protocol Fuzzing & Mutation This test stresses the protocol stacks of the DUT by exposing it to traffic from various protocol randomizer and mutation tools. Several of the tools in this category are based on the ISIC test suite and other well known test tools/suites. Traffic load is a maximum of 350Mbps and 60,000 packets per second (average packet size is 690 bytes). Results are presented as a simple PASS/FAIL the DUT is expected to remain operational and capable of detecting and blocking exploits throughout the test. 5.5 Power Fail Power to the DUT is cut whilst passing a mixture of legitimate and disallowed traffic. Firewalls should always be configured to fail closed no traffic should be passed once power has been cut. 2013 NSS Labs, Inc. All rights reserved. 28

5.6 Redundancy Does the DUT include multiple redundant critical components (fans, power supplies, hard drive, etc.)? (YES/NO/OPTION). 5.7 Persistence Of Data The DUT should retain all configuration data, policy data, and locally logged data once restored to operation following power failure. 5.8 High Availability (HA) Option High availability (HA) is important to many enterprise customers, and this test is designed to evaluate the effectiveness of available HA options. If no HA offering is available, all results in this section will be marked as N/A. 5.8.1 Failover Legitimate Traffic Two identical devices will be configured in an active- passive configuration and legitimate traffic will be passed through the DUT at 50% of the maximum rated load as determined in Test 4.4.2 (21KB HTTP response size.) Switch connectivity to the primary device will be terminated and the DUT will be expected to failover seamlessly with zero loss of legitimate traffic (some retransmissions are acceptable.) 5.8.2 Failover Malicious Traffic Two identical devices will be configured in an active- passive configuration and a mix of legitimate and malicious traffic will be passed through the DUT at 50% of the maximum rated load as determined in Test 4.4.2 (21KB HTTP response size.) Switch connectivity to the primary device will be terminated and the DUT will be expected to failover seamlessly while continuing to deny all malicious traffic. 5.8.3 Time To Failover Time to failover to the standby device will be recorded. 5.8.4 Stateful Operation Is full state maintained across all connections throughout the period of failover? 5.8.5 Active- Active Configuration Is active- active configuration available (YES/NO)? 2013 NSS Labs, Inc. All rights reserved. 29

6 Management & Configuration Security devices are complicated to deploy; essential systems such as centralized management console options, log aggregation, and event correlation/management systems further complicate the purchasing decision. Understanding key comparison points will allow customers to model the overall impact on network service- level agreements (SLAs), estimate operational resource requirements to maintain and manage the systems, and better evaluate required skill/competencies of staff. As part of this test, NSS will perform in- depth technical evaluations of all the main features and capabilities of the centralized enterprise management systems offered by each vendor, covering the following key areas: General Management and Configuration how easy is it to install and configure devices, and deploy multiple devices throughout a large enterprise network? Policy Handling how easy is it to create, edit, and deploy complicated security policies across an enterprise? Alert Handling how accurate and timely is the alerting, and how easy is it to drill down to locate critical information needed to remediate a security problem? Reporting how effective and customizable is the reporting capability? For additional information concerning enterprise management testing, refer to the separate management questionnaire document. 2013 NSS Labs, Inc. All rights reserved. 30

7 Total Cost of Ownership & Value Organizations should be concerned with the ongoing, amortized cost of operating security products. This section evaluates the costs associated with the purchase, installation, and ongoing management of the DUT, including: Product Purchase the cost of acquisition. Product Maintenance the fees paid to the vendor (including software and hardware support, maintenance and updates). Installation the time required to take the device out of the box, configure it, put it into the network, apply updates and patches, initial tuning, and set up desired logging and reporting. Upkeep the time required to apply periodic updates and patches from vendors, including hardware, software, and firmware updates. 2013 NSS Labs, Inc. All rights reserved. 31

Appendix A: Test Environment The goal of this procedure is to provide a thorough test of all the main components of a routed firewall device in a controlled and repeatable manner and in the most real- world environment that can be simulated in a test lab. The Test Environment The NSS test network is a multi- Gigabit infrastructure that can accommodate both Gigabit copper and 10 Gigabit fiber interfaces. The firewall is configured for the use- case according to the test methodology. Traffic generation equipment, such as the hosts generating exploits, is connected to the external network, while the receiving equipment, such as the vulnerable hosts for the exploits, is connected to the internal network. The firewall is connected between two gateway switches, one at the edge of the external network and one at the edge of the internal network. All normal network traffic, background load traffic, and exploit traffic is transmitted through the firewall, from external to internal (responses will flow in the opposite direction). The same traffic is mirrored to multiple SPAN ports of the external gateway switch, to which network monitoring devices are connected. The network monitoring devices ensure that the total amount of traffic per port pair reflects the amount being sent and received by the traffic generating device and DUT. The management interface is used to connect the appliance to the management console on a private subnet. This ensures that the firewall and console can communicate even when the target subnet is subjected to heavy loads, in addition to preventing attacks on the console itself. 2013 NSS Labs, Inc. All rights reserved. 32

Appendix B: Change Log Version 5.4 30 September 2013 Capitalized SYN in subsection title Removed SSL section Clarified section 3.4.4 Removed references to specific test tools Version 5.3 04 December 2012 No Change Log available. Change Log Appendix added with version 5.4. 2013 NSS Labs, Inc. All rights reserved. 33

Contact Information NSS Labs, Inc. 206 Wild Basin Rd, Building A, Suite 200 Austin, TX 78746 USA +1 (512) 961-5300 info@nsslabs.com www.nsslabs.com v.130930a This and other related documents available at: www.nsslabs.com. To receive a licensed copy or report misuse, please contact NSS Labs at +1 (512) 961-5300 or sales@nsslabs.com. 2013 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors. Please note that access to or use of this document is conditional on the following: 1. NSS Labs reserves the right to modify any part of the methodology before, or during, a test, or to amend the configuration of a device under test (DUT) where specific characteristics of the DUT or its configuration interfere with the normal operation of any of the tests, or where the results obtained from those tests would, in the good faith opinion of NSS Labs engineers, misrepresent the true capabilities of the DUT. Every effort will be made to ensure the optimal combination of security effectiveness and performance, as would be the aim of a typical customer deploying the DUT in a live network environment. 2. The information in this document is believed by NSS Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this document are at the reader s sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this document. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON- INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This document does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader s expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This document does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this document are the trademarks, service marks, and trade names of their respective owners. 2013 NSS Labs, Inc. All rights reserved. 34