TEST METHODOLOGY. Next Generation Firewall (NGFW) v6.0

Transcription

1 TEST METHODOLOGY Next Generation Firewall (NGFW) v6.0

2 Table of Contents 1 Introduction The Need for Next Generation Firewalls (NGFWs) About This Test Methodology and Report Inclusion Criteria Product Guidance Recommended Neutral Caution Security Effectiveness Firewall Policy Enforcement Baseline Policy Simple Policies Complex Policies Static NAT (Network Address Translation) Dynamic/Hide NAT (Network Address Translation) SYN Flood Protection IP Address Spoofing TCP Split Handshake Spoof Application Control Block Block Specific Action (Depends on the application) User/Group ID Aware Policies Intrusion Prevention Exploit Library False Positive Testing Coverage by Attack Vector Coverage by Impact Type Coverage by Date Coverage by Vendor Coverage by Result Evasion IP Packet Fragmentation Stream Segmentation RPC Fragmentation URL Obfuscation HTML Obfuscation Payload Encoding FTP Evasion

3 3.5.8 Layered Evasions Performance Raw Packet Processing Performance (UDP Traffic) Byte Packets Byte Packets Byte Packets Byte Packets Byte Packets Byte Packets Latency Byte Frames Byte Frames Byte Packets Byte Packets Byte Packets Byte Packets Maximum Capacity Theoretical Maximum Concurrent TCP Connections Theoretical Maximum Concurrent TCP Connections with Data Maximum TCP Connections per Second Maximum HTTP Connections per Second Maximum HTTP Transactions per Second HTTP Capacity with No Transaction Delays KB HTTP Response Size 2,500 Connections per Second KB HTTP Response Size 5,000 Connections per Second KB HTTP Response Size 10,000 Connections per Second KB HTTP Response Size 20,000 Connections per Second KB HTTP Response Size 40,000 Connections per Second Application Average Response Time: HTTP HTTP Capacity with Transaction Delays KB HTTP Response with Delay KB HTTP Response with Delay Real-World Traffic Real-World Protocol Mix (Enterprise Perimeter) Real-World Protocol Mix (Financial) Real-World Protocol Mix (Education) Real-World Protocol Mix (Data Center) Real-World Protocol Mix (US Mobile Carrier) Real-World Protocol Mix (European Mobile Carrier)

4 5 Stability & Reliability Blocking Under Extended Attack Passing Legitimate Traffic under Extended Attack Behavior of the State Engine Under Load Attack Detection/Blocking Normal Load State Preservation Normal Load Pass Legitimate Traffic Normal Load State Preservation Maximum Exceeded Drop Legitimate Traffic Maximum Exceeded Protocol Fuzzing & Mutation Power Fail Persistence of Data Total Cost of Ownership & Value Contact Information

5 1 Introduction 1.1 The Need for Next Generation Firewalls (NGFWs) Firewall technology is one of the largest and most mature security markets. Firewalls have undergone several stages of development, from early packet filtering and circuit relay firewalls to application layer (proxy-based) and dynamic packet filtering firewalls. Throughout their history, however, the goal has been to enforce an access control policy between two networks, and they should therefore be viewed as an implementation of policy. A firewall is a mechanism used to protect a trusted network from an untrusted network, while allowing authorized communications to pass from one side to the other, thus facilitating secure business use of the Internet. With the emergence of new web applications and security threats, however, firewalls are evolving further. With the emergence of Web 2.0 trends pushing critical business applications through firewall ports that were previously reserved for a single function, such as HTTP, the legacy firewall technology is effectively blinded. Unable to differentiate between actual HTTP traffic and non-http services tunneling over port 80, such as VoIP or instant messaging, the addition of application-level monitoring rather than simply port and destination is essential. As a consequence of this, firewalls are evolving further. This means it is no longer possible to rely on port and protocol combinations alone to define network applications. The next generation firewall (NGFW) therefore requires the capability to perform deep packet inspection of all packets, on all ports, and over all protocols in order to determine which applications are running over which ports and thus secure them effectively. 1.2 About This Test Methodology and Report NSS Labs test reports are designed to address the challenges faced by IT professionals in selecting and managing security products. The scope of this particular report includes: Security effectiveness Performance Stability and reliability Total cost of ownership (TCO) As NGFWs are deployed at critical choke points in the network, the stability and reliability of an NGFW is imperative. Therefore, regardless of any new deep inspection capabilities, the main requirement of any NGFW is that it must be as stable, as reliable, as fast, and as flexible as the firewall that it is replacing. 5

6 Based on the needs identified in NSS research, the following capabilities are considered essential in any NGFW device: Traditional first generation firewall features, including: o o o o o Basic packet filtering Stateful multi-layer inspection NAT VPN Highly stable Next generation firewall features, including: o Application awareness/control o User/group control o Integrated intrusion prevention system (IPS) o Ability to operate at Layer 3 ( traditional ) o External intelligence to enhance blocking decisions (i.e., reputation services ) Tuning: Security engineers typically will tune an IPS to ensure its protection coverage matches the needs of the environment where it is being placed. Though this strategy works well for data centers and demilitarized zones (DMZs), protecting desktops is a different matter. In surveying enterprises, NSS researchers discovered that many enterprises do not strictly control the desktop, and that in larger enterprises there can be a wide range of applications running on the typical endpoint. As such, enterprises are expecting IPS and NGFW vendors to provide maximum security for desktop client applications with their out-of-the-box recommended or default policies. In addition, research indicates that enterprises are not ready to replace their dedicated IPS solutions with NGFW in the data center. This leads us to conclude that the intrusion prevention functionality within an NGFW typically will be used to protect desktop clients, with optimal protection predefined via a vendor-supplied recommended/default policy, and this is how the devices will be tested. 1.3 Inclusion Criteria In order to encourage the greatest participation, and to allay any potential concerns of bias, NSS invites all security vendors claiming NGFW capabilities to submit their products at no cost. Vendors with major market share, as well as challengers with new technology, will be included. The NGFW should be supplied as a single appliance, where possible (cluster controller solutions are also acceptable), with the appropriate number of physical interfaces capable of achieving the required level of connectivity and performance (minimum of one in-line port pair per Gigabit of throughput, or one in-line 10Gbps port pair per 10Gbps of throughput). Firewall products should be implemented as Layer 3 (routing) devices. Multiple separate 1Gbps or 10Gbps connections will be made from the external to internal switches via the device under test (DUT), subject to a minimum of one in-line port pair per Gigabit of throughput. Thus, an 8Gbps device with only four port pairs will be limited to 4Gbps. The maximum number of port pairs will be connected to determine the overall throughput of the DUT. 6

7 Once installed in the test lab, the DUT will be configured for the use-case appropriate to the target deployment (corporate network perimeter). The DUT should also be configured to block all traffic when resources are exhausted or when traffic cannot be analyzed for any reason. 7

8 2 Product Guidance NSS issues summary product guidance based on evaluation criteria that is important to information security professionals. The evaluation criteria are weighted as follows: Security effectiveness The purpose of an NGFW is to separate internal trusted networks from external untrusted networks through policy and routing, and to identify and block attacks against assets, while allowing select controlled traffic to flow between trusted and untrusted networks. Resistance to evasion Failure in any evasion class permits attackers to circumvent protection. Stability Long-term stability is particularly important for an in-line device, where failure can produce network outages. Performance Correctly sizing an NGFW is essential. Value Customers should seek low TCO and high effectiveness and performance rankings. Products are listed in rank order according to their guidance rating. 2.1 Recommended A Recommended rating from NSS indicates that a product has performed well and deserves strong consideration. Only the top technical products earn a Recommended rating from NSS, regardless of market share, company size, or brand recognition. 2.2 Neutral A Neutral rating from NSS indicates that a product has performed reasonably well and should continue to be used if it is the incumbent within an organization. Products that earn a Neutral rating from NSS deserve consideration during the purchasing process. 2.3 Caution A Caution rating from NSS indicates that a product has performed poorly. Organizations using one of these products should review their security posture and other threat mitigation factors, including possible alternative configurations and replacement. Products that earn a Caution rating from NSS should not be short-listed or renewed. 8

9 3 Security Effectiveness This section verifies that the DUT is capable of enforcing a specified security policy effectively. NSS firewall analysis is conducted by incrementally building upon a baseline configuration (simple routing with no policy restrictions) to a complex, real-world, multiple-zone configuration supporting many addressing modes, policies, applications, and inspection engines. At each level of complexity, test traffic is passed across the firewall to ensure that only specified traffic is allowed, and the rest is denied, and that appropriate log entries are recorded. The NGFW must support stateful firewalling either by managing state tables to prevent traffic leakage, or as a stateful proxy. The ability to manage firewall policy across multiple interfaces/zones is a required function. At a minimum, the firewall must provide a trusted internal interface, an untrusted external/internet interface, and (optionally) one or more DMZ interfaces. In addition, a dedicated management interface (virtual or otherwise) is preferred. 3.1 Firewall Policy Enforcement Policies are rules that are configured on a firewall to permit or deny access from one network resource to another, based on identifying criteria such as source, destination, and service. A term typically used to define the demarcation point of a network where policy is applied is a demilitarized zone (DMZ). Policies are typically written to permit or deny network traffic from one or more of the following zones: Untrusted This is typically an external network and is considered to be unknown and non-secure. An example of an untrusted network would be the Internet. DMZ This is a network that is being isolated by the firewall restricting network traffic to and from hosts contained within the isolated network. Trusted This is typically an internal network; a network that is considered secure and protected. The NSS firewall tests verify performance and the ability to enforce policy between the following: Trusted to Untrusted Untrusted to DMZ Trusted to DMZ Note: Firewalls must provide at a minimum one DMZ interface in order to provide a DMZ or transition point between untrusted and trusted networks 9

10 3.1.1 Baseline Policy Routed configuration with an allow all policy Simple Policies Simple outbound and inbound policies allowing basic browsing and access for internal clients and no external access Complex Policies Complex outbound and inbound policies consisting of many rules, objects, and services Static NAT (Network Address Translation) Inbound network address translation (NAT) using fixed IP address translation with one-to-one mapping Dynamic/Hide NAT (Network Address Translation) Outbound network address translation (NAT) (from internal to external), where all outbound traffic hides behind the IP address of the external interface of the firewall utilizing a pool of high ports to manage multiple connections SYN Flood Protection The basis of a SYN flood attack is to fail to complete the three-way handshake necessary to establish a legitimate session. The objective of SYN flooding is to disable one side of the TCP connection, which will result in one or more of the following: The server is unable to accept new connections. The server crashes or becomes inoperative. Authorization between servers is impaired. The DUT is expected to protect against SYN floods IP Address Spoofing This test attempts to confuse the firewall into allowing traffic to pass from one network segment to another. By forging the IP header to contain a different source address from where the packet was actually transmitted, an attacker can make it appear that the packet was sent from a different (trusted) machine. The endpoint that receives successfully spoofed packets will respond to the forged source address (the attacker). The DUT is expected to protect against IP address spoofing TCP Split Handshake Spoof This test attempts to confuse the firewall into allowing traffic to pass from one network segment to another. The TCP split handshake blends features of both the three-way handshake and the simultaneous-open connection. 10

11 The result is a TCP spoof attack that allows an attacker to bypass the firewall by instructing the target to initiate the session back to the attacker. Popular TCP/IP networking stacks respect this handshaking method, including Microsoft, Apple, and Linux stacks, with no modification. 1 The DUT is expected to protect against TCP split handshake spoofing. 3.2 Application Control Complex outbound and inbound policies consisting of many rules, objects, and applications, verifying that the DUT is capable of correctly determining the correct application from deep packet inspection (regardless of port/protocol used), and taking the appropriate action. Popular social networking web sites (web applications) Instant messaging (IM) Skype and other VoIP Torrents Other applications TBD For each application, NSS will test the DUT s ability to perform the following functions: Block The DUT should be able to correctly identify the application and block it Block Specific Action (Depends on the application) For example, with instant messaging, the DUT should allow text communications while blocking file transfers. 3.3 User/Group ID Aware Policies Complex outbound and inbound policies consisting of many rules, objects and applications, verifying that the DUT is capable of correctly determining the correct user/group ID from deep packet inspection, and taking the appropriate action. Due to the unique needs and topology requirements of enterprises today, NSS recommends that each enterprise perform an Active Directory integration proof-of-concept deployment for any NGFW(s) under consideration. 3.4 Intrusion Prevention Policies consisting of threat protection signatures, verifying that the DUT is capable of correctly blocking malicious traffic based on a comparison of packet/session contents against signatures/filters/protocol decoders. The latest signature pack is acquired from the vendor s support site, and the DUT is deployed with the out-of-thebox recommended or default security policy. No tuning of the product is allowed by the vendor. NSS considers it unacceptable for a product of this nature to be sold without a recommended or default policy. No custom 1 The TCP Split Handshake: Practical Effects on Modern Network Equipment, Tod Alien Beardsley & Jin Qian, 11

12 signatures are permitted in the testing. All signatures used must be available to the general public at the time of testing. Although intrusion detection systems operate in detection-only mode, an NGFW is required to block and log exploit attempts and hostile traffic Exploit Library NSS security effectiveness testing leverages the deep expertise of our engineers who utilize multiple commercial, open-source, and proprietary tools, including NSS network live stack test environment 2 as appropriate. With thousands of exploits, this is the industry s most comprehensive test to date. Most notably, all of the live exploits and payloads in the NSS exploit test have been validated in our lab such that one or more of the following is true: A reverse shell is returned A bind shell is opened on the target allowing the attacker to execute arbitrary commands Arbitrary code execution A malicious payload is installed A system is rendered unresponsive Etc. This test goes far beyond replaying packet captures or pressing the button on a test tool. In short, NSS engineers trigger vulnerabilities for the purpose of validating that an exploit was able to pass through the DUT False Positive Testing The ability of the DUT to identify and allow legitimate traffic while maintaining protection against threats and exploits is just as important as its abilility to protect against malicious content. This test will include a varied sample of legitimate application traffic, which should be identified and allowed, or blocked, based on policy rules. 2 For more information on the NSS Live Testing harness and methodology, please refer to the latest Security Stack (IPS): Test Methodology located at 12

13 3.4.3 Coverage by Attack Vector Threats and exploits can be initiated by either the target or the attacker targeting either local or remote vulnerabilities. As a result, NSS categorizes threats and exploits into the following matrix: Network Local Attacker RPC Exploit Root Kit Target Browser Exploit Trojan *Example exploits included above for reference purposes Attacker Initiated Also referred to as server-side exploits, the threat/exploit is executed remotely by the attacker against a vulnerable application and/or operating system Target Initiated The threat/exploit is initiated by the vulnerable target. The attacker has little or no control as to when the target user or application will execute the threat. Given that NGFW devices are typically deployed to protect end users, this class of exploit is the main focus of the NGFW security effectiveness test Network Threats/exploits that are initiated as a result of network communication Local Local execution that requires existing access to the target (not applicable to NGFW). Protective ratings are reported in raw percentages of mitigated attacks and their resulting impact: system, service, fault, reconnaissance. Although a system or service exploit may be partially mitigated by the DUT, the service could have crashed because of the residual communications resulting in a fault impact on the service or operating system Coverage by Impact Type The NSS threat and attack suite contains thousands of publicly available exploits (including multiple variants of each exploit) from which groups of exploits are carefully selected to test based on appropriate usage. Each exploit has been validated to impact the target vulnerable host(s). Based on the impact of the threat against the target, the following metrics are reported: System Exposure Attacks resulting in remote system compromise and the ability of the attacker to execute arbitrary system-level commands. Most exploits in this class that are weaponized will provide the attacker with a fully interactive remote shell on the target client or server. 13

14 Service Exposure Attacks resulting in an individual service compromise but not arbitrary system-level command execution. Typical attacks in this category include service specific attacks such as SQL injection that enable the attacker to execute arbitrary SQL commands within the database service. These attacks are somewhat isolated to the service and do not immediately result in full system-level access to the operating system and all services. However, using additional localized system attacks, it may be possible for the attacker to go from the service level to the system level System or Service Fault Attacks resulting in a system or service-level fault that crashes the targeted service or application and requires administrative action to restart the service or reboot the system. These attacks do not enable the attacker to execute arbitrary commands. However, the resulting impact to the business could be severe given that the attacker could crash the protected system or service Coverage by Date The typical enterprise will run a mix of both old and new applications, and NSS research shows that crimeware kits will frequently include exploits that date back several years. Therefore, NSS security effectiveness testing will include exploits current at the time of the test, as well as targeting vulnerabilities covering multiple years dating backwards from the time of the test. Results will be reported by year for up to 10 years prior to the year of the test. Results prior to that time period, where applicable, will be aggregated into the oldest bucket. 14

15 3.4.6 Coverage by Vendor NSS live exploit test contains many vendors, including but not limited to the following list. Protection capabilities are indicated as percentages Coverage by Result 3Com Adobe Alt-N Apache Apple Atrium Avast BEA BitDefender Borland CA Cisco Citrix ClamAV EMC Facebook GNU Google HP IBM IPSwitch ISC Kaspersky LanDesk lighttpd Linux Macromedia MacroVision Mailenable McAfee Mercury Microsoft MIT Mozilla Mplayer Multiple Vendors MySQL NOD32 Novell Nullsoft OpenLDAP OpenOffice OpenSSH OpenSSL Oracle Other Misc Panda RealNetworks Samba SAP Snort Sophos SpamAssassin Squid Sun Microsystems Symantec Trend Micro Trillian UltraVNC Veritas VideoLan VMWare WinAmp WinFTP Winzip Yahoo The following results of exploitation are represented in NSS live exploit test. Protection capabilities are indicated as percentages Arbitrary Code Execution A software bug that allows an attacker to execute any commands of the attacker's choice on a target machine or in a target process 15

16 Buffer Overflow The exploitation of a software bug due to improperly establishing memory bounds allows an attacker to overwrite adjacent memory and execute a command Code Injection The exploitation of a software bug that allows the processing of invalid data within a program. Code injection can be used by an attacker to introduce code into a computer program to change the course of execution Cross-Site Script The exploitation of a web application that enables attackers to insert malicious script into web pages, which can then be executed by other users Directory Traversal The exploitation of a lack of security in an application (as opposed to exploiting a bug in the code) that allows usersupplied input with characters representing traverse to parent directory to be passed through to the file APIs. The goal of this attack is to order an application to access a file or executable that is not intended to be accessible Privilege Escalation This exploit type allows an attacker to gain access to resources that would not normally have been available Target Type The following list of web target types is represented in NSS live exploit test. Protection capabilities are indicated as percentages. Web Server ActiveX Web Browser JavaScript Browser Plug-ins/Add-ons 3.5 Evasion Attackers can modify basic attacks to evade detection in a number of ways. If a DUT fails to detect a single form of evasion, any exploit can pass through the device, rendering it ineffective. NSS verifies that the DUT is capable of detecting and blocking basic exploits when subjected to varying common evasion techniques. Wherever possible, the DUT is expected to successfully decode the obfuscated traffic to provide an accurate alert relating to the original exploit, rather than alerting purely on anomalous traffic detected as a result of the evasion technique itself. A number of common exploits are executed across the DUT to ensure that they are detected in their unmodified state. These will be chosen from a suite of older/common basic exploits for which NSS is certain that all vendors will have signatures. None of the exploits that were used in section 3.4 will be used as evasion baselines. This ensures that vendors are not provided with any information on the content of any part of the main NSS exploit library in advance of the test. 16

17 3.5.1 IP Packet Fragmentation These tests determine the effectiveness of the fragment reassembly mechanism of the NGFW. Fragments from 8 32 bytes in size Ordered, out-of-order, or reverse order fragments Fragment overlap, favoring new and favoring old data Interleaved, duplicate, duplicate with or without incrementing DWORD, duplicate packets with random payload, or duplicate packets scheduled for later delivery Any combination of the above methods It is a requirement of the test that the DUT submitted should have all IP fragmentation reassembly options enabled by default in the shipping product Stream Segmentation These tests determine the effectiveness of the stream reassembly mechanism of the NGFW. Segments from bytes in size Ordered, reverse ordered, or out-of-order segments, with favor old or favor new Duplicate, duplicate interleaved, duplicate last packet, or overlapping segments Invalid or NULL TCP control flags Sequence resync requests, random initial sequence number, or out-of-window sequence numbers Faked retransmits, PAWS elimination, or segments containing random data Endianness interchanged Any combination of the above methods It is a requirement of the test that the DUT submitted should have all TCP stream reassembly options enabled by default in the shipping product RPC Fragmentation Both Sun/ONC RPC and MS-RPC allow the sending application to fragment requests, and all MS-RPC services have a built-in fragmentation reassembly mechanism. An attacker can transmit the BIND followed by a single request fragmented over a hundred actual requests with small fragments of the malicious payload. Alternatively, the attacker could transmit both the BIND and request fragments in one large TCP segment, thus foiling any signatures that use a simple size check. NSS uses test tools that combine large writes with many tiny MS-RPC fragments and provide multiple levels of fragmentation. These tests determine the effectiveness of the RPC reassembly mechanism of the NGFW: Sun/Open Network Computing (ONC) RPC byte fragmentation Fragments sent in one or more TCP segments, with or without last fragment RPC fragmentation may or may not be performed in a single segment Etc., Any combination of the above methods 17

18 3.5.4 URL Obfuscation Random URL encoding techniques are employed to transform simple URLs that are often used in pattern-matching signatures to apparently meaningless strings of escape sequences and expanded path characters, using a combination of the following techniques: Escape encoding (% encoding) Microsoft %u encoding Path character transformations and expansions ( /./, //, \ ) These techniques are combined in various ways for each URL tested, ranging from minimal transformation to extreme (every character transformed). All transformed URLs are verified to ensure they still function as expected after transformation. URL encoding Levels 1 8 (minimal to extreme) Premature URL ending Long URL Fake parameter TAB separation Case sensitivity Windows\delimiter Session splicing Any combination of the above methods HTML Obfuscation The ability to recognize malicious HTML documents is becoming increasingly important when protecting the enterprise. Malicious HTML documents exploit flaws in common web browsers, browser plug-ins, and add-ons to gain control of the client system and silently install malware such as Trojans, rootkits, and key loggers. Therefore, it is becoming increasingly important that security products charged with protecting end systems must correctly interpret HTML documents. Many security products use simple pattern matching systems with very little semantic or syntactic understanding of the data they are analyzing. This leaves them vulnerable to evasion through the use of redundant, but equivalent, alternative representations of malicious documents. This test suite uses a number of malicious HTML documents that are transferred from server to client through the DUT. Each malicious HTML document is served with a different form of obfuscation, as follows: UTF-16 and UTF-32 character set encoding (big-endian) UTF-16 and UTF-32 character set encoding (little-endian) UTF-7 character set encoding Chunked encoding (random chunk size, fixed 8-byte chunk size, or chaffing/arbitrary numbers inserted between chunks) Compression (Deflate) Compression (Gzip) Base-64 character set encoding with or without bit shifting or chaffing Any combination of the above methods 18

19 The UTF-16 character set specifies a 2-byte sequence for most characters and a 4-byte sequence for the others (a small percentage). Recoding an HTML document in UTF-16 significantly changes its appearance. A document that contains just the ASCII subset of characters will appear to have a null byte between every one of the original characters. There are also two different forms of the UTF-16 encoding, depending on whether the null high byte comes first (big-endian) or second (little-endian) this test uses big-endian byte ordering The UTF-32 character set specifies a 4-byte sequence. Like the UTF-16 character set encoding, there are two variations big-endian and little-endian and this test case uses big-endian byte ordering. The UTF-7 character set encodes most ASCII characters as themselves. However, in addition to recoding non- English characters as other encodings do, it also recodes many punctuation symbols, including many of the symbols that are important to the HTML specification. Therefore, recoding an HTML document in UTF-7 significantly changes its appearance Chunked encoding allows the server to break a document into smaller chunks and transmit them individually. The server needs only to specify the size of each chunk before it is transmitted and then indicate when the last chunk has been transmitted. Since chunked encoding intersperses arbitrary numbers (chunk sizes) with the elements of the original document, it can be used to greatly change the appearance of the original document as observed on the wire. In addition, the server can choose to break the document into chunks at arbitrary points. This makes it difficult for simple pattern matching systems to reliably identify the original HTML document from the raw data on the network. Per RFC 2616, the HTTP protocol allows the client to request and the server to use several compression methods. These compression methods not only improve performance in many circumstances, they completely change the characteristic size and appearance of HTML documents. Furthermore, small changes in the original document can greatly change the final appearance of the compressed document. This property of these algorithms could be used to obfuscate hostile content for the purpose of evading detection. The deflate compression method is a Lempel-Ziv coding (LZ77), specified in RFC The gzip compression method is specified in RFC For each of the above, it is verified that a standard web browser (such as Internet Explorer) is capable of rendering the results of the evasion. 19

20 3.5.6 Payload Encoding This test attempts to confuse the NGFW into allowing an otherwise blocked exploit to pass using various encoding options that are standard within the Metasploit Framework and/or other evasion tools. A partial list includes: x86/call4_dword_xor This encoder implements a Call+4 Dword XOR Encoder x86/countdown This encoder uses the length of the payload as a position-dependent encoder key to produce a small decoder stub. x86/fnstenv_mov This encoder uses a variable-length mov equivalent instruction with fnstenv for getip. x86/jmp_call_additive This encoder implements a Jump/Call XOR Additive Feedback Encoder x86/shikata_ga_nai This encoder implements a Polymorphic XOR Additive Feedback Encoder. The decoder stub is generated based on dynamic instruction substitution and dynamic block ordering. Registers are also selected dynamically FTP Evasion When attempting FTP exploits, it is possible to evade some products by inserting additional spaces and telnet control sequences in FTP commands. These tests insert a range of valid telnet control sequences that can be parsed and handled by IIS FTP server and wu-ftpd, and which also conform to Section 2.3 of RFC 959. Control opcodes are inserted at random, ranging from minimal insertion (only one pair of opcodes), to extreme (opcodes between every character in the FTP command): Inserting spaces in FTP command lines Inserting non-text Telnet opcodes Levels 1 8 (minimal to extreme) Any combination of the above methods Layered Evasions This test attempts to bypass the DUT by performing any legitimate combination of the evasion techniques specified in section 3.5. It will be verified that the target machine s standard network stack is capable of decoding the evasion correctly while maintaining the exploit viability. 20

21 4 Performance This section measures the performance of the DUT using various traffic conditions that provide metrics for realworld performance. Individual implementations will vary based on usage; however, these quantitative metrics provide a gauge as to whether a particular DUT is appropriate for a given environment. 4.1 Raw Packet Processing Performance (UDP Traffic) This test uses UDP packets of varying sizes generated by traffic generation appliances. A constant stream of the appropriate packet size with variable source and destination IP addresses transmitting from a fixed source port to a fixed destination port is transmitted bi-directionally through each port pair of the DUT. Each packet contains dummy data, and is targeted at a valid port on a valid IP address on the target subnet. The percentage load and frames per second (fps) figures across each in-line port pair are verified by network monitoring tools before each test begins. Multiple tests are run and averages are taken where necessary. This traffic does not attempt to simulate any form of real-world network condition. No TCP sessions are created during this test, and there is very little for the detection engine to do (although each vendor will be required to write a signature to detect the test packets to ensure that they are being passed through the detection engine and not fast-tracked from the inbound to outbound port). The goal of this test is to determine the raw packet processing capability of each in-line port pair of the DUT, and its effectiveness at forwarding packets quickly in order to provide the highest level of network performance and lowest latency Byte Packets Maximum 1,488,000 frames per second per Gigabit of traffic. This test determines the ability of a device to process packets from the wire under the most challenging packet processing conditions Byte Packets Maximum 844,000 frames per second per Gigabit of traffic Byte Packets Maximum 452,000 frames per second per Gigabit of traffic Byte Packets Maximum 234,000 frames per second per Gigabit of traffic. This test provides a reasonable indication of the ability of a device to process packets from the wire on an average network Byte Packets Maximum 119,000 frames per second per Gigabit of traffic. 21

22 Byte Packets Maximum 81,000 frames per second per Gigabit of traffic. This test has been included to demonstrate how easy it is to achieve good results using large packets. Readers should use caution when taking into consideration those test results that only quote performance figures using similar packet sizes. 4.2 Latency The goal of the latency and user response time tests is to determine the effect the firewall has on the traffic passing through it under various load conditions. Test traffic is passed across the infrastructure switches and through all in-line port pairs of the DUT simultaneously (the latency of the basic infrastructure is known and is constant throughout the tests). The packet loss and average latency (s) are recorded for each packet size (64, 128, 256, 512, 1024, and 1514 bytes) at a load level of 90% of the maximum throughput with zero packet loss as previously determined in Test 4.1 (Raw Packet Processing Performance (UDP Traffic) Byte Frames Maximum 1,488,000 frames per second per Gigabit of traffic Byte Frames Maximum 844,000 frames per second per Gigabit of traffic Byte Packets Maximum 452,000 frames per second per Gigabit of traffic Byte Packets Maximum 234,000 frames per second per Gigabit of traffic Byte Packets Maximum 119,000 frames per second per Gigabit of traffic Byte Packets Maximum 81,000 frames per second per Gigabit of traffic. 4.3 Maximum Capacity The use of traffic generation appliances allows NSS engineers to create true real-world traffic at multi-gigabit speeds as a background load for the tests. The goal of these tests is to stress the inspection engine and determine how it handles high volumes of TCP connections per second, application layer transactions per second, and concurrent open connections. All packets contain valid payload and address data, and these tests provide an excellent representation of a live network at various connection/transaction rates. Note that in all tests, the following critical breaking points where the final measurements are taken are used: 22

23 Excessive concurrent TCP connections Latency within the NGFW is causing unacceptable increase in open connections. Excessive concurrent HTTP connections Latency within the NGFW is causing excessive delays and increased response time. Unsuccessful HTTP transactions Normally, there should be zero unsuccessful transactions. Once these appear, it is an indication that excessive latency within the NGFW is causing connections to time out Theoretical Maximum Concurrent TCP Connections This test is designed to determine the maximum concurrent TCP connections of the DUT with no data passing across the connections. This type of traffic would not typically be found on a normal network, but it provides the means to determine the maximum possible concurrent connections figure. An increasing number of Layer 4 TCP sessions are opened through the device. Each session is opened normally and then held open for the duration of the test as additional sessions are added up to the maximum possible. Load is increased until no more connections can be established, and this number is recorded Theoretical Maximum Concurrent TCP Connections with Data This test is identical to 4.3.1, except that once a connection has been established, 21 KB of data is transmitted (in 21 KB segments). This ensures that the DUT is capable of passing data across the connections once they have been established Maximum TCP Connections per Second This test is designed to determine the maximum TCP connection rate of the DUT with one byte of data passing across the connections. This type of traffic would not typically be found on a normal network, but it provides the means to determine the maximum possible TCP connection rate. An increasing number of new sessions are established through the DUT, ramped slowly to determine the exact point of failure. Each session is opened normally, one byte of data is passed to the host, and then the session is closed immediately. Load is increased until one or more of the breaking points defined earlier is reached Maximum HTTP Connections per Second This test is designed to determine the maximum TCP connection rate of the DUT with a 1 byte HTTP response size. The response size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. A 1 byte response size is designed to provide a theoretical maximum HTTP connections per second rate. Client and server are using HTTP 1.0 without keep-alive, and the client will open a TCP connection, send one HTTP request, and close the connection. This ensures that all TCP connections are closed immediately upon the request being satisfied, thus any concurrent TCP connections will be caused purely as a result of latency the DUT introduces on the network. Load is increased until one or more of the breaking points defined earlier is reached Maximum HTTP Transactions per Second This test is designed to determine the maximum HTTP transaction rate of the DUT with a 1 byte HTTP response size. The object size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. A 1 byte response size is designed to provide a theoretical maximum connections per second rate. 23

24 Connections per Second Mbps NSS Labs Test Methodology Next Generation Firewall v6.0 Client and server are using HTTP 1.1 with persistence, and the client will open a TCP connection, send ten HTTP requests, and close the connection. This ensures that TCP connections remain open until all ten HTTP transactions are complete, thus eliminating the maximum connection per second rate as a bottleneck (one TCP connection = 10 HTTP transactions). Load is increased until one or more of the breaking points defined earlier is reached. 4.4 HTTP Capacity with No Transaction Delays The goal of these tests is to stress the HTTP detection engine and determine how the DUT copes with network loads of varying average packet size and varying connections per second. By creating genuine session-based traffic with varying session lengths, the DUT is forced to track valid TCP sessions, thus ensuring a higher workload than for simple packet-based background traffic. This provides a test environment that is as close to real world as it is possible to achieve in a lab environment, while ensuring absolute accuracy and repeatability. Each transaction consists of a single HTTP GET request,and there are no transaction delays (i.e., the web server responds immediately to all requests). All packets contain valid payload (a mix of binary and ASCII objects) and address data, and this test provides an excellent representation of a live network (albeit one biased towards HTTP traffic) at various network loads. 44Kbyte Response 21Kbyte Response 10Kbyte Response 4.5Kbyte Response 1.7Kbyte Response CPS 2,500 5,000 10,000 20,000 40,000 Mbps 1,000 1,000 1,000 1,000 1, KB HTTP Response Size 2,500 Connections per Second Maximum 2,500 new connections per second per Gigabit of traffic with a 44 KB HTTP response size maximum 140,000 packets per second per Gigabit of traffic. With relatively low connection rates and large packet sizes, all hosts should be capable of performing well throughout this test KB HTTP Response Size 5,000 Connections per Second Maximum 5,000 new connections per second per Gigabit of traffic with a 21KB HTTP response size maximum 185,000 packets per second per Gigabit of traffic. With average connection rates and average packet sizes, this is a good approximation of a real-world production network, and all hosts should be capable of performing well throughout this test KB HTTP Response Size 10,000 Connections per Second Maximum 10,000 new connections per second per Gigabit of traffic with a 10KB HTTP response size maximum 225,000 packets per second per Gigabit of traffic. With smaller packet sizes coupled with high connection rates, this represents a very heavily used production network. 24

25 KB HTTP Response Size 20,000 Connections per Second Maximum 20,000 new connections per second per Gigabit of traffic with a 4.5KB HTTP response size maximum 300,000 packets per second per Gigabit of traffic. With small packet sizes and extremely high connection rates, this is an extreme test for any host KB HTTP Response Size 40,000 Connections per Second Maximum 40,000 new connections per second per Gigabit of traffic with a 1.7KB HTTP response size maximum 445,000 packets per second per Gigabit of traffic. With small packet sizes and extremely high connection rates, this is an extreme test for any host. 4.5 Application Average Response Time: HTTP Test traffic is passed across the infrastructure switches and through all in-line port pairs of the DUT simultaneously (the latency of the basic infrastructure is known and is constant throughout the tests). The results recorded at each response size (44KB, 21KB, 10KB, 4.5KB, and 1.7KB HTTP responses) load level of 90% of the maximum throughput with zero packet loss as previously determined in section 4.4 (HTTP Capacity With No Transaction Delays). 4.6 HTTP Capacity with Transaction Delays Typical user behavior introduces delays between requests and reponses, for example, as users read web pages and decide which links to click next. This next set of tests is identical to the previous set except that these include a 5- second delay in the server response for each transaction. This has the effect of maintaining a high number of open connections throughout the test, thus forcing the sensor to utilize additional resources to track those connections KB HTTP Response with Delay Max 5,000 new connections per second per Gigabit of traffic with a 21KB HTTP response size maximum 185,000 packets per second per Gigabit of traffic. 5-second transaction delay resulting in an additional 50,000 open connections per Gigabit over the test described in section With average connection rates and average packet sizes, this is a good approximation of a real-world production network, and all sensors should be capable of performing well throughout this test KB HTTP Response with Delay Max 10,000 new connections per second per Gigabit of traffic with a 10KB HTTP response size maximum 225,000 packets per second per Gigabit of traffic. Repeated with background traffic loads of 25%, 50%, 75%, and 100% of maximum throughput of NGFW. 5-second transaction delay resulting in an additional 100,000 open connections over the test described in section With large average packet sizes coupled with very high connection rates, this represents a very heavily used production network, and is a strenuous test for any sensor. 4.7 Real-World Traffic Where previous tests provide a pure HTTP environment with varying connection rates and average packet sizes, the goal of this test is to simulate a real-world environment by introducing additional protocols and real content while still maintaining a precisely repeatable and consistent background traffic load. 25

26 The result is a background traffic load that is closer to what may be found on a heavily-utilized normal production network Real-World Protocol Mix (Enterprise Perimeter) Traffic is generated across the DUT comprising a protocol mix typically seen in an enterprise perimeter Real-World Protocol Mix (Financial) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large financial institution Real-World Protocol Mix (Education) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large educational environment Real-World Protocol Mix (Data Center) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large data center Real-World Protocol Mix (US Mobile Carrier) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large US mobile carrier Real-World Protocol Mix (European Mobile Carrier) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a European mobile carrier. 26

27 5 Stability & Reliability Long-term stability is particularly important for an in-line device, where failure can produce network outages. These tests verify the stability of the DUT along with its ability to maintain security effectiveness while under normal load and while passing malicious traffic. Products that are not able to sustain legitimate traffic (or that crash) while under hostile attack will not pass. The DUT is required to remain operational and stable throughout these tests, and to block 100% of previously blocked traffic, raising an alert for each. If any non-allowed traffic passes successfully, caused by either the volume of traffic or the DUT failing open for any reason, this will result in a FAIL. 5.1 Blocking Under Extended Attack The DUT is exposed to a constant stream of security policy violations over an extended period of time. The device is configured to block and alert, and thus this test provides an indication of the effectiveness of both the blocking and alert handling mechanisms. A continuous stream of security policy violations mixed with legitimate traffic is transmitted through the DUT at a maximum of 100Mbps for 8 hours with no additional background traffic. This is not intended as a stress test in terms of traffic load (covered in the previous section), it is merely a reliability test in terms of consistency of blocking performance. The DUT is expected to remain operational and stable throughout this test and to block 100% of recognizable violations, raising an alert for each. If any recognizable policy violations are passed, caused by either the volume of traffic or the DUT failing open for any reason, this will result in a FAIL. 5.2 Passing Legitimate Traffic under Extended Attack This test is identical to Test 5.1, where the external interface of the DUT is exposed to a constant stream of exploits over an extended period of time. The DUT is expected to remain operational and stable throughout this test, and to pass most/all of the legitimate traffic. If an excessive amount of legitimate traffic is blocked throughout this test, caused by either the volume of traffic or the DUT failing for any reason, this will result in a FAIL. 5.3 Behavior of the State Engine Under Load This test determines whether the DUT is capable of preserving state across a large number of open connections over an extended time period. At various points throughout the test (including after the maximum has been reached), it is confirmed that the DUT is still capable of inspecting and blocking traffic that is in violation of the currently applied security policy, whilst confirming that legitimate traffic is not blocked (perhaps as a result of exhaustion of the resources allocated to state tables). The DUT must be able to apply policy decisions effectively based on inspected traffic at all load levels. 27