TEST METHODOLOGY Distributed Denial- of- Service (DDoS) Prevention v1.0
Table of Contents 1 Introduction... 5 1.1 The Need for Distributed Denial- of- Service Prevention... 5 1.2 About This Test Methodology and Report... 5 1.3 Inclusion Criteria... 6 2 Product Guidance... 7 2.1 Recommended... 7 2.2 Neutral... 7 2.3 Caution... 7 3 Security Effectiveness... 8 3.1 DDoS Prevention Enforcement... 8 3.1.1 In- Line Protection... 8 3.1.2 Out- of- Path Protection... 8 3.2 DDoS Attack Categories... 8 3.2.1 Volumetric... 9 3.2.2 Protocol... 9 3.2.3 Application... 9 3.2.4 Layered Attacks... 10 3.3 Evasion... 10 3.3.1 IP Packet Fragmentation... 10 3.3.2 Stream Segmentation... 11 3.3.3 Layered Evasions... 11 4 Performance... 12 4.1 Raw Packet Processing Performance (UDP Traffic)... 12 4.1.1 64 Byte Packets... 12 4.1.2 128 Byte Packets... 12 4.1.3 256 Byte Packets... 12 4.1.4 512 Byte Packets... 12 4.1.5 1024 Byte Packets... 12 4.1.6 1514 Byte Packets... 12 4.2 Latency... 13 4.2.1 64 Byte Frames... 13 4.2.2 128 Byte Frames... 13 4.2.3 256 Byte Packets... 13 4.2.4 512 Byte Packets... 13 4.2.5 1024 Byte Packets... 13 4.2.6 1514 Byte Packets... 13 4.3 Maximum Capacity... 13 2014 NSS Labs, Inc. All rights reserved. 2
4.3.1 Theoretical Maximum Concurrent TCP Connections... 14 4.3.2 Theoretical Maximum Concurrent TCP Connections with Data... 14 4.3.3 Maximum TCP Connections Per Second... 14 4.3.4 Maximum HTTP Connections Per Second... 14 4.3.5 Maximum HTTP Transactions Per Second... 14 4.4 HTTP Capacity with No Transaction Delays... 15 4.4.1 44 KB HTTP Response Size 2,500 Connections Per Second... 15 4.4.2 21 KB HTTP Response Size 5,000 Connections Per Second... 15 4.4.3 10 KB HTTP Response Size 10,000 Connections Per Second... 15 4.4.4 4.5 KB HTTP Response Size 20,000 Connections Per Second... 15 4.4.5 1.7 KB HTTP Response Size 40,000 Connections Per Second... 15 4.5 HTTP Connections Per Second and Capacity (with Delays)... 16 4.5.1 21 KB HTTP Response Size with Delay... 16 4.5.2 10 KB HTTP Response Size with Delay... 16 4.6 Application Average Response Time: HTTP... 16 4.7 Real- World Traffic... 16 4.7.1 Real- World Protocol Mix (Data Center Virtualization Hub)... 16 4.7.2 Real- World Protocol Mix (Data Center Financial)... 16 4.7.3 Real- World Protocol Mix (Data Center Mobile Users and Applications)... 17 4.7.4 Real- World Protocol Mix (Data Center Web- Based Applications and Services)... 17 4.7.5 Real- World Protocol Mix (Data Center Internet Service Provider (ISP) Mix)... 17 4.8 Performance Under Attack... 17 4.8.1 Real- World Protocol Mix (Data Center Financial) While Under DDoS Attack... 17 4.8.2 Real- World Protocol Mix (Data Center Web- Based Applications and Services) While Under DDoS Attack... 17 4.8.3 Real- World Protocol Mix (Data Center Internet Service Provider (ISP) Mix) While Under DDoS Attack... 17 5 Stability and Reliability... 18 5.1 Mitigating Under Extended Attack... 18 5.2 Passing Legitimate Traffic Under Extended Attack... 18 5.3 Protocol Fuzzing and Mutation... 18 5.3.1 Protocol Fuzzing and Mutation Detection Ports... 18 5.3.2 Protocol Fuzzing and Mutation Management Port... 19 5.4 Power Fail... 19 5.5 Redundancy... 19 5.6 Persistence of Data... 19 6 Management and Configuration... 20 7 Total Cost of Ownership and Value... 21 2014 NSS Labs, Inc. All rights reserved. 3
Appendix A: Test Environment... 22 Appendix B: Change Log... 24 Contact Information... 25 2014 NSS Labs, Inc. All rights reserved. 4
1 Introduction 1.1 The Need for Distributed Denial- of- Service Prevention Over the past decade, the threat landscape has changed as more enterprises and large organizations have moved their mission- critical services online. Competing in global markets driven by just- in- time demand, these enterprises rely on continuous uptime to perform business transactions on a 24/7/365 model. This shift in the business model has, however, engendered a new breed of cyber attacks designed to limit access to these resources. Although distributed denial- of- service (DDoS) attacks technically are not new, they are more effective today than ever before. The relative ease with which DDoS attacks can be launched, the diverse methods by which such attacks can be executed, and the amount of damage that can be caused by a single attack make DDoS attacks a challenge to defend against. Such attacks have proved an effective way to wreak havoc, causing high- profile outages and interruptions to transaction processing. They can be motivated by a wide range of factors, and taking down websites or blocking transactions remain effective ways to make statements or cause visible and potentially far- reaching business disruptions. As enterprises look to defend against DDoS attacks, they are turning to DDoS prevention solutions, which offer protection against the different categories of DDoS attacks, and which can take the form of on- premise devices or managed services. Many vendors have entered the DDoS prevention market in recent years, and their solutions should be evaluated carefully. 1.2 About This Test Methodology and Report NSS Labs test reports are designed to address the challenges faced by IT professionals in selecting and managing security products. The scope of this particular report includes: Security effectiveness Performance Stability and reliability Management and configuration Total cost of ownership (TCO) In order to prevent downtime to mission- critical systems, DDoS prevention solutions must be able to be deployed without impacting legitimate user access. These solutions should improve the security stance of an enterprise by limiting potential threats attackers may use. Also, these solutions must function properly without requiring network infrastructure or network security redesign when being added to existing security architecture. The following capabilities are considered essential as part of a DDoS prevention solution: Detect and mitigate all categories of DDoS attacks Resistant to known evasion techniques Provide legitimate access to protected resources while under DDoS attack Be highly resilient and stable Ability to operate at layer 3 2014 NSS Labs, Inc. All rights reserved. 5
1.3 Inclusion Criteria In order to encourage the greatest participation, and allay any potential concerns of bias, NSS invites all leading DDoS prevention vendors to submit their products at no cost. Vendors with major market share, as well as challengers with new technology, will be included. DDoS prevention solutions should be implemented as in- line devices (whether routing or transparent) or as out- of- path solutions capable of interacting with an existing routing and switching environment using industry- supported protocols (including routing protocols like BGP). Solutions should be supplied with the appropriate number of physical interfaces capable of achieving the required level of connectivity and performance (i.e., a minimum of one port pair per Gigabit of throughput or one port pair per 10 Gbps of throughput). Solutions are subject to a minimum of one port pair per Gigabit of throughput. Thus, an 8 Gbps device with only four port pairs will be limited to 4 Gbps. The minimum number of port pairs will be connected to support the claimed maximum bandwidth of the solution being tested (thus an 8 Gbps DUT with ten port pairs will have eight 1Gbps connections tested). 2014 NSS Labs, Inc. All rights reserved. 6
2 Product Guidance NSS issues summary product guidance based on evaluation criteria that is important to information security professionals. The evaluation criteria are weighted as follows: Security effectiveness The primary reason for buying a DDoS prevention solution is to protect mission critical applications, data centers, and enterprises from experiencing any downtime due to an external attack. Resistance to evasion Failure in any evasion class allows attackers to circumvent protection. Stability Long- term stability is particularly important for a DDoS prevention solution, where failure can produce network outages. Performance Correctly sizing a DDoS prevention solution is essential. Management In particular, how difficult is it to configure the highest degree of protection across multiple devices or to make updates quickly as needs evolve? Value Customers should seek low TCO and high effectiveness and performance rankings. Products are listed in rank order according to their guidance rating. 2.1 Recommended A Recommended rating from NSS indicates that a product has performed well and deserves strong consideration. Only the top technical products earn a Recommended rating from NSS regardless of market share, company size, or brand recognition. 2.2 Neutral A Neutral rating from NSS indicates that a product has performed reasonably well and should continue to be used if it is the incumbent within an organization. Products that earn a Neutral rating from NSS deserve consideration during the purchasing process. 2.3 Caution A Caution rating from NSS indicates that a product has performed poorly. Organizations using one of these products should review their security posture and other threat mitigation factors, including possible alternative configurations and replacement. Products that earn a Caution rating from NSS should not be short- listed or renewed. 2014 NSS Labs, Inc. All rights reserved. 7
3 Security Effectiveness This section verifies that the DDoS prevention solution, or device under test (DUT), can detect and mitigate DDoS attacks effectively. NSS analysis is conducted first by testing every category of DDoS attack individually to determine that the DUT can successfully detect and mitigate the attack. Once a baseline of security effectiveness is determined, NSS builds upon this baseline by combining multiple DDoS attacks from different categories in an attempt to overwhelm the DUT and allow attack leakage to occur. At each point during testing of security effectiveness, NSS validates that legitimate traffic is still allowed and is not inadvertently mitigated by the DUT while mitigating the DDoS attack. 3.1 DDoS Prevention Enforcement There are two primary approaches that DDoS prevention solutions use to provide mitigation of attacks. Regardless of the approach, the DDoS solution must detect and mitigate the attack or attacks launched against a protected environment. For each attack tested, time to detection is measured along with the efficacy of the mitigation (i.e., the percentage of the attack was successfully mitigated). 3.1.1 In- Line Protection In- line DDoS prevention solutions adopt the traditional network security device posture of mitigating, or dropping, malicious traffic in- line, and as such, typically consist of a single appliance (or multiple appliances for high availability scenarios). These solutions are often deployed in front of or behind the perimeter security device. These types of DDoS prevention solutions can either be dedicated standalone appliances, or they can be integrated into other traditional security products such as intrusion prevention systems (IPS) and next generation firewalls (NGFW). This type of solution is generally deployed in enterprises and small- to- medium data centers; however, it is not limited to these environments, as it can be designed to handle high throughput scenarios. 3.1.2 Out- of- Path Protection Out- of- path posture is where the DDoS prevention solution actively monitors traffic at an ingress point for malicious activity. Once malicious activity is detected, the DDoS prevention solution uses technologies such as routing protocols, for example, BGP, to redirect the traffic to a dedicated appliance for inspection and to reintroduce the legitimate (i.e., not malicious) traffic into the network. This type of DDoS prevention solution commonly consists of more than one appliance and is designed to work in higher throughput environments such as large data centers and ISPs. 3.2 DDoS Attack Categories Attackers can target any part of the TCP/IP stack to be successful in their goal of network downtime. As such, NSS engineers test all layers to confirm the DUT can successfully mitigate attacks regardless of their layer in the stack. DDoS attacks can be broken into three distinct categories. Each category has a unique approach to deny access to the target; however, the goal is the same. 2014 NSS Labs, Inc. All rights reserved. 8
3.2.1 Volumetric On occasion, the easiest way to prevent access to a target is to consume all of the network bandwidth that the target of the attack has available. This is the goal behind a volumetric DDoS attack. The attacker, through various means, launches an attack designed to cause network congestion between the target and the rest of the Internet. This volume of traffic can be generated through multiple hosts, for example, a botnet, and leaves no available bandwidth for legitimate users of the resource (whether it is an ecommerce website or financial services group). Volumetric DDoS attacks generally target protocols that are stateless and do not have built- in congestion avoidance. Examples of volumetric DDoS attacks include (but are not limited to): ICMP packet floods (including all ICMP message types) Malformed ICMP packet floods UDP packet floods (usually containing no application layer data) Malformed UDP packet floods Spoofed IP packet floods Malformed IP packet floods Ping of Death Smurf attack 3.2.2 Protocol Attackers can also prevent access to a target by consuming other types of resources. Protocol DDoS attacks are designed to exhaust resources available on the target or on a specific device between the target and the Internet. The devices can include routers, load balancers, and even some security devices. When the DDoS attack consumes a resource such as a device s TCP state table, no new connections can be opened because the device is waiting for connections to close or expire. Protocol DDoS attacks need not consume all of the target s available bandwidth to make it inaccessible. Examples of protocol DDoS attacks include (but are not limited to): SYN floods ACK floods RST attacks TCP connection floods Land attacks TCP state exhaustion attacks Fragmentation attacks TCP window size attacks 3.2.3 Application Attackers also attempt to prevent access by exploiting vulnerabilities in the application layer. These vulnerabilities can be within an application layer protocol as well as within the application itself. Attacks on unpatched, vulnerable systems do not require as much bandwidth as either protocol or volumetric DDoS attacks in order to be successful. This style of DDoS attack may require, in some instances, as little as one or two packets to render the target unresponsive. Application DDoS attacks can also consume application layer or application resources by slowly opening up connections and then leaving them open until no new connections can be made. Examples of application DDoS attacks include (but are not limited to): 2014 NSS Labs, Inc. All rights reserved. 9
HTTP GET floods HTTP POST floods HTTP partial connection floods HTTP overlapping range header attacks DNS amplification attacks SSL exhaustion attacks SIP Invite floods 3.2.4 Layered Attacks This section attempts to bypass the DUT by layering DDoS attacks from the different categories defined in section 3.2 in an attempt to overwhelm the DUT. While the DUT is successfully mitigating a volumetric DDoS attack, its resources may be exhausted, and an application DDoS attack may bypass the DUT and render the target of the attack inoperable. All layered DDoS attacks will be verified prior to being added to the NSS DDoS attack library. 3.3 Evasion Attackers can modify basic DDoS attacks to evade detection in a number of ways. If a DUT fails to detect a single form of evasion, any attack can pass through the DUT and render it ineffective. NSS verifies that the DUT is capable of detecting and mitigating basic DDoS attacks when subjected to varying common evasion techniques. Wherever possible, the DUT is expected to successfully decode the obfuscated traffic in order to provide an accurate alert relating to the original attack, rather than alerting purely on anomalous traffic detected as a result of the evasion technique itself. A subset of DDoS attacks successfully detected and mitigated in their unmodified state during the testing of section 3.2 will be chosen for testing of section 3.3 as NSS is certain the vendor has coverage for them. It is a requirement of the test that the DUT submitted have all evasion detection options enabled by default in the shipping product. 3.3.1 IP Packet Fragmentation This test measures the effectiveness of the fragment reassembly mechanism of the DUT. Examples of IP packet fragmentation include (but are not limited to): Fragments from 8 32 bytes in size Ordered, out- of- order, or reverse order fragments Fragment overlap, favoring new and favoring old data Interleaved, duplicate, duplicate with or without incrementing DWORD, duplicate packets with random payload, or duplicate packets scheduled for later delivery Any combination of the above methods 2014 NSS Labs, Inc. All rights reserved. 10
3.3.2 Stream Segmentation This test measures the effectiveness of the stream reassembly mechanism of the DUT. Examples of stream segmentation include (but are not limited to): Segments from 1 2048 bytes in size Ordered, reverse ordered, or out- of- order segments, with favor old or favor new Duplicate, duplicate interleaved, duplicate last packet, or overlapping segments Invalid or NULL TCP control flags Sequence resync requests, random initial sequence number, or out- of- window sequence numbers Faked retransmits, PAWS elimination, or segments containing random data Endianness interchanged Any combination of the above methods 3.3.3 Layered Evasions This test attempts to bypass the DUT by performing any legitimate combination of the evasion techniques specified in section 3.3. It will be verified that the target machine s standard network stack is capable of decoding the evasion correctly while maintaining the attack viability. 2014 NSS Labs, Inc. All rights reserved. 11
4 Performance This section measures the performance of the DUT using various traffic conditions that provide metrics for real- world performance. Individual implementations will vary based on usage, however these quantitative metrics provide a gauge as to whether a particular DUT is appropriate for a given environment. NSS testing dictates that the product is configured to meet all requirements before testing is performed. If any changes are made to the DUT configuration to allow for the following performance tests to be completed, they will be noted. 4.1 Raw Packet Processing Performance (UDP Traffic) This test uses UDP packets of varying sizes generated by traffic generation tools to validate packet performance of the DUT. A constant stream of the appropriate packet size with variable source and destination IP addresses transmitting from a fixed source port to a fixed destination port is transmitted bi- directionally through each port pair of the DUT. Each packet contains dummy data and is targeted at a valid port on a valid IP address on the target subnet. The percentage load and frames per second (fps) figures across each in- line port pair are verified by network monitoring tools before each test begins. Multiple tests are run and averages taken where necessary. This traffic does not attempt to simulate any form of real- world network condition. No TCP sessions are created during this test, and there is very little for the state engine to do. The aim of this test is purely to determine the raw packet processing capability of each in- line port pair of the DUT and its effectiveness at forwarding packets quickly in order to provide the highest level of network performance and the lowest latency. 4.1.1 64 Byte Packets Maximum 1,488,000 frames per second per Gigabit of traffic. This test determines the ability of a device to process packets from the wire under the most challenging packet volume processing conditions. 4.1.2 128 Byte Packets Maximum 844,000 frames per second per Gigabit of traffic 4.1.3 256 Byte Packets Maximum 452,000 frames per second per Gigabit of traffic 4.1.4 512 Byte Packets Maximum 234,000 frames per second per Gigabit of traffic. This test provides a reasonable indication of the ability of a device to process packets from the wire on an average network. 4.1.5 1024 Byte Packets Maximum 119,000 frames per second per Gigabit of traffic 4.1.6 1514 Byte Packets Maximum 81,000 frames per second per Gigabit of traffic. This test has been included to demonstrate how easy it is to achieve good results using large packets. Readers should use caution when taking into consideration those test results that only quote performance figures using similar packet sizes. 2014 NSS Labs, Inc. All rights reserved. 12
4.2 Latency The aim of the latency and user response time tests is to determine the effect the DUT has on the traffic passing through it under various load conditions. Test traffic is passed across the infrastructure switches and through all in- line port pairs of the DUT simultaneously (the latency of the basic infrastructure is known and is constant throughout the tests). The packet loss and average latency (µs) are recorded for each packet size (64, 128, 256, 512, 1024 and 1514 bytes) at a load level of 90% of the maximum throughput with zero packet loss as previously determined in section 4.1. 4.2.1 64 Byte Frames Maximum 1,488,000 frames per second per Gigabit of traffic 4.2.2 128 Byte Frames Maximum 844,000 frames per second per Gigabit of traffic 4.2.3 256 Byte Packets Maximum 452,000 frames per second per Gigabit of traffic 4.2.4 512 Byte Packets Maximum 234,000 frames per second per Gigabit of traffic 4.2.5 1024 Byte Packets Maximum 119,000 frames per second per Gigabit of traffic 4.2.6 1514 Byte Packets Maximum 81,000 frames per second per Gigabit of traffic 4.3 Maximum Capacity The use of traffic generation tools allows NSS engineers to create true real world traffic at multi- Gigabit speeds as a background load for the tests. The aim of these tests is to stress the inspection engine and determine how it handles high volumes of TCP connections per second, application layer transactions per second, and concurrent open connections. All packets contain valid payload and address data, and these tests provide an excellent representation of a live network at various connection/transaction rates. Note that in all tests, the following critical breaking points where the final measurements are taken are used: Excessive concurrent TCP connections Unacceptable increase in open connections on the server- side Excessive response time for HTTP transactions Excessive delays and increased response time to client Unsuccessful HTTP transactions Normally, there should be zero unsuccessful transactions. Once these appear, it is an indication that excessive latency is causing connections to time out. 2014 NSS Labs, Inc. All rights reserved. 13
4.3.1 Theoretical Maximum Concurrent TCP Connections This test is designed to determine the maximum concurrent TCP connections of the DUT with no data passing across the connections. This type of traffic would not be typical of a normal network, but it provides the means to determine the maximum possible concurrent connections. An increasing number of Layer 4 TCP sessions are opened through the device. Each session is opened normally and then held open for the duration of the test as additional sessions are added up to the maximum possible. Load is increased until no more connections can be established, and this number is recorded. 4.3.2 Theoretical Maximum Concurrent TCP Connections with Data This test is identical to section 4.3.1, except that once a connection has been established, 21 KB of data is transmitted (in 1 KB segments). This ensures that the DUT is capable of passing data across the connections once they have been established. 4.3.3 Maximum TCP Connections Per Second This test is designed to determine the maximum TCP connection rate of the DUT with one byte of data passing across the connections. This type of traffic would not be typical of a normal network, but it provides the means to determine the maximum possible TCP connection rate. An increasing number of new sessions are established through the DUT, ramped slowly to determine the exact point of failure. Each session is opened normally, one byte of data passed to the host, and then the session is closed immediately. Load is increased until one or more of the breaking points defined earlier is reached. 4.3.4 Maximum HTTP Connections Per Second This test is designed to determine the maximum TCP connection rate of the DUT with a 1 byte HTTP response size. The response size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. A 1 byte response size is designed to provide a theoretical maximum HTTP connections per second rate. Client and server are using HTTP 1.0 without keep- alive, and the client will open a TCP connection, send one HTTP request, and close the connection. This ensures that all TCP connections are closed immediately upon the request being satisfied, thus any concurrent TCP connections will be caused strictly as a result of latency of the DUT. Load is increased until one or more of the breaking points defined earlier is reached. 4.3.5 Maximum HTTP Transactions Per Second This test is designed to determine the maximum HTTP transaction rate of the DUT with a 1 byte HTTP response size. The object size defines the number of bytes contained in the body, excluding any bytes associated with the HTTP header. A 1 byte response size is designed to provide a theoretical maximum connections per second rate. Client and server are using HTTP 1.1 with persistence, and the client will open a TCP connection, send 10 HTTP requests, and close the connection. This ensures that TCP connections remain open until all 10 HTTP transactions are complete, thus eliminating the maximum connection per second rate as a bottleneck (1 TCP connection = 10 HTTP transactions). Load is increased until one or more of the breaking points defined earlier is reached. 2014 NSS Labs, Inc. All rights reserved. 14
4.4 HTTP Capacity with No Transaction Delays The aim of these tests is to stress the HTTP detection engine and determine how the DUT copes with network loads of varying average packet size and varying connections per second. By creating genuine session- based traffic with varying session lengths, the DUT is forced to track valid TCP sessions, thus resulting in a higher workload than for simple packet- based background traffic. This provides a test environment that is as close to real world as it is possible to achieve in a lab environment, while ensuring absolute accuracy and repeatability. Each transaction consists of a single HTTP GET request, and there are no transaction delays (i.e., the web server responds immediately to all requests). All packets contain valid payload (a mix of binary and ASCII objects) and address data, and this test provides an excellent representation of a live network (albeit one biased towards HTTP traffic) at various network loads. Connections per Second 44Kbyte Response 21Kbyte Response 10Kbyte Response 4.5Kbyte Response 1.7Kbyte Response CPS 2,500 5,000 10,000 20,000 40,000 Mbps 1,000 1,000 1,000 1,000 1,000 Mbps 4.4.1 44 KB HTTP Response Size 2,500 Connections Per Second Maximum 2,500 new connections per second per Gigabit of traffic with a 44 KB HTTP response size average packet size 900 bytes maximum 140,000 packets per second per Gigabit of traffic. With relatively low connection rates and large packet sizes, all DUTs should be capable of performing well throughout this test. 4.4.2 21 KB HTTP Response Size 5,000 Connections Per Second Maximum 5,000 new connections per second per Gigabit of traffic with a 21 KB HTTP response size average packet size 670 bytes maximum 185,000 packets per second per Gigabit of traffic. With average connection rates and average packet sizes, this is a good approximation of a real- world production network, and all DUTs should be capable of performing well throughout this test. 4.4.3 10 KB HTTP Response Size 10,000 Connections Per Second Maximum 10,000 new connections per second per Gigabit of traffic with a 10 KB HTTP response size average packet size 550 bytes maximum 225,000 packets per second per Gigabit of traffic. With smaller packet sizes coupled with high connection rates, this represents a very high utilization production network. 4.4.4 4.5 KB HTTP Response Size 20,000 Connections Per Second Maximum 20,000 new connections per second per Gigabit of traffic with a 4.5 KB HTTP response size average packet size 420 bytes maximum 300,000 packets per second per Gigabit of traffic. With small packet sizes and extremely high connection rates, this is an extreme test for any DUT. 4.4.5 1.7 KB HTTP Response Size 40,000 Connections Per Second Maximum 40,000 new connections per second per Gigabit of traffic with a 1.7 KB HTTP response size average packet size 270 bytes maximum 445,000 packets per second per Gigabit of traffic. With small packet sizes and extremely high connection rates, this is an extreme test for any DUT. 2014 NSS Labs, Inc. All rights reserved. 15
4.5 HTTP Connections Per Second and Capacity (with Delays) Typical user behavior introduces delays between requests and reponses, e.g., think time, as users read web pages and decide which links to click next. This group of tests is identical to the previous group except that these include a 5 second delay in the client request for each transaction. This has the effect of maintaining a high number of open connections throughout the test, thus forcing the DUT to utilize additional resources to track those concurrent connections. 4.5.1 21 KB HTTP Response Size with Delay Maximum 5,000 new connections per second per Gigabit of traffic with a 21KB HTTP response size average packet size 670 bytes maximum 185,000 packets per second per Gigabit of traffic. 5 second transaction delay resulting in an additional 50,000 open connections per Gigabit over the test described in section 4.4.2. With average connection rates and average packet sizes, this is a good approximation of a real- world production network, and all DUTs should be capable of performing well throughout this test. 4.5.2 10 KB HTTP Response Size with Delay Maximum 10,000 new connections per second per Gigabit of traffic with a 10 KB HTTP response size average packet size 550 bytes maximum 225,000 packets per second per Gigabit of traffic. 5 second transaction delay resulting in an additional 100,000 open connections per Gigabit over the test described in section 4.4.3. With large average packet sizes coupled with very high connection rates, this represents a very heavily used production network and is a strenuous test for any DUT. 4.6 Application Average Response Time: HTTP Test traffic is passed across the infrastructure switches and through all port pairs of the DUT simultaneously (the latency of the basic infrastructure is known and is constant throughout the tests). The results are recorded at each response size (44 KB, 21 KB, 10 KB, 4.5 KB, and 1.7 KB HTTP responses) load level and at 90% of the maximum throughput with zero packet loss, as previously determined in section 4.4 and section 4.5. 4.7 Real- World Traffic Where previous sections provide a pure HTTP environment with varying connection rates and average packet sizes, the aim of this test is to emulate a real- world environment by introducing additional protocols and real content while still maintaining a precisely repeatable and consistent background traffic load. The result is a background traffic load that is closer to what may be found on a heavily- utilized normal production network. 4.7.1 Real- World Protocol Mix (Data Center Virtualization Hub) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large data center, focusing on virtualization traffic (VMotion, Hyper- V migration, etc.). 4.7.2 Real- World Protocol Mix (Data Center Financial) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large financial institution data center. 2014 NSS Labs, Inc. All rights reserved. 16
4.7.3 Real- World Protocol Mix (Data Center Mobile Users and Applications) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a large mobile carrier. 4.7.4 Real- World Protocol Mix (Data Center Web- Based Applications and Services) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a web hosting data center. 4.7.5 Real- World Protocol Mix (Data Center Internet Service Provider (ISP) Mix) Traffic is generated across the DUT comprising a protocol mix typical of that seen in a typical ISP installation, covering all types of traffic. 4.8 Performance Under Attack Previous sections provided a baseline of throughput using a pure HTTP environment with varying connection rates with average packet sizes as well as different real- world protocol mixes to emulate a heavily- utilized normal production network. None of these sections demonstrated the impact on normal or legitimate traffic while the DUT was providing DDoS mitigation. The goal of this section is to show impact to normal, legitimate traffic while the DUT is actively mitigating DDoS attacks. It provides the best representation of the DUT s ability to allow legitimate users access to protected resources while providing protection to the same resources under attack. The following real- world traffic mixes will be run at a specified percentage of the previously- determined maximum throughput, with a percentage of the remaining available bandwidth being consumed by DDoS attacks from section 3.2. 4.8.1 Real- World Protocol Mix (Data Center Financial) While Under DDoS Attack Traffic is generated across the DUT comprising a combination of the Real- World Data Center Financial protocol mix along with DDoS attacks targeting services included in this protocol mix. 4.8.2 Real- World Protocol Mix (Data Center Web- Based Applications and Services) While Under DDoS Attack Traffic is generated across the DUT comprising a combination of the Real- World Data Center Web- Based Applications and Services protocol mix along with DDoS attacks targeting services included in this protocol mix. 4.8.3 Real- World Protocol Mix (Data Center Internet Service Provider (ISP) Mix) While Under DDoS Attack Traffic is generated across the DUT comprising a combination of the Real- World Data Center Internet Service Provider (ISP) mix along with DDoS attacks targeting services included in this protocol mix. 2014 NSS Labs, Inc. All rights reserved. 17
5 Stability and Reliability Long- term stability is particularly important for an in- line device, where failure can produce network outages. These tests verify the stability of the DUT along with its ability to maintain security effectiveness while under normal load and while passing malicious traffic. Products that are not able to sustain legitimate traffic (or that crash) while under hostile attack will not pass. The DUT is required to remain operational and stable throughout these tests and to mitigate 100% of previously mitigated traffic, raising an alert for each. If any prohibited traffic passes successfully caused by either the volume of traffic or the DUT failing open for any reason this will result in a FAIL. 5.1 Mitigating Under Extended Attack The DUT is exposed to a constant stream of DDoS attacks over an extended period of time. The DUT is configured to mitigate and alert, and thus this section provides an indication of the effectiveness of both the mitigating and alert handling mechanisms. A continuous stream of DDoS attacks is mixed with legitimate traffic and transmitted through the DUT for a minimum of 8 hours. This is not intended as a stress test in terms of traffic load (covered in the previous section) but is merely a reliability test in terms of consistency of mitigating performance. The device is expected to remain operational and stable throughout this test, and to mitigate 100% of recognizable violations, raising an alert for each. If any DDoS attacks are allowed to pass through the DUT caused by either the volume of traffic or the DUT failing open for any reason this will result in a FAIL. 5.2 Passing Legitimate Traffic Under Extended Attack This test is identical to section 5.1, where the external interface of the DUT is exposed to a constant stream of DDoS attacks over an extended period of time. The DUT is expected to remain operational and stable throughout this test, and to pass all legitimate traffic. If an amount of legitimate traffic is mitigated during this test caused by either the volume of traffic or the DUT failing for any reason this will result in a FAIL. 5.3 Protocol Fuzzing and Mutation This test stresses the protocol stacks of the DUT by exposing it to traffic from various protocol randomizer and mutation tools. Several of the tools in this category are based on the ISIC test suite and other well- known test tools/suites. Traffic load is a maximum of 350Mbps and 60,000 packets per second (average packet size is 690 bytes). Results are presented as a simple PASS/FAIL the device is expected to remain operational and capable of detecting and mitigating attacks throughout the test. 5.3.1 Protocol Fuzzing and Mutation Detection Ports DUT is exposed to protocol fuzzing and mutation traffic across its inspection ports. 2014 NSS Labs, Inc. All rights reserved. 18
5.3.2 Protocol Fuzzing and Mutation Management Port DUT is exposed to protocol fuzzing and mutation traffic directed to the management port. 5.4 Power Fail Power to the DUT is cut whilst passing a mixture of legitimate and disallowed traffic. The option should exist to configure the DUT to fail closed such that no traffic is passed once power has been cut. This option does not have to be the default. 5.5 Redundancy Does the DUT include multiple redundant critical components (fans, power supplies, hard drive, etc.) (YES/NO/OPTION)? 5.6 Persistence of Data The DUT should retain all configuration data, policy data, and locally logged data once restored to operation following power failure. 2014 NSS Labs, Inc. All rights reserved. 19
6 Management and Configuration Security devices are complicated to deploy; essential systems such as centralized management console options, log aggregation, and event correlation/management systems further complicate the purchasing decision. Understanding key comparison points will allow customers to model the overall impact on network service level agreements (SLAs); estimate operational resource requirements to maintain and manage the systems; and better evaluate required skill/competencies of staff. As part of this test, NSS will perform in- depth technical evaluations of all the main features and capabilities of the centralized enterprise management systems offered by each vendor, covering the following key areas: General Management and Configuration How easy is it to install and configure devices and to deploy multiple devices throughout a large enterprise network? Policy Handling How easy is it to create, edit and deploy complicated security policies across an enterprise? Alert Handling How accurate and timely is the alerting? How easy is it to drill down to locate critical information needed to remediate a security problem? Reporting How effective and readily customizable is the reporting capability? For additional information concerning enterprise management testing, refer to the separate management questionnaire document. 2014 NSS Labs, Inc. All rights reserved. 20
7 Total Cost of Ownership and Value Organizations should be concerned with the ongoing, amortized cost of operating security products. This section evaluates the costs associated with the purchase, installation, and ongoing management of the DUT, including: Product Purchase the cost of acquisition Product Maintenance the fees paid to the vendor (including software and hardware support, maintenance and updates) Installation the time required to take the device out of the box, configure it, put it into the network, apply updates and patches, initial tuning, and set up desired logging and reporting Upkeep the time required to apply periodic updates and patches from vendors, including hardware, software, and firmware updates 2014 NSS Labs, Inc. All rights reserved. 21
Appendix A: Test Environment The aim of this methodology is to provide a thorough test of all of the main components of the DUT in a controlled and repeatable manner and in the most real- world environment that can be emulated in a test lab. The Test Environment The NSS Labs test network is a multi- Gigabit infrastructure that can accommodate Gigabit (copper or fiber), 10 Gigabit SFP+ interfaces, and 40 Gigabit QSFP+ interfaces. The DUT is configured for its use- case deployment defined in section 3.1 of this test methodology. Figure 1 In- Line Prevention Figure 2 Out- of- Path Prevention 2014 NSS Labs, Inc. All rights reserved. 22
Test equipment, such as the hosts generating DDoS attack traffic and other traffic generation tools, is connected to the external network to transmit emulated traffic. Test equipment, such as the vulnerable targets susceptible to attack or exploitation and test generation equipment emulating hosted services, is connected to the internal network. The DUT, depending on its deployment scenario, as is defined in Section 3.1, is connected between two gateway switches, one at the edge of the external network and one at the edge of the internal network. All normal network traffic, background load traffic, and DDoS attack traffic is transmitted through the environment containing the DUT, from external to internal (responses will flow in the opposite direction). The management interface is used to connect the appliance to the management console on a private subnet. This ensures that the DUT and console can communicate even when the target subnet is subjected to heavy loads, in addition to preventing attacks on the console itself. 2014 NSS Labs, Inc. All rights reserved. 23
Appendix B: Change Log Version 1.0 31 January 2014 Original Document 2014 NSS Labs, Inc. All rights reserved. 24
Contact Information NSS Labs, Inc. 206 Wild Basin Rd, Building A, Suite 200 Austin, TX 78746 USA +1 (512) 961-5300 info@nsslabs.com www.nsslabs.com v140131 This and other related documents available at: www.nsslabs.com. To receive a licensed copy or report misuse, please contact NSS Labs at +1 (512) 961-5300 or sales@nsslabs.com. 2014 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, photocopied, stored on a retrieval system, or transmitted without the express written consent of the authors. Please note that access to or use of this document is conditional on the following: 1. NSS Labs reserves the right to modify any part of the methodology before, or during, a test, or to amend the configuration of a device under test (DUT) where specific characteristics of the DUT or its configuration interfere with the normal operation of any of the tests, or where the results obtained from those tests would, in the good faith opinion of NSS Labs engineers, misrepresent the true capabilities of the DUT. Every effort will be made to ensure the optimal combination of security effectiveness and performance, as would be the aim of a typical customer deploying the DUT in a live network environment. 2. The information in this document is believed by NSS Labs to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this document are at the reader s sole risk. NSS Labs is not liable or responsible for any damages, losses, or expenses arising from any error or omission in this document. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY NSS LABS. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON- INFRINGEMENT ARE DISCLAIMED AND EXCLUDED BY NSS LABS. IN NO EVENT SHALL NSS LABS BE LIABLE FOR ANY CONSEQUENTIAL, INCIDENTAL OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This document does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet the reader s expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This document does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this document are the trademarks, service marks, and trade names of their respective owners. 2014 NSS Labs, Inc. All rights reserved. 25