TEST METHODOLOGY. Distributed Denial-of-Service (DDoS) Prevention. v2.0

TEST METHODOLOGY Distributed Denial-of-Service (DDoS) Prevention v2.0

Table of Contents 1 Introduction... 4 1.1 The Need for Distributed Denial-of-Service Prevention... 4 1.2 About This Test Methodology and Report... 4 1.3 Inclusion Criteria... 5 2 Product Guidance... 6 2.1 Recommended... 6 2.2 Neutral... 6 2.3 Caution... 6 3 Security Effectiveness... 7 3.1 DDoS Prevention Enforcement... 7 3.1.1 Inline Protection... 7 3.2 DDoS Attack Categories... 7 3.2.1 Volumetric... 7 3.2.2 Protocol... 8 3.2.3 Application... 8 3.2.4 Layered Attacks... 8 4 Performance... 9 4.1 HTTP Capacity with No Transaction Delays... 9 4.1.1 2880 KB HTTP Response Size 40 Connections per Second... 10 4.1.2 768 KB HTTP Response Size 150 Connections per Second... 10 4.1.3 192 KB HTTP Response Size 600 Connections per Second... 10 4.1.4 44 KB HTTP Response Size 2500 Connections per Second... 10 4.1.5 21 KB HTTP Response Size 5000 Connections per Second... 10 4.1.6 10 KB HTTP Response Size 10,000 Connections per Second... 10 4.1.7 1.7 KB HTTP Response Size 40,000 Connections per Second... 10 4.2 Application Average Response Time: HTTP... 10 4.3 Real-World Traffic... 10 4.3.1 Real-World Protocol Mix (Data Center Virtualization Hub)... 10 4.3.2 Real-World Protocol Mix (Data Center Financial)... 11 4.3.3 Real-World Protocol Mix (Data Center Mobile Users and Applications)... 11 4.3.4 Real-World Protocol Mix (Data Center Web-Based Applications and Services)... 11 4.3.5 Real-World Protocol Mix (Data Center Internet Service Provider (ISP) Mix)... 11 5 Stability and Reliability... 12 5.1 Mitigating Under Extended Attack... 12 5.2 Passing Legitimate Traffic Under Extended Attack... 12 5.3 Protocol Fuzzing and Mutation... 12 5.3.1 Protocol Fuzzing and Mutation Detection Ports... 12 2

5.4 Power Fail... 13 5.5 Persistence of Data... 13 6 Total Cost of Ownership and Value... 14 Appendix A: Test Environment... 15 Appendix B: Change Log... 16 Contact Information... 17 3

1 Introduction 1.1 The Need for Distributed Denial-of-Service Prevention Over the past decade, the threat landscape has changed as more enterprises and large organizations have moved their mission-critical services online. Competing in global markets driven by just-in-time demand, these enterprises rely on continuous uptime to perform business transactions on a 24/7/365 model. This shift in the business model has, however, engendered a new breed of cyberattacks designed to limit access to these resources. Although distributed denial-of-service (DDoS) attacks are not new, they are more effective today than ever before. The relative ease with which DDoS attacks can be launched, the diverse methods by which such attacks can be executed, and the amount of damage that can be caused by a single attack make DDoS attacks a challenge to defend against. Such attacks have proved an effective way to wreak havoc, causing high-profile outages and interruptions to transaction processing. They can be motivated by a wide range of factors, and taking down websites or blocking transactions remain effective ways to make statements or cause visible and potentially far-reaching business disruptions. As enterprises look to defend against DDoS attacks, they are turning to DDoS prevention solutions, which offer protection against the different categories of DDoS attacks, and which can take the form of on-premises devices or managed services. Many vendors have entered the DDoS prevention market in recent years, and their solutions should be evaluated carefully. 1.2 About This Test Methodology and Report NSS Labs test reports are designed to address the challenges faced by IT professionals in selecting and managing security products. The scope of this particular report includes: Security effectiveness Performance Stability and reliability Management and configuration Total cost of ownership (TCO) In order to prevent downtime to mission-critical systems, DDoS prevention solutions must be able to be deployed without impacting legitimate user access. These solutions should improve the security stance of an enterprise by limiting potential threats attackers may use. Also, these solutions must function properly without requiring network infrastructure or network security redesign when being added to existing security architecture. The following capabilities are considered essential as part of a DDoS prevention solution: Detect and mitigate all categories of DDoS attacks Resistant to known evasion techniques Provide reasonable legitimate access to protected resources while under DDoS attack Be highly resilient and stable 4

1.3 Inclusion Criteria In order to encourage the greatest participation, and allay any potential concerns of bias, NSS invites all leading DDoS prevention vendors to submit their products at no cost. Vendors with major market share, as well as challengers with new technology, will be included. DDoS prevention solutions should be implemented as in-line devices (whether routing or transparent). Solutions should be supplied with the appropriate number of physical interfaces capable of achieving the required level of connectivity and performance (i.e., a minimum of one port pair per Gigabit of throughput or one port pair per 10 Gbps of throughput). Solutions are subject to a minimum of one port pair per Gigabit of throughput. Thus, an 8 Gbps device with only four port pairs will be limited to 4 Gbps. The minimum number of port pairs will be connected to support the claimed maximum bandwidth of the solution being tested (thus an 8 Gbps system under test (SUT) with ten port pairs will have eight 1Gbps connections tested). 5

2 Product Guidance NSS Labs issues summary product guidance based on evaluation criteria that are important to information security professionals. Key evaluation criteria are as follows: Security effectiveness Resistance to evasion Stability Performance Management Value Products are listed in rank order according to their guidance rating. 2.1 Recommended A Recommended rating from NSS indicates that a product has performed well and deserves strong consideration. Only the top technical products earn a Recommended rating from NSS regardless of market share, company size, or brand recognition. 2.2 Neutral A Neutral rating from NSS indicates that a product has performed reasonably well and should continue to be used if it is the incumbent within an organization. Products that earn a Neutral rating from NSS deserve consideration during the purchasing process. 2.3 Caution A Caution rating from NSS indicates that a product has performed poorly. Organizations using one of these products should review their security posture and other threat mitigation factors, including possible alternative configurations and replacement. Products that earn a Caution rating from NSS should not be short-listed or renewed. 6

3 Security Effectiveness This section verifies that the DDoS prevention solution, or SUT, can detect and mitigate DDoS attacks effectively. NSS analysis is conducted first by testing every category of DDoS attack individually to determine that the SUT can successfully detect and mitigate the attack. Once a baseline of security effectiveness is determined, NSS builds upon this baseline by combining multiple DDoS attacks from different categories in an attempt to overwhelm the SUT and allow attack leakage to occur. At each point during testing, NSS validates that legitimate traffic is still allowed and is not inadvertently mitigated by the SUT. 3.1 DDoS Prevention Enforcement There are two primary approaches that DDoS prevention solutions use to provide mitigation of attacks: in-line protection and out-of-band signaling. This methodology will focus on in-line enterprise systems. Regardless of the approach, the DDoS solution must detect and mitigate the attack or attacks launched against a protected environment. For each attack tested, success will be measured by performance and availability of the protected application along with the efficacy of the mitigation. 3.1.1 Inline Protection Inline DDoS prevention solutions adopt the traditional network security device posture of mitigating, or dropping, malicious traffic inline, and as such, typically consist of a single appliance (or multiple appliances for high availability scenarios). These solutions are often deployed in front of or behind the perimeter security device. These types of DDoS prevention solutions can either be stand-alone appliances, or they can be integrated into other traditional security products such as intrusion prevention systems (IPS) and next generation firewalls (NGFW). This type of solution generally is deployed in enterprises and small-to-medium data centers; however, it is not limited to these environments, as it can be designed to handle high throughput scenarios. 3.2 DDoS Attack Categories Attackers can target any part of the TCP/IP stack to be successful in their goal of network downtime. As such, NSS engineers test all layers to confirm the SUT can successfully mitigate attacks regardless of where they are located in the stack. DDoS attacks can be broken into three distinct categories. Each category has a unique approach to deny access to the target; however, the goal is the same. 3.2.1 Volumetric On occasion, the easiest way to prevent access to a target is to consume all of the network bandwidth that the target of the attack has available. This is the goal behind a volumetric DDoS attack. The attacker, through various means, launches an attack designed to cause network congestion between the target and the rest of the Internet. This volume of traffic can be generated through multiple hosts, for example, a botnet, and leaves no available bandwidth for legitimate users of the resource (whether it is an ecommerce website or a financial services group). Volumetric DDoS attacks generally target protocols that are stateless and do not have built-in congestion avoidance. Examples of volumetric DDoS attacks include (but are not limited to): ICMP packet floods (including all ICMP message types) 7

Malformed ICMP packet floods UDP packet floods (usually containing no application layer data) Malformed UDP packet floods Spoofed IP packet floods Malformed IP packet floods 3.2.2 Protocol Attackers can also prevent access to a target by consuming other types of resources. Protocol DDoS attacks are designed to exhaust resources available on the target or on a specific device between the target and the Internet. The devices can include routers, load balancers, and even some security devices. Once the DDoS attack consumes a resource such as a device s TCP state table, no new connections can be opened because the device is waiting for connections to close or expire. Protocol DDoS attacks need not consume all of the target s available bandwidth to make it inaccessible. Examples of protocol DDoS attacks include (but are not limited to): SYN floods ACK floods RST attacks TCP connection floods Fragmentation attacks TCP window size attacks NTP reflection DNS reflection 3.2.3 Application Attackers also attempt to prevent access by exploiting vulnerabilities in the application layer. These vulnerabilities can be within an application layer protocol as well as within the application itself. Attacks on unpatched, vulnerable systems do not require as much bandwidth as either protocol or volumetric DDoS attacks in order to be successful. This style of DDoS attack may require, in some instances, as little as one or two packets to render the target unresponsive. Application DDoS attacks can also consume application layer or application resources by slowly opening up connections and then leaving them open until no new connections can be made. Examples of application DDoS attacks include (but are not limited to): HTTP floods (e.g., various tools such as Low/High Orbit Ion Cannon/Anon Cannon) HTTP resource exhaustion (e.g., Slowloris, slowpost, RUDY) SSL exhaustion attacks (e.g., THC-SSL-DOS) SIP invite floods 3.2.4 Layered Attacks These tests attempt to bypass the SUT by layering DDoS attacks from the different categories defined in section 3.2 in an attempt to overwhelm the SUT. While the SUT is successfully mitigating a volumetric DDoS attack, its resources may be exhausted, and an application DDoS attack may bypass the SUT and render the target of the attack inoperable. All layered DDoS attacks will be verified prior to being added to the NSS DDoS attack library. 8

4 Performance This section measures the performance of the system using various traffic conditions that provide metrics for realworld performance. Individual implementations will vary based on usage; however, these quantitative metrics provide a gauge as to whether a particular SUT is appropriate for a given environment. The net difference between the baseline (without the SUT) and the measured capacity of the SUT is recorded for each of the following tests. 4.1 HTTP Capacity with No Transaction Delays The aim of these tests is to stress the HTTP detection engine and determine how the SUT copes with network loads of varying average packet size and varying connections per second. By creating genuine session-based traffic with varying session lengths, the SUT is forced to track valid TCP sessions, thus resulting in a higher workload than for simple packet-based background traffic. This provides a test environment that is as close to real world as it is possible to achieve in a lab environment, while ensuring absolute accuracy and repeatability. Each transaction consists of a single HTTP GET request, and there are no transaction delays (i.e., the web server responds immediately to all requests). All packets contain valid payload (a mix of binary and ASCII objects) and address data, and this test provides an excellent representation of a live network (albeit one biased towards HTTP traffic) at various network loads. 9

4.1.1 2880 KB HTTP Response Size 40 Connections per Second Maximum 40 new connections per second per Gigabit of traffic with a 2,880 KB HTTP response. With low connection rates and large packet sizes, all SUTs should be capable of performing well throughout this test. 4.1.2 768 KB HTTP Response Size 150 Connections per Second Maximum 150 new connections per second per Gigabit of traffic with a 768 KB HTTP response size. With low connection rates and large packet sizes, all SUTs should be capable of performing well throughout this test. 4.1.3 192 KB HTTP Response Size 600 Connections per Second Maximum 600 new connections per second per Gigabit of traffic with a 192 KB HTTP response. With medium packet sizes and increased connection rates, this represents an average production network. 4.1.4 44 KB HTTP Response Size 2500 Connections per Second Maximum 2,500 new connections per second per Gigabit of traffic with a 44 KB HTTP response size. With decreased packet sizes and increased connection rates, this may also represent an average production network. 4.1.5 21 KB HTTP Response Size 5000 Connections per Second Maximum 5,000 new connections per second per Gigabit of traffic with a 21 KB HTTP response size. 4.1.6 10 KB HTTP Response Size 10,000 Connections per Second Maximum 10,000 new connections per second per Gigabit of traffic with a 10 KB HTTP response. With small packet sizes and high connection rates, this is an strenuous test for any SUT. 4.1.7 1.7 KB HTTP Response Size 40,000 Connections per Second Maximum 40,000 new connections per second per Gigabit of traffic with a 1.7 KB HTTP response size. With very small packet sizes and extremely high connection rates, this is an extreme test for any SUT 4.2 Application Average Response Time: HTTP Test traffic is passed across the infrastructure switches and through all port pairs of the SUT simultaneously (the latency of the basic infrastructure is known and is constant throughout the tests). The results are recorded at each HTTP response size (2880 KB, 768 KB, 192 KB, 44 KB, 21 KB, 10 KB, and 1.7 KB) and at 90% of the maximum throughput with zero packet loss, as previously determined in section 4.1. 4.3 Real-World Traffic Where previous sections provide a pure HTTP environment with varying connection rates and average packet sizes, the aim of this test is to emulate a real-world environment by introducing additional protocols and real content while still maintaining a precisely repeatable and consistent background traffic load. The result is a background traffic load that is closer to what may be found on a heavily-utilized normal production network. 4.3.1 Real-World Protocol Mix (Data Center Virtualization Hub) Traffic is generated across the SUT comprising a protocol mix typical of that seen in a large data center, focusing on virtualization traffic (e.g., VMotion, Hyper-V migration). 10

4.3.2 Real-World Protocol Mix (Data Center Financial) Traffic is generated across the SUT comprising a protocol mix typical of that seen in a large financial institution data center. 4.3.3 Real-World Protocol Mix (Data Center Mobile Users and Applications) Traffic is generated across the SUT comprising a protocol mix typical of that seen in a large mobile carrier. 4.3.4 Real-World Protocol Mix (Data Center Web-Based Applications and Services) Traffic is generated across the SUT comprising a protocol mix typical of that seen in a web hosting data center. 4.3.5 Real-World Protocol Mix (Data Center Internet Service Provider (ISP) Mix) Traffic is generated across the SUT comprising a protocol mix typical of that seen in a typical ISP installation, covering all types of traffic. 11

5 Stability and Reliability Long-term stability is particularly important for an in-line device, where failure can produce network outages. These tests verify the stability of the SUT along with its ability to maintain security effectiveness while under normal load and while passing malicious traffic. Products that are not able to sustain legitimate traffic (or that crash) while under hostile attack will not pass. The system is expected to remain operational and stable throughout this test, and to mitigate recognizable violations, raising an alert for each. 5.1 Mitigating Under Extended Attack The SUT is exposed to a constant stream of DDoS attacks over an extended period of time. The SUT is configured to mitigate and alert, and thus this section provides an indication of the effectiveness of both the mitigating and alert handling mechanisms. A continuous stream of DDoS attacks is mixed with legitimate traffic and transmitted through the SUT for a minimum of 8 hours. This is not intended as a stress test in terms of traffic load (covered in the previous section) but is merely a reliability test in terms of consistency of mitigating performance. The system is expected to remain operational and stable throughout this test, and to mitigate recognizable violations, raising an alert for each. If any DDoS attacks are allowed to pass through the SUT caused by either the volume of traffic or by the SUT failing open for any reason service impact will be rated as a percentage. 5.2 Passing Legitimate Traffic Under Extended Attack This test is identical to section 5.1, where the external interface of the SUT is exposed to a constant stream of DDoS attacks over an extended period of time. The SUT is expected to remain operational and stable throughout this test, and to pass all legitimate traffic. If any amount of legitimate traffic is mitigated during this test caused by either the volume of traffic or by the SUT failing for any reason service impact will be rated as a percentage. 5.3 Protocol Fuzzing and Mutation This test stresses the protocol stacks of the SUT by exposing it to traffic from various protocol randomizer and mutation tools. Several of the tools in this category are based on the ISIC test suite and other well-known test tools/suites. Traffic load is a maximum of 350 Mbps and 60,000 packets per second (average packet size is 690 bytes). Results are presented as a simple PASS/FAIL the device is expected to remain operational and capable of detecting and mitigating attacks throughout the test. 5.3.1 Protocol Fuzzing and Mutation Detection Ports The SUT is exposed to protocol fuzzing and mutation traffic across its inspection ports. 12

5.4 Power Fail Power to the SUT is cut whilst passing a mixture of legitimate and disallowed traffic. The system is expected to maintain connectivity for application availability and should be configured to fail open. 5.5 Persistence of Data The SUT should retain all configuration data, policy data, and locally logged data once restored to operation following power failure. 13

6 Total Cost of Ownership and Value Implementation of security solutions can be complex, with several factors affecting the overall cost of deployment, maintenance, and upkeep. All of these should be considered over the course of the useful life of the solution. Product purchase the cost of acquisition Product maintenance the fees paid to the vendor (including software and hardware support, maintenance, and updates) Installation the time required to take the device out of the box, configure it, deploy it in the network, apply updates and patches, perform initial tuning, and set up desired logging and reporting Upkeep the time required to apply periodic updates and patches from vendors, including hardware, software, and firmware updates 14

Appendix A: Test Environment The aim of this methodology is to provide a thorough test of all of the main components of the SUT in a controlled and repeatable manner and in the most real-world environment that can be emulated in a test lab. The Test Environment The NSS test network is a multi-gigabit infrastructure that can accommodate Gigabit (copper or fiber), and 10 Gigabit fiber SFP+ interfaces. The SUT is configured for its use-case deployment (see section 3.1). Figure 1 In-Line Prevention Test equipment, such as the hosts generating DDoS attack traffic and other traffic generation tools, is connected to the external network to transmit emulated traffic. Test equipment, such as the vulnerable targets susceptible to attack or exploitation and test generation equipment emulating hosted services, is connected to the internal network. The SUT, depending on its deployment scenario (see section 3.1), is connected in line via an aggregation switch. All normal network traffic, background load traffic, and DDoS attack traffic is transmitted through the environment containing the SUT, from external to internal (responses will flow in the opposite direction). The management interface is used to connect the appliance to the management console on a private subnet. This ensures that the SUT and console can communicate even when the target subnet is subjected to heavy loads, in addition to preventing attacks on the console itself. 15

Appendix B: Change Log Version 2.0 17 October 2014 Original Document Changes added for review to 2.0 8 October 2015 Correction (reinserted 21 KB test in Section 4.1) 16

Contact Information NSS Labs, Inc. 206 Wild Basin Rd, Building A, Suite 200 Austin, TX 78746 USA info@nsslabs.com www.nsslabs.com This and other related documents available at: www.nsslabs.com. To receive a licensed copy or report misuse, please contact NSS Labs. 2015 NSS Labs, Inc. All rights reserved. No part of this publication may be reproduced, copied/scanned, stored on a retrieval system, e-mailed or otherwise disseminated or transmitted without the express written consent of NSS Labs, Inc. ( us or we ). Please read the disclaimer in this box because it contains important information that binds you. If you do not agree to these conditions, you should not read the rest of this report but should instead return the report immediately to us. You or your means the person who accesses this report and any entity on whose behalf he/she has obtained this report. 1. The information in this report is subject to change by us without notice, and we disclaim any obligation to update it. 2. The information in this report is believed by us to be accurate and reliable at the time of publication, but is not guaranteed. All use of and reliance on this report are at your sole risk. We are not liable or responsible for any damages, losses, or expenses of any nature whatsoever arising from any error or omission in this report. 3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY US. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, ARE HEREBY DISCLAIMED AND EXCLUDED BY US. IN NO EVENT SHALL WE BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, PUNITIVE, EXEMPLARY, OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF ADVISED OF THE POSSIBILITY THEREOF. 4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or software) tested or the hardware and/or software used in testing the products. The testing does not guarantee that there are no errors or defects in the products or that the products will meet your expectations, requirements, needs, or specifications, or that they will operate without interruption. 5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations mentioned in this report. 6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of their respective owners. 17