Deciphering Detection Techniques: Part III Denial of Service Detection

Deciphering Detection Techniques: Part III Denial of Service Detection By Dr. Fengmin Gong, Chief Scientist, McAfee Network Security Technologies Group January 2003 networkassociates.com

Table of Contents I. Introduction... 3 II. What Is a Denial of Service Attack?... 3 III. What Is a Distributed Denial of Service Attack?... 3 IV. DOS/DDOS Detection Available Today... 4 V. The Requirements of an Effective DOS/DDOS Detection System... 4 1. Multiple Detection Mechanisms... 4 2. Attack Coverage... 4 3. Granularity of Attack Detection... 5 4. Consolidation of Alarms... 5 5. Response Actions... 5 VI. IntruShield System: Next-Generation Network IDS Designed for Effective DoS/DDoS Protection... 5 IntruShield Architecture... 6 IntruShield DoS and DDoS Detection Mechanisms... 6 IntruShield: DoS and DDoS Detection Coverage... 7 IntruShield: Unmatched Granularity of Detection... 8 IntruShield: Data Management and Consolidation of Alarms... 8 IntruShield: DoS/DDoS Response Actions... 8 VII. About the Author... 8 VIII. About McAfee Network Protection Services... 8 McAfee IntruShield... 9 IX. About Network Associates... 9 2 2003 Network Associates

I. Introduction Nothing is more crippling to your business than a network outage, and no one is immune to them. They cause operational problems, and it takes time to deal with them. Thousands of dollars are lost every second your services are unavailable to your customers. You may have invested in the personnel or infrastructure to respond to an equipment failure, but what do you do when you find yourself a victim of a Denial of Service (DoS) or a Distributed DoS (DDoS) attack? Can you afford to be disabled by hackers? DoS and DDoS attacks deny legitimate users access to critical network services. Hackers achieve this by launching attacks that consume excessive network bandwidth, host processing cycles, or other network infrastructure resources. DoS attacks have caused some of the world s largest companies to disappoint customers and investors as their Web sites became inaccessible to customers, partners, and usersæsometimes for up to twenty-four hours. For the victim, the impact can be extensive. Tools that enable DoS attacks are maturing to the point that even unsophisticated intruders could do serious damage. This paper details: The severe nature of DoS and DDoS attacks The requirements of an effective DoS/DDoS detection in a network IDS The highly accurate DoS and DDoS detection and pro-active response capabilities of the IntruShield System II. What Is a Denial of Service Attack? Unlike most other hacks, a Denial of Service (DoS) does not require the attacker to gain access or entry into the targeted server. The primary goal of a DoS attack is instead to deny legitimate users access to the service provided by that server. Attackers achieve their DoS objective by flooding the target until it crashes, becomes unreachable from the outside network, or can no longer handle legitimate traffic. The actual volume of the attack traffic involved depends on the type of attack traffic payload used. With crafted payload such as malformed IP fragments, several such packets may be sufficient to crash a vulnerable TCP/IP stack; on the other hand, it may take a very large volume of perfectly conforming IP fragments to overwhelm the defragmentation processing in the same TCP/IP stack. Sophisticated attackers may choose to use a mixture of normal and malformed payloads for a DoS attack. DoS attacks can vary in impact from consuming the bandwidth of an entire network, to preventing service use of a single targeted host, or crashing of a single service on the target host. Most DoS attacks are flood attacks; that is, attacks aimed at flooding a network with TCP connection packets that are normally legitimate, but consume network bandwidth when sent in heavy volume. The headers of malicious packets are typically forged, or spoofed, to fool the victim into accepting the packets as if they are originating from a trusted source. III. What Is a Distributed Denial of Service Attack? A Distributed Denial of Service (DDoS) attack is a DoS attack that is coordinated across many systems all controlled by a single attacker, known as a master. Prior to the attack, the master had compromised a large number of hosts, typically without their owners knowledge, and installed software that will later enable the coordinated attack. These compromised hosts, called zombies (a.k.a. daemons, agents, slaves, or bots), are then used to perform the actual attack. When the master is ready to launch the attack, every available zombie is contacted and instructed to attack a single victim. The master is not a part of the attack, thus tracing the true origin of a DDoS attack is very difficult. As with a DoS attack, packets sent from each zombie may be spoofed to fool the victim into accepting data from the trusted source. DDoS allows the attackers to utilize the network to multiplex low-volume sources into a high-volume stream in order to overwhelm the targets. Through the master-zombie communications, the real attackers can potentially hide their identities behind the zombies. 3 2003 Network Associates

IV. DOS/DDOS Detection Available Today Your ability to detect attacks directly affects your ability to react appropriately and to limit the damage caused by a DoS/DDoS attack. While Intrusion Detection Systems (IDS) have grown quite sophisticated and most products available today successfully detect most types of attacks, DoS and DDoS attacks are still difficult to detect with accuracy. The problem with DoS attacks is the sheer number of ways in which they can be executed, the increasingly sophisticated attack methods, and the growing range of systems targeted. Most of today s IDS products use a very simplistic method of detection. They compare current traffic behavior with acceptable normal behavior to detect DoS attacks, where normal traffic is characterized by a set of pre-programmed thresholds. These techniques establish a baseline and then look for jumps æsituations where the volume of network traffic jumps from low to very high levels. This simplistic approach suffers from several shortcomings. First, the threshold is typically set statically and thus requires user setting for every new environment and cannot adapt to changes to the environment. Second, only a small number of thresholds can be defined because very detailed statistics for protocol breakdown are not available to users. Third, thresholds can only be applied at high aggregate levels, e.g., per subnet, due to the lack of monitoring granularity. These shortcomings can lead to false positives and false negatives in detection depending on the threshold errors. Even if a detection is made correctly, a lack of granularity can limit one s ability to accurately identify and block the attack traffic. To be effective, a DoS/DDoS detection system requires additional features to help accurately detect an attack, and distinguish it from normal activity. V. The Requirements of an Effective DOS/DDOS Detection System An effective DoS/DDoS detection system is capable of detecting and responding to denial-of-service attacks on networked computers in real time. The method described above, threshold-based detection, is only one basic method, and there are ways to evade it. Combining that technique with others greatly improves your system s ability to determine benign traffic from an attack. 1. Multiple Detection Mechanisms With any baseline-based product, its effectiveness depends on how accurately your thresholds are set. These thresholds do not take into account flash crowds, or legitimate surges in traffic caused by benign behavior, such as everyone logging in to their systems around 9 am. Thresholds are static. Instead, the ability to learn the network, also known as statistical anomaly analysis, provides a solid view of how the network looks over time, and thus the product can quickly detect when something different takes place. Statistical anomaly analysis provides a more realistic picture of normal network behavior, and should be adaptive and intelligent, repeating the learning process automatically, and enabling itself to view network behavior over different time periods to acquire a varied and larger data sample. 2. Attack Coverage There are several general categories of DoS attacks. The CERT Coordination Center (CERT/CC), a federally funded research and development center operated by Carnegie Mellon University, divides attacks into three classes: bandwidth (or flood) attacks, protocol attacks, and logic attacks. Bandwidth attacks are relatively straightforward attempts to consume resources, such as network bandwidth or equipment throughput. Protocol attacks take advantage of the inherent design of common network protocols. These attacks use the expected behavior of protocols such as TCP, UDP, and ICMP to the attacker s advantage, and essentially befuddle the victim with specially crafted packets, as it tries to conform to standard protocol practice. Logic attacks exploit specific known vulnerabilities in network software, such as a Web server, or the underlying TCP/IP stack. While the categorization of DoS attacks is not well standardized, an effective IDS must be able to detect a DoS attack regardless of the actual means of attack. 4 2003 Network Associates

3. Granularity of Attack Detection To cover all of one s myriad, vulnerable bases, an effective IDS would protect the entire network. However, most sensorbased IDS products today are limited in their scope of coverage, in that one sensor is able to protect one link on the network. Thus to deploy enough IDS hardware to provide adequate coverage for the entire network would be costprohibitive. Worse, even when enough sensors are available for adequate coverage, there is no way to distinguish or protect a subset of the traffic analyzed by a single sensor. These products analyze all traffic or no traffic. This is a problem with many existing systems claiming anomaly-based detection, because their limited processing capacity forces them to work only with highly aggregated traffic. Today s networks are extremely heterogeneous, comprised of so many environments, systems, and servers that a much more granular approach to detection is necessary. The IDS must be able to monitor, with distinct and specialized security policies, each subset of the aggregate network traffic. Granular profiling is necessary in order to learn the normal traffic behavior accurately. Furthermore, only with granular separation will it be feasible to accurately isolate the attacking traffic and take countermeasures with little side effect. 4. Consolidation of Alarms It s not just a matter of detecting an attack, it s how the alarm is raised. Given the tendencies of DoS and DDoS attacks to last for more than a minute or two, imagine the number of alarms raised if your IDS issued alerts every second throughout the duration of a DDoS attack. CAIDA, the Cooperative Association for Internet Data Analysis at the University of California, San Diego, published a paper in 2001 called Inferring Internet Denial-of-Service Activity. Using a technique they call backscatter analysis, over a three-week-long time period they attempted to provide an estimate of worldwide denial-of-service activity. Their findings, which include more than 12,000 distinct attacks against 5,000 hosts belonging to more than 2,000 organizations worldwideæsites such as Amazon, @Home, or Hotmail to small foreign ISPsæshow that 50 percent of attacks are less than ten minutes in duration, 80 percent are less than thirty minutes, and 90 percent last less than an hour. Two percent of attacks are greater than five hours, 1 percent are greater than ten hours, and dozens spanned multiple days. Approximately.5 percent lasted less than one minute. An effective IDS must recognize the episodes of attacks and provide consolidated alerts without losing critical attack information. 5. Response Actions How fast you respond to an attack could make the difference between being buried under a mountain of packets and escaping unscathed. Accurate detection is the first step. Having your IDS respond in real time, particularly at wire speed, is the goal. Application of ingress and egress filtering using ACLs is a standard recommended response. Imagine if your IDS did this automatically, selectively, and in real-time. Of course, the user must be in control and you will enable this automatic response only upon reliable detection. VI. IntruShield System: Next-Generation Network IDS Designed for Effective DoS/DDoS Protection The industry s first real-time network intrusion prevention platform that has taken intrusion detection in general, and DoS and DDoS specifically to a new level. McAfee IntruShield network security products deliver an integrated hardware and software solution, which delivers comprehensive protection from known, first strike (unknown), DoS, and DDoS attacksæfrom several hundred Mbps to multi-gigabit speeds. The award-winning IntruShield architecture integrates patented signature, anomaly, and Denial of Service detection on a single purpose-built appliance. The IntruShield architecture not only enables highly accurate detection, but also empowers administrators with smart tools and processes, and enables flexible and scalable deployment for global businesses and vital government agencies. 5 2003 Network Associates

IntruShield Architecture The IntruShield architecture employs a combination of threshold-based and patented self-learning, profile-based detection techniques that delivers unmatched intelligence to detecting DoS and DDoS attacks and to blocking them. With straightforward threshold-based detection, IntruShield administrators can configure data traffic limits to ensure their servers will not become unavailable due to overload. These thresholds are selected based on coverage of different DDoS attacks and on the availability of statistics that will help the users to configure them. Meanwhile, self-learning methodologies enable IntruShield to study the patterns of network usage and traffic over time; thus understanding the wide variety of lawful, though unusual, usage patterns that may occur during legitimate network operations. The learning algorithm takes into account the bursty behavior that is common in all network traffic, and differentiates it from the real onset of DDoS traffic. In addition to learning the intensity behavior, it also learns the correlational behavior of different types of packets, which reliably captures TCP/IP protocol behavior, route configuration, and so on. Highly accurate DoS detection techniques are essential because popular Web sites and networks do experience legitimateæand sometimes unexpectedætraffic surges during external events, such as those which news sites experienced following the September 11 disaster, or for a particularly compelling new program, service, or application. The combination of these two techniques yields the highest accuracy of detection for the full spectrum of DoS and DDoS attacks, when hundreds or even thousands of hosts are co-opted by a malicious programmer to strike against a single victim. IntruShield DoS and DDoS Detection Mechanisms IntruShield sensors support two detection mechanisms: Learning-based and Threshold-based. IntruShield administrators can select either or both approaches for DoS/DDoS policy application. 1. Learning-Based Detection: IntruShield sensors can detect DoS and DDoS attacks by learning your network s normal traffic behaviors and detecting attacks based on deviations from these normal behaviors using a patent-pending algorithm. When a sensor is first installed or upon administrator instruction, the sensor enters a Learning Mode, during which normal traffic behaviors are learned by observing traffic for a short period of time. (The initial Learning Mode lasts 48 hours.) Example behaviors include relative distribution of ICMP echo request and echo reply, and bursty rate distribution of TCP SYN, TCP RST, and UDP packets. After the initial learning period, a long-term traffic profile is created and the sensor enters DoS/DDoS Detection Mode. In DoS/DDoS Detection Mode, the sensor compiles short-term traffic profiles to match against the learned longterm profile. Alerts are generated based precisely on deviations from the long-term profile using algorithms. Once operating in DoS/DDoS Detection Mode, the sensor continues to learn (at a more gradual rate) and automatically updates the long-term traffic profile. The system learns and models the traffic behavior on a two-week sliding window. When the onset of an attack is detected, the sensor suspends learning until the attack desists, thereby protecting the integrity of the profile. The learned profile is also saved on the sensor at configurable intervals, and the detection engine can be rolled back to a saved profile at any time. The administrator can discard the current long-term profile and roll back to an earlier state, or simply enter Learning Mode once again using the IntruShield Manager console. Long-term profiles can also be uploaded to the Manager from the sensor for archival purposes. An uploaded profile can be pushed to the sensor from the Manager at any time. Multiple profiles based on different behavioral aspects of the traffic provide the ability to differentiate between typical flash-crowd scenarios and real DDoS attacks. For example, in a typical Web-traffic flash crowd situation (i.e., all users logging in to the network at 9:00 A.M. Monday morning), there may be a legitimate traffic volume surge that could trigger a volume-based anomaly alert, assuming that the surge has reached a level of significance; however, since all of the traffic is legitimate with the proper three-way TCP handshake, there is no TCP control segment anomaly that typically comes with a SYN-flood attack. 6 2003 Network Associates

2. Threshold-Based Detection: IntruShield sensors can detect DoS/DDoS attacks based on administrator-configured thresholds by packet counts and rates for various types of packets such as ICMP, TCP SYN, UDP, IP fragments, and so forth. These threshold types have been selected so that a typical network administrator will be able to determine a reasonable threshold based on what is seen in normal operation and the limits of their resources (link bandwidth, server processing capacity, etc.). IntruShield users can also utilize some of the packet count statistics gathered by the IntruShield sensor (available from the IntruShield Manager) to guide the setting of these thresholds. In general, it does not make sense to use default thresholds because different environments have very different traffic mixes (packet types) and volume behaviors. IntruShield: DoS and DDoS Detection Coverage IntruShield sensors provide coverage for logic, and can detect the following DoS and DDoS attacks: Logic attacks, for example: o Land attack o Ping of Death o Teardrop Bandwidth, or Flood attacks, for example: o ICMP echo request Flood o TCP data segment Flood o TCP SYN/RST Flood o IP fragment Flood Protocol attacks, for example: o SYN flood o ICMP echo reply flood (aka Smurf) o UDP Flood (aka Fraggle) Additionally, the sensor is able to detect activity indicative of DDoS flooding attack tools, based on signatures detected in the traffic between the Attack Handler (master) and Attack Zombies (slaves). This can accurately detect zombie machines within your network. For example, IntruShield can detect traffic indicating the presence of the following tools: Trinoo TFN TFN2K Stacheldraht shaft mstream IntruShield is also able to detect reflective DDOS attacks. These are insidious attacks wherein a zombie sends Request packets (e.g., TCP SYN requests) to a reflector host with the source IP of the victim, thereby causing a huge number of response packets (e.g., SYN+ACK) from the reflector to the victim. In a reflective attack, it might seem to the victim, for example, that is actually under attack by a Web site. In general, the learning measures deployed in the IntruShield system accounts for all packets on the IPv4 network. Therefore, DDoS attacks based on any such packets are detectable. 7 2003 Network Associates

IntruShield: Unmatched Granularity of Detection IntruShield sensors are unique in that they provide multiple detection ports, enabling a single sensor to monitor multiple of a network. Thus a single sensor provides broad detection coverage. On a more granular level, an IntruShield sensor can detect attacks either at the aggregate link level or for a protected network, where a protected network is a subset of the traffic carried by the link. A protected network is defined using Virtual LAN (VLAN) tags, if the link is carrying VLAN traffic, or by specifying one or more Classless Inter-Domain Routing (CIDR) blocks that identify a subset of the traffic. Thus a protected network could be a single critical host such as DNS, Mail Relay, etc., or a subnetwork of the enterprise such as the enterprise's Web servers, HR servers, data center, desktop PCs, and so forth. For every protected network, the IntruShield sensor will learn the traffic behavior, and provide detection and response independently for that corresponding subset of traffic. This feature allows enterprise networks to specifically detect which hosts or group of hosts are under attack as well as the type of attack. Furthermore, it allows the administrator to be able to configure different detection policies (either Threshold- or Learning-based) and different response actions for each of the protected networks. IntruShield can support a few thousand protected networks, enabling a DoS/DDoS profile to be created for each protected network. IntruShield: Data Management and Consolidation of Alarms The IntruShield system recognizes the onset of attacks and understands how an ongoing attack manifests in observable traffic behavior. It consolidates the attack into a single episode, and will only issue alarms based on consolidated information of attack episodes and types. For example, it does not make sense to generate the same alert every second if a DDoS attack has occurred and is ongoing, since, as discussed earlier, a DDoS attack typically lasts longer than one minute. The IntruShield system recognizes such an onset of attacks and reports at the episode level to avoid overwhelming the users with repeated alerts. IntruShield: DoS/DDoS Response Actions IntruShield supports a rich set of response actions for DoS/DDoS attacks. IntruShield can accurately discriminate between good and bad traffic based on full protocol state maintenance and a patent-pending algorithm, which allows IntruShield to take the user-configurable response actions on only the bad traffic. Examples of bad traffic might include packets with illegal TCP/IP header information, application packets with illegal content fields (e.g., DNS), TCP packets without proper connections, and an abnormally large volume of packets of a given type from a previously unknown source that may overwhelm a protected resource. IntruShield can generate an alert with detailed DDoS attack information, which can be used by your firewalls to modify their ACLs and block the attack traffic in real time. For example, is it a flood of TCP SYN, ICMP, or fragmented IP? Drop DDoS traffic (available when the IntruShield sensor is operating in In-line Mode) based on what is outside the learned profile. The sensor can discriminate between DDoS (bad) traffic and non-ddos (good) traffic based on full protocol state maintenance and a patent pending algorithm. VII. About the Author Dr. Fengmin Gong is the Chief Scientist for the McAfee Network Security Technologies Group, where he is responsible for driving the continued innovation of IntruShield s security architectureæleveraging his expertise in areas such as signature, anomaly, and denial of service detection. Before to his work on IntruShield, Dr. Gong was Director of Advanced Networking Research at MCNC, a provider of sophisticated electronic and information technologies and services aimed at businesses and government agencies. 8 2003 Network Associates

While at MCNC and earlier at Washington University, he was involved in advanced security and networking projects for agencies such as DARPA, NSA, NSF, NLM, and NASA. During his time at MCNC he was also Adjunct Assistant Professor of Computer Science at North Carolina State University. In a distinguished academic and research career, Dr. Gong has written and contributed to nearly forty research papers on network intrusion, anomaly detection, secure collaboration, multi-media content delivery, and network quality of service. Dr. Gong has presented his research at industry events such as IEEE technical forums, as well as SIGGRAPH, DISCEX, NOMS, and ISCEX. VIII. About McAfee Network Protection Services McAfee Network Protection Solutions keep both large and smaller distributed networks up and protected from attacks. Best-of-breed network protection solutions in the portfolio include the Sniffer Network Protection Platform for performance management and fault identification, InfiniStream performing security forensics on network activity, Network Performance Orchestrator (npo) for centralizing and managing network activity, and McAfee IntruShield delivering network-based intrusion prevention. McAfee IntruShield McAfee IntruShield, a part of Network Associates McAfee Network Protection Solutions family of products, is a unique cutting-edge technology that prevents intrusions on the wire before they hit critical systems. Highly automated and easily managed, McAfee IntruShield is designed with such flexibility that it can be implemented in a phased approach - that overcomes the false positives inherent with today s legacy intrusion detection systems - and thus enables you to develop the right policy for blocking in your unique IT infrastructure. For example, you can deploy in-line to notify and block known attacks, and to notify-only on unknown attacks. Or you can implement complete blocking but just for business-critical network segments. IntruShield is delivered in a high-speed appliance which is able to scan traffic and assess threat levels with blinding speed, even on gigabit networks. It can be used at the edge or in front of key core resources. IntruShield has been crafted to satisfy both the security and network administrators as it stops a wide range of network attacks but does so with network latencies typically less than 10 milliseconds. IntruShield also looks for anomalous behavior and includes specialized analysis to find new denial of service mass attacks. IX. About Network Associates With headquarters in Santa Clara, California, Network Associates, Inc. is a leading supplier of network security and availability solutions. Network Associates comprises three product groups: McAfee Security, delivering world-class anti-virus and security products; Sniffer Technologies, a leader in network availability and system security; and Magic Solutions,a leader in innovative service management solutions. For more information, Network Associates can be reached in the United States at 972-308-9960 or on the Internet at http://www.networkassociates.com All Network Associates products are backed by our PrimeSupport program and Network Associates Laboratories. Tailored to fit your company s needs, PrimeSupport service offers essential product knowledge and rapid, reliable technical solutions to keep you up and running. Network Associates Laboratories, a world leader in information systems and security, is your guarantee of the ongoing development and refinement of all our technologies. Network Associates, Sniffer, McAfee, Magic Soultions, IntruShield, and PrimeSupport are either registered trademarks or trademarks of Network Associates, Inc. and/or its affiliates in the US and/or other countries. Sniffer brand products are made only by Network Associates, Inc. All other registered and unregistered trademarks in this document are the sole property of their respective owners. 2003 Networks Associates Technology, Inc. All Rights Reserved. 6-avd-ins-DoS-001/0603 9 2003 Network Associates