An Efficient and Reliable DDoS Attack Detection Using a Fast Entropy Computation Method

Transcription

1 An Efficient and Reliable DDoS Attack Detection Using a Fast Entropy Computation Method Giseop No and Ilkyeun Ra * Department of Computer Science and Engineering University of Colorado Denver, Campus Box 109, 1200 Larimer St., Denver, CO80204, USA Tel: , kafa46@hanmail.net * Tel: , Fax: , ilkyeun.ra@ucdenver.edu Abstract The threat of Distributed Denial of Service (DDoS) has become a major issue in network security and is difficult to detect because all DDoS traffics have normal packet characteristics. Various detection and defense algorithms have been studied. One of them is an entropy-based intrusion detection approach that is a powerful and simple way to identify abnormal conditions from network channels. However, the burden of computing information entropy values from heavy flow still exists. To reduce the computing time, we have developed a DDoS detection scheme using a compression entropy method. It allows us to significantly reduce the computation time for calculating information entropy. However, our experiment suggests that the compression entropy approach tends to be too sensitive to verify real network attacks and produces many false negatives. In this paper, we propose a fast entropy scheme that can overcome the issue of false negatives and will not increase the computational time. Our simulation shows that the fast entropy computing method not only reduced computational time by more than 90% compared to conventional entropy, but also increased the detection accuracy compared to conventional and compression entropy approaches. I. INTRODUCTION The main features of an Internet network are its open environment and scalability. On one hand, these characteristics have led the growth of the Internet. On the other hand, vulnerabilities in the network have occurred simultaneously. The threat of Distributed Denial of Service (DDoS) attacks now has become a major issue in network security. Launching a DDoS attack becomes easier for DDoS attackers while the defenders have a more difficult detecting malicious network flow since the DDoS attacker now uses normal packets flow with spoofed packet information. A burden for the defenders is to process all packet information within a limited time because a DDoS attacker sends a lot of normal packets to a victim. Although there is a good monitoring scheme against DDoS attacks, it still needs relatively high computational time to identify an attack from a normal packet flow. The few current studies have focused mainly on reducing computation resources to detect a DDoS attack. This paper concentrates on designing an efficient DDoS attack detection method that can both significantly reduce computational time and increase detection accuracy. To yield less computation time, we use the information entropy concept as an attack detection estimator. We introduce two entropy computational approaches: compression entropy using a data compression scheme and a modified entropy estimator, called Fast Entropy. The compression entropy provides the possibility of greatly reduced computational power by using a lossless data compression scheme. However, it is too sensitive to detect intrusion effectively. To moderate sensitivity of compression entropy, we designed the Fast Entropy. We find that our Fast Entropy scheme has better performance in terms of speed and accuracy than conventional entropy-based network detection and can reduce computational time nearly 90% compared to a conventional entropy scheme. This paper is presented in the following order. Section II presents the related work that includes Information Entropy and network intrusion detection with Conventional Entropy. New entropy approaches are introduced in Section III. DDoS detector design is described in Section IV. Simulation and analysis follow in Section V. Finally, our conclusion is discussed in Section VI. II. RELATED WORK A. Information Entropy Entropy is a concept identified by Shannon in 1948 [1]. Entropy is a quantity, a measure of the uncertainty of a random variable. Let X be a discrete random variable with alphabet χ and probability mass function p(x) = Pr {X=x}, x χ. The entropy H(X) of a discrete random variable X is defined as H(X) = -Σ x χ p(x)log p(x), (1) where 0log0 = 0, and H(X) 0 since 0 p(x ) 1 The function of the basic properties of entropy is defined as a concave function of the distribution. The entropy value equals 0 when p = 0 or 1. Similarly, the entropy is maximum when p = 1/2. This property easily can be used in network traffic monitoring. If network traffic changes from normal to abnormal status such as when the DDoS attacker sends a bulk of packets with the same port number to saturate a certain port, the entropy of this port number will be decreased. By contrast, under normal conditions, the entropy of the port number will /09/$ IEEE 1223 ISCIT 2009

2 be increased. This phenomenon can be applied to various network information such as source IP address, destination IP address, source port, destination port, total number of packets, and even in the data clustering schemes. B. DDoS Detection Approaches Various studies introduced several detection approaches. First, a Signature Based Approach (SBA) was introduced based on knowledge of known attack patterns. General SBAs work as follows: 1) find a pattern or a signature of an attack, 2) generate attack signatures and save them in a database, and 3) update the attack database if there is a new attack(s). SBA is efficient because it is easy to implement. Also, it identifies known attacks with low false negatives. However, SBA has several disadvantages [2]: 1) all systems having SBA must be trained, 2) SBA has potential false negatives because it may not detect even simple variations of attacks (it only detects exactly the same attack patterns according to the knowledge of the database), 3) SBA has false negative characteristics if an attack failed or a system was poorly configured, and 4) if the signature is stolen, the detection system no longer works properly. Since a DDoS attack has no attack signature (it increases normal packets to saturate network capacity), the SBA cannot work efficiently against DDoS attacks. Due to the limitations of SBA such as human errors, false positives, and false negatives, a safer detection approach, called an Anomaly Based Approach (ABA) has been proposed. It uses distribution analysis approaches, data mining, and statistical approaches. Usually, ABA is considered to be a more complex architecture. In fact, however, ABA needs to consult every pattern of incoming traffic, which means more work load to maintain a relatively high security level. Distribution-based approaches use a distribution of traffic information Base Distribution and Inverse Distribution were used in network intrusion detection [3, 4]. In the case of Statistical Anomaly-based approaches, a GAIA sensor (local DDoS sensor based on statistical analysis of the traffic), which uses modeling, mediation, detection, and alert generation components, is used in early detection scheme [5] using statistical quantity. Using chi-square statistic, researchers in [6] presented a DDoS attack detection algorithm. However, these approaches still need computational power to calculate the estimation statistic. Therefore, distribution-based and statistical approaches are not sufficient since they require a large number of computations to yield statistics or distribution of network flow. Entropy-based approaches to DDoS detection were introduced and turned out to be powerful network intrusion detection schemes. Entropy-based approaches have significant benefits in intrusion detection [7]: 1) the use of entropy can increase the sensitivity of detection to uncover anomalous incidents, 2) the use of such traffic features provides additional diagnostic information into the nature of the anomalous incidents, and 3) entropy of traffic feature distributions offers useful information to measure distance among traffic groups (clusters). Even though using Entropy has several advantages, it still needs an efficient algorithm to reduce computational time and memory usage in a high speed network. As previously shown in (1), to calculate entropy of a packet stream, the algorithm must have a probability (pi) that there will be a frequency of xi out of the arrived packets. We need to store every packet in the packet repository with its counter. Therefore, every packet needs to search for whether or not its packet is already stored in the repository. III. NEW ENTROPY APPROACHES A. Compression Entropy To reduce the computational time of the entropy estimator, we investigated several approaches and learned that data compression could be an efficient way of reducing computation time, and we studied how to use the compression scheme to get faster entropy computation by using linear regression to analyze the relation between original source entropy and compressed entropy for worm detection [8]. However, the relation from linear regression does not guarantee the concrete relation, which means that it has a potential error factor. Ratko Tomic introduced a Fast, Optimal Entropy Coder (FOEC) by using a combination scheme in data compression [9]. We find that the information entropy can be generated as a by-product during the FOEC process. We introduce a way to calculate entropy values based on Tomic s lossless data compression method. In this section, we will discuss the general procedures of Tomic s method, and then show how to use these values in a DDoS detection approach with heap structure. Packet information (source/destination IP address, source/destination port number) can be represented with a binary string (0 and 1). Also, the information is nothing but distinct bits (0s and 1s). If we use an indexing lattice path of the given source string, we can compute entropy with only the number of 1s counted [9]. I n (b 1 b 2 b n ) =, where 0 n 1 < n 2 < < n k < n (2) bi : i-th binary element n : total number of bi where k is the number of 1 s in Sn{ nj } : a subsequence of the sequence { i 1}, retaining only those values of (i-1) for which bi = 1, or in words, nj : a zero-based bit index which picks out only the 1 s from the input string Sn. The size (in bits) of the path index In with k ones is log(n(n-k,k)) = log(c(n,k)), where the binomial coefficient C(n,k) is the path count for n-bit strings with k ones. Applying the Stirling-Approximation for the factorials C(n,k), the path index size log(c(n,k)) which is compression entropy becomes: Log(C(n,k)) n[ plog(1/p) + qlog(1/q)] 1/2log(2π n p q), Where the probabilities through: p(1) p= k/n and p(0) q = (n k ) /n 1224

3 We can use the idea of the Multi Alphabet Source scheme from FOEC [9]. As shown in equation (2), the entropy of the binary source can be computed with the count of 1s. Let a sequence of n symbols be S n taking values from an alphabet A q = { a 1, a 2,..., a q }, and be a list of q counts k 1, k 2, k 2,... k q (each count k i counting the corresponding symbol a i ) adding up to n. k 1 + k 2 + k k q = n (3) The number of different arrangements of these n symbols with the given symbol counts k i satisfying (3) is the multinomial coefficient: N( n, k1, k2, kq) = =!!!! = = = (4) If we take a log of both sides, we can get the entropy of sequence of alphabet. Entropy(A q ) = log( ) + log () + + log( ) = log( ) + + log ( ) + log() (5) With this entropy value from FOEC, we can start the interpretation of the lossless Compression Entropy from the first term of (5), log( ) until the last term, log( ). 1 st Term: C (k 1 + k k q, k q ) = C (n, k q ) by 2 nd Term: C (k 1 + k k q-1, k q-1 ) = C (n-k q, k q-1 ) 3 nd Term: C (k 1 + k k q-2, k q-2 ) = C (n-k q -k q-1, k q-2 ) : : (q-1)-th Term: C (k 1 + k 2, k 1 ) = C (n- k q - k q k q-2, k 1 ) q-th Term: C (k 1, k 1 ) = C (n- k q - k q k q-2 - k q-1, k 1 ) = 1 Since in every i-th term computation, the number of objects from which we can choose (let this number be n i ) is subtracted by k q-i-1 element, the size of n i keeps being decreased. The number of same packet information (for example, address or port number) means k i in equation (4). For instance, if we insert all of arrived source IP address into heap, we can pull out the element until pulled data meets different address value. If we count those number of packets and define this number as k i, we can calculate entropy value in each step (herein, i-th term). It is very similar to heap sort. Also, the k i means the number of 1s in the stream in the i-th stage. We can get total entropy by adding up the entropies from all steps. B. Fast Entropy An entropy, H, is a quantity of disorder. If the disorder (the number of types of symbols) is increased, H must be increased. Otherwise, H must be decreased. Therefore, we can simplify the expression of entropy as follows: H = log (the number of possible states within input information) =, where p i is the probability i-th symbol occurrences from all observed symbols. One idea to reduce the consumption of computational resources during calculating information entropy is that we use only the number of different types of symbols without computing probabilities. Thus, we can redefine the entropy as H = log (the number of possible system state). But, the new definition of entropy has one problem since it doesn t reflect the total number of symbols. Usually, an attacker significantly increases the number of different symbols (packets) to paralyze a victim s system by saturating a victim s system capacity during DDoS attack. As a result, we need to add one more criterion the total number of symbols. Now, we define entropy as: H = =, where, m is the number distinct packets, n is total number of packets in an input. However, this new entropy could still increase the false positives that are very critical in monitoring systems. For instance, if an attacker increases the number of packets, he/she may also increase the number of packet types simultaneously. In that case, the ratio will not be changed noticeably, and the entropy value will stay almost same, which makes the algorithm miss attacks. To overcome the issue of the false positive, we propose an entropy calibration factor to increase the sensitivity to increasing packet numbers. The proposed entropy calibration is presented as follows. Let ni be the total number of packets in monitoring interval ti. We now monitor 2 variables, which are ni-1 and ni. These two values are adjacent values in the monitoring time series. Let us define calibration entropy as follows: H = Where, If n i n i-1,, if n i n i-1, if n i < n i-1, then. 1225

4 With the same fashion, n i < n i-1 will have same range We will use the value ratio value between 0 and 1, since this region among various log base values is very significantly changing. By taking absolute value, we can monitor the entropy of the change in the number of packets. Also, we will use log base 2 since a significant change of entropy is between 5 and 10 if the network changes abruptly because the packet number ratio of (0.5, 1] has a similar pattern between 5 and 10. With the value of lower than 0.5, the logarithmic value drops significantly, thus reflecting remarkable changes in the traffic flow of the network. If there is no traffic change in terms of packet number, the ratio is 1, and the entropy change is zero. (log1 = 0, no impact in detection facility). In order to detect DDoS attacks, we should continuously monitor entropy values sequentially per every monitoring interval, called window size. Thus, each entropy value should be calculated with respect to each fixed moving average window. There are variants of the simple moving average method. We will use the simple moving average, since we assume the traffic packet arriving is identically and independently distributed (i.i.d), memoryless, and a stationary process. The Figure 1 shows the concept of attack monitoring with simple moving average with size k. Assume we monitor the entropy values for m intervals (i.e., window size of k). If we have a monitoring interval of t seconds, we monitor the entropy value for m t seconds. In every monitoring interval t, an entropy value is computed. Let τ be log We can write if n i n i-1, and log if n i < n i-1 H = + τ, where, m is the number distinct packets, n is total number of packets in an input, and τ is packet number calibration factor (same as H ). To acquire only the number of distinct packets, we use heap structure similar to the lossless compression entropy. However, unlike the lossless compression entropy, we need only the insert operation because we can know n during the insert operation via the comparison step in heap. This fact can contribute to reduce the computational time of fast entropy more than the lossless compression entropy computation. If we monitor the packet every t time over T time period, and if we also monitor m packets in t, and then the total packets over T T is n. In other words, the number of packets is m. Thus, the runtime of computing the number of types can be defined as follows: + + =, Where, is average packet number in t, can be represented as = T = n t T. Now, let t T be α, and then we can simplify the above equation as follows: = t T = α log (n * α -1 ) = α log n α log α Therefore, the runtime of the fast entropy can be represented by O (α log n α log α), which is faster than total runtime of conventional entropy O ( ) and compression entropy O(n log n). IV. DDOS DETECTOR DESIGN Let s define as follows: μ i : i-th average of Moving Average Window σ: Standard Deviation of H n-m ~ H n-1 with μ i D i : absolute value of difference between μ i and H n (i.e., D i = μ i - H n ) β: threshold multiplication factor, positive integer value (default μ = 3) ω: threshold (ω = β * σ) Fig 1. Monitoring Concept with Moving Average Once μi is computed, it will be compared with Hn. To detect a traffic pattern change, if Di ω, we decide that we have an attack (under an attack) in the current monitoring interval n. Otherwise, the traffic condition is still the normal condition (out of attack). Once a comparison is done, the Moving Average Window will be moving forward along with time evolution (μi will start at tn-m+1). V. SIMULATION AND ANALYSIS A. Input Data We wanted to run our simulation program with various datasets from different institutes, but we could not get many actual datasets that have DoS/DDoS attacks. We have collected four different datasets to verify our proposed method: 1) one normal dataset (University of Colorado Denver BSS Computer Lab Traffic) and three different DoS/DDoS datasets (1999 DARPA and two 2000 DARPA datasets). We mixed or interleaved DoS/DDoS datasets with the normal dataset to see how our proposed algorithm will 1226

5 detects attacks. The detailed description of each dataset is described as follows: Normal Data Flow (University Computer Lab) University computer lab filled with normal computer user (student) packets at the Behavioral Science (BSS) Lab of University of Colorado Denver. DoS and Port Sweep Attack (1999 DARPA Dataset) We mixed two network flows together. One is the normal data flow from the BSS lab. The other is DoS attack traffic extracted from a DARPA dataset on April (99 DARPA). We have precisely separated DoS and Probe attack packets from 99 DARPA, which are 5 DoS attacks and 2 stealthy Probing attacks. We interleaved 99 DARPA attacks in BSS lab traffic. Typical DDoS (Non-Stealthy 2000 DARPA Dataset) The 2000 DARPA dataset (00 DARPA) is a typical dataset of DDoS attack traffic. We can categorize the five attack phases in the 00 DARPA dataset: phase1 (IP sweeping), phase2 (Probing IPs), phase3 (Penetrating via vulnerability), phase4 (Install attack software), and phase5 (Launching DDoS). Stealthy DDoS (Stealthy 2000 DARPA Dataset) The 2000 DARPA Stealthy dataset (00 Stealthy DARPA) is stealthier than the 1999 DARPA dataset but also has 5 attack phases as does the Non-Stealthy 2000 DARPA dataset. We set up only BSS lab traffic at the beginning of 30 minutes; after that we laid two mixed datasets (BSS lab dataset and 2000 DARPA Stealthy dataset). Fig. 2. Conventional Entropy Distribution with Typical DDoS Dataset (Non-Stealthy DARPA 2000) Fig. 3. Compression Entropy Distribution with Typical DDoS Dataset (Non-Stealthy DARPA 2000) B. Simulation Procedure We implemented our proposed algorithm with C language and ran our proposed detection program many times using all four datasets on a desktop PC with an MS Windows XP operating system. For our simulation, we fixed our moving average window size for this paper and ran many simulations using different datasets to find out the suitable threshold value (ω). The threshold value (ω) will be another key parameter to determine the accuracy of the detection. In this paper, we have selected 4σ as the threshold value (ω) and we tested our detection program for all datasets. We will present the number of packets at every detection point, detection accuracy tables (conventional, combination, and our fast entropy detectors), and adaptive detector performance for the 99 DARPA, 00 DARPA, and 00 stealthy DARPA datasets. C. Simulation Result Analysis and Evaluation The simulation results suggest that our Fast Entropy Scheme has higher accuracy in DoS, DDoS or Stealthy DDoS attack cases. Figures 2~4 portray the entropy distributions with a typical DDoS dataset. The compression entropy scheme yields huge entropy values with extreme sensitivity (see Fig. 3). Fig. 4. Fast Entropy Distribution with Typical DDoS Dataset (Non-Stealthy DARPA 2000) We simulated three entropy schemes to evaluate the detection accuracy with three datasets within the threshold range between 2σ and 6σ. A result with ω = 4σ is presented in TABLE I. Under the DoS attack, the conventional entropy scheme has the worst performance with high false negatives in TABLE I, while the Fast Entropy Scheme shows the best performance without any false negatives. TABLE I also shows that our Fast Entropy scheme shows a higher detection accuracy than conventional entropy and compression scheme against typical and stealthy DDoS attacks. From the simulation with threshold range (2σ ~ 6σ), our Fast Entropy approach yields powerful results in almost all ranges (3σ~5σ) against a typical DDoS attack, since it has both low false positives and relatively low false negatives compared to conventional and compression entropy schemes. 1227

6 For the stealthy DDoS input, the compression entropy scheme represents the network flow changes since it is very sensitive to changes of the network channel, which was the main fault of the typical DoS/DDoS detector. However, our Fast Entropy scheme shows us the main change in phase 5 (DDoS launching stage), which makes a detector capable of detecting an anomaly. Meanwhile, the conventional entropy scheme does not display any pattern change, which implies that it has poor performance under stealthy DDoS attack D. Runtime Analysis The lossless compression entropy and fast entropy can reduce the computational time almost 90% by using a heap structure compared to conventional entropy calculation. However, as we discussed in detection accuracy, the Compression Entropy Scheme has high false negatives over all thresholds, which means it doesn t work well as the network monitoring algorithm with information entropy, even though it has fast data compression ability Entropy Type Conventional Compression 0 Fast DoS TABLE I. DETECTION ACCURACY RESULT Threshold :ωσ Input Data Normal DDoS Stealthy DDoS Fig. 5. Runtime Distribution Error Type False False Positives Negatives DoS a 6 1 Typical DDoS b 4 1 Stealthy DDoS c 4 0 DoS 3 3 Typical DDoS 3 6 Stealthy DDoS 2 12 DoS 4 0 Typical DDoS 2 3 Stealthy DDoS 2 1 a DoS: 99 DARPA Dataset b Typical Dataset: Non-Stealthy 00 DARPA Dataset c Stealthy DDoS: Stealthy 00 DARPA Dataset The Fast Entropy Detection Scheme needs the smallest amount of time among the three entropy schemes (even faster Fast Compression Conventional than compression entropy). Our Fast Entropy DDoS detection scheme is the best fit in terms of information entropy approaches, with high probability of detection and very low computational time. VI. CONCLUSIONS Conventional entropy is known as the efficient algorithm to monitor changes of network conditions. It needs time to calculate probabilities of distinct packet types; computing probabilities of distinct packets take very long computational time. In this paper, we have proposed the fast entropy approach that combines the lossless compression entropy of the FOEC method, and the entropy calibration that uses the number of packet types and the number of packets based on the idea that DDoS attacks rely mainly not on packet types alone as in conventional entropy, but both the packet types and traffic volume (the number of packets). We report that our Fast Entropy scheme reduced computational time by 90% of conventional entropy scheme while maintaining detection accuracy. Fast Entropy is even faster than compression entropy scheme in computing entropy values with same or better detection accuracy. For our future work, we have been developing an adaptive fast entropy algorithm that will further reduce the false positives as well as false negatives without adding overhead by introducing dynamic moving average and detection threshold value with respect to behavior of attacks. REFERENCES [1] C.E. Shannon, A Mathematical Theory of Communication, Bell System Technical Journal, vol. 27, pp & , Oct [2] T. Ditcheva and Lisa Fowler, Signature-based Intrusion Detection class notes for COMP , University of North Carolina at Chapel Hill, Feb [3] S. Singh, C. Estan, G.Varghese, and S. Savage, Automated Worm Fingerprinting, Proceedings of the 6 th OSDI 6, Dec [4] V. Karamcheti, D. Geiger, Z. Kedem, and S. Muthukrishnan, Detecting Malicious Network Traffic Using Inverse Distribution of Packet Content, Proceedings of ACM SIGCOMM 2005 workshop on Mining Network Data, [5] E. Besson, A. Gouget, and H. Sibert, The GAIA Sensor: an Early DDoS Detection Tool, ACM SIGMETRICS Performance Evaluation Review 34, 2006, pp.7-8. [6] B. Song, J Heo, and C. S. Hong, Collaborative Defense Mechanism Using Statistical Detection Method against DDoS attacks, IEICE TRANS. COMMUN E90-B, 2007, pp [7] A. Lall, V. Sekar, M. Ogihara, J. Xu, and H. Zhang, Data Streaming Algorithms for Estimating Entropy of Network Traffic, ACMSIGMETRICS PERFORMANCE Evaluation Review 34, 2006, pp [8] A. Wanger, Entropy-Based Worm Detection for Fast IP Networks, Ph.D. dissertation, Swiss Federal Institution of Zechnology, Zurich, Swiss, [9] R. V. Tomic, Fast, Optimal Entropy Coder, 1 st Works Corporation Technical Report TR ,