On Tracing Attackers of Distributed Denial-of-Service Attack through Distributed Approaches

Transcription

1 On Tracing Attackers of Distributed Denial-of-Service Attack through Distributed Approaches WONG, Tsz Yeung A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Computer Science and Engineering c The Chinese University of Hong Kong September 2007 The Chinese University of Hong Kong holds the copyright of this thesis. Any person(s) intending to use a part or the whole of the materials in the thesis in a proposed publication must seek copyright release from the Dean of the Graduate School.

2 Thesis/Assessment Committee Professor YOUNG Fung Yu (Chair) Professor WONG Man Hon (Thesis Supervisor) Professor LEE Moon Chuen (Committee Member) Professor LEONG Hong Va (External Examiner)

3 Abstract The denial-of-service attack has been a pressing problem in recent years. Denial-of-service defense research has blossomed into one of the main streams in network security. Various techniques such as the pushback message, the ICMP traceback, and the packet filtering techniques are the remarkable results from this active field of research. The focus of this thesis is to study and devise efficient and practical algorithms to tackle the flood-based distributed denial-of-service attacks (flood- -based DDoS attack for short), and we aim to trace every location of the attacker. In this thesis, we propose a revolutionary, divide-and-conquer traceback methodology. Tracing back the attackers on a global scale is always a difficult and tedious task. Alternatively, we suggest that one should first identify Internet service providers (ISPs) that contribute to the flood-based DDoS attack by using a macroscopic traceback approach. After the concerned ISPs have been found, one can narrow the traceback problem down, and then the attackers can be located by using a microscopic traceback approach. For the macroscopic traceback problem, we propose an algorithm, which leverages the well-known Chandy-Lamport s distributed snapshot algorithm, so that a set of border routers of the ISPs can correctly gather statistics in a coordinated fashion. The victim site can then deduce the local traffic intensities of all the participating routers. Given the collected statistics, we provide a method for the victim site to locate the attackers who sent out dominating flows of packets. Our finding shows that the proposed methodology can i

4 pinpoint the location of the attackers in a short period of time. In the second part of the thesis, we study a well-known technique against the microscopic traceback problem. The probabilistic packet marking (PPM for short) algorithm by Savage et.al. has attracted the most attention in contributing the idea of IP traceback. The most interesting point of this IP traceback approach is that it allows routers to encode certain information on the attack packets based on a pre-determined probability. Upon receiving a sufficient number of marked packets, the victim (or a data collection node) can construct the set of paths the attack packets traversed (or the attack graph), and hence the victim can obtain the locations of the attackers. In this thesis, we present a discrete-time Markov chain model that calculates the precise number of marked packets required to construct the attack graph. Though the PPM algorithm is a desirable algorithm that tackles the microscopic traceback problem, the PPM algorithm is not perfect as its termination condition is not well-defined in the literature. More importantly, without a proper termination condition, the traceback results could be wrong. In this thesis, we provide a precise termination condition for the PPM algorithm. Based on the precise termination condition, we devise a new algorithm named the rectified probabilistic packet marking algorithm (RPPM algorithm for short). The most significant merit of the RPPM algorithm is that when the algorithm terminates, it guarantees that the constructed attack graph is correct with a specified level of confidence. Our finding shows that the RPPM algorithm can guarantee the correctness of the constructed attack graph under different probabilities that the routers mark the attack packets and different structures of the network graphs. The RPPM algorithm provides an autonomous way for the original PPM algorithm to determine its termination, and it is a promising means to enhance the reliability of the PPM algorithm. ii

5 摘 要 這 數 年 間, 分 散 式 阻 斷 服 務 攻 擊 已 成 為 一 個 迫 切 需 要 解 決 的 問 題 故 防 治 分 散 式 阻 斷 服 務 攻 擊 的 研 究 已 成 為 一 個 主 要 的 網 絡 保 安 課 題 這 活 躍 的 研 究 疇 產 生 了 多 個 卓 越 的 研 究 結 果, 如 pushback 信 息 技 術 ICMP 追 蹤 技 術 及 封 包 過 濾 技 術 等 本 論 文 主 要 研 究 防 治 洪 水 式 阻 斷 服 務 攻 擊 ( 簡 稱 洪 水 攻 擊 ) 的 方 案, 及 設 計 可 行 的 高 效 的 演 算 法 以 防 治 洪 水 攻 擊 本 論 文 主 要 研 究 方 向, 是 研 究 方 案 用 以 找 出 洪 水 攻 擊 的 發 動 地 點 我 們 提 出 一 種 創 新 的 以 分 治 法 為 本 的 追 蹤 技 術, 以 追 蹤 發 動 洪 水 攻 擊 的 地 點 洪 水 攻 擊 的 規 模 往 往 是 全 球 性 的, 故 追 蹤 發 動 攻 擊 的 地 點 亦 往 往 是 困 難 與 煩 瑣 的 因 此, 我 們 提 出 一 個 二 步 的 追 蹤 方 案, 以 追 蹤 全 球 性 洪 水 攻 擊 的 發 動 地 點 第 一 步, 所 有 的 網 絡 供 應 商 要 合 作, 以 找 出 那 些 網 絡 供 應 商 包 含 了 洪 水 攻 擊 的 發 動 地 點 這 一 步, 我 們 稱 之 為 宏 觀 追 蹤 方 案 下 一 步, 當 發 現 了 那 些 網 絡 供 應 商 包 含 了 洪 水 攻 擊 的 發 動 地 點, 有 關 的 網 絡 供 應 商 便 會 採 用 微 觀 追 蹤 方 案, 以 追 蹤 在 網 絡 供 應 商 內 的 所 有 的 洪 水 攻 擊 的 發 動 地 點 本 論 文 提 出 一 宏 觀 追 蹤 演 算 法 該 宏 觀 追 蹤 演 算 法 是 建 基 於 有 名 的 Chandy-Lamport 分 佈 式 快 照 演 算 法, 以 進 行 分 佈 式 的 追 蹤 我 們 命 名 該 演 算 法 為 快 照 追 蹤 演 算 法 快 照 追 蹤 演 算 法 是 在 各 網 絡 供 應 商 的 邊 界 路 由 器 上 執 行 的, 而 這 些 路 由 器 將 按 照 演 算 法 的 指 示 合 作 地 收 集 數 據, 再 把 數 據 送 給 洪 水 攻 擊 的 受 害 網 站 根 據 路 由 器 的 數 據, 受 害 網 站 即 可 以 排 列 出 各 網 絡 供 應 商 輸 出 的 攻 擊 流 量 從 而, 受 害 網 站 即 可 以 找 出 有 可 能 的 攻 擊 發 動 地 點 根 據 我 們 的 研 究 發 現, 快 照 追 蹤 演 算 法 是 一 個 高 效 的 演 算 法, 能 在 短 時 間 內 找 出 攻 擊 發 動 地 點 iii

6 本 論 文 接 著 探 討 微 觀 追 蹤 演 算 法 或 然 性 封 包 編 碼 演 算 法 ( 簡 稱 PPM 演 算 法 ) 是 一 個 備 受 著 目 的 IP 追 蹤 演 算 法, 而 該 演 算 法 亦 是 適 合 成 為 微 觀 追 蹤 演 算 法 PPM 演 算 法 值 得 留 意 的 特 點, 在 於 其 根 據 一 個 預 先 設 定 的 或 然 率, 稱 為 編 碼 或 然 率, 在 網 絡 供 應 商 內 的 路 由 器 上, 把 封 包 選 擇 性 的 編 碼 當 洪 水 攻 擊 的 受 害 網 站 收 到 足 夠 的 已 編 碼 封 包,PPM 演 算 法 便 可 以 計 算 出 攻 擊 封 包 的 行 走 路 線 從 而,PPM 演 算 法 便 可 以 找 出 洪 水 攻 擊 的 發 動 地 點 在 本 論 文 中, 我 們 研 究 出 一 個 馬 爾 可 夫 鏈 模 型, 能 讓 受 害 網 站 準 確 地 計 算 出 需 要 的 已 編 碼 封 包 數 量, 以 計 算 出 準 確 的 攻 擊 封 包 行 走 路 線 縱 使 PPM 演 算 法 是 一 個 優 秀 的 微 觀 追 蹤 演 算 法, 可 惜 的 是, 由 於 現 在 沒 有 研 究 項 目 把 它 的 停 止 運 作 條 件 作 明 確 的 定 義, 因 此 PPM 演 算 法 並 不 算 是 一 個 完 美 的 演 算 法 更 重 要 的 是, 若 果 PPM 演 算 法 的 停 止 運 作 條 件 是 錯 誤 的 話, 它 的 追 蹤 結 果 ( 即 攻 擊 封 包 的 行 走 路 線 ) 將 會 是 錯 誤 的 本 論 文 將 為 PPM 演 算 法, 研 究 出 一 個 精 確 的 停 止 運 作 條 件 由 於 新 的 停 止 運 作 條 件 將 為 PPM 演 算 法 帶 來 改 變, 我 們 把 新 的 演 算 法 命 名 為 修 正 的 或 然 性 封 包 編 碼 演 算 法 ( 簡 稱 RPPM 演 算 法 ) RPPM 演 算 法 最 重 要 的 價 值, 在 於 它 能 保 證 RPPM 演 算 法 的 追 蹤 結 果 是 在 一 個 指 定 的 準 確 度 以 上 我 們 的 研 究 發 現, 在 不 同 的 編 碼 或 然 率 及 網 絡 架 構 之 下,RPPM 演 算 法 都 可 以 保 證 追 蹤 結 果 是 在 指 定 的 準 確 度 之 上 總 結 RPPM 演 算 法 的 優 點, 是 其 能 為 PPM 演 算 法 帶 來 自 動 化 的 停 止 運 作 條 件, 從 而 提 高 了 PPM 演 算 法 的 可 靠 性 iv

7 Acknowledgement In completing this thesis, I am most grateful to my thesis advisor, Dr. Manhon Wong, and my former thesis advisor, Dr. John Chi-shing Lui, who have been giving continuous support and guidance to me throughout the past five years. I am also glad to have my colleagues in the Department Computer Science and Engineering, especially Mr. C. M. Lee, Mr. Ray Lam, Mr. Y. T. Ma, Mr. T. B. Ma, Mr. Y. K. Liu, Mr. Y. K. Hui, Ms. Catherine Zhou, and Dr. L.C. Lau. They have given me invaluable advice and support through my years of research life. Last but not least, I am most glad to have Ms. Elaine Chan who has been giving me unconditional love and the strength to get through the difficulties I encountered. v

8 Contents 1 Defense Against Denial-of-Service Attack Overview of Attack Methodology Vulnerability-based attack Flood-based attack Worm attack Flash crowd Scope of the Thesis General assumptions A divide-and-conquer traceback approach Structure and Contribution of the Thesis Related Work Distributed Snapshot Algorithm DDoS Defense Mechanisms Distributed Snapshot Traceback Algorithm Overview and Problem Definition Overview Problem definition Traceback methodology How to perform the traceback Difficulties of a distributed traffic measurement vi

9 2.2 Distributed Algorithm Reasons for incorrect traceback result Measuring the correct local traffic The distributed snapshot algorithm Pseudocode and execution of snapshot algorithm Example in calculating the traceback result Interpreting the Traceback Result Investigation of the traffic inequality Calculating bounds for the number of packets arrived at the victim site Performance Evaluations Implementation Issues Topology construction System overhead Implementation issue based on ICMP traceback An alternative to aggregate congestion control and pushback Special deployment - acyclic network Partial deployment Chapter Summary Probabilistic Packet Marking Algorithm Structure of This Chapter Goal and Structure of the PPM Algorithm Global network and attack graph Constructed graph Structure of the PPM algorithm Assumptions Marked packets and PPM markings vii

10 3.3.2 Router Packet marking probability Attack source and attack pattern Attack graph and packet routing Graph Reconstruction Example Packet marking Attack graph reconstruction Chapter Summary Termination Condition of PPM Algorithm Using the Upper-Bound Packet Number as the Termination Condition Failure under the multiple-attacker environment Simulation findings Chapter structure Packet-Type Model Packet-Type probability Pseudocode of the calculation of the packet-type probabilities Illustration of the calculation of the packet-type probability Using Markov Chain Model to Find the Sufficient Packet Number The Markov process Example on discrete-time Markov chain modeling Fundamental matrix Example on calculating E[X] Disproving the Upper-Bound Packet Number as the Termination Condition Chapter Summary viii

11 5 Rectified Probabilistic Packet Marking Algorithm Structure of This Chapter Overview of the RPPM Algorithm Working principle Flow of rectified graph reconstruction procedure Execution Diagram of the RPPM algorithm Types of states Types of transitions Worst-case, average-case, and best-case scenarios Role of the execution diagram Derivation of Termination Packet Number Technique State-change probability TPN derivation Section summary and TPN calculation subroutine Graph Reconstruction Example State C State C State C Simulation Result Simulation environment Simulation: different values of the marking probability Simulation: different graph structures Section summary Supporting Routers with Multiple Victim Routes Problem of multiple victim routes Formulating an extra set of extended graphs Reformulation of packet-type probability Simulation: support for multiple victim routes ix

12 5.7.5 Section summary Deployment Issues of the RPPM Algorithm Choice of the marking probability Execution time comparison between the PPM and the RPPM algorithms Scalability issue in PPM algorithm Precision problem Chapter Summary Bibliography 176 x

13 List of Figures 1.1 The architecture of a typical flood-based DDoS attack The architecture of a reflector attack The overview of the divide-and-conquer traceback approach An example network topology Asynchronous reading of outgoing traffic counters in Example B Correct accumulative local traffic without clock synchronization An example execution of the snapshot algorithm B ji = 0 under all circumstances A ji is the channel state A network topology with two attackers who reside in the local domains of R 3 and R A timing diagram that shows the progress of the distributed snapshot traceback algorithm Classification of pre-monitoring, monitoring and post-monitoring packets The channel state of Link 3,2 contains pre-monitoring (monitoring) packets from both R 3 and R 4 in the first (second) instance of the snapshot algorithm (a) Network topology and (b) Legend for Simulations A and B Simulation A.1. Bounds for the real local traffic under constant traffic rate xi

14 2.13 Simulation A.2. The real local traffic under exponential on/off process Simulation A.3. Effect of multiple attackers on the real local traffic bounds Simulation A.4. Effect of new attackers locations Simulation A.5. On different attack traffic rates Simulation B. Simulation for large scale Internet Topology (a) An acyclic network with one attacker who resides in the local domain of R 3. (b) R 3 maintains two accumulative outgoing traffic counters C 3,1 (t) and C 3,2 (t) for the links Link 3,1 and Link 3,2, respectively Another timing diagram that shows the progress of the distributed snapshot traceback algorithm (a) The same example network as Figure 2.7 with attacking domains R 3 and R 4. But, the router R 3 is an undeployed router. (b) Logically, a virtual link between the router R 2 and R 4 is formed A timing diagram that shows the progress of the DDoS traceback algorithm under the partial deployment environment (a) In this example network, the router R 2 is an undeployed routers while the others are deployed routers. (b) As the undeployed router is transparent to the traceback protocol, the router R 1 records the channel state of the virtual links Link 3,1 and Link 4, The timing diagram under a partial deployment environment. A drawback is that the channel states of the virtual links Link 3,1 and Link 4,1 become indistinguishable at router R A typical case of a DDoS attack toward the victim V xii

15 3.2 The illustration of an attack graph: (a) an attack graph is not the entire network; the attack graph is the paths traversed by attack packets; (b) the attack graph may become larger than the actual one due to the lack of legitimacy of the packets The pseudocode of the packet marking procedure of the PPM algorithm The pseudocode of the path reconstruction procedure of the PPM algorithm The failure of the router R 1 causes the route tables of R 2, R 3, and R 4 to change. This results in a constructed graph with routers having multiple outgoing edges A step-by-step illustration of the reconstruction of the attack graph based on the incoming packet sequence in Table A six-router binary tree network: the upper-bound equation cannot be applied under this multiple-attacker environment A eight-router tree network with four independent linear paths: another multiple-attacker environment Simulation result: Number of marked packets required versus number of independent paths An increasing yet chaotic trend of the rate of change of the number of marked packets required The pseudocode of the packet-type probability calculation it calculates the packet-type probability of every edge in the graph G (a) G a : A simple example linear network with three edges. (b) G b : An example network with multiple paths leading from R 3 and R 4 to the victim xiii

16 4.7 Example network G 1 : it is a linear network with three routers and one victim Illustration of the Markov chain model of the PPM algorithm with network G 1 in Figure The constructed transition probability matrix formulated of the Markov chain shown in Figure Simulation result versus theoretical result: for network G 1 in Figure 4.7, we obtain two close sets of results for the distribution of the sufficient packet number X Example network G 2 : totally 16,384 Markov states Probability distribution of the sufficient packet number on the 14-router binary-tree network, G Fundamental matrix calculated by Equation (4.13) with transition probability matrix P shown in Figure The comparison between the simulation and the theoretical results: both results disprove the linear property proposed by previous work The design goal of the RPPM algorithm: to have a correct constructed graph with probability greater than P The pseudocode of the rectified graph reconstruction procedure of the RPPM algorithm An execution diagram of the rectified graph reconstruction procedure of the RPPM algorithm constructing a graph with n edges Extended graph example: a constructed graph and its set of extended graphs The pseudocode of the termination packet number (TPN) calculation subroutine xiv

17 5.6 State C 1 : a constructed graph with one edge, and its extended graphs State C 2 : a constructed graph with two edge, and its extended graphs The simulations show that the larger the marking probability is, the closer to the worst-case execution the simulation result is RPPM algorithm simulation: 15-node linear network with random marking probability RPPM algorithm simulation: 14-router binary-tree network with random marking probability RPPM algorithm simulation: 14-router random-tree network with random marking probability RPPM algorithm simulation: 100-router random-tree network with marking probability = RPPM algorithm simulation: 500-router random-tree network with marking probability = RPPM algorithm simulation: 1,000-router random-tree network with marking probability = When the routers have more than one victim route, the RPPM algorithm cannot guarantee the correctness of the constructed graph when the confidence level is larger than An illustration of the extended graph with the support of multiple victim routes The pseudocode of the packet-type probability calculation subroutine which supports multiple victim routes With the support for multiple victim routes, the RPPM algorithm can provide the guarantee of the correctness of the constructed graph xv

18 5.19 Average number of marked packets required for a correct graph reconstruction against different values of the marking probability Average number of total packets (marked packets plus unmarked packets) required for a correct graph reconstruction against different values of the marking probability The number of marked packets recorded for the set of RPPM algorithm simulations carried out on a random-tree networks with 14 routers The number of marked packets recorded for the set of RPPM algorithm simulations carried out on random-tree networks with 50 routers and 100 routers The number of marked packets recorded for the set of RPPM algorithm simulations carried out on random-tree networks with 500 routers and 1000 routers The percentage of number of marked mpackets increased when comparing the RPPM algorithm to the PPM algorithm with different network scales Scalability analysis: average number of marked packets collected by the PPM algorithm versus the size of the attack graph The pseudocode of repeating the RPPM algorithm to increase the runtime probability xvi

19 List of Tables 2.1 Computation of the accumulative local traffic in Example A by using Equation (2.1) Computation of L i (t 1, t 2 ): the local traffic within [t 1, t 2 ] in Example A by using Equation (2.2) Computation of accumulative local traffic at time t i,1 and t i, The local traffic intensity counts only the packets in between the two instances of the snapshot algorithm A sequence of packets collected by the victim Packet-type probabilities for G a in Figure Packet-type probabilities for G b in Figure 4.6 : after the path (R 3, R 2, R 1, v) of G b is considered Packet-type probabilities for G b in Figure 4.6: after both paths (R 3, R 2, R 1, v) and (R 4, R 1, v) of G b are considered The marked packet-type probabilities of the extended graph G 1,1 and G 1, The marked packet-type probabilities of the extended graphs G 2,1,G 2,2, and G 2, The average number of packets and time required to form a correct constructed graph in a 100BaseT Ethernet xvii

20 Chapter 1 Defense Against Denial-of-Service Attack If you know your enemies and know yourself, you will win hundred times in hundred battles. The Art of War, Sun Tzu. The emergence of the Internet as a pervasive form of communication has led to the recent enormous deployment of E-business and information distribution services. However, the success of the Internet also attracts malicious attackers who abuse system resources and expose the inherent security problems of the Internet. Distributed denial-of-service (DDoS) attack is one of the most pressing problems on the Internet. Well-known commercial sites such as Yahoo!, Amazon, and ebay were attacked and were out of service for many hours due to a series of DDoS attacks on February 2000[1, 2]. Since then, DDoS attacks have increased in size, frequency, sophistication, and severity. In this chapter, we are going to understand what a distributed denial-of- -service attack is. We dissect the methodologies of common DDoS attacks in Section 1.1. After we are familiar with the nature of the DDoS attacks, we define the scope of this thesis in Section 1.2: to trace the location of the attackers of a DDoS attack. In the same section, we suggest our approach against DDoS attacks in a world-wide scale, and we name it the divide-andconquer traceback approach. In Section 1.4, we introduce previous work that is 1

21 Chapter 1 Defense Against Denial-of-Service Attack 2 related to this thesis. Roughly speaking, this covers the methodologies that will be introduced in later chapters, including the distributed snapshot algorithm, the packet filtering technique, and the IP traceback technique. 1.1 Overview of Attack Methodology The goal of the DDoS attacks is to degrade or even disable the service(s) provided by the target. For the example attack case in [1], the targeted services are the web services provided by Yahoo!, CNN, and Amazon. We classify a DDoS attack in terms of the attack methodology. A denial-of-service attack can be realized in either two techniques: 1. exploiting the vulnerability in network protocols and software; and 2. leveraging high volume of address-spoofing, bogus traffic. We name the former type of attack the vulnerability-based attack and the latter type of attack the flood-based attack. These two kinds of attacks are usually mixed together in order to bring about a large amount of damage. Note that an attacker always wants to disguise himself or herself as a set of legitimate users. There is a loophole in the TCP protocol that no components, devices, or authorities on the Internet can check the identity of any packets sent. Say the attacker is sending a packet from a machine with address A, he or she can easily change the source address of the packet to address B without anyone noticing. We name this kind of packets spoofed packets since the source address is spoofed. The advantage of the attacker sending spoofed packets is to keep his or her location secret. Then, the DDoS countermeasures would not target him or her so easily. Though this exploits the vulnerability of the TCP protocol, we choose not to classify attacks using spoofed packets as vulnerability-based

22 Chapter 1 Defense Against Denial-of-Service Attack 3 attacks because every attack uses spoofed packets. Henceforth, throughout the text, we always assume that every attacker sends spoofed packets Vulnerability-based attack In the following sections, we introduce two severe kinds of vulnerability-based attacks. This kind of attack leverages the flaws in protocol designs and the defects in software. Once such vulnerabilities are exploited, the service provided by the victim will be shut or degraded. TCP-SYN flood attack The TCP-SYN flood attack[3] (or SYN attack) is an infamous vulnerabilitybased attack. Though the attack carries the word flood, what the attack does is to exploit the vulnerability in the implementation of the SYN packet handling of the TCP/IP protocol. In a nutshell, this attack targets the three-way handshake protocol of the TCP protocol [4]. The attack brings down a host by flooding the host with enough spoofed SYN packets so that these spoofed SYN packets occupy all the available connections of the hosts. Eventually, there are no more resources left for further connections. The countermeasure of this threat is the SYN cookies introduced in [5]. Nowadays, most operating systems already have SYN cookies implemented (inside the operating system s kernel). Low-rate TCP attack In [6], the authors proposed and realized a new form of attack that targets the congestion control mechanism of the TCP protocol. The attacker carefully orchestrates the periodic attack packets to exploit the fixed minimum TCP retransmission timeout so as to shut off most, if not all, legitimate TCP flows. Though there is are incident reports on the low-rate TCP attack, there are

23 Chapter 1 Defense Against Denial-of-Service Attack 4 already solutions [7, 8, 9] proposed in the literature Flood-based attack The flood-based attack aims to disable a victim host by leveraging a high volume of spoofed traffic. Once this type of DDoS attack is launched, the victim will experience increasing load. The service will usually be impacted significantly, and there are cases when the victims have broken down. To realize such an attack, computers in an order of tens of thousands are needed in order to generate a significantly large burden on the victim. The attacker in reality cannot own such a scale of resource but steals them. The attacker usually obtains computing resources by compromising a large number of computers in order to launch a large-scale flood-based attack. This can be realized by exploiting known vulnerabilities in widespread operating systems such as Microsoft Windows. When such a exploitation is done, the attacker usually gains the highest privilege of the compromised computer and can perform whatever acts he or she likes. We called those compromised computers zombies [10, 11]. Although the attack involves the technique of exploiting vulnerabilities, the technique is not the payload of the DDoS attack, and such an exploitation neither brings the zombies down nor degrades the computing performance of the zombies after all. Zombie attack Once a large group of zombies has been gathered, the attacker loads attack programs to the zombies. The zombies are then turned into unwitting attackers, and the DDoS attack is then launched. Figure 1.1 shows the deployment scenario and the entities involved in a DDoS attack using zombies [12]. The attacker seated in front of his or her own computer controls a set of handlers that are, again, obtained by exploiting vulnerabilities. These handlers are used

24 Chapter 1 Defense Against Denial-of-Service Attack 5 A Attacker H H Handlers Z Z Z Z Zombies ν Victim Figure 1.1: The architecture of a typical flood-based DDoS attack. to control the zombies so that the attacker can become stealthy during the attack. The zombies are the ones that are sending spoofed traffic to the victim. There are occasions that, when an outbreak happens, the Internet becomes paralyzed because this kind of attack usually targets on widespread software. It is always difficult to hunt down the attacker of a zombie attack. The attacker always protects the communication between the handlers and the zombies by encrypting the communication channels [12]. What we can do is to ask the Internet service providers (ISPs) to help locate and filter the attack traffic so as to ease the pain of the victim. Also, replacing legacy and buggy software is a crucial step to reduce the number of handlers and zombies that can be obtained by attackers. Moreover, intrusion detection systems (IDS) [13, 14] should always be installed in order to detect and stop intrusions by attackers promptly and effectively. Reflector attack There is another kind of automated attack using a similar architecture called the reflector attack [15]. As shown in Figure 1.2, the main feature of this attack is that the zombies are not attacking the victim directly but through a set of

25 Chapter 1 Defense Against Denial-of-Service Attack 6 Z Z Z Z Zombies R R R R Reflectors ν Victim Figure 1.2: The architecture of a reflector attack. reflectors. The zombies send spoofed packets with the source addresses set to the victim s address and the destination addresses set to the reflectors addresses. The reflectors are usually some public servers, such as domain name servers, and the content of the spoofed packets is usually a request for service from the reflectors. The reflectors will then generate replies without knowing that the requests are frauds. As a result, the reflectors send the replies to the victim as the source addresses of the requests are set to the victim s address. The reflector attack is, therefore, by its nature, more detrimental than using the zombie attack model alone because: 1. it amplifies the effect of the DDoS attack. Let us imagine that the attacker has only one zombie. By sending spoofed packets to different reflectors, one zombie is already enough to attack the victim in a distributed way; 2. it also degrades the services provided by the reflectors. During the reflector attack, the reflectors are loaded by the requests from the zombies, and this degrades the services provided by the reflectors; and

26 Chapter 1 Defense Against Denial-of-Service Attack 7 3. it is more difficult to be traced. Since the reflecting flow is coming from innocent hosts (given that the reflectors are not compromised), the tracing can be done readily, but to find that they are just reflectors. Peer-to-peer attack This is an emerging type of attack mechanism. The peer-to-peer (P2P for short) DDoS attack does not attack the P2P file sharing network but makes use of the P2P network to launch a DDoS attack [16]. A P2P file transfer network usually has ten of thousands of clients joining it. One type of the P2P attack is to poison the file records shared by the clients. The attack writes a bogus entry saying that a certain location, which is the victim, is providing a certain set of files (and usually the victim is not a member of P2P network). When innocent clients follow the bogus entry for a file sharing service, it will end up in an error. But, the victim is bombarded with tens of thousands irrelevant file requests. Automatic attack tools Several well-known DDoS attack tools adopt the above attack architectures. These tools are designed to be versatile so that they can mount different types of attack payloads to the zombies. Several famous tools include the Tribe Flood Network 2000 (TFN2K for short) [17], the Trinoo [18], and the Stacheldraht [19]. These automatic attack tools are well designed and are effective in launching DDoS attacks Worm attack The worm attack is another form of automatic attack tool. To define, a worm is a piece of software that runs on a computer, and the computer is unwillingly having the worm running. The worm has the ability to duplicate itself, and

27 Chapter 1 Defense Against Denial-of-Service Attack 8 has the duplicated copies infect other computers. From a functional point of view, a worm infects a computer by exploiting vulnerabilities of the software used on the target computer. A worm also has its payload: some payloads just infect other computers, some payloads harm the hosting computer, or some payloads attack target sites in a cooperative manner. Code red The Code Red [20] is a famous worm that roamed the Internet during the summer of The worm exploits a vulnerability in the Microsoft IIS server which, is widely deployed around the globe (around 20% of market share by 2001 [21]). The payload of this worm was twofold: first the worm tried to infect as many IIS servers as possible, and then all the worms were coordinated to launch a DDoS attack toward several victims such as the web server of the U.S. White House. In response, Microsoft announced the vulnerabilities with the corresponding software patches provided. The attack ceased when the vulnerabilities were fixed, and at the same time, the ISPs filtered the payload of the worm. Slammer Again, Microsoft was the target of another famous worm attack. The worm named Slammer demonstrated a severe attack on the Internet in 2003 [22] by using a vulnerability of Microsoft SQL server. The Slammer is actually an interesting worm attack incident. The only payload it carried was to propagate itself with a blitz tactic. Once the worm affected a vulnerable Microsoft SQL server, the immediately probed the network for other vulnerable Microsoft SQL servers by rapidly firing malicious traffic with random IP addresses. The malicious traffic brought down many routers and then initiated a wave of routing table updates. When the failed routers were fixed and were online again, the worm started another wave of routing table updates. The bombardment

28 Chapter 1 Defense Against Denial-of-Service Attack 9 of the malicious traffic, the failures of routers, and the changes of the routing tables together shut the Internet down partially. This was, as a matter of fact, a DDoS attack that targeted the Internet infrastructure Flash crowd Despite the mentioned explicit attacks, there are scenarios in which the services provided by the victim are degraded because of legitimate traffic. The flash crowd happens when many users simultaneously send requests to one Web site, usually because of special events attracting the interest of the mass population. These events could be scheduled ones such as broadcasts of World Cup matches, unpredictable events such as earthquakes, or links from popular Web sites (see [23] for details). In our context, the flash crowd is certainly not a DDoS attack. Nevertheless, the flash crowd behaves similarly to a DDoS attack. The victim and the network itself can be overloaded by a flash crowd event, and the aggregated volume of the legitimate traffic is comparable to a DDoS attack. In the literature, publications have mentioned this problem and suggested solutions have been provided [24, 25]. To conclude, the DDoS attack may take different attack forms, strategies, and patterns. Interested readers can refer to survey articles [26, 27] for more details. 1.2 Scope of the Thesis In this thesis, we target the flood-based attack, and we aim to stop such an attack when one can detect it. According to industrial practices against DDoS attacks [28], one should do the following steps in response to a DDoS attack:

29 Chapter 1 Defense Against Denial-of-Service Attack Preparation. Service providers have a high chance of successful defense against a DDoS attack if they have laid the groundwork against it. 2. Detect. The ability to quickly identify an attack is critical to minimizing the damage that the attack can cause. 3. Traceback. Once a service provider has detected an attack, the next step is to traceback trying to determine the source of the attack so that the service provider can apply mitigation techniques, or, if the source of the attack is from another network, inform the corresponding peer. 4. Containment. When an organization knows where an attack is coming from, the organization should apply containment and filtering mechanisms to stop the malicious traffic. 5. Postmortem. After a security incident, it is important for the organization to review what was most effective during an attack and what could be improved. The target of this thesis is to trace back: to locate the sources of the attack flows that are contributing to the DDoS attack. In the following section, we present some general assumptions General assumptions We aim to locate the sources of the attack flows. Hence, if the attacker(s) are using the attacking architecture mentioned in Section 1.1.2, we are concerned only with the locations of the zombies or the locations of the reflectors in the reflector attack. We assume that the victim has the ability to detect that the providing service is being degraded by overwhelming traffic. We also assume that the victim is allowed to report the incident to the victim s ISP, and the ISP will then handle the incident.

30 Chapter 1 Defense Against Denial-of-Service Attack 11 We are not interested in discriminating between a legitimate flow and an attack flow. We are also not interested in distinguishing between a flash crowd or a DDoS attack. What we are concerned with is identifying flows that degrade the service provided by the victim. Last but not least, since we are concerned with the flood-based attack only, we are not going provide solutions to remedy vulnerability-based attack such as the low-rate TCP attack A divide-and-conquer traceback approach As DDoS attacks are becoming more violent and the attack scale is enlarging, tracking down attackers across the globe is becoming more difficult and more tedious. To provide relief from such a adversary reality, we propose a divide-and-conquer approach so that the global-scale traceback problem can be divided into tractable sub-problems. Overview From a technical point of view, in the case of launching a global-scale attack, attack sources are spread across different Internet service providers (ISPs for short), and these sources send attack traffic toward the ISP where the victim resides. As shown in Figure 1.3, attackers located in ISPs C, D, and E send traffic toward ISP A, where the victim resides. We propose that the ISPs should be coordinated, and together discover the ISPs that are contributing overwhelming traffic, and we call this problem the macroscopic traceback problem. After the problematic ISPs have been identified, in the next step, each ISP should trace the location of attackers within its administrative domain, and we call this problem the microscopic traceback problem. Specifically, a macroscopic traceback algorithm should be deployed within

31 Chapter 1 Defense Against Denial-of-Service Attack 12 ν E A D B C Intra-ISP network Inter-ISP network R ISP border router ISP backbone router Attacker ν Victim site Macro-traceback processing node Micro-traceback processing node Figure 1.3: The overview of the divide-and-conquer traceback approach.

32 Chapter 1 Defense Against Denial-of-Service Attack 13 the inter-isp network. Referring to Figure 1.3, the border routers and the coupling links between the border routers together form the inter-isp network. To facilitate the deployment of the macroscopic traceback algorithm, every border router is connected to a macro-traceback processing node, which executes the macroscopic traceback algorithm. On the other hand, a microscopic traceback algorithm should be deployed within the intra-isp network, and the intra-isp network is constructed by a network of backbone routers of an ISP. Again, a processing node, namely the micro-traceback processing node, is added to help trace the attackers within the network inside an ISP. An example divide-and-conquer traceback execution In addition to the architecture of the divide-and-conquer traceback approach, Figure 1.3 also sets up an attack scenario. In the figure, we have five attacking sources with the distribution that ISP C contain three attackers while both ISPs D and E contains one. In the beginning, at the moment that a DDoS attack is detected, the victim, which resides in ISP A, calls for the DDoS defense service from its ISP. In turn, the border router of ISP A diverts the traffic sent toward the victim to the macro-traceback processing node, and the macro-traceback processing node initiates a macroscopic traceback algorithm. The macro-traceback processing nodes of the remaining ISPs join the algorithm accordingly. The traceback result of the macroscopic traceback algorithm should discover that ISPs C, D, and E contain the sources of the attack. Next, ISP A would inform ISPs C, D, and E about the traceback result. In response, each border router of the concerned ISPs diverts all the outgoing traffic sent toward the victim to the micro-traceback processing node. Each processing node, running the microscopic traceback algorithm, aims to locate the attack sources, which are sending traffic toward it. Once the traceback result is ready, the concerned ISP can discover the locations of the attackers,

33 Chapter 1 Defense Against Denial-of-Service Attack 14 and follow-up actions, such as packet filtering, will then be carried out. Justification First of all, it will be attractive to the ISPs if the traceback algorithms are deployed only within their administrative domains. To justify, from the ISPs points of view, they do not want to disclose any information about their networks. The reason is simple: their peers are actually competitors, not partners. Thus, any algorithms that execute across multiple ISPs have difficulties in deployment, and this is the reason for confining the microscopic traceback algorithm within the intra-isp network. On the other hand, the divide-and-conquer approach not only narrows down the traceback scope by using the macroscopic traceback algorithm but also speeds up the traceback process by having multiple execution instances of the microscopic traceback algorithm concurrently at different ISPs. We believe that there is no silver bullet that can handle every kind of floodbased DDoS attack. We believe that one should use the right tool against the right problem, the right model, and the right scenario. Therefore, in this thesis, we choose to investigate the DDoS attack defense mechanism from two different angles. 1.3 Structure and Contribution of the Thesis In Chapter 2, we devise a macroscopic traceback algorithm. Leveraging the well-known Chandy-Lamport s distributed snapshot algorithm, we propose a distributed algorithm that can correctly collect statistics (in a distributed sense) from programmable routers in a coordinated fashion [29]. Then, by analyzing the collected data, a victim can deduce the intensity of the traffic generated by the network that is attached to every participating router. The contribution of the algorithm is twofold. Firstly, this is the first piece of work

34 Chapter 1 Defense Against Denial-of-Service Attack 15 that applies a classical distributed algorithm in a DDoS attack defense mechanism effectively. Second, this work also provides a theoretical foundation to measure Internet traffic in a distributed sense. In Chapter 3, we analyze a promising microscopic traceback algorithm. The probabilistic packet marking algorithm (PPM algorithm for short) by Savage et al. [30] is an effective way to locate attackers using flood-based DDoS attacks. In this chapter, we present an overview of the PPM algorithm. Yet, the PPM algorithm is not perfect as its termination condition is not welldefined in the literature. More importantly, it is found that, without a proper termination condition, the attack graph constructed by the PPM algorithm would be wrong. In Chapter 4, we study the termination condition of the PPM algorithm. This is the first piece of work in the literature that studies the termination condition of the PPM algorithm [31]. We present a discretetime Markov chain model that provides a precise calculation for the termination condition for the PPM algorithm. Nevertheless, the mechanism requires knowledge of the attack graph in advance. This contradicts the purpose of the traceback algorithm, which is designed to find the attack graph. This leads to the surrender of the current termination condition of the PPM algorithm To improve the termination condition of the PPM algorithm, we present a new algorithm, the rectified probabilistic packet marking algorithm (RPPM algorithm for short) in Chapter 5 [32]. The most significant merit of the RPPM algorithm is that when the algorithm terminates, the algorithm guarantees the correctness of the traceback result with a specified level of confidence. Our findings show that the RPPM algorithm can guarantee such a correctness under different deployment scenarios. As one of the major contributions of this thesis, the RPPM algorithm provides an autonomous way, which is missing in the original PPM algorithm, to determine its termination, and it is a promising means to enhance the reliability of the PPM algorithm.

35 Chapter 1 Defense Against Denial-of-Service Attack Related Work The macroscopic snapshot algorithm that will be introduce in Chapter 2 leverages the well-known Chandy-Lamport distributed snapshot algorithm. In this section, we first introduce the importance of this distributed snapshot algorithm. Then, we introduce the development of the techniques against DDoS attacks, mainly the packet filtering technique and the IP traceback technique Distributed Snapshot Algorithm The very first distributed snapshot algorithm was proposed by Dijkstra and Scholten [33]. Later, Chandy and Lamport proposed the consistent global snapshot algorithm in [34], and the algorithm is derived from Lamport s earlier work on logical time [35]. Fischer et. al. designed another algorithm for consistent global snapshots, and this algorithm is tailored for transaction-based systems [36]. The distributed snapshot algorithm has been applied in capturing consistent global state of a distributed system. The primary use of the snapshot algorithm is in checkpointing and rollback recovery [37]. The checkpointing and recovery are vital properties that allows systems to make progress in the presence of failures. In brief, checkpointing [38] is a technique to save the states of an executing process. Processes achieve fault tolerance by saving recovery information periodically during failed-free executions. Upon a failure, a failed process uses the saved information to restart the computation from an intermediate state, thereby reducing the amount of lost computation. The recovery information includes the states of the participating processes, called checkpoints. In a distributed system, a global checkpointing scheme requires a coordinated checkpointing of the participating processes. The Chandy-Lamport distributed snapshot algorithm provides a proofed consistent global state with