Network Intrusion Prevention Systems: Signature-Based and Anomaly Detection

Transcription

1 Network Intrusion Prevention Systems: Signature-Based and Anomaly Detection Thesis for the degree of DOCTOR of PHILOSOPHY by Shimrit Tzur-David SUBMITTED TO THE SENATE OF THE HEBREW UNIVERSITY OF JERUSALEM April 2011

2 i This work was carried out under the supervision of: Prof. Danny Dolev and Dr. Tal Anker

3 ii Acknowledgements First and foremost, I owe my deepest gratitude to my advisor Prof. Danny Dolev. It is really difficult to express my gratitude to Danny. I thank him for being a true role model and mentor and for his never ending support. His endless patience and didactic attitude, combined with his wisdom and brilliancy, make him a remarkable guide. I would like to thank him for his permanently-opendoor policy, and for giving real meaning to the title of an academic father. I thank him from the bottom of my heart. It has truly been an honor. I would also like to thank my co-advisor Dr. Tal Anker for always being willing to help, for his support, his original ideas and valuable advices. This thesis would not have been possible without him. Furthermore, I would like to thank all the members of the Distributed Algorithms, Networking and Secure Systems Group (DANSS). Specifically, I would like to thank my extraordinarily talented friend and partner Yaron Weinsberg for his support and contribution to my research. I would also like to thank Harel Avissar and Kiril Lashchiver for their valuable contributions. Many thanks to my parents-in-law for their patience through these years, for their love and support. My warmest and deepest thanks go to my parents for a lifetime of love and support. There are no words to express how indebted and grateful I am to both of you. Thank you. Last but not least, I am very grateful to my beloved husband Danny for the loving support and never ending acceptance. And finally to my best results, my three wonderful sons: Moran, Maayan and Yali. Although you only recently joined the team, you are a powerful source of inspiration and energy, your smiles make all the different.

4 iii Abstract Intrusion Detection systems (IDS) were developed to identify and report attacks in the late 1990s, as hacker attacks and network worms began to affect the internet. Traditional IDS technologies detect hostile traffic and send alerts but do nothing to stop the attacks. Network Intrusion Prevention Systems (NIPS) are deployed in-line with the network segment being protected. As the traffic passes through the NIPS, it is inspected for the presence of an attack, when an attack is identified, the NIPS blocks the offending data. Attacks can be divided into two categories. One is attacks that have some sort of signatures. The other category includes zero-day attacks which are new and previously unknown to security protectors. In order to detect and prevent attacks of the first category, the NIPS should include a patternmatching algorithm. There is an alleged trade-off between the accuracy of detection and algorithmic efficiency. Both are paramount in ensuring that legitimate traffic is not delayed or disrupted as it flows through the device. For this reason, the pattern-matching algorithm must be able to operate at line-speed, while simultaneously detecting the intrusions. With networking speeds doubling every year, it is becoming increasingly difficult for software based solutions to keep up with the network rates. This dissertation presents two novel pattern-matching algorithms (for the first category of attacks). The first algorithm uses a Ternary Content Addressable Memory (TCAM) and is capable of matching multiple patterns in a single operation. Since a TCAM-based solution might be too expensive for some potential users, the other algorithm is cost effective and it is based on Field Programmable Gate Arrays (FPGAs) and standard RAM. Both solutions achieve line-rate speed of several orders of magnitude faster than similar works, without compromising accuracy of detection. Furthermore, our systems are fully compatible with Snort s rules syntax, which is the de facto standard for intrusion prevention systems. In order to detect attacks of the other category, the network must deploy anomaly-based detection engine. Anomaly-based engine protects the network from new and unknown threats before the vulnerability is discovered and an attack is launched. Thus, the engine should integrate intelligent

5 iv learning capabilities. The principal way for achieving this goal is to model anticipated network traffic behavior, and to use this model for identifying anomalies. The main challenges include accuracy and minimization of the false-positive rate and memory consumption. In this dissertation we present four anomaly-based engines. The first two, the MULAN-filter and SPADE are designed to protect the network against DoS and DDoS attacks. Both MULAN (MUlti-Level Adaptive Network) and SPADE (a Statistical Packet Acceptance Defense Engine) are accurate engines that use multi-level adaptive structure for specifically detecting suspicious traffic. They both use a relatively small memory footprint and therefore can be easily applied on hardware. MULAN creates a profile of normal network behavior and detects anomalies of that profile. On the other hand, SPADE starts to operate only when the network is under a risk of crashing. SPADE is based on the assumption that during DoS/DDoS attacks, a significant portion of the traffic that is seen belongs to the attack, therefore, SPADE applies a statistical mechanism to primarily filter the attack s traffic. NUMA (NIPS for URL Manipulation Attacks) is our third engine and it is designed to protect web servers. Web servers have become a major target for attacks. Web server attacks and website defacements increase significantly every year. Prevention of web-related attacks is currently based on identifying the signature of the attack in a dictionary of predefined signatures of known attacks. Thus, prevention of network attacks requires a high-speed pattern matching algorithm. Besides the drawback of not detecting zero-day attacks, it is hard to keep intrusion prevention signature sets constantly updated with the large numbers of vulnerabilities discovered on a daily basis. Therefore, a web related NIPS should be enhanced with a system that protects against zeroday attacks. NUMA provides protection against web-related attacks that are generated through URL manipulations. The key challenges in designing NUMA include addressing high-speed system with low memory consumption that generates low false-positive. In order to target these three challenges, NUMA uses a dynamic data structure that can be deployed in hardware. This dynamic data structure enables NUMA to track and filter suspicious traffic. The great accuracy is achieved by the ability to analyze all the incoming traffic without the need to sample the data.

6 v The last engine protects the Domain Name System protocol, or shortly - DNS. DNS is a protocol that is used for a naming system for computers, services, or any other network resources. The most important purpose of the protocol is to translate names, which are meaningful to humans, into IP addresses, which are meaningful to computers, thus allowing us to easily locate every computer or service around the globe. DNS protocol was not designed to deal with security issues, therefore many hacking methods were developed in order to insert wrong data into the DNS lookup tables, or in its cache. In cache poisoning attack, the attacker inserts wrong data into the DNS cache. The main protection against those attacks in the DNS protocol is the transaction ID (TID) of the request and the source port of the originator. Unless the reply to the source port is returned with the same TID as the original request, it is ignored. Therefore, in order to poison the cache of the DNS server, the hacker has to know the source port and guess the right TID for the request. As far as we know, all the solutions today that does not change the protocol, try to increase entropy to make this guessing harder. In this paper we assume that the hacker is an all knowing person. Our solution does not change the DNS protocol and moreover, does not change the behavior of the name servers. The main idea is to search for anomalies in the times in which the name servers get the response. This dissertation covers a major part of today s network attacks. All of the engines are based on algorithms that achieve line-speed rate. In order to achieve this rate, they either use a special hardware or analyze only suspicious data. In addition, they are all accurate with a negligible rate of false-positives.

7 Contents Contents vi 1 Introduction Signature-Based NIPS Anomaly-Based NIPS Internet Threats Research Objectives Dissertation Outline Signature-based NIPS Related Work Software Based Pattern Matching Hardware Based Pattern Matching Problem Definition General Definitions Snort Specific Notations ClamAV Anti Virus RTCAM Populating the TCAM High-Level Runtime Operation Data Structures vi

8 CONTENTS vii CONTENTS 3.4 A Packet Flow within the NIPS Patterns with Negations Long Patterns Optimization False Positives Worst Case Considerations Window Size Considerations Effects on the Shift Value Possible Attack Experimental Results Results on ClamAV Pattern Set Results on Snort Pattern Set Scanning Time Results Porting RTCAM to SRAM Motivation FPGA Algorithm High-Level Runtime Operation Data Structures A Packet Flow within the NIPS Dealing with Short Patterns Experimental Results Anomaly-based NIPS Bandwidth Attacks URL Manipulation Attacks DNS Cache Poisoning Attacks Related Work MULAN Known DoS Attacks

9 CONTENTS viii CONTENTS 6.2 Notations and Definitions The MULAN-filter Design Anticipated Traffic Behavior Profile Data Structure The Algorithm Training Mode Verification Mode The Algorithm Parameters Modeling Non-Self-Similar Traffic Handling Short Attacks Optimal Implementation Experimental Results Scalability Accuracy Controlling the Tree Size SPADE SPADE Design Data Structures Predicting the Rate of Next Sample The Algorithm Optimal Implementation Experimental Results Scalability Accuracy NUMA URL Manipulation Attacks NUMA Design Overview

10 CONTENTS ix CONTENTS High-Level Runtime Operation Scores and Threshold Computation The anomalous characteristics Optimal Implementation HTTP Traffic Characteristics Bloom Filter Implementation TCAM-based Implementation Throughput Experimental Results DFP Cache Poisoning The DFP Algorithm Experimental Results The Window α and β Considerations FactorWindow Considerations Slow Packets Consideration Memory Consumptions Attacks Detection Conclusions 100 Bibliography 101 Appendix 113

11 CHAPTER 1. INTRODUCTION 1 1. Introduction Today s businesses rely increasingly on corporate IT networks and their connection with the global Internet. However, the convenience of global connectivity comes at a cost - the vulnerability of network and systems to the malicious actions of cyber criminals. Symantec s survey [21] from January 2010 examines 2100 businesses and government agencies from 27 countries. Astoundingly, every one of those businesses had suffered a cyber loss in the preceding 12 months, which could have been anything from theft of customer credit card or financial data to theft of intellectual property. 92% of the respondents claim that the cyber theft led to significant costs due to loss of customer trust, falling employee productivity and decreased revenues. According to a MIS director at a mid-size enterprise, new viruses, spyware and backdoors are seen every day. 42% of companies ranked cyber security as a top priority, ranking it over traditional criminal activity (17%), brand-related issues (17%), natural disasters (14%) and terrorism (10%). Another security survey [22] has revealed that the top eight threats were viruses (78% of respondents), system penetration (50%), denial of service (40%), insider abuse (29%), spoofing (28%), laptop theft (22%), data/network sabotage (20%) and unauthorized insider access (16%). Although viruses were the most significant threats, 66% of the companies said that they perceive system penetration to be the largest threat. The survey also revealed that despite the fact that 86% of the companies used firewalls, the companies did not feel secure against penetrations. A typical firewall will allow or deny incoming packets based on the port that the TCP or UDP request is arriving on. It is designed to deny clearly suspicious traffic but is also designed to allow some traffic through. This behavior has a major disadvantage, as any packet is allowed through an open port in the firewall. Many exploits take advantage of weaknesses in the protocols that are allowed through the perimeter firewalls and once the web server has been compromised, this can often be used as a springboard to launch additional attacks on other internal servers. Once a rootkit or back door has been installed on a server, the hacker has unfettered access to that server. Moreover, according to the most recent Symantec Internet Security Threat Report [20], attackers are now creating global networks that support coordinated criminal activity. All this sophisticated criminal activity has driven up the costs of defense and recovery. The inadequacies inherent in current defenses have driven the development of a new breed of security products known as Intrusion Detection Systems (IDS). Today s IDS is a combination

12 2 CHAPTER 1. INTRODUCTION of signature analysis, traffic monitoring, and anomaly detection technologies. The heart of most solutions today is signature analysis. Typically, a signature-based IDS is configured with thousands of rules that detect malicious attacks and codes. Similar to anti-virus software, an attack must be analyzed before a signature is developed to recognize it. This time lapse can be critical. In the last years, the time between a new vulnerability and its associated exploit has been decreasing, placing more pressure on IDS manufacturers to rush signatures to the market. For example, the MyDoom attack [13] benefited from this time interval, allowing it to become one of the fastest spreading worm in history. Timely delivery of signatures is integral to overall IDS effectiveness. An alternative to signature-based IDS called behavioral or anomaly-based IDS. These systems assume that normal network traffic behaves within certain patterns. For example, opening network ports in rapid succession is typically not seen in normal traffic, so a behavioral or anomaly-based IDS may flag that traffic as abnormal and identify it as a port scan [14] (generally a precursor to an attack). These systems seek to provide an alternative to the drawbacks of signature-based systems. Monitoring for intrusions is a critical component of any network security policy. The greatest challenge, when working with IDS systems, is sifting through and utilizing the large volume of data generated. As IDS systems deal with detection, the role of IDS systems is postmortem reporting. In order to prevent attacks, we need systems that can detect attacks online and prevent malicious data to enter the network. The new brand of IDSs is IPSs (Intrusion Prevention Systems). Early intrusion prevention systems were spawned from intrusion detection systems. IPS vendors began to block detected attacks. The earliest generation of the IPS blocked attacks by integrating with firewalls. The IPS creates a rule, sends it to the firewall and the firewall blocks traffic from or to a particular IP address or port. These systems took action on attacks and were particularly effective in blocking systematic attacks from particular networks or hosts. However, first generation IPS solutions had a number of drawbacks. First, they could prevent only repetitive attacks. I.e. at the first occurrence of the attack, the system detects it, creates a rule and sends it to the firewall. If this attack repeats itself, the firewall prevents it from entering the network. Second, a Firewall examines packets by their IP address. Since Proxies and NAT (Network Address Translation) are widely used, there is a difficulty to determine if a packet is legal or malicious. Last, a Firewall can block specific host by filtering its IP address. Since the same attack can be originated by another host, hackers could evade these systems with relative ease and even generate distributed attacks. The modern IPS systems are integrated with Intrusion Prevention capabilities in order to dis-

13 CHAPTER 1. INTRODUCTION SIGNATURE-BASED NIPS card or block the offending data from passing through the system. These Intrusion Prevention Systems (IPS) are proactive defense mechanisms designed to detect malicious packets within normal network traffic. These systems block the offending traffic automatically before it does any damage rather than simply raising an alert as, or after, the malicious payload has been delivered. Similar to the evolution of firewalls, IPS systems today are extremely effective and scalable because they perform deep packet inspection to ensure that only legitimate traffic makes it into the network. This method of attack blocking alleviates the issue of blocking entire networks as well as stopping distributed attacks. IPS solutions have evolved into proactive devices that sit in-line on a network, just as a firewall or router does. As a consequence of sitting in-line, IPS solutions are required to match high-traffic, gigabit-level requirements. As traffic has increased, the number of attacks has grown as well. Clever algorithms have to be developed to filter the real attacks. While today s IPS technology is a significant step forward, the debate continues. In addition to the accuracy requirement, IPS solutions have to minimize their latency. This is a major challenge since today s Inline IPSs introduce latency as every packet needs to be inspected. The obvious need for increased accuracy in IPS systems is rapidly raising the performance bar for IPS products in the marketplace today. The optimal system should be independent of the firewall, fully automated, eliminate false-positives, and block almost all attacks that try to enter the network. There are two types of NIPS, signature-based detection NIPS and anomaly detection NIPS. In signature-based NIPS, the system has a database of predefined signatures of known attacks, and the detection of attacks has to utilize a high-speed pattern matching algorithm. On the other hand, anomaly detection NIPS is supposed to supply a protection against zero-day attacks, which are new and unknown threats, before the vulnerability is discovered and the exploit is created and launched. 1.1 Signature-Based NIPS Signature-based detection NIPS is usually comprised of two components: a pattern matching engine and a complementary packet classification engine. The pattern matching engine s input is a received packet and its output is a set of patterns which are a subset of a set of signatures. The signatures identify worms, viruses and protocol specific keywords. The classification engine tracks each connection traversing the packet processor, and ensures that the packets are valid. Today s pattern matching algorithms must be able to operate at wire speeds. With networking speeds doubling every year, it is becoming increasingly difficult for software based solutions to keep up with

14 1.2. ANOMALY-BASED NIPS 4 CHAPTER 1. INTRODUCTION the line rates. This has underscored the need for specialized hardware-based solutions. 1.2 Anomaly-Based NIPS Anomaly detection NIPS has to protect the network against unknown threats before the vulnerability is discovered and the exploit is launched. Thus, NIPS has to integrate intelligent capabilities in order to cope with novel attacks. The main idea is modeling normal network traffic behavior and then finding anomalies of that behavior. Our goal is prevention of attacks and not just detection, thus, the false-positive rate is a major challenge. Since we discard packets that were categorized as part of an attack, any false-positive is a normal traffic that is being blocked. 1.3 Internet Threats In order to generate an attack, the attacker usually has a motive and it uses some method to exploits some vulnerability. There are several ways to generate an attack, the popular ways are listed below. Virus. Harmful code that infects computers (or other devices), making them unusable. Botnets. A botnet is a collection of software agents, or robots, that run autonomously and automatically. These agents are usually compromised computers (also called zombie machines) that are combined into a network. The agents in the botnet are used to send out spam messages, or to perform a denial of service attack against a remote target. A botnet s originator (aka bot herder ) can control the group remotely, usually for malicious purposes. The herder can send instructions to millions of zombie machines. Botnet is used as a tool to generate the attacks. Spam. Spam is flooding the Internet with many copies of the same message, in an attempt to force the message on people who would not otherwise choose to receive it. Botnets are used to send about 80% of spam. Most spam is commercial advertising. spam lists are often created by scanning Usenet postings, stealing Internet mailing lists, or searching the Web for addresses. Any user that uses phone service to read his s, pays to read also these spams. In addition, the transmission of spams costs money for ISPs and online services, these costs are transmitted directly to subscribers. Spam is an important vector for malware propagation. Sophos, IT security firm, published a report from 2010 [18] that revealed an alarming rise

15 CHAPTER 1. INTRODUCTION INTERNET THREATS in attacks on users of social networks, such as Facebook and Twitter, by cyber criminals. According to the report, criminals have increasingly focused attacks on social networking users in the last 12 months, with an explosion in the reports of spam and malware: 57% of users report they have been spammed via social networking sites, a rise of 70.6 % from Commtouch published a security report in 2010 that revealed that spam levels averaged 88% of all traffic throughout the third quarter of Malware. Malware is a class of malicious software-viruses, worms, trojans, and spyware - that is designed to infect computers and systems and steal critical information, delete applications, drives and files, or convert computers into an asset for an outsider or attacker. Such scams have taken advantage of the full gamut of vectors to reach new audiences, such as links sent out via promising lottery winnings, or messages spread via social networking sites such as Twitter or Facebook. One key profit-driven malware trend of 2009 was scareware - faking antivirus security product. These attacks fool users into believing their computer has a security problem. Typically, scareware is planted on web-sites in the form of pop-up advertisements or disguised downloads. Phishing. Phishing is an Internet fraud that aims to steal valuable information such as credit cards, user IDs and passwords by creating a web-site similar to that of a legitimate organization, then directing traffic to the fraudulent site to harvest what should be private information for financial or political gain. Denial of service (DoS). DoS attacks are an attempt to make a computer resource unavailable to its intended users. Usually this is done by flooding of incoming messages that essentially force the targeted system to shut down, thereby denying use by legitimate users. In Distributed DoS (DDoS), these messages arrive from multiple resources. Today, the trend is to generate multi-stage attacks, i.e. attacks that combine messaging and web elements. In the first stage, the attacker entices the user to get into a malicious site. This, for example, can be done by sending an that entice the user to click on an embedded link, or manipulating search engines so the attacker link appear higher than legitimate results. In the last stage, the attacker tries to convince the victim to install malware, complete a phishing form or submit personal information. These attacks can be blocked by preventing any of these stages.

16 1.4. RESEARCH OBJECTIVES 6 CHAPTER 1. INTRODUCTION 1.4 Research Objectives We defined 4 main objectives. In order to fulfill the following objectives, we designed and developed smart data structures that can be implemented in hardware as described below. Threats Prevention. Threats prevention (and not just detection) adds difficulties to the system, most of them stem from the fact that the system has to work at line-speed. The NIPS device has the potential to act as a bottleneck, increasing latency and reducing throughput. In all of our engines, we developed algorithms that either use a special hardware or analyze only suspicious data so they can operate at line-speed. Accuracy. The system should be as accurate as possible. Accuracy is measured by falsenegative and false-positive rates. False-negative is a case in which the system does not detect a threat and false-positive is the case in which the system drops normal traffic. Scalability. One of the major problems is memory consumption. There is a trade-off between accuracy and memory consumption. One of our goals is to supply a scalable system with a reasonable amount of memory without compromising the system accuracy. Building Normal Traffic Behavior. Anomaly-based system uses thresholds in order to decide whether a packet/flow is part of an attack or not. These thresholds are determined according to the normal traffic behavior. Thus, the normal traffic behavior should reflect the real traffic behavior, moreover, it should be updated according to the traffic changes. 1.5 Dissertation Outline This dissertation is comprised of two parts. The first part presents signature-based engines and the second part presents anomaly-based engines. The rest of this dissertation is organized as follows: Chapter 2 describes the concept of signaturebased NIPS and it also presents several common algorithms that are used in various systems. Most of the algorithms are software oriented but some can be implemented in hardware. Chapters 3 and 4 present our two algorithms, the TCAM-based algorithm and the Hash-based algorithm. Both algorithms are efficient and operate at wire speed. The second part of this dissertation describes anomaly-based NIPS. This part begins in Chapter 5 where we present the attacks we target and present several related works. Chapters 6, 7, 8 and 9 present our anomaly-based engines. The last Chapter 10 conclude this dissertation and presents several directions to proceed in this research.

17 CHAPTER 2. SIGNATURE-BASED NIPS 7 2. Signature-based NIPS A pattern matching algorithm is a building block in any signature-based NIPS. Filtering traffic based on the packet headers is not enough. Many attacks, like the well known internet worms Nimda, Code Red and Slammer contain a string of bytes as signature. The location of a signature in the packet payload is not deterministic, so the algorithm must be able to detect patterns of different lengths starting at arbitrary locations. The pattern matching algorithm allows to deeply digging into traffic flows to spot hidden attacks on targets like Web, , and DNS servers. The algorithm must be able to operate at wire speeds that are doubling every year. Our string matching algorithms function by analyzing the text using a search window and then systematically shifting the window along the text. This is known as the sliding window mechanism. The patterns length may be smaller or wider than the window width. We make use of the bad character heuristic in order to reduce the number of comparisons required. A brute force approach requires O(mn) comparisons, where m is the pattern length and n is the text length. In the bad character approach the algorithm needs to preprocess the set of patterns to create a shift table. The shift table contains shift-values for each block of characters from the patterns alphabet. The algorithm starts by comparing the rightmost character in the sliding window with the m th character of the pattern. If the mismatching character appears in the search window, the search window is shifted so that the mismatching character is aligned with the rightmost position of the mismatching character in the search window. If the mismatching character does not appear in the search window, the search window can be shifted by its width. In the case of a mismatch, the corresponded shift table entry determines how further we can skip the search window. When a match occurs, an exact match algorithm is invoked to efficiently compare the text and the set of patterns. The string matching algorithms have several characteristics. First, the algorithms match multipattern strings of various sizes while their runtime complexity is independent of patterns length or count. Moreover, many of the processing of the algorithms and data structures can be done offline. Another main characteristic is the algorithms space complexity, we need to minimize it so the algorithms can be easily implemented in hardware, either in regular memory (SRAM) or in TCAM. Last, the algorithms worst-case performance still enables processing packets in several Gbps. We devise two algorithms: One ia based on a TCAM and second uses a standard memory.

18 2.1. RELATED WORK 8 CHAPTER 2. SIGNATURE-BASED NIPS 2.1 Related Work This section surveys the common algorithms for multiple string pattern matching. Since a string matching algorithm is an essential building block for numerous applications it has been extensively studied [100, 108] Software Based Pattern Matching This section briefly describes some of the best known software based algorithms which are: Knuth- Morris-Pratt [66, 81], Boyer-Moore [39, 46], and Aho-Corasick [29, 45]. KMP Algorithm The naive algorithm forgets all information about previously matched symbols after the pattern shifts. Thus, it is possible that it re-compares a text symbol with different pattern symbols again and again. This leads to its worst case complexity of (nm), where n is the length of the text and m is the length of the pattern. The algorithm of Knuth, Morris and Pratt [66, 81] makes use of the information gained by previous symbol comparisons. It never re-compares a text symbol that has matched a pattern symbol. As a result, the complexity of the searching phase of the Knuth-Morris- Pratt algorithm is in O(n). However, a preprocessing of the pattern is necessary in order to analyze its structure. The preprocessing phase has a complexity of O(m). Since m n (pattern length text length), the overall complexity of the Knuth-Morris-Pratt algorithm is in O(n). BM Algorithm The basic idea of the Boyer-Moore [39,46] algorithm is that more information is gained by matching patterns from the right than from the left. This allows reducing the number of the needed comparisons. We denote the pattern to search in the string as pat and the patterns length as patlen. pat is aligned with the string such that the first character of pat is aligned with the first character of string. We will call the patlen th character of the string char. The algorithm uses three main observations. One, if char is known not to occur in pat, then there is no possibility of an occurrence of pat starting at string positions 1 to patlen. The second observation is a generalization of the first one. If the last occurrence of char in pat is delta 1 characters from the right end of pat, pat can slide down delta 1 positions without checking for matches. If char does not occur in pat, delta 1 is patlen. In the case where the last character of pat matches char, there is a need to determine if

19 CHAPTER 2. SIGNATURE-BASED NIPS RELATED WORK the previous character of the string matches the previous character of pat. The algorithm continues with this examination until it reaches the first character of pat or until it notices a mismatch. In the later case, m is the number of matched characters until the algorithm recognized the mismatch. Lastly, in the case where the algorithm matches part of pat (until some mismatched character), it shifts pat by m + k characters where k depends on where char occurs in pat. If the last occurrence of char in pat is from the right to the mismatched character (within the characters the algorithm already matched), the algorithm does not gain anything from delta 1 so k = 1. In this case the algorithm shifts pat in 1 + m characters. On the other hand, if the last occurrence of char is from the left side of the mismatched character, the algorithm can set k to delta 1 (char) m, so pat can be shifted in delta 1 (char) characters. Both algorithms build a shift table that contains shift-values for each character from the patterns alphabet. They use the shift table to avoid back tracking and to shift the text when possible. The search time for an m bytes pattern in a n bytes text is O(n + m). If there are r patterns, the search time is O(r(n + m)), which grows linearly with r. AC Algorithm The Aho-Corasick [29] algorithm matches multiple patterns simultaneously. It pre-processes the patterns and builds a finite-state-machine which can process the text in a single path in O(n) time. The behavior of the pattern matching machine is dictated by three functions: a goto function g, a failure function f and an output function output. g maps a pair consisting of a state and an input symbol into a state or a fail message. Each pattern has a track in the pattern matching machine, g simply proceeds to the next state with the given input or fails in case there is no such a transition. f maps a state into a state. f is consulted whenever g reports fail and it returns the state where the algorithm can proceed from with the next input character. Certain states are designated as output states which indicate that a pattern has been found. Whenever the pattern matching machine encounters one of these states, s, output emits the set output(s ). The problem with this technique is the exponential state explosion (see [54]) Hardware Based Pattern Matching NIPS must be able to operate at wire speeds. With networking speeds doubling every year, it is becoming increasingly difficult for software based solutions to keep up with the line rates. This

20 2.1. RELATED WORK 10 CHAPTER 2. SIGNATURE-BASED NIPS has underscored the need for specialized hardware-based solutions which are portable and operate at wire speed. This section presents some of the best known hardware based algorithms. Parallel Bloom Filters The Parallel Bloom filters algorithm [48, 98] can handle thousands of patterns but it uses a Bloom filter for each possible pattern length. A Bloom filter is a data structure that stores a set of signatures compactly by computing multiple hash functions on each member of the set. Each Bloom filter computes k hash functions of each pattern in its set and produces k hash values ranging from 1 to its corresponding patterns length. It sets the k bits in a m bits vector. It repeats this process for each pattern in its set. Each Bloom filter scans a substring of its corresponding length from the input and detects suspicious signatures. If all the k hash functions match the input key, the Bloom filter declares this pattern as suspicious and the analyser determines if the string is indeed a member of the set or a false positive. Multiple engines can be instantiated to monitor the data, thus the byte stream can be advanced by more than one byte at a time. With four parallel engines, the algorithm can push four bytes in a single clock cycle and the throughput is over 2.46 Gbps. The fact that each pattern s length requires a separate Bloom filter is a limiting factor, especially when dealing with very long virus definitions that can be thousands of bytes long. Network Processor Pattern Matching The work of Liu et al. [72] uses a shift based algorithm that uses a network processor enhanced with a memory based hashing engine. It uses a prefix sliding window (PSW) of length w, which shifts from the leftmost byte to the rightmost byte of the text. The shift value is determined as follows: if the w sequential bytes covered by the PSW contains k bytes of pattern P j such that P SW w k...p SW w 1 = a 0...a k 1 where 1 k w, the algorithm can shift by w k bytes. If there is no such pattern, the algorithm can shift by w bytes. Their solution focuses on a single pattern or compositions of patterns without any correlation among them. Their algorithm is optimized for patterns of length of 4. The memory requirement of their solution is a function of the width of the patterns, therefore a wider PSW is not practical. The integer value of the bytes in the PSW is used as the entry address for looking up the skip distance (the skip table size is (2 8 ) width ). At that time, analysis of Snort s rules pattern supported their assumption since the majority of the patterns was indeed of length of four (which is not the case anymore). This algorithm can get a shift average of around 2.

21 CHAPTER 2. SIGNATURE-BASED NIPS RELATED WORK TCAM Pattern Matching The time required to find an item stored in memory can be reduced considerably if the stored data item can be identified for access by the content of the data itself rather than by its address. Memory that is accessed in this way is called content-addressable memory (CAM) [82]. CAMs can be binary or ternary. A Ternary CAM, TCAM, can store three binary values for every bit: zero, one and don t care. This extra feature enables more advanced algorithms to exploit the memory. The work of Lakshman, Yu and Katz [54] presents a TCAM based pattern matching algorithm. A TCAM key is constructed for every byte in the packet. The width of the key is configurable and equals the TCAM width. Each row in the TCAM presents a pattern. The algorithm repeatedly extracts a key comprised of w consecutive bytes from the packet. Once the TCAM reports a hit, the algorithm reports the matched pattern. If the packet length is n, the algorithm has n TCAM lookups. Our RTCAM solution (presented in Chapter 3) also uses TCAM but it skips many bytes in the text, thus, it searches the TCAM for only a subset of keys. They assume that TCAM lookup time is 4 ns, thus, the algorithm matching speed is 8 n/4n = 2 Gbps. This algorithm maintains a matching table in order to recover the long patterns. This table stores all the valid combination of prefixes and suffixes patterns. For any combination it stores the prefix index, the suffix index and the distance between them. If a combination yields a new valid prefix, it is also added to the table as a new prefix. The algorithm also maintains a Partial Hit List (PHL). When it matches a prefix, it records it in this list. At every TCAM hit, the algorithm checks if it matches a prefix or a suffix. If it matches a prefix, it adds it to the PHL but if it matches a suffix, it checks if a combination of the matched pattern with one of the matched prefixes yields a valid combination. Since the lookup process requires searching the matching table, the matching table is a three-dimensional array. The total memory consumption for this array is w a b, where w is the TCAM width, a is the number of prefixes, b is the number of suffixes and they are both equal to i ( m i/w 1) where m i is the length of pattern i. Most of the entries in this array are empty. Another drawback of their solution is the fact that it relies solely on TCAM memory. The increased updates to signatures suggest that designing a solution solely based on TCAM memory can be quite expensive. For example, at the time that the paper was published, populating ClamAV viruses definition within a TCAM memory required 240KB 1 (TCAM width of 128 Bytes). In today s ClamAV version 2, it requires more than 1 Version 0.15, 1768 definitions. 2 Version , 846,214 definitions.

22 2.2. PROBLEM DEFINITION 12 CHAPTER 2. SIGNATURE-BASED NIPS 100MB, which is very expensive in today s TCAM prices. Note that Snort s signatures are fewer (around 9300, version 9.3) and can be realized in a TCAM of about 690KB (for TCAM width of 32 Bytes). They define a scan ratio as the total scanning time (including memory lookups) vs. the time spent on TCAM lookups only. Their simulation shows that 60% of the packets have a scan ratio of 1, meaning that there are no memory hits at all and 80% of the packets have a scan ratio of 1.2. The max scan ratio for all packets is less than 2, resulting in a scan rate of 1 Gbps. 2.2 Problem Definition This section provides the necessary terms and notations, which are commonly used in the field of signatures-based intrusion detection systems. We also briefly survey the very popular Snort IDS [17] and the widely used GPL anti-virus library, ClamAV [6] General Definitions DEFINITION Define a pattern P to be a string of characters from an alphabet Σ which needs to be identified within the input text. Define a sub-pattern P s to be a sub-string of a pattern P. DEFINITION Define a search window to be a sequential part of the input text within which a sub-pattern is looked for. DEFINITION Define a string-matching algorithm as follows: We assume that the text is an array T [1..n] of length n and that the pattern is an array P [1..m] of length m. We further assume that the elements of P and T are characters drawn from a finite alphabet Σ. We say that pattern P occurs with shift s in text T (or, equivalently, that pattern P occurs at position s + 1 in text T ) if 0 s n m and T [s+1..s+m] = P [1..m] (that is, if T [s+j] = P [j], for 1 j m). If P occurs with shift s in T, then we call s a valid shift; otherwise, we call s an invalid shift. The string-matching problem is the problem of finding valid shifts, which a given pattern P occurs in a given text T. The extended problem of finding multiple patterns in a given text is called multiple pattern matching. DEFINITION Define a multiple pattern string-matching algorithm as follows: The text is an array T = t [1..n] and the set of r patterns, P j, where 1 j r (the patterns may have different lengths). The algorithm goal is to output the positions of all occurrences of any of the patterns in the text.

23 CHAPTER 2. SIGNATURE-BASED NIPS PROBLEM DEFINITION Since we use Snort and ClamAV rules set in our simulation, we overview them both and present their specific notation Snort Specific Notations Snort [17] is an open source NIDS, which is available free of cost. Snort uses rules stored in text files that can be modified by a text editor. Snort comes with a rich set of pre-defined rules to detect intrusion activity and it is possible to add more rules to the set at will. Each rule contains header and content fields. The header part checks the protocol, source and destination IP address and port. The content part scans packets payload for one or more patterns. Rules with more than one pattern are called correlated rules. Rules can also contain negation patterns. Negation of patterns stands for no occurrence of the pattern. The matching pattern may be in ASCII, HEX or mixed format. HEX parts are between vertical bar symbols. An example of a Snort rule is: alert tcp any any -> / (content: "idc 3a3b "; msg: "mountd access";) The above rule looks for a TCP packet, with any source IP and port, destination IP: and port 111. To match this rule, packet payload must contain pattern idc 3a3b, which is ASCII characters i, d and c, and also bytes 3a and 3b in HEX format. In contrary to previous works, our solutions are Snort compatible. Rule s Syntax Snort rules are divided into two logical sections, the rule header and the rule contents. The rule header contains the rule s action, protocol, source and destination IP addresses and the source and destination ports information. The rule contents contains the patterns to match, alert messages and options that define which parts of the packet should be inspected to determine if the rule action should be taken. Snort defines a set of rule options. All Snort rule options are separated from each other using the semicolon character. Rule option keywords are separated from their arguments with a colon character. This section provides some of the more important keyword definitions that are used to specify where in the text the string matching algorithm should operate. They also enable writing correlated rules with position relation among them.

24 2.2. PROBLEM DEFINITION 14 CHAPTER 2. SIGNATURE-BASED NIPS DEFINITION Define a rule to be a set of patterns with some correlation among them. A rule is matched only if all its patterns are also matched with the expected correlation. DEFINITION Define offset to be the position in the text where to start searching for a pattern. It specifies how far into a packet the algorithm should ignore before starting to search for the specified pattern relative to the beginning of the packet. For example, an offset of 5 would tell the string-matching algorithm to start looking for the specified pattern after the first 5 bytes of the payload. DEFINITION Define depth to be the position in the text where to stop searching for a pattern. It specifies how far into a packet the algorithm search for the specified pattern. For example, a depth of 5 would tell the string-matching algorithm to only look for the specified pattern within the first 5 bytes of the payload. DEFINITION Define distance to be the number of characters the algorithm should ignore before starting to search for the specified pattern relative to the end of previous pattern match. Distance can be thought of as exactly the same thing as offset, except it is relative to the end of the last pattern match instead of the beginning of the packet. DEFINITION Define within to be the number of characters in which the algorithm should search for the specified pattern relative to the end of the previous pattern match. It specifies the maximum number of bytes between two pattern matches. Within can be thought of as exactly the same thing as depth, except it is relative to the end of the last pattern match instead of the beginning of the packet ClamAV Anti Virus ClamAV [6] is an anti-virus toolkit for UNIX, initially designed for scanning on mail gateways. It provides a flexible and scalable multi-threaded daemon, a command line scanner and an advanced tool for automatic database updating via Internet. The package also includes a virus scanner shared library. ClamAV includes a virus database that currently contains nearly 850, 000 signatures (version ). Although this number is smaller than those of major commercial virus scanners, the number of viruses recognized by ClamAV has been steadily increasing [78]. All ClamAV signatures are simple patterns with no keywords or any other options. The next two chapters present two novel multiple pattern string-matching algorithms.