Kingston University London

Transcription

1 Kingston University London Analysis and Testing of Intrusion Detection/Prevention Systems (IDS/IPS) XYLANGOURAS ELEFTHERIOS Master of Science in Networking and Data Communications THESIS

2 Kingston University London Thesis Title Analysis and Testing of Intrusion Detection/Prevention Systems (IDS/IPS) Dissertation submitted for the Degree of Master of Science in Networking and Data Communications By XYLANGOURAS ELEFTHERIOS SUPERVISOR Dr. PANAYIOTIS KOTZANIKOLAOU KINGSTON UNIVERSITY, FACULTY OF COMPUTING, INFORMATION SYSTEMS & MATHEMATICS ΤEI OF PIRAEUS, DEPARTMENTS OF ELECTRONICS AND AUTOMATION JULY 2011

3 Table of Contents Chapters Pages 1 Introduction Review The Intrusion Detection Systems Misuse Detection Anomaly Detection Host-based Intrusion Detection Systems (HIDS) System Logs System Call File System Monitoring Advantages Disadvantages of HIDS Network Intrusion Detection Systems (NIDS) Topology Packet Sniffing Advantages Disadvantages of NIDS Popular IDS OSSEC Bro Suricata Network scanning Network scanning tools Nemesis Nesus Practical Experience System Analysis and Configuration Network analysis Security Policy Firewall Configuration Router Configuration Test-Bed Environment Snort Overview Snort Configuration... 27

4 Setting up the network variables Configuration of the decoder Configuration of the Base detection engine Configuration of the dynamic loaded libraries Configuration of the preprocessors Output Plug-in configuration Customization of the rule set Customization of the preprocessors and the decoder rule set Customization of shared object Nmap Nmap Features Performance Test Scan Detection Host Discovery Port Scanning Fragmentation False Alarms FTP server protection Internal Network Protection Conclusions References Table of Figures Figures Pages 2.1 Network intrusion Detection System Architecture Enterprise Network DMZ and Internal Network of the Enterprise Network Test-Bed Environment (VMware workstation) Virtual Machine (Windows XP) Virtual Machine (Backtrack 5) Nmap sp scan for host with IP address sp scan on the DMZ Snort detects the sp Nmap scan... 32

5 4.3 sp scan on the Internal Network Snort detects the sp Nmap scan SYN scan for the ftp server Snort detects the SYN scan FIN scan for the client of the Internal Network with the IP address Snort detects the FIN scan NULL scan for client of the Internal Network with the IP address Snort detects the NULL scan Xmas scan for the client of Internal Network with the IP address Snort detects the Xmass scan Nmap scans the port Snort detects the scan that performed on port SYN scan for the ftp server with the fragmentation option enabled Snort detects the SYN scan despite the fragmentation Snort detects a client that is visiting the web site: 46

6 Abstract Intrusion Detection Systems (IDS) are network security tools that were developed in order to detect and identify possible intrusions. This dissertation presents an overview of the IDS/IPS technology along with their various categories. It also presents the implementation of Snort on a test-bed environment. Snort is one of the most popular IDS. The test-bed environment will be used for the performance test of Snort. Snorts ability to detect intrusion will be tested with the use of a specialized scanning tool (Nmap) that will generate attacks. The results from these tests will be discussed and analyzed to provide a general view of IDS technology.

7 1. Introduction History has shown that in our world individuals always tried to protect their privacy and their valuable resources. History has also shown that the security measures that individuals use in order to protect their privacy or their valuable resources will somehow fail. The same thing happens with the computers networks. No matter how much money are spend for the protection of a computers network there is always the probability of failure. The computers networks security tools are designed according to three basic principles: the first principle is the prevention of possible attacks; the second one is the detection of any malicious activities the last but not least the active response to any kind of attack (Kizza, Joseph 2005). Many tools have been developed over the years for the protection of computer networks. One of the most common security tools that is installed almost in every computers network is the firewall. It is the first line of protection. It works like a filter that filters the incoming traffic and allows only the requested data to enter the protected system. On the other hand there are specific tools that are developed in order to automate attacks against protected networks that use firewalls. These tools can trick the firewall, for example by masquerade the attacker s profile. If the intruder manages to enter the system without being recognized then he can act like an authorized user and trigger the attack. As was mentioned in the previous paragraph there are not security measures that will last forever. It is certain that the battle against malicious attacks will go on forever. The best way to defend a computer system is to accept that the intruder will somehow enter the protected system. The ability of detecting a possible intrusion is the key element to an integrated protection against any kind of malicious attacks. There are tools that are made for detecting and preventing any kind of malicious attack. These tools are well known as Intrusion Detection Systems (IDS). They are used by network administrators in order to detect possible unauthorized intrusion made in their protected network. 1

8 The first publication about Intrusion Detection Systems was made back in 1980 where the authors tried to present the importance of monitoring systems in order to audit trails that lead to misuse (Kizza, Joseph 2005). The Intrusion Detection tools are sets of techniques that are made for identifying any kind of malicious activities (Vokorokos et al. 2006). The IDS systems provides: (a) continuous monitoring of the actions made by the users of a network system or an individual user, (b) configuration of the security mechanisms of a system, (c) control the integrity of valuable resources, detect possible attacks by comparing with patterns of well known attacks, (c) analysis of any abnormal activities and (e) operation system management. There are two main categories for the IDS systems; the Host-Based and the Network-Based systems. The host-based systems are used for the protection of a single computer and they use information related to the computers operating system (Yeung, Yuxin 2003). On the contrary the network-based systems use information from the packets that travel over the network (Wang et al. 2010). In both cases the system sends alarm to the system administrator. The alarms can be visual or audible signs or even s that inform the administrator about a possible intrusion. Except from the Intrusion Detection Systems there are also security tools that not only detect malicious activities but also prevent them. These tools are well known as Intrusion Prevention Systems and were first presented in the late 90s. Intrusion Prevention and Intrusion Detection Systems have similar operations, the IPS is considered to be an extension of the IDS (Rowan, Tom 2007). There are two main techniques for the Intrusion Detection Systems, the anomalybased and the signature analysis detection. IDS systems can use both techniques and in some cases at the same time. The first technique, the anomaly-based detection, detects the not normal activities of a system (Kizza, Joseph 2005). For example, a web server has the 90% of its traffic load during the day, in this case an unexpected increase of data traffic during the night it is considered to be a not normal activity. For that mechanism a list of normal and expected activities (statistical description) is compulsory in order to configure the IDS. Although there are cases of false alarms 2

9 for the IDS systems that use the anomaly based mechanism. In many cases system s behavior is out of the ordinary without being attacked. The administrators who choose to use the anomaly-based configuration face the problem caused by false alarms. The signature-based detection uses a database in order to detect possible attacks. The database contains attack signatures and the IDS system compares these signatures with the patterns made by audit files (Jia, Chen 2009). Every time the IDS system finds a match it triggers the alarm. As with the anomaly-based detection, there is always the possibility of a false alarm. Moreover, both mechanisms may fail to detect a possible intrusion. As was mentioned in the first paragraph there are no security measures that guarantee the protection against all possible attacks. This dissertation focuses on the protection provided by Intrusion Detection and Intrusion Prevention Systems against malicious attacks. In more details, this dissertation analyses the IDS/IPS systems by testing their performance against malicious attacks provided by specialized attack tools. Chapter 1 of this thesis describes how the IDS and IPS systems work along with the related work of the subject. The following chapter (Chapter 2) describes a test-bed environment suitable for the testing of the IDS/IPS systems. The installation of an IDS system on a personal computer is likely to cause problems to its latter use due to the changes made by the IDS to the ports of the system. For that reason the tests for the IDS performance will take place on a virtual machine. Chapter 3 part presents the implementation of an IDS/IPS system on the test-bed environment that was mentioned on the previous part. The IDS system that will be used for the dissertation is snort. It is an open source IDS developed by Sourcefire. 3

10 Then the IDS system will be tested against a specialised attack tool. The results of these tests will be gathered and presented analytical. The last chapter (Chapter 4) of this dissertation provides the conclusions from these tests and the performance analysis of the IDS/IPS system. 2. Review 2.1 The Intrusion Detection Systems Intrusion Detection Systems are systems that provide security protection against malicious attacks. They were developed in order to monitor the normal operation of computer systems or networks and detect suspicious events. The IDS alerts the administrator of the protected computer system/network when an intrusion has occurred but sometimes the IDS generates false alarms. The false alarms are produced by non malicious events that triggered the detection mechanism of the IDS. An IDS is usually an additional security measure for computer networks that already use a firewall for their protection. The IDSs do not always manage to detect all the possible intrusions because attackers use techniques that can trick both firewalls and IDSs. Computer administrators use IDSs for two main reasons; the first is that firewalls do not provide enough protection against malicious attacks (there are several firewall bypass techniques) and the second one is that they do not know if they do not know if they succeeded to protect the system or not. The IDSs are systems that can protect both networks and its hosts. There are two major categories; the Host-based Intrusion Detection Systems (HIDS) and the Network-based Intrusion Detection Systems (NIDS). A HIDS analyzes the files and the applications of the operating system while a NIDS analyzes the packets that travel through the network. Both types use the same techniques for detection. The first technique is the misuse detection where the IDS searches to for something already known by using misuse patterns; and the anomaly detection where the IDS searches inside the system for out of the ordinary operations (Bace, Mell 2001). 4

11 2.2 Misuse Detection The misuse detection technique compares the events of the system with the events of a predefined pattern characterized as malicious. For every positive result produced by the comparison the IDS informs the administrator of the system for a possible attack. These predefined patterns are also called signatures. Each signature defines a specific attack. The misuse detection technique is also known as signature-based detection. This technique does not produce false alarms very often. It can also detect with precision the attack technique that tries to penetrate the protected system. The precise detection helps the administrator to deal better with the attack. Moreover the misuse technique can help system administrators with lack of experience in the security field to effectively protect a system (Bierman et al. 2001). The signature-based technique detects only the known threats, in other words it can detect only the modeled threats (Mutz et al. 2003). This technique works like the antivirus, it must be up to date during its operation in order to provide maximum protection to the user. In addition, the design of this technique makes sometimes the protected system vulnerable to common threats. 2.3 Anomaly Detection The Anomaly Detection method focuses on the behavior of the system, the IDS in this case triggers the alarm for every behavior that is considered to be out of the ordinary. The theory for this detection method is that malicious activities can be detected due to their abnormal behavior. In other words, the attacks have different activity than the normal procedures of the system (Kizza, Joseph 2005). In order to do that, the IDSs that use the anomaly detection technique, creates a profile that has all the characteristics of a legitimate user of the system and they compare its behavior with any other from inside the system. The construction of this profile requires data from the normal operation of the system during a period of time. There are many measures and techniques that are used by the detection systems for the anomaly detection technique. One common technique that is used in many commercial IDSs is the threshold detection. In this technique the behavior of the 5

12 system is measured by counts. For example, how many times a user tries to login to the system, the number of file queries made in a certain amount of time, the amount of memory (RAM) that was used for a specific procedure. For all these examples the IDS has an acceptable level that defines the normal activity, any activity that exceeds these level is characterized as a malicious attack (Aydin et al.2009). Another common technique used by the commercial IDSs is the statistical measure technique. There are two cases for this technique. In the first case the attributes of the user s profile should agree with a specific pattern that indicated the legitimate profiles of the users. The second case uses patterns that are made by a collection of data from a reasonable amount of time. These statistical measures are parametric and non parametric respectively. The IDSs can also use rule-based measures that use rule patterns instead of the patterns that are used in the non-parametric technique which uses countable values. There are also other measure techniques that are used by the non-commercial IDSs that contains generic algorithms. The main drawback of anomaly detection is that it produces a great number of false alarms. It is very difficult to represent the normal activity with the use of a static pattern. The behavior of a legitimate user inside a system can be sometimes out of the ordinary. In addition, the construction of a legitimate profile requires a great amount of historical data in order to represent the normal behavior. On the other hand the anomaly detection method can detect common and specialized attacks without the use of detailed information about them. Furthermore, it can be used in order to create signatures used by detection systems that use the misuse detection technique (Bace, Mell 2001)(Bierman et al. 2001). 2.4 Host-based Intrusion Detection Systems (HIDS) The Host-based Intrusion Detection Systems are used to protect a single host. HIDSs use information form an individual system. The most popular detection 6

13 technique for HIDS is the signature recognition. The signatures are patterns that are recognized as possible attacks by the administrator of the computer system. In other words, signatures are rules that describe events and also how these events interact with each other. The signatures are pre-defined by the detection policy made by the computer administrator (Bierman et al. 2001) System logs The IDSs use the system log files as a source of information. The IDSs usually perform log monitoring and alert the administrator every time they find anomalies in the sequence of events System Call The HIDSs can also use the kernel of the operating system in order to observe the system calls. From this point of view the HIDS can alert the administrator (security officer) for suspicious requests made by programs of the computer systems. The HIDS spots the system calls that are supposed to be suspicious according to the detection policy File System Monitoring Except from the systems calls and logs the HIDS can also detect malicious activities by observing the files of the file system. In particular the HIDS monitors the size and the attributes of the files that consists the file system of the computer. This technique has a major advantage against the others; it has a very fast response to malicious attacks Advantages and Disadvantages of HIDS The HIDSs have the ability to detect intrusions that the Network-based IDSs fail to detect since they have the ability to use information about the use of the applications or the access to files of the system. They can also analyze packets that are used for encrypted communication between hosts (the Network-based IDSs do not have this ability). Moreover, they can be used for the detection of malicious attacks that involve integrity breaches (Kozushko 2003). 7

14 The HIDS is part of the computer system that protects; in a successful attack against the system, the intruder could also disable the operation of the HIDS. They only analyze the packets that reach their host; they are unable to analyze the traffic of the rest of the network so they are vulnerable to network attacks. In addition, they are vulnerable to denial of service attacks. Sometimes the HIDSs may have to deal with a great amount of information (HIDS monitor trails of the operation system), in that case they must use additional storage resources. The use of additional storage resources along with process resources affects the performance of the computer system they protect. 2.5 Network Intrusion Detection Systems (NIDS) Most of the intrusion detection systems are made for network protection; two of the most popular network intrusion detection systems is snort and Bro. These systems monitor the data traffic inside the network. The position of the IDS inside a network is very important. The IDS must be able to collect information from all the hosts of the network without affecting the performance of the entire network. As a result, the location of the NIDS differs from network to network. The NIDS may look like an internet firewall but it is not. There are some critical differences between these two security tools. Firewalls make binary decisions according to the rules that they use in order to protect a network. The IDS systems perform further analysis of the data that travel through the network and warn the users for possible attacks (Davies 2002). The firewalls work like doors. Every door opens with a specific key. If the intruder has the key he can enter the protected system and act like an authorised user. The NIDS, on the other hand, works like a security camera that monitors the protected area 24 hours a day and detects the out of the ordinary behaviour. Most protected networks use both firewalls and NIDS in order to provide its users with further protection. Firewall and NIDS supplement each other. 8

15 2.5.1 Topology It is not easy for the network designers to decide for the location of a NIDS inside their network. The decision needs serious consideration. The NIDS uses sensors in order to collect information about the traffic inside the network. The NIDS is supposed to control the data traffic of the entire network which usually contains besides its hosts, routers and switches, servers and other components. If the sensor is placed at the end of a line that caries several of these components then it would be difficult for the NIDS to have a clear view of the traffic. The correct implementation of an NIDS starts with a question: which are the valuable sources of the network? The NIDS should be designed according to the security policy of the network. The position of the NIDS is chosen according to following parameters (Beale 2003): The topology of the network (ring, star etc.) The position of the firewall inside the network If it is necessary to filter traffic both before and after the firewall The components of the network (routers, switches, servers etc.) Figure2.1 shows a network that is protected by an NIDS system. As it is shown the multiple NIDSs are placed in strategic locations inside the network. There are three NIDSs, the first one is used for monitoring the data traffic from and to the Server1 and 2. Another NIDS is used for the analysis of data form and to the clients of the networks. And the last NIDS is used for the protection of a single server (Server 3). Each NIDS is configured in order to protect the hosts of the subnet that is installed. The more NIDS are used the merrier because each NIDS can be configured according to the services used by the hosts of the network. 9

16 Figure2.1. Network that uses NIDS for its protection Packet Sniffing Packet sniffing is one of the most effective ways to perform detection over a network. The NIDS uses the promiscuous mode of the interfaces in order to sniff packets that travel through the network wires. The promiscuous mode is one of the most common techniques of packet sniffing but there are also other techniques that are used for the same reason. A Network Interface Card (NIC) that is set to work in promiscuous mode accepts all the packets that are passing through its interface even those which are not addressed to it (Ansari 2002). Many intruders are aware of packet sniffing and they use techniques in order to avoid being sniffed by the NIDS. Most of the evasion techniques split payload into packets that are undetectable from the NIDS since they do not carry enough information to trigger the alarm, this techniques are also known as fragmentation attacks. As a result IDSs use techniques that detect fragmentation by collecting the packets (fragments) and put them in the right order so that they represent the original payload. Intruders then must find other ways to trick the reassembling capability of the IDS and the battle goes on and on. 10

17 2.5.3 Advantages and Disadvantages of NIDS The Network-based IDSs like all the security tools have some advantages against intrusions and some drawbacks. The advantages are: 1. Large networks do not require a great number of IDSs for their protection. The location of the IDSs inside a network will be according to a specific strategy. 2. Moreover, the implementation of an NIDS has little impact on the networks performance. The IDS do not actually affect the way that data travel through the network wires since they are passive systems. 3. In addition, NIDSs are invisible to intruders. There are no security tools that can identify whether a network is protected by an NIDS or not. The NIDSs have also some major drawbacks: 1. The NIDSs usually fail to detect attacks over busy networks. Most NIDSs are unable to process all the packets of a network during the high traffic hours. Although there are hardware NIDSs that have the requisite process power to perform effective packet processing under high traffic periods. The hardware NIDS increases the overall cost of the network and also affects its scalability. 2. The performance of a NIDS sensor is affected by the operation limits of a network component. Some network switches do not use universal monitoring for their ports which means that the monitoring of sensor over a single host will be limited. In addition, in some cases single ports are unable to mirror the whole amount of data that travel through the switch (Kozushko 2003). 3. The NIDS do not have the ability of analyzing encrypted data. 4. The NIDS is able to inform the network administrator for a possible attack but on the other hand is unable to tell if the attack was successful. Each time the NIDS generates the alarm, the network administrator must search for 11

18 penetration evidence in order to determine whether the attack was successful or not. 2.6 Popular Intrusion Detection Systems The most intrusion detection systems are open source programs, like most security tools, that can be used for free by user who want to protect their systems. Four of the most popular of them are Snort, OSSEC HIDS, Bro and Suricata (Paxson 2004) (Hay et al. 2008)( This paragraph focuses on the presentation of the last three IDSs; snort will be analyzed briefly on the next paragraph OSSEC HIDS OSSEC is an open source Host-based Intrusion Detection System which was generally developed by Daniel Cid along with several others developers. It is a very popular security tool that thousands of users use for free in order to protect their systems. The OSSEC HIDS uses a analysis and correlation engine and can check file s integrity, analyze the log files. It can also perform monitoring for the Windows registry and is designed to use a policy with a centralized architecture. Moreover, as an HIDS its alerts are on real-time and its responses are also active. Despite the fact that the OSSEC was originally developed to work as an HIDS it can also be used for analyzing logs (authentication logs), firewalls, other IDSs or even web servers. The OSSEC is suitable for many operating systems (Windows Microsoft, Mac, Linux and others). There are three installation types for the OSSEC HIDS, the local, the agent and the server installation. The first installation is the most common installation for all the IDSs and is designed to provide protection to a single host. The agent installation is used by several hosts which report to an OSSEC server. The last installation is made to a server that collects information from hosts that use the agent installation. As most of the open source software tools the OSSEC has its own web page where developers from around the world can submit bugs or suggests for the further 12

19 development of the tool. In general, OSSEC is a scalable security tool that becomes better through time Bro This security tool is a Network Intrusion Detection System (NIDS) based on Unix. Bro uses the anomaly detection technique in order to detect possible threats. In other words, it compares network traffic with rules that characterize malicious events. It uses a very effective technique for packet filtering which makes it suitable for monitoring the traffic of a site over an internet connection. The hardware required for this job is similar with the hardware of a common personal computer. The Bro IDS monitors network traffic like the others NIDSs. Some of the features of the Bro NIDS are: the syntax of scripts is made with the use of its own language, the Bro Language. It does not require additional installations on every single computer host. In addition, the Bro language, which is considered to be a very rich language, provides the ability to this NIDS to have a very strong signature matching facility. It can also store information from previous activities and use them for the analysis of signatures. Another interesting feature of Bro is that it contains a snort2bro tool that can convert signatures of Snort into signatures suitable for Bro. The main drawback of this tool is that the actions made by Bro are made through the operating system of the protected system Suricata Suricata is one of the newest open source Intrusion Detection Systems and represents the next generation of security tools. Suricata can work as an IDS and IPS as well, it use the misuse detection technique for detection which generates alerts during events that are characterized as suspicious. The developers of Suricata wanted to create an IDS/IPS that will be used in combination with other security tools in order to create a more secure environment. For that reason Suricata is compatible with many others security tools and can also use calls from these tools for the detection of threats. Suricata is suitable for detecting multiple threats and due to its ability to perform fast analysis of data it can be used effectively for the traffic analysis of networks. This IDS/IPS has the unique ability to have keywords not only 13

20 for the common protocols (IP, TCP, UDP etc.) but also for HTTP, FTP or even SMB. It is clear that Suricata will provide the users with more detection abilities based on the application layer. Another usnique feature of the Suricata IDS/IPS is that it can save the information of traffic in a variable in order to use it for a match at a later time. The Suricata IDS was designed to be able to take advantage of the high process power of a system and easily implemented in any computer system. 2.7 Network Scanning The techniques that intruders use in order to attack a computers system are similar to those used by burglars. The burglars inspect every possible entrance of the house that they want to brake in before their attack. They collect information about all the doors of the house like where that door lead to, how difficult is to break in through that door and if that door triggers some kind of alarm. After that the burglar is ready to perform a successful attack and with the less possibilities of being caught. The same thing happens with the attacks performed by intruders that want to enter a computer system. They use specified tools for the collection of information that they will use later during their attack. This process is called network scanning and is usually by intruders that want to discover possible targets (active hosts of the network). The network scanning tools can give important information about the host like services that he uses, the active ports that those services use and information about the applications that run on the host s computer along with the operating system that is implemented. These information are gathered through four network scanning techniques, the network mapping, the port scanning the service and version detection and the operating system detection ( The network mapping is the first process during the attack because it discovers all the possible targets. The targets are hosts that reply to requests made by the intruder that performs a network scanning. Network mapping use ICMP, TCP and UDP protocols in order to create requests for the possible targets. The next step during the intrusion is the port and service scanning. After the port scanning the attacker will know which ports of the target s system are used and for 14

21 what reason. The same thing happens with the service scanning. There are three main techniques for the port and service scanning the connect scan, the half open scan and the stealth scan. During the connect scan the intruder generates a three way TCP handshake in order to establish a connection with the target. The half-open scan on the other hand waits only for the SYN/ACK of the target and then breaks the communication (also known as SYN scan). The stealth scan is more sophisticated methods like fragmentation, flag settings in order to perform an undetected scan. Some popular stealth scans are the SYN/ACK scan, the XMAS scan, the NULL and the FIN scan. The first scanning method can be easily stopped by firewalls or IDSs, the second one which performs uncompleted connections that are not being logged by the host s system can be detected only by systems that use IDS. The stealth scans are far more difficult to be detected, although most IDSs are designed to protect systems against these kinds of scans. The operation systems detection is a process where the intruder gathers information about the operating system of the host. It is also called fingerprinting and is performed by sending packets to the target that have different settings. In this dissertation the Nmap is used as a network scanning tool that will be used for the attack. The next paragraph describes the operation of Nmap along with its basical features Network scanning tools There are many scanning tools that can be used in order to provide information about networks or single hosts. In this dissertation the Nmap scanning tool is used which is described briefly in the next chapter. Bellow some other popular scanning tools are described that can also be used instead of Nmap Nemesis Nemesis is a packet crafting tool designed for UNIX and Windows Microsoft operating systems. It is used as an attack tool by network administrators who want to test the security mechanisms of their networks. Nemesis is a command-line utility 15

22 that can craft packets of the following protocols: DNS, ETHERNET, ARP, ICMP, IP, IGMP, OSPF, RIP, UDP, and TCP. The Nemesis has the ability to produce almost any kind of packet ( Nessus Nessus is a specialised tool that is used for vulnerability assessments over networks and was first presented in It is a ubiquitous scanner that works perfectly with any kind of networks. This vulnerability scanner is connected to a community of users large enough to provide updates when it is necessary (Rogers et al. 2008). 2.8 Practical Experience There are many publications about Intrusion Detection Systems since the early 80s where the first IDS was designed. The most recent of them present testing techniques and tools that can evaluate the performance of the IDS. The most common testing method is to generate attacks by using attack tools (usually packet craft tools). The results should be taken after various kinds of intrusions generated and several changes on the IDS s configuration in order to simulate a great range of attacks. In the An Experience Developing an IDS Stimulator for the Black-Box Testing of Network Intrusion Detection Systems (2003) a basic method for testing an IDS is presented. The paper analyzes the results after the attack on several IDSs. For that reason the attacks are generated by a tool named Mucus which uses the signatures of Snort in order to create traffic that will force the IDSs to generate their alarms. The attack technique in this paper is to randomize the synthetic events of a modeled attack. The results show that most Intrusion Detection Systems fail to detect this kind of attacks (Mutz et al. 2003). In Testing Network-based Intrusion Detection Signatures Using Mutant Exploits another testing method for Intrusion Detection Systems is presented. This paper describes the attack generated by mutant users. These mutant users attack a single host that runs the IDS with several attack techniques. The results from the different kind of attacks are analyzed in order to evaluate the performance of the IDS (Vigna et al. 2004). 16

23 3. System Analysis and Configuration 3.1 Network Analysis This dissertation analyses and describes how Snort, a Network Intrusion Detection System, reacts during the attacks made by attackers that use network scanning tools (Nmap). A specified scenario is necessary in order to demonstrate the operation of Snort under these situations. For that reason in the next paragraphs the architecture of a network is described that will be used for the analysis of Snort reactions. As was mentioned in the previous chapter (Chapter 2) the Network Intrusion Detection Systems, in many cases, can analyse all the traffic that pass through the network s wires. They can also have custom settings that can make their operation more effective according to the characteristics of the network that they protect. The network that will be used in this dissertation is a small enterprise network and it is shown in Figure 3.1. The enterprise network has an ftp server and four hosts. All the clients can have access to the enterprise server and also have access to the internet. In addition the ftp server should be accessible only to the network s clients. Figure3.1. Enterprise Network Architecture 17

24 As it is shown in Figure3.1 the network consists of an enterprise router that is connected to the internet via a modem and between them there is a firewall that protects all the network elements of the enterprise. The router uses two gateways to connect the server and the hosts. The first gateway is dedicated to the server and the other one is dedicated to the clients. The server is the network element of the demilitarized zone (DMZ). The hosts are placed on the internal network which is part of the enterprise network. The purpose of this architecture is to separate the hosts from the two servers in order to provide additional security to the server s subnet. In addition this architecture enables Snort to analyze data traffic in a more efficient way with two sensors, each for every subnet. Since the two subnets have different types of data traffic, the configuration of the two sensors is not the same. An alternative implementation of Snort can be done with a single sensor placed behind the firewall, although a single sensor will affect the performance of the network. The process power of two sensors and their customized configuration outweighs the one sensor. The Figure 3.2 shows the location of the sensors inside the network. DMZ FTP server INTERNET MODEM Snort Sensor Firewall Snort Sensor INTERNAL NETWORK Switch Router Computer Computer Computer Computer Figure3.2. DMZ and Internal Network 18

25 3.2 Security Policy Before the implementation of any security tool it is important to address the security policy of the network. The security policy describes the rules for the behavior of every element of the network. The rules set constraints on the behavior of the users of the network and also control the access of the network from the outside world. The goals of the security policy are: Protect the clients and the services of the network from any kind of malicious attacks and abuse. Establish security mechanisms that will detect and prevent any of these malicious attacks. The enterprise network consists of four clients and one ftp server. All clients are connected to a switch which is connected to the enterprise router (client subnet). The server is also connected to a connected to the enterprise router. This subnet will be the DMZ of the network. The enterprise router is also connected to the internet. All the clients of the network must be able to access the internet. For the protection of the network a firewall and a Network Intrusion Detection System are used. The firewall is placed at the edge of the network and must provide security to both clients and servers of the network. The NIDS will use two sensors one for every subnet. The subnet of the DMZ is: /24 The subnet of the internal network is: / Firewall configuration The firewall must protect both clients and the server of the network. On the other hand it must use rules that will not affect the operation of the ftp server from the clients of the server. The network does not have servers that are reachable from the outside world, for that reason all the well known (0-1023) ports will be closed. There are although 19

26 many other ports above the 1023 that are open for the services used by the clients of the network. In addition, the firewall will must be able to block several web sites that the clients are not allowed to visit. These web sites are and As a result the firewall blocks all the traffic from these two servers by using their IP. The IP address of the miniclip web site is The facebook on the other hand has more than one IP addresses in order to share the traffic to different servers. The IP addresses which are used by the facebook web site are , , , and Router configuration The router enables traffic from the DMZ to the internal network and backwards. On the other hand it blocks any traffic from and to the internet for the ftp server. The clients of the internal network are able to connect to the internet. The name of the interface of the router which is connected to the internet is eth1. The eth2 and the eth3 are the interfaces of the DMZ and the internal network respectively. The configuration for the interface of the enterprise router for the DMZ is: # Accept tcp packets on destination port 21 (ftp) from internal network # The subnet of the internal network is /24 iptables -A INPUT -p tcp -s /24 --dport 21 -j ACCEPT # Block all the incoming packets from the interface eth1 iptables -I INPUT -i eth1 -s -j DROP The configuration for the interface of the enterprise router for the internal network is: # Accept traffic from both interfaces iptables -I INPUT -i eth1 -s -j ACCEPT iptables -I INPUT -i eth2 -s -j ACCEPT 3.5 Test-bed Environment In this dissertation the operation of Snort is presented under the attack by a network scanning tool. The network scanning tool that will be used is Nmap and it will perform various scans in order to test the detection abilities of Snort. 20

27 For that reason a safe test-bed environment is needed. In order to create a safe test bed environment for the test a virtual machine is installed on a computer that will contain two different hosts. The first host will operate the Nmap and the other one will operate the Snort. The software that will be used as the test bed environment is VMware workstation. The VMware workstation has the ability to run virtualised two or more operation systems at the same time. Figure 3.3 shows the user s interface of the VMware workstation. Figure3.3. Test-bed Environment (Vmware Workstation) The test-bed environment of this dissertation requires two individual hosts, for that reason two virtual machines are installed on the VMware workstation. The BackTrack 5 is installed on the first virtual machine because it is an operating system that contains the Nmap. On the other virtual machine the Microsoft s Windows XP is installed. The Snort will be then installed on the virtual machine that contains the 21

28 Windows XP operating system. In Figures 3.4 and 3.5 it is shown the VMware workstation after the installations of Windows XP and BackTrack 5 respectively. Figure3.4. Virtual Machine (Microsoft Windows XP) Figure3.5. Virtual Machine (Backtrack5) 22

29 The use of this test bed environment allows any kind of configuration on the operating systems of the two hosts without warring for any malefaction that may occur after the use of Nmap or Snort. The computer that will run the virtual machines will not be affected from any of the two virtual systems. 3.6 Snort Overview Snort is an Intrusion Detection System designed for network protection (NIDS) and can be used for packet sniffer and packet logger as well. It uses several techniques in order to monitor traffic, set security rules and generate alarms for every possible attack. The Snort has many add-on programs that can provide the user with many different features. It mainly works with the TCP/IP protocol but it can also work with other protocols that are used in networks. Martin Roesch (Marty), the developer of the Snort, wanted to create a security tool flexible and compatible with the majority of the operating systems. He also created a packet sniffer that can layout both hex and ASCII formats of the packets in a solid format. The Snort is a quite flexible tool but it has some system requirements for its implementation. These requirements depend on the size of the network that needs to be protected. The larger the network the more Snort sensors will be used for implementation of the Snort. The alerts generated by Snort can be thousands in a small time space. Disk space and process power are also some of the Snort s requirements in order to work properly. The alerts generated by Snort can be thousands in a small time space (Yang 2010). The disk space is used by Snort for the storage of its alerts; the process power along with the memory depends on the traffic of the network and the selected features that are being monitored. Snort inspects the packets that pass through its sensors and alerts the administrator of the system whenever it finds a malicious behaviour. In order to do that the Snort compares the data traffic with a set of behaviours that are specified as malicious. These specified behaviours can be customised by the rules that are selected by the administrator. The packet decoder is the first stop for the packets that arrive at the Snort s sensors. The decoder identifies the protocol used by the packet and it can also 23

30 trigger the alarm if the protocol is characterized as part of a malicious attack. The packet decoder can also generate the alarm for excessively long packets and errors on protocol s headers. The packet that passes through the packet decoder for parsing reaches the pre-processors. Snort may use more than one pre-processors, each preprocessor is a plug-in which helps the administrator s to inspect the incoming packets in a more detailed way. Without the pre-processors the Snort inspects each packet separately from the other packets. This means that the Snort will be unable to detect sophisticated attacks that use fragmentation techniques. The fragmentation techniques overwrite data in several packets that do not have the expected order. The pre-processors can help the Snort on the detection of these attacks. The final destination of the data is the detection engine where the rules set there by the administrator justify their legitimacy (Paxson 2004) Snort Configuration As was mentioned in the first chapter of this dissertation, no matter how many security mechanisms are installed on the network there will always be some weaknesses. For this reason the configuration of Snort must focus on these security weaknesses in order to eliminate the vulnerabilities of the protected system. The enterprise network that is tested uses two different snort sensors placed strategically inside the network. Their positions ease Snort along with the analysis of data due to the fact that they use different configurations since they analyze different kind of data. The first Snort sensor analyzes the data from and to the ftp server and the second one the data that travel from and to the internal network. The configuration of Snort has many option, is one of the privileges that its user have. There are nine basic steps for the customization of the configuration of Snort. Some of the changes that can be made are critical for the customization, although there are many parts of the configuration that remain the same in most occasions. Here are the nine steps for the configuration of Snort: 24

31 Setting up the network variables This step is used in order to set the variables of the network. In this step actually the Snort is introduced to the network. The only modification has to do with the address of the network that is protected. # Setup the address of the protected network var HOME NET any In the case of the sensor that protects the ftp server # Setup the address of the protected network var HOME NET /24 In the case of the sensor that protects the internal network # Setup the address of the protected network var HOME NET /24 There are many other modifications like the address of the external network, the addresses of any servers that may be installed on the network and even set the ports that are used for services. At the end of this step it is very important to set the path where the Snort rules are inside the system. # Setup the address of the protected network var HOME NET any # Setup the address of the external network var EXTERNAL NET any # Give the path of the folder where the rule files are # Snort is installed on a computer that uses the Microsoft Windows XP so the path needs to be absolute var RULE PATH c:\snort\rules var SO RULE PATH c:\snort\so_rules var PREPROC RULE PATH c:\snort\preproc_rules 25

32 Configuration of the decoder The first process that the packets pass through is the decoding made by the decoder of Snort. Here the underlying protocols of the packets are determined. The decoder saves these data and the same think happens with the location of the payload which is then processed by the preprocessors and the dynamic engine. The decoder generates alerts for anomalies that may occur on the header of the payload or for packets that have the wrong size according to the decoders configuration Configuration of the Base detection engine Here are the rules that are used by the snort in order to work as an NIDS. The detection engine receives the data from the packet decoder and the preprocessors and compares them with the rules from the snort.conf file. The default configuration of the detection engine will generate an alert when the first match is produced after the comparison between packets and rules. There is also the opportunity to generate more than one alert for a packet by enabling the multiple matching mode of Snort. This mode does not stop at the first match but compares the packets with all the available rules and then generates the alarms according to the matches that where caused (there will be no changes at this step, Snort will use the default settings) Configuration of the dynamic loaded libraries In this step the path of preprocessor libraries, preprocessor engine, dynamic rules libraries is defined Configuration of the preprocessors The preprocessors are very important for the well operation of Snort. They are used for reassembling the packets or decoding protocols. In other words they inspect data in order to find packets that their original form has been alternate. A common technique that is used by attackers that they want to hide their tracks is fragmentation. Attackers brake their packets into pieces so it is not detectable by the IDS. The preprocessors gather all the broken pieces together and reassemble the original packet. For the protection of the enterprise network all the preprocessors will be used for maximum security. For example without the use of the preprocessor 26