How To Write A Network Intrusion Prevention System

Transcription

1 THESIS THE USE OF INTRUSION PREVENTION SYSTEM TO INCREASE COMPUTER SECURITY NAJIB BIN LIMUN UNIVERSITI TEKNOLOGI MARA NOVEMBER 2005

2 THESIS

3 NOVEMBER 2005 DECLARATION I hereby declare that the work in this report is my own except for quotations and summaries which have been acknowledge. 24 th JANUARY 2006 NAJIB BIN LIMUN

4 ACKNOWLEDGEMENT In the name of Allah, the Most Gracious and Most Merciful. All praise be to Allah, for all the bless that he gave, finally I can complete my project upon the date end. First and foremost, I would like to express my deepest appreciation and thanks to my dedicated supervisor, Mr. Adzhar Abdul Kadir, for his guidance, ideas, supporting and advice in completing the project. I am very lucky to have him as a supervisor as he is an experience lecturer. Special thanks to other lecturers and staffs who have also help me in completing my project. Not forgotten, my friends who have willingly to share their knowledge with me. I feel very fortunate to have the advice and guidance of many talented people who have many experience and knowledge in all aspect of network, open source and security tools. They shared their ideas with me. Lastly, I would like to express my gratitude to my beloved parent who have supporting me. These are the people who have always giving me a moral support in completing my project. Thank you for all the person who have help me. May Allah bless you. i

5 ABSTRACT Network intrusion prevention systems provide an important proactive defense capability against security threats by detecting and blocking network attacks. This task can be highly complex and traditional firewall system are currently not capable of handling fast attack through the operating system. The problems arise when many exploits attempt to take advantage of weaknesses in every protocols that are allowed through our perimeter firewalls, and once the Web server has been compromised, this can often be used as a springboard to launch additional attacks on other internal services. Once a rootkit or back door has been installed on a computer, the hacker has ensured that he will have unfettered access to that machine at any point in the future. Thus, iptables also called as Netfilter can also be implement as an intrusion prevention system. Iptables works by filtering the traffic flow between your computer and the Internet. It can limit access to and from the Internet to only specific computers on your network. It can also limit the type of communication, selectively permitting or denying various Internet services. Hence, to harden the iptables rule, another tool need to be apply to work with the iptables rule script. The psad tool is good in implementing some additional feature like an alert and logfile analysis. ii

6 TABLES OF CONTENT ACKNOWLEDGEMENT ABSTRACT TABLE OF CONTENTS LIST OF FIGURES i ii iii v CHAPTER 1: PROBLEM DESCRIPTION 1.1 Introduction Problem Statement Project Objectives Project Scope Project Significant 8 CHAPTER 2: LITERATURE REVIEW 2.1 Introduction History of Intrusion Prevention System Network based Intrusion Prevention Host based Intrusion Prevention Server based Intrusion Prevention Network Associates Intrusion Pevention Inline Network Intrusion Detection System Layer Seven Switches Application Firewall /IDS Hybrid Switches Deceptive Applications Related Works Conclusion 26 iii

7 CHAPTER 3: METHODOLOGY 3.1 Introduction Data Collection Project Methodology Planning Analysis Development Iptables Module Firewall rule Implementation and Testing Implementation on Linux Fedora Core Conclusion 32 CHAPTER 4: RESULTS AND FINDINGS 4.1 Introduction Data Log Data Capture Testing with PHLAK Testing with Metasploit Framework Conclusion 42 CHAPTER 6: CONCLUSION AND RECOMMENDATIONS 6.1 Conclusion Recommendations 45 APPENDIX 46 REFERENCES iv

8 LIST OF FIGURES Figure 2.1 Figure 2.2 Figure 2.3 Figure 2.4 Figure 2.5 Figure 2.6 Figure 2.7 Figure 2.8 Figure 2.9 Figure 4.1 Figure 4.2 Figure 4.3 Figure 4.4 Figure 4.5 Figure 4.6 Figure 4.7 Figure 4.8 Figure 4.9 Figure 5.0 Figure 5.1 Figure 5.2 Figure 5.3 Integrated IPS Deployment Inline Network IDS architecture Inline Network IDS operation Packet Scrubbing Layer Seven Switches Application Firewall/IDS Hybrid Switches Deceptive Applications operation Deceptive Applications response to attacker Nmap scanning Nmap sscanning Queso Strobe Nmap scanning countermeasure Nmap scanning countermeasure Nmap scanning countermeasure Nmap scanning countermeasure Queso countermeasure strobe countermeasure Metasploit Framework System logfile Iptables analysis v

9 CHAPTER 1 INTRODUCTION 1.1 INTRODUCTION Today's knowledgeable hackers have advanced well beyond scanning for open ports on network firewalls and are now targeting applications directly. They have devised sophisticated attacks that easily circumvent traditional intrusion detection systems (IDS) and network firewalls. In a recent survey commissioned by VanDyke Software, some 66 per cent of the companies who responded said that they perceive system penetration to be the largest threat to their enterprises. The survey revealed that the top eight threats experienced by those surveyed were viruses (78 per cent of respondents), system penetration (50 per cent), DoS (40 per cent), insider abuse (29 per cent), spoofing (28 per cent), data/network sabotage (20 per cent), and unauthorised insider access (16 per cent). This trend has given rise to two different types of next-generation security products --Intrusion Prevention Systems (IPS) and Application Firewalls. Both IPS products and application firewalls are capable of blocking attacks that bypass traditional firewalls. Hence both have been successfully deployed in some of the largest networks in the world. Whilst it is true that firewalls, routers and even Intrusion Detection System devices all have intrusion prevention technology included in some form, it is believe that there are sufficient grounds to create a new market sector for true Intrusion Prevention System (IPS). These systems are proactive defence mechanisms designed to detect malicious packets within normal network traffic (for example, something that the current 1

10 breed of firewalls do not actually do) and stop intrusions, blocking the offending traffic automatically before it does any damage rather than simply raising an alert. Within the IPS market place, there are two main categories of product: Host IPS and Network IPS (Neil Desai, February 2003). In Host IPS, the Host IPS relies on agent that are installed directly on the system that will be protected. It binds closely with the operating system kernel and services closely in order to prevent the attacks as well as log them. While in Network IPS (NIPS), it is about the combination of a standard IDS, an IPS and a firewall. Sometimes, it is call as an In-line IDS or Gateway IDS. The NIPS has at least two network interfaces, one is design as internal and another one as external. As packet appeared at either interface, they are passed through the detection engine for inspection whether the packet should be transmit or drop. When the NIPS detect a malicious packet, rather than raising an alert, it will discard the packet and mark that flow as bad. Many people do not realize that iptables can also act as an Intrusion Prevention System. The iptables firewall or also referred as Netfilter is the default firewall tool for Linux operating system. Iptables is generally considered to be more complex than ipchains. However, iptables provide more powerful and flexible feature. The iptables feature works by having IP packets which is network data that enter or leave the firewall computer, traverse a set of chains that define the tasks that are done with the packet. Each rule that are add essentially does both of the following ; (1). Checks if a particular criterion is met such as that a packet requests a particular service or comes from a particular address. (2). Takes an action (such as dropping, accepting, or further processing a packet). (Oscar Andreasson, Iptables Tutorial, 2005) Different set of rules are implemented for different types of tables. However, most of the rules you create will relate to the filter table. A lot of great features are built into iptables. Some of the features are as a transparent proxy, port forwarding and intrusion prevention system. 2

11 1.2 PROBLEMS STATEMENT The implementation of intrusion prevention system (IPS) provide many advantages in the network security development. Rather than using the tradisional firewall and intrusion detection system, it is better to use intrusion prevention system as a new generation protection for our network environment. The number of vulnerabilities continue to increase, with the average time from vulnerability announcement to actual exploit release decreasing, further compounding the difficulties in ensuring effective security patching to protect the network. By the time the Intrusion Detection System (IDS) has detected the offending packet, raised an alert and transmit the TCP Resets, the payload of the exploit has been already delivered. It shows how fast an attack can be done on the network system that are poorly protected. From the review on other research paper and observation, it can be found that some of the usual challenges faced by the network administrator is how to maintain the security level of their network environment. Lack of maintenance and observation on network performance can lead to the network failure due to too many trojans and backdoor in the system. Security is increasingly top of mind for IT directors and network managers.this is because organizations look to them to protect the network s critical resources and sensitive information. They are responsible for ensuring that authorized users are accessing only the information they should and preventing infiltration of their corporate networks by unauthorized individuals. In recent years, however, this task has grown increasingly more difficult due to a variety of factors. The number of users and the ways in which they access the network continues to expand, making it harder to tightly control and opening up many avenues for 3

12 inappropriate use of resources. The usage of peer-to-peer protocol consume a high bandwidth of data stream and could cause traffic congestion. Besides, the usage of P2P (peer-to-peer) network has become the popular way of downloading movies, MP3 and other related media as it allowed peering and seeding. These are the new ways on how the virus, trojan and hackers try to attack our network environment. These protocols should be blocked from accessing the Internet. The quantity and complexity of attacks continues to grow, often exploiting vulnerabilities in the application-layer that require sophisticated attack detection and analysis to identify and mitigate. Hacking or attacking tools are widely available on the Internet and have become significantly less complicated (a lot of tools even have their own GUI s), making it possible for almost any Internet user to download and run an exploit against an organization. For some novice hacker, they used to attack their own inbound connection in order to gain a knowledge on hacking. Attacks increasingly target Windows components, rather than server software, which translates into more potentially vulnerable systems. As Windows system is full of vulnerability, it needs frequent patches. A good network security policy should be setup on the server to protect the clients which use Windows as an operating system. While firewalls are certainly the first-line of defense and an absolute requirement for any company connecting to the Internet, organizations have realized that firewalls cannot be the only line of defense. As a result, most organizations have adopted a layered approach to network security to try to minimize the risks to their critical assets. A key layer, specifically when it comes to attack protection and mitigation, is an intrusion prevention system. However, not all systems that claim they can perform intrusion prevention are the same. (Juniper Networks, June 2004) As a result, it is imperative for us to understand the implications of deploying an intrusion prevention system in the network and the problems faced by the communities that were involved directly in security area. To develop and implement a successful 4

13 intrusion prevention system using iptables, it is important to identify the process and function of intrusion prevention system. 5

14 1.3 PROJECT OBJECTIVES The objectives of this project are as follows ;- 1) Implement iptables as an intrusion prevention system based on content based filtering. 2) Analyze the output from several attack that try to reach the target computer. 1.3 PROJECT SCOPE The implementation of the intrusion prevention system focuses on those who are involve directly in security area and the students who want to know more about the network security area. This project emphasizes more on the implementation of iptables as an intrusion prevention system rather than developing brand new system. This is because the intrusion prevention system is using iptables, which is an open source tools that was already built in Linux. The implementation focused on the firewall rules and script to suit the needs and requirements. The tools that are used to implement the intrusion prevention system based on content-based filtering. Network intrusion prevention system is implemented and tested on two different platforms, which are PHLAK-0.3, debian based attacking tools and Redhat Fedora Core PHLAK act as an attacker while the Fedora will act as an target computer which will be implement with the iptables-intrusion prevention system. Thus, we will also use the Metaesploit Framework tool to test the vulnerability of the iptables-ips towards some exploit. Due to this, the hardware specifications for the target computer are as follows : AMD Athlon XP MB of RAM 6

15 40 GB of hard disk 3Com Etherlink XL 10/100 NIC For this project, we analyzed the result of the intrusion prevention system based on the data control, data capture and alerting. 7

16 1.4 PROJECT SIGNIFICANT Contribution to the Lecturers Lecturer can use the development of the intrusion prevention system as one of the way to protect the computer by changing the iptables function as host based IPS. Besides, it can be implement in the faculty network environment to ensure that the faculty system is well protected from interval and external attack. Later, lecturers can compile their own customize firewall script for their private computer Contribution to the Students Iptables is a powerfull tool for IP filtering firewall. Many students may not concern on protecting their Linux machine because they assume that Linux is very secure. However, there are many exploits and attacks that can make Linux vulnerable to the hackers. Hence, we must take the oppurtunities to implement such system that can protect our whole network environment and system. Students can learn to utilize all the tools that are provided in the Linux distribution as some of them may become a powerfull usefull tools. Customizing the iptables firewall script can change the overall function of the tools. 8

17 CHAPTER 2 LITERATURE REVIEW 2.1 INTRODUCTION This chapter presents the related literature on several study on intrusion prevention system and review on some description of some known or similar project or issues based on intrusion prevention system. 2.2 HISTORY OF INTRUSION PREVENTION SYSTEM An intrusion prevention system (a computer security term) is any device which exercises access control to protect computers from exploitation. "Intrusion prevention" technology is considered by some to be an extension of intrusion detection (IDS) technology, but it is actually another form of access control, like an application layer firewall. (Wikipedia, the free encyclopedia) Intrusion prevention systems were invented independently by Jed Haile and Vern Paxon to resolve ambiguities in passive network montoring by placing detection systems in-line. A considerable improvement upon firewall technologies, IPS make access control decisions based on application content, rather than IP address or ports as traditional firewalls had done. (Wikipedia, the free encyclopedia) Some time later IPS was commercialized by One Secure which was eventually acquired by NetScreen Technologies which was in turn aquired by Juniper Networks in 9

18 2004. As IPS systems were originally a literal extension of intrusion detection systems, they continue to be related. (Juniper Networks, 2004) Based on Juniper Network, Intrusion Prevention System Buyer s Guide (2004), it stated that prior to conducting a feature-by-feature comparison, decision-makers should frame their evaluation using the following five criteria. an effective intrusion detection and prevention device should: 1. Facilitate investigations for quick incident resolution. A key value of an intrusion prevention system is how quickly and effectively it enables the resolution of security incidents. Obviously the goal is to prevent an attack before it ever reaches its destination, however, administrators know that until they have an idea of what is going on in the network it is very difficult to create a security policy to effectively prevent attacks. As a result, it is important that the intrusion prevention system provide you with visibility into the network activity, both at the network and application-level, so that you can understand what is going on and quickly move to remediate incidents and develop a strong security policy to prevent future attacks of the same nature. It is important that the solution enables both a quick summary of the most important types of events, as well as a way to quickly drill into the raw data and manipulate it to analyze individual incidents. Generally when an attack occurs, you only have a single data point from which you must extrapolate what is going on and figure out how to contain and eradicate the attack. An intrusion prevention solution, however, should make it easy for you to take that data point and correlate it with others to quickly achieve a clear understanding of exactly what is going on, so that you can take corrective action and protect your resources. 2. Deliver ease of use. The ease of use of an intrusion detection and prevention system translates directly into greater control and a higher degree of security. If the system enables administrators to quickly view pertinent, critical information and make adjustments, network managers can ensure the network is efficiently protected from the latest threats and the most up-to-date 10

19 security policy is in force. If a device is hard to control and understand, administrators are going to waste time trying to find the information they need to do their job. Quick security policy definition and easy, global updates from a single, centralized location ensure that enterprise IT teams scattered around the globe can have a comprehensive, real-time view of the system and the network. Providing a granular level of control in an intuitive manner to security managers, not only ensures that the system meets the company s specific security requirements, but also that valuable IT time and resources are not misspent or wasted. Simply put, a complicated and unintuitive IPS system adds little-to-no value to a corporation s information security program. 3. Provide comprehensive protection. It is important to understand that no solution can protect against absolutely all attacks. Due to the dynamic attack landscape, it is impossible to predict and protect against everything that could potentially be used against a network. The inherent complexity of network traffic, which includes the vast number of protocols at both the network (IP, TCP, UDP, ICMP, etc.) and application (HTTP, FTP, SMTP, DNS, POP3, IMAP, etc.) layers, provides attackers ample vulnerabilities to exploit. Combine the inherent complexity with the fact that attacks come in different shapes and forms, and attackers have a virtual buffet to choose from when they are attacking your network. The key is to minimize your exposure to attacks. As a result, the comprehensiveness of protection provided by an intrusion detection and prevention system is critical to its ability to help organizations maintain an acceptable risk level. The solution must support a broad range of protocols and protect against a diverse set of attack types to provide value. 4. Prevent an attack. Whether an intrusion detection and prevention system can stop the attack from ever reaching its victim is the cornerstone to its prevention capabilities. How effective is an intrusion detection system that has to rely on another system to try to prevent an attack? The answer is obvious, but many intrusion detection products do just that, sending a request to a firewall or even the victims themselves to try to end the attack. All of these mechanisms come after the attack has already reached the victim, so even when 11

20 successful, they require the network administrator to investigate exactly how much the attack was able to do before it was stopped. Any device that introduces latency to the prevention response, is not able to offer true prevention. A truly effective solution can actively prevent attacks during the detection process and drop the malicious traffic. This ensures it never reaches its intended victim, keeping the enterprise network and sensitive, mission-critical data safe and secure. 5. Fit seamlessly into the network. In today s highly distributed, global enterprises, the intrusion detection and prevention solution must be both easy to deploy and meet the performance requirements for the network segment it is protecting. Companies simply cannot spare the time and resources required to update each individual device within the corporate network every time a change is made to the enterprise s security policy or a new attack signature is released, so the system should offer some centralized controls that simplify the installation and ongoing maintenance. Nor can an organization afford to re-architect the network or suffer any performance degradation when the system is deployed. As a result, the system must process traffic quickly, make security decisions instantly, and present that information to the network manager in a timely fashion, ensuring the administrator has a real-time view of the system at all times. A slow system that cannot keep up with the rapid flow of network traffic can mean missed attacks and an increasingly vulnerable corporate network. The device needs to perform in an optimal manner, so that the administrator knows exactly what is going on in the network at any given time. 2.3 NETWORK-BASED INTRUSION PREVENTION Network intrusion monitors are attached to a packet-filtering router or packet sniffer to detect suspicious behavior on a network as they occur. They look for signs that a network is being investigated for attack with a port scanner, that users are falling victim to known traps like.url or.lnk, or that the network is actually under an attack such as through SYN flooding or unauthor ized attempts to gain root access (among other types of attacks). (Ulf T. Mattson, 2004) 12

21 Based on user specifications, these monitors can then record the session and alert the administrator or, in some cases, reset the connection. Some examples of such tools include Cisco s NetRanger and ISS RealSecure as well as some public domain products like Klaxon that focus on a narrower set of attacks. (Neil Desai, February, 2003) 2.4 HOST-BASED INTRUSION PREVENTION As with Host IDS systems, the Host IPS relies on agents installed directly on the system being protected. It binds closely with the operating system kernel and services, monitoring and intercepting system calls to the kernel or APIs in order to prevent attacks as well as log them. (Kostantinos Sinidis, Institute of Computer Science, Foundation for Research and Technology Hellas, 2004) It may also monitor data streams and the environment specific to a particular application (file locations and Registry settings for a Web server, for example) in order to protect that application from generic attacks for which no signature yet exists. 2.5 SERVER-BASED INTRUSION PREVENTION These tools analyze log, configuration and data files from individual servers as attacks occur, typically by placing some type of agent on the server and having the agent report to a central console. Some examples of these tools include Axent s OmniGuard Intrusion Detection (ITA), Security Dynamic s Kane Security Monitor and Centrax s entrax as well as some public domain tools that perform a much narrower set of functions like Tripwire which checks data integrity. Tripwire will detect any modifications made to operating systems or user files and send alerts to ISS' RealSecure product. Real-Secure will then conduct another set of security checks to monitor and combat any intrusions. (Neil Desai, February, 2003) 13

22 2.6 NETWORK ASSOCIATES INTRUSION PREVENTION Combining Best of Breed Host and Network IPS technology results in a more comprehensive and robust defensive posture, meaning fewer successful attacks, more efficient use of scarce security resources and lower operating costs than simply deploying one technology or the other. An intrusion or compromise consists of multiple stages: Reconnaissance, Scanning, Gaining Access, Maintaining Access, and Clearing Tracks. Although both Host and Network IPS have the ability to prevent each stage, both technologies are not equally adept at detecting and blocking each stage (Figure 1). Integrating the strengths of each architecture provides a solution whose sum is greater than its parts. (CISCO Systems) Figure INLINE NETWORK INTRUSION DETECTION SYSTEM Although it is called as Inline Network Intrusion Detection System but it performs as an intrusion prevention system. Most NIDS would be configured with two NICs, one for management and one for detection (Figure 2). The NIC that is configured for detection 14

23 usually does not have an IP address assigned to it, making it a stealth interface. Since it does not have an IP address assigned to it no one can send packets to it or cause the NIDS to reply using that interface. (McAfee Network Protection Solution) Figure 2 The inline NIDS works like a layer two bridge, sitting between the systems that need to be protected and the rest of the network (Figure 3). Figure 3 All traffic will pass through the inline NIDS. Unlike a regular bridging device though, the inline NIDS will inspect the packet for any vulnerabilities that it is configured to look for. If a packet contains a piece of information that trips a signature the packet can be forwarded or dropped and either logged or unlogged. Hogwash can take it a bit further 15

24 though: it has the added ability to rewrite the offending packet(s) to something that won t work, a procedure known as packet scrubbing (Figure 4). This type of IPS is useful if you don t want the attacker to know that their attacks are unsuccessful or if you want the attacker to continue to attack one of your systems in an attempt to gather more evidence. It is also useful when deploying a honeynet so that only the outbound traffic, from the honeynet, is scrubbed. (McAfee Network Protection Solution) Figure 4 An inline NIDS offers the great capabilities of a regular NIDS with the blocking capabilities of a firewall. As with most NIDS, the user can monitor, in this case protect, many servers or networks with a single device. This can be both a blessing and a curse. If the system were to fail or crash the traffic would not get through the device. (ISS Guard actually fails open when the product crashes). If you are concerned about uptime and SLAs, this might cause a big issue for your network. These IPSs will feel most comfortable in the hands of security teams that already deal with NIDS. Because these IPSs are variants of existing NIDS, writing rules for them is very easy and offers a way to catch new attacks. To block unknown attacks with a signature-based inline NIDS, you would have to have some generic rules, like looking for NOOP sleds. This does not, however, stop all new attacks. In the case of a protocol 16