It iiumiuimmnuu. Pusat Khidrpp; Maklumat Akadendk UNIVERSITI MALAYSIA SARAWAK Kote ßarnprjn HOST-BASED INTRUSION DETECTION SYSTEM (HIDS)

Transcription

1 P. KH IDMAT MAKLUMAT AKADEMIK UNIMAS Pusat Khidrpp; Maklumat Akadendk UNIVERSITI MALAYSIA SARAWAK Kote ßarnprjn It iiumiuimmnuu HOST-BASED INTRUSION DETECTION SYSTEM (HIDS) ZAID AFFENDI BIN ZAINOL (Network Computing) This Project is submitted in partial fulfillment of the requirement for the degree of Bachelor of Computer Science with Honours Faculty of Computer Science and Information Technology Universiti Malaysia Sarawak 2005

2 DECLARATION No portion of the work referred to in this report has been submitted in support of an application for another degree or qualification of this or any other university or institution of higher learning. ( Qi-- /G 4 i-. ), a7cdc- (Zaid Affendi bin Zainol) (Date) ii

3 ACKNOWLEDGEMENTS In the name of God, the Most Merciful and the Most Compassionate, praise to be the one, the sustainer of the entire universe. First of all, I would like to acknowledge my parents, who give me supports and cares from the very beginning. Without their understanding, patience, love and care, I would not be able to complete my system and report. Secondly, I wish to express my sincere thanks and deepest appreciation and gratitude to my supervisor, Mr. Nazim bin Jambli, for his invaluable advice, guidance and continuous assistance in the accomplishment and completion of this project. A special appreciation to my examiner, Puan Azni Haslizan for helping and patiently guide me to complete my thesis. Only God will be able to give the reward for their contributions. Last but not least, I would like to thank to contribution made by all my friends. Their help, and support have motivates me to work extra hard than I use before. A similar thanks is also extended to Nur Edayu Dahlan for helping and inspiring me to complete my thesis. Thank you very much to all of you. iii

4 Pusat Khidmat Makiumat Akademik UNIVERSITI MALAYSIA SARAWAIC Kota Samarahan TABLE OF CONTENTS DECLARATION 11 ACKNOWLEDGMENT 111 TABLE OF CONTENTS iv LIST OF FIGURES ix LIST OF TABLES X LIST OF ABBREVIATIONS X1 ABSTRACT X11 ABSTRAK X111 CHAPTER ONE: AN OVERVIEW 1.0 Introduction Purpose of the Report Problem Statement Project Objectives Project Scope Project Significant Project Potential Outcomes Report Overview 5 CHAPTER TWO: LITERATURE REVIEW 2.0 Introduction 7 iv

5 2.1 Reviewing of Existing System Types of Attacks and Intrusions Reconnaissance Attacks Exploits Denial of Service (DoS) Attacks Implementation of IDS Categories of IDS Host-based IDS Network Based IDS Application Based IDS IDS Models Knowledge based IDS Behavioral IDS Intrusion Detection System Components Sensors Analyzer User interface Strengths of Host-Based Intrusion Detection Systems Analysis of Existing IDS CMDS NetRanger Tripwire Analysis Of Software Used Windows Operating System 29 V

6 2.4.2 Visual Basic Programming Language Conclusion 30 CHAPTER THREE: METHODOLOGY 3.0 Introduction Definitions System Development Life Cycle (SDLC) System Planning System Analysis Conceptual Design System Evaluation Detailed System Design System Implementation System Maintenance Conclusion 37 CHAPTER FOUR: SYSTEM DESIGN 4.0 Introduction System design organization Section 1: General conceptual plan of the system design Section 2: Specific details of State Process (Child Diagram) Section 3: System Workflow and Idea 42 V1

7 4.1.4 Section 4: System Layout and Implementation Design Technical Design Components Architecture - Functional Architecture - Physical Use Case System Use Case Description HIDS Design Features of the System Component of the System IP Monitoring Port Scanning Access IP by Hostname System Architecture Conclusion 54 CHAPTER FIVE: SYSTEM IMPLEMENTATION, TESTING AND EVALUATION 5.0 Introduction Software Implementation System Testing and Evaluation Host-Based Intrusion Detection System Program Opening the Monitoring IP Activities How Intrusion Detected Conclusion 64 Vll

8 CHAPTER SIX: CONCLUSION 6.0 Introduction Objectives Achievement Work Independently Have A Log Have an Interface System Limitation Platform Dependent Host-based Detection Techniques Report Future Work and Recommendation Network-based IDS Platform Independent Automatic Generated Report Security Enhancement Summary and Conclusion 69 BIBLIOGRAPHY 70 viii

9 LIST OF FIGURES Figure 2.1: IDS location between Intranet and Internet 9 Figure 3.1: System Development Life Cycle (SDLC) 32 Figure 4.1: A general conceptual plan for HIDS 39 Figure 4.2: Context Diagram 0 40 Figure 4.3: Process 1 Details 41 Figure 4.4: System Workflow 42 Figure 4.5: Sample Monitoring IP Activities 44 Figure 4.6: Sample Monitoring Possible Intrusion 44 Figure 4.7: Sample of Port Scanner 45 Figure 4.8: Sample of Trace Route 45 Figure 4.9: HIDS Use Case Diagram 47 Figure 4.10: IP Monitoring 50 Figure 4.11: Port Scanning 51 Figure 4.12: Access IP by Hostname 52 Figure 4.13: HIDS System Architecture 53 Figure 5.1: Connection between Host and Client 58 Figure 5.2: Connection between Server and Client (LAN) 58 Figure 5.3: Host gets the IP Activities 59 Figure 5.4: Interface of IP Activities Monitoring 60 Figure 5.5: Command Line to detect Intrusion 61 Figure 5.6: Possible Intrusion Monitoring Interface and Result 62 Figure 5.7: Sample of Log Files 63 Figure 5.8: Port Scanner on local host 63 Figure 5.9: Port Scanner for Remote (Network/Client) 64 ix

10 LIST OF TABLES Table 2.1: Advantages and Disadvantages of IDS 13 Table 2.2: Overview of Computer Misuse Detection System (CMDS) 21 Table 2.3: Overview of Netranger 24 Table 2.4: Overview of Tripwire 28 X

11 LIST OF ABBREVIATIONS IDS Intrusion Detection System HIDS Host-based Intrusion Detection System TCP Transmission Control Protocol IM Instant Messaging CPU Central Processing Unit DOS Denial of Service CMDS Computer Misuse Detection System SAIC Science Applications International Corporation GUI Graphical User Interface DES Data Encryption Standard ACL Access Control List CRC Cycle Redundancy Check PLC Project Life Cycle RAD Rapid Application Development DFD Data Flow Diagram LAN Local Area Network UDP User Datagram Protocol SMTP Simple Main Transfer Protocol IP Internet Protocol ERD Entity Relationship Diagram DFD Data Flow Diagram QOS Quality of System SDLC System Development Life Cycle X1

12 ABSTRACT Host-based Intrusion Detection System (RIDS) is one of alternative approach in computer security that been widely developed by experts in protecting the privacy and integrity in network environment. Nowadays, implementations firewall with HIDS is taking its leap by storm. The collaboration among IDS and firewall has improved and reduce the security problem faces in network. By implementing IDS with firewall, it ensures that prevention and detection roles cover the security management effectively and widely. This project describes a prototype Host-based Intrusion Detection System (HIDS) based on Windows platform. This system supports monitoring IP activities, detecting possible intrusion, and port scanning. Besides that, activities of possible intrusion that been gathered can be save as a log for future references and action for administrator. xii

13 ABSTRAK Pengkalan Sistem Pengesanan Pencerobohan (PSPP) adalah satu daripada alternatif di dalam sekuriti komputer yang kini giat dibangunkan oleh pakar-pakar untuk memastikan keselamatan di dalam rangkaian komputer. Kini, pengunaan ` rrewall' bersama PSPP telah digunakan secara mendadak. Pengunaan ` irewall ' bersama PSPP telah mengurangkan masalah yg dialami di dalam pencerobohan sistem rangkaian. Ia juga memastikan aktivitri mencegah dan mengesan meliputipengurusan keselamatan secara efektif dan meluas. Projek ini menunjukkan satu prototaip untuk Pengkalan-hos Sistem Pengesahan Pencerobohan (PSPP) berdasarkan platform Windows. Sistem ini menyediakan pengawasan aktiviti IF, mengesan pencerobohan, dan pemeriksaan port. Selain itu, maklumat mengenai pencerobohan ini dapat disimpan sebagai log untuk rujukan dan tindakan kepada pendaftar. X111

14 CHAPTER ONE: AN OVERVIEW 1.0 Introduction According to Cort (2004), computer and network intrusions have been with us since the introduction of the computer, but intrusion detection systems are still somewhat new to the market. The first implementation of intruders detection system just started in the early of 90's. Intrusion is known as any intentional event where an intruder gains access that compromises the confidentiality, integrity, or availability of computers, networks, or the data residing on them (ISRC, 2004). Intrusion Detection Systems (IDS) inspects traffic for known attack signatures and issues alerts on detection. There are basically three categories of IDS that being used today. The categories are Host Based Intrusion Detection Systems (HIDS), Network Based Intrusion Detection Systems (NIDS), and Application Based Intrusion Detection Systems. This project will focus on HIDS based on the concept of pattern matching. Pattern matching is based on looking for a fixed sequence of bytes in a single packet. In this system prototype, it will include a specific starting point and endpoint for inspection within the packet. It is an approach and approach to match the pattern of the suspect packet. For instance, the system can specify the TCP flags for packets to be considered. I

15 1.1 Purpose of the Report The purpose of this report is to document the final year project on host-based intrusion detection system or also known as RIDS. It will discuss about host-based intrusion detection system countermeasures that are currently available, review their strength and weakness, and provide some practical recommendations for when and how to use RIDS. This report will include the findings on HIDS. It also will describe about HIDS design and systems prototype. 1.2 Problem Statement According to Grimes (2002), many people thought that having a firewall and antivirus scanner is good enough. They believe with firewall and antivirus software, the computer is secure enough. However, intruders are deviously clever and adaptive. So an Intrusion Detection System (IDS) is a must have tool for any serious in-depth computer plan. Antivirus scanner captures known worms, viruses, and Trojan horses, while firewall stop port intruders. An HIDS can sniff network packets to see what really happening (Roger, 2002). For example, an IDS can detect whether your port 80 traffic is a Web request or an Instant Messaging (IM) file transfer. Firewalls and scanners can't stop a buffer-overflow attack or recognize the latest SQL injection attack, but an IDS is able to recognize and respond to attacks of these types. Advanced IDS can drop the packet before it causes harm or can modify a security parameter so that the malicious packet becomes harmless. 2

16 Enterprise networks are facing ever-increasing security threats from worms, port scans, DDoS, and network misuse, and thus effective monitoring approaches to quickly detect these activities are greatly needed. So an intrusion detection system (IDS) is the additional technology that can be valuable enhancement. By monitoring and implementing HIDS, it will fill the gaps left by the other security tools. 1.3 Project Objectives The main objective of this project is to minimize or reduce the security, intrusion and hacking problem faces in network environment. Other objectives of the project are: " Minimize or reduce the security, intrusion and hacking problem faces in network environment. " Produce a prototype of HIDS. The RIDS can be use at the outside or inside the firewall independently. " Have a log to monitor intruder activities. It should provide tool to captures data for analysis. Then it should produce an appropriate action to notice the system administrator about the information gathered. " Have an interface to make it more user-friendly and convenient. 3

17 1.4 Project Scope The scopes of the project are: " This project will be run on Windows platform because it is the most widely used platform in the world. Besides that, to make the implementation of the HIDS successful, Visual Basic programming language will be used to implement the system prototype. " This system will work independently on a host based system to monitor and gather intruder's activities. " This system will focus more on collecting intruder's activities in detail, not to detect the intruders. 1.5 Project Significant I choose to implement HIDS project because it's needed in our network environment nowadays. This project will minimize the risk of internet based attack. Besides that, it will also prevent the attackers from gathering or manipulating important data. This project will be able to collect intruder's activities and at the same time it should take appropriate actions such as log the activities and inform the system administrator about the unauthorized activities. Therefore, it will provide the integrity of the system. 4

18 1.6 Project Potential Outcomes The expected outcomes from this project are host-based IDS that can: " To provide a log about intruders activities " To breakdown a few types of denial of services attack " Produce appropriate action to inform the system administrator 1.7 Report Overview This is brief information about this report. All the chapter and related topics will be described in the later sections. For the first chapter, this report will gives an overview and some brief information what the project is all about. It covers the purpose of the reports, problem statement, project objectives, project significances, and project potential outcomes. Chapter Two is the literature review, which covers some security topics related to the topics especially the existing IDS. This chapter also states the advantages and disadvantages of the IDS. In this chapter, brief information of software used is also included. Chapter Three provides information about the systematic approach used to build the system prototype. System Development Life Cycle is used in order to complete the project objectives. Chapter Four will describe the system design of the HIDS prototype. Firstly, the typical attack scenarios such as intrusion and malicious actions will be described briefly. Then, this chapter will discuss in detail about the system prototype design for the HIDS. 5

19 In Chapter Five, all process involved in the system implementation are covered. This chapter will discuss the software implementation, system testing and evaluation, and implementation issues on the development of HIDS prototype. The overall conclusion of the project is represented in Chapter Six. In this chapter, it will discuss on the project status based on its limitation and whether the project has achieve its target and objectives. The system's future works and recommendation are included in this chapter. 6

20 CHAPTER TWO: LITERITURE REVIEW 2.0 Introduction Host based intrusion detections systems (HIDS) relatively a new technology. As HIDS evolves, it is quickly becoming a key component of a security policy. HIDS is also becoming a valuable and sometimes inexpensive tool for PC users that are concerned with the security of their systems. HIDS is designed to monitor malicious events aimed at host machines. This chapter will review the existing system. It will focus on types of attack and intrusion in IDS. Then it will discuss about the implementation of IDS, categories of IDS including their advantages and disadvantages, and IDS models. It also will review existing IDS in the market nowadays. 2.1 Reviewing Of Existing System Intrusion Detection System (IDS) In traditional IDS deployment, most organization installed these devices in the perimeter either between the router and firewall or placed it outside the router. Having IDS in either of these locations, it will have it own functions and provide a tool that can captures data for analysis. 7

21 2.1.1 Types of Attacks and Intrusions According to Graham (2004), he defines three categories of attacks and intrusions: " Reconnaissance attacks " Exploits " Denial of services (DoS) attacks Reconnaissance attacks Reconnaissance attacks includes port scans, ping sweeps, recons, DNS zone transfers, and public web server indexing to find holes in the network through which they can gain access Exploits Exploits is using bugs or hidden features in applications, servers, and operating systems which allow unauthorized access to the system Denial of Service (DoS) Attacks Dos attacks are typically indiscriminate attempts by an attacker to crash systems or overload network connections, memory buffers, and CPU registers with the intent of denying access to your system by everyone else. While these categories all seem to address attacks from the 8

22 outside, we must not forget that attacks and intrusions can come from both outside and inside the organization. Hackers, industrial spies, and other people may try to compromise the security of the system from the outside, but disgruntled employees or individuals who have gained physical access to the organization's systems may similarly compromise security Implementation of IDS In general, IDS can be implemented in the following locations as shown in this simple diagram (Figure 2.1): FIREWALL INTERNET GS3 ýý ý -ý ý IDS I IDS 2 ý ýýýý Figure 2.1: IDS location between Intranet and Internet (Cort, A. (2002). Algorithm based approach to intrusion detection, SANS Institute) i. IDS I can detect attacks against the firewall ii. IDS 2 detects traffic which has penetrated the firewall iii. IDS 3 represent implementation of one or more IDS at various nodes throughout the network, and can detect attacks by insiders 9

23 2.1.3 Categories of IDS There are basically 3 main types of IDS being used today: " Host based (looking for instance at system logs for evidence of malicious or suspicious application activity in real time) " Network based (a packet monitor) " Application Based IDS (monitor only specific applications) Host Based Intrusion Detection Systems (RIDS) Host-based systems were the first type of IDS to be developed and implemented. These systems collect and analyze data that originate on a computer that hosts a service, such as a Web server. Once this data is aggregated for a given computer, it can either be analyzed locally or sent to a separate/central analysis machine. One example of a host-based system is programs that operate on a system and receive application or operating system audit logs. These programs are highly effective for detecting insider abuses. On the down side, host- based systems can get unwieldy. With several thousand possible endpoints on a large network, collecting and aggregating separate specific computer information for each individual machine may prove inefficient and ineffective. Possible host-based IDS implementations include Windows NT/2000 Security Event Logs, RDMS audit sources, Enterprise Management systems audit data (such as Tivoli), and UNIX Syslog in their raw forms or in their secure forms such as Solaris' BSM; host-based commercial products include RealSecure, ITA, Squire, and Entercept, etc. 10

24 Network Based Intrusion Detection System (NIDS) NIDS are used to monitoring the activities that take place on a particular network, Network- based intrusion detection analyzes data packets that travel over the actual network. These packets are examined and sometimes compared with empirical data to verify their nature: malicious or benign. They have network interface in promiscuous mode. Because they are responsible for monitoring a network, rather than a single host, Network-based intrusion detection systems (NIDS) tend to be more distributed than host-based IDS. Instead of analyzing information that originates and resides on a computer, network-based IDS uses techniques like "packet-sniffing" to pull data from TCP/IP or other protocol packets traveling along the network. This surveillance of the connections between computers makes network- based IDS great at detecting access attempts from outside the trusted network. In general, network-based systems are best at detecting the following activities: " Unauthorized outsider access: When an unauthorized user logs in successfully, or attempts to log in, they are best tracked with host-based IDS. However, detecting the unauthorized user before their log on attempt is best accomplished with network-based IDS " Bandwidth theft/denial of service: these attacks from outside the network single out network resources for abuse or overload. The packets that initiate/carry these attacks can best be noticed with use of network-based IDS 11