1 DETECTING AND ANALYZING NETWORK ATTACKS USING VIRTUAL HONEYNET By NUR ATIQAH BT. HASAN In partial fulfillment of requirement for the BACHELOR OF SCIENCE (Hons.) IN DATA COMMUNICATION AND NETWORKING Major Area: Network Security Approved by the Examining Committee: Pn. Rozita bt. Yunos Project Supervisor En. Mohd Ali bin Mohd Isa Examiner UNIVERSITI TEKNOLOGI MARA SHAH ALAM, SELANGOR MAY 2006
2 DETECTING AND ANALYZING NETWORK ATTACKS USING VIRTUAL HONEYNET By NUR ATIQAH BINTI HASAN ( ) A project paper submitted to FACULTY OF INFORMATION TECHNOLOGY AND QUANTITATIVE SCIENCES UNIVERSITI TEKNOLOGI MARA In partial fulfillment of requirement for the BACHELOR OF SCIENCE (Hons) IN DATA COMMUNICATION AND NETWORKING Major Area: Network Security Approved by the Examining Committee: Pn. Rozita bt. Yunos Project Supervisor... En. Mohd Ali bin Mohd. Isa Examiner
3 CERTIFICATION OF ORIGINALITY This is to certify I am responsible for the work submitted in this project that the original work is my own except as specified in the reference and acknowledgement and the original work contained herein have not been taken or done by unspecified sources or persons.... MAY 2006 NUR ATIQAH BINTI HASAN ii
4 ACKNOWLEDGEMENT Assalammualaikum w.b.t By the name of Allah, the Most Gracious and the Most Merciful. First of all, Alhamdulillah and the most gratitude to the Mighty Allah, the One and Only One for giving and led me a great chances and ability to accomplished my final year project for this year About three years doing my degree, now this is the time to me to proof myself with the hard and patient to contribute what I had done for my future with knowledge and experienced. Without a full commitment and guidance from my supervisor, Puan Rozita Yunos I would not finished this project. Special thanks to her because gave me an ability to work and co-operated with her. I also like to thank to Associates Professor Dr. Saadiah binti YAhya and En. Mohd. Adzhar Abd Kadir for giving me advises and guidance from the beginning of the project. Thanks also to my examiner, En. Mohd. Ali Mohd. Isa for his guidance and support. Thanks to Puan Salmah Abd Aziz for giving me permission doing my research at our faculty. Special thanks to my beloved mum and dad, my family for their support and understanding me along I ve been a student. Last but not least, thanks to all lecturers and all my friends for helping me in completing my research project. Wassalam iii
5 ABSTRACT The security concern is the most important things about a networking environment and computer. To know how secure our computer and network, we must doing a study on how it can be work and defense it from any malicious attack. A virtual honeynet is a technology is designed to capture and give information from a bad guy. Many of the honeypot is designed with the open source operating system. Therefore, this project is made and running with Windows environment operating system that matching with the real network and operating system used at PSMB. We will be captured the unknown activities in the real network. This virtual honeynet will be set up in one single machine by using Honeywall as a tool to capture an unknown activity at the network. Then, we will be analyzing the data that we had captured. Here, we will be focusing only at PSMB network and only captured the port attacks. iv
6 TABLE OF CONTENTS PAGE CERTIFICATION OF ORIGINALITY ii ACKNOWLEDGEMENT iii ABSTRACT iv TABLE OF CONTENTS v LIST OF FIGURES x LIST OF TABLES xi LIST OF GRAPHS xi 1.0 INTRODUCTION 1.1 Background Problem Statement Objectives of the Research Scope of the Research Significance of the Research Organization of the Research LITERATURE REVIEW 2.1 Introduction What is a hacker? Honeypot What is honeypot and what are the types? Production Honeypot Research Honeypot Value of Honeypot 9 v
7 2.3.3 Classes of Honeypots Low-interaction honeypot High-interaction honeypot Honeynet Virtual Honeynet Self-Contained Virtual Honeynet Hybrid Virtual Honeynet IDS-Intrusion Detection System Types of IDS Network Intrusion Detection System Host-based Intrusion Detection System System Integrity Verifiers Log File Monitor Similar Studies A Study of Possible Attacks Against FTMSK Network Honeypots in Windows Environment Using Honeypot to Detect Internal Attacks at FTMSK Usage of Honeypot for Detection and Analysis of 18 Unknown security Attacks A Honeypot Architecture for Detecting and Analyzing Hands in Honeypot Honeypots and Honeynets Security through Deception Monitoring VMware Honeypots Honeypotting with VMware basics Conclusion 22 vi
8 3.0 METHODOLOGY 3.1 Introduction Knowledge Attainment Data Collection Primary Data Secondary Data Planning, Design and Analyzing Planning Selecting hardware for the machines Selecting honeypot tool Selecting virtual machine for deploying 27 virtual honeynet Design Analyzing Analyzing Honeywall CDROM Analyzing VMware Workstation Implementation and Data Collection Implementation Hardware installation Software installation and configuration Microsoft Windows XP Pro VMware Workstation Honeywall Honeypot (Windows 2000 Server) Data Collection 36 vii
9 3.5 Data Analysis and Findings Documentation Conclusion FINDING AND ANALYSIS Introduction Data Collection Network Traffic Network traffic captured on weekday Network traffic captured on weekend Ports Attacked Types of Attacks Data Analysis Network Traffic Analysis Network traffic on weekday Network traffic on weekend Ports Attacked Analysis Port 137 Netbios Name Service Port UPnP Simple Service Discovery 53 Protocol Port NETBIOS Datagram Service Port 445 Microsoft Domain Service Port DCOM Service Control Manager Conclusion 54 viii
10 5.0 CONCLUSION AND RECOMMENDATION 5.1 Conclusion Recommendation 56 REFERENCES 57 APPENDIX APPENDIX A: Installation Honeywall ix
11 LIST OF FIGURES Figure 3.1 Research Methodology Phases Diagram 24 Figure 3.2 Network Diagram for Virtual Honeynet 28 Figure 3.3 Screen shot of Vmware workstation 30 Figure 3.4 Screen shot of Honeywall running simultaneously 31 Figure 3.5 Honeywall booting up screen shot 33 Figure 3.6 Installation Honeywall screen shot 34 Figure 3.7 Honeywall configuration set up screen shot 35 Figure 3.8 Honeypot IP Address screen shot 35 Figure 3.9 Honeypot running Windows 2000 Server 36 Figure 4.1 Network traffic captured at 3 to 4 a.m 39 Figure 4.2 Network traffic captured at 9 to 10 a.m 39 Figure 4.3 Network traffic captured at 16 to 17 p.m 40 Figure 4.4 Network traffic captured at 1 to 2 a.m 40 Figure 4.5 Network traffic captured at 15 to 16 p.m 41 Figure 4.6 Network traffic captured at 19 to 20 p.m 41 Figure 4.7 Port 445 had been attacked at inbound connection 42 Figure 4.8 Port 135 had been attacked at inbound connection 42 Figure 4.9 Port 137 had been attacked at inbound connection 43 Figure 4.10 Port 138 had been attacked at inbound connection 43 Figure 4.11 Port 1900 had been attacked at inbound connection 44 Figure 4.12 Snort alert on SCAN UPnP service 44 Figure 4.13 Snort alert on http_inspect 45 Figure 4.14 Snort alert on SNMP AgentX and spp_stream4 45 Figure 4.15 Snort alert on SNMP trap tcp 46 Figure 4.16 Snort alert on ICMP PING NMAP 46 Figure 4.17 Snort alert on port unreachable 47 Figure 4.18 Snort alert on MISC UPnP malformed advertisement 47 x
12 LIST OF TABLES Table 4.1 Network traffic on weekday 48 Table 4.2 Network traffic on weekend 50 Table 4.3 Port had been attacked 51 LIST OF GRAPHS Graph 4.1 Network traffic on weekday at different time 49 Graph 4.2 Network traffic on weekend at different time 50 Graph 4.3 Port had been attacked 52 xi
13 CHAPTER 1 INTRODUCTION 1.1 BACKGROUND The honeypot is a relatively new technology. Although it was first publicly discussed more than 10 years ago, only recently has this new tool begun to be widely adopted. A honeypot is unique in that it does not solve a specific problem, which is the case with most traditional security technologies. For example, firewalls are used to prevent unauthorized access to resources, while intrusion detection systems (IDS) are used to detect attacks or failures in security. Instead, a honeypot is a very flexible security tool with several different applications. Honeypots can be used to prevent attacks by deceiving attackers, to detect attacks by capturing probes, or to gather information by logging attackers' activity. Although honeypots can achieve many different goals, they all share the same concept. They are not part of an organization's network and do not run any real services. Thus, nothing should be interacting with them. In a perfect world, any resulting activity a honeypot captured would be an anomaly. In reality, organizations are surrounded by people who want to harm them so connections to the honeypot are most likely probes, scans, or attacks against the company. This simple concept gives honeypots great advantages over other security tools. The other type of honeypot recently used is honeynet. The concept of the honeynet first began in 1999 when Mr. Lance Spitzner, founder of the Honeynet Project published the paper To Build a Honeypot. He proposed that instead of developing technology that emulated systems to be attacked, why not deploy real systems behind firewalls waiting to be hacked.
14 Basically, a honeynet is a type of honeypot, more specifically, a type of high interaction honeypot. And thus being a high interaction honeypot, nothing is emulated which all services, applications and operating systems are as real as in any production environment. A main feature that separates a high interaction honeypot from a honeynet is that a honeynet contains one or more honeypots. It is a network of multiple systems creating an illusion of a production network. It is through this network, specifically through the network access device, is where hacker activity is monitored, recorded and controlled. A honeynet works by creating a highly controlled environment. Honeynets as opposed to honeypots though takes the concept one step further. Instead of just one computer or a number of unconnected computers, a network is set up in such a way that everything in the honeynet appears like a normal network. All applications and services are real though all systems running within the honeynet are considered honeypots. This type of setup makes the honeynet the most interactive and reliable of all honeypots. Virtual Honeynets take the concept of honeynet technologies, and implement them into a single system. Virtual honeynets are not a new concept. Instead they take the existing concept of Honeynets and implement them in a different fusion. This implementation has its unique advantages and disadvantages over traditional honeynets. The advantages are reduced cost and easier management, as everything is combined on a single system. However, this simplicity comes at a cost. First, you are limited to what types of operating system you can deploy by the hardware and virtualization software. Second, virtual honeynets come with a risk, specifically that an attacker can break out of the virtualization software and take over the Honeynet system, bypassing data control and data capture mechanisms.
15 1.2 PROBLEM STATEMENT Nowadays, security system is very important to any organization to protect their data or any information kept in their computer from the intruders to access. Unauthorized user is able to connect to the organization s computers and control it in some form to view or access the files. Many of us know how to use the computer but do not have enough information to secure the computer especially for the system administrators. The frequency of computer intrusion has been increasing rapidly for several years. But today, analyzing intrusions is difficult to do, largely manual task because system administrators lack the information and tools needed to understand easily the sequence of steps that occurred in an attack. Building honeynet previously has been a costly effort, which holds true as price grows exponentially with mission critical deployments. Even the home user interested in deploying honeynet technologies must provide a dedicated machine for data capture, data control, as well as the decoy system itself not to mention the cost of a dedicated connection to larger networks such as the Internet. 1.3 OBJECTIVE OF THE RESEARCH The basic objectives of this research are: To deploy virtual honeynet as a tool to detect an attack that enables the automated detection from any malicious and unknown attack over the network. To detect and analyze an attack that enables the automated detection from any unknown attack using the virtual honeynet.
16 1.4 SCOPE OF THE RESEARCH There will be scopes on this project that will help to make this project successful without going further than its objectives. This project will be using one physical machine based on Windows environment. The tool that will be using here is Honeywall which is one of the honeynet tools. All internal networks at PSMB will be monitor and analysis to collect the result for certain time period. We will be focusing only at the port attacks. 1.5 SIGNIFICANCE OF THE RESEARCH All computer users must know that it is important to protect their own computer from any intruders who is trying to access their system. Therefore, they must prepare and alert for every activity of the network. This project, it addresses the methods and tools an administrator uses to understand how an intruder gained access to the computer. It can alert or give a warning to prevent the system from the intruders or attackers by using a honeypot as an extra security for personal computer. Beside that, for any companies or organization by using virtual honeynet on one single machine will reducing the cost then implement honeynet using several machines.
17 1.6 ORGANIZATION OF THE RESEARCH This is the detailed about the thesis organization to make it more efficient and a guide to do the project research. Through this stage, we should obtain clear view of overall of the thesis and problems about this project. Chapter 1: Introduction In this chapter, we had been discussed about the overall of the research in general. It contains the overview of the problems, objectives, scope and significance of the total project. Chapter 2: Literature Reviews This chapter will discuss about literature reviews that related to the research. Studied from the previous project and similarity with others, it helps and gives us an idea to start and how to organize the project. Chapter 3: Methodology This chapter will discuss about the approaches and methodologies employed to the project. The discussion is consisting of all method from the beginning until the end of the project. Chapter 4: Findings and Analysis This chapter will be focus about the finding for what we have got from the methodologies. Based on the approaches, all findings will be analyzed and capture from the result of the project. This will be present in the proper manner through this chapter
18 Chapter 5: Conclusion and Recommendation This chapter is the last topic or the overall of the research that has been done. It will be summarize to provide the conclusion of the project and recommendation to suggest for new project to others in future.
19 CHAPTER 2 LITERATURE REVIEW 2.1 INTRODUCTION In this chapter, we will discuss about the review that had been made to understand the concept of the research. From the literature review that had been done here, we had identified the meaning of the hackers and attackers, security tools and also the similar project or work to my research project that had been done. 2.2 WHAT IS HACKER? This article adapted from tells us that a hacker is a term used to describe people who use computers. Hacker has multiple meanings. In computer programming, hacker means a programmer who hacks or reaches a goal by employing a series of modifications to exploit or extend existing code or resources. In computer security, hacker translates to a person able to exploit a system or gain unauthorized access through skill and tactics. This usually refers to a black hat hacker. In other fields, hacker is extended to mean a person who makes things work beyond perceived limits through their own technical skill, such as a hardware hacker, or reality hacker. However, for some the word has a negative connotation and refers to a person who "hacks" to accomplish programming tasks that are ugly, inelegant, and inefficient. The negative form of the noun "hack" is even used among users of the positive sense of "hacker".
20 From the computer and information technology prospective and term, attacker means and pointed to the people who are trying to make a problem to others. These problems consist of the safety of the data inside the computer and the computer system itself by considering the software and hardware. As a user, we must always alert and secure our computer from those people around us especially in our internal network or outsider such as internet. 2.3 HONEYPOT To make sure the security concern is number one of the organization, we must choose the perfect and right tool to detect an anomaly situation or any attack may occurs during our processing time What is honeypot and what are the types? Honeypots are closely monitored network decoys serving several purposes such as they can distract adversaries from more valuable machines on a network, they can provide early warning about new attack and exploitation trends and they allow deeply examination of adversaries during and after exploitation of a honeypot. Honeypots are a highly flexible security tool with different applications for security. They don't fix a single problem. Instead they have multiple uses, such as prevention, detection, or information gathering. Honeypots all share the same concept that is a security resource that should not have any production or authorized activity. In other words, deployment of honeypots in a network should not affect critical network services and applications. A honeypot is a security resource whose value lies in being probed, attacked, or compromised. There are two general types of honeypots:
21 Production Honeypot Production honeypots refers to protect and mitigate a risk to organization or company. This kind of honeypot is help to secure environment, such as detect attacks. Production honeypot are easy to build and deploy than a research honeypot because they require less functionality. Their job is deal with a bad guy and captures all activity inside the network. (Spitzner, 2001) Research Honeypot Research honeypot refers on how to use to learn from it. This kind of honeypot does not add direct value to a specific network of the organization. Research honeypot is to research the threat of organization may face, who are trying to attack, how it organized, what kind of tool they used to attacks and where they obtain the tools. (Spitzner, 2001) Value of Honeypot A honeypot can not be used to fix anything. It is even worse, a honeypot can attract more interest in a specific network than one would like. So what can a honeypot provide, what can it be used for? A honeypot is a resource which is intended to get compromised. Every traffic from and to a honeypot is suspicious because no productive systems are located on this resource. In general, every traffic from and to a honeypot is unauthorized activity. All data collected by a honeypot is therefore interesting data. A honeypot will in general not produce an awful lot of logs because no productive systems are running on that machine which makes analyzing this data much easier. Data collected by a honeypot is of high value and can lead to a better understanding and knowledge which in turn can help to increase overall network security. One can also argue that a honeypot can be used for prevention because it can deter attackers from attacking other systems by occupying them long enough and bind their resources. Against most attacks nowadays a honeypot does not help deceiving individuals as there are no persons to deceive. If a honeypot does not
22 get attacked, it is worthless. Honeypots are normally located at a single point and the probability can be quite small that an attacker will find the honeypot. A honeypot does also introduce a certain risk - blackhats could get attracted to the whole network or a honeypot may get silently compromised. (Baumann and Plattner, 2002) Classes of Honeypots Honeypot is coming with many shape and size to make the attackers difficult to get into the system. To better understand honeypots, it has two general categories, low-interaction and high-interaction honeypots Low-interaction honeypot Low-interaction are the primarily production honeypots that are used to help protect a specific organization. The attackers is limited to how much he or she can interact with by emulated services like FTP, telnet, HTTP and others services. Example of the lowinteraction honeypots is the BackOfficer Friendly, honeyd, Mantrap, Specter and others High-interaction honeypot High-interaction honeypot is the actual system with full-blown operating systems and application. It can be learning much more from the attackers because there is actual operating system that the attackers can compromise and interact with. Example of the high-interaction honeypot is the honeynet. It designs to as architecture for entire network to be attacked. It will control the network and captured all the activity that running to the operating system.
23 2.4 HONEYNET A honeynet is a type of honeypot. Specifically, it is a high-interaction honeypot designed to capture extensive information on threats. High-interaction means a honeynet provides real systems, applications, and services for attackers to interact with (as opposed to lowinteraction honeypots such as Honeyd which provide emulated services and operating systems. It is through this extensive interaction we gain information on threats, both external and internal to an organization. What makes a honeynet different from most honeypots is that it is an entire network of systems. Instead of a single computer, a honeynet is a network of systems desinged for attackers to interact with. These victim systems (honeypots within the honeynet) can be any type of system, service, or information you want to provide. Conceptually honeynets are very simple. They are simply a network that contains one or more honeypots. Since honeypots are not production systems, the honeynet itself has no production activity, no authorized services. As a result, any interaction with a honeynet implies malicious or unauthorized activity. Any connections intiated inbound to your honeynet are most likely a probe, scan, or attack. Almost any outbound connections from your honeynet imply someone has compromised a system and has initiated outbound activity. This makes analyzing activity within your honeynet very simple. In many ways, it s the classic needle in the haystack problem, as you attempt to find the critical incident amongst volumes of information. Since a honeynet is nothing more than a network of honeypots, all captured activity is assumed to be unauthorized or malicious. All you are doing is capturing needles. It s up to you to prioritize which of those needles has the greatest value to you, and then analyze them in great detail. (Honeynet Project, May 2005)
24 2.5 VIRTUAL HONEYNET It s a solution that allows you to run everything you need on a single computer. We use the term virtual because it all the different operating systems have the 'appearance' to be running on their own, independent computer. These solutions are possible because of virtualization software that allows running multiple operating systems at the same time, on the same hardware. Virtual Honeynets are not a radically new technology; they simply take the concept of Honeynet technologies, and implement them into a single system. This implementation has its unique advantages and disadvantages over traditional Honeynets. The advantages are reduced cost and easier management, as everything is combined on a single system. Instead of taking 8 computers to deploy a full Honeynet, you can do it with only one. However, this simplicity comes at a cost. First, you are limited to what types of operating system you can deploy by the hardware and virtualization software. For example, most Virtual Honeynets are based on the Intel X 86 chips, so you are limited to operating systems based on that architecture. You most likely cannot deploy an Alteon switch, VAX, or Cray computer within a virtual Honeynet. Second, virtual Honeynets come with a risk. Specifically, an attacker may be able to compromise the virtualization software and take over the entire Honeynet, giving them control over all the systems. Last, there is the risk of fingerprinting. Once the bad guys have hacked the systems within your virtual Honeynet, they may be able to determine the systems are running in a virtual environment. (Honeynet Project, January 2003) Self-Contained Virtual Honeynet A Self-Contained Virtual Honeynet is an entire Honeynet network condensed onto a single computer. The entire network is virtually contained on a single, physical, system. A Honeynet network typically consists of a firewall gateway for Data Control and Data