Intrusion Detection. Tianen Liu. May 22, paper will look at different kinds of intrusion detection systems, different ways of

Transcription

1 Intrusion Detection Tianen Liu May 22, 2003 I. Abstract Computers are vulnerable to many threats. Hackers and unauthorized users can compromise systems. Viruses, worms, and other kinds of harmful code can cause great damage. In order to have computer systems be more secure, there are three main types of defenses against attacks: antivirus software, firewalls, and intrusion detection systems. This paper will discuss this third kind of defense: intrusion detection systems (IDS). This paper will look at different kinds of intrusion detection systems, different ways of detecting intrusions, and system response after detection. II. Introduction An intrusion detection system is a defense mechanism whose goal is to detect when a system or network is being used inappropriately or without correct authorization 1. Work on this began in 1980 with James Anderson who introduced a surveillance system that could detect malicious activity using event tracking records or audit logs. In 1985, Dorothy Denning and Peter Neumann provided a model on an intrusion detection expert system. Beginning with these researches, intrusion detection systems (IDS) were born. They are needed because the other two major kinds of defenses, antivirus software and firewalls, are not adequate to cover all kinds of attacks. Antivirus software 1

2 protects only against malicious programs such as viruses, but not against hackers and many other kinds of threats. Firewalls limit the kind of traffic that can flow in and out of a system so that they do not allow unauthorized access to important information. But these do not protect entirely. The traffic left to flow freely can be harmful. Intrusion detection systems are needed because they can sense a variety of unusual activities, and notify the proper authorities and prevent further attacks. IDS adds to system security, especially when they are used in addition to antiviruses and firewalls. Intrusion detection systems fall into three major categories depending on what kind of system they monitor. Though each has benefits and limitations, an IDS that can monitor the largest number of machines is the best. This paper will explain why this is so. It will also discuss a variety of detection methods and compare and contrast them. Lastly, this paper will look at how an IDS responds to an attack. Work related to IDSes are intrusion prevention systems, which improve upon how IDSes respond to attacks. This paper will not cover all information related to intrusion detection systems, but will examine most of the major issues. III. Main Body Components It is important to discuss, first of all, the basic components of an intrusion detection system. Almost all intrusion detection systems have audit logs. These keep track of what is happening on a system. The logs generated are sent to the management portion of an IDS for analysis to detect intrusions. Most detection systems have sensors that examine traffic and keep records of the packets they view flowing in and out of a 2

3 host or network system. The third major component is the afore mentioned management system, which analyzes data from logs and generates alerts when intrusions are detected. IDSes can be software or hardware. When used on a network, they are usually placed behind a firewall. Kinds of Intrusion Detection Systems An intrusion detection system can be divided into several kinds. The type of detection system is determined by what type of system is being monitored. What is monitored can be a host, a network, or a large portion of the internet. A host-based system keeps and examines audit records about a host. It checks the integrity of system files, watches for suspicious processes, looks at communication traffic in and out of the computer. A network-based system looks at packets on a network. These typically use sensors to log all the activity on a network segment and a console to collect and analyze the logs for suspicious activity. The main difference between host and network systems is that a host monitors its own system whereas a network detection system monitors the segment of the network it is attached to. A host-based intrusion detection system or HIDS is able to detect insider attacks by checking its own files and processes, and outsider attacks by watching incoming and outgoing traffic of the host. A network intrusion detection system or NIDS looks at every packet on a segment of a network, and thereby can get more of an idea of traffic flow to and from the machines in the network. However, NIDS must keep up with the speed of the packets on the LAN. If too many packets are going by too fast, a network detection system will not be able look at all of them. 3

4 An organization can choose to use host-based or network-based solution. If it decides to use host-based systems, it must install a host-based IDS on every host to get a better view of the network. If it uses a network-based system, it may run the risk of missing packets, but it can monitor traffic among more than one machine without having to install many systems. A third solution is distributed intrusion detection systems (DIDS). Distributed systems can better handle a problem that neither host nor network systems are able to deal with very well. For example, a system administrator may see something unusual happen to a host or a portion of the network they are monitoring. They wonder whether other hosts or networks are receiving the same unusual traffic. If they are, this traffic may be part of a concerted attack that spans a large section of the internet. Neither HIDS or NIDS can see a large part of the internet. If there is a way to see other hosts and networks that are not part of the local network that a system administrator may be monitoring, then such an attack can be detected early and wide area epidemics can be prevented. Distributed systems offer this kind of protection. They can have three components. A host agent module generates logs about a single host and sends them across the internet to a central manager module. A LAN monitor agent module generates logs about traffic on the local area network that it is monitoring. It also sends its logs to the central manager module. The central agent module collects these data, and data from other LANs and hosts 2. Thus, it checks for intrusion on a larger scale. DIDS provide for more accurate attack detection and quicker response because administrators can see more network activity. 4

5 DIDS can have modular components like the one described. Some can also support host-based and network-based IDSes that send their log files to the distributed system. Therefore, by using this kind of DIDS, one can have the benefit of HIDS and NIDS also. Someone using HIDS and DIDS can get an idea of host events while also seeing what is happening to other systems in the internet. People using NIDS and DIDS can see traffic on their local network while also knowing about major events happening on the internet. DIDS offer a multi-level way of monitoring systems. They can see what is going on at the host, network, and global level. This is better than looking at possible intrusions only on a host or only on a network. Information about insider attacks (people misusing a host system), outsider attacks (people trying to attack a particular network), and attacks on other hosts and networks can all be viewed. Seeing what is going on at multiple levels is the key to a better detection system. Methods of Intrusion Detection There are several different kinds of approaches to actually detecting intrusions. These include statistical anomaly detection, rule-based anomaly detection, and rule-based penetration identification. Statistical anomaly detection uses statistics formed from audit logs to detect anomalies from normal user behavior. Most statistical anomaly systems rely on learning about past behavior of users. Analysis of audit logs over time determine what behavior is normal for users. Any deviations generate alerts. Tests for determining 5

6 normal behavior include mean and standard deviation. This test examines data from logs to see if they fall into the range of average behavior and how much the data points vary from one another. The multivariate test looks at correlation between two or more variables, such as login frequency and time between sessions. If these two variables taken together exceed what is normal, then an alert will be generated. The Markov process examines transition probabilities between certain states. For example, it can look at the transitions between commands to see if they fit normal usage. The time series test determines whether something happens too quickly or too slowly. Finally, the operational test suspects intrusion if the number of occurrences of an event surpasses a predetermined limit. These tests can be used together to determine deviations, because each test measures different aspects of a single event. Another approach is rule-based anomaly detection. In this approach, the system analyzes data from audit logs and automatically develops a set of rules to describe normal behavior. While statistical anomaly detection inputs data into statistical tests to see whether this data falls into previously learned statistics, rule-based anomaly detection relies on the rules generated from previous statistics. So data about each new event is tested against the rules to see whether it is normal. Because rules are generated from statistics, a large database of rules is needed for rule-based anomaly systems to work well. The number of rules could reach or even 1 million. Nevertheless, the rulebased anomaly approach is as effective and strong as the statistical anomaly approach 3. despite the large volume of rules. Intrusion Detection Expert System (IDES) is an example of an statistical anomaly system and Wisdom and Sense (W&S) is an example of a system with rule-based anomaly detection. 6

7 An important advantage of anomaly intrusion detection systems is that they can catch new intrusion attempts, which are anomalous behavior. The disadvantage is that if what is normal is abnormal traffic (like during the time of the Code Red worm), then the system will not generate appropriate alerts. Therefore, an anomaly IDS needs to be placed on a network or used in a host system for a long period of time to learn what is normal. This is because the longer it gets a chance to learn, the more accurate are the alerts. A third way of detecting intrusions is rule-based signature or penetration identification. This method is often used along with anomaly systems. In this method, rules are not set up based on analysis of audit logs. Rules can be set up by defining known intrusions and by defining suspicious activity. The usual procedure is to collect information from system administrators and security analysts. Based on this information, one can set up some heuristics. For example, users should not be able to open other users personal directories. They should not write to other users files. They should not copy system programs. They should not be able to log in several times to the same system 4. Besides these rules that are set up against suspicious activity, a signature system also includes rules against known intrusions. Alerts are generated if an event or sequence of events matches the way known intrusions were launched--that is, they match signatures of known intrusions. Usually, vendors provide about 500 to 1500 rules for their products 5. As new intrusions are discovered, a system administrator must update his system with new signatures. The obvious disadvantage of a signature rule-based system is that a system administrator must constantly update his intrusion detection system whenever new 7

8 intrusions are discovered and new signatures are available. This constant updating can be a hassle, and if one does not do it in time an intrusion can occur. Another disadvantage is that this system cannot detect new intrusions because new intrusions do not follow any known patterns. An anomaly based system does not have these disadvantages. It can serve to complement signature detection s weaknesses. On the other hand, the signature based model is able to quickly detect misuse and known ways of intrusion more accurately than anomaly systems. In anomaly based systems, alerts may be generated by small, non-critical deviations, thereby creating many false alarms (or false positives). Similarly, if a network often has a large volume of abnormal traffic, actual harmful packets may escape detection, thereby creating false negatives. By using signature based systems to detect known intrusions, the accuracy of detection increases because signatures are designed to catch known intrusions whereas anomaly systems can either miss the intrusion or create so many false positives that the true positive is overlooked. Therefore, signature detection and anomaly detection serve well to complement each other. A rule of thumb is to use systems together. Operating together, they become a stronger system and their weaknesses can be covered by one another s strengths. NetScreen Intrusion Detection and Prevention: an example It is worthwhile to learn more about an intrusion detection system by looking at an actual IDS product. This paper will look at an intrusion detection product called NetScreen Intrusion Detection and Prevention (IDP). IDP is a network-based system that utilizes several different mechanisms to maximize the number and types of attacks that 8

9 can be detected. These mechanisms include stateful signature, protocol anomaly, traffic anomaly, backdoor detection, network honeypot, among others. IDP, like many other detection products, employs both anomaly and signature detection. Let us see how these methods are used in a real IDS product. IDP s signature detection is unlike other products on the market. Other products compares every packet to signatures of known attacks to see if there is a match. So this kind of packet signature detection processes unnecessary information, because it applies the mechanism to all traffic even when attacks cannot be generated at that point. However, stateful signature detection looks only at relevant traffic where attacks can be generated. For example, in order to determine whether a user is trying to login to a server as a root user, stateful signature detection would not look for the word root in all transactions. It would look for it only in the login sequence 6. Thus, precious resources are not wasted; a system administrator can look at packets that are of genuine concern. IDP s anomaly detection chooses to focus on the usage of communication protocols and traffic flow rather than other kinds of system usage. IDP s protocol anomaly detection analyzes traffic, comparing them with normal traffic that follows protocols. Usually, abnormal traffic is ambiguous, that is, it does not follow protocol specifications so that they can avoid detection. But normal packets are unambiguous. Thus this mechanism tests traffic to see whether it deviates from how normal traffic follows protocol specifications. Those packets that deviate are flagged with alerts. The effectiveness of protocol anomaly depends on the number of protocols the detection system recognizes. Abnormal packets that use a protocol not supported by the detection system may successfully escape detection. Therefore, the greater number of protocols a 9

10 system supports, the better. Some example protocols that IDP supports are IP, TCP, ICMP, ECHO, FINGER, DNS, POP3, IMAP, and many others 7. IDP also supports traffic anomaly detection. This kind of anomaly detection detects attacks that continue throughout a number of sessions. For example, an attack could begin by gathering some information (port scanning). Later, an attacker can penetrate through an open port that his earlier port scan discovered. Traffic anomaly detection would recognize port scanning as abnormal traffic, because port scans deviate from most normal traffic patterns. Thus, traffic anomaly detection that focuses on analyzing traffic patterns can detect port scans and other kinds of probes. After detection, the victim of the potential attack can close the vulnerable port and watch out for future activity from the attacker s IP address. IDP has another mechanism, which may protect against backdoor attacks. Backdoor attacks are those that enter the system through another program without the user s knowledge. While a user executes one program, a hidden program may also be running. By running this program that the user does not see, an attacker gains control of the system. An example of a backdoor attack is a Trojan horse. After the user opens the Trojan, for example a singing birthday card, the Trojan downloads the malicious code. The attacker interacts with the malicious program and gains control of the system. IDP detects backdoor attacks by searching for interactive traffic such as that between the attacker and the program in the compromised system. Upon finding such traffic, IDP tests this against what is defined by the administrator to be allowed. This is similar to comparing current traffic to a set of rules or heuristics. Unexpected interactive traffic that violates these rules generate alarms. 10

11 Another way to detect intrusions is through a network honeypot, which IDP uses. A network honeypot is a system that impersonates real services. When an attacker scans ports and finds such a system, the system sends fake information to the attacker so that the attacker would try to access these false services. Since legitimate users only access services through ports that they know offer real services, whoever tries to access services of honeypots are not legitimate users. They have done a port scan and found the honeypot; they can be assumed to be attackers. Since honeypots make it difficult for attackers to use them to compromise other systems, catching attackers this way spares other systems of potential attacks by these hackers. After looking at some of the detection mechanisms that Netscreen s IDP offers, one can see how each of them work together to maximize detection of attacks. Signature and anomaly detections can catch known and new attacks. Together they can also detect any type of abnormal and wrong use of the system. Backdoor intrusions like Trojans can be handled by IDP s backdoor detection. Honeypots can get an idea of what the attacker wants to do, such as the services he wants to access as he plans to take over a system. Honeypots also notify the IDP of these port scanning attackers, and proper responses can be taken to watch out for traffic from these attackers. Used by themselves, only some attacks may be detected. Anomaly detections used alone may not catch everything or may generate many false positives. Signature and honeypot detections help lower false positives, because traffic that match signatures and people who access honeypots are usually guilty of attacking a system. In this example, we have seen a variety of methods in a real intrusion detection system. The reason for these methods to be employed in one 11

12 IDS is to catch a wide variety of attacks. Many methods are better than one in detecting intrusions. Response after Detection There are many methods of intrusion detection. But what happens after an attack is detected? The rest of this paper will discuss some ways intrusion detection systems can respond to detected attacks. Usually, all suspected packets can be displayed on screen. So an administrator can analyze suspected packets and respond to attacks personally. For example, he or she can close a vulnerable port or disconnect an affected system. However, it would be too much for a security professional to respond to each alarm. Automated responses are necessary. One type of automated response is called session sniping. Session sniping is performed on TCP connections. Two systems, communicating through TCP, must establish a connection through handshaking before any data can flow between them. When users want to terminate the session, they must close the connection. In order to stop illegitimate traffic, an IDS can knock down this connection so that the attack can be stopped. An IDS that uses session sniping sends packets to the attacker and the victim with the TCP reset bit set to 1. When they receive these packets, the connection would stop. Another way to respond to attacks is to signal the firewall or router. After detection, an IDS tells the firewall or router to block future packets from the intruder s IP address. 12

13 But these responses have drawbacks. They are too late to stop the effects of an attack. Before a TCP connection can be dropped, some malicious packets may have already entered the system. Firewalls and routers only block malicious packets in the future. Those that already got through the firewalls and routers can still have an effect on the system. Unfortunately, most intrusion detection systems rely on these response mechanisms. Because of this, intrusion detection systems usually do not have adequate prevention. Some security professionals are turning to intrusion prevention systems (IPS) to overcome this problem. IPS is a new development, whose goal is to prevent attacks from affecting a system at all. IPS usually drops malicious packets as they are detected. So a system does not have to recover from an attack since they will not be affected in the first place. IV. Summary This paper has covered some of the major topics related to understanding an intrusion detection system. The best intrusion detection system, which can accurately detect a variety of attacks, is one that detects intrusions at different levels and applies a multi-method approach. Host-based and network-based detection systems have their benefits and limitations. A distributed system may be better because it can know what is going on at the host, network, and global level. A multi-method approach, using signature, anomaly, honeypot, backdoor defense, etc., increases the chances of catching all kinds of attacks. 13

14 The way to a secure system is not through a cure-all that could detect and prevent all intrusions. No such system exists. Indeed, the approach to intrusion detection is to use many methods instead of one. The way to securing a system is to use many products instead of one. Anti-virus software and firewalls should preferably be used along with intrusion detection systems. An able and attentive system administrator is also needed. The reason for such an army is that attacks come in various forms. No single method can stop them. While good IDSes are able to detect many intrusions, most IDSes cannot prevent attacks effectively. The focus in the future will probably be on intrusion prevention systems that both effectively detect and prevent attacks. 14

15 References Carr, Jim. Intrusion Detection Systems: Back to Front? Network Magazine. 5 Sep URL: Farshchi, Jamil. Statistical based approach to Intrusion Detection. URL: Hawrylkiw, Dan. Network Intrusion and use of automated responses. URL: Hrivnak, Allison. Host Based Intrusion Detection: An Overview of Tripwire and Intruder Alert. 29 Jan URL: Kemmerer, Richard A. Computer Security. URL: Netscreen.com. Products. URL: Sans.org. What is host-based intrusion detection? URL: Sans.org. What is network based intrusion detection? URL: Sink, Michael. The Use of Honeypots and Packet Sniffers for Intrusion Detection. 15 April URL: Spafford, Eugene, and Diego Zamboni. Data collection mechanisms for intrusion detection systems. 2 June URL: Spitzner, Lance. Honeypots: Tracking Hackers. San Francisco: Pearson Education, Inc., Stallings, William. Network Security Essentials: Applications and Standards. New Jersey: Prentice Hall, Inc., Zuver, Robert. A Thousand Heads Are Better Than One The Present and Future of Distributed Intrusion Detection. 30 April URL: 15

16 Endnotes 1. Hrivnak. 2. Stallings, p Ibid, p Ibid, p Farshchi. 6. Netscreen.com. 7. Ibid. 16