A PEER-BASED HARDWARE PROTOCOL FOR INTRUSION DETECTION SYSTEMS

Transcription

1 A PEER-BASED HARDWARE PROTOCOL FOR INTRUSION DETECTION SYSTEMS Major Gregory B. White and Captain Mark L. Huson Department of Computer Science United States Air Force Academy USAF Academy, Colorado Abstract A number of intrusion detection systems have been developed to detect intrusive activity on individual hosts and networks. The systems developed rely almost exclusively on a software approach to intrusion detection analysis and response. In addition, the network systems developed apply a centralized approach to the detection of intrusive activity. The problems introduced by this approach are twofold. First the centralization of these functions becomes untenable as the size of the network increases. A heirachical network organization incorporating Internetwork Security Monitors (ISMS) has been proposed as a solution to this problem. However, the introduction of intermediate security systems increases the number of potential targets and introduces communication delays which are unacceptable for high bandwidth data transfers. Second, and more importantly, the combination of centralization and software implementation as an approach to network intrusion detection introduces a dangerous vulnerability. As intruders gain access to the system, they target the security software itself and the centralization ensures the compromise of the entire network. The solution to these problems is a hardware implementation of a decentralized approach to intrusion detection. This paper describes the hardware platform necessary to implement such a system. It also proposes an intrusion detection protocol which would be used by this hardware to communicate relevant intrusive activity events between heterogeneous systems connected in a network or internetwork. This work is based on the Cooperating Security Mangers; a peer-based approach to intrusion detection developed at Texas A&M University. f Introduction Dr. Dorothy Denning presented a paper in 1986 on the use of audit trail records for intrusion detection purposes[l]. In her paper, Dr. Denning described how exploitation of a computer system generally involves an abnormal use of the system or its resources which can potentially be detected by searching for abnormal system usage patterns. This idea led to the use of a system s audit trail to establish a profile of normal behavior for each user. Abnormal activities would then be indicated by a current session s activities falling outside of the established user s profile. Three metrics were proposed by Dr. Denning to define a user s normal activities. These metrics include the number of times a specific command is used during a session, the quantity of a specific resource used, and the length of time between related events[l, 21. Several specific parameters can be chosen to provide the data called for by these metrics such as the time between successful logins, the number of pages printed, the amount of disk space used, and the frequency that commands such as the UNIX 1s and cd are used. The methods described by Dr. Denning are not without problems. For example, if an individual knows the system is monitored, they can gradually modify their behavior in order to escape detection. It may also be possible for intruders to use low-level features of the system which may not be monitored in order to perform subtle forms of intrusion[l, 21. An additional problem with this User-Profiling approach to intrusion detection is that researchers have found it difficult to identify the parameters which best describe the normal activity of a user and which provide the best chance at detecting intrusive activity. As a result of the problems encountered when relying on user-profiling techniques to identify intrusive activity, several other methods have been used to supplement or replace the method proposed by Dr. Denning. The first of these is an Action-Based approach which relies on the identification of specific actions as being indicative of intrusive behavior. The purpose in this case is to determine what activities, no matter who performs them, are indicators of an intrusion into the system. The system can then be monitored and if any of these actions are performed an alarm can be raised. The biggest drawback to this method is that it assumes that the intrusion detection system is aware of all activity that indicates intrusive behavior. If an intruder develops a new method to gain access to a system, or discovers a new security hole, an action-based intrusion detection system will not detect the break-in. One way to look at the difference between action-based and user-profiling methods is that action-based techniques attempt to define proper system behavior as opposed to the attempt to define normal behavior seen in user-profiling. A second supplemental method to perform intrusion detection is Intruder-Profiling. This method is similar to userprofiling except instead of attempting to define user activity, it defines the normal activity of an intruder. In other words, it attempts to determine what the normal actions of an $ IEEE 468

2 intruder are and then search for patterns of activity that match these actions. This is somewhat similar to a law enforcement agency s profile of a serial killer. The problem with this technique is that it assumes that there is a pattern to intruder activity (and that intruders follow this pattern) and that enough data on intrusive activity exists to establish these profiles. The last method that can be used to supplement userprofiling in intrusion detection systems is Signature Analysis. Just as an individual s handwriting is unique and can be used for identification, so is an individuals ((typing signature. It has been shown that an individual s keystrokes can be analyzed and used to positively identify them[3]. The obvious drawback to this technique is that it requires specialized hardware to perform and is impractical over a network. Instead, however, a modification of this technique can be used to supplement other intrusion detection methods in order to add a second level of assurance. This modification involves the analysis of common typing errors and commands that a user makes/issues. This method cannot be used as the sole means of authentication because a user s error rate may be too low, but it can be used as a secondary mechanism. Since Dr. Denning s presentation in 1986, a number of organizations have developed intrusion detection systems. These systems generally use a combination of the actionbased and user-profiling systems. Some have been extended to detect intrusive activity over a network as well as on individual hosts. An examination of several of these is useful in order to see where the deficiencies and problems still lie. 2 Existing Intrusion Detection Systems One of the earlier intrusion detection systems (IDS) developed is the Multics Intrusion Detection and Alerting System (MIDAS)[4, 51. MIDAS is designed to work on NSA s Dockmaster computer system. Dockmaster is NSA s unclassified computer system used by the computer security research community. A portion of the intrusion detection processing is performed on the Multics system itself but the heart of it resides on a separate, stand-alone Symbolics machine. Audit records generated on the Multics machine are first preprocessed to filter out data not used by MIDAS. The processed audit records are then transferred to the Symbolics machine where they are entered into a fact base as assertions. A separate command monitor captures command data not audited by the Multics system and also transfers it to the Symbolics machine. The introduction of an assertion into the fact base may cause one or more bindings to take place between the record and a rule found in the rule base. A statistical data base contains both system and user statistics used to define normal behavior on the Dockmaster system and the assertion of a fact my also trigger a warning should the current user s activities fall outside of the defined norm. The architecture of MIDAS is depicted in figure 1. Preprocessor Network Interface. I Fact Base Multics 1 - symbolics Statistical Data Base Figure 1: MIDAS architecture [4, 51. System Security Monitor It can be seen that MIDAS uses a combination of both action-based and user-profile techniques in its intrusion detection activities. The system is specific to the Multics environment and requires a separate, stand-alone processor to perform the IDS functions. Another early IDS is the Intrusion Detection Expert System (IDES) developed at SRI International[6, 71. Like MI- DAS, this system uses both a user-profile and action-based approach to intrusion detection. Like MIDAS, IDES receives an audit record, parses it, and compares it against both an established user profile as well as an expert system containing information about specific actions viewed as indicators of intrusive activity. Should anomalous activity be identified, again like MIDAS, an administrator is notified so that further investigation and corrective action can be taken. A problem with IDES and all other single-host IDS is that certain types of intrusive activity may not be detected because it is not considered important enough to raise an alarm. An example of such an activity is a single failed login attempt. All users have at one time or another either forgotten their password or simply made a typographical error when entering it. An intrusion detection system which raises an alarm every time a failed login occurs will soon be ignored. If viewed from a network perspective, however, and a dozen machines on the network all had a single failed login attempt for a specific user, the picture suddenly changes. Now this can be seen to be a special type of attack known as doorknob rattling where a potential intruder is repeatedly trying to gain access to an account but doesn t spend too much time on any one host in order to avoid detection. To detect this type of attack requires a different approach, a network approach, to intrusion detection systems. Later extensions to IDES have led to a system designed to detect intrusions in just such a networked environment. One of the first systems designed to perform intrusion detection in a network environment is the Network Security 469

3 Monitor (NSM). This system, developed at the University of California, Davis, does not use audit trails to perform intrusion detection. Instead, NSM monitors the broadcast channel, observing all network traffic. This method results in two significant advantages for NSM. First, NSM can be used to monitor a network consisting of heterogeneous platforms since it monitors the traffic between systems and not the systems themselves. Second, an intruder cannot tell that the network is being monitored since NSM is only listening to the broadcast channel. One drawback to NSM is that it doesn t perform single host intrusion detection. NSM also encounters a problem when the network traffic is encrypted. Of particular interest to the security community was a test of NSM conducted at UC-Davis over a two month period. During that time, over 110,000 connections were made to the network NSM was monitoring and analyzed. NSM identified over 300 intrusions during this period, of which only about 1% had been detected by system administrators. One of the best network intrusion detection systems designed for a local area network (LAN) is the Distributed Intrusion Detection System (DIDS), also developed at UC- Davis[S]. DIDS was designed to address the problems experienced by NSM and consists of three components. The first are individual host monitors which reside on the hosts connected to the network to be monitored. The second component is the DIDS Director which is responsible for coordinating the intrusion detection activities for the network. The last component is the LAN monitor (currently NSM) whose responsibility is to monitor those systems connected to the net which do not have an individual host monitor. An interesting aspect of DIDS is its ability to track individual users across the network despite attempts to hide their trail by using multiple accounts and user ids. The major problem with DIDS is the limitation imposed by the centralized a p proach it takes in directing the intrusion detection activities. Since all host monitors communicate with the DIDS Director, the director can become saturated with messages as the size of the LAN grows. Additionally, the intrusion detection activities are limited to the LAN that DIDS is running on. A proposal to extend the concept to an internetworked environment has been made which organizes multiple networks into a hierarchical structure with monitors (known as Internetwork Security Monitors) at each of the levels[9]. The problem with this is approach that this hierarchical structure does not extend throughout the Internet. A system designed to address the problem found in centralized intrusion detection systems are the Cooperating Security Managers (CSM). Like DIDS, CSM tracks users as they travel between systems but unlike DIDS, it does not use a dedicated centralized director. Instead, each host running a CSM is responsible for the activities of its own users. CSMs coordinate intrusion detection among themselves as users travel between various hosts on different networks. Since there is no central director (each CSM assumes this role for its own users) there is no single point that can become over burdened with network traffic from every monitored host. The individual components of CSM are depicted in figure 2. A problem with the current implementation of CSM is that all components are implemented in software. It has been shown in the past that a common target of intruders are security packages running on a host which has been broken into. A safer implementation would place some or all components in hardware. 7 Interface 0 t her Manager Handler Figure 2: CSM Functional Components. 3 Hardware-Based Protocols for Intrusion Detection Systems In order to lessen the chance that a host s security systems become corrupted, as many of the security functions as possible should be moved to hardware. If machine code level instructions or sequences of instructions that indicate intrusive activity is taking place can be identified it may even be possible to eliminate the need for all software portions of an intrusion detection system. Recognizing intrusions at this level, however, is extremely difficult so a more practical approach is to place the directing functions of the intrusion detection system in hardware and leave the system specific functions in software. The question, then, is what sort of directing functions can be placed in the hardware and what system specific functions will need to remain software based? The proposed solution mirrors the components of the software based CSM but places certain key portions in hardware. The CSM hardware platform will be responsible for the detection of intrusive activity on a host as well as the coordination of activities among hosts; the Security Manager and Local IDS functions of Figure 2. These functions depend on the generation of messages from the system specific portions of the CSM and on the ability to monitor network traffic and receive messages from other CSMs on the network. For example, user tracking, will be accomplished by the hardware platform. When a user attempts to login to a host from across a network, three separate indicators will be used to establish the connection. The first indication is the packet containing the request to establish the connection. The hardware platform, which receives a copy of all network traffic, can make an entry into a table of connections when it receives such a packet. It then waits for the two other 470

4 indications in order to verify the connection establishment. The other two messages will be sent to the hardware portion of the CSM from the originating and receiving hosts. The originating host s CSM (the one from which the connection request was sent) will send a message to the new host s CSM stating which user generated the request. This message will be recorded by the hardware platform as a message received from the network. The originating and receiving hosts will send messages to the hardware portion of their own CSMs indicating the success or failure of the attempted connection. The originating host s CSM would then send this failure backwards to each intermediate host CSM until it reached the user s point of entry to the network. Detection of intrusive activity is dependent upon communication between the software and hardware components of the host s CSM. The software component is responsible for the detection of security relevant events (the Command Monitor functions) and communicating the occurrence of these events to the CSM hardware component. While the detection of security relevant events is system unique, the general categories they fit into is not. It is these categories that are of concern to the hardware components. The categories of security relevant events include: connection requests attempts to gain access to privileged accounts (e.g. su) browsing related commands invocation of an editor invocation of a compiler printing of files deletion of files movement of files copying of files electronic mail web related commands attempts to exploit known security holes Each CSM hardware component will also be responsible for keeping track of where all users originated and where they may have then ventured. Should any individual CSM determine that a user s activity is indicative of intrusive behavior, it will notify the other CSM s in the user s trail of its discovery. Intrusive behavior can be detected by either an originating or receiving host. Each CSM can then take appropriate action to handle the intruder (whether this is simply to notify a system administrator or something more drastic such as severing the connection is up to the individual sites). It is important to note that, for security purposes and to avoid spoofing, all communication between CSMs must be encrypted. The User Interface and Intruder Handling functions of the CSM are system specific, and will be implemented in software. In terms of system response, the individual system managers can select the level of response for their system. Once an intruder is detected, the response can range from simple sysadmin notification to terminating the session and locking the account. The sysadmin will pre-select the level of response which will be automatically carried out by the CSM when an intrusion is detected. An additional benefit of a hardware-based CSM is that it solves the problem associated with the legal issues surrounding backhacking. Currently, when a system is broken into, certain agencies want to track the user back to the place of origin in order to catch, and prosecute, the individual(s) involved. Since there is no current trail kept, this involves quite a bit of coordination between system administrators, or the use of tools to hack backwards along the same trail the intruder used to arrive at the current host. Such activity is strictly regulated and is done infrequently. With the way the CSM s are designed to maintain knowledge of all connections, it is an easy matter to immediately notify all systems involved, and to provide the users trail and place of origin without having to employ any tools to break into affected systems. The current drawback to the hardware-based CSM is that it requires that the majority of systems connected to the network have the hardware platform attached. Techniques for maintaining packet integrity have been discussed in the literature when unmonitored hosts are connected to the network [9]. When a significant number of hosts in an intruder s trail are unmonitored, the effectiveness of CSM is reduced. This CSM concept works best when unmonitored hosts are in the minority. Mixing of software and hardware based systems would open the entire network up to the vulnerabilities associated with current software-based systems. The CSM concept will receive acceptance only if the hardware platform is inexpensive enough for vendors to consider its inclusion in new system designs. Preliminary discussion with specific hardware vendors indicate that this goal is indeed possible. 4 Conclusions The security software is a common target of computer system intruders. It is logical, therefore, to place as many of the security related functions as possible in hardware. This paper proposes just such a system based on the Cooperating Security Manager approach developed at Texas A&M University. While system specific portions of the CSM would remain as part of the system s security software, the Local IDS and Security Manager portions (see figure 2) are system independent and can therefore be implemented in hardware. References [l] Dorthy Denning, 1986 Symposium An intrusion-detection model, in on Security and Privacy, April 1986, 47 1

5 pp [2] Gregory B. White, E. Fisch, and U. Pooch, Cooperating security managers: A peer-based intrusion detection system, IEEE Network, vol. 10, no. 1, pp , January/February [3] R. Gaines, W. Lisowski, J. Press, and N. Shapiro, AUthenitication by keystroke timing: Some preliminary results, RAND Technical Report R-2526-NSF, Rand Corporation, [4] Gregory B. White, E. Fisch, and U. Pooch, Computer System and Network Security, CRC Press, Boca Raton, FL, [5] M. Sebring, E. Shellhouse, M. Hanna, and R. Whitehurst, Expert systems in intrusion detection: A case study, in 11th National Computer Security Conference, Gaithersburg, MD, October 1988, pp [6] T. Lunt, R. Jagannathan, R. Lee, S. Listgarten, D. Edwards, P. Neumann, H. Javitz, and A. Valdes, Ides: The enhanced prototype, a real-time intrusion-detection expert system, SRI International SRI-CSL-88-12, SRI International, [7] T. Lunt, et al., Knowledge-based intrusion detection, in 1989 AI Systems in Government, March [8] S. Snapp, J. Bentano, G. Dias, T. Goan, L. Hebprlein, C. Ho, K. Levitt, B. Mukherjee, S. Smaha, T. Grance, D. Teal, and D. Mansur, Dids (distributed intrusion detection system) - motivation, architecture, and an early prototype, in 14th National Computer Security Conference, Washington, D.C., October 1991, pp [9] L. Heberlein, B. Mukherjee, and K. Levitt, Internetwork security monitor: An intrusion detection system for large-scale networks, in 15th National Computer Security Conference, Washington, D.C., October 1992, pp