Formal Reasoning about Intrusion Detection Systems. Abstract. Intrusion detection is an appealing approach to improving the security of systems.

Transcription

1 Tao Song March 2007 Computer Science Formal Reasoning about Intrusion Detection Systems Abstract Intrusion detection is an appealing approach to improving the security of systems. The goal of intrusion detection is to detect attacks with low false positive rate and low false negative rate. New approach is needed to evaluate the effectiveness of the intrusion detection systems (IDSs. In this dissertation, we present a formal framework for the analysis of intrusion detection systems that employ declarative rules for attack recognition. Our approach allows reasoning about the effectiveness of an IDS by formalizing and proving security properties of the IDS. Detection rules of IDSs and security requirements of the system are formalized and properties about the IDSs are proved within the framework. To illustrate the validation of our approach, we formalize and proved properties about three typical IDSs. SHIM (System Health and Intrusion Monitoring is used as an exemplary host-based IDS to validate our approach. We formalized all specifications of SHIM which together with a trusted file policy enabled us to reason about the soundness and completeness of the specifications by proving that the specifications satisfy the policy under various assumptions. DRCP (Dynamic Registration and Configuration Protocol is an auto configuration protocol in mobile Ad Hoc networks. With respect to this protocol, our approach defines a global security requirement for a network that characterizes the good behavior of individual nodes to assure the global property. We formally prove that the local detection rules (identifying activity that is monitored imply the global

2 2 security requirement. OLSR (Optimized Link State Routing protocol is a proactive, table-driven routing protocol in MANETs. We analyze a specification-based intrusion detection mechanism to detect insider attacks in the OLSR protocol. We proved that the intrusion detection approach, which focuses on monitoring of local behavior, achieves a global integrity of network routing information. Our approach, novel to the field of intrusion detection, can, in principle, yield an intrusion detection system that detects any attack, even unknown attacks, that can imperil the security requirements of the system. The originality of our formal analysis is that it is completely based on analytical methods and does not rely on simulation or experimental evaluation.

3 Formal Reasoning about Intrusion Detection Systems By TAO SONG B.S. (Peking University 1997 M.S. (Peking University 2000 DISSERTATION Submitted in partial satisfaction of the requirements for the degree of DOCTOR OF PHILOSOPHY in Computer Science in the OFFICE OF GRADUATE STUDIES of the UNIVERSITY OF CALIFORNIA DAVIS Approved: Committee in charge 2007 i

4 Formal Reasoning about Intrusion Detection Systems Copyright 2007 by Tao Song

5 To my wife, Xiaohan Gu, and my daughter, Sophia Song. iii

6 Acknowledgments I would like to express my sincere gratitude and appreciation to my advisor, Professor Karl Levitt, for his guidance, understanding, patience and support at all levels. In every sense, none of this work would have been possible without him. I would also like to thank Professor Devanbu and Professor Bishop for reading this dissertation and offering constructive comment. Dr. Poornima Balasubramanyam, Dr. Jeff Rowe, Dr. Calvin Co and Dr. Marcus Tylutki offered much-appreciated advice, support and thought-provoking ideas throughout my research in computer security. Many specifications and data I used in the formal verification were provided by many people inside and outside UC Davis. In particular, Calvin Co provided the spec of SHIM and Chinyang Henry Tseng simulated the OLSR detection mechanism on the GloMoSim simulation platform. I thank them and other friends at UC Davis for their encouragement and support in my research. In addition, I am grateful to my mother and father who have always been supportive of my academic pursuit in the United States. And I thank my brother and sister for supporting my study and taking care of our parents when I am far away from home. My final, and most heartfelt, acknowledgment goes to my wife, Xiaohan Gu. Her support, encouragement, and companionship has turned my journey through graduate school into a pleasure. When I was preparing this dissertation, our lovely daughter, Sophia Song, was brought to the world. She brings endless surprise and joy to my life. I dedicate this work and my love to them. iv

7 Contents List of Figures List of Tables viii ix 1 Introduction Overview of Intrusion Detection Limitations of Earlier Work Contribution of this Thesis Introducing Terminology Dissertation Outline Motivating examples Examples of Intrusion Scenarios on Host-based Systems Ftp Vulnerability Lpr Vulnerability Detecting Attack Scenarios Overview of Intrusion Detection Systems Developing Detection Rules Enforcement of Security Policies Trusted File Policies Policies Regarding Race Condition Attacks Need For a New Approach The Framework Introduction Overview Hierarchical Model of the Framework High level Security Requirements (Policy and Attacks Modeling the Behavior of Systems Specifications of Intrusion Detection Rules in the Model Verification Specification Completeness Environment Assumptions v

8 4 Formal Reasoning about Host-based IDSs Introduction Analysis of Intrusion-Detection Rules Framework Hierarchical framework of Verification Formalization of the Model Mechanization of the model Specification and Verification of SHIM Introduction to SHIM Formalization of Specifications Verification of SHIM rules Performance Discussion Summary Formal Reasoning about a Specification-based Intrusion Detection for Dynamic Auto-configuration Protocols in Ad hoc Networks Introduction Formal network model and hierarchical framework A Hierarchical Framework for Formal Reasoning A Formal network model Automated verification with ACL Overview of DRCP DRCP Vulnerabilities and Attacks Example DRCP attacks A Specification-based Intrusion Detection applied to DRCP Global Security Requirement Motivation for local behavioral specifications Generation of local behavioral specification Formalization and Verification Formalization of security requirements Formalization of Specifications of DRCP Verification about enforcement of security requirements Discussions Summary Formal Reasoning about a Specification-based Intrusion Detection for OLSR protocol Introduction Security Analysis of OLSR Overview of OLSR Attacks Against OLSR A Specification-based Intrusion Detection for OLSR Overview A Correct Behavior Model of OLSR Simulation Results The Framework and Formal Network Model The Framework vi

9 6.4.2 A Formal Network Model Formal Reasoning about OLSR Formalization of OLSR Validation of the OLSR Model Verification of the Intrusion Detection Mechanism for OLSR Discussion Summary Conclusions and Future Work Conclusions On the Hierarchical Model of Formal Analysis On Formal Reasoning about Host-based IDSs On Formal Reasoning about a Specification-based Intrusion Detection for Dynamic Auto-configuration Protocols in Ad hoc Networks Formal Reasoning about a Specification-based Intrusion Detection for OLSR Protocol Future Work Compositional Verification Improving Specifications for Privileged Programs Distributed Monitoring Detection of Temporary Inconsistency Verification on other Systems Bibliography 98 A Specifications for Privileged Programs in UNIX systems 103 B Functions and Theorem of the Verification 126 vii

10 List of Figures 3.1 Framework of Our Approach Verification Hierachy Relationship among security policy, specifications and attacks Mechanism of SHIM to filter concurrent execution audit log Hierarchical Framework for Verification Example Operation of DRCP Example DRCP Attack Global Requirements and Local Specification for DRCP protocol An EFSM Model of DRCP Server Part Generation of a route from Topology Table OLSR Routing Finite State Automata (FSA Hierarchical Framework for Verification viii

11 List of Tables 2.1 The ftp buffer overflow attack The lpr race condition attack Important functions of ACL System Requirement for DRCP Attack techniques in MANETs OLSR Routing Table Establishment ix

12 Abstract Intrusion detection is an appealing approach to improving the security of systems. It involves the runtime gathering of data from system operations, and the subsequent analysis of the data. There are three different kinds of detection models: anomaly detection, misuse detection and specification-based intrusion detection. Anomaly detection compares characteristics of the system with normal behavior profiles and an alert is reported if they do not match. Misuse detection compares current behavior to signatures of known attacks and reports an alert if there is a match. Specification-based intrusion detection monitors behavior of systems according to specifications, which describe desired functionality for securitycritical entities. The goal of intrusion detection is to detect attacks with low false positive rate and low false negative rate. New approach is needed to evaluate the effectiveness of the intrusion detection systems (IDSs. In this dissertation, we present a formal framework for the analysis of intrusion detection systems that employ declarative rules for attack recognition, e.g. specificationbased intrusion detection. Our approach allows reasoning about the effectiveness of an IDS by formalizing and proving security properties of the IDS. Detection rules of IDSs and security requirements of the system are formalized and properties about the IDSs, like security requirements are satisfied by deploying detection rules, are proved within the framework. The formal framework and analysis methodology will provide a scientific basis for one to argue that the IDS detects all attacks that would violate security requirements of the system. To illustrate the validation of our approach, we formalize and proved properties about three typical IDSs including a host-based IDS (SHIM, an IDS for an auto configuration protocol (DRCP and an IDS for a routing protocol (OlSR. For each IDS, detection rules are formalized and security requirements of the system are proved to be satisfied with the enforcement of these detection rules. SHIM (System Health and Intrusion Monitoring is used as an exemplary hostbased IDS to validate our approach. We formalized all specifications of SHIM which together with a trusted file policy enabled us to reason about the soundness and completeness of the x

13 specifications by proving that the specifications satisfy the policy under various assumptions. DRCP (Dynamic Registration and Configuration Protocol is an auto-configuration protocol in mobile Ad Hoc networks. With respect to this protocol, our approach defines a global security requirement for a network that characterizes the good behavior of individual nodes to assure the global property. We formally prove that the local detection rules (identifying activity that is monitored imply the global security requirement. OLSR (Optimized Link State Routing protocol is a proactive, table-driven routing protocol in MANETs. We analyze a specification-based intrusion detection mechanism to detect insider attacks in the OLSR protocol. The specifications, also called local constraints, describe valid behavior of local network. We proved that the intrusion detection approach, which focuses on monitoring of local behavior, achieves a global integrity of network routing information. Our approach, novel to the field of intrusion detection, can, in principle, yield an intrusion detection system that detects any attack, even unknown attacks, that can imperil the security requirements of the system. The originality of our formal analysis is that it is completely based on analytical methods and does not rely on simulation or experimental evaluation. xi

14 1 Chapter 1 Introduction 1.1 Overview of Intrusion Detection Critical computer systems and networks, for economic, functionality, and compatibility reasons, will likely be comprised of commercial-off-the-shelf (COTS products. These COTS systems and software are not designed and built in a way that provides users with confidence in their ability to meet security requirements; nor are these systems secure. It seems new attacks that exploit vulnerabilities in existing systems occur daily. Therefore, there is an imminent need for new techniques that can increase the trustworthiness of COTS systems. Complementing preventive strategies, intrusion detection has been a prevalent technology that is becoming accepted by the community. Intrusion detection is an appealing approach to improving the security of existing systems, as it can be retrofitted onto security-critical programs. Briefly, intrusion detection involves the runtime gathering of data from system operation, and the subsequent analysis of the data; the data can be audit logs generated by an operating system, packets sniffed from a network, or reports from instrumented programs, which could be applications such as a database system. We can categorize IDSs with many different criteria. IDSs can be divided into network-based IDS and host-based IDS by the systems they monitor.. IDSs that monitor network backbones and look for attack scenarios are called network-based IDSs, whereas those that operate on hosts defend and monitor the operating and file systems for signs of

15 2 intrusion are called host-based IDSs. Normal behaviors of a computer system have the characteristics of predictable statistics, the absence of a known attack scenario and a conformation to specifications [13]. Intrusion detection systems are developed on the assumption that attacks will break at least one of the characteristics. There are three different kinds of intrusion detection models: anomaly detection, misuse detection and specification-based intrusion detection. Anomaly detection, first proposed by Anderson [22], is based on the assumption that the characteristics of attacks are significantly different from normal behavior. The characteristics of the system are compared with the expected values and an alert is reported if they do not match. Anomaly detection is capable of detecting unknown attacks or variants of known attacks if such attacks significantly change the monitored characteristics of the system. But the trade-off is that detection often comes with high false-positive rate, which means tat normal system behaviors will be reported as potential attacks. Misuse detection compares current behavior to signatures of known attacks and reports an alert if there is a match [49, 41]. Misuse systems have difficulty detecting unknown attacks as well as variants of known attacks. Whether generalized attack signatures are capable of detecting unknown attacks still remains an open question. Specification-based intrusion detection monitors current behavior of systems according to specifications that describe desired functionality for security-critical entities [26, 30, 56, 29, 67]. A mismatch between current behavior and the specifications will be reported as an attack. Privileged programs, also called root-setuid programs, are often analyzed by a specification-based intrusion detection system because of their significant impact on system security. The effective user of a privileged program has root privileges and attacks against a privileged program often abuse the privilege to access security critical objects that are not supposed to be accessed by the victim programs. During program operation, the system accesses associated with the operation of a program are recorded in audit logs and matched against the specification by the IDS. Mismatches are reported and almost always indicate an attack. Theoretically, specification-based intrusion detection is capable of detecting unknown attacks or variant of known attacks and a report is issued as soon as a specification violation occurs.

16 3 While intrusion detection systems can improve the security of a system, it is hard to evaluate the effectiveness of the IDS with respect to the primary objective users have for the deployment of such systems: the ability to detect large classes of attacks (including variations of known attacks and unknown attacks with a low false alarm rate. In addition, it is hard to assess, in a scientific manner, the security posture of a system with an IDS deployed. Even more important to users than detecting attacks is to know that IDS will generate alerts if there is any violation of a security policy. 1.2 Limitations of Earlier Work So far, experimental evaluation and testing have been the only approaches that have been attempted to evaluate the effective of an intrusion detection system. First, the intrusion detection system is subjected to test data, which are normal (e.g. attack-free, to develop a profile of normal behavior. Then it will be challenged by test data that contain attacks, like DARPA Off-line Intrusion Detection Evaluation [50]. Results, including the detection rate for real attacks and false alarm rate, are used to assess the effectiveness of the intrusion detection system. The usefulness of these evaluations depends on the coverage of the test data and the sensitivity of the test results with respect to changes in the environments. However, it is hard to obtain test data from real environment that are known to be free of attacks or obtain test data that cover attacks representing most attack categories. Attacks experienced by a system quite often change with time as variations of known attacks surface or new attack emerge. Thus it is difficult to develop a set of test data that is comprehensive and that has strong prediction power with respect to changes in operating environments. There is a critical need to establish a scientific foundation for evaluating and analyzing the effectiveness of intrusion detection systems. This research employs formal reasoning to analyze intrusion detection systems, attempting to provide a scientific foundation for assessing the security of the overall system. We consider host-based systems and network-based systems, reasoning about intrusion detection systems in isolation and their composition.

17 4 1.3 Contribution of this Thesis This thesis makes two contributions in evaluation of the effectiveness of intrusion detection systems. The key contribution of this research is a mechanized methodology to formally reason about the effectiveness of intrusion detection systems in satisfying specific security requirements. This is one of the first approaches to analyzing detection rules of an IDS with formal methods. A hierarchical verification model is proposed, in which we formalize an abstract system model, detection rules and security requirements. Theorems are created and proved to reason about the soundness and completeness of detection rules. Second, we implement the framework in our analysis of a host-based intrusion detection system, SHIM and network intrusion detection systems. The specifications in SHIM are formalized and reasoned about with respect of simulated attacks and access control policies. In the analysis of network intrusion detection systems in ad hoc networks, we develop global requirements to describe the security properties of ad hoc networks and local specifications that restrict behavior of nodes. We formally prove that the global requirements are enforce by the local specifications together with some assumptions. Parts of the research are represented in my papers[54, 59, 64, 60]. Taken together, this research puts formal evaluation of intrusion detection systems on much further footing, and provides direction for the continued development of the area. 1.4 Introducing Terminology This section explains some terms that are used throughout this proposal. Some of the terms have well-accepted definitions among security professional while others have specific use in this dissertation. Attack: An attack is any set of actions whose purpose is to compromise the integrity, confidentiality, or availability of a resource. The set of actions may be performed by a single attacker or by a group of cooperating attackers. An attacker exploits vulnerabilities in a system to gain necessary privileges to achieve his/her goal. Auditing: Auditing is the analysis of log records to represent information about

18 5 the system in a clear and understandable manner [34]. Exploitation: An exploitation is a set of actions that result in a violation of the security policy of a computer system. Intruders exploit system vulnerabilities or flaws to gain unauthorized access to the system. Logging: Logging is the recording of events or statistics to provide information about system use and performance [34]. Security policy: A security policy is a statement that partitions the state of the system into a set of authorized, or secure states and a set of unauthorized or nonsecure states [34]. Specification: A specification is a description of characteristics of a computer system or program. Specification-based intrusion detection: An intrusion detection method which monitors current behavior of systems according to specifications that describe desired functionality for security-critical entities and raises an alert if there is a violation. Privileged program: A privileged program is a program that is executed with special privileges, enabling it to bypass the system security mechanism in order to accomplish its task. In Unix, examples are those programs that are owned by root and have the setuid bit on. Vulnerability: A vulnerability is a weakness in automated system security procedures, administrative controls, or internal controls that could be exploited by a thread to gain unauthorized access to information or to disrupt critical processing. 1.5 Dissertation Outline The rest of this dissertation is organized as follows. Chapter 2 presents a framework of our approach. Chapter 3 describes our analysis of a host-based IDS, SHIM. In chapter 4 we formalize security requirements for DRCP protocol and formally address the enforcement of these security requirements by a specification-based IDS. Chapter 5 presents a distributed IDS for OLSR protocol and analyze this IDS in fulfilling security requirements for OLSR. Chapter 6 discusses various issues of our work. Chapter 7 concludes our work and gives

19 some suggestions about future research. 6

20 7 Chapter 2 Motivating examples This chapter presents several examples aimed at illustrating the subtleties involved in developing detecting rules and security policies to detect intrusions and the associated pitfalls. The examples motivate the need for a formal framework for reasoning about detection rules in intrusion detection systems. The overall approach is to prove properties about intrusion detection systems. Our proof method depends on detecting rules and a security policy, where these rules are reasoned about with respect to the policy. 2.1 Examples of Intrusion Scenarios on Host-based Systems We present two well-known intrusion scenarios, each exploiting a different class of vulnerabilities (buffer overflow, race condition. Detection rules and security policies, which detect or prohibit these attacks, will be introduced in the following sections Ftp Vulnerability Ftp daemon is a Unix utility for providing file transfer service to remote clients. Ftp daemon is a setuid root program, or previleged program, because root privileges are required to monitor the ftp port, which is a system port in UNIX. The fact that ftp daemon runs with root privilege enables it to perform operations inaccessible by normal users. Therefore if ftp daemon is comprised, the attacker may acquire root privilege. This is the basis for the exploitation. There is a buffer overflow vulnerability for an early version

21 8 of wuftp daemon(2.4.2-beta-18. The vulnerability can be exploited by an attack which enables the attacker to invoke any arbitrary system call. Specifically, wuftp daemon invoke a strcat( C library call without restrict the size of the input. As a result, when wuftp daemon is given a very long request, the data could overwrite the memory location beyond the buffer, causing unexpected behavior. An elaborated organized request may even invoke arbitrary system calls. Due to the fact that wuftp daemon is a setuid program, the attacker can invoke a shell with root privilege and use it to access any files of the system [8]. Table 2.1 depicts a trace of the exploit. When strcat( is called, the run-time stack consists of the activation record of strcat(, which include the parameters, the return address, and local variables. When an attacker sends a long request message consisting of code to the ftp daemon, strcat( fills the read buffer and beyond with the message so that the buffer now contains the injected code and the return address is overwritten with an address pointing to the buffer. When the subroutine returns, it branches into the buffer and executes the attacker s code. Obviously, we can detect aattacks that exploit this vulnerability by checking the parameters of strcat(. However, most auditing systems do not record the content of strcat( operations because of the huge volume of data associated with these operations. Therefore, it is not possible to tell whether from the audit trails, wuftp daemon is under a buffer overflow attack. However, if we look at the operations performed by wuftp daemon, we can detect the intrusion by comparing the operations performed by wuftp daemon in a normal execution and under an attack. Another idea is to detect the buffer overflow attacks by examining audit data for attack signatures, like sequences of operations or strings which are used to exploit the vulnerabilities. However, it is not difficult to develop variants of known attacks which have different signatures but still exploit the same vulnerabilities. Therefore, the signature-based approach may have high false alert rate in detecting attacks exploit specific vulnerabilities Lpr Vulnerability This section describes a race-condition vulnerability in an early version of lpd(line printer Daemon utility and related intrusions that exploit this and similar vulnerabilities. Lpd is a UNIX utility which provides printing services. After a printing job is submitted

22 9 ftp daemon attack comments Len =... Set the length of the buffer t=xcalloc(len,... Malloc memory Strcat(t,... Buffer overflow attack 7... Following attack behavior 8 Open(O RW, /etc/passwd... Change the passwd file 9... Table 2.1: The ftp buffer overflow attack by a client, lpd will check whether the client has the privilege to access the file. After that lpd will read the file and send it to the printer. Lpd is a setuid root program because root privileges are required in order to send files to printing devices. Therefore, lpd run with root privileges and can access any arbitrary file in a UNIX system. Lpd has a race condition flaw, which enables an attacker to read files which is inaccessible to him. The flaw relates to the way lpd check the permission of a file before printing it as well as its semantics in dealing with symbolic links. Specifically, the flaw enables a nonprivileged user to print out any file that he does not have privileges to read. The lpr attack exploits the race condition vulnerability. Please refer to table 2.2 for a simple exploit. First, the attacker requests the printing of some benign file (for example, his own file doc.txt. His rights are checked by the lpr command and, if access is granted, the request is put into the printer daemon queue. Then, before the printing actually starts, the attacker removes the printed file and replaces it by a link to some file he is not allowed to access (for example, /etc/shadow. As a result, the latter file will eventually be printed on his behalf. We can detect this intrusion in two ways. First, we can detect it by just looking at the operations performed by lpd. Although the sequence of calls and parameters performed by lpd are the same when it is under the race condition attack, the actual physical files printed by the calls are different. In Unix, each physical file is identified by a unique inode number. In the normal sequence, lpd prints the file corresponding to the inode of the

23 10 lpd daemon attack 1 lpr doc.txt 2 read /var/spool/lpd/job-**** 3 rm doc.txt 4 ln -s /etc/shadow doc.txt 5 read doc.txt 6 send document to printer Table 2.2: The lpr race condition attack file whose permissions have been checked. However, in the sequence of audit records of lpd under the exploitation, the file being printed and the file being checked have different inodes. Thus, by checking the inode of the files being printed, which are usually available in the audit trail, we can detect occurrences of this intrusion. Second, we can detect the intrusion by checking whether operations that change the meaning of the file are performed by other processes between the time lpd checks the file and prints files. Some ways of specifying the valid order among these operations are needed in order to detect the intrusion in that way. In section 3, policies regarding noninterference will be introduced and they will provide general solutions for race condition attacks. 2.2 Detecting Attack Scenarios In this section, we give an overview of intrusion detection systems first. Then a few examples are presented to indicate the subtleties involved in developing detect rules Overview of Intrusion Detection Systems We can categorize IDSs according to different criteria. IDSs can be divided into network-based IDS and host-based IDS by the systems they monitor.. IDSs that monitor network backbones and look for attack scenarios are called network-based IDSs, whereas those that operate on hosts defend and monitor the operating and file systems for signs of intrusion are called host-based IDSs. Normal behaviors of a computer system have the characteristics of predictable statistics, the absence of a known attack scenario and a conformation to specifications

24 11 [13]. Intrusion detection systems are developed on the assumption that attacks will break at least one of the characteristics. There are three different kinds of intrusion detection models: anomaly detection, misuse detection and specification-based intrusion detection. Anomaly detection, first proposed by Anderson [22], is based on the assumption that the characteristics of attacks are significantly different from normal behavior. The characteristics of the system are compared with the expected values and an alert is reported if they do not match. Anomaly detection is capable of detecting unknown attacks or variants of known attacks if such attacks significantly change the monitored characteristics of the system. But the trade-off is that detection often comes with high false-positive rate, which means tat normal system behaviors will be reported as potential attacks. Misuse detection compares current behavior to signatures of known attacks and reports an alert if there is a match [49, 41]. Misuse systems have difficulty detecting unknown attacks as well as variants of known attacks. Whether generalized attack signatures are capable of detecting unknown attacks still remains an open question. Specification-based intrusion detection monitors current behavior of systems according to specifications that describe desired functionality for security-critical entities [26, 30, 56, 29, 67]. A mismatch between current behavior and the specifications will be reported as an attack. Privileged programs, also called root-setuid programs, are often analyzed by a specification-based intrusion detection system because of their significant impact on system security. The effective user of a privileged program has root privileges and attacks against a privileged program often abuse the privilege to access security critical objects that are not supposed to be accessed by the victim programs. During program operation, the system accesses associated with the operation of a program are recorded in audit logs and matched against the specification by the IDS. Mismatches are reported and almost always indicate an attack. Theoretically, specification-based intrusion detection is capable of detecting unknown attacks or variant of known attacks and a report is issued as soon as a specification violation occurs.

25 Developing Detection Rules In designing an intrusion detection system, one must select which kind of detection rules will be developed to detect intrusions. These detection rules include specifications of a specification-based intrusion detection system and attack signatures of misuse detection systems. Anomaly detection systems will not be analyzed because most of them are statistics-based and difficult to analyze with logic while specifications and signatures are declarable. Development of correct intrusion-detection rules is a very difficult and error-prone task: it involves extensive knowledge engineering on attacks and most components of the system; it requires a deep and correct understanding of most of the components in a system and how they work together; it requires the rule developers to be cautious and careful to avoid mistakes and gaps in coverage. Often, it is very difficult to assess whether a given set of intrusion-detection rules is correct (they detect the attacks or enforce security policies. We discuss the subtleties involved in writing valid behavior specifications for a program. Traditionally, in specification-based IDSs, a valid behavior specification for a program declares what operations and system calls are allowed for the program. Whether an operation is allowed or not depends on the attributes of the process and the reference, and attributes of the system calls. In SHIM, a specification for a program is a list of rules describing all the operations valid for the program. For example, the following rule in the ftp daemon (ftpd specification allows the program to open any file in the /var/log/wtmp directory to write. (OPEN W R,$F.path == /var/log/wtmp With specifications that list valid operations of security-critical programs, an IDS can monitor the execution of the programs and raise an alert if the specifications are violated. If the specifications are correct and complete, the IDS is capable of detecting all, including known and unknown, attacks that change the behavior of the programs. However, the draft specifications, when used to monitor the program execution, will produce false positives (i.e., valid operations performed by the program reported as erroneous because they are

26 13 not included in the specification. Then, one augments the specification to include rules to cover these operations. However, one needs to be very careful in writing the specification for a program to avoid some attacks going undetected. For example, given the above rule, if /var/log/wtmp somehow is writable by attackers, they can create a link from /var/log/wtmp/file to the /etc/passwd file. A specification-based IDS with this rule in the specification of ftpd will permit this operation and the attack will go undetected. For the intrusion scenarios that we introduced in the previous section, the ftp buffer overflow attack can be detected by a specification-based IDS if the attacker intend to invoke a shell or open some security-critical objects, like the passwd file; the lpr race condition attack, however, may not be detected because the attack does not change the observed behavior of the lpd program. We can augment the specification of the lpd to check the number of links to the target object and to generate a warning if the number of links to the object is greater than one, thus preventing this kind of attacks. Based on our experience, writing specifications for a program is subtle and tricky, thus demanding an approach to rule validation. 2.3 Enforcement of Security Policies In the current view, rules are evaluated against known attacks and hypothesis unknown attacks. How good are these rules? This thesis addresses this question by proposing that the rules can be reasoned about with respect to the security goal, e.g. a security policy. One interesting question about intrusion detection is, given the detection rules, whether the high-level security policies, like access control policies, can be satisfied. In this context, we mean that actions that violate the security policies are either prohibited or detected. Most computer systems employ access controls policies as the first line of defense to protect resources and information in the systems. The access control policies limit access to files and other operating system objects based on their ownership and the settings of the owner, group and others permission bits. To ensure the security of a system, the access

27 14 control mechanism should work properly. And, an intrusion detection system should be able to monitor violations of the standard access control policy and raise an alarm. In addition, we identify a suite of policies at different levels of abstraction to specify the full breadth and depth of intrusion detection mechanisms and detect different attacks. This hierarchy will be addressed in the formal specification and verification described in this dissertation Trusted File Policies Initially, we start with a simple and easily understood integrity policy, which states that certain files cannot be modified. For example, a UNIX system relies on a number of privileged programs and devices drivers. These programs often read data from configuration files, trusting the integrity of these files. These trusted files must be specified as special files by the policy and then protected as such. Trusted file access policies are security policies that we developed to keep trusted files from unauthorized access. In UNIX systems, a discretionary access control (DAC mechanism defines whether a subject can access an object or not depending on the privilege of the subject and the access permission of the object. Some files are intended to be accessed by specific users or using specific programs. Thus, trusted file access policies are defined in our format as: (trusted file, authorized user, program, access where trusted file is the file to be protected, authorized user defines the user that can aaccess the file with any programs and program defines the program that can be used by other users to access the file. As an example, the passwd file access policy is defined as: (/etc/passwd, root, passwd, (open-wr,create, chmod, chown, rename. It indicates the passwd file of a UNIX system should be editable by root using any program or by an ordinary user using the passwd program. Therefore, if an attacker compromises a ftp daemon and try to access the passwd file, the passwd file access policy will be violated. In fact, this policy prohibited any unauthorized access to the passwd file. Therefore, if this policy is enforced by an IDS, it means any unauthorized access to the passwd file will be detected by the IDS.

28 Policies Regarding Race Condition Attacks A few policies are proposed to detect race condition attacks. Zimmermann et al. proposed a reference flow control model to policy-based intrusion detection [70, 71]. The authors claim that their approach is capable of detecting unknown noninterference behaviors which violate security policy specifications of a system [71]. In the reference flow control model, a security policy is enforced by defining domains that specify sets of method calls, each combination of these calls producing authorized information flows. Any cross-domain information flow operation will be considered as an intrusion. Race-condition attacks can be detected by observing information flows from an object created by an attacker to objects that he is not authorized to access. Ko et al. proposed another approach to detect race-condition attacks by checking valid interleaving between privileged and unprivileged processes [28]. They developed a noninterference model which ensures that concurrent behavior of two programs will have no impact on behavior of each programs. Their approach only bases on the knowledge of system call execution interleaving legality which is defined once and for all by the implementation and is considered to be more simple and efficient in detecting race-condition attacks. The lpr race condition attack, which we introduced in the previous section, can be detected by both approaches. Formal [42, 53, 3] techniques for assuring the security of a system have been investigated. It is our view that a security policy should support formal reasoning. 2.4 Need For a New Approach As illustrated by the examples, writing intrusion-detection rules for specificationbased and misuse detection is tricky. Existing techniques for evaluating the intrusiondetection rules, primarily based on testing, have limitations. A major problem is the development of a comprehensive suite of test cases, which are typically manually generated by intrusion detection experts, based on observed attack instances and analysis of captured attack tools. The process of developing the test suite is thus labor-intensive and errorprone. Moreover, the trend toward increasing sophistication of attacks such as multi-step

29 16 attacks, distributed coordinating attacks, and the use of code obfuscation and IDS evasion techniques makes IDS testing more and more difficult. An automated tool that can assist IDS developers to generate, analyze, and verify the correctness of the rules, that they are both necessary and sufficient, will be of benefit to the process of evaluating the effectiveness of intrusion-detection rules. In the next section, we will present a complementary approach, based on a formal framework, for reasoning about intrusion-detection rules.

30 17 Chapter 3 The Framework 3.1 Introduction We develop a formal framework that permits analysis of the detection rules in intrusion detection systems. This analysis can be used to identify errors and improve the quality of the rules. In particular, we focus our effort on misuse detection and specificationbased detection, both employing declarative rules to detect intrusive activity. Based on our experience, developing good misuse signatures and good specifications for programs is tricky and error prone. It often requires insights into the attacks and critical security aspects of a system. In addition, one needs to be very careful when writing them to avoid gaps in coverage. In particular, it is hard to judge whether changes to the rules actually improve or degrade the ability to detect new attacks. This research explores the application of techniques from formal method research to assist the development and analysis of detection rules. We employ verification techniques to prove that a given rule set, together with the operating system, can enforce a given security policy. In addition, we enumerate assumptions of specific security policies to identify possible ways the policies can be violated without being detected by the detection rules. Thus we abstract any special attacks and strict to prove that our IDS will detect any attack that threatening the security policy. The assumptions cover the kind of attacks that cannot or very unlikely to occur. In particular, we model the expert rules in host-based IDSs and network IDSs in this research. We envision that the analysis allows one to provide a solid argument that the

31 18 given intrusion detection rules can detect certain attacks and violations of certain security policies, thus providing assurance of the security of the overall system beyond the level of assurance achieved by experiments and testing. We build specifications of network protocols and use them to analyze packet traffic sniffed from the network. We verify security invariants that can be enforced by these specifications. In addition, to increasing the coverage of intrusion detection systems and our confidence in that coverage, we provide a methodology to make future enhancements and to allow others to continue this work in other areas, with other intrusion detection systems, and on other platforms. -.* / :( +:, 6(4 +, * - ** +, ' ( *+, -.* / - 4 5' ( +, 8 : 6(4 +7,!"#$%& ;< / ( Overview Figure 3.1: Framework of Our Approach Our approach to formal analysis of intrusion-detection rules is inspired by the formal methods research on designing and building trusted computer systems (Figure 3.1:

32 19 Left side. Here, a model of a system is shown to satisfy a security policy. Formal methods involve the use of mathematical logic to model a system that enables one to reason about the properties of the system. Formal methods have been used to discover errors that occur in the process of designing and building a computer system. Formal methods are feasible for reasoning specifications with respect to a security policy, e.g. A1 certification as in the Orange Book. Our reasoning, although new and novel, offer a similar complexity. We reason about the detection rules with respect to security requirements of a system (Figure 3.1: Right side. The process of designing and building a trusted system involves the development of a security model. A security model usually consists of a specification of a security policy (which defines the security requirements or what is meant by secure and an abstract behavioral model of the system. Usually, the security policy can be stated as a mapping of system states into authorized (secure and unauthorized (insecure states; or can be a property of the system (e.g., noninterference. The model is an abstraction of the actual system which provides a high level description of the major entities of the system and operations on those entities. There may be many layers of abstraction within the model, each a refinement of the higher level abstraction. Given the security policy and model, one should be able to prove that the model satisfies the security policy assuming some restriction on the state transition functions. The behavioral model itself can be used to guide the design of the system. In addition, one can then formally specify the design and prove that the design actually adheres to the model. Further, one can also mathematically prove that the implementation (software and hardware correctly implements the abstract system model, or rigorously test the code. Usually, the verification is done in a hierarchical manner in which a hierarchy of abstract machines, or refinements of the system, is defined. 3.3 Hierarchical Model of the Framework We develop a security model for IDS. This IDS security model will consist of an abstract behavioral model of an existing system, specifications of high-level security properties, and specifications of intrusion-detection rules. Rather than relying on a system with