Stellar: A Fusion System for Scenario Construction and Security Risk Assessment

Transcription

1 Stellar: A Fusion System for Scenario Construction and Security Risk Assessment Stephen Boyer, Oliver Dain, and Robert Cunningham MIT Lincoln Laboratory Information Systems Technology Group 244 Wood St., Lexington, MA {boyer,odain,rkc}@ll.mit.edu Abstract Stellar aggregates and correlates alerts from heterogeneous network defense systems, building scenarios and estimating the security risk of the entire scenario. Prior work considered Stellar scenario formation; in this paper we explore the advantages provided by using scenario context to assess the risk of actions occurring on a network. We describe the design and an evaluation of Stellar and its Security Assessment Declarative Language (SADL), a fast, stateful, simple-to-use language for assessing the priority of scenarios, on a high traffic network under constant attack. The evaluation of the Stellar system deployed on a large, operational enterprise network demonstrated its ability to scale to high alert volumes while accurately forming and prioritizing scenarios. Stellar not only produced high priority scenarios matching all incidents reported by human analysts, but also discovered additional scenarios of concern that had initially gone unnoticed. Furthermore, by following the simple formalism embedded in example SADL rules, system administrators quickly develop a correct understanding of the network they are protecting 1. Keywords: security, fusion, scenario, correlation, intrusion detection 1 Introduction Many security-conscious organizations protect their networks with multiple computer network defense (CND) systems such as intrusion detection systems (IDSs), firewalls, 1 This work is sponsored by the Department of Defense under the Air Force Contract F C Opinions, interpretations, conclusions and recommendations are those of the author and are not necessarily endorsed by the United States Government. and routers with access control lists. While this defensein-depth strategy may improve attack detection rates, it also increases the workload of security analysts. Analysts now must expend more time and employ other resources to monitor and appropriately act on information produced by these sensors. Inundated by the large alert volume, some analysts cope by ignoring alerts that describe communication to known patched services or tune sensors to suppress such alerts. The remaining alert volume is often still large, making cross-sensor alert correlation difficult. Attack detection becomes even more difficult when network services are dynamic, multiple organizations share networks, or when users are allowed to enable and disable services on their machines. Fusion systems attempt to address these problems by providing tools to enhance the presentation and management of sensor data. In this paper we present Stellar; a real-time system that combines the alerts produced by multiple CND systems into prioritized scenarios. Stellar consists of two distinct components: the scenario building engine and the security risk assessment engine. The scenario building engine combines alerts that share a common cause. At this stage no effort is made to differentiate real attacks from false alarms. The purpose of grouping alerts is to provide a context for making security risk assessments. Security risk is determined by a set of rules written in Security Assessment Declarative Language (SADL), an easy to learn declarative language similar to the Structured Query Language (SQL) [1]. SADL is able to combine knowledge of the network environment (critical servers, mission, known vulnerabilities, etc.) and the context in which an alert appears when making an assessment. The ability to exploit context in risk assessment allows Stellar to simultaneously increase accuracy and decrease false alarm rates. Stellar was recently evaluated on a large operational network. This evaluation demonstrated the system s ability 1

2 to accurately prioritize scenarios. The system reduced the amount of data analysts had to consider by several orders of magnitude without missing any of the high priority security events identified by security professionals considering the entire alert set. Furthermore, the dynamic distributed nature of the network made it nearly impossible for the analysts to be completely aware of all the services offered on the protected network despite their skill and expertise. Stellar provides a unified framework to describe the network environment. We discovered that when administrators views of that environment (as embodied in SADL statements ) were incorrect, Stellar enabled them to quickly update their view of the network. A brief overview of related work is presented in Section 2. Stellar s approach to alert fusion is summarized in Section 3. The system architecture is described in Section 4. Section 5 provides a look at the SADL language, and the results of the recent system evaluation are presented in Section 6. 2 Related Work Network security alert correlation has become an area of active research. Much of the research has focused on discovering complex, multi-step attacks from alert data. One popular approach to this problem is to utilize sets of rules or knowledge bases to determine which alerts should be correlated. SCYLLARUS [8] and AlertSTAT [23] both employ expert knowledge to determine alert correlations. SCYLLARUS relies on the maintenance of a knowledge base which contains a list of assets, security goals, possible alerts, and the impact each alert would have on security goals. This information is used to produce hypotheses and determine which hypotheses are likely enough to report. AlertSTAT gets its expert knowledge from a set of rules written in the STATL language. Each rule specifies a set of state transitions that define the steps in a multi-step attack. A distinct rule is required for each attack scenario type. A similar approach is employed by prerequisite systems. These systems maintain information about the conditions necessary for an attack step to succeed and the conditions that are produced by a successful attack step. Multistep attack scenarios can then be discovered by searching for chains of alerts such that the post-conditions of earlier attacks satisfy the pre-conditions of later attacks. Ning and Cui describe one such system which is designed for use offline [17]. The CRIM component of MIRADOR is also designed to run offline, but includes a component to decide which scenario to report when it is possible to construct multiple plausible scenarios from a set of alerts [11, 3]. The EMERALD suite of intrusion detection tools includes a probabilistic alert correlation component [?, 22]. This system treats alerts as vectors and fuses alerts if their similarity is high enough. It is necessary to maintain a database that maps each alert type produced by the component IDSs to an attack class and a matrix that specifies the similarity between the attack classes. Most of the above systems require the maintenance of a lot of expert knowledge. For example, the prerequisite systems require the maintenance of a database of alert types, the attacks to which they correspond, and the pre and post conditions of those attacks. Stellar s correlation component requires no expert knowledge encoding. The Stellar prioritization engine, based on the SADL language, does require the user to encode some expert knowledge, but every effort is made to keep this to a minimum. Furthermore, SADL separates information about the local network configuration, which most systems administrators know, from information about common alerts and attack patterns. See Section 5 for details. The above systems assume a reasonably high fidelity alert stream. False alarms can lead to the construction of scenarios which never occurred and false negatives will fail to reveal attack steps critical to the success of the correlation mechanism. To combat the false negative problem systems must hypothesize attack steps that did not produce an alert (Cuppens and Miège call this abductive correlation [11]) but this can have an adverse effect on run time and can exacerbate the false positive problem as it is now more likely that false alarm alerts will be erroneously combined with other alerts. Combating the false positive problem is critical for systems that intend to correlate the output of the current state of the art intrusion detection tools as these sensors are not very accurate and produce a lot of false positives (some researchers have suggested that as much as 99% of typical alarm traffic is false alarms [2, 13]). If network defense systems provided the kind of detailed data described by the M2D2 data model [16] it might be possible to determine if the alerts correspond to real attacks and if those attacks were successful. However, currently deployed systems generally do not provide this kind of information making complex correlation difficult. Additionally, complex correlation systems can produce false scenarios even when the alerts are completely accurate. For example, the hosts on most networks we have examined are scanned and attacked by worms multiple times per day. There are thus many scans followed by many vulnerability exploit attempts. There are now many ways to combine these alerts (n scans followed by m exploit attempts yields n m possible scenarios) and these unrelated events may be erroneously combined to yield a scenario that looks directed and sinister. Because of these difficulties Stellar does not attempt to form complex scenarios except in the case where there is overwhelming evidence to support such a hypothesis. Instead the system performs simple correlation and focuses 2

3 on fine grained prioritization that accounts for alert context. All of the systems described above focus primarily on correlation and place little emphasis on prioritization. The M-Correlator extension to the EMERALD system (which is not the same as the probabilistic component described earlier) allows analysts to write rules, an Alert Cluster Policy Specification, to combine similar alerts [20]. These rules specify how to combine multiple alerts which correspond to the same incident but they do not attempt to correlate alerts from multiple steps of an attack scenario. An extension to the Tivoli management system correlates first by finding duplicates (multiple alerts for the same event) and then consequences (one alert being caused by another) [7]. Both types of groupings are formed by consulting a set of rules. An additional level of correlation can be performed which attempts to discover more complex scenarios called situations. While these situations can discover attack scenarios, the rules are biased toward very simple scenarios. Situations require all the alerts to have the same source IP, the same destination IP, or be of the same type. This is the type of simple correlation advocated by the authors. In contrast to Stellar, both of these systems perform prioritization on each alert in isolation instead of leveraging the context correlated alerts provide. M-Correlator s prioritization algorithm accounts for mission, alert vulnerability dependencies, and network topology, but the calculation is performed on a per-alert basis. Furthermore, like the prerequisite approaches described above, the vulnerability dependency information must be provided for each alert. The Tivoli system s duplicate and consequence rules can change the severity of an alert (e.g. raising the severity if duplicates are found) but these are configured statically and do not account for network topology or mission. Ourston et. al have developed a system that performs simple correlation and complex prioritization [18, 19]. The correlation component combines all alerts between a single pair of hosts. Prioritization is performed by Hidden Markov Models (HMMs) that attempt to match the alerts to a series of attack steps. Because the Markov Model allows for attack steps that did not produce any alerts (the hidden Markov states) the system can handle the false negative problem. The false positive problem is not directly addressed. Each possible attack scenario is encoded in the HMM and different scenarios have different priorities. This prioritization mechanism is quite different from Stellar s which utilizes a set of rules written by a domain expert. Julisch and Dacier advocate data mining historical alarms to discover alarm patterns [13]. A domain expert can then convert the discovered patterns into filters which remove common false alarms. As SADL is a powerful language that allows analysts to write the kind of rules discussed by Julisch and Dacier it might be beneficial to combine the approach they advocate with the Stellar system. Morin and Debar propose using Chronicles, a language for expressing sequences of alerts for correlation in [15]. While the time complexity of Chronicles as a function of the number of rules is addressed, the complexity as a function of the number of alerts, which is more likely to be an issue, is not addressed and could be problematic as the number of alerts grow large. 3 Stellar Approach The Stellar system defines scenarios to be collections of alerts that share a common cause. Scenarios are not predefined and the cause need not be malicious behavior. A misconfiguration or a group of false alarms could form a scenario. The constructed scenario then provides a broader context in which a priority assessment can be made. The following examples illustrate types of scenarios that an analyst might encounter: User Error Scenario: A valid user mistypes his password and fails to authenticate to a remote file server. This failed remote login attempt could trigger multiple alerts from host-based as well as network-based intrusion detection systems. All alerts caused by the failed login form a scenario. Misconfiguration Scenario: A host on the network has its default gateway incorrectly configured. When the host attempts to send a packet to a remote host, the packet is accepted through the firewall and reaches the router. The router then replies to the host with an ICMP Redirect message that is then again accepted through the firewall and received by the host. This exchange could trigger alerts from network-based IDSs, the firewall, and the router. All alerts caused by the misconfiguration form a scenario. Reconnaissance and Exploit Attempt Scenario: An adversary probes the network by scanning for hosts running a vulnerable server. At a later time, the adversary launches a known exploit. The scanning and attempted exploit activity could be observed by the firewall, as well as host-based and network-based intrusion detection systems. Once again, all alerts produced by these sensors as a result of the scanning and exploit attempt form a scenario. The intent of scenario construction is to provide context so that priority might be better assessed. Grouping alerts obviously reduces the amount of data. However, data reduction is not the only benefit. Grouping affords additional context that may reduce the number of false alarms and improve detection rates. 3

4 Alerts typically describe a single network or host event as seen from one sensor. When alerts are assessed in isolation, the analyst and/or correlation system must make a priority determination based solely on the information that a single alert provides. Alerts often provide information about real events that are nonetheless benign in order to also report on the isolated case when an event is due to a malicious operation. The benign but common operation swamps the malicious but rare operation with the predictable result being that many administrators disable the alerts that both the malicious and benign activities generate. Administrators that do not adopt this practice are faced with continuously sifting through benign alerts to determine the root cause of an event. As a result, the false alarm rates are still unacceptably high and certain types of attacks can be missed as the corresponding alerts have been disabled. Assessing risk in context allows analysts to keep low accuracy rules enabled on the devices. Typically these alerts represent false alarms and Stellar will assign the scenarios containing them a low priority. However, when the context warrants more concern, the priority of the scenario will be elevated and attacks will not be overlooked. Additionally, since alerts have not been disabled on the CND devices, potentially valuable forensic information is not lost. For example, consider the User Error Scenario above. Analysts will not normally be concerned with failed login attempt alerts as such alerts generally reflect a user forgetting or mistyping a password. As a result, security analysts might choose to disable these alerts in order to reduce the volume of false alarms. However, disabling these alerts leaves the analyst blind to password guessing attacks. Using Stellar, failed login alerts are grouped into scenarios. Those scenarios containing only a few failed login attempts could be assigned low priority and would not adversely affect analyst work load. However scenarios containing many failed login attempts could be assigned a higher priority so that password guessing attacks would not go unnoticed. In general, the evolving context allows Stellar to differentiate benign login errors from serious security threats. 4 Architecture The Stellar architecture is depicted in Figure 1. The architecture was designed with scalability in mind. Each component can run on a separate machine on different network segments. The modular design distributes the computational load and facilitates deployment customization. 4.1 Sensors, Bridges and Data Storage Because Stellar is a fusion and correlation system, it relies on the quality of the alert information produced by the CND devices deployed. As nearly every CND device produces alert data in its own format, the Stellar system employs custom bridges to aggregate, normalize, and store different sensors data in a relational database. 2 Integrating a new CND device into the Stellar system requires writing a bridge to convert the sensor s output to SQL statements that can be executed on the Stellar database. Due to the ubiquity of SQL, developers can design bridges using a wide variety of tools and computer languages. Currently, the Stellar system has bridges that accept information from RealSecure Network and Server sensors [12], Snort [21], Checkpoint firewall [14], and Battlefield Intrusion Detection System [4], a host based intrusion detection system. 4.2 The Engine, Correlation, and Prioritization The Stellar Engine has two distinct tasks. The first task is to read the alert information from the database at configurable intervals and form scenarios. Scenarios are formed based on the Scenario Generation Rules (SuGaR) that can be either explicitly written or learned through training [5],[6]. Once the scenarios are formed, the Engine assigns a priority to the entire scenario based on the SADL rules specified by a analyst. 4.3 Using Stellar The Stellar Graphical User Interface (GUI) is used for both system training and scenario monitoring. Training mode facilitates interactive learning of SuGaR, helping the analyst to discover scenarios and build examples. Modifications to the SuGaR rules would allow us to form complex scenarios using combinations of different attributes and measures. In order to evaluate the SADL engine for this paper, a simple SuGaR rule that builds scenarios from alerts arising from the same source IP address was used. Although capable of much more sophisticated correlation, this simple grouping was found to provide the best performance in terms of scenario formation on many data sets [5]. Monitoring mode displays scenarios as they are formed and prioritized. Users can assign names and comments to developing scenarios as well as produce and print scenario reports. As scenarios form, analysts can sort the scenarios by priority and quickly focus their attention on those which have been assigned high priority. Multiple analysis can cooperate by running their own local copy of the GUI and observing scenarios, SuGaR, and SADL rules. 2 IDMEF [10] is a standard for CND sensor output that could eliminate the need for bridges but is not yet widely adopted. 4

5 RealSecure NT Server Sensor Stellar Analyst GUI 1 RealSecure Network IDS Snort Database Stellar Scenario and Prioritization Engine BIDS Checkpoint Stellar Analyst GUI 2 Figure 1. Stellar Architecture 5 Security Assessment Declarative Language Once scenarios are constructed a priority is assigned. It is very useful to prioritize based on context formed from the collection of alerts in the scenario, the organization s network, mission, and behavior. The decoupling of scenario formation and prioritization to incorporate such rich context is extremely powerful and unique to Stellar. For example, an analyst may want to be able to designate that a scenario with port scan alerts from an external host targeting critical web servers is high priority. However, if the port scan is coming from the system administrator s machine between 2:00 AM and 3:00 AM, the scan does not represent a threat and the scenario is assigned a low priority. SADL allows for this kind of flexibility in scenario prioritization. Other approaches to the prioritization problem were considered. Machine learning and probabilistic approaches were rejected as building a detailed representation of the network, mission, and configuration is extremely difficult using these types of models. Expert system approaches were rejected as the languages used to specify rules tend to be complex and evaluation speed is often inadequate for this application. Using attack trees to dynamically assess security risk was also rejected due to the fact that attack trees do not account for every possible attack vector, suffer from a combinatorial explosion, and do not solve the problem of known network noise such as broken routers or weekly port scans. It is also difficult to assess ones position in an attack tree when the underlying sensor data is predominately false alarms. Having considered these as well as alternative techniques and approaches discussed in Section 2, we decided to develop an easy to use declarative language. SADL was designed to be very similar to SQL, a database query language supported by many relational databases [9]. We selected a language akin to SQL because of its power, and familiarity to target users. Conceptually, each scenario is treated as a database table. Each alert in the scenario is treated as a row in the table. Properties of the alert, such as source IP address, alert name, and time, are considered columns in the table. A SADL rule can then be viewed as a SQL statement that evaluates to either true or false. SADL rules are evaluated in order. The first rule that matches assigns the priority. For example, consider the User Error Scenario described in Section 3. We wish to differentiate Windows_Access_Error alerts caused by users forgetting or mistyping their password from brute force password guessing attacks. The goal is to assign low priority to those scenarios which contain only alerts generated by users making common errors, including Windows_Access_Error alerts, while allowing the system to generate high priority alerts for password guessing attacks and attacks that are followed by more malicious attacks. To accomplish this the group USER_ERROR_ALERTS is defined that consists of all alerts about common user errors. The two rules found in Figure 2 will then assign priority one (low priority) to scenarios containing only these alerts as long as the number of Windows_Access_Error alerts is less than 10. If more than 10 such alerts appear in a scenario the first rule will no longer match and the second rule will assign the scenario a priority of eight. A SADL rule base is typically constructed by focusing on 5

6 1: contains_only(type in USER_ERROR_ALERTS) and count(type= Windows_Access_Error) < 10; 8: count(type= Windows_Access_Error) > 9; Figure 2. Windows Access Error Example lowering the priority of the most common alerts and raising the priority of the most serious attacks. The false alarm reduction rules should be written to take advantage of all the available context and should be as specific as possible. This reduces the amount of data an administrator must consider without accidentally lowering the priority of attack. The rule above lowers the priority of scenarios containing only common user error alerts and less than ten Windows_Access_Error alerts and is a good example of a rule that will deemphasize alerts while maintaining visibility of potential attacks. A better rule would add context about the network, perhaps allowing guessing only to those systems that are known to allow remote logins. If certain assets are particularly important to an organization, rules can be written to raise the priority of attacks against these assets; particularly if the assets are vulnerable to a known class of attacks (e.g. web servers will be more vulnerable to HTTP attacks). Rules can also be written to ensure that security policy is enforced. For example, in the evaluation described in Section 6, the Windows domain controllers were of particular concern. Thus rules were written to raise the priority of any type of behavior involving these hosts that deviated from the security policy. These rules written, alerts indicating that a domain controller performed a DNS lookup against a server other than those on the list of allowed DNS servers would result in a high priority scenario. 5.1 Security Assessment Declarative Language Syntax Valid SADL rules may take one of two forms: a group statement or a priority assignment statement. A priority assignment statement consists of a number between 0 and 9 followed by a colon and a scenario level expression. A group statement consists of the keyword group, a unique group identifier, and a comma separated list in parentheses. All SADL rules are terminated with a semicolon. SADL is not case sensitive. All of the rules found in Figure 3 are valid SADL rules. The first rule the is a group statement. The other two rules are priority assignment statements. Each priority assignment statement contains a scenario level expression. Scenario level expressions are applied to scenarios and evaluate to either true or false. If the scenario level expression evaluates to true then the scenario is assigned the priority associated with the scenario level expression. Group statements simply facilitate the composition of priority assignment statements and do not assign priority to scenarios. Alert level expressions apply to properties of a single alert and are always enclosed by scenario level functions or predicates. There are only four data types in SADL: number, subnet, string, and boolean. A number is a numerical value such as 1243 or A subnet is a valid IPv4 subnet. Subnets are specified using CIDR block notation with an IPv4 address followed by a slash ( / ) and a number between 0 and 32. The number after the IP address specifies the number of 1 bits in the subnet mask. For example /16 specifies a 16 bit subnet mask which is equivalent to a mask of If no subnet mask is provided the address refers to a single host. String constants are always enclosed in quotes. Examples of valid strings include Port_Scan, RealSecure, and Monday. A boolean is a truth value: either true or false. Boolean values are usually produced by evaluating SADL predicates. Alert level expressions are formed by combining alert level properties, predicates and operators into an expression that evaluates to either true or false for a given alert. Valid alert properties are listed in Table 2. Valid operators are listed in Table 1. There are two alert level predicates: time, and *. The time predicate is used to specify a time window. This is probably best explained by example: time(02:00:00,2m) evaluates to true on any alert that occurred between 02:00 and 02:02 hours. Times are always specified in military time helping avoiding any confusion between A.M. and P.M. hours. The m after the numeral 2 in the example specifies that the time scale for the window is in minutes. Valid time scales are s for seconds, m for minutes, and h for hours. Thus time(13:00:00,2h) evaluates to true on any alert that occurred between 13:00 and 15:00 hours. There is also one special alert level predicate, the * predicate. * always returns true just like the SQL * predicate. It is useful in conjunction with some of the scenario level functions. For example, count(*) returns the number of alerts in the scenario. Scenario level functions are similar to SQL domain aggregate functions. They operate on the alerts that match the alert level expressions they enclose. For example, the 6

7 group important_ips( , /24); 8: CONTAINS(from in important_ips); 2: contains_only(type= Port_Scan ); Figure 3. Example SADL rules Operator Type Valid Data Types Example = binary string, subnet, number from= > binary number count(from= )>5 < binary number rate(src_ip= )<5 >= binary number count(dest_ip= )>=5 <= binary number count(to= )<=5 not unary boolean not type= Port_Scan in unary boolean Port_Scan in scan_group Table 1. Valid Operators Property Data Type Description Example Values from, src_ip subnet source IP address to, dest_ip subnet destination IP address type string type of alert Port_Scan protocol string protocol associated with the alert UDP, ICMP src_port number source port of the alert 2765, 80 dest_port number destination port of the alert 80, 21 day string day of the week Monday, Tuesday sensor subnet IP address of the sensor Table 2. Valid Alert Level Properties Function Return Type Description contains boolean True if the scenario contains at least 1 alert matching the alert level expression contains_only boolean True if the scenario contains only alerts matching the alert level expression contains_pct number Returns the percent of alerts in the scenario that match the alert level expression count number Returns the number of alerts that match the alert level expression count_distinct number Returns the number of distinct alerts that match the alert level expression rate number Returns the number of alerts that match the alert level expression divided by the number of seconds between the first and last such alert. Table 3. Valid Scenario Level Functions 7

8 count function takes an alert level expression as an argument and returns the number of alerts in the scenario that match the given expression. Thus count(src_ip = and type= Port_Scan ) would return the number of alerts in the scenario whose source address is and whose alert type is Port_Scan. Valid Scenario level functions are listed in Table Predicate Complexity A naive implementation of scenario level functions would require a computation time that is linear in the number of alerts in the scenario. For example, the count function must examine each alert in the scenario to see if it matches the alert level expression and count the number of times an alert does in fact match. Since scenarios are evaluated against all SADL rules each time they change, a constant time solution is desired. All of the functions listed in Table 3 maintain state, enabling a scenario s priority to be evaluated in constant time, using information stored during the last evaluation of the scenario. By maintaining state, the value of the function can be determined by examining only the alerts that were added to the scenario since it was last evaluated. Since a scenario s prioritization is re-evaluated each time alerts are added, functions can generally determine their value by examining only a single alert. 6 System Evaluation In order to evaluate the performance of Stellar we deployed the system on a large, operational enterprise network. We ran Stellar in parallel with an existing system that had separate user interfaces for each sensor and required human analysts to manually correlate and prioritize alerts into incident reports. The site s analysts helped us install, run, and provided feedback on the Stellar s performance and usability. 6.1 Site Deployment The trial network is composed of two class B networks and contains over fifty different web, DNS, mail and domain controller servers in addition to thousands of desktop hosts. Daily aggregate ingress and egress traffic rates averaged approximately 13 Mbps. Defending the network were three Checkpoint firewalls and four Snort sensors located across multiple network segments. Other CND devices were deployed but were not included in this evaluation primarily due to unsupported sensor types and time synchronization problems. The organization overseeing the network is extremely security conscious and has a team of knowledgeable and dedicated security analysts monitoring alerts twenty four hours a day, seven days a week. 6.2 Deployment Architecture and Configuration In an effort to simplify the installation process and decrease the operational impact of the evaluation, the authors pre-installed the Stellar components on laptops. The machine on which the Stellar Engine ran was a Pentium III 1.3 GHz machine with 1 GB of RAM running Windows 2000 Professional. The evaluation was intended to focus on SADL performance. As prior work had indicated that forming scenarios based on source IP address alone could be effective [5], the scenario building engine was configured to combine alerts into a scenario if and only if the source IP addresses of the alerts were identical. The evaluation took place over three days. On the first day the system was configured and deployed. Some of the analysts familiar with the installation site wrote SADL rules with the help of the authors. The rules written on the first day were deployed on the second day when Stellar was run on live operational data. During this period, the team of approximately 3 local analysts performed their duties as usual, ignoring the output from Stellar. Analysts at this site determine if detected activity might compromise network security and write incident reports. On the third day, the incident reports written during the second day were compared to the scenarios given high priority by the Stellar system. If the system was effective, all security incidents identified by the analysts would have been given high priority by the Stellar system. The third day was also used to gather feedback from the analysts about the usability and usefulness of the system. The majority of the deployment and configuration effort was composing the site specific SADL rules. SADL rule composition efforts required collaboration of several analysts in order to provide information about critical network assets, services, sensors, and policies. The process of writing such a rule set was more time intensive than originally anticipated. This experience leads us to believe that systems which require additional information will be difficult to maintain unless data input can be automated. While writing the prioritization rules took only a couple of hours (including the time required to explain the SADL language and syntax to the analysts), filling in the group statements which identify network assets was time consuming. Knowledge of all the network s services did not reside in one place or with one analyst, although nearly all said they had a mental map of the network. Compiling a comprehensive list of servers required several phone calls to other, remotely located analysts and system administrators. Additionally, initial Stellar runs revealed legitimate servers that the analysts were not aware of. These servers came to light when SADL produced high priority scenarios indicating a modest number of successful connections to unknown servers. This occurred 8

9 several times before all legitimate servers were identified. While composing the SADL rules it became apparent that many of rules could be written once and applied generally at many different sites; requiring changes only to the group statements. The rule in Figure 4 is designed to identify port 80 connections to hosts which are not known to be web servers where the connection was allowed by the firewall. To use this rule at a different site, only the elements of the two group statements (for http_servers and system_networks) would need to be modified. Many similarly valuable rules can be written. Despite the potential general applicability of a few base rules, each site will pursue its unique mission and enforce different policy necessitating the composition of custom rules to achieve the best performance. Work to reduce the composition and maintenance burden is addressed in Section 6.4. Despite the time required to compose and test the SADL rule base, the language proved to be powerful and flexible, allowing the analysts to write complex rules with few limitations. The final rule base used to perform the evaluation consisted of 18 group and 26 priority assignment statements. The group statements primarily characterized the network defining lists of hosts, services, and classes of alert types. The priority statements made use of the group statements in defining complex policies and known network behavior. Approximately one half of the priority statements addressed scenarios that were considered high priority. The remainder of the priority statements lowered priority for scenarios that were indicative of false alarms or known network behavior (i.e. misconfigurations, faulty hardware, etc.). 6.3 Quantitative System Performance and Results Once an acceptable set of SADL rules were composed and the system installed, a twenty four hour uninterrupted run of Stellar was initiated on the live network. Within the twenty four hour period, the seven sensors produced a total of 845,236 alerts. From those 845,236 alerts, Stellar formed 12,137 scenarios. Of the 12,137 scenarios formed, only sixty two were assigned a priority of seven or above Stellar Speed During the run the Stellar system did not have any problems keeping up with the alert volume. Sensors produced an average of ten alerts per second during the twenty four hour period. Experiments after the evaluation showed that the Engine was able to start, retrieve all 845,236 alerts from the database, form the 12,137 scenarios, assign priorities, and update the database in less than twelve minutes, processing and average of 1,380 alerts per second. Stellar is capable of processing alerts streams over one hundred times the rate at which the 7 sensors in the evaluation produced them Stellar Memory Use and Scenario Time Out Because the Stellar system is designed to run in real time, all the scenarios and the alerts they contain must be held in memory. When the machine on which the Engine is running runs out of physical memory, the Engine must time out scenarios. Scenarios which were the last to have alerts added to them are removed from memory first. When a scenario is removed from memory it still exists in the database and may be viewed by the analyst, but the Engine will no longer prioritize or add new alerts to timed out scenarios. Since attackers may employ long latencies in order to avoid detection, the Stellar system tries to avoid timing out scenarios by minimizing the memory requirements of the stored scenario information. Experimentation revealed that an alert requires an average of 350 bytes of memory. Configurable runtime parameters allow users to set engine thresholds for scenario time outs. During the twenty four hour run we configured the Engine to maintain no more than 400,000 alerts in memory. When this threshold was exceeded, alerts were timed out in fifty thousand alert blocks. Each time the Engine reached the 400,000 alert threshold, fifty thousand alerts were removed from memory. During the run Stellar never consumed more than 150 MB of memory. We could have maintained more alerts in memory before timing out, but we wanted to test the time out mechanism and so chose this lower threshold to ensure that time outs would be triggered several times during our test. At the end of the twenty four hour period, 3,723 scenarios remained active in memory and 8,429 scenarios had been time timed out Prioritization Accuracy During the same twenty four hour period that the Stellar system ran, site analysts continued their task of network surveillance and incident reporting. Incident reports submitted by site analysts gave us a base line from which we could assess detection performance. Site analysts submitted twelve incident reports during the twenty four hour evaluation period. All twelve incident reports concerned reconnaissance or probing activity and were among the sixty two Stellar assigned a high priority (a priority seven or greater). All alerts referenced in the incident reports appeared in the corresponding high priority scenario. The number of alerts per scenario ranged from 16 to 7456 with an average of Having confirmed successful scenario construction and prioritization for all twelve incidents reported by analysts, we further examined scenarios for which no incident had been reported but had been prioritized high. This analysis identified an additional eight scenarios that site analysts confirmed would have been the subject of an incident report had they been identified. Several of these incidents 9

10 8: contains( dest_port = 80 AND not dest_ip in http_servers AND type = accept AND dest_ip in system_networks ); Figure 4. Unaccounted for web server connection SADL rule indicated a probe followed by HTTP and SMTP exploit attempts against site servers. Fortunately none of the targeted machines were vulnerable to the attacks. The formation and high prioritization of the multi-step attack scenarios is significant in that alerts combined across multiple sensors to reveal a broader vision of attack activity. The eight unreported scenarios usually incorporated alerts from both the firewalls and the Snort sensors and did not match any existing SADL rule. Unexpected activity is assigned the default priority which is set high to catch novel attack behavior. An example of a SADL rule that assigned a high priority to scenarios confirmed to have been the subject of analysts incident reports is presented in Figure 5. This rule is a conjunction of two scenario level predicates that assigned a priority seven to any scenario that contains more than 10 firewall drop alerts to distinct hosts with fewer than five distinct destination ports. This rule is designed to identify behavior that is indicative of a horizontal network scan. Since the rule does not specify a time window is can detect slow, stealthy scans which are overlooked by Snort s port scan preprocessor. Of the sixty two high priority scenarios, a total of twenty were confirmed to warrant further investigation. Not every scenario that received a high priority necessitated an incident report. The majority of the high priority scenarios were intentionally prioritized as such by the analysts in order to keep potentially suspicious activity in view where it could immediately be acted on or dismissed. Scenarios of this sort typically received a priority of 7 where scenarios of real concerned received a 8 or 9. The overwhelming majoring of the remaining scenarios prioritized high but not reported upon constituted initial indications of reconnaissance activity. There were three scenarios for which we could not assign the desired prioritization due to limitations in SADL. This limitation is due to the fact that SADL cannot account for alert ordering in prioritization rules. Work since the evaluation has addressed this limitation and is explained in Section Qualitative Results Analysts noted that it would be easier to investigate incidents if the GUI could indicate which alert in a scenario caused the priority to change. The ability to cross reference additional data, such as port to service mappings, packet dump data, and vendor alert information was also requested. The most difficult part of writing the SADL rules was compiling a list of servers and services known to be running on the protected network. Although once the system is configured to model the network, we anticipate that it will change only slowly. Sometimes analysts will want to know about these changes, and sometimes analysts will want the system to automatically learn about these changes and adapt. To address this we are working to incorporate LANScape, a passive network monitoring tool. LANScape tracks of all hosts on the network and the services they are running. If a new host or service appears on the network the administrator is notified. Newly discovered servers and services can be marked as allowed or unknown by the administrator. Thus, for example, newly discovered HTTP servers that are marked as allowed by the administrator can be automatically added to http_servers group. Active scanning could also be used to help populate and maintain such groups. Many signature based IDSs suffer from high false positive rates partly due the fact that they are not aware of all of the services running on all the variety of OSs that the IDSs are tasked to protect. The addition of LANScape will also enable SADL to assign priority based on automatically integrated additional context such as knowledge about whether a host is listening on a targeted port or running a particular version of a vulnerable operating system. Figure 6 contains an example of what a SADL rule could look like with this extension. This rule assigns high priority to scenarios that contain web attacks destined for a host which is not on the list of allowed web servers but is none the less listening on port 80. A limitation to the SADL implementation briefly mentioned in Section does not allow one to prioritize based on the the order in which alerts occur. In a few instances, analysts wanted the ability to prioritize scenarios containing the same types of alerts differently based on the order in which those events occurred. One ordering was benign where another ordering was indicative of malicious intent. In order to overcome this temporal ordering limitation we are currently developing the ability to assign attributes to events that match arbitrary length sequences. Once events match the sequence and are assigned the sequence attribute, we will be able to prioritize based on that attribute just like any other such as type or protocol. 10

11 7: count_distinct(dest_ip, type="drop") > 10 and count_distinct(dest_port, type="drop") < 5; 9: contains(type in web_attacks and not (to in web_servers) and is_listening( dest_ip, 80 )); Figure 5. Scan and Attack SADL rule Figure 6. Potential SADL rule with machine service information 7 Conclusion The Stellar system demonstrates the benefits of exploiting context in prioritization. To support this Stellar first combines related alerts into scenarios and then prioritizes the entire scenario using a declarative prioritization language. A test deployment in an operational enterprise network demonstrated Stellar s ability to reduce analyst workload without compromising security. Stellar had no difficulty processing nearly one million alerts per day. The correlation and prioritization technique discovered all the incidents identified and reported by human analysts as well as many others that would have necessitated an incident report submission had analysts been aware of scenario s existence. The massive amount of data reduction provided by Stellar allows analysts to more quickly focus on incidents of interest as well as understand the network they protect and the traffic traversing it. References [1] M. M. Astrahan and D. D. Chamberlin. Implementation of a structured english query language. Communications of the ACM, 10: , [2] Tim Belcher and Elad Yoran. Riptech internet security threat report VII. Technical report, Riptech, July [3] S. Benferhat, F. Autrel, and F. Cuppens. Enhanced correlation in an intrusion detection process. In Proceedings of the Second International Workshop Mathematical Methods, Models and Architectures for Computer Networks Security, St. Petersburg, Russia, September [4] Robert K. Cunningham, Richard P. Lippmann, David Kassay, Seth E. Webster, and Marc A. Zissman. Hostbased bottleneck verification efficiently detects novel computer attacks. In IEEE Military Communications Conference Proceedings, Atlantic City, NJ, [5] Oliver Dain and Robert K. Cunningham. Fusing A Heterogeneous Alert Stream Into Scenarios. Kluwer Academic Publishers, [6] Oliver Dain, Robert K. Cunningham, and Stephen Boyer. IREP++, a faster rule learning algorithm. In Proceedings of the Fourth SIAM International Conference on Data Mining, Orlando, FL, April [7] Hervé Debar and Andreas Wespi. Aggregation and correlation of intrusion-detection alerts. In W. Lee, L. Mé, and A. Wespi, editors, Recent Advances in Intrusion Detection, page 85, Springer, October [8] Robert P. Goldman, W.L. Heimerdinger, Steven A. Harp, Christopher Geib, Vicraj Thomas, and Robert L. Carter. Information modeling for intrusion report aggregation. In Proceedings of the DARPA Information Survivability Conference and Exposition, pages IEEE Computer Society, June [9] James R. Groff and Paul N. Weinberg. SQL The Complete Reference. McGraw Hill, [10] Intrusion Detection Working Group. Intrusion detection message exchange format. IETF-IDWG Draft, January [11] IEEE Computer Society Technical Committee on Security and Privacy. Alert Correlation in a Cooperative Intrusion Detection Framework, Los Alamitos, CA, May [12] Internet Security Systems. RealSecure console user guide. Atlanta, GA, [13] K. Julisch and M. Dacier. Mining intrusion detection alarms for actionable knowledge, [14] Check Point Firewall Technologies LTD. FireWall-1 User Guide, 3.0 edition, html. 11

12 [15] Benjamin Morin and Hervé Debar. Correlation of intrusion symptoms: an application of chronicles. In Proceedings of the 6th symposium on Recent Advances in Intrusion Detection (RAID 2003), Pittsburg, PA, September Springer LNCS. [16] Benjamin Morin, Ludovic Mé, Hervé Debar, and Mireille Ducassé. M2D2 : a formal data model for IDS alert correlation. In Proceedings of the 5th symposium on Recent Advances in Intrusion Detection (RAID 2002), pages , Zurich, Switzerland, October Springer LNCS. [17] Peng Ning and Yun Cui. An intrusion alert correlator based on prerequisites of intrusions. Technical Report TR , [18] Dirk Ourston, Sara Matzner, William Stump, and Bryan Hopkins. Applications of hidden markov models to detecting multi-stage network attacks. In Proceedings of the 36th Annual Hawaii International Conference on System Sciences, Big Island, Hawaii, January IEEE Computer Society. [19] Dirk Ourston, Sara Matzner, William Stump, and Bryan Hopkins. Coordinated internet attacks: Responding to attack complexity. Journal of Computer Security, 12(2): , [20] Phillip A. Porras, Martin W. Fong,, and Alfonso Valdes. A mission-impact-based approach to IN- FOSEC alarm correlation. In Lecture Notes in Computer Science, Proceedings Recent Advances in Intrusion Detection, pages , Zurich, Switzerland, October [21] Marty Roesch. Snort: Lightweight intrusion detection for networks. In Usenix 13th Systems Administration Conf., Usenix Assoc. [22] Alfonso Valdes and Keith Skinner. Probabilistic alert correlation. In Recent Advances in Intrusion Detection (RAID 2001), number 2212 in Lecture Notes in Computer Science. Springer-Verlag, [23] G. Vigna, F. Valeur, and R.A. Kemmerer. Designing and Implementing a Family of Intrusion Detection Systems. In Proceedings of the European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE 2003), Helsinki, Finland, September