Cyber Security: Evaluating the Effects of Attack Strategy and Base Rate through Instance-Based Learning

Transcription

1 Cyber Security: Evaluating the Effects of Attack Strategy and Base Rate through Instance-Based Learning Aman Arora School of Computing and Electrical Engineering Indian Institute of Technology, Mandi, India Varun Dutt School of Computing and Electrical Engineering School of Humanities and Social Sciences Indian Institute of Technology, Mandi, India Abstract Cyber attacks, the disruption of normal operations in a computer network due to malicious events called cyber threats, are becoming widespread. In order to check the prevalence of cyber attacks the role of security analysts, human decision makers whose job is to prevent cyber attacks, is becoming extremely important. However, currently very little is known on how security analysts might respond to different attack strategies of an attacker in cyber attacks. Also, little is known on how the proportion of threats (i.e., base rate) in an attack scenario influences the analyst s timely and accurate detection of such attacks. In this paper, we use an existing cognitive model of the security analyst, based upon Instance-Based Learning Theory, and we evaluate the effects of attack strategy and base rate on the model s accurate and timely detection of cyber-attacks in a simulated scenario. The attack strategy was manipulated as: impatient (attacker injects all threats in the beginning of the scenario) and patient (attacker waits till the end of the scenario to inject threats); and, base rate was manipulated as: common (13 out of 25 scenario events (52%) were threats) and rare (3 out of 25 scenario events (12%) were threats). Results reveal that the attack strategy influences only the analyst s accuracy and not her timeliness; however, the base rate influences her timeliness and not the accuracy. We discuss the implications of our results for training analyst in their job. Keywords: cyber attacks; base rate; attack strategy; Instance-based Learning Theory; accuracy; timeliness. Introduction According to the U.S. White House, Cyberspace touches nearly every part of our daily lives. It is the broadband networks beneath us and the wireless signals around us, the local networks in schools, hospitals, work-places and business...today s world is more interconnected than ever before. These lines explicitly describe the inevitableness of Internet. Yet, for all its advantages, increased connectivity brings increased risk of theft, fraud, and abuse. As we are becoming more reliant on modern technology, we are also becoming more vulnerable to cyber attacks such as Corporate Security Breaches, Spear Phishing, and Social Media Fraud. With the prevalence of Anonymous and LulzSec hacking groups and other threats to corporate and national security, guarding against cyber attacks is becoming a significant part of IT governance, especially because most government agencies and private companies have moved to online systems (Sideman, 211). Thus, in a cyber-world like ours, we must have defenses to keep us safe. In order to create appropriate defenses against cyber attacks, the role security analysts, a human decision maker whose main role is to protect computer networks from cyber attacks, is becoming indispensable (Jajodia, Liu, Swarup, & Wang, 21). Given the growing demand, companies have started employing security analyst. However, currently not much emphasis has been laid on how different environmental factors (e.g., attack strategy and base rate) would affect the analyst s accurate and timely detection of cyber attacks. One of the researches done in this field indicates that the analyst s accurate and timely threat detection is likely to be a function of prior experiences with cyber attacks and tolerance to perceived threats (Dutt et al., in press; McCumber, 24; Salter, Saydjari, Schneier, & Wallner, 1998). Dutt et al. (in press) have given primary predictions about a simulated analyst s performance according to her experience and tolerance to threats (or risk-taking). These authors created a cognitive model of an analyst s decisions, based upon Instance-Based Learning Theory (IBLT; Gonzalez & Dutt, 211), and populated the model s memory with cyber threat and cyber non-threat experiences. The model s tolerance was determined by the number of events it perceived as threats before it declared the sequence of network events to be a cyber attack. Accordingly, a model with a greater proportion of threat experiences was more accurate and timely in detecting threats compared with one with a smaller proportion of such experiences; whereas, the tolerance did not influence model s accuracy and timeliness. Although Dutt et al. (in press) have highlighted the role of prior experience and risk-taking on cyber threat detection, it is unclear how their results would vary due to the nature of adversarial behaviors (an external factor). Thus, here we simulate an analyst s decision process through the computational model developed by Dutt et al. (in press) and derive predictions about analyst s decisions in scenarios that differ in adversarial behaviors. One characteristic of adversary s behavior is the attack strategy (patient or impatient) being followed to infiltrate the network. An impatient strategy is the strategy in which attacker passes all threat events in the beginning of a sequence of network events; whereas, a patient strategy will make attacker to 336

2 wait till the end to inject the threats in a sequence of network events. We also consider proportion of threat events (i.e., base rate; common or rare) that an attacker uses to attack a network. A common base rate is when the attacker uses a large number of threat events in a cyber attack; however, a rare base rate is when the analyst uses only a smaller number of threat events in a cyber attack. Next, a cyber-infrastructure and an IBL model of analyst s cyber threat detection are detailed. Then, we describe the manipulations in the attack scenarios and detail the results of our manipulations. We close this paper by presenting a discussion of our findings and their implications to training human analysts in their job. Cyber Infrastructure A corporate network may consist of different types of servers and multiple layers of firewalls. We used a simplified network configuration consisting of a web server, a fileserver, and two firewalls (Ou, Boyer, & McQueen, 26; Xie, Li, Ou, & Levy, 21). An external firewall ( firewall 1 in Figure 1) controls the flow of traffic between the Internet and the Demilitarized zone (DMZ; a subnetwork that separates the Internet from the company s internal LAN network). Second firewall ( firewall 2 in Figure 1) controls the traffic between the webserver and the fileserver, a company s internal LAN network. The webserver interacts with outside world and it is behind the DMZ. FileServer contains necessary information that internal users (employees) use to do their daily operations. FileServer is connected to workstations, which employees use as a medium to interact with the fileserver and are authorized to run executable files resting on the fileserver. Generally, an attacker is identified as a computer on the Internet that is trying to gain access to the internal corporate servers and workstations. For this cyber-infrastructure, attackers follow a pattern of island-hopping attack (Jajodia et al., 21; pp. 3), where the webserver is compromised first, and then it is used to originate attacks on the fileserver and other company workstations. An attacker, in order to gain access to fileserver, will need to pass at least 3 events (to get to webserver, fileserver, and then to execute binaries inside fileserver). Because of this reason, we define the rare base rate to contain 3 threats out of total 25 network events (as it is minimum an attacker may require). The model discussed in this paper is presented with different cyber events in a sequence, where these events follow an attacker s strategy, i.e., the sequence of events are a combination of a base rate and timing strategy. Attack sequences may be classified as rare-impatient, rare-patient, common-impatient and common-patient depending upon the combination of attack strategies being followed by attacker. Rare-impatient attack sequence contains 3 out of total 25 network events as threats and are all being injected in the beginning. Rare-patient also contains 3 threats but here these are being injected in the end of sequence of 25 network events. Figure 1: An Example Attack Scenario. In common-impatient attack sequence there are 13 threats out of total 25 network events and these are being injected with impatient strategy, i.e., all of them are being injected in the beginning of the sequence; however, in common-patient attack sequence these 13 threats are being injected at the end of sequence. Nature of these attack sequences is not known to the model; however, the model is able to get alerts corresponding to some network events (that are regarded as threats) generated from the intrusiondetection system (IDS) (Jajodia et al., 21). Out of 25 events, some are threats that are initiated by an attacker (the rest of the events are initiated by inside users). The model does not know which events are generated by the attacker and which are generated by corporate employees. By perceiving network events in a sequence as threats or nonthreats, the model needs to identify, as early and accurately as possible, whether the sequence constitutes a cyber attack. Instance-Based Learning Theory (IBLT) IBLT is a theory of how people make decisions from experience in dynamic environments (Gonzalez & Dutt, 211). Computational models based on IBLT have been shown to generate accurate predictions of human behavior in many dynamic decision-making situations similar to those faced by analysts (Dutt et al., in press; Dutt & Gonzalez, 212; Gonzalez & Dutt, 211; Gonzalez et al., 211). IBLT proposes that every decision situation is represented as an experience called an instance that is stored in memory. Each instance in memory is composed of two parts: situation (S) (the knowledge of attributes that describe an event), a Decision (D) (the action taken in such situation), and utility (U) (a measure of expected result of a decision that is to be made for an event). For a situation 337

3 involving securing a network from threats, the situation attributes are those that can discriminate between threat and non-threat events: the IP address of a computer (web-server, file-server, or workstation, etc.) where the event occurred, the directory location in which the event occurred, whether the IDS raised an alert corresponding to the event, and whether the operation carried out as part of the event(e.g., a file execution) by a user of the network (which could be an attacker) succeeded or failed. In the IBL model of a analyst, an instance s S part refers to the situation attributes defined above; and the U slot refers to the expectation in memory that a network event is a threat or not. For example, an instance could be defined as [webserver, c:\, malicious code, success; threat], where webserver, c:\, malicious code, and success constitutes the instance s S part; and threat is the instance s U part (the decision being binary: threat or not, is not included in this model). An instance is retrieved in the recognition phase from memory according to an activation mechanism (Gonzalez et al., 23; Lejarraga et al., in press). The activation of an instance i in memory is defined using a simplified version of ACT-R s activation equation: where i refers to the i th instance that is pre-populated in memory, and i = 1,2, constitutes the total number of prepopulated instances in memory; B i is the base-level learning mechanism and reflects both the recency and frequency of use for the i th instance since the time it was created; and is the noise value that is computed and added to an instance i s activation at the time of its retrieval attempt from memory. The B i equation is given by: ( { } In this equation, the frequency effect is provided by, the number of retrievals of the i th instance from memory in the past. The recency effect is provided by, the time since the th past retrieval of the i th instance (in equation 2, t denotes the current event number in the scenario). The d is the decay parameter and has a default value of.5 in the ACT-R architecture, and this is the value we assume for the IBL model. refers to the similarity between the attributes of the situation and the attributes of the i th instance. is defined as, Table 1: The coded values in the S part of instances in memory and attributes of a situation event. Attributes Values Codes IP Webserver 1 ) Fileserver 2 Workstation 3 Directory Missing value -1 1 File X 1 Alert Present 1 Absent Operation Successful 1 Unsuccessful is the similarity component and represents the mismatch between a situation's attributes and the situation (S) part of an instance i in memory. The k is the total number of attributes for a situation event that are used to retrieve the instance i from memory. The value of k=4 as there are 4 attributes (IP, directory, alert, and operation) that characterize a situation in the network. The match scale ( ) reflects the amount of weighting given to the similarity between an instance i s situation part l and the corresponding situation event s attribute. is generally a negative integer with a common value of -1. for all situation slots k of an instance i, and we assume this value for the. The or match similarities represents the similarity between the value l of a situation event s attribute and the value in the corresponding situation part of an instance i in memory. Typically, is defined using a squared distance between the situation event s attributes and the corresponding instance's situation slots (Shepard, 1962). Thus, is equal to the sum of squared differences between a situation event s attributes and the corresponding instance's S part. In order to find the sum of these squared differences, the situation events attributes and the values in the corresponding S part of instances in memory were coded using numeric codes. Table 1 shows the codes assigned to the S part of instances and the situation events attributes. The noise value (Anderson & Lebiere, 1998; Gonzalez & Dutt, 211) is defined as ( ) where, is a random draw from a uniform distribution bounded in [, 1] for an instance i in memory. We set the parameter s in an IBL model to make it a part of the activation equation (equation 1). The s parameter has a default value of.25 in the ACT-R architecture, and we assume this default value in the IBL model. We used IBLT to study the accurate and timely detection of threats by cyber analysts because IBLT is known to make better decisions compared to other models and techniques. Basava, Ramakrishna and Varun in there paper Cyber Situation Awareness: Rational Methods versus Instance- Based Learning Theory for Cyber Threat Detection; ICCM 338

4 in this conference compare IBLT with Naïve Bayes classifier which is a rational approach to make decisions. There results depict that Naïve Bayes approach is poor in terms of timeliness and accuracy as compared to the IBLT. Hence, we ran our experiments with IBLT to study the interaction of base rate and attack strategy in timely and accurate detection of cyber attacks. Experiments The IBL model used here has been taken from Dutt et al. (in press). This model is presented with sequences of network events that represent four strategies (commonpatient, common-impatient, rare-patient, rare-impatient) of the attacker with tolerance fixed at 5% of base rate. All sequences contained 25 network events. Model s memory was pre-populated with instances that represent analysts with different experiences and the model was fixed to use tolerance level of.5. The IBL model retrieved instance with highest activation and made a decision about an event being a threat or a non-threat. We use only 5 simulations of the model as they were sufficient for generating stable model results (Dutt et al., in press). We ran 5 simulations (each simulation consisting of 25 network events) and the model s effectiveness was evaluated using its accuracy and detection timing in four groups defined by: strategy (patient and impatient) and base rate (common and rare). Accuracy was determined by computing the d (Z(hitrate) Z(false-alarmrate)), hitrate (hits/(hits + misses)), and false-alarm rate = (falsealarms/(false-alarms + correct-rejections)) (Wickens, 21) over the course of 25 network events and averaged across the 5 simulations. The decision of the model for each network event was marked as a hit if an instance with its U slot indicated a threat for an actual threat event in the sequence. Similarly, the model s decision was marked as a false-alarm if an instance with its U slot indicated a threat for an actual non-threat event in the sequence. Hits and false-alarms were calculated for all events before model declared a cyber attack and stopped, or when all the 25 events had occurred (whichever came first). Furthermore, detection timing was calculated in each simulation as the proportion of attack steps, defined as 1% - the percentage of threat events out of a total 25 that have occurred after which the model classifies the event sequence as a cyber attack and stops. Therefore, higher percentages of attacks steps would indicate the model to be timelier in detecting cyber attacks. Accuracy As shown in Figure 2, it was seen that d corresponding to common base rate (-.26) was greater than that for the rare base rate (-.75). So, the base rate did influence the model s accuracy (d ). Furthermore, as shown in Figure 3, the accuracy for the impatient strategy (d = 3.48) was greater than that for the patient strategy (d = -4.25). Thus, attack strategy also influenced the accuracy. However, the base rate did not play as big a role as that played by strategy (see Figure 4). Strategy was found to influence accuracy and its effect was irrespective of the base rate. From Figure 4, an impatient strategy had a greater d (common-impatient = 4.23, rare-impatient = 2.72) compared to a patient strategy (common-patient = -4.28, rare-patient = -4.23) Figure 2: Accuracy(d ) as affected by base rate d' d' Figure 3: Accuracy(d ) as affected by strategy. Common Rare Impatient Patient Results We will be explaining the results obtained with the help of figures presented in this paper from Figure 2 to Figure 7. Each of these figures is a histogram where y axis corresponds to accuracy or timeliness as mentioned across respective figure and the bars represents attack scenario, those can be decoded by the legend in each figure. 339

5 Figure 4: Accuracy(d ) as affected by different combinations of base rate and strategy Timeliness After running the model it was observed that the timeliness (Figure 5) for the rare scenario (36%) was higher than that for the common scenario (47%). Thus, timeliness was influenced by base rate. However, strategy did not influence timeliness, as can be seen from Figure 6 both strategies have almost same timeliness (impatient strategy with timeliness of 42% and patient strategy with timeliness of 41%). This means that strategy plays no role in the time it takes to determine if there is an attack. Interaction of base rate and strategy in determining the timeliness was evaluated in Figure 7. Timeliness for common scenarios (common-patient = 36%, common-impatient = 35%) were lower than that in the rare scenarios (rare-patient = 46%, rare-impatient = 49%). Also, timeliness did not vary much for both strategies in both scenarios. Thus, there is only an influence base rate on timeliness and not of strategy on timeliness: The rare base rate caused model to be timelier. Thus, to conclude, timeliness was affected by base rate d' 47 Timeliness (%) Common Patient Figure 5: Timeliness as affected by base rate Common Impatient Rare Patient Rare Impatient Common Rare Timeliness (%) Figure 6: Timeliness as affected by strategy Timeliness (%) 49 Impatient Patient Common Patient Common Impatient Rare Patient Rare Impatient Figure 7: Timeliness as affected by different combinations of base rate and strategy. Discussion and Conclusions In this paper, we studied the effects of base rate and attack strategy on the model s accurate and timely detection of cyber-attacks. Such an analysis is important as unlike the cognitive factors (recency and tolerance) that are under the direct control of analyst, the environmental factors (attack strategy and base rate) are controlled by the attacker and outside the direct control of the analyst. We find that both these environmental factors, being outside the control of the analyst, influence the analyst s accuracy or timeliness. First, the analyst s accuracy was influenced by the strategy. The model was more accurate when the strategy was impatient compared to when it was patient. The likely reason for this result is that an impatient strategy s early threats increase the activation of threat instances in the model s memory early on. Therefore, the increase in activation is likely to make the model perform more accurately against an impatient strategy compared to a patient strategy. Second, the timeliness was influenced by the base rate: The model was timelier for the rare base rate compared to the common base rate. This observation can be explained based upon the definition of base rate, i.e., the proportion of threats being passed among the 25 network events. If the 34

6 proportion of threats is less (i.e., in rare scenario), it will take less time for the model to declare a sequence of network events as an attack compared to when the proportion of threats is more (i.e., in the common scenario). The activation of threat instances is likely increase faster for the scenario with the rare base rate compared with the scenario with the common base rate (because the instances with the rare base rate are fewer in number and easy to identify). This increase in activation in the rare scenario would cause these activated instances to be retrieved from memory often, causing the model to stop early. Our results have important implications for training analysts in their job. First, as both the base rate and strategy influence the cyber threat detection, it is prudent to train analysts on scenarios that differ in both these environmental factors. Second, it is expected that analyst should be trained in the common scenario and for a patient attack strategy as in these cases the model s performance was the poorest. In these experiments, we overlooked the effects of tolerance (risk-taking) by setting it up at 5% of base rate. However, one expects that the model s tolerance would likely vary from one individual to another. More specifically, the model s ability would likely be influenced by its risk-taking and varying the tolerance may have significant effects on the model s accurate and timely detection of cyber-attacks. Thus, the next step in this research would be to introduce tolerance as another parameter and see its interaction with base rate and strategy. We plan to undertake this idea as part of our ongoing research on this topic. Jajodia, S., Liu, P., Swarup, V., & Wang, C. (21). Cyber situational awareness. New York, NY: Springer. McCumber, J. (24). Assessing and managing security risk in IT systems: A structured methodology. Boca Raton, FL: Auerbach Publications. Ou, X., Boyer, W. F., & McQueen, M. A. (26). A scalable approach to attack graph generation. In Proceedings of the 13th ACM Conference on Computer and Communications Security (pp ). Alexandria, VA: ACM. doi: / Salter, C., Saydjari, O., Schneier, B., & Wallner, J. (1998). Toward a secure system engineering methodology. In Proceedings of New Security Paradigms Workshop (pp. 2-1). Charlottesville, VA: ACM. doi: / Shepard, R.N. (1962). The analysis of proximities: multidimensional scaling with an unknown distance function. Part I. Psychometrika, 27, Sideman, A. (211). Agencies must determine computer security teams in face of potential federal shutdown. Retrieved from Wickens, T. D. (21). Elementary signal detection theory. New York, NY: Oxford University Press, USA. Xie, P., Li, J. H., Ou X., Liu, P., & Levy, R. (21). Using Bayesian networks for cyber security analysis. In Proceedings of the 21 IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) (pp ). Hong Kong, China: IEEE Press. doi: 1.119/DSN Acknowledgements We are very thankful to Dr. Varun Dutt, Assistant Professor, Indian Institute of Technology, Mandi for guiding us in difficult times on this project. Also, we are grateful to the Indian Institute of Technology, Mandi for providing the necessary capital and financial resources that made this project possible in the first place. This research was also partially supported by the Multidisciplinary University Research Initiative Award on Cyber Situation Awareness (MURI; #W911NF ) from Army Research Office to Cleotilde Gonzalez, Carnegie Mellon University, USA. References Anderson, J. R., & Lebiere, C. (1998). The atomic components of thought. Hillsdale, NJ: Lawrence Erlbaum Associates. Dutt, V., Ahn, Y. S., & Gonzalez, C. (in press). Cyber situation awareness: modeling detection of cyber attacks with instance-based learning theory. Human Factors. Dutt, V., & Gonzalez, C. (212). Cyber Situation Awareness through Instance- Based Learning: Modeling the Security Analyst in a Cyber-Attack Scenario. In C. Onwubiko & T. Owens (Eds.), Situational Awareness in Computer Network Defense: Principles, Methods and Applications (pp ). Hershey, PA: IGI Global. Gonzalez, C., & Dutt, V. (211). Instance-based learning: Integrating decisions from experience in sampling and repeated choice paradigms. Psychological Review, 118(4), doi: 1.137/a24558 Gonzalez, C., Dutt, V., & Lejarraga, T. (211). A loser can be a winner: Comparison of two instance-based learning models in a market entry competition. Games, 2(1), doi: 1.339/g