Optimal Personalized Filtering Against Spear-Phishing Attacks

Transcription

1 Optimal Personalized Filtering Against Spear-Phishing Attacks Aron Laszka and Yevgeniy Vorobeychik and Xenofon Kotsokos Institte for Software Integrated Systems Department of Electrical Engineering and Compter Science Vanderbilt University Nashville, TN Abstract To penetrate sensitive compter networks, attackers can se spear phishing to sidestep technical secrity mechanisms by exploiting the privileges of careless sers. In order to maximize their sccess probability, attackers have to target the sers that constitte the weakest links of the system. The optimal selection of these target sers takes into accont both the damage that can be cased by a ser and the probability of a malicios being delivered to and opened by a ser. Since attackers select their targets in a strategic way, the optimal mitigation of these attacks reqires the defender to also personalize the filters by taking into accont the sers properties. In this paper, we assme that a learned classifier is given and propose strategic per-ser filtering thresholds for mitigating spear-phishing attacks. We formlate the problem of filtering targeted and non-targeted malicios s as a Stackelberg secrity game. We characterize the optimal filtering strategies and show how to compte them in practice. Finally, we evalate or reslts sing two real-world datasets and demonstrate that the proposed thresholds lead to lower losses than nonstrategic thresholds. 1 Introdction To sccessflly breach highly secre systems, attackers have to focs on the weakest link in the chain of secrity, which is often the sers (Sasse, Brostoff, and Weirich 2001). One particlarly pernicios form of attack on sers is spear phishing, that is, targeting specific sers (or classes of sers) throgh malicios , making se of their individal characteristics, sch as who their bosses or friends are, to bild trst (Hong 2012). In recent years, we have seen several spear-phishing attacks that sccessflly breached highly secre organizations. For example, in 2011, the Oak Ridge National Laboratory, which condcts classified and nclassified energy and national secrity work, was breached by a spear-phishing attack (Zetter 2011). In this incident, the attackers sent an , which claimed to be from hman resorces, to the lab employees. This contained a link to a malicios website, which infected the employees Copyright c 2015, Association for the Advancement of Artificial Intelligence ( All rights reserved. compters with a malware that sbseqently stole sensitive data and sent it to an nknown destination. As another example, in 2012, one of the White Hose internal networks was breached sing spear phishing (McCllagh 2012). The attackers, who are believed to have sed servers in China, were allegedly able to access the network of the president s military office, which is in charge of, for example, strategic nclear commands. Finally, compters at the Nclear Reglatory Commission (NRC) of the U.S., which contain sensitive information that cold be sed for srveillance or sabotage, were breached three times in the past three years (Rogers 2014). In the most recent incident, the attackers first compromised an NRC employee s personal e- mail accont, which they then sed to send s to 16 other employees. The contained a malicios PDF attachment, which infected the compter of an employee who opened the attachment (Rosenblatt 2014). The defining characteristic of spear-phishing attacks which differentiates them from reglar phishing or spam is that they are targeted at specific, careflly chosen individals or grops. Since sending a large nmber of similar s (e.g., with the same malicios attachment) wold almost certainly raise an alarm, the attackers focs on a sbset of the sers who constitte the weakest links of the system. Moreover, the emergence of digital and social media has made it easier for attackers to know mch abot their prospective targets, sch as where they work, what they are interested in, and who their friends are (McAfee Labs 2014; Jagatic et al. 2007). Typical mitigation for phishing attacks is the same as for spam: there is an filtering system, often based in part on machine learning, which comptes a risk score for each e- mail and filters those for which the risk score exceeds some pre-specified threshold. The vale of this filtering threshold has to be careflly chosen, since overzealos filtering may also remove many non-malicios s. Hence, defenders have to find the right balance between secrity and sability (Sheng et al. 2009). Frthermore, these thresholds can be personalized, as different sers have different levels of carefllness and different potential to case damage. For example, a recent report fond based on a large-scale experiment that the departments which hold the most sensitive data in a bsiness, sch as HR, acconting, and finance, are the worst at detecting frad (McAfee Labs 2014).

2 However, the targeted natre of spear phishing makes the problem qalitatively different: since the attacker selects the target sers by taking into accont both their individal properties and their filtering thresholds, the defender has to set the thresholds in a strategic way. In this paper, we investigate the problem of optimally setting personalized filtering thresholds against spear-phishing attacks, given an classifier with its associated false-negative / false-positive probability tradeoff. Specifically, we model this problem as a Stackelberg game, characterize the optimal filtering strategies, and show how these filtering strategies can be compted in practice at scale. We also evalate the proposed filtering strategies sing real data, demonstrating that or approach leads to better otcomes for the defender. The remainder of this paper is organized as follows. In Section 2, we discss related work on filtering malicios e- mails. In Section 3, we introdce or game-theoretic model. In Section 4, we present analytical reslts on or model. In Section 5, we present nmerical reslts. Finally, in Section 6, we give or conclding remarks. 2 Related Work There are many research reslts on measring sers ssceptibility to phishing attacks and the detection and classification of potentially malicios s. These reslts are complementary to ors, since we assme that the sers ssceptibility has been measred and a classifier has been trained, and we bild or model on these assmptions. Several experiments have been condcted to measre individals ssceptibility to phishing attacks. For example, the athors of (Jagatic et al. 2007) performed an experimental stdy at Indiana University to measre individals probabilities of falling victim to phishing. To measre these probabilities, the athors lanched an actal (bt harmless) phishing attack targeting stdents and tilizing pblicly available acqaintance data mined from social-network websites. The reslts show that certain characteristics of the targeted stdents, sch as gender, academic major, and grade, have a significant effect on the probabilities. As another example, the athors of (Sheng et al. 2010) performed an online srvey to stdy the relationship between demographic and phishing ssceptibility. The stdy, which was based on an online roleplaying task, fond that certain factors, sch as gender and age, have a significant effect. The problem of detecting malicios s has also been extensively stdied. For example, the athors of (Fette, Sadeh, and Tomasic 2007) apply machine learning to a featre set designed to highlight ser-targeted deception. When evalated on a real-world dataset, their method correctly identified over 96% of the phishing s while misclassifying only approximately 0.1% of the non-malicios e- mails. More recently, the problem of classifying malicios s has also been stdied as an adversarial data-mining problem. In adversarial data mining (or adversarial machine learning), the classification problem is viewed as a game between the classifier and an adversary, who maniplates the instances to be classified in order to increase the nmber of false negatives (Dalvi et al. 2004). For example, the athors of (L Hillier, Weber, and Figeroa 2009) bild an adversary-aware classifier for detecting phishing s sing an online version of Weighted Margin Spport Vector Machines, and they present experimental reslts showing that it is highly competitive compared to previos online classification algorithms. Besides their textal content, phishing s can also often be identified by detecting links to malicios websites, which can initiate a drive-by download or install. The athors of (Ma et al. 2009) stdy the problem of detecting malicios websites and propose a website classifier, which ses statistical methods, lexical featres of the URL, and host-based featres, sch as WHOIS and geographic properties. As another example, the athors of (Choi, Zh, and Lee 2011) propose a method sing machine learning to detect malicios URLs and to identify the natre of the attack. The proposed method ses a variety of featres, inclding lexical featres of the URL, link poplarity of the website, content featres of the webpage, and DNS featres. 3 Model Now, we introdce or game-theoretic model of filtering targeted and non-targeted malicios s. For a list of symbols sed in this paper, see Table 1. Symbol Table 1: List of Symbols Description FP (f) false-positive probability given that the false-negative probability is f A nmber of sers targeted by the attacker L expected damage for delivering targeted malicios s to ser N expected damage for delivering nontargeted malicios s to ser C expected loss from filtering ot nonmalicios s to ser f T optimal false-negative probability of ser given that the ser is targeted f N optimal false-negative probability of ser given that the ser is not targeted We assme that the classifier of the organization otpts a maliciosness score for each received , and an is delivered to the recipient if and only if the score is below a given threshold. We call misclassified malicios s false negatives (i.e., when a malicios is below the threshold) and misclassified non-malicios s false positives (i.e., when a non-malicios is above the threshold). By adjsting the filtering threshold, the organization can increase the probability of false positives and decrease the probability of false negatives, or vice versa. We represent the attainable false-positive and falsenegative probability pairs sing a fnction FP : [0, 1] [0, 1], where FP (FN) is the probability of false positives when the the probability of false negatives is FN. In any practical classifier, FP is a non-increasing fnction of FN.

3 For analytical tractability, we frther assme that FN is continos, strictly decreasing, and strictly convex fnction of FN. Note that, in Section 5, we show that or reslts can be applied sccessflly to FP fnctions that do not satisfy these additional assmptions. We let L denote the expected amont of damage (i.e., loss) that the organization sstains for delivering malicios targeted s to ser. This amont L can be compted as L = E[damage to organization ser falls victim] Pr[ser falls victim is delivered] rate of targeted attacks. (1) In practice, any organization that aims to be prepared against cyber-attacks needs to have some estimate of its cyberassets vale and the expected freqency of attack attempts; hence, it shold be able to estimate the first and the third factors. Moreover, the second factor (i.e., the probability of falling victim) can be measred by sending probe s to the sers. Besides spear phishing, the organization also receives non-targeted malicios s. We let N denote the loss that the organization sstains for delivering malicios nontargeted s to ser. Finally, an organization also has to take into accont the prodction and sability loss sstained when a non-malicios is filtered ot. We let C denote the amont of loss sstained for not delivering non-malicios s addressed to ser. Attacker-Defender Game We model the conflict between the targeting attacker and the organization as a Stackelberg secrity game, where the defender s role is played by the organization. The attacker s strategic choice is to select a sbset of sers A to whom she sends malicios s. Since a large nmber of s containing the same malware or linking to websites distribting the same malware cold easily be detected, the attacker tries to stay covert by sending only a limited nmber of s. Formally, we model this limitation by assming that the attacker s strategy has to satisfy A A, where A is a constant. The defender s strategic choice is to select the falsenegative probability f for each ser. Recall that the reslting false-positive probability for ser is FP (f ). For a given strategy profile (f, A), the players payoffs are defined as follows. The attacker s payoff is U attacker = A f L, (2) and the defender s loss (i.e., inverse payoff) is L defender = U attacker + = f L + A f N + FP (f )C (3) f N + FP (f )C. (4) In the analysis, or goal will be to find the attacker s best response and the defender s optimal strategies, which are defined as follows. Definition 1. An attacker strategy is a best response if it maximizes the attacker s payoff, taking the defense strategy as given. As is typical in the secrity literatre, we consider sbgame perfect Nash eqilibria as or soltion concept (Korzhyk et al. 2011). We will refer to the defender s eqilibrim strategies as optimal strategies for the remainder of the paper. Note that, as we will discss at the beginning of Section 4, or model allows the attacker to break ties between mltiple best-response strategies in an arbitrary manner. Definition 2. We call a defense strategy optimal if it maximizes the defender s payoff given that the attacker will always play a best-response strategy. 4 Analysis We begin or analysis with characterizing the attacker s best-response strategies and then stdy the problem of finding an optimal defense strategy. From Eqation (2), it follows immediately that, against a given defense strategy f, the targeting attacker s bestresponse strategy is to choose the set of A sers with the highest f L vales. Frthermore, if there are mltiple bestresponse strategies (i.e., mltiple sets of sers attaining the same sm), then these strategies all yield the same payoff to the defender as well, since the defender s payoff depends on the attacker s strategy only throgh the attacker s payoff (see first term in Eqation 4). In other words, the attacker can break ties between best responses in an arbitrary way. To facilitate or analysis, we now introdce some additional notation. Let f T denote the optimal vale of f given that A, and let f N denote the optimal vale of f given that A. Formally, for each ser, f T and f N are the vales at which the minima of and f (L + N ) + FP (f )C (5) f N + FP (f )C (6) are attained, respectively. Note that it is fairly easy to show these vales are well-defined and niqe for each ser. Optimal Defense Sbproblem First, we stdy an important sbproblem of finding an optimal defense strategy. Sppose that a set of sers A is given, and we are looking for the optimal defense strategy against which A is a best-response strategy for the attacker. In other words, we restrict or search space to defense strategies in which the sers of A have the highest f L vales. We begin with a special case, in which the parameter vales of the sers in A differ sbstantially from those of the remaining sers. Proposition 1. Sppose that a set of sers A is given, and the defender s choice is restricted to strategies against which A is a best response. If min A f T L max A f N L, then choosing f T for every A and choosing f N for every A is the optimal defense strategy.

4 Proof. (Sketch.) Firstly, A is a best response for the attacker, since the sers in A have the highest f L vales. Secondly, for each A, f = f T is optimal by definition, and for each A, f = f N is also optimal by definition. Then, as the defender s loss is the sm of the losses for the individal sers, the strategy f mst also be optimal for the given A. It is noteworthy that this strategy profile (i.e., the defender s strategy f given by Proposition 1 and the attacker s strategy A) wold actally be a niqe Nash eqilibrim in a simltaneos version of the game, as both players strategies are best responses. 1 However, this Nash eqilibrim is not necessarily a sbgame perfect Nash eqilibrim in or Stackelberg game. The above proposition provides a complete characterization of the optimal defense strategy for a special case. Next, we consider the general case, where the condition of Proposition 1 might not hold, and provide necessary conditions on the optimal defense strategy. Theorem 1. Sppose that a set of sers A is given, and the defender s choice is restricted to strategies against which A is a best response. Then, in an optimal defense strategy, there exists a vale sch that for every A, f L = if f T L <, and f = f T otherwise, for every A, f L = if f N L >, and f = f N otherwise. Intitively, the above theorem states that, in an optimal defense, sers in A with a sfficiently high f T L will have f L = f T L, sers not in A with a sfficiently low f N L will have f L = f N L, and all other sers will have a niform f L vale, which we let be denoted by. See Figre 1 for an illstration. f L Figre 1: Illstration for Theorem 1 with for sers and A = 2. Ble dots represent f N L vales, and red dots represent f T L vales. 1 The niqeness of the eqilibrim follows from the observation that, if a set A satisfies the condition of the above lemma, then no other set can satisfy it. Frthermore, it can easily be shown that the game has a Nash eqilibrim only if there exists a set A satisfying the condition of the lemma. A Proof. (Sketch.) It is obvios that min A f L max A f L is a necessary and sfficient condition for A to be a best response. Then, given an optimal defense strategy f, let be max A f L. We have to show that each f L takes the vale f T L, f N L, or given by the lemma. First, if A, then the optimal vale for f L wold be f T L ; however, the actal vale cannot be lower than, since A wold not be a best response otherwise. Using the convexity of FP (f ), it can then be shown that f L = is an optimal choice whenever f T L <. Second, if A, then the optimal vale for f L wold be f N L ; however, the actal vale cannot be higher than by definition (recall that we let = max A f L for the proof). Again, sing the convexity of FP (f ), it can be shown that f L = is an optimal choice whenever f N L >. Note that, based on the above theorem, we can easily find the optimal vale for any given set A sing, for example, a binary search. Conseqently, we can find an optimal defense strategy by iterating over all A-sized sbsets of the sers and solving each defense sbproblem. Generally, this approach is not feasible in practice, as the nmber of possible sbsets increases exponentially with A. However, if the nmber of sers that can be targeted by the attacker is very limited, we can find the attacker s best response sing an exhastive search. In the case A = 1, this simply means iterating over the set of sers. For the general case, we provide an efficient approach in the following sbsection. Optimal Defense The previos theorem establishes that, in an optimal defense strategy, the sers f vales are either f T, f N, or some L. Now, we discss how this observation can be sed to find an optimal defense strategy. The following theorem shows how to find an optimal strategy for a given vale. Note that this differs from the assignments in Theorem 1, where the set A was given. Theorem 2. Sppose that we are given a constant, and the defender s choice is restricted to strategies where max A f L and min A f L for a best response A 2. Then, the otpt of the following algorithm is an optimal defense strategy: 1. For each ser, compte the loss of ser when it is not targeted as follows: if f N L <, then the loss is f N N + FP (f N )C ; otherwise, the loss is L N + FP ( L )C. 2. For each ser, compte the loss of ser when it is targeted as follows: if f T L >, then the loss is f T (L + N ) + FP (f T )C ; otherwise, the loss is L (L + N ) + FP ( L )C. 2 Recall that the attacker always targets the A sers with the highest f L vales; hence, both max A f L and min A f L are niform over the best responses.

5 3. For each ser, let the cost of ser being targeted be the difference between the above compted loss vales. 4. Select a set A of A sers with the lowest costs of being targeted. 5. For every A, let f = f T if f T L >, and let f = L otherwise. 6. For every A, let f = f N if f N L <, and let f = L otherwise. 7. Otpt the strategy f. Proof. (Sketch.) First, sppose that besides a best response A is also given. In other words, the defender s choice is restricted to strategies against which A is a best response, max A f L, and min A f L. Then, we can show that Steps 5 and 6 of the above algorithm are optimal sing an argment similar to the one in the proof of Theorem 1. Second, we show that Steps 1 to 4 yield an optimal set A. For the sake of contradiction, sppose that for some instance of the game, there exists a set A that leads to lower expected loss for the defender. Note that, since we already have that Steps 5 and 6 give an optimal assignment for any set, we can assme that the defense strategies corresponding to the sets A and A are given by Steps 5 and 6. Now, let + be a ser that is in A bt not in A, and let be a ser that is in A bt not in A. By removing + and adding to A, the defender s expected loss is decreased by the difference between the costs of + and being targeted. Since A consists of the A sers with the lowest costs of being targeted (see Step 4), this difference has to be nonnegative; hence, the expected loss is not increased by sch changes to A. Then, sing at most A sch changes, we can transform A into A, withot increasing the expected loss. However, this contradicts the assmption that A leads to lower expected loss than A; therefore, the original claim mst hold. Efficient Search Let L defender () denote the minimm loss that the defender can achieve for a given vale (i.e., the defender s loss for the defense strategy otpt by the algorithm of Theorem 2 and the attacker s best response against it). Then, finding an optimal defense strategy is eqivalent to finding argmin L defender () (see Figres 3(a) and 3(b) for an illstration). Hence, we redced the problem of finding an optimal defense strategy to the problem of optimizing a single scalar vale. 5 Experiments In this section, we evalate or model sing real-world datasets and compare or optimal strategic thresholds to niform thresholds. Please note that the goal of these experiments is not to find a classifier that performs better than other classifiers in the literatre, since or model assmes that a classifier and the reslting false positive / false negative crves are given. The goal of these experiments is to demonstrate the practical feasibility of or approach for setting the classification thresholds and to show that it otperforms non-strategic soltions. FP FN (a) UCI dataset FN (b) Enron dataset Figre 2: False-positive probability as a fnction of falsenegative probability. Datasets We sed two pblicly available datasets for or nmerical examples. For both datasets, we trained a naïve Bayes classifier. UCI The first dataset is from the UCI Machine Learning Repository (Bache and Lichman 2013), which is a labeled collection of messages. Each has 57 featres, most of which indicate freqencies of particlar words or characters. We sed 80% of the collection for training or classifier, and the remaining 20% for testing it, that is, for obtaining the false negative / false positive trade-off crve. Enron The second dataset is the Enron dataset 3 (Klimt and Yang 2004). For each message, we compted 500 featres based on the content of the message. We sed 12 thosand s from the dataset for training or classifier and s for testing it. Figres 2(a) and 2(b) show the false-positive probability (FP) as a fnction of the false-negative probability (FN) for the UCI and Enron datasets, respectively. Finding Optimal Strategies Recall that, in Section 3, we assmed the fnction FP (FN) to be strictly convex. However, the actal crves shown by Figres 2(a) and 2(b) are only approximately convex, since they have a nmber of smaller irreglarities. We now discss how the overcome the challenges posed by these irreglarities to the application of or theoretical reslts. First, the vales of f N and f T might be non-niqe if the fnction FP (FN) is not strictly convex. However, in practice, the probability of mltiple global minima is negligible. 4 Nevertheless, if there were mltiple vales minimizing the defender s loss for ser when A, we cold simply define f T to be the maximal vale. It is easy to see that this is the best choice, since it will allow s to se the optimal vale f T L instead of as long as possible. Similarly, we can define f N to be the minimal vale that minimizes the defender s loss for ser when A In any case, we are limited by the size of the classifier s testing set and the actal precision of floating point nmbers, so the existence of mltiple global minima is mostly a pecliarity of the implementation.

6 Expected loss Ldefender (a) UCI dataset (b) Enron dataset Figre 3: Expected loss as a fnction of for A = 3. Second, finding the vales of f N and f T cold be challenging, since the defender s loss can be a non-convex fnction of f. However, in practice, the fnction FN(FP ) is actally given by a set of datapoints, whose cardinality is necessarily pper bonded by the cardinality of the testing set of the classifier. Conseqently, even a simple exhastive search is feasible, since its rnning time will be linear in the size of the inpt. Finally, finding the optimal vale of cold also be challenging, since the objective fnction (i.e., the defender s expected loss) can be a non-convex fnction of. Figres 3(a) and 3(b) show the defender s expected loss as fnctions of for strategies compted sing the algorithm of Theorem 2 for the UCI and Enron datasets, respectively. However, we can see that the objective fnction is relatively smooth in practice, it has only a few local minima, all of which are in the vicinity of the global minimm. Frthermore, we can even se an exhastive search, since the fnction L defender () is again given by a set of data points, whose cardinality is pper bonded by the nmber of sers cardinality of the testing set of the classifier. Hence, the rnning time of an exhastive search will be qadratic in the size of the inpt. Comparison with Non-Strategic Thresholds Now, we stdy the main qestion regarding or reslts: can the strategic setting of thresholds decrease the expected amont of losses? To answer this qestion, we compare or strategic thresholds with two non-strategic, niform thresholds. These niform thresholds do not expect the attacker to select the targets in a strategic manner, bt they are otherwise optimal (i.e., minimize the expected losses). Uniform Threshold #1 The first baseline assmes that the attacker targets the sers niformly at random; hence, the niform false-negative probability f is compted as argmin f f ( N + A ) 1 L + FP (f) C. Uniform Threshold #2 The second baseline assmes that the attacker targets those sers who have the most potential to case losses (i.e., have the highest L vales); hence, the Expected loss Ldefender Nmber of sers targeted A (a) UCI dataset Nmber of sers targeted A (b) Enron dataset Figre 4: Expected loss as a fnction of A for the optimal strategy (solid line) and niform thresholds (dashed and dotted lines). niform false-negative probability f is compted as ( ) argmin f N + max L + FP (f) f A: A =A A C. For the nmerical examples, we generated a set of 31 sers as follows: For every ser, potential losses de to ndelivered nonmalicios and delivered targeted malicios s are approximately ten times higher than losses de to delivered non-targeted s. Formally, for each ser, L, C 10 N. The motivation behind this choice is the standard assmption that ndelivered non-malicios s are mch worse than delivered non-targeted malicios s, sch as spam. Frthermore, based on examples of spear-phishing attacks, it is reasonable to assme that targeted malicios s are also mch worse. The potential damage vales L, C, and N follow a power law distribtion. Formally, the nmber of sers with damage vales between some l and l + 1 is approximately twice as mch as the nmber of sers with vales between l + 1 and l + 2. Finally, the vale of L ranges from 0.5 to 5.5. The motivation behind modeling the potential damage vales with a power law distribtion is the typical hierarchical strctre of organizations, where the nmber of employees at a higher level is typically smaller. Figres 4(a) and 4(b) compare or strategic soltion to niform thresholds at varios attack sizes for the UCI and Enron datasets, respectively. The solid line ( ) shows the defender s expected loss for or optimal strategy, the dashed line ( ) shows the loss for niform threshold #1, and the dotted line ( ) shows the loss for niform threshold #2. Note that, for every threshold, we compted the defender s loss based on the attacker s best response in or model, as the goal is to compare how different thresholds perform against a targeting attacker. We can see that the proposed strategic soltion is clearly sperior in every case. Frthermore, the improvement over the non-strategic thresholds is qite stable with respect to A, that is, the improvement does not diminish as the attacker targets more and more sers. Finally, by comparing the reslts for the two datasets, we can see that the relative

7 improvement is higher for the more detailed dataset (i.e., Enron), which sggests that it is possible that or soltion cold lead to even higher improvements for more detailed datasets. 6 Conclsion Since the weakest link in the chain of secrity is often hman behavior, thwarting spear-phishing attacks is a crcial problem for any organization that aims to attain a high level of secrity. Besides ser edcation, the most typical defense against phishing attacks is the filtering of malicios s. In this paper, we focsed on the problem of finding optimal filtering thresholds against targeted and non-targeted malicios s. The targeted, strategic natre of spear-phishing attacks presents an interesting problem, which we modeled as a Stackelberg secrity game. While characterizing the attacker s best response is trivial, characterizing and finding the defender s optimal strategy is mch more challenging. However, sing Theorem 2, we can redce this problem to a mch simpler scalar optimization, which as we discssed in Section 5 can be efficiently solved in practice, even for large datasets. Finally, we evalated or theoretical reslts sing two real-world datasets, which reslt in typical false-negative / false-positive crves. We compared or strategic thresholds to two non-strategic thresholds, and fond that or strategic thresholds are clearly sperior. Frthermore, we also fond that the improvement over the non-strategic thresholds is higher for the more detailed dataset and it does not diminish as the nmber targeted sers increases. This shows that or method scales well not only comptationally, bt also performance-wise. Acknowledgment This work was spported in part by the National Science Fondation nder Award CNS , by the Air Force Research Laboratory nder Award FA , and by Sandia National Laboratories. References Bache, K., and Lichman, M UCI Machine Learning Repository. Choi, H.; Zh, B. B.; and Lee, H Detecting malicios web links and identifying their attack types. In Proceedings of the 2nd USENIX Conference on Web Application Development, USENIX Association. Dalvi, N.; Domingos, P.; Sanghai, S.; Verma, D.; et al Adversarial classification. In Proceedings of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), Fette, I.; Sadeh, N.; and Tomasic, A Learning to detect phishing s. In Proceedings of the 16th International Conference on World Wide Web, WWW 07, New York, NY, USA: ACM. Hong, J The state of phishing attacks. Commnications of the ACM 55(1): Jagatic, T. N.; Johnson, N. A.; Jakobsson, M.; and Menczer, F Social phishing. Commnications of the ACM 50(10): Klimt, B., and Yang, Y The Enron corps: A new dataset for classification research. In Proceedings of the 15th Eropean Conference on Machine Learning (ECML), Springer. Korzhyk, D.; Yin, Z.; Kiekintveld, C.; Conitzer, V.; and Tambe, M Stackelberg vs. Nash in secrity games: An extended investigation of interchangeability, eqivalence, and niqeness. Jornal of Artificial Intelligence Research 41(2): L Hillier, G.; Weber, R.; and Figeroa, N Online phishing classification sing adversarial data mining and signaling games. In Proceedings of the ACM SIGKDD Workshop on CyberSecrity and Intelligence Informatics, ACM. Ma, J.; Sal, L. K.; Savage, S.; and Voelker, G. M Beyond blacklists: learning to detect malicios web sites from sspicios URLs. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM. McAfee Labs Threats report - Agst reports/rp-qarterly-threat-q pdf. McCllagh, D White Hose confirms spearphishing intrsion. CNET, white-hose-confirms-spearphishingintrsion/. Rogers, J Hackers attack Nclear Reglatory Commission 3 times in 3 years. FOX News, foxnews.com/tech/2014/08/20/hackersattack-nclear-reglatory-commission/. Rosenblatt, S Nclear reglator hacked 3 times in 3 years. CNET, nclear-commission-hacked-3-times-in-3- years/. Sasse, M. A.; Brostoff, S.; and Weirich, D Transforming the weakest link A hman/compter interaction approach to sable and effective secrity. BT Technology Jornal 19(3): Sheng, S.; Kmaragr, P.; Acqisti, A.; Cranor, L.; and Hong, J Improving phishing contermeasres: An analysis of expert interviews. Proceedings of the 4th APWG ecrime Researchers Smmit. Sheng, S.; Holbrook, M.; Kmaragr, P.; Cranor, L. F.; and Downs, J Who falls for phish? A demographic analysis of phishing ssceptibility and effectiveness of interventions. In Proceedings of the SIGCHI Conference on Hman Factors in Compting Systems, ACM. Zetter, K Top federal lab hacked in spear-phishing attack. WIRED, oak-ridge-lab-hack/.