KEEPING UP WITH THE JONESES: ASSESSING PHISHING SUSCEPTIBILITY IN AN EMAIL TASK Kyung Wha Hong 1, Christopher M. Kelley 2, Rucha Tembe 2, Emerson Murphy-Hill 1 & Christopher B. Mayhorn 2 1 Department of Computer Science, 2 Department of Psychology, North Carolina State University, Raleigh, NC Most prior research on preventing phishing attacks focuses on technology to identify and prevent the delivery of phishing emails. The current study supports an ongoing effort to develop a user-profile that predicts when phishing attacks will be successful. We sought to identify the behavioral, cognitive and perceptual attributes that make some individuals more vulnerable to phishing attack than others. Fifty-three participants responded to a number of self-report measures (e.g., dispositional trust) and completed the Bob Jones email task that was designed to empirically evaluate phishing susceptibility. Over 92% of participants were to some extent vulnerable to phishing attacks. Additionally, individual differences in gender, trust, and personality were associated with phishing vulnerability. Application and implications for future research are discussed. INTRODUCTION Cybersecurity involves a complex interaction between users and technology. While security threats might take a variety of forms such as viruses or worms delivered via nefarious websites or USB drives, theft using social engineering tactics such as phishing are becoming increasingly common and costly. Loss of time and increased stress levels are the immediate personal costs (Hardee, West, & Mayhorn, 2006). Long term personal costs are likely as well, such as decreased trust and usage of the internet for banking, shopping, and other conveniences (Dhamija & Tygar, 2005; Kelley, Hong, Mayhorn, & Murphy-Hill, 2012). In terms of economic losses, a recent survey (Gartner, 2007) indicates phishing attacks caused a loss of 3.2 billion dollars based on a sample of 4500 adults with an average of $866 lost per phishing occurrence. Moreover, phishing targeted at administrators can compromise entire systems and user communities (Schwartz, 2011). The goal of this research is to develop a userprofile that predicts when and where phishing attacks will be successful. Such a user-profile could be useful to help identify behavioral, cognitive, and perceptual differences that make some users more susceptible to phishing than others. For instance, individual differences in trust and cognitive and attentional capacity have been identified separately as contributing to phishing susceptibility. However, no one has constructed a unified user-profile that combines individual differences to proactively identify individual users who are prone to being successfully phished. Participants METHOD Fifty-three undergraduate students were recruited to complete an experiment (Table 1). Participants were tested individually in sessions that lasted approximately two hours and given extracredit as compensation. Materials The experiment was completed in two stages such that participants completed an online survey and then a laboratory session.
Self-report measures. Participants completed a survey that measured demographic characteristics such as age, gender, and primary language as well as previous experiences with phishing, online purchasing behavior, and general computing behavior (based on Eveland, Shah, & Kwak, 2003; Yoshioka, Washizaki, & Maruyama, 2008). Participants also responded to measures of dispositional trust (Merritt & Ilgen, 2008), impulsivity (Neyste & Mayhorn, 2009), and personality (Gosling, Rentfrow, & Swann, 2003). Table 1 Participant Characteristics M SD Range be from careerbuilder.com, a legitimate website representing a real company (even with their logo). Also it seems to give useful information to the user. However, if a user clicks on the links included in the email, it actually leads them to a website that is not related to careerbuilder s official website. Disguising the sender or source of an email by making it look like a legitimate company is a typical tactic used to create phishing emails. Figure 1 Example Phishing Email Age 20.20 2.33 18-27 Frequencies Gender Male Female Race Caucasian/Non-hispanic Language English Primary Major Computer Science Psychology 60% 40% 80% 96% 34% 66% Participants were given the following instructions: Behavioral measures. To empirically assess phishing susceptibility, participants completed an email task where they were asked to access a Google Mail account for a character named Bob Jones and categorize 14 email messages (Table 2). Table 2 Email Messages Divided by Category Email Category Phishing 7 Spam 1 Malware 1 Legitimate 5 Total 14 Figure 1 shows one of the phishing emails we used as stimuli in this experiment. This email appears to n When you are going through each email, do as you normally do. For example, if you normally read each email carefully do as you usually do. Or if you usually skim through each message quickly that s also fine, too. After going through an email you have to make a decision about the email. If you think email is legitimate and you d like to respond (e.g., reply, click on a link, download a file) to the email, then mark Important. If you think email is legitimate but doesn t need any response and would like to just archive, leave it as it is. If you think email is not legitimate, suspicious, or spam, then Delete. Procedure
After providing informed consent and completion of the self-report measures delivered online, participants visited the laboratory where a battery of cognitive tests and the Bob Jones email task were administered. The cognitive tests included a measure of working memory capacity (WMC) (Unsworth, Heitz, Schrock, & Engle, 2005), crystallized intelligence (Shipley, 1986), spatial ability (Peters et al., 1995; Vandenberg & Kuse, 1978), and sustained attention (Temple et al., 2000). Upon completion of the cognitive tests, instructions for the Bob Jones email task were delivered. Finally, participants were debriefed and dismissed. RESULTS Responses to self-report measures were captured via an online survey tool, Qualtrics, and the results of the cognitive tests and the Bob Jones email task were entered into SPSS for analysis. Survey Results Prior phishing experience. Many respondents indicated that they had previous phishing experience via email. For instance, 25% reported glancing at the contents of a phishing email whereas 36% admitted to completely reading a phishing message. Thirty percent were compelled to ask someone else whether they thought the email was authentic whereas 11% reported contacting an authority (e.g., bank). The most severe phishing consequences seemed to be relatively rare with 15% clicking on a link, 8% installing a virus/malware, and 6% entering personal information. Of those who entered personal information, name (6%) and password (6%) comprised the information provided to phishers. Most frequent consequences of worst experience included noticed unusual activity in an online account (15%) and reduced online activity (15%). Based on this previous experience, 89% agreed that they were confident that they can tell the difference between a legitimate email and one sent by a scammer. Behavioral Results Bob Jones email task performance. To ascertain phishing susceptibility, a score that ranged from 0 (perfect ability) to 100 (no ability) was calculated for participant s ability to identify phishing emails. The data suggested more than 92% of participants were susceptible to phishing with only 4 participants (7.5% of the sample) successfully identifying all of the phishing emails and approximately 52% misclassifying more than half of the phishing emails. Since phishing also impacts the ability of people to identify legitimate emails, the number of authentic emails that were incorrectly deleted was assessed. Fifty-four percent deleted at least one authentic email. Individual differences correlated with accuracy. The ability to correctly identify phishing emails revealed gender, trust, and personality were correlated with phishing vulnerability. For example, women were less likely than men to correctly identify phishing emails, t(51) = -2.15, p <.036. Dispositional trust, extraversion and openness to new experience were correlated with deleting legitimate emails. Specifically, less trusting individuals, r(52) = -.30, p <.034, introverts, r(53) = -.29, p <.054, and those less open to new experiences, r(53) = -.435, p <.002, were more likely to delete legitimate emails. Severity of email misclassification. In addition, because misclassifying some emails could have more severe consequences than others, five classes of email severity were created that ranged from 1 to 5. (Class 1:legitimate email no danger, Class 2:spam email or email sent to numerous recipients no danger but less useful, Class 3:phishing email redirecting to unexpected site no danger, Class 4:phishing email with a danger of loosing less critical information, Class 5: phishing email with a danger of losing money or critical information). Thus, when an email was misclassified a severity score was assigned based on the participant s response (e.g., their classification) and the consequence of misclassifying that particular email (Table 3). For example, if a participant responded with important for a phishing email in email severity class 4, the severity score for this response was assigned a score of 4. However, if this participant responded with delete for a phishing email in email severity class 5, the severity score for this response was assigned a score of 0. A total severity score due to misclassification was calculated as the sum of severity scores for
each email response and ranged from 0 (no consequence) to 23 (severe consequence). Table 3 The Severity Score based on Email Severity Class and Participants responses Results revealed an average severity score of 14.24. What s more, only 2% of participants correctly classified all emails indicating approximately 98% would have experienced adverse consequences resulting from email misclassification. DISCUSSION While the topic of phishing and social engineering is not new, the current focus on the human side of the HCI equation promises to expand our knowledge in this area. The preliminary results of the current study illustrate a number of findings. First, results suggest a disconnect between participants self-reported data and the empirical data collected from the Bob Jones email task. Specifically, approximately 92% of participants misclassified phishing emails even though 89% indicated they were confident of their ability to identify phishing emails. These results suggest a majority of participants were not only susceptible to phishing attacks, but overconfident in their ability to protect themselves from such attacks. Second, only 2% of the participants suffered no adverse consequences due to misclassification of emails during the task. Third, individual differences such as gender, dispositional trust, and personality appear to be associated with the ability to correctly categorize emails as either legitimate or phishing. Limitations While these results are interesting, they should be interpreted with caution given several potential methodological and analytical limitations. For instance, reliance on self-report of prior behavior may be subject to memory biases. Likewise, the behavioral measure (Bob Jones email task) could be described as artificial because participants were asked to role play; however, this methodology has been validated with prior research (Sheng et al., 2010). Moreover, analysis of the consequences of participants email misclassification severity was based on a preliminary coding scheme developed by an individual rater. Current efforts are underway to provide inter-rater reliability for this measure and additional measures used in the Bob Jones email task. The sample recruited for the current study consisted of college students. However, efforts are currently underway to recruit a more diverse set of participants (i.e., a non-student sample of working professionals). Recently, we collected data from volunteers employed at a government agency. Future analyses will compare the students and nonstudents to determine whether there are similarities that are common to the two groups and more importantly, how they vary in terms of phishing susceptibility. Future Research and Application These results contribute to an ongoing effort to develop a user profile that identifies those most at risk of being phished. One implication might be the ability to recommend a tailored anti-phishing training tool to a user who is determined to be vulnerable to phishing attack. Moreover, our efforts to investigate individual differences in phishing susceptibility are exemplified in a recent paper that describes how people from different cultures conceptualize phishing (Tembe, Hong, Murphy- Hill, Mayhorn, & Kelley, 2013). Further research will focus on refining this profiling procedure and using it to inform the design of a usable and effective tool to help users combat phishing attacks. Our plan is to develop a training tool that includes training contents reflecting the results from this study in addition to conventional training tools contents (e.g., disguised email source, poor grammar, urgency cues, etc.). Moreover, we will analyze how our anti-phishing tool contributes to protecting users from the severe
consequences of phishing attacks compared to other tools that are currently on the market. ACKNOWLEDGEMENTS This research was supported by a National Security Agency Grant to the fourth and fifth authors. REFERENCES Dhamija, R., & Tygar, J. D. (2005). The battle against phishing: Dynamic security skins. Paper presented at the ACM International Conference Proceeding Series. Eveland, W. P., Shah, D. V., & Kwak, N. (2003). Assessing causality in the cognitive mediation model: A panel study of motivations, information processing, and learning during campaign 2000. Communication Research, 30(4), 359-386. doi: 10.1177/0093650203253369 Gartner. (2007). Gartner survey shows phishing attacks escalated in 2007; more than $3 billion lost to these attacks. Retrieved from http://www.gartner.com/newsroom/id/565125 Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003). A very brief measure of the big-five personality domains. Journal of Research in personality, 37(6), 504-528. Hardee, J. B., West, R., & Mayhorn, C. B. (2006). To download or not to download: An examination of computer security decision making. interactions, 13(3), 32-37. Kelley, C. M., Hong, K. W., Mayhorn, C. B., & Murphy-Hill, E. (2012). Something smells phishy: Exploring definitions, consequences, and reactions to phishing. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 56(1), 2108-2112. doi: 10.1177/1071181312561447 Merritt, S. M., & Ilgen, D. R. (2008). Not all trust is created equal: Dispositional and history-based trust in humanautomation interactions. Human Factors: The Journal of the Human Factors and Ergonomics Society, 50(2), 194-210. Neyste, P. G., & Mayhorn, C. B. (2009). Perceptions of cybersecurity: An exploratory analysis. Proceedings of the 17th world congress of the international ergonomics association. Beijing, China. Peters, M., Laeng, B., Latham, K., Jackson, M., Zaiyouna, R., & Richardson, C. (1995). A redrawn vandenberg and kuse mental rotations test-different versions and factors that affect performance. Brain and cognition, 28(1), 39-58. Schwartz, M. J. (2011). Spear phishing attacks on the rise, InformationWeek. Retrieved from http://www.informationweek.com/security/attacks/spearphishing-attacks-on-the-rise/230500025 Sheng, S., Holbrook, M., Kumaraguru, P., Cranor, L. F., & Downs, J. (2010). Who falls for phish?: A demographic analysis of phishing susceptibility and effectiveness of interventions. Proceedings of the 28th international conference on Human factors in computing systems. Atlanta, Georgia, USA Shipley, W. C. (1986). Shipley institute of living scale. Los Angeles, CA: Western Psychological Services. Tembe, R., Hong, K. W., Murphy-Hill, E., Mayhorn, C. B., & Kelley, C. M. (2013). American and Indian Conceptualizations of Phishing. Proceedings of the 3 rd Workshop on Socio-Technical Aspects in Security and Trust. Temple, J. G., Warm, J. S., Dember, W. N., Jones, K. S., LaGrange, C. M., & Matthews, G. (2000). The effects of signal salience and caffeine on performance, workload, and stress in an abbreviated vigilance task. Human Factors: The Journal of the Human Factors and Ergonomics Society, 42(2), 183-194. doi: 10.1518/001872000779656480 Unsworth, N., Heitz, R. P., Schrock, J. C., & Engle, R. W. (2005). An automated version of the operation span task. Behavior Research Methods, 37(3), 498-505. Vandenberg, S. G., & Kuse, A. R. (1978). Mental rotations, a group test of three-dimensional spatial visualization. Perceptual and motor skills, 47(2), 599-604. doi: 10.2466/pms.1978.47.2.599 Yoshioka, N., Washizaki, H., & Maruyama, K. (2008). A survey on security patterns. Progress in Informatics, 5(5), 35-47.