There s Hope for Johnny: Comparing Automatic and Manual Encryption in

Transcription

1 SAND C There s Hope for Johnny: Comparing Automatic and Manual Encryption in t* t t Scott Ruoti, Jeff Andersen, Travis Hendershot, * t t Yung Ryn Choe, Daniel Zappala, Kent Seamons Brigham Young Universityt {ruoti, andersen, isrl.byu.edu, {zappala, cs.byu.edu * Sandia National Laboratories yrchoe@sandia.gov ABSTRACT Usable, secure remains an open problem. Recent research examined manual encryption, the practice of allowing users to view encrypted ciphertext, and found that it helped users better understand secure and avoid mistakes. In this work, we formally test whether manual encryption affects usability, users understanding of secure , and users ability to avoid mistakes. We test these hypotheses using two versions of the Private WebMail (Pwm) system, one that supports manual encryption and one that is fully automatic. Our results demonstrate that after accounting for confounding usability problems, manual encryption does not have a significant effect on usability, users understanding of secure , or users ability to avoid mistakes. Additionally, our results show that our versions of Pwm score high on the System Usability Scale (SUS), falling in the 85th to 90th percentile of a large number of systems tested with SUS. We also find that participants are eager to embrace secure webmail and begin using Pwm regularly. 1. INTRODUCTION Usable, secure is still an open problem more than 15 years after Whitten and Tygar s seminal paper, Why Johnny Can t Encrypt [16]. Recently, there is a renewed interest in secure communications (including ) motivated in part by the recent revelations of broad * Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energys National Nuclear Security Administration under contract DE-AC04-94AL85000 Copyright is held by the author/owner. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee. Symposium on Usable Privacy and Security (SOUPS) 2015, July 22-24, 2015, Ottawa, Canada. Internet surveillance. The Electronic Frontier Foundation (EFF) has published a secure messaging scorecard as the first phase of a campaign to promote the development of tools (including ) that are secure and usable for ordinary users.1 However, to the best of our knowledge, only three laboratory usability studies for secure have appeared in the research literature since Whitten and Tygar s work [8, 12, 10]. In a recent book, Garfinkel and Lipford summarize the work on secure and identify a key open problem in this area: whether automatic and transparent encryption improves the usability of encrypted or messaging systems [6]. Here, automatic encryption refers to both hiding the key management details that proved to be so problematic in the Whitten and Tygar s study, as well as hiding ciphertext from end users. The alternative approach discussed by Garfinkel and Lipford is manual encryption, which refers to showing the user ciphertext to help them understand that encryption has occurred. Two recent studies on this issue contain mixed results. A study by Fahl et al. [5] examined manual and automatic encryption for Facebook messaging and found that both manual and automatic message encryption fared equally well in terms of usability, but did not explore how they affected users understanding of what had occurred. A study by Ruoti et al. [10] compared Private WebMail (Pwm), an automatic encryption system that is tightly integrated with Gmail in the browser, with MP, a manual encryption system that operates as a stand-alone desktop application and requires users to cut-and-paste ciphertext into their . During a laboratory user study, participants using MP s manual encryption made fewer mistakes and answered more questions correctly on a quiz that tested their understanding of the system. However, the results of the Ruoti study are inconclusive because there are significant differences between Pwm and MP that introduce confounding factors, thus raising questions about the better scores seen with MP. In this work, we formally test three hypotheses regarding automatic and manual encryption: automatic and manual encryption differ in their usability, manual encryption does better than automatic encryption at helping users understand secure , and manual encryption does 1 1

2 better than automatic encrypt at helping users avoid mistakes. To test these hypotheses, we use a modified version of Ruoti et al. s Pwm system. We first fix several existing usability problems with Pwm, providing a fairer comparison between automatic and manual encryption and reducing confounding factors. We then test our hypotheses by conducting an A/B test using two identical versions of Pwm that differ only in their support for automatic and manual encryption. We evaluate the hypotheses based on usability scores derived via the System Usability Scale (SUS), a questionnaire that assesses user understanding, and quantitative measures of how often users sent sensitive information without encryption. Our results demonstrate that after accounting for confounding usability problems, manual encryption does not have a significant effect on usability, users understanding of secure , or users ability to avoid mistakes. In addition, our improved version of the Private WebMail system scores an 80.0 on the System Usability Scale (SUS), rating in the excellent category for usability and receiving an A grade. This score is in the 85th to 90th percentile of a large number of systems tested with SUS [11]. We also find that participants are eager to embrace secure webmail and most want to start using Pwm. 2. RELATED WORK Whitten and Tygar [16] conducted the first formal user study of a secure system (i.e., PGP 5). They found serious usability issues with key management and users understanding of the underlying public key cryptography. The majority of the users were unable to successfully send encrypted in the context of a hypothetical political campaign scenario. Garfinkel and Miller [8] repeated the Johnny experiment with a slightly modified scenario in the context of automatic key management. They focused on S/MIME and its ability to include the sender s public key along with the message. They implemented co-pilot, a client-side tool in Outlook Express that supports Key Continuity Management (KCM) with S/MIME. Once the user obtains the key from a sender of an message (trust on first use), the system will alert the user if they ever receive an message from that sender that is signed or encrypted with a different key. The results showed that automatic key management was more usable than the manual key management in the original Johnny experiment. However, the study revealed that the tool was a little too transparent in how well it integrated with Outlook Express. Some users failed to read the instructions associated with visual indicators. Sheng et al. [12] conducted a small pilot study by repeating the Johnny experiment to determine whether PGP 9 had made usability advances in the eight years since PGP 5 was tested. Even though improvements had been made, some users still struggled with key management. The encryption and decryption had become so transparent that users were unsure if a message they received had actually been encrypted. Ruoti et al. [10] conducted a series of user studies with Private WebMail (Pwm), a secure prototype that tightly integrates with Gmail. Even though results showed the system to be quite usable, they found that some users made mistakes and were hesitant to trust the system since the automatic encryption was too transparent. They also conducted a study comparing Pwm to a desktop application (MP) that supported manual encryption. They found that participants using MP s manual encryption made fewer mistakes and answered more questions correctly on a quiz that tested their understanding of the system. However, the results are inconclusive because in the Ruoti study there are significant differences between the two systems that introduce confounding factors, thus raising questions concerning how to interpret the results of the study. Their results provided a direct impetus for our research, as we conduct an A/B test of a modified version of Pwm to formally comparing manual and automatic encryption. Fahl et al. [5] explored manual and automatic encryption in the design of a Facebook chat system. Their user study demonstrated that users preferred automatic key management, but found no significant difference between automatic and manual encryption. They raised the issue that transparency could impact users feelings of trust and recommended the issue be addressed in future work. Garfinkel and Lipford [6] recently published a book that gives an overview of results from the usable security field, including lessons learned and challenges ahead. In their discussion of secure , they focus on the issue of automatic, transparent encryption (see pages 53-54). They give a detailed overview of the findings from Ruoti et al. [10] that raised concerns about the transparency of automatic encryption and contrast that with the favorable results for automatic encryption found by Fahl et al. [5]. The results in our paper are aimed at addressing this open problem described therein. 3. VALIDATING MANUAL ENCRYPTION The goal of our research is to determine whether manual encryption is able to increase usability, improve understanding and help users avoid mistakes when sending secure . More specifically, we are interested in determining if these benefits always exist or only when other usability issues are present, as in Ruoti et al. s former work [10]. Based on these benefits, we form three sets of hypotheses. Increased Usability H0 Automatic and Manual encryption have similar usability. H1 Automatic and Manual encryption differ in their usability. Improved Understanding H2 Automatic and Manual encryption do equally well at helping users understand secure . H3 Manual encryption does better than automatic encryption at helping users understand secure . Helped Users Avoid Mistakes H4 Automatic and Manual encryption do equally well at helping users avoid mistakes. H5 Manual encryption does better than automatic encryption at helping users avoid mistakes. 2

3 * Encrypted Sensitive Information - This is a plaintext greeting giving context to the message. This message has been encrypted using Pwm Figure 1: An encrypted in the inbox Since this research is verifying the claims made by Ruoti et al., we decided to use their Private WebMail (Pwm) system to test our hypotheses. After obtaining the source code for Pwm, we modified it and created two versions: a version that uses automatic encryption and a version that uses manual encryption. Other than these two differences, the systems were otherwise identical. This allows us to conduct an A/B test that specifically examines the benefits of manual encryption as compared to automatic encryption. The remainder of this section describes Pwm, explains the differences between the automatic and manual encryption versions of Pwm, and details other modifications we made to Pwm's interface to remove usability issues that might otherwise have been confounding factors in our study. 3.1 Private WebMail (Pwm) Pwm implements secure through tight integration with Gmail. When users read or compose secure , Pwm replaces portions of Gmail's interface with Pwm's own secure interface. Pwm's interfaces are styled differently than Gmail's, providing a clear demarcation of which information is being protected by Pwm. All secure sent using Pwm includes instructions on how to setup Pwm for first-time users (see Appendix D.) Pwm's design includes a key escrow server, so anyone can receive a secure message without having taken any prior steps, such as creating and publishing a public key. The setup instructions direct new users to the Pwm website, where they are able to add a bookmarklet to their browser's bookmark storage.2 The new user then returns to Gmail and clicks on the Pwm bookmarklet to run Pwm. The benefits of using a bookmarklet to run Pwm are that users do not need installation permission on the machine and also avoid any worries related to installing extensions [10]. The drawback is that users are required to click on the Pwm bookmarklet each time they reopen Gmail. When Pwm is running, secure messages are automatically decrypted for users. The decrypted contents of the message are displayed in place of the instructions and ciphertext that were in the message's body (see Figure 2). Optionally, the sender of the message can also include a plaintext greeting with the encrypted message. This message is shown to the user above the decrypted contents of the message. Encrypted messages in the user s inbox are marked as Encrypted (see Figure 1). By default, Pwm does not encrypt all sent by a user, but instead requires that they manually enable encryption on a per-message basis (see Figure 3). Once enabled, Pwm replaces the compose interface with its own interface (see Figure 4). It also annotates the to and subject fields to describe how Pwm handles these two fields. Finally, it adds an area where users can include a plaintext greeting that 2A bookmarklet is a browser bookmark that contains executable JavaScript. This JavaScript is run on the page that users are viewing when the bookmark is clicked. Figure 2: The read interface for an encrypted . Figure 3: Gmail compose interface before enabling encryption appears before the encrypted part of their message. Pwm s key management is handled by a key escrow server. Authentication is done using -based identity and authentication [7, 15]. Pwm is able to retrieve users and is able to complete authentication without ever prompting users for input. 3.2 Automatic and Manual Encryption The automatic encryption version of Pwm encrypts and sends as soon as the user clicks Send encrypted (see Figure 4). We created a manual encryption version that splits this operation into two distinct steps. In the first step, instead of clicking the Send encrypted button, the user instead clicks the Encrypt button. Upon doing so, the user s message is encrypted and both the instructions for decrypting the message and the encrypted ciphertext are shown to the user inside Gmail s compose interface (see Figure 5). This allows the user to confirm that encryption has actually taken place. In the second step, the user then clicks Gmail s Send button to send the encrypted Modifications to Pwm s Interface Ruoti et al. s work indicated that manual encryption improved users understanding and helped them avoid mistakes, but it also noted several issues with Pwm s 3

4 Figure 4: Pwm s compose interface Figure 5: Ciphertext shown to users after manual encryption design that might have contributed to these problems. To better isolate the benefits of manual encryption we have fixed these potentially confounding usability issues Delayed Encryption One concern expressed by a significant portion of participants in the studies by Ruoti et al. was that Pwm encrypted so quickly that they were unsure if it had really done anything. Also, participants indicated the encryption process was so transparent that they were unsure who could read their encrypted message. To address both of these problems we added an artificial delay after users click the Send encrypted or Encrypt buttons. For each recipient, users are shown a message lasting 1.5 seconds that states the is being encrypted for that user (e.g., Encrypting for bob@gmail.com ). This helps users understand who will be able to read the encrypted and also lets them feel that something substantial has happened during the encryption process. Because most messages have a small number of recipients, this artificial delay does not significantly impact the overall experience Compose Interface In order to prevent the contents of users drafts from being readable by Gmail, Pwm provides an encrypted-mode compose interface, which is overlayed on top of Gmail s own interface when encryption is enabled. To better conform to the flow of composing messages in Gmail (moving from top to bottom), we moved the button which enables encryption to the very top of Gmail s compose interface (see Figure 3).3 When encryption is enabled, we modified the placeholder text for the recipients and subject fields, explaining how Pwm uses these two fields (see Figure 4). We also added informative text to the top of the compose interface to make clear the current message s encryption status (see Figure 4 and Figure 5). Furthermore, in the automatic encryption version we modified Gmail s Send button to read Send unencrypted, to ensure that users knew that Gmail s messages are not encrypted by default. We did not make the same change for the manual encryption version as the same Send button is used to send both encrypted and unencrypted messages. Finally, we added functionality allowing users to compose plaintext greetings that are included with the encrypted (see Figure 2 and Figure 4). Although Ruoti et al. s Pwm mentioned this functionality, their interface had no place for users to compose greetings. We include the greeting as it can help engender trust in a new user that receives an encrypted , giving them confidence to setup and use Pwm. It also provides context for encrypted messages before they are decrypted Look and Feel We designed a new website for helping users install Pwm.4 This website was created using the Bootstrap Freelancer theme,5 and has a more professional look and feel than the original Pwm website. Additionally, we standardized the look and feel of Pwm to use the same colors and styles as the website, giving a consistent experience to all Pwm-based user activity Tutorials To improve the user experience, we also added tutorials to Pwm. These tutorials appear the first time a user runs Pwm, reads an encrypted , or composes an encrypted . The tutorials are intended to help users understand how secure through Pwm works. Participants have the option to skip the tutorials and can also choose to rewatch tutorials at their convenience. The tutorials provide step-by-step instructions on how to use Pwm. Each step uses simple language and instructs about a single feature of Pwm (see Figure 6). Some tutorial steps require explicit action from users before they can move on to the next tutorial step, helping users internalize correct behavior (see Figure 7). 3This button was originally on the bottom of Gmail s compose interface, next to the Send button. 4 [URL redacted at this time] 5http: //startbootstrap.com/template-overviews/ freelancer/ 4

5 Figure 6: Style of the tutorial window Figure 7: Tutorial waiting for action from the user Introduction. This tutorial is shown to users the first time they run Pwm. It informs users that Pwm will help protect their and tells them how to identify whether Pwm was running. Read. The first time users read an encrypted message, they are shown this tutorial. The tutorial shows users how to identify an encrypted , explains the plaintext greeting, and identifies which portions of the message were encrypted. Additionally, it informs them that messages encrypted with Pwm are provided authenticity and confidentiality (even from Gmail). Compose. The first time users compose an message while Pwm is running, they are given the option to watch a tutorial describing how to compose encrypted . This tutorial teaches users the correct order of operations for composing an encrypted . It also informs them about who can read their message, the purpose of the optional greeting, and where to type sensitive information. 4. METHODOLOGY We conducted an IRB-approved user study comparing automatic encryption with manual encryption. This section gives an overview of the study and describes the scenario design, user tasks, study questionnaire, survey development, and limitations. 4.1 Study Setup The study ran for two weeks, beginning Tuesday, February 17, 2015 and ending Tuesday, March 3, In total, 52 individuals participated in our study. Participants were randomly assigned to test either the automatic or manual encryption version of Pwm. Participants took a minimum of 30 minutes and a maximum of 60 minutes to complete the study. We compensated each participant $10 USD for their efforts. Studies were conducted in a room that had been set up specifically for this study. When participants first entered the room, they were read a brief introduction to the study (see Appendix A.1). For the remainder of the study, all instructions were written and provided to them, either via a printed information sheet or via . Participants completed all tasks on a virtual machine, ensuring that the computer started in the exact same state for all participants. Two study coordinators were involved in the study. One coordinator sat in the same room as the participant and sat within the participant s peripheral vision. This coordinator was instructed to avoid prompting participants and to only assist participants if they had not made any progress within five minutes. The second coordinator sat in another room and corresponded with the participant over as part of the study tasks Quality Control We removed one participant from the results of the survey. This participant s Gmail configuration was such that Pwm did not work. In the process of trying to address the problem, this participant interacted with the coordinators significantly more than any other participant. Additionally, they were the only participant to not use their own address. We chose to discard this participant s results as the two preceding effects may have biased their responses. For the remainder of this work we report results based on the 51 remaining participants Participant Demographics We recruited Gmail users for our study at a local university. Participants were evenly split between genders: male (25; 49%), female (26; 51%). Participants skewed young: years old (45; 88%), years old (3; 6%), years old (2; 4%), 55 years or older (1; 2%). We distributed posters broadly across campus to avoid biasing our results by any particular major. The poster is available in Appendix C. All participants were affiliated with the university,6 with the overwhelming majority being undergraduate students: undergraduate students (44; 86%), graduate students (2; 4%), faculty and staff (5; 10%). Participants had a variety of majors, including both technical and non-technical majors. No major was represented by more than three participants, with the vast majority only having a single participant. Participants were asked how often they logged into Gmail on the web. Most participants indicated they checked their on the web many times a day: many times a day (39; 76%), once a day (3; 6%), 2-3 times a week (2; 4%), once a week (2; 4%), 2-3 times a month (2, 4%). 4.2 Scenario Design During the course of the study, participants were given two scenarios to complete: being hired for a new job and sending credit card information to a spouse. Both of these scenarios were designed to simulate situations that participants could realistically imagine 6 We did not require this affiliation. 5

6 using secure to complete. Participants completed tasks for the first scenario before beginning the second scenario. Prior to beginning each scenario, participants were provided with a written description of the scenario (see Appendix A.2). This description included information that participants should send in place of their own personal information.7 Participants were asked to treat this sensitive information with the same care as they would treat their own information. 1. Being hired for a new job. Participants were told that they had recently returned from an interview at National Citadel, a fictitious company created for this study. As part of this scenario they had to submit receipts in order to be reimbursed for travel expenses, accept an offer of employment, and complete several hiring tasks. 2. Sending credit card information to a spouse. In this scenario participants were told that their spouse had texted them asking for login information to a credit card website. Participants were also told that they wanted to send this information encrypted over , but that their spouse had never before used secure . Participants were asked to send their spouse the requested information, taking whatever steps they felt necessary to ensure their spouse could read the encrypted Task Design Based on the two scenarios, we designed several tasks according to two goals. First, we wanted to have realistic tasks that users could envision completing as part of the scenarios. Second, we wanted tasks that used as many features as possible. Additionally, we designed the tasks without considering Pwm s feature set, as we wanted to avoid biasing the tasks to reflect favorably on Pwm. Tasks were completed entirely using , and participants used their own accounts. Participants were presented with tasks sequentially. If participants accidentally sent sensitive information without encryption they were notified of their mistake in an and asked to resend the information using encryption. The exact wording of the task s is given in Appendix A.3. Being hired for a new job. Task 1. Participants received an from National Citadel containing instructions on how to be reimbursed for expenses from their recent interview. Participants were told to send their Social Security number and a picture of their receipts. They were informed that this information must be encrypted as per National Citadel s policies. This also instructed users to set up and use Pwm to encrypt their messages.8 This task was designed to test whether participants were able to set up and use Pwm having only been provided 7We took this approach to avoid situations where sensitive information might have been leaked to Gmail. 8These instructions were generated by Pwm and are shown in Appendix D with instructions that might be reasonably expected from a company requesting information to be encrypted.9 Task 2. Participants were first asked to close their browser and then reopen Gmail. They were informed that this simulated several weeks passing after the completion of task 1. Participants were then sent an encrypted that contained an offer letter. They were asked to reply with their acceptance. They were also asked to CC their new manager. This task was designed to test whether participants would remember how to enable Pwm. It also tested whether they could use Pwm to CC a new recipient. Task 3. Participants were sent an instructing them to information to a background check company. They were instructed to encrypt the information. This task was designed to test whether users could enable encryption, either by forwarding the request for information or by composing a new message. Task 4. Participants were instructed to send bank account information to National Citadel s payroll department. They were not reminded to encrypt this information. This task was designed to test whether users would remember to encrypt information if they were not explicitly prompted to do so. Unlike the preceding tasks, if they sent information without encryption, we still considered this task complete. Sending credit card information to a spouse. Task 5. As described in the scenario, participants sent login information to their spouse using Pwm. It was left up to the user to decide how to best prepare their spouse for receiving their first Pwm message. This task was designed to see how participants would induct a new person into using secure . Regardless of what instructions were sent, we considered this task complete when the information had been sent.10 Task 6. Participants received another from their spouse asking them for additional credit card information. This request was not encrypted and did not instruct the participant to send the additional credit card information encrypted. This task was designed to test whether users would remember to encrypt information if they were not explicitly prompted to do so. Unlike most of the preceding tasks, if they sent information without encryption, we still considered this task complete. To test hypotheses H4 and H5, we measure the number of mistakes participants make while attempting these tasks. 9This task also shows that we designed the tasks around normal usage and not Pwm. Pwm does not support attachments and only allows for images to be inserted inline with the body. This task could potentially be confusing to users as they cannot use their normal work-flow for attaching an image. 10Pwm includes instructions by default, and since they were sufficient to help the participant start using Pwm, it was reasonable to assume that the participant believed they would be sufficient to help their spouse start using Pwm. 6

7 4.4 Study Questionnaire We administered our study using the Qualtrics web-based survey software. Before beginning the survey, participants were read an introduction by the study coordinator and asked to answer a set of demographic questions. Participants then completed the six study tasks, following which they were asked to complete a questionnaire regarding their experience. The full text of this questionnaire can be found in Appendix B.2. To test hypotheses H0 and H1, regarding the usability of each version of Pwm, we had participants complete the ten System Usability Scale (SUS) questions [3, 4]. Answers to these questions are used to derive each version s SUS score, a single numeric score from 0, the least usable, to 100, the most usable, that provides a rough estimate of the version s overall usability. Recent research has shown that SUS scores are effective for comparing systems across different study populations and is the usability statistic used in Ruoti et al. s original work [10]. Moreover, Tullis and Stetson compared SUS to four other usability metrics (three standard metrics from the usability literature and their own proprietary measure) and determined that SUS gives the most reliable results [14]. After completing the SUS questions, participants were asked several questions designed to ascertain whether they would want to use Pwm in their day-to-day lives. These included questions on whether they wanted to have all or only some of their encrypted, whether they wanted to begin using Pwm, and whether they would feel comfortable using Pwm with their family and friends. Participants were also asked to describe what they liked about Pwm, what they would change about Pwm, and how Pwm could be more applicable to their own lives. To test hypotheses H2 and H3, we ask participants questions to examine how well they understood the cryptographic properties of Pwm. To test understanding of confidentiality, they were asked to indicate which parties could read a message encrypted with Pwm. Similarly, participants were asked whether Pwm provided authenticity and integrity for secure messages composed with Pwm. Each question was asked in language that would be approachable to users and did not require a technical background. For each property, participants were given the option to indicate that they were unsure whether that property was provided by Pwm. The survey also asked participants how they preferred to send sensitive information. They indicated whether they had ever used secure before, and if so for what and why. Participants were asked whether they would be more likely to use to send sensitive information if secure was available to them. Then they were asked whether they would be willing to pay for secure , and if so, how much. Finally, we asked participants how they would prefer to have secure implemented. The options were, Integrated tightly into a web-based system. Integrated tightly into a desktop/mobile application. A desktop/mobile application that has a separate interface for encrypting messages. A browser extension that has a separate interface for encrypting messages. A web page that has a separate interface for encrypting messages. A provider that only supports encrypted . Participants were asked to rank these options according to their preferences, with the options initially being ordered randomly. Participants were also asked why they preferred their top-ranked preference. 4.5 Survey Development We began the design of our study before completing the implementation of Pwm. As mentioned earlier, we designed the study to test secure first and Pwm second. This study development approach allows us to feel more confident that participants were interacting with Pwm in a realistic fashion. To help ensure that this goal was met, we included researchers who are not involved in our secure research to help design the scenarios and tasks. After creating scenarios and tasks that were acceptable to all parties, we then conducted a pilot study with three participants. We did not identify any significant flaws during the pilot study. 4.6 Limitations While our studies included students with a diverse set of majors and technical expertise, it would be beneficial for future studies to verify our results using non-student populations. Gmail users may also not be representative of the general population s preferences regarding secure , and further studies should be conducted with users of other systems. Our study is short-term and is not necessarily representative of how participants would use secure over a longer period of time. Once our implementation of Pwm has been proven to be sufficiently usable and secure in short-term studies, it is important that it then be analyzed in a more comprehensive long-term study. This could potentially reveal additional information about the advantages and disadvantages of manual encryption. Our study is a lab study and has limitations common to all studies run in a trusted environment [9, 13]. While there are indications that some participants treated the provided sensitive information as they would their own (e.g., refusing to the provided social security number), there is still no guarantee that participants reactions mimic their real life behaviors. Additionally, our studies did not test participants ability to resist attacks. 5. RESULTS In this section we report the quantitative results from our user study. First, we report data related to our hypotheses. Second, we report other quantitative data related to participants willingness to adopt secure , preferences for how secure is implemented, and prior experience using secure Usability We evaluated Pwm using the System Usability Scale (SUS). The automatic encryption version of Pwm had a SUS score of 79.1 and the manual encryption version had a SUS score of This difference was not statistically significant (two tailed student t-test, unequal variance p = 0.43). Further breakdown of the SUS scores, along 7

8 Percentiles j j 65 j j 85 j F D c B A A+ OK Good Excellent Best Not acceptable Marginal Low M.Hiqh Acceptable SUS Score Figure 8: Adjective-based ratings to help interpret SUS scores Confidentiality Authenticity Integrity Automatic % 4% 11% 63% 4% 33% 81% 0% 19% Manual % 4% 13% 63% 0% 42% 71% 4% 25% Overall % 4% 12% 63% 2% 37% 76% 2% 22% Table 2: Participants understanding of Pwm s confidentiality, authenticity, and integrity properties Count Mean Standard Deviation Min O Median CO a Max Automatic Manual Overall Ruoti et al Table 1: SUS scores with SUS scores from Ruoti et al. s earlier study, can be found in Table 1. Based on these results, we fail to reject the null hypothesis H0 for usability. To give greater context we leverage the work of several researchers. Bangor et al. [2] analyzed 2,324 SUS surveys, and derived a set of acceptability ranges that describe whether a system with a given score is acceptable to users in terms of usability. Bangor et al. also associated specific SUS scores with adjective descriptions of the system s usability. Using this data, we generated ranges for these adjective ratings, such that a score is correlated with the adjective it is closest to in terms of standard deviations. Sauro et al. [11] also analyzed SUS scores from Bangor et al. [1], Tullis et al. [14], and their own data. They calculate the percentile values for SUS scores and assign letter grades based on percentile ranges. The above contextual clues are presented in Figure 8. Using this context, Pwm s SUS score of 80.0 is rated as having excellent usability and given an A grade. It falls between the 85th and 90th percentile. We used Ruoti et al. s SUS results from their earlier studies and calculated that their implementation of Pwm had an average SUS score of This score is rated as having good usability, given a B grade, and falls between the 70th and 80th percentile. The difference between our implementation and that of Ruoti et al. is statistically significant (two tailed student t-test, unequal variance p < 0.01). 5.2 Understanding We asked three questions to determine whether each participant understood the confidentiality, authenticity, and integrity properties provided by Pwm. The responses indicated whether each participant correctly understood the principle, had some misunderstanding, or was unsure. Table 2 summarizes these results. In all three cases, the difference between manual and automatic encryption was not statistically significant: Confidentiality X2[2, N = 51] = 0.03, p = 0.98 Authenticity X2[2, N = 51] = 1.10,p = 0.58 Integrity X2[2, N = 51] = 1.56, p = 0.46 Based on these results, we fail to reject the null hypothesis H2 for understanding. 5.3 Avoiding Mistakes During the study, we recorded all instances of a participant taking an action that deviated from the study parameters. Using this data, we identified several instances where a participant s actions leaked sensitive information. These results are reported by task in Table 3. Tasks two, five, and six had no mistakes, and the differences in the remaining three tasks were not statistically significant. Task 1 X2[1,N = 51] = 2.85, p = 0.09 Task 2 X2[1,N = 51] = 0.00, p =1.00 Task 3 X2[1,N = 51] = 0.003,p =

9 5 O i (M OO ^ lo ^ ^ ^ ^ ^ ^ CO CO CO CO CO CO &&&&&& H H H H H H Automatic Manual Overall Table 3: Number of participants who sent sensitive information without encryption Of the four mistakes in Task 1, only one was related to manual encryption: the participant had encrypted their message correctly, but after encryption used Gmail s compose interface to modify the encrypted and add the sensitive information. In the second mistake, the participant had also encrypted their message correctly, but then reloaded the page, which deactivated Pwm, and used Gmail s compose interface to modify the encrypted draft and add the sensitive information. The remaining two mistakes were from participants who never attempted to install Pwm and sent the sensitive information using Gmail. Based on these results, we fail to reject the null hypothesis H4 for avoiding mistakes. 5.4 Other results Acceptability of Pwm Participants were asked for their opinions about the need for secure and also asked if they would be willing to use Pwm in their day-to-day lives. These responses are summarized in Figure 9. Participants are unanimous (51; 100%) in wanting the ability to be encrypt sensitive , but only a quarter of participants (12; 24%) want all encrypted. More than half of the participants (27; 53%) disagree with encrypting all . Over four-fifths of participants (42; 83%) want to start using Pwm. More than nine out of ten people (47; 92%) agree that their friends and family could easily start using Pwm, and three-fourths (37; 73%) indicate they would use Pwm to send encrypted to their friends and family. The majority of participants agree that Pwm protects their (35; 69%). The rest are not sure (16; 31%). No one disagreed with the statement that Pwm protects their Implementation Preferences Pwm is an implementation of secure that tightly integrates with existing web apps. We asked participants to rank other potential methods for implementing secure . Their reported preferences are given in Figure 10. Nearly all participants (44; 87%) prefer a system that is tightly integrated with their existing webmail solution: most preferred (28; 55%) or second most preferred (16; 31%). The next most popular option is having secure implemented into an existing desktop or mobile application: most preferred (10; 20%) or second most preferred (17; 33%). The remaining implementation methods had the majority of participants rank them as their fourth to sixth choice Sending Sensitive Information Half of the participants (25; 49%) indicated that they have sent sensitive information over . Common types of information include bank account information, social security numbers, passwords, and job specific information. Of the participants who have sent sensitive information over , most indicated doing it infrequently: less than once a month (19, 76%), once a month (4; 16%), 2-3 times a week (1; 4%), many times a week (1; 4%). Participants were asked to indicate which forms of communication they preferred to use to send sensitive information. Most participants indicated they preferred to share sensitive information over the phone or in person: over the phone (37; 73%), in person (36; 71%), using (10; 20%), in a cellular text message (6; 12%), using a secure website (2, 4%), using a fax (1, 2%). Four-fifths of participants indicated that they would be willing to use encrypted to send sensitive data: willing (40; 79%), depends on who and what (8; 16%), unwilling (3; 6%) Willingness to Pay Participants were undecided about whether they were willing to pay for encrypted willing(1; 2%), it depends (35; 69%), unsure (4; 8%), unwilling (11; 22%). The participants that were unsure mentioned several factors that would determine if they were willing to pay for secure the cost, whether the cost was one-time or recurring, and whether secure was required for their job. Excluding the participants that were unwilling to pay for encryption, the remaining participants (40; 79%) were willing to pay on average $5 USD per month (median $5 USD, mean $6.105 USD, mode $2 and $5 USD) Prior Use of Secure Three participants (3; 6%) had previously used secure . One had used PGP, another had used Lotus Notes built-in encryption, and the third had used a secure prototype but wasn t sure what it was called. The participant who had used PGP described his experience, saying, Had to generate a key pair, configure the plugin, then distribute the public key (and no one I used it with could figure out how to unencrypt anyway, so I abandoned it). 6. DISCUSSION In this section we begin with a discussion of automatic and manual encryption. We follow this by discussing participant experiences, opinions, and preferences regarding secure . Finally, we detail lessons learned about our implementation of Pwm. In this section, participants are referred to as R1 - R52, with the number corresponding to the order in which they participated in our study. 6.1 Automatic and Manual Encryption For each set of hypotheses, we failed to reject the null hypothesis. This indicates that by adding additional features to automatic encryption (e.g., tutorials, delayed 9

10 I want to be able to encrypt all of my . I want to be able to encrypt sensitive . I want to start using Pwm. My friends and family could easily start using Pwm. I would use Pwm with my friends and family. Pwm protects my . Strongly Agree ^ Agree (Disagree Neither Agree nor Disagree Strongly Disagree Figure 9: Participants opinions regarding Pwm (number of participants) encryption, improved task flow) we are able to replicate the benefits of manual encryption. Additionally, several participants who used the manual encryption were surprised that their messages were not automatically sent. R8 and R29 stated, respectively, [...] I had to click to encrypt it and then click to send. I liked the thought of it being one action. The one thing I would change is the interface of the after you click encrypt. The first time I encrypted a message, it seemed to disappear. I wasn't sure if it had been sent until I realized that I had to click the send button. I think you could improve this if you made a big blue or green bar in the that said, This contains your encrypted message and then if you clicked on it you could see what you wrote. Still, this is not an indictment of manual encryption. Our results demonstrate that manual encryption does not negatively impact usability, user understanding, or mistake avoidance. While other features were able to provide the same benefits as manual encryption, we still believe that manual encryption is a valid tool that system designers could choose Understanding Our results demonstrated that most participants understood what cryptographic features were provided by Pwm. Still, there was a significant number of participants that indicated they were unsure about the protections provided by Pwm. This indicates that Pwm could do more to instruct and aid users in their understanding of how their is protected. Additionally, care should be taken in evaluating our results on understanding. The questions on authenticity and integrity asked users whether Pwm provided these cryptographic properties, but did not verify this understanding with additional questions. It is possible that participants answered yes to these questions in response to the study s trusted environment [13]. For example, R26 indicated that Pwm would warn him if the message had been modified in transmission, which, while a true statement, was not something experienced by any participant or mentioned in the tutorials: Yes, but Pwm will notify you if it has been motified [sic]. Still, there is evidence that at least some of the participants truly understood these properties. Participant R20 stated the following about integrity: Define modified,... it probably can t be changed in a meaningful way, but could be garbled or corrupted (just by altering the encrypted bits) 6.2 Acceptability of Pwm Participants seemed to be interested in encrypting their . While over half rejected the idea of having all of their encrypted, every single participant was interested in encrypting their sensitive . Additionally, all but nine participants wanted to start using Pwm. Furthermore, participants overwhelmingly believed Pwm could be used with their friends and family. Examples of participants positive experiences with Pwm were consistently attested to in their responses. For example, R26, R39, and R42 expressed, respectively, Pwm was very simple, but definitely carried a [sic] aura of confidence in actually protecting your information. Now, how well it does is something that I cannot determine due to my lack of knowledge, but it was very well put together. It was very concise and user friendly and did not require esoteric knowledge to operate. I would definitely feel more comfortable sending sensitive information over if I were using Pwm versus just sending it via an provider. I liked how I could encrypt sensitive information like bank account information, 10

11 Integrated tightly into a web-based system. Integrated tightly into a desktop/mobile application. A browser extension that has a separate interface for encrypting messages. A provider that only supports encrypted . A web page that has a separate interface for encrypting messages. A desktop/mobile application that has a separate interface for encrypting messages. 60 1st 2nd 3rd 4th 5th 6th Figure 10: Participants ranking of secure implementation alternatives (number of participants) credit card, and other things. It wasn t that hard to use. I didn t have to download anything; all I had to do was just save a bookmark and then click on it. It was really easy to use. I liked how it made encrypting important information so easy. The tutorial was fast and easy. I like that it is easy and convenient to use with day to day s. I liked that the background was blue so I knew when it was encrypting. Based on these responses and our implementation s high SUS score, the highest SUS score for secure systems in the literature, we believe that our implementation of Pwm is usable enough to potentially see widespread adoption. Although significant changes still need to be made to Pwm (e.g., work with more providers, bug fixes) and a long-term study needs to be conducted to verify our results, it is encouraging to see that users are ready and willing to embrace encrypted through Pwm. 6.3 Implementation Preferences Our results indicate that participants are most interested in secure systems that integrate with existing systems (see Figure 10). There may exist some bias created by asking this question after allowing participants to use a tightly integrated system. Nevertheless, participants free response answers to why they selected their top preference give an indication that these are their true preferences. Most participants cited ease-of-use and the ability to continue to use existing systems and applications as their prime reasons for preferring tight integration. For example, participants R3 and R26 stated, respectively, I want to be able to choose the mail client I want and still have the protection I need I think that an encryption system is more likely to be applied and enjoyed if a user, such as myself, does not have to access another completely separate application every time I want to send or read an encrypted . This would become very tedious and time consuming, and I think many people might just give up or find another option. The way it was employed in the scenarios was pretty good, and I think integrating it into your actual, individual account on a web system like would make that even better! Although only asked why they preferred their top option, a third of participants (18; 35%) explicitly stated that they did not like the idea of switching between multiple applications. R30 and R35 shared, respectively, Having a separate app/web page/browser extension would be a hassle. I would prefer the tightly integrated approach like what we did in the scenarios. I chose my first option for simplicity. It was easier to have the encryption program integrated into the I was already using. Having to copy and paste through a different interface would be more challenging, especially when initially trying to learn the program. More research should be done to verify whether these preferences hold true across a larger population. If that is the case, then more research emphasis should be placed on solutions that tightly integrate with existing systems. 6.4 Mixed mode The optional plaintext greeting that can accompany encrypted by Pwm is intended to help senders give confidence to recipients who have never used Pwm that the is authentic and not spam. Surprisingly, during our study we noted that a small, but significant number of users would include a plaintext greeting in many 11

12 Gmail's Icon Pwm's Icon Figure 11: Gmail s and Pwm s icons for attaching images. of their encrypted messages. For example, in Task 4 eight participants (8; 16%) including a greeting stating that they were sending their direct deposit information. Interestingly, this was actually a feature that seven participants (7; 14%) listed as one of their favorite features of Pwm. R12 and R51 stated, respectively, It wasn t rigid, I could write part of a message and have the other parts encrypted if I wanted. It was very clear what was encrypted and what wasn t [...] That is lets me chose when to encrypt and, when not to. It s also nice to have the option of writing a message before the encrypted part so others know it s not spam. 6.5 Lessons Learned The following are lessons we learned about our implementation of Pwm during the study Instructing New Users While Pwm automatically includes instructions on how to setup and use Pwm, most participants were unaware of this. During Task 5, participants would often spend several minutes trying to open an old Pwm message to grab the instructions, just to have the message immediately decrypted and the instructions disappear. It would be helpful to make it more clear that these instructions are always included. Perhaps we could even add functionality that allows users to explicitly add these instructions Look and Feel Quite a few participants indicated enjoying the look and feel of Pwm. They also indicated that having a color-scheme that was distinct from Gmail helped them more easily understand what information was encrypted and what information was in the clear. This intuitive understanding potentially helped participants avoid mistakenly entering sensitive information where it would not be protected. Additionally, participants mentioned that it helped them feel more confident in the system. Thus it is clear that when designing tightly integrated systems, it is important to have a distinct look and feel Icon Selection One potential area of concern when integrating tightly with existing webmail systems is that participants may become confused if the secure system s interface does not match the underlying webmail system s interface. In Task 1, participants were asked to attach an image to their message. Pwm s button that exposes this functionality differs from Gmail (see Figure 11), and we noticed that participants became confused when they were unable to find Gmail s attachment icon. Many participants spent several minutes trying to find the button that would allow them to attach an image, often completely ignoring the button that would actually allow them to do so. The failed search behavior exhibited here was surprisingly uniform across participants. When asked about this, R17 indicated, I think like in my normal Gmail interface there s just like an attach button, and I find it at the bottom, and so it kind of threw me off, cause that s usually how I m used to attaching them, so I almost had to like start over and try it again and then I figured it out. I dunno, I think it was just that I was trying to do it weird. This issue illustrates the need to match visual cues when presenting users with an alternative interface. Explanatory text is not always sufficient Tutorials We recorded the number of participants that completed each of the tutorials. The video recording for one participant s session was corrupted, and we were unable to determine if they had completed the tutorials. The results of the remaining 50 participants demonstrate that participants were willing to watch tutorials: watched introductory tutorial (46; 92%), watched tutorial on reading secure (46; 92%), watched the tutorial on composing secure (27; 54%). These result surprised us, as we expected most users to ignore the tutorials. In their user studies, Ruoti et al. reported that nearly all participants ignored a tutorial video that was displayed on their Pwm website next to Pwm s setup instructions [10]. This result indicates that participants are willing to watch tutorials, though two criteria seem to be crucial: they appear in-page as participants need them and they contain simple and direct wording CONCLUSION We examined the open question on how manual encryption affects the usability of secure . Based on this question, we formulated three hypotheses that examined how automatic and manual encryption differed in terms of usability, users understanding of secure , and users ability to avoid mistakes, respectively. We then tested our hypotheses by conducting an A/B test using two versions of the Private WebMail (Pwm) system [10] that differed only in their support for automatic and manual encryption. Our results demonstrate that after accounting for confounding usability problems, manual encryption does not have a significant effect on usability, users understanding of secure , or users ability to avoid mistakes. In addition, our improved version of Pwm scores an 80.0 on the System Usability Scale (SUS), rating in the 11We believe less participants watched the compose tutorial, which appeared the first time participants clicked on GMail s Compose button, because it was not clear they needed to see it at this point and because the tutorial drew attention to the option that they could skip it. Better tutorial design could possibly address this issue. 12

13 excellent category for usability and receiving an A grade. This score is in the 85th to 90th percentile of a large number of systems tested with SUS [11]. All participants in our study indicated that they were interested in encrypting sensitive . Four-fifths of participants expressed a desire to begin using Pwm. Moreover, nearly all participants believed that their friends and family could easily start using Pwm. Although it is necessary to verify these results using a long-term user study, participants responses still give hope that we are significantly closer to the goal of providing ordinary users with a usable system to exchange secure s. 8. REFERENCES [1] A. Bangor, P. Kortum, and J. Miller. An empirical evaluation of the System Usability Scale. International Journal of Human-Computer Interaction, 24(6): , [2] A. Bangor, P. Kortum, and J. Miller. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4(3): , [3] J. Brooke. SUS a quick and dirty usability scale. In Usability Evaluation in Industry. CRC Press, [4] J. Brooke. SUS: A retrospective. Journal of Usability Studies, 8(2):29-40, [5] S. Fahl, M. Harbach, T. Muders, M. Smith, and U. Sander. Helping Johnny 2.0 to encrypt his Facebook conversations. In Proceedings of the Eighth Symposium on Usable Privacy and Security, page 11. ACM, [6] S. Garfinkel and H. R. Lipford. Usable security: History, themes, and challenges. Synthesis Lectures on Information Security, Privacy, and Trust, 5(2):1-124, [7] S. L. Garfinkel. -based identification and authentication: An alternative to PKI? IEEE Security & Privacy, 1(6):20-26, [8] S. L. Garfinkel and R. C. Miller. Johnny 2: a user test of key continuity management with S/MIME and Outlook Express. In Proceedings of the First Symposium on Usable Privacy and Security, pages ACM, [9] S. Milgram and E. Van den Haag. Obedience to authority, [10] S. Ruoti, N. Kim, B. Burgon, T. Van Der Horst, and K. Seamons. Confused Johnny: when automatic encryption leads to confusion and mistakes. In Proceedings of the Ninth Symposium on Usable Privacy and Security, page 5. ACM, [11] J. Sauro. A practical guide to the system usability scale: Background, benchmarks & best practices. Measuring Usability LLC, [12] S. Sheng, L. Broderick, C. Koranda, and J. Hyland. Why Johnny still can t encrypt: evaluating the usability of encryption software. In Proceedings of the Second Symposium On Usable Privacy and, Security - Poster Session, [13] A. Sotirakopoulos, K. Hawkey, and K. Beznosov. I did it because i trusted you : Challenges with the study environment biasing participant behaviours. In SOUPS Usable Security Experiment Reports (USER) Workshop, [14] T. S. Tullis and J. N. Stetson. A comparison of questionnaires for assessing website usability. In Usability Professional Association Conference, pages 1-12, [15] T. W. Van Der Horst and K. E. Seamons. Simple authentication for the web. In Third International Conference on Security and Privacy in Communications Networks (SecureComm), pages IEEE, [16] A. Whitten and J. D. Tygar. Why Johnny can t encrypt: A usability evaluation of PGP 5.0. In 8th USENIX Security Symposium,

14 APPENDIX A. PWM USER STUDY A.1 Introduction At the beginning of each user study the following was read to the participant by the study coordinator. Welcome to our Gmail study. I am the study coordinator and am here to assist you as needed. In this study, you will be using Gmail to complete several scenarios. In each scenario you will play the role of another person. I will provide you with information about this person. During the scenarios, please use this provided information and not your own personal information. Please protect any sensitive information for this person just as if it was your own. During the course of the study we will record what is happening on your screen. This video will not be seen by anyone besides the researchers and will be destroyed once our research is complete. We will not collect any personally identifying information. Any data, besides the screen recording and answers to the study survey, will be deleted automatically upon your completion of the study. You will receive $10.00 as compensation for your participation in this study. The expected time commitment is approximately 30 minutes. If you have any questions or concerns, feel free to ask me. You can end participation in this survey at any time and we will delete all data collected at your request. You may now proceed with the survey on the left-most computer. I will remain in the room to observe the study and also to answer any questions you may have. A.2 Scenarios These were the two scenarios used in the study. The first scenario covers Task 1 - Task 4 and the second scenario covers Task 5 and Task 6. A.2.1 Scenario 1 In this scenario, you have applied for a job with National Citadel. Last weekend they flew you out for a final interview. They told you they would you instructions for getting your expenses reimbursed. All information you will need to complete this scenario is provided below. During this scenario you will receive and respond to several s. Until you receive an with a confirmation code, please continue to check your inbox for new messages. Be aware that sometimes there will be a slight delay (30-60 seconds) before you receive an . At this time, please log into your account where you should see an message from National Citadel. If you don t see such a message, ask the study coordinator for help. Scenario 1 Persona 1 Social Security Number State of residence 1 [redacted] 1 1 Date of birth February 20, Bank account number Bank routing number A.2.2 Scenario 2 In this scenario, you have received a text message from your spouse (spouse [redacted]) asking for help logging in to your credit card website. Your spouse has asked you to him/her the account username and password. This information is sensitive, so you want to encrypt it. You know that your spouse has never used encrypted before. Please do whatever you think you would do in real life to send them this information encrypted with Pwm. All information you will need to complete this scenario is provided below. During this scenario you will receive and respond to several s. Until you receive an with a confirmation code, please continue to check your inbox for new messages. Be aware that sometimes there will be a slight delay (30-60 seconds) before you receive an . Scenario 2 Persona Account username family343 Account password b@nkp@ssword Credit card number Credit card CCV 992 A.3 Task s A.3.1 Task 1 Initial . Encrypted: Yes From: finances@nationalcitadel.com Subject: Receipt Reimbursement Greeting: Hi [participant s name], thank you for interviewing with National Citadel this week. In order to process your expense reimbursement, please reply to this with your Social Security Number and a picture of your receipts for your purchases. Company policy requires that you send us this information encrypted. We use Pwm to encrypt . This includes directions for setting up Pwm. After setting up Pwm, you will be able to encrypt the required information. Thanks, -Jen Cobb Encrypted Body: Now that you are running Pwm, you can simply reply to this message, and your information will automatically be encrypted. If the participant encrypts the requested data. Encrypted: No From: study@[redacted] Subject: Task Complete Body: Hi [participant s name], congratulations on completing your first task. Please continue to watch your inbox for more communications from National Citadel. -ISRL Research If the participants does not encrypt the requested data. Encrypted: No From: finances@nationalcitadel.com Subject: Re: Receipt Reimbursement Body: Hi [participant s name], it looks like you didn t send those details securely. Please resend with encryption enabled. Thanks, -Jen Cobb 14

15 A.3.2 Task 2 Participants instructed to close browser and then reopen Gmail. Encrypted: No From: study@[redacted] Subject: Scenario Instructions Body: Before we proceed with the scenario, please close Chrome. This simulates time passing after your interview. After Chrome is shut down, you may re-open Chrome and navigate to Gmail. Thanks, -ISRL Research Initial . Encrypted: Yes From: hiring@nationalcitadel.com Subject: National Citadel Offer Greeting: This encrypted contains details about your employment offer with National Citadel. Please decrypt to view. Encrypted Body: Congratulations [participant s name]! We were very impressed with your performance in the interviews and are excited to extend a full-time offer. See below for the details. Please look over the offer and, if it is acceptable, reply to us so we can proceed with the hiring. Also, please CC your acceptance to your manager, Kaylee Clark, at kclark@nationalcitadel.com. We look forward to your reply. Regards, -Rory Tam ^J^National Citadel [TODAY'S DATE] Dear [PARTICIPANTS NAME], On behalf of National Citadel (the "Company"), I am very pleased to offer you the position of Project Manager. This letter clarifies and confirms the terms of your employment with the Company Start Date and Salary Unless we mutually agree otherwise in writing you will commence employment on June 1, 2015 ("Start Date"). Your salary will be $6, per month payable in accordance with the Company's standard payroll practice and subject to applicable withholding taxes. Background Check This offer is contingent on the successful completion of a background check. This offer and all terms of employment stated in this letter will expire March 24, JOHN, we are very excited about the possibility of you joining us. I hope that you will accept this offer and look forward to a productive and mutually beneficial working relationship Please let me know if I can answer any questions for you about any of the matters outlined in this letter Sincerely -Zoe Walker Recmiting Manager If the participant encrypts the requested data. Encrypted: Yes From: hiring@nationalcitadel.com Subject: Re: National Citadel Offer Encrypted Body: Hi [participant s name], we received your acceptance and are excited to begin the hiring process. As part of the onboarding procedure, you will be receiving s from our Background Check and Payroll departments. Please reply to these s with the requested information, so we can proceed with your hire. Congratulations once again on joining our team. Regards, -Rory Tam If the participants does not CC Kaylee Clark. Encrypted: Yes From: hiring@nationalcitadel.com Subject: Re: National Citadel Offer Encrypted Body: Hi [participant s name], it looks like you didn t CC your acceptance to your manager, Kaylee Clark, at kclark@nationalcitadel.com. Please resend your acceptance and be sure to CC her. Thanks, -Rory Tam If the participants does not encrypt the requested data. Encrypted: Yes From: hiring@nationalcitadel.com Subject: Re: National Citadel Offer Greeting: There was a problem with your reply. Please decrypt this message to view. Encrypted Body: Hi [participant s name], it looks like you didn t encrypt your response. Please resend your acceptance and be sure to encrypt it. Thanks, -Rory Tam If the participants does not CC Kaylee Clark and does not encrypt the requested data. Encrypted: Yes From: hiring@nationalcitadel.com Subject: Re: National Citadel Offer Greeting: There was a problem with your reply. Please decrypt this message to view. Encrypted Body: Hi [participant s name], it looks like you didn t CC your acceptance to your manager, Kaylee Clark, at kclark@nationalcitadel.com. Also, you seem to have sent your reply insecurely. Please resend your acceptance; be sure to CC her, and be sure to use encryption. Thanks, -Rory Tam A.3.3 Task 3 Initial . Encrypted: No From: hiring@nationalcitadel.com Subject: National Citadel Background Check Body: Hello [participant s name], we have received your hiring details and are proceeding with a background check. Please fill out the following details and forward them to our Background Check provider, backgroundcheck@[redacted]. As a reminder, please encrypt all sensitive communications with other entities regarding your employment with National Citadel. - Full name - Date of birth - Social Security Number - State of residence Thanks, -Simon Turner 15

16 If the participant encrypts the requested data. Encrypted: Yes From: Subject: Re: Fwd: National Citadel Background Check Encrypted Body: Hi [participant s name], thanks for forwarding your background check details. We ll get started right away. National Citadel will inform you when your check is complete. Regards, -Inara Sanchez If the participants does not encrypt the requested data. Encrypted: Yes From: backgroundcheck@[redacted] Subject: Re: Fwd: National Citadel Background Check Greeting: Hi [participant s name], it looks like you sent us these background check details insecurely. Please re-forward with encryption enabled. Thanks, -Inara Sanchez If the participants replies instead of sending the data to the background check company. Encrypted: No From: hiring@nationalcitadel.com Subject: Re: National Citadel Background Check Body: Hi [participant s name], it looks like you replied to us instead of forwarding your details to our background check provider. Please forward to backgroundcheck@[redacted]. Thanks, -Inara Sanchez A.3.4 Task 4 A.3.6 Task 6 Initial . Encrypted: No From: spouse@[redacted] Subject: Re: Credit card details Body: Hi, thanks for the login details. Now it s saying I need the last four digits of the credit card, and the CCV number. Can you send those to me? Thanks Upon participants sending the requested data. Encrypted: No From: study@[redacted] Subject: Re: Fwd: Task Complete Body: Hi [participant s name], you have completed this scenario. Please return to Qualtrics and enter your confirmation code. Confirmation code: 8472 Thanks, -ISRL Research Initial . Encrypted: No From: payroll@nationalcitadel.com Subject: National Citadel Direct Deposit Information Body: Hi [participant s name], we are proceeding with your hire and need your direct deposit information. Please reply to this with your bank account number and routing number so that we can ensure your paychecks are properly deposited. Thanks, -Matthew Reynolds Upon participants sending the requested data. Encrypted: No From: study@[redacted] Subject: Re: Fwd: Task Complete Body: Hi [participant s name], you have completed this scenario. Please return to Qualtrics and enter your confirmation code. Confirmation code: 1071 Thanks, -ISRL Research A.3.5 Task 5 If the participant encrypts the requested data. Continue to Task 6. If the participants does not encrypt the requested data. Encrypted: No From: study@[redacted] Subject: Task Incomplete Body: Hi [participant s name], please try again to send the credit card details securely. It looks like the message you just sent was unencrypted. Thanks, -ISRL Research 16

17 B. QUESTIONNAIRES B.1 Demographic Questionnaire What is your gender? Male, Female What is your age? years old years old years old years old 55 years or older What is the highest degree or level of school you have completed? Some school, no high school diploma High school graduate, diploma or the equivalent (for example: GED) Some college or university credit, no degree College or university degree Graduate or professional degree What is your occupation or major? How often do you use Gmail through your browser? Less than, Once a Month Once a Month 2-3 Times a Month Once a Week 2-3 Times a Week Once a Day Many Times a Day B.2 Study Questionnaire You have completed all of the scenarios. You will now be asked several questions concerning your experience with Pwm. Please answer the following question about Pwm. Try to give your immediate reaction to each statement without pausing to think for a long time. Mark the middle column if you don t have a response to a particular statement. Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree I think that I would like to use this system frequently. I found the system unnecessarily complex. I thought the system was easy to use. I think that I would need the support of a technical person to be able to use this system. I found the various functions in this system were well integrated. I thought there was too much inconsistency in this system. I would imagine that most people would learn to use this system very quickly. I found the system very cumbersome to use. I felt very confident using the system. I needed to learn a lot of things before I could get going with this system. Please answer the following questions about Pwm. Try to give your immediate reaction to each statement without pausing to think for a long time. Mark the middle column if you don t have a response to a particular statement. Strongly Disagree, Disagree, Neither Agree nor Disagree, Agree, Strongly Agree I want to be able to encrypt all of my . I want to be able to encrypt sensitive . I want to start using Pwm. I would use Pwm with my friends and family. My friends and family could easily start using Pwm. Pwm protects my . What did you like most about using Pwm? What would you change about Pwm? Is there anything else you think that Pwm could do to be more useful for you personally? If I encrypt an using Pwm, who is able to read it? Multiple selections allowed Me The recipients Gmail Hackers who steal the during transmission Hackers who break into my account Government Agencies, for example the NSA Unsure Can an encrypted with Pwm have a fake from address? Yes, No. It depends, Unsure Can an encrypted with Pwm be modified by a third party (for example, Google or a hacker)? Yes, No. It depends, Unsure Do you ever send sensitive information through ? Yes, No Only seen if selected "Yes" to sending sensitive information through . What type of sensitive information do you send through ? Only seen if selected "Yes" to sending sensitive information through . How often do you send sensitive information through ? Less than Once a Month Once a Month 2-3 Times a Month Once a Week 2-3 Times a Week Once a Day Many Times a Day 17

18 What method do you prefer to use to send sensitive information? Multiple selections allowed , Phone call, Text message, Fax, In person, Other Prior to this study, had you ever encrypted your before? Yes, No Only seen if selected "Yes" to having previously used encryption. What software have you used to encrypt your ? Only seen if selected "Yes" to having previously used encryption. Why did you need to encrypt your ? If easy-to-use encryption was available, would you be more likely to send sensitive information over ? Yes, No, It depends, Unsure There are many ways to implement encrypted . One approach is to tightly integrate with existing systems. For example, Pwm appears in the same page as Gmail. Another approach is to create a separate interface where you encrypt messages, and then copy these encrypted s into your client.the following are a list of possible approaches to encrypted . Please rank them according to which you would prefer to use. (Higher is better, drag and drop to arrange). Initial ordering of options randomized. Integrated tightly into a web-based system. Integrated tightly into a desktop/mobile application. A desktop/mobile application that has a separate interface for encrypting messages. A browser extension that has a separate interface for encrypting messages. A web page that has a separate interface for encrypting messages. A provider that only supports encrypted . Please explain why you prefer your top ranked option. Would you be willing to pay for a system that encrypts ? Yes, No, It depends, Unsure Only seen if did not select "No" to whether willing to pay for encryption. How much money would you be willing to pay a month to encrypt your ? 18

19 C. USER STUDY POSTER Recruitment poster used during study. Gmail User Study We are conducting research on how to improve Gmail. Please come and help us learn how to better protect everyone's ! Sign up at redacted!.voucanbook. me The study will take approximately 30 minutes Compensation will be $10 Must have a Gmail account Redacted Contact Info 19

20 D. PWM INSTRUCTIONS Instructions for installing Pwm included with all encrypted by Pwm. These instructions are hidden once participants are running Pwm. 20