Measuring the Effectiveness of Encrypted Surveys as a Non-Response Reduction Technique

Transcription

1 Measuring the Effectiveness of Encrypted Surveys as a Non-Response Reduction Technique Herson Esquivel-Vargas, Sandip Pandey and Tamunomiebaka Dibi {h.t.esquivelvargas, s.pandey, t.c.dibi}@student.utwente.nl University of Twente Surveys are an important research tool in applied statistics, though sometimes it is difficult to get enough responses to reach statistically significant results. Because of the low cost, electronic surveys are becoming more and more popular. We developed a hypothesis which claims that encrypted surveys could be used as a non-response reduction technique and then we did our own survey applying this approach in order to gather evidence to support or refute our hypothesis. We measured the response rate of encrypted surveys over plain text surveys to investigate the effectiveness of responses in both groups. We sent in the same survey to s publicly available in PGP servers: 50% plain text and 50% encrypted. Although the questions of the survey were not relevant for our hypothesis (which is focused on the response or non-response variable) we developed five questions about Security Concerns of Mobile Phones. With 2 replies out of 249 encrypted s and 4 replies out of 249 plain text s, the results gotten refute our hypothesis. Finally, we discuss the implications of our results in other somewhat related fields like spam and briefly identify some common user practices related to security in mobile phones. Keywords - Encrypted Survey, PGP (Pretty Good Privacy) Users, Non-response reduction technique. Introduction Sampling theory posits that data collected from a randomly selected sample may be projected onto a larger population. Depending on the kind of research being developed, surveys can be the more appropriate way to gather data. Surveys are an important research tool in applied statistics, though sometimes it is difficult to get enough responses to reach statistically significant results. The response rate indicates the percentage of persons that participate in a research project out of all of those who are asked to participate. It is commonly known the fact that You cannot manage what you cannot monitor and that is why International workshops like the one carried out since 1990 by nonresponse.org have as mission...the focus of the Workshop is non-response in household surveys. The main goal of the Workshop is to bring adjusters and reducers together and to initiate cooperation on different projects. People s participation in surveys can be classified (Singer, 2002) as those who respond for altruistic reasons e.g. the survey is useful for some purpose important to the respondent, or the respondent is fulfilling a social obligation; those who respond for survey-related reasons e.g. they are interested in the survey topic, or find the interviewer appealing; and those who cite what the authors call personal reasons e.g. they promised to do it. In spite of that a 2010 study by SurveyMonkey (Brent, 2010) looking at 100,000 of the online surveys they host found a drop in the response rate by about 3% at 10 questions surveys and about 6% at 20 questions surveys, with drop off slowing (for example, only 10% reduction at 40 questions). (Converse, 2006) also pointed out that the quality of response is often found to be degraded toward the end of long surveys. Because of the low cost, electronic surveys are becoming more and more popular. The aim of the present study is to test a new non-response reduction technique based on PGP to send encrypted (personalized) messages using each individual s public key, which could trigger the recipient s curiosity and at the end, help in the reduction of non-response rates. There are many reasons for wanting to develop new nonresponse reduction techniques: Governments usually pay attention to their demographic measurements and statistics to strategically plan the investment in fields like education, healthcare, welfare homes, among others. Many recommendations have been proposed towards a non-response reduction (Lynn, 2008) in telephone or faceto-face surveys (Dillman, 1978), some of which can be used in surveys as well. Advance letter: A short letter is sent in advance to inform the sampled respondents about the upcoming survey.

2 2 TEAM BOBBYPHISHER Short introduction: The interviewer should always start with a short instruction about him or herself. Respondent-friendly survey questionnaire: The questions asked must be clear, non-offensive and easy to respond to for the subjects under study. Incentives: Something is offered to the respondents to motivate the participation sacrificing anonymity. In the pilot survey we developed to test our technique we used additionally the latter three in all the surveys we sent. We sent encrypted surveys using user s public keys commonly available in PGP key servers. Using this technique we reach a demographic group with very particular technical characteristics/skills. Taking advantage of this, we develop a survey on Security concerns in mobile phones. A control group from the same population receive the same survey in plain text, which allow us to compare the results from both groups. To be more elaborate on PGP, Pretty Good Privacy is a standard format defined in [RFC-4880] used to encrypt and decrypt . Subjects were chosen in an unbiased manner so that we could cover a wide range of people from different demographic areas. There are many legitimate reasons for an internet user to be convinced to remain anonymous and/or communicate confidentially online including their rights to privacy and personal security. The Internet Society published an online survey results regarding confidential communications on the Internet which recorded 1347 responses from all around the world indicating global interest in this issue (Internet.Society, 2014). The survey was to collect information about Internet users to help gaining a more complete understanding of their needs and expectations for confidential communications online. Survey showed that about 95% considered it important to be able to communicate online with others confidentially both in personal and work purposes. Another result of the survey indicated that 52.6% considered encryption as tools or strategies to use for private communications with other users out of which 70% used PGP (Pretty Good Privacy). The above result motivated the hypothesis of the paper. Method The research methodology can be highlighted in following points. 1. Subject Selection 2. Generation of Personalized s 3. Sending s 4. Collection and analysis of responses Figure 1. Distribution of domains among subjects Subject Selection Random PGP users were chosen from the MIT s PGP keyserver (see appendix B for script used to find PGP users) from which we extracted name, and the public key. The quality of the data hosted in those servers can not be verified a priori and we can reasonably expect the presence of some careless users whose private key (required to decrypt messages) has been lost, fake/absurd names, closed accounts (like temporary addresses e.g. student.utwente.nl) and even dead people ( 0x E79F0DF4B). Since to the best of our knowledge there are no available statistics about the correctness of the data available in PGP servers we can not quantify such expectations. We chose a batch of 908 users divided in 454 for the control group (plain text survey recipients) and 454 for the experimental group (encrypted survey recipients). To avoid being black-listed by providers at the time of sending the s, we divided our subjects in three batches of 200, 216 and 454 users. As a result of the random selection of subjects we got the domain distribution shown in figure 1. Generation of Personalized s For both groups of subjects the started with Hello [name]. To craft such messages we used the name registered in the PGP server and collected in the first phase. An example of a complete message is shown in appendix A. For the encrypted survey group, the personalized was encrypted using the recipient s public key using the GPG (GNU Privacy Guard) software implementation of PGP. Encrypted messages could be considered more personalized because each one of those messages was encrypted using recipient s public key such that only the intended recipient of the message would be able to decrypt and read the message. We couldn t identify any risks for our subjects since of course,

3 TEAM BOBBY PHISHER 3 Figure 2. Aggregated survey statistics including total sent, failure, success delivery and response received they could freely ignore the message or participate in the survey. delivery After the preparation of personalized s, the delivery of both plain text and encrypted s was automated through a tailored software (appendix B) with a 30 seconds delay in between two s of each category. The delay was introduced to prevent from being detected as spam. Moreover, to send the s we used addresses in the student.utwente.nl domain which (we believe) can hardly be considered a source of spam. All messages were sent during the second half of November (16th, 24th and 28th each batch), Collection and analysis of responses After two weeks waiting for responses from the subjects we started the process of data collection. This decision is motivated from the fact shown in literature that more than 90% of users click on spam links within the first week (Kanich et al., 2008). Although it is not exactly the same situation we believe that a time window of two weeks is enough to effectively measure the response rate. plain text or encrypted (the sender s public key is in PGP servers too) answers were considered equally as successful surveys. The answers to our questions were recorded for further analysis. Results The units of analysis were individuals (i.e. survey recipients) who could either ignore or reply the survey. Accordingly with the hypothesis that we wanted to test, encrypted surveys should have had a greater response rate compared to plain text surveys but the experimental results show no evidence that this is the case. After sending 908 survey s, 46% of the total bounced with delivery failure. The number of subjects receiving plain text and encrypted surveys was exactly 249 in each group but less than 2% of those subjects actually responded the survey as shown in figure 2. Table 1 Actual delivery/response statistics. Date Delivered s Response Plain text Encrypted Plain text Encrypted Nov Nov Nov Total Out of 6 responses obtained from the survey, 2 of the responses were not useful to analyse the survey questions because reply didn t include the answers to survey question. However, those were still replies to survey and which made them important not to ignore since we were trying to measure the non-responsiveness. The responses of each questions are listed below along with questions. 1. Which operating system runs on your Smartphone? Android 1 ios 2 Other 1 2. What is the average data consumption (download) on your smartphone per week? <100MB 2 100MB-500MB 1 500MB-1000MB 1 >1000MB 0 3. On average, how often do you check for application updates on your device? Daily 0 Weekly 2 Monthly 0 Never 1 It is done automatically 1 4. Do you have an antivirus installed on your phone? Yes 0 No 4 5. Do you pay attention to the permissions required by the apps when you install? Yes but I accept anyway 0 Yes and sometimes I cancel 4 No, I don t care 0 Discussion In this research we started with the hypothesis that encrypted surveys could be used as a non-response reduction technique. Although encrypting something that 1) is not confidential and 2) we want to divulge can be thought as counterproductive, the reasoning behind it was that receiving an

4 4 TEAM BOBBYPHISHER encrypted survey could trigger the user s curiosity and therefore could be used to increase the response rate. The poor response rate obtained from the experiment refutes our hypothesis. Among the reasons of this results we can think of two: 1. Since the demography chosen is located in many different countries around the world, the language could have been a barrier because we used English for all the surveys without distinction. 2. Due to the high usage of mobile devices nowadays, most people tend to read and reply their personal mail using those while they are traveling or at home. The lack of PGP applications for mobile devices or the lack of trust of the users to put their secret key on their mobile devices could have deter them from effectively decrypt and read the survey. The obtained results are also useful to explain why publicly available public keys like those in PGP directories, are not used to deliver encrypted spam. It was important for the experiment that the s sent weren t marked as spam on the subject s inbox. To discard that possibility, we sent 5 encrypted and 5 plain text surveys to our own student address hosted in googleapps which was an important fact given that maximum number of our subjects have their addresses in gmail.com and it was reasonable to expect the spam filter behaves similarly. However, after some experiment with sending encrypted s with various titles, we found out that even encrypted could be marked as spam by Mail Provider. Gmail and Hotmail constituted the majority of subjects address, so study was constrained to above two providers only. Hence, the title and structure of body of were carefully chosen to avoid common spam patterns, and was tested to be not marked as spam on major provider in experiment. To avoid being blacklisted for sending large volume of s, sufficient delay (30 seconds in case of Gmail) was introduced between two consecutive s. Several delay and failure messages were obtained within a week after we sent the which suggested that such addresses no longer exists. Those survey s were discarded from the experiment and rest of s were assumed to be received by the subjects. However, we had no way of finding out whether the subject actually read the . Despite poor survey results, we can clearly see that all surveyed subjects doesn t use any antivirus application in their phone as in Figure 4. Another distinct result we see is that surveyed subjects agreed on paying attention to app permissions and often they cancel the application that request unnecessary permissions as shown in Figure 5. These results are in line with the expectation of PGP users to be self-aware about the privacy and application permissions (see Appendix C for survey result charts). The major limitation of the experiment is that the lack of responses support the fact that encrypted surveys is not a non-response reduction technique. However, this experiment can serve as base for conducting further research on the effectiveness of electronic communications from the point of view of the sender interested in statistics, marketing, etc. with different selection of subjects. Another contribution of this research is the GPL ed software developed to retrieve and send encrypted and plain text messages in batch. The reader can find the links in appendix B. In our experiment the subjects were randomly chosen from all around the world with no restriction in place of residence and age of key. Conducting similar research with more specific target group based on demographic region, and age of public key could help understand the effectiveness of encrypted surveys better to have meaningful conclusions. References Brent, C. (2010, December). Does adding one more question impact survey. Retrieved from /08/survey_questions_and_completion_rates/ Converse, P. E. (2006). The nature of belief systems in mass publics (1964). Critical Review, 18(1-3), Dillman, D. A. (1978). Mail and telephone surveys : the total design method. New York : Wiley. (1978 ed.) Internet.Society. (2014, December). Communicating online confidentially. Retrieved from default/files /Communicating%20Online %20Confidentially_GDP1.pdf/ Kanich, C., Kreibich, C., Levchenko, K., Enright, B., Voelker, G. M., Paxson, V., & Savage, S. (2008). Spamalytics: An empirical analysis of spam marketing conversion. In Proceedings of the 15th acm conference on computer and communications security (pp. 3 14). Lynn, P. (2008). The problem of non-response, chapter 3, International Handbook of Survey Research Methods. Singer, E. (2002). The use of incentives to reduce nonresponse in household surveys. Survey nonresponse,

5 TEAM BOBBY PHISHER 5 Appendix A Sample Survey Hello John Doe, You received this message because you have been randomly chosen to participate in this survey. This is part of a research project developed at the "UNIVERSITY OF TWENTE". Your answers will not be disclosed to third parties and you will not receive additional survey s from us. We will raffle an Amazon gift card ($20) among those who participate in the survey. You could be the lucky one. We will inform you through if you are the lucky winner of the raffle. Please take out two minutes of your time to reply this writing an X on the appropriate option for every question. Feel free to add any comments on those cases when you think none of the options suits your situation. 1. Which operating system runs on your smartphone? [ ] Android [ ] ios [ ] Other 2. What is the average data consumption (download) on your smartphone per week? [ ] <100MB [ ] 100MB-500MB [ ] 500MB-1000MB [ ] >1000MB 3. On average, how often do you check for application updates on your device? [ ] Daily [ ] Weekly [ ] Monthly [ ] Never [ ] It is done automatically 4. Do you have an antivirus installed on your phone? [ ] Yes [ ] No 5. Do you pay attention to the permissions required by the apps when you install? [ ] Yes but I accept anyway [ ] Yes and sometimes I cancel [ ] No, I don extquotesinglet care Thank you for your time. Figure 3. Figure 2: Average Data usage Figure 4. Figure 3: Software update pattern in Smartphone Kind Regards, Sandip Pandey<s.pandey@student.utwente.nl>, Herson Esquivel-Vargas<h.t.esquivelvargas@student.utwente.nl>, Tamunomiebaka Dibi<t.c.dibi@student.utwente.nl> Appendix B Source code Project source for sending encrypted s Program source code for finding PGP users post-page?startkey=[%22picking-random-pgp-users%22] Appendix C Survey Results Charts and graphs shows the result of survey. Figure 5. Use of antivirus software in phone