Contacting respondents for survey research Is email a useful method? Joanna d Ardenne and Margaret Blake November 2012
Background
The question Is email a useful way of contacting potential respondents about surveys? Might it help increase response? Might it reduce non-response bias? Might it save money? 2 Might it allow more frequent contact on longitudinal studies?
Background Traditionally advance letters used to contact respondents But the way in which people communicate has changed Use of letters declining and nearly 50% of letters delivered are marketing Increasingly public bodies communicate online Increase in postage costs 3
Internet and email use 2012 80% of population use the internet 2010 90% of internet users use email 2012-67% of internet users use internet every day 2011-45% connect to internet using mobile devices 2011-27% submit forms on line 2011 32% look at public authority websites 4
But Much email traffic is also marketing Do people actually prefer to be contacted by letter? Increasing use of social media 60% women in 2012 54% men in 2012 Non-users are not random: e.g. 36% of single people aged 65+ are internet users 2008 highest income households more than three times as likely to have internet access than lowest 2011 21% think they do not have sufficient knowledge to protect their personal data Email addresses may be shared 5
Why contact respondents by email? For follow-up study. For next wave on a longitudinal study. To provide feedback/ findings from study. For panel maintenance e.g. asking for address updates. To inform respondents about appointments etc mid study. Conducting quality checks. 6
Background NatCen collected email on some surveys No standard question across surveys Varying effectiveness in obtaining email address Wanted to find out: Best way of collecting email addresses What respondents feel about it Accuracy of email addresses How respondents react when contacted by email 7
Three tests. Test 1: Cognitive testing of email address requests Test 2: Piloting email address requests on Omnibus Test 3: Validation and response exercise 8
Cognitive Testing of Email requests
Background and aims The aims of the cognitive testing were to collect qualitative information on... How respondents felt about being asked to provide an email address Reasons why respondents may refuse to provide an email address 10
Methods Cognitive testing of email requests piggy-backed on to three different question testing projects: Understanding Society Welsh Health Survey European Social Survey Requests administered in a real-life context Respondents thought requests were legitimate rather than part of the question testing Afterwards respondents were asked a number of probes about their views on the email request 69 respondents took part in total Sampling and recruitment varied by project 11
The questions tested EM1: From time to time NatCen contacts people for further research and may wish to contact you again about a future study. Would this be all right? Yes No EM2:If we contact you to see if you are willing to help us again we may use email. Do you have an e-mail address we can contact you on? Your email address would only be used for research purposes and would not be passed on to anyone outside NatCen. Yes No- have not got email. No do not wish to give email address EM3: What is your e-mail address? 12
Reasons for not providing an email All respondents understood the request: Asking for an email address was not considered unusual or inappropriate Reasons for reticence or refusal to provide an email address were: Not having an email account or having limited access to a computer (generally from older respondents) Concerns about the volumes of emails they might receive 13 Recommendation: Respondents could be frustrated if they receive too many emails on follow-up studies. Procedures need to detail the maximum number of requests we should send in a given time period. Interviewers could be provided with an instruction to reassure respondents about this if it is required.
Reasons for not responding to an email Emails not being received at a convenient time Emails being deleted from shared accounts Respondents described having joint email accounts with partners or family members. Other account users may delete emails thinking them as unsolicited Recommendation: Ideally follow-up emails should specify the name of the person to whom they are directed to, to reduce the likelihood that they will be deleted or responded to by another person. Data security concerns 14
Data Security Concerns How do I know that if you send me an email it has really come from you? Female, Aged 36-59 Recommendation: 15 Respondents raised concerns about data security in relation to email requests to take part in surveys i.e. that emails could be sent by anyone. This suggests that other modes of contact should be considered in conjunction with web to confirm the legitimacy of the email request.
Piloting email requests on the Omnibus
Background and aims The aims of this test were to pilot the email request in a real survey setting to ascertain what proportion of respondents provide an email address in a survey context the characteristics of respondents who agree to give an email address Sex Age group 17
Methods Questions were piloted on the NatCen Omnibus Survey (2011) Random probability PAF sample Face-to-face CAPI interview with one adult in the hhld Questions were asked at the end of the interview of 1,839 Rs Questions tested were the same as those used in cognitive testing Interviewers had to enter the email address twice as a form of check If there was any difference between the two email address entries an error message appeared 18
Key findings 1 83% of respondents consented to some form of recontact Of these: 43% agreed to contact by email 34% did not have an email address 23% did not want to be contacted by email Put another way 35.5% of Omnibus respondents gave permission to be contacted by email 19
Breakdown of responses to the email request 28.1% 0.1% 16.9% No permission to recontact Permission to email Do not wish to give email address Have not got email 19.5% 35.5% Don't Know 20 N=1,839, unweighted data.
Key findings 2 There was no clear pattern between sex and providing an email address Amongst those willing to be re-contacted older respondents were less likely to provide an email address: 56% of 16-24 year olds 27% of 65-74 year olds 16% of 75+ year olds Reasons for not providing an email address varied by age 21
Response to EM2 by age-group 80% 70% 60% 50% 40% 30% Yes No email Not willing 20% 10% 0% 16-24 25-34 35-44 45-54 55-64 65-74 75+ 22 N=1,527 unweighted data.
Validation and response exercise
Background and aims The aim the validation and response exercise was to ascertain Percentage of emails given which were valid The length of time that elapses between sending an email request and receiving replies How many of the Omnibus respondents would read and respond to an email from NatCen The characteristics of the respondents who respond to the email Sex Age group 24
Methods Omnibus respondents who provided an email address during their Omnibus interview were sent an email (n=544) The email was sent 5-7 months after the Omnibus interview The email asked the respondents to click on a link Emails were sent via a bulk email distributor Constant Contact. We were able to monitor: Bounce-backs and reasons for emails not being delivered Whether respondents opened the email (even if they didn t respond) Whether respondents clicked on the link Results were collated after 30 days 25
Email Content From: NatCen Omnibus Email heading: Thank you for taking part in our study 26 {Dear INSERT TITLE/ SURNAME} You recently took part in our Topical Issues study. At the end we asked you for an email address we could contact you on. This was the first time we have done this and we would like to check whether our procedures for collecting and using email addresses work. Please click on the below link to confirm you received this email http://www.natcen.ac.uk/emailtesting/index.htm Clicking on the link confirms to us that the interviewer recorded your email address correctly. We are not asking you to take part in further research. We will not contact you again about this email address test. Thank you for your help. Jo d Ardenne Senior Researcher. National Centre for Social Research Email: Joanna.d ardenne@natcen.ac.uk For more information about the research we do please visit our website: www.natcen.ac.uk
Key findings 1 If emails were opened this tended to happen soon after the test email was sent: 53% of openings occurred on the same day the test email was sent By the 7th day 89% of the openings had occurred By the 14th day 98% of the openings had occurred Respondents who clicked on the link typically did so on the same day they opened the email 27
Diminishing returns each day Cumulative % of emails opened 100 90 80 70 60 50 40 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 Days Recommendation: Reminders, if used, should be sent 1-2 weeks after the initial email request. It would be useful to do further tests to establish the effect of reminders sent at different times using different modes. 28 Cumulative % of when each email was opened
Key findings 2 The majority of the emails sent were not opened Of the 544 emails sent 10.5% were not delivered (bounce-backs) 51.8% were delivered but not opened 0.6% were opened but the respondent declined to click on the link 28.1% of validation respondents clicked on the link Therefore, in total 10% of Omnibus respondents clicked on the link 29
Results of validation and response test 10.5% 28.1% Email not delivered 9.6% 51.8% Email delivered but not opened Email opened but link not clicked on Email opened and link clicked on 30 N=544
Reasons for bounce-backs 1 10.5% of emails were bounce-backs 6% bounced due to non-existent email addresses 4% bounced due to other reasons (email was blocked, inbox was full, other unknown reasons) Non-existent addresses could be due to two reasons: Respondent error when giving the address (either accidentally or as a covert means of refusal) Interviewer error in typing the address 31
Reasons for bounce-backs 2 The 35 non-existent email addresses were checked by hand. Errors in commonly used domain name were spotted in 5 cases e.g. @otmail (meant to be @hotmail) @couk (meant to be @co.uk). Errors were entered twice indicating the check question is not always working as intended Recommendation: The procedures for briefing interviewers should be reviewed to stress the importance of accuracy. It could be worthwhile getting respondents to verify their own email addresses on the interviewer s laptop. Likewise it may be worth considering some form of incentive for interviewers collecting accurate email addresses. 32
Emails not being opened 51.8% of emails were delivered but not opened We speculate that respondents either: Did not see the email because it was sent to a junk email box via a spam filter; or Ignored the email or deleted it without opening it. Recommendation: As we have no control over respondents spam filters the strategy we should pursue is encouraging respondents to open the email in the first place. The email title will be important for this and could merit further investigation. It would be possible to run a split-ballot experiment where emails are sent with different titles to see what impact, if any, title has on response. 33
Not clicking the link The majority of respondents who opened the email went on to click on the link (approx 75%). 9.6% of test respondents opened the email but did not click on the link. We speculate reasons for not clicking the link could be: Concern about the legitimacy of the request (i.e. whether the email was actually from NatCen) Concern that responding could lead to additional email requests Lack of interest Opening the email at an inconvenient time (e.g. whilst working) Recommendation: Further investigation could be done to look at whether email text can both reassure respondents and motivate them to take part. 34
Key findings 3 Males more likely to open the email than females 44% of males opened the email 34% of females opened the email Although more younger people provided an email address older people were more likely to open the email and reply to it 27% of 16-24 year olds opened the email (with 16% replying) 51% of 64-74 years olds opened the email (with 49% replying) There was a drop off in opening and clicking in the 75+ age group 35
Opening and clicking by age group 60% 50% Respondents 40% 30% 20% 10% Opened email Clicked on link 0% 16-24 24-34 35-44 45-54 55-64 65-74 75+ Age group 36
Discussion
Discussion 1 Could email be used in follow-up studies of PAF samples? ONLY in combination with other methods Is this cost effective? Further work is required to establish the likely response rates to follow-up surveys by email: Subject saliency Incentives Number of reminders Mode of reminders (F2F/Telephone/Mail) Variations in email title?? 38
Discussion 2 Will findings transfer to longitudinal/ panel surveys? Cost effective? What training should we give interviewers in relation to collecting email addresses? Accuracy? Reassurances? How often should we email respondents? What other new methods should we be considering? Mobile devices Social media 39
Thank you If you want further information or would like to contact the authors, Joanna d Ardenne Senior Researcher T. 020 7549 7108 E. joanna.d ardenne@natcen.ac.uk Visit us online: www.natcen.ac.uk