Contacting respondents for survey research

Transcription

1 Contacting respondents for survey research Is a useful method? Joanna d Ardenne and Margaret Blake November 2012

2 Background

3 The question Is a useful way of contacting potential respondents about surveys? Might it help increase response? Might it reduce non-response bias? Might it save money? 2 Might it allow more frequent contact on longitudinal studies?

4 Background Traditionally advance letters used to contact respondents But the way in which people communicate has changed Use of letters declining and nearly 50% of letters delivered are marketing Increasingly public bodies communicate online Increase in postage costs 3

5 Internet and use % of population use the internet % of internet users use % of internet users use internet every day % connect to internet using mobile devices % submit forms on line % look at public authority websites 4

6 But Much traffic is also marketing Do people actually prefer to be contacted by letter? Increasing use of social media 60% women in % men in 2012 Non-users are not random: e.g. 36% of single people aged 65+ are internet users 2008 highest income households more than three times as likely to have internet access than lowest % think they do not have sufficient knowledge to protect their personal data addresses may be shared 5

7 Why contact respondents by ? For follow-up study. For next wave on a longitudinal study. To provide feedback/ findings from study. For panel maintenance e.g. asking for address updates. To inform respondents about appointments etc mid study. Conducting quality checks. 6

8 Background NatCen collected on some surveys No standard question across surveys Varying effectiveness in obtaining address Wanted to find out: Best way of collecting addresses What respondents feel about it Accuracy of addresses How respondents react when contacted by 7

9 Three tests. Test 1: Cognitive testing of address requests Test 2: Piloting address requests on Omnibus Test 3: Validation and response exercise 8

10 Cognitive Testing of requests

11 Background and aims The aims of the cognitive testing were to collect qualitative information on... How respondents felt about being asked to provide an address Reasons why respondents may refuse to provide an address 10

12 Methods Cognitive testing of requests piggy-backed on to three different question testing projects: Understanding Society Welsh Health Survey European Social Survey Requests administered in a real-life context Respondents thought requests were legitimate rather than part of the question testing Afterwards respondents were asked a number of probes about their views on the request 69 respondents took part in total Sampling and recruitment varied by project 11

13 The questions tested EM1: From time to time NatCen contacts people for further research and may wish to contact you again about a future study. Would this be all right? Yes No EM2:If we contact you to see if you are willing to help us again we may use . Do you have an address we can contact you on? Your address would only be used for research purposes and would not be passed on to anyone outside NatCen. Yes No- have not got . No do not wish to give address EM3: What is your address? 12

14 Reasons for not providing an All respondents understood the request: Asking for an address was not considered unusual or inappropriate Reasons for reticence or refusal to provide an address were: Not having an account or having limited access to a computer (generally from older respondents) Concerns about the volumes of s they might receive 13 Recommendation: Respondents could be frustrated if they receive too many s on follow-up studies. Procedures need to detail the maximum number of requests we should send in a given time period. Interviewers could be provided with an instruction to reassure respondents about this if it is required.

15 Reasons for not responding to an s not being received at a convenient time s being deleted from shared accounts Respondents described having joint accounts with partners or family members. Other account users may delete s thinking them as unsolicited Recommendation: Ideally follow-up s should specify the name of the person to whom they are directed to, to reduce the likelihood that they will be deleted or responded to by another person. Data security concerns 14

16 Data Security Concerns How do I know that if you send me an it has really come from you? Female, Aged Recommendation: 15 Respondents raised concerns about data security in relation to requests to take part in surveys i.e. that s could be sent by anyone. This suggests that other modes of contact should be considered in conjunction with web to confirm the legitimacy of the request.

17 Piloting requests on the Omnibus

18 Background and aims The aims of this test were to pilot the request in a real survey setting to ascertain what proportion of respondents provide an address in a survey context the characteristics of respondents who agree to give an address Sex Age group 17

19 Methods Questions were piloted on the NatCen Omnibus Survey (2011) Random probability PAF sample Face-to-face CAPI interview with one adult in the hhld Questions were asked at the end of the interview of 1,839 Rs Questions tested were the same as those used in cognitive testing Interviewers had to enter the address twice as a form of check If there was any difference between the two address entries an error message appeared 18

20 Key findings 1 83% of respondents consented to some form of recontact Of these: 43% agreed to contact by 34% did not have an address 23% did not want to be contacted by Put another way 35.5% of Omnibus respondents gave permission to be contacted by 19

21 Breakdown of responses to the request 28.1% 0.1% 16.9% No permission to recontact Permission to Do not wish to give address Have not got 19.5% 35.5% Don't Know 20 N=1,839, unweighted data.

22 Key findings 2 There was no clear pattern between sex and providing an address Amongst those willing to be re-contacted older respondents were less likely to provide an address: 56% of year olds 27% of year olds 16% of 75+ year olds Reasons for not providing an address varied by age 21

23 Response to EM2 by age-group 80% 70% 60% 50% 40% 30% Yes No Not willing 20% 10% 0% N=1,527 unweighted data.

24 Validation and response exercise

25 Background and aims The aim the validation and response exercise was to ascertain Percentage of s given which were valid The length of time that elapses between sending an request and receiving replies How many of the Omnibus respondents would read and respond to an from NatCen The characteristics of the respondents who respond to the Sex Age group 24

26 Methods Omnibus respondents who provided an address during their Omnibus interview were sent an (n=544) The was sent 5-7 months after the Omnibus interview The asked the respondents to click on a link s were sent via a bulk distributor Constant Contact. We were able to monitor: Bounce-backs and reasons for s not being delivered Whether respondents opened the (even if they didn t respond) Whether respondents clicked on the link Results were collated after 30 days 25

27 Content From: NatCen Omnibus heading: Thank you for taking part in our study 26 {Dear INSERT TITLE/ SURNAME} You recently took part in our Topical Issues study. At the end we asked you for an address we could contact you on. This was the first time we have done this and we would like to check whether our procedures for collecting and using addresses work. Please click on the below link to confirm you received this Clicking on the link confirms to us that the interviewer recorded your address correctly. We are not asking you to take part in further research. We will not contact you again about this address test. Thank you for your help. Jo d Ardenne Senior Researcher. National Centre for Social Research Joanna.d [email protected] For more information about the research we do please visit our website:

28 Key findings 1 If s were opened this tended to happen soon after the test was sent: 53% of openings occurred on the same day the test was sent By the 7th day 89% of the openings had occurred By the 14th day 98% of the openings had occurred Respondents who clicked on the link typically did so on the same day they opened the 27

29 Diminishing returns each day Cumulative % of s opened Days Recommendation: Reminders, if used, should be sent 1-2 weeks after the initial request. It would be useful to do further tests to establish the effect of reminders sent at different times using different modes. 28 Cumulative % of when each was opened

30 Key findings 2 The majority of the s sent were not opened Of the 544 s sent 10.5% were not delivered (bounce-backs) 51.8% were delivered but not opened 0.6% were opened but the respondent declined to click on the link 28.1% of validation respondents clicked on the link Therefore, in total 10% of Omnibus respondents clicked on the link 29

31 Results of validation and response test 10.5% 28.1% not delivered 9.6% 51.8% delivered but not opened opened but link not clicked on opened and link clicked on 30 N=544

32 Reasons for bounce-backs % of s were bounce-backs 6% bounced due to non-existent addresses 4% bounced due to other reasons ( was blocked, inbox was full, other unknown reasons) Non-existent addresses could be due to two reasons: Respondent error when giving the address (either accidentally or as a covert means of refusal) Interviewer error in typing the address 31

33 Reasons for bounce-backs 2 The 35 non-existent addresses were checked by hand. Errors in commonly used domain name were spotted in 5 cases (meant (meant to Errors were entered twice indicating the check question is not always working as intended Recommendation: The procedures for briefing interviewers should be reviewed to stress the importance of accuracy. It could be worthwhile getting respondents to verify their own addresses on the interviewer s laptop. Likewise it may be worth considering some form of incentive for interviewers collecting accurate addresses. 32

34 s not being opened 51.8% of s were delivered but not opened We speculate that respondents either: Did not see the because it was sent to a junk box via a spam filter; or Ignored the or deleted it without opening it. Recommendation: As we have no control over respondents spam filters the strategy we should pursue is encouraging respondents to open the in the first place. The title will be important for this and could merit further investigation. It would be possible to run a split-ballot experiment where s are sent with different titles to see what impact, if any, title has on response. 33

35 Not clicking the link The majority of respondents who opened the went on to click on the link (approx 75%). 9.6% of test respondents opened the but did not click on the link. We speculate reasons for not clicking the link could be: Concern about the legitimacy of the request (i.e. whether the was actually from NatCen) Concern that responding could lead to additional requests Lack of interest Opening the at an inconvenient time (e.g. whilst working) Recommendation: Further investigation could be done to look at whether text can both reassure respondents and motivate them to take part. 34

36 Key findings 3 Males more likely to open the than females 44% of males opened the 34% of females opened the Although more younger people provided an address older people were more likely to open the and reply to it 27% of year olds opened the (with 16% replying) 51% of years olds opened the (with 49% replying) There was a drop off in opening and clicking in the 75+ age group 35

37 Opening and clicking by age group 60% 50% Respondents 40% 30% 20% 10% Opened Clicked on link 0% Age group 36

38 Discussion

39 Discussion 1 Could be used in follow-up studies of PAF samples? ONLY in combination with other methods Is this cost effective? Further work is required to establish the likely response rates to follow-up surveys by Subject saliency Incentives Number of reminders Mode of reminders (F2F/Telephone/Mail) Variations in title?? 38

40 Discussion 2 Will findings transfer to longitudinal/ panel surveys? Cost effective? What training should we give interviewers in relation to collecting addresses? Accuracy? Reassurances? How often should we respondents? What other new methods should we be considering? Mobile devices Social media 39

41 Thank you If you want further information or would like to contact the authors, Joanna d Ardenne Senior Researcher T E. joanna.d [email protected] Visit us online: