Challenges and pitfalls of introducing web surveys at NSIs

Transcription

1 Challenges and pitfalls of introducing web surveys at NSIs Ana Bozic ESSnet workshop DCSS, Wiesbaden 4-5 September 2014

2 Challenges and pitfalls Increasing non-response Budget cuts / Need to optimize costs (maintaining the required level of quality) Communication with the respondents Preparation (adaptation) of questionnaires Combine data from different survey modes An efficient system for managing the respondents

3 Our experience with mixed-mode data collection (incl. web survey ) 1) Tourism travels of domestic population (2012) A) Statistical office (SURS): CATI B) Faculty of Social Sciences: Mixed-mode experiment - web, CATI, mail 2) Survey on the careers of doctorate holders (2013) Statistical office (SURS): web, CATI, mail

4 Mixed-mode experiment design (Tourism travels of domestic population) Parallel survey with SURS (CATI) in 2012 N= 3000 n 1 = 1720 with known telephone number n 2 = 1280 undefined telephone number Web Invitation to web survey for n 1 + n 2 ( ) CATI CATI survey for nonrespondents from n 1 ( ) Mail Mail survey for nonrespondents from n 1 and n 2

5 60% 50% Response rates (Tourism travels of domestic population) 49% 55% 43% 60% 40% 30% 33% 34% 20% 10% 10% 10% 0% Web Web + CATI Web + CATI + Mail SURS (CATI) All units Units with known telephone number

6 Results/Findings (Tourism travels of domestic population) - With mixed-mode we reach more young and educated - In comparison with SURS survey mixed-mode survey significantly overestimates variables about travel activities - Mixed-mode survey was not less expensive then SURS CATI survey - With mixed-mode survey was reach more different populations (travel much more)

7 Survey on the careers of doctorate holders (Sample) in 2013 Published address (81%) Published telephone number (46%) e-address = 0 e-address = 1 Total Tel = 0 Tel = 1 Total % 9% 19% % 37% 81% % 46% 100% Survey design: Web-1 (+reminder) + CATI + Web-2 + Mail (Web-3)

8 1. phase: Web-1 n web-1 =2236 response=36% response web-1 = 806 nonresponse web-1 =1372 no eligible web-1 =58 2. phase: CATI n CATI =632 - fixed or mobile phone number is applied Waiting for web-2 response=30% cumulative=44% response CATI =189 nonresponse CATI =417 no eligible CATI =26 n REST =740 - No answer n= phase: Web-2 - With e-address - No e-address Waiting for mail response=6% cumulative=47% response web-2 =52 4. phase: web-3 and mail response web-3 =45 response=16% cumulative=54% response mail =112 n=837 nonresponse web-2 =783 n=990 nonresponse web-3,mail =831 no eligible web-2 =2 - Questionnaire not completeed no eligible web-3 =2 no eligible mail =0 n REST =213

9 Results/Findings (Survey on the careers of doctorate holders) Response rate is higher within persons with known e-address (38%) compared to those with unknown address (31%). Response rate of particular fields of doctorate is about 50%, less are cooperative those with doctorate in medical sciences (41%). Young people respond close 60%, the elderly under 40%. There are no differences in responses between genders. In 2010 we managed to determine the phone number to 59% persons (2013; 46%), response rate (CATI+mail) was 62% (2013; 54%) survey was not less expensive then 2010 survey.

10 Exchanging experiences - consultation SURS, authorised producers of national statistics, faculties, marketing research companies Outlines: What is the future of independent survey modes? Can the introduction of web surveys (as one of modes) increase response rate? How/When can the introduction of mixed modes reduce the cost of the survey? How to define the appropriate mode, population, content, length of the survey? What is the appropriate strategy of combining (cost dimension, optimal coverage of the population, content...)? To what extent is necessary to adapt the questionnaire to keep consistency? Are we able to assess the impact of survey mode (mode effect)? Can we estimate the bias?

11 Conclusions: Possibility to offer online surveys is inevitable (Internet penetration in Slovenia approx. 75% of households). Combining of different modes is used only when we are forced to do so (combination of too many modes; negative effects - lack of control; several effects - very important to evaluate the effects are experiments). Rewards to respondents have an impact, but only if they are monetary. Monetary rewards may also affect responses (web panels - desirable answers). The implementation of household surveys online still impose several limitations. More than 50% of the questions should be changed (transformed) with the transition to a web survey. We obtain different values of variables with different modes of surveying - often we are more interested in the trend as the population value. The response rate is particularly important when variables are not robust. Time series (options: new beginning, try to keep parallel survey, additional modelling, ).

12 Project Introducing of web surveys ( ) A systematic approach to the introduction of web survey at SURS (substantive, organizational and technical solutions) Exclud. management and determining surveys that go on web Pilot - Consumer survey Cognitive tests - first findings

13 First findings Communication on web must be kindly as there is an interviewer Short introduction letter, a simple website to access the questionnaire (colours, pictures affect the response) Simple link, username, password (but safe) Appropriate formulation of questions (consistency), output harmonisation

14 Adequate display of explanations Testing functionality (technical literacy) Paradata - used also for testing the questionnaire Reminders (efficiency vs. cost) testing in Oct. Particular challenge to produce a system for managing the respondents 2nd stage

15 Thank you! Ana Bozic, M.Sc. Head of Interviewing of People and Households Department Project leader Introducing web survey (for persons) at SURS