Alcohol, CAPI, drugs, methodology, non-response, online access panel, substance-use prevalence,

Transcription

1 METHODS AND TECHNIQUES doi: /j x The utility of online panel surveys versus computerassisted interviews in obtaining substance-use prevalence estimates in the Netherlandsadd_ Renske Spijkerman 1, Ronald Knibbe 2, Kim Knoops 3, Dike van de Mheen 4 & Regina van den Eijnden 5 Radboud University, Behavioural Science Institute, Nijmegen, the Netherlands, 1 Maastricht University, Department of Health Promotion, Maastricht, the Netherlands, 2 Statistics Netherlands, Heerlen, the Netherlands, 3 ErasmusMc, Department of Public Health/ IVO Addiction Research Institute, Rotterdam, the Netherlands 4 and Utrecht University, Department of Interdisciplinary Social Sciences, Utrecht, the Netherlands 5 ABSTRACT Aims Rather than using the traditional, costly method of personal interviews in a general population sample, substance-use prevalence rates can be derived more conveniently from data collected among members of an online access panel. To examine the utility of this method, we compared the outcomes of an online survey with those obtained with the computer-assisted personal interviews (CAPI) method. Design Data were gathered from a large sample of online panellists and in a two-stage stratified sample of the Dutch population using the CAPI method. Setting The Netherlands. Participants The online sample comprised Dutch online panellists (15 64 years) of Survey Sampling International LLC (SSI), and the CAPI cohort 7204 respondents (15 64 years). Measurements All participants answered identical questions about their use of alcohol, cannabis, ecstasy, cocaine and performance-enhancing drugs. The CAPI respondents were asked additionally about internet access and online panel membership. Both data sets were weighted statistically according to the distribution of demographic characteristics of the general Dutch population. Findings Response rates were 35.5% (n = ) for the online panel cohort and 62.7% (n = 4516) for the CAPI cohort. The data showed almost consistently lower substance-use prevalence rates for the CAPI respondents. Although the observed differences could be due to bias in both data sets, coverage and non-response bias were higher in the online panel survey. Conclusions Despite its economic advantage, the online panel survey showed stronger non-response and coverage bias than the CAPI survey, leading to less reliable estimates of substance use in the general population. Keywords web survey. Alcohol, CAPI, drugs, methodology, non-response, online access panel, substance-use prevalence, Correspondence to: Renske Spijkerman, Behavioural Science Institute, PO Box 9104, 6525 HE, Nijmegen, the Netherlands. r.spijkerman@pwo.ru.nl Submitted 28 November 2007; initial review completed 19 August 2008; final version accepted 31 March 2009 INTRODUCTION Most research on substance use prevalence rates is conducted by means of written self-report questionnaires or oral interviews in random samples of the national population [1]. However, both methodologies are costly and time-consuming [2,3]. Online panel research may be a more economic alternative. Here, members of online access panels are invited to participate in research via , after which the respondents complete a digital questionnaire. Apart from being cheaper, online research is also quicker than personal interviewing and thus allows easy data acquisition from large samples. In addition, online research is generally more convenient for participants. Despite these advantages, online panel surveys may have the important disadvantage of causing considerable selection biases resulting from coverage errors and self-selection mechanisms among panellists [4,5]. In the present study we compare two data-collection methods, i.e. web surveys completed by Dutch online

2 1642 Renske Spijkerman et al. panel members and computer-assisted personal interviews (CAPI) obtained in a stratified sample of the general Dutch population. We follow the assumption of Duffy et al. [5] that a statistically weighted procedure will correct partly for selection effects caused by using a self-selected pool of online panellists. However, even with weighting, differences in substance-use prevalence estimates may remain between the two survey modes due to other types of selection and survey-mode effects [4,6,7]. We therefore examined potential differences in substance-use prevalence rates as obtained by the two research modes. To determine whether any discrepancies might be due to coverage bias, we questioned the CAPI respondents about online panel membership and subsequently compared the substance use prevalence rates of the CAPI respondents with and without such membership. METHODS Data collection took place during May December The sampling frame of the CAPI survey comprised residents aged years who were included in the municipal population registers on 1 January 2005, excluding institutionalized individuals. After defining 40 strata of municipalities, we selected municipalities per stratum. The probability that a municipality was selected was proportionate to the size of the municipality. Subsequently, we selected residents randomly from the municipalities included, with their number depending upon the size of the municipality [8]. The sampling frame of our online survey consisted of members (aged years) of the Dutch online panel of Survey Sampling International LLC (SSI). In 2005 SSI had compiled a large Dutch online panel of more than members, from which respondents of the present study were selected in a stratified sample that corresponded to the distribution of population characteristics. We defined 18 quota cells delineated by gender, age, educational level and region. The sample distribution was monitored and if an insufficient number of panellists responded, additional s were sent to new respondents. In both surveys we used the same yes/no questions [9] to measure substance use. The panel members answered by completing an electronic questionnaire, while the CAPI respondents did so during a personal interview. None of the respondents were paid for their participation. Participants were first asked whether they had used any of the indicated substances (alcohol, cannabis, ecstasy, cocaine and performance enhancing drugs) in their lifetime. Next, those who answered affirmatively were asked whether they had used the substance(s) in the past 12 months and, only for alcohol, during the last 30 days. Finally, all CAPI participants received yes/no questions about whether they could access the internet and, if so, whether they participated in an online panel or online research. The CAPI sample comprised 7204 people and we achieved a response rate of 62.7% (n = 4516). Of the invited online panel members, 35.5% responded (n = ). Although we achieved most of the planned targets for the different strata, the actual response was low, particularly among the youngest panel members (15 24 years), where the response rate was 20 26%. Additional scrutiny of the differences in substance use between respondents and nonrespondents showed that infrequent drinkers and non-smokers were over-represented among the nonrespondents in both samples. To assess the characteristics of the non-respondents in the CAPI survey, a capture recapture study was conducted by selecting a random sample of respondents and non-respondents (n = 1250). Of the re-approached respondents 84.5% (n = 299) responded, while of the former nonrespondents 54.7% (n = 491) did so. For the online survey, a non-response study in a convenience sample of non-respondents (n = 1991) was conducted. Details of non-response studies can be obtained from the authors. A comparison of the costs between both studies showed that the online panel study generated a markedly lower price for a completed interview ( 2.81) than the CAPI study ( 76.76). After the fieldwork, data were weighted statistically according to several demographic variables of the general population as reported by Statistics Netherlands, using a similar weighting procedure for both data sets. (Further details on the exact weighting procedure are provided in Appendix S1, which is available with the online version of this paper; see details at the end.) Both samples showed an equal distribution of males and females and age. Relative to the CAPI respondents, more online respondents had a higher educational level and fewer had a nonwestern cultural background. RESULTS Table 1 shows the substance-use prevalence rates in each sample for three different age bands. As shown, the weighting procedure had not led to large adjustments in our estimates. The overall pattern of results reflected that substance use in the online respondents was higher. Only alcohol consumption in the past 30 days was higher in the CAPI respondents aged between 45 and 64 years. The elevated rates for 12-month alcohol and cocaine use and for both prevalence rates of ecstasy were present only in the middle and youngest age categories. In contrast, the elevated life-time and 12-month use of performance-

3 Comparison of CAPI and online panel data on substance use 1643 Table 1 Substance-use prevalence rates (%) per age category for the online and computer-assisted personal interviews (CAPI) respondents. Online panel (n = ) CAPI (n = 4516) Age (years) Unweighted Weighted Unweighted Weighted Pearson c 2 Alcohol Past 12 months *** ** Past 30 days ** Cannabis Life-time *** *** *** Past 12 months *** *** *** Ecstasy Life-time *** *** Past 12 months * ** Cocaine Life-time *** *** ** Past 12 months *** *** Performance-enhancing drugs Life-time *** ** Past 12 months *** ** Pearson c 2 based on the weighted estimates. *P < 0.05; **P < 0.01; ***P < enhancing drugs were observed only in the middle and highest age band. Finally, higher rates for life-time and 12-month cannabis use and for life-time cocaine use were found in all age categories. Of the CAPI respondents, 84.9% reported having internet access. Table 2 shows that the estimates for alcohol use were higher for the respondents with internet access. Conversely, respondents without internet access reported having used illegal drugs such as cannabis, ecstasy and cocaine more frequently. Of the respondents with internet access, 13.2% reported being a member of an online panel, which amounts to 11.2% of the total CAPI sample. This group reported more life-time cannabis and ecstasy use than respondents who had no panel membership. Because the data collection was the same for all CAPI respondents, the observed differences could not be attributed to mode effects. DISCUSSION With the present study we gained more insight into the utility of the online survey method to research substance use prevalence rates in the Netherlands. Although the online panel survey produced a larger data set against markedly lower costs, its response rate was much lower. Further, our findings showed that the web-based survey yielded higher prevalence rates for all substances under investigation, except alcohol use, in the past 30 days. Three types of biases, i.e. coverage, non-response and measurement bias, might explain these elevated prevalence rates. With regard to coverage bias, we compared the substance use estimates of the CAPI respondents with and without panel membership. Indeed, cannabis and ecstasy use was higher among respondents with panel memberships than among their peers without such a

4 1644 Renske Spijkerman et al. Table 2 Differences in demographic characteristics and substance use for the computer-assisted personal interviews (CAPI) respondents in relation to internet access and online panel membership. Internet access Panel membership No Yes No Yes n = 646 n = 3868 Pearson c 2 n = 3370 n = 489 Pearson c 2 Sex Male Female Age (years) *** * Education level Low *** *** Middle High Alcohol Past 12 months *** Past 30 days *** Cannabis Life-time *** Past 12 months *** Ecstasy Life-time ** * Past 12 months *** Cocaine Life-time ** Past 12 months Performance-enhancing drugs Life-time Past 12 months Results are based on weighted data. Only respondents indicating having internet access (n = 3868) were asked after their online panel membership. Two cases with missing data on internet access and nine cases with missing data on online panel membership were excluded from the analyses. *P < 0.05; **P < 0.01; ***P < membership. We conclude, therefore, that the differences in estimates between the CAPI and the online group are due partly to online panel members having specific characteristics that are not representative of the general population. The between-sample differences are explained further by the strong non-response bias in the online panel. The low response rate indicates that we reached only a select group of online panellists. We attempted to correct part of this selection effect by weighting for demographic features, but this procedure did not correct for other sample characteristics. Hence, the online respondents were probably not representative of the Dutch general population. The contrasting prevalence rates may also be due to different data collection modes. The presence of an interviewer may increase a person s inclination to give socially desirable answers. In substance-use research the impact of socially desirable answers can be particularly high, as substance use is regarded generally as a sensitive topic [10]. Thus, relative to respondents of the online panel survey, CAPI respondents who were interviewed in the presence of an interviewer may have been more likely to under-report their substance use. In sum, our online survey to gauge substance use in a general population sample was more cost-effective, but suffered from higher non-response and coverage bias than the traditional CAPI study. For policy makers, our findings suggest that the expensive CAPI method should be preferred when aiming at the acquisition of reliable substance use prevalence rates among general populations. However, to answer questions that are less dependent upon sample representativeness, policy makers should favour online panel surveys over the traditional CAPI method. To solve the problem of coverage bias, we recommend the setting-up of representative panels. Currently, such a panel is set up and financed by the Dutch government. This investment will enable policy makers to obtain more reliable information against much lower costs. Declarations of interest None. Acknowledgements The present study was funded by the Dutch Ministry of Health, Welfare and Sport. We thank Ferdy Otten, Frans Frenken, Birthe Onverwagt and Nienke Nottelman for their assistance with the collection and preparation of the data. We would also like to thank Manja Abraham and Christianne Hupkens for their helpful suggestions and advice.

5 Comparison of CAPI and online panel data on substance use 1645 References 1. Richter L., Johnson P. B. Current methods of assessing substance use: a review of strengths, problems, and developments. J Drug Issues 2001; 31: De Leeuw E., Nicholls W. II. Technological innovations in data collection: acceptance, data quality and costs. Soc Res Online 1996; 1. Available from org.uk/1/4/leeuw.html (accessed 8 June 2009). 3. Dillman D. A. Why choice of survey mode makes a difference. Publ Health Rep 2006; 121: Duffy B., Smith K., Terhanian G., Bremer J. Comparing data from online and face-to-face surveys. Int J Market Res 2005; 47: Lynn P., Jowell R. How might opinion polls be improved? The case for probability sampling. J Roy Stat Soc 1996; 159: Aquilino W. S., LoSciuto L. Effects of interview mode on self-reported drug use. Publ Opin Q 1990; 64: Beebe T. J., McRae J. A. Jr, Davern P. A., Harrison, M. E., Quinlan K. B. Mail surveys resulted in more reports of substance use than telephone surveys. J Clin Epidemiol 2005; 58: Statistics Netherlands. POLS: Basis Documentatie [Integrated survey on household living conditions (POLS): basic documentation]. Heerlen, the Netherlands: Statistic Netherlands; European Monitoring Centre for Drugs and Drugs Addiction (EMCDDA). Handbook for Surveys on Drug Use among the General Population. Lisbon: EMCDDA; Tourangeau R., Smith T. W. Asking sensitive questions: the impact of data collection mode, question format, and question context. Publ Opin Q 1996; 60: Supporting information Additional Supporting Information may be found in the online version of this article: Appendix S1. The utility of online panel surveys versus computer-assisted interviews in obtaining substance-use prevalence estimates in the Netherlands. Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.

6