Evidence-based Survey Operations

Size: px

Start display at page:

Download "Evidence-based Survey Operations"

Easter Lorraine Haynes
7 years ago
Views:

1 Evidence-based Survey Operations Prof. Dr. Michael Bosnjak #GESISPanel 1

2 What is Survey Operations? Survey operations (i.e., survey implementation) encompasses all measures and procedures to field a data collection project. Overall goal: Reducing survey errors and biases in light of available resources and other restrictions, such as: type of target population feasible sampling procedures available measurement options legal and professional regulations financial resources Core tasks: Developing, monitoring, and documenting fielding-related study implementation solutions. 2

3 What is Survey Operations? Selection of topics addressed: Survey mode selection Contact and interaction plan development Incentive scheme Survey staff selection and development Field monitoring strategy and documentation Survey operations is ideally evidence-based, that is rooted in past research and best practices about what works, what doesn't, and why. 3

4 What is Evidence-based Survey Operations? Evidence-based survey operations (EBSO) emphasizes the use of evidence from well designed and conducted empirical research in survey operationsrelated decision making.... advocates that to the greatest extent possible decisions, guidelines and policies should be based on (ideally meta-analytic) evidence. favors a hierarchical model to weigh available evidence (see the Evidence Pyramid ).... promotes the use of formal, explicit methods to analyze evidence and makes it available to scholars, decision makers, and practitioners. 4

5 5 The Evidence Pyramid

6 Meta-Analysis: Generic Procedure Research questions Inclusion/ exclusion criteria Coding Generic Procedure Sources (databases) form Coding reliability ES synthesis Sign. testing Problem statement Systematic retrieval of studies Coding/ Transformation Analysis Interpretation/ Communication Dependent ES nonpublished articles Unifying ES metrics Moderator analysis ES metric Data generation model FE/RE Describes data generation process (of observed ES) Coding manual Homogeneity analysis

7 Meta-Analysis: Analysis Part Study universe True rho U2 True rho U1 Meta-analytic integration mean r Primary studies Rep. Effect sizes (documented) r 2 r 1 r 3 r k r Raw data (x 1, y 1 ), (x 2, y 2 ) (x n, y n ) (x 1, y 1 ), (x 2, y 2 ) (x n, y n ) (x 1, y 1 ), (x 2, y 2 ) (x n, y n )

8 Examples for Applying EBSO GESIS Panel An open, probability-based omnibus access panel for the social sciences. (since Jan 2013, submission accepted since Oct 2013) GESIS SDM Survey Guidelines A free resource summarizing various survey methodology guidelines, each answering a common core question: What works in survey methodology? (to be released in January 2015) 8

9 GESIS Panel: Characteristics Probability-based access panel of individuals: General population in Germany, German speaking, years Recruited panelists from population registers (270 sampling points) and 7599 face-to-face interviews (CAPI) N= 4961 (2014 starting sample) Multi-mode surveys: 9 Web-based surveys (approx. 65% of panelists) Mail surveys (because of Internet coverage deficits in DE and sample composition bias concerns; approx. 35% of panelists) Experiments with 4 groups max. possible Bi-monthly data collection waves, approx. 20 minutes of interviewing time each.

10 20 minutes GESIS Panel: Waves and Designs Wave 1 Wave 2 Wave 3 Wave k S1.W1 S3.W2 S3.W S2.Wk S2.W1 S4.W2 S2.W3 Sx.Wk Single-wave studies: Cross-sectional designs (e.g., S1, S4) Multiple-wave, longitudinal studies (e.g. S2, S3), including: Multiple cross-sectional designs; Splitted or partitioned cross-sectional design (e.g., S2) Panel designs (e.g., S3) NOT considered: Cohort (sub-sample) studies time 10

11 GESIS Panel: Reducing Nonresponse Evidence-based movement mindset: Using quantitative research syntheses (systematic reviews, meta-analysis) on what works findings Examples in other scientific disciplines: Education: What Works Clearinghouse Health sciences: Cochrane Collaboration Behavioral sciences: Campbell Collaboration Survey methodology in an infancy stage, but some quantitative syntheses to develop specific measures and procedures to reduce nonresponse

12 Examples: Nonresponse Syntheses Heberlein, T.A. & Baumgartner, R. (1978). Factors affecting response rates to mailed questionnaires: A quantitative analysis of the published literature. American Sociological Review, 43(4), Fox, R.J., Crask, M.R., & Jonghoom, K. (1988). Mail survey response rate: A meta-analysis of selected techniques for inducing response. Public Opinion Quarterly, 52(4), Church, A. (1993). Estimating the effect of incentives on mail survey response rates: A meta-analysis. Public Opinion Quarterly, 57(1), Cook, C., Heath, F., & Thompson, R.L. (2000). A Meta-Analysis of Response Rates in Web- or Internet-Based Surveys. Educational and Psychological Measurement, 60(6), Edwards, P. et al. (2002). Increasing response rates to postal questionnaires: Systematic review. British Medial Journal. 324, 1-9. Shih, T.-H., & Fan, X. (2007).Response Rates and Mode Preferences in Web- Mail Mixed-Mode Surveys: A Meta-Analysis. International Journal of Internet Science, 2(1), Lozar Manfreda, K., Bosnjak, M., Berzelak, J., Haas, I., & Vehovar, V. (2008). Web surveys versus other survey modes: A meta-analysis comparing response rates. International Journal of Market Research, 50(1), Shih, T.-H., & Fan, X. (2009). Comparing response rates in and paper surveys: A meta-analysis. Educational Research Review, 4,

13 GESIS Panel: EBSO Decisions Sampling Frame (270 SP) Fieldwork Contact Interview Compliance Recruitment Interview (CAPI) Welcome Survey (Mail, Web) Initial Survey Waves (Mail, Web) Wave aa: August (Mail, Web) Wave ab: October (Mail, Web) Wave ac: December (Mail, Web) Development Stages and Surveys 13 Non-cooperating municipalities Non-Contact Rejection (Interview) Rejection (Panlist) Non-Response (Unit-/Item-) Non-Response (Unit-/Item-), Within and Between Waves Mode- Noncompliance Address register -> Random selection of households -> Identification of respondents via last birthday method. Postal letters; Four contacts; Contact cards; Hotline; Electronic AAPOR-compliant contact protocol; Experienced interviewers; Interviewer training. Short interview (max. 20 mins), Two-sided incentives, Personalization; Tailored mode offered (online, offline) Prompt invitation; Prepaid monetary incentive by mail; Reminder strategy; Mode-switch option; 24/7 Hotline; Mixed-mode and Unimode design Monetary incentives per wave; Reminder strategy; Mode-switch option (offline > online); 24/7 Hotline; Self-service environment for online panelists; Mixed-mode and Unimode design Measures and Procedures

14 14 GESIS SDM Survey Guidelines

15 15

The AmeriSpeak ADVANTAGE

The AmeriSpeak ADVANTAGE OVERVIEW Funded and operated by NORC at the University of Chicago, AmeriSpeak TM is a probabilitybased panel (in contrast to a non-probability panel). Randomly selected households are sampled with a known,