Incentives and response rates - first experiences from

Transcription

1 Incentives and response rates - first experiences from the SOEP-Innovation-Sample 2009 Jürgen Schupp - joint work with Martin Kroh, Elisabeth Liebau, Nico Siegel, Simon Huber, and Gert G. Wagner 2 nd Panel-Survey-Methodology-Workshop Mannheim, July, 5-6 th

2 Wave 1 response rates in SOEP by sample in % ,6 68,0 70,0 55,0 54,0 51,0 48,3 40, A (1984) - West Germany B (1984) Foreigners in West Germany C (1990) - Citizen of the GDR D (1994/5) - Immigrants since 1984 (average) E (1998) - Crosssection '98 F (2000) - Crosssection 2000 G (2002) - High- Income 2002 H (2006) - Crosssection 2006 Source: SOEP. SOEP-Samples 2 27

3 Motivation Current debate on different forms and values of respondent incentives on response rates, unit non response bias and various measures of data quality (Laurie/Lynn 2009) Since start of SOEP in 1984 no systematic evaluation of best practice of incentives Open questions: Do cash-incentives lead to higher response rates than lotterytickets? Cross-sectional and longitudinal response-rates Do high cash-incentives have significant increasing responserates? Cross-sectional and longitudinal response-rates Are there significant reductions in longitudinal response rates converting high to low cash incentives? 3 27

4 Personalized Incentives in SOEP since 2008 (The default model for the longitudinal Announcement by mail two weeks prior to pop. interview ) including a unconditional ticket for monthly nationwide lottery (5 incentive) but conditional incentive for those households that did not participate with an interview in the last year Interviewer will additional have per household a door opener (value about 4 Euro/per household), a brochure Living in Germany For each successful interview, any respondent will receive a thank you letter (during summer) including and porto-card (stamp for a letter) and the offer to get more information about the study upon request Additional incentives for some age-groups (DJ-questionnaire, mother & child) 4 27

5 The SOEP Innovation Sample in fall/winter 2009 a new sample (I) of the SOEP has been drawn with a gross sample size of 5,000 households same questionnaires as core-soep 2009 (no additional biographical questionnaires an experimental design to measure the impact of four different incentives has been carried out different incentives in terms of their overall effect on response rates and their effect on cooperation patterns In spring 2010 a mailed non-response survey started with a short questionnaire was sent out to 3,215 households that did not participate not finally finished yet In fall 2010 the second wave of sample I will start; including biographical questionnaire incentive design up to now not finalized 5 27

6 4 randomized split groups of grosssample within each sample point (1,250) 1. SOEP-Default - Lottery a. Household door-opener b. Lottery ticket (sent by mail some weeks later) 2. low cash a. Household door-opener b. 5 Euro instant cash per household c. 5 Euro instant cash incentive per adult respondent 3. high cash a. Household door-opener b. 5 Euro instant cash per household c. 10 Euro instant cash incentive per adult respondent 4. choice cash/lottery ticket a. Household door-opener b. Choice of low instant cash incentive per household and adult respond or lottery ticket incentive per adult respondent 6 27

7 Design of Innovation Sample First step: selection of sample points n=250; Area sample; Regional stratification of sample Second step: Listing of addresses N=12,000, 48 addresses per sample point, random walk Third step: assignment of gross-sample Oversampling potential migrants (doubling the migrant percentage per point) by onomastic-check for n= 12,000 Total n= 5,000 Final step: random split for incentives n=4*1,

8 Data Collection of Innovation Sample Compared to other SOEP-samples short duration for fieldwork (5 months) Due to budget cuts no special interviewer training 49 % of all interviewers (247) without SOEP-experience No mixed mode strictly CAPI (no biographic questionnaires or age specific questionnaires) 8 27

9 Descriptive results of fieldwork Crosstab Randomized split groups - incentives Total low cash high cash 5+10 choice - lottery or low cash lottery ticket Response status participation not possible(lang died, not uage, health realized eligible refusal no contact problems) interview Total ,8% 49,8% 9,6% 4,3% 32,4% 100,0% ,0% 49,7% 9,8% 3,8% 31,7% 100,0% ,9% 50,7% 11,7% 3,5% 30,2% 100,0% ,9% 52,2% 11,0% 3,7% 28,2% 100,0% ,4% 50,6% 10,5% 3,8% 30,6% 100,0% 9 27

10 Corrected results for units out of scope Crosstab Randomized split groups - incentives Total low cash high cash 5+10 choice - lottery or low cash lottery ticket Response status participation not possible(lang uage, health realized refusal no contact problems) interview Total ,8% 10,0% 4,5% 33,7% 100,0% ,3% 10,3% 4,0% 33,4% 100,0% ,8% 12,2% 3,7% 31,4% 100,0% ,9% 11,5% 3,9% 29,7% 100,0% ,0% 11,0% 4,0% 32,0% 100,0% 10 27

11 % Development of response rate for SOEP 51 40, F H I SOEP-response rate 11 27

12 Models for Survey Response (cooperation) SOEP-Innovation- Sample 2009 Influence of Incentives on Survey Response (Model Comparison) M1 Logit M2 Logit, clustered M3 xtlogit, re (intnr) dep. var: response(1/0) odds ratios odds ratios odds ratios Incentives (ref: lottery) choice (0.90) (0.84) (0.88) low_cash 1.20* * (2.10) (1.88) (2.20) high_cash * (1.93) (1.82) (2.12) sigma_u : rho : N groups:

13 Some structural information of the sample Migrant (by onomastic characteristics of the name (migrant) Region (East-/West-Germany) (east) Area Rural area small town (<20,000) (country) Big town (>100,000 and < 500,000) (bigtown) Metropolitan towns (> 500,000) (metro) Telephone-Number (generated by surveyinstitute via public available data-base checks) (openno) Number of contacts by interviewer (contacts) Subjective evaluation of neighborhood by interviewer (untended) 13 27

14 Models for Survey Response (cooperation) SOEP-Innovation- Sample 2009 Determinants of Survey Response M0 M1 M2 M3 M4 M5 dep. var: response(1/0) odds ratios odds ratios odds ratios odds ratios odds ratios odds ratios Incentives (ref: lottery) choice (0.86) (0.87) (0.98) (1.03) (1.03) low_cash 1.23* 1.23* 1.23* 1.21* 1.21* (2.20) (2.20) (2.22) (2.04) (2.04) high_cash 1.23* 1.23* 1.24* 1.23* 1.23* (2.18) (2.18) (2.27) (2.22) (2.22) east (1.10) (0.78) (0.64) (0.66) country (1.26) (1.13) (1.09) (1.02) bigtown (-0.70) (-0.56) (-0.51) (-0.52) metro (-0.56) (-0.31) (0.06) (0.05) migrant 0.65*** 0.68*** 0.68*** (-4.39) (-4.02) (-4.04) openno (0.93) (0.74) (0.76) contacts (-0.52) (-0.53) untended 0.57*** 0.57*** (-4.36) (-4.36) interviewer variables SOEP experience 1.22 (1.48) sigma_u rho Wald-chi² (df) *** 55.18*** 61.45*** (df=3) (df=7) (df=9) (df=11) (df=21) N *p<0.05, **p<0.01,***p<0.001 SOEP

15 Further analysis Control of additional neighborhood Determinants Individual and Neighborhood Determinants of Survey Nonresponse An Analysis Based on a New Subsample of the German Socio-Economic Panel (SOEP), Microgeographic Characteristics and Survey-Based Interviewer Characteristics By Jörg-Peter Schräpler, Jürgen Schupp, and Gert G. Wagner (2010) 15 27

16 Further checks for data quality Partial Unit Non-Response Randomized * punr Crosstabulation Randomized split groups - incentives Total low cash high cash 5+10 choice - lottery or low cash lottery ticket punr,00 1,00 Total ,1% 11,9% 100,0% ,2% 9,8% 100,0% ,6% 10,4% 100,0% ,3% 12,7% 100,0% ,8% 11,2% 100,0% 16 27

17 Summary Cash-Incentives show significant improvement in participation rate Migrants show significant lower participation rate Indicators of lowest housing standard show lower participation rate (causality???) First view - no interviewer characteristics show significant variation of participation rate 17 27

18 SOEP Non-Response Study 2010 combined with randomized experiment Positive framing of non-participation (part of majority) in motivation letter Mailed short questionnaire with conditioned incentive end of May 2010 a. Porto-Card (2 stamps for letters) - no additional incentive b. Porto-Card (2 stamps for letters) - 5 Euro c. Porto-Card (2 stamps for letters) - 15 Euro 18 27

19 Socio-Demographic Age, Sex, Marital Status, Education (secondary, vocational) (Un-)Employment Status Native language Household characteristics Household size, age of household members Household income (screener - stratified) Neighborhood Description Subjective level of security in neighborhood Contact to neighbors SOEP - Non-Response Study 2010 Questionnaire 7 pages Subjective indicators Leisure time activities Political interests Voting participation Personality (Big Five) Risk aversion Subjective well-being Identification of individual refusal Specific reasons for nonparticipation in SOEP General reasons for nonparticipation in surveys Future participation Invitation to participate No future participation and delete address 19 27

20 Incentive Design of SOEP-Non- Response All Non-Participants (without non-eligible) of SOEP-Innovation-Study Total = addresses refusals (not including 33 refusals with explicit deleted address) 204 participation temporarily not possible 512 non contacts Three different versions of motivation letters 20 27

21 Incentive Design of SOEP-Non- Response Non-Response Group * Incentive Crosstabulation Non-Response Group Total refusal participatoin temporarily not possible non contacts Incentive no incentive 5 Euro 15 Euro Total ,3% 100,0% 100,0% 77,7% ,9%,0%,0% 6,3% ,8%,0%,0% 15,9% ,0% 100,0% 100,0% 100,0% 21 27

22 Incentive Designs of SOEP-Non- Response & Innovation Sample 2009 Incentive Design Innovation Subsample 2009 * Incentive Crosstabulation Incentive Design Innovation Subsample 2009 Total low cash high cash 5+10 choice - lottery or low cash lottery ticket Incentive no incentive 5 Euro 15 Euro Total ,5% 24,9% 24,0% 24,6% ,2% 23,8% 25,0% 24,2% ,3% 26,1% 24,6% 25,4% ,0% 25,2% 26,5% 25,8% ,0% 100,0% 100,0% 100,0% 22 27

23 Preliminary Results of SOEP- Non-Response final response status * Non-Response Group Crosstabulation final response status Total 1 completed short questionnaire 2 died, nursing case 4 confirmed refusal 99 up to now no information/ % within Non-Response Group % within Non-Response Group % within Non-Response Group % within Non-Response Group % within Non-Response Group Non-Response Group 2 participatoin temporarily 3 non 1 refusal not possible contacts Total ,2% 10,3% 14,5% 12,5% ,0% 1,5%,2%,2% ,0%,0%,4% 1,6% ,8% 88,2% 85,0% 85,8% ,0% 100,0% 100,0% 100,0% 23 27

24 Preliminary Results of SOEP- Non-Response only refusals final response status * Incentive Crosstabulation final response status Total completed short questionnaire died, nursing case confirmed refusal up to now no information Incentive no incentive 5 Euro 15 Euro Total ,0% 12,5% 12,3% 12,2% ,1%,0%,0%,0% ,1% 1,6% 2,4% 2,0% ,8% 85,9% 85,3% 85,8% ,0% 100,0% 100,0% 100,0% 24 27

25 Preliminary Results of SOEP-Non- Response only respondents Incentive * re-interview status Crosstabulation Incentive Design Non-Response Total no incentive 5 Euro 15 Euro re-interview status yes, ready for participation in future no Total ,2% 55,8% 100,0% ,4% 51,6% 100,0% ,9% 58,1% 100,0% ,4% 54,6% 100,0% 25 27

26 60 % Final response rate for SOEP- Innovation Sample ,9 38,4 0 Initial Response rate Initial Rate for wave 2 incl. non-response Information Final Response Rate 26 27

27 Outlook How much we must get knowing about respondents (non-respondents) in wave 1? (Socio-)demographic characteristics of nonrespondents can be controlled by comparisons with official statistics (and geo-coded data about the environment of non-respondents) non-observed characteristics of nonrespondents can be controlled by miniquestionnaires and may be short phone calls (e.g. risk aversion, personality characteristics) 27 27