Experiment on Web based recruitment of Cell Phone Only respondents

Transcription

1 Experiment on Web based recruitment of Cell Phone Only respondents 2008 AAPOR Annual Conference, New Orleans By: Chintan Turakhia, Abt SRBI Inc. Mark A. Schulman, Abt SRBI Inc. Seth Brohinsky, Abt SRBI Inc. Benjamin Phillips, Brandeis University 1

2 Acknowledgements Survey Sampling International Pew Research Center for the People and Press National Center for Health Statistics (NCHS) 2

3 Objectives Explore the feasibility of Web based recruitment of Cell Phone Only (CPO) respondents as a cost effective substitute for RDD based CPO screening. Examine the validity of Web based CPO data against benchmark data. Examine the impact of demographic weighting in resolving difference against benchmark data. Key driver analysis to assess predictive quality of against benchmark data. 3

4 Methodology Used of Survey Sampling International (SSI) Internet Panel. SSI Internet Panel consists of 1.2 million panelists and has been operating since A total of about 171,000 invitations sent to SSI panelists between October 15 - October 31, 2007 in order to recruit 300 CPO respondents. Controlled sample to represent age proportions of CPO respondents of 2007 NHIS (National Health Interview Survey) data. Weighted data by gender within each age for analysis. Eliminated 55+ age category from analysis due to small sample size. Used NHIS or PEW Cell Phone Only data for benchmark comparisons depending on availability of relevant data. Age Category Gender Unweighted n Weighted n Male Female Male Female

5 Caveats Smaller sample size made exploring sub segments of the population challenging. Mode differences may have impacted some benchmark comparisons. : MODE: WEB. NHIS CPO: MODE: IN-PERSON. PEW CPO: MODE: PHONE. Survey timing may have impacted report on personal opinion attributes. 5

6 Marital Status No significant difference in Marital Status between Web CPO and NHIS CPO % 45% 41% 44% 28% 15% 15% 6% 3% NHIS CPO 2% 2% 35% 25% 15% 5% 22% 18% 7% 23% NHIS CPO 7% 7% Never married Married Living with a partner Divorced Separated/ Widowed Never married Married Living with a partner Divorced Separated/ Widowed 6

7 Racial and Ethnic Composition are LESS Hispanic and MORE Non-Hispanic White than NHIS CPO % 63% % % 11% 12% 9% 18% 8% 12% 6% 16% White (Non-Hispanic) Black (Non-Hispanic) Hispanic White (Non-Hispanic) Black (Non-Hispanic) Hispanic NHIS CPO NHIS CPO 7

8 Educational Attainment: Educational attainment is higher among as compared to NHIS CPO. 6 48% 38% 24% NHIS CPO 17% 17% 11% 5% High school graduate or less Some college College graduate Post-graduate 8

9 Educational Attainment: Educational attainment is higher among older as compared to NHIS CPO. 6 49% 46% 29% 26% 16% 14% NHIS CPO 11% 8% High school graduate or less Some college College graduate Post-graduate 9

10 are more affluent than NHIS CPO Annual Income % 6 46% 46% NHIS CPO* 35% 28% NHIS CPO* 18% 17% 9% 13% 9% 13% 13% 22% 19% Less than $35,000 $35,000-54,999 $55,000-74,999 $75,000 + Less than $35,000 $35,000-54,999 $55,000-74,999 $75, *$20,000 or more (no detail) set as system missing

11 Homeownership Young (18-34) rent significantly less than NHIS CPO respondents (72%) NHIS CPO NHIS CPO Own 24% 41% Own 49% 57% Rent 72% 51% Rent 48% 11

12 Households with Children Under 18 No significant difference in presence of children under 18 between the two samples. 37% 34% 33% NHIS CPO NHIS CPO

13 Health Status Significant differences in report on health status between and NHIS CPO. 8 75% % % NHIS CPO 44% 43% NHIS CPO 29% 21% 7% Excellent/Very good Good Fair/Poor 4% 13% Excellent/Very good Good Fair/Poor 14% 13

14 Smoking Habits: No significant difference between the two samples on life time smoking. Greater proportion younger are current non-smokers than NHIS CPO. Lifetime Smoking Behavior Have you smoked at least 100 cigarettes in your entire life? Current Smoking Behavior Do you now smoke cigarettes everyday, some days, or not at all? 7 6 NHIS CPO 58% 61% 7 6 NHIS CPO 42% 39% Of those who said Yes 43% 52% 39% 27% 18% 21% Yes No Everyday Somedays Not at all 14

15 Smoking Habits: No significant difference between the two samples among older CPO respondents in lifetime or current smoker status. Lifetime Smoking Behavior Have you smoked at least 100 cigarettes in your entire life? Current Smoking Behavior Do you now smoke cigarettes everyday, some days, or not at all? 7 63% NHIS CPO 7 NHIS CPO 6 54% 37% 45% Of those who said Yes 6 49% 56% 42% 36% 9% 9% Yes No Everyday Somedays Not at all 15

16 Registered Voters No significant difference in voter registration status between the and PEW CPO % 64% 77% 75% PEW CPO PEW CPO

17 Party Identification No significant difference in Party ID between and PEW CPO PEW CPO 45% 6 PEW CPO 51% 47% 25% 21% 29% 24% 31% 29% Republican Democrat Independent / No preference Republican Democrat Independent / No preference 17

18 Political Ideology No significant difference in Political Ideology among Older report being more Moderate in political ideology % 65% 45% 35% 25% 15% 5% 26% 31% 44% 41% PEW CPO 28% 6 55% 45% 35% 25% 15% 5% 24% 34% 57% 45% 19% PEW CPO 21% Conservative Moderate Liberal Conservative Moderate Liberal 18

19 Personal Life Significantly less are satisfied with their personal life than PEW CPO respondents % PEW CPO % PEW CPO % 46% 28% 9% 13% Satisfied Dissatisfied Satisfied Dissatisfied 19

20 Religious Service Attendance respondents attend religious service less frequently as compared to PEW CPO PEW CPO 63% 54% PEW CPO 67% 22% 29% 14% 17% 24% 35% 9% 14% At least once a week Once or twice a month Few time a year/seldom/never At least once a week Once or twice a month Few time a year/seldom/never 20

21 Importance of Religion respondents are less religious than PEW CPO respondents % 43% 6 57% 35% 32% 34% PEW CPO 36% 33% PEW CPO 25% 15% 5% 14% 18% 7% 26% 11% 7% 8% Very Somewhat Not too Not at all Very Somewhat Not too Not at all 21

22 Significant Differences: vs. Benchmark Data Demographic Variables Demographic Characteristics Marital Status Racial and Ethnic Composition Educational Attainment Annual Income Homeownership Households with Children 22 - No significant differences exist Significant differences exist

23 Significant Differences: vs. Benchmark Data Substantive Variables Substantive Characteristics Registered Voters Party Identification Political Ideology Smoking Habits Personal Life Satisfaction Health Status Religious Service Attendance 23 - No significant differences exist Significant differences exist Importance of Religion

24 Sample Balanced Results (Raking) In an effort to resolve differences between the two samples, we weighted data to reflect NHIS demographic characteristics on: Gender Race Education 24

25 Sample Balanced: Registered Voters Improvement in group, gap widened for group % 67% 64% 77% 69% 75% 6 Pre Weight Post Weight PEW CPO Pre Weight Post Weight PEW CPO

26 Sample Balanced: Party Identification Gap widened for Republicans and Independents among group Pre Weight Post Weight PEW CPO 45% 49% 6 Pre Weight Post Weight PEW CPO 51% 55% 47% 25% 21% 21% 29% 14% 24% 31% 32% 29% Republican Democrat Independent / No preference Republican Democrat Independent / No preference 26

27 Sample Balanced: Political Ideology Gap Widened for Moderates for both age groups % 24% 31% 44% 41% Pre Weight Post Weight PEW CPO 26% 28% 65% 6 55% 45% 35% 25% 24% 34% 57% 62% 45% 19% Pre Weight Post Weight PEW CPO 18% 21% 15% Conservative Moderate Liberal 5% Conservative Moderate Liberal 27

28 Sample Balanced: Smoking Habits: No significant difference between Pre-weight and Post-weight. Lifetime Smoking Behavior: Have you smoked at least 100 cigarettes in your entire life? Current Smoking Behavior: Do you now smoke cigarettes everyday, some days, or not at all? % 45% Pre Weight Post Weight NHIS CPO 39% 58% 54% 61% Of those who said Yes % 44% 52% Pre Weight Post Weight NHIS CPO 39% 38% 27% 18% 18% 21% Yes No Everyday Somedays Not at all 28

29 Sample Balanced: Smoking Habits: Current Smokers: Improvement in Non-smokers, Gap widened for Infrequent Smokers Lifetime Smoking Behavior: Have you smoked at least 100 cigarettes in your entire life? Current Smoking Behavior: Do you now smoke cigarettes everyday, some days, or not at all? % 67% 54% Pre Weight Post Weight NHIS CPO % Pre Weight Post Weight NHIS CPO 37% 33% 45% Of those who said Yes 49% 49% 42% 36% 36% 15% 9% 9% Yes No Everyday Somedays Not at all 29

30 Sample Balanced: Health Status No significant difference between Pre-weight and Post-weight % % 61% 31% 21% Pre Weight Post Weight NHIS CPO 7% 9% 4% % 44% 42% 43% 44% 29% Pre Weight Post Weight NHIS CPO 13% 14% 14% Excellent/Very good Good Fair/Poor Excellent/Very good Good Fair/Poor 30

31 Sample Balanced: Religious Service Attendance Gap widened for At least once a week and Few times a Year among older Pre Weight Post Weight PEW CPO 9 8 Pre Weight Post Weight PEW CPO 78% 7 63% 67% 7 67% 6 54% 6 29% 22% At least once a week 14% 12% 17% Once or twice a month Few time a year/seldom/never 35% 24% 19% At least once a week 9% 3% 14% Once or twice a month Few time a year/seldom/never 31

32 Sample Balanced: Importance of Religion No significant difference between Pre-weight and Post-weight % 32% 32% 43% 34% 9% 14% Pre Weight Post Weight PEW CPO 18% 17% 7% 36% 33% 33% 31% 26% 13% 11% 7% Pre Weight Post Weight PEW CPO 24% 8% Very Somewhat Not too Not at all Very Somewhat Not too Not at all 32

33 Significant Differences: PRE and POST WEIGHT COMPARISON Demographic Variables Demographic Characteristics Pre Weight Post Weight Pre Weight Post Weight Marital Status Racial and Ethnic Composition Educational Attainment Annual Income Homeownership Households with Children 33 - No significant differences exist Significant differences exist

34 Significant Differences: PRE and POST WEIGHT COMPARISON Substantive Variables Substantive Characteristics Pre Weight Post Weight Pre Weight Post Weight Registered Voters Party Identification Political Ideology Smoking Habits Health Status Religious Service Attendance Importance of Religion 34 - No significant differences exist Significant differences exist

35 CPO Key Driver Analysis In assessing the predictive quality of panel against benchmark data, regression analysis was run for: Predictors of Smoking Ordinal logistic regression Predictors of Health Status Ordinal logistic regression Predictors of Voter Registration Binary logistic regression 35

36 Dependent variable 1 = Never smoke 2 = Smoke some days 3 = Smoke every day Smoking Ordinal Logistic Regression Model developed using NHIS data then applied to Web sample Web CPO and NHIS CPO samples show similar direction for all attributes as predictors of smoking behavior. No significant difference between the two sample in key drivers of smoking behavior ECEPT: Web CPO college graduates are 57% less likely to be smokers than less than college group whereas NHIS CPO college graduates are 75% less likely to be smokers than less than college group. 36

37 Predictors of Smoking Variable Web NHIS Suppressed Category Female * Male Age College degree.306***.428**.581***.252*** Age LT college degree Nonwhite Own West ***.684**.682** White Rent and other Northeast, South, Midwest p <.1 * p <.05 ** p <.01 *** p <.001 Notes: Ordinal logistic regression. Odds ratios shown. Red circles indicate significant differences between coefficients. Where multiple categories included in suppressed category of dummy variable, no significant differences between coefficients were found in the full model. 37

38 Health Status Dependent variable ( Would you say that in general your health is ) 1 = Fair/poor 2 = Good 3 = Very good Ordinal Logistic Regression Model developed using NHIS data then applied to Web sample. and NHIS CPO samples show similar direction for most attributes as predictors of Health Status. No significant difference between the two samples in key drivers of Health Status except: Hispanics are 1 MORE likely to report very good health than all other races. Whereas NHIS CPO Hispanics are LESS likely to report very good health than other races. 38

39 Predictors of Health Status Variable Web NHIS Suppressed Category Female.608*.715*** Male Age **.490*** Age Age *** Age Hispanic 2.405* **.783* ** All other races Smoke.730*.817*** (ordinal measure) Some college *** HS or less College degree 2.424* 3.179*** HS or less p <.1 * p <.05 ** p <.01 *** p <.001 Notes: Ordinal logistic regression. Odds ratios shown. Red circles indicate significant differences between coefficients. Where multiple categories included in suppressed category of dummy variable, no significant differences between coefficients were found in the full model. 39

40 Dependent variable 0 = Not registered voter 1 = Registered voter Registered voter Binary Logistic Regression Model developed using Pew data then applied to Web sample and PEW CPO samples show similar direction for most attributes as predictors of Voter Registration. No significant difference between the two samples in key drivers of Voter Registration except: Females are LESS likely to be a registered voter than Males. Whereas PEW CPO Females are 86% MORE likely to be a registered voter. 40

41 Predictors of Registered Voter Variable Web Pew Suppressed Category Female Age * *** 1.860** *** ** Male Age Some college ** HS or less College degree 4.320** 2.584** HS or less Graduate degree 4.045* 4.843** HS or less Independent.364***.446*** Republican or Democrat p <.1 * p <.05 ** p <.01 *** p <.001 Notes: Ordinal logistic regression. Odds ratios shown. Red circles indicate significant differences between coefficients. Where multiple categories included in suppressed category of dummy variable, no significant differences between coefficients were found in the full model. 41

42 CONCLUSION Significant differences found between and Benchmark CPO data in Health Status and Religion attributes. These differences could not be resolved by demographic weighting. exhibit greater similarity with benchmark data in Political attributes. This offers potential for use of in political research. Regression Analysis reveals general agreement in sign and magnitude of coefficients between Web CPO and benchmark data ECEPT two cases where differences between the two samples pointed in opposite direction. is significantly cheaper (CPI $35) as compared to Phone CPO (CPI $200). As more high quality data on CPO population become available, we will be able to test more dependent variables, expanding knowledge about ways in which Web panel CPO population differs from CPO population as a whole and ways to resolve these differences. 42

43 43 Thank you Contact: Chintan Turakhia