Experiment on Web based recruitment of Cell Phone Only respondents

Experiment on Web based recruitment of Cell Phone Only respondents 2008 AAPOR Annual Conference, New Orleans By: Chintan Turakhia, Abt SRBI Inc. Mark A. Schulman, Abt SRBI Inc. Seth Brohinsky, Abt SRBI Inc. Benjamin Phillips, Brandeis University 1

Acknowledgements Survey Sampling International Pew Research Center for the People and Press National Center for Health Statistics (NCHS) 2

Objectives Explore the feasibility of Web based recruitment of Cell Phone Only (CPO) respondents as a cost effective substitute for RDD based CPO screening. Examine the validity of Web based CPO data against benchmark data. Examine the impact of demographic weighting in resolving difference against benchmark data. Key driver analysis to assess predictive quality of against benchmark data. 3

Methodology Used of Survey Sampling International (SSI) Internet Panel. SSI Internet Panel consists of 1.2 million panelists and has been operating since 2000. A total of about 171,000 invitations sent to SSI panelists between October 15 - October 31, 2007 in order to recruit 300 CPO respondents. Controlled sample to represent age proportions of CPO respondents of 2007 NHIS (National Health Interview Survey) data. Weighted data by gender within each age for analysis. Eliminated 55+ age category from analysis due to small sample size. Used NHIS or PEW Cell Phone Only data for benchmark comparisons depending on availability of relevant data. Age Category Gender Unweighted n Weighted n 18-34 Male 87 98 Female 99 92 35-54 Male 40 51 Female 48 35 4

Caveats Smaller sample size made exploring sub segments of the population challenging. Mode differences may have impacted some benchmark comparisons. : MODE: WEB. NHIS CPO: MODE: IN-PERSON. PEW CPO: MODE: PHONE. Survey timing may have impacted report on personal opinion attributes. 5

Marital Status No significant difference in Marital Status between Web CPO and NHIS CPO. 18-34 35-54 6 49% 45% 41% 44% 28% 15% 15% 6% 3% NHIS CPO 2% 2% 35% 25% 15% 5% 22% 18% 7% 23% NHIS CPO 7% 7% Never married Married Living with a partner Divorced Separated/ Widowed Never married Married Living with a partner Divorced Separated/ Widowed 6

Racial and Ethnic Composition are LESS Hispanic and MORE Non-Hispanic White than NHIS CPO. 9 8 7 6 72% 63% 18-34 35-54 9 81% 8 7 6 64% 11% 12% 9% 18% 8% 12% 6% 16% White (Non-Hispanic) Black (Non-Hispanic) Hispanic White (Non-Hispanic) Black (Non-Hispanic) Hispanic NHIS CPO NHIS CPO 7

Educational Attainment: 18-34 Educational attainment is higher among as compared to NHIS CPO. 6 48% 38% 24% NHIS CPO 17% 17% 11% 5% High school graduate or less Some college College graduate Post-graduate 8

Educational Attainment: 35-54 Educational attainment is higher among older as compared to NHIS CPO. 6 49% 46% 29% 26% 16% 14% NHIS CPO 11% 8% High school graduate or less Some college College graduate Post-graduate 9

are more affluent than NHIS CPO Annual Income 18-34 35-54 6 59% 6 46% 46% NHIS CPO* 35% 28% NHIS CPO* 18% 17% 9% 13% 9% 13% 13% 22% 19% Less than $35,000 $35,000-54,999 $55,000-74,999 $75,000 + Less than $35,000 $35,000-54,999 $55,000-74,999 $75,000 + 10 *$20,000 or more (no detail) set as system missing

Homeownership Young (18-34) rent significantly less than NHIS CPO respondents (72%). 18-34 35-54 NHIS CPO NHIS CPO Own 24% 41% Own 49% 57% Rent 72% 51% Rent 48% 11

Households with Children Under 18 No significant difference in presence of children under 18 between the two samples. 37% 34% 33% NHIS CPO NHIS CPO 12 18-34 35-54

Health Status Significant differences in report on health status between and NHIS CPO. 8 75% 18-34 35-54 6 57% 7 6 63% NHIS CPO 44% 43% NHIS CPO 29% 21% 7% Excellent/Very good Good Fair/Poor 4% 13% Excellent/Very good Good Fair/Poor 14% 13

Smoking Habits: 18-34 No significant difference between the two samples on life time smoking. Greater proportion younger are current non-smokers than NHIS CPO. Lifetime Smoking Behavior Have you smoked at least 100 cigarettes in your entire life? Current Smoking Behavior Do you now smoke cigarettes everyday, some days, or not at all? 7 6 NHIS CPO 58% 61% 7 6 NHIS CPO 42% 39% Of those who said Yes 43% 52% 39% 27% 18% 21% Yes No Everyday Somedays Not at all 14

Smoking Habits: 35-54 No significant difference between the two samples among older CPO respondents in lifetime or current smoker status. Lifetime Smoking Behavior Have you smoked at least 100 cigarettes in your entire life? Current Smoking Behavior Do you now smoke cigarettes everyday, some days, or not at all? 7 63% NHIS CPO 7 NHIS CPO 6 54% 37% 45% Of those who said Yes 6 49% 56% 42% 36% 9% 9% Yes No Everyday Somedays Not at all 15

Registered Voters No significant difference in voter registration status between the and PEW CPO. 10 9 8 7 6 72% 64% 77% 75% PEW CPO PEW CPO 16 18-34 35-54

Party Identification No significant difference in Party ID between and PEW CPO. 18-34 35-54 6 PEW CPO 45% 6 PEW CPO 51% 47% 25% 21% 29% 24% 31% 29% Republican Democrat Independent / No preference Republican Democrat Independent / No preference 17

Political Ideology No significant difference in Political Ideology among 18-34. Older report being more Moderate in political ideology. 18-34 35-54 6 55% 65% 45% 35% 25% 15% 5% 26% 31% 44% 41% PEW CPO 28% 6 55% 45% 35% 25% 15% 5% 24% 34% 57% 45% 19% PEW CPO 21% Conservative Moderate Liberal Conservative Moderate Liberal 18

Personal Life Significantly less are satisfied with their personal life than PEW CPO respondents. 18-34 35-54 10 10 9 8 7 7 86% PEW CPO 9 8 7 85% PEW CPO 6 6 54% 46% 28% 9% 13% Satisfied Dissatisfied Satisfied Dissatisfied 19

Religious Service Attendance respondents attend religious service less frequently as compared to PEW CPO. 18-34 35-54 8 7 6 PEW CPO 63% 54% 9 8 7 6 PEW CPO 67% 22% 29% 14% 17% 24% 35% 9% 14% At least once a week Once or twice a month Few time a year/seldom/never At least once a week Once or twice a month Few time a year/seldom/never 20

Importance of Religion respondents are less religious than PEW CPO respondents. 18-34 35-54 45% 43% 6 57% 35% 32% 34% PEW CPO 36% 33% PEW CPO 25% 15% 5% 14% 18% 7% 26% 11% 7% 8% Very Somewhat Not too Not at all Very Somewhat Not too Not at all 21

Significant Differences: vs. Benchmark Data Demographic Variables Demographic Characteristics 18-34 35-54 Marital Status Racial and Ethnic Composition Educational Attainment Annual Income Homeownership Households with Children 22 - No significant differences exist Significant differences exist

Significant Differences: vs. Benchmark Data Substantive Variables Substantive Characteristics 18-34 35-54 Registered Voters Party Identification Political Ideology Smoking Habits Personal Life Satisfaction Health Status Religious Service Attendance 23 - No significant differences exist Significant differences exist Importance of Religion

Sample Balanced Results (Raking) In an effort to resolve differences between the two samples, we weighted data to reflect NHIS demographic characteristics on: Gender Race Education 24

Sample Balanced: Registered Voters Improvement in 18-34 group, gap widened for 35-54 group. 10 9 8 7 72% 67% 64% 77% 69% 75% 6 Pre Weight Post Weight PEW CPO Pre Weight Post Weight PEW CPO 25 18-34 35-54

Sample Balanced: Party Identification Gap widened for Republicans and Independents among 35-54 group. 18-34 35-54 6 Pre Weight Post Weight PEW CPO 45% 49% 6 Pre Weight Post Weight PEW CPO 51% 55% 47% 25% 21% 21% 29% 14% 24% 31% 32% 29% Republican Democrat Independent / No preference Republican Democrat Independent / No preference 26

Sample Balanced: Political Ideology Gap Widened for Moderates for both age groups. 18-34 35-54 6 26% 24% 31% 44% 41% Pre Weight Post Weight PEW CPO 26% 28% 65% 6 55% 45% 35% 25% 24% 34% 57% 62% 45% 19% Pre Weight Post Weight PEW CPO 18% 21% 15% Conservative Moderate Liberal 5% Conservative Moderate Liberal 27

Sample Balanced: Smoking Habits: 18-34 No significant difference between Pre-weight and Post-weight. Lifetime Smoking Behavior: Have you smoked at least 100 cigarettes in your entire life? Current Smoking Behavior: Do you now smoke cigarettes everyday, some days, or not at all? 7 6 42% 45% Pre Weight Post Weight NHIS CPO 39% 58% 54% 61% Of those who said Yes 7 6 43% 44% 52% Pre Weight Post Weight NHIS CPO 39% 38% 27% 18% 18% 21% Yes No Everyday Somedays Not at all 28

Sample Balanced: Smoking Habits: 35-54 Current Smokers: Improvement in Non-smokers, Gap widened for Infrequent Smokers Lifetime Smoking Behavior: Have you smoked at least 100 cigarettes in your entire life? Current Smoking Behavior: Do you now smoke cigarettes everyday, some days, or not at all? 7 6 63% 67% 54% Pre Weight Post Weight NHIS CPO 7 6 56% Pre Weight Post Weight NHIS CPO 37% 33% 45% Of those who said Yes 49% 49% 42% 36% 36% 15% 9% 9% Yes No Everyday Somedays Not at all 29

Sample Balanced: Health Status No significant difference between Pre-weight and Post-weight. 18-34 35-54 8 75% 7 6 63% 61% 31% 21% Pre Weight Post Weight NHIS CPO 7% 9% 4% 8 7 6 57% 44% 42% 43% 44% 29% Pre Weight Post Weight NHIS CPO 13% 14% 14% Excellent/Very good Good Fair/Poor Excellent/Very good Good Fair/Poor 30

Sample Balanced: Religious Service Attendance Gap widened for At least once a week and Few times a Year among older. 18-34 35-54 9 8 Pre Weight Post Weight PEW CPO 9 8 Pre Weight Post Weight PEW CPO 78% 7 63% 67% 7 67% 6 54% 6 29% 22% At least once a week 14% 12% 17% Once or twice a month Few time a year/seldom/never 35% 24% 19% At least once a week 9% 3% 14% Once or twice a month Few time a year/seldom/never 31

Sample Balanced: Importance of Religion No significant difference between Pre-weight and Post-weight. 18-34 35-54 6 6 57% 32% 32% 43% 34% 9% 14% Pre Weight Post Weight PEW CPO 18% 17% 7% 36% 33% 33% 31% 26% 13% 11% 7% Pre Weight Post Weight PEW CPO 24% 8% Very Somewhat Not too Not at all Very Somewhat Not too Not at all 32

Significant Differences: PRE and POST WEIGHT COMPARISON Demographic Variables Demographic Characteristics 18-34 Pre Weight 18-34 Post Weight 35-54 Pre Weight 35-54 Post Weight Marital Status Racial and Ethnic Composition Educational Attainment Annual Income Homeownership Households with Children 33 - No significant differences exist Significant differences exist

Significant Differences: PRE and POST WEIGHT COMPARISON Substantive Variables Substantive Characteristics 18-34 Pre Weight 18-34 Post Weight 35-54 Pre Weight 35-54 Post Weight Registered Voters Party Identification Political Ideology Smoking Habits Health Status Religious Service Attendance Importance of Religion 34 - No significant differences exist Significant differences exist

CPO Key Driver Analysis In assessing the predictive quality of panel against benchmark data, regression analysis was run for: Predictors of Smoking Ordinal logistic regression Predictors of Health Status Ordinal logistic regression Predictors of Voter Registration Binary logistic regression 35

Dependent variable 1 = Never smoke 2 = Smoke some days 3 = Smoke every day Smoking Ordinal Logistic Regression Model developed using NHIS data then applied to Web sample Web CPO and NHIS CPO samples show similar direction for all attributes as predictors of smoking behavior. No significant difference between the two sample in key drivers of smoking behavior ECEPT: Web CPO college graduates are 57% less likely to be smokers than less than college group whereas NHIS CPO college graduates are 75% less likely to be smokers than less than college group. 36

Predictors of Smoking Variable Web NHIS Suppressed Category Female.962.790* Male Age 18-24 College degree.306***.428**.581***.252*** Age 25-54 LT college degree Nonwhite Own West.539.836.982.457***.684**.682** White Rent and other Northeast, South, Midwest p <.1 * p <.05 ** p <.01 *** p <.001 Notes: Ordinal logistic regression. Odds ratios shown. Red circles indicate significant differences between coefficients. Where multiple categories included in suppressed category of dummy variable, no significant differences between coefficients were found in the full model. 37

Health Status Dependent variable ( Would you say that in general your health is ) 1 = Fair/poor 2 = Good 3 = Very good Ordinal Logistic Regression Model developed using NHIS data then applied to Web sample. and NHIS CPO samples show similar direction for most attributes as predictors of Health Status. No significant difference between the two samples in key drivers of Health Status except: Hispanics are 1 MORE likely to report very good health than all other races. Whereas NHIS CPO Hispanics are LESS likely to report very good health than other races. 38

Predictors of Health Status Variable Web NHIS Suppressed Category Female.608*.715*** Male Age 35-44.419**.490*** Age 25-34 Age 45-54.498.320*** Age 25-34 Hispanic 2.405* **.783* ** All other races Smoke.730*.817*** (ordinal measure) Some college 1.319 1.488*** HS or less College degree 2.424* 3.179*** HS or less p <.1 * p <.05 ** p <.01 *** p <.001 Notes: Ordinal logistic regression. Odds ratios shown. Red circles indicate significant differences between coefficients. Where multiple categories included in suppressed category of dummy variable, no significant differences between coefficients were found in the full model. 39

Dependent variable 0 = Not registered voter 1 = Registered voter Registered voter Binary Logistic Regression Model developed using Pew data then applied to Web sample and PEW CPO samples show similar direction for most attributes as predictors of Voter Registration. No significant difference between the two samples in key drivers of Voter Registration except: Females are LESS likely to be a registered voter than Males. Whereas PEW CPO Females are 86% MORE likely to be a registered voter. 40

Predictors of Registered Voter Variable Web Pew Suppressed Category Female Age 35-54.500* *** 1.860** *** 1.074 1.929** Male Age 25-34 Some college 1.800 2.191** HS or less College degree 4.320** 2.584** HS or less Graduate degree 4.045* 4.843** HS or less Independent.364***.446*** Republican or Democrat p <.1 * p <.05 ** p <.01 *** p <.001 Notes: Ordinal logistic regression. Odds ratios shown. Red circles indicate significant differences between coefficients. Where multiple categories included in suppressed category of dummy variable, no significant differences between coefficients were found in the full model. 41

CONCLUSION Significant differences found between and Benchmark CPO data in Health Status and Religion attributes. These differences could not be resolved by demographic weighting. exhibit greater similarity with benchmark data in Political attributes. This offers potential for use of in political research. Regression Analysis reveals general agreement in sign and magnitude of coefficients between Web CPO and benchmark data ECEPT two cases where differences between the two samples pointed in opposite direction. is significantly cheaper (CPI $35) as compared to Phone CPO (CPI $200). As more high quality data on CPO population become available, we will be able to test more dependent variables, expanding knowledge about ways in which Web panel CPO population differs from CPO population as a whole and ways to resolve these differences. 42

43 Thank you Contact: Chintan Turakhia (c.turakhia@srbi.com)