Alternate Strategies for Including Cell Phone Only Households in Surveys: Comparing the Use of ABS and Cell Phone RDD Samples in Massachusetts

Transcription

1 Alternate Strategies for Including Cell Phone Only Households in Surveys: Comparing the Use of ABS and Cell Phone RDD Samples in Massachusetts David Dutwin, PhD State Health Research Policy Interest Group Academy Health Annual Research Meeting Seattle, WA June 11, 2011

2 Overview Comparison of alternate sampling strategies to capture cell phone households Address-based sample (ABS) Cell phone random digit dial sample (RDD) Case study in Massachusetts in 2010 Massachusetts Health Insurance Survey (MHIS) -- ABS and landline RDD Massachusetts Health Reform Survey (MHRS) -- Landline RDD and cell phone RDD Recommendations

3 Alternative Approaches in MA MHIS ABS & landline RDD ABS: Random sample of households from the delivery sequence file used by the United States Postal Service -- Includes households regardless of phone status Landline RDD: Random sample of landline numbers Estimated coverage rate: 98% of households MHRS Landline RDD & cell phone RDD Landline RDD: Random sample of landline numbers Cell phone RDD: Random sample of cell phone numbers Estimated coverage rate: 90 to 95% of households

4 Similarities in MHIS and MHRS Both fielded every year, including 2010 Similar questionniares focused on health insurance coverage, access and use Both draw on SHADAC s Coordinated State Coverage Survey (CSCS) Similar sample sizes Similar costs Same organizations involved in both surveys, including same survey firm

5 Differences in MHIS and MHRS Sponsor: state vs. foundation Sample design ABS & landline RDD vs. landline RDD & cell phone RDD Size of landline RDD sample Oversampling of selected populations Instrument design Data collection mode: Phone, internet & mail vs. phone Focus: all members of household vs. random adult Respondent selection Survey fielding Time period of the survey: spring vs. fall Modes of contact: mail & some phone vs. phone & some mail Use of incentives Response rates & sample coverage

6 Comparison of Selected Elements ELEMENT 2010 MHIS 2010 MHRS Sample Size Contact Method Incentives Estimated coverage rate Response rate 4,478: 1,253 landline RDD, 3,225 ABS Advance letter w/ call-in and web invitation; 2 reminders; hardcopy questionnaire; callout (if phone number available) 3,040: 2,418 landline RDD, 622 cell phone RDD Refusal letter w/ call-in invitation (if address available); call-out $20 for those without available phone number, plus Red Sox $10 for all tickets or $100 sweepstakes for all ~98% ~90-95% 40%--49% landline RDD, 37% ABS 35%--42% landline RDD, 31% cell phone RDD

7 Advantages for Cell Phone Coverage MHIS--ABS Very high coverage rates for USPS in Massachusetts Share of cell phone only households reached was close to the NHIS estimated proportion of cell phone only households in Massachusetts Bonus coverage: Also reached people who were using voice over internet phones as well as a few nontelephone households MHRS--Cell phone RDD A very high percentage of all cell phone numbers are in cell phone banks that can be sampled Based on estimates from other sources: Only about 10% of all cell phone owners in MA have non-ma area codes

8 Other Advantages MHIS--ABS Able to contact all sample members by mail Creating survey weights is straightforward using traditional survey procedures since population totals for the sample are available MHRS--Cell phone RDD Single mode for entire sample All CATI telephone interviews using the same core instrument Cell phone offers potential to reach respondents who are seldom at home

9 Disadvantages MHIS--ABS Cannot obtain phone numbers for some households so that only method of contact was mail, leading to lower response rates for those households Some addresses will be vacant or summer homes, etc. More item non-response on the mail questionnaire (12% of MHIS ABS surveys from mail) Greater operational complexity and higher costs MHRS--Cell phone RDD Cannot obtain addresses to mail advance letter Cell phones expire, not in service, business, etc. Costly to screen out respondents who do not live in Massachusetts (28% of cell phone RDD sample) Survey weights are challenging since we don t know the population totals for the cell phone population

10 Comparing All Adults in MHIS and MHRS 2010 MHIS ABS (N=2,103) 2010 MHRS RDD (N=3,040) Age 19 to Age 26 to Non-white Hispanic Female Born in the U.S Less than high school Employed Married/partner Excellent/Very good health status Fair/Poor health status Sample is adults 19 to 64; Estimates are weighted; * p <.05 Greater Boston area

11 Comparing Cell-phone-only Adults in MHIS and MHRS MHIS Target adult (N=514) MHRS Respondent (N=275) Age 19 to Age 26 to Non-white* Hispanic Female Born in the U.S Less than high school Employed Married/partner Excellent/Very good health status Fair/Poor health status Sample is adults 19 64; Estimates are weighted; * p <.05 Greater Boston area

12 Summary Despite their differences, the MHIS and MHRS yielded similar overall samples of the nonelderly adult population AND similar samples of cell-phone-only nonelderly adults in Massachusetts Both ABS and cell phone RDD appear to be credible strategies for capturing cell-phone only households However, important to remember: Since don t know actual population characteristics of the cell-phone-only population we can t assess whether missing important components of the population with both strategies

13 Recommendations To states: ABS and cell phone RDD are credible strategies for including cell-phone households in surveys To researchers/survey methodologists: Explore methods/develop strategies to increase response rates to surveys across sample frames Explore methods/develop standards for estimating consistent response rates and household coverage rates across sample frames Explore implications of mode effects for surveys that rely on multiple modes Explore methods/develop strategies for combining cell phone RDD and landline RDD samples

14 Thanks and Contact Information Acknowledgement: The MHIS was funded by the Massachusetts Division of Health Care Finance and Policy. The MHRS was funded by the Blue Cross Blue Shield of Massachusetts Foundation. We appreciate that they allowed us to use the survey data to conduct this analysis. Contacts: Nicole Buttermore: Research Director, SSRS David Dutwin: Vice President, SSRS Timothy Triplett: Survey Methodologist, Urban Institute Doug Wissoker: Senior Research Associate, Urban Institute Sharon K. Long: Professor, University of Minnesota