1 Guidelines & Considerations: Telephone Surveys of Respondents Reached via Cell Phone AAPOR Cell Phone Task Force: Paul J. Lavrakas Chair, Charlotte Steeh, Stephen Blumberg, John Boyle, Michael Brick, Mario Callegaro, Howard Fienberg, Anna Fleeman, Donna Gillin, John Hall, Scott Keeter, Courtney Kennedy, Michael Link, Linda Piekarski, Chuck Shuttles, and Trevor Thomson
4 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Subcommittee Members: Chair Linda Piekarski,, Survey Sampling Intl. Stephen J Blumberg, U.S. Centers for Disease Control and Prevention John Boyle, Abt SRBI Inc. Michael Brick, Westat Scott Keeter, Pew Research Center
5 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Cell Phone Sample is available from most sample suppliers Frames are constructed from industry databases Administrative data and subject to some error Contain information on the types of service for prefixes and 1000-blocks where pooling is in effect Contain Rate Center and Service Provider information
6 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Just a frame of possible cell phone numbers No database of directory listed numbers No address information No information on where subscriber resides Don t t know if subscriber also has landline service or is Cell Phone Only No demographic information No Measure of Size for Cell or Cell-Only population below Census Region
7 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Just a frame of possible cell phone numbers Can t t identify working numbers or blocks Can t t screen for disconnects (TCPA) Can t t identify business numbers Can t t identify pre-paid paid phone cards or disposable phones Can t t identify cell-only
8 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Types of service that might contain cell phone numbers: 04 Dedicated to Cellular 55,60 Special/Selective Billing Option Cellular 65 Miscellaneous Service (non-500 PCS) 67,68 Special/Selective Billing Options PCS 58,63 Special/Selective Billing Options - Cell, paging, mobile 50 Shared 3 or more POTS, Cellular, Paging, etc. 54 Shared POTS and Cellular 66 Shared POTS and PCS (non-500)
9 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Frame considerations when purchasing sample Is the frame based on prefixes, 1000-blocks or 100-blocks? Prefix restricted frames have more coverage error What types of service are included: dedicated, shared, cellular, PCS, special billing? Dedicated service frames have more coverage error Shared service only means multiple services provided by single service provider
10 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Prefix level information inaccurate coverage error A BELLSOUTH SO CNTL CENTREVL ALA BELLSOUTH SO CNTL CENTREVL AL NEXTEL COMM INC CENTREVL AL NEXTEL COMM INC CENTREVL AL NEXTEL COMM INC CENTREVL AL SOUTHERN COMM SVCS CENTREVL AL NEXTEL COMM INC CENTREVL AL NEXTEL COMM INC CENTREVL AL NEXTEL COMM INC CENTREVL AL Prefix Type: 00 = POTS; 04 = Cellular Company Type: 0=BOC; 5 =Cellular Service Provider
11 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Prefix level information inaccurate potential overlap A NEXTEL COMM INC CENTREVL AL NEXTEL COMM INC CENTREVL BELLSOUTH SO CNTL CENTREVL BELLSOUTH SO CNTL CENTREVL BELLSOUTH SO CNTL CENTREVL BELLSOUTH SO CNTL CENTREVL BELLSOUTH SO CNTL CENTREVL BELLSOUTH SO CNTL CENTREVL BELLSOUTH SO CNTL CENTREVL NEXTEL COMM INC CENTREVL Prefix Type: 00 = POTS; 04 = Cellular Company Type: 0=BOC; 5 =Cellular Service Provider
12 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Frame considerations when purchasing sample How are shared service numbers handled? May be identified only at prefix or 1000-block Potential for coverage error or overlap with RDD frame What levels of geography are available? FIPS State and County is lowest level for which coding can be done - based on location of Rate Center How was coding accomplished? Estimate of precision of geo-coding?
13 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Frame considerations when purchasing sample Are there alternatives for including landline numbers ported to wireless service? NeuStar s Intermodal Ported TN Identification Service license limits use to scrubbing or efforts to comply with TCPA regulations prohibiting cells to cell phones using automated telephone equipment. May not be used to construct or enhance a cell phone frame.
14 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Coverage considerations for cell phone samples National coverage of cell phone numbers is excellent It is much more important than for landline RDD to determine residence of respondent Prefix geography areas are much larger than for landline prefixes Respondent is more likely not to live in the county of the Rate Center Subscribers can move to a different city or state and keep their phone number
15 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Defining sample geography by county can result in coverage error when: No rate center is located in one or more of the sampled counties Unknown number of subscribers live in a county not in the sampled geography Unknown number of subscribers live in a sampled county but have phone numbers in a rate center in a county not being sampled Subscribers can move to a different city or state and keep their phone number
16 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Integrating sample from RDD and Cell frames No consensus on best approach for combining sample from both frames Population frame vs. household frame Situation of households that can be reached by both a landline and a cell phone These topics will be covered in more depth in the weighting section
17 CONSIDERATIONS REGARDING COVERAGE AND SAMPLING Researchers should obtain enough cell phone cases, or cell phone only cases, so that it is not necessary to apply large weights to cell or cell only cases
18 CONSIDERATIONS REGARDING NONRESPONSE
19 CONSIDERATIONS REGARDING NONRESPONSE Subcommittee Members: Chair Anna Fleeman,, Arbitron Mario Callegaro,, Knowledge Networks Michael Link, Nielsen Chuck Shuttles, Nielsen Charlotte Steeh, CDC
20 CONSIDERATIONS REGARDING NONRESPONSE Problematic aspect of cell surveys is low response rates At least points below landline Need to approach problem in three ways: Identify true sources of nonresponse Use strategies that reduce nonresponse Modify the AAPOR RR calculations to address differing features of CP surveys
21 CONSIDERATIONS REGARDING NONRESPONSE Types of Nonresponse akin to landline Noncontacts Refusals Other noninterviews Undetermined eligibility But effect on survey and response are quite different
22 NONCONTACTS Most noncontacts perform similarly in landline and cell phone surveys, yet VMs much higher with cell As more people make their cell phone the primary contact, the noncontact rates are expected to decrease
23 REFUSALS Respondent pays for call or uses minutes May be engaged in an activity not conducive to answering phone or talking Cell phones are private and/or personal More difficult to convert refusals in cell phone - not reaching another HH member
24 UNDETERMINED ELIGIBILITY Cell phones can be used for business and/or personal Or, used only for emergencies Operator messages often unclear and confusing Number not accepting calls at this time Please enter your mailbox number
25 OPERATIONAL CONSIDERATIONS TO IMPROVE RESPONSE Calling Protocols Safety concerns Time zone issues Voic messages Caller ID and callbacks Refusal conversions Reimbursement & Incentives
26 NONRESPONSE SUMMARY Nonresponse greater in cell surveys Ensure best practices and tested methodologies are used Disclose sample types LL and/or Cell Phone surveys targeting high CPO demo should include cell numbers If not, indicate why and how excluding may affect results
27 CONCLUSIONS Cell phone sampling is a necessity in survey research Coverage bias must be reduced for estimates to remain reliable And, the good news Survey research using cell phone samples has been going on since 2002 and has indeed been successful!
28 LEGAL & ETHICAL CONSIDERATIONS
29 LEGAL & ETHICAL CONSIDERATIONS Subcommittee Members: Chair Howard Feinberg, CMOR Stephen J. Blumberg, CDC Chuck Shuttles, Nielsen Trevor Thompson, AP
30 Ethical & Legal DISCLAIMER The information provided in this presentation is not a substitute for legal advice. It is provided for informational purposes only. It is advisable to consult with private legal counsel regarding the scope and application of any laws.
31 Legal Issues Directly Related to Cell Phone Calls Federal Law Federal Telephone Consumer Protection Act (TCPA) Applies to both intra-state calls (calls made and originating from within the same state), as well as interstate calls (calls from one state to another). Autodialer/Cell Phone Provisions restrict calls for research purposes!
32 Telephone Consumer Protection Act The Concern for Survey Research ANY call made (without( consent) ) using an automatic dialing device to a cellular telephone number, when the party is charged for the call Automatic telephone dialing system as equipment that has the capacity to store or produce telephone numbers to be called using a random or sequential number generator and to dial such numbers. It includes predictive dialers!!
33 How to Comply With the Current TCPA Rules? Permissible Calls: If placed manually by telephone interviewers (instead of using an automatic telephone dialing system) If the called party is not charged for the call If the call is made with consent of the called party Remove cell phone numbers from the sample
34 Other Legal & Ethical Issues Call Back Attempts State harassment laws Caller ID Time of Day 8 9pm
35 Other Ethical Considerations Remember the MOBILE in mobile phone! Driving a car Are you in a place where you can safely talk on the phone and answer my questions?
36 Other Ethical Considerations Respondent answering a cell phone in a public place (protect their privacy) Time constraints for cell phone calls Minutes/cost/expectations expectations
37 More Information CMOR - Telephone Research Resources FCC/FTC Fact Sheets Fact Sheet on Predictive Dialers - FCC Fact Sheet with Information about the Cell Phone Provisions of the TCPA - Fact Sheets about Cell Phones and the Do-not not-call registry s.html; Donna Gillin ;
38 CONSIDERATIONS REGARDING MEASUREMENT
39 CONSIDERATIONS MEASUREMENT Subcommittee Members: Chair Scott Keeter, Pew Research Center Patrick Glaser, CMOR Paul Lavrakas, Independent Consultant Charlotte Steeh, CDC
40 CONSIDERATIONS MEASUREMENT Overview Additional survey items Dual frame and weighting purposes Data Quality Is data quality lower for cell phone respondents versus landline demographically similar respondents, and if so, under what circumstances?
41 CONSIDERATIONS MEASUREMENT Additional Survey Items What (if any) additional items are needed? Merging with landline frame survey Weighting Key analyses This is a measurement issue, but will be addressed in weighting section (next)
42 CONSIDERATIONS MEASUREMENT Data Quality Overview Unique nature of a cell phone call Interviewer and respondent interaction Data quality impacts?
43 CONSIDERATIONS MEASUREMENT Data Quality Initial Findings* No evidence to show significant quality differences to landline Steeh (2005) item nonresponse,, correlations, demographic distribution Brick et al. (2007) missing data, length of open-ended, ended, or response to sensitive items Also see Kuusela,, Callegaro, & Vehovar (2007) and Kennedy (2007) * What we know currently
44 CONSIDERATIONS MEASUREMENT Consider that the quality of respondents answers may be affected by: Respondents location work, bus, with spouse, etc. Research opportunity data quality indicators by dichotomous home vs. out of home survey call Interviewer training (e.g., determining location, refusal conversion, etc.) Sensitive Survey Topics sexually transmitted disease, race-related related attitudes, income, etc.
45 CONSIDERATIONS MEASUREMENT Consider that the quality of respondent s answers may be affected by: Connection bad connection, ambient noise Phone Equipment lower volume
46 CONSIDERATIONS MEASUREMENT Measurement Overall Summary Additional items may be needed for dual frame and other analyses Data Quality No significant differences to landline surveys detected yet Stay vigilant more research is needed Interviewer training specific to cell phone surveys could help
47 CONSIDERATIONS REGARDING WEIGHTING
48 CONSIDERATIONS REGARDING WEIGHTING Subcommittee Members: Chair John Hall, Mathematica Policy Research John Boyle, Abt SRBI Inc. Mike Brick, Westat Mario Callegaro, Knowledge Networks Courtney Kennedy, University of Michigan Linda Pierkarski, Survey Sampling Intl.
49 OVERVIEW OF WEIGHTING CONSIDERATIONS Types of studies considered Question researchers should ask When are weights needed (in general) Factors affecting weighting choices
50 TYPES OF STUDIES THESE CONSIDERATIONS ADDRESS Samples selected from RDD frames landline cell, or both Study population comprises Households and/or Residents of households
51 INITIAL QUESTIONS ABOUT WEIGHTING Are weights needed? How does the study design affect the weighting approach? What variables should be used for post- stratification? What other issues must be considered in weighting? What data will the survey need to collect?
52 WHEN ARE WEIGHTS REQUIRED Generally weights are used to compensate for: differential probabilities of selection (oversampling) differential propensities to respond sampling frame coverage problems Would almost always be required in dual frame (landline and cell) design Especially if those with both services interviewed from both frames
53 WHEN ARE WEIGHTS REQUIRED (CONT.) Examination of unweighted frequencies is advisable even if weights are to be used Researchers should report Whether weights were used Why weights were or were not used Weighting procedures used (if any were used)
54 FACTORS AFFECTING WEIGHTING DECISION Population being studied (as defined by telephone service) No phone service Cell only Landline only Landline and cell service
55 FACTORS AFFECTING WEIGHTING DECISION (CONT.) Design approaches Landline and cell frames used, but screening on type of service eliminates multiplicity Landline and cell frames are used and there is no screening
56 FACTORS AFFECTING WEIGHTING DECISION (CONT.) Design approaches (continued) Only a landline frame is used but cell only group is part of study population Only one frame is used and inference is only to those on that frame
57 DIFFERENT WEIGHTING ADJUSTMENTS Probabilities Study Sampling of Selection Population Frame(s) Within Frame Cell only Cell At phone level; at person level depending on approach to "sharing" All phone (plus non phone) All phone (plus non phone) Landline Landline and cell Within household selection if any See above for frame specific Adjustments for Differences In Within Frame Multiplicity Number of cell phones Number of landlines Number of cell phones; number of landlines Multiplicity Across Frames Not Applicable Not Applicable Depends on Screening Rule See next "slide" on Multiplicity Service Interruption Possibly "Keeter," modified to cover periods of cell only May use Keeter or modified Keeter Response Rates Among strata or other groups Among strata or other groups Within frame; among strata or other groups Post Stratification Demographic and geography if data are available By whether had interruption if data available; by age and other factors correlated with phone usage By phone usage if data available; by age and other factors correlated with phone usage
58 APPROACHES TO DEALING WITH MULTIPLICTY IN DUAL FRAME SURVEYS Linear combinations (composite or Hartley weights) Computing probabilities of selection to account for overlap Raking or post-stratification stratification to totals for usage groups (cell only, etc.) to totals for age and other factors associated with phone usage
59 OTHER ISSUES AFFECTING WEIGHTING CELL PHONE SURVEYS If sample is from cell phone frame only weighting fairly straightforward post-stratification stratification problematic especially for sub-national studies frame information (geography and exchange level demographic estimates) used by many in landline surveys is missing or not as accurate for cell phone frames
60 OTHER ISSUES AFFECTING WEIGHTING CELL PHONE SURVEYS (CONT.) Issues in dual frame surveys accounting for multiplicity has caused difficulties data for post-stratification stratification is sparse Cell frame usually has lower response rate differential response between frames among households with both cell and landline
61 COLLECTING DATA FOR WEIGHTING Has respondent been reached on cell or landline? What phone service does respondent have cell, landline, both? Does anyone besides respondent use cell phone that was called? Other cell phones? Other landlines?
62 COLLECTING DATA FOR WEIGHTING (CONT.) Usage patterns for cell and landline Demographic data associated with telephone usage age sex others
63 RECOMMENDATIONS All telephone surveys should disclose whether the sample includes landline and/or cell numbers and how the numbers were selected from the frames Surveys containing cell phone numbers should fully disclose if any weights were used and what population estimates have been used to post-stratify stratify
64 RECOMMENDATIONS Surveys targeting subgroups with substantial percentages of cell phone only households should sample cell phone numbers For example, year olds, renters, etc. If not feasible, discuss how excluding cell phone numbers affected the results
Climate Surveys: Useful Tools to Help Colleges and Universities in Their Efforts to Reduce and Prevent Sexual Assault Why are we releasing information about climate surveys? Sexual assault is a significant
WORKING P A P E R Improving and Enhancing Telephone-based Disease Surveillance Systems in Local Health Departments David J. Dausey, Anita Chandra, Agnes Gereben Schaefer, Benjamin Bahney, Amelia Haviland,
NUMBERS, FACTS AND TRENDS SHAPING THE WORLD APRIL 3, 2014 FOR FURTHER INFORMATION ON THIS REPORT: Aaron Smith, Senior Researcher 202.419.4500 RECOMMENDED CITATION: Pew Research Center, April 2014, Older
2014 Point-in-Time Count Methodology Guide Table of Contents Acknowledgements... 1 About this Document... 2 1. Introduction to the PIT Count Methodology Guide... 3 1.1 Purpose... 3 1.2 The Importance of
Office of Adult Education Follow-Up Manual State of Michigan Workforce Development Agency Office of Adult Education revised September 2014 CONTENTS OVERVIEW OF NRS REQUIREMENTS...2 OVERVIEW OF THIS DOCUMENT...
Key Measurement Issues in Screening, Referral, and Follow-Up Care for Young Children s Social and Emotional Development April 2005 Prepared by Colleen Peck Reuland and Christina Bethell of the Child and
The Definitive IP PBX Guide Understand what an IP PBX or Hosted VoIP solution can do for your organization and discover the issues that warrant consideration during your decision making process. This comprehensive
Contract No.: TLG98-014-1690.03 MPR Reference No.: 8583-495 Evaluation of Home and Community Based Waiver Program Survey Methodology September 23, 2005 Larry P. Snell Zhanyun Zhao Cathy Lu Frank Potter
STEPS TO SUCCESS FOR RURAL ENTREPRENEURS: STARTING A SMALL ENGINE REPAIR SHOP PREPARED FOR U.S. DEPARTMENT OF COMMERCE ECONOMIC DEVELOPMENT ADMINISTRATION MARCH 2008 University of Alaska Center for Economic
What is a Survey By Fritz Scheuren Harry Truman displays a copy of the Chicago Daily Tribune newspaper that erroneously reported the election of Thomas Dewey in 1948. Truman s narrow victory embarrassed
Investigation of the Use of Mobile Phones While Driving Prepared by Alasdair Cain Mark Burris Center for Urban Transportation Research College of Engineering, University of South Florida 4202 E. Fowler
Data protection Subject access code of practice Dealing with requests from individuals for personal information Contents 3 Contents 1. About this code of practice 4 Purpose of the code 4 Who should use
NSSE Multi-Year Data Analysis Guide About This Guide Questions from NSSE users about the best approach to using results from multiple administrations are increasingly common. More than three quarters of
Health Insurance Exchange Study Minnesota Department of Health February, 2008 Division of Health Policy Health Economics Program PO Box 64882 St. Paul, MN 55164-0882 (651) 201-3550 www.health.state.mn.us
A Cooperative Agreement Program of the Federal Maternal and Child Health Bureau and the American Academy of Pediatrics Acknowledgments The American Academy of Pediatrics (AAP) would like to thank the Maternal
Exploration of the Fiscal Impact of Domestic Violence on Local Criminal Justice Systems in the MAG Region Maricopa Association of Governments Regional Domestic Violence Council Victim Services Stakeholder
General Principles of Software Validation; Final Guidance for Industry and FDA Staff Document issued on: January 11, 2002 This document supersedes the draft document, "General Principles of Software Validation,
2014 Report Broadband Adoption Survey Table of Contents About This Report 4 Executive Summary 4 Methodology 9 Internet Use 11 Internet Use by Age Group 11 Internet Use by Household Income 11 Internet Use
Fair Lending Laws and Regulations Introduction This overview provides a basic and abbreviated discussion of federal fair lending laws and regulations. It is adapted from the Interagency Policy Statement
The author(s) shown below used Federal funds provided by the U.S. Department of Justice and prepared the following final report: Document Title: Author: The Private Security Industry: A Review of the Definitions,
A Consumer Guide to Understanding Health Plan Networks Table of Contents steps you can take to understand your health plan s provider network pg 4 What a provider network is pg 8 Many people are now shopping
United States Department of Agriculture Economic Research Service Economic Information Bulletin Number 138 March 2015 Where Do Americans Usually Shop for Food and How Do They Travel To Get There? Initial
Code of Conduct and Good Practice for Members of the Marketing Research and Intelligence Association December 2007 TABLE OF CONTENTS SECTION A: INTRODUCTION... 2 Foreword... 2 The Marketing Research and
ECC Report 213 Impact of Number Portability between Fixed and Mobile Services (Service Portability) Approved 09 April 2014 ECC REPORT 213 - Page 2 0 EXECUTIVE SUMMARY The term service portability describes
Personal Over-indebtedness in Northern Ireland February 2006 M. Woods OFMDFM Equality Directorate Research Branch Table of Contents 1.0. Introduction... 2 2.0. Consumer Credit in the UK... 3 3.0. Extent
Audit Manual PART TWO SYSTEM BASED AUDIT Table of content 1. Introduction...3 2. Systems based audit...4 2.1. Preparing for & planning the audit assignment...5 2.2. Ascertaining and recording the system...7
I. Automatic Vehicle Location and the Potential for Pupil Transportation Applications Automatic vehicle location (AVL) is an automated vehicle tracking system made possible by navigational technologies