Unintended Consequences: The Cost of Purging Business Numbers in RDD Surveys. Daniel Merkle, Gary Langer, Jon Cohen ABC News

Transcription

1 Unintended Consequences: The Cost of Purging Business Numbers in RDD Surveys Daniel Merkle, Gary Langer, Jon Cohen ABC News Linda Piekarski Survey Sampling International Robert Benford, David Lambert TNS Contact: Daniel Merkle ABC News 47 W. 66 th St. 6 th Floor New York, NY Paper presented at the annual conference of the American Association for Public Opinion Research, Montreal, Canada, May 18-21, 2006.

2 Survey researchers commonly use RDD samples that are purged of listed business telephone numbers to increase interviewer productivity by removing numbers assumed to be ineligible for household surveys. Very little research has been presented that evaluates this practice. We find that purging listed business numbers by conventional means reduces household coverage by removing numbers that in fact are eligible, with no appreciable improvement in productivity. Neither of two prominent previous studies on this topic directly assesses the effects of business number purging. In Battaglia et al. (1995), interviewers called telephone numbers initially identified as business or nonworking; 3.1 percent were found to be residential, but the study did not break out the results for business and nonworking numbers separately. Battaglia, Ryan and Cynamon (2005) summarized the 1995 study, saying the cost benefits swamped the false removal of a very small percentage of the residential telephone numbers in an RDD sample (p. 6). However, an analysis of the cost-benefit considerations was not presented. Battaglia, Ryan and Cynamon (2005) also provide an in-depth discussion of procedures used for purging out-of-scope numbers, but without directly evaluating traditional business number purging done using directories. Instead they evaluate a procedure in which all listed business numbers are purged, then all remaining sampled numbers are pre-dialed in an attempt to identify additional business numbers, as well as nonworking or modem-dedicated numbers. When the numbers classified in this predialing procedure as businesses were later called by interviewers, 28.5 percent were found to be, in fact, residential numbers. The authors say this should not be a great 2

3 concern because the incidence of numbers classified as businesses, as a proportion of the entire sample, was small (p. 19). Our study evaluates traditional business number purging using directories and explores the consequences of excluding these numbers. We evaluate the completes produced by these numbers to determine if they re valid household interviews. We also examine the characteristics of respondents interviewed on business-listed numbers and explore the implications for noncoverage error. Finally, we consider the cost-benefit implications of including these numbers in the sample. Methodology The data for this study come from three ABC News or ABC News/Washington Post polls conducted by telephone in June and August 2005 among national random samples of, respectively, 1,004, 1,002 and 1,006 adults age 18 and over. For each poll, a sample of telephone households in the continental United States was selected via RDD. Interviewers asked to speak to the household member age 18 or over at home who had the last birthday. The response rates were 29.3, 31.6 and 30.8 percent. 1 Data collection and tabulation were conducted by TNS of Horsham, PA. Table One Methodological Details of Polls Field Dates Sponsor Sample Size Response Rate 6/23-26/2005 ABC News/Wash. Post 1, /18-21/2005 ABC News 1, /25-28/2005 ABC News Wash. Post 1, The response rates are computed using AAPOR rate RR3: completed interviews / [(completed interviews + partial interviews) + (refusals + noncontacts + other) + e*(unknown households + unknown other)]. We set e at.20 based on estimates from previously published RDD telephone surveys with similar field period lengths (Keeter, Miller, Kohut, Groves and Presser (2000, p. 127)). 3

4 The samples were produced by Survey Sampling International (SSI) from a sampling frame that includes all active telephone area codes, exchanges and 100-blocks. Sampled phone numbers are pre-dialed via a nonringing auto-dialer to purge nonworking numbers. SSI uses a database of 14.5 million business telephone numbers to flag business lines. About three million listed business numbers are excluded from the frame entirely because they are in exchanges and 100-blocks whose only listed numbers are businesses. The business database, updated quarterly, is compiled from sources such as Yellow Pages directories and the Dun and Bradstreet Business Data database. Survey researchers customarily exclude listed business numbers from their RDD samples. For this study, numbers flagged as businesses in the sampling frame were retained in the sample, and respondents were asked about their phone line usage. Overall, 27,036 telephone numbers were dialed across the three surveys, including 16,129 unlisted, 9,440 residential-listed, and 1,467 business-listed. To evaluate the validity of completed interviews obtained via listed business numbers, we asked respondents where we reached them (e.g., at home or work), whether the phone line was used for personal or business use, who in the household could use the phone line and whether it was listed in a residential and/or business telephone directory. The analyses reported in this paper were conducted using unweighted data. Weighting for the probabilities of selection (i.e., number of adults in the household and number of phone lines) produced similar results. 4

5 Results The assumption underlying the practice of purging business numbers is that these numbers are ineligible for household surveys. The results of our study indicate that this is not always the case. Interviews were completed with respondents at 10 percent of the telephone numbers flagged as businesses (Table Two). As a share of total completes, five percent of all respondents were interviewed on lines classified as business numbers that normally would have been purged from the sample. Table Two Percent of Numbers Resulting in Interviews Survey % of business % of numbers completes All / / / Were these valid household interviews, or should these numbers have been excluded? The data from the phone line usage questions indicate that these are, in fact, primarily households (Table Three). Ninety-four percent of these numbers rang at residences (including 76 percent at a home and 18 percent at a home office). 2 This is similar to the results from non-business-listed numbers, 98 percent. Six percent of those contacted via the business-listed sample were reached outside their homes a total of nine respondents, of whom five were reached at non-home-based workplaces, three on cell phones and one at someone else s home. These are ineligible respondents in an RDD household survey and arguably should be removed from the dataset. Their number, however, is not consequential; nor are business-listed numbers the 2 See Appendix A for question wordings. 5

6 unique source of such respondents. Two percent of residential-number respondents also were reached outside their homes a total of 65 individuals, including 17 at non-homebased workplaces, 13 at someone else s home, five on cell phones and 30 elsewhere. It's evident that not-at-home completes exist as sampling noise with or without the use of business-listed numbers. Among business-listed respondents reached at their home office or outside their home, 77 percent said the phone line on which they were reached was used at least in part for personal use (including 71 percent who said it was for personal and business use alike, and 6 percent who said it was for personal use only). Among non-business-listed numbers this was slightly higher, 87 percent, although the difference was not statistically significant given the small sample sizes. Six percent of respondents contacted on business-listed numbers said the number was listed only in the Yellow Pages, and nine percent said it was unlisted. Eighty-five percent reported a residential listing either in the White Pages only (72 percent) or in both the Yellow and White Pages (13 percent). As would be expected, the incidence of unlisted numbers was higher, 24 percent, among those who were contacted via traditional, business-purged sample. In practice, moreover, almost all business-flagged phone numbers were used as household rather than business-only lines. In households with more than one adult, 93 percent of those contacted on listed business numbers said any adult in the household could answer the phone line in question; another five percent said only they answered the phone but others could speak on it. These results are similar to those who were contacted on non-business-listed numbers. 6

7 Table Three Phone Line Usage by Type of Sample Business Listed: Yes No Where reached: (n=142) (n=2,788) Home or home/office Somewhere else Chi-square=8.81, p<.05 Usage of phone line: a (n=34) (n=145) Personal or personal/business Business only Chi-square=2.34, p=n.s. Telephone listing: (n=139) (n=2,683) White pages or White/Yellow 84.9% 74.5% Yellow pages only Not listed Chi-square=34.77, p<.05 Who can use phone line: b (n=110) (n=2,035) Any HH adult can answer Only respondent answers, but others can speak on it Only respondent uses it Chi-square=3.45, p=n.s Note: a Asked of those reached in a home office or somewhere outside the home. b Asked of those with two or more adults living in the household. Coverage and Productivity A more important finding is that business number purging increases noncoverage. Sixty-five percent of those contacted on numbers that normally would have been purged from the sample said they had no other phone line in the household. These households would have been excluded from the sample had these purported business numbers been purged, producing a noncoverage rate of three percentage points. We evaluate potential noncoverage error by examining the characteristics of respondents who would have been missed if their business-flagged numbers had been purged (i.e., those with a businesslisted number as the only phone line in the household). Table Four compares the 7

8 noncovered and covered populations for all questions that were common across the three surveys. 3 Eligible respondents who would have been missed if business purging had been implemented were more likely to be middle age, had higher incomes and were more apt to be politically conservative. The two groups didn t differ significantly on education, race, sex, region or political party identification (Table Four). While the differences on age, income and ideology are statistically significant, the practical impact of the noncoverage error caused by purging business numbers was not great (Table Five), because the noncoverage encompasses just three percent of total households. The noncoverage error introduced by business number purging was half a percentage point or less in the eight questions we studied, averaging 0.28 of a percentage point. Listed business numbers are purged from RDD samples to decrease costs because these numbers are assumed to be ineligible. The small noncoverage error introduced by this practice could be justified by cost savings. In fact, however, we do not find any real cost savings by implementing business purging. The productivity of the listed business numbers is quite similar to that of the rest of the sampled numbers (Table Six). Across all three surveys, 10.1 percent of purported business numbers resulted in completes compared with 11.2 percent of the other sampled numbers. Business-flagged respondents were reached in a mean of 2.40 dialings, compared with a nearly identical 2.39 dialings to reach listed residential respondents, and 2.44 to reach unlisted residential respondents. 3 Noncovered consists of those respondents who would have been missed had business purging been implemented (i.e., those with business-listed numbers as the only phone line in the household); Covered represents those who would have had a chance to be interviewed if business purging were implemented (i.e., those who do not have business-listed numbers or those with a business-listed number and one or more other phones lines in the household.) 8

9 Table Four Characteristics of Noncovered vs. Covered Population When Business Number Purging Is Implemented Noncovered Covered population population (n=92) (n=2,920) Age: % 29.2% Chi-square=9.47, p<.05 Income: Under $50, $50,000 or more Chi-square=3.92, p<.05 Education: < High school High school graduate Some college Chi-square=4.75, p=n.s. Race: White Nonwhite Chi-square=1.74, p=n.s. Sex: Men Women Chi-square=.61, p=n.s. Region Northeast Midwest South West Chi-square=7.04, p=n.s. Ideology: Conservative Liberal Moderate Chi-square=11.29, p<.05 Party ID: Republican Democrat Independent/other Chi-square=2.64, p=n.s Note: Sample sizes vary somewhat because of missing data. 9

10 Table Five Evaluation of Noncoverage Error When Business Number Purging Is Implemented Covered Total Absolute population population difference (n=2,920) (n=3,012) Ages % 51.5%.4 Income: Under 50K Education: < High school Race: White Men Region: West Ideology: Conservative Party ID: Democrat Note: The largest difference is shown for each question. Sample sizes vary somewhat because of missing data. Table Six Productivity of Sampled Numbers: Business-Listed vs. Other Sampled Numbers Percent of Numbers Resulting in Interviews Business Other All / / / Sample dispositions help explain this phenomenon. There were far fewer nonworking or disconnected numbers in the business-flagged sample (five percent, versus 24 percent of the purged sample) (Table Seven). Instead, numbers in the businesslisted sample were more apt to be dispositioned as nonresidential (38 percent, versus 6 percent of the purged sample). These differences suggest a small productivity cost associated with dialing the business-listed sample, because nonresidential numbers need 10

11 to be dispositioned by interviewers while nonworking numbers are more likely to be dispositioned by the CATI system without interviewer intervention. However, the cost likely is minimal, given both the comparatively few business-listed numbers in any given sample and the short time it typically takes an interviewer to disposition a nonresidential number. The two samples had roughly equal numbers of ring/no answer dispositions (22 and 25 percent, respectively) and other noncontacts (13 and 14 percent, respectively). And the business-listed sample had a net total of 11 percent refusals and other out-ofscope numbers, compared with 21 percent of the business-purged sample. In sum, efficiencies of the business-listed sample compensated for its inefficiencies. Table Seven Sample Dispositions: Business-Listed vs. Other Sampled Numbers Absolute Business Other difference Nonresidential Nonworking Refusals Other out-of-scope Ring/No answer Completed interview Other noncontact Note: Refusals includes respondent refusals, household refusals, hang-ups, stopped interviews, midterm break-offs and respondents who claimed to have done the survey; Other out-ofscope includes fax/modems, language barriers, respondent disabled, and no eligible respondent; Other noncontact includes answering machine, busy, callbacks, and caller id block. Discussion and Conclusion Purging business lines from RDD samples is accepted orthodoxy in survey research. But it is a largely unexamined orthodoxy, one that has carried forward from an earlier and simpler time. Business telephone numbers once may have existed as entirely discrete and separate objects from household numbers. Those lines have blurred; our 11

12 results show that in 21 st century America there is no iron curtain between household and business lines. Our results are buttressed by findings from the 2000 U.S. Census, which found that nearly 4.2 million Americans worked at home in 2000, a 23 percent increase from 1990 double the rate of growth of the workforce overall (U.S. Census Bureau, 2004). Today's reality is that a business-listed telephone line is not necessarily a businessonly telephone line. Given this reality, we find that when business numbers were retained rather than purged from RDD samples, 10 percent of such numbers yielded completed interviews, and they accounted for five percent of all completes. Results from questions on how these phone lines are used show that these were valid household interviews, appropriately included in our RDD sample. While some might argue for a weighting adjustment for number of phone lines, we also found that approximately two-thirds of these purported business lines represented the only phone line on which the contacted household could have been reached. A recent study of phone-line weighting, moreover, found it to be at best an unnecessary procedure and at worst a risky one. (Merkle and Langer, 2004). This evaluation also finds an unintended consequence of business number purging, an increase in household noncoverage. Three percent of eligible households in this study those with a listed business number and no other phone line in the household would have had zero probability of selection if business-line purging were implemented. Those who would have been missed differed on age, income and ideology. The overall magnitude of the noncoverage error was minimal because the size of the noncovered population was so small. Nonetheless, maximizing coverage is highly desirable in survey research, particularly given other challenges to full coverage, notably 12

13 cell-phone-only households. Any reduction in coverage should be tolerated only if justified by other considerations. The justification for business-line purging is budgetary, based on the presumption that dialing business-listed numbers is unproductive. Our findings run contrary to this conventional yet previously untested wisdom: productivity of the business-listed and other sampled phone numbers was very similar. The only other apparent cost impact of including business listings would be in replicating our phone-use questions for respondents contacted at business-listed numbers as an ongoing belt-and-suspenders check of their inclusion. But the cost would be very minimal, given the small number of these completes, and our findings suggest the approach would be unnecessary. In sum, our results argue for the inclusion of listed business telephone numbers in RDD surveys. Previous research (Battaglia et al., 1995; Battaglia, Ryan and Cynamon, 2005), has been more supportive of business line purging, perhaps because of the different approach of these studies. Battaglia et al. (1995) considered business and nonworking numbers together, whereas we only looked at business numbers. Battaglia, Ryan and Cynamon (2005) evaluated a business identification procedure beyond the traditional directory-based approach to purging. Further study is warranted, focused in particular on exploring the breakdown between presumed and actual business-only lines, confirming our productivity findings and extending our examination of the extent of noncoverage error created by the exclusion of purported business-only numbers from RDD samples. 13

14 Appendix A Question Wordings 51. Including yourself, how many adults, 18 years old or older, live in this household? 52. Is the telephone number at which I have reached you listed in a published telephone directory? IF YES: Is it listed in the white pages, the yellow pages, or in both? 1 Yes, WHITE PAGES 2 Yes, YELLOW PAGES 3 Yes, BOTH WHITE AND YELLOW PAGES 4 NOT A LISTED NUMBER DK/No opinion NA/Refused 53. Have I reached you at home, at a home office where you live, at an office or other workplace outside your own home, or somewhere else? 1 At home 2 At a home office where you live 3 At an office or other workplace outside your own home 4 (VOL) Someone else s home 5 (VOL) Home line forwarded elsewhere 6 (VOL) Cell phone or home line ported to cell 7 (VOL) SOMEWHERE ELSE (SPECIFY) NA/Refused ***ASK IF Q53 NOT=1 (NOT REACHED AT HOME)*** 54. Do you use this phone number only for personal use, only for business purposes, or for BOTH business and personal use? 1 Personal use only 2 Business purposes only 3 Both business and personal use NA/Refused ***ASK IF Q51 > 1 (>1 ADULT IN HH)*** 55. Thinking about the adults who live in your household, which of these three statements best describes how you use this phone line for incoming calls: 1 Any adult in your household can answer this phone line 2 Only you answer this phone line, but others can speak on it 3 Or, only you answer this phone line AND only you speak on it DK/No opinion NA/Refused 56. Not counting cell phones or phone lines used ONLY for faxes or modems, how many different telephone numbers, including this one, are there in your home that I could have reached you on for this call? This includes listed or unlisted numbers. 14

15 References Battaglia, Michael P., Starer, Amy, Oberkofler, Jerry, and Zell, Elizabeth R "Pre- Identification of Nonworking and Business Telephone Numbers in List-Assisted Random-Digit-Dialing Samples." Proceedings of the Section on Survey Research Methods, Alexandria, VA: American Statistical Association, pp Battaglia, Michael P., Ryan, Meg, and Cynamon, Marcie "Purging Out-of-Scope and Cellular Telephone Numbers from RDD Samples." Proceedings of the Section on Survey Research Methods, Alexandria, VA: American Statistical Association. Keeter, S, Miller C., Kohut A., Groves R.M. and Presser S. (2000). Consequences of Reducing Nonresponse in a National Telephone Survey. Public Opinion Quarterly 64: Merkle, Daniel M. and Langer, Gary (2004). How Too Little Can Give You Too Much: Determining the Number of Household Phone Lines in RDD Surveys. Paper presented at the annual conference of the American Association for Public Opinion Research, Phoenix, AZ, May U.S. Census Bureau (2004). Census Bureau Releases Information on Home Workers. Washington, D.C.: October, 20. Last accessed April, 20, 2004 at: 15