mobilesampling new challenges and developments in mobile research methodology

Transcription

1 mobilesampling new challenges and developments in mobile research methodology

2 Mobile Telephone Methodology: Present and Future The growth of the percent of the U.S. population that owns a mobile telephone but no traditional landline telephone continues unabated. The major difference between mobile and landline telephones, from a sampling perspective, is that mobile telephones at present are connected to very little other data. With landlines, for example, one can append a name and an exact address to about half all numbers, and for the remainder, a ZIP code that is reasonably accurate. One can also append mean demographic information for the residents of a telephone exchange, such as share of racial and ethnic groups, income and age. And finally, directory-listed information can be appended indicating race, age and many other demographics of members of a landline household. These tools have afforded great power for researchers to sample landlines with a high degree of geographic accuracy and the ability to oversample or cluster interviews by a range of geographic and demographic properties. Mobile phones, in effect, have no comparable data: They are, grossly, a blank slate. How then does one oversample for respondents of specific geographies or demographics? How can the expected incidence and coverage (two critical measures greatly impacting price and quality) be estimated? How can one sample at the local level? Is there hope for listed information on mobile phones? At the current track, it seems as though the future of telephone survey research will not include landline telephones. What criteria are there by which a decision to eschew landlines in telephone surveys entirely might be made? And how close are we to this becoming common practice? The future of mobile telephonic research is to some degree still unwritten. But a number of developments have significantly increased the utility, efficiency and power of mobile telephone sampling. First, there are rate centers, which are specific billing geographies first developed to denote local area networks of landlines in the early 20th century (in short, the demarcations at which, to oversimplify, a telephone call became long distance. ) While sample methodologists have had the luxury of about 81,000 landline telephone exchanges with which to stratify geographically, here there are only 18,107 rate centers in the U.S. That said, over 18,000 rate centers still afford a significant ability to stratify mobile telephones nationally, within states and even within specific metropolitan areas (for comparison, there are about 43,000 ZIP codes in the U.S., and 3,143 counties.) About half of all rate centers reside within a single county, while the rest cross at least one county line. 1

3 Rate Centers in the Washington DC Metro Area Every mobile telephone is associated with the rate center from which, generally, the mobile phone was purchased. But importantly, rate center assignment tends to be based on brick and mortar point of purchase stores. If a mobile phone was purchased over the Internet, it becomes associated with the rate center of the closest brick and mortar store for that carrier (again, generally speaking.) There are many exceptions. If one were to cross state lines to purchase a mobile phone and request that the area code for the mobile phone number be for one s state of residence, the rate center associated with that phone will be the closest rate center to the area code and exchange associated with that phone. But overall, the important take-home statistic of note is that currently 50% of all mobile phone owners live in the rate center associated with their mobile telephone number. And half of those with a mismatch (those who live somewhere outside of their mobile telephone rate center,) nevertheless live 25 miles or less from the centroid of their mobile phone s rate center. Thus, insofar as one selects multiple rate centers in a given study, over half of persons in the target area of interest will report living in that target area. Two important limitations of rate center selection are: First, that rate centers without a brick and mortar mobile phone store do not have any mobile numbers associated with them...they are empty rate centers. Persons who live in such rate centers will have telephone numbers connected to adjacent filled rate centers. Therefore, one must be very careful not to design a study without first understanding which centers are empty and which are filled. The good news is that, while over 11,000 rate centers are empty, only about 16% of the U.S. population resides in those rate centers, since they are typically not very densely populated. The second limitation of rate center sampling is that the size of rate centers is quite inconsistent, and a few select metropolitan areas are dominated by a single large rate center reaching into over a dozen counties. In such areas, which include Denver, Las Vegas, Minneapolis/St. Paul, Phoenix, and to a lesser extent Houston, Miami, San Antonio and Dallas, there is no ability to oversample select areas of the metro area. 2

4 Denver Rate Center (Purple; county lines in yellow) Oklahoma Rate Centers (Blue=filled, Gray=empty) The key statistics of sampling are incidence and coverage. What incidence can we expect in a study? The lower the incidence, the more expensive it is to reach one s target population. And, what coverage will one get? The lower the coverage, the fewer number of persons in a target population that have any chance of being selected for a given study. The good news is that the average metropolitan area, if one were to select all the rate centers associated with that area, will attain a respectable 77% incidence and 75% coverage. Large metro areas, such as Houston, will attain higher rates, specifically 89% incidence at 92% coverage. Small metro areas like Providence, RI, will likely attain incidence and coverage in the mid 60% range. It is important to note however that those people who are not covered differ from those who are. Specifically, those not covered tend to be more educated, younger, own homes at a greater rate and are more likely to be Caucasian. Given a very high coverage rate, the potential for bias is low. 3

5 If one is interested in sampling more precisely in terms of geography, there is another option: billing ZIP codes. Simply put, sampling companies are able to append the billing ZIP code for mobile telephones, and then those billing ZIP codes can be used to sample specific geographic areas. But as might be expected, here too there are limitations. First, only 60% of mobile telephone owners have a billing ZIP code associated with them. Those on prepaid phone plans do not have billing ZIP codes associated with their phones (more on this shortly.) Second, only about half of all mobile phone owners with a billing ZIP code self-report living in this ZIP code area. The good news, however, is over half of the mismatches live within 20 miles of the centroid of their billing ZIP code, and two-thirds live within 35 miles. So as with rate centers, assuming that an aggregate of ZIP codes are selected for any given study, the survey incidence will be much greater than 50%. Rather than incidence, it is the coverage that is concerning, given that the baseline is 60% (who have billing ZIP codes,) and in reality it will be lower given the mismatch rate. Typically, a billing ZIP code selection of a small geographic area is unlikely to attain less than 50% coverage. The solution of course is to not only utilize billing ZIP code sample, but to use non-billing sample in the associated rate center as well, oversampling the billing ZIP code sample to maximize incidence. Even still, the potential for bias is not insignificant, as it appears that persons who own mobile phones without a billing ZIP code are more likely to be in lower income brackets, less educated and are significantly less likely to be Caucasian. 4

6 Another append available to researchers and methodologists is the prepaid phone flag. All mobile samples are capable of being flagged as a prepaid phone or not. This flag can be an effective way to oversample select populations, including non-whites, persons of low income, and mobile-phone-only respondents. One key difference between mobile phone surveys and landline surveys is productivity. While landlines can use auto-dialing systems, mobile phones have to be hand-dialed. And more critically, landlines can be scrubbed by a variety of methods, and filtered to 1+ listed sample (basically, nearly every study using landline samples in the last 25 years has utilized 1+ listed sample.) Such techniques extract the majority of non-working and business numbers, resulting in dramatically increased dialing efficiencies. Indeed, the advent of 1+ listed sampling allowed for telephone surveys to become truly affordable in the first place. One key difference between mobile phone surveys and landline surveys is productivity. There is no such ability to create 1+ listed mobile samples, and many scrubbing techniques utilized by the major sampling companies for landlines are not financially feasible, or they utilize databases that just don t exist for mobile telephones. Luckily, there are now two techniques available to scrub non-working numbers from mobile telephone samples, one offered by Targus and the other from MSG (Cell-WINS.) In short, such methods are able to look back at telephone use and define telephone numbers as either being active or inactive. The inactive sample is sequestered and never given to a research firm for dialing. Such procedures unfortunately extract about 5-6% of actual households, or actual working mobile phones. But that said, the same has been true of the 1+ listed technique, and if this loss of coverage has been satisfactory for landline sample for a quarter century, surely it is acceptable for mobile phones. Patterns of bias are similar to what has been found in 1+ listed samples as well. That is to say, there are a handful of variables in which the 5-6% of extracted households look considerably different than the 94-95% that were not extracted, but given that it is only 5-6% of all households (less in the context of a dual frame design that includes landline households and in which some of the 5-6% will be covered for also owning a landline,) the potential for bias is practically negligible. Importantly, while it costs money to have these procedures enacted, telephone productivity is improved about 10-20%, depending on level of effort (studies with only a few call attempts will save more than studies with 10 or more call attempts.) This significantly exceeds the cost of such sample. 5

7 Overall, the future of mobile phone sampling and survey research looks generally bright. Landline sample has for some time had the ability to append data from listed database. For example, if one wanted to conduct a survey of persons age 65 and older, one could append listed information in which there is an expected probability of a given telephone number reaching a household with someone age 65 or older present. As with most things in the sampling world, such listed databases vary in the degree to which they cover all (in this example) 65+ households and to which they are accurate in their flagging. For illustration, such databases have been able to flag about 80% of all households that have a person age 65 and older with about an 80% accuracy rate. That said, this is traditionally one of the best appends in the market; comparatively, appends for year olds are at about a 20% coverage and 40% accuracy. This makes logical sense, as the older someone s age, the more time the listing companies have to find them, and the less likely they are to have disconnected from the landline. In any event, being able to append not just age, but income, ethnicity, the presence of children and a host of other variables has made landline stratification and sampling by such variables an excellent way to target such populations of interest. Unfortunately, while listed databases for mobile phones are growing, they are a mere shadow of what is available on landlines, and really are not at the point of being particularly useful for sampling. In most testing thus far, researchers have found coverage rates of under 10% for most appends, with an accuracy rate of under 20%. Again, at these levels, there is little utility for such appends, but the hope is that as people use their mobile phone numbers for billing and other purposes, such data sets will improve over time. A final topic to consider is the future of mobile phone surveying in general. In a mere decade, the percentage of persons who own a landline telephone has dropped from about 97% to 55%. And consequently, dual-frame surveys have gone from being 10% or 20% mobile phones to 30%, then 40%, and now, quite commonly, 50% or more. How high can it go? The short answer is 100%. Researchers have already found that studies that only gather data via mobile phones contain only a small amount of bias, largely due to the holdout ~7% of the population that own a landline but not a mobile phone. And importantly, because refusal rates on mobile phones have flattened over the past few years, combined with the new scrubbing techniques noted above, and the fact that typical studies do not do lengthy household selection screeners on mobile phones like they typically do on landlines, the net result is that despite the requirement for hand dialing mobile phones are now only modestly more expensive to dial than landlines. In absence of financial constraints and meaningful bias, it is entirely possible that we will begin to see 100% mobile phone studies in the future. Well not quite yet. Landlines are still more efficient to dial, and still have many more times the ability to stratify in order to reach populations of interest. It is likely that dual-frame sampling will continue to be the norm for the next decade, with ever-increasing shares of mobile phone interviews. Overall, the future of mobile phone sampling and survey research looks generally bright. The ability to target geographically is improving, and sampling companies continue to search for ways to improve billing ZIP and other geographic appends. The ability to target using listed data appending is in its infancy but the hope is continued improvement over time. And as persons continue to treat mobile phones as normal, primary communication devices, they should similarly find the idea of participating in a survey over a mobile phone to be normal. In all, it adds up to a likely future in which mobile phones are the primary method by which random sampled surveys are administered. 6

8 About the Author: David Dutwin David Dutwin is Executive Vice President and Chief Methodologist at SSRS, where his primary responsibilities include sampling designs, project management, executive oversight, weighting and statistical estimation. David is an active member of the survey research community, having served in the American Association for Public Opinion Research as a member and a chair of special task forces, a member of the Standards, Communications, and Heritage Committees, teaching multiple short courses and webinars, and as the Student Paper winner of He was elected to the AAPOR Executive Council in 2013 and serves as the 2016 Conference Chair. He attained his Bachelors in Political Science and Communication from the University of Pittsburgh, a Masters of Communication from the University of Washington and his doctorate in Communication and Public Opinion from the Annenberg School for Communication at the University of Pennsylvania. For over a decade he has taught Research Methods, Rhetorical Theory, Media Effects and other courses as an Adjunct Professor at West Chester University. David is also a Research Scholar at the Institute for Jewish and Community Research and Principal Methodologist for JPAR/Jewish Policy and Action Research. His publications are wide-ranging, including a 2008 book on media effects and parenting; methodology articles for Survey Practice, the MRA magazine Alert!, and other publications; and a range of client reports, most recently on Hispanic acceptance of LGBT, which he presented to a Congressional briefing in