Mixed-Mode Methods for Conducting Survey Research

Transcription

1 Mixed-Mode Methods for Conducting Survey Research Herbert M. Baum, Ph.D.; Anna Chandonnet M.A.; Jack Fentress M.S., M.B.A.; and Colleen Rasinowich, B.A.

2 In November 2010 the Federal government held The Workshop on the Future of Federal Household Surveys where Dr. Don Dillman presented a paper entitled Making New Data Collection Modes Effective in Household Surveys and in 2011 Public Opinion Quarterly published an article The Future Modes of Data Collection (Couper). Both these distinguished authors note that there is growing interest in mixed-mode surveys, because of the limitations posed by each of the major existing modes (paper/pencil, personal, telephone, and web). The typical limitations cited by mode are: Cost paper/pencil and personal Time paper/pencil and personal Coverage telephone and web Decreasing response rates paper/pencil and telephone Different preferences of respondents all modes To overcome these limitations purchasers of surveys as well as the survey research community have been seeking innovative solutions. In addition to exploring mixedmode methods from the perspective of its impact on the quality of data being gathered, this topic is important because with declining response rates there is greater interest in offering respondents options of how they want to respond to a survey. To increase response rates, survey research needs to move from our needs to those of the respondent. For example, people are accessing information via the Internet, but increasingly doing so via smartphones. Can we use that to invite people to participate in a survey? What is a mixed-mode survey? Couper (2011) presents a lengthy discourse on what is meant by mode and notes that there can be as much variation within mode as between modes. For example, when conducting telephone interviewing is the survey being conducted using computer-assisted telephone interviewing (CATI) or by interactive voice response (IVR)? The former has a live person conducting the interview while the latter does not. Between CATI surveys there may be differences in how the sample was selected random digit dialing (RDD) versus dual-frame sampling; thus we need to acknowledge that intra-mode variations do need to be considered as well. Intra-mode variation is not covered in this paper. For this paper, mode refers to the four general survey methods: paper/pencil, telephone (CATI or IVR), personal (including Computer Assisted Personal Interviewing CAPI) and web. We define mixed-mode as using one survey instrument with two or more data collection modes (e.g., gathering data via paper/pencil and the web). By this definition, conducting multiple surveys of a population by different modes is not mixed-mode. That definition leads to the questions: When is it best to use a mixed-mode survey? What challenges are posed when fielding a survey using mixed-modes? The answer to the former relates to the survey instrument, coverage, access, and respondent preferences. The latter to formatting the instrument and the introduction of bias in one mode that is not present in the other mode (e.g., satisficing or social desirability). In this white paper we review what a mixed-mode method survey means, when it is best to use that approach, the challenges posed in conducting a mixed-mode survey, the potential benefits of a mixed-mode survey, and DRC s experience in conducting mixed-mode surveys. 1

3 When should you use a mixed-mode approach? There is no general answer other than letting the potential survey instrument dictate how the survey is best conducted. As a result, there are instances when the length and complexity of the survey dictate only using one mode. Specifically, it would not be advisable to conduct adaptive conjoint or complex interviews on paper. The complicated skip patterns are best suited for an interviewer conducting either CATI or CAPI or possibly a web mode that leverages available analytic software. Cost and time would dictate which one of these methods you would select. Similarly, there are instances when the mode selected requires a mixed mode approach. For a general population web survey, there is potential bias related to coverage issues. As of 2010 Internet access at home was limited to 71% of the population and this percentage varies by both age and race (Smith, 2010). To correct for this differential in access, potential respondents without Internet access must be offered an alternative means of responding (e.g., paper/pencil); making the survey dual mode by design. If however you had a closed population, one where all the potential respondents are identified including addresses (e.g., college students or employees of a company), a single mode web-based survey might be the most appropriate choice. In addition to theoretic reasons for using either a single or mixed-mode approach, there are the pragmatic reasons of cost and time. To boost response rates and keep the cost of survey reasonable, it might be decided to follow up with respondents of a paper/pencil survey with CATI. By conducting the mail response first, you limit the number of calls that need to be made. What are the challenges? The challenges relate to not introducing bias in the results by using different modes for gathering the data; thus mode effects become an important design consideration with the goal of minimizing their impact. The literature on mixed-modes primarily focuses on differences in paper/pencil versus the web; probably because these two modes deliver the survey instrument via visual stimuli. In 2000, Young et al. found virtually no differences between paper/pencil and web modes of a survey. Church (2001) went beyond the paper/web comparision and also considered another respondent-administered mode (IVR). He found that two dual-mode surveys, one utilizing paper-and-pencil and web and the other using paper-and-pencil and IVR, showed a small percentage of unique variance in the data when comparing modes. Church recommends that with equivalent data quality results across mode, one should choose the methodology based on the project parameters (e.g., culture fit or ease of implementation). A mixed mode study that used both self-administered and interviewer administered questions would be prone to bias being introduced from the use of an interviewer. Dillman, et al. (2009) reported that for the satisfactiondissatisfaction questions respondents to the aural modes are significantly more likely than are respondents to the visual modes to give extreme positive responses, a difference that cannot be accounted for by a tendency toward recency effects with the telephone. This type of effect is known as social desirability bias (i.e., providing a positive response when there is somebody asking the question). Chang and Krosnick (2009) compared an internet survey version with a telephone interview and found that the telephone data manifested more random measurement error, more survey satisficing, and more social desirability response bias than did the Internet data. Kraut, Oltrogge, and Block (1998) observed social 2

4 desirability bias when they found the telephone mode of an Employee Opinion survey to have significantly more favorable results than the paper/pencil mode of the survey. These studies indicate a measurement effect that is present when questions are asked by an interviewer as opposed to self-administered. Social desirability bias, can be minimized to the extent that questions are factual or by using the phone as a mode without an interviewer (IVR). The latter is substantiated by Knapp and Kirk (2003) who found no significant differences in responses to sensitive questions between paper/pencil, web, and automated touchtone phone response modes. As noted, mixed methods might be used to increase response rate. Dillman, et al. (2009) utilized different combinations of modes for the initial phase compared to the follow up to non respondents (second phase). Phase 1 data collection was conducted by telephone interview, mail, interactive voice response, and the Internet, while Phase 2 consisted of nonrespondents to Phase 1, and was conducted by telephone or mail. In general, results from [the] study suggest that switching to a second mode is an effective means of improving response. If conducting a mixed-mode survey, should there be different surveys for each? Doing so could introduce response bias associated with the survey instrument. Dillman and Christian (2003) recommend using unimode construction where the instrument is written for the more restrictive environment. This is the practice employed by DRC. If conducting a survey via paper/pencil and using CATI to increase the response rate, the latter would be considered the more restrictive environment. CATI being administered aurally requires that questions be fairly short and answer options brief so that respondents can remember both the question and answers as they respond. A paper/pencil survey can have longer questions and answer options, because the respondent can go back and review them. Though not a challenge directly related to mixed-mode, many organizations are considering changing the mode they use to conduct a survey. If longitudinal trends are important, this can introduce bias. If a test of both modes is not made, then at least all reports should indicate when the change occurred. What are the benefits of conducting a mixed-mode survey? As noted earlier there are pragmatic benefits, usually cost and time. A mail survey with CATI follow up may increase response rate and it will be much less expensive than conducting the entire survey via CATI. De Leeuw (2005) notes Survey designers choose a mixed-mode approach because mixing modes gives an opportunity to compensate for the weaknesses of each individual mode at affordable cost. Offering a paper survey option with a web survey may help to alleviate some concerns about web surveys, such as implementation issues (bandwidth), confidentiality, perceived versus actual accessibility, respondent burden, tolerance level with technological glitches, perceived versus actual cost savings, data representativeness and accuracy, and lower response rate (Church, 2001). This is not meant to imply that there are not benefits in conducting a web survey. The benefits of include: a quicker turnaround time; lower item nonresponse; use of longer open-ended questions (Kwak & Radler, 2002); the time and cost savings associated with not printing/mailing; and less transcription or scanning errors because the data are returned in an electronic format (Kaplowitz, Hadlock, & Levine, 2004). As noted above, there is some evidence that a mixedmode approach does increase response rates. This has been demonstrated in general population surveys, special 3

5 populations, among professionals, and in establishment surveys (de Leeuw, 2005). A mixed mode approach also demonstrates to respondents that you respect their time and are offering them options that may allow them to respond when it is most convenient for them. This is particularly relevant for branded surveys (customer satisfaction or loyalty). If you are using technology in innovative ways (e.g., IVR or web surveys), clients will view your approach as leading edge but by also offering a traditional option they see that you are attempting to appeal to a spectrum of respondents. Increasing the response rate has two benefits. It reduces the likelihood of nonresponse bias and it decreases the standard errors, if you are weighting the data. The latter occurs because when making population estimates there has to be a weight accounting for nonresponse, and a larger weight usually means larger standard errors. Opinion Research (AAPOR) Annual Conference (Randolf, 2001). And as a supplement to this white paper, an additional DRC white paper is available with the specific focus on comparing paper-and-pencil and web survey modes (Fenlason & Christianson DeMay, 2002). For additional information on this topic or working with DRC s professional staff on your mixed-mode survey please contact Jon Leinen, Vice President, Business Development, Survey Services group by (JLeinen@DataRecognitionCorp.com) or by telephone ( ) Overall, the benefits of using a mixed-mode approach, when feasible, are many. The challenges, with proper attention and expertise can be overcome or minimized. DRC s experience in conducting mixed-mode surveys DRC has a long history of providing mixed-mode expertise on a variety of survey types (e.g., 360 feedback, employee opinion, customer satisfaction, parent satisfaction with school services) for both commercial and federal clients. As thought leaders on this topic, DRC has presented or supported several multi-mode research papers at the following conferences: The Society for Industrial and Organizational Psychology (SIOP) Annual Conference (Christianson DeMay & Armstrong, 2000; Christianson DeMay & Toquam-Hatten, 2001), The International Military Testing Association Annual Conference (Elig & Waller, 2001; Quigley, Riemer, Cruzen, & Rosen, 2000), and the American Association for Public 4

6 References (Those in bold are written or supported by DRC staff) Chang, L., & Krosnick, J. A. (2009). National surveys via RDD telephone interviewing versus the internet: Comparing sample representativeness and response quality. Public Opinion Quarterly, 73(4), Christianson DeMay, C., & Armstrong, S. (2000, April). Dual process surveying: A technical perspective. In W. H. Macey (chair), Making the move to internet-based surveys: Lessons from experience. Practitioner Forum presented at the annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA. Christianson DeMay, C., & Toquam-Hatten, J. (2001). 360-degree assessment at The Boeing Company: A dual-process (web/paper) approach. In M. M. Harris (chair), The internet and I-O Psychology: Applications and issues. Practitioner Forum presentation at the 16th annual meeting of the Industrial and Organizational Psychology, San Diego, CA, April Church, A. H. (2001). Is there a method to our madness? The impact of data collection methodology on organizational survey results. Personnel Psychology, 54, Couper, M.P. (2011). The future modes of data collection. Public Opinion Quarterly, 75(5), de Leeuw, E. D. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21(2), Dickinson, J. R., Faria, A. J., & Friesen, D. (1994). Live vs. automated telephone interviewing: Study finds automated interviews yield comparable results for less money. Marketing Research, 6(1), Dillman, D. A., & Christian, L. M. (2003). Survey mode as a source of instability in responses across surveys. Retrieved 01/13/05 from edu/dillman/papers.htm Dillman, D. A., Phelps, G., Tortora, R., Swift, K., Kohrell, J., Berck, J., & Messer, B. L. (2009). Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response, and the internet. Social Science Research, 38(1), Dillman D.A. (2010) Making New Data Collection Modes Effective in Household Surveys. Presented at the Workshop on the future of Federal Household Surveys, November 4-5. Washington, DC. Elig, T., & Waller, V. (2001). Internet versus paper survey administration: Impact on qualitative responses. Paper presented at the 43rd Annual Conference of The International Military Testing Association, Canberra, Australia. Fenlason, K. J. & Christianson DeMay, C. (2002). Paper-and-pencil and web surveys: Different methods, same results? Published electronically on datarecognitioncorp.com/survey/websur.htmlhttp:// Kaplowitz, M. D., Hadlock, T. D., & Levine, R. (2004) A comparison of web and mail survey response rates. Public Opinion Quarterly, 68, Knapp, H., & Kirk, S. A. (2003). Using pencil and paper, Internet and touch-tone phones for self-administered surveys: Does methodology matter? Computers in Human Behavior, 19,

7 Kraut, A. I., Oltrogge, C. G., & Block, C. J. (1998, April). Written vs. telephone surveys of employees: Are the data really comparable? Paper presented at the 13th annual conference of the Society for Industrial and Organizational Psychology, Dallas, TX. Kwak, N., & Radler, B. (2002). A comparison between mail and web surveys: Response pattern, respondent profile, and data quality. Journal of Official Statistics, 18(2), Quigley, B., Riemer, R. A., Cruzen, D. E., & Rosen, S. (2000). Internet versus paper survey administration: Preliminary findings on response rates. Paper presented at the 42nd Annual Conference of the International Military Testing Association, Edinburgh, U.K. Randolph, J. (2001). Internet versus paper survey administration: Impact on qualitative responses. Presented at the American Association for Public Opinion Research Annual Conference, Montreal, Quebec. Smith, A. (2010). Home Broadband Pew Research Center s Internet & American Life Project. August 11, Washington DC. Young, S. A., Daum, D. L., Robie, C., & Macey, W. H. (2000, April). Paper versus Web survey administration: Do different methods yield different results? Paper presented at the 15th annual conference of the Society for Industrial and Organizational Psychology, New Orleans, LA. 6

8