The Evaluation and Analysis of Survey Paradata. Instructors: Bob Groves and Mick Couper Winter, 2008

Transcription

1 Draft, January 29, 2009 The Evaluation and Analysis of Survey Paradata Instructors: Bob Groves and Mick Couper Winter, 2008 Overview of the Course Paradata are empirical measurements about the process creating survey data themselves. They consist of visual observations of interviewers, administrative records about the data collection process, computer-generated measures about the process of the data collection, external supplementary data about sample cases, and observations of respondents themselves about the data collection. The definition is not well-evolved and is subject to debate. This course will have four parts: a. Review of the (brief) literature on survey paradata b. Review of analytic approaches to paradata c. Student proposals on analysis of paradata d. Analysis projects on paradata Student Activities Students will be responsible for reading the small amount of literature assigned, discussing it in class. Particular attention will be paid to identifying gaps in the past uses of paradata to answer questions about costs and errors of survey estimates. Students will then expand the bibliography of relevant literature by their own searches, producing a more comprehensive set of readings. Each student will propose an analysis of paradata to be performed on data supplied by the instructors. The proposal will be written and presented in the class. Each student will conduct the proposed analyses and write a technical paper describing the analysis. Student Prerequisites Limited to PhD students in Survey Methodology or to others by explicit permission of one of the instructors.

2 Key Perspectives on Paradata Motivating the Class 1. Cost Efficiency of data collection 2. Building propensity models on paradata 3. Management interventions based on paradata 4. Using paradata as proxy indicators of measurement error 5. Error structures of paradata 6. Ethical consideration re paradata Thursday, January 29\ Class 1 - Orientation Overview and Class Scheduling Student-level goals Class Schedule Saturday, February 7, 9-11AM Class 2 Conceptual Framework of Paradata Issues Couper and Lyberg, The Use of Paradata in Survey Research. Proceedings of the 55th Session of the International Statistical Institute (CD-ROM), 2005 Durrant, G.B. and Steele, F Multilevel Modelling of Refusal and Noncontact Nonresponse in Household Surveys: Evidence from Six UK Government Surveys. Journal of the Royal Statistical Society. Series A. 172, 2: Heerwegh, D. (2003), Explaining Response Latencies and Changing Answers Using Client-Side Paradata from a Web Survey. Social Science Computer Review, 21 (3): Friday, February 13, 3-5PM Class 3 Issues in Paradata for Nonresponse Error Investigations Beaumont, J.F., On the Use of Data Collection Process Information for the Treatment of Unit Nonresponse Through Weight Adjustment, Survey Methodology, 31, 2005, pp Durrant, G.B. and Steele, F Multilevel Modelling of Refusal and Noncontact Nonresponse in Household Surveys: Evidence from Six UK Government Surveys. Journal of the Royal Statistical Society. Series A. 172, 2: 1-21 Greenberg, B. S. and S. L. Stokes (1990). "Developing an Optimal Call Scheduling Strategy for a Telephone Survey." Journal of Official Statistics 6(4): Groves, R. M. and S. G. Heeringa (2006). "Responsive design for household surveys: tools for actively controlling survey errors and costs." Journal of the Royal

3 Statistical Society: Series A (Statistics in Society) 169(3): Johnson, T. P., Y. I. K. Cho, et al. (2006). "Using Community-Level Correlates to Evaluate Nonresponse Effects in a Telephone Survey." Public Opinion Quarterly 70(5): Kristal, A.R., White, E., Davis, J.R., Corycell, G., Raghunathan, T., Kinne, S., Lin, T. (1993), Effects of Enhanced Calling Efforts on Response Rates, Estimates of Health Behavior, and Costs in a Telephone Health Survey Using Random-Digit Dialing. Public Health Reports, 108 (3): Kulka, R.A., Weeks, M.F. Toward the development of Optimal Calling Protocols for Telephone Surveys: A Conditional Probabilities Approach, Journal of Official Statistics, 4, 319, 1988, pp Lynn, P. (2003). "PEDAKSI: Methodology for Collecting Data about Survey Non- Respondents." Quality and Quantity 37(3): Potthoff, R.F., Manton, K.G. and Woodbury, M.A. (1993), Correcting for Nonavailability Bias in Surveys by Weighting Based on Number of Callbacks. Journal of the American Statistical Association, 88 (424): Purdon, S., P. Campanelli, et al. (1999). "Interviewers Calling Strategies on Face- to- Face Interview Surveys." Journal of Official Statistics 15(2): Rao, R. S., M. E. Glickman, et al. (2007). "Stopping rules for surveys with multiple waves of nonrespondent follow-up." Statistics in Medicine Simester, D. I., P. Sun, et al. (2006). "Dynamic Catalog Mailing Policies." Management Science 52(5): Snijkers, G., J. Hox, et al. (1999). "Interviewers' Tactics for Fighting Survey Nonresponse." Journal of Official Statistics 15(2): Smith, T.W. (1984), Estimating Nonresponse Bias with Temporary Refusals. Sociological Perspectives, 27 (4): Sturgis, P. and Campanelli, P The Scope for Reducing Refusals in Household Surveys: An Investigation based on Transcripts of Tape-recorded Doorstep Interactions. Journal of the Market Research Society. 40, 2: Traugott, M.W., (1987), The Importance of Persistence in Respondent Selection for Preelection Surveys. Public Opinion Quarterly, 51: Van der Vaart, W., Ongena, Y., Hoogendoorn, A. and Dijkstra, W Do Interviewers Voice Characteristics Influence Cooperation Rates In Telephone Surveys?. International Journal of Public Opinion Research. 18, 4: Weeks, Kulka, and Pierson, Optimal Call Scheduling for a Telephone Survey, The Public Opinion Quarterly, Vol. 51, No. 4 (Winter, 1987), pp Wednesday, February 18, 10AM-12PM Class 4 Issues in Paradata for Measurement Error Investigations Couper, M.P. (1998), Measuring Survey Quality in a CASIC Environment. Invited paper presented at the Joint Statistical Meetings of the American Statistical Association, Dallas, August.

4 Couper, M.P., Hansen, S.E., and Sadosky, S.A. (1997), Evaluating Interviewer Performance in a CAPI Survey. In L. Lyberg et al. (eds.), Survey Measurement and Process Quality. New York: John Wiley, Couper, M.P., Tourangeau, R., and Marvin, T. (2009), Taking the Audio out of Audio- CASI. Public Opinion Quarterly, forthcoming. Couper, M.P., and Lyberg, L.E. (2005), The Use of Paradata in Survey Research. In Proceedings of the 55 th Session of the International Statistical Institute, Sydney, Australia, April [CD]. NO CLASS WEEK OF FEBRUARY 23 (UMi Spring Break) Friday, March 6, 3-5PM Class 5 Miscellaneous Other Issues Heerwegh, D. (2003), Explaining Response Latencies and Changing Answers Using Client-Side Paradata from a Web Survey. Social Science Computer Review, 21 (3): Stern, M.J. (2008), The Use of Client-Side Paradata in Analyzing the Effects of Visual Layout on Changing Responses in Web Surveys. Field Methods, published online July 2, Yan, T., and Tourangeau, R. (2008), Fast Times and Easy Questions: The Effects of Age, Experience and Question Complexity on Web Survey Response Times. Applied Cognitive Psychology, 22 (1): Friday, March 13, 3-5PM Class 6 NO CLASS WEEK OF MARCH 16 (UMd Spring Break) Wednesday, March 25, 9-11AM Class 7 Friday, April 3, 3-5PM Class 8 Friday, April 10, 3-5PM Class 9 Wednesday, April 15, 9-11AM Class 10 Friday, April 24, 3-5PM Class 11

5 Friday, May 1, 3-5PM Class 12 Saturday, May 9, 11AM-1PM Class 13 Wednesday, May 13, 9-11AM Class 14