Survey and Statistical Methods Core Population Research Seminar Series Session 4: Standardized Survey Interviewing Part 1: Controlling Interviewer Related Error Jack Fowler, PhD Center for Survey Research UMass Boston
What Interviewers Are Supposed to Do in Order to Be Consistent (Standardized) Read questions as worded Use nondirective probes when questions are not answered adequately Among other things, those probes should not materially change the content of the questions that other respondents are answering so that all respondents are answering as close to the same question as possible
Directive Probes Any probe that can be answered with a yes or no is a directive probe
Why Does It Matter? Error 1: The more interviewers innovate (eg. when they probe) the more they affect the data Error 2: Small changes in question wording sometimes have big effects on answers
Why Are There Problems? Bad question design Inadequate interviewer training Inadequate interviewer monitoring Interviewer fatigue
Question Design and Wording Complex terms and concepts Multi-barrelled questions Unclear response tasks
Three Barrels This question is about automobile injuries, including injuries from crashes, burns, and any other kind of accidents. Were you ever injured in a car accident because of your driving?
Poor Sequencing Have you been the victim of a crime in the last 12 months? What kind of crime was that?
Poor Sequencing II Do you do any kind of activity for exercise on a regular basis? What are the different kinds of things you do for exercise?
Poor Sequencing III Including any days spent in a hospital, between [START DATE] and [END DATE], did illness or injury keep [PERSON] from work more than half of the day, not counting work around the house? What were the dates that [PERSON] missed more than half a day from work between [START DATE] and [END DATE]? How many actual days was that between [FIRST DATE] and [END DATE]? Above Qs asked acceptably 72%, 42%, and 15% respectively
Unclear How to Respond When did you first vote in a Presidential election? While I was in college During the Vietnam war About 10 years ago In 1972 In what year did you first vote in a Presidential election?
Unclear How to Respond II What kind of museums have you visited in the past year? *(field coded) Art museums Science museums History museums Other *Borrowed from Hootkoop-Steenstra
Unclear How to Respond III How did you get to the game yesterday? I drove I got a ride with friends By car We circled around on Elm Street to avoid the traffic I stayed up the night before to get my homework done
Cognitive Testing Identify ambiguous terms and define them Identify complex questions and break them up Identify questions for which it is not clear how to answer and build that information into the question
Behavior Coding During pretests, identify questions That interviewers have trouble reading as written That respondents have difficulty answering without additional probing For which respondents frequently ask for clarification
Intraclass correlation This is a measure of how much interviewers affect the answers they obtain. Obviously, the ideal would be if the value is 0.
Biggest correlates of interviewer-related error The rate at which questions have to be probed, which in turn is related to: Lack of clarity of what the question is asking Lack of clarity about how to answer the question
Train Interviewers I % Who read questions acceptably by length of training Training Days < 1 40% 2 62% 5 79% 10 60% Read Acceptably
Train Interviewers II Percent who probe open ended questions acceptably Training Days < 1 13% 2 31% 5 29% 10 47% Probe Acceptably
Train Interviewers III Percent who probe closed questions acceptably Training Days < 1 48% 2 67% 5 72% 10 80% Probe Acceptably
Why Monitoring Matters Compared interviewers who were tape recorded with those whose interviewing was not reviewed by a supervisor
Interviewer Effects On Answers Measure of Interviewer effect on data (lower is better)* Monitored *Intraclass correlation P<.05 Not Monitored.008.011
Measure of how much interviewers biased results (higher is better)* *P<.04 Monitoring and Bias Monitored Not Monitored 34 2
What Are We Monitoring For? Reading questions Good probing The way interviewers interact with respondents
Interviewers Set Standards For Respondents The way an interviewer behaves has an important effect on what respondents do and how well they perform How do we know?
Experience Can Be A Negative (NSDUH) Reporting illicit drug use in lifetime Experienced Interviewer Percent Yes 33% New Interviewer 40%
Experience Can Be A Negative II Percentage of known visits to doctors reported by week of data collection Interviewer Week 1 18% 2 21% 3 23% 4 27% 5 29% Percent NOT Reported
Interviewers Communicate Interviewer concern about efficiency Low Expectations Respondent said interviewer wanted Exact Answers General Ideas TOTAL 77% 23% 100% Medium 62% 38% 100% High 66% 34% 100%
Interviewers Communicate Expectations II Interviewer concern about accuracy Low Exact Answer Respondent said interviewer wanted General Ideas TOTAL 62% 38% 100% High 76% 24% 100%
Interviewer Expectations Affect Data Please take your time and think carefully about your answers. It is important that you answers be as accurate as possible
Reported Media Use by Interview Procedure (Miller) Experimental Control Watched TV previous day 86% 66% Listened to radio previous day 67% 65% Read newspaper previous day 83% 77% Seen an X-rated movie 61% 51% Mean number of books read last 3 months 2.9% 5.3%
Monitoring Matters To show we care about reading as worded and probing To help experienced interviewers not get sloppy and tired To make sure interviewers are communicating that interviewing is a serious data collection effort
How about recruitment? Other than the obvious skills following directions, understandable speaking, computer skills as needed there are essentially no guidelines for selecting good interviewers
Matching If the obvious characteristics of interviewers are related to answers, may see an effect For most questions, demographic characteristics unrelated to answers if interviewers are well trained
How to minimize interviewerrelated error 1. Write good questions and test them 2. Train the interviewers in good standardized practices 3. Script interviewers to emphasize data quality 4. Monitor how the data collection process is carried out
THANK YOU