Population Research Seminar Series

Size: px

Start display at page:

Download "Population Research Seminar Series"

Victoria Chandler
8 years ago
Views:

Interviewing Part 1: Controlling Interviewer Related

1 Survey and Statistical Methods Core Population Research Seminar Series Session 4: Standardized Survey Interviewing Part 1: Controlling Interviewer Related Error Jack Fowler, PhD Center for Survey Research UMass Boston

2 What Interviewers Are Supposed to Do in Order to Be Consistent (Standardized) Read questions as worded Use nondirective probes when questions are not answered adequately Among other things, those probes should not materially change the content of the questions that other respondents are answering so that all respondents are answering as close to the same question as possible

things, those probes should not materially change the content of the questions that other

3 Directive Probes Any probe that can be answered with a yes or no is a directive probe

4 Why Does It Matter? Error 1: The more interviewers innovate (eg. when they probe) the more they affect the data Error 2: Small changes in question wording sometimes have big effects on answers

5 Why Are There Problems? Bad question design Inadequate interviewer training Inadequate interviewer monitoring Interviewer fatigue

6 Question Design and Wording Complex terms and concepts Multi-barrelled questions Unclear response tasks

7 Three Barrels This question is about automobile injuries, including injuries from crashes, burns, and any other kind of accidents. Were you ever injured in a car accident because of your driving?

8 Poor Sequencing Have you been the victim of a crime in the last 12 months? What kind of crime was that?

9 Poor Sequencing II Do you do any kind of activity for exercise on a regular basis? What are the different kinds of things you do for exercise?

10 Poor Sequencing III Including any days spent in a hospital, between [START DATE] and [END DATE], did illness or injury keep [PERSON] from work more than half of the day, not counting work around the house? What were the dates that [PERSON] missed more than half a day from work between [START DATE] and [END DATE]? How many actual days was that between [FIRST DATE] and [END DATE]? Above Qs asked acceptably 72%, 42%, and 15% respectively

What were the dates that [PERSON] missed more than half a day from work between [START DATE] and [END DATE]?

11 Unclear How to Respond When did you first vote in a Presidential election? While I was in college During the Vietnam war About 10 years ago In 1972 In what year did you first vote in a Presidential election?

12 Unclear How to Respond II What kind of museums have you visited in the past year? *(field coded) Art museums Science museums History museums Other *Borrowed from Hootkoop-Steenstra

13 Unclear How to Respond III How did you get to the game yesterday? I drove I got a ride with friends By car We circled around on Elm Street to avoid the traffic I stayed up the night before to get my homework done

14 Cognitive Testing Identify ambiguous terms and define them Identify complex questions and break them up Identify questions for which it is not clear how to answer and build that information into the question

them up Identify questions for which it is not

15 Behavior Coding During pretests, identify questions That interviewers have trouble reading as written That respondents have difficulty answering without additional probing For which respondents frequently ask for clarification

respondents have difficulty answering without additional

16 Intraclass correlation This is a measure of how much interviewers affect the answers they obtain. Obviously, the ideal would be if the value is 0.

17 Biggest correlates of interviewer-related error The rate at which questions have to be probed, which in turn is related to: Lack of clarity of what the question is asking Lack of clarity about how to answer the question

turn is related to: Lack of clarity of what the

18 Train Interviewers I % Who read questions acceptably by length of training Training Days < 1 40% 2 62% 5 79% 10 60% Read Acceptably

19 Train Interviewers II Percent who probe open ended questions acceptably Training Days < 1 13% 2 31% 5 29% 10 47% Probe Acceptably

20 Train Interviewers III Percent who probe closed questions acceptably Training Days < 1 48% 2 67% 5 72% 10 80% Probe Acceptably

21 Why Monitoring Matters Compared interviewers who were tape recorded with those whose interviewing was not reviewed by a supervisor

22 Interviewer Effects On Answers Measure of Interviewer effect on data (lower is better)* Monitored *Intraclass correlation P<.05 Not Monitored

23 Measure of how much interviewers biased results (higher is better)* *P<.04 Monitoring and Bias Monitored Not Monitored 34 2

24 What Are We Monitoring For? Reading questions Good probing The way interviewers interact with respondents

25 Interviewers Set Standards For Respondents The way an interviewer behaves has an important effect on what respondents do and how well they perform How do we know?

26 Experience Can Be A Negative (NSDUH) Reporting illicit drug use in lifetime Experienced Interviewer Percent Yes 33% New Interviewer 40%

27 Experience Can Be A Negative II Percentage of known visits to doctors reported by week of data collection Interviewer Week 1 18% 2 21% 3 23% 4 27% 5 29% Percent NOT Reported

28 Interviewers Communicate Interviewer concern about efficiency Low Expectations Respondent said interviewer wanted Exact Answers General Ideas TOTAL 77% 23% 100% Medium 62% 38% 100% High 66% 34% 100%

29 Interviewers Communicate Expectations II Interviewer concern about accuracy Low Exact Answer Respondent said interviewer wanted General Ideas TOTAL 62% 38% 100% High 76% 24% 100%

30 Interviewer Expectations Affect Data Please take your time and think carefully about your answers. It is important that you answers be as accurate as possible

31 Reported Media Use by Interview Procedure (Miller) Experimental Control Watched TV previous day 86% 66% Listened to radio previous day 67% 65% Read newspaper previous day 83% 77% Seen an X-rated movie 61% 51% Mean number of books read last 3 months 2.9% 5.3%

32 Monitoring Matters To show we care about reading as worded and probing To help experienced interviewers not get sloppy and tired To make sure interviewers are communicating that interviewing is a serious data collection effort

33 How about recruitment? Other than the obvious skills following directions, understandable speaking, computer skills as needed there are essentially no guidelines for selecting good interviewers

34 Matching If the obvious characteristics of interviewers are related to answers, may see an effect For most questions, demographic characteristics unrelated to answers if interviewers are well trained

35 How to minimize interviewerrelated error 1. Write good questions and test them 2. Train the interviewers in good standardized practices 3. Script interviewers to emphasize data quality 4. Monitor how the data collection process is carried out

36 THANK YOU

WHEN TWO QUESTIONS ARE BETTER THAN ONE

WHEN TWO QUESTIONS ARE BETTER THAN ONE WHEN TWO S ARE BETTER THAN ONE Floyd Jackson Fowler, Jr., Gregory Fitzgerald, UMASS Boston, Paul Beatty, NCHS Floyd Jackson Fowler, Jr., UMASS Boston, 100 Morrissey Blvd. Boston, MA 02125-3393 Key Words: