Questionnaire Design in Telephone Surveys: Interviewers and Research Call Center Managers Experience

Transcription

1 Questionnaire Design in Telephone Surveys: Interviewers and Research Call Center Managers Experience Wojciech Jablonski University of Lodz Poland AAPOR 69th Annual Conference Anaheim, CA, USA May 15-18, 2014

2 AGENDA Introduction Method Results Conclusions 2

3 AGENDA Introduction Method Results Conclusions 3

4 CATI CHARACTERISTICS There is no visual contact between the interviewer and the respondent its has an impact on the quality of the obtained data (de Leeuw 2002) satisficing (Krosnick 1999) - CATI interviewers are much more constrained than f2f interviewers as far as using visual aids is concerned 4

5 CATI CHARACTERISTICS SPECIAL TECHNIQUES keyword summary (Dillman 1978) unfolding (Groves 1979) split questions (Dillman 1978; Dillman et al. 2009) branching (Malhotra, Krosnick & Thomas 2009) end-point only scales (Frey 1989; de Leeuw 2002) / with midpoint (Courser & Lavrakas 2012) pre-categorised questions 5

6 RESEARCH QUESTION What are the CATI interviewers and call center managers opinions and attitudes towards different response scales formats used by survey organizations? We consider the interviewers opinions of an interview to be reliable indicators of the quality of the conversation and the validity of the obtained data 6

7 AGENDA Introduction Method Results Conclusion 7

8 METHOD Data collection: & 2013 (follow-up) 12 major Polish survey research organizations V 846 interviewer questionnaires & 32 IDIs with well-experienced interviewers + IDIs with CATI fieldwork managers 8

9 AGENDA Introduction Method Results Conclusion 9

10 RESULTS ADVANTAGES/DISADVANTAGES The job of a telephone interviewer, as any other job, has both disadvantages and advantages. Below there are several features of CATI interviewer s job listed. For each of them, say to what extent you consider it an advantage or a disadvantage. Use the scale of 1 to 5 shown below, where 1 means this feature is a disadvantage and 5 means that it is an advantage. The other values on the scale may be used to express opinions in-between the two extremes. Communication with the respondent is verbal only there is no eye contact 14% 35% 48% disadvantage advantage 3% hard to say 10

11 RESULTS GENERAL TRAINING Below is a list of different elements, which make up the general training organized for interviewer candidates. For each of them, state to what extent, in your opinion, is it useful for participants. Use the scale of 1 to 5 shown below, where 1 means this issue is completely redundant (and so there is no point in discussing it) and 5 means it is very useful (and so discussing it is necessary). The other values on the scale may be used to express opinions in-between the two extremes. Information about the types of questions used in CATI scripts 5% 16% 78% redundant necessary element element 1% hard to say 11

12 RESULTS GENERAL TRAINING ELEMENT NECESSARY (points 4-5) Information about the types of questions used in CATI scripts Information about the research company (outline of the history, scope of operation) 78% 47% Information about the ESOMAR code regulations 55% Information about the Data Protection Act 65% Information about the methods of sampling in telephone research Information about the methods of selecting respondent within a household 66% 72% 12

13 RESULTS PROBLEMS Situation considered to be particularly troublesome: handling respondents who have difficulty in understanding the questions According to the interviewers the questions used in CATI scripts are often formulated using complicated vocabulary and syntax. The questions have not been adapted to suit the intellectual skills of an average respondent. 13

14 Sometimes questions are so long that when I finish reading them I don t understand what the beginning is. The respondent without written text all the more. RESULTS PROBLEMS Some questions are just too clever. Even when I look through them I have to really get into them. ( ) The questions are not designed for ordinary people but for the ones better educated. Imagine a question in which you have 39 items, an 11 point scale and you must read it 4 times for different companies. It s a nightmare. When you conduct such interviews, you can notice a moment when the respondents let their minds wander, they give answers at random. 14

15 RESULTS INTERVIEWERS STRATEGIES A majority of interviewers claim that they sometimes deviate from interviewing rules Unfolding questions with semantic (fully-labeled) scales Transforming fully-labeled scales into end-point only scales Transforming check-all-that-apply questions into itembased yes-or-no questions 15

16 RESULTS CATI FIELDWORK MANAGERS OPINION CATI call center managers partially share interviewers opinion and consider CATI script design to be one of the major factors influencing the difficulty of the interview. Reasons: Sometimes survey organization is responsible only for fieldwork has no influence on the content of the questionnaire design Pretesting questionnaires for more understandability is seldom used in market research + There are researchers who only have hazy idea of the reality of fieldwork in (CATI) research No elements of conversational interviewing are used in CATI scripts 16

17 AGENDA Introduction Method Results Conclusion 17

18 CONSLUSION Interviewers demonstrate good methodological intuition they recognize questions that according to professional guidelines should and should not be used in CATI scripts The design of some questions used in CATI scripts do not follow these regulations. As a result, interactional problems between the interviewers and the respondents occur. The interviewers sometimes solve such problems by destandardizing the interview protocol 18

19 REFERENCES Courser, M.W., Lavrakas, P.J. (2012). Item Nonresponse and the 10-Point Response Scale in Telephone Surveys. Survey Practice 5(4): 1-5. de Leeuw, E.D. (2002). Data Quality in Mail, Telephone and Face to Face Surveys. Amsterdam: TT-Publikaties. Dillman, D.A. (1978). Mail and Telephone Surveys. The Total Design Method. New York: John Wiley & Sons. Dillman, D.A., Smyth, J.D., Christian, L.M. (2009). Internet, Mail and Mixed-Mode Surveys. The Tailored Design Method. Hoboken: John Wiley & Sons. Frey, J.H. (1989). Survey Research by Telephone. Newbury Park: Sage Publications. Groves, R.M. (1979). Actors and Questions in Telephone and Personal Interview Surveys. Public Opinion Quarterly 43(2): Krosnick, J.A. (1999). Survey Research. Annual Review of Psychology 50: Malhotra, N., Krosnick, J.A., Thomas, R.K. (2009). Optimal Design of Branching Questions to Measure Bipolar Constructs. Public Opinion Quarterly 73(2):

20 THANK YOU! Wojciech Jablonski 20