Making Surveys Work for You: Some Important Design Considerations

Transcription

1 Making Surveys Work for You: Some Important Design Considerations By Don A. Dillman* For Net-Conference on Survey Methods and Measurement June 15,11 *Regents Professor in the Department of Sociology and Deputy Director of the Social and Economic Sciences Research Center at Washington State University, Pullman, Washington Welcome to this web conference on designing effective surveys. I hope that you will find it useful. My focus today is on how survey methods are, changing, and to provide helpful suggestions for navigating through these changes. Let me begin with some of the questions you asked in the material sent to me, and how I ll go about responding. 2 A few of your questions. Providing answers is a challenge? What errors are introduced when modes are combined? Can landline and cell phones be combined effectively? Is it okay to use mail primarily, with telephone follow-up? Is there bias from internet surveys? How can response rates be improved (Incentives?) and how much is enough? What can we do to get youth and minorities to respond? What is the state of the science on framing opinion questions? Can you give us practical guidelines for urban vs. rural areas? Different answers may fit different situations. When I looked at these (and other) questions, I thought I could be most useful to focus on criteria for decisionmaking and give examples of what s possible or practical and then, responding to questions will allow us to go deeper into the explanations. First, some background. 3 4 None of our main data collection methods are working as well as we would like when used alone. Telephone has ~70% coverage; response rates have declined steady; now as low as-15% in national surveys. The Internet has effective household coverage < 70%; (access + skill +frequency of use) and response rates are often low. In-person methods affordable for only a few surveys, and coverage is increasing problem. Attempts to circumvent these limitations are prevalent but also have problems Cell phones added to RDD samples can improve coverage of the general public, but are also expensive. In addition, cell phone numbers are often carried from one geographic area to another when people move. Effective household Internet access is lower than the number of connections, because of infrequent use. We cannot use to access general public households. We have no sampling algorithm equivalent to RDD. We can only obtain web response by approaching through postal mail

2 Mail Survey Methods are experiencing a revival of interest The U.S. Postal Service makes available (through vendors) all residential addresses (The Delivery Sequence File, or DSF) It is now our best household sampling frame, and may be our only source for accessing nearly all (~ 95-97%) households. One can request by mail the return of a mail questionnaire and/or a response via web. Two design considerations now seem to dominate survey methodology Tailored Design matching data collection modes to populations, topics, and survey task. (Your studies intrigue me in this regard.) Mixed-mode surveys, using 2 nd, 3 rd mode to overcome problems of single mode, is becoming increasingly important as a way of dealing with the shortcomings of individual modes. Thus, we need to figure out where mail, web and telephone can and cannot help us collect better quality survey data in both situations. 7 8 Are Mixed Modes More of a Solution or More of a Problem? Perhaps there is no choice! To talk about these the need for tailoring and for mixed-mode designs I need to provide some context: Response rates are declining (telephone) Often we cannot reach everyone by single modes. Sample Frames are less adequate (random digit dialing) Cannot contact random samples of internet addresses But, are we getting improvement in data quality or perhaps producing even larger problems by mixing modes? Do we get the same measurement across modes? Do we reduce non-response error through adding a new mode or only increasing measurement problems? Doing successful sample surveys means that we must conduct sample surveys in a way that reduces error to the extent practical. We also need to recognize that there are multiple sources of error in sample surveys. 9 What a Survey Can Do: Provide the distribution of a characteristic in a population by surveying only a small portion of the members of that population. which allows us to answer questions like these: What percent of Minnesota residents now smoke? What percent of unemployed adults in Minneapolis use tobacco each day? Do students in suburban high schools have higher rates of tobacco use than students in inner city schools? How do sample surveys allow us to make precise estimates for the opinions and behaviors of millions from a survey of hundreds or a few thousand respondents?

3 In Order for Surveys to Make Estimates With Known Precision, We Must Limit Four Kinds of : Coverage Sampling Coverage How it happens: Occurs because all members of the population are not given an equal or known (non-zero) chance of being selected for the survey. Measurement Non-response Incomplete, poorly constructed and out-of-date lists (e.g. employees, members, and/or customers). All households do not have telephones. Many people do not have Internet access or the skills to use it For Example ~25%+ of households do not have landlines. Only ~65% of households have internet access from home (additional 11% from somewhere else). Some internet t connections are by modem. Households without internet access or landline access differ from others in significant ways. Sampling Occurs because only some members of the population are allowed to participate in the survey while others are systematically excluded. The general (but incomplete) indicator of precision is completed sample size. Number of completed interviews and precision of estimates at 95% confidence (quadrupling number of completed questionnaires decreases sampling error by one-half). 0 = +/- % 11 = +/- 3% 0 = +/- 5% 20 = +/- 2% 15 If % of our 0 respondents say they smoke cigarettes each day we can be 95% confident that between 55% and 65% of all people in our population smoke cigarettes each day (ignoring other possible errors). 16 The Pascal Triangle and its Application An Inappropriate Perspective for an Election Survey How is it possible to make estimates with stated confidence (e.g. 95%) that tens of thousands or millions of people have a particular characteristic, while surveying so few people? Voters on election day All Voters are Unique

4 A Necessary Perspective in an Election Survey Why Does Random Sampling with Complete Coverage Work? Voters on election day It lets probabilities work, as described in a Pascal Triangle. The Pascal Triangle shows the probable outcomes of 1 to random draws of respondents from populations in which one-half of respondents have each characteristic. In a two candidate election there are TWO types of voter. Let s assume % are going to vote for each candidate. 19 Chances in Ten Consecutive Draws of Randomly Selecting All Combinations of Red and Blue Voters Interpretation of the Draws Number of Draws Sum of Chances Chances of Blue - 1/24 Chances of 5 red 5 blue - 252/24 Chances of 4-6 of each color - 672/ Number of Randomly Needed Depends Mostly on Sample Size and Not on Population Size An Observation Sample Size for +/- 3% Precision ,057 1,066 1,068 Population of this size 0 0 1,000 5,000,000 0,000 1,000,000 0,000,000 Much of the traditional design work in random sample surveys has been to assure high coverage so that random sampling will be effective, i.e. all people who vote have an equal chance of being selected in the sample. But, there are also other sources of error to consider

5 Measurement Occurs because of inadequate question wording, placement in the questionnaire, the interviewer and/or the respondent. For example, Interviewer Presence May Have a Normative Influence: How would you rate your health? Some known sources of measurement error: Mode effects: Social desirability Primacy/Recency Acquiescence Question order effects Different visual layouts Measurement differences can be a significant problem when trying to use more than one mode for the same study. Percent Excellent Good Fair Poor Mail Telephone Face-to-face Respondents more likely to give culturally acceptable answer on telephone or in-person Non-response Occurs because some people do not respond when requested to answer the survey questions, and they are different in some way from those who do answer. How $2 Token Cash Incentive in Advance Influenced Response Rates Across Age Groups 1993 Survey of Washington State New Drivers License Holders 80 For example: 70 With $2 Without $2 Are older people more likely to respond than younger people? Are employed people more likely to respond than unemployed people in an employment rate survey? 0 < Age To summarize, it s important to reduce each kind of error. Maximizing one at expense of others is NOT a good idea. A Case Study on seeking improvement by mixing modes Coverage Measurement Sampling Nonresponse the Gallup Organization tried to switch from telephone to web for periodic cross-sectional employee studies in 1999, and noticed significant differences in measurement. At the same time, some response rates appeared to be falling. These concerns led to an experiment on the effects of four survey modes on measurement and switching modes as a means of improving response rates. 29 5

6 Case Study: an attempt to improve data quality by offering multiple modes Four modes were examined: Telephone, Interactive Voice Response (IVR), web and mail. Can response rates be improved by offering two modes in sequence? What are the measurement consequences? For details, see: Dillman, Tortora, Phelps, Swift, Kohrell, Berck, and Messer, 09. Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response, and the Internet. Social Science Research, 38(1), Case Study Comparison: Telephone, IVR, Mail, and Web Questions were comparable across modes, using the preferred format for telephone. Q2. Overall, how satisfied are you with your long distance company? 1 Not at all satisfied Extremely satisfied Q3. Considering what you get and what you pay for it, how would you rate the overall value of your long distance company s products and services? 1 Terrible Outstanding 32 How the study was implemented A sample of individuals that submitted warranty information on new purchases in order to get access by either mail or telephone. A two-step implementation process; offer initial mode with multiple contacts and after a pause of 2-3 weeks offer the second mode. Mail to telephone. Telephone to mail. IVR to telephone. Web to telephone 33 Response rates improved significantly after a mode switch was made % 81% 51% 48% Additional Response Initial Response 34 Effects of Initial Modes on choosing most positive category for five longdistance satisfaction questions Aural modes produced different results than visual modes ent Perce 0 Percent choosing positive labeled end-point Telephone IVR Mail Web T I M W T I M W T I M W T I M W T I M W Q2 Q3 Q4 Q5 Q6 Responses to aural modes (telephone, IVR) were similar. Responses to visual modes modes (web, mail) grouped together. Control panels showed that primacy/recency not a cause of measurement differences;. Traditional explanations of differences e.g. social desirability seemed unlikely

7 Was the higher response rate worth the cost? Probably not, because of the measurement differences that resulted. Telephone obtained more extreme answers regardless of whether first mode or last mode Additional panels showed no primacy/recency effects. The demographics did not change in a way that suggested different kinds of respondents answered the second mode. Either mail or telephone alone would probably have been adequate, except for the measurement difference. Why this example? Mixing survey modes is not always a desirable solution. Doing that requires us to deal with many measurement issues. But, using mail-alone alone may also be undesirable. To sort out the possibilities and implications, we need to think about coverage, non-response and measurement Two Ways that Switching Modes May Change Measurement? 1. Mode choice influences question structure and wording. 2. Presence vs. absence of interviewer (e.g. telephone vs. web) influences respondent answers. 1. How Changes in Wording and Construction Influence Answers Each mode has evolved to favor certain ways of asking questions to maximize the potential to obtain quality data. This tendency comes from the tradition of how each mode was designed to maximize its potential use, regardless of how it might be done in another survey mode. 39 Two Examples of Question Construction Differences: Not offering the same answer categories in all modes. Asking an open-ended question in one mode and a closed-ended d d question in the other modes. Unintentional Construction Effect: Volunteered Responses on Telephone To what extent do you favor or oppose expansion of NATO? Telephone: Mail: 1 Strongly favor 1 Strongly favor 2 Somewhat favor 2 Somewhat favor 3 Somewhat oppose 3 Somewhat oppose 4 Strongly oppose 4 Strongly oppose 5 NO OPINION 6 DON T KNOW 7 REFUSE

8 Unintentional Construction Difference: Open vs. Closed Ended Marital status was asked differently by one organization on telephone and web (Tortora, 04). Telephone: What is your marital status? Web: Which of the following best describes your marital status? Single Married Separated Divorced Widowed 43 Unintentional Construction Difference: Results from a Telephone and Web Comparison What is your marital status? Differences were significant and tracked across monthly data collections. Telephone (n =,380) Web (n = 9,713) Difference Single Married Separated Divorced Widowed TOTAL How does the presence of interviewer/human interaction influence respondent answers A substantial literature shows that mail obtains different answers to many questions than for interview surveys. Among the reasons: Presence of the interviewer (interaction expectations) Presumed characteristics ti of interviewer i and desire to be seen in a favorable way. Aural and visual information are perceived using different sensory systems and processed using separate neural centers in the brain. Aural communication places higher demands on respondent memory. Is the presence of an interviewer helpful or not? Interviewers control the delivery of survey information whereas respondents have control over how they complete the survey in self-administered surveys. Interviewers can have positive impacts by clarifying information and motivating respondents. Interviewers can have negative effects because respondents are more likely to give socially desirable answers and acquiesce (rather than disagree) in the presence of an interviewer Interviewer Presence May Have a Normative Influence: Social Desirability An Effect of Interviewer Presence: Acquiescence, the Tendency to Agree Percent Respondents are more likely to give culturally acceptable answers on the telephone How often do you drive a car after drinking alcoholic beverages? Frequently Occasionally Seldom Never Don t Know Mail Telephone ( 2 = 14.8 p <.001) Percent Percent of respondents who agree with each of 9 opinions on how to encourage seat belt use Mail n=884 Telephone n= (Dillman & Tarnai, 1991, pp ) 8

9 Greater Extremeness on Four-Point Scales in General Public Survey Percent choosing not a problem (vs. small, moderate, or serious) Telephone Mail Will Scalar Questions With Little or No Social Desirability Produce the Same Results Across Modes? Telephone respondents tend to give more extreme positive answers than do web respondents In some cases social desirability, acquiescence or primacy/recency do not seem to explain why Mean for 9 Questions: Telephone, 47.1 Mail, 31.9 Difference, (Dillman & Mason, 1984) Does Presence of an Interviewer Influence Response to Scalar Questions? A series of identically worded scalar questions was tested in a web telephone experiment. Please note at this point that we are starting to bridge between interviewer presence and visual vs. aural communication. These attributes are intertwined. If we use exactly the same wording for scalar questions in aural vs. visual surveys Will we get the same answers A series of evaluations were made for a variety of scalar questions in random sample student surveys at Washington State University Question formats evaluated in Telephone vs. Web Experiments Fully labeled, 1-step. 5-category scales 9 comparisons. 7-category scales 6 comparisons. Polar-point labeled, 1-step. 5-category scales 13 comparisons. 7-category scales 4 comparisons. 11-category scales 4 comparisons. 5-Category Fully Labeled Scales Telephone Survey How satisfied are you with Washington State University as a place to go to school? Would you say you are Very Satisfied, Somewhat Satisfied, Neutral, Somewhat Dissatisfied or Very Dissatisfied? Web Survey Branched 2-step scales. 7-category scales comparisons

10 5 4 3 Mean Responses by Mode: 5-Category Fully Labeled Scales * * Telephone * * * * * * Web 5-Category Polar-Point Labeled Scales Telephone Survey On a 5-point scale where 5 means very satisfied and 1 very dissatisfied, how do you feel about the quality of INSTRUCTION in the classes you have taken at WSU? Web Survey 2 1 Q2 Q16 Q24 Q5 Q21 Q25 Q5 Q21 Q25 Satisfied/ Dissatisfied Agree/Disagree 55 Construct Specific * p Mean Responses by Mode: 5-Category Polar-Point Summary of Findings on Mode Effects in Scalar Questions * * * * * * * Telephone * * Web * * All six scale formats we tested produced a slightly more positive response on telephone than on web for the vast majority of test items. 5 and 7 category fully labeled 5, 7, and 11 category polar point labeled 7 category 2-step scales 2 1 Q2 Q4 Q6 Q7 Q16 Q24 Q5 Q21 Q25 Q18 Q19 Q9 Q17 Satisfied/Dissatisfied Agree/Disagree Extremely Likely/ Best Possible/ Not Likely Worst Possible 57 * p.05 See: Christian, Leah Melani, Don A. Dillman, and Jolene D. Smyth. 08. The Effects of Mode and Format On Answers to Scalar Questions in Telephone and Web Surveys. In Lepkowski, et al. Advances in Telephone Survey Methodology. New York: Wiley-Interscience. pp Conclusions: Interviewer Effects in Scalar Questions Respondents answer scales more positively and are more likely to select the extreme positive category on the telephone compared to web, mail, or face-to-face (when a showcard is provided). Finding a solution to scalar questions for which social desirability seems not to apply may be as difficult as when it does apply. Implications Presence of Interviewer May need to avoid combining telephone and either web or mail modes, especially for panel surveys, where attempts to measure change from Time 1 to Time 2 are being made. Statistical adjustment may be a possibility, but I leave that to others to figure out. 59

11 However, What if we combined mail and web. They are both visual, and measurement differences should be quite small for most questions. Also, if we can use Postal Service addresses without names, maybe we can get people to go to the web. Let s see what might happen if we do that ; Mail-only surveys with incentives added achieve reasonable response rates Four general public surveys with 12 page questionnaires, $5 token cash incentive with request, 3-4 follow-ups, respondent-friendly design. (Stern and Dillman, 06; Smyth et al. ; Messer and Dillman, Forthcoming) 05 regional Lewiston/Clarkston social participation survey, Telephone directories 66% 07 Lewiston/Clarkston Quality of Life Survey Addressed-based sample (ABS), 71% 08 Washington state-wide community satisfaction Survey, ABS, most recent birthday 55% 09 State of Washington Household Economic Survey Knowledgeable respondent, ABS 68% Study Procedures (Smyth, Dillman, Christian and O Neill,, Applied Behavioral Science) 12 page questionnaire, items, up to 80 responses (depending upon branching); a -25 minutes survey Four contacts. Pre-notice letter. Questionnaire (or web request). Thank-you post card. Replacement questionnaire (adjusted by treatment). $5 token cash incentive included with initial mail questionnaire or web request. Data collected November 7, 07, to January, Additional background on building mail implementation systems. (Results of an Experiment, Contacts x Incentive Sent With the Request vs. Post payment). Incentive 1 st Mailing (%) 2 nd Mailing (%) 3 rd Mailing (%) 4 th Mailing (%) No incentive $1 Cash $5 Cash $5 Check $ Check $ Check $ Check Promise of $ Note: each treatment group contained 1 subjects (James and Bolstein 1992) 64 Treatments for Lewston/Clarkston Study 1. Mail preference: Send mail questionnaire and mention web with initial request. 2. Mail preference: Send mail questionnaire but withhold mention of web for about two weeks. 3. Web preference: Web invitation with no mail questionnaire, but explain that mail questionnaire will be sent in about two weeks. 4. Equal preference: Encourage response by either mode equally. Sample Sizes Treatment n A. Mail Preference (mention web) 0 B. Mail Preference (no web mention) 0 C. Web Preference (mail in 2 weeks) 0 D. Equal preference (respondent s choice) 0 TOTAL

12 Tailoring to the survey topic and location and creating a unified design between mail and web In the slides that follow Pictures of location to be surveyed. Creating common screens for mail and web. Tailoring/personalizing the survey to the location and population Photos taken of local landmarks, artwork, and symbols to make survey recognizable and visually attractive. Use of common branding for mail and web. Choice of stationary, envelopes and content based upon rethinking of personalization strategies given that name could not be used Mail Questionnaire Unified design between mail (on left) and web (on right) Use of personalized images to help respondents feel connected to survey and to place an emphasis on study area instead of on survey source. Also used similar color and design to connect to web survey The opening web page Used an entry page similar to front cover of paper survey, still focusing on making the survey recognizable through familiar images. Design of the web survey focus on population not sponsor Question 2 Similar design format to paper survey, and use of familiar image in upper left-hand corner of the screen

13 Personalized Correspondence All letters used WSU stationary. Exterior of Envelopes (2 nd and 4 th Contacts) Used WSU address labels. Used a return label showing the photo from survey cover and the survey title to increase familiarity. Photo of questionnaire cover used to tie different elements together Initial vs. Final Response Rates Treatments Mail preference (mention web early) Initial Response Rate (%) Final Response Rate (%) Difference Mail preference (no web in early contacts) Web preference (no mail in early contacts) Equal preference (choice) Final response rate, paper preference vs. web preference Treatments Web (%) Paper(%) Total (%) Mail preference (mention web early) Mail preference (no mention web) Web preference Equal preference (choice) Response rate summary Postal DSF with addresses only (i.e., no names) can produce reasonably high response rates 71% mail preference and 55% web preference. Using a postal letter to drive people to the web was quite effective 41% responded by web. Adding the mail option after two weeks brought in considerably more respondents 14% Is non-response error a problem? It does not help much to improve response rates if our respondents are different from non-respondents on variables important to the study objectives. Thus, we need to compare respondent characteristics on web vs. mail within the different treatment groups. Adding a mail option to the web was more effective than adding a web option to the mail 14% vs. 1%

14 Lewiston/Clarkston: Demographics in the web preference treatment by mode of response Responded by Mail (n = 80) Responded by Web (n = 232) Difference Mean age < yrs > 65 yrs % Female % White % Employed (full time) % Retired % With 4-year degree % Married % Widowed Red indicates significant difference at.05 level. 79 Web respondents were quite different from mail on 7 of 24 substantive attitudes/opinions Mail Web Diff. % attached to the area % think willingness for community involvement has increased % think fish population increased % more internet use improves quality of life % think more cell use improves quality of life % think enviro. protection is too weak % gray wolves not threat to domestic animals Red indicates significant difference at.05 level. 80 But web + mail (all web preference respondents) quite similar to mail preference group. Mail Preference (n=261) Web Preference (n=312) Difference Mean age < yrs > 65 yrs % Female % White % Employed (full time) % Retired % With 4-year degree % Married % Widowed Red indicates significant difference at.05 level. 81 Summary: types of respondents by mode Web on its own brings in specific types of respondents and leaves others out. Web and mail used together bring in a wider range of respondents comparable to mail used alone. Our best chance of reducing nonresponse error is to use both. 82 What happens when respondents are given a choice of which mode to use? Treatments Web (%) Mail (%) Total (%) Mail preference (mention web early) Mail preference (no mention web) Web preference Expansion to state-wide data collection A similar model was used, i.e. personalize questionnaire to the state with pictures. We pushed to the web by withholding mail. Two tests in successive years. Equal preference (choice) Response rates for the choice treatments are lower than for the mail preference (no mention) treatment but higher than for the web preference treatment

15 Statewide Mail Questionnaire with map and regional pictures 12 X 8.5 booklet with personalized images to help respondents feel connected to survey and to place an emphasis on study area instead of on survey source. Also used similar color and design to connect to web survey. (Messer and Dillman, Forthcoming) Web Questionnaire Designed to emulate the mail questionnaire and appeal to the local population Response Rates From Mail only is higher than web + mail in three studies. But, 2/3 of respondents were successfully pushed to respond via web. Mail Only Web + Mail Lewiston/Clarkston Rural Region Survey-07 71% 41% + 14% = 55% Washington Community Survey-08 57% 31% + 15% = 46% Washington Economic Survey % 34% + 18% = 52% A few conclusions We no longer have a good sampling frame for telephone surveys If we mix telephone (and web or mail) we stand a good chance of obtaining different measurement. Mail-alone may provide a good option, but implementation methods affect response. Web + mail has lower response than mail-alone, but may be a good option for future Your questions You may have questions about what I presented here, or you may wish to go beyond what I have done here and ask about things not yet discussed