Introduction to Survey Methodology. Professor Ron Fricker Naval Postgraduate School Monterey, California

Transcription

1 Introduction to Survey Methodology Professor Ron Fricker Naval Postgraduate School Monterey, California 1

2 Goals for this Lecture Introduce professor and course Define what we mean by the term survey Characteristics of typical surveys Distinction between polls and surveys Lay out the basic steps in conducting a rigorous research survey Discuss various survey modes Pros and cons Trade-offs Considerations in making mode selection 2

3 Contact Information Professor Ron Fricker Phone Commercial: DSN: Fax:

4 Professional Experience Academic credentials Ph.D. and M.A. in Statistics, Yale University M.S. in Ops Research, The George Washington University B.S. in Mathematics from the United States Naval Academy Teaching credentials Started teaching post-graduate courses in mid-80s Have taught at NPS, RAND Graduate School, and USC Real world credentials Former active duty naval officer Senior Statistician specializing in survey and military research at the RAND Corporation Can find out more at 4

5 Class is All About Collecting and Analyzing Survey Data In this class you ll learn: How to design good survey questions and craft an effective survey instrument About the trade-offs between various survey modes The rudiments of sampling for surveys Some statistical methods useful for analyzing survey data 5

6 Course Schedule 6

7 Course Materials Lecture notes: Binder with all slides Textbook: Survey Methodology Also excerpts from Mail and Internet Surveys and Sampling: Design and Analysis Software: JMP version 7.0 All on-line at Readings password protected: MCOTEA 7

8 What is a Survey? A survey is a: systematic method for gathering information from (a sample of) entities for the purposes of constructing quantitative descriptors of the attributes of the larger population of which the entities are members In many ways, a survey is just a form of data collection 8

9 A Special Type of Data Collection Typically surveys: Gather information by asking people questions Collect information by either (1) interviewers asking questions and recording responses or (2) respondents reading and recording their own answers Collect information from a subset of the population, a sample, rather than from all members 9

10 Polls vs. Surveys No clear distinction between the two terms Poll often used for private sector opinion studies Use many of the same design features as studies that would be called surveys Poll rarely used to describe government or scientific surveys To me, the term poll implies either a commercial or less-scientific study, or a quick turn-around survey whose results may be of short-term interest 10

11 Steps in Conducting a Survey (1) Clearly state research objective(s) Decide on survey mode(s) How will you contact potential respondents? Contact mode web, , phone, etc. In what media will the survey be given? Survey or response mode web, , phone, etc. How will you follow up with non-respondents? Follow-up mode web, , phone, etc. Determine fielding strategy How to maximize response rates? Design the survey questions and the survey instrument 11

12 Steps in Conducting a Survey (2) Determine sample size and sampling strategy Obtain Institutional Review Board (IRB) or other approval as necessary Are respondents promised confidentiality? What is the impact if their survey responses become known? Pre-test, pre-test, pre-test Give the survey to some test subjects and get their feedback What works and what doesn t? Are you getting correct data/information? Revise and re-pre-test as necessary Draw sample and field the survey 12

13 Steps in Conducting a Survey (3) Follow up with non-respondents Assemble the survey data Clean data as necessary Analyze the data and report results Summarize the data Weight as necessary and appropriate to infer back to population Calculate and report margin of error Evidence of bias? Unit and item non-response Communicate survey results to respondents if promised 13

14 One View of the Steps 14

15 Another View of the Steps Excerpted from What is a Survey? by Fritz Scheuren 15

16 Examples of Large-scale Surveys (from your textbook) National Crime Victimization Survey National Survey on Drug Use and Health Survey of Consumers National Assessment of Educational Progress Behavioral Risk Factor Surveillance System Current Employment Statistics Program 16

17 National Crime Victimization Survey (NCVS) 17

18 National Survey on Drug Use and Health (NSDUH) 18

19 Survey of Consumers (SOC) 19

20 National Assessment of Educational Progress (NAEP) 20

21 Behavioral Risk Factor Surveillance System (BRFSS) 21

22 Current Employment Statistics (CES) Program 22

23 Examples of DoD Surveys Annual Active Duty Personnel Survey Survey of MWR Services and Family Programs Military Recruiter Quality of Life Survey Adult Influencer Poll (AIP) Youth Attitude Tracking Study (now JAMRS Youth Poll) 23

24 Which Survey Mode Should I Use? You are going to conduct a research survey Goal is to make the best inference possible from sample to population within budget constraints To important questions: What is the most appropriate method to choose for a particular research question? What is the impact of choosing a particular method on survey cost and accuracy? 24

25 Traditional Survey Modes Mail Paper questionnaire sent to respondents Self-administered and mailed back Telephone Interviewers call respondents on telephone Interviewer-assisted In-person Interviewers go to respondent s home or office Interviewer-assisted 25

26 Options Have Multiplied in the Age of Computers and Telecom CATI TDE IVR/T-ACASI CAPI Text- CASI Audio- CASI Video- CASI CAI Methods 26

27 Mode Considerations Modes differ by: Degree of interviewer involvement Degree of interaction with respondent Degree of respondent privacy Channels of communication Technology usage Considerations: Cost Timeliness Accuracy 27

28 Broadly speaking Interviewer-assisted surveys more costly than selfadministered Face-to-face most expensive, then telephone, then mail, then web Interviewer-assisted surveys less prone to various errors and item nonresponse Particularly compared to self-administered paper survey More or less compared to self-administered computerassisted surveys depending on sophistication of program Self-administered less subject to sensitivity bias Can combine some self-administered questions into a predominantly interviewer-assisted mode 28

29 However, timeliness is not just about the length of time a survey is in the field Really, it s from instrument design through analysis Much of that is not affected by survey mode Compared to a mail survey, web-based surveys save on two main things: Mailing time to send out and return surveys Time required to do data entry, coding, and cleaning for paper surveys Yet, it can also take a lot of time to code, test, and de-bug very complicated web (more generally, computer-assisted) survey programs 29

30 And, cost is not just about manhours On large mail surveys, printing and postage costs can be significant 1,500 surveys x 4 mailings at $1.50 each = $9,000! 1,500 $1 incentives = $1,500 On telephone surveys must consider fixed costs of phone and CATI equipment + variable costs On face-to-face interviews, travel costs can be significant (not to mention interviewer time) But in terms of manhours, don t forget: (1) instrument design time; (2) in non-electronic modes, data entry and coding time; (3) non-response follow-up efforts; and (4) analysis time 30

31 And accuracy is not just about response rates and missing items Total survey error: what mode or modes will most decrease all types of error? Lots of considerations Frame coverage Nonresponse rates Sensitivity bias, etc 31

32 32

33 Cost and Frame Availability Often Drive Mode Choice Survey budgets never unlimited Frequently cost will eliminate one or more modes And likely drive other survey design decisions I.e., face-to-face interviews too expensive On the other hand, surveys using area probability frames usually have to start out using face-to-face interviews Similarly, if a complete frame is available, web survey is possible option But general population surveys eliminate web 33

34 Some Modes Logically Group (1) Telephone and face-to-face surveys Telephone often considered an alternative to faceto-face Both interviewer-assisted and can have similar frame coverage issues Many longitudinal surveys start out with a face-toface interview with telephone follow-ups Mail sometimes an alternative to telephone If both telephone and addresses available 34

35 Some Modes Logically Group (2) Web surveys often considered a replacement for mail surveys Coverage is still an issue for web surveys Web superior in terms of automation and speed However, literature consistently finds that nonresponse rates are higher for web surveys 35

36 Using Multiple Modes Reasons to use multiple modes: Maximize response rates Reduce/eliminate selection & coverage bias Allow respondents to use most convenient mode Reduce costs Cross-sectional surveys: start with least costly mode and move to more expensive modes as necessary to get response Longitudinal surveys: Use less expensive modes in follow-on survey rounds 36

37 Considerations When Using Multiple Modes Focus should be on designing instruments to ensure equivalency across modes Rather than optimizing for a particular mode E.g., make the follow-on web survey resemble its counterpart paper survey Ensure operation is set up to track cases across the modes Must avoid duplications and uncoordinated respondent contact Must design and field so that mode effects can be disentangled from other sample characteristics May have to randomize survey mode over a subset 37

38 Other Data Collection Methods Surveys are not the only (nor necessarily the best) way to collect data Other methods include Administrative records Focus groups and qualitative investigations Randomized experiments Which is best depends on the research question(s) and/or the purpose for which the data will be used 38

39 What We Have Covered Defined the term survey, including the Characteristics of types of surveys this class will focus on Distinction between polls and surveys Discussed the basic steps in conducting a rigorous research survey Considerations when choosing a survey mode In various ways, choice likely to affect survey Timeliness Accuracy Cost And cost may drive mode choice possibilities 39