Survey Design Assessment Spotlight Hosted by the Student Affairs Assessment Council University of North Carolina at Chapel Hill
Surveys are Self-report data collection instruments (Gideon, 2012) Tools for collecting information on a variety of topics and times (past, present, future) Administered in a variety of formats
Gideon (2012) identified 7 steps to design a survey Define conceptual and construct variables Formulate preliminary survey items Examine preliminary questionnaire items for a number of factors Conduct a pilot study Correct and rephrase items based on pilot study Write an introduction and instructions Make any final adjustments or modifications
Step #1: Define conceptual and construct variables Conceptual definition (nominal definition) definition of a term so everyone will know what the researcher means when referring to variables in the study Construct definition (operational definition) actual traits in the empirical world that will be measured Face validity are we examining the trait we intend to measure
Step # 2: Formulate preliminary items Map the topics to be covered by the questionnaire List general topics then specific items to capture the topic
Formulate preliminary items Lived Experiences I have experienced verbal harassment I have witnessed verbal harassment Institutional commitment to diversity Senior leaders at this institution value diversity Diversity is a core value of this institution
Formulate preliminary items Use simple and direct language Use terms that will be understandable to the participants of the survey
Step #3: Examine preliminary items for a number of factors Do items fit Double-barreled questions Double negatives Leading or loaded questions Accurate and adequate response options Relevance to other items
Open-Ended vs. Closed-Ended Open ended items are good for capturing personal experiences and detail that aren t predictable based on literature review Closed-ended items are good for something that has been examined previously
Direct, Implied, and Indirect Questions Direct question asks openly about respondents opinion, belief, attitude, or behavior Implied question asks for respondent s evaluation of another s opinion, belief, attitude, or behavior. Indirect questions ask about issues in disguise; good for sensitive topics
Avoid double barrel questions Do you think women and children should be given first available flu shots? Avoid questions with the following: and, as well as, in addition to
Avoid double negatives Are you unfamiliar with the negative question rule? Better to ask: Are you familiar with the negative question rule?
Avoid leading or loaded questions Leading items create bias which increases response error Example: Do you agree that the government should do everything in its power to stop foreclosures?
Response Options Likert scale Multiple choice 5 response options for stable statistical analysis 4 response options to elicit an unequivocal opinion
Definite vs Relative Ranking Definite ranking respondent ranks each item independent of others Relative ranking respondent ranks items in relationship to each other
Check relevance to other items Items should be presented in logical order and grouped by subtopics halo effect items can create a prejudice against another item
Triangulation Several items examining same construct Good for assessing complex constructs Good for assessing stability of opinion, belief, or behavior
Step #4: Pilot Study Select a sample of people that have the same characteristics as those you wish to include in the study. Test survey and document feedback Examine the psychometric properties of reliability and validity
Step #5: Amend items as needed Step #6: Write the survey introduction Generates interest in the topic Identifies purpose of study Gives information about who is conducting the study Length of time to complete Contact information of researcher
Step #7: Final check and prep Typos, grammar, spelling Organization Clear directions or definitions Train those administering the survey
Questions & Discussion
Reference L. Gideon (ed.), Handbook of Survey Methodology for the Social Sciences, pp. 91-107, Springer Science and Business Media, New York. DOI: 10.1007/978-1-4614-3876-2_7