Survey Design. Assessment Spotlight Hosted by the Student Affairs Assessment Council University of North Carolina at Chapel Hill

Size: px

Start display at page:

Download "Survey Design. Assessment Spotlight Hosted by the Student Affairs Assessment Council University of North Carolina at Chapel Hill"

Ilene Hutchinson
8 years ago
Views:

1 Survey Design Assessment Spotlight Hosted by the Student Affairs Assessment Council University of North Carolina at Chapel Hill

2 Surveys are Self-report data collection instruments (Gideon, 2012) Tools for collecting information on a variety of topics and times (past, present, future) Administered in a variety of formats

3 Gideon (2012) identified 7 steps to design a survey Define conceptual and construct variables Formulate preliminary survey items Examine preliminary questionnaire items for a number of factors Conduct a pilot study Correct and rephrase items based on pilot study Write an introduction and instructions Make any final adjustments or modifications

for a number of factors Conduct a pilot study Correct and rephrase items based on

4 Step #1: Define conceptual and construct variables Conceptual definition (nominal definition) definition of a term so everyone will know what the researcher means when referring to variables in the study Construct definition (operational definition) actual traits in the empirical world that will be measured Face validity are we examining the trait we intend to measure

referring to variables in the study Construct definition (operational definition) actual

5 Step # 2: Formulate preliminary items Map the topics to be covered by the questionnaire List general topics then specific items to capture the topic

6 Formulate preliminary items Lived Experiences I have experienced verbal harassment I have witnessed verbal harassment Institutional commitment to diversity Senior leaders at this institution value diversity Diversity is a core value of this institution

harassment Institutional commitment to diversity Senior

7 Formulate preliminary items Use simple and direct language Use terms that will be understandable to the participants of the survey

8 Step #3: Examine preliminary items for a number of factors Do items fit Double-barreled questions Double negatives Leading or loaded questions Accurate and adequate response options Relevance to other items

9 Open-Ended vs. Closed-Ended Open ended items are good for capturing personal experiences and detail that aren t predictable based on literature review Closed-ended items are good for something that has been examined previously

10 Direct, Implied, and Indirect Questions Direct question asks openly about respondents opinion, belief, attitude, or behavior Implied question asks for respondent s evaluation of another s opinion, belief, attitude, or behavior. Indirect questions ask about issues in disguise; good for sensitive topics

for respondent s evaluation of another s opinion, belief, attitude, or

11 Avoid double barrel questions Do you think women and children should be given first available flu shots? Avoid questions with the following: and, as well as, in addition to

12 Avoid double negatives Are you unfamiliar with the negative question rule? Better to ask: Are you familiar with the negative question rule?

13 Avoid leading or loaded questions Leading items create bias which increases response error Example: Do you agree that the government should do everything in its power to stop foreclosures?

14 Response Options Likert scale Multiple choice 5 response options for stable statistical analysis 4 response options to elicit an unequivocal opinion

15 Definite vs Relative Ranking Definite ranking respondent ranks each item independent of others Relative ranking respondent ranks items in relationship to each other

16 Check relevance to other items Items should be presented in logical order and grouped by subtopics halo effect items can create a prejudice against another item

17 Triangulation Several items examining same construct Good for assessing complex constructs Good for assessing stability of opinion, belief, or behavior

18 Step #4: Pilot Study Select a sample of people that have the same characteristics as those you wish to include in the study. Test survey and document feedback Examine the psychometric properties of reliability and validity

19 Step #5: Amend items as needed Step #6: Write the survey introduction Generates interest in the topic Identifies purpose of study Gives information about who is conducting the study Length of time to complete Contact information of researcher

purpose of study Gives information about who is conducting

20 Step #7: Final check and prep Typos, grammar, spelling Organization Clear directions or definitions Train those administering the survey

21 Questions & Discussion

22 Reference L. Gideon (ed.), Handbook of Survey Methodology for the Social Sciences, pp , Springer Science and Business Media, New York. DOI: / _7

Survey research. Contents. Chaiwoo Lee. Key characteristics of survey research Designing your questionnaire

Survey research. Contents. Chaiwoo Lee. Key characteristics of survey research Designing your questionnaire Survey research Chaiwoo Lee Postdoctoral Associate MIT AgeLab chaiwoo@mit.edu agelab.mit.edu Contents Key characteristics of survey research Designing your questionnaire Writing the questions Putting them