Tips for Writing Questionnaire Items

Transcription

1 - Office f Educatinal Assessment Last Updated: Octber 2006 Tips fr Writing Questinnaire Items This handut is prvided by the UW Office f Educatinal Assessment (OEA) as a resurce fr WebQ wrkshp attendees wh want sme basic tips fr writing questinnaire items. It des nt address brader aspects f survey and research design, but additinal resurces can be fund n ur web site: OEA als prvides a variety f cnsulting and research supprt services; please feel free t cntact us if yu are interested in assistance with a research, assessment, r evaluatin prject (baytam@u.washingtn.edu, ). TIP 1: BE RELEVANT Befre starting yur survey, yu shuld specify ne r several ver arching questins yu wish t address (i.e., research questins r evaluatin questins). Yur questinnaire items shuld be fcused arund these questins. Different Dmains f Questins: Behavir, Beliefs, and Evaluatin Des prviding ptinal, nline practice quizzes help students learn the material in my curse? Behavir Of the ten ptinal nline practice quizzes, hw many did yu cmplete? Beliefs Cmpleting the nline practice quizzes helped me learn the material in this curse. Neutral Evaluatin Hw satisfied were yu with the nline quizzes? The three items abve have different fci. The first culd help answer yur research questin if yu were able t crrelate respnses t the item with actual grades. Cpyright 2006 University f Washingtn Office f Educatinal Assessment

2 Different Types f Questins: Open- and Clsed-Ended Hw d transfer students benefit frm ging t Transfer Thursday sessins in the Gateway center? Clsed ended: Which f the fllwing d yu see as benefits f attending Transfer Thursday sessins? (check all that apply) Learning abut majr requirements Meeting with a UW adviser Meeting ther transfer students Other (specify): Open ended: What d yu see as the benefits f attending Transfer Thursday sessins? Data frm an pen ended questin can help yu create check all that apply lists r ther clsed frmat items. The primary drawback f pen ended questins is the time it takes t analyze the data; if resurces are limited, use pen ended questins sparingly. TIP 2: BE SPECIFIC Avid abstract terms and jargn The in-class activities in this class facilitate active learning. Neutral The in-class activities (e.g., illusry crrelatin demnstratin, eye-witness testimny activity) help keep me engaged in class. Neutral In the revisin, the terms in class activities and active learning have been specified. Prvide clarifying details Hw many times did yu visit CLUE? Hw many times did yu visit the Center fr Learning and Undergraduate Enrichment (CLUE) -- evening tutring in Mary Gates Hall? Write ut acrnyms r abbreviatins the first time. Prvide further descriptrs if apprpriate. Frequency estimates Hw many times d yu visit CLUE? Never Nt ften Fairly Often All the time During a typical quarter, hw many times d yu visit CLUE? Never 1-3 times Abut every ther week Abut nce per week 2 r mre times per week The less definite the respnse ptins, the mre ʹniseʹ r errr in the data (i.e., participants will vary in their interpretatin f the ptins). UW OEA: Tips fr Writing Questinnaire Items 2

3 TIP 3: AVOID CONFUSION Avid duble-barreled questins The TAs in this curse were helpful and respnsive. Neutral Rate the perfrmance f the TAs n the fllwing dimensins. Helping yu understand the material Respnding t inquires abut yur grade Pr Pr Excellent 5 Excellent 5 Avid duble- wrding The instructr fr this curse did nt respnd t students' cmments during class. Neutral The instructr fr this curse was respnse t students' cmments during class. Neutral In the first case, students wh thught the instructr was respnsive (a trait) wuld have t chse strngly disagree (a answer). Watch ut fr such mental puzzles. TIP 4: USE APPROPRIATE SCALES Prvide a spectrum f ptins Did yu enjy the interactive nline activity abut Newtn's 3rd Law? N Yes T what extent did yu enjy the interactive nline activity abut Newtn's 3rd Law? Nt at all A little much Typical scales ment Neutral Satisfactin Evaluatin Pr Fair Gd gd Excellent unfair Unfair Fair fair Nt at all Nt t Knwledge Nt at all Nt t UW OEA: Tips fr Writing Questinnaire Items 3

4 Uniplar vs. Biplar scales Uniplar Nne Sme Biplar Often, the presence f ne attribute des nt necessitate the absence f anther. Fr example, ne can enjy bth arts and sciences, s they shuld nt be presented with a scale that frces them t chse ne r the ther. Beware f impsing a structure r mdel that may nt be valid. Prviding a (neutral) midpint With midpint Neither nr Withut midpint Opinins vary regarding the apprpriateness r usefulness f a midpint, particularly fr agreement scales. Research suggests a frced respnse will yield rughly the same prprtin f agree/disagree as a scale with a midpint. Fr example: Results with a midpint: Results withut a midpint: 30% cn, 50% neutral, 20% pr 60% cn, 40% pr Number f respnse anchrs Shuld ne label all f the respnse ptins? In general, the mre specific yu are abut the respnse ptins, the less yu will have t wrry abut idisyncratic interpretatins f thse ptins by the respndents. In sme instances (e.g., frequency estimates) it is imperative that yu label all ptins. In ther cases, yu culd increase errr by trying t ʹfrceʹ a label. Whatever system yu chse, make sure that yur labeling is symmetrical (e.g., d nt label nly ʺ1ʺ and ʺ2ʺ and ʺ5ʺ n a 5 pint scale). N/A: Use sparingly! NO I wuld have cmpleted the nline practice quizzes, even if they did nt cunt twards my grade. N/A OK The ptinal nline practice quizzes imprved my grade n the midterm. N/A Anther alternative wuld be t direct respndents t skip a set f items if they had nt cmpleted any f the ptinal nline practice questins, making the N/A ptin unnecessary. ʺNt applicableʺ and ʺdnʹt knwʺ shuld be avided fr pinin and attitude items. "Check all that apply" lists Make lists as exhaustive as pssible. Use previusly cllected pen ended data if available. Unless all ptins are knwn, it is almst always apprpriate t prvide a space fr a write in ʺtherʺ respnse. UW OEA: Tips fr Writing Questinnaire Items 4

5 TIP 5: AVOID BIAS Avid leading questins The UW is cnsidering having wireless capacity fr the entire campus within 3 years. Given the ptential impact n students' behavir in the classrm, what d yu think f this prpsal? IFFY Hw d yu think all-campus wireless capacity will affect students' behavir in the classrm? BEST Hw, if at all, d yu think all-campus wireless capacity will affect students' behavir in the classrm? D yu think all-campus wireless capacity will affect students' behavir in the classrm? Definitely nt Pssibly Definitely yes If yes r pssibly, in what ways d yu think it will affect students' behavir? In the first example, there is an implicit assumptin that expanded capacity will affect students behavir in class. The secnd wrding is better, but it still des nt make clear that respndents are free t deny any effect; the third wrding prvides that ptin. In the last example, the additin f a clsed frmat intrductry questin allws fr easy tabulatin f verall pinins frm the pen ended questin. Beware hidden assumptins and cntingencies During the past quarter, hw ften have yu skipped evening scial events t study? In the curse f a typical seven-day week, hw many hurs d yu spend ding curse-related wrk utside f class? In the curse f a typical seven-day week, hw many hurs d yu spend scializing with friends fr nn-academic purpses? The verall questin r bjective is t find ut hw students in different majrs balance their scial lives and study time. The first example assumes that the respndent wuld nrmally take part in scial events. Furthermre, there is an implicatin that all scial events take place in the evening, that nly evening scial events ʹcuntʹ, r that respndents typically nly study during the evening. Finally, the respndent wuld be bliged t think back and average ver an entire quarter. REFERENCES/RESOURCES Bradburn, N. M., Sudburn, S. & Wansink, B. (2004). Asking questins: The definitive guide t questinnaire design. San Francisc, CA: Jssey Bass. Czaja, R. & Blair, J. (2005). Designing surveys: A guide t decisins and prcedures. Thusand Oaks, CA: Sage Publicatins. Fwler, F. J. (2002). Survey research methds, 3rd Ed. Thusand Oaks, CA: Sage Publicatins. Fwler, F. J. (1995). Imprving survey questins: Design and evaluatin. Thusand Oaks, CA: Sage Publicatins. Petersn, R. (2000). Cnstructing effective questinnaires. Thusand Oaks, CA: Sage Publicatins UW OEA: Tips fr Writing Questinnaire Items 5