Greg Pope, Analytics and Psychometrics Manager 11am-12:30pm, Monday March 15

Transcription

1 Greg Pope, Anlytics nd Psychometrics Mnger 11m-12:30pm, Mondy Mrch 15 My bckground in eductionl nd psychologicl mesurement (psychometrics) Been in the testing industry for bout 14 yers Spent 7 yers s psychometricin with the high stkes testing progrm in Albert, Cnd (Albert Eduction) After tht spent 3 yers designing psychometric softwre (sttisticl cheter nlysis, computer dptive testing) Joined Questionmrk in lte 2006 in product mngement Designed the Test Anlysis Report Worked on the Results Mngement System (RMS) Currently working on Questionmrk s new reporting pltform nd course evlution reports Slide 2 1

2 By show of hnds: How mny people work in corportions? How mny work in cdemic institutions? How mny work in government/militry? Other? By show of hnds: How mny people feel very comfortble with psychometrics (item nd test sttistics)? How mny feel somewht comfortble? How mny feel not comfortble nd wnt to be more comfortble? Slide 3 This session will present some of the theory behind ssessment nlysis to put the numbers into context The discussion will be s non-technicl s possible with more pplied pproch Questionmrk tools tht cn help evlute the performnce of ssessments nd ssessment items will be presented with pplied exmples If you hve questions plese do sk during the session Slide 4 2

3 A brief review of the theory Putting theory into prctice, some Questionmrk tools Item nlysis report Test nlysis report Results Mngement System (RMS) Summry Guidelines nd stndrds Question nd nswer period Slide 5 CTT (Clssicl Test Theory) is wht we ll know nd love: P-vlues Discrimintion sttistics (point-biseril correltions, high minus low performnce, etc.) Been round long time (most of the 20 th century) Works very well for most pplictions nd by fr the most widely used form of item/test nlysis Ability to work with smller smple sizes (e.g., or less) Reltively simple to compute (not fitting dt to model) Hs different set of ssumptions from IRT Slide 6 3

4 IRT (Item Response Theory) is n lterntive tht some of us my hve herd of or use: -prmeter: Item discrimintion b-prmeter: Item difficulty c-prmeter: Item pseudo-guessing informtion Been round since the 1960s (Lord) Mkes things like computer dptive testing (CAT) nd dvnced test development techniques possible More complex to compute (fitting dt to model) Requires lrger smple sizes depending on the number of prmeters, more prmeters more prticipnt responses needed (e.g., 700+ for 3-PL) More informtion: Slide 7 CTT item nlysis IRT item nlysis: Item chrcteristic curve (ICC) Slide 8 4

5 Wht is better CTT or IRT? Ech re used for their own purposes nd hve pros nd cons Questionmrk currently uses CTT in our products Flexible in terms of smple sizes Fst to compute with few or no computtionl gotchs People re fmilir with these sttistics nd so do not require lerning new mesurement model CTT meets the needs of 99% or more of customers CTT sttistics re relted to IRT sttistics to some degree P-vlues re highly correlted with b-vlues Point-biseril correltions re highly correlted with -vlues We will be discussing CTT tody Slide 9 Relibility is used in every dy lnguge: My cr runs relibly mens it strts very time We re going to be tlking bout test score relibility Essentilly: How consistently the test scores mesure construct We cn t go into ll the detil here tody, for good primer into the theory see: Trub, R.E. (1994). Relibility for the Socil Sciences: Theory & Applictions. Thousnd Oks: Sge. Slide 10 5

6 An ssessment is mesurement instrument comprised of mny individul mesurements (questions/items) Wht is being mesured is the bility, trit, construct, ltent vrible of interest (n investment bnking test my mesure the construct knowledge of investment bnking ) All mesurement instruments hve error in their estimtes, so the trditionl view of test score relibility sys tht person s: Slide Mesurements mde by thermometer of temperture re imperfect (tmospheric vribles, sunlight, etc.) To mitigte this tke lots of mesurements using different, high qulity thermometers Slide 12 6

7 Q1 Mesurements mde by test question of construct re imperfect (prticipnt ftigue, psychologicl vribles, etc.) To mitigte this tke lots of mesurements using different, high qulity questions Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Slide 13 Four pproches for mesuring relibility: 1. Internl consistency: Correltions of items comprising the test (how well do they hng together ) 2. Split-hlf (split-forms): Correltion of two forms (splits) of the test (first 25 items versus lst 25) 3. Test-retest: Correltion between multiple dministrtions of the sme test 4. Inter-rter relibility: Correltion between two or more rters (mrkers) who rte the sme thing (e.g., provide essy scores) Slide 14 7

8 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Slide 15 Kuder-Richrdson Formul 20 (KR-20) First published in 1937 Designed for dichotomous (1/0, right/wrong) items Vlues rnge from 0 to 1 (closer to 1 = higher relibility) Cronbch s Alph Cronbch published in 1951 Designed for dichotomous nd nondichotomous (continuous, 1 to 5) items Generlly vlues rnge from 0 to +1 (closer to +1 = higher relibility) Used in the Test Anlysis Report nd RMS Slide 16 8

9 1. Item difficulty Items tht re extremely hrd or extremely esy ffect discrimintion nd therefore relibility. If lrge number of prticipnts do not hve time to finish the test this ffects item difficulty 2. Item discrimintion Items tht hve higher discrimintion vlues will contribute more to the mesurement efficcy of the ssessment (more discriminting questions = higher relibility). Prt of this reltes to sound question development, if questions re well crfted nd non-mbiguously worded they re more likely to hve cceptble discrimintion 3. Construct being mesured If ll questions re mesuring the sme construct (e.g., from the sme topic) relibility will be incresed 4. How mny prticipnts took the test With very smll numbers of prticipnts the relibility coefficient will be less stble Slide Composition of people tht took the test If the smple of prticipnts tking n ssessment is not representtive (e.g., no-one studied!) the relibility will be negtively impcted 6. How mny questions re dministered Generlly the more questions dministered the higher the relibility (to point, we cn t hve 10,000 question test!) 7. Environmentl dministrtion fctors Conditions in the testing re such s noise, lighting levels, etc. cn cuse distrction wy from the mesurement of wht the prticipnts know nd cn do 8. Person fctors Test nxiety, ftigue, nd other humn fctors cn reduce the ccurcy of mesurement of wht people know nd cn do Slide 18 9

10 High stkes Medium stkes Low stkes Cronbch s Alph: 0.90 or higher (excellent) Cronbch s Alph: 0.70 or higher (good/cceptble) Cronbch s Alph: The higher the better but not lwys possible to get high vlues (e.g., short formtive quizzes) Slide 19 Vlidity (of test scores) refers to: Whether the test is mesuring wht it should be mesuring The processes followed to crete the test nd test questions is sound nd defensible Tht experts hve hd chnce to review nd rubber stmp the processes Tht the test results predict the intended outcomes Tht the scores re use ppropritely So vlidity is not number it hs to do with following best prctices, conducting studies nd reserch, using results firly, etc. In order for n ssessment to be vlid it must be relible Slide 20 10

11 Orgnized into three ctegories: 1. Criterion-relted (predictive/concurrent): Are scores relted to outcome mesures? (e.g., do SAT scores predict post-secondry performnce) 2. Content-relted: Is the test content pproprite for wht is being mesured (e.g., is content coverge representtive of the curriculum) 3. Construct-relted: Is the test mesuring the correct trit/ttribute/bility/construct? (e.g., is this mth bility test mesuring mth bility) Slide 21 The scores tht re reported to prticipnts nd other stkeholders must be derived from high qulity questions In order for the prticipnt to obtin mening nd chieve lerning from ssessment results the ssessments must mesure wht they re supposed to mesure relibly Two core spects of question qulity hs to do with the nlysis of difficulty nd discrimintion Slide 22 11

12 Number of results How mny prticipnts hve nswered the question Importnt to know how stble the sttistics re, generlly more prticipnts nswering the question mens more stbility of sttistics Number not nswering How mny prticipnts hve not nswered the question Importnt to know if there ws problem with the question wording (prticipnts got confused nd didn t nswer) or the disply (n imge wsn t displying properly onscreen) tht need to be ddressed Also impcts the number of results Question difficulty (p-vlue) The difficulty of question represented s proportion of the prticipnts tht nswered the question correctly (in the cse of multiple choice items) A p-vlue for question of mens tht 55% of prticipnts choose the correct nswer for the question Question discrimintion The degree to which questions differentite or discriminte between prticipnts who know the mteril well nd those tht do not know the mteril well. Exmples of discrimintion sttistics: High minus Low: The proportion of exminees of the top 27% of prticipnts (in terms of ssessment score) minus the lowest 27% (should be lrge positive difference between high nd low) Item-totl correltion coefficient: A Point-biseril correltion between exminee ssessment scores nd item scores (higher item scores should men higher ssessment scores) Slide 23 How prticipnt responds to question sys something bout wht they know nd cn do Question difficulty hs to do both with the question nd the prticipnt This question is esy becuse lots of prticipnts selected the correct nswer This prticipnt got this question right nd so they hve demonstrted knowledge/skills t this level Question difficulty is on scle tht is relted to prticipnt bility (knowledge/skills) Slide 24 12

13 Discrimintion refers to how well n item discrimintes/differentites between prticipnts of different knowledge/skill levels Experts in n re should get higher scores on the question nd higher scores on the overll ssessment Novices in the sme re should get lower scores on the question nd lower scores on the overll ssessment Slide 25 Composed of severl sections: Informtion section Item difficulty (p-vlue) histogrm nd Item discrimintion (outcome discrimintion) histogrm Question by question detiled nlysis Summry informtion Informtion section provides detils regrding when the report ws creted, etc. Slide 26 13

14 Summry informtion (t the bottom of the report) Provides summry of the verge p-vlue, discrimintion nd item totl correltion Slide 27 Item difficulty (p-vlue) histogrm nd Item discrimintion (outcome discrimintion) histogrm Provides summry of the # of item by difficulty nd discrimintion Some hrder questions Most verge Some esier questions Some poor Most verge Some gret Slide 28 14

15 Rn out of time? Question by question detiled nlysis Provides detiled nlysis of ech question Too hrd, why? Upper/Lower splits Are distrcters pulling people? The more the better You wnt high positive vlues Slide 29 Slide 30 15

16 Slide 31 Liner reltionship between outcome discrimintion (horizontl xis: high-low) nd outcome correltion (verticl xis: point-biseril) y = x r = R² = Slide 32 16

17 I m going to present severl exmples of rel life questions nd their psychometric performnce Lets try to see why the questions re performing well or re not performing well sttisticlly These exmples re ll tken from the Questionmrk Blog where I hve posted them over the pst severl months: Slide 33 Good smple size Not too hrd or too esy Nice nd high Expected ptterns ll the wy Everyone nswered Lots of diff between upper nd lower Summry: This question performs well psychometriclly speking! Slide 34 17

18 Good smple size Hrd question Not good, should be positive Wit second Boise should be the correct nswer! This response option shows the expected response pttern Everyone nswered Not good, negtive diff between upper nd lower Summry: This question ws mis-keyed, once the keyed correct nswer is chnged in the RMS it performs quite well. Slide 35 Problem with the question or with instruction? Fir smple size Not too hrd or too esy Too low, not cceptble Upper people seem to like the Pltykurtic option, why? Everyone nswered Not gret, smll diff between upper nd lower Summry: This question is borderline, for medium/high stkes ssessment this item does not perform well enough sttisticlly. Slide 36 18

19 Good smple size Esy question but tht is lright Pretty good Joke distrcters tht don t drw nyone The correct nswer performs well though Everyone nswered Resonbly good diff between upper nd lower Summry: This question hs clssicl problem, two of the distrcters don t work (don t drw ny prticipnts). SME should come up with more plusible distrcters for this question nd hve it re-bet tested. Slide 37 Smll smple size Very hrd question Negtive, not good People re ll over the plce in terms of selecting n nswer Lots of no responses Very low discrimintion Summry: This question is quite poor. There re too mny vribles to consider to be ble to sy how mny questions should be on ny type of ssessment (there probbly is not correct nswer for this question). Slide 38 19

20 Good smple size Very hrd question High negtive vlue, not good People re ll over the plce with no ide wht the nswer is Lots of no responses Extremely low discrimintion Summry: This question is bsolutely terrible! The person who wrote it should be relesed from service nd the question stricken from the Perception repository, it is not slvgeble. Slide 39 The Test Anlysis Report provides informtion to evlute the psychometric performnce nd qulity of your ssessments Slide 40 20

21 Composed of severl sections: Informtion section Tble of test sttistics Topic level sttisticl brekdown Frequency distribution Histogrm Informtion section provides detils regrding when the report ws creted, etc. Slide 41 Remember, reducing smple size reduces mesurement robustness Tble of test sttistics nd Topic level sttisticl brekdown Provides the sttisticl detils for the overll ssessment s well s t the topic levels Slide 42 21

22 The more prticipnts the better The more The lower the better for these (except skew, closer to Tble of test sttistics items the nd Topic level sttisticl brekdown better to 0 better) Provides the sttisticl detils for the overll ssessment s well s point t the topic levels Becuse it is bout mesurement consistency Slide 43 The more items in ech topic the better to point The higher the better The higher 2010 Users Conference the Mimi better Frequency distribution nd Histogrm Displys the ssessment results in tbulr form nd grphiclly Middle line medin, most scores re between percentiles Some higher scores some lower scores Dt displyed grphiclly Slide 44 22

23 The Item Anlysis Report nd Test Anlysis Report produce snpshot of informtion in sttic form Wht if you need/wnt to drop questions, chnge question scoring nd see dynmiclly wht the effects of mking chnges to your ssessment results would be? Welcome to the RMS Slide 45 Relese in 2008, dd-on to Questionmrk Perception Review items in test nd drop, credit or lter scoring Review test results nd define pss or cut score Get rel-time preview of how proposed chnges will impct overll item sttistics nd test relibility Publish results into flt file dtbse for ccess by reporting tools Mintin chnges within n udit tril to id ssessment defensibility Slide 46 23

24 Drop or Credit Questions Review Item Difficulty Edit Angoff Estimte Borderline Item Discrimintion Flgged Low Item Discrimintion Flgged Rel-time summry Slide 47 Slide 47 Distribute, Clculte, Sve Understnding some of the theory cn help determine why questions re performing well or re not performing well Questionmrk tools, such s the Item Anlysis Report, Test Anlysis Report, nd Results Mngement System, provide the mechnisms to put theory into prctice nd get the most out of your ssessments Applying (s much s possible) medium/high stkes stndrds to low stkes ssessments will improve the informtion glened from ssessments for ll stkeholders Slide 48 24

25 Relted resources Slide 49 Questionmrk Blog: Test Anlysis Report guide: ls/er/report_types/test_nlysis.htm RMS user guide: x White ppers: x Trining sessions: Creting Assessments Tht Get Results ( Slide 50 25

26 Guidelines nd stndrds informtion Slide 51 Guidelines nd stndrds documents provide the informtion needed to mke/ensure tht progrm is defensible For exmple, recent inititive tht Questionmrk is involved with: ATP Test Security Guidelines ( Designed to provide ssistnce for testing orgniztions who wish to develop Test Security Pln An orgniztion tht implements nd follows these guidelines will be defensible in terms of test security Slide 52 26

27 Regultory Principles for e-assessment (2007). Developed by the Qulifiction nd Curriculum Authority (QCA) in collbortion with the Council for the Curriculum, Exmintions nd Assessment in Northern Irelnd (CCEA), the Welsh Assembly Government, nd the Scottish Qulifictions Authority (SQA). For more informtion, nd to downlod copy of the document, plese visit: Tidbits: QCA requires orgniztions to provide prctice tests to prticipnts before they tke the ctul ssessments (Perception does this nicely nd cost effectively) It lso requires orgniztions to produce regulr reports on the performnce of different lnguge versions of questions/tests Slide 53 Guidelines for Computer-Bsed Testing (2000; The Guidelines for Computer-Bsed Testing (2000) is publiction produced by the Assocition of Test Publishers (ATP), of which Questionmrk is n ctive member ATP Security Pln Guidelines (in progress, Provides informtion on wht is required to ensure ssessment security More informtion t product centrl, check it out Slide 54 27

28 NCTA Professionl Stndrds nd Guidelines for Post- Secondry Test Centers ( This resource in intended for orgniztions tht conduct testing t the post-secondry level nd require informtion on test centre best prctices Guidelines for Computerized Adptive Test Development nd Use in Eduction (1995; This resources is designed for those involved in the development of computerized dptive testing (CAT) progrms Slide 55 Stndrds for the Accredittion of Certifiction Progrms (2005). Progrms developed by the Ntionl Commission for Certifying Agencies (NCCA). Downlod from: /93/Defult.spx Interntionl Test Commission (ITC) Projects ( These projects re trnslted into severl lnguges, provide best prctices in severl res nd re intended for ny orgniztion involved in low-, medium-, nd high-stkes ssessment: ITC Guidelines on Adpting Tests ITC Guidelines on Test Use ITC Interntionl Guidelines on Computer-Bsed nd Internet- Delivered Testing Slide 56 28

29 ISO 17024: Conformity ssessment Generl requirements for bodies operting certifiction of persons. The interntionl stndrd provides the requirements necessry to ensure tht certifiction orgniztions conduct the processes surrounding certifiction in consistent, comprble, nd relible wys. The stndrd is designed to provide guidnce on how to estblish relibility cross certifiction orgniztions vi confidentility, best prctices, nd ethicl conduct. Obtin vi the ISO online store ( Slide 57 Stndrds for Eductionl nd Psychologicl Testing (1999; Presents best prctices for brod udience, gered towrds orgniztions developing eductionl (e.g., stte or provincil level chievement tests) nd psychologicl tests (e.g., personlity tests) t vrious levels of stkes (low to high). The guidelines nd stndrds outlined within re pplicble to ny re of ssessment, not just K-12 eduction or psychologicl tests Very thorough, well orgnized, well written Mny well respected people contributed to the guidelines Slide 58 29

30 Uniform Guidelines on Employee Selection Procedures (1978). These guidelines incorporte single set of principles which re designed to ssist employers, lbor orgniztions, employment gencies, nd licensing nd certifiction bords to comply with requirements of Federl lw prohibiting employment prctices which discriminte on grounds of rce, color, religion, sex, nd ntionl origin. Access: ml Slide 59 Code of Fir Testing Prctices in Eduction ( This publiction pplies brodly to testing in eduction (dmissions, eductionl ssessment, eductionl dignosis, nd student plcement) for ll modes of presenttion (computer, pper-nd-pencil, etc.) Principles for Fir Student Assessment Prctices for Eduction in Cnd ( s/eng_prin.pdf). This document is intended for students, prents, techers, nd orgniztions involved in eduction bsed ssessment Hs some relly useful info, not just for Cndins Slide 60 30

31 Generlly they re lid out into numbered sections (exmple from the ATP Test Security Guidelines): Slide 61 Not ll the stndrds nd guidelines out there re pproprite for ll orgniztions It is importnt to choose document which best fits your orgniztion s context, needs nd purposes Obviously QCA my be pproprite if you re certifiction orgniztion in the UK Some of the generl documents provide useful informtion for ny progrm nd following the guidelines cn help improve n ssessment progrm For exmple, the Stndrds for Eductionl nd Psychologicl Testing provide detiled guidnce on how to develop nd run high qulity, defensible ssessment progrm regrdless of stkes Slide 62 31

32 Once guidnce document is chosen n orgniztion should: Go through ech section of guidelines document nd determine whether they currently meet the requirement or whether they need to do work in order to meet the requirement If requirement is met then it should be internlly documented how the requirement is met If the requirement is not met then it should be internlly documented wht needs to be done in order to meet the requirement A risk nlysis should be conducted to determine how urgent it is to meet the requirement. This cn guide resource decisions nd timelines Slide 63 Setting up tble listing the guideline requirements nd summry of whether the requirements hve been met or not is useful first step: Slide 64 32

33 If requirement is not met in the document there could be severl resons why (which should be documented): It is not pplicble to n orgniztion s sitution It is not possible to ttin due to specific situtions It is possible to ttin t some point in the distnt future It is possible to ttin in the short/medium terms Slide 65 Mny stndrds nd guidelines hve supplementl mteril tht help mke it cler wht needs to be done in order to comply QCA hs lots of cse studies nd supplementl mterils on their web site ( ANSI supplementl mteril web site ( Items.spx) ATP Test Security Guidelines Supplementl Hndbook (currently in progress, drfts vilble on wiki) Most guideline nd stndrds documents hve supplementl guides on their web sites or upon request Slide 66 33

34 The most importnt thing to do is centrlly store: Documenttion of the nlysis of the different sections of the stndrd/guidelines (wht is being done, wht is not nd why) Documenttion of the processes followed for those requirements which re met Documenttion of n ction pln to meet the requirements which re currently not met nd n pproximte timeline Slide 67 When ANSI conducts n evlution of n orgniztion for complince with 17024, documenttion is extremely importnt They require severl copies (digitl or pper) of ll documenttion for ech section of the requirements They lso conduct on site inspections (e.g., to see if test mterils re in fct stored in secure loctions) They provide n opportunity for n orgniztion to updte/modify their processes nd documenttion before finl decision is mde Slide 68 34

35 Your orgniztion my not decide to go through forml review process through ANSI Your orgniztion could do n internl udit using the most pproprite guideline/stndrd document to see where you re most defensible nd where you re lest defensible in terms of complince with the requirements An internl ction pln cn help fcilitte chnges over time to ddress res of wekness internlly nd highlight res of strength to your stkeholders externlly Bottom line you don t need to go through the forml ccredittion process to obtin business vlue from using one or more guidelines documents Slide 69 We cn t go through ll the stndrds/guidelines individully tody but we cn identify common requirement themes nd tips on how to meet the requirements: 1. Psychometric qulity of ssessments 2. Item nd test design nd development 3. Security of mterils, informtion, nd personnel 4. Orgniztionl budgeting, structure, nd role definitions 5. Accessibility Slide 70 35

36 Test Anlysis Report detils on sttistics Slide 71 A mesure of the symmetry of the distribution of scores (i.e., whether scores re pushed or skewed to one side or the other). Rnges from bout -2 to +2 Negtive skew 20 p r t 14 # i 12 c o i 10 f p 8 6 n t 4 s 2 0 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Prticipnt scores Norml distribution (no skew) 20 p r t 14 # i 12 c o i 10 f p 8 6 n t 4 s 2 0 0% 20% 40% 60% 80% 100% Prticipnt scores Positive skew 20 p r t 14 # i 12 c o i 10 f p 8 6 n t 4 s 2 0 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Prticipnt scores Slide 72 36

37 Negtive skew # o f p r t i c i p n t s % 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Prticipnt scores Slide 73 Norml distribution (no skew) # o f p r t i c i p n t s % 20% 40% 60% 80% 100% Prticipnt scores Slide 74 37

38 Positive skew # o f p r t i c i p n t s % 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Prticipnt scores Slide 75 ( n n( n 1)( n 1) 2)( n 3) z 4 3( n 1) ( n 2)( n 2 3) A mesure of the symmetry of distribution of scores (i.e., how peked/pointed versus flt re the distribution of scores wht is hppening t the tils). Norml rnge from bout -3 to +3 Negtive kurtosis (flt: pltykurtic) Norml distribution (zero kurtosis: mesokurtic) Positive kurtosis (pointed: leptokurtic) 18 p 16 r 14 t 12 # i c 10 o i 8 f p 6 n t 4 s 2 0 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Prticipnt scores 20 p r t 14 # i 12 c o i 10 f p 8 6 n t 4 s 2 0 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Prticipnt scores 20 p r t 14 # i 12 c o i 10 f p 8 6 n t 4 s 2 0 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Prticipnt scores Slide 76 38

39 Positive kurtosis (pointed: leptokurtic) # o f p r t i c i p n t s % 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Prticipnt scores Slide 77 Negtive kurtosis (flt: pltykurtic) # o f p r t i c i p n t s % 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Prticipnt scores Slide 78 39

40 Norml distribution (zero kurtosis: mesokurtic) # o f p r t i c i p n t s % 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Prticipnt scores Slide 79 The most commonly used mesure of centrl tendency (refers to the middle of distribution of scores) Rnge of vlues depends on scores Slide 80 40

41 Another mesure of centrl tendency, less sensitive thn the men to outliers Rnge of vlues depends on scores Where 50% of prticipnts obtined higher scores nd 50% of prticipnts obtined lower scores Slide 81 A third mesure of centrl tendency, used gret del in survey nlysis The most common score in distribution of scores Rnge of vlues depends on scores 34% 43% 56% 56% 56% 63% 67% 76% 88% Mode = 56% Slide 82 41

42 Prticipnt s scores minus the men gives sense of the spred/vrition of test scores The spred or vrition of scores between prticipnts Are the scores spred out (e.g., 0 to 100%) or clustered together (e.g., ll scores between 55 nd 62%) Rnge of vlues depends on scores Rick s test score = 75% Slly s test score = 83% Mrk s test score = 53% Ell s test score = 91% Stndrd devition = 16.36% Slide 83 Another mesure of vrition The first step in clculting stndrd devition: The stndrd devition is the squre root of the vrince Rnge of vlues depends on scores Used in some dvnced clcultions (e.g., Anlysis of vrince: ANOVA, Multiple nlysis of vrince: MANOVA) Slide 84 42

43 The "spred" or stndrd devition of test scores for to relibility prticipnt if tht prticipnt hd been theoreticlly ssessed repetedly using the sme test Refers to the inherent error found surrounding ny test score observed test score = theoreticl true score + error Relted to test relibility (the more relible the test the lower the stndrd error) Rnge is dependent on size of stndrd devition (which is dependent on scores) nd test relibility coefficient mgnitude Typicl rnge: 1 to 20 Slide 85 Hey wht s this? It s relibility! The mount of error on test is inversely relted Product knowledge test Theoreticl test score = 65.2% Theoreticl test score = 66.4% Theoreticl test score = 63.7% Theoreticl stndrd devition = 1.26% Rick s observed score = 66.1% Slide 86 Theoreticl test score = 67.1% Theoreticl test score = 65.8% Theoreticl test score = 67.5% Theoreticl test score = 65.9% 1.26% of error surrounding Rick s observed score 43

44 Hey wht s this? It s the number of prticipnts Conceptully very similr to the stndrd who took the test! The more error of mesurement but rther thn prticipnts the closer we get to referring to error in n individul prticipnt s popultion representtion score, this refers to how much error there is in determining the true popultion men The lrger the smple size (i.e., number of prticipnts) the smller the stndrd error of the men The more prticipnts in smple the greter likelihood tht it pproximtes the popultion Slide 87 Smple (153 prticipnts) Popultion (87,000 prticipnts) p 30 r t 25 # i c 20 o i f p 15 n 10 t s 5 p r t # i c o i f p 8000 n 6000 t 4000 s % 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 0 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Prticipnt scores Prticipnt scores Smple men = 56.78% Smple stndrd devition = 15.21% Stndrd error of men = 1.23% within plus or minus 1 stndrd error of the men vlue, 68 times out of 100, the true popultion men will reside Slide 88 Typiclly the popultion informtion is not known, but if you could see the informtion Popultion true men = % 44