Item Response Theory and Its Application

Transcription

1 Item Response Theory and Its Applcaton Analyss of Bomat Potentals for CAT ntroducton of the program that s beng developed Densa Denglerová Tomáš Urbánek Department of Psychology Academy of Scences Czech Republc

2 BOMAT advanced (Bochum`s Matrces Test) Authors: Rudger Hossep, Danela Turck, Mchele Hasella. Publshed n 1999 n Germany. In 2002 Czech verson. BOMAT: measures the common ntellgence smlar to Spearman`s concept of g factor, undmensonal test t s based on the smlar prncples as Raven test dfferentates better between people wth hgher ablty of ntellgence nonverbal test conssts of 39 tems tme lmt s 80 mnutes

3 Example of a tranng tem from BOMAT

4 Sample (respondents) conssts of students of Czech unverstes number of respondents s 363 equal dstrbuton between men and women age 18 44, average age 21,9 Software The parameter estmaton of tems (dffculty, dscrmnaton and guessng parameter) was calculated by the software Blog 3.

5 Three IRT Models Used for Analyss of BOMAT Items Rasch s model ) ( ) ( 1 ) ( b b e e P + = θ θ θ Brnbaum s model ( ) ) ( ) ( 1 b Da b Da e e P + = θ θ θ Guessng model ( ) ( ) ( ) ( ) b Da b Da e e c c P + + = θ θ θ 1 1

6 Comparson of dffculty parameter n 1, 2 and 3 parameters models b1 b2 b dffculty parametr parameter obtížnost -5 tem číslo number položky

7 ,8 1,6 1,4 1,2 1 0,8 0,6 0,4 0,2 0 Comparson of dscrmnaton parameter n 2 and 3 parameters models a2 a Item číslo number položky 2 dskrmnač ní parametr Dscrmnaton parameter

8 Guessng parameter for partcular tems Guessng parameter Item number

9 Item 21

10 Estmatons of Ablty dependng on the Selecton of Items one of the advantages of IRT effcency only 1 parameter model 4 selectons of tems Odd Items Even Items Frst 20 Items Fnal 20 Items All Items Odd Items 1,00 0,65 0,61 0,85 0,92 Even Items 0,65 1,00 0,66 0,82 0,89 Frst 20 Items 0,61 0,66 1,00 0,41 0,70 Fnal 20 Items 0,85 0,82 0,41 1,00 0,93 All Items 0,92 0,89 0,70 0,93 1,00 Table of correlatons among dfferent samples of tems

11 All Items Odd Items Even Items Frst 20 Items Fnal 20 Items Mean dfference -0,009-0,008-0,041-0,011 Pared t-test Standard devaton 0,340 0,480 0,620 0,320

12 Software CAT The program for admnstraton of arbtrary tems based on prncples of computerzed adaptve testng. START Intal estmaton of the latent trat (ablty) BODY OF PROGRAM Selecton of the approprate tem from the tem pool dependng on the level of the latent trat Respondent s answer New estmaton of the latent trat END Several possbltes, whch wll be dscussed n the next secton

13 I. Secton Selecton of sutable model (good model-data ft) the program works wth 3 models (1PL, 2PL, 3PL) currently t s stll possble to use other models, especally desgned for personalty testng II. Secton Selecton of procedure for estmaton of ablty margnal maxmum lkelhood estmaton margnal maxmum a posteror estmaton (Bayes estmaton) III. Secton The begnng of test several (three) randomly selected tems are admnstred to respondent, the tems should have lower parameter of dffculty (estmaton of ablty - postve motvaton of respondent) the tem wth the partcular parameter of dffculty s chosen (f the level of ablty has been approxmately known)

14 IV. Secton Selecton of the Item to Admnster the tem wth the hghest nformaton functon s value random selecton from tems, the nformaton contrbuton of whch s hgher then the a pror set value random selecton from several tems wth the hghest nformaton functon s values together wth all tems wth nformaton functon s values hgher than the a pror set value V. Secton Format of Items a respondent chooses an answer from offered choces (dfferent answers) open ended statement the answer s compared wth the lst of possble answers

15 VI. Secton Item pool All tems are taken from 1 tem pool (undmensonal ntellgence test) There are more tem pools, every pool measures dfferent trat. The tem to admnster s chosen from the pool wth the hghest standard error wthn the estmaton of the latent trat. As a result we receve more estmates of dfferent trats (Eysenck test one tem pool for neurotcsm, and other pool for extroverson). Stratfed tem pool tems n the pool are dstrbuted to groups whch are regularly alternated durng the admnstraton of the pool (f there s not a sutable tem n the current group, t s skpped). The result s one estmaton of one skll (test measures mathematcs skll of pupls and the groups are addton, subtracton, multplcaton, dvson).

16 VII. Sectons End of program Our program can be desgned to stop when: the maxmum (a pror set number of tems) test length s reached the pool has run out of the sutable tems (n the case of small tem pool) the estmaton of a latent trat exceeds the pass-fal crteron the level of the trat s estmated wth the suffcent precson the standard error s lower than the set value the dfference of standard errors between two last tems s suffcently small

17 Example of the smulated test sesson (4) Admnstered tem 7 Response was 1 Skll estmate s Standard error s Standard error delta s (5) Admnstered tem 10 Response was 1 Skll estmate s Standard error s Standard error delta s (7) Admnstered tem 5 Response was 1 Skll estmate s Standard error s Standard error delta s (8) Admnstered tem 28 Response was 0 Skll estmate s Standard error s Standard error delta s (6) Admnstered tem 29 Response was 0 Skll estmate s Standard error s Standard error delta s

18 Questons Your experence wth varous software for estmaton of parameters wthn dfferent models Usage of IRT n personalty testng, approprate models CAT n personalty testng

19 Thank you for your attenton. Contact: Densa Denglerová Tomáš Urbánek Czech Academy of Scences Department of Psychology Veveří Brno