Item Analysis for Key Validation Using MULTILOG. American Board of Internal Medicine Item Response Theory Course

Item Analysis for Key Validation Using MULTILOG American Board of Internal Medicine Item Response Theory Course

Overview Analysis of multiple choice item responses using the Nominal Response Model (Bock, 1972). The NRM can be estimated using Multilog.

Multilog The Multilog gprogram estimates the parameters of several IRT models for polytomous data: Graded Response Model Nominal Response Model Multiple Choice Model (Thissen and Steinberg, 1984) Like BILOG-MG and PARSCALE, Multilog was written specifically for IRT analyses, so estimation is very fast and efficient. Much of the syntax for Multilog is similar in syntax to that of BILOG-MG and PARSCALE.

Fitting the Nominal Response Model in Multilog

Multilog Example To demonstrate how to use Multilog to produce estimates of the NRM, we will use data from: A 20-item test (simulated for demonstration purposes). Each item had four score categories. Analysis of each item s distracter options will happen. 1,000 examinees took the test.

First Step: Data File Formats Prior to using Multilog we must set our data into a useable form for the program. For our example data, we will the same file used in our BILOG-MG example. The file had the response option selected by each examinee. In BILOG-MG, we needed a key file to score each examinee s repsonse profile. Unlike BILOG-MG, an answer key is not needed. Just the option each examinee selected for each item.

Data File Format

Running Multilog Multilog is a script-based program that features a graphical system to help set up the script files. To get started, open Multilog. Start All Programs Multilog 7.03 Multilog 7.03

Creating Multilog Script To begin, we will run the NRM on the example data. Click on File New. This will create a new *.mlg file containing the script that will run the analysis. Be sure to place the script in a folder that has fewer than 80 characters total (perhaps on the root drive directly).

Multilog Script NRM example...distractor analysis. >PROBLEM RANDOM, INDIVIDUAL, NITEMS=15, NGROUPS=1, NEXAMINEES=1000, NCHAR=5,DATA='EXAMPL01.DAT'; >TEST ALL, NOMINAL, NC=(6(0)15), HIGH=(5,5,5,5,5,5,5,5,5,5,5,5,5,5,5); >SAVE FORMAT; >ESTIMATE NC=2500; >TGROUPS NU=10 QP=(-4.5(1.00)4.5); >END; 6 123459 111111111111111 222222222222222 333333333333333 444444444444444 555555555555555 666666666666666 (5A1,15A1)

Multilog Script, Annotated The initial portion of the script contains the name of the analysis NRM example...distractor analysis. Unlike BILOG and PARSCALE, a comment section cannot be included. All other command sections must start with a > and be terminated by a ;

PROBLEM Command Line >PROBLEM RANDOM, INDIVIDUAL, NITEMS=15, NGROUPS=1, NEXAMINEES=1000, NCHAR=5,DATA='EXAMPL01.DAT'; DAT' The PROBLEM command supplies the general information used by the Multilog program (like the GLOBAL line in BILOG-MG). RANDOM specifies that both item and examinee parameters are to be estimated. INDIVIDUAL specifies that the data are in the form of individual response vectors (can have aggregated data, too). NITEMS specifies the number of items. NGROUPS specifies the number of groups. NEXAMINEES specifies the number of examinees. NCHAR specifies the number of characters in the ID string. DATA provides the name of the data file for the analysis.

SAVE Command Line >SAVE FORMAT; The SAVE command instructs Multilog to save estimated parameters to files. FORMAT will write the file in a useable format.

ESTIMATE Command Line >ESTIMATE NC=2500; The ESTIMATE command supplies the estimation specifics for the program. NC specifies the maximum of EM algorithm iteration cycles.

TGROUPS Command Line >TGROUPS NU=10 QP=(-4.5(1.00)4.5); The TGROUPS command supplies the number and location of the theta quadrature points. NU specifies the number of quadrature points. QP gives the location of the points.

END Command Line >END; The END command tells Multilog that the command part of the syntax has terminated. More syntax is needed still: The mapping of category codes to response parameters. The variable format statement.

Response Mapping Syntax 6 123459 111111111111111 222222222222222 333333333333333 444444444444444 555555555555555 666666666666666 The response mapping provides the key for Multilog to map the character responses in the data file to those which will be modeled by the NRM. The 6 gives the number of response options. The 123459 gives the actual options. Each line after that codes each of the 15 items into a response for each.

Variable Format Statement (4A1,1X,15A1) The variable format statement is a required statement that lists the way the data are stored in the data file. This statement uses FORTRAN-like syntax for reading data from files. The first statement, 4A1, lets BILOG know there is a column of data for the examinee id files. 4A1 gives the information that the column has a width of four characters. The 1X states that there is an empty column between the examinee id and the data. The 15A1 states the data are contained in the next 15 columns, each with zero breaks in between.

Running Multilog Now that the syntax has been created, the Multilog program must be run. Multilog runs in single phase. To run the analysis: Save the input script file first. Click on Run Run What should happen after you run the Multilog analysis. After a few minutes, you will hopefully see a set of text files pop-up in the Multilog window saying things completed successfully. Other times a pop-up p p box will appear that indicates an error within the program. Debugging can be difficult in Multilog.

Multilog Output To view the output from the Multilog session click on View and select the only option (named after your syntax file name, but ending with.out). An example from one item s output is shown on the next slide.

Multilog Item Output ITEM 12: 6 NOMINAL CATEGORIES, 5 HIGH CATEGORY(K): 1 2 3 4 5 6 A(K) -1.08-0.04 0.38 0.15-0.38 0.96 C(K) 1.22 0.64 0.45 0.42-2.55-0.18 Item Parameter Estimates CONTRAST-COEFFICIENTS (STANDARD ERRORS) FOR: A C CONTRAST P(#) COEFF.[ DEV.] P(#) COEFF.[ DEV.] 1 111 1.04 (0.15) 116-0.58 (0.13) 2 112 1.46 (0.18) 117-0.78 (0.15) 3 113 1.23 (0.18) 118-0.81 (0.15) 4 114 0.70 (0.61) 119-3.77 (0.52) 5 115 2.04 (0.20) 120-1.40 (0.19) @THETA: INFORMATION: (Theta values increase in steps of 0.2) -3.0 - -1.6 0.055 0.068 0.085 0.105 0.130 0.159 0.193 0.231-1.4-0.0 0.274 0.319 0.364 0.406 0.442 0.468 0.482 0.484 0.2-1.6 0.474 0.453 0.425 0.393 0.360 0.328 0.299 0.272 1.8-3.0 0.248 0.227 0.208 0.191 0.176 0.162 0.149 OBSERVED AND EXPECTED COUNTS/PROPORTIONS IN CATEGORY(K): 1 2 3 4 5 6 OBS. FREQ. 390 173 161 144 7 125 OBS. PROP. 0.3900 0.1730 0.1610 0.1440 0.0070 0.1250 OBS. PROP. 0.3900 0.1730 0.1610 0.1440 0.0070 0.1250 EXP. PROP. 0.3927 0.1727 0.1597 0.1432 0.0070 0.1246

Viewing Item Parameters Multilog makes viewing the item parameters a bit easier with the inclusion of IRT Graphics, a package for plotting the estimated IRFs. To view some of the item results: Close the output window (the lower x at the top right). Go to Run Plot. The IRT Graphics program should open.

IRT Graphics To view the item parameter results, click on the ICC button (at the top right). For each item, the ICC is plotted.

IRT Graphics Example Item Characteristic Curve: 5 Nominal Response Model 1.0 2 0.8 Probability 0.6 0.4 0.2 1 3 4 5 6 0-3 -2-1 0 1 2 3 Ability Category legends Item: 5 Solid Lines: 1= Black 2= Blue 3= Magenta 4= Green 5= Red Dotted Lines: 6= Black (High Category: 5)

IRT Graphics Interpretation Consider Item 5 Here, the correct option was Category #4. Category # 4 is the most likely response option when theta values are high. Category #2 is more often selected when theta values are low. Probab bility Item Characteristic Curve: 5 Nominal Response Model 1.0 2 0.8 0.6 0.4 0.2 1 3 4 5 6 0-3 -2-1 0 1 2 3 Ability Category legends Item: 5 Solid Lines: 1= Black 2= Blue 3= Magenta 4= Green 5= Red Dotted Lines: 6= Black (High Category: 5)

Conclusion This afternoon s introduction to Multilog scratched the surface of the things the program can accomplish. The NRM can be fit using Multilog. Distracter analyses are one outcome of the NRM.

Next Transforming Cutscores into Theta Values. Standard setting. Score reporting.