Random Effects Logistic Regression Models as Item Response Models

Transcription

1 Random Effects Logistic Regression Models as Item Response Models Carolyn J. Anderson Department of Educational Psychology I L L I N O I S university of illinois at urbana-champaign c Board of Trustees, University of Illinois Spring 2014

2 Outline Motivation Basic Concepts HLM Example Logistic Regression As a multilevel model. Connections with IRT Example of Vocabulary Test Rasch Model (1PL) 2PL 2PL with predictors for θ References C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

3 Motivation Often we wish to use θ as a response variable in a regression model (i.e., see what influences or explains variability in θ). Estimating θ and using it in a regression is non-optimal because it introduces error error due to estimation of θ. Solution: Put predictors of θ into the IRT model, e.g., θ j = ν 1 (age) j +ν 2 (HS degree) j +ν 3 (Elementary) j +e j. May also want to put predictors for difficulty parameters into the IRT model, e.g., b i = b i +η 1 (Instructions) i. C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

4 Models have Different Names For normally distributed response variable, Hierarchical Linear Models Multilevel Analysis using Linear Mixed Models Variance Components Analysis Random coefficients Models Growth curve analysis All are special cases of Generalized Linear Mixed Models (GLMMs) C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

5 Examples of Hierarchies Individuals within groups Level 2 Group 1... Level 1 person 11person12 Longitudinal Level 2 Level 1 Person 1... time 11 time12 time 1t1 Group 2... Group N... person 1n1 person 21person22 person 2n2 person N1 personn2 person NnN Person 2... Repeated Measures Level 2 Person 1 Person Level 1 trial 11 trial12 trial n1 trial 1 trial Person N... time 21 time22 time 2t2 time N1 timen2 time NtN... Person N... trial n2 trial N1 trialn2 trial NnN C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

6 Item Response Data as Clustered Data Clustered structure leads to Dependencies in data. Standard errors for parameters that are too small. Random Effects Models provide opportunities to Include covariates at both levels of data. Greater efficiency. IRT Data Structure: Level 2 Level 1 Examinee 1... item 11 item12 item 1n itemn2 Examinee 2... Examine 3 item 21 item22 item 2n2 item N1 item NnN C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

7 A Little Example: NELS88 data National Education Longitudinal Study conducted by National Center for Education Statistics of the US department of Education. Data consist of the first in a series of longitudinal measurements of students starting in 8th grade. Data were collected Spring I obtained the data used here from deleeuw/sagebook We will look at 23 out of the 1003 schools. Y ij = Math scores of student i in school j. X ij = Time per day that student i in school j spends doing homework. C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

8 Math Scores by HOMEW C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

9 Math Scores by HOMEW C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

10 All Regressions in One Figure C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

11 Random Effects Model Level 1: Y ij = β 0j +β 1j x ij +ǫ ij where ǫ ij N(0,σ 2 ) i.i.d.. Level 2: β 0j = γ 00 +U 0j β 1j = γ 10 +U 1j where ( U0j U 1j ) (( 0 M.V.N. 0 ) ( τ00 τ, 10 τ 10 τ 20 )) i.i.d. and independent of ǫ ij. C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

12 Linear Mixed Model Mixed Random and Fixed Effects. Y ij = γ 00 +γ 10 x ij +U 0j +U 1j x ij +ǫ ij }{{}}{{} Fixed Random C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

13 Logistic Regression Y ij is a dichotomous response (e.g., correct/incorrect) and assumed to be a binomial random variable. Linear predictor, e.g., Link is logarithm of odds log β 0 +β 1 x ij ( ) P(Yij = 1) P(Y ij = 0) Putting these together gives us logistic regression model, ( ) P(Yij = 1) log = β 0 +β 1 x ij P(Y ij = 0) exp(β 1 ) = odds ratio associated with a 1 unit change in x ij. C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

14 Multilevel Logistic Regression Just change Level 1 Model to a logistic regression with random regression coefficients, Level 1: log Level 2: (the same) ( ) P(Yij = 1) = β 0j +β 1j x ij P(Y ij = 0) β 0j = γ 00 +U 0j β 1j = γ 10 +U 1j where ( U0j U 1j ) (( 0 M.V.N. 0 ) ( τ00 τ, 10 τ 10 τ 20 )) i.i.d. C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

15 Logistic Mixed Model Replacing Level 2 Models for the β pj s into Level 1 model yields ( ) P(Yij = 1) log = γ 00 +γ 10 x ij +U 0j +U 1j x ij P(Y ij = 0) or exp(γ 00 +γ 10 x ij +U 0j +U 1j x ij ) P(Y ij = 1) = (1+exp(γ 00 +γ 10 x ij +U 0j +U 1j x ij )) C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

16 Using Framework To get a Rasch Model Suppose we have 4 items where Y ij = 1 for a correct response and Y ij = 0 for incorrect. Explanatory Variables are indicator variables, { 1 if resonding to item 1 x 1j = 0 otherwise { 1 if resonding to item 4...x 4j = 0 otherwise Level 1: The linear predictor η ij = β 0j +β 1j x 1j +β 2j x 2j +β 3j x 3j +β 4j x 4j and the link is the logit: P(Y ij = 1 η ij ) = exp[η ij] 1+exp[η ij ] C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

17 To get Rasch Model (continued) Level 2: β 0j = U 0j N(0,τ 00 ) β 1j = γ 10 β 2j = γ 20 β 3j = γ 30 β 4j = γ 40 Our model for each item P(Y 1j x 1j,...,x 4j,U 0j ) = exp[γ 10 +U 0j ]/(1+exp[γ 10 +U 0j ]) P(Y 2j x 1j,...,x 4j,U 0j ) = exp[γ 20 +U 0j ]/(1+exp[γ 20 +U 0j ]) P(Y 3j x 1j,...,x 4j,U 0j ) = exp[γ 30 +U 0j ]/(1+exp[γ 30 +U 0j ]) P(Y 4j x 1j,...,x 4j,U 0j ) = exp[γ 40 +U 0j ]/(1+exp[γ 40 +U 0j ]) C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

18 Set The IRT Connection γ i0 = b i, the difficulty for item i U 0j = θ j, value on latent variable for examinee j. For item i, P(Y ij = 1 x 1j,...,x 4j,U 0j ) = exp(γ i0 +U 0j )/(1+exp(γ i0 +U 0j )) = exp(b i +θ j )/(1+exp(b j +θ j )) One Parameter Logistic Regression Model. The model can be fit in more standard format of P(Y ij = 1 x 1ij,...,x 4j,θ j ) = exp( b i +θ j )/(1+exp( b j +θ j )) C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

19 Rasch Example Data are response to 10 vocabulary items from the 2004 General Social Survey from n = 1155 respondents. I used the more standard format of the model. The model was fit using SAS PROC NLMIXED. Edited output: NOTE: GCONV convergence criterion satisfied. Fit Statistics -2 Log Likelihood AIC (smaller is better) AICC (smaller is better) BIC (smaller is better) C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

20 Rasch Parameter Estimates b are easiness parameters: Standard t Pr Parameter Estimate Error DF Value > t Gradient b < b < b < b < b < b < b < b < b < b < std < var(theta) <.0001 C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

21 Item Characteristic Curves: Rasch/1PL C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

22 2 Parameter Logistic Model Back to just 4 items: For 2PL, we need a slope (discrimination parameter) for each item. This is a generalized non-linear mixed model. Change the model for intercept to β 0j = (a 1 x 1j +a 2 x 2j +a 3 x 3j +a 4 x 4j )U 0j Model for item i becomes P(Y ij = 1) = = exp(γ i0 +a i U 0j ) (1+exp(γ i0 +a i U 0j )) exp(b i +a i θ j ) (1+exp(b i +a i θ j )) C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

23 2 Parameter Logistic Model (continued) Model for item i becomes P(Y ij = 1) = = exp(γ i0 +a i U 0j ) (1+exp(γ i0 +a i U 0j )) exp(b i +a i θ j ) (1+exp(b i +a i θ j )) Can be re-parameterized if desired: P(Y ij = 1 θ) = exp( b i +a i θ j ) (1+exp( b i +a i θ j )) or P(Y ij = 1 θ) = exp(a i ( b i +θ j )) (1+exp(a i ( b i +θ j ))) C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

24 Example of 2PL: 10 vocabulary items Model fit as P(Y ij = 1 θ) = exp( b i +a i θ j ) (1+exp( b i +a i θ j )) NOTE: GCONV convergence criterion satisfied. Fit Statistics -2 Log Likelihood AIC (smaller is better) AICC (smaller is better) BIC (smaller is better) (compare to 1PL: AIC= and BIC= 10275) C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

25 b i Parameter Estimates easiness Standard Pr t Parameter Estimate Error DF Value > t Gradient b < b < b < b < b < b < b < b < b < b < Looks good so far... C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

26 a i Parameter Estimates Standard Pr t Parameter Estimate Error DF Value > t Gradient a < a < a < a < a < a < a < a < E-6 a < a < All a i are significant for the hypothesis H o : a i = 0, but this is not the only null that interests us. C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

27 a i Parameter Estimates 1PL is a special case of 2PL where a 1 = a 2 =... = a 10. Likelihood ratio test of H o : a 1 = a 2 =... = a 10. LR = 2(LogLike 1PL LogLike 2PL ) = = 113 df = 9, p <.01. Data support conclusion that 2PL is the better model. C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

28 Item Characteristic Curves: Rasch/1PL C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

29 Model for θ j where θ j = ν 1 (age) j +ν 2 (HSplus) j +ν 3 (Elementary) j +U 0j. (age) j is age of examinee j. (age) j = 45.05, sd = 16.34, 25th percentile= 32, 50th percentile= 44, and 75th percentile= 56. Highest degree earned HSplus j = { 1 HS or more 0 otherwise Elementary j = { 1 elementary d 0 otherwise C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

30 2PL with Model for θ j Using the model for theta, the IRT model for item i is, P(Y ij = 1 x 1j,...,x 10j,U 0j ) exp(b i +a i (ν 1 (age) j +ν 2 (HSplus) j +ν 3 (Elementary) j +U 0j )) = (1+exp(b i +a i (ν 1 (age) j +ν 2 (HSplus) j +ν 3 (Elementary) j +U 0j )) Adding the model for θ j, Yield better estimates of θ j. Allows testing hypotheses about what effects help to explain variability of θ j without introducing extra variability due to measurement error. C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

31 2PL with 3 Covariates for θ: b i easiness Standard Pr t Parameter Estimate Error DF Value > t Gradient b b b < b b b b < b < b b < C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

32 2PL with 3 Covariates for θ: a i Discrimination Standard Pr t Parameter Estimate Error DF Value > t Gradient a < a < a < a < E-6 a < a < a < a < a < a < C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

33 Effect of Covariates Covariates of θ Std Pr Parameter Est Error DF Value > t Gradient age < elementary < high school plus < C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

34 Interpretation of Covariates θ j = (age) j (Elementary) j (HSplus) j +U 0j where HSplus j = { 1 if HS or more 0 otherwise Elementary j = { 1 elementary deg 0 otherwise Notes: (age) j = and sd = τ 00 = 1 (fixed for identification). C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

35 Impact of Covariates C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

36 Summary of Models Number 2 Model parms (LnLike) AIC BIC Rasch (1PL) PL PL 3 covariates PL w/ highest degree Rasch w/ 3 covariates C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

37 Extensions This approach can be extended to Models for polytomous items. Models with covariates for difficulty parameters. Setting equality or inequality restrictions on parameters is possible. Can get estimates of θ j (i.e., U 0j ) after fitting model, empirical Bayes estimates. Drawbacks: Can be slow, e.g., 2PL with covariates took about 45 minutes. Must assume θ j N(0,τ 00 ). Multidimensional models are difficult. Empirical Bayes estimates of θ j are biased or shrunken. C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38

38 References Multilevel Models: Snijder & Bosker (2012). Multilevel Analysis, 2nd Edition. Thousand Oaks, CA: Sage. IRT Models with covariates: DeBoeck & Wilson (eds). (2004). Explanatory Item Response Models. NY: Springer. For more references, see pdf These slides and SAS code are at (bottom of the page). C.J. Anderson (Illinois) Logistic Regression as IRT Model Spring / 38