Illustration (and the use of HLM) Chapter 4 1 Measurement Incorporated HLM Workshop
The Illustration Data Now we cover the example. In doing so we does the use of the software HLM. In addition, we will discuss typical questions that can be answered. 2 Measurement Incorporated HLM Workshop
The Illustration Data So the data set consists of 7,185 students who are nested in schools (there are 160 schools). Variables that we will use: MATHACH (Math Achievement at student level). CSES (group centered SES at student level). MEANSES (Average School SES, school level). l) SECTOR (Public 0 versus Catholic 1, school level). 3 Measurement Incorporated HLM Workshop
Student Label School Label Level-1 Variables Level-2 Variables
The Questions 1. How much do U.S. high schools vary in their mean math achievement? 2. Do schools with high MEANSES also have high math achievement? 3. Is the strength of association between student CSES and math achievement similar across schools? s? Or is CSES S a better predictor of student math achievement in some schools than others? 4. How do public and Catholic schools compare in terms of mean math achievement and in terms of the strength of the SES-math achievement relationship, after we control for MEAN SES? 5 Measurement Incorporated HLM Workshop
1 st Question 1. How much do U.S. high schools vary in their mean math achievement? Notice that this question only addresses the variance of the observations, We will use a random effects ANOVA, 6 Measurement Incorporated HLM Workshop
1 st Question We assume that u 0j has a normal distribution with mean 0 and variance τ 00 School j s mean MATHACH β γ + 0 = γ + u 0 j 00 0 j MATHACH β + ij = 0 j r ij Here all we want to know is the amount of variability of the mean math achievement across the schools 7 Measurement Incorporated HLM Workshop
HLM The program does allow one to keep two separate files in the case of a 2 level analysis, SO in our case we could have a file for each examinee and a file for school, You will need to give the ID variable (school) in both files, Here we have it as an SPSS file and so we will just use it. 8 Measurement Incorporated HLM Workshop
HLM To begin using HLM open the program and click on file This will pop-up a question about what format of dt data you are using At this point we want to start a new analysis so click on Make a new MDM file 9 Measurement Incorporated HLM Workshop
HLM This brings up an option window. For now we only discuss how to do a two level analysis so click on HLM2 10 Measurement Incorporated HLM Workshop
HLM Pick variables Pick variables Pick the SPSS file that you are using Pick the SPSS file that you are using 11 Measurement Incorporated HLM Workshop
HLM Then save the mdmt file, a control file Add name of data file Finally, click on Make MDM file, completes your set up. Examine basic statistics file Click Done 12 Measurement Incorporated HLM Workshop
HLM That will get you to this part where you define your model. 13 Measurement Incorporated HLM Workshop
HLM Output The output for HLM is given as a txt file This file can be opened from HLM Here I will just provide some of the main parts 14 Measurement Incorporated HLM Workshop
Summary Information This gives us the random components 15 Measurement Incorporated HLM Workshop
Fixed Effects Fixed effect for our model (intercept,,γγ 00 ) 16 Measurement Incorporated HLM Workshop
Random Effects Same random components as before only now we have SE and significance tests 17 Measurement Incorporated HLM Workshop
Estimates Think back to the original question. 1. How much do U.S. high schools vary in their mean math achievement? We have a parameter that measured that for us, which was τ 00 =8.61. Also, the average e achievement ement across all school averages (i.e., the grand mean). γ 00 =12.34. 18 Measurement Incorporated HLM Workshop
Range of School Averages Given the mean and its variance I can even compute a confidence interval to describe a range that includes 95% of all schools average math achievement. ˆ γ ± 1.96( ˆ τ ) 00 00 1 2 1 12.64 ± 1.96(8.61) 2 = (6.89,18.39) 19 Measurement Incorporated HLM Workshop
CI for Grand Mean We can also put a CI around the our estimate of the grand mean using the same equation. Only now we use the standard error of our estimate for γ 00 1 ± 2 12.64 ± 1.96(.24) = (12.17,13.11) 17 13 11) 20 Measurement Incorporated HLM Workshop
Interclass Correlation We can also compute the intraclass correlation. ˆ τ 8.61 ˆ τ 00 + σ 861 8.61+ 3915 39.15 00 ρ = = = 2 0.18 21 Measurement Incorporated HLM Workshop
Reliability Lastly we can compute the reliability of our estimate of School j s mean. ˆ ( ) ˆ00 λ j = reliability( Y. j) = ( 2 ˆ τ ) 00 + ( σ / n j ) τ 22 Measurement Incorporated HLM Workshop
Reliability If we compute this value for all schools and average it we will get a summary of reliability which is the same that is given in the HLM output. 23 Measurement Incorporated HLM Workshop
2 nd Question 1. Do schools with high MEANSES also have high math achievement? Focus is how a level-2 variable effects math achievement (average math achievement). Means as outcomes model. 24 Measurement Incorporated HLM Workshop
2 nd Question MATHACH β + ij = 0 j r ij β 0 j = γ00 + + γ01meanses u0 j 25 Measurement Incorporated HLM Workshop
HLM 26 Measurement Incorporated HLM Workshop
Model Summary Information 27 Measurement Incorporated HLM Workshop
Results (Fixed and Random) The estimate of our effect for MEANSES, γ 01 Hypothesis test Here are our variance This tells us that t there are still components differences across schools and that we do need the random effect
Additional Results There are other things we can compute. For example, we can compute the expected range of school means adjusted for MEANSES. 1 1 γˆ 1.96 ˆ 2 12.65 1.96 2.64 2 00 ± τ00 = ± = ( ) ( ) ( 9.47,15.83) 29 Measurement Incorporated HLM Workshop
Additional Results We can also compute the proportion of variance that can be explained by MEANSES using. τˆ 00 00 = = ( ANOVA) τˆ ( MEANSES) τˆ00 ( ANOVA) 8.61 2.64 8.61 0.69 30 Measurement Incorporated HLM Workshop
Additional Results Lastly we can still compute the intraclass correlation as before. Now it is conditional on MEANSES. A measure of dependence within school given that we are now accounting for MEANSES. τˆ 2.64 ρ = 00 = =.06 τˆ +σ 2 2.64 39.16 00 + 31 Measurement Incorporated HLM Workshop
3 rd Question 1. Is the strength of association between student CSES and math achievement similar across schools? Or is CSES a better predictor of a students math achievement in some schools than others? Now the focus is on the effect of a level-1 variable on the dependent variable. Also how this effect various across schools. 32 Measurement Incorporated HLM Workshop
3 rd Question That means that we are interested in: MATHACH = β0 + β1 ( CSES ) + r β β = γ + u 0 j 00 = γ + u j j i ij oj 1j 10 1j 33 Measurement Incorporated HLM Workshop
HLM 34 Measurement Incorporated HLM Workshop
HLM Output Notice now that we have two level-2 random effects we can estimate each variance and their covariance, this is their covariance matrix 35 Measurement Incorporated HLM Workshop
HLM Output 36 Measurement Incorporated HLM Workshop
Average Regression Line Can determine the average regression line across school based on the level-2 fixed effects. β = γ + u 0 00 β j = γ + u oj 1 j 10 1 j So the average line is: MATHACH = 12.65 + 2.19( CSES) 37 Measurement Incorporated HLM Workshop
Variability of Line Can determine variability of the regression lines across schools. Based on variance components (or random effects) τ 00 and τ 11 12.64 ± 1.96(8.68) = 6.87,18.41 1 1 2 ( ) 2.19 1.96(0.68) 68) 0.57,3.81 ± 2 = ( ) 38 Measurement Incorporated HLM Workshop
Relationships of Coefficients Can interpret the relationship between average math achievement with in a school and the relationship between CSES and MATHACH. Looking at τ 01 39 Measurement Incorporated HLM Workshop
Additional Results Lastly, we could look at the proportion of error, at level 1, that can be explained by CSES. Because we are at level-1 we use σ 2 2 2 σ ( ANOVA ) σ ( CSES ) 39.15 36.70 = =.063 σ 2 ( CSES ) 39.15 Remember that the school MEANSES accounted for 60% 40 Measurement Incorporated HLM Workshop
4 th Question 4. How do public and Catholic schools compare in terms of mean math achievement and in terms of the strength of the SES-math achievement relationship, after we control for MEAN SES? 41 Measurement Incorporated HLM Workshop
4 th Question Do MEANSES and SECTOR significantly predict the intercept. Do MEANSES and SECTOR significantly predict the slope. How much variation in the intercept and slope is explained by MEANSES and SECTOR. 42 Measurement Incorporated HLM Workshop
HLM 43 Measurement Incorporated HLM Workshop
HLM Output 44 Measurement Incorporated HLM Workshop
Fixed Effects 45 Measurement Incorporated HLM Workshop
Random Effects 46 Measurement Incorporated HLM Workshop
HLM Output Lastly, I could, as shown on page 85, compute the amount of variability reduced, compared to the random regression line discussed previously. 47 Measurement Incorporated HLM Workshop
Summary The purpose this morning was to get you familiar with terminology, notation, and the types of tests and conclusions we can make with HLM. Are there any questions? 48 Measurement Incorporated HLM Workshop