Iteratioal Educatio Joural, 6, 7(), 174-188. ISSN 144-1475 6 Shao esearch Press. http://iej.cjb.et 174 Accoutability o teachers ad schools: A value-added approach I Gusti Ngurah Darmawa School o Educatio, Uiversity o Adelaide igusti.darmawa@adelaide.edu.au Joh P. Keeves School o Educatio, Fliders Uiversity joh.keeves@liders.edu.au Curretly, there has bee substatial iterest, i Australia ad iteratioally, i policy activities related to outcomes-based educatioal perormace idicators ad their lik with growig demads or accoutability o teachers ad schools. I order to achieve a air compariso betwee schools, it is commoly agreed that a correctio should be made or lack o equity. It is argued that studet perormace is ilueced by three geeral actors: the studet backgroud, classroom ad school cotext, ad idetiied school policies ad practices. I this article the eects o these three actors o sciece achievemet amog studets i Caberra, Australia have bee addressed. The eects are discussed with reerece to Type A, Type B, Type X, ad Type Z eects. Type A eects are school eectiveess idicators cotrolled or studet backgroud. Type B school eects are cotrolled or both studet backgroud ad cotext variables. Type X eects are estimated with studet eects, cotext eects ad omalleable policy eects cotrolled or. Fially, Type Z eects ivoke school eectiveess idicators, cotrolled or studet, cotext, ad all idetiied policy eects. Value-added, accoutability, sciece achievemet, social psychological measures, equity, school eectiveess idicators ACCOUNTABILITY OF TEACHES AND SCHOOLS Durig the past two decades there has bee a growig iterest i the perormace ad accoutability o teachers ad schools both i Australia ad iteratioally (owe, ). Educatioal outcome idicators are requetly used to measure the perormace o teachers, schools, programs, ad policies. eliace o such idicators is largely the result o a growig demad to hold these etities accoutable or perormace, deied i terms o outcomes, such as stadardised test scores i sciece, rather tha iputs such as studet prior achievemet, teacher quality, class size, or quality o acilities (Meyer,, ). The use o such idicators, or example average or media test scores, has some major shortcomigs. owe () poited out that the aalyses o test scores teded to be ocused o a comparative rakig o schools rather tha o idetiyig actors that explaied school diereces. Moreover, Meyer () coteded that average test scores (a) were ilueced by actors other tha school perormace; (b) were a relectio o the accumulated learig that had occurred; (c) teded to be cotamiated due to studet mobility; ad (d) ailed to localise school perormace to a speciic classroom or grade level.
Darmawa ad Keeves 175 Give these problems associated with the use o commo educatioal outcome idicators, the papers by Ballou et al. (4), De Fraie et al. (), audebush ad Willms (1995), ubi et al. (4), ad Willms ad audebush (1989) have approached the estimatio o school ad teacher eects through the use o a variety o statistical models, kow as value-added models i the educatio literature. The essece o the value-added approach is to isolate statistically the cotributio o teachers ad schools to growth i studet achievemet at a give grade level rom all other sources o studet achievemet growth. Failure to isolate these cotributios could result i highly cotamiated idicators o perormace. Cosequetly, the emphasis i cross-atioal achievemet surveys, as well as atioal studies o educatioal achievemet that compare the perormace o schools usig the rak orderig or scalig o outcomes ail to examie i a meaigul way diereces i perormace uless urther aalyses that estimate value-added eects are carried out. Type A, Type B, Type X, ad Type Z Eects FOU TYPES OF SCHOOL EFFECTS audebush ad Willms (1995, p. 1) ad Willms ad audebush (1989, pp. 1-14) argued that studet perormace (Y) was ilueced by three geeral actors: the studet backgroud characteristics (S), school cotext (C) ad idetiied school policies, practices, ad stratiicatios (P), as well as each studet s uique cotributio (e). Y = µ S C P e (1) ij j ij ij ij ij This model ca be exteded to accommodate classroom or teacher eects by splittig school cotext (C) ito its compoets, amely classroom cotext (CC) ad school cotext (SC). Furthermore, school policies ad practices (P) ca be divided ito idetiied policies ad practices (IP) ad uidetiied policies ad practices (UP). Idetiied policies ad practices (IP) ca be urther subdivided ito malleable policies ad practices (MP) ad o-malleable policies ad practices (NP). It should be oted that o-malleable polices ad practices (NP) ca be idetiied, but a school has o cotrol over them sice they are determied at the system level, while malleable policies ad practices (MP) are uder a school s cotrol. Hece we may write Y = µ S CC SC NP MP UP e () ijk jk ijk ijk ijk ijk ijk ijk ijk Equatio () ca also be writte with urther error terms ( u ad r jk ) icluded: Y = γ S CC SC NP MP UP u r e () ijk ijk ijk ijk ijk ijk ijk k jk ijk Four types o teachers or school eects ca be distiguished: Type A, Type B eects (audebush ad Willms, 1995; Willms ad audebush, 1989), Type X eects (Hugi, ; Keeves et al., 5) ad Type Z eects. Type A eects reer to how well the studets i a school perorm i compariso with the perormace o similar studets i other schools. Type A eects are o iterest or studets ad parets i choosig a school. Parets wat to kow which school ca help their child to excel. Parets ad studets will choose the school with the largest Type A eect, that is the school with the largest value added eect whe idividual studet characteristics are take ito accout. The Type A eects ca be speciied as: A = CC SC NP MP UP u r e (4) ijk ijk ijk ijk ijk ijk k jk ijk k
176 Accoutability o teachers ad schools: A value-added approach Type B eects reer to how well the studets i a classroom withi a school perorm, compared to similar studets i classrooms ad schools with similar cotexts. Type B eects are o iterest or those who are lookig or accoutability o the teacher ad school. Teachers ad pricipals are more iterested i the Type B eects o their ow schools because they look or a idicatio o their school s perormace, excludig actors that lie beyod their cotrol. Type B eects are also o iterest or admiistrators ad educatio policy makers, lookig or accoutability. Schools should ot be held accoutable or the cotext i which they operate. The Type B eects ca be speciied as: B = NP MP UP u r e (5) ijk ijk ijk ijk k jk ijk There are some o-malleable polices ad practices (NP), polices ad practices that ca be idetiied but the school has o cotrol over them, ad they should be removed, such as whether the school is urba or rural, or the size o the school i situatios where the school has o cotrol over its size, as well as other stratiyig variables such as State or School Type. Thereore, Type X eects reer to how well the studets i a classroom withi a school perorm, whe compared to similar studets i classrooms ad schools with similar cotexts as well as similar o-malleable policies ad practices. It may be argued that the Type X estimate is the most appropriate estimate o value added, with studet eects, cotext eects (CC ad CS), ad idetiied o-malleable policy eects (NP) removed rom the value added estimates. Type X eects ca be speciied as: X = MP UP u r e (6) ijk ijk ijk k jk ijk However, it would seem appropriate to judge a school by the eect o idetiied malleable polices ad practices as well. A example o malleable policy ad practice at the school level would seem to be that o streamig. Ater cotrollig or the malleable policy ad practices, the remaiig eects ca be labelled as Type Z eects ad ca be writte as Z = UP u r e (7) ijk ijk k jk ijk DATA SAMPLE The data used i this study were collected rom 1,984 juior secodary studets i 71 classes i 15 schools i Caberra, Australia. These 15 schools cosisted o ie govermet schools, our Catholic schools ad two idepedet schools. Nie o these schools were co-educatioal schools ad six were sigle sex (three boys ad three girls schools). I additio, te out o the 15 schools had a streamig policy o placig high achievig studets i larger classes. The sample represets a cohort o approximately studets, who traserred rom Grade 6 to Grade 7 withi a small school system. HYPOTHESIZED MODEL Testig o hypotheses i multilevel models ca be carried out usig multilevel data aalyses sotware such as HLM 6 or Widows (audebush et al., 4). The HLM program was iitially developed to id a solutio or the methodological weakess o educatioal research studies durig the early 198s, which was the ailure o may aalytical studies to atted to the hierarchical, multilevel character o much o educatioal ield research data (Bryk ad audebush, 199). This ailure came rom the act that the traditioal liear models used by most researchers require the assumptio that subjects respod idepedetly to educatioal programs (audebush ad Bryk; 1994, p. 59). I practice, most educatioal research studies select studets as a sample who are ested withi classrooms, ad the classrooms are i tur ested withi schools, ad schools exist withi geographical regios. I this situatio, the studets
Darmawa ad Keeves 177 selected i the study are ot idepedet, but rather ested withi orgaisatioal uits ad igorig this act results i the problems o aggregatio bias ad misestimated precisio (audebush ad Bryk, 1994, p. 59). I Figure 1 the three-level model proposed or testig i this study is show. The ames, codes ad descriptio o the predictor variables tested or iclusio at each level o the three-level model have bee provided i Table 1. Apart rom Class size (CSIZE) at class level ad school classiicatios at school level, all the other variables at the class ad school levels were costructed by aggregatig the studet-level data. SC MP NP Level School Level CC Level Class Level Figure 1. Hypothesised three-level hierarchical model or sciece achievemet ANALYSES The multilevel models were built step-by step. The irst step was to ru a model without explaatory variables, which is also called the ull model. Thus ull model was itted to provide estimates o the variace compoets at each level (audebush ad Bryk, ). The ull model ca be stated i equatio orm as ollows. Level-1 model Y ijk = π jk e ijk Level- model π jk = β j r jk Level- model S ACH β k = γ u k (8) where: Y ikj is the sciece achievemet o studet i i class j i school k. Level 1 Studet Level The secod step udertake was to estimate Type A eects i which studet characteristics were added, thereby cotrollig or studet itake. At this stage, a step-up approach was ollowed to examie which o the eight studet-level variables (listed i Table 1) had a sigiicat (at p.5) iluece o the outcome variable, ACH. Four variables (FOCC, EXPED, LIKSCI ad PIOACH) were oud to be sigiicat ad thereore were icluded i the model at this stage. These our studet-level variables were grad-mea-cetred i the HLM aalyses so that the
178 Accoutability o teachers ad schools: A value-added approach itercept term would represet the average ACH score or the studets with average studet characteristics. Whe a variable was cetred aroud its grad mea, the zero value idicated its average value. Table 1. Variables tested at each level o the hierarchy Level Variable code Variable descriptio Level-1 (Studet-level) (S) FOCC Father's occupatio (1=Uskilled labourer,..., 6= Proessioal) EXPOCC Expected occupatio (1=Uskilled labourer,..., 6= Proessioal) EXPED Expected educatio (1=Year 1 ad Below,... ; 6=Higher Degree) ACAMOT Academic motivatio (=Lowest motivatio,..., 4=Highest motivatio) LIKSCH Like school (=Likes school least,..., 4=Likes school most) LIKSCI Like sciece (1=Likes sciece least,..., 4=Likes sciece most) SELEG Sel regard (1=Lowest sel regard,..., 4=Highest sel regard) PIOACH Prior sciece achievemet (=Lowest score,..., 5=Highest score) Level- (Class-level) (CC) CSIZE Class size (8=Smallest,..., 9=Largest) FOCC_ Average athers' occupatio at class-level EXPOCC_ Average expected occupatio at class-level EXPED_ Average expected educatio at class-level ACAMOT_ Average academic motivatio at class-level LIKSCH_ Average like school at class-level LIKSCI_ Average like sciece at class-level SELEG_ Average sel regard at class-level PIO_ Average prior sciece achievemet Level- (School-level) (SC) CSIZE_ Average class size FOCC_ Average athers' occupatio at school-level EXPOCC_ Average expected occupatio at school-level EXPED_ Average expected educatio at school-level ACAMOT_ Average academic motivatio at school-level LIKSCH_ Average like school at school-level LIKSCI_ Average like sciece at school-level SELEG_ Average sel regard at school-level PIO_ Average prior sciece achievemet (NP) GOVT Govermet school (=No-govermet; 1=Govermet) CATH Catholic school (=No-Catholic; 1=Catholic) IND Idepedet school (=No-Idepedet; 1=Idepedet) BOYS Boys' school (=Girls ad Co-ed; 1=Boys oly) GILS Girls' school (=Boys' ad Co-ed; 1=Girls oly) COED Co-educatioal school (=Boys oly ad Girls' oly; 1=Co-ed) (MP) STEAM Streamig i school (=No streamig; 1=Streamig) Outcome ACH Sciece Achievemet (1 =lowest score.55=highest score) The third step udertake was to estimate Type B eects, which ivolved addig the classroom cotext ad school cotext variables ito the model usig the step-up strategy metioed above. At this stage, the Level- ad Level- exploratory aalysis sub-routies available i HLM 6 were employed or examiig the potetially sigiicat classroom ad school cotext variables (as oud i the output) i successive HLM rus. Followig the step-up procedure, two classroom cotext variables (PIO_ ad CSIZE) were icluded i the model or the itercept. I additio, two cross-level iteractio eects (betwee PIOACH ad FOCC_ ad betwee PIOACH ad LIKSCI_) were icluded i the model. The ourth step ivolved addig the sigiicat o-malleable school policies ad practices ito the model usig the Level- exploratory aalysis sub-routie ad the step-up strategy. At this stage, two cross-level iteractio eects (betwee FOCC ad GOV ad betwee EXPED ad
Darmawa ad Keeves 179 IND) were icluded i the model. I additio, the estimated coeiciets or FOCC_ were ixed at the school level because the reliability estimate o this coeiciet was below.1. The ial step ivolved addig the malleable school policy ito the model (STEAM). Estimates o ixed eects or Types A, B, X, ad Z models have bee give i Table. The Type A model ca be deoted as ollows. Level-1 model Y = π π FOCC π EXPED π LIKSCI π PIOACH e ijk jk 1jk ijk jk ijk jk ijk 4jk ijk ijk Level- model π jk = β k r jk π 1jk = β 1k r 1jk π jk = β k r jk π jk = β k r jk π 4jk = β 4k r 4jk Level- model β k = γ u k β 1k = γ 1 u 1k β k = γ u k β k = γ u k β 4k = γ 4 u 4k (9) The Type B model ca be deoted as ollows. Level-1 model Y = π π FOCC π EXPED π LIKSCI π PIOACH e ijk jk 1jk ijk jk ijk jk ijk 4jk ijk ijk Level- model π jk = β k β 1k PIO_ jk β k CSIZE jk r jk π 1jk = β 1k r 1jk π jk = β k r jk π jk = β k r jk π 4jk = β 4k β 41k FOCC_ 4jk r 4jk Level- model β k = γ u k β 1k = γ 1 u 1k β k = γ u k β 1k = γ 1 u 1k β k = γ u k β k = γ u k β 4k = γ 4 γ 41 LIKSCI_ u 4k 4k β 41k = γ 41 u 41k (1) The Type X model ca be deoted as ollows.
18 Accoutability o teachers ad schools: A value-added approach Level-1 model Y = π π FOCC π EXPED π LIKSCI π PIOACH e ijk jk 1jk ijk jk ijk jk ijk 4jk ijk ijk Level- model π jk = β k β 1k PIO_ jk β k CSIZE jk u jk π 1jk = β 1k r 1jk π jk = β k r jk π jk = β k r jk π 4jk = β 4k β 41k FOCC_ 4jk r 4jk Level- model β k = γ u k β 1k = γ 1 u 1k β k = γ u k β 1k = γ 1 γ 11 GOV 1k u 1k β k = γ γ 1 IND k u k β k = γ u k β 4k = γ 4 γ 41 LIKSCI_ u 4k 4k β 41k = γ 41 (11) The Type Z model ca be deoted as ollows. Level-1 model Y = π π FOCC π EXPED π LIKSCI π PIOACH e ijk jk 1jk ijk jk ijk jk ijk 4jk ijk ijk Level- model π jk = β k β 1k PIO_ jk β k CSIZE jk u jk π 1jk = β 1k r 1jk π jk = β k r jk π jk = β k r jk π 4jk = β 4k β 41k FOCC_ 4jk r 4jk Level- model β k = γ γ 1 STEAM k u k β 1k = γ 1 u 1k β k = γ u k β 1k = γ 1 γ 11 GOV 1k u 1k β k = γ γ 1 IND k u k β k = γ u k β 4k = γ 4 γ 41 LIKSCI_ u 4k 4k β 41k = γ 41 (1) VAIANCE EXPLAINED The cocept o variace explaied is very commo i multiple regressio aalysis. It gives the idea o how much o the variability o the depedet variable is accouted or by liear regressio
Darmawa ad Keeves 181 o the predictor variables. The usual measure o the proportio o variace explaied is the square multiple correlatio,. Oe way to approach this cocept is to treat separately proportioal reductios i the estimated variace compoets,,, ad ϕ at Level 1,, ad respectively as aalogues o values at each level. Variace compoets or the ull model:,, ad ϕ. Variace compoets or the ial model:,, ad ϕ. Proportio o variace explaied at each level i the ial model: At Level 1: 1 = At Level : = At Level : ϕ ϕ ϕ = (1) However, this approach ca be somewhat problematic. It sometimes happes that addig explaatory variables icreases rather tha decreases some o the variace compoets. Thereore, it is possible to obtai egative values o. Sijders ad Bosker (1999) gave a suitable multilevel versio o or the two-level model where the average class size was as ollows: At Level 1: 1 1 = At Level : / / 1 = (14) Equatio (8) ca be exteded to a three-level model where o average each school cosists o classrooms. At Level 1: 1 1 ϕ ϕ = At Level : / / 1 ϕ ϕ = At Level : / ) * * ( / ) * /( 1 ϕ ϕ = (15) Variace compoets preseted i Table were calculated usig equatio (15). ESULTS The Null Model: Diereces Betwee Schools ad Betwee Classes The aalysis was started by ittig the ull model. This model provides estimates o the diereces betwee studets, betwee classes ad betwee schools. The sum o these three
18 Accoutability o teachers ad schools: A value-added approach compoets is the total variace. It ca be see i Table that or sciece achievemet, 5. per cet (8.7/71.45) o the total variace is situated at the studet level ad aother 46.6 per cet (.4/71.45) o the total variace is located at the class level. These large compoets idicate that there are large diereces betwee studets ad betwee classrooms. The percetage o the variace at the school level is very small (.4/71.45=.1%) which suggests that the schools are very similar to each other i terms o studet achievemet i sciece. I other words, the Level itraclass correlatio expressig the likeess o studets i the same school is estimated to be.1, while the itraclass correlatio expressig the likeess o studets i the same classes ad the same schools is estimated to be.47. Sice most o the variace compoets at the school ad class levels are situated at the class level, it is importat to localise school perormace to a speciic classroom or grade level. Type A Model: Addig Studet Characteristics At the studet-level, the results i Table show that Sciece achievemet is directly ilueced by Father's occupatio (FOCC), Expected occupatio (EXPED), Like sciece (LIKSCI) ad Prior achievemet (PIOACH). Whe other actors were equal, studets whose athers had high status occupatios outperormed studets whose athers had low status occupatios. Studets who aspired to pursue educatio to higher levels were estimated to achieve better whe compared to studets who had o such ambitios, while studets who liked sciece were estimated to achieve better whe compared to studets who did ot like sciece. I additio, studets who had high prior achievemet scores were estimated to achieve better tha studets who had low prior achievemet scores. Addig the studet level variables to the model explais a large part o the diereces betwee studets (5.7 %), classes (69.9 %), ad betwee schools (69.8 %) i sciece achievemet. I other words, sciece achievemet diereces betwee schools ad betwee classes were largely due to itake diereces at the grade level uder survey. The remaiig diereces betwee classes ad betwee schools were idicators o the variace i Type A school eects ad i Type A teachig eects. The residuals o schools ad classes ca be see i Figure ad Figure respectively, with little variability betwee schools. Type B Model: Addig Classroom ad School Cotexts From Table it ca be see that at the class-level, Sciece achievemet is directly ilueced by Average prior achievemet (PIO_) ad Class size (CSIZE). Whe other actors were kept equal, studets i classes with high prior achievemet scores were likely to achieve better whe compared to studets i classes with low prior achievemet scores. Importatly, there was cosiderable advatage (i term o better achievemet i sciece) associated with beig i larger classes. Nevertheless, i iterpretig the eects o class size, it eeds to be recogised that 1 out o the 15 schools i these data had a streamig policy that ivolved placig high achievig studets i larger classes ad low achievig studets i smaller classes or eective teachig. Thereore, the better perormace o the studets i larger classes i these data is ot surprisig. Studets i the schools that implemeted streamig policy achieved better i sciece.
Darmawa ad Keeves 18 Table. Fial estimatio o ixed eects Type A model Type B model Type X model Type Z model Fixed Eects Coeiciet S.E p-value Coeiciet S.E p-value Coeiciet S.E p-value Coeiciet S.E p-value Itercept γ 8.5.5. 8.4.1. 8.1.. 7.17.5. STEAM γ 1 1.6.59.17 PIO_ γ 1.7.8..8.7..9.6.1 CSIZE γ.8.6.1.9.7.1..6. FOCC, γ 1.7.1.11.7.14.16.68.19.4.71.19. Iteractio with GOV γ 11 -.5..44 -.55..9 EXPED γ.58.9..5.1..44.1..4.1.1 Iteractio with IND γ 1.54.8.7.66.9.41 LIKSCI γ.14.1..15.1..14.1..15.1.1 PIOACH γ 4.97.5..9.4..94.4..94.4. Iteractio with LIKSCI_ γ 41.1..4.1..7.1..8 Iteractio with FOCC_ γ 41.7...5...6..6 Table. Variace Compoets Number o Available Explaied (%) Uexplaied (%) Model Deviace Parameter Studet Class School Studet Class School Studet Class School Estimated (N=1984 (K=71) (J=15) (N=198 (K=71) (J=15) (N=1984) (K=71) (J=15) ) 4) Null model 1,78 4 8.7.4.4 Prior Achievemet 1,14 9 4. 9.58. 5.7 69.9 69.8 47..1. Type A Model 11,879 6.68 6.49.14 61.8 78.8 77.4 8. 1..6 Type B Model 11,79 61.56 1.6.81 67.9 9.9 8.. 9.1 17.8 Type X Model 11,786 55.54 1.6.71 67.8 91.1 8.6.1 8.9 16.4 Type Z Model 11,78 56.54 1.74.1 68.4 9. 88.7 1.6 8. 11.
184 Accoutability o teachers ad schools: A value-added approach.5 1 Itercept. Itercept -.5 14 11 1 5 1 8 1 9 1 4 6 7 15-1 5 4 9 7 6 54 65 1 4 44 14 19 66 48 55 56 9 5 8 7 8 6 7 45 8 49 18 1 57 67 11 5 46 7 64 1 71 59 41 1 15 4 69 6 68 74 6 58 5 4 4 47 16 6 1 51 1 7 5 17 6 61 9 Class ID Figure. Type A school residuals Figure. Type A class residuals I Table there are two sigiicat cross-level iteractio eects. These cross-level iteractio eects are betwee (a) PIOACH ad FOOC_ at Level (class level); ad (b) PIOACH ad LIKSCI_ at Level (school level). It ca be see i Figure 4 ad Figure 5 that the eect o prior achievemet is stroger i classes with higher status o athers occupatio ad i schools with higher level o likig sciece. Higher achievig studets were better o i classes that had higher status o athers occupatio as well as i schools with higher levels o likig sciece. Sciece Achievemet 4 5 5 High Occupatioal Low Occupatioal Sciece Achievemet 4 5 5 High Like Sciece Low Like Sciece 15 Low Averag High Prior Achievemet 15 Low Average High Prior Achievemet Figure 4. Impact o iteractio eect o FOCC_ ad PIOACH o Sciece Achievemet at the classroom level Figure 5. Impact o iteractio eect o LIKSCI_ ad PIOACH o Sciece Achievemet at the school level Ater cotrollig or studet characteristics, class cotext ad school cotext, the proportio o variace explaied is icreased by 9.1 per cet at the studet level, 8.9 per cet at the class level, ad 7.6 per cet at the school level. The residuals o 15 schools ad 71 classes ca be see i Figure 6 ad Figure 7, with a icrease i variability betwee schools ad a decrease i the variability betwee classes..5 1 Itercept. Itercept -.5 8 7 5 1 4 14 15 1 1 9 6 11 1-1 1 5 5 7 41 6 15 51 46 1 58 5 61 74 98 18 68 56 16 71 1 4 6 8 45 54 17 656 44 66 1 64 6 1 41 7 8 9 57 6 67 47 59 4 79 Class ID 14 57 19 4 48 55 6 7 49 11 69 4 5 Figure 6. Type B school residuals Figure 7. Type B class residuals
Darmawa ad Keeves 185 Type X Model: Addig No-Malleable School Policies ad Practices Whe o-malleable policies were etered ito the equatios at Level, two additioal iteractio eects were oud. These iteractio eects icluded iteractios betwee (a) FOCC ad GOV ad (b) EXPED ad IND. From Figure 8 it ca be see that whe other actors are equal, ather s occupatio had less impact i govermet schools tha i o-govermet schools. I other words, studets with high ather s occupatioal status gaied smaller advatage i govermet schools compared o-govermet schools. Likewise, rom Figure 9 it ca be see that studets i idepedet schools achieve higher scores i sciece whe they have high expected educatio. However, studets with low levels o expected educatio have oticeably lower levels o achievemet i they are i idepedet schools. 1 Sciece Achievemet 9 8 7 6 5 No-Govermet School Govermet School Sciece Achievemet 9 8 7 6 5 Idepedet School No-Idepedet School 4 Low Average High 4 Low Average High Father's Occupatio Expected Educatio Figure 8. Impact o iteractio eect o Govermet School ad FOCC o Sciece Achievemet at the school level Figure 9. Impact o iteractio eect o Idepedet School ad EXPED o Sciece Achievemet at the school level Ater addig o-malleable policies ad practices, oly 16.4 per cet ad 8.9 per cet o variace compoets at the school ad class levels are let uexplaied. The Type X residuals o 15 schools ad 71 classes ca be see i Figure 1 ad Figure 11 respectively..5 1 Itercept. Itercept -.5 7 5 6 4 15 8 1 9 1 1 14 11 1-1 5 4 5 1 46 6 7 51 1 1 74 58 61 5 15 18 9 8 68 1 56 71 64 6 4 65 8 66 7 1 54 17 6 1 45 44 16 41 6 57 9 7 4 6 9 59 47 67 8 5 19 6 7 4 48 7 14 49 11 69 55 4 5 Class ID Figure 1. Type X school residuals Figure 11. Type X class residuals Type Z Model: Addig Malleable School Policies ad Practices At the School level, the results i Table show that Sciece Achievemet is also directly ilueced by streamig policy (STEAM). Studets i the schools that implemeted streamig policy achieved better i sciece. I this model, oly 1.6 per cet, 8.9 per cet ad 11. per cet o variace compoets at studet, class, ad school levels are let uexplaied. The Type Z residuals o 15 schools ad 71 classes ca be see i Figure 1 ad Figure 1 respectively.
186 Accoutability o teachers ad schools: A value-added approach.5 1 Itercept. Itercept -.5 7 15 5 4 6 8 9 1 1 14 1 11 1-1 5 1 4 5 7 6 46 511 15 1 58 5 61 74 68 18 98 71 56 1 6 4 45 16 65 64 44 8 17 546 Class ID 66 1 6 1 7 67 41 57 8 9 47 59 69 4 77 19 14 4 5 48 6 49 55 11 7 69 4 5 Figure 1. Type Z school residuals Figure 1. Type Z class residuals Iitially the diereces betwee schools are very small as show by the residuals o the Null model. Ater cotrollig or studet characteristics, there are still o sigiicat diereces betwee schools. Addig classroom cotext ad school cotext variables oticeably chage the residuals or School ad School 1. Makig allowace or additioal o-malleable policies chaged school residuals eve urther. However, ater cotrollig or the sigiicat malleable policy variable the average levels o perormace or most schools are ot sigiicatly dieret rom each other. School ad School 1 are the two schools that have oticeably lower ad higher perormace respectively. School is sigiicatly worse tha other schools, but School 1 is sigiicatly better tha other schools ater cotrollig or studet characteristics, cotext variables as well as idetiied school policies ad practices. These chages are oticeable rom compariso o Figures, 6, 1, ad 1 as well as rom Figure 14, ater allowace is made or the Type A, Type B, Type X ad Type Z eects.. 1.5 1. esidual.5. -.5-1. -1.5 7 15 5 4 6 8 9 1 1 14 1 11 1 Z Null Ach 68 Type A Type B Type X Type Y Figure 14. Types A, B, X ad Z school residuals CONCLUSIONS This article is cocered with the statemet that, studet outcomes are oly partially ilueced by the school where they are erolled. Other actors that have a impact o the studet outcomes are studet characteristics ad cotext variables. I this study, Type A, Type B, Type X ad Type Z eects are estimated by allowig or studet backgroud, class ad school cotexts, omalleable school policies ad malleable school policies respectively i successive regressio equatios. The mai eects reported rom the aalysis at the studet level, idicate that i additio to prior achievemet, it was the social psychological measures associated with the diereces betwee studets withi classrooms that were havig eects, amely, socioecoomic status, educatioal aspiratios, ad attitudes towards learig sciece. About per cet o the variace betwee
Darmawa ad Keeves 187 studets withi classrooms is let uexplaied, idicatig that there are other studet-level actors likely to be ivolved i iluecig studet achievemet. At the classroom level, about eight per cet o the total classroom variace or oly 1.7 per cet o the total variace is let uexplaied, with the average level o prior achievemet o the class group havig a sigiicat eect. I additio, class size has a positive eect at this level o sciece achievemet, with studets i larger classes doig sigiicatly better tha studets i smaller classes. This eect is likely to be coouded with actors associated with the qualities o the teachers assiged to teach the larger ad the smaller class groups. Perhaps, this idicates the skill o the admiistratio o the schools, particularly i those schools that adopt streamig practices to select the best teachers ad allocate them to the higher perormig studets i larger classes. I additio, a iteractio eect also reveals that the eect o prior achievemet is stroger i classes with high status o athers occupatio. High achievig studets are better o i classes that have higher status o athers occupatio. At the school level o aalysis, streamig directly explais some o the diereces i levels o perormace betwee schools i spite o the very small betwee school variace. The ilueces o the o-malleable variables ivolvig school type ad whether a school is sigle-sex or coeducatioal do ot have direct eects o the educatioal outcome o sciece achievemet, but they do have moderatig or iteractio eects. Thus whether the school is a Govermet or a Idepedet school iteracts with Father s occupatio ad Expected educatio respectively to have small eects o the outcome variable. Nevertheless, it is this actor o school type that has had ad cotiues to have a marked iluece o chages i the provisio o educatio i the Australia school systems. Uortuately it is o loger possible to udertake research ito this issue, because over-simplistic value added comparisos, that were made prior to the itroductio o multilevel aalytical procedures have cotamiated this ield o iquiry i Australia. Two importat idigs emerge rom this study. First, cosiderable variace is situated at the class level. Thereore i examiig value added across schools, the class level ca ot be igored. Otherwise, the class level variace compoets may be coouded with studet level ad school level variace compoets ad lead to a overestimatio o school diereces. I educatioal eectiveess research, eglectig class cotext variables may lead to icorrect coclusios. Secod, very little variace is let uexplaied at the school ad class levels to be accouted or by characteristics associated with school resources or by the direct eects o teachers. I the qualities o teachers are havig eects they are associated with ad are subordiate to the levels o iitial achievemet o the studets whom they are assiged to teach, with high achievig studets beig placed i larger classes possibly with the better teachers. However, the use o a value added approach i assessig school eectiveess is ot without problems. There is still room or argumet whether Type A, Type B, Type X or Type Z eects should be cosidered. Careul thought also eeds to be give whe cosiderig which o the variables should be used i estimatig Type A, Type B, Type X ad Type Z eects. Moreover, how to obtai iormatio o classroom eects is yet aother questio to address. Should logitudial data rather tha cross-sectioal data be used? Apart rom these problems which still eed to be debated, the value added approach is providig a way to assess better the eectiveess ad the accoutability o schools as well as classrooms ad teachers. Furthermore, it is clearly iappropriate to rak schools o terms o their perormace ad ideed to rak coutries, without givig some cosideratio to these complex statistical problems. Noetheless, research ad scholarly debate eeds to be carried out to develop a better uderstadig o the issues addressed.
188 Accoutability o teachers ad schools: A value-added approach EFEENCES Ballou, D., Saders, W. ad Wright, P. (4) Cotrollig or studet backgroud i value-added assessmet o teachers. Joural o Educatioal ad Behavioral Statistics, 9(1), 7-65. Bryk, A.S. ad audebush, S.W. (199) Hierarchical Liear Models: Applicatios ad Data Aalysis Methods. Newbury Park, CA.: Sage Publicatios. De Fraie, B., Va Damme, J. ad Oghea, P. () Accoutability o School ad Teachers:what should be take ito accout? Europea Educatioal Joural, 1(), 4-48. Hugi, N. () Measurig School Eects Across Grades. Fliders Uiversity Istitute o Iteratioal Educatio esearch Collectio. No.6. Adelaide: Shao esearch Press. Keeves, J.P., Hugi, N. ad Arassa, T. (5). Measurig value added eects across schools: should schools be compared i perormace? Studies i Educatioal Evaluatio, 1(-), 47-66. Meyer,.H. () Value-added idicators: A powerul tool or evaluatig sciece ad mathematics programs ad policies. NISE Brie, (), 1-7. Meyer,.H. () Value-Added Idicators: Do They Make a Importat Dierece? Evidece rom the Milwaukee Public Schools, Paper preseted at the Aual Meetig o the America Educatioal esearch Associatio, April, : New Orleas. audebush, S.W. ad Willms J.D. (1995). The estimatio o school eects. Joural o Educatioal ad Behavioral Statistics, (4), 7-5. audebush, S.W. ad Bryk, A.S. (1994). Hierarchical liear models. I T. Husé ad T.N. Postlethwaite (Eds.), Iteratioal Ecyclopedia o Educatio: esearch ad Studies (d ed., pp. 59-596). Oxord: Pergamo Press. audebush, S.W. ad Bryk, A.S. (). Hierarchical Liear Models: Applicatios ad Data Aalysis Methods (d ed.). Thousad Oaks, CA: Sage. audebush, S.W., Bryk, A.S., Cheog, Y.F. ad Cogdo,.T. (4). HLM 6: Hierarchical Liear ad Noliear Modelig. Licolwood, IL: Scietiic Sotware Iteratioal. owe, K.J. () Assessmet, league tables ad school eectiveess: Cosider the issues ad let s get real!, Joural o Educatioal Equiry, 1(1), 7-98. ubi, D.B., Stuart, E.A. ad Zautto, E.L. (4). A potetial outcomes view o value-added assessmet i educatio. Joural o Educatioal ad Behavioral Statistics, 9(1), 1-116 Sijders, T. ad Bosker,. (1999) Multilevel Aalysis: A Itroductio to Basic ad Advaced Multilevel Modellig. Lodo: Sage Publicatios. Willms, J.D. ad audebush, S.W. (1989) A logitudial hierarchical liear model or estimatig school eects ad their stability. Joural o Educatioal Measuremet, (6), 9-. IEJ