Abstract. Introduction

Transcription

1 P5: INTRACLASS CORRELATION AND VARIANCE COMPONENTS AS POPULATION ATTRIBUTES AND MEASURES OF SAMPLING EFFICIENCY IN PIRLS 001 Pierre Foy, IEA Data Processig Ceter, Hamburg, Germay Keywords: Itraclass Correlatio, Variace Decompositio, Sample Desig, Stratificatio Abstract The Itraclass Correlatio is a measure of homogeeity amog aalytical uits. I the cotext of PIRLS 001, the itraclass correlatio measures the homogeeity of readig achievemet betwee schools i a atioal educatioal system. It allows the researcher to address policy cocers related to equity ad disparity of learig opportuities. The derivatio of variace compoets, through variace decompositio, allows the researcher to probe deeper ito the various sources of variace i readig achievemet. Specifically, we look at studets, classes ad schools as sources of variace. We further break dow the school variace compoet ito a school stratificatio compoet to ivestigate disparity of readig achievemet betwee groupigs of schools, as defied by the stratificatio variables applied i atioal sample desigs. This latter compoet also allows us to measure the efficiecy of the school-level stratificatios, as applied atioally, i reducig the stadard errors of the survey estimates. Itroductio This paper presets two related cocepts, as applied to the PIRLS 001 data. The itraclass correlatio is a measure of homogeeity amog uits of aalysis, ad thus provides a summary glimpse ito the variace structure of the data. Variace compoets are derived from variace decompositio ad offer a more detailed perspective ito the variace structure. We will describe i detail these two cocepts; demostrate how they are computed, with results take from the PIRLS 001 data, ad show how the results ca be iterpreted, both as populatio attributes ad as measures of samplig efficiecy. Defiitio Itraclass Correlatio The Itraclass Correlatio is a importat populatio attribute used by sample survey statisticias i derivig efficiet sample desigs ad sample sizes for hierarchical populatios. The PIRLS data are hierarchical i ature, with studets withi classes, ad classes withi schools. At miimum, we ca derive two variace compoets: the betwee-school variace ad the withischool variace. The itraclass correlatio simply expresses the betwee-school variace as the proportio of the sum of the two variace compoets, as described i the followig equatio: IC B B W (1) B where is the betwee-school variace, ad is the withi-school variace. W A low itraclass correlatio, say less tha 0.5, idicates relatively small betwee-school variatios. I other words, schools ted to perform at comparable levels. As the itraclass correlatio icreases, beyod 0.5, the schools perform with ever-icreasig variatios; some schools achievig very high levels of performace ad others very low levels of performace. For the samplig statisticia, a school system with a low itraclass correlatio requires a sample desig that focuses more o the withi-school compoet. Thus he will devise a sample desig that samples fewer schools, but more studets withi schools. As the itraclass correlatio icreases, the focus shifts to samplig more schools, ad perhaps fewer studets withi schools. Page 1

2 As a populatio attribute, the itraclass correlatio offers a measure of equity, or disparity, of learig opportuity. Systems with a low itraclass correlatio have achieved a measure of equity whereby all schools perform at roughly equivalet levels. Systems with a high itraclass correlatio demostrate disparity of learig opportuity whereby some schools perform well, yet others i the same system perform poorly. Estimatio The Itraclass Correlatio is a simple case of variace decompositio. It is derived from a sigle-level aalysis of variace, as preseted i figure 1 (Neter & Wasserma, p. 44). The ANOVA table presets the geeral case with uequal studet sample sizes withi schools, ad is computed usig samplig weights 1. Figure 1: ANOVA Table for Computig the Itraclass Correlatio The quatity is estimated as follows (Neter & Wasserma, p. 58): 1 1 i 1 i i 1 i i 1 i () where is the umber of sampled schools, ad i is the umber of sampled studets i school i. This quatity ca be iterpreted as the average i i the case of a ubalaced ANOVA. If all i = k, as is the case i a balaced ANOVA, the = k. The itraclass correlatio ca thus be estimated from the mea squares i the ANOVA table ad usig equatio (1), as follows: IC MS B MS ( B MSW 1) MS W (3) where MSB ad MSW are defied i the ANOVA table of figure 1. I a ubalaced ANOVA, the quatity must be used. I a balaced ANOVA, ca be replaced with the costat studet sample size k i each school. For the aalysis of variace model to work properly, some basic assumptios o the ature of the data beig aalysed must hold: 1. Two levels: The data come from a hierarchical structure with two levels, amely schools ad studets. This may seem obvious, but eeds to be stated sice i some respects we may be applyig a simpler model to a more complex data structure. 1 The samplig weights used are the house weights, as described i the PIRLS 001 User Guide for the Iteratioal Database (Gozalez & Keedy, 003). Page

3 . Equal probability samplig: Uits at each level must be selected with equal probabilities. I this case, schools should be sampled with equal probabilities, ad studets withi schools should be selected with equal probabilities, although these studet probabilities eed ot be the same betwee schools. 3. Equal variace: We require equal variace at all levels. I this case, the studet withi school variace should be equal for each school. 4. Radom effects: Uits at each level must cosist of a radom sample from a larger populatio. Thus schools must be sampled from a larger populatio ad studets must be sampled from a larger populatio of studets withi each school. As we will see later i the iterpretatio of actual results, these assumptios do ot always hold. Ad we might therefore wat to recosider our approach to this problem. The assumptio of radom effects is perhaps the least restrictive sice we usually have school ad studet samples take from larger populatios. All of these assumptios must hold if we are to make valid statistical tests from the resultig data. The violatio of ay of these assumptios, however, does ot prevet us from derivig estimates for descriptive purposes. Applicatio We have applied equatio (3) to the PIRLS 001 data. The data were computed usig the procedure NESTED from SAS, versio 8. The PIRLS study was carried out i 001 ad admiistered a readig test at the 4 th grade i 35 coutries. The results are give i Table 1. The table gives itraclass correlatios for the overall readig score, as well as for its two compoets: readig for literary experiece ad readig to acquire ad use iformatio. All calculatios are based o the first plausible value of each score. The coutries are raked i ascedig order of the overall readig itraclass correlatio. The coutries have bee arbitrarily divided ito three groups. The first group cosists of coutries with a low itraclass correlatio, the cut-off beig set at 0.5. This meas that, for these coutries, less tha 5% of the total variace is betwee schools; for Icelad, the betwee-school variace for overall readig represets oly 8% of the total variace. At the other extreme of the itraclass correlatio spectrum, we fid a group of coutries with relatively large itraclass correlatios. This group is arbitrarily defied as those with itraclass correlatios grater tha 0.35, i.e., the betwee-school variace represets more tha 35% of the total variace; for Sigapore, the betwee-school variace accouts for as much as 56% of the total variace. Betwee these two groups, we fid a third oe, where itraclass correlatios lie betwee 0.5 ad The break poits used to defie the three groups of coutries are to some extet arbitrary, we ca oetheless refer to coutries havig either a low itraclass correlatio, a high itraclass correlatio, or somewhere i betwee. Coutries with a low itraclass correlatio have achieved some measure of equity i learig achievemet. All coutries should be iterested i discoverig the cotext for achievig this result, usig data from this group of coutries. I all three groups, we should be iterested i better uderstadig the source of the betwee-school variace, albeit to varyig degrees. This lie of equiry takes o more relevace as the itraclass correlatio icreases. Although coutries with a low itraclass correlatio might provide cotextual iformatio to describe how they have reached some state of equity i learig opportuity, they might also wat to check this iformatio as cofirmatio of their efforts. For example, it might be iterestig to discover that a particular coutry still maages equity without ecessarily havig a perfectly fair allocatio of certai resources at the school or classroom level. It must be said that havig equated a low itraclass correlatio with equity i educatioal achievemet, equity beig perceived as a positive outcome, this does ot ecessarily etail high educatioal achievemet. I fact, a simple regressio betwee atioal achievemet scores ad their Page 3

4 itraclass correlatio yields a small egative slope (-0.001, yet statistically sigificat). Oe eed oly look at the four coutries with the lowest itraclass correlatios (Icelad, Sloveia, Norway ad Cyprus) to realise this. This is further cofouded by the fact that these coutries are amog those with the lowest studet mea ages. This relatioship eeds further study; ivestigatig the relatioship betwee age ad studet achievemet across coutries i a grade-based study. We ca speculate that the itraclass correlatio might icrease as the studets progress through the grades, but this has ot bee demostrated i past IEA studies (i.e., TIMSS), where the itraclass correlatio at the 8 th grade, for both mathematics ad sciece, remais at comparable levels for these four coutries i particular. Table 1: Itraclass Correlatios for PIRLS 001 It is also importat to metio that coutries with the larger itraclass correlatios ted to be amog the low achievers. Cosequetly, coutries with a high itraclass correlatio should ivestigate the causes of their large betwee-school variaces, as potetial meas for improvig their overall performace levels. Page 4

5 From the samplig perspective, the magitude of the itraclass correlatio is a idicator of the itrisic iefficiecies i drawig samples from a hierarchical populatio. As the itraclass correlatio icreases, samplig becomes less efficiet ad requires more efficiet sample desigs, larger sample sizes, or both. Therefore, populatios with low itraclass correlatios place fewer demads o the samplig statisticia i developig a suitable sample desig. Populatios with large itraclass correlatios require additioal kowledge of the ature of the disparities betwee schools, iformatio that ca be used i defiig strata i such a way as to miimise the impact of the large itraclass correlatio. The fial weapo available is simply to icrease the sample size, although, oce agai, additioal kowledge of the true ature of disparity might require either a larger sample of schools, or a larger sample of studets or classrooms withi schools, depedig o which is the major source of variace. Iterpretatio Because the PIRLS 001 data are derived from complex, stratified, multi-stage sample desigs, proper iterpretatio of the calculated itraclass correlatios must take these sample desig features i cosideratio. This is particularly importat to the extet that the data derived from these complex sample desigs allow us to meet, or violate, the uderlyig assumptios supportig the aalysis of variace model, as preseted earlier. PIRLS participats first defied strata, the selected schools, classrooms ad fially studets. Most participats sampled oly oe classroom per school, whereas a few others sampled more tha oe. This is a importat distictio to make sice classrooms withi schools ca be aother source of variace. Give the assumptios behid the simplified aalysis of variace model i Figure 1, this withi-classroom variace compoet will either be cofouded with the betwee-school variace compoet for coutries that sampled oly oe classroom per school, or cofouded with the withischool variace compoet for coutries that sampled more tha oe classroom per school. Therefore, we have some imbalace i the iterpretatio of the itraclass correlatios i Table 1, sice coutries that sampled oly oe classroom per school will ted to have a larger itraclass correlatio tha if they had sampled more tha oe classroom per school. This of course depeds o the magitude of the withi-classroom variace compoet. If this compoet is small or o-existet, the samplig more tha oe classroom per school makes little differece. But if a coutry sampled oly oe classroom per school, it is ot possible to establish the magitude of the betwee-classroom variace compoet. The followig coutries systematically sampled more tha oe classroom per school: Colombia, Frace, Germay, Icelad, Ira, Kuwait, Netherlads, Norway ad Swede. For these coutries, the calculated itraclass correlatios are better reflectios of their true betwee-school variace compoets. The itraclass correlatios for all other coutries will ted to be a over-estimatio of the true betwee-school variace compoets, depedig o the magitude of their ukow betweeclassroom variace compoets. I a sceario whereby oly oe classroom per school is sampled, we ca claim to have a sample of classrooms, as opposed to a sample of schools, from the whole populatio. If we were to redefie the structure of the ANOVA table i Figure 1 i this cotext, we would label the sources of variatio as betwee classrooms ad withi classrooms. Thus the itraclass correlatio becomes more a measure of the disparity betwee classrooms i the populatio tha betwee schools. We make here a distictio betwee betwee classrooms i the populatio ad betwee classrooms withi schools, the former expected to be greater or equal to the latter. Thus if we were to calculate the itraclass correlatios cosiderig classrooms as the first level, rather tha schools, we would obtai more comparable results betwee coutries sice we would cosider all samples o the same footig, that is to say as samples of classrooms as opposed to samples of schools. The data i Table allow us to make this compariso. The school level colum cosiders the PIRLS samples as samples of schools ad is idetical to the overall readig colum of Table 1. The classroom level colum cosiders the PIRLS samples as samples of classrooms, rather tha schools, ad thus we would expect the umbers i this colum to be greater for coutries that sampled more tha oe classroom per school. This is ideed what we observe i the icrease colum, which describes the icrease i the classroom colum, relative to the school colum. This icrease gives us a Page 5

6 glimpse of the presece of a betwee-classroom withi school variace compoet, without explicitly measurig it. Table : Comparig Itraclass Correlatios Betwee Schools ad Betwee Classrooms The reader should ote that there are more coutries here with positive icreases tha the coutries listed earlier that systematically sampled more tha oe classroom per school. This is due to some coutries havig sampled more tha oe classroom per school, but i a limited way, albeit eough to be reliably estimated i this table. It is iterestig to ote that the itraclass correlatio i Icelad icreases dramatically, suggestig a possibly large classroom variace compoet. This is i stark cotrast to Norway, where we observe a much smaller icrease. We should poit out the limitatios of samplig oly oe classroom per school i beig able to make a distictio betwee school ad classroom variaces. Takig Cyprus as a example, had we sampled more tha oe classroom per school, we would expect to estimate the same value i the classroom level colum, but a value less or equal i the school level colum, the magitude of the differece depedet o the magitude of the betwee-classroom withi school variace compoet. Page 6

7 The iterplay betwee the betwee-school ad betwee-classroom variace compoets raises other potetial shortcomigs i the proper iterpretatio of the itraclass correlatios. Our first assumptio behid the aalysis of variace used to derive the itraclass correlatio is that the data are derived from a two-level model, amely schools ad the studets withi. This assumptio does ot quite hold i the cotext of PIRLS where the sample desig is at miimum a three-level model: schools with classrooms ad studets. Havig said that, we are still permitted to view the data through a two-level model, either schools ad studets, or classrooms ad studets, provided the various sources of variace remai ucorrelated ad we do ot violate the other uderlyig assumptios. Our secod assumptio is that all uits withi a level are selected with equal probabilities. This actually holds for classrooms ad studets, but does ot hold for schools, sice they are selected usig a stratified PPS (probabilities proportioal to size) sample desig. If the betwee-school variace compoet is costat, regardless of school size, the we ca expect miimal disturbace from the PPS sample selectio. This assumptio of equal probabilities is more readily met if we cosider classrooms istead of schools as the first level, sice the classroom selectio probabilities have almost equal probabilities by desig. Our third assumptio is oe of equal variace at each level. Specifically, the betweeclassroom variace should be equal i all schools. Ituitively, we should questio whether this assumptio holds or ot. Let us first cosider the studet level variace withi schools. Whether a school has oe classroom of studets, or te classrooms of studets, let us for the sake of argumet assume that we have equal studet-level variace. The betwee-variace compoet arises from the partitioig of studets ito classes, ad this process apportios some of the studet-level variace to a betwee-classroom compoet ad the remaider to a withi-classroom compoet. Our cotetio is that the magitude of the betwee-classroom compoet ca be highly depedet o the umber of classrooms to be formed i a give school. The total withi-school variace may be the same i all schools, but there ca be more classroom differetiatio i a school with te classes tha i a school with oly two classrooms. Thus we would expect a larger betwee-classroom variace compoet i larger schools tha i smaller schools, thereby ivalidatig our assumptio of equal variace at the classroom-level. This pheomeo remais to be tested ad validated. We formulated a fourth assumptio, oe of radom effects at all levels. This is clearly validated give that we have sampled uits at all levels: schools, classrooms ad studets. Oly for a hadful of coutries did we select all eligible schools, ad thus this assumptio might ot hold. For these coutries, we could ifer a super-populatio model, whereby the relatively small umbers of schools i their samples are coceptually samples of their respective super-populatios. This discussio leads us to cosider a more elaborate aalysis of variace model, better suited for the complex structure of the PIRLS data. This brigs us the to cosider a four-level model, which would allow us to explore i greater detail, the may variace compoets, some of which we have alluded to already, preset i the PIRLS data. Defiitio Variace Decompositio Recogizig that the PIRLS data are hierarchical i ature studets withi classrooms, classrooms withi schools ad schools withi samplig strata we ca explore the variace structure of these data usig a four-level aalysis of variace model. This model is preseted i Figure (Bortz, p. 53), ad is a extesio of the simpler model preseted i Figure 1. Thus all terms i the figure are aalogous ad hopefully obvious, except perhaps for the derivatio of the quatities ad. They are derived i a aalogous maer as ad are meat to reflect the average studet sample size at the school level ad at the stratum level respectively. The uderlyig assumptios i the four-level aalysis of variace model are similar to the oes discussed earlier for the two-level model: 1. Four levels: We ow cosider that the data come from a hierarchical structure with four levels, amely strata, schools, classrooms ad studets. Page 7

8 . Equal probability samplig: Uits at each level must be selected with equal probabilities. We have a difficulty here sice strata are ot actually sampled, but this is more relevat uder our fourth assumptio. 3. Equal variace: We agai require equal variace at all levels. We have already discussed a potetial difficulty i this regard cocerig the betwee-classroom variace compoet. 4. Radom effects: Uits at each level must cosist of a radom sample from a larger populatio. This clearly does ot hold for the strata, ad it is difficult to coceive of a super-populatio model suitable for this level. Figure : ANOVA Table for Variace Decompositio From the ANOVA table, we ca estimate four variace compoets: A betwee-strata variace, str A betwee-schools withi strata variace, sch A betwee-classrooms withi schools variace, A betwee-studets withi classrooms variace, We ca estimate the total variace as the sum of these four variace compoets, ad the express each variace compoet as a percetage of this total variace. This is aalogous to the defiitio of the itraclass correlatio. Applicatio Usig the PIRLS 001 data, we estimated all four variace compoets for all participatig coutries. This was doe oly with the overall readig score, usig its first plausible value, ad the house weights. The estimates are preseted i Table 3. The Mea Square Error is a estimate of the total variace, ad the variace compoets, labelled STR, SCH, CLS ad STD respectively, are expressed as percetages of the total variace. Coutries are preseted i alphabetical order. At first glace, we will otice that the classroom compoet is estimated as zero (0) for most coutries. This simply reflects the fact that these coutries sampled oly oe classroom per school, or more tha oe classroom i too few schools, thereby makig the estimatio of this variace compoet impossible, or ureliable. These data are also preseted graphically i Figures 3 ad 4. The pie charts display the relative magitude of the four variace compoets. The total area of each pie chart is proportioal to each coutry s mea square error, thus graphically displayig the promiece of atioal total variaces. Page 8 cls std

9 For example, sice Morocco has the largest total variace, it also has the largest chart. Icelad havig roughly half the total variace of Morocco, it has a chart whose area is half that of Morocco. Table : Variace Decompositio Data for PIRLS 001 Iterpretatio I theory, we should be able to observe the itraclass correlatios from these data. We say i theory because this is ot always the case. It is uclear at this time why this is so, but we assume it is related to the ubalaced ature of these ANOVAs. A balaced ANOVA has equal sample sizes at all levels ad all the desired properties for properly estimatig all of its parameters. The PIRLS data are ot at all balaced, especially regardig the school sample sizes withi strata, ad to lesser degrees regardig classrooms withi schools ad studets withi classrooms. Aother possible explaatio for this less tha perfect fit could be related to the potetially poor estimatio of the classroom variace compoet. Earlier we discussed the possibility that it may ot be appropriate to assume equal variace at the classroom level. This may be maifestig itself here, ad defiitely warrats further ivestigatio. Argetia is a example where we ca readily observe the itraclass correlatio as was estimated i Table 1. Recall that it was 0.41, meaig that the betwee-school variace represets 4.1% of the total variace. This betwee-school variace is ow divided ito two compoets: the Page 9

10 stratum ad school compoets. Summig up these two compoets, we obtai a comparable estimate of 4.7% of total variace explaied. Figure 3: Variace Decompositio for PIRLS 001 Part 1 If we take Bulgaria as a example, its itraclass correlatio was estimated at i Table 1. The sum of its stratum ad school compoets yields 50.7%. This rather large discrepacy caot be explaied ad warrats further study. It is, however, the oly coutry amog those without a classroom compoet to display such a large discrepacy. All other coutries without a classroom compoet behave much like Argetia, although with some varyig mior discrepacies. The situatio is less obvious for those coutries with a o-zero classroom variace compoet. If we were to sum up their stratum, school ad classroom compoets, we would geerally come close to their itraclass correlatios, as estimated i the classroom level colum of Table. This would ted to cofirm our earlier ifereces o the ature of the cofoudig of classroom variace whe samplig oly oe classroom per schools versus samplig more tha oe classroom per school. We ca the ifer that whatever classroom variace may exist is curretly cofouded i the school variace for those coutries that sampled oly oe classroom per school. Page 10

11 We must ote the uusual results for the Czech Republic ad the Slovak Republic. Although both coutries did defie samplig strata, we were ot able to reliably estimate the stratum variace compo et. From closer examiatio of their data, sample sizes withi strata appear very ubalaced ad sparse, thus perhaps leadig to the ureliability. Figure 4: Variace Decompositio for PIRLS 001 Part Stratificatio Efficiecy Despite the may caveats expressed regardig the relevace of the results derived from this multi-level ubalaced aalysis of variace model, we ca still glea may iterestig ad useful fidigs. From the samplig perspective, our primary iterest is to explai as much as possible of the school variace compoet sice this is the level which primarily dictates the ultimate reliability of the survey estimates derived from these data. Our purpose is to defie school strata that will cluster schools accordig to their achievemet levels. Coutries did ideed defie school strata, with varyig levels of detail, based mostly o readily available admiistrative data sources. Our objective the i defiig a stratum variace compoet i this variace decompositio was to evaluate the efficiecy of these atioal stratificatio strategies. As we ca see by the widely varyig magitudes of the Page 11

12 stratum compoets amog participatig coutries, the efficiecy of atioal stratificatio strategies varies greatly. Efficiet samplig strategies are evidet i may coutries, with Argetia ad Colombia as obvious examples. I Argetia, 19.5% of the total variace is explaied by differeces betwee strata; a regioal stratificatio cotributig greatly to this efficiecy. O the substative side, this fidig ca also be useful sice we ca ow ackowledge importat regioal differeces i educatioal achievemet. This has obvious policy implicatios, i settig objectives ad mechaisms towards reducig regioal disparities i the delivery of learig opportuities. Regioal stratificatio is a geeral tred i settig efficiet samplig strategies amog coutries. Most of the coutries with large itraclass correlatios ted to highlight regioal disparities (Argetia, Belize, Kuwait, Moldova, Morocco, Romaia, Russia Federatio & Turkey); at miimum we ca fid disparities betwee urba ad rural parts (Colombia, Macedoia & Romaia). School type, amely, public ad private schools, ca also be idicators of disparities i achievemet (Argetia, Colombia & Ira). Laguage has also bee used efficietly to stratify (Macedoia, Moldova & Turkey). Kuwait implemeted a geder stratificatio that proved to be efficiet. The use of efficiet stratificatio strategies does ot beefit oly coutries with high itraclass correlatios. Coutries with low itraclass correlatios also produced efficiet implemetatios of similar stratificatio strategies, although with lesser impact sice there was geerally less betweeschool variace to apportio to a stratificatio variace compoet. At the other ed of the spectrum, we ca readily fid some coutries with iefficiet, or iadequate, stratificatio strategies. They ca be idetified by their small stratum variace compoet, such as Cyprus, Icelad, Lithuaia, Morocco, the Netherlads, Scotlad, Sigapore, Sloveia, Swede ad Turkey. Those with low itraclass correlatios ca be grated some form of dispesatio i this regard, cosiderig there is less school variace to be apportioed to a stratum variace compoet. But some cosideratio of alterate stratificatio strategies should ot be discouted. Those coutries with large itraclass correlatios ca ill afford iefficiet stratificatio strategies sice the alterative is to cosider larger sample sizes, geerally regarded as a burde o resour ces. Morocco ad Sigapore have the largest school variace compoets (43.% & 41.3% respectively). Both should be ecouraged to cosider alterative stratificatio strategies. Prelimiary ivestigatios lead us to believe Morocco might wat to cosider school type i its stratificatio. Sigapore did ot implemet ay type of stratificatio, which seems plausible cosiderig that early all schools are selected i the sample. Noetheless, some from of stratificatio should perhaps be cosidered i the future. For TIMSS 003, Sigapore has chose to sample two classrooms per school at the 8 th grade, sice they expect to have a large classroom compoet at that grade level. If this proves successful, the it could also be cosidered for the 4 th grade i PIRLS. The use of efficiet stratificatio strategies is imperative for coutries that have large stadard errors i their published results (Marti & Mullis, p. 6). I PIRLS, we would iclude Argetia, Moroc co ad Sigapore as cadidates, sice their stadard errors are greater tha the targeted 5 poits. We have already discussed Morocco ad Sigapore. Sice Argetia appears to have implemeted a soud stratificatio strategy, the oly other recourse would appear to be a icrease i sample size. Coclusio We have examied the coceptual models behid the estimatio of the itraclass correlatio i particular, ad of variace compoets i geeral. The models make uderlyig assumptios about the data to which they are applied. We have discussed how some of those assumptios may ot hold uder specific circumstaces, thereby forcig us to qualify ay ifereces made based o our fidigs. We would therefore wat to ivestigate two major aspects related to the proper applicatio of the ANOVA models to data such as PIRLS: How does the ubalaced ature of PIRLS data affect the proper estimatio of variace compoets i a ANOVA model? Page 1

13 Ca we be assured that the assumptio of equal variace for the betwee-classroom withi school variace compoet holds? If ot, how does it affect the proper estimatio of this variace compoet? Based o our ivestigatio of the itraclass correlatios from the PIRLS data, we would be wise to coside r itraclass correlatios usig classrooms as the first-level uits, rather tha schools, if they are to be used to compare coutries. The resultig itraclass correlatio is more readily comparable amog participatig coutries. This is ot to say that itraclass correlatios based o schools as firstlevel uits are iappropriate i geeral. Those ca still be used for sample size determiatio, for example. The applicatio of a four-level ANOVA model to the PIRLS data has provided very useful isight ito the atioal stratificatio strategies employed by the various participatig coutries. We are i a positio to determie where successful strategies have bee implemeted ad, more importatly, where alterate strategies would prove beeficial i future survey cycles. This latter poit is particularly crucial if we cosider that coutries with large itraclass correlatios ted to have low levels of achievemet. Havig a better uderstadig of the variace structure i those coutries would be highly relevat both for policy implicatios ad sample desig developmet. REFERENCES BORTZ, Juerge, Statistik Fuer Sozialwisseschaftler, Spriger-Verlag, 1989 COCHRAN, William G., Samplig Techiques, Third Editio, Joh Wiley ad Sos, 1977 MARTIN, Mick, MULLIS, Ia V.S., et al., PIRLS 001 Iteratioal Report, Bosto College, 003 GONZALEZ, Eugeio, Keedy, A, PIRLS 001 User Guide for the Iteratioal Database, Bosto College, 003 NETER, Joh, & WASSERMAN, William, Applied Liear Statistical models, Irwi, 1974 Page 13