Perre Foy Statstcs Canada 4 Calculaton of Samplng Weghts 4.1 OVERVIEW The basc sample desgn used n TIMSS Populatons 1 and 2 was a two-stage stratfed cluster desgn. 1 The frst stage conssted of a sample of schools; the second stage conssted of samples of one ntact mathematcs classroom from each elgble target grade n the sampled schools. The desgn requred schools to be sampled usng a probablty proportonal to sze (PPS) systematc method, as descrbed by Foy, Rust, and Schlecher (1996), and classrooms to be sampled wth equal probabltes (Schlecher and Snscalco, 1996). Whle TIMSS had a basc desgn for how the natonal representatve samples of students n Populatons 1 and 2 were to be drawn, aspects of the desgn were adapted to natonal condtons and analytcal needs. For example, many countres stratfed the school samplng frame by varables of natonal nterest. As another example, some countres chose to sample two classrooms from the upper grade of the target populaton. Chapter 2 of ths report documents n detal the natonal samples for TIMSS Populatons 1 and 2. Whle a mult-stage stratfed cluster desgn greatly enhances the feasblty of data collecton, t results n dfferental probabltes of selecton; consequently, each student n the assessment does not necessarly represent the same number of students n the populaton, as would be the case f a smple random samplng approach were employed. To account for dfferental probabltes of selecton due to the nature of the desgn and to ensure accurate survey estmates, TIMSS computed a samplng weght for each student that partcpated n the assessment. Ths chapter documents the calculaton of the samplng weghts for students sampled for the Populatons 1 and 2 man assessment and for those students subsampled to also take part n the performance assessment. 2 4.2 WEIGHTING PROCEDURES The general weghtng procedure for TIMSS requred three steps. The frst step for all target populatons conssted of calculatng a school weght. The school weght also ncorporates weghtng factors from any addtonal front-end samplng stages requred 1 The target populatons are defned as follows: 2 Populaton 1: Students enrolled n the two adjacent grades where most 9-year-old students are found at the tme of testng (thrd and fourth grades n many countres) Populaton 2: Students enrolled n the two adjacent grades where most 13-year-old students are found at the tme of testng (seventh and eghth grades n many countres). See Harmon and Kelly (1996) for detals of the samplng procedures for the performance assessment. 71
by some TIMSS partcpants. 3 A school-level nonresponse adjustment was appled to the school weght; t was calculated ndependently for each desgn doman or explct stratum. The second step conssted of calculatng a classroom weght. A classroom-level nonresponse adjustment was not necessary snce n most cases a sngle classroom was selected per school at each grade level. When only one of the sampled classrooms n a school partcpated, a grade-specfc school-level response adjustment was used. When one of two selected classrooms n a school (when a country chose to sample two classrooms per grade) dd not partcpate, the classroom weght was calculated as though a sngle classroom had been selected n the frst place. The classroom weght was calculated ndependently for each school and grade. The fnal step conssted of calculatng a student weght. A student-level nonresponse adjustment was appled to the student weght. The student weght was calculated ndependently for each sampled classroom. The overall samplng weght attached to each student record s the product of the three ntermedate weghts: the frst stage (school) weght, the second stage (classroom) weght, and the thrd stage (student) weght. The overall samplng weght attached to each student n the performance assessment sub-sample s the product of the frst stage weght adjusted for the subsamplng of schools requred, the second stage weght, and the thrd stage weght adjusted for the subsamplng of students requred at ths stage. 4.2.1 Frst-Stage (School) Weght The frst stage weght represents the nverse of the frst stage selecton probablty assgned to a sampled school. The TIMSS sample desgn requred that school selecton probabltes be proportonal to school sze, wth school sze beng enrollment n the target grades. The basc frst stage weght for the th sampled school was thus defned as M -------------- n * m where n s the number of sampled schools, m s the measure of sze for the th school and M m 1 1 where N s the total number of schools n the stratum. The basc frst stage weght also ncorporates a weghtng factor or factors resultng from addtonal front-end samplng stages that were requred by some TIMSS partcpants. Ths occurred when geographcal regons were sampled before schools were se- N 72 3 For example, the Unted States sampled school dstrcts as prmary samplng unts (PSUs), and then schools wthn the sampled PSUs.
lected. The calculaton of such weghtng factors s smlar to the frst stage weght snce samplng geographcal regons was also done wth probablty proportonal to sze (PPS). The resultng frst stage weght s smply the product of the "regon" weght and the frst stage weght as descrbed earler. In some countres, schools were selected wth equal probabltes. Ths generally occurred when no relable measure of school sze s avalable. In ths case, the basc frst stage weght for the th sampled school was defned as N ---- n where n s the number of sampled schools and N s the total number of schools n the stratum. It should be noted that n ths case the basc weght for all sampled schools s dentcal. 4.2.1.1 School-Level Response Rate (Partcpaton Rate) A school-level response rate, weghted and unweghted, was calculated to measure the proporton of orgnally selected schools that ultmately partcpated n the assessment. Snce replacement schools were used to mantan the sample sze, school-level response rates have been reported both wth and wthout the use of replacement schools. The calculaton of the response rate used the followng terms, derved from the data collecton: n ex number of sampled schools that should have been excluded number of orgnally sampled schools that partcpated n rp number of replacement schools that partcpated n nr number of non-respondng schools (nether the orgnally selected schools nor ther replacements partcpatng.) Note that the followng equaton holds: n ex + + n rp + n nr n The unweghted school-level response rate s defned as the rato of orgnally sampled schools that partcpated to the total number of sampled schools mnus any excluded schools. It was calculated by the followng equaton: sc R unw ----------------------------------- + n rp + n nr 73
The weghted school-level response rate s defned n a smlar manner. The weght assgned to the th sampled school for ths purpose s the samplng nterval used to select t,. The weghted school-level response rate, based solely on orgnally selected schools, s therefore the rato of the weghted sum of orgnally sampled schools that partcpated to the weghted sum of all sampled schools less any excluded schools. It was calculated by the followng equaton: R w sc 1 1 ----------------------------------------------------------------------- n rp 1 + + n nr 1 The weghted school-level response rate, ncludng replacement schools, was calculated by the followng equaton: sc R wrp, 1 1 + n rp 1 n rp 1 ----------------------------------------------------------------------- + + n nr 1 4.2.1.2 School-Level Nonresponse Adjustment Frst stage weghts were calculated for orgnally sampled schools and replacement schools that partcpated. Any sampled schools that were no longer elgble were removed from the calculaton of ths nonresponse adjustment. Examples are secondary schools ncluded n the samplng frame by mstake and schools that no longer exsted. The school-level nonresponse adjustment was calculated separately for each desgn doman and explct stratum. The school-level nonresponse adjustment was calculated as follows: n n A ex sc -------------------- + n rp and the fnal frst stage weght for the th school thus becomes FW sc A sc * 74
In the event that a sampled school had partcpatng classrooms n only one grade when both grades were n fact present, the school-level nonresponse adjustment becomes grade-specfc. Such a school was consdered a partcpant for the grade n whch students were tested but as a non-partcpant for the grade n whch no students were tested. Ths led also to the calculaton of separate school-level response rates by grade. 4.2.2 Second-Stage (Classroom) Weght The second stage weght represents the nverse of the second stage selecton probablty assgned to a sampled classroom. Classrooms were sampled n one of two ways n Populaton 1 and Populaton 2: Equal probablty f there was no subsamplng of students wthn a classroom Probablty proportonal to classroom sze f subsamplng of students wthn a classroom was requred The second stage weght was calculated ndependently for each grade wthn a sampled school n Populaton 1 and Populaton 2. A nonresponse adjustment was not requred for the second stage weght. Where the classroom selected n one target grade dd not partcpate but the sampled classroom n the other target grade dd, the separate frst stage nonresponse adjustments were appled by target grade. 4.2.2.1 Equal Probablty Weghtng For grade g wthn the th school, let C g, be the total number of classrooms and c g be the number of sampled classrooms. Usng equal probablty samplng, the fnal second stage weght assgned to all sampled classrooms from grade g n the th school was g FW, C g, cl1 -------- c g As a rule, c g takes the value 1 or 2 and remans fxed for all sampled schools. In cases where c g has the value 2 and only one of the sampled classrooms partcpated, a classroom-level nonresponse adjustment was appled to the second stage weght by multplyng t by the factor 2. 4.2.2.2 Probablty Proportonal to Sze (PPS) Weghtng For grade g wthn the th school, let k g,,j be the sze of the jth classroom. Usng PPS samplng, the fnal second stage weght assgned to the jth sampled classroom from grade g n the th school was K g, FW cl2 --------------------- c g * k 75
where c g s the number of sampled classrooms as defned earler and c g K g, k j 1 Agan, as a rule, c g takes the value 1 or 2 and wll reman fxed for all sampled schools. In cases where c g has the value 2, and only one of the sampled classrooms partcpated, a classroom-level nonresponse adjustment was appled to the second stage weght by multplyng t by the factor 2. 4.2.3 Thrd-Stage (Student) Weght The thrd stage weght represents the nverse of the thrd stage selecton probablty attached to a sampled student. If ntact classrooms were sampled as specfed n Foy, Rust, and Schlecher (1996), then the basc thrd stage weght for the jth grade g classroom n the th school was BW st 1.0 If, on the other hand, subsamplng of students was requred wthn sampled classrooms, then the basc thrd stage weght for the jth grade g classroom n the th school was gj BW,, k st ---------- s g where k g,,j s the sze of the jth grade g classroom n the th school, as defned earler, and s g s the number of sampled students per sampled classroom. The latter number usually remans constant for all sampled classrooms n a grade. 4.2.3.1 Student-Level Response Rate (Partcpaton Rate) and Adjustment The basc thrd stage weght requres an adjustment to reflect the outcome of the data collecton efforts. The followng terms were derved from the data collecton for each sampled classroom: s ex number of sampled students that should have been excluded s rs number of sampled students that partcpated s nr number of sampled students that dd not partcpate. 76
Note that the followng equaton holds: s ex s rs s nr + + s where s g,,j s the number of sampled students per sampled classroom. Ths number should be constant f subsamplng of students s done wthn each sampled classroom and represents the classroom sze, k g,,j, when ntact classrooms are tested. The student-level response rate, for a gven classroom, was calculated as follows: R st s rs ----------------------------------- s rs + s nr Excluded students (.e., those meetng the gudelnes for student-level exclusons specfed n Foy, Rust, and Schlecher, 1996) were not ncluded n the calculaton of the response rate. The student-level nonresponse adjustment was calculated as follows: s A rs st s nr + ---------------------------------- s rs Note that the student-level nonresponse adjustment s smply the nverse of the student-level response rate. The fnal thrd stage weght for the jth grade g classroom n the th school thus becomes FW st A st * BW st The weghted overall student-level response rate was computed as follows: R w st rs 1 rs + nr 1 BW cl1 * * BW cl1 * * BW st ------------------------------------------------------------------------- BW st where the numerator s the summaton of the basc weghts over all respondng students, and the denomnator s the summaton of the basc weghts over all respondng and nonrespondng students. Weghted student response rates were reported separately by grade n the TIMSS nternatonal reports. 77
4.2.4 Overall Samplng Weghts The overall samplng weght s smply the product of the fnal frst stage weght, the approprate fnal second stage weght, and the approprate fnal thrd stage weght. If ntact classrooms were tested, then the overall samplng weght was W FW sc FW sc * * FW st If subsamplng wthn classrooms was done, then the overall samplng weght was W FW sc FW cl2 * * FW st It s mportant to note that samplng weghts vared by school, grade, and classroom. However, students wthn the same classroom have the same samplng weghts. The use of samplng weghts s crtcal to obtanng proper survey estmates when samplng technques other than smple random samplng are used. TIMSS has produced a samplng weght for each student sampled for the TIMSS man (wrtten) assessment and subsampled for the performance assessment. Secondary analysts usng the TIMSS data wll need to be aware of ths and use the proper weghts when conductng analyses and reportng results. 78
REFERENCES Foy, P., Rust, K., and Schlecher, A. (1996). Sample desgn. In M.O. Martn and D.L. Kelly (Eds.), TIMSS techncal report, volume I: Desgn and development. Chestnut Hll, MA: Boston College. Harmon, M. and Kelly, D.L. (1996) Performance assessment. In M.O. Martn and D.L. Kelly (Eds.), TIMSS techncal report, volume I: Desgn and development. Chestnut Hll, MA: Boston College. Schlecher, A. and Snscalco, M.T. (1996). Feld operatons. In M.O. Martn and D.L. Kelly (Eds.), TIMSS techncal report, volume I: Desgn and development. Chestnut Hll, MA: Boston College. 79