Sample Design in TIMSS and PIRLS

Sample Desgn n TIMSS and PIRLS Introducton Marc Joncas Perre Foy TIMSS and PIRLS are desgned to provde vald and relable measurement of trends n student achevement n countres around the world, whle keepng to a mnmum the burden on schools, teachers, and students. Both programs employ rgorous school and classroom samplng technques so that achevement n the student populaton as a whole may be estmated accurately by assessng ust a sample of students from a sample of schools. TIMSS assesses mathematcs and scence achevement at two grade levels and so TIMSS has two target populatons all students enrolled at the fourth grade and all students enrolled at the eghth grade. Countres may assess ether or both student populatons. As an assessment of readng comprehenson n prmary school, the target populaton for PIRLS s all students enrolled at the fourth grade. TIMSS and PIRLS both employ the same two-stage random sample desgn, wth a sample of schools drawn as a frst stage and one or more ntact classes of students selected from each of the sampled schools as a second stage. Intact classes of students are sampled rather than ndvduals from across the grade level or of a certan age because TIMSS and PIRLS pay partcular attenton to students currcular and nstructonal experences, and these typcally are organzed on a classroom bass. Samplng ntact classes also has the operatonal advantage of less dsrupton to the school s day to day busness than ndvdual student samplng. Natonal Samplng Plan Each country partcpatng n TIMSS or PIRLS needs a plan for defnng ts natonal target populaton and applyng the TIMSS and PIRLS samplng methods to acheve a natonally representatve sample of schools and students. The development and mplementaton of the natonal samplng plan s a collaboratve exercse nvolvng the country s Natonal Research Coordnator (NRC) and the TIMSS and PIRLS samplng experts. Statstcs Canada s responsble for advsng the Natonal Research Coordnator on all samplng matters and for ensurng that the natonal samplng plan conforms to the TIMSS and PIRLS standards. In cooperaton SAMPLING IMPLEMENTATION 1

METHODS AND PROCEDURES 2 SAMPLING IMPLEMENTATION wth samplng staff from the IEA Data Processng and Research Center (DPC), Statstcs Canada works wth the Natonal Research Coordnator to select the natonal school sample(s) and produce all supportng documentaton for trackng the sampled schools. Ths ncludes ensurng that the school samplng frame (the school populaton lst from whch the school sample s drawn) provded by the Natonal Research Coordnator s complete and satsfactory; checkng that categores of excluded students are clearly defned, ustfed, and kept to a mnmum; assstng the Natonal Research Coordnator n determnng the sample sze and a stratfcaton plan that wll meet both nternatonal and natonal obectves; and drawng a natonal sample of schools. When samplng has been completed and all data collected, Statstcs Canada documents populaton coverage and school and student partcpaton rates and constructs approprate samplng weghts for use n analyzng and reportng the results. The TIMSS & PIRLS Internatonal Study Center, n cooperaton wth Statstcs Canada and the IEA DPC, provdes Natonal Research Coordnators wth a seres of manuals to gude them through the samplng process. More specfcally, TIMSS & PIRLS Survey Operatons Procedures Unt 1: Samplng Schools and Obtanng ther Cooperaton (TIMSS & PIRLS Internatonal Study Center, 2008) descrbes the steps nvolved n defnng the natonal target populaton and selectng the school sample, and TIMSS & PIRLS Survey Operatons Procedures Unt 3: Contactng Schools and Samplng Classes (TIMSS & PIRLS Internatonal Study Center, 2010) descrbes the procedure for samplng classes wthn the sampled schools and makng preparatons for conductng the assessments. Wthn-school samplng procedures for the feld test are documented n TIMSS & PIRLS Survey Operatons Procedures Unt 2: Preparng for and Conductng the Feld Test (TIMSS & PIRLS Internatonal Study Center, 2009). The TIMSS or PIRLS Natonal Research Coordnator s responsble for provdng Statstcs Canada wth all nformaton and documentaton necessary to conduct the natonal samplng, and for conductng all samplng operatons n the country. In partcular, the NRC s expected to dentfy the grade(s) that correspond to the nternatonal target populaton(s); create a samplng frame by lstng all schools n the populaton that have classes wth students n the target grade(s); determne natonal populaton coverage and exclusons, n accordance wth the TIMSS and PIRLS nternatonal gudelnes; work wth Statstcs Canada to develop a natonal samplng plan and dentfy sutable stratfcaton varables, ensurng that these varables are present and correct

for all schools; contact all sampled schools and secure ther partcpaton; keep track of school partcpaton and the use of replacement schools; and conduct all wthn-school samplng of classes. Each NRC s requred to complete a seres of samplng forms documentng the completon of each of these tasks. A crucal feature of each nternatonal meetng of Natonal Research Coordnators s a one-to-one meetng between each NRC and samplng staff at Statstcs Canada and the IEA DPC. At these meetngs, each step of the samplng process s documented and revewed n detal, and NRCs have the opportunty to rase ssues and ask questons about ther natonal stuaton and any challenges they face. Statstcs Canada consults wth the TIMSS & PIRLS Internatonal Study Center or the Internatonal Samplng Referee, as necessary, to resolve ssues and questons. Fnal approval of TIMSS and PIRLS natonal samplng plans s the responsblty of the TIMSS & PIRLS Internatonal Study Center, based upon the advce of Statstcs Canada and the Internatonal Samplng Referee. Defnng the Target Populaton As nternatonal studes of the comparatve effects of educaton on student achevement n mathematcs, scence, and readng, TIMSS and PIRLS defne ther nternatonal target populatons n terms of the amount of schoolng students have receved. The number of years of formal schoolng s the bass of comparson among partcpatng countres. Thus, the nternatonal target populaton for PIRLS and TIMSS at the lower grade s all students n ther fourth year of formal schoolng, and for TIMSS at the upper grade, all students n ther eghth year of formal schoolng. UNESCO s Internatonal Standard Classfcaton of Educaton (ISCED) provdes an nternatonally accepted classfcaton scheme for descrbng levels of schoolng across countres. The ISCED system descrbes the full range of schoolng, from pre-prmary (Level 0) to the second level of tertary educaton (Level 6). ISCED Level 1 corresponds to prmary educaton or the frst stage of basc educaton. The frst year of Level 1 should mark the begnnng of systematc apprentceshp of readng, wrtng, and mathematcs (UNESCO, 1999). Four years after ths would be the target grade for PIRLS and fourth grade TIMSS, and s the fourth grade n most countres. Smlarly, eght years after the frst year of ISCED Level 1 s the target grade for eghth grade TIMSS and s the eghth grade n most countres. However, gven the cogntve demands of the SAMPLING IMPLEMENTATION 3

assessments, TIMSS and PIRLS want to avod assessng very young students. Thus, TIMSS and PIRLS recommend assessng the next hgher grade (.e., ffth grade for PIRLS or fourth grade TIMSS and nnth grade for eghth grade TIMSS) f, for fourth grade students, the average age at the tme of testng would be less than 9.5 years and, for eghth grade students, less than 13.5 years. The fourth-grade and eghth-grade target populatons of students are defned as follows: Fourth grade (PIRLS and TIMSS): All students enrolled n the grade that represents four years of schoolng countng from the frst year of ISCED Level 1, provdng the mean age at the tme of testng s at least 9.5 years. Eghth grade (TIMSS only): All students enrolled n the grade that represents eght years of schoolng countng from the frst year of ISCED Level 1, provdng the mean age at the tme of testng s at least 13.5 years. All students enrolled n the target grade, regardless of ther age, belong to the nternatonal target populaton and should be elgble to partcpate n TIMSS and PIRLS. Because students are sampled n two stages, frst by randomly selectng a school and then randomly selectng a class from wthn the school, t s necessary to dentfy all schools n whch elgble students are enrolled. Essentally, elgble schools for TIMSS or PIRLS are those that have any students enrolled n the target grade, regardless of type of school. All schools of all educatonal sub-systems that have students learnng full-tme n the target grade are part of the nternatonal target populaton, ncludng schools that are not under the authorty of the natonal Mnstry of Educaton. Natonal Target Populatons For most countres, the target grade for PIRLS or TIMSS s the fourth and/ or eghth grade. However, because educatonal systems vary n structure and n polces and practces wth regard to age of startng school and promoton and retenton, there are dfferences across countres n how the target grades are labelled and n the average age of students. To ensure that the approprate natonal target grades are selected, each NRC completes Samplng Form 1, whch dentfes the target grades, the country s name for those grades, and the average age of students n those grades at the tme of data collecton. (To vew an example of a completed Samplng Form 1 for TIMSS and PIRLS n 2011, please clck here.) METHODS AND PROCEDURES 4 SAMPLING IMPLEMENTATION

Natonal Coverage and Exclusons TIMSS and PIRLS are desgned to descrbe and summarze student achevement across the entre target grade (fourth or eghth), and so t s very mportant that natonal target populatons am for comprehensve coverage of elgble students. However, n some cases, poltcal, organzatonal, or operatonal factors make complete natonal coverage dffcult to attan. Thus, n some rare stuatons, certan groups of schools and students may have to be excluded from the natonal target populaton. For example, t may be that a partcular geographcal regon, educatonal sub-system, or language group cannot be covered. Such excluson of schools and students from the target populaton s referred to as reduced populaton coverage. Even countres wth complete populaton coverage fnd t necessary to exclude at least some students from the target populaton because they attend very small schools, have ntellectual or functonal dsabltes, or are non-natve language speakers. Such students may be excluded at the school level (.e., the whole school s excluded) or wthn the school on an ndvdual bass. SCHOOL LEVEL EXCLUSIONS Although t s expected that very few schools wll be excluded from the natonal target populaton, NRCs are permtted to exclude schools on the followng grounds when they consder t necessary: Inaccessblty due to ther geographcally remote locaton; Extremely small sze (e.g., four or fewer students n the target grade); Offerng a grade structure, or currculum, radcally dfferent from the manstream educatonal system; or Provdng nstructon solely to students n the student-level excluson categores lsted below (.e., caterng only to specal needs students). STUDENT LEVEL EXCLUSIONS The nternatonal wthn-school excluson rules are specfed as follows: Students wth functonal dsabltes These are students who have physcal dsabltes such that they cannot perform n the PIRLS and/ or TIMSS testng stuaton. Students wth functonal dsabltes who are able to perform should be ncluded n the testng. Students wth ntellectual dsabltes These are students who are consdered, n the professonal opnon of the school prncpal or by SAMPLING IMPLEMENTATION 5

other qualfed staff members, to have ntellectual dsabltes or who have been tested as such. Ths ncludes students who are emotonally or mentally unable to follow even the general nstructons of the test. Students should not be excluded solely because of poor academc performance or normal dscplnary problems. It should be noted that students wth dyslexa, or other such learnng dsabltes, should be accommodated n the test stuaton f possble, rather than excluded. Non-natve language speakers These are students who are unable to read or speak the language(s) of the test and would be unable to overcome the language barrer n the test stuaton. Typcally, a student who has receved less than one year of nstructon n the language(s) of the test should be excluded. Because dsablty crtera vary from country to country, NRCs are asked to translate the TIMSS and PIRLS nternatonal excluson standards nto the local equvalent. Students should be consdered for excluson strctly n accordance wth the nternatonal standards. If a sampled school contans a class consstng entrely of students from one of the excluson categores, such a class s excluded pror to classroom samplng. NRCs understand that excluson rates must be kept to a mnmum n order that natonal samples accurately represent the natonal target populaton. The overall number of excluded students must not account for more than 5% of the natonal target populaton of students n a country. The overall number ncludes both school-level and wthn-school exclusons. The number of students excluded because they attend very small schools must not account for more than 2% of the natonal target populaton of students. To document populaton coverage and exclusons, each NRC completes Samplng Form 2, whch lsts the number of students n the natonal target populaton and the number of students excluded at both the school level and wthn the school for each populaton to be assessed. (To vew an example of a completed Samplng Form 2 for TIMSS and PIRLS n 2011, please clck here.) Requrements for Samplng the Target Populaton TIMSS & PIRLS set hgh standards for samplng precson, partcpaton rates and sample mplementaton n order to acheve natonal samples of the hghest qualty and survey estmates that are unbased, accurate and nternatonally comparable. METHODS AND PROCEDURES 6 SAMPLING IMPLEMENTATION

SAMPLING PRECISION AND SAMPLE SIZE Because TIMSS and PIRLS are fundamentally studes of student achevement, the precson of estmates of student achevement s of prmary mportance. To meet the TIMSS & PIRLS standards for samplng precson, natonal student samples should provde for a standard error no greater than.035 standard devaton unts for the country s mean achevement. Wth a standard devaton of 100 on the TIMSS and PIRLS achevement scales, ths standard error corresponds to a 95% confdence nterval of ± 7 score ponts for the achevement mean and of ± 10 score ponts for the dfference between achevement means from successve cycles (e.g., the dfference between a country s achevement mean on TIMSS 2007 and TIMSS 2011). Sample estmates of any student-level percentage estmate (e.g., a student background characterstc) should have a confdence ntervals of ± 3.5%. For most countres, the TIMSS and PIRLS precson requrements are met wth a school sample of 150 schools and a student sample of 4,000 students for each target grade. Dependng on the average class sze n the country, one class from each sampled school may be suffcent to acheve the desred student sample sze. For example, f the average class sze n a country were 27 students, a sngle class from each of 150 schools would provde a sample of 4,050 students (assumng full partcpaton by schools and students). Some countres choose to sample more than one class per school, ether to ncrease the sze of the student sample or to provde a better estmate of school-level effects. A school sample larger than the mnmum of 150 schools may be requred under the followng crcumstances: The average class sze n a country s so small that, even when samplng more than one classroom per school, t s not possble to reach the student sample sze requrements by selectng only 150 schools. Prevous cycles of TIMSS and/or PIRLS showed that the samplng precson requrements cannot be met unless a larger school sample s selected. Classes wthn schools are tracked by student performance (more common at eghth grade than at fourth grade). Ths ncreases varaton between classes n student achevement and can reduce samplng precson. In ths stuaton, t s advsable to sample at least two classrooms per school whenever possble, n addton to samplng more schools. A hgh level of non-response s antcpated, leadng to sample attrton and reduced sample sze. Note that whle a larger school sample helps to SAMPLING IMPLEMENTATION 7

mantan sample sze n the face of non-response, t does not compensate for non-response bas. Feld Test Sample The school sample for the feld test for TIMSS and PIRLS s drawn at the same tme and from the same populaton of schools as the full sample. The feld test sample sze requrement s 200 students per feld test achevement booklet. The total feld test sample sze s a functon of the number of achevement booklets beng feld tested. Typcally, PIRLS has four feld test booklets and so requres a feld test sample of 800 students, whereas TIMSS wth sx booklets requres a sample of 1200 students at each grade. PARTICIPATION RATES To mnmze the potental for non-response bas, PIRLS and TIMSS am for 100% partcpaton by sampled schools, classrooms and students, whle recognzng that some degree of non partcpaton may be unavodable. For a natonal sample to be fully acceptable t must have ether: A mnmum school partcpaton rate of 85%, based on orgnally sampled schools; AND A mnmum classroom partcpaton rate of 95%, from orgnally sampled schools and replacement schools; AND A mnmum student partcpaton rate of 85%, from sampled schools and replacement schools; OR A mnmum combned school, classroom and student partcpaton rate of 75%, based on orgnally sampled schools (although classroom and student partcpaton rates may nclude replacement schools). Classrooms wth less than 50% student partcpaton are deemed to be not partcpatng. Developng and Implementng the Natonal Samplng Plan Although Natonal Research Coordnators are responsble for developng and mplementng natonal samplng plans, Statstcs Canada and the IEA DPC work closely wth NRCs to help ensure that these samplng plans fully meet the standards set by the TIMSS & PIRLS Internatonal Study Center, whle also adaptng to natonal crcumstances and requrements. Natonal samplng plans must be based on the nternatonal two-stage sample desgn (schools METHODS AND PROCEDURES 8 SAMPLING IMPLEMENTATION

as the frst stage and classes wthn schools as the second stage) and must be approved by Statstcs Canada. (For more nformaton about the TIMSS & PIRLS nternatonal sample desgn, please clck here.) Stratfcaton Stratfcaton conssts of arrangng the schools n the target populaton nto groups, or strata, that share common characterstcs such as geographc regon or school type. Examples of stratfcaton varables used n TIMSS and PIRLS nclude regon of the country (e.g., states or provnces); school type or source of fundng (e.g., publc or prvate); language of nstructon; level of urbanzaton (e.g., urban or rural area); soco-economc ndcators; and school performance on natonal examnatons. In TIMSS and PIRLS, stratfcaton s used to: Improve the effcency of the sample desgn, thereby makng survey estmates more relable; Apply dfferent sample desgns, such as dsproportonate sample allocatons, to specfc groups of schools (e.g., those n certan states or provnces); Ensure proportonal representaton of specfc groups of schools n the sample. School stratfcaton can take two forms: explct and mplct. In explct stratfcaton, a separate school lst or samplng frame s constructed for each stratum and a sample of schools s drawn from that stratum. In TIMSS and PIRLS, the maor reason for consderng explct stratfcaton s dsproportonate allocaton of the school sample across strata. For example, n order to produce equally relable estmates for each geographc regon n a country, explct stratfcaton by regon may be used to ensure the same number of schools n the sample for each regon, regardless of the relatve populaton sze of the regons. Implct stratfcaton conssts of sortng the schools by one or more stratfcaton varables wthn each explct stratum, or wthn the entre samplng frame f explct stratfcaton s not used. The combned use of mplct strata and systematc samplng s a very effectve and smple way of ensurng a proportonal sample allocaton of students across all mplct strata. Implct stratfcaton also can lead to mproved relablty of achevement estmates, provded the mplct stratfcaton varables are correlated wth student achevement. (To SAMPLING IMPLEMENTATION 9

vew a summary of the stratfcaton varables used by countres n TIMSS & PIRLS n 2011, please clck here.) Natonal Research Coordnators consult wth Statstcs Canada and the IEA DPC to dentfy the stratfcaton varables to be ncluded n ther samplng plans. The school samplng frame s sorted by the stratfcaton varables pror to samplng schools so that adacent schools are as smlar as possble. Regardless of any other explct or mplct varables that may be used, the school sze s always ncluded as an mplct stratfcaton varable. To document the stratfcaton varables used n ther samplng plans, each NRC completes Samplng Form 3, whch lsts the varables to be used for explct and mplct stratfcaton, and the number of levels of each stratfcaton varable. (To vew an example of a completed Samplng Form 3 for TIMSS and PIRLS n 2011, please clck here.) School Samplng Frame One of the Natonal Research Coordnator s most mportant samplng tasks s the constructon of a school samplng frame for the target populaton. The samplng frame s a lst of all schools n the country that have students enrolled n the target grade, and s the lst from whch the school sample s drawn. A wellconstructed samplng frame provdes complete coverage of the natonal target populaton wthout beng contamnated by ncorrect or duplcate entres or entres that refer to elements that are not part of the defned target populaton. If the natonal samplng plan calls for explct stratfcaton, there should be a separate samplng frame for each explct stratum. A sutable school measure of sze (MOS) s a crtcal aspect of the natonal samplng plan, because the sze of the school determnes the school s probablty of selecton. The most approprate school measure of sze s an up-to-date count of the number of students n the target grade. If the number of students n the target grade s not avalable, the total student enrolment n the school may be the best avalable substtute. Samplng Form 4 provdes some basc nformaton about the school samplng frame, ncludng the average class sze at the target grade, the number of classrooms to be sampled per school, the school measure of sze (MOS) to be used for school samplng, and the school year for whch the frame was constructed. (To vew an example of a completed Samplng Form 4 for TIMSS and PIRLS n 2011, please clck here.) The school samplng frame s usually a spreadsheet contanng a sngle METHODS AND PROCEDURES 10 SAMPLING IMPLEMENTATION

entry for each school. Ths entry ncludes a unque dentfcaton number and contact nformaton (f approprate gven the country s prvacy laws), the values of the stratfcaton varables for the school, and the school measure of sze. It s useful f the school entry also ncludes the number of classes n the school n the target grade because ths provdes a mechansm for predctng n advance the sze of the eventual student sample. Ths predcted sample sze may be compared wth the eventual student sample sze as a check on the samplng process. For an example of a partal samplng frame for a country assessng TIMSS 2011 at the eghth grade, please clck here. In ths example, regon and urbanzaton could be used as stratfcaton varables. Samplng Schools Once the school samplng frame s structured to meet all nternatonal and natonal requrements, Statstcs Canada can draw the school sample. If the samplng frame s explctly stratfed, t s necessary to decde how the school sample s to be allocated among the explct strata (.e., the number of schools to be sampled n each stratum.) When ths has been decded, a sample of schools s selected wthn each explct stratum usng systematc samplng wth probabltes proportonal to sze. (For a descrpton of the school samplng procedure, please clck here.) The PPS technque means that the larger schools, those wth more students, have a hgher probablty of beng sampled than the smaller schools. However, ths dfference n the selecton probabltes of larger and smaller schools s largely offset at the second stage of samplng by selectng a fxed number of classes (usually one or two) wth equal probablty from the sampled school. Classes n large schools wth many classes at the target grade have a lower probablty of selecton than classes n smaller schools that have ust one or two classes. Even though the feld test s scheduled n the school year before the year of data collecton n most countres, the preferred approach n TIMSS and PIRLS s to select both samples of schools at the same tme. Ths ensures that both the feld test and data collecton samples consttute random samples representatve of all schools n the country, and that no school s selected for both samples. Replacement Schools Although deally all schools sampled for TIMSS and PIRLS should partcpate n the assessments, and although NRCs work hard to acheve ths goal, t s antcpated that a 100 percent partcpaton rate may not be possble n all countres. To avod sample sze losses, the samplng plan dentfes, a pror specfc replacement schools for each sampled school. Each SAMPLING IMPLEMENTATION 11

orgnally sampled school has two pre-assgned replacement schools, usually the school mmedately precedng the orgnally sampled school on the school samplng frame and the one mmedately followng t. Replacement schools always belong to the same explct stratum as the orgnal, but although they may come from dfferent mplct strata f the school they are replacng s ether the frst or last school of an mplct stratum. The man ustfcaton for replacement schools n TIMSS and PIRLS s to ensure adequate sample szes for analyss of subpopulaton dfferences. Although the use of replacement schools does not elmnate the rsk of bas due to school nonpartcpaton, employng mplct stratfcaton and orderng the school samplng frame by school sze ncreases the chances that a sampled school s replacements would have smlar characterstcs. Ths approach mantans the desred sample sze whle restrctng replacement schools to strata where nonresponse occurs. Snce the school frame s ordered by school sze, replacement schools also tend to be smlar n sze to the school they are desgnated to replace. NRCs understand that they should make every effort to secure the partcpaton of all of the sampled schools and that only after all attempts to persuade a sampled school to partcpate have faled should the use of ts replacement school be consdered. Samplng Classes Wthn each sampled school, all classes wth students at the target grade are lsted, and one or more ntact classes are selected wth equal probablty of selecton usng systematc random samplng. Ths procedure s mplemented usng the WnW3S samplng software. The selecton of classes wth equal probablty, combned wth the PPS samplng method for schools, n general results n a self-weghtng student sample. If the school has mult-grade classes (.e., the class contans students from more than one grade level), only students from the target grade are elgble for samplng. Because small classes tend to ncrease the rsk of unrelable survey estmates and can lead to reduced overall student sample sze, t s necessary to avod samplng too many small classes. Based on consderaton of the sze dstrbuton of classes and the average class sze, a lower class sze lmt or mnmum class sze (MCS) s specfed for each country. Pror to samplng classes n a school, any class smaller than the MCS s combned wth another class n the school to form a pseudoclass for samplng purposes. The procedure for samplng classes METHODS AND PROCEDURES 12 SAMPLING IMPLEMENTATION

wthn schools s descrbed n more detal n Survey Operatons and Qualty Assurance. Samplng Weghts Natonal student samples n TIMSS and PIRLS are desgned to accurately represent the target populatons wthn a specfed margn of samplng error, as descrbed prevously. After the data have been collected and processed, sample statstcs such as means and percentages that descrbe student characterstcs are computed as weghted estmates of the correspondng populaton parameters, where the weghtng factor s the samplng weght. A student s samplng weght s essentally the nverse of the student s probablty of selecton, wth approprate adustments for nonresponse. In prncple, the stratfed two-stage samplng procedure used n TIMSS and PIRLS, where schools are sampled wth probablty proportonal to school sze and classes are sampled wth probablty nversely proportonal to school sze, provdes student samples wth equal selecton probabltes. However, n practce dsproportonate samplng across explct strata varyng the number of classes selected, and dfferental patterns of nonresponse can result n varyng selecton probabltes, requrng a unque samplng weght for the students n each partcpatng class n the study. The student samplng weght n TIMSS and PIRLS s a combnaton of weghtng components reflectng selecton probabltes and samplng outcomes at three levels school, class, and student. At each level, the weghtng component conssts of a basc weght that s the nverse of the probablty of selecton at that level, together wth an adustment for nonpartcpaton. The overall samplng weght for each student s the product of the three weghtng components: school, class (wthn school), and student (wthn class). School Weghtng Component Gven that schools n TIMSS and PIRLS are sampled wth probablty proportonal to school sze, the basc school weght for the th sampled school (.e., the nverse of the probablty of the th School beng sampled) s defned as: BW sc M n m where n s the number of sampled schools, m s the measure of sze for the th school, and N M m 1 SAMPLING IMPLEMENTATION 13

where N s the total number of schools n the explct stratum. 1 School Nonpartcpaton Adustment If a sampled school does not partcpate n TIMSS or PIRLS and nether does ether of ts two desgnated replacement schools, t s necessary to adust the basc school weght to compensate for the reducton n sample sze. The school-level nonpartcpaton adustment s calculated separately for each explct stratum, as follows: A sc n + n 1 + n 2 + n n + n + n s r r nr s r1 r2 where n s s the number of orgnally sampled schools that partcpated, n r1 and n r2 the number of frst and second replacement schools, respectvely, that partcpated, and n nr s the number of schools that dd not partcpate. Sampled schools that are found to be nelgble 2 are not ncluded n the calculaton of ths adustment. In countres where the TIMSS or PIRLS sample s a census of all schools n the target grades (e.g., Bahran, Cyprus, Kuwat, Malta, and Qatar), the followng school-level adustment s used: A sc m s + m m where m s s the sum of the measures of sze (number of students) from schools that partcpated and m nr the sum of the measures of sze from schools that dd not partcpate. Combnng the basc school weght and the school nonpartcpaton adustment, the fnal school weghtng component for the th school becomes: s nr FWsc Asc BWsc It should be noted that, as well as beng a crucal component of the overall student weght, the fnal school weghtng component s a samplng weght n ts own rght, and should be used n analyses where the school s the analytc unt. Class Weghtng Component The class weghtng component reflects the class-wthn-school selecton 1 For countres such as the Russan Federaton that nclude a prelmnary samplng stage, the basc school weght also ncorporates the probablty of selecton n ths prelmnary stage. The basc school weght n such cases s the product of the prelmnary stage weght and the school weght. 2 A sampled school s nelgble f t s found to contan no elgble students (.e., no students n the target grade). Such schools usually are n the samplng frame by mstake or are schools that recently have closed. METHODS AND PROCEDURES 14 SAMPLING IMPLEMENTATION

probablty. After a school has been sampled and has agreed to partcpate n TIMSS or PIRLS, one or two classes are sampled wth equal probablty from the lst of all classes n the school at the target grade. Because larger schools have more classes from whch to sample than smaller schools, the probablty of class selecton vares wth school sze, wth students n small schools more lkely to have ther class selected than students n large schools. Ths relatvely greater selecton probablty for students n small schools offsets ther lower selecton probablty at the frst stage, where probablty-proportonal-to-sze school samplng results n hgher selecton probabltes for larger schools. The basc class-wthn-school weght for a sampled class s the nverse of the probablty of the class beng selected from all of the classes n ts school. For the th sampled school, let C be the total number of elgble classes and c the number of sampled classes. Usng equal probablty samplng, the basc class weght for all sampled classes n the th school s: BW cl C c For most TIMSS or PIRLS partcpants, c takes the values 1 or 2. Class Nonpartcpaton Adustment Basc class weghts are calculated for all sampled classes n the sampled and replacement schools that partcpate n TIMSS or PIRLS. A class-level nonpartcpaton adustment s appled to compensate for classes that do not partcpate or where the student partcpaton rate s below 50 percent. Such sampled classes are assgned a weght of zero. Class nonpartcpaton adustments are appled at the explct stratum level rather than at the school level to mnmze the rsk of bas. The adustment s calculated as follows: A cl s+ r1+ r2 s+ r1+ r2 δ / c where c s the number of sampled classes n the th school, as defned earler, and d gves the number of partcpatng classes n the th school. Combnng the basc class weght and the class nonpartcpaton adustment, the fnal class weghtng component, assgned to all sampled classes n the th school, becomes: 1 SAMPLING IMPLEMENTATION 15

FWcl Acl BWcl Student Weghtng Component The student weghtng component represents the student-wthn-class selecton probablty. The basc student weght s the nverse of the probablty of a student n a sampled class beng selected. In the typcal TIMSS and PIRLS stuaton where ntact classes are sampled, all students n the class are ncluded, and so ths probablty s unty. However, under certan crcumstances, students may be sampled wthn the class, wth probablty less than unty. For an ntact class wth no student subsamplng, the basc student weght for the th class n the th school s computed as follows: BW st 1 1. 0 For classes wth student subsamplng, the basc student weght for the th class n the th school s: BW n n st rg bs 2 + n, where n rg s the number of students n the th class of the th school selected to partcpate n TIMSS or PIRLS and n bs s the number of students n the class not selected. Adustment for Student Nonpartcpaton The student nonpartcpaton adustment for the th classroom n the th school s calculated as: A s A,, st1 st2 rg rs, + snr srs, where s rs s the number of partcpatng students (.e., students that partcpated n TIMSS or PIRLS and have assessment scores) n the th class of the th school, and s nr s the number of students sampled n ths class and expected to have assessment scores but dd not partcpate n the assessment (for ntact classes, ths wll be the total number of students lsted n the class, not countng excluded students). The fnal student weghtng component for students n the th classroom of the th school s: st std std FW A BW METHODS AND PROCEDURES 16 SAMPLING IMPLEMENTATION

where equals 1 when there was no student subsamplng (ntact classes) and 2 when a sample of students was drawn from the students n the class. Overall Student Samplng Weght The overall student samplng weght s the product of the fnal weghtng components for schools, classes, and students, as follows:, sc cl st W FW FW FW All student data reported n the TIMSS or PIRLS nternatonal reports are weghted by the overall student samplng weght, known as TOTWGT n the TIMSS and PIRLS nternatonal databases. Partcpaton Rates Because nonpartcpaton can result n sample bas and msleadng results, t s mportant that the schools, classes, and students that are sampled to partcpate n TIMSS or PIRLS actually take part n the assessments. To show the level of samplng partcpaton n each country, TIMSS and PIRLS calculate both unweghted partcpaton rates (.e., based on smple counts of schools, classes, and students) and weghted partcpaton rates based on the samplng weghts descrbed n the prevous secton. Unweghted partcpaton rates provde a prelmnary ndcator that may be used to montor progress n securng the partcpaton of schools and classes, whereas weghted partcpaton rates are the ultmate measure of samplng partcpaton. TIMSS and PIRLS report weghted and unweghted partcpaton rates for schools, classes, and students, as well as overall partcpaton rates that are a combnaton of all three. To dstngush between partcpaton based solely on orgnally sampled schools and partcpaton that also reles on replacement schools, school and overall partcpaton rates are computed separately for orgnally sampled schools only and for orgnally sampled together wth replacement schools. Unweghted School Partcpaton Rates The unweghted school partcpaton rate s the rato of the number of partcpatng schools to the number of orgnally sampled schools, excludng any sampled schools found to be nelgble. A school s consdered to be SAMPLING IMPLEMENTATION 17

a partcpatng school f at least one of ts sampled classes has a student partcpaton rate of at least 50 percent. The two unweghted school partcpaton rates are calculated as follows: sc s unweghted school partcpaton rate for orgnally sampled R unw schools only sc R r unw unweghted school partcpaton rate, ncludng orgnally sampled and frst and second replacement schools. R R sc s unw ns n + n + n + n s r1 r2 nr n + n 1 + n 2 n + n + n + n sc r s r r unw s r1 r2 nr UNWEIGHTED CLASS PARTICIPATION RATE The unweghted class partcpaton rate s the rato of the number of sampled classes that partcpated to the number of classes sampled, as follows: R cl unw s+ rl+ r2 s+ rl+ r2 where c s the number of sampled classes n the th school, and c* s the number of partcpatng classes n the th school. Both summatons are across all partcpatng schools. UNWEIGHTED STUDENT PARTICIPATION RATES The unweghted student partcpaton rate s the rato of the number of students that partcpated n TIMSS or PIRLS to the total number of students n the partcpatng schools and classes. Classes where less than 50 percent of the students partcpate are consdered to be not partcpatng, and so students n such classes also are consdered to be nonpartcpants. The unweghted student partcpaton rate s computed as follows: R st unw s c c * srs, rs, + snr METHODS AND PROCEDURES 18 SAMPLING IMPLEMENTATION

OVERALL UNWEIGHTED PARTICIPATION RATES The overall unweghted partcpaton rate s the product of the unweghted school, class, and student partcpaton rates. Because TIMSS and PIRLS compute two versons of the unweghted school partcpaton rate, one based on orgnally sampled schools only and the other ncludng replacements as well as orgnally sampled schools, there also are two overall unweghted partcpaton rates: ov s unweghted overall partcpaton rate for orgnally sampled R unw schools only ov R r unw unweghted overall partcpaton rate, ncludng orgnally sampled and frst and second replacement schools. ov s sc s cl st Runw Runw Runw Runw ov r sc r cl st R R R R unw unw unw unw WEIGHTED SCHOOL PARTICIPATION RATES The weghted school partcpaton rate s the rato of two estmates of the sze of the target student populaton. The numerator s derved from the measure of sze of those sampled schools that partcpated n TIMSS or PIRLS and the denomnator s the weghted estmate of the total student enrollment n the populaton. Weghted school partcpaton rates are computed for orgnally sampled schools and for orgnally sampled and replacement schools combned, as follows: sc R s wtd weghted school partcpaton rate for orgnally sampled schools only sc R r wtd weghted school partcpaton rate, ncludng orgnally sampled and frst and second replacement schools. R R sc s wtd, s+ rl+ r 2 sc r wtd s s+ rl+ r2 s+ rl+ r2 BW FW FW sc FW BW FW sc sc sc cl, FW FW FW cl cl, cl, st, FW FW st, st, FW st Summatons n both the numerator and denomnator are over all respondng students and nclude approprate class and student samplng weghts SAMPLING IMPLEMENTATION 19

were used. Note that the basc school weght appears n the numerator, whereas the fnal school weght appears n the denomnator. WEIGHTED CLASS PARTICIPATION RATES The weghted class partcpaton rate s computed as follows: R cl wtd s+ rl+ r2 s+ rl+ r 2 BW BW FW BW sc sc FW cl, cl FW where both the numerator and denomnator were summatons over all respondng students from classes wth at least 50 percent of ther students partcpatng n the study, and the approprate student-level samplng weghts were used. In ths formula, the basc class weght appears n the numerator, whereas the fnal class weght appears n the denomnator. And, the denomnator n ths formula s the same quantty that appears n the numerator of the weghted school partcpaton rate for all schools, whether orgnally sampled or replacement. WEIGHTED STUDENT PARTICIPATION RATES The weghted student partcpaton rate s computed as follows: R st wtd s+ rl+ r2 s+ rl+ r 2 BW BW BW BW sc sc BW cl, cl FW where both the numerator and denomnator are summatons over all respondng students from partcpatng schools. In ths formula, the basc student weght appears n the numerator, whereas the fnal student weght appears n the denomnator. Also, the denomnator n ths formula s the same quantty that appears n the numerator of the weghted class partcpaton rate for all partcpatng schools, whether orgnally sampled or replacement. WEIGHTED OVERALL PARTICIPATION RATES The overall weghted partcpaton rate s the product of the weghted school, class, and student partcpaton rates. Because there are two versons of the weghted school partcpaton rate, one based on orgnally sampled schools st, st, st, st, METHODS AND PROCEDURES 20 SAMPLING IMPLEMENTATION

only and the other ncludng replacement as well as orgnally sampled schools, there also are two overall weghted partcpaton rates: ov R s wtd weghted overall partcpaton rate for orgnally sampled schools only ov R r wtd weghted overall partcpaton rate, ncludng sampled, frst and second replacement schools. ov s ov s cl st Rwtd Rwtd Rwtd Rwtd ov r sc r cl st R R R R wtd wtd Weghted school, class, student, and overall partcpaton rates are computed for each TIMSS and PIRLS partcpant usng these procedures. wtd wtd References TIMSS & PIRLS Internatonal Study Center. (2008). TIMSS and PIRLS 2011 survey operatons procedures unt 1: Samplng schools and obtanng ther cooperaton. Chestnut Hll, MA: TIMSS & PIRLS Internatonal Study Center, Boston College. TIMSS & PIRLS Internatonal Study Center. (2009). TIMSS and PIRLS 2011 survey operatons procedures unt 2: Preparng for and conductng the PIRLS 2011 and TIMSS 2011 feld test. Chestnut Hll, MA: TIMSS & PIRLS Internatonal Study Center, Boston College. TIMSS & PIRLS Internatonal Study Center. (2010). TIMSS and PIRLS 2011 survey operatons procedures unt 3: Contactng schools and samplng classes for the TIMSS & PIRLS 2011 data collecton. Chestnut Hll, MA: TIMSS & PIRLS Internatonal Study Center, Boston College. UNESCO. (1999). Operatonal manual for ISCED-1997 (nternatonal standard classfcaton of educaton). Pars: UNESCO Insttute of Statstcs. SAMPLING IMPLEMENTATION 21