School tracking and development of cognitive skills additional results

Transcription

1 ömmföäflsäafaäsflassflassflas fffffffffffffffffffffffffffffffffff Dscusson Papers School trackng and development of cogntve sklls addtonal results Sar Pekkala Kerr Wellesley College Tuomas Pekkarnen Aalto Unversty, IZA and IFAU Roope Uustalo HECER, IZA and IFAU Dscusson Paper No. 350 August 2012 ISSN HECER Helsnk Center of Economc Research, P.O. Box 17 (Arkadankatu 7), FI Unversty of Helsnk, FINLAND, Tel , Fax , E-mal Internet

2 HECER Dscusson Paper No. 350 School trackng and development of cogntve sklls addtonal results* Abstract Ths paper evaluates the effects of selectve vs. comprehensve school systems on mltary test scores n mathematcal, verbal and logcal reasonng sklls tests. We use data from the Fnnsh comprehensve school reform whch replaced the old two-track school system wth a unform nne-year comprehensve school. The paper uses a dfferences-n-dfferences approach and explots the fact that the reform was mplemented gradually across the country durng a sx-year perod. We fnd that the reform had a small postve effect on the verbal test scores, but no effect on the mean performance n the arthmetc or logcal reasonng tests. However, the reform sgnfcantly mproved scores on all tests for the students whose parents had only basc educaton. JEL Classfcaton: H52, I21 Keywords: educaton, school system, trackng, comprehensve school, test scores. Sar Pekkala Kerr Tuomas Pekkarnen Wellesley College Aalto Unversty School of Economcs WCW, 106 Central Street P.O. Box Wellesley, MA Aalto USA FINLAND e-mal: skerr3@wellesley.edu e-mal: tuomas.pekkarnen@aalto.f Roope Uustalo Helsnk Center of Economc Research (HECER) P.O. Box 17 (Arkadankatu 7) FI Unversty of Helsnk FINLAND e-mal: roope.uustalo@helsnk.f * Ths workng paper contans addtonal results that dd not ft nto the verson submtted to Journal of Labor Economcs. The authors would lke to thank Davd Autor, Alexs Brownell, Lda Farré, Wllam Kerr, Sandra McNally, Eva Mörk, as well as semnar partcpants at London School of Economcs, Unverstat de Alcante, Wellesley College, ESPE Conference n London, and EALE Conference n Amsterdam for helpful comments. Pekkarnen and Pekkala Kerr are grateful for fnancal assstance from the Academy of Fnland and Yrjö Jahnsson Foundaton, respectvely.

3 1. Introducton Internatonal comparsons of student achevement, such as the OECD s Programme for Internatonal Student Assessment (PISA), have generated a growng nterest n the effect of school systems on student outcomes. Accordng to these comparsons, the dfferences n average test results across countres wth roughly equal school resources are very large. Also, the dsperson of test scores vares consderably across countres. One potental explanaton for cross-country dfferences n the level and, n partcular, the varance of achevement scores s the extent and tmng of trackng, or ablty groupng of students. For example, the OECD has repeatedly argued that varaton n student performance tends to be hgher n countres wth early trackng polces (e.g. OECD, 2003). Hgh varance of student achevement and ts correlaton wth famly background have been seen as problematc from the perspectve of equalty of opportunty. On the other hand, postponng trackng could lower the qualty of teachng at least for the hgh-ablty students. Implct n ths debate s an effcency-equty trade-off: postponement of trackng could mprove equalty but mght decrease the average achevement. The tmng of trackng dffers sgnfcantly between comprehensve and selectve school systems. In the selectve system, trackng students nto dfferent types of schools occurs early, and choces made as early as age ten largely determne later schoolng optons. In the comprehensve system, students often stay n the same schools untl the end of secondary school. In ths study we evaluate the effect of the Fnnsh comprehensve school reform on cogntve sklls tests. Fnland had a selectve two-track school system untl the 1970 s, when the school reform replaced the old two-track system wth a unform comprehensve school system that s smlar to those n other European countres. As a result of the reform the trackng age was postponed from age 10 to 15. The dfferences between the pre- and post-reform systems are smlar to the cross-country dfferences n school systems n the OECD countres today. The effects of the Fnnsh reform are therefore nformatve for the current schoolng polcy debate. Prevous studes such as Meghr and Palme (2005) as well as Pekkarnen et al. (2009) have shown that comprehensve school reforms dd mprove the equalty of opportunty by decreasng the ntergeneratonal correlaton of earnngs. However, the earnngs effects 1

4 reported n prevous studes could be due to peer effects, socal networks, openng of new educatonal opportuntes or drect mpact on productve sklls. Usng prevously unavalable data from the Fnnsh Defense Forces we can partally open the black box relatng school systems to labor market outcomes. We use scores from cogntve sklls tests taken at the begnnng of mandatory mltary servce. These data allow us to examne how the comprehensve school system affected arthmetc and verbal sklls as well as logcal reasonng ablty. Ths s partcularly mportant as recent research emphaszes the role of cogntve sklls, rather than mere school attanment, as mportant determnant of ndvdual earnngs, the dstrbuton of ncome and economc growth. (Hanushek and Wößmann, 2008). Yet exstng evdence on the mpact of major school reforms on cogntve sklls s stll scarce. We use a smlar dfferences-n-dfferences approach as n Pekkarnen et al. (2009) and explot the fact that the school reform was mplemented gradually across regons. However, we evaluate the effects of the school reform drectly on the dstrbuton of sklls that the students are supposed to learn n school. Our results show that the reform had a small postve effect on verbal test scores, but lttle effect on the mean performance n the arthmetc or logcal reasonng tests. However, the reform sgnfcantly mproved the scores n all tests for students whose parents had only basc educaton or low ncome. At the same tme, the reform had no negatve effects on the test scores of students from more advantaged backgrounds. These results are qualtatvely n lne wth the results n Pekkarnen et al. (2009) where t was shown that the comprehensve school reform had a substantal negatve effect on the ntergeneratonal ncome elastcty. However, we fnd that the effects on cogntve sklls are far too small to fully explan the effects on ncome. The rest of the paper proceeds as follows. In the next secton, we revew the lterature on the effects of school trackng and compare our approach to the prevous studes. We then descrbe the content and the mplementaton of the Fnnsh comprehensve reform. The fourth secton descrbes the data and the Fnnsh Army Basc Sklls Test, the results of whch we use as a dependent varable. We then move on to present the dfferences-n-dfferences and maxmum lkelhood estmaton of the effect of the reform on test scores n secton fve and n the sxth secton we dscuss the results. The seventh secton concludes. 2

5 2. Prevous lterature Economc theory provdes somewhat ambguous predctons on the effect of comprehensve versus selectve school systems on student achevement. A comprehensve system, where the entre cohort s n the same class, ncreases heterogenety n the classroom. Ths probably makes classes more dffcult to teach and thereby may lower student achevement (Lazear, 2001). However, any changes n the class composton may also affect student achevement due to peer effects. The effect on the mean achevement depends on whether good students are harmed by bad students more than bad students beneft from beng around good students. 1 Even f the effect on the average student achevement s ambguous, a comprehensve school system should decrease the varance of test scores. Ths s due both to a more homogenous currculum and to less segregated peer groups. Furthermore, as early educatonal choces are more lkely to be determned by famly background (Brunello and Checch, 2007), later trackng age n the comprehensve system may reduce the correlaton between the test scores and the famly background. The most convncng emprcal evdence on the effects of ablty trackng on test scores comes from a feld experment n Kenya where randomly selected schools mplemented trackng and non-trackng polces. Duflo et al. (2011) show that trackng wthn schools seems to beneft all students. However, t s not clear whether these results can be generalzed to developed countres where the student populaton s less heterogeneous. Furthermore, n selectve systems students are typcally not tracked wthn schools but to dfferent types of schools, whch necessarly mples that teacher qualty and currculum may vary consderably across the tracks. In addton, the debate on the relatve advantages of dfferent school systems s prmarly concerned wth trackng nto dfferent types of schools rather than trackng wthn schools. Hence, even a well-desgned randomzed experment of trackng wthn schools s unlkely to settle the polcy queston of whether the entre school system should be selectve or comprehensve. In developed countres, most of the exstng evdence on the potental benefts of selectve versus comprehensve system orgnates from cross-country comparsons. For example, Hanushek and Wößmann (2006) fnd that the varance n test scores n nternatonal student 1 In the Lazear (2001) model the peer effects cannot be dstngushed from the currculum effects. In hs model ll-behavng students stop the teachng for the entre class. Hence, average student qualty determnes how much tme the teacher can spend n teachng and how much materal can be covered. 3

6 assessments s hgher n countres where trackng takes place at an early age. At the same tme, early trackng seems to have generally negatve effects on mean performance. In contrast, Brunello and Checch (2007) and Waldnger (2006), who use a smlar cross-country approach, fnd no effects on the test score varance. These conflctng results from prevous studes reflect, n part, the dffcultes n analyzng the effects of school system based on cross-country data. Whle these studes try to control for varaton due to other factors by controllng for early test scores (Hanushek and Wößmann, 2006; Waldnger, 2006) or by usng tme varaton n the trackng age (Brunello and Checch, 2007), t s far from clear that all relevant cross-country dfferences are relably accounted for. Analyzng changes n test scores when a country swtches from a tracked to a comprehensve system appears to be a more promsng approach to dentfy the effects of the school system on student achevement. Prevous attempts to do ths nclude Kerckhoff et al. (1996) and Galndo-Rueda and Vgnoles (2005), both of whom study the effect of a gradual movement from a selectve school system to a comprehensve system n England. However, as noted by Mannng and Pschke (2006), the areas that frst swtched to the comprehensve system n England were on average poorer than the areas whch retaned the tracked system. It s therefore dffcult to dsentangle the effect of school systems from the regonal dfferences usng common data sources such as the Natonal Chld Development Survey, that only contan a sngle cohort. Relatve to the earler studes, the dstnct advantage of the Fnnsh reform s the avalablty of panel data from several cohorts, whch avods the need to rely exclusvely on crosssectonal varaton. The Fnnsh comprehensve school reform was mplemented gradually regon by regon between 1972 and Ths gradual mplementaton allows us to control for regonal varaton and any tme trends n student achevement usng a dfference-ndfferences approach, whch avods bases such as those dscussed by Mannng and Pschke. Furthermore, the data also nclude nformaton on famles, whch makes t possble to estmate the effect of the reform usng data on brothers who were placed nto dfferent school systems because they dffered n age. 4

7 3. Comprehensve school reform Background Fnland ntroduced a wde-rangng comprehensve school reform n the 1970 s. Smlar reforms had already taken place n Sweden n the 1950s and n Norway n the 1960s (Meghr and Palme, 2005; Aakvk et al. 2010). The Fnnsh comprehensve school reform abolshed the old two-track school system and created a unform nne-year comprehensve school. The man motvaton of the reform was to provde equal educatonal opportuntes to all students, rrespectve of place of resdence or socal background. In the pre-reform system all students entered prmary school ( kansakoulu ) at the age of seven. After four years n the prmary school, at age 11, the students were faced wth the choce of applyng to general secondary school ( oppkoulu ) or contnung n the prmary school. Admssons to the general secondary school were based on an entrance examnaton, a teacher assessment and prmary school grades. Those who were admtted to the general secondary school (52% of the cohort n 1970) contnued frst n the junor secondary schools for fve years, and often went on to the upper secondary school for three addtonal years. At the end of the upper secondary school the students took the matrculaton examnaton that provded elgblty for unversty-level studes. Those who were not admtted or who dd not apply to the general secondary school track contnued n the prmary school. The prmary school lasted altogether eght years. The last two years of prmary school concentrated on teachng vocatonal sklls and were called contnuaton classes or cvc school. After an amendment n 1963 muncpaltes could further extend these cvc school courses by a year, thereby creatng a nne-year prmary school. The mnmum school leavng age, regardless of the track, was sxteen, unless the student had already completed all requred prmary school courses. The pre-reform system s descrbed schematcally n the left-hand panel of Fgure 1. [Fgure 1: SCHOOL SYSTEMS] 2 Ths secton draws on Pekkarnen et al. (2009). 5

8 3.2 Content of the comprehensve school reform The reform ntroduced a new currculum and changed the structure of prmary and secondary educaton. The new currculum ncreased the academc content of educaton compared to the old prmary school currculum by ncreasng the share of mathematcs and scences. In addton, one foregn language became compulsory for all students. The new comprehensve school currculum resembled the old general secondary school currculum and exposed the pupls who n the absence of the reform would have stayed n the prmary school, to a sgnfcantly more academc educaton. The structure of the post-reform school system s descrbed n the rght-hand panel of Fgure 1. The prevous prmary schools, cvc schools and junor secondary schools were replaced by nne-year comprehensve schools. At the same tme, the upper secondary school was separated from the junor secondary school nto a dstnct nsttuton. After the reform, all pupls followed the same currculum n the same establshments (comprehensve schools) up to age sxteen. After nne years n the new comprehensve school, students could choose between applyng to upper secondary school or to vocatonal schools. Admsson to both tracks was based solely on comprehensve school grades. Unlke comprehensve school reforms n many other European countres, the Fnnsh reform dd not extend the length of compulsory schoolng or change the mnmum school leavng age, whch had been sxteen ever snce Although the pre-reform system oblged the muncpaltes to provde only eght years of prmary school, analyss of qunquennal census data reveals that by the tme of the comprehensve school reform most muncpaltes were already offerng a full nne years of prmary school for pupls who dd not go to junor secondary school. In 1975, when the reform had not yet reached the nnth grade, 92.6% of the ffteen-year-olds that would be n the nnth grade f they progressed at the typcal speed were stll n school. Ths fracton remaned at 92.6% n 1980 when the reform had reached the nnth grade n all but the last reform regon. In 1985, well after the reform was completely mplemented, the fracton of those turnng ffteen whle stll n school was only slghtly hgher, 93.9%. To us, these numbers suggest that the comprehensve school reform dd not ncrease the actual mnmum school leavng age. The effect of the reform can thus be nterpreted as comng through changes n the currculum and n the tmng of trackng choces, whch naturally also mples changes n peer groups. 6

9 3.3 The mplementaton of the comprehensve school reform The mplementaton of the reform was preceded by a process of plannng that lasted for two decades. The frst expermental comprehensve schools started ther operaton n In 1968, the Parlament approved the School Systems Act (467/1968), accordng to whch the two track school system would be gradually replaced by a nne-year comprehensve school system. The adopton of the new school system was to take place between 1972 and A regonal mplementaton plan dvded the country nto sx mplementaton regons and dctated when each regon would mplement the comprehensve school system. Regonal school boards were created to oversee the transton process. The muncpaltes that were responsble for operatng the school system could not select the reform date but were forced to follow the plan desgned by the Natonal Board of Educaton. In each regon, pupls n the fve lowest prmary school grades started n the comprehensve school durng the fall term of the year stated n the mplementaton plan. After that, each ncomng cohort of frst graders started ther schoolng n the comprehensve school. The pupls who were already above the ffth grade n the year when the reform was mplemented n ther home regon completed ther schoolng accordng to the pre-reform system. Thus, n each regon t took approxmately four years to fully complete the reform. Fgure 2 llustrates how the reform spread through the Fnnsh muncpaltes durng 1972 to The muncpaltes n whch the reform was mplemented n 1972 were predomnantly stuated n the northernmost provnce of Lapland. In 1973, the reform was mplemented n the north-eastern regons, n 1974 n the northwest, n 1975 n the south-east, n 1976 n the south-west, and fnally, n 1977 n the captal regon of Helsnk. [Fgure 2: COMPREHENSIVE SCHOOL REFORM MAP] The comprehensve school reform faced ntense resstance. The most common argument aganst the reform was that abolshng trackng would reduce the qualty of educaton. As a compromse, ablty trackng was partally retaned so that math and foregn languages were taught at dfferent levels wthn the comprehensve school. Ths ablty groupng was eventually abolshed n A reform of ths scale naturally mpled mportant changes n teacher educaton and the nternal organzaton of schools. In the old system prmary school teachers were educated n 7

10 separate non-unversty nsttutons. Intally, these teachers contnued to teach n the new comprehensve schools. Eventually, teacher educaton was moved nto newly founded schools of educaton wthn unverstes. Over tme the reform therefore led to an ncrease n the average duraton and an mprovement n the qualty of teacher tranng. The mplementaton of the Fnnsh comprehensve school reform makes t n many ways a promsng natural experment for evaluatng the effects of trackng on student outcomes. A partcularly useful setup was created by the regonal mplementaton plan that dctated when each muncpalty moved nto the comprehensve school system. Ths allows us to use a fxed-effects approach to control for the changes over tme and any regonal dfferences between the muncpaltes that were assgned to dfferent mplementaton regons. Gven that we use data from the very frst cohorts that were affected by the reform, the effects are lkely to be somewhat attenuated. For example, n the early years ablty trackng was retaned n certan subjects, most teachers stll only had the shorter prmary school teacher educaton, and the mergng of separate schools nto the same physcal unts probably caused dsruptons n the organzatonal structure. 4. Data A fundamental problem n assessng the effects of school reform on student performance s that students n separate school systems rarely partcpate n comparable tests. Sometmes t s possble to use naton-wde performance evaluatons or nternatonal comparsons of student achevement. However, snce most large-scale school reforms took place n 1960s and 1970s when testng was not as wdespread as today, t s dffcult to fnd tests admnstered to representatve and reasonably large samples of students from both pre- and post-reform school systems. Ths paper uses the results from the Basc Sklls test of the Fnnsh Army. Snce mltary servce s mandatory for men n Fnland, almost the entre male cohort takes the test. The average age at tme of testng s 20, so obvously factors other than the school system may also have had an effect on the test results. On the other hand, the longer-lastng outcomes of school systems are probably more nterestng than the mmedate effects on test results. In addton, the Basc Sklls test s also a strong predctor of earnngs and occupaton later n lfe, 8

11 so any effect of school system on the test scores wll have mportant consequences for lfetme earnngs. 3 The Fnnsh Army Basc Sklls test s desgned to measure general abltes. The Army uses these test results n selectng conscrpts to offcer tranng. Unlke the Swedsh and Norwegan mltary test score data, used for example by Black, Devereux, and Salvanes (2010) and by Lndqvst and Vestman (2011), the Fnnsh test score data are avalable n a dsaggregated form. The test conssts of three subcategores: verbal, arthmetc, and logcal reasonng. Each subtest ncludes forty multple choce questons. In the verbal reasonng subtest, the subject has to choose synonyms or antonyms of gven words, select words that belong to the same category as a gven word, exclude words from a group of words, and dentfy smlar relatonshps between word pars. The arthmetc reasonng test asks the subject to complete number seres, solve verbally expressed mathematcal problems, compute smple arthmetc operatons, and choose smlar relatonshps between pars of numbers. The logcal reasonng test s a standard culture free ntellgence test based on Raven s progressve matrces and ts results should therefore be less affected by pre-test schoolng. 4 On the other hand, both the verbal and arthmetc reasonng categores test sklls that are prmarly taught n school. The test was orgnally created n 1955 and re-desgned n Exactly same test was used over the entre tme span analyzed here. From 1982 onward, the test results are stored n the Army database that also ncludes personal dentfcaton numbers, makng t possble to lnk the test results to nformaton on test takers from other data regsters. Our data nclude all conscrpts who were born between 1962 and 1966 and who were found n the Army database,.e., those who started ther mltary servce after January There s some selectvty n the data due to the fact that t s possble to enter mltary servce as a volunteer before age 20. Thus some men n the oldest cohorts served before the Army regster was created. We expermented wth several solutons to ths problem. For example, we lmted the analyss to those who served n the army at age 20. We also restrcted the data to men born between 1964 and 1966,.e., those that we can observe even f they volunteered for early servce at age 18 or 19. Snce ths made qualtatvely lttle dfference, we only report the results from usng the full sample and smply control for age at test. 5 3 Uustalo (1999) reports that recruts who scored one standard devaton hgher n the Basc Sklls tests earn on average 6% more than recruts wth smlar educaton and experence but lower test scores. 4 The contents of the tests are descrbed n detal n Thonen et al. (2005). 5 Results based on restrcted samples can be found n Appendx 3. 9

12 It s possble to be exempted from the mltary servce due to relgous or ethcal convcton though n 1980s ths was rare. More common reasons for beng exempt from mltary servce were severe health condtons, most often related to mental health problems. However, even these crtera were substantally strcter n the 1980s than today. A comparson of the number of observatons by brth cohort n our data and the correspondng cohort sze n the 1980 populaton census reveals that our test score data contan nformaton on 85.3 percent of the relevant male brth cohorts 6. Ths corresponds closely to the reported fracton of the cohort that served n the mltary n the 1980s (Fnnsh Defense Command, 2000). Fgure 3 plots the dstrbuton of the raw scores,.e. the number of correct answers n each subtest. The dstrbuton of the average score s plotted n the bottom rght corner. These hstograms clearly show that there s plenty of varaton n the test scores; the raw scores are dstrbuted over the whole range from zero to forty. Also, the dstrbuton of the average scores, s symmetrc and not very far from normal dstrbuton. [Fgure 3: DISTRIBUTION OF THE TEST SCORES] Per our request, Statstcs Fnland lnked the test scores to census data on the Fnnsh populaton. The Statstcs Fnland longtudnal census fle contans data on the entre populaton lvng n Fnland n 1970, -75, -80, -85 and -90. From 1990 onwards nformaton s avalable annually. Fnnsh census data are based almost entrely on admnstratve regsters. For example, nformaton on the place of resdence n each census year s based on the Populaton Regster. In general, these regster data are of very hgh qualty. Only a few persons have any mssng data, and the man reasons for not beng ncluded n the census data are resdng abroad and death. Therefore practcally all conscrpts were found n the regster data and our data does not suffer from attrton problems that often plague smlar studes. Census data were used to gather nformaton on the pupls date of brth and place of resdence n 1970, -75 and -80, whch jontly determne whether the ndvdual attended a tracked or a comprehensve school. Statstcs Fnland does not release these data wth a muncpalty-dentfer, but per our request created an ndcator classfyng muncpaltes nto sx categores accordng to the year n whch the comprehensve school reform was 6 Our data are collected from the Fnnsh Army database and contan no nformaton on those who dd not serve n the mltary. It s also clear that those dsmssed from the army due to health and mental reasons dffer n many ways from the rest of the populaton. However, a smple comparson of the number of observatons per cohort and regon n our data to the overall cohort sze n each regon ndcates that the comprehensve school reform had no effect on the lkelhood of servng n the mltary. 10

13 mplemented n each. Except for those who moved between census years between two muncpaltes that mplemented the reform at dfferent years, t s possble to accurately determne whch school system was n place when the students were at the relevant age. The movers were dropped from the data used below, resultng n a reducton of the sample by 10%. 7 The census data also nclude famly codes that can be used to dentfy brother pars and to gather nformaton on parents educaton and earnngs. To be more exact, these famly codes are based on persons lvng n the same household, not necessarly bologcal famly members. We use the famly codes from the 1975 census, when the oldest men n the sample were thrteen years old and most lkely stll lvng at home. Table 1 reports the mean test scores by cohort and reform mplementaton regon as devatons from overall sample means n standard devaton unts. Implementaton regons are defned by the year when the reform took place n the regon. There are large dfferences across regons and a general ncrease n the test scores over tme. These regonal dfferences are correlated wth the average parental educaton and ncome levels of the regons, reported n the last two rows of the table. An ncrease n test scores over tme, generally known as the Flynn effect, has also been documented usng the same data by Kovunen (2007) for a longer tme perod. However, ths effect also reflects dfferences between cohorts other than those due to the school system. The shaded area of the table ndcates the students who attended comprehensve school. Snce these students are younger and are concentrated n the regons wth below average test scores, t s obvous that a cross-secton comparson of regons or a tme-seres comparson of subsequent cohorts would not produce relable estmates for the effect of the comprehensve school reform. Smlar tabulatons are shown for the three subtests n Appendx 1. [Table 1: MEAN SCORE BY COHORT & REGION] 7 As a robustness check we ncluded the movers and determned the reform year based on the place of resdence n the 1975 census. Ths made practcally no dfference for the results. 11

14 5. Estmaton methods Our goal s to estmate the causal effect of the school regme on the Army test scores. That s, to determne how an average student, or a student wth certan characterstcs, would have fared, had she or he been assgned to the reformed comprehensve system nstead of the prevous selectve early trackng system. A fxed effects approach s used to control for regonal dfferences as well as general trends over tme. The effect of the comprehensve school reform s dentfed because the tmng of the reform dffers across regons. Most of the estmates are based on the followng regresson model: y D j Dt A C j t (1) ( ) ( ) ( ) ( ) where y s the army test score of ndvdual, attendng school n regon j and belongng to cohort t. D j and D t are regon and cohort specfc dummes and s the age at test. C j()t() s an ndcator, varyng across cohorts and regons, that pupl attended comprehensve school. The parameter of nterest n (1) s. The dentfyng assumpton s that the comprehensve school ndcator, C j()t(), s uncorrelated wth the error term condtonal on the other regressors. Ths assumpton, and the fact that D j and D t enter (1) addtvely, reflect the basc dfferencesn-dfferences assumptons. Note, n partcular, that we make no assumptons regardng the smlarty of the regons where the reform took place early to those where reform took place later, nor do we clam that reform dates were randomly assgned. The parameter s an unbased estmate of the average causal effect of comprehensve schoolng, f the tmng of the reform s uncorrelated wth other regon-specfc changes n student outcomes. It s mportant to notce that regresson (1) controls for the regonal dfferences wth sx mplementaton regon dummes but not for the full set of more than fve hundred muncpal fxed effects. The man reason s that we have no access to the muncpalty codes due to data protecton regulatons. However, the only reason to nclude muncpalty dummes n regresson (1) would be the concern that the reform took place earler n non-randomly selected muncpaltes. But ths s only a problem f the reform dummy s correlated wth the muncpalty fxed effects. Ths correlaton s fully absorbed by ntroducng the sx mplementaton regon fxed effects, snce wthn these regons the mplementaton year does not vary across muncpaltes. 12

15 The fact that we only have 5 (cohorts) x 6 (regons) = 30 observatons for dentfcaton purposes obvously has mplcatons for statstcal nference. We deal wth ths by clusterng the standard errors at regon level usng the Moulton-procedure programmed for Stata by Angrst and Pschke (2009). Followng Cameron, Mller, and Gelbach (2007), we also make a small sample correcton for the standard errors by magnfyng the resduals by G /( G 1), where G s the number of regons, and use crtcal values from a t-dstrbuton wth G-2 degrees of freedom. 8 To facltate the nterpretaton of the statstcal sgnfcance of the results wth non-standard crtcal values, we report the 95% confdence ntervals nstead of standard errors of our regresson coeffcents n the tables. We also estmate (1) by nteractng C j()t() wth parental educaton and ncome. These results are nformatve n evaluatng whether the reform was partcularly successful n mprovng the cogntve sklls of students from dsadvantaged famly backgrounds relatve to other students. Furthermore, we also evaluate the effect of the reform on the varance of the test scores. A straghtforward approach for examnng ths s to model smultaneously the effect of the reform on both the mean and the varance of the test scores. Assumng that the error term follows a normal dstrbuton, the test scores wll be dstrbuted as y ~ j( ) t( ) exp 1 2 y ( D j( ) D t( ) 2 j( ) t( ) A C j()t() ) 2 (2) 2 The subscrpts n jt ndcate that the varance n the test scores may vary across regons and cohorts and may therefore be affected by the reform. The model s parameterzed by assumng that log-varance s an addtve functon of the regon, cohort and reform dummes. 8 We were also concerned that seral correlaton wthn regons could lead to a bas n the standard errors. However, calculatng mean resduals by regon and cohort from equaton 1 and regressng these mean resduals on the lagged resduals ndcates that the frst order autocorrelatons of the resduals n dfferent test tems are between -.09 and.11. These estmates are consstent wth the null of no autocorrelaton accordng to an autocorrelaton test suggested by Wooldrdge (2002),.e. regressng the frst dfferenced resduals on ther lagged values and testng whether the resultng coeffcent s equal to -.5. As a fnal check we also estmated the confdence ntervals usng a wld bootstrap-t procedure that retans the cluster structure n the data and, accordng to Monte Carlo results by Cameron et al. (2008), performs well even wth only 6 clusters. In general, the wdth of the confdence ntervals does not vary much across the dfferent cluster correcton procedures, nor are the cluster corrected standard errors very dfferent from the OLS standard errors once cohort and regon fxed effects are ncluded n the model. We report alternatve estmates of confdence ntervals of key parameters n Appendx 2, but our man concluson s that, gven the lack of sgnfcant autocorrelaton n the data, the choce of cluster correcton method s not very mportant. Ths fndng s consstent wth the Monte Carlo experments n Bertrand et al. (2004). 13

16 14 ) exp( 2 jt t j jt C D D (3) The log-lkelhood functon of the normal heteroskedastc model s N t j t j t j t j N t j t j C A D D C A D D y C A D D N L 1 ) ( ) ( ) ( ) ( 2 ) ( ) ( ) ( ) ( 1 ) ( ) ( ) ( ) ( exp ) ln( 2 2 ln (4) where measures the effect of the reform on the mean score and ts effect on the varance. 6. Results 6.1 Average effects The baselne results are reported n Table 2. To facltate the quanttatve nterpretaton of the results, the test scores are converted nto standard devaton unts. Column (1) smply regresses the average test score on the comprehensve school dummy, and shows that those who attended comprehensve school scored on average standard devatons lower n the Army test. However, the results n column (2) reveal that ths negatve correlaton reflects the fact that regons wth on-average lower test scores mplemented the reform frst. When full sets of brth cohort and regon dummes are ncluded n the regresson, ths negatve correlaton s removed and we fal to fnd any sgnfcant effect of comprehensve school on average test scores. The causal nterpretaton of the result n column (2) of table 2 reles on the standard dfference-n-dfferences assumpton that the changes n test scores n the reform regons would have been smlar to the changes n the control regons n the absence of the reform. Gven that the panel spans several perods, ths assumpton can be relaxed somewhat by addng regon-specfc lnear trends n test scores as we do n column (3). The effect of the reform s now postve and close to beng statstcally sgnfcant at 5% level but very small at standard devaton unts.

17 Column (4) adds famly fxed effects to the equaton, thus dentfyng the effect of the reform from the dfferences between brothers that attended dfferent school systems. The estmates become mprecse but are stll very close to those reported n column (3). Interestngly, addng famly fxed effects also reverses the postve trend n the test scores, ndcatng that the brth order effect on the test scores s larger than the dfference across the brth cohorts. 9 [Table 2: BASIC RESULTS] Table 3 examnes the effect of the school reform on dfferent tests n turn. Column (1) regresses each test score separately on the regon and cohort dummes and a dummy varable ndcatng whether the person attended a comprehensve school. Column (2) agan adds separate regon-specfc lnear trends and column (3) controls for the famly fxed effects. For brevty, only the coeffcents of the comprehensve school dummy are reported n each column. The comprehensve school reform had no sgnfcant effects on ether math or logcal reasonng tests. The effect on the verbal ablty test s postve and sgnfcant f regonal trends are ncluded n the model. The sze of the effect on verbal test scores ranges between and standard devaton unts. Famly fxed effect estmates tend to be much less precse than the estmates that explot between-famly varaton, and are therefore never sgnfcantly dfferent from zero or sgnfcantly dfferent from the pont estmates reported n columns (1) and (2). The fndng that the comprehensve school reform had ts largest effects on the verbal test was perhaps to be expected. After all, verbal sklls are learned n schools, and hence the changes n school system may have effects on these sklls. If ndeed the logcal reasonng test truly measures nnate reasonng abltes, pre-test schoolng should have lttle or no effect on the test. Fnally, the changes n the mathematcs teachng resultng from the reform were perhaps not as sgnfcant. As noted above, the ablty groupng was retaned n mathematcs and, as a result, math classes contnued to be taught at three dfferent ablty levels after the reform. [Table 3: EFFECTS ON DIFFERENT TEST ITEMS] Table 4 reports the maxmum-lkelhood estmates measurng the effects of the reform on both the mean and the log-varance of the test scores. These equatons are estmated separately for each test. All equatons nclude cohort and regon effects, regonal trends and 9 The brth order effect was also found n a Norwegan study of the Army test scores (Krstenssen and Bjerkdal, 2007). 15

18 age-at-test dummes on both the mean and the varance, but only the effects of the comprehensve school reform are reported. The maxmum-lkelhood method produces very smlar estmates for the effect of the reform on the mean scores as the lnear regresson model used n tables 2 and 3. The effects on means are sgnfcant only for the verbal test. The effects on the varance of the test scores are small but generally negatve. In the math test the effect s close to zero. In the verbal and logcal reasonng test the reform reduced the varance about 2.5 percent. None of these effects, however, are statstcally sgnfcant. [Table 4: EFFECTS ON MEAN AND VARIANCE] As shown n tables 2 and 3, the estmated effect of the reform on test scores s somewhat senstve to the ncluson of regon-specfc lnear trends. Ths rases a concern that the standard dfferences-n-dfferences specfcaton may fal to separate out pre-exstng trends from dynamc polcy responses, as ponted out by Wolfers (2006) n a dfferent context. To explore ths possblty, we re-estmated equaton (1) by addng dummes for years two and three pror to the reform and separate dummes for each of the fve years after the reform, thus omttng the dummy for one year pror to the reform. Ths flexble specfcaton should trace out any pre-reform trends and dynamc polcy responses followng the reform. The results are reported n table 5 and a vsual representaton for the average test scores s plotted n fgure 4. There s lttle ndcaton of any pre-reform trends. However, especally the verbal test scores, and to a lesser extent the average test scores, ncrease sgnfcantly n the frst year after the reform and stay at a hgher level or even grow n the years followng the reform. These results suggest that the pattern found n tables 2 and 3 does not reflect pre-exstng trends but rather stems from the true reform effect. [Table 5: THE EFFECT OF THE SCHOOL REFORM ALLOWING FOR LEADS & LAGS] [Fgure 4: LEADS AND LAGS] 6.2 Effects by parental background Tables 6A and 6B examne the effect of the comprehensve school reform on average test scores by famly background. Column (1) n table 6A estmates regresson models smlar to those reported n column (1) of table 3 but adds an ndcator of parents educaton and ts nteracton wth the reform dummy. Parents are classfed as beng hghly educated f at least 16

19 one parent has completed at least 12 years of educaton. In the pre-reform schoolng system ths generally refers to a stuaton where the parent attended the more academc track. In column (2), we add lnear regon-specfc trends to the regresson. Snce the nterest here s n the dfference of the effect of the reform between recruts from hgh- and low-educaton famles, we can control for regonal trends n a more flexble way n these regressons. In column (3), we add nteractons of brth cohorts and regonal dummes thus controllng also for any non-lnear regonal trends. After addng these nteractons the man effect of the reform on test scores s no longer dentfed, but the dfference of effect between recruts from hgh and low educated famles stll s. Fnally, n column (4), we further ntroduce the full nteractons of parental educaton wth cohort and mplementaton regon fxed effects as well as wth the test age dummes. Accordng to table 6A, parental educaton has a clear effect on test scores. Men wth hghly educated parents score, on average, standard devatons hgher. The effect of the reform, now referrng to the effect on men wth less educated parents, s postve and larger than n table 2. However, the most remarkable result n table 6A s the negatve and sgnfcant estmate for the nteracton of parental educaton and comprehensve school reform. Accordng to the results the reform had sgnfcantly less effect on men wth more educated parents. Ths result s robust to ncludng regon-specfc lnear trends or full cohort regon nteractons although no longer sgnfcant once full nteractons are ntroduced n column (4). The pont estmates n table 6A ndcate that the reform ncreased the test scores of men from low-educaton famles by standard devaton unts and but lttle or no effect on men from hgh-educaton famles. The results are qualtatvely smlar for ndvdual tests (not reported n table) and agan the effect s strongest n the case of verbal test scores where the reform ncreased the test scores of recruts from low educated famles by 0.06 standard devaton unts. Ths effect s szeable, amountng to a quarter of the effect of parental educaton. In table 6B we repeat the same analyss now usng parents ncome as the measure of famly background. The parents ncome s measured by summng the annual taxable ncome of both parents, nflatng the ncomes to the 2002 prce level and takng an average over the census years 1970, -75 and -80. To facltate the nterpretaton of the coeffcents, parental earnngs were normalzed by subtractng the sample mean. Ths demeanng has no mpact on the estmate of the nteracton of the reform and parental earnngs, but t makes t possble to 17

20 nterpret the man effect of the reform n table 6B as the effect of the reform on sons from famles wth sample mean ncome. The results are qualtatvely smlar to those usng parents educaton. Men wth rcher parents tend to have hgher average scores and the nteracton between the parents ncome and the reform dummy s negatve n all models but column (4) where t s not statstcally dfferent from zero. [Tables 6A and 6B: EFFECTS BY FAMILY BACKGROUND] 6.3 Magntude of the effects The results presented above suggest that the comprehensve school reform ncreased verbal test scores by approxmately 0.04 standard devaton unts. Ths may seem lke a small ncrease and t s therefore nstructve to put our results nto perspectve by comparng them to the effect szes found wth other educatonal reforms. The closest comparsons are the varous studes on the effect of trackng. Duflo et al. (2008) report that trackng students wthn schools n Kenya led to an average ncrease of test scores by 0.14 standard devaton unts. Comparng dfferent countres, Hanushek and Wößmann (2006) fnd no effect of early school trackng on mean test scores. Jacob and Ludwg (2008) survey the effect szes of varous educatonal polces targeted at chldren from low ncome famles n the US. The effects of these polces on test scores vary from a hgh of 0.22, related to a large reducton n class sze, to effects as low as 0.03 of polces such as Teach for Amerca. In addton, the polces surveyed by Jacob and Ludwg (2008) typcally target much younger chldren, where we should expect to see larger effects. Hence, n the lght of these results, the effects of the Fnnsh comprehensve school reform are small but not out of lne wth results from other educaton polces, especally gven that the effects here are measured on average four years after the completon of comprehensve school whle other studes tend to focus on more mmedate outcomes. Perhaps the most nterestng result reported above s the consstent postve effect of the reform on the test scores of recruts from low-educaton and low ncome famles. As ths mples a reducton n test score dfferences along socoeconomc lnes, t s nformatve to compare the test score results to the earler estmates of the effect of the comprehensve school reform on the ntergeneratonal ncome elastcty. 18