Central exit examinations increase performance... but take the fun out of mathematics

Transcription

1 SCHUMPETER DISCUSSION PAPERS Central ext examnatons ncrease performance... but take the fun out of mathematcs Hendrk Jürges Kerstn Schneder SDP ISSN by the author

2 Central ext examnatons ncrease performance... but take the fun out of mathematcs Hendrk Jürges & Kerstn Schneder Hendrk Jürges MEA Unverstät Mannhem L13, Mannhem Germany Fax: Emal: Kerstn Schneder Department of Economcs Unverstät Wuppertal Gaußstr Wuppertal Germany Fax: Emal: Revsed verson: August 2008 Abstract: In response to PISA, all German federal states but one have adopted central ext examnatons (CEEs) at the end of all secondary school tracks. Theoretcally, the advantages of CEEs are farly undsputed. CEEs make teachng and learnng output observable and comparable across schools, and provde ncentves for teachers and students to ncrease ther effort. In lne wth earler research, we confrm that CEEs have a postve causal effect on student performance. We also nvestgate what actually drves ths effect. We fnd that the teachers' man reacton to CEEs s to ncrease the amount of homework, and to check and dscuss homework more often. Students report ncreased learnng pressure, whch has szeable negatve effects on student atttudes towards learnng. Students who take central ext exams n mathematcs lke mathematcs less, thnk t s less easy and they are more lkely to fnd t borng. Keywords: Hgh-stakes testng, student achevement, teacher qualty JEL-Codes: I28, H42, D02 1

3 1. Introducton Snce the publcaton of TIMSS and PISA test results, school reform has ganed renewed nterest n the German publc. In partcular the results of PISA have sparked ntense poltcal dscussons about the need to reform the German school system. Part of the dscusson has focused on nsuffcent fnancal resources flowng nto the school system, as exemplfed by repeated complants about too large class szes. Although ncreasng fnancal nputs nto the educaton system mght rase outputs (measured e.g. as average student performance) to some degree, t has to be kept n mnd that the educaton system operates under decreasng margnal returns. In a developed country lke Germany t s at most unclear f the returns are suffcently hgh to warrant a general ncrease n the educaton budget. In fact, estmated effects of school resources on student achevement are often small and sometmes even nconsstent (as exemplfed by the class sze dscusson). Increasng resources alone does not appear to be a very promsng approach, especally when dealng wth a broad target populaton (e.g. Hanushek, 1996, Hoxby, 2000). An alternatve to an nput-orented approach s to change the nsttutonal setup of the school system (or the educaton system n general). From an economst's pont of vew, creatng the rght ncentves for students and schools can ncrease the average performance wth gven fnancal nputs. Thus, changng the nsttutonal setup appears to be a more cost-effcent approach. But whch nsttutons provde the rght ncentves n schools? In general, economsts favor output-orented governance of the publc school system: defne the goals of educaton, gve ncentves for attanment of these goals, and allow schools to choose the approprate means to reach these goals. The man thrust s to ntroduce more competton nto the school system and to develop ndcators that allow comparng the performance of schools. Standardzed tests are a requrement to for rankng schools publcly accordng to ther average performance. German educaton polcy has reacted to the "PISA shock" wth what s sometmes termed a "paradgm shft" (see Kultusmnsterkonferenz 2005): the move from the old nput-orented to a more output-orented governance. One key element of ths new paradgm are natonal performance standards, whch have become mandatory n the school year 2005/2006, and whch defne expected competences and performance levels for students at dfferent ages and n dfferent secondary school tracks (see Secton 2 for a descrpton of the German school system). A closely related ssue that has receved a great deal of attenton s settng common 2

4 standards by establshng central ext examnatons (CEEs) throughout the country. Ths dscusson s of partcular nterest n Germany because federal states that already employed CEEs n the past generally outperformed non-cee states n achevement tests. In response to PISA, all German federal states but one that have not had CEEs have ntroduced central ext examnatons. Further, a group of seven German federal states have recently ntroduced regular standardzed tests of student sklls at dfferent grades n prmary and secondary schools (VERA). The mplct assumpton behnd these polcy changes s that states wth CEEs (or standardzed tests n general) outperform non-cee states because of the benefcal effects of central ext examnatons and not because of some other, omtted varable at the federal state level. The theoretcal lterature almost unanmously shows that CEEs and central standards mprove student performance and mght even rase welfare (Costrell 1997, Effnger & Polborn 1999, Jürges, Rchter & Schneder 2005). Central ext examnatons are purported to functon better as ncentves for students, teachers and schools than decentralzed examnatons (e.g. Bshop 1997, 1999). Students, for example, beneft because the results of CEEs are more valuable sgnals on the job market than the results of non-central examnatons, smply because the former are comparable. Furthermore, students who have to meet an external standard at the end of ther school career have no ncentve to establsh a low-achevement cartel n class, possbly wth the tact consent of the teachers. Much of the exstng emprcal lterature has been devoted to estmatng the effect of CEEs on student performance wthout tryng to fgure out how exactly CEEs work. In ths paper, we am at havng a look nto the black box and study the possble channels through whch central exams rase performance. In partcular, we look at changes n teacher and student effort as well as atttudes towards learnng that mght have benefcal effect on the learnng process. For nstance, teachers whose students have to face standardzed examnatons at the end of secondary school mght adopt more effcent teachng styles but they mght also smply ncrease the students' workload. In ths paper, we focus on the effect of ext exams at the end of lower secondary educaton, where a "natural experment" provded by the German school system helps to nfer the causal effect of CEES on performance and teachng practces. In CEEs, students are generally examned n only one of the two subjects tested n TIMSS, namely mathematcs. We calculate the between-state dfferences n the mathematcs-scence dfferences n test scores, teachng 3

5 practces (as perceved by students and teachers), student behavor and atttudes, as well as teachng careers and teacher atttudes. Under farly weak dentfyng assumptons descrbed below, these dfferences-n-dfferences can be nterpreted as the causal effect of CEEs on outcomes. The paper proceeds as follows: n Secton 2 we descrbe the relevant features of the German school system. Secton 3 gves a schematc overvew of our conceptual framework of the learnng process and Secton 4 explans our dentfcaton strategy n detal. In Secton 5 we gve a bref descrpton of the German TIMSS 1995 data and Secton 6 contans the estmaton results for dfferences-n-dfferences n a large number of educaton process outcomes. Fnally, we draw some conclusons n Secton Insttutonal background We now gve a concse descrpton of the German school system, tryng to emphasze those aspects that are most relevant for understandng central ext examnatons n the German context (a detaled descrpton of the German school system can be found n Jonen and Eckardt (2004)). Fgure 1 gves a stylzed overvew of prmary and secondary educaton n Germany. All chldren n Germany attend prmary school, whch covers grades 1 to 4, or n some states grades 1 to 6. There s no formal ext examnaton at the end of prmary schoolng. Rather, students are generally allocated to one of the three secondary school types on the bass of ther ablty and performance n prmary school (see Jürges & Schneder, 2007, for an emprcal analyss of bases n the secondary school track allocaton process). --- Fgure 1 about here --- Hauptschule, Realschule, and Gymnasum are the three man types of secondary school; each leadng to a specfc leavng certfcate. The Hauptschule provdes ts students wth basc general educaton, and usually comprses grades 5 to 9 (or 10 n some states). The Realschule provdes a more extensve general educaton, usually comprsng grades 5 to 10. The Gymnasum provdes an n-depth general educaton coverng both lower and upper secondary level, and usually comprses grades 5 to 13 (or 12 n some former GDR states). Dependng on 4

6 ther academc performance, students can at least theoretcally swtch between school types. 1 At the end of lower secondary level, Hauptschule and Realschule students who complete grade 9 or 10 successfully are awarded a leavng certfcate. They are only requred to take central ext examnatons n some states (Table 1 descrbes the stuaton n 1995, the year n whch the TIMSS data were collected). Sx states had central ext examnatons at the end of Realschule, and only four had them at the end of Hauptschule. 2 Students leavng Hauptschule and Realschule usually embark on vocatonal tranng n the "dual" system, so called because t combnes part-tme educaton n a vocatonal school wth on-the-job tranng wth a prvate or publc sector employer. Gymnasum students are not ssued a leavng certfcate after completng lower secondary level, but are admtted to the upper level of secondary educaton, whch eventually leads to a unversty-entrance dploma (Abtur). Central ext examnatons are most common at the end of upper secondary educaton. However, as Table 1 shows, decentralzed systems of ext examnatons at the end of upper secondary educaton exst as well. In the absence of central exst examnatons teachers devse exams on ther own, subject to the approval of the school supervsory authorty. --- Table 1 about here --- German ext examnatons never cover all of the subjects taught at school. For the Abtur,, students can choose four or fve subjects (the choce s lmted and vares from state to state). Ths leads to self-selecton problems, whch are unlkely to be solved convncngly wth the avalable TIMSS data. At Hauptschule and Realschule, German and mathematcs are always tested n the ext examnatons,.e., mathematcs s compulsory for all students n these two school types takng ext examnatons. In order to assess the causal effect of CEEs, we wll thus concentrate on the mathematcs performance, teachng practces, student behavor and atttudes n Hauptschule and Realschule as the man outcome varables to be affected by CEEs. 1 A fourth type of school, the Gesamtschule (comprehensve school), does not appear n our fgures. Ths type of secondary school offers all lower secondary level leavng certfcates, as well as provdng upper secondary educaton. It only plays a mnor role n most federal states wth less than 10 percent of all students n grade 8 attendng a comprehensve school. 2 As mentoned n the ntroducton, CEEs have now been ntroduced n Saarland (2001), Hamburg (2005), Brandenburg (2005), Hesse (2006), Lower Saxony (2006), Berln (2007), North Rhne-Westphala (2007), Bremen (2007), Schleswg-Holsten (2008). 5

7 3. Conceptual framework A stylzed conceptual model of the educaton process underlyng our study s shown n Fgure 2. Student achevement s typcally vewed as the man outcome of the educaton process and educaton polcy s often evaluated based on ths outcome only. What s often mssng s an analyss of how educaton polcy and nsttutons are affectng the process of teachng and learnng. In the case of CEEs, the earler emprcal lterature has manly analyzed the effect of CEEs on student achevement and has tred to dentfy the causal effect of external exams. Theoretcal models, however, also consder the channels through whch CEEs work n more detal. For nstance, CEEs are thought to affect students' and teachers' effort and thereby mprove student achevement. But clearly, rasng effort s costly for students and teachers, as more effort negatvely affects utlty. Ths could result n a more negatve atttude of students towards school. However, a better knowledge of for nstance mathematcs mght well ncrease the student s nterest n mathematcs and result n a more postve atttude. And even teachers mght fnd t more enjoyable to teach more motvated students. If ths s the case, CEEs promse to grant a free lunch. In ths paper, we try to fnd out how CEEs affect effort, motvaton and atttudes of students and teachers, and hope to shed more lght on the costs and benefts of CEEs. --- Fgure 2 about here --- Whle t s wdely accepted that the man determnant of ndvdual educatonal success s parental background, the nfluence of the parents, e.g. the electorate, on nsttutons s of mportance as well. We ndcate ths by the (dashed) arrow from parental background to nsttutons. For estmatng the causal effect of CEEs, ths consttutes an mportant potental source of endogenety of nsttutons, whch we dscuss n the next secton. 4. Identfcaton The most basc approach to dentfy the effect of CEEs on any outcome would be to estmate smple dfferences between average outcomes n CEE states and non-cee states, controllng for student background and other varables of nterest. Smple dfferences n outcomes across CEE and non-cee states are of lmted value, however, because they gnore two potentally confoundng effects: a composton effect and endogenety of CEEs. The frst problem, the 6

8 composton effect, stems from the fact that n CEE states more students attend Haupt- and Realschule and fewer students attend Gymnasum than n non-cee states. Snce students are selected nto secondary schools manly on the bass of ther achevement n prmary school, student achevement n CEE states (condtonal on school type) wll be hgher smply because there are, on average, relatvely more able students n each type of school. We wll use nformaton on the proporton of students n each school type to account for ths knd of composton effect. Dfferent compostons of the student body n German secondary schools across states are nterpreted as the result of dfferent crtcal ablty levels α chosen to sort students. As a proxy for α, we wll use 1 Φ (1 a), the a percent quantle of the standard normal dstrbuton, where a s the proporton of 8 th grade students asprng to a hgh school dploma (see Table 1). Besdes the dffcultes due to a composton effect, the attempt to estmate the effect of CEE s subject to the fundamental problem of causal nference, namely that t s mpossble to observe the ndvdual treatment effect (Holland, 1986). One cannot observe the same teacher or student at the same tme as beng teacher or student n a state wth and wthout CEE. Only f selecton nto treatment s purely random, ths poses no problem. However, self-selecton nto treatment s one of the most frequent problems encountered by researchers tryng to evaluate the causal effects of polcy measures. In our context, ths can happen f parents vote wth ther feet and move to another state n order to send ther chldren to schools wth a central ext examnaton (or to avod CEEs). Parents n non-cee states who lve near a CEEstate may choose to send ther chldren to school n the neghborng state. However, ths wll not apply to many parents. In the short run, the treatment status mght be consdered exogenous, gven the nsttutonal arrangement n each state. In the long run, however, nsttutons can change and affect all parents. But clearly, not only parents can vote wth ther feet; teachers mght well be more moble than parents when decdng where to work. However, the between state-moblty of teachers, who are mostly state cvl servants, s rather lmted. As an example, consder the moblty between Bavara (one of the large southern CEE-states) and the rest of Germany (see Table 2). In 2001 a total of 102 teachers appled to be transferred from a non-bavaran school to a Bavaran school. Only 22 teachers were granted the transfer. The number of teachers from Bavara who appled to be transferred to another German state was even smaller (38). Moreover, the observed moblty has been manly between Bavara and neghborng Baden-Württemberg, whch s another large CEE state. 7

9 --- Table 2 about here --- Even f moblty of parents and teachers s low, the exstence of CEEs mght reflect unobserved varables such as the mportance attached to educaton by the electorate of a partcular state,.e., parental atttudes towards educaton and achevement n school (see the dashed arrow n Fgure 2). If CEEs are correlated wth such atttudes, smple dfferences between CEE and non-cee states wll be a based measure of the causal CEE effect. Our strategy to solate CEE effects from dfferental parental atttudes and other unobserved varables draws on varaton wthn states. As explaned above, the fact that CEEs only apply to a narrow range of subjects offers a source of exogenous varaton that can be used to dentfy the causal effect of CEEs. When mathematcs s a CEE subject but scence s not and f CEEs have a causal effect, the observed outcome dfferences should be larger n mathematcs than n scence. Formally, our estmator can be descrbed as follows (for smplcty, let us assume for a moment that all outcomes are measured contnuously). Consder two regressons: one to explan outcomes related to mathematcs denotes the student) m y and measured at the student level (the ndex y m = µ + X β + C δ + ε m, (1) and another to explan outcomes related to scence s y y s = µ + X γ + ε s, (2) where s some student-specfc characterstc (e.g. general ablty), X s a vector of µ covarates that mght affect mathematcs and scence outcomes dfferently, C s a dummy varable for central exams n mathematcs, and (2) from (1) yelds k ε, k = m, s are..d. error terms. Subtractng d = y m y s = X ( β γ) + C δ + ( ε m ε s ), (3) where δ s the parameter of nterest. The man advantage of ths estmator s that each student serves as her own control group. By takng dfferences, µ s swept out of the regresson. Estmatng (3) allows us to control for a lot of unobserved heterogenety on the ndvdual level, such as general ablty, general atttudes towards learnng and academc success, or soco-economc background. When lookng at "subjectve" outcome varables, the dfference- 8

10 n-dfference estmator has another mportant advantage: t wll sweep out all dfferences between CEE- and non-cee-states that are due to dfferences n survey response styles across both types of states. In order for δ to dentfy the causal effect of CEEs on outcomes, we need dentfyng assumptons, specfcally m s E[ ( ε ε )] = 0. There are several ways n whch ths C assumpton mght be volated, dependng on the outcome varable. For nstance, n the case of student test scores, there could be systematc ndrect effects n the form of spllover from mathematcs (more general sklls) to scence (more specfc knowledge and sklls). Negatve spllovers from mathematcs to scence are also concevable f students dvert resources away from learnng scence to learnng mathematcs because the latter s tested aganst an external, and possbly hgher, standard. If mathematcs teachers also teach scence, spllover can be thought of as teachers transferrng more successful teachng strateges from one subject to another. In the analyss of test scores and student atttudes, the above assumpton can be volated f CEE and non-cee states dffer systematcally n ther relatve preference for mathematcs rather than scence. Also, unobserved student background (e.g. nnate mathematcs and scence sklls) must not dffer between federal states. Usually, one can plausbly assume that such characterstcs are equally dstrbuted across German states. But as was mentoned n the dscusson of the composton effect, we use selectve sub-samples of the student populaton. Mathematcs sklls may be more mportant than scence sklls when students are allocated to secondary school types. If the Gymnasum skms off the students wth the best mathematcs sklls (and mathematcs ablty s not perfectly correlated wth scence ablty), students n states wth a hgh proporton of students n Hauptschule and Realschule (hgh α, see above) may have better mathematcs sklls than ther peers n low-α states, but comparable scence sklls. Fnally, t s also mportant that mathematcs and scence outcomes are comparable. The deal comparson subject for our dfference-n-dfference-strategy s one (a) for whch we have TIMSS test scores and a lot of ancllary nformaton about the teachng process (b) that s "unrelated" to mathematcs to avod problems of knowledge spllover or of teachers teachng both subjects, and (c) that s not tested n central exams. Unfortunately, ths subject does not exst. The subject that n our opnon comes closest to meetng all three requrements smultaneously s bology. However, bology test scores and separate nformaton on hours spent learnng bology at home are not avalable n the TIMSS data. 9

11 Thus we use the correspondng nformaton for scence n general for our comparsons of outcomes across exam types. Jürges, Schneder & Büchel (2005) gve a detaled dscusson of the plausblty of our dentfyng assumptons wth respect to student achevement as the outcome varable. They argue that spllover from good mathematcs sklls to good performance n the TIMSS scence test s lkely to be very small, because of the 87 (released) scence tems, only four requre mathematcs sklls, such as dvdng by a fracton (see IEA TIMSS 1998). Negatve spllover s lkely, so that strctly speakng, we are only able to measure the sze effect of a partal ntroducton of CEEs (that ncludes the effect of students to dvert tme away from non-tested to tested subjects). Szeable spllover on the teacher level s probably less of a problem. Less than 15 percent of the teachers teach both mathematcs and bology. Relatve preferences for mathematcs versus scence are most lkely to be very smlar n CEE and non-cee states. Mathematcs are a core subject n every state, accountng for roughly one-ffth of teachng tme n prmary schools and about one-seventh of teachng tme n lower secondary schools and there are no sgnfcant dfferences n relatve teachng tme between CEE and non-cee states (Frenck 2001). Fnally, we can account for the possblty of relatve composton or selecton effects by controllng for α n our dfference-n-dfferences framework. 5. Data descrpton The nternatonal data set of TIMSS Germany contans data on a total of th and 8 th grade students and 566 teachers n 137 schools, collected n the 1994/95 school year. Data were collected n 14 of the 16 German states (Baden-Württemberg and Bremen dd not partcpate), and from all major types of secondary schools. However, for reasons explaned above, we consder only the Haupt- and Realschule data. Moreover, we deleted data from Saxony (where both mathematcs and bology are tested centrally) from our sample. 3 Our workng sample conssts of 1,976 students n non CEE-states and 1,219 students n CEE states. In addton to the actual test results n mathematcs and scence, the TIMSS data contan a wde range of context varables on student backgrounds and atttudes, as well as on teachers and schools. 3 As has been ponted out by one referee, Saxony would make good comparson state to corroborate our results f students n Saxony (passng central exams n mathematcs and scence) are compared to those wth CEEs only n mathematcs. The man problem wth ths strategy s that the number of ndependent observatons (classes, not 10

12 Despte the wealth of data avalable, we take a rather parsmonous approach and select a lmted number of control varables for student and school background that have proven to have szeable explanatory power for student achevement. Table 3 contans varable defntons and descrptve statstcs, by the type of ext examnaton, for these varables. Student background, measured n terms of the number of books at home, dffers only slghtly by ext examnaton type the proporton of students wthn each range s very smlar n CEE and non-cee states. There are far more students wth an mmgrant background n the non- CEE group than n the CEE group. Ths s largely attrbutable to the relatvely low rates of mmgraton to eastern Germany, where most states have central ext examnatons (a legacy of the former GDR educaton system). Another major dfference between students n CEE and non-cee states s that n the latter, a larger proporton of students have repeated class at least once. --- Table 3 about here --- Table 4 contans varable descrptve statstcs for our dependent varables. Exact queston wordngs are shown n Table A1 n the Appendx. The most notable dfference between students n states wth and wthout CEEs s ther achevement n mathematcs and scence (scores were standardzed to have a mean of 0 and a varance of 1, dfferences can thus be nterpreted n terms of standard devatons). In mathematcs, students n states wth CEEs score on average nearly 0.6 standard devatons hgher than those n states wthout CEEs. In scence the dfference s somewhat less than 0.5 standard devatons. In both types of states, roughly three quarters of the students agree or agree strongly to the statement that they are usually good n mathematcs or bology. --- Table 4 about here --- There are a number of statstcally sgnfcant dfferences between CEE and non-cee states n teachng practces as reported by the students. For nstance, n mathematcs t appears that CEE students more often copy notes from the board but less often work from textbooks or worksheets on ther own. Teachers also appear to gve homework less often but homework s students) n Saxony s very low (12). However, there s evdence that n Saxony, bology homework s taken more serously relatve to mathematcs homework than n other CEE states, whch supports our hypothess. 11

13 more often checked. Overall, however, the percentage dfferences are relatvely small. In bology, the dfferences are much larger, n partcular wth respect to gvng, checkng, and dscussng homework. Whle about half of the students n non-cee states say that teachers gve, check, and dscuss homework pretty often or always, 25 to 37 percent of the students n CEE states do so. Such large dfferences shed some doubt on the cross-state comparablty of the ordnal response scales such as the one used for these queston. It rather seems as f there s dfferental tem functonng at work,.e. "pretty often" may mean dfferent thngs n absolute terms dependng on whether a student lves n a CEE or a non-cee state. Another noteworthy dfference between students n CEE and non-cee states s ther atttude towards mathematcs. CEE students are consstently less lkely to lke or enjoy mathematcs, or to fnd t an easy subject, but they are more lkely to fnd t borng. Dfferences wth respect to bology are smaller and less often statstcally sgnfcant. Agan, dfferental tem functonng mght be an ssue here. However, for our dfference-n-dfference analyss of causal effects of CEEs, ths s less of a problem as t mght seem at frst. Snce we use ntrastudent varaton n ordnal judgements, the only measurement assumpton we make s that of response consstency,.e. that students use the same response categores n the same way, ndependent of the subject they refer to (mathematcs or bology/scence). Students may dffer n the way they use these answer categores. For example, the constructon of our dependent varables allows that "once n whle" means the same frequency to one student as "pretty often" to another. Indvdual students' response styles may also dffer across questons. For nstance, "pretty often" may mean a dfferent frequency when used wth the statement "We have a quz or test" rather than wth "The teacher gves us homework". We only requre that "pretty often" means the same when used for the same questons related to mathematcs classes and to bology classes. 6. Regresson Results Regresson results are shown n Table 5. We only report the coeffcents for the CEE-dummy varable, whch measure the effect of CEEs on varous dmensons of student achevement, teacher and student behavor and student atttudes. In other words, Table 5 shows the results of 24 regressons wth dfferent dependent varables but the same set of explanatory varables. Besdes CEE, we use the explantory varables descrbed n Table 3 above: the number of books n the student s home, student s sex, grade, and mmgraton background, whether a student repeated class, regon (East/West Germany), type of school (Realschule/Hauptschule), 12

14 and alpha, the varable that reflects the selectvty of the student body n Real/Hauptschule n the respectve federal state. Wth the excepton of class behavor, each dependent varable measures achevement, behavor and atttudes n mathematcs relatve to bology (or n some nstances, scence n general). Thus the CEE coeffcent dentfes dfferences-n-dfferences, as explaned above. Before actually dscussng our results, a note on the nterpretaton of the effects shown for multnomal logt models mght be helpful. All models are 3-category models. The values shown n Table 6 are relatve rsks and ther standard errors (computed wth the delta method). Sgnfcance levels are based on t-tests usng the orgnal logt coeffcents, however. The three categores of the dependent varable are defned n the same generc way. Wth two four category outcome varables, there are 16 dfferent combnatons of answers. Take selfrated performance as an example. The orgnal tems read: "I usually do well n mathematcs" and "I usually do well n bology", respectvely. Students are asked whether they "strongly dsagree", "dsagree", "agree", or "strongly agree" to that statement. We reduce the 16 possble combnatons to three. (1) Agree less to do well n mathematcs than to do well n bology (put dfferently: to state one does worse n mathematcs than n bology) (2) Agree equally to do well n mathematcs and bology (put dfferently: to state one does about equally well n mathematcs and n bology) (3) Agree more to do well mathematcs than to do well n bology (put dfferently: to state one does better n mathematcs than n bology) For example, a student who "dsagreed" to both statements s assgned to the second category, a student who "dsagreed" to the mathematcs tem but "agreed" to the correspondng bology tem s assgned to the frst category, and a student who "strongly agrees" to the math statement but "dsagrees" wth the bology statement s assgned to the thrd category, etc. In the multnomal regressons, the mddle category s always the baselne category. In Table5 we show two relatve rsks: the frst mrrors the effect of central ext examnatons on the probablty of thnkng one does worse n mathematcs than n bology relatve to the probablty of thnkng that one does about equally well n mathematcs than n bology: the second reflects the effect on the probablty of thnkng one does better n mathematcs than n 13

15 bology relatve to the probablty of thnkng that one does about equally well n both subjects. To llustrate, consder the the bvarate relatonshp between central ext exams and the relatve self-evaluaton n mathematcs versus bology shown n Table about here Tables 5 and The relatve rsk of those n CEE-states to judge themselves worse n math than n bology can be computed as: P( Y = 1 CEE = 1) P( Y = 1 CEE = 0) RR ( 1) = = P( Y = 2 CEE = 1) P( Y = 2 CEE = 0) A multnomal regresson of the dfferences n self-ratngs on CEE wthout covarates would yeld exactly the same result. Snce RR(1) s larger than one ths means that students n CEEstates have a hgher rsk of thnkng they do worse n mathematcs than n bology relatve to non-cee students. RR(2) equals 0.829,.e. students n CEE-states have a lower relatve rsk of thnkng they do better n mathematcs than n bology than non-cee students. As a shorthand, we wll smply state that students n CEE states are less lkely to thnk they do well n mathematcs than students n non-cee states, bearng n mnd that ths need not be true n absolute terms but relatve to bology or scence n general. Although the dfference-ndfferences rsk ratos are admttedly a bt cumbersome to nterpret, they have the advantage that we can assume that all ordnal varables are measured on the same scale,.e. we thnk that under farly weak assumptons we have no dfferental tem functonng problem Objectve and self-perceved student achevement We now dscuss our results, startng wth student achevement. TIMSS test scores were rescaled to have mean zero and a standard devaton of one. Thus the dfference n mathematcs scores between CEE and non-cee states s 0.11 standard devatons larger than the same dfference n scence scores. CEE state students thus do relatvely better than non- CEE state students, whch ndcates that there s some causal effect of CEEs on achevement. Comparng ths to the one grade year dfferences n mathematcs scores of 0.28 shows that the effect amounts to a lttle more than one thrd of a school year. 14

16 One potental objecton aganst ths result s that the regresson nclude a small proporton of students (from Bavara ) who wll pass CEEs n scence. Earler studes (Jürges et al. 2005) have addressed ths ssue by excludng students who are lkely to take a central ext examnaton n scence, e.g. those who (a) say they do well n scence or (b) the 40% Hauptschule and 25% Realschule students who do best n scence as measured by ther TIMSS test score. 4 We have tred both sample restrctons. It turns out that mposng the restrctons results n an ncreased estmate of the effect of CEE on test scores. Ths s of course expected as we essentally censor the left-hand sde varable. However, the results for the other dependent varables n ths paper (dscussed below) are very smlar as those n the basc regressons ncludng all Bavaran students. Returnng to the students self-assessment shows that controllng for covarates does only slghtly change the results presented above. Thus despte the fact that the relatve mathematcs performance of students n CEE-states s superor to that of ther peers n non-cee states, studentes themselves appear to thnk the opposte. One explanaton for ths seemngly contradtory fndng s that relatve expectatons are hgher n CEE states, for example because teachers put more pressure on ther students to perform well, knowng that the centralzed exams are due n only one or two years Teachng practces The results for teachng practces (as reported by the students) shows major dfferences between CEE- and non-cee-states for all homework related tems. Rsk ratos smaller than one n the column labeled "RR(1)" ndcate that t s less common n CEE-states to gve less homework n mathematcs than to gve the same amount of homework n mathematcs and bology. Rsk ratos greater than one n the column labeled "(RR2)" ndcate that t s more common n CEE-states to gve more homework n mathematcs than to gve the same amount of homework n mathematcs and bology. Both effects go n the same drecton. It s thus much more common for teachers n CEE-states to gve, check and dscuss homework n class. Moreover, effect szes (relatve rsks) for these tems are substantal. Overall, t seems as f the mportance of homework s the man systematc dfferences between CEE- and non-cee states. Of the other nne tems, only two show sgnfcant 4 In Bavara between 25 percent (Realschule) and 40 percent (Hauptschule) of the students take the central ext examnatons n scence. 15

17 dfferences: how often teachers let students copy notes from the board, and how often they start a new topc by solvng an example. The rsk ratos smaller than one n the "RR(1)" column show that ths s a less common practces n non CEE-states mathematcs lessons than n CEE-states mathematcs lessons Class behavor (student dscplne) The tems concernng the behavor of the students n class can only be analysed n terms of smple dfferences between CEE and non-cee states. Ths s because there are no correspondng tems for scence or bology classes n the data. Hence we use ordered logt models to compare levels of dscplne n mathematcs classes across states. The results do not suggest that there are any systematc dfferences between CEE and non-cee states n student dscplne n math classes. However, dfferental tem funtonng across the two types of states could mask factual dfferences n dscplne. 6.4 Student effort and motvaton TIMSS asked students how much tme they spend outsde school learnng mathematcs and scence. Our results ndcate that students n CEE-states spend relatvely more tme learnng mathematcs at home than ther peers n non-cee states. Two varables that am at capturng the general motvaton for learnng mathematcs and scence are how much students agree to the statement that mathematcs/bology s mportant n everyone s lfe and. whether students would lke a job that nvolves mathematcs/bology. Here we only fnd weak and/or nconsttent relatonshps wth the presence of central ext examnatons. Students n CEEstates have a slghtly lower chance to thnk that mathematcs s mportance n everyone's lfe but the relatonshp s not sgnfcant. There s also a hgher probablty that students n CEEstates want to get a job that nvolves mathematcs rather than bology, but they are also more lkely to want to have t the other way round Student atttudes The fnal set of tems measures the dfference n ndvdual atttudes towards mathematcs and scence. Here we fnd strong and consstent evdence for causal effects of central ext examnatons, and ths evdence clearly ponts nto the drecton that CEEs mpose costs on students. Students n CEE-states are consstently less lkely to lke mathematcs, to enjoy 16

18 dong mathematcs, and to fnd that mathematcs s an easy subject. They are also more lkely to fnd mathematcs borng. Thus despte the better performance, CEE state students have a worse atttude towards mathematcs. 7. Summary and concluson Ths paper studes the costs and benefts of central ext examnatons (CEEs) at the end of lower secondary school n Germany. The theoretcal lterature almost exclusvely focuses on the benefts of central examnatons, whch arse n the form of hgher student achevement. The costs, however, have been neglected so far by the economc lterature. By costs we mean potentally negatve effects on students' and teachers' morale and atttudes towards learnng. The dentfcaton of (postve or negatve) causal effects of CEEs s by no means easy. Cauton s warranted when nterpretng observed dfferences between jursdctons wth and wthout CEEs as the effect of CEEs on student achevement, because CEEs are most lkely the outcome of a poltcal process (reflectng the preferences of the electorate) and thus potentally endogenous. In ths paper, we make use of some unque regonal varaton n Germany that allows us to develop a dfference-n-dfferences dentfcaton strategy to estmate the causal effect of CEEs on academc performance, teachng practces, and student atttudes. In the German school system, only some states have CEEs and these exams are restrcted to core subjects such as German, mathematcs and the frst foregn language (mostly Englsh). We use data from the Thrd Internatonal Mathematcs and Scence Study (TIMSS) 1995 to explot ths nsttutonal varaton and uncover the causal effect of CEEs on student achevement n mathematcs, teachng practces and students' atttudes towards mathematcs by comparng a range of outcome varables across subjects and types of ext examnaton. The fundamental dea s that a CEE affects only mathematcs-related outcomes but not scence-related outcomes. In most of our analyses, we use bology outcomes as our man comparson subject. Bology s almost never examned centrally and there s no mathematcs nvolved n lower secondary bology topcs. There are three man nsghts from ths study. Frst, CEEs have a small but statstcally sgnfcant causal effect on student test scores. Second, teachers n CEE-states are more lkely to gve, check and dscuss homework. Thrd, students n CEE-states do lke mathematcs less, fnd t less easy and fnd t more borng than those n non-cee states. They are also somewhat 17

19 more dlgent n learnng mathematcs at home. We fnd only lttle dfference n (studentreported) teachng practces other than those that are homework-related, lttle dfference n student behavor n class, and lttle dfference n general student motvaton to learn mathematcs. Broadly speakng, ths evdence s consstent wth the vew that that the man effect of CEEs s that teachers ncrease the pressure on students rather than employ more sophstcated or nnovatve teachng methods. However, gvng, checkng, and dscussng homework certanly nvolves also ncreased teacher effort. But all n all, achevement gans n mathematcs appear to result largely from ncreased student effort. One (certanly unntended) consequence s that students n CEE states less often thnk that mathematcs s fun to do. Ths mght actually offset some of the postve achevement effects of CEEs. Workng harder but beng less motvated could be less effcent than workng hard but at the same enjoyng t. 18

20 References: Bshop, John H. (1997). "The effect of natonal standards and currculum-based exams on achevement." Amercan Economc Revew, 87, Bshop, John H. (1999). "Are natonal ext examnatons mportant for educatonal effcency?" Swedsh Economc Polcy Revew, 6, Costrell, Robert M. (1997). "Can educatonal standards rase welfare?" Journal of Publc Economcs, 65, Effnger Matthas R. & Mattas K. Polborn (1999). "A model of vertcally dfferentated educaton." Journal of Economcs, 69, Frenck, Isabella (2001). "Stundentafeln der Prmar- und Sekundarstufe I m Länderverglech - ene emprsche Stude am Bespel der Fächer Deutsch und Mathematk." Unpublshed master thess, Unversty of Essen. Hanushek, Erc A. (1996). "School Resources and Student Performance." In Does Money Matter? The Effect of School Resources on Student Achevement and Adult Success, edted by Gary Burtless. Brookngs Insttuton. Holland, P.W., 1986, "Statstcs and Causal Inference." Journal of the Amercan Statstcal Assocaton 81; Hoxby, Carolne M. (2000). "The effects of class sze on student achevement: New evdence from populaton varaton." Quarterly Journal of Economcs, 115, IEA TIMSS (1998). "TIMSS scence tems: Released set for populaton 2 (seventh and eghth Grades)." [2002, March 1]. Jonen, Gerd & Thomas Eckardt (2006). "The Educaton System n the Federal Republc of Germany 2004." Secretarat of the Standng Conference of the Mnsters of Educaton (KMK). dosser/dosser_en_ebook.pdf [2007, Jan 12]. Jürges, Hendrk & Kerstn Schneder (2004). "Internatonal dfferences n student achevement: An economc perspectve." German Economc Revew, 5, Jürges, Hendrk & Kerstn Schneder (2007). "What can go wrong wll go wrong: Brthday effects and early trackng n the German school system" MEA Dscusson Paper Unversty of Mannhem. Jürges, Hendrk, Wolfram F. Rchter & Kerstn Schneder (2005). "Teacher qualty and ncentves, Theoretcal and emprcal effects of standards on teacher qualty", FnanzArchv 61 (3) 2005, Jürges, Hendrk, Kerstn Schneder & Felx Büchel (2005). "The effect of central ext examnatons on student achevement: Quas-expermental evdence from TIMSS Germany." Journal of the European Economc Assocaton 3 (5) 2005, Kultusmnsterkonferenz (ed.) (2005): Bldungsstandards der Kultusmnsterkonferenz Erläuterungen zur Konzepton und Entwcklung. München/Neuwed: Luchterhand. Wößmann, Ludger (2002). Central Exams Improve Educatonal Performance: Internatonal Evdence." Kel Dscusson Papers 397, Kel Insttute for World Economcs. 19

21 upper secondary Vocatonal track (dual system) (2 to 3 years) Gymnasum (11 th to 12 th /13 th grade) lower secondary Hauptschule (5 th /7 th to 9 th /10 th grade) Realschule (5 th /7 th to 10 th grade) Gymnasum (5 th /7 th to 10 th grade) Prmary school (1 st to 4 th /6 th grade) Fgure 1: The German school system Student achevement Parental background Student effort, motvaton, nterest, and atttudes Teacher effort, teachng practces and atttudes Insttutonal background (CEE) Fgure 2: Conceptual framework 20

22 Tables: Table 1: CEE by federal state and type of certfcate (as of 1995); proporton of students by school/type of certfcate (n 1999) a Hauptschule Realschule Gymnasum (Abtur) 8 th grade students n Hauptschule tracks c 8 th grade students n Realschule tracks d 8 th grade students n Abtur tracks e CEE-States Baden-Württemberg G/M/F/O G/M/F A Bavara G/M/F/S b / G/M/F/S b / A O O Mecklenburg-West Pomerana G/M/F A Saarland A Saxony G/M/F G/M/S A Saxony-Anhalt G/M A Thurnga G/M G/M/F A Average (unweghted) th grade students n other schools Non-CEE states Berln Brandenburg Bremen Hamburg Hesse Lower Saxony North Rhne-Westphala Rhneland-Palatnate Schleswg-Holsten Average (unweghted) Notes: G = German; M = Mathematcs; F = Foregn Language (mostly Englsh); S = Scence; O = Other; A = Any subject chosen for the wrtten exams a Percentages add up to less than 100. Students n specal schools (e.g. for slow learners) are not lsted. b Optonal subject. c Hauptschule students and students n mddle or comprehensve schools asprng to the Hauptschule certfcate. d Realschule students and students n mddle or comprehensve schools asprng to the Realschule certfcate. e Gynnasum students and students n comprehensve schools asprng to the Abtur. 21

23 Table 2: Cross-state moblty of teachers (2001) Applcatons Transfers To Bavara From Bavara To Bavara From Bavara Baden-Württemberg Berln Brandenburg Bremen Hamburg Hesse Lower Saxony Mecklenburg-West Pomerana North Rhne-Westphala Rhneland-Palatnate Saarland Saxony Saxony-Anhalt Schleswg-Holsten Thurnga Source: Bavaran Mnstry of Educaton and Culture webste. Table 3: Descrptve Statstcs, Student background varables Varable non-cee CEE t-test of dfference a Sex (1 = female) Books at home: ** Books at home: Books at home: ** Books at home: Books at home: Immgrant background *** School type (1 = Realschule) Grade (1 = 8 th grade) Repeated grade *** East Germany *** N obs * p<10%; ** p<5%; *** p<1%; a t-values allow for clusterng on the class level 22

24 Non-CEE CEE t-test of Table 4: Descrptve Statstcs, Dependent varables Mathematcs Bology (Scence) Varable Non-CEE CEE t-test of dfference a dfference a Achevement TIMSS test score *** *** Self-rated performance b * Teachng practce Teacher shows how to do problems c ** Copy notes from board c *** Have quz or test c Work from textbooks on our own c *** *** Use everyday thngs n solvng problems c Teacher gves homework c * *** Teacher checks homework c * *** Teacher dscusses homwork c *** Teacher explans rules and defntons c Teacher dscuss practcal problems c Teacher asks what students already know c *** Try to solve examples related to problems c ** ** Class behavour Students neglect school work b Students are orderly and quet b Students do as teacher says b Student effort and motvaton Hours spend studyng subject at home d Subject mportant to everyone's lfe b Lkes to do job that nvolves subject b ** Student atttudes Lke subject e ** Enjoy learnng subject b *** Subject s borng b Subject s easy b ** ** * p<10%; ** p<5%; *** p<1%; a t-values allow for clusterng on the class level (student varables) or the teacher level (teacher varables) b 1 = agree, strongly agree; 0 = dsagree, strongly dsagree c 1 = pretty often, always; 0 = never, once n a whle d 0 = no tme, less than 1 hour, 1-2 hours; 1 = 3-5 hours, more than 5 hours e 1 = lke, lke a lot; 0 = dslke, dslke a lot f 1 = yes; 0 = no g 1 = qute a lot, a great deal; 0 = not at all, a lttle h n percent; j n mnutes 23

25 Table 5: Estmated effects of central ext examnatons on achevement, student behavor and student atttudes Dependent Varable Estmaton method RR (1) (math<bo) Standard error RR (2) (math>bo) Standard error Achevement TIMSS test score OLS 0.110* Self-rated performance b Mult. Logt 1.631*** Teachng practce Teacher shows how to do problems c Mult. Logt Copy notes from board c Mult. Logt 0.635** Have quz or test c Mult. Logt Work from textbooks on our own c Mult. Logt Use everyday thngs n solvng problems c Mult. Logt Teacher gves homework c Mult. Logt *** Teacher checks homework c Mult. Logt 0.558*** *** Teacher dscusses homwork c Mult. Logt 0.640** *** Teacher explans rules and defntons c Mult. Logt Teacher dscuss practcal problems c Mult. Logt Teacher asks what students already know c Mult. Logt Try to solve examples related to problems c Mult. Logt 0.724** Class behavour Students neglect school work b Ord. Logt Students are orderly and quet b Ord. Logt Students do as teacher says b Ord. Logt Student effort and motvaton Hours spend studyng subject at home d Mult. Logt * Subject mportant to everyone's lfe b Mult. Logt Lkes to do job that nvolves subject b Mult. Logt * Student atttudes Lke subject e Mult. Logt 1.868*** Enjoy learnng subject b Mult. Logt 1.813*** * Subject s borng b Mult. Logt ** Subject s easy b Mult. Logt 1.655*** * Notes: Standard errors for odds ratos are computed by the delta method. Sgnfcance levels based on orgnal t-statstcs: * p<10%; ** p<5%; *** p<1% Control varables: number of books at home, mmgrant, sex, school type, repeated grade, grade, East German state, state proporton of students attendng Gymnasum. RR denotes the relatve rsk that the ratng pertanng to mathematcs s larger of smaller then the same ratng pertanng to bology. Table 6: Crosstabulaton of central ext examnaton ndcator and relatve performance self-ratngs (column percentages) Subjectve performance Non-CEE state CEE state 1 (does worse n mathematcs) (does about equally well) (does better n mathematcs) Total