Equity in Learning? A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES

Transcription

1 Equity in Learning? A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES

2 The opinions expressed in this publication are those of the contributors and do not necessarily reflect the policies or views of UNICEF. The designations employed in this publication and the presentation of the material do not imply on the part of UNICEF the expression of any opinion whatsoever concerning the legal status of any country or territory or of its authorities or the delimitations of its frontiers. Extracts from this publication may be freely reproduced with due acknowledgement using the following reference: UNICEF, Equity in Learning? A Comparative Analysis of the pisa 2009 Results in Central and Eastern Europe and the Commonwealth of Independent States, Geneva: UNICEF Regional Office for Central and Eastern Europe and the Commonwealth of Independent States (CEE/CIS). For further information and to download this or any other publication, please visit the UNICEF CEE/CIS website at All correspondence should be addressed to: UNICEF Regional Office for CEE/CIS Education Section Palais des Nations CH 1211 Geneva 10 Switzerland Copyright: 2013 United Nations Children s Fund (UNICEF) Editing: Stephen Boyle Design: Cover Photo: UNICEF/SWZ/00503

3 Equity in Learning? A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES

4

5 TABLE OF CONTENTS Table of Contents Boxes, Figures and Tables...5 Abbreviations and Acronyms...8 Acknowledgements...9 Foreword...10 Executive Summary...12 Chapter 1: Introduction...20 Chapter 2: Overall Performance...28 Overview of performance...31 Absolute disadvantage...33 Reading literacy...34 Mathematics literacy...35 Science literacy Performance on different reading competencies...37 Key findings on overall performance Chapter 3: Investigating Equity in Performance...44 Within-country disparities in performance...46 Gender differences in performance...49 Relationship between socio-economic background and reading performance...51 Immigrant background and reading performance...54 Resilient students Distribution of resources across schools...57 School location...59 Between-school variance in reading performance and in socio-economic background...60 Key findings on equity in performance Chapter 4: Trends in Performance Over Time...66 Trends in reading since Trends in reading since 2000, 2003 and Trends in mathematics and science...72 Trends in variance in performance...73 Trends in the relationship between socio-economic background and performance

6 Equity in Learning? A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES The case of Poland...75 The case of Turkey...76 The case of Kyrgyzstan...76 Key findings on trends...78 Chapter 5: School- and System-Level Factors Associated WITh Performance...80 Country wealth and student performance Pre-primary school attendance...84 Differentiation: school selection and ability grouping...86 School autonomy School choice...91 Accountability School resources Learning time...94 School climate Key findings on school- and system-level factors Chapter 6: Summary Conclusions and Policy Suggestions Country and system factors associated with student performance Policy challenges: ensuring quality education for all by tackling low performance and reducing large disparities Mitigating the impact of socio-economic background on performance Tailoring policy interventions to meet specific country challenges Achieving high quality with equity in educational outcomes: tradeoffs and realities Specific strategies to tackle low performance in reading literacy Overall conclusion References Annex 1: Overview of Performance in CEE/CIS Countries Annex 2: Tables and Figures

7 Boxes, Figures and Tables Boxes, Figures and Tables Boxes Box 1 Country groupings Box 2 Concepts of literacy in PISA Box 3 Tasks students can typically do at and below the baseline levels of proficiency...33 Box 4 PIRLS and TIMSS studies in learning achievement...40 Figures Figure 1 Percentage of 15-year-old students scoring below Level 2 in reading, PISA Figure 2 Percentage of 15-year-old students scoring below Level 2 in mathematics, PISA Figure 3 Percentage of 15-year-old students scoring below Level 2 in science, PISA Figure 4 Performance difference between the PISA 2009 combined reading scale and each aspect subscale...39 Figure 5 Performance difference between the PISA 2009 combined reading scale and each text format subscale...40 Figure 6 Mean performance vs. 95 th to 5 th percentile difference: average science, reading and mathematics, PISA Figure 7 Performance difference between girls and boys, PISA Figure 8 Score point difference in reading associated with one unit increase in the PISA index of economic, social and cultural status and percentage of variance in student reading performance explained by student socio-economic background...53 Figure 9 Percentage of 15-year-old students scoring below Level 2 in reading, by immigrant background...55 Figure 10 Percentage of resilient students among disadvantaged students...57 Figure 11 Simple correlation between the school mean socio-economic background and school resources Figure 12 Difference in reading score of students in schools from various locations from students in rural schools, after accounting for the PISA index of economic, social and cultural status...59 Figure 13 Between-school variance in student reading performance (and in socio-economic background), as a percentage of total variance between and within schools

8 Equity in Learning? A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES Figure 14 School-level score point difference in reading associated with half a unit increase in the school mean PISA index of economic, social and cultural status (between-school gradient), and student-level score point difference associated with one unit increase in the student-level socio-economic background index (within-school gradient) Figure 15 a) Observed changes in mean reading performance between 2000 and 2009, and changes adjusted for socio-demographic differences b) Changes between 2000 and 2009 of percentage of boys and of girls below Level Figure 16 Annual observed changes in mean reading performance between 2000 and 2009, 2003 and 2009 or 2006 and 2009 (largest gap available); changes adjusted for socio-demographic differences; and linear trends across all available PISA assessments...71 Figure 17 Annual observed changes in mean mathematics and science performance between 2003 and 2009 or 2006 and Figure 18 Change in variance in reading performance between 2000 and 2009 (as a percentage of 2000 variance) and difference in percentage of between-school variance in reading performance ( )...73 Figure 19 Change between 2000 and 2009 in the student-level score point difference in reading performance associated with one unit increase in the PISA index of economic, social and cultural status (overall association); in student-level difference associated with unit increase in student-level ESCS (within-school association); and in school-level difference associated with unit increase in school mean ESCS (between-school association)...74 Figure 20 gdp per capita in equivalent US dollars (converted using purchasing power parities) vs. average mean performance in PISA reading, mathematics and science Figure 21 Percentage of 15-year-old students in PISA having attended pre-primary education for more than one year, one year of less, or not at all (based on student self-reports)...85 Figure 22 Performance difference in reading between students who report having attended pre-primary school for more than one year and those without pre-primary school attendance, before and after accounting for the socio-economic background of students...86 Figure 23 How much autonomy individual schools have over: a) resource allocation; b) curriculum and assessment...90 Figure 24 Difference in performance on the reading scale between public and private schools...92 Figure 25 Comparing countries mean performance in reading, PISA Figure 26 Comparing countries mean performance in mathematics, PISA Figure 27 Comparing countries mean performance in science, PISA Figure 28 Comparing countries range of performance in reading, PISA

9 Boxes, Figures and Tables Figure 29 Comparing countries range of performance in mathematics, PISA Figure 30 Comparing countries range of performance in science, PISA Figure 31 Strength of the socio-economic gradient and reading performance Figure 32 Mean performance in PISA 2009 vs. PIRLS 2006 (4 th grade) Figure 33 Mean performance in PISA 2009 vs. TIMSS 2007 (8 th grade) Figure 34 Mean performance in PISA 2009 vs. TIMSS 2007 (4 th grade) Tables Table 1 Countries participating in PISA 2009, and whether they participated in previous years Table 2 Mean performance in reading, mathematics and science literacy, PISA Table 3 Within-country disparities in performance: difference between 95 th and 5 th percentile in reading, mathematics and science literacy, PISA Table 4 Vertical and horizontal differentiation policies of school systems...88 Table 5 Mean performance in reading, mathematics and science literacy, PISA Table 6 PISA index of economic, social and cultural status (ESCS): Average among all students, within-country disparities (difference between 95 th and 5 th percentile), interquartile range at the student and school levels, average by pre-primary school attendance Table 7 Assessment and accountability practices Table 8 Resources and learning time Table 9 School climate

10 Equity in Learning? A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES Abbreviations and Acronyms CEE/CIS ESCS EU GDP ICT IEA OECD PIRLS PISA TIMSS Central and Eastern Europe and the Commonwealth of Independent States Economic, Social and Cultural Status European Union Gross Domestic Product Information and Communication Technology International Association for the Evaluation of Educational Achievement Organisation for Economic Cooperation and Development Progress in International Reading Literacy Study Programme for International Student Assessment Trends in International Mathematics and Science Study 8

11 Acknowledgements Acknowledgements Equity in Learning? A Comparative Analysis of the PISA 2009 Results in Central and Eastern Europe and the Commonwealth of Independent States (CEE/CIS) was commissioned by the UNICEF Regional Office for CEE/CIS. Giorgina Brown was the lead author of the paper. She was responsible for all data analysis and discussion in chapters 1 through 5 and for assembling the whole report. Aaron Benavot was responsible for writing chapter 6 on Conclusions and Policy suggestions. Special thanks go to both authors. Philippe Testot-Ferry was responsible for the overall design, development and coordination of the study. Erin Tanner provided comments and revisions, and Petronilla Murithi provided valuable administrative assistance. The data came from the Organisation for Economic Cooperation and Development (OECD) Programme for International Student Assessment (PISA). OECD/PISA bears no responsibility for the way data are used or presented. Copy-editing was by Stephen Boyle. Design and layout was by 9

12 Equity in Learning? A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES Foreword As the 2015 target date for achieving the Millennium Development Goals (MDGs) approaches, the numerous debates which are taking place to discuss the post-2015 agenda are increasingly focusing on the issue of the quality of education. With the combined concern of many partners and stakeholders for equity and quality of education, interest has shifted to focus on the results of education processes, in terms of learning outcomes and trends across social groups. These discussions are particularly relevant for countries of Central and Eastern Europe and the Commonwealth of Independent States (CEE/CIS) which, since the transition, have been facing a crisis in education quality. Education systems in the region are in fact generating growing inequalities in learning outcomes. Countries spending the least on education show the worst results; disparities in learning outcomes are wide and increasingly stratified by socio-economic status and gender. Reforms to improve the quality and relevance of education have been initiated but have not penetrated into the classrooms, especially in poor and rural areas; outdated curricula and teaching methods prepare students for memorisation of facts rather than application of the skills which are critical for performance in knowledge economies. Despite the efforts of governments to reform their education systems, about half the adolescents in the region leave basic education without mastering core skills in reading and mathematics, and without the competencies necessary in today s societies. Adolescents from marginalized communities are far more likely to leave school with low levels of learning achievement, less knowledge, and fewer skills and proficiencies than their peers from better-off communities. Inequities in learning and performance eventually lead to dropping out and exclusion from schooling. This can have important consequences for countries, both in terms of economic performance as well as internal stability. This publication, Equity in Learning?, offers a foundation for a better understanding of the factors influencing students achievement as well as for future reflection and action to improve the outcomes of basic education in the region. It uses sound statistical methods to analyse the most recent data on learning achievement from the OECD s Programme for International Student Assessment (PISA), from an equity perspective. The study: 1) Presents data on equity gaps in learning outcomes for reading, mathematics and science among 15-year-old students based on the results of PISA ) Identifies trends over time and across countries, with particular attention to the link between quality and equity. 3) Proposes policy recommendations that, as evidenced by the data, respond to the challenges in reaching high levels of learning achievement. Participation in PISA and/or other international assessments demonstrates countries strong commitment to monitoring and improving learning outcomes and to being open to understanding their challenges in education quality. This study concentrates on the 13 participating countries which had, in 2009, a Programme of Cooperation in education with UNICEF: Albania, Azerbaijan, Bulgaria, Croatia, Georgia, Kazakhstan, Kyrgyzstan, Moldova, Montenegro, Romania, Russian Federation 1, Serbia and Turkey. For the purpose of comparison, this report also presents results from eight 1 As of 2013, UNICEF and the Russian Federation no longer have a programme of cooperation. However, in this report, data from the Russian Federation is included because at the time of the data analysis UNICEF and the Russian Federation were cooperating in education. 10

13 Foreword countries in the region that became EU member states in 2004: Czech Republic, Estonia, Hungary, Latvia, Lithuania, Poland, Slovakia and Slovenia. This research calls on all countries in the region to: 1) Strengthen and refocus policy efforts to improve equity in the quality and relevance of basic education education systems must improve teacher quality and student engagement in learning. 2) Tackle low student performance with targeted strategies to reduce disparities in learning. 3) Ensure full and equitable implementation of education policies. We hope that this report will contribute to a better understanding of students academic achievement in the CEE/CIS region. We also hope that it will serve as a basis for triggering national dialogue on the importance of monitoring, assessing and analysing learning outcomes and their trends across social groups, and will inspire initiatives to reduce equity gaps in learning. Marie-Pierre Poirier Regional Director for CEE/CIS UNICEF 11

14 Equity in Learning? A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES Executive Summary This UNICEF study examines educational outcomes in the 13 participating countries in the CEE/ CIS region which had, in 2009, a Programme of Cooperation in education with UNICEF: Albania, Azerbaijan, Bulgaria, Croatia, Georgia, Kazakhstan, Kyrgyzstan, Moldova, Montenegro, Romania, the Russian Federation 2, Serbia and Turkey. It focuses on new data which have become available since the publication in 2009 of the study Learning Achievement in the CEE/CIS Region, which was commissioned by the UNICEF Regional Office for CEE/CIS. In particular, it examines the most recent results of the Programme for International Student Assessment a large-scale international assessment conducted by OECD in 74 countries. For the purpose of comparison, results of eight countries that became member states of the European Union (EU) in 2004 the Czech Republic, Estonia, Hungary, Latvia, Lithuania, Poland, Slovakia and Slovenia (referred to as the EU8 countries) are also presented (see country grouping in Box 1). The last round of PISA was undertaken in 2009 and 2010 in order to assess students preparedness for adult life as they near the end of secondary education; it focused on reading literacy, but also provided performance results on mathematics and science literacy. The term literacy is used in PISA to point to its broad approach in measuring knowledge and skills focusing on ability to use these in novel situations. This report starts by describing overall performance levels in the 21 study focus countries (CEE/ CIS and EU8 countries) in reading, mathematics and science in PISA It looks also at average literacy scores on different reading competencies, and at the percentage of students who are at a disadvantage in absolute terms, being below the baseline level of achievement identified by PISA (defined as the common international benchmark Level 2). The PISA results are compared with two other large ongoing surveys of learning achievement: the Trends in International Mathematics and Science Study (TIMSS) and the Progress in International Reading Literacy Study (PIRLS). In order to investigate equity in performance, the report looks at the extent of overall within-country disparities in achievement levels across the 15-year-old student population, in terms of the size of the gap between top achievers and the lowest achievers (relative disadvantage). It then examines inequality in performance in various sub-national populations (by sex, socio-economic background, immigrant status and school location) and inequality in the distribution of resources across schools. It also looks at the percentage of resilient students in each country, who despite their disadvantaged background manage to achieve high levels of performance, and examines the extent to which variability in performance and in socio-economic background is concentrated between (rather than within) schools, resulting in academic and social exclusion. The report continues with an analysis of trends in overall performance, adjusting for demographic changes, in within-country disparities, and in the relationship between socio-economic background and reading performance. It then looks at school- and system-level factors affecting performance, presenting the context by looking at the level of national wealth available for educational expenditure and its relationship to performance. The factors described relate mainly to the following: preprimary school attendance; school selection and ability grouping; school autonomy; school choice; accountability; school resources; learning time and school climate. The report concludes with a summary and policy recommendations. The following are some of the main results: 2 As of 2013, UNICEF and the Russian Federation no longer have a programme of cooperation. However, in this report, data from the Russian Federation is included because at the time of the data analysis UNICEF and the Russian Federation were cooperating in education. 12

15 EXECUTIVE Summary Overview of performance and absolute disadvantage In terms of performance averages in the three literacy domains of science, reading and mathematics, there is a clear separation in ranking among the study focus countries between the EU8 countries, at the top (with averages similar to those of OECD countries), and CEE/CIS countries, at the bottom. In reading, an average of 48 per cent of students in the CEE/CIS countries do not reach the baseline level of achievement, compared with 19 per cent among EU8 and OECD countries. In mathematics, the figure rises to 53 per cent of students in CEE/CIS countries failing to reach the baseline level, compared with 21 per cent in EU8 countries, while in science the number of students failing to reach the baseline figure in CEE/CIS countries is 47 per cent, compared with 15 per cent in EU8 countries. The best-performing country is Estonia, which performs significantly above the OECD average in all three literacy domains. Poland performs above the OECD average in reading and science and at a level with the OECD average in mathematics, while Slovenia is above the OECD average in mathematics and science. Hungary is at a level with the OECD average in all three literacy domains, as is the Czech Republic in mathematics and science, and Slovakia in mathematics. All other country means among the study focus countries are significantly lower than the OECD average. The bestperforming CEE/CIS countries are Croatia (in reading), and Croatia together with the Russian Federation (in mathematics and science), followed by Turkey. Croatia is at a level with EU8 countries Czech Republic, Slovakia and Lithuania in reading, and Slovakia, Lithuania and Latvia in science. At the other extreme, Kyrgyzstan is by far the lowest-performing country among all 74 PISA countries, as was the case in the previous PISA survey (in 2006). In Kyrgyzstan, more than 80 per cent do not reach the baseline level of achievement in the three literacy scales; these 15-year-old students may not be capable of the basic tasks that will enable them to participate effectively and productively in life situations. Other CEE/CIS countries where the majority of students do not reach the baseline level of performance are Albania, Georgia and Kazakhstan (in all three literacy scales), Azerbaijan (in reading and science), Moldova (in reading and mathematics) and Montenegro (in mathematics and science). Looking at performances in different reading competencies, students in the study focus countries seem to be better at obtaining information and understanding the meaning of a text, rather than at reflecting on the implications of its content; the reflect and evaluate aspect seems to be the most problematic for several of the low performers in the region. In PISA 2006, which focused on science literacy, we saw that students in the region were better at explaining phenomena scientifically than at identifying scientific issues or using scientific evidence. Comparing the results of different surveys, CEE/CIS and EU8 countries tend to do better, relative to the OECD countries, in TIMSS and PIRLS than in PISA. One common explanation is that PISA assesses the use and application of knowledge and skills in real-life situations, while TIMSS and PIRLS focus more on measuring the mastery of an internationally agreed formal curriculum. Equity in performance In terms of equity measures and other characteristics measured by PISA, there is no such separation between EU8 and OECD countries, on the one side, and CEE/CIS countries, on the other, as there is for average performance. In the study focus countries, there are generally fewer disparities in performance than in OECD countries, with EU8 and CEE/CIS countries having on average similar disparities. Differences between countries in average performance are small compared with differences in performance within countries. On average, within-country differences in performance in the study focus countries (represented by the gap between the top achievers, at the 95 th percentile, and the lowest achievers, at the 5 th percentile) range from a minimum of about the equivalent of 13

16 Equity in Learning? A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES six years of schooling in Azerbaijan 3 to nine years in Bulgaria. While within-country differences are large in all countries, they are relatively small in some countries compared with others: as well as Azerbaijan, the three countries Estonia, Latvia and Romania also have relatively narrow gaps between high and low performers over the three literacy scales. The example of Estonia reassures us that high absolute standards of performance are not incompatible with low levels of within-country disparities. There are a number of performance inequalities in various sub-national populations. In terms of gender differences, girls have significantly higher average scores in reading than boys in all 74 participating countries. The average gender gap in reading achievement is larger in the study focus countries (48 points) than it is in the OECD (39 points). Azerbaijan has the smallest gap and Albania the largest. For the countries that participated in the survey of digital reading, the gender gap is usually smaller than with print reading. In mathematics, boys have an advantage in seven countries (Montenegro, Croatia, Turkey, Serbia, Hungary, Estonia and Azerbaijan) and girls in four countries (Albania, Georgia, Lithuania and Kyrgyzstan). There is no significant difference in the remaining 10 study focus countries. Gender differences in mathematics and science are much smaller than in reading. In science, differences in average scores of boys and girls in the majority of PISA countries are not statistically significant. However, among the study focus countries there is a tendency for girls to perform better than boys, with girls outperforming boys in 14 of the 21 countries. Overall, girls outperform boys by an average of 10 points across the study focus countries (while the average difference in the OECD region is zero). The socio-economic background of students and schools has generally a strong influence on performance, with most students who perform poorly coming from socio-economically disadvantaged backgrounds. On average in the study focus countries, 13 per cent of differences in reading performance within countries are associated with differences in socio-economic background. A student from a more socio-economically advantaged background (the top one-seventh) scores on average 36 points more than a student with an average background equivalent to almost one year of schooling. Differences range from 21 points in Azerbaijan to 51 points in Bulgaria. Nevertheless, despite such divergence there are always some students who despite their disadvantaged background manage to achieve high levels of performance; the OECD defines these as resilient students. Generally, top-performing countries have also the largest percentage of resilient students. Among the study focus countries, the largest percentage of resilient students is found in Turkey, where 42 per cent of all disadvantaged students (i.e. those from the bottom quarter of the distribution of socio-economic background in their own country) score in the top quarter in reading among students with similar socio-economic background from all countries. While the quantity of resources tends to be more favourable (i.e. there are lower student-teacher ratios) in schools with lower socio-economic intake, in many countries the more disadvantaged students attend schools with lower quality resources (i.e. they have a lower proportion of full-time teachers with advanced university degrees). In general, students in urban schools perform better than students in rural schools, and students in larger towns perform better than those in smaller towns. Taking into account socio-economic background, there are still more than 80 points difference in reading performance (equivalent to about two years of schooling) between students in city schools and rural schools in Bulgaria, Kyrgyzstan and Hungary. Whatever the reason, large between-school differences in what students learn are an indication of inequality in the school system. Countries in which students are not separated in different schools 3 This equates to six times the typical gap in OECD countries between the average reading performance of 15-year-old students in two adjacent grades. 14

17 EXECUTIVE Summary according to socio-economic background also tend to have small between-school differences in performance and high overall performance, as is the case in Estonia and Latvia. On the other hand, the proportion of between-school variance in performance is more than 50 per cent in Romania, Slovenia, Hungary and Turkey, indicating that there is a larger variation in the average performance of schools in these countries than in individual scores within schools 4. In most countries, it is the school s economic, social and cultural status that appears to have a much greater effect on students performance than the individual student s background. Trends in time By monitoring changes in educational outcomes over time which has been possible because PISA is an ongoing triennial survey it emerges that in some cases overall performances have improved and inequalities reduced. Of course, data for each subject area are comparable only from the point in time when the first in-depth assessment took place. Consequently, results in reading are comparable only from 2000, in mathematics from 2003, and in science from Among the eight study focus countries with data on reading since 2000, there was an improvement in Albania, Hungary, Latvia and Poland, a decline in the Czech Republic, and no significant change in Bulgaria, Romania and the Russian Federation. Part of the change can be attributed to changes in sampling methods and the socio-demographic profile of students; by adjusting for such changes, the performance change in the Czech Republic and Hungary becomes non-significant. The percentage performing below the baseline was reduced in this nine-year period by 14 per cent in Albania, 13 per cent in Latvia, and eight per cent in Poland. Because Latvia and Poland raised the performance of their lowest-achieving students while maintaining the performance level among the highestachieving students, variation in reading performance decreased too. The largest decrease in variance was seen in Latvia, with a 39 per cent decrease making Latvia one of the countries with the least within-country disparity followed by Romania (with a 22 per cent decrease) and Poland (20 per cent decrease). In Romania, there was a decline in the performance of high-achievers, with no change for low-achievers. The share of between-school variance in reading performance remained fairly similar in most countries except in Poland, which had been one of the study focus countries with the largest share of between-school variance in 2000 (62 per cent). By 2009, it had the smallest share of all (19 per cent). This change has been attributed to the institutional reform as a result of which 15-yearold students are no longer separated into different types of schools. The relationship between socio-economic background and reading performance diminished in the Czech Republic, Albania and Latvia, while it increased in Romania. The between-school association of socio-economic intake and performance at the school level decreased most in Latvia and Poland, and it also fell in the Russian Federation. For countries that did not participate in PISA 2000, reading performance has improved in Serbia and Turkey since 2003, and in Kyrgyzstan and Montenegro since 2006, while in Slovenia there has been a significant decline since In mathematics, performance has since 2003 increased in Turkey and declined in the Czech Republic. Of countries that did not participate in 2003, Kyrgyzstan and Romania have increased their mean mathematics performance since 2006, while in Lithuania performance has declined. In science, Turkey s and Poland s mean performance has increased since 2003, while in the Czech Republic, Montenegro and Slovenia it has declined. 4 The performance variation that can be attributed to differences in student results in different schools (as by comparing school averages) is larger than the performance variation that can be attributed to differences in the performance of students within schools (that cannot be attributed to differences between schools). So, students within a given school tend to have similar performance, compared with students in different schools. 15

18 Equity in Learning? A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES School- and system-level factors High-performing countries which have a weak relationship between socio-economic background and performance (the OECD s main indicator of equity) share a number of commonalities. The OECD describes them as: a high value is placed on education; there are clear and ambitious standards; there is an emphasis on ensuring the quality of teachers and principals; and high-quality learning is provided consistently to every student. One study focus country, Estonia, is included in the small group of eight school systems deemed to be successful, among the 65 PISA 2009 countries, for a performance above the OECD average and a lower-than-average impact of socio-economic background. Unlike in the richer OECD countries, where the relationship between national income and educational performance is considerably weaker, in the study focus countries there is a close association between student performance and GDP per capita. It seems that money matters below a certain threshold, and indeed the lowest performers in PISA are all countries with lower levels of GDP per capita. Nevertheless, despite certain study focus countries lacking the economic resources to provide sufficient educational opportunities, and the fact that all study focus countries have GDP per capita rates well below the OECD mean, some countries in the region perform significantly better in PISA than the OECD mean. The implication is that once there are enough resources for the basics, it is the way those resources are used that affect educational performance. Some systems are comprehensive, with all 15-year-olds following the same programme, as in Estonia, Latvia and Poland, while others are stratified, with a selection of students streamed into different programmes or schools. Among the eight school systems with above-average performance and below-average socio-economic inequalities in PISA, including Estonia, none show high levels of student differentiation. Early differentiation and selection contributes to between-school inequalities and academic exclusion. Also, in countries where more students repeat grades, or where it is more common to transfer weak or disruptive students out of a school, overall results tend to be worse and socio-economic differences in performance wider. Another important feature of school organisation is the degree of autonomy that schools have in taking decisions on various matters. Overall, across the range of tasks involving resource allocation, principals in the EU8 countries report more autonomy (with 59 per cent reporting sole responsibility of schools) than in OECD countries (45 per cent), with the lowest degree of autonomy reported in the CEE/CIS countries (37 per cent). In general, there is more joint responsibility between schools and higher authorities reported in curriculum and assessment than in resource allocation. The highest levels of autonomy are again found in the EU8 countries (with on average 66 per cent of principals reporting sole responsibility of schools, 27 per cent joint responsibility and seven per cent reporting that responsibility is solely that of higher authorities). This degree of autonomy was higher than the OECD average (where 60 per cent of principals report sole responsibility of schools, 24 per cent report joint responsibility and 16 per cent report responsibility is that of higher authorities only). CEE/CIS countries report the lowest levels of autonomy (36 per cent sole responsibility of schools, 23 per cent joint responsibility and 41 per cent higher authorities only). In PISA, countries that grant greater autonomy to schools to design curricula, decide courses, establish student assessment policies, and determine course content and textbooks used tend to show better performance in reading than those that do not, while there is no correlation at the country level between performance and autonomy in resource allocation. The only type of resource that PISA shows to be correlated with student performance is the level of teachers salaries relative to national income. The OECD observes that raising teacher quality is 16

19 EXECUTIVE Summary more effective in improving student outcomes than is creating smaller classes, noting that systems prioritising teachers pay over smaller classes tend to achieve higher levels of performance. Within countries, socio-economically advantaged schools tend to have more educational resources and tend to perform better, suggesting the need for a more equitable distribution of resources across schools. However, after accounting for socio-economic background, resources do not seem to make a difference. Among all PISA countries, principals in Turkey had the highest perception of problems with instruction due to teacher shortages, followed by Kyrgyzstan and Kazakhstan in CEE/CIS. Kyrgyzstan, followed by Turkey, was the most likely to report that instruction in schools is hindered by a lack of adequate material resources. Learning is disrupted the most by both student and teacher behaviour in Turkey, according to principals, to a degree far above that of any other PISA country. Schools with better disciplinary climates, more positive behaviours among teachers and better teacher-student relations tend to achieve higher scores in reading. Most of this effect is connected to socio-economic background, since more disciplined classes are generally attended by students with advantaged backgrounds (who tend to perform better and may reinforce a climate conducive to learning). But even controlling for socio-economic background, part of the performance advantage remains. The challenge is to weaken this association between background and climate, partly by changing the social mix of students in some schools. Generally, a large proportion of differences are accounted for by socio-economic factors, if not solely then at least jointly with other factors. Nevertheless, there are still factors related to school characteristics which can be affected by public policy that seem to make a difference, both to absolute levels of performance as well as to disparities across the system. If disadvantage becomes established at an early age, given the importance of home background, then attempts to mitigate such disadvantage need to begin before a child even starts compulsory school. Even when socio-economic background is kept constant, students who have attended preprimary education for more than a year tend to show higher performance in reading than those who have not, in all the study focus countries except Estonia, Latvia and Croatia. The average difference is 18 points, while the largest difference is observed in Kyrgyzstan where it corresponds to more than one year of schooling (47 points). Summary conclusions and policy suggestions At present, CEE/CIS countries confront a number of pressing policy challenges in education. First and foremost, there is an acute need to improve the provision of quality education for all students. Among other things, this means establishing school systems that improve teacher quality and effectiveness and enhance student learning experiences and outcomes. Second, countries need to tackle low student performance and find ways to reduce disparities in the knowledge, skills and proficiencies which students obtain after completing a compulsory school cycle. A third challenge involves the loose linkages between educational reforms and policy intentions, on the one hand, and school realities and classroom dynamics, on the other. In many countries the actual implementation of educational policies and intentions in local schools is uneven and partial. Finally, it is vital for CEE/CIS countries to create learning environments which move beyond the rote memorization of facts and teacher-dominated pedagogy, and focus instead on the application of knowledge and skills to new situations, with an emphasis on innovation, creativity, problem-solving and open dialogue. These policy challenges are widespread throughout the region, and are especially pronounced in the school systems serving poor and rural communities. This report makes a convincing case for an essential assertion: public policies in education matter. They have made, and will continue to make, a real difference in the lives of children and youth, and 17

20 Equity in Learning? A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES in the prosperity and social solidarity of countries and communities in the CEE/CIS region. For public policies to foster quality and equity in education and to provide effective learning environments in schools and classrooms, they must be based on the collection and examination of rigorous evidence from different sources. The PISA assessment programme, which systematically explores countryspecific and cross-country relationships in student performance in reading, mathematics and science, provides a strong platform to examine and debate the efficacy of different policy options in education. For many countries in the CEE/CIS region, well designed, methodologically rigorous and transparent studies of student achievement and learning deficits, such as PISA, have been far too rare in the past. In addition, there are several other vital messages that emerge from this report: (1) It is imperative that CEE/CIS countries strengthen and refocus their policy efforts to improve the quality, equity and relevance of basic education. Access to quality basic education is a basic right for each and every child in CEE/CIS. The results and analyses of the 2006 and 2009 PISA assessments provide concrete evidence of which policy options and reforms are more or less likely to bring about concrete changes towards this goal. This report has paid particular attention to policies that improve the performance of low-achieving students and schools; that expand access to pre-primary education, especially for disadvantaged and minority children; that help all students learn to read, and read for enjoyment; and that focus on ways to improve teacher quality and effectiveness. (2) Quality education is not simply a matter of securing adequate inputs to schooling, such as sufficient schools, laboratories, textbooks, computers, instructional time and trained teachers. Quality education is also a matter, and increasingly so in the light of globalization, of the quality of knowledge, competencies, skills and attitudes that children take away from their school experiences. Thus, there is a well-defined need for a shift in how quality education is conceived, measured and monitored in the CEE/CIS region. All policy stakeholders should be involved in supporting student learning and monitoring learning outcomes. Especially useful in this regard are current international assessments such as PISA, PIRLS and TIMMS. Other assessments (regional, national or sub-national) which are appropriate, relevant, transparent and methodologically rigorous would also be useful. (3) Most CEE/CIS countries have made great strides in increasing enrolment and attendance rates in basic education (up to, and sometimes including, upper-secondary education). However, these achievements mask deep-rooted inequalities in the quality of education provided and the benefits for personal development, work opportunities and lifelong learning to young people who have completed their formal schooling. The monitoring of learning deficits and disparities through international assessments is a powerful tool for addressing equity challenges in education and society, especially in the CEE/CIS region, where such inequalities are rampant. 18

21 EXECUTIVE Summary 19

22 CHAPTER 1 INTRODUCTION UNICEF

23

24 Equity in Learning? A COMPARATIVE ANALYSIS OF THE PISA 2009 RESULTS IN CENTRAL AND EASTERN EUROPE AND THE COMMONWEALTH OF INDEPENDENT STATES Chapter 1: Introduction The transition states of Central and Eastern Europe and the Commonwealth of Independent States, which are moving from Soviet education systems to more modern and nationally relevant systems, are confronted with major obstacles to achieving successful outcomes for school quality and learning. Education systems in the region are also confronted with inequalities in learning outcomes, commonly stratified by socio-economic status. Since the publication in 2009 of the study Learning Achievement in the CEE/CIS Region, which was commissioned by the UNICEF Regional Office for CEE/ CIS, new data on educational outcomes in the region have become available. In particular, the most recent results of the Programme for International Student Assessment a large-scale international assessment conducted by the OECD provide an excellent opportunity for analysing not only the quality and relevance of basic education in this region, but also equity issues, including new countries and trends over time. This study is based mainly on results as published by the OECD in the several volumes on PISA 2009 Results 5. PISA is an ongoing triennial survey launched by the OECD in order to assess students preparedness for adult life as they near the end of secondary education, and evaluate the quality, equity and efficiency of school systems 6. PISA measures the performance of 15-year-old students in three core competencies: reading, mathematics and science, and has thus far been administered four times, in 2000, 2003, 2006 and In each year, PISA measured students overall performance in the three competencies and conducted an in-depth investigation of students skills in one of the three subjects. In the last run (2009) the main focus was on reading, as it was in PISA s first edition, in 2000; PISA 2003 focused on mathematics, and PISA 2006, on science. In total, 65 countries participated in PISA Ten additional participants administered the same assessments in 2010 (within the PISA 2009 plus project) 7. This study focuses primarily on the 13 participating countries with which UNICEF had a Programme of Cooperation in education in 2009, referred to in the report as CEE/CIS countries : Albania, Azerbaijan, Bulgaria, Croatia, Georgia, Kazakhstan, Kyrgyzstan, Moldova, Montenegro, Romania, the Russian Federation 8, Serbia and Turkey. Compared with the last PISA edition (2006), four more CEE/CIS countries participated in the 2009 survey 9. Kazakhstan participated for the first time in Albania did not participate in 2003 and 2006, but did participate in PISA Georgia and Moldova participated for the first time in 2010 within the PISA 2009 plus project 10. For the purpose of comparison, results of the eight EU countries that became member states of the EU in 2004 the Czech Republic, Estonia, Hungary, Latvia, Lithuania, Poland, Slovakia and Slovenia are also presented ( EU8 countries ). Together, the CEE/CIS and EU8 countries are referred to as study focus countries. The OECD average is used as a benchmark. Since the last round of PISA, four new countries have joined the OECD: Chile, Estonia, Israel and Slovenia (see Box 1 for the complete list). 5 OECD 2010 vols.i-v. For more information on the PISA survey please refer to this publication (e.g. examples of test questions are included) or to where also the microdata can be downloaded. 6 OECD 2010 vol. I, p.3. 7 This makes a total of 74 participating countries, since the data for Dubai (2009) is included in the United Arab Emirates (2009 plus). 8 As of 2013, UNICEF and the Russian Federation no longer have a programme of cooperation. However, in this report, data from the Russian Federation is included because at the time of the data analysis UNICEF and the Russian Federation were cooperating in education. 9 The results of PISA 2006 for the CEE/CIS countries were discussed in UNICEF Results for the PISA 2009 plus participants are contained in Walker 2011, which came out as this report was being finalised. Limited data is available in Walker 2011, compared with the main PISA 2009 volumes published by OECD, which explains why Georgia and Moldova are not always present in the tables and graphs of this report. 22