School Value Added Measures in England

Transcription

1 School Value Added Measures in England A paper for the OECD Project on the Development of Value-Added Models in Education Systems Andrew Ray Department for Education and Skills October 2006

2 School Value Added Measures in England Contents Page 1. Introduction 3 2. The objectives and development of value added models 5 3. Data sources for the calculation of value added The value added models The use and presentation of value added models 54 References 67 Annex A The National Curriculum and Key Stage tests 71 Annex B MLM residuals and the shrinkage factor 73 2

3 1. Introduction This paper has been prepared for the OECD Project on the Development of Value- Added Models in Education Systems. It is structured according to the requirements of the OECD group, with an emphasis on a description of the methodology for the value added models currently in use. Although it provides some explanation of the development and objectives of value added modelling, the paper does not discuss in detail the rationale for using value added models or similar data within schools or as part of initiatives to raise standards in attainment. Nor does it cover the wider academic debates concerning approaches to school improvement and school effectiveness and the ways these have affected education policy. The paper discusses the value added models in England it does not cover the other countries of the UK where there are no national value added systems. The focus here is on pupils of compulsory school age (age 4 to 16) and the paper does not discuss value added for older students at colleges. Many of the same issues would apply for measurement of post-16 value added but there are differences in terms of the qualification structure, the available data on contextual factors, the relationships between prior attainment and outcome, the types of institutions where study takes place, the extent of full and part-time study, and differences in the ages at which courses are taken 1. This paper also does not discuss directly the issues in providing value added models for special schools, a type of school that makes provision for pupils with statements of special educational needs. Before discussing value added models, a few background facts are useful in order to put the English school system into an international context: There are approximately 8.2 million pupils in 25,200 state-maintained and independent schools (DfES 2006e) - 7% of pupils were in the independent sector. Some pupils with special educational needs are educated in maintained schools; others are educated separately in special schools. There are about 17,500 primary schools, which generally cover ages 4-11, and about 3,400 secondary schools, normally covering ages (some have sixth forms covering post-16 as well). 1 For more information on current plans see the Learning and Skills Council Site: ult.htm. For an example of post-16 value added analysis, see O Donoghue et al (1997). 3

4 The average size of a secondary school is 980 pupils (approximately 140 pupils per year group on average); primary schools have about 240 pupils on average, 40 per year. Maintained schools are funded through local government: there are 150 Local Authorities covering England. Local Authorities vary considerably in size and characteristics. The smallest is the Scilly Isles with just one school and the largest is Kent with 103 secondary schools and 470 primary schools. The school year runs from September July (dates vary), divided into three terms. Assessment of attainment is made at the end of the school year, so data for the calculation of value added becomes available in the following autumn. This means that, for example, the model shown in Table 4.3 below for 2005 relates to assessments made in summer

5 2. The objectives and development of value added models Summary In England value added models were developed first in various projects for particular groups of schools. The establishment of a national curriculum and new sources of linked national data enabled consistent value added models to be based on pupillevel data. There have been two main phases in the development of value added models, both of which are discussed here: (1) simple value added scores based on prior attainment only; (2) more complex contextualised value added scores based on a range of factors and calculated using multilevel models. In addition to school level scores, value added and pupil progress information more generally has also been used and presented in graphs and tables. Value added modelling is now used: (1) In Performance Tables to provide information to parents and hold schools to account (2) In systems for school improvement, where data is used for self-evaluation and target setting (3) To inform school inspections, which are now tied into the school improvement process (4) To help select schools for particular initiatives (5) To provide information on the effectiveness of particular types of school or policy initiatives This section provides a brief overview of the development of value added models (other accounts can be found in the literature, e.g. Schagen and Hutchison, 2003). The key objectives of the current system are then outlined. 2.1 Local models for school improvement Before the introduction of the National Curriculum (see Annex A) the only examinations sat by all pupils in maintained schools were the GCSEs and similar qualifications at age 16. Although it was possible to benchmark or adjust these results using data about the school 2, there was no information on pupil level 2 See for example Sammons et al (1994) which discussed ways of benchmarking results in the absence of national pupil level data on prior attainment. 5

6 characteristics and no earlier test data with which to calculate value added scores. However, it was possible to provide value added analysis where pupils in a group of schools took specific tests. Some of these analyses were undertaken by academics, such as Mortimer et al (1988) and Goldstein et al (1993), others were carried out by analysts in Local Authorities. They varied in complexity and purpose: some involved the provision of specific feedback to promote school improvement, others were more concerned with developing value added methods and the evidence on school effectiveness. There is a summary of these early studies in SCAA (1994). Another model for local value added analysis in the absence of national data is the specialised centre. For example, starting in the 1980s, the CEM centre at Durham University has provided value added analysis for an increasing number of schools and continues to use its own tests, which are not based on the National Curriculum (Tymms and Coe, 2003). Another example is the National Foundation for Education Research s school improvement work using their QUASE data. Some analysis of the use made by schools of these kinds of value added data is in Saunders (2000). A more recent development has been the Fischer Family Trust s provision of value added analyses to a growing number of Local Authorities 3. This has utilised the National Curriculum test data (which will be discussed below) rather than alternative types of test. 2.2 Developments in school accountability and improvement data In the years leading to the introduction of the first national value added models, several significant changes were made in the English school system. In 1988 the National Curriculum was introduced in England, setting out the subjects and programmes of study which maintained schools are obliged to cover from ages For Key Stages 1-3, covering 5 to 14 year olds, a national system of testing and teacher assessment was established: attainment is assessed against criterionreferenced national curriculum levels at the end of each key stage (see Annex A). The testing system is run by the independent Qualifications and Curriculum Authority (QCA) and National Assessment Agency (NAA). 3 The Fischer Family Trust is an independent, non-profit organisation which, among other projects, provides analyses and data to support the processes of self-evaluation and targetsetting. From a total of 55 authorities in July 2001, the project now covers all LAs in England and Wales. 6

7 In 1992 the Performance Tables 4 for schools were introduced with the aim of informing parents in their choice of school and providing schools with an incentive to raise standards. The first tables showed results in the GCSE exams taken by 16 year olds (along with one indicator for A-levels taken by 18 year olds). In 1996 the first tables for primary schools were produced with results for the new Key Stage 2 tests taken by 11 year olds. Over time the tables have included more indicators, partly as a result of the greater quantity of information available at national level. The first value added scores for all secondary schools were included in 2002, with value added for primary schools following a year later. In the same year that Performance Tables were introduced, school inspection was reformed with the creation of Ofsted (the Office for Standards in Education). Ofsted inspects all maintained schools and Local Authorities in England and its inspectors have access to school attainment data, in the form of the Performance AND Assessment (PANDA) Reports 5. The data in these reports has therefore played an important part in school accountability as they form part of the evidence base used by inspectors to make judgements about school effectiveness. Ofsted s overall inspection reports are published. 6 Schools are graded as Outstanding, Good, Satisfactory or Inadequate; schools in this last category may be put into special measures or given a Notice to Improve. The development of national data on attainment and the ability to link outcomes with prior attainment facilitated the provision of a greater range of information to promote school improvement. Although, as noted above, school improvement analysis could already be provided by Local Authorities and academic institutions, not all schools had access or made use of this kind of data. To fill this gap, the first Autumn Package was produced in 1998, with national patterns of value added figures and statistics for groups of schools, allowing individual schools to benchmark their performance and set targets 7. The Autumn Package supplemented the Ofsted PANDA, which already contained data for specific schools (although no value added measures until they were introduced into Performance Tables). In recent years the Autumn Package has evolved into an interactive software system, called the Pupil 4 Now called the School and College Achievement and Attainment Tables, but for brevity referred to in this paper as Performance Tables. 5 Formerly the Pre-inspection Context and School Indicator (PICSI) Report. 6 Inspection reports can be seen at 7 See DfEE (1998a). Some limited value added information was actually made available one year earlier (QCA, 1998); prior to that there was no possibility of matching pupil level prior attainment to outcomes for the key stages in schools. 7

8 Achievement Tracker 8. In 1997 the government had set in train the development of better pupil-level data and consulted on the introduction of a unique pupil identifier that would help data to be matched throughout the school system. These UPNs were introduced in 1999, following work to consider the practical and data protection issues. Another key development was the move to an annual pupil-level census of schools, which would collect background characteristic data that schools recorded for administrative purposes. The Pupil Level Annual Schools Census (PLASC) was introduced in Underpinning it there had been discussion of a common basic dataset, i.e. a core set of agreed definitions of all variables that would be collected, to ensure consistency. The most recent key developments in the systems that use value added are part of an overarching policy called the New Relationship with Schools (NRwS). One of the aims of NRwS was to improve use of data, taking advantage of the new possibilities opened up by the introduction of UPNs and PLASC. This has led to the introduction of new products and more complex value added analyses. The PANDA and Pupil Achievement Tracker are being merged to create a new software system, RAISEonline, due to be released this autumn. The analysis in RAISEonline, which includes the results of multilevel value added models, will be used by school inspectors and the new School Improvement Partners established under NRwS. Another strand of NRwS is the introduction of School Profiles, which supplement the data in Performance Tables (including value added information) with more background information on schools. 2.3 National value added based on prior attainment In the early 1990s, the new emphasis on performance data to hold schools accountable gave rise to concerns that schools could not be judged fairly in the absence of value added measures. At the same time, the development of the Key Stage tests offered the possibility of calculating value added scores for each school based on progress between each Key Stage, once national data was available for the relevant cohorts of pupils. In 1994 the School Curriculum and Assessment Authority (forerunner of the QCA, the organisation that administers the Key Stage tests) published a report on value added 8 See 8

9 performance indicators (SCAA, 1994). Following this report, the Department for Education clarified in a briefing paper its aims in developing school and college value added measures: (1) to compare these institutions consistently at national level; (2) to allow detailed planning and targeting at local level; (3) to help school inspectors make more informed judgements (DfE, 1995). The Department also stated that value added measures should be based on prior attainment. Including additional socio-economic factors would be a challenge because of the difficulty in measuring them, with a danger that adjustments to value added measures on these grounds which were not sufficiently rigorous could justify poor performance or legitimise low expectations. A further study, the Value Added National Project, was commissioned from researchers at the University of Durham (Fitz-Gibbon, 1997). The remit of this National Project was to advise government on the development of a national system of value-added reporting for schools based on prior attainment, which will be statistically valid and readily understood. It defined the two major applications of such a system as internal school management and public accountability. The National Project concluded that simple statistical approaches were preferable to more complex multi-level models. It reported that headteachers were concerned about publication of value added measures, but that nevertheless a majority wanted to see value added in the Performance Tables. The researchers noted concerns about year-on-year variability in value added measures and the potential difficulty of taking into account pupil mobility, i.e. the movement of pupils between schools during the key stages. Taking the findings of the National Project into account, a simple value added method was piloted in the 1998 Performance Tables, making use of the fact that it was now possible to link Key Stage 4 outcomes to Key Stage 3 prior attainment (DfEE 1998b). This method compared each pupil s expected outcome, based on the national median GCSE result for each level of Key Stage 3 prior attainment, with their actual outcome. Value added scores for schools were then taken as the average of these differences for all their pupils. This method will be referred to here as the median method and its pros and cons are discussed in Section 4. This method for deriving schools scores was consistent with the approach taken in presenting national charts and information on value added in the new Autumn Package (DfEE, 1998a). 9

10 Although there were positive responses to this 1998 pilot, a major problem identified was that in covering only Key Stage 3 to 4 (the last two years of secondary schools), these measures gave an incomplete picture of overall value added. For example, a school could achieve a good Key Stage 3-4 VA score whilst underperforming between Key Stages 2 and 3. The decision was therefore taken not to publish VA scores nationally until the full secondary age range could be covered. In the first year this was possible, 2001, a further pilot of both KS 2-3 and KS 3-4 measures was undertaken, and the following year value added scores for these key stages were published for all schools (with a few exceptions, e.g. some private independent schools). Value added scores for primary schools were calculated using the same method, piloted in 2002 and published in The secondary and primary value added scores have appeared in Performance Tables in each subsequent year and were also used by Ofsted in their PANDA reports. It was eventually possible to match Key Stage 2 directly to Key Stage 4 results and value added scores on this basis were added to the Tables in The move to contextualised value added Whilst the publication of value added measures in the Performance Tables was generally seen as a positive advance on the publication of raw results only, there were some concerns about aspects of the methodology and presentation (see Section 4). At the same time, the developments in linking data, via the unique pupil numbers, and the first Pupil Level Annual Schools Census (PLASC) in 2002, offered scope to reconsider the possibility of including contextual data alongside prior attainment in value added models. Once PLASC data matched to information on pupil progress became available, towards the end of 2002, statisticians in the Department began analysing it to understand the relationships between the variables and what they said about national performance. Views of a selection of academics in the field were sought on the future direction of the value added work and, although there was no consensus of opinion, there was strong support from some for the development of more complex models that used the new data. Outside the Department, statisticians at the National Foundation for Education Research (NFER) and the Fischer Family Trust also began building value added models that took into account contextual factors. On the basis of some of the early work by NFER, the National Audit Office (2003) recommended that performance information should take into account not just prior attainment, but 10

11 also other external influences on performance, based on the new PLASC data. In January 2004 the schools minister made a speech announcing the New Relationship with Schools (Miliband, 2004). He recognised that there is a flourishing debate as to whether we should take account of more than the prior attainment when we calculate the value added by schools and said that over the coming months, we shall be consulting widely as we move towards a model of value added which commands the confidence of all. This consultation and development process has taken the views of schools and Local Authorities throughout the country, as well as further ongoing discussion with academics and between statisticians in the Department and Ofsted. In October 2004 a prototype contextualised value added (CVA) model covering the Key Stage 2-4 age range was discussed with schools. The following year a system of KS2-4 CVA scores was piloted for use in Performance Tables. The model used PLASC data on pupil background factors and a different methodology for calculating school scores: a two-level multi-level model. Ofsted also started to use this pilot CVA model, along with equivalents for other key stages, in their PANDA reports. The model is discussed in Section 4.2 below. Following the pilot some small amendments are being made to the method and a new 2006 contextualised value added model will be used this autumn. The last ten years has been a continued period of transition and this process will continue. The evolution of value added analysis has allowed concepts to be tested and more complex approaches to be introduced after schools have begun to utilise simpler data. The downside of this approach has been the difficulty in maintaining continuity and the way that some aspects of the current system are partly based on earlier approaches (e.g is a transition year for Performance Tables, with CVA used for secondary schools but the earlier VA measure used for primary schools). There continues to be discussion on which models to use for various purposes and how to present the results, for example in the new School Profiles and the new school improvement software system, RAISEonline. 2.4 Objectives of the current system This brief outline has already indicated that there have been several objectives in developing value added analyses and value added scores. The main uses of the current system are summarised below. 11

12 (1) Performance Tables The objectives of the tables remain to provide consistent accessible national data on the performance of schools, to inform parents and the public more generally, and ensure that schools are accountable for their results. The tables are resource intensive to produce accurately every year and are deliberately kept to a limited range of key indicators. They therefore do not, for example, provide results or value added for every subject taken at Key Stage 4. Users are directed to Ofsted inspection reports for a fuller picture of a given school. They are also told that value added measures represent a better estimate of school effectiveness than the raw results that take no account of prior attainment. As noted above, the new School Profiles, aimed at parents as a new source of general information on specific schools, will also include the Performance Tables value added measures. (2) Data for school improvement and inspections RAISEonline will provide a more extensive range of data than the Performance Tables, including value added for a wider range of outcome measures and for subgroups of pupils within the school. The main objective of RAISEonline is to provide all schools with a free software product that allows them to analyse their own data and compare it against national patterns or the results and value added achieved by high performing schools. Schools will use RAISEonline as part of the self-evaluation and target setting process that they undertake with the help of School Improvement Partners. The data will also be available to Ofsted s inspectors for use in judging the extent to which the school is improving or has the capacity to improve. The statistics will not be made available to the public more generally. It remains the case that schools and Local Authorities use a variety of data sources for school improvement in addition to the DfES and Ofsted products, from simple Excel spreadsheets to the analyses of specialists in value added like the Fischer Family Trust (Kirkup et al, 2005). (3) Selection of schools for particular initiatives Although value added is not used directly in funding schools, it has been used as a way of selecting out particular schools, for example, some schools are currently designated as High Performing and given additional responsibility for helping 12

13 weaker local schools or engaging in other projects 9. The data can also be used to look at specific issues, e.g. underperformance in particular subjects, and then targeted for additional support from the National Strategies (consultants employed to undertake activities to raise attainment standards nationally). (4) Monitoring policy initiatives The availability of value added results also provides useful information for monitoring progress in groups of schools subject to specific policies or administrative arrangements. Results of this kind using the linked key stage data were published by the Department in a statistical bulletin (DfES, 2002) and have subsequently been provided regularly as value added scores for groups of schools. External researchers have also used the data to construct regression models showing the relative progress made by pupils in certain types of school (examples are the results in NAO, 2003). However, there are limitations to this approach and it is debatable how far relatively simple cross-sectional results can be seen as robust estimates of the impact of a policy. 9 For more information see the section on High Performing Specialist Schools here: 13

14 3. Data sources for the calculation of value added Summary The value added models use national linked data on tests and various pupil characteristics. Tests are administered by an external body, the QCA, who are responsible for maintaining standards and ensuring that the levels of attainment are correctly specified. Pupil characteristics data are taken each January from all schools in England. The advantages and disadvantages of the data can be summarised as follows: Advantages Near universal coverage of pupils, allowing the data to be used extensively in school accountability and improvement The testing framework has a clear system of education levels and point scores that can be used in measuring value added Contextual information is now collected on every pupil and only a very small proportion of pupils and schools have missing data Pupil postcode data can be matched to a range of local area indicators The system of unique pupil numbers facilitates accurate matching and has led to the creation of a national database which can be accessed by researchers Disadvantages The system of high stakes national testing may be seen as having potential disadvantages in terms of its effects on pupils and teaching Only data on the core subjects are collected at Key Stages 1-3 Some privately educated pupils are excluded Contextual information does not cover all the background factors that would be relevant, e.g. there is no direct measure of social class. There is no means of linking results to teachers or subject departments. This section only covers the data sources used in the two generations of value added models used by the Department. It would in theory be possible to include additional variables from other existing data sources in these models, especially at school level this is discussed further in Section 4. Although there are data on the school as a 14

15 whole, there is no means of results directly to individual teachers or subject departments. 3.1 Test result data This section describes some of the features of the test data and points to some general issues relevant to the development of value added systems more generally: Should there be a national testing system? Who should administer and collect the test data? How reliable are the tests? Is data available for every school? Which pupils results are excluded? For which subjects should data be collected? Should test data be used exclusively or should teacher assessments also be included? How should different attainment levels be scored relative to each other? Does the test data capture any other information (e.g. gender) which could be used in a value added model? Examinations at age 16 have a long history, but the tests at age 7, 11 and 14 are recent developments. The tests themselves measure attainment in the curriculum subjects and do not measure other related but different qualities like aptitude or general intelligence. Without the collection of national data for these age groups it would not be possible to calculate comparable value added scores for all schools. However, it is worth bearing in mind that the tests were set up to provide outcome measures at the end of the different stages and were not designed specifically with the aim of calculating value added scores. The introduction of testing was a major undertaking and one that met with concerns about an over-emphasis on test results at the expense of broader educational aims. There was also concern about the effects of testing on younger pupils. On balance the government has judged the advantages of testing, both within schools and for wider accountability, to outweigh the disadvantages. However, the Welsh National Assembly voted in 2004 to phase out the Key Stage tests in Welsh schools. The English Department for Education and Skills as an organisation does not administer the testing system directly. This is done instead by external agencies, the QCA and NAA, partly to ensure the independence of the system (since rising pass 15

16 rates are used by the Department as a success measure). Information on the processes for developing tests (which, being public, have to be different each year) and on the ways marks are converted into levels can be found on the QCA website ( As part of their remit, the QCA undertake regular analysis of the tests reliability and quality; for example, they provide Cronbach s alpha measures for the Key Stage tests: in 2005 these ranged from 0.86 to 0.92 for Key Stage The NAA also survey schools for their views: for example in 2005, 94% of schools said that the KS2 maths tests were a very or fairly accurate reflection of pupils abilities; the figures for science and English were 95% and 83% (SMSR, 2005). There have been different systems for collecting and publishing the test results at each key stage. Currently the Key Stage 1 assessments are collected from schools by the 150 Local Authorities and then from these by the Department. For Key Stages 2 and 3 there is an external contractor to organise the external marking and collate the results. For Key Stage 4 the awarding bodies (five main ones and many smaller organizations) send data to an external contractor who assembles a database for the Department. Data are collected and published for almost all schools and pupils, including pupils in state maintained special schools (for pupils with special educational needs). The system is such that there would be no advantage in producing results for just a sample of schools. However, coverage of the tests is not universal. At Key Stage 2, for example, many private independent schools do not take the tests and value added scores can only be produced for a subset 11. Rather than include only those independent schools for which data are available, the KS2 Performance Tables excluded all independent schools. There has been a slightly different approach at Key Stage 4, where independent schools have been included and given the choice of whether to have their value added scores published when these are available. As explained in Annex A, test data for Key Stages 1-3 are only collected for the core 10 See the report here QCA s explanation of the Cronbach s alpha measure is that it measures how far the test is measuring a single concept such as spelling or reading or science. If the test were perfect, the result would be If some of the questions are measuring something else (for example if a lot of the items in the mathematics tests demanded high level skills in reading before you could understand what was required) the Cronbach s alpha would be low. In the national curriculum tests, most have coefficients above 0.80, and some over 0.90, so they do appear to be measuring what they claim. 11 In 2005 only about half the independent school pupils had data that could allow value added calculations at Key Stage 2. 16

17 subjects defined in the National Curriculum. Clearly other educational systems might collect data on a different range of subjects. The balance of subjects in the group used for either the input or output measure in a value added model will affect the results, for example more emphasis on reading in a group of subjects would tend to mean higher results by girls for that overall group 12. At Key Stage 4, by contrast, there are no core subjects: all qualifications can contribute to the outcome measure. At Key Stage 4 all the qualifications are externally assessed. However, at Key Stages 2 and 3 national data are collected on both test results and teacher assessments. Either or both could be used in value added models, but so far test data has been preferred because it is externally verified and therefore in theory more robust and consistent. At Key Stage 1, tests were not externally marked and there have been concerns about robustness of the data (see Tymms and Dean, 2004, who quote some head teachers views, although no firm evidence of biases). Since 2005 all Key Stage 1 levels are based on teacher assessment. Whilst this might introduce potential for bias (in contrast to a more objective test), there are good reasons for supposing it will make data more robust (since teacher assessment draws on a range of evidence rather than one test sat by 7 year old pupils). Some of the issues relating to the scoring system can be illustrated in relation to Key Stage 2. Pupils taking Key Stage 2 tests (aged 11) are given marks in each subject. These marks are converted into overall levels by the QCA. At Key Stage 2 there are basically three main levels. Level 4 is seen as the level expected of pupils at age 11 (when the levels were set up, the median pupil was half way through the Level 4s; since then targets have been set to try to increase the number achieving at least a Level 4). Level 5 is a higher level of attainment than Level 4; Level 3 is lower. Table 3.1 shows the current distribution for Mathematics, taken from DfES (2006a). Clearly the main levels cover a broad range of pupils and the system does not distinguish very finely between different pupils (for a discussion of the use of marks data in disaggregating the levels, see Section 4.2). It is also apparent from this table that there is potential for ceiling effects since the maximum level is 5 and some of the 33% of pupils achieving this may have been able to achieve higher levels if these existed. Table 3.1 Key Stage 2 test levels and points 12 In the PISA 2000 and 2003 studies, girls achieved higher marks for reading in all countries: see OECD (2004). 17

18 KS2 test outcome Points Distribution (%) for Mathematics in 2006 Level Level Level Compensatory N (not awarded a test level) 15 2 B (working below the level of the test) 15 4 Disapplied Disregarded 0 Absent Disregarded 1 Table 3.1 also shows the way levels have been assigned a point value (each level worth six additional points), for convenience in calculating averages across the three core subjects (English, mathematics and science). So, for example, a pupil achieving two level 4s and a level 5 would be given ( )/3 = 29 points. Pupils below level 3 all receive 15 points 13. Pupils who were absent or disapplied because they could not access the test 14 are not assigned points and are disregarded from the calculations. However the aim is to disregard as few pupils as possible so, for example, pupils with special educational needs (disabilities or learning difficulties) are included, even though some may not reach Level 3. The average point scores (APS) are published at school level in Performance Tables and have been used as the input measures in the simple median method value added models. The levels at each key stage are aligned by QCA to measure equal levels of attainment across subjects, over time and even across key stages. This means, for example, that a Level 5 achieved in mathematics at Key Stage 2 in 2002 is broadly equivalent to a Level 5 achieved in Science at Key Stage 3 in A system of this kind has numerous practical advantages but the task of maintaining these equivalencies is difficult and there is research that indicates that in some cases there may have been misalignment (Massey et al, 2003). Nevertheless, it is worth bearing in mind that none of these equivalencies is essential for a value added system. As long as all schools have taken the same tests, they can be compared against each 13 Pupils whose marks are relatively close to the Level 3 boundary receive a Level 2, others are given an N. Some pupils (often with special educational needs) are not entered for the test because they are known to be working below the level 3 standard ( B ). All three categories are given 15 points. It could be argued that the Bs and Ns should have lower point scores than 15 - decisions on these detailed aspects of the scoring system will have a small impact on the eventual value added models. 14 An example might be a pupil who is blind but unable to read brail (the tests are available in brail). 18

19 other within a given year. It would of course be possible to move away from the point score system of Table 3.1 or rescale the results (e.g. normalise them) so that, for example, the English and mathematics results have the same distribution within a year. However, the Department s models preserve the Levels used in the tests because they have a concrete meaning in terms of what can be achieved. They are also well understood by schools and parents. Table 3.2 The new point scoring system at Key Stage 4 Qualification Grade Points Qualification Grade Points GCSE Vocational GCSE GCSE Short Course A* 58 D 220 A 52 GNVQ Full M 196 B 46 Intermediate P 160 C 40 U/ X 0 D 34 D 110 E 28 M 98 GNVQ Part 1 F 22 P 80 G 16 U/ X 0 U / X / Q 0 D 136 A*A* 116 GNVQ Full M 112 AA 104 Foundation P 76 BB 92 U/ X 0 CC 80 D 68 DD 68 M 56 GNVQ Part 1 EE 56 P 38 FF 44 U/ X 0 GG 32 U / X / Q 0 A* 29 A 26 B 23 C 20 D 17 E 14 F 11 G 8 U / X / Q 0 At Key Stage 4 (16 year olds) there is a different qualification structure and point score framework. Table 3.2 gives an indication of the complexity of the system at this age: it only includes the main types of qualification but still suggests how wide the range of possible outcomes can be. For example, a pupil gaining an A in one of 19

20 his/her GCSE s would get 52 points for that qualification, whilst a B in a GCSE short course would contribute 23 points. An average point score across all the subjects can be calculated. The table illustrates the need for an elaborate framework of point scores and equivalency rules when all the less common qualifications available are included, as is now the case in England. In addition to the test levels, marks and exam grades, information is also available from the test data on the pupil s name, date of birth and gender. Names may reflect aspects of social background or ethnicity but could not reasonably be used in a value added model. Even though pupils normally sit the key stage tests at a given age, their age within year is correlated with outcomes and so can be used in a value added model (see Section 4). Obviously gender can also be used in value added modelling. At present there are no plans for additional national tests. There are optional tests available for intermediate years, administered by QCA, but these are not taken in all schools. There are also Pscales designed to measure very low levels of attainment for pupils with special educational needs: these are used to inform teacher assessment but are not yet part of the statutory national data collection system. The Foundation Stage Profile is completed by schools for all 5 year olds, but only a 10% sample is collected nationally so value added measures for individual schools could not currently be based on this data. 3.2 The Pupil Level Annual School Census PLASC was introduced in 2002 with the aim of collecting contextual data from schools administrative records on all pupils annually (i.e. not just at the end of each key stage). The main variables in the PLASC data, all of which are used in the contextualised value added model, either directly or transformed in some way, are discussed below. The numbers of pupils falling into some of the groups (ethnic minorities, special educational needs etc.) are shown in the context of Section 4 s discussion of the model (Table 4.4). This section focuses on the key variables and does not discuss more general problems in collecting data for every pupil, e.g. duplicate records or pupils without a Unique Pupil Number. These kinds of error are relatively rare and are not significant enough to be a concern for the production of value added scores. As will be apparent, PLASC has continued to evolve since its introduction in 2002, 20

21 with new variables and new classifications, and it will continue to change, offering new possibilities for value added modelling. This year a termly count has been introduced, although most of the variables here will still only be collected once a year. Data will also be collected on absences and exclusions from school. It would not be appropriate to include these directly in a value added model because although they may well improve the model s explanatory power, schools should to some extent be responsible for these factors. However, data will be available on reason for absence and so it may be possible to include a flag for pupils who have, for example, been sick for a long period. Gender Gender was available before the introduction of PLASC but is now collected as part of PLASC. There are no particular issues with this variable, although given the millions of pupil records collected there are inevitably a few anomalies (0.1% of pupils are recorded as having a different gender in 2006 compared to 2005). Entitlement to Free School Meals Children whose parents receive the social welfare benefit Income Support, and some related benefits, are entitled to claim free school meals (FSM). To become entitled to FSM the parents have to indicate a wish for their child to have a school meal and give a proof of benefit receipt. FSM is the only direct measure on PLASC that relates to the pupil s family income. It is a useful variable, but cannot be considered an accurate proxy for social class or income more generally because it is a simple binary flag. The 85% of pupils who are not entitled to FSM vary considerably in their home background and socio-economic status, and there is also variation in circumstances within the FSM group itself (see for example Table 5.7 in DfES, 2006b). Even as a simple proxy measure for deprivation FSM has disadvantages. The fact that parents have to register an interest in their pupils having school meals may discourage some from applying. Those not applying but eligible may be an unrepresentative group, e.g. choosing not to register for cultural reasons or on the basis of dietary preferences that may correlate with attitudes to education. The benefit rules may also exclude some families who could be considered deprived, e.g. some low paid workers. Clearly a pupil's entitlement to a free meal may change when family circumstances change and this may not always get recorded, although it 21

22 is possible to trace movements in and out of FSM status on PLASC (see DfES (2006b) Appendix D). There is also a specific problem currently with using data from one Authority, Hull, where a healthy eating campaign was introduced in 2004 that gave all primary school pupils, regardless of income, free school meals. More generally, the propensity of parents to register as entitled to FSM may vary in response to the quality of meals, which may vary from school to school or between Local Authorities 15. Ethnicity PLASC collects data for 18 main ethnic groups, with a 19 th code available for unclassified since provision of this data is voluntary. The unclassified group represented 3.4% of the pupils included in the Key Stage 2-4 contextualised value added model. The proportion of unclassified pupils has fallen since the introduction of PLASC, although the current proportion is only a slight reduction on the 2005 figures. The unclassified pupils are not nationally representative they tend to have relatively low attainment. In 2006 there were two schools that declined to return any data on ethnicity 16. Even with 18 ethnic groups, the codes obviously have to cover pupils with very different characteristics. For example, the Black African category covers pupils who may or may not speak English, or who may come from recent or long established immigrant communities. PLASC includes extended codes which Authorities can use, that record, for example, specific African countries of origin. However these extended codes are not used in value added model they are not universally collected and they would add considerably to the complexity of the model. Between 2002 and 2003 the available ethnicity codes changed (see Godfrey, 2004) and although no further changes are planned, it is worth noting that any change of this nature would affect the specification and consistency of value added models. The source of the data on ethnicity could be the school, parents or the pupil themselves: this question of who is best placed to supply the data is something that needs to be considered separately for each variable. There is no firm guidance on ethnicity, although parents are now the main source and would be considered the most reliable supplier for young pupils in particular. In 2006, across all pupils in 15 Recent criticism of the quality of school meals in the media has been cited by some schools as a reason for the fall in their reported proportion with FSM in One of these is a Jewish school which objects to the ethnicity codes because Jewish is not one of the possible codes. 22

23 PLASC in secondary and primary schools, 86% had an ethnicity code provided by their parents and almost 70% of schools had 90%+ of their pupils ethnicity codes supplied by parents. There would be some concerns about the quality of data in the small number of schools (especially primary schools) where the majority of data appears to come from pupils. There are also a few schools that supply all the information themselves rather than asking pupils or parents. Clearly a change in source for a given pupil may result in an ethnicity code changing. For PLASC as a whole, 2.3% of pupils changed ethnic category between 2005 and Ideally we would want ethnicity to be fixed and stable, but some of the changes are actually from the unclassified or other status to a specific ethnic group and these represent improvements in the quality of the data (Table 3.3) Table 3.3 Improvements to ethnicity coding between 2005 and 2006 Percentage of pupils of compulsory school age and over [1] as at January 2005 and January 2006 Matched on Pupil UPN Specific ethnic category Pupils codes in 2005 Other Ethnic category Unclassified ethnic category [2] Pupils codes in 2006 Specific ethnic category Other Ethnic category Unclassified ethnic category [2] Pupil ethnicity is only collected for those pupils aged 5 or over as at 31 August preceding the start of the academic year. 2. Includes both pupils where ethnicity has not been collected and pupils who have refused to give their ethnicity. Special Educational Needs Special Educational Needs (SEN) covers a wide range of needs that are often interrelated as well as specific needs that usually relate to particular types of impairment. Children with SEN will have needs and requirements which may fall into at least of one of four areas: communication and interaction; cognition and learning; behaviour, emotional and social development; and sensory and / or physical needs. The SEN Code of Practice sets out a graduated response to meeting children's SEN: School Action, where the class teacher or SEN Co-ordinator provides interventions that are additional to or different from those provided as part of the school's usual differentiated curriculum and strategies 23

24 School Action Plus, where the intervention at School Action has not resulted in improvement and external advice is sought; A statement of SEN setting out the child s needs, issued by the Local Authority when School Action Plus has not resulted in improvement or the child's needs are particularly complex. Funding for SEN can differ in each Authority depending on the funding formula agreed with their schools. A child with very similar needs may be at School Action Plus in one Authority, but have a statement in another. For the purposes of the Department's modelling, the two interventions which involve external advice and support School Action Plus and a statement of SEN - are combined. More detailed data is collected on the type of primary SEN need for pupils with a statement and those at School Action Plus. The four areas of need are further split to include autistic spectrum disorder, type of learning difficulty, visual and hearing impairment. Pupils identified as having learning difficulties are more likely to be low attainers than those with a physical or other impairment without additional learning needs (DfES, 2005). This detailed data could in theory be used in the modelling but has not so far, partly so as not to complicate the model, but partly because there may be inconsistencies in the reporting of this information across Authorities. In addition there are still about 5% of pupils who are not recorded as having one of the specific primary need categories. In PLASC overall, 91% of pupils had the same SEN status in 2006 as they had in The overall percentage of pupils with SEN has been under 20% for the last 5 years. Where pupils with SEN are educated will be affected by Local Authority changes to their schools. For example, the closure or contraction of a special school (which caters for pupils with a statement of SEN) would lead to more SEN pupils in other local schools both mainstream and special. Conversely, the opening of a special school, or of a special needs unit or resource base within one of the maintained schools, could reduce the number of pupils with SEN in surrounding schools. First Language This data item aims to collect data on the first language of pupils English or other than English (in addition, some pupils are coded as not known but believed to be English or not known but believed to be other than English ). First language is 24

25 defined as the first language to which the child was initially exposed during early development and if they were exposed to more than one language including English, the variable should be coded as English. It does not distinguish different languages other than English. As with ethnicity, there are some small changes year-on-year in this variable, even though it ought to remain constant: in % of pupils in all schools were recorded differently than in Looked-After Children This variable counts pupils who are in the care of their Local Authority. These children may be living with foster parents or prospective adopters, placed in children's homes or some other form of residential care, or placed at home with their parents. It is a relatively small proportion of pupils, but an important group who are relatively likely to be vulnerable and educationally disadvantaged. There have been some concerns that the numbers of these pupils are under-counted in PLASC, partly because schools may not know whether pupils are in the care of their Authority (which may not be the same Authority in which they go to school). In 2007 the plan is to try to merge data from the Local Authority directly onto PLASC, rather than rely on schools to supply this information. This is an example of how the range of contextual factors can be extended by bringing in other administrative data, but it has been necessary to overcome both technical difficulties and data protection issues. Date of entry PLASC records the date of entry into the school for each pupil. This information can be used to obtain measures of pupil mobility: the current method is to flag pupils who joined at non-standard times - any month other than July, August or September. It is also possible to treat pupils differently depending on how recently they joined the school. The precise definitions used in the CVA model are described in Section 4. There is obviously considerable variation at school level in the numbers of pupils joining at non-standard times. The PLASC checking process picks up a few obvious errors, e.g. 6 primary schools who may have submitted a date of entry which corresponds to the introduction of new software rather than the real dates. Another way of measuring mobility is simply to see which pupils had moved schools between one year s PLASC and the previous one (after taking into account schools which cover unusual age ranges). This method has been used in Machin et al (2006). In future, with more regular school census returns it will be possible to trace 25

26 pupils from school to school each term (i.e. three times a year) rather than once a year. Home postcode Postcodes are the codes used in identifying postal addresses in the UK. There are about 1.8 million of them, typically covering about 15 houses each. At a higher level of aggregation there are postal districts, sectors and areas. These can in turn be linked to other geographical classifications, such as the Super Output Areas (for further information see Appendix C in DfES, 2006b). The postcodes or larger areas could be included in the model directly, but what the Department and other analysts have done is to link these postcodes to small area data which gives some indication of the characteristics of a pupil s local environment. This local data is not part of the schools data collection process, it is derived from other sources. Various possible local indicators have been tried but at present the models use a measure called IDACI the Income Deprivation Affecting Children Index. This is the percentage of a Super Output Area s children under 16 who are living in families in receipt of Income Support and Job Seekers Allowance, or in receipt of Working Families Tax Credit and whose equalised income is below the 60% median income before housing costs. In other words, it is a measure of how many children in the local area are living in low income families. The index is calculated by another government department using data and a methodology which can be checked and is open to scrutiny. This makes it more suitable for an official value added system than the various commercial indices such as ACORN or MOSAIC which are widely used to classify small areas by their socio-economic characteristics 17. Postcode data collected on PLASC is checked to ensure the postcode exists: only a tiny proportion of them are invalid. At present there are no validity checks on the distance between postcodes and schools; some pupils live a long way from the school they attend so these kinds of checks would be time consuming. 3.3 Linking data Before the introduction of unique pupil numbers in 1999 it was possible to link a proportion of the test data using pupils names and dates of birth, but UPNs have 17 ACORN has been used by the Fischer Family trust in their value added models. For a discussion of ACORN and MOSAIC in the context of value added modelling, see Webber and Butler (2005). 26

27 made matching much more fast and accurate. It is not yet the case that every single pupil has a UPN - a few errors can occur when schools assign them. However, where UPNs are absent it is still possible to fall back on fuzzy matching methods, including the use of geographical data to narrow down the search for missing data. The result is a very complete match enabling the calculation of robust VA estimates. The development of UPNs has allowed the Department to construct a National Pupil Database, linking test data to the PLASC characteristics 18. Among many other things, this database allows academics, Local Authorities and the Department to produce consistent value added analyses for current and future years. The diagram below shows the current structure of the database. The earliest date for which data has been loaded varies according to the key stage. The dotted lines trace the progress of a particular cohort from Key Stage 2 through to Key Stage 4 it is not yet possible to link over a longer period 19. Figure 3.1 Datasets linked in the National Pupil Database in 2006 Dataset 95/96 96/97 97/98 98/99 99/00 00/01 01/02 02/03 03/04 04/05 PLACS (All ages) Key Stage 4 (Year 11 - Age15) Key Stage 3 (Year 9 - Age13) Key Stage 2 (Year 6 - Age10) Key Stage 1 (Year 2 - Age 6) Linkable data Data linkable in future Data not linkable 18 More information is at the PLASC/NPD User Group website: 19 The Year 7 progress tests are an intermediate test that is not taken by all pupils, so it is not used generally for value added modelling. 27

28 4. The value added models 4.1 Using prior attainment only: the median method Summary This method is based on matched pupil level data and uses prior attainment only, comparing a single input measure with a single output measure. It can cover one or more key stages, depending on the availability of data. National information has been published in the form of charts showing median outcome from each prior attainment point. This was adapted for the calculation of school scores, which are derived as the average for each school of the differences between each pupil s actual result and the national median result for pupils with their prior attainment score. The method was designed to be simple to calculate and understand. It also had to be easily integrated into the production cycle for Performance Tables. The pros and cons of the median method can be summarised as follows: Advantages A simple method based on the idea of a median line which has been used elsewhere for school improvement. Although it only uses prior attainment, this variable is easily the most important in explaining test results. Avoids using a regression model, but is able to cope with a non-linear relationship between prior attainment and outcomes. Use of the median makes the expected outcomes robust to the effect of outliers. Disadvantages The approach does not make use of contextual information which can influence school effectiveness. Scores are not centred on an easily interpreted figure. There are some ceiling effects that affect value added for the high achievers. The VA scores for small schools will be relatively unstable This section describes the median method and some of its advantages and 28

29 disadvantages, including points that have been picked up by external commentators. Naturally one of the possible criticisms is that it does not take sufficient account of other contextual variables (see Tymms and Dean, 2004) 20. However as contextual factors (including pupil mobility) are addressed in the contextualised value added model, they are not discussed in this section. The presentation of VA scores and the use made of them is addressed in Section The calculation of school value added scores The median method was designed to be simple for schools to understand and allow them to calculate their own value added scores with reference to information on the national expected results. Rather than use a regression model, the method was based on the median lines familiar to schools from the Autumn Package 21. In this system, a school can look at the prior attainment of each pupil and compare it to the median line, the difference being that pupil s contribution to the value added score. Figure 4.1 provides an example using a median line. One pupil has made 50 points more than expected at Key Stage 4, given their prior attainment at Key Stage 2, another has made 50 points less than expected. The sum total of the vertical distances to the median line, divided by the total number of pupils, is the school s value added score. Figure 4.1 Example median line and the calculation of value added Key Stage 4 Capped Point Score X X Key Stage 2 Average Point Score 20 An example of the effect of excluding contextual factors is that school VA scores are correlated with school prior attainment levels, i.e. schools with more low attainers in their intake tend to have lower VA scores. Since school prior attainment levels are included in the contextualised value this is no longer a feature of the new models. 21 The method was actually first piloted in 1998, the same year as the first Autumn Package, but the kind of median lines shown in the 1998 Autumn Package had appeared previously in information on GCSE to A-level value added. By 2002, when the first Performance Tables value added appeared, the Autumn Package was well established. 29

30 There is one final step in the calculation of a school s value added score which is done for presentation purposes. Schools have consistently said over the years that they do not want to see value added scores that are negative numbers because this may be misinterpreted as meaning that pupils are going backwards in terms of their progress. The decision was therefore taken to add the number 100 to all value added figures, so a school score of -3 points became 97 and a score of +3 points became 103. Whilst this adjustment makes the figures less directly interpretable, it remains possible to say that the school with 103 adds 6 more points to pupil progress than the school with 97. When the Key Stage 4 point score system became more complex (see Table 3.2) it was decided to use 1000 rather than 100 for this purpose (it has the advantage of looking less like an index number than the previous system, although centring round 100 is currently still being done for the other key stages) Input and output measures In designing the median method, a key decision was the precise definition of the input and output measures. The aim was to have one overall measure per key stage rather than separate measures for different subjects. It was also decided to use the points achieved in the tests rather than a simpler indicator of pass or fail relative to a given threshold (an example would be whether or not a pupil achieves five or more GCSEs at grade C or above). The advantage of this is that points provide more information for individual pupils in schools and therefore give value added models that take account of attainment at all ability levels. However, it can be argued that a threshold based measure is easier for schools to understand. At Key Stages 1-3, subjects were given equal weight and combined in a simple average (as noted already, the subject mix was determined by the core subjects at each key stage). At Key Stage 4 pupils do not take a set number of qualifications. Here total points rather than average points per subject was used, to encourage schools to enter pupils for as many examinations as possible. However, a cap of the best eight was also imposed to discourage schools from entering pupils for too many subjects, with the possible risk of damaging the quality of their education. This cap of eight is to some extent arbitrary an alternative figure like ten could have been chosen but it was arrived at after consultation and had the support of the teaching trade unions. 30

31 Table 4.2 Input and output measures for median method Key Stage 1 to 2 2 to 3 3 to 4 Input Average points from reading, writing and maths Average points from English, maths and science Average points from English, maths and science Output Average points from English, maths and science Average points from English, maths and science Total points from up to eight KS4 qualifications (best eight) The reason average rather than total points is used at Key Stages 1-3 is so as not to disadvantage a pupil who was absent from one of them (e.g. through illness). At Key Stage 4, which covers a range of subjects, many based on course work, there is no specific equivalent to being absent on the day of the test. However, an important decision in defining the Key Stage 4 output was to include pupils who are not entered for any tests and give them zero points. This is intended to discourage schools from not entering pupils simply because they are concerned they will get low marks Skewness in the value added scores Figure 4.3 Distribution of Key stage 4 outcomes Key Stage 4 outcomes in ,000 60,000 Pupils (state maintained schools) 50,000 40,000 30,000 20,000 10, Key Stage 4 total capped point score

32 One consequence of including pupils with no entries at Key Stage 4 is that it gives a spike to the distribution (Figure 4.3). This departure from a simple bell curve poses potential problems for the calculation of value added models 22. In the median method, some pupils who would be expected to get points based on the national median end up with zero and this tendency skews the pupil level value added distribution (Figure 4.4). Since school scores are calculated as the mean of the pupil value added scores, they will be weighted down by pupils who scored zero. As a result, more than half the schools have had mean differences below zero, and therefore value added scores below The same applies at other levels of aggregation: for example in 2005, only 17 of the 149 Local Authorities with maintained secondary schools had mean value added scores over 1000 (DfES, 2006c). Figure 4.4 Distribution of Key Stage 2-4 value added scores Key Stage 4 pupil value added scores in , ,000 Pupils (state maintained schools) 100,000 80,000 60,000 40,000 20, Key Stage 2-4 value added score Critchlow and Coe (2003) recommended moving to a consistent method to eliminate this problem, using either medians or means throughout. However, there is another reason why the value added scores are not guaranteed to centre on The national median line is calculated on data before it has been checked by schools and is not subsequently revised in light of the checked data. Since on balance the 22 The value added models of the Fischer Family Trust exclude the zero entries altogether. With a regression approach, this kind of distribution could be treated with a Tobit model, but this would add to the complexity of the VA system. 32

33 revisions tend to be upwards, the original median line will be slightly lower than the corrected medians. The comparison of unamended medians with amended results will therefore give more negative than positive differences and tend to reduce value added scores. These issues have particularly affected Key Stage 4 value added. The distributions are less skewed at other key stages (see DfES (2004), Part 2). Also, the spike at zero for Key Stage 4 is less prominent now than in earlier years, because of the low scoring courses now included in the calculations which enable pupils to obtain basic skills in vocational or other subjects. The fact that school value added scores are not centred on 1000 does not affect their usefulness as a measure of relative effectiveness, but it does make the figures more difficult to explain. The Performance Tables website has given information on the distributions of VA scores, but the possibility remains that a school with above average value added could have seen themselves as below average if their score was below The median method compared to a simple regression approach As noted above, the reason for choosing the median method over a regression approach was to make the system simple and to retain consistency with the school improvement Autumn Package. Both of these factors also had resource implications: a simple method that could be used once for different purposes required less time and staff to implement. Schools can easily calculate or check their own value added scores with reference to the expected results along a national median line. However, the same could also be said of a simple OLS regression model, which provides a formula for the calculation of expected results. The main difference in complexity is therefore in understanding how the expected results were calculated. The median is simply the result of the typical pupil. The regression formula can be described in terms of a summary of the line of best fit, but its calculation method (OLS) is not obvious to a non-statistical audience. An additional factor in choosing the median line method was that the relationship between prior attainment and outcomes is non-linear. The median method deals with this in a simple way because the median line is not straight (see Figure 4.1 for example). A regression model would need to include a non-linear term (adding a layer of complexity to the non-statistician audience) but even this would not 33

34 guarantee a good fit at the ends of the distribution. If the relationship between prior attainment and output changed at different points of the range, it may be necessary to partition prior attainment or introduce spline functions. Another advantage of the median line is that it is not influenced by outliers in the data. A mean method or a regression method could result in expected outcomes that were unduly affected by some unusual results. However, given the number of pupils involved in the calculations, this would only be an issue for the extremes of the distribution where there are relatively few pupils Stability of the value added scores Value added is designed to estimate school effectiveness and we would not expect this to fluctuate widely from year to year 23. There is a potential concern therefore in using VA scores for schools which do show large changes. A statistical analysis of the year-on-year volatility in results and value added scores for secondary schools was published in DfES (2004). For example, 1% of all schools moved from the upper quartile to the lower quartile of Key Stage 3 to 4 value added scores between 2002 and Volatility is not only a problem for the median method, it is an issue for any annual set of value added scores, and the smaller the school, the more likely results are to vary considerably from year to year. There is clearly a trade-off between statistical reliability and the desire to include data on as many schools as possible. for Performance Tables the number of schools is maximised by only excluding (i) primary schools if they have 10 or fewer pupils; (ii) any school where fewer than half of the pupils have matched data with which to calculate VA. The Value Added National Project (Fitz-Gibbon, 1997) recommended a minimum cut-off of 30 pupils. 24 A more recent report (Tymms and Dean, 2003) has suggested 50, but acknowledges that this would exclude most primary schools and imply that value added for primary schools should not in fact be published in Performance Tables at all. Another option for dealing with small cohorts is to combine data for more than one year, e.g. a three year average, but this inevitably means that results will be less upto-date. Volatility can also be dealt with by warning the reader about the reliability of 23 A useful discussion of issues in measuring and interpreting trends in value added scores will be published in Thomas et al (2007, in press). 24 The report also suggested a more sophisticated decision rule based on the calculated reliability of the value added score see Annex E in Fitz-Gibbon (1997). 34

35 results for small schools the notes accompanying the Performance Tables do this Possible ceiling effects It is in the nature of testing systems that they can restrict the full range of attainment demonstrable by very high or very low attainers. Ceiling effects may lead, for example, to a few high ability pupils not being able to demonstrate their true attainment level and thus contribute their full share to a value added score. Tymms and Dean (2004) suggest that a ceiling effect will introduce a bias into value added between Key Stage 1 and 2. The highest score now possible at Key Stage 1 is Level 3 in all subjects (21 points). The median KS3 points score for these pupils in 2005 was 33, which is the maximum obtainable at KS3. Hence nobody with Level 3s in all subjects at KS1 in 2001 could achieve a positive value added score (or, since 100 is added, a value added score greater than 100) 25. Pupils with lower levels of prior attainment can make a small positive contribution to their school s value added, but may also be subject to some ceiling effect. This remains a problem for the median method in its current form. However, this issue has been addressed in designing a contextualised model, as will be discussed below. 4.2 Contextual value added: a multilevel model 26 Summary This method uses a more complex definition of prior attainment and a range of contextual variables to predict attainment. The choice of contextual variables was based on statistical, educational and practical criteria. As with the median method, value added scores for each school are derived from the difference between predicted and observed attainment. After consultation with academics and practitioners it was decided to use a simple version of a multilevel model (MLM) to calculate these figures; MLM takes into account the fact that pupils are grouped into schools. The value added scores for many schools would be similar if an Ordinary Least Squares model had been used. However, a significant feature of MLM is the application of shrinkage, where the value added scores for small schools tend to be closer to the national mean, making it less likely that extreme value added scores 25 In 2001 it was actually still possible to get up to 27 points at KS1, but only 0.2% of pupils achieved more than 21 points. For those 0.2% there would also be a ceiling. 26 For an introduction to multilevel models see Kreft & De Leeuw (1998). Goldstein (2003) gives a more advanced treatment. Harvey Goldstein was one of the academics who provided advice to the DfES. 35

36 would be recorded for these schools. The model has deliberately been kept relatively simple: it could in theory have more levels of analysis, and more explanatory variables both in the fixed and random parts of the model. Advantages Contextual factors are taken into account. The hierarchical structure of the data is taken into account through multilevel modelling. The modelling framework gives external experts more confidence in the approach (although not all agree that MLM is appropriate for the purpose of providing value added scores). VA scores for small scores are less volatile (but also less likely to reveal differences from the mean). Disadvantages A multilevel model may be hard to explain to schools, but so far the consultation process has been positive. Scores are still not centred on an easily interpreted figure due to the requirements of the Performance Tables which mean that the model has to be calculated on early unamended data. It is difficult to get a good fit to the data at the extreme ends of the range. There are still some ceiling effects which are being mitigated with special adjustments. Section 3 described the data sources for the contextualised value added (CVA) model. This section describes the model specification and decisions made on the way the variables were defined and used. It focuses on one of the CVA models developed so far, estimating school effectiveness for Key Stage 2 to 4 in This model has been piloted for use in Performance Tables and used in the information provided by Ofsted and the School Improvement Partners. A slightly different version was discussed initially with schools a year earlier (October 2004) and some further amendments are being planned for later this year. These future developments and the issues that have emerged so far with the current model are discussed further below. As in Section 3.1, the model here is described in terms of the objective of providing school value added scores. Other uses, like allocating schools to particular initiatives are discussed briefly in Section 5. 36

37 4.2.1 Outline of the 2005 Key Stage 2-4 CVA model Table 4.3 shows the KS2-4 CVA multi-level model for The choice of fixed effects (equivalent to the standard regression coefficients) will be discussed shortly. In a multilevel model, the residual variance is partitioned: here the partition is into two levels: the pupil (Level 1) and the school (Level 2). These are the model s random effects. Within an education system it is possible to have other levels. For example, within schools, pupils are grouped into classes, but as there is no national data on teaching groups, this level cannot be modelled here. Above the level of the school there are Local Authorities, which fund and provide services to the maintained schools in their area. Researchers have often included these Authorities as a third level and this might be a sensible future amendment to the current model. However, to begin with the aim was to keep the model relatively simple. Table 4.3 The 2005 Key Stage 2-4 CVA regression model Multilevel model to predict capped Key Stage 4 points in logL = n = 548,222 pupils Explanatory factor Variable Estimate Std. Error P value Intercept ** KS2 student APS ** Prior attainment KS2 APS (using fine grades) squared ** KS2 English PS deviation ** KS2 Maths PS deviation ** Deprivation Does student have FSM? ** Deprivation of pupil's local area Deprivation indicator IDACI score ** Special Educational Needs Does student have SEN - Action Plus? ** Does student have SEN - school action? ** Mobility Student joined other than Jul/Aug/Sep? ** Student joined within last 2 yrs? ** Gender Is student female? ** Age Age within year ** Language Is English not the student's first language? ** Ethnic group Is the student White Irish? Is the student a White Irish traveller? ** Is the student White Gypsy/Roma? ** Is the student White other? ** Is the student Mixed White/Black Caribbean? Is the student Mixed White/Black African? * Is the student Mixed White/Asian? ** Is the student any other Mixed ethnic group? ** Is the student Indian? ** Is the student Pakistani? ** 37

38 Is the student Bangladeshi? ** Is the student any other Asian ethnic group? ** Is the student Black Caribbean? ** Is the student Black African? ** Is the student any other Black ethnic group? ** Is the student Chinese? ** Is the student any other ethnic group? ** Is the student in an unclassified ethnic group? ** In care Has the student ever been in care at this school? ** Level of school prior attainment School KS3 APS (using fine grades) for CVA ** Spread of school prior attainment School std dev of KS3 APS for CVA ** Random components: Estimate Std. Error Between school variance Within school variance Variance partition coefficient 0.07 This multilevel model was run in MLwiN, a software package that calculates the estimates in Table 4.2 and outputs the Level 1 and Level 2 residuals 27. Level 1 residuals show variation in pupils outcomes in relation to their schools. The Level 2 residuals show schools outcomes in relation to the national expected results, given the factors measured by the fixed effects. These Level 2 residuals are the value added scores. Another way in which this model has been kept simple is that there are no explanatory variables for the random part of the model. It assumes that a school is uniformly more or less effective for all its pupils, and that this can be encapsulated in a number, the value added score. A more complex approach is to assume schools vary in their effectiveness, e.g. between levels of prior attainment or for different ethnic groups. This could produce a range of measures for each school, or alternatively a set of charts showing the schools value added compared to national expected levels. Although there are significant differences in, for example, the slope of attainment outcomes relative to prior attainment, this has not been taken into account for value added in Performance Tables, where the aim was to provide a single indicator. Value added information for school improvement (e.g. in the forthcoming RAISEonline) is differentiated within schools according to whether pupils have low, medium or high prior attainment. The model mainly consists of pupil-level variables but also includes two school 27 For more on MlWin see 38

39 composition variables, i.e. variables that take the same value for every pupil in a given school. Most of the pupil-level variables are simple flags relating to the PLASC data discussed above. Table 4.4 shows how many pupils were coded 1 for each of these flags in the 2005 KS2-4 model. Most of these categories applied to thousands of pupils nationally, although of course many individual schools have no pupils from some of these groups. Table 4.4 The prevalence of different pupil characteristics Binary variables in the KS CVA model Number Percentage Does student have FSM? % Has the student ever been in care at this school? % Does student have SEN - school action? % Does student have SEN - Action Plus? % pupil joined school after Sept Yr % pupil joined school not in July /AUG/ Sept Yr % Is student female? % Is English not the student's first language? % Is the student White Irish? % Is the student a White Irish traveller? % Is the student White Gypsy/Roma? % Is the student White other? % Is the student Mixed White/Black Caribbean? % Is the student Mixed White/Black African? % Is the student Mixed White/Asian? % Is the student any other Mixed ethnic group? % Is the student Indian? % Is the student Pakistani? % Is the student Bangladeshi? % Is the student any other Asian ethnic group? % Is the student Black Caribbean? % Is the student Black African? % Is the student any other Black ethnic group? % Is the student Chinese? % Is the student any other ethnic group? % Is the student in an unclassified ethnic group? % The choice of variables The decisions over which variables to include in the model were based on a mixture of statistical, educational and practical considerations. Given the need to provide value added information for every school, it was necessary to restrict the choice to information for which there is national data. Since the aim was to generate value added residuals, the explanatory fixed effect variables needed to cover factors that are outside the school s control. Prior attainment and pupil characteristics are 39

40 included because they are facts about the inputs to a school. Information on, for example, attendance levels, would not be included because attendance can be seen as, to some extent, an output of the school (even though poor attendance could affect results). With CVA, the variables relating to prior attainment are still the most important in explaining results. There were three important changes to the way prior attainment was treated compared to the earlier median method: The move to a more complex methodology offered the chance to reconsider the use of a simple combined-subject indicator for prior attainment. The average point score (APS) was still included, but two extra terms were used, measuring the difference between the English and Maths results and this overall APS. Another important addition was the quadratic term, reflecting the fact that the relationship between KS4 outcomes and KS2 prior attainment is non-linear. With the median method this non-linearity was taken into account because the median line did not need to be straight. The APS prior attainment measure was based on the levels achieved in the three subjects. However, the level distribution for Key Stage 2 does not offer a very fine grained distinction between pupils (see Table 3.1). Clearly the underlying marks data give more information and would improve the fit of a value added model. As these marks are collected centrally, they are available for use. They were not used in the first median method model, partly to keep the model simple and partly because a pupil s actual marks are not officially recognised in the way that a level is (also, the quality of the marks data was less reliable in the earlier years). A compromise between marks and levels has therefore been developed, with levels divided into sublevels using marks data. This retains the well-understood and widely recognised level structure but utilises the more detailed information on pupils performance available from the marks. In deciding how many additional contextual variables to include, there has not been time to test exhaustively every possibility offered by the national data discussed in Section 3. The starting point was a consideration of the most obvious possibilities, taking into account what is known from previous internal and external research about 40

41 factors that explain variation in test results. In deciding on the specific definition of the variables, some exploratory analysis was necessary: for example it was decided to include two mobility variables to capture (i) the general effect and (ii) the additional impact of being mobile just before the exams 28. Overall the criteria of simplicity and intelligibility to schools were not abandoned it was felt that the model still needed to be kept as simple as possible. This is why, for example, the initial CVA model discussed here has no interaction terms (although the 2006 models will have a limited number of interactions). The model was built up stepwise and the stability of coefficients checked. With so many observations, almost all variables are statistically significant and adding extra variables tends to improve the fit of the model. As an example of a more limited model, Table 4.5 shows a version which has the prior attainment variables only. The 2 Log Likelihood statistic (which is used in MLM to compare models) is significantly larger, which implies that the more complex CVA model is a better fit to the data. Prior attainment nevertheless provides most of the explanatory power of the CVA fixed effects. In an OLS context (using the R-squared figures), the equivalent prior attainment only model explains 49% of the variance, whereas the CVA version with all the additional variables explain 57%. Table 4.5 A 2005 Key Stage 2-4 prior attainment only model Multilevel model to predict capped Key stage 4 points in logL = n = 548,222 pupils Explanatory factor Variable Estimate Std. Error P value Intercept * KS2 student APS ** KS2 English PS deviation ** KS2 Maths PS deviation Prior attainment KS2 APS (using fine grades) - squared ** 28 For KS2-4, there is one flag for pupils joining the school after September of year 10 (i.e. close to their KS4 exams) and another for pupils joining at a non-standard time: not in July, August or September of years 7, 8 or 9. 41

42 Random components: Estimate Std. Error Between school variance Within school variance Variance partition coefficient 0.11 In looking at the range of possible explanatory variables, some significant ones were not included if it was felt that they would add complexity without greatly enhancing the quality of the model. For example, it would have been possible to include FSM status information from previous years, since these categories are not stable over time and pupils with FSM every year tend to get slightly lower results than pupils who move in and out of FSM status. Conversely, a couple of non-significant ethnic categories were included because it was felt that for practical and presentational reasons it would be better to include all the categories rather than combine two of them with other groups on the basis of the 2005 data (including these variables makes very little difference to the overall model.) The degree of correlation among the variables was checked if high correlation were to lead to unstable or odd looking estimates, it would make the model harder for schools to accept. With so many observations (n = 548,222) it is not surprising that many of the variables are statistically significantly correlated, although most correlations (Pearson correlation coefficients) are less than 0.2. Prior attainment is correlated with some of the variables unsurprising given that these variables are being considered as explanatory factors for attainment at a subsequent key stage. There are correlations above 0.2 for some ethnic groups and first language, and some ethnic groups and FSM status. There is also, unsurprisingly, a reasonable level of correlation between FSM status and local deprivation measured by the IDACI variable (0.356); but it was agreed that both should be included in order to use as much information as possible about the social background of the pupil. The decision to move to a CVA model meant accepting that levels of deprivation, special educational needs and so on should be taken into account when assessing value added. It would have been possible to choose a subset of factors for this purpose, e.g. if it was felt important to take into account deprivation but not ethnicity, but there were no strong reasons for making an exception of any variables on educational grounds. The coding and description of variables that could involve 42

43 political sensitivities has been checked with relevant policy experts: for example, combining similarly performing ethnic groups may be acceptable on statistical grounds but would not necessarily be appropriate from the point of view of presenting the model to schools and parents. For similar reasons, there may be concerns with some of the names used to classify local areas in a measure like MOSAIC which is used for other purposes like advertising (the Department s CVA model uses IDACI, which simply gives areas a number between 0 and 1). In designing a value added system it is important to avoid perverse incentives or unintended consequences. A potential problem with some contextual variables is that schools could manipulate them to affect their value added score. For example, the unclassified ethnicity variable has a negative coefficient, i.e. the more unclassified pupils a school has, the lower will be its predicted results and thus, potentially, the higher will be its value added. This gives schools a possible disincentive to find out or report pupils ethnicity. A similar problem relates to SEN and the School Action category. It is up to schools to assign pupils to this type of SEN: if they assign lots of pupils it will depress their predicted results and boost their value added. In general these kinds of disincentive effects can be overcome if the data items are used elsewhere. For example, schools with more ethnic minority pupils have in the past attracted higher funding and the SEN figures are published independently in Performance Tables (where a school might not wish it to be seen that they have lots of pupils with SEN and in school action ). The last two variables in the model are at school level. The overall level of prior attainment has long been acknowledged as a factor in explaining pupil progress (the median method VA data was benchmarked by school prior attainment levels in the Autumn Package). The CVA model not only includes the level of prior attainment, it also includes the spread (as measured by the standard deviation), on the basis that schools with a narrow ability range have an advantage and can more easily appear effective. In practice, the schools with narrow ability ranges tend to have pupils with relatively high overall prior attainment there is a high correlation (-0.806). However, both variables have been included so as reduce the bias that would otherwise occur in favour of schools with a more restricted intake. These two school level variables relate to the prior attainment of the cohort whose progress is being modelled. This means that the variables describe these pupils direct peers but do not measure the overall characteristics of the schools pupils 43

44 (which could impact on the learning environment). An alternative would be to define school level variables based on all pupils in the school at the time of the KS4 results, or even all pupils in the school during the cohort of interest s whole time in the school. Many other school level variables could be derived from the pupil level variables, e.g. the percentage with first language other than English, the percentage of girls, the percentage of SEN pupils, or the percentage of FSM pupils. Some of these variables have been used in other value added models (e.g. NAO, 2003). An advantage of these variables is that they can proxy pupil level characteristics that are not measured. For example among the undifferentiated group of non-fsm pupils, it might be expected that those in schools with high levels of FSM would tend to be from relatively less affluent backgrounds (although the inclusion of IDACI already helps to differentiate the non-fsm pupils). Once again, the main motivation for not including more school level variables has been to keep the model as simple as possible. There are also grounds for minimising the number of school level context variables because they can be correlated with their pupil-level equivalents. There may also be a case for not taking factors like the percentage of SEN into account if it is felt that schools should not be given higher value added scores when they have more of these disadvantaged pupils (although this is to some extent a version of the general argument against contextual factors, whether at pupil or school level). School level context variables can also be hard to interpret and explain to schools. For example at Key Stage 4 progress is relatively greater for pupils in schools with both low and high numbers of FSM pupils. This could be reflected in the CVA model, but expecting more progress from these two groups of schools and less progress from schools in the middle of the FSM range would need a clear rationale in educational terms. Finally, it is worth considering the degree of similarity between different models. Most value added models, for different key stages, for different subjects 29 or for different years would be similar i.e. most of the variables discussed here could be included on statistical and educational grounds and could be specified in the same way. If, say, a particular ethnic category was statistically insignificant for one particular subject or key stage there may be grounds for still including it in order to 29 Models for different subjects are not included in Performance Tables, where the aim is to have one model for each key stage, but are used elsewhere for school self-evaluation. 44

45 maintain consistency. However, some differences in specification may be appropriate: for example in the Key Stage 1-2 CVA model, school level prior attainment variables are not used because the smaller cohort sizes make them harder to estimate. In general, year-on-year consistency in the models would be reassuring for schools and help in interpreting changes over time. However, models can be open to amendment if one of the explanatory variables altered in some way or a useful new variable became available The choice of regression method Once it was decided to include various contextual factors, there was no longer a possibility of using a simple non-regression approach like the median method. Given the need for a multiple regression model taking into account prior attainment and contextual factors, the two basic options were Ordinary Least Squares or Multilevel Modelling. Multilevel modelling offers a more complex set of modelling options which take into account the structuring of the data, i.e. the fact that pupils are organised into schools. OLS is the standard regression method: widely familiar to those with statistical training, although obviously not familiar to most teachers or parents. An OLS version of the CVA model is shown in Table 4.6. Before discussing the differences between OLS and MLM it is worth noting the R 2 figure here of 0.57 (there is no direct equivalent for the multilevel model see Kreft & De Leeuw (1998) p115). This shows that the prior attainment and contextual variables only explain just over half the variance in Key Stage 4 results. This may seem relatively low, but it needs to be born in mind that this model measures value added over a five year period. In most models, the majority of variance in outcomes is explained by the prior attainment term and here, the model is trying to predict results from the full range of KS4 qualifications taken by 16 year olds (including vocational courses, art and design, physical education etc.), from prior attainment data that only covers English, maths and science from the tests at age 11. Other CVA models have higher R 2 values. 30 Changes to the model for 2006 mean that two version of the 2005 CVA estimates will have to be used this year: the original estimate and an equivalent based on a model amended to the new methodology. 45

46 Table Ordinary Least Squares version of the CVA model OLS model to predict capped Key stage 4 points in 2005 R 2 = ,222 n = pupils Explanatory factor Variable Estimate Std. Error P value Intercept ** KS2 student APS ** KS2 English PS deviation ** KS2 Maths PS deviation Prior attainment KS2 APS (using fine grades) - squared ** Deprivation Does student have FSM? ** Deprivation of pupil's local area Deprivation indicator - IDACI score ** Does student have SEN - Action Plus? ** Special Educational Needs Does student have SEN - school action? ** Student joined other than Jul/Aug/Sep? ** Mobility Student joined within last 2 yrs? ** Gender Is student female? ** Age Age within year (corrected) ** Language Is English not the student's first language? ** Is the student White Irish? * Is the student a White Irish traveller? ** Is the student White Gypsy/Roma? ** Is the student White other? ** Is the student Mixed White/Black Caribbean? Is the student Mixed White/Black African? ** Is the student Mixed White/Asian? ** Is the student any other Mixed ethnic group? ** Is the student Indian? ** Is the student Pakistani? ** Is the student Bangladeshi? ** Is the student any other Asian ethnic group? ** Is the student Black Caribbean? ** Is the student Black African? ** Is the student any other Black ethnic group? ** Is the student Chinese? ** Is the student any other ethnic group? ** Ethnic group Is the student in an unclassified ethnic group? ** In care Has the student ever been in care at this school? ** Level of school prior attainment School KS3 APS (using fine grades) for CVA ** Spread of school prior attainment School std dev of KS3 APS for CVA ** One of the main reasons for using an MLM approach is that OLS will tend to underestimate the standard errors for the estimates of the model parameters. Here the MLM standard errors are wider for the school level composition variables (as 46

47 would be expected once school differences are explicitly modelled as a level) although the estimates for these two coefficients are still significant under MLM. For the pupil-level variables, some of the standard errors are wider in the MLM model, others are narrower. Both OLS and MLM produce unbiased estimates for the regression coefficients. A comparison of the OLS model with the MLM model in Table 4.3 reveals small differences in most of the coefficients. Some differences are almost negligible (for example, the effect of FSM is very similar in both). In other cases there are differences that would have a minor impact on some schools value added: e.g. the coefficient for Black Caribbean pupils is 6 points lower in MLM, which means that each Black Caribbean pupil is predicted a grade in one subject less in the MLM model compared to the OLS model. Overall therefore the models are similar. In the multilevel CVA model, the intra-school correlation (or variance partition coefficient) is The main differences between OLS and MLM model coefficients would tend to occur for relatively high correlations, i.e. when the data is highly structured. Figure 4.5 Illustration of the effect of shrinkage in the CVA model Shrinkage of raw residuals in the KS2-4 CVA model Shrunk CVA residual Size of school Raw school residual Given the similarity of the models, the main difference to the results in using MLM rather than OLS has been the way that MLM shrinks the value added estimates (shrinkage is explained in Annex B). The degree of shrinkage depends on the size of the school: smaller schools are shrunk towards the national mean. Figure 4.5, based on the shrinkage factor for the KS2-4 CVA model, shows how some example 47

48 raw residuals are brought closer to the national average (zero). A school with a raw residual of +10 starts to be affected by shrinkage for cohorts of less than about 50. For a school with a more extreme raw residual, +50, some shrinkage occurs even in quite large schools, but the main impact would again be for particularly small schools. Table 4.7 provides some information on the numbers of schools where shrinkage has an impact. Here a shrinkage factor of, for example, 0.9, means that the value added score is reduced to 90% of its raw size, moving slightly closer to the national average. Only 14% of secondary schools had a shrinkage factor below 0.9. Even among primary schools, 86% had a CVA score that was at least three quarters the size of the raw residual. Primary schools with cohort sizes of 10 or fewer are currently excluded from the Performance Tables. Schools with 11 pupils have a shrinkage factor of Table 4.7 Impact of shrinkage on CVA scores for schools Primary (KS1-2) 2006 Secondary (KS2-4) 2005 Shrinkage No. of Schools Percentage No. of schools Percentage % 0 0% % 1 0% % 14 0% % % % % Total % % There is no easy solution to the problem of interpreting value added for small schools (see the earlier discussion of volatility in Section 4.1.5). In OLS, a small school s value added is based on the pupils it has, with a confidence interval implying a wide margin of error. Multilevel modelling provides a different approach, where the point estimate is calculated from both the school s pupils and the national school distribution (with narrower confidence intervals around the resulting estimates). The national distribution is used to modify the estimate when robust information on the school is limited because of its size 32. This shrinkage process could be seen as problematic because it prevents a genuinely successful or ineffective small school 31 These primary school statistics relate to new 2006 data rather than the 2005 data quoted elsewhere, but the pattern would be similar from year to year. Primary schools with fewer than 11 pupils have been given CVA scores in the information provided for school improvement. 32 This approach is reflected in the fact that the shrunken residuals are sometimes known as the empirical Bayes estimates or posterior mean estimates. 48

49 from registering a significantly high or low value added score. However, in such circumstances, there is no way of telling from one year s data whether these small schools raw residuals are good estimates of effectiveness, or whether their results have been subject to random error of some kind. The advantage of MLM is that it explicitly allows for this uncertainty in calculating the value added score. Views differ on the appropriateness of using shrunken residuals in the context of a system for providing value added scores. MLM residuals may not be ideal for ranking schools (Kreft & De Leeuw (1998) p52) but there would also be drawbacks in using other methods for this purpose (and it needs to be born in mind that the Performance Tables do not explicitly provide such a ranking). As noted already, the National Value Added Project concluded in 1997 that for most purposes simple regression models would be sufficient. However, many of the experts consulted by the DfES more recently favour MLM and it was decided to move ahead on this basis. In terms of intelligibility for a lay audience, there is a big difference between the median line method and a CVA regression model, but not too much difference between regression models based on OLS and on MLM. Ofsted have recognised some of the issues around shrinkage by provided their inspectors with information on how to unshrink (i.e. divide by the school s shrinkage factor) so as to see what the raw residual would have looked like. This raw residual gives some additional information which can be used when the Inspector considers the overall performance of the school. The Inspector may be able to judge from other data and impressions of the school the extent to which the raw residual appears to be an accurate reflection despite the possible uncertainty associated with it, e.g. that the school really is as effective as the raw residual implies The process of calculating value added scores Figure 4.6 sets out the process planned for calculation of CVA scores in Ideally the process of providing school value added scores would involve collecting the data, running the modelling software and distributing the results. However, this is complicated by the need to allow schools access to provisional CVA data before it is published. This is done in the interests of transparency and the desire to retain schools confidence in the methods, as well as giving schools the chance to check the contextual variables that have been used in the model. If they want to, schools can check how the actual calculation of value added scores works using their own 49

50 data and a ready reckoner provided on the Department s website 33. Figure 4.6 The process of calculating CVA for the Performance Tables CVA in Performance Tables Process Test data collected Organization QCA / Examining bodies Matched to prior attainment and PLASC Multilevel modelling: CVA coefficients calculated Adjustments for ceiling effects specified Calculation of CVA scores and other indicators Results and scores checked Amended data calculated Contractor DfES DfES Contractor Schools Contractor Performance Tables published Contractor Any changes to the test or contextual data made by schools at this checking stage would ideally feed through into a revised model (and then potentially a further round re-notifying schools of the results). But this would be difficult, given the need to publish results as soon as possible. The relatively small numbers of amendments 33 See 50

51 should not change the model significantly, which is why the model is fixed at the stage of the unamended data. School residuals are then calculated from the amended data, with reference to this unamended model. As with the median method, this means that the school value added scores differ slightly from those that would be obtained if the whole model had been run on amended data. This is not ideal but is a practical solution given the agreement with schools that they are shown provisional CVA scores in advance of publication. The upshot of this approach is that although multilevel modelling is used to produce the initial equation (Table 4.3 above), the final value added scores and their associated confidence intervals are not direct outputs from a multilevel model. They need to be calculated from the model equation and the formula for shrinkage. The specification for calculating the residuals is given to the Performance Tables contractors (who therefore do not actually need to have multilevel modelling software or expertise). The contractors also calculate the initial unamended results in the same way, using a formula that uses the fixed effects and variance terms estimated by the Department (working jointly with Ofsted). The process shown in Figure 4.6 also includes an adjustment to impose a ceiling on value added scores so that the predicted KS4 results for pupils are not higher than the theoretical maximum. These small alterations are implemented by the contractors according to standard rules and are another reason why the value added scores do not come directly from MLM software. The advantage of this approach is that the adjustment is easily understood and programmed. A neater solution would involve changing the model so that these adjustments are not necessary, but this could add considerably to the model s complexity (ceiling effects are discussed further below). The initial multilevel modelling is carried out within the Department and Ofsted using MLwiN 34. MLwiN is used quite widely in the UK and therefore has the advantage of being supported by an infrastructure of training courses and advice. The Department is also looking into whether SPSS can be used for the relatively simple type of multilevel model being used here; this would have the advantage of being a package familiar to a wider range of analysts. The pros and cons of other packages have not been tested, so the fact that MLwiN is currently used does not necessarily mean that it is the best package for running a large-scale annual exercise in calculating value

52 added scores Issues for development The Key Stage 2-4 CVA model was piloted for use in Performance Tables in autumn Feedback from the pilot schools was very positive and there were no widespread objections to the use of particular contextual variables. However, there was a call for investigation of some interaction terms and as a result of subsequent analysis it has been decided to augment the model with (1) first language interacted with prior attainment and (2) ethnic groups interacted with the FSM measure of deprivation. Both these interactions provide statistically significant improvements to the model and while the impact on school estimates was negligible for most schools, some school estimates changed moderately. Another interaction, between ethnic groups and first language was investigated but found not to improve the model enough to warrant its inclusion in addition to (1) and (2). A further adjustment to the model for 2006 will be to address the ceiling effect problems. As noted already, some ceiling effect is inevitable given the practical constraints of a testing framework that cannot cater for every possible level of attainment. However, the CVA model in 2005 had an additional problem: the mean observed result was generally in line with the predicted result except at the extremes of the attainment distribution. This meant that, for example, the model tended to overestimate the predicted results for high achievers leaving them with little scope to achieve positive value added, with knock-on effects for a few schools with relatively large numbers of these pupils. The model already has an adjustment to ensure predicted results cannot exceed the theoretical maximum. This has been amended so that, at the extremes, predicted results that lie above the mean actual result (for a small bin range of attainment outcomes) are reduced to the level of the mean. This new approach will continue to be looked at it may be preferable to try to improve the fit of the original model, even at the expense of adding further complexity to the model equation. Another option would be to standardise residuals at each prior attainment point, stretching small differences in the actual point scale for high and low attaining pupils so that they 35 Information on the pilot is on the Performance Tables website: As mentioned in Section 2, the model was used more extensively during 2005 and 2006 as part of the New Relationship with Schools, for inspections and school improvement. 52

53 have more chance of affecting the school VA scores (Schagen, 2006). However, this approach would involve changing to a new metric rather than using the National Curriculum and Key Stage 4 points. There are various further issues that can be investigated for future versions of the model. For example, more school level variables, like the percentage of minority ethnic pupils, will be considered. Pupil mobility is another issue that could be addressed in more depth in the future now that there will be termly school censuses. Although the CVA model includes two mobility measures as factors that may explain test results, it still ascribes the pupil s CVA to the school in which the pupil took the test. An alternative approach would be to apportion progress between schools, although this would add an additional layer of complexity. 53

54 5. The use and presentation of value added models Summary The Performance Tables include a limited range of statistics on schools: value added data is presented alongside facts about overall attainment and school context. For school improvement and inspection, a wider range of value added measures and charts have been given. Both school value added scores and other types of value added analysis have been used elsewhere: in publishing information for parents and schools, in selecting schools for particular purposes and as part of the approach to target setting. General issues in presentation of value added are: How should schools and other stakeholders be consulted on the development of the models? How should value added scores and analysis be described for different audiences? Which graphical presentations are the most appropriate? Can the media be encouraged to present value added accurately? More specific issues are: When and how should confidence intervals be given? Should schools be measured as significant in relation to the national average? Should schools ranks be given as well as their value added score? Should schools be grouped and then labelled (e.g. high performing, or Category 1 ) or coloured coded? 5.1 Consultation and presentation of pilot value added models As will be clear from Sections 2 and 4, the development of value added models has involved a combination of: (1) internal development work; (2) advice from external experts (both commissioned and more generally through discussion); (3) consultation with schools and Local Authorities. Consultation has centred on a series of pilot conferences where models and results are discussed with representatives from schools and Authorities (Table 5.1 below relates to CVA there were similar 54

55 conferences for the original VA measures). There has also been an evaluation questionnaire which all schools included in the pilots were invited to complete (a 40% response rate for the 2005 secondary CVA pilot). Feedback is evaluated and used as the basis for further model development (an example being the request to introduce interaction terms into the CVA model) and for improvement to the presentation and documentation of value added scores in the Performance Table. Ofsted have also consulted on the use of value added, both with Inspectors and with Local Authorities. Table 5.1 Contextualised value added consultation conferences Date Delegates Model Audience Location 05 October CVA prototype Schools Birmingham 07 October CVA prototype Schools London 08 October CVA prototype Schools York 24 January Secondary CVA Schools Birmingham 26 January Secondary CVA Schools London 31 January Secondary CVA Schools York 02 February Secondary CVA Schools London 02 February Secondary CVA Authorities London 21 February Secondary CVA Authorities Leeds 27 June Primary CVA Schools Leeds 29 June Primary CVA Schools London 11 July Primary CVA Schools York The presentations used at the conferences in Table 5.1 are available and give an indication of how CVA has been presented to schools and Authorities in these conferences. Figure 5.1 shows one example slide, comparing VA and CVA scores and demonstrating the effect of contextualisation and the change to a multilevel modelling framework. In general the approach in the conferences has been to explain technical aspects as simply as possible, but without skipping important points like the shape of the prior attainment distribution (non-linear and modelled using a quadratic term), the role of the shrinkage factor and the calculation of confidence intervals. 55

56 Figure 5.1 Value added v Contextualised value added: example slide from the consultation presentation 5.2 The presentation of value added scores in Performance Tables Performance Tables data are published both online and in booklets for each Local Authority. Figure 5.1 shows how the 2005 VA scores, based on the median method and prior attainment only, appeared on screen for an example secondary school 36. The result is included alongside raw results and some contextual information. Here the KS2-4 score of means that the pupils in this school achieved, on average, 10.2 points less than the median pupils for each prior attainment level. However, this school actually had slightly higher than average value added since in 2005 the mean school VA score was (see the earlier discussion in section 4.1.3). 36 See 56

57 Figure 5.2 Screenshots showing VA on the website for Performance Tables 57

58 The Performance Tables website has guidance on how to use and interpret the VA figures. For example, the 2005 site includes the message reproduced below, designed to reinforce the idea that VA is a better measure of school effectiveness than raw results. The reference to significance is needed because the VA scores are not accompanied by confidence intervals: instead the website gives guidance on the range of scores that can be considered broadly average depending on the size of school. What a school's value added measures tell you The value added measures give the best indication in these Tables of schools' overall effectiveness. But the significance that can be attached to any particular school's value added measure depends, among other things, on the number of pupils included in the value added calculation. The smaller the number of pupils, the less confidence can be placed on the value added measure as an indicator of whether the effectiveness of a school is significantly above or below average. Although VA scores are published on-line, many parents will first become aware of them through the media. There is no space here to discuss the media treatment of value added in detail, but a few examples can be given. Figure 5.3 is an extract from The Guardian (19/1/06) which in common with the other broadsheet newspapers published the school figures for each Authority in alphabetical order (although note that the title is League Tables ). These newspapers also provide a key explaining the figures, based on information on the Performance Tables website. The Times (19/1/06) did give one league table where schools were ranked (Figure 5.4), showing the schools with the highest Key Stage 2-4 value added (many of which were small independent schools). The tabloid press did not report the value added scores. Practice will have varied among local newspapers, but the emphasis is normally still on raw results An example report for the Sussex Evening Argus is here: 58

59 Figure 5.3 Extract from The Guardian showing value added and other data Figure 5.4 Extract from The Times showing a value added league table 59

60 School 1 Street name Vilage / Town City/County Postcode Tel School 2 Street name Vilage / Town City/County Postcode Tel School 3 Street name Vilage / Town City/County Postcode Tel School 4 Street name Vilage / Town City/County Postcode Tel School 5 Street name Vilage / Town City/County Postcode Tel School 6 Street name Vilage / Town City/County Postcode Tel School 7 Street name Vilage / Town City/County Postcode Tel CY CY IND IND CYS CYS NMSS COMP MIXED COMP FD MIXED SEL MIXED SEL GIRLS N/A MIXED N/A MIXED N/A MIXED R B R B Measure centred on 1000 CVA measure based on progress between KS2 and KS published Not published Not entered Not entered Local Authority (excluding independentschools) Upperand lowerlimits ofks2-ks4 CVA Confidence Intervals Not published Not published Not entered Not entered Coverage %ofpupils athe end ofks4 included in CVA calculation Numberof qualifications 100% % 99% 100% % 99% Not published Not published Not entered Not entered Average numberof qualifications taken by pupils in KS2 -KS4 CVA calculation Not published Not published Not entered Not entered Numberof pupils at the end of KS4 Percentage ofpupils at the end of KS4,aged 14 orless Percentage ofpupils at the end of KS4,aged % 99% 35 1% 99% 4 1% 99% 17 1% 99% Numberand percentage ofpupils at the end ofks4 with SEN with statements or supported at School Action Plus % 7.9% 16.1% 14.3% Supported at School Action % 15.7% 25.8% 2.9% Level 2 Level 1 (5 ormore grades A*-C) (5 ormore grades A*-G) %of15 yearold pupils achieving 5+A*-C (and equivalent) % 40% 41% 40% 49% 51% 48% 50% 40% 80% 85% 50% 96% 98% % % 49% 46% 40% 49% 51% 48% 49% 40% 78% 85% 49% 97% 98% % 80% 81% 85% 86% 84% 85% 70% 80% 89% 85% 97% 97% % % 90% 95% 94% 95% 85% 97% 80% % 95% 99% 100% 97% 100% 100% Not entered Not entered Not entered Not entered Not entered Not entered Not entered Not entered Not entered Not entered Not entered Not entered Not entered Not entered % 5.5% 3.0% 6.1% % 15.0% 10.4% 3.9% authorised absence unauthorised absence % 0.5% School % 0.4% School %! School % 0.0% School6 4 N/A N/A N/A N/A N/A N/A N/A N/A % 0.0% School7 17 N/A N/A N/A N/A N/A N/A N/A N/A % 3.7% School10 70% 75% 73% 51% 53% 91% % 99% 50% 60% 80% 73% 82% 82% % 50% 42% 97 N/A N/A N/A School11 4.4% 13.6% 5 ormore grades A*- C including English and Maths GCSEs Level 2 in 'functional English and maths' %ofpupils achieving: 47.9% 80.4% 86.5% 43.3% 87.8% 95.0% %39.8%43.1%43.3% 40.5%39.8%43.1%43.3% 4.4% 13.6% 8.8% 1.4% LocalAuthority England (including al schools) % 10.2% 49.9% 81.2% 85.5% 51.5% 88.9% 95.8% %49.2%50.0%51.5% 47.9%49.2%50.0%51.5% 3.1% 10.2% 7.5% 1.1% England Level 1 in 'functional English and maths' %ofpupils achieving at leastone qualification Average total pointscore per pupil %of15 yearold pupils achieving 5+A*-C (and equivalent)including English and Maths GCSEs Total numberof pupils (al ages) Total numberand percentage ofpupils with SEN with statements or supported at School Action Plus Supported at School Action Absence record for day pupils of Numberofday pupils of compulsory school age compulsory schoolage %ofhalfdays missed due to The contextualised value added scores have not yet been published in Performance Tables. Figure 5.5 shows two views of a mock-up for the 2006 Tables booklet (the website is not yet designed). The coverage of each score is given as usual but now for the first time upper and lower bounds of a 95% confidence interval are presented alongside the figures, derived from the multilevel modelling. The confidence intervals are interpretable as the range of uncertainty around each school s estimates of school effectiveness. Although all pupils, not just a sample, are used to calculate CVA, confidence intervals are useful because the pupil s results are still effectively a sample of the possible outcomes that could have been achieved by pupils with these characteristics in these schools 38. Note that the confidence intervals do not say directly whether a given School X is significantly different from another School Y. Figure 5.5 Proposed presentation of CVA scores for the 2006 results L EA Key Stage 2 tokey Stage 4 ContextualValue Added Cohort Informationfor pupils at the endof Key Stage 4 GCSE andequivalent achievements of students at the endof Key Stage 4 Number of 15 Year OldPupils onrolin 2005/06 Year onyear comparisons BackgroundInformation SCHOOLS Upper Lower SCHOOLS! 4-19 Not SPECIAL SCHOOLS SPECIAL SCHOOLS! Detail (top left): Key Stage 2 to Key Stage 4 Contextual Value Added LEA Measure centred on 1000 Upper Lower Coverage Number of qualifications SCHOOLS CVA measure based on progress between KS2 and KS4 Upper and lower limits of KS2-KS4 CVA Confidence Intervals % of pupils at the end of KS4 included in CVA calculation Average number of qualifications taken by pupils in KS2 - KS4 CVA calculation School 1 Street name Village / Town City/County Postcode Tel CY COMP MIXED R B % 8.4 School 2 Street name Village / Town City/County Postcode Tel CY COMP FD MIXED R B % 9.0 School 3 Street name Village / Town City/County Postcode Tel IND SEL MIXED! 4-19 Not published Not published Not published Not published 38 The Department will not be putting confidence intervals around the raw scores because these are seen less as effectiveness estimates and more as simple statements of fact about the results in a given year. 60

61 5.3 The presentation of value added scores for school improvement Whilst the Performance Tables focus on a few key indicators, information for school improvement, first in the Autumn Package and now via RAISEonline, is more extensive and detailed. The data in RAISEonline will not be made public: it is meant for use by the school and contains data on each individual pupil. Local Authorities and independent providers like the CEM Centre at Durham University have treated their school improvement data as confidential to the school to encourage an open dialogue and accurate self-evaluation. The difference between these sources and RAISEonline is that results in the latter are available to the Department and Ofsted inspectors. The main method of presenting value added information at a national level in the Autumn Package has already been discussed in relation to the median line method. This information is still being provided each year and the illustrations below are taken from the 2005 Key Stage 4 data 39. Figure 5.6 shows the median line and interquartile range of the KS4 outcomes from different KS2 starting points. Figure 5.7 relates to one particular band of prior attainment and shows the frequency of different KS4 outcomes. The website explains that the median line graphs enable schools to compare the progress of their pupils with progress achieved nationally taking into account prior performance. The Progress Charts provide information to support schools in raising their expectations of pupil achievement and can be used in setting realistic but challenging targets. Figure 5.6 Value added line chart from the Autumn Package 2005 GCSE & Equivalent Capped Value Added Line GCSE & Equivalent Total Points Score and Average 2000 Key Stage 2 Points Score and below above 39 See 61

62 Figure 5.7 Value added progress chart from the Autumn Package 29 <= Key Stage 2 Average Points Score <= 30 50% 40% 36% 30% 25% 20% 19% 10% 0% 7% 7% 2% 2% 3% 0% U G F E D C B A A* 2005 Average GCSE & Equivalent Grade Per Entry RAISEonline 40 will replace the Autumn Package and will contain data both on specific pupils as well as schools. At the time of writing the system is not yet complete, but will be available later in the autumn of The Autumn Package charts shown above will be part of RAISEonline, as will some of the new features and reports that featured in the Ofsted PANDAs compiled for each school for use during the 2005/06 academic year 41. This new information included both VA and CVA, for the schools as a whole and for pupil subgroups, with figures colour coded according to whether they are significantly above or below average. There are also graphical presentations: snake plots showing where the school CVA score fits in the national distribution (Figure 5.8), plots that compare the CVA with raw attainment (Figure 5.9) and plots showing CVA scores for subgroups within the school (Figure 5.10). RAISEonline will have CVA and VA scores for the last three years and shows whether these have increased or declined significantly. 40 For more information see 41 PANDAs for example schools are available from Ofsted here: 62

63 Figure 5.8 Example CVA snake plot Figure 5.9 Example plot comparing CVA and attainment Figure 5.10 Example chart showing CVA for pupil subgroups 63

64 This paper has not so far discussed the calculation of value added scores for subgroups within schools, as shown in Figure One option would have been to include the various subgroups as random effects in the model, e.g. allowing each school to have a different modelled effect for FSM pupils, or for boys, or for boys on FSM. However, this has so far been difficult to implement and so a simpler method was chosen which calculates the schools subgroup value added scores as averages relative to the national CVA model. The estimates are shrunk based on the size of the group and confidence intervals are calculated. The approach is therefore similar to the method for calculating the schools overall CVA scores. One disadvantage of this method is that at a national level, the school scores for some of the subgroups do not average out at 100 (or 1000 for Key Stage 4) - the addition of more school level factors would improve this fit. The general point this illustrates is that if a model is to be used for more detailed modelling than the provision of school scores, it may be necessary to increase its complexity. 5.4 Other uses of the value added scores Having described the two main vehicles for value added information, others can be discussed more briefly. For example, the School Profiles are another example of the presentation of value added data to parents. They have recently been launched and will be amended in light of initial feedback 42. The current plan is to include in them the Performance Tables value added information, presented as part of a snake plot like Figure 5.5. School value added data is also included in the London Families publication (DfES, 2006d). This is produced as part of the London Challenge policy and it defines families of similar schools based on prior attainment and deprivation, measured by FSM and IDACI. The book provides a range of comparative data for London schools, including their VA and CVA scores (no confidence intervals or value added charts are included). Value added has not so far been extensively used in allocating funding. One exception is the High Performing Specialists Schools policy 43. This identifies schools which are given additional funding to support work in providing assistance to neighbouring schools and in various special activities like a concentration on vocational learning or on pupils with special educational needs. The criteria that 42 See 43 See 64

65 need to be met are based on value added measures at different Key Stages, for the latest three years. The VA measures are benchmarked by school prior attainment, i.e. high performing schools need to have high VA relative to schools with similar levels of prior attainment. Some additional criteria are also included, e.g. a minimum absolute level of achievement in the most recent year. This policy illustrates the way that several different indicators need to be used in combination to define high performance. Value added scores can be used to identify schools for particular attention in internal discussion or for additional support from consultants in the National Strategies. In connection with this there has been discussion of various ways of segmenting the school population, e.g. into schools that are improving or declining in terms of value added. However, as it is not possible to isolate clear clusters of schools (due in part to the volatility of value added scores) any such segmentation would just be a first step towards further investigation. In England various initiatives have been targeted at groups of schools, e.g. the Specialist Schools programme, Leadership Incentive Grant, the Excellence in Cities policy and others. As noted in Section 2, value added scores can be used as information to monitor policy initiatives of this kind, although by themselves they do not measure the impact of the policy. In addition to providing information on overall value added, the school scores show how much between-school variation there is within the policy. Target setting in schools is an important part of the school improvement process in England. Targets are set in relation to test results rather than value added (targets would not be straightforward: since value added scores are calculated relative to the national average, not all schools can improve). Value added scores are not used directly to set targets but the value added approach, taking into account pupil prior attainment, underpins guidance on targets. Care is taken to encourage the setting of stretching targets for pupils, schools and Local Authorities that are not simple extrapolations of previous performance. There are various ways of doing this but the general approach has been to supply information on the sort of results that would be expected in the future if a school (for example) improved its value added to the level of similar schools, in terms of average prior attainment, that currently have high value added. This approach is being used in RAISEonline. There is an ongoing debate about the inclusion of more contextual variables in value added for target setting. 65

66 The risk of this is that low expectations may be built in for pupils that currently make less progress on average, e.g. pupils entitled to free school meals. On the other hand, schools with high prior attainment levels but few FSM pupils could be set more stretching targets if the FSM data were factored in. This section has discussed the use of value added scores for schools. There are many other ways of presenting linked data on progress between key stages, whether at national level, for schools or Local Authorities or for individual pupils. Moreover, the regression models used to calculate value added scores can be extended for a variety of purposes, e.g. to look at the value added for groups of schools or the impact of funding on results. Some of this analysis is done within the Department but an increasing amount of value added-related quantitative research is being undertaken by external experts and academics using the National Pupil Database. 66

67 References Note that DfES statistical bulletins and first releases are available on-line here: Web links here and in the main text were accessed in September and October Critchlow, J. and Coe, R. (2003) Serious flaws arising from the use of the median in calculating value added measures for School Performance Tables in England, Paper presented to the 29th International Association for Educational Assessment (IAEA) Annual Conference. ( DfE (1995) Value Added in Education: A Briefing Paper from the Department for Education. London: Department for Education DfEE (1998a) The Autumn Package, London: Department for Education and Employment DfEE (1998b) 1998 Value Added Pilot: Supplement to the Secondary School Performance Tables, London: Department for Education and Employment DfES (2002) Pupil Progress in Secondary Schools by School Type in England: 2001 Statistics of Education Bulletin 05/02. London: Department for Education and Skills DfES (2004), Variation in Pupil Progress 2003, Statistics of Education Bulletin 02/04. London: Department for Education and Skills DfES (2005), The Characteristics of Low Attaining Pupils, Statistics of Education Bulletin 02/05. London: Department for Education and Skills DfES (2006a), National Curriculum Assessments at Key Stage 2, and Key Stage 1 to Key Stage 2 Value Added Measures for England 2004/2005 (Final), Statistical First Release SFR22/2006 DfES (2006b), Trends in Attainment Gaps: 2005 Statistics of Education Bulletin. London: Department for Education and Skills DfES (2006c), GCSE and Equivalent Results and Associated Value Added 67

68 Measures in England 2004/05 (Final), Statistical First Release SFR26/2006 DfES (2006d), Families of Schools: May 2006 Secondary Schools, London: Department for Education and Skills DfES (2006e), Schools and Pupils in England: January 2006 (Final), Statistical First Release: SFR38/2006 Fitz-Gibbon, C.T. (1997) The Value Added National Project Final Report: Feasibility Studies for a National System of Value-Added Indicators. London: School Curriculum and Assessment Authority Godfrey, R. (2004). Changes in Ethnicity Codes in the Pupil Level Annual Schools Census London: Department for Education and Skills ( Goldstein, H. (2003) Multilevel Statistical Models: Third Edition. London: Arnold Goldstein, H., Rasbash, J., Yang, M., Woodhouse, G., Pan, H., Nuttall, D. & Thomas, S. (1993) A Multilevel Analysis of School Examination Results, Oxford Review of Education 19(4), Kirkup, C., Sizmur, J., Sturman, L. and Lewis, K. (2005) Schools Use of Data in Teaching and Learning, DfES Research Report 671. London: Department for Education and Skills Kreft, I. & De Leeuw, J. (1998) Introducing Multilevel Modelling. London, Thousand Oaks and New Delhi: Sage Publications Machin, S., Telhaj, S. & Wilson, J., The Mobility of English School Children, CEE Discussion Paper CEEDP0067. London: Centre for the Economics of Education Massey, A., Green, S., Dexter, T., and Hammet, L. (2003) Comparability of national tests over time: KS1, KS2 and KS3 standards between 1996 and Final Report to the QCA of the Comparability Over Time Project, London: Qualifications and Curriculum Authority Miliband, D. (2004) Personalised Learning: Building A New Relationship With 68

69 Schools, Speech By David Miliband, Minister Of State For School Standards, to the North Of England Education Conference, Belfast, 8th January 2004 ( Mortimore, P., Sammons, P., Stoll, L., Lewis, D. and Ecob, R. (1988) School Matters: The Junior Years. Wells: Open Books. National Audit Office (2003) Making a Difference: Performance of maintained secondary schools in England. London: The Stationery Office OECD (2004) Learning for Tomorrow s World: First Results from PISA Paris: Organization for Economic Cooperation and Development. O Donoghue, C., Thomas, S., Goldstein, G. & Knight, T. (1997), 1996 Study on Value Added for year olds in England, DfEE Research Studies RS52, London: Department for Education and Employment. QCA (1998) 1996 and 1997 LEA Benchmark Compendium. London: Qualifications and Curriculum Authority. Sammons, P., Thomas, S., Mortimore, P., Owen, C. & Pennell, H. (1994) Assessing School Effectiveness: Developing Measures to put School Performance in Context, London: Office for Standards in Education Saunders, L. (2000) Understanding schools use of value added data: the psychology and sociology of numbers, Research Papers in Education 15(3), pp SCAA (1994) Value Added Performance Indicators for Schools. London: School Curriculum and Assessment Authority Schagen, I. & Hutchison, D. (2003) Adding Value in Educational Research the marriage of data and analytical power, British Educational Research Journal, Vol. 29, No. 5, October 2003 SMSR (2005), Evaluation of the National Curriculum Tests: 2005 Test Evaluation: Key stage 2 English, mathematics and science. Final Report, report for the National Assessment Agency by Social and Market Strategic Research Ltd. ( 69

70 Thomas, S., Peng, W-J., Gray, J. (2007, in press) Value added trends in English secondary school performance over ten years, Oxford Review of Education, 33 (3) Tymms, P. & Coe, R. (2003) Celebration of the Success of Distributed Research with Schools: The CEM Centre, Durham, British Educational Research Journal, Vol. 29, No. 5, October 2003 Tymms, P. and Dean, C. (2004), Value Added in the Primary School League Tables, A Report for the National Association of Head Teachers, May Durham: CEM Centre, University of Durham Webber, R. and Butler, T. Classifying pupils by where they live: how well does this predict variations in their GCSE results?, CASA Working Paper Number 99. London: Centre for Advanced Spatial Studies, University College London. 70

71 Annex A The National Curriculum and Key Stage tests The English value added models relate to the Key Stages described below. The national curriculum and Key Stage tests are maintained by the QCA (Qualifications and Curriculum Authority) 44. Independent schools do not need to operate the national curriculum or the Key Stage tests. Foundation stage (age 3-5) This covers children in nurseries and the reception year at primary school. In 2002 the national curriculum was extended to cover this age group with six areas of learning 45. The Foundation Stage Profile was introduced into schools and settings in 2002/3, with 13 summary scales covering the six learning areas. Attainment on these scales is assessed by the teachers for each child receiving government-funded education by the end of the pupil s time in the foundation stage. Key Stage 1 (age 5-7) This covers Year 1 and Year 2 in primary schools, with pupils assessed at the end of Year 2 when most are 7 years old. The national curriculum specifies learning across a range of subjects such as history, art and information technology, but the three core subjects are English, mathematics and science. Pupils take tests in reading, writing and mathematics, but since 2005 these tests have only been used to inform overall teacher assessments the marks have not been collected centrally. Key Stage 2 (age 7-11) Twice the length of Key Stage 1, this takes pupils from Year 3 to Year 6, up to the age of 11 which is usually seen as the end of primary education : the following year most pupils in maintained schools move to secondary schools. At the end of the four years pupils are assessed by teachers and take tests in 44 For more information see 45 These are: personal, social and emotional development; communication, language and literacy; mathematical development; knowledge and understanding of the world; physical development; creative development. 71

72 English, mathematics and science. Key Stage 3 (age 11-14) This covers the first three years of secondary schooling (Year 7 to Year 9). Again there is a national curriculum across a range of subjects, with teacher assessment and tests in English, mathematics and science. Key Stage 4 (age 14-16) This covers the final period (Year 10 and 11) of compulsory schooling during which pupils are working towards a range of academic and vocational qualifications, partly assessed via coursework. Most of the assessment is at the end of Year 11. The qualifications are set by various independent awarding bodies. The main qualification is the GCSE there are currently over forty academic subjects on offer and eight vocational subjects. However there are also a very wide range of other qualifications which can be taken by this age group and the QCA has for which the QCA has established equivalences (i.e. established that a given qualification is worth, say, half of a GCSE). The number of subjects taken by pupils will vary (up to eight subjects are included in the value added models discussed here). Key Stage 5 or Post-16 After the age of 16 many, but not all, pupils stay on in full-time education. Those that do, study a range of academic and vocational qualifications, both in schools and separate colleges. The main academic qualification taken after two years (i.e. mainly by 18 year olds) is the A-level; the AS level is similar but is equivalent to half an A-level. 72

73 Annex B MLM residuals and the shrinkage factor Consider a simple two level model as in (1) below. The y ij could be the capped GCSE results for the ith pupil in the jth school, as in the DfES CVA model. For simplicity, the example here has just has one explanatory variable, x ij, which could, for example, be the Key Stage 2 prior attainment average point score used in the median method approach. The unexplained variance is partitioned between, u 0 j, for schools, and e 0 ij, for each pupil. yij = 0 +! 1xij + u0 j + e0ij! (1) The standard OLS approach assumes no systematic variance at school level, i.e. it assumes that there is no need for the 2 u 0 j term because! u0 is zero. Hence with OLS, the true unexplained variance in results is estimated with the simple raw residuals, which are the differences between observed and predicted results: r = y "! ˆ "! ˆ x ij ij ( 0 1 ij ) Since in OLS all schools are treated as if they were simply the sum of their pupils rather than having a different systematic effect, the value added scores would be the mean of these r ij residuals, which can be written r j. In the MLM context we want to estimate not just the overall residual for each pupil, but the component school and pupil parts on the basis that! is not zero. To do this, MLM uses an iterative process to produce maximum likelihood estimators for both the regression coefficients and the variance of the residuals at school and pupil level. In this process, the rj figures do not provide efficient estimators for level residuals. If instead the 2 u0 uˆ 0 j, the school uˆ 0 j are defined as a function of the raw residuals u ˆ = c 0 j r j it can be shown that the expected difference between the true and estimated school 73

74 level residuals, E! is actually minimised when 2 ( u ˆ0 ) j u0 j! = (2) 2 u0 c 2 2!! e0 u0 + n j The solution to the multilevel model provides sample estimates for these variance terms 2 2! u0 and! e0. Hence it is possible to calculate a constant c j for each school. The multilevel school residuals uˆ 0 j are equal to the raw residuals r j that would be obtained from the model s estimated fixed effects, adjusted by the values of c j. The constant that the c j is called the shrinkage factor because it is bounded by 0 and 1, so uˆ 0 j residuals will be smaller (either negative or positive) than the r j. As the size of the school, increases, c j becomes closer to 1. In other words, for large schools, the MLM and OLS value added residuals will be similar. For small schools on the other hand, the shrinkage factor will have an impact. And the impact is larger when the within-school variation is large relative to the between-school variation (which is the case for the CVA model discussed in Section 4.2). n j, 74