A technical guide to 2014 key stage 2 to key stage 4 value added measures

Transcription

1 A technical guide to 2014 key tage 2 to key tage 4 value added meaure CONTENTS Introduction: PAGE NO. What i value added? 2 Change to value added methodology in Interpretation: Interpreting chool value added core 5 Interpreting pupil group value added core 7 Methodology: Calculating key tage 4 attainment 8 Calculating key tage 2 prior attainment 10 Calculating pupil value added core 13 Calculating chool value added core 15 Calculating pupil group value added core 17 Annex A 2014 amended attainment etimate 20 Annex B Interpreting chool value added core 21 Annex C Interpreting chool value added core for pupil group diadvantaged pupil 22 1

2 WHAT IS VALUE ADDED? The purpoe of a value added meaure i to etimate how individual pupil progreed in relation to their peer from the beginning to the end of their compulory econdary education. Thi i ued for comparing progre of pupil between chool. Thi document detail how value added meaure are calculated at pupil level and how thee are ued to calculate aggregate meaure for pecific pupil group and at chool level (a publihed in the performance table). Guidance i alo given on how to interpret thee meaure. Pupil attainment at end of key tage 4 i chiefly affected by two key factor: pupil attainment at the end of key tage 2 and the impact of their econdary chool up to the end of key tage 4. A value added meaure aim to etimate the effect of the econdary chool on the pupil attainment taking their key tage 2 reult into account. Value added meaure are etimated for each individual pupil by comparing their key tage 4 reult with all other pupil with imilar key tage 2 reult 1. The difference between a pupil actual key tage 4 performance and their etimated key tage 4 performance give the pupil their value added core. Each individual pupil value added core i relative to the performance of other pupil. Naturally ome pupil will progre more than other independently of which chool they attend, and a chool value added core would certainly be different if they had a different cohort of pupil. Confidence interval are provided a a proxy for a range within which you can be 95% certain the true value added core lie. Thee interval hould be taken into account when making comparion between chool, group or national average. Therefore, the chool value added meaure that are publihed in the performance table are preented alongide the repective 95% confidence interval. School value added meaure etimate the effect of the econdary chool on all of their pupil attainment. A ummary diagram on page 21 how how to interpret thee aggregated core. 1 Calculated a a key tage 2 fine level, ee page 20 for detail on thi calculation and grouping 2

3 For key tage 4, value added core the following meaure are calculated: Key tage 2 to key tage 4 bet 8 plu Englih and mathematic bonu value added meaure The bet 8 plu Englih and mathematic bonu value added meaure etimate how pupil perform in their bet 8 GCSE (or equivalent qualification) with pupil receiving an additional bonu for their performance in GCSE Englih and mathematic, a the point core for thee ubject are counted twice. Key tage 2 to key tage 4 Englih Baccalaureate (EBacc) value added meaure Value added core are alo calculated for the five EBacc ubject area: Englih mathematic cience humanitie language A pupil ha a eparate etimated key tage 4 outcome calculated for each of the five EBacc ubject area. Thee etimate are then compared againt a pupil bet core in each EBacc ubject area. For example, if a pupil achieved a B in GCSE geography and a C in GCSE hitory then it i the GCSE geography reult (the pupil bet reult in the EBacc humanitie ubject area) which i ued to compare againt the etimated key tage 4 outcome for the EBacc humanitie ubject area value added meaure 2. A pupil then ha a value added core calculated for each ubject area by finding the difference between their actual key tage 4 attainment in the ubject area and their etimated key tage 4 attainment in the ubject area. All pupil are included in the Englih and mathematic ubject area value added meaure. However, only the pupil that have taken the required qualification at the end of key tage 4 are included in the cience, language and humanitie ubject area value added meaure. 2 The bet core in each EBacc ubject area i taken after uual dicounting rule have been applied. See page 8 for further guidance on dicounting 3

4 CHANGES TO VALUE ADDED METHODOLOGY IN 2014 Due to the implementation of the Wolf and early entry reform 3, which changed the way in which qualification are counted in the 2014 performance table, the ditribution of the key tage 4 average point core (APS) changed. A a reult, the exiting modelling approach no longer provided a good a fit to the data, which meant biae in value added core would have been introduced for pupil with very high and very low prior attainment. Therefore, to remove thee model biae, which may impact on chool value added core, a revied and implified value added model ha been developed to better repreent the progre pupil make. Thi revied model ue a impler mean line methodology baed upon analying pupil performance at each prior attainment band eparately. Thi ha the benefit of both removing unwanted bia in the methodology and implifying the model o it can be more readily undertood. See Annex A for the attainment etimate for each prior attainment band. Uing the mean line methodology the value added hrinkage factor adjutment i no longer applied. The hrinkage factor wa previouly applied to provide better value added etimate for chool with a mall number of pupil. Thi wa calculated uing the within chool and between chool variance that were routinely etimated a part of the previou methodology. However, on reflection including the hrinkage factor did not make enough of an impact to warrant the complexity it brought to the methodology. To promote implicity, thi will no longer be applied. Pleae ee page 13 and 15 for further detail of how pupil and chool value added core are calculated under the mean line methodology. 3 Further detail on thee reform i provided 4

5 INTERPRETING SCHOOL VALUE ADDED SCORES When evaluating a chool performance we mut be careful to note that it i baed on a given et of pupil' reult. A chool could have been equally effective and yet the ame et of pupil might have achieved lightly different key tage 4 reult, and the chool would almot certainly have hown different key tage 4 reult with a different et of pupil. Thi element of uncertainty need to be taken into account when interpreting a chool value added core; 95% Confidence Interval are provided a a proxy for a range in which you can be 95% certain the true value added core lie. A chool confidence interval i alway centred on the chool value added core. For example, if a chool value added core i 1,010 and the ize of the chool confidence interval i 5 point, then the confidence interval range between 1,005 and 1,015 (ie 5 point either ide of the chool value added core). The ize of a confidence interval i determined by the number of pupil in the chool at the end of key tage 4 and the tandard deviation. In thi cae the tandard deviation ued i at the national level for all pupil o the number of pupil at the chool i the only influencing factor. Smaller chool have wider confidence interval becaue their value added core i baed on a maller number of pupil, o there i le evidence on which to judge the chool effectivene. To judge a chool effectivene, both the chool value added core and the aociated confidence interval need to be taken into account. If the whole range of the confidence interval i above 1,000 (ie the lower confidence limit i greater than 1,000), we can ay the chool core i above the national average and i tatitically ignificant, and we can be confident the chool i helping it pupil make better than average progre. An illutration of how to interpret chool value added core i given in Annex B. Similarly, when the entire range of the confidence interval i below 1,000 (ie the upper confidence limit i le than 1,000), we can ay the chool core i below the national average and i tatitically ignificant. Finally, if the confidence interval traddle the national average of 1,000, then we can ay that the chool i not ignificantly different from the national average, in other word, we cannot confidently ay that the chool value added core i definitely above or definitely below the national average. The table and diagram overleaf how how a chool value added core and confidence interval hould be interpreted to reach one of the three definition above. School A i an example of a chool that i ignificantly above national 5

6 average; chool B i not ignificantly different from national average; and chool C i ignificantly below national average. School A School B School C School VA Score 1,010 1, Upper Confidence Interval 1,015 1, Lower Confidence Interval 1, For more information on the calculation of confidence interval, pleae ee the ection, calculating pupil value added core. Comparion of EBacc ubject area value added core Confidence interval mut alo be taken into account when comparing two or more EBacc ubject area value added core within a chool (eg when comparing a chool EBacc Englih value added core with their EBacc humanitie value added core). The ize of the confidence interval for each of the five EBacc value added meaure for a chool will vary in ize (a they are baed on different model and different number of pupil) each interval provide a proxy for the range of value within which we are tatitically confident that the chool true value added core for the repective EBacc ubject area lie. 6

7 INTERPRETING PUPIL GROUP VALUE ADDED SCORES When comparing pupil group value added core, it i important to note that Thee core are baed on a given et of pupil reult (who belong to a pupil group, for example diadvantaged pupil or a pecific prior attainment band) for a particular tet paper on a particular day. Confidence interval are alo calculated to give a proxy for a range of core within which we are tatitically confident that a chool pupil group value added core will fall. There are two way in which a pupil group value added core can be compared; to the national average for all pupil (1,000) or the national pupil group average. For an explanation of how to interpret confidence interval, pleae refer back to page 5 (Interpreting School Value Added Score) and an illutration of how to interpret pupil value added core i alo given in Annex C. 7

8 CALCULATING KEY STAGE 4 ATTAINMENT Qualification included in the meaure Bet 8 4 value added i baed on pupil performance capped to eight entrie. The 2014 bet 8 core follow the new performance table rule; a. a reduced number of eligible qualification are included compared to previou year 5 b. the ize of each qualification ha been capped at a maximum of one GCSE per entry c. a limit of two non-gcse qualification that contribute toward the meaure and; d. early entry rule 6 apply, in which only a pupil firt entry in an EBacc ubject i counted rather than their bet entry. The qualification included are alo ubject to uual performance table dicounting rule. Further detail i available here: ance_for_school_2014.pdf ubject value added meaure include only reult in qualification that qualify for the EBacc. Further information on which qualification count toward each ubject area can be found here: Key tage 4 point core A pupil performance in their bet 8 qualification (plu Englih and mathematic bonu) continue to be expreed a a point core with a maximum of 580 point available. Subject point core calculation alo remain the ame a before with value up to 58 point a hown in table 1 below. 4 Bet 8 value added core i baed on a pupil bet 8 qualification once early entry rule have been applied. 5 Further information on qualification that are included in the 2014 table can be found here: 6 Further information on early entry rule can be found in the dicounting and qualification ection of the performance table webite. 8

9 Table 1 Qualification point core Qualification Point core GCSE grade A* 58 GCSE grade A 52 GCSE grade B 46 GCSE grade C 40 GCSE grade D 34 GCSE grade E 28 GCSE grade F 22 GCSE grade G 16 Further information on point core can be found here: ument.pdf 9

10 CALCULATING KEY STAGE 2 PRIOR ATTAINMENT A pupil average performance in key tage 2 Englih, mathematic and cience tet will continue to be ued a a baeline to compare pupil progre. The average key tage 2 point core are converted into a key tage 2 fine level. The key tage 2 fine level i a imple converion of dividing a pupil average key tage 2 point core by 6 and then rounding to 1 decimal place. Worked Example Samantha mark in her key tage 2 tet (taken in 2009) were 77, 76 and 74 in Englih, mathematic and cience repectively. The diagram below et out how thee are converted into an average key tage 2 fine level to be ued a the prior attainment input into value added meaure. KS2 SUBJECTS KS2 ENGLISH TEST OUTCOME KS2 MATHEMATICS TEST OUTCOME KS2 SCIENCE TEST OUTCOME Tet mark for each ubject Tet mark converted to fine point TEST LEVEL 5 TEST LEVEL 4 TEST LEVEL 5 KS2 average point core Converted to a KS2 fine level 5.3 For Englih, mathematic and cience aement where a level of 3 to 5 i awarded in the tet, the fine grade i calculated by: Baic level + actual tet mark bottom of level threhold top of level threhold bottom of level threhold + 1 Uing the Englih tet mark above, 77 i a level 5 a it i between 67 and 100 which were the level 5 threhold for The fine grade i therefore calculated a follow: 10

11 The fine point are calculated by multiplying the fine grade by 6: 5.29*6=31.74 The ame calculation are applied to the mathematic and cience tet outcome and the average of the three ubject i calculated to give an average point core of Thi i then converted to a fine level by dividing by 6 and rounding to 1 decimal place to give a fine level of 5.3. Adjutment are made for pupil with incomplete tet reult: if a pupil doe not have a tet reult in a ubject then their teacher aement level i ued (ee table below) if a pupil ha a reult miing in one or more ubject, then prior attainment i calculated from the remaining ubject() if a pupil doe not have a tet core or teacher aement reult in any ubject then they are excluded from the meaure Table 2 teacher aement adjutment If tet level = 3-5 Ue pupil fine point core 2 If teacher aement available, ue; Award: W = 3 Level 1 = 9 Level 2 = 15 Any higher = ue pupil fine point core A,D,F,L,P,Z = Exclude pupil If no teacher Exclude Pupil aement available B, N If teacher aement available, ue; If no teacher aement available Award: W = 3 Level 1 = 9 Level 2 = 15 Any higher = 15 (capped) A,D,F,L,P,Z = Exclude pupil Exclude Pupil 11

12 A, M, Q, S, T, X If teacher aement available, ue; If no teacher aement available Award: W = 3 Level 1 = 9 Level 2 = 15 Level 3 = 21 Level 4 = 27 Level 5 = 33 Any higher = 33 (capped) A,D,F,L,P,Z = Exclude pupil Exclude Pupil Note on grade code A Abent B Working below the level of the tet D Diapplied F Key tage 2 pupil not at end of key tage 2 and taking thi ubject in future year L Left N Not awarded a tet level M Miing P Reult for ubject found in previou year dataet S Pending maladminitration Q Maladminitration T Working at the level of the tet but not able to acce them W Working toward level 1 X Lot Z Ineligible 12

13 CALCULATING PUPIL VALUE ADDED SCORES Individual pupil value added core are calculated firt before a chool value added core can be produced. The firt tep i to calculate a key tage 4 etimated outcome for all pupil that are at the end of key tage 4 in Thi i baed on the actual key tage 4 outcome of all pupil nationally with the ame level of achievement at key tage 2 (prior attainment). A pupil prior attainment i defined a the average of their key tage 2 Englih, mathematic and cience reult, in fine level. A pupil value added core i the difference between their etimated and actual key tage 4 outcome. Worked Example Samantha ha an average key tage 2 point core of giving a fine level of 5.3. Her bet 8 plu Englih and mathematic bonu core i 508. The national average bet 8 plu Englih and mathematic bonu core for pupil who hare the ame key tage 2 reult a Samantha i Samantha value added core i the difference between her actual bet 8 core and the etimated bet 8 core, that i, = In addition, a key tage 2 to key tage 4 pupil level value added ready reckoner can be found in the Methodology and technical guide ection of the performance table webite. Thi provide the option for uer to input data and tet different cenario. Pupil in pecial chool The etimated key tage 4 attainment for pupil in pecial chool i baed on a comparion with pupil of the ame prior attainment in maintream chool. 13

14 Thi mean that their value added core are calculated baed on average derived from maintream chool only. Similarly, confidence interval in pecial chool and their pupil group are calculated uing the value from the maintream pupil population. Thi approach enable pecial chool to compare the progre their pupil make relative to thoe in maintream chool. Pupil eligibility for incluion in value added model Pupil are included in the bet 8 value added, EBacc Englih and E Bacc mathematic model if: their key tage 4 attainment can be matched to their attainment at key tage 2 they have a key tage 2 average point core that i greater than zero; they do not have a diregarded outcome in all three key tage 2 tet / teacher aement; they attend a maintained maintream chool (including academie and city technology college) Further pupil eligibility criteria exit in the cae of EBacc cience, humanitie and language value added meaure: they mut have completed a coure of tudy in eligible ubject() within each repective ubject area, ie have entered the ubject Note: ubject entry i not a pre-requiite for incluion in EBacc Englih and mathematic value added meaure. All maintained maintream and pecial chool will have a value added core for all ix key tage2 to key tage4 value added meaure, provided they have at leat one eligible pupil for each meaure. 14

15 CALCULATING SCHOOL VALUE ADDED SCORES The chool value added core i the mean average of it pupil value added core. Worked Example continued Samantha i one of 142 pupil in her chool key tage 4 cohort, Pupil # Pupil name Value added core 1 Samantha George Jame Sum +1, The chool bet 8 value added core i therefore /142 = For preentation purpoe, thi core i added to 1000 and rounded to one decimal place. A uch, thi chool would have a bet 8 value added core of A 95% confidence interval i calculated around each chool value added core, defining a proxy for the range of value within which we are tatitically confident that the true value of the progre core for the chool lie. The confidence interval, denoted LowCI, UppCI, i given by the formula: where: LowCI, UppCI VA CI, VA CI, LowCI UppCI VA CI i the lower confidence limit for the chool value added core i the upper confidence limit for the chool value added core i the chool value added core i the ize of the confidence interval for the chool value added core 15

16 where: CI N n 1.96 i the critical value for a 95% confidence interval; N i the tandard deviation of the value added core for all eligible pupil nationally; n i the number of eligible pupil that belong to the chool The national average value added core of all maintained maintream chool core will be 0. When a chool ha their lower confidence interval limit higher than 1,000 ( LowCI > 1,000), the chool value added core i above average and the reult i tatitically ignificant. When a chool ha their upper confidence interval limit lower than 1,000 ( UppCI < 1,000), the chool value added core i below average and the reult i tatitically ignificant. When the confidence interval pan 1,000 ( LowCI < 0 < UppCI ), we cannot ay with confidence whether the chool value added core i above or below average, and ay the reult i not tatitically ignificant. Worked example continued We can calculate the ize of the confidence interval for the chool bet 8 value added core uing CI : We derive the confidence interval for the chool bet 8 value added core: = [ , ] = [1000.5, ] Hence, a LowCI > 1,000 we can ay with confidence that thi chool bet 8 value added core i above average and i tatitically ignificant. 16

17 CALCULATING PUPIL GROUP VALUE ADDED SCORES The value added core for a pupil group (eg diadvantaged pupil) in a chool i calculated a the average value added core of all pupil that belong to that group in the chool. Similarly, the value added core for a group nationally i the average value added core of all pupil in maintream chool that belong to that group nationally. Worked Example continued Samantha i one of 30 diadvantaged pupil (defined, a pupil who are either eligible for free chool meal or are children who are looked after) among the 142 pupil in her chool key tage 4 cohort, who gain a range of bet 8 value added core: Diadvantaged pupil # Diadvantaged pupil name Value added core 1 Samantha George Alion Sum The diadvantage pupil group value added core for the chool i therefore /30 = For preentation purpoe, thi core i added to 1,000 and rounded to one decimal place. A uch, thi chool would have a value added core of for it diadvantaged pupil. More generally, the pupil group key tage 2 to key tage 4 value added core for any chool, VA g, i given by: VA 1000 g VA pg, 17

18 where: VA pg i the average value added core for all eligible pupil that belong to the pupil group within the chool, given by: VA pg npg p 1 n VA pg p, where: n pg p1 n pg VA p i the number of eligible pupil that belong to the pupil group within the chool i the um of the value added core of eligible pupil that belong to the pupil group within the chool We calculate the diadvantaged pupil group value added core for the chool, VA g, by calculating the average value added core of the diadvantaged pupil within the chool, a follow: VA g 1000 VA pg 1000 npg p1 n VA pg p (to 3 d.p.) Note: We would publih thi core a , but retain the decimal place for thi example for illutrative purpoe for the confidence interval calculation. 18

19 Pupil group confidence interval Pupil group confidence interval can be calculated in the ame way a thoe around chool value added meaure. For example if looking at diadvantaged pupil you would replace with the number of diadvantaged pupil within the chool. The national tandard deviation i till ued. n 19

20 Annex A 2014 AMENDED ATTAINMENT ESTIMATES The etimated attainment core i the average core of all pupil nationally with the ame prior attainment at key tage 2. The following table how the key tage 4 etimate for each key tage 2 average fine level, baed on the 2014 amended data. KS2 average fine level Bet 8 plu E&M bonu etimate EBacc Englih etimate EBacc math etimate EBacc cience etimate 7 EBacc humanitie etimate 6 EBacc language etimate 6 <= >= There i ome volatility in the lowet prior attainment band for the cience, humanitie and language value added model, thi i due in part to a low number of pupil and i conitent with previou year' model. 20

21 Standard Deviation

22 Annex B INTERPRETING SCHOOL VALUE ADDED SCORES Example of chool value added information Diplaying a chool value added information viually on a chart How to interpret the information and chart Intitution Name VA meaure baed on progre between Key Stage 2 and Key Stage 4 (centred around 1,000) Limit of Key Stage 2 to 4 VA Confidence Interval Upper Lower School A 1, , ,011.1 School B , School C School A 1,000 [NATIONAL AVERAGE SCHOOL VA SCORE] 1, , ,011.1 School A VA core i above the national average and thi i tatitically ignificant Thi i becaue the whole range of the confidence interval i above 1,000 Thi tell u that the pupil in thi chool make more progre than average The higher the chool VA core, the more progre the pupil in the chool are making, with 1,000 repreenting the national average progre. Confidence interval then allow u to ae whether the chool VA core i ignificantly above or below the national average KEY: Upper Confidence Limit School VA Score Lower Confidence Limit School B 1,000 [NATIONAL AVERAGE SCHOOL VA SCORE] School C 1,000 [NATIONAL AVERAGE SCHOOL VA SCORE] 1, School B VA core i not ignificantly different from the national average Thi i becaue the range of the confidence interval traddle the national average of 1,000 Thi tell u that the pupil in thi chool make progre comparable with the average School C VA core i below the national average and thi i tatitically ignificant Thi i becaue the whole range of the confidence interval i below 1,000 Thi tell u that the pupil in thi chool make le progre than average

23 Annex C - INTERPRETING SCHOOL VALUE ADDED SCORES FOR PUPIL GROUPS DISADVANTAGED PUPILS Example of a School VA Information from Performance Table for diadvantaged pupil Diplaying a School VA Information for Diadvantaged Pupil Viually on a Chart How to Interpret the Information and Chart Diadvantaged Pupil Bet 8 VA meaure - diadvantaged pupil Bet 8 VA Other lower example 95% confidence of poible limit for outcome diadvantaged pupil Bet 8 VA upper 95% confidence limit for diadvantaged pupil Other example of poible outcome NATIONAL AVERAGE SCHOOL VA SCORE = 1,000 NATIONAL AVERAGE VA SCORE FOR DISADVANTAGED PUPILS The VA core for diadvantaged pupil i above the national average for all pupil and the national average for diadvantaged pupil and both reult are tatitically ignificant. Thi i becaue the whole range of the confidence interval i above 1,000 and the national average for diadvantaged pupil. Thi tell u that diadvantaged pupil in thi chool make more progre than average for all pupil and diadvantaged pupil nationally KEY: Upper Confidence Limit School VA Score ALL PUPILS DISADVANTAGED BELOW NO SIG ALL PUPILS DISADVANTAGED NO SIG ABOVE ALL PUPILS DISADVANTAGED NO SIG BELOW ALL PUPILS DISADVANTAGED ABOVE NO SIG Lower Confidence Limit National Average VA Score for Diadvantaged Pupil ALL PUPILS DISADVANTAGED BELOW BELOW ALL PUPILS DISADVANTAGED NO SIG NO SIG ALL PUPILS DISADVANTAGED ABOVE BELOW ALL PUPILS DISADVANTAGED BELOW ABOVE NO SIG ABOVE BELOW Not Significantly Different Above average and thi i tatitically ignificant Below average and thi i tatitically ignificant Note: VA core are alo available for non-diadvantaged pupil and low, middle and high attainer Guide to KS2-4 Value Added

24 Guide to KS2-4 Value Added