A GUIDE TO LEVEL 3 VALUE ADDED IN 2013 SCHOOL AND COLLEGE PERFORMANCE TABLES

Transcription

1 A GUIDE TO LEVEL 3 VALUE ADDED IN 2013 SCHOOL AND COLLEGE PERFORMANCE TABLES Cotets Page No. Summary Iterpretig School ad College Value Added Scores 2 What is Value Added? 3 The Learer Achievemet Tracker 4 Which studets are icluded i Level 3 Value Added ad how is their attaimet calculated? Calculatig Studet Value Added scores for Idividual Qualificatios 8 Calculatig School ad College Value Added scores for Idividual Qualificatios Calculatig School ad College Value Added scores for Qualificatio Types Calculatig School ad College academic ad vocatioal Value Added scores Iterpretig School ad College Value Added scores 14 Whe ad Where will Level 3 Value Added be published 16 Other Useful Resources 17 Aex A What s ew i Level 3 Value Added for 2012/13? 18 Techical Aex - Cotets

2 SUMMARY INTERPRETING SCHOOL AND COLLEGE VALUE ADDED SCORES Example of how a school or college's value added might be displayed i a table Displayig a school or college's VA visually o a chart How to Iterpret the iformatio ad chart Istitutio Name VA measure based o progress betwee Key Stage 4 ad Level 3 Qualificatios (measure cetred aroud zero ad expressed i grades) Limit of Level 3VA Cofidece Itervals Upper Lower School A School B School C School A NATIONAL AVERAGE SCHOOL VA SCORE = ZERO School A's VA score is above the atioal average ad this is statistically sigificat. This is because the whole rage of the cofidece iterval is above zero This tells us that studets i this school make more progress tha average The higher a school or college's VA score i a qualificatio type, the more progress the studets i the school or college are makig, with zero represetig the atioal average. Cofidece itervals the allow us to assess whether the school or college's VA score is sigificatly above or below the atioal average College B NATIONAL AVERAGE SCHOOL VA SCORE = ZERO College B's VA score is ot sigificatly differet from the atioal average. This is because the rage of the cofidece iterval straddles the atioal average of zero This tells us that studets i this college make progress comparable with the average KEY: Upper Cofidece Limit School VA Score Lower Cofidece Limit School C NATIONAL AVERAGE SCHOOL VA SCORE = ZERO School C's VA score is below the atioal average ad this is statistically sigificat. This is because the whole rage of the cofidece iterval is belowzero This tells us that studets i this school make less progress tha average 2

3 WHAT IS VALUE ADDED? Whe measurig how effective a school or college is, it is importat to look at how well its studets perform i their tests ad examiatios. However, whe evaluatig examiatio performace, it is also importat to take ito cosideratio that whe studets first joi a school sixth form or college, they have varyig levels of ability, i.e. studets have may differet startig poits. So a measure is eeded that looks at how much progress studets have made from the begiig to the ed of their Level 3 qualificatio (i.e. betwee the ed of Key Stage 4 ad the completio of their Level 3 qualificatio). This is the purpose of Value Added (VA). Aalysis shows that there is a very strog relatioship betwee examiatio performace of studets at the ed of the previous Key Stage ad curret Level 3 qualificatios. A VA measure uses this relatioship to estimate how well all studets atioally, perform i the curret Level 3 qualificatios they have opted to study. For Level 3 Value Added, a idividual studet s estimated outcome i a Level 3 qualificatio is calculated by lookig at the actual performace of all studets atioally that studied the same Level 3 qualificatio ad that demostrated similar ability i their exams at the ed of KS4. More specifically, we estimate a studet s Level 3 qualificatio outcome as the average poits achieved by studets atioally i this qualificatio of similar ability at KS4. This estimated outcome i the Level 3 qualificatio ca the be compared agaist what the studet actually achieved i their Level 3 qualificatio, to see whether or ot they exceeded their estimate. The differece betwee a studet s actual performace ad their estimated performace gives the studet a Value Added score i the Level 3 qualificatio. The VA score of all studets studyig a particular Level 3 qualificatio i a school or college ca the be averaged to fid the school or college s VA score i the Level 3 qualificatio. This score is used to idetify the schools ad colleges that are helpig their studets make more or less progress tha average. The summary diagram o page 2 shows how to iterpret these scores. Value added is expressed as a proportio of a grade: as a proportio of oe A Level grade for a academic qualificatio, ad as a proportio of a BTEC Level 3 Subsidiary Diploma grade for a vocatioal qualificatio. 3

4 THE LEARNER ACHIEVEMENT TRACKER Prior to the 1 st April 2012, the Youg People s Learig Agecy (YPLA) was resposible for deliverig ad publishig the Learer Achievemet Tracker (LAT) Value Added measure. However, o the 1 st April 2012, the YPLA coverted to become the Educatio Fudig Agecy ad the resposibility for producig the LAT passed to the Departmet for Educatio (DfE). Followig the hadover of resposibility for the LAT, the DfE have made some chages to the way that the Value Added iformatio is preseted ad to how schools ad colleges access their Value Added data. To reflect these chages, the DfE have decided to re-ame the measure as Level 3 Value Added. All documetatio from the 1 st April 2012 owards ow refers to the measure by this ame. Ay referece to LAT relates to Value Added data published prior to April 1 st 2012 by the YPLA. To see a example of how iformatio will be preseted i Level 3 Value Added, please see the Ready Reckoer ad the Ready Reckoer User Guide o the Level 3 Value Added webpage (see the Other Useful Resources sectio of the documet for a lik to the webpage) If you would like to fid out more iformatio about how to access Level 3 Value Added data, please see the Whe ad Where will Level 3 Value Added be published sectio of this documet. 4

5 WHICH STUDENTS ARE INCLUDED IN LEVEL 3 VALUE ADDED AND HOW IS THEIR ATTAINMENT CALCULATED? Level 3 Value Added is a measure that looks at the progressio studets make betwee Key Stage 4 ad the ed of their Level 3 qualificatio. The measure covers all studets that: Were aged 16, 17 or 18 at the begiig of the Academic Year (31 st August) Have completed a qualificatio approved for Level 3 Value Added Have results at the ed of Key Stage 4 Are i a School or College with a valid istitutio idetifier How to calculate each Studet s Key Stage 4 startig poit The startig poit for Level 3 Value Added is the average attaimet of the studet at the start of their Level 3 qualificatio, also kow as prior attaimet. For studets aged 17 ad 18, this is calculated as the average of their attaimet at Level 2 ad below (or Key Stage 4) i exams take up to two years before their outcome i the Level 3 qualificatio. For studets aged 16, this is calculated as the average of their attaimet at Level 2 ad below (or Key Stage 4) i exams take up to oe year before their outcome i the Level 3 qualificatio. This is summarised i the table below: Example Age of Studet* Eligible Qualificatios for a Studet's Prior Attaimet at Key Stage 4 A 18 Level 2 qualificatios ad below up to age 16 (that is two years before) B 17 Level 2 qualificatios ad below up to age 15 (that is two years before) C 16 Level 2 qualificatios ad below up to age 15 (that is oe year before) *Age at the begiig of academic year i which the Level 3 qualificatio was completed Each studet s Key Stage 4 outcome is calculated by addig together the total poits for their applicable Key Stage 4 qualificatios ad dividig the total through by the combied size of the qualificatios. For example, GCSEs are equivalet to a size of 1 i the size system admiistered by the Teachig Agecy. All other qualificatios are sized relative to oe GCSE. Examples of how to calculate Key Stage 4 average poit scores are show below: Example A: Studet A takes te GCSEs at age 16 ad progresses to complete a Level 3 qualificatio at age 18. The studet s prior attaimet is calculated as the total poit score i the te GCSEs divided by a volume of 10 (each GCSE is equivalet to a size of 1). 5

6 Example B: Studet B takes four GCSEs ad a OCR Level 2 Natioal Certificate at age 15; the studet the goes o to take a Level 3 qualificatio at age 17. The studet s prior attaimet is the total poit score i the four GCSEs ad the OCR Level 2 Natioal Certificate divided by a volume of 8 (that is, four GCSEs with a size of 1 ad oe OCR Level 2 Natioal Certificate with a size of 4). Example C: Studet C attais two Bs, three Cs ad oe U i GCSEs at age 15 ad a merit i a OCR Level 2 Natioal Certificate at age 15; studet C the goes o to take a Level 3 qualificatio at age 16. The studet s prior attaimet is the total poit score i the six GCSEs ad the OCR Level 2 Natioal Certificate (2 x 46) + (3 x 40) + (1 x 0) + (1 x 196) = 408 divided by a volume of 10 (six GCSEs with size 1 ad oe OCR Level 2 Natioal Certificate with size 4). This gives a average poit score of (408 / 10) = How to measure achievemet i Level 3 qualificatios for Value Added As with the calculatio of prior attaimet, a poit score system is used to assig poits to grades for Level 3 qualificatios. However, the iclusio of fails i the Value Added measure meas that there are uequal gaps betwee the lowest grades possible i Level 3 qualificatios ad a fail. For example, i A Levels there is a 30 poit gap betwee each grade from A* to E but there is a 150 poit gap betwee a E ad a fail. This creates issues with the statistical modellig. For this reaso, Level 3 Value Added re-bases the poit scores to create eve sized gaps betwee grades ad also betwee the lowest grade ad a fail for all qualificatio types. This allows fails to be icluded i Level 3 Value Added but also retais the scale for the other grades. The diagram below demostrates how re-basig works. A B C D E Rebase Large gap betwee a E ad a FAIL A B C D E Equal distace betwee all outcomes FAIL The table below shows how poit scores are re-based for a selectio of Level 3 Value Added qualificatio types. 6

7 A Level / Applied GCE Sigle Award Grade Poit Score Rebased Poit Score AS Level / Applied GCE AS Sigle Award Grade Poit Score Rebased Poit Score BTEC Level 3 Subsidiary Diploma Grade Poit Score Rebased Poit Score A* A D* A B D B C M C D P D E FAIL 0 0 E FAIL 0 0 FAIL 0 0 7

8 CALCULATING STUDENT VALUE ADDED SCORES FOR INDIVIDUAL QUALIFICATIONS Before examiig how to calculate studet Level 3 Value Added scores, it is importat to first uderstad how the measure is structured. The ature of provisio i sixth forms ad colleges meas that there is a very wide rage of differet subjects offered to studets as well as a wide rage of differet types of qualificatios that studets ca study. To reflect this diverse rage of qualificatios, Level 3 Value Added is structured so that there is a separate value added measure for every qualificatio (subject) offered withi each qualificatio type. The area highlighted blue i the diagram below shows the qualificatio level of the hierarchy. The result of usig the approach above is that each studet will have a separate value added score for every Level 3 qualificatio that they study. I order to calculate a studet s value added score i a particular qualificatio, the first step is to use a statistical model to calculate a estimated outcome for the studet i the qualificatio. This estimate is calculated based o the actual outcomes of all studets atioally that have take the same Level 3 qualificatio ad with the same level of prior achievemet at the ed of Key Stage 4. For example, a studet that scored a average of 52 poits at Key Stage 4 would have their estimated outcome i A Level Geography calculated by the statistical model based o the actual outcomes of all studets atioally takig A Level Geography that also scored a average of 52 poits at the ed of Key Stage 4. A studet s Value Added score is the calculated by subtractig their estimated outcome i the qualificatio from their actual outcome i the qualificatio. Usig the same example, if a studet achieves a A i their A Level Geography but they were estimated to achieve a B by the statistical model, the the studet has a Value Added score of +1 grade. 8

9 The positive score tells us that this studet has exceeded their estimated A Level Geography outcome. If the Value Added score was egative, the this would tell us that the studet scored less tha their estimated A Level Geography outcome. The tables below summarises the calculatio described above. Example 1 Studet's Actual KS4 Average Poit Score Performace of all studets takig A Level Geography with a average score of 52 at KS4 used to estimate studet's performace i A level Geography (usig a statistical model) Studet's Estimated A Level Geography Outcome Studet's Actual A Level Geography Outcome Differece (Actual - Estimate) 52 Poits B A +1 Grade Example 2 Studet's Actual KS4 Average Poit Score 46 poits Performace of all studets takig A Level Frech with a average score of 46 at KS4 used to estimate studet's performace i A level Frech (usig a statistical model) Studet's Estimated A Level Frech Outcome Betwee a B ad C grade Studet's Actual A Level Frech Outcome B Differece (Actual - Estimate) +0.4 Grades The techical aex (sectio 1) provides a more detailed descriptio of how studets estimated scores ad their Value Added scores are calculated. 9

10 CALCULATING SCHOOL AND COLLEGE VALUE ADDED SCORES FOR INDIVIDUAL QUALIFICATIONS Oce studet VA scores have bee calculated for a particular qualificatio (e.g. OCR Natioal Certificate i Busiess Studies), we take the average of all the studet VA scores i that qualificatio withi the school or college. We the apply the shrikage factor, a adjustmet that provides a better estimate of VA scores for schools ad colleges with small umbers of pupils. The diagram below shows a example of how a school/college VA score is calculated from a example of five studet VA scores i a idividual qualificatio. STEP 1 - FIND THE AVERAGE OF STUDENT SCORES IN THE QUALIFICATION Studet 1 VA Score Studet 2 VA Score Studet 3 VA Score Studet 4 VA Score Studet 5 VA Score Average of the Studet VA Scores = School/College Ushruke VA Score i Qualificatio STEP 2 - APPLY THE SHRINKAGE FACTOR School/College Ushruke VA Score x Shrikage Factor = School/College Shruke VA Score i Qualificatio For more iformatio o calculatig school ad college Value Added scores, icludig the applicatio of shrikage factors, please see sectio two of the techical aex. 10

11 CALCULATING SCHOOL AND COLLEGE VALUE ADDED SCORES FOR QUALIFICATION TYPES The previous sectio showed how to calculate value added scores for each qualificatio offered by a school or college; it is also possible to aggregate a school or college s qualificatio scores up to calculate qualificatio type value added scores for the school or college. The blue shaded area i the diagram shows the qualificatio type level of the hierarchy. A qualificatio type Value Added score (e.g. AS Levels) is calculated by fidig the average of all the qualificatio Value Added scores that belog to the qualificatio type. A school or college s AS Level qualificatio type score would be foud by averagig all the AS Level qualificatio Value Added scores (e.g. AS Level History, AS Level Ecoomics, AS Level Maths) offered by the school. The calculatio is also weighted by the umber of studets takig each qualificatio; this gives greater weight to qualificatios beig take by more studets. The example below demostrates how to calculate a AS Level qualificatio type Value Added score for a school/college offerig three AS levels: AS Level VA Score = (50 x +0.25) + (20 x -0.70) + (100 x +0.35) ( ) = grades AS Level History VA score = Cohort Size = 50 AS Level Ecoomics VA score = Cohort Size = 20 AS Level Maths VA score = Cohort Size =

12 Which Qualificatio Types are Icluded? I order for a qualificatio type (e.g. A Levels) to be icluded withi Level 3 Value Added, it must first be a Level 3 qualificatio type but it also must have a graded outcome. This meas that the qualificatio type eeds to have four or more possible outcomes, for example, A Levels have seve differet outcomes (A*, A, B, C, D, E ad FAIL). Additioally, there eeds to be a miimum of 80 studets ad 5 istitutios offerig the qualificatio type atioally i order for it to be icluded i Level 3 Value Added. The list below shows some of the qualificatio types icluded i Level 3 Value Added for the publicatio based o 2012/13 examiatio data. Qualificatio Type A Levels AS Levels Exteded Project (Diploma) Pre-U Qualificatios Applied GCE A Level Qualificatios Iteratioal Baccalaureate Free Stadig Maths Qualificatios BTEC Level 3 Qualificatios 12

13 CALCULATING SCHOOL AND COLLEGE ACADEMIC AND VOCATIONAL VALUE ADDED SCORES It is also possible to group together school ad college value added scores ito a score for all academic qualificatios ad a score for all vocatioal qualificatios. Aggregatig to academic ad vocatioal level ivolves combiig value added scores from differet qualificatio types, for example, if a school has a value added score i AS Level Maths ad i A Level Biology, both VA scores would cotribute to the school s academic value added score. Similarly if a college has a value added score i BTEC Level 3 Subsidiary Diploma Busiess Studies ad a value added score i Level 3 Foudatio Diploma i Art ad Desig, the both of these would cout towards the college s vocatioal value added score. The relative size of the qualificatio type is also take ito accout i the calculatio of academic ad vocatioal value added scores. Academic VA scores are give as a proportio of oe A Level grade ad vocatioal VA scores are give as a proportio of oe BTEC Level 3 Subsidiary Diploma grade. The example below demostrates how to calculate a academic VA score for a school/college offerig oe A Level ad oe AS Level: Academic Qualificatios (50 x +0.5) + (100 x -0.15) (50 x 1.0) + (100 x 0.5) = A Level Grades A Level Chemistry VA Score = Cohort Size = 50 Qualificatio type size = 1.0 AS Level Eglish Literature VA Score = Cohort Size = 100 Qualificatio type size = 0.5 Please refer to the table o page 12 to see which qualificatio types are cosidered academic ad which are vocatioal. 13

14 INTERPRETING SCHOOL AND COLLEGE VALUE ADDED SCORES The previous three sectios have show that schools ad colleges have value added scores at qualificatio, qualificatio type ad academic ad vocatioal level. The approach for iterpretig the value added scores at the differet levels of aggregatio is the same. A school or college s VA score at qualificatio, qualificatio type or academic ad vocatioal level ca be used as a measure of their effectiveess, but it is importat to ote that a value added score is based o a give set of studets' results for a particular test paper o a particular day. A school or college could have bee equally effective ad yet the same set of studets might have achieved slightly differet results i their Level 3 qualificatio o the day. Ad the school or college would almost certaily have show slightly differet results i the Level 3 qualificatio with a differet set of studets. This elemet of ucertaity eeds to be take ito accout whe iterpretig a school or college s VA score; this is doe usig cofidece itervals. A cofidece iterval is a rage of scores withi which we are statistically cofidet that a school or college s true VA score will fall. A school or college s cofidece iterval is always cetred o the school or college s VA score. For example, if a school or college s VA score is +1 ad the size of their cofidece iterval is 0.5 grades, the the cofidece iterval rages betwee +0.5 ad +1.5 (i.e. half a grade either side of the VA score). The size of the cofidece iterval is largely determied by the umber of studets i the school or college that completed the Level 3 qualificatio. Schools ad colleges with fewer studets completig the qualificatio have wider cofidece itervals because their VA score is based o a smaller umber of studets, so there is less evidece o which to judge the school or college s effectiveess. To judge a school or college s effectiveess, both the VA score ad the associated cofidece iterval eed to be take ito accout. If the whole rage of the cofidece iterval is above the atioal average of zero (i.e. the lower boud is greater tha zero), we ca say the school/college score is above the atioal average ad this is statistically sigificat. Therefore, we ca be cofidet the school/college is helpig its studets make better tha average progress i the qualificatio. A illustratio of how to iterpret school/college VA scores is give overleaf. Similarly, whe the etire rage of the cofidece iterval is below zero (i.e. the upper boud is less tha zero), we ca say the school/college score is below the atioal average for the Level 3 qualificatio ad this is statistically sigificat. Fially, if the cofidece iterval straddles the atioal average of zero, the 14

15 we ca say that the school/college is ot sigificatly differet from the atioal average for the qualificatio, i other words, we caot cofidetly say that the school or college s VA score is defiitely above or defiitely below the atioal average for the qualificatio. The table ad diagram below shows how a school/college s VA score ad cofidece itervals should be iterpreted to reach oe of the three defiitios above. School A is a example of a school that is sigificatly above atioal average, College B is ot sigificatly differet from atioal average, ad School C is sigificatly below atioal average. School A School B School C VA Score Upper Cofidece Limit Lower Cofidece Limit For more iformatio o the calculatio of cofidece itervals, please see the techical aex. 15

16 WHEN AND WHERE WILL LEVEL 3 VALUE ADDED BE PUBLISHED? There are two Level 3 Value Added publicatios based o 2012/13 examiatio data; the first is o Thursday 17th October 2013 ad was based o uameded examiatio data; the secod will be i Jauary 2014 ad will be based o ameded examiatio data. The Level 3 Value Added publicatio based o uameded data will be available o the performace tables checkig website. Please ote that 2012/13 reports will ot be accessible through the provider gateway, ad the checkig website is the oly place you ca access a school or college report. The Excel Level 3 VA report ad the Level 3 VA Summary pdf report are made available through the Performace Tables checkig website. The two reports ca be foud o the documets page i folders called Level 3 VA Report ad Level 3 VA Summary. Schools ad colleges that are part of a cosortium have a additioal Excel report ad a additioal pdf report published o the Checkig website. This is a aoymised report that shows Level 3 Value Added iformatio for all schools ad colleges withi the cosortium. The Level 3 Value Added publicatio based o ameded data is made available o two websites, which are; The Performace Tables Checkig Website The Performace Tables Website As with the uameded publicatio, the Excel Level 3 VA report ad the Level 3 VA summary pdf report will be available i the same locatios o the Performace Tables Checkig website. Schools ad Colleges i a cosortium will agai receive two further aoymised reports showig Level 3 Value Added data for all for all schools ad colleges withi the cosortium. Level 3 Value Added scores based o ameded data are also published o the Departmet for Educatio s Performace Tables website. The Performace Tables website shows academic ad vocatioal value added scores, as well as qualificatio type value added scores for every school ad college (e.g. a A Level VA score or a BTEC Natioal Diploma VA Score). 16

17 OTHER USEFUL RESOURCES As well as this guide, there are a umber of other useful resources o Level 3 Value Added. The resources detailed below ca all be foud o the checkig website util the publicatio of ameded data, whe they will also be located o the Level 3 Value Added Webpage: Level 3 Value Added Webpage There is also a mailbox which you ca cotact if you have ay questios o Level 3 Value Added though we advise that you check the FAQs documet first: Level3VA.HELPDESK@educatio.gsi.gov.uk Frequetly Asked Questios: A list of Frequetly Asked Questios ca be foud at the lik above o the Level 3 Value Added webpage. Level 3 Value Added Ready Reckoer ad Guide: A Level 3 Value Added ready Reckoer is available o the checkig website. Ulike the Level 3 VA Reports, the Ready Reckoer allows the user to iput their ow data to calculate Value Added scores for specific groups of pupils. For cosistecy, the output from the Ready Reckoer is i the same format as the VA Reports. A guide o how to use ad iterpret the VA Reports ad Ready Reckoer ca also be foud o the checkig website. 17

18 ANNEX A - WHAT'S NEW IN LEVEL 3 VALUE ADDED FOR 2012/13? There are a umber of ew features i Level 3 Value Added for 2013, this page explais some of these ad the impact they might have o your school or college. 1) Overall Academic ad Vocatioal VA Scores We ow calculate oe overall score for academic qualificatios ad aother for vocatioal qualificatios. These scores are the Departmet s headlie measures ad will be published i Performace Tables i Jauary. To calculate this score, we aggregate together all of the scores from academic subjects to create the academic score ad aggregate all vocatioal subject scores ito a sigle vocatioal score. Both these scores will be give with a cofidece iterval to help ascertai whether your school or college is above, below, or comparable to the atioal average i academic ad vocatioal qualificatios. 2) Value Added measured i Grades From ow o, Level 3 Value Added scores are give as the proportio of a grade. VA scores for academic qualificatios are give as the proportio of oe A Level grade, ad scores for vocatioal qualificatios are give as a proportio of oe BTEC Level 3 Subsidiary diploma grade. For example a academic VA score of +0.5 idicates that o average, studets achieved half a A Level grade above the atioal average for studets with similar Key Stage 4 attaimet. Reportig value added i grades makes iterpretatio easier, allowig schools ad colleges to determie which qualificatio types they are performig best i, as all academic ad vocatioal VA scores will be o comparable scales. We have chose the A Level ad Subsidiary Diploma qualificatios as the respective academic ad vocatioal referece stadards because they have the greatest umber of etries atioally, ad are therefore the easiest qualificatios for schools, colleges, studets ad parets to idetify with. 3) Two AS Level model types You will otice that there are two differet types of AS Level models i your report - the first beig AS Level (Not cotiued to A2) ad the other beig AS Level (All). The AS Level (Not Cotiued to A2) models are based o AS Levels that were ot cotiued to A Level. These cout towards the aggregatio of academic qualificatios. The AS Level (All) models are based o all AS Levels, ad are icluded as a school improvemet tool. These do ot cout i the academic aggregatio, so as ot to double cout studets takig AS levels. We have icluded the AS Level (All) models because they better idicate progress levels made for all studets i their AS Level year. 18

19 4) Pupils are allocated differetly I previous years, studets have had their Level 3 VA results allocated based o where they sat the exam, istead of the school or college at which they are erolled. This has ow bee chaged so that from 2012/13 studets are allocated based o where they are deemed to be o roll. This meas that Level 3 VA ow allocates results i the same way to other Performace Tables measures ad schools are held accoutable for the studets they teach, as opposed to those they provide a exam veue for. 5) Chage of cohort Previously, Level 3 VA would report o qualificatios take i-year. This meat that if a studet re-took a qualificatio, it would cout towards the school or college s VA score for two years ruig. We ow oly iclude studets oce they have reached the ed of Key Stage 5. Therefore, ay retake qualificatios, or cases where a studet takes a more substatial qualificatio i the same subject, will be discouted. Oly the fial studet results will be couted towards Level 3 VA. Agai, this aligs Level 3 VA to other performace tables measures. For the 2012/2013 Ameded publicatio, we have applied the academic ad vocatioal cohort rules used by other performace table measures to Level 3 VA. A studet s academic qualificatios will oly cout towards their school s academic Value Added score if that studet is eligible for the academic cohort. Likewise, a studet s vocatioal qualificatios will oly cout towards a school s vocatioal Value Added score if that studet is eligible for the vocatioal cohort. Please see the Level 3 VA Frequetly Asked Questios documet for details of who is eligible for each cohort. 19

20 A GUIDE TO LEVEL 3 VALUE ADDED IN 2013 SCHOOL AND COLLEGE PERFORMANCE TABLES TECHNICAL ANNEX Cotets Sectio 1 Calculatio of exam level value added scores for studets Sectio 2 Calculatio of qualificatio value added scores for schools ad colleges Sectio 3 Calculatio of qualificatio type value added scores for schools ad colleges Sectio 4 Calculatio of Academic / Vocatioal value added scores for schools ad colleges Sectio 5 Calculatio of statistical sigificace of school ad college value added scores Page Number

21 SECTION 1 - CALCULATION OF EXAM LEVEL VALUE ADDED SCORES FOR STUDENTS 1. Before value added scores ca be calculated for schools ad colleges, a iitial stage is required to calculate idividual exam level value added scores for studets. These exam level value added scores provide the buildig blocks for calculatig the aggregated school ad college value added scores. 2. The first step i calculatig a studet s value added score i a idividual exam is to derive their estimated poit score i the qualificatio. The studet s estimated attaimet is calculated usig the atioal lies produced as part of the statistical modellig. The studet s qualificatio value added score is the foud by subtractig their estimated poit score from their actual poit score (both re-based) for that qualificatio. 3. The iformatio required to perform this calculatio is detailed below: 0, 1, 2, 3, 4 Coefficiets for give qualificatio at atioal level S 2 Natioal covariace matrix Natioal variace of error Number of exam records i qualificatio per school/college Variace ratio per qualificatio per school/college Shrikage factor per qualificatio per school/college x Prior attaimet for th exam record y Outcome attaimet for th exam record u 2 Array of exam record VA scores Stadard error per qualificatio per school/college Scalar variace of qualificatio per school/college (itermediate value) Formula for calculatio of idividual exam level VA scores 4. Idividual exam level value added scores are calculated usig the followig equatio: u Y X T T 21

22 5. u is a 1 matrix of the idividual exam record VA scores ad (upper case gamma) is a 1 5 matrix cotaiig the five coefficiets for this particular qualificatio. Y is a 1 matrix of the outcome attaimet y of each exam record for this particular qualificatio ad X is a x 5 matrix, where each row of X is give , x, x, x, x 6. For example, for a sigle case, if a learers had a average prior attaimet score of x = 45 ad a outcome attaimet of y = 210 ad the key coefficiets were: 0 = = = = = The applyig the equatio for a sigle studet would give: u Y u y Would be equal to: X T , x 1, x 2, x, 3, x 4 u = 210 ( x x x x ) u = u = Expressed as a matrix of three cases this would be equivalet to: Y, X x x x x x 4 x Therefore a example of u would be: 5.95 u If a cadidate has achieved the highest possible grade for a qualificatio the their value added score should ot be lower tha 0. I some cases 22

23 the atioal lie of expected attaimet fitted by the statistical model ca lead to studets with high prior attaimet values beig give a estimated attaimet value above the maximum grade. 9. For example, o a AS level, a studet could be give a estimated attaimet value of 76 rebased QCA poits whe the maximum attaiable grade, a A grade, is oly worth 75 poits. I this case the studet would usually be give a score of -1 however this is overwritte with a score of 0. Studets whose estimates are below the maximum grade do ot have their scores modified. 23

24 SECTION 2 - CALCULATION OF QUALIFICATION VALUE ADDED SCORES FOR SCHOOLS AND COLLEGES 10. Now that the exam level value added scores have bee calculated for each studet, it is ow possible to calculate school ad college value added scores i each qualificatio that they offer. Here, the term qualificatio meas a subject withi a qualificatio type, for example, A Level Frech. 11. A qualificatio value added score for a school or college is calculated by fidig the average of all the exam level value added scores i that qualificatio ad i that istitutio. The result is multiplied by a shrikage factor. 12. The shrikage factor is a umber betwee zero ad oe. If the umber of studets takig the qualificatio is large, the shrikage factor is close to oe, ad thus has little effect o the school or college s value added score. If the umber of studets takig the qualificatio is small, the shrikage factor is close to zero, ad brigs the school or college s qualificatio value added score towards the atioal average. By doig this, the results of qualificatios with low studet umbers are ow more reliable ad less volatile, while the value added scores for qualificatios with large umbers of studets remai virtually uchaged. Calculatio of istitutio VA score for a qualificatio 13. The first step is to calculate a average (mea) value added score usig the exam level value added scores for each idividual i the school or college that has take the qualificatio. u VAavg 1 u deotes the VA score of the th exam record. 14. Next calculate the average prior attaimet x avg for the same set of studets usig the formula below: x avg The ext step is to calculate the shrikage factor deoted by λ. First a itermediate tau squared value τ 2 must be calculated usig the formula below: 2 x xs x T 24

25 2 I this formula, x is defied as, x, x ad S is the Natioal 1 avg avg Covariace matrix for the particular qualificatio for which the calculatios are beig performed The calculate the variace ratio usig the error term from the MLM output: Fially, calculate the shrikage factor : 18. The overall istitutio VA score U for the give qualificatio is the give by: U VA Calculatio of cofidece itervals aroud a school or college s qualificatio VA score 19. Usig the stadard error, it is possible to calculate cofidece itervals aroud a school or college s qualificatio value added score. Cofidece itervals represet the rage withi which we ca be cofidet that the avg school or college s true value added score lies. The stadard error of U is give by: The 95% cofidece iterval aroud a school or college s qualificatio value added score is the give by: U

26 SECTION 3 - CALCULATION OF QUALIFICATION TYPE VALUE ADDED SCORES FOR SCHOOLS AND COLLEGES 21. This sectio describes how to calculate qualificatio type value added scores for a school or college. A example of a qualificatio type would be A Levels or BTEC Natioal Diplomas. 22. A qualificatio type value added score is calculated by fidig the weighted average (based o umber of studets) of all the school/college qualificatio value added scores that belog to the qualificatio type. For example, if a school offered A Level Geography ad A Level Maths, the the school s A Level qualificatio type value added score would be calculated by fidig the weighted average of the school s A Level Geography ad A Level Maths value added scores. 23. The iformatio required to perform this calculatio is detailed below: VA QualSubj VA score for a particular qualificatio withi a give qualificatio type at school/college level VA Aggregate VA score across qualificatios Qual withi a give qualificatio type at school/college level ExamQualSu bj Number of exams for a particular qualificatio at school/college level ExamQuals Number of exams across all qualificatios withi a give qualificatio type at school/college level QualSubj Stadard error for a give qualificatio at school/college level VA Qual Stadard error for a give qualificatio type at school/college level Weightig factor for selected qualificatios. = 1 for all qualificatios, except Geeral Studies, where = The formula below is used to calculate aggregate value added scores for qualificatio types for a school or college. The formula should be used for each qualificatio type that a school or college offers. TotalSubjs VA 1 VAQual QualSubj Exam QualSu b Exam QualSu bj 26

27 25. It is the possible to calculate 95% cofidece itervals aroud the school or college s qualificatio type value added score. The first step i doig this is to calculate the stadard error for the qualificatio type: VA Qual ExamQualSu bj 1 ExamQualSu bj ExamQuals 2 2 QualSubj 26. The fial step is to the use the qualificatio type stadard error to calculate cofidece itervals aroud the value added score usig the followig equatio: VAQual 1.96 VA qual 27

28 SECTION 4 - CALCULATION OF ACADEMIC AND VOCATIONAL VALUE ADDED SCORES FOR SCHOOLS AND COLLEGES 27. It is also possible to calculate academic ad vocatioal value added scores that go across qualificatio types. For example, it is possible to calculate a vocatioal value added score that aggregates together qualificatio value added scores from differet qualificatio types such as BTEC Subsidiary Diploma Hospitality ad BTEC Exteded Diploma Busiess Studies. 28. The iformatio required to perform this calculatio is detailed below. Variable Descriptio School or College s overall academic / vocatioal value added score The umber of academic / vocatioal qualificatios for that school or college School or college s VA score for give academic or vocatioal qualificatio (e.g. A level maths VA score) Natioal average VA score for give academic or vocatioal qualificatio Number of etries withi school or college withi give academic or vocatioal qualificatio The volume of the qualificatio type for the give academic or vocatioal qualificatio, i relatio to A Levels (for academic qualificatios) / BTEC Level 3 Subsidiary Diplomas (for vocatioal qualificatios) 29. The formula below is used to calculate aggregated value added scores for academic ad vocatioal qualificatios. As this value added score combies iformatio from differet qualificatio types, the Vol variable is icluded i the formula. Qual 28

29 This step icludes a small adjustmet to correct for aggregatio error, which meas that the studet average VA score is zero rather tha the istitutio average. This may mea there is a small icosistecy with qualificatio type ad idividual qualificatio scores. For example, if a istitutio oly offered A Levels, the their A Level score could be slightly differet from their aggregate academic score, eve though they are both calculated from the same results. 30. It is also possible to calculate cofidece itervals aroud each SSA (across qualificatio types) value added score. To do this, the stadard error must first be calculated which is give by the formula below, with additioal variables required i the table after: ( ( ) ) ( ) Variable Descriptio Stadard error of overall academic / vocatioal value added score Stadard error for the VA score for the give academic / vocatioal qualificatio 31. The fial step is to the use the academic / vocatioal stadard error to calculate cofidece itervals aroud the value added score usig the followig equatio: 29

30 SECTION 5 CALCULATING THE STATISTICAL SIGNIFICANCE OF SCHOOL AND COLLEGE VALUE ADDED SCORES Statistical Sigificace at Qualificatio Level 32. A school or college qualificatio value added score (deoted U) is defied to be below the atioal average ad statistically sigificat whe their value added score is below 0 ad the upper ed of the 95% cofidece iterval is below zero. This ca be expressed formulaically as: U U 0 & 33. A school or college qualificatio value added score is defied to be above the atioal average ad statistically sigificat whe their value added score is above 0 ad the lower ed of the 95% cofidece iterval is above zero. This ca be expressed formulaically as: U U 0 & Statistical Sigificace at Qualificatio Type Level 34. A school or college qualificatio type value added score is defied to be below the atioal average ad statistically sigificat whe their value added score is below 0 ad the upper ed of the 95% cofidece iterval is below zero. This ca be expressed formulaically as: 1.96 VA 0 VA 0 & Qual VA Qual 35. A school or college qualificatio type value added score is defied to be above the atioal average ad statistically sigificat whe their value added score is above 0 ad the lower ed of the 95% cofidece iterval is above zero. This ca be expressed formulaically as: Qual 1.96 VA 0 VA 0 & Qual VA Qual Statistical Sigificace at Academic / Vocatioal Level 36. A school or college academic / vocatioal value added score is defied to be below the atioal average ad statistically sigificat whe their value added score is below 0 ad the upper ed of the 95% cofidece iterval is below zero. This ca be expressed formulaically as: Qual ( ) 37. A school or college academic / vocatioal value added score is defied to 30

31 be above the atioal average ad statistically sigificat whe their value added score is above zero ad the lower ed of the 95% cofidece iterval is above zero. This ca be expressed formulaically as: ( ) 31