Schools Value-added Information System Technical Manual

Transcription

1 Schools Value-added Information System Technical Manual Quality Assurance & School-based Support Division Education Bureau 2015

2 Contents Unit 1 Overview... 1 Unit 2 The Concept of VA... 2 Unit 3 Control Variables... 4 Unit 4 Interpreting VA Information... 5 Unit 5 Reporting VA Information Appendix: Calculating VA Scores... 18

3 Unit 1 Overview The Education Bureau (EDB) is committed to supporting schools to make use of data and evidence to conduct self-evaluation. The EDB also provides schools with different evaluation tools and data to meet their needs. Value-added (VA) information is one of the self-evaluation tools provided by the EDB through the online platform, namely the Schools Value-added Information System (SVAIS), since 2003 and it has been widely used by schools. The SVAIS provides schools with quantitative and objective data on students academic performance. In conjunction with other school-based data, VA information can facilitate schools to understand more clearly the academic performance of students. With the changes brought about by the implementation of the New Academic Structure (NAS), such as the phasing-out of the Hong Kong Certificate of Education Examination and the Hong Kong Advanced Level Examination and the introduction of the Hong Kong Diploma of Secondary Education (HKDSE) Examination, the EDB has reviewed the VA model and the SVAIS. In the course of the revision, the EDB has sought advice from academic experts and maintained its communication with schools so as to enhance the transparency of the system and to have a better understanding of the schools needs. Under the NAS, the SVAIS has begun to provide VA information for the 2012 and 2013 S1/S6 cohort of students based on their HKDSE Examination results since Starting from 2015, two new control variables, namely non-chinese speaking (NCS) and special educational needs (SEN) have been added to enhance the reliability and validity of the VA model. VA information is rarely definitive and never explains why a school is performing better or worse. This can only be made clearer by conducting an in-depth analysis of individual schools. Schools should make reference to other related evaluation data (for instance, students performance in the Pre-S1 Hong Kong Attainment Test, internal assessment results and among others) when examining the effectiveness of learning and teaching. In addition, VA information reflects only students academic performance. Schools should make use of other sources of information such as the Assessment Program for Affective and Social Outcomes (APASO), school-based questionnaires and teachers observation to understand the whole person development of students, in order to formulate and implement follow-up action plans for continuous improvement. 1

4 Unit 2 The Concept of VA In evaluating subject learning, public examination results are important indicators of absolute standards and can be helpful in identifying instances of unacceptably low standards. Nevertheless, they may not be good indicators of the extent to which schools have contributed to the levels of attainment of their students. When comparisons are made of the average results of students attending different schools, what is revealed often indicates more about the nature of the students attending those schools than the effectiveness of the schools. VA information, on the other hand, takes into account factors such as student ability and school characteristics and can better reflect the effectiveness of schools in improving the academic performance of their students. Therefore, schools are advised to refer to both the public examination results and the VA information in self-evaluation when evaluating the effectiveness of their learning and teaching strategies. The multi-level regression model used to compile VA scores can be described by the following formula: Student actual attainment in the HKDSE Examination (outcome variable) = Student predicted attainment (after adjusting for control variables) in the HKDSE Examination + school residual + student residual The formula shows that there is a difference between student actual attainment in the HKDSE Examination (outcome variable) and the statistically predicted attainment after taking into account student academic ability in S1 and other background characteristics (control variables). This difference the unexplained part or what is often called the residual can be further partitioned into two components, namely the schools attended by the students (school residual), and unexplained individual differences (student residual). It is through the estimation of the school residual that VA estimates can be obtained. For technical details of the multi-level regression model, please refer to the Appendix. 2

5 Other control variables e.g. School average AAI Gender All boys/ All girls school The logic behind this operational definition is as follows: given that examination scores have been adjusted statistically for the most important factors known to influence student performance, then if significant differences in examination scores among schools remain, it is reasonable to infer that these differences in scores reflect differences in the effectiveness of schools. 3

6 Unit 3 Control Variables An important issue in compiling VA estimates is to adjusting student predicted attainment by control variables. Since VA estimates are residuals, or what is left over after taking into account the effect of all the control variables, it is important to include control variables that are known to significantly affect student performance as far as practicable. The best predictor of student actual attainment is almost always student prior attainment. When a reliable and valid student prior attainment is available, this will explain a relatively large proportion of variance, while other control variables will only explain a small amount of variance. These other control variables may nonetheless still have a statistically significant but small influence on student actual attainment. The general advice in regression analysis is to strike a balance between maximising the percentage of variance explained while minimising the number of control variables. In the VA model, the student prior attainment and other control variables currently used are described below: Control variable Description of variable The AAI is the normalised scores of Secondary School Places Academic Ability Allocation (SSPA). It is a measure of the academic ability of Index (AAI) students on entry to Secondary One. It serves as the student prior attainment in the VA model. It is the average AAI score of all students within a school taking a School Average AAI given subject. It adjusts the peer AAI effect on student performance. Gender All Girls School All Boys School Non-Chinese Speaking (NCS) Special Educational Needs (SEN) It adjusts the different rates of progress made by boys and girls. It adjusts the effect of being in an all girls school. It adjusts the effect of being in an all boys school. It adjusts the effect of being a non-chinese speaking student. It adjusts the effect of being a student with special educational needs. 4

7 Unit 4 Interpreting VA Information Value-added Scores Value-added scores (VA scores) are normalised (so that they follow a normal distribution) and placed on a scale that ranges from -10 (the lowest score) to +10 (the highest scores), with an overall mean of zero (centred on zero). In other words, a score of zero does not imply that students in the school have made no progress, but rather that the progress of students in the school is average within the territory. VA scores are empirical Bayes estimates, which means that they are weighted to reflect the reliability of each school s score. In other words, the VA scores of a school with a small number of students analysed in a particular subject are biased towards the overall average. This means that for small schools that actually added high value, their true performance will not be reflected fully in their VA scores, which will instead be shrunk towards the average (zero). The same applies in the opposite direction to small schools that actually added low value. Their inferior performance will not be reflected fully in their VA scores. The VA scores compiled in such a way, on the one hand, is an advantage since they are conservative estimates. On the other hand, this shrinkage complicates the ranking of VA scores (i.e. Stanine), since high-scoring small schools and low-scoring small schools may end up close together in rank. In this regard, schools are advised not to rely solely on VA scores or Stanine to make judgment on a school s VA performance, but also the confidence interval which is explained in the next section. Confidence Intervals All VA estimates are associated with a degree of uncertainty. In the SVAIS, this uncertainty is quantified and a 95% confidence interval is constructed around each VA score. A confidence interval is a range of values which contains the estimated parameter (i.e. the true VA score of a school) with a certain probability. A 95% confidence interval on a VA estimate can then be interpreted as an interval which contains the true VA score of a school with a 95% probability. A shorter 95% confidence interval implies that the corresponding VA estimates are more accurate. The 95% confidence interval of a VA estimate of a subject will usually be shorter if the percentage of variance explained by the control variables for that subject is 5

8 large, which is in turn affected by the predictive power of the control variables on student actual attainment in the HKDSE Examination, the number of students analysed for that subject and the variations among individual students, ceteris paribus. For example, as shown in the multi-subject graph below, it can be observed that the 95% confidence interval of Liberal Studies is obviously shorter than that of Chinese Literature. One of the reasons is that the number of students available for analysis of Liberal Studies is much larger than that of Chinese Literature. On the other hand, given a similar number of students analysed in Chinese Language and English Language, the 95% confidence interval of English Language is slightly shorter than that of Chinese Language since a larger percentage of variance of English Language is explained by the control variables in the model than that of Chinese Language. 6

9 All regression analyses are carried out for each subject and each year separately. Hence, the confidence intervals of VA scores will differ across subjects and across years. This needs to be borne in mind when comparing VA performance across subjects and across years. Value-added Performance There are three categories of value-added (VA) performance: Below average: The VA score is significantly lower than the territory average. This happens when the whole length of the confidence interval is below the territory average (i.e. zero). If this happens, the text below average will be displayed under Value-added performance in the VA reports. (See Subject A below) On a par with average: The VA score is not significantly different from the territory average. This happens when the confidence interval straddles the territory average (i.e. zero). If this happens, the text on a par with average will be displayed under Value-added performance in the VA reports. (See Subject B below) Above average: The VA score is significantly higher than the territory average. This happens when the whole length of the confidence interval is above the territory average (i.e. zero). If this happens, the text Above average will be displayed under Value-added performance in the VA reports. (See Subject C below) VA score Upper Confidence Bound Estimated VA Score Lower Confidence Bound 0 Subject C: VA is significantly above average Territory Average VA=0 = 0 Subject B: VA is not significantly different from average Subject A: VA is significantly below average 7

10 Stanines The VA scores of all schools in each subject are ranked in ascending order and subsequently transformed into nine groups (stanines). Distribution of schools by the nine groups (stanines) could be diagrammatically represented by the following figure. The stanine is a normalised standard score ranging from 1 to 9, with a mean of 5 and a standard deviation of 2. The stanine for each school provides a crude indication of the VA performance of the user s school relative to other schools. The stanine can be found in the All School Reports, All Years Report and the Multi-Subject Table. Schools are advised not to rely solely on stanines to review their VA performance since stanines do not take into consideration the confidence interval, i.e. the accuracy of the VA estimates. Percentage of Students Analysed The percentage of students analysed is the ratio of the number of students included in VA analysis to the total number of students taking the examination of a particular subject in a school. For a student to be included in VA analysis, there should be HKDSE Examination results as well as his/her own Academic Ability Index (AAI) six years before the HKDSE Examination which can be matched with a personal identifier. Hence, students under the following circumstances cannot be included in VA analysis: 8

11 1. Students who came from regions outside Hong Kong (including the Mainland and foreign countries) and joined local secondary schools directly; 2. Students repeated/suspended/skipped any levels in secondary school, leading to unmatched SSPA scores 6 years ago; or 3. Students who did not have SSPA scores 6 years ago as their primary schools did not participate in the SSPA scheme. The VA information should be interpreted with caution if the percentage of students analysed is low, since a large proportion of the school s students taking that subject have not been included in VA analysis. In that case, the school s VA information may not be representative enough to reflect the performance of all S6 students of the school. Nonetheless, the VA information is still useful for school development in the sense that it could reflect the performance of those students who are included in the analysis. In addition, since VA information based on a very small number of students is unreliable, if the number of students analysed in a particular subject of a school is fewer than ten, the VA information of that subject will not be released. 9

12 Unit 5 Reporting VA Information VA Information of Individual Subjects The standard marks of individual subjects compiled by the Hong Kong Examinations and Assessment Authority (HKEAA) are taken as the scores of student actual attainment and they are normalised before being included in VA analysis. In principle, VA analyses are conducted for all Category A subjects, but it does not follow that there is VA information for all HKDSE subjects. If the VA estimates of a particular subject are not accurate enough, such as English Literature, Visual Arts and Music, there will not be VA information for that subject. On the other hand, since there are no standard marks for Category B (Applied Learning) and Category C (Other Language) subjects, no VA information is provided for these subjects either. VA Information of Subject Groups The VA information of Core 4 and Best 5 subject groups may reflect the holistic VA performance of a school. Since the standard marks compiled by the HKEAA are taken as the scores of student actual attainment in the VA model, the scores of different Category A subjects are directly comparable. The Core 4 score is defined as the sum of the standard marks of Chinese Language, English Language, Mathematics (Compulsory Part) and Liberal Studies. The Best 5 score is defined as the sum of the five highest standard marks out of all the HKDSE subjects taken by a student (including Category A and Category B subjects). These five subjects may include or exclude the core subjects. When a student sits both the Mathematics (Compulsory Part) and the Mathematics (Extended Part) at the same time, the better of the two will be selected. Attained with Distinction and Attained in Category B subjects are deemed comparable to level 3 and level 2 in Category A subjects respectively. As there are no standard marks of Category C subjects, these subjects will not be included in the Best 5 calculation. In addition, if a student takes fewer than five HKDSE subjects, that student will not be included in the Best 5 VA analysis. Same as the standard marks of individual subjects, both Core 4 and Best 5 scores are normalised before being included in VA analysis. 10

13 School VA information is provided for individual subjects and for specific subject groups. So far there is no single VA score for summarising the VA performance of the whole school. SVAIS Reports In the SVAIS, each school is provided with a school account that allows users to access their own VA information as well as summary statistics for the territory. Five types of reports are provided through the SVAIS. 1. All Schools Report The All Schools Report shows the distribution of VA scores of all schools, schools with a similar intake of students and schools within the same district of a particular subject in a specific year as shown below. 11

14 The box-and-whisker plot of all schools displays the distribution of VA scores of all schools in the territory. As shown in the diagram below, the top whisker represents the 95th percentile, the top of the box represents the 75th percentile, the bold line in the middle of the box the 50th percentile (i.e. the median), the bottom of the box the 25th percentile, and the bottom whisker the 5th percentile. The red line indicates the VA score of the user s school. 95 th Percentile 75 th Percentile Your school 50 th Percentile 25 th Percentile 5 th Percentile The table below displays the distribution of VA scores of all schools in the territory and the position of the user s school. Value-added Score Percentiles 5.50 or above Top 5% of schools in the territory (95th percentile or above) 2.25 to 5.50 Next 20% of all schools in the territory (75th percentile to 95th percentile) 0.84 to 2.25 Next 15% of all schools in the territory (60th percentile to 75th percentile) to 0.84 Middle 20% of all schools in the territory (40th percentile to 60th percentile) to Next 15% of all schools in the territory (25th percentile to 40th percentile) to Next 20% of all schools in the territory (5th percentile to 25th percentile) and below Bottom 5% of all schools in the territory (Below 5th percentile) The table below displays the detailed VA information of the user s school, including the VA score, the 95% confidence interval, the VA performance, the stanine, and the percentage of students analysed. 12

15 Value-added Score 95% Confidence Interval Lower Bound Upper Bound Value-added Performance Stanine (1-9) % of Students Analysed On a par with average (0) The box-and-whisker plot of similar intake schools displays the distribution of VA scores of the intake group which the user school belongs to for a particular subject. Similar intake schools are categorised by the School Average AAI. All School Average AAI of the same subject is ranked in ascending order and subsequently transformed into 9 groups with a mean of 5 and a standard deviation of 2. To avoid too few schools appearing in the highest and lowest groups, the highest two groups (Groups 8 and 9) are combined into one similar intake group. Likewise, the lowest two groups (Groups 1 and 2) are combined into one similar intake group. Altogether there are seven intake groups as shown in the diagram below. Combined into one intake group Combined into one intake group The box-and-whisker plot of schools from same district displays the distribution of VA scores of the District Council district which the user school belongs to for a particular subject. If there are fewer than three schools in a particular district for a particular subject, this plot will not be reported in the SVAIS to avoid the identification of individual schools. 13

16 2. Differential Effectiveness Report The SVAIS displays the differential effectiveness for students of different abilities in a statement. In the SVAIS, differential effectiveness refers to the extent to which a school is more or less effective for more or less able students. The following figures illustrate different cases of differential effectiveness in detail. The blue regression line in the graph summarises the relationship between the AAI (student prior attainment) and the HKDSE Examination scores in a hypothetical school, School A. The red regression line summarises the relationship between the AAI and the HKDSE Examination scores averaged across all schools. In the case of Panels 1 3, the line for School A is significantly steeper than the line for All Schools. This means that School A is relatively more effective for more able students and less effective for less able students. This may happen when the overall VA performance of the whole school is above average (Panel 1), on a par with average (Panel 2) or below average (Panel 3). In the case of Panels 4 6, the line for School A is significantly flatter than the line for All Schools. This means that School A is relatively more effective for less able students and less effective for more able students. This may happen when the overall VA performance of the whole school is above average (Panel 4), on a par with average (Panel 5) or below average (Panel 6). 14

17 In the case of Panels 7 9, the slope of the line for School A does not differ significantly from the slope of the line for All Schools. This means that School A is equally effective for students of different abilities. This may happen when the overall VA performance of the whole school is above average (Panel 7), on a par with average (Panel 8) or below average (Panel 9). 3. All Years Report Users may refer to this report for the VA trend of a particular subject. It consists of a chart and a table. The chart below provides a graphical display of VA scores and confidence intervals for all years for which VA information is available for that subject. 15

18 The table below displays the detailed VA information of the user s school of the same subject over the years, including the VA score, the 95% confidence interval, the VA performance and the stanine. Years Value-added Scores 95% Confidence Interval Value-added Lower Bound Upper Bound Performance Stanine (1-9) Above average (+) On a par with average (0) On a par with average (0) 5 This report is particularly useful for identifying consistently high or low performance over the years, but caution should be exercised to avoid reading too much into seemingly random fluctuations from year to year. 4. & 5. Multi-Subject Graph and Table The Multi-Subject Graph and Table display the VA information of all subjects in a single year. The Multi-Subject Graph below provides a graphical display of VA scores and confidence intervals for all subjects with available VA information. 16

19 The Multi-Subject Table below displays the detailed VA information of the user s school in all subjects, including the VA score, the 95% confidence interval, the VA performance, the stanine and the percentage of students analysed. Subject Name Value-added Score 95% Confidence Interval Lower Bound Upper Bound Chinese Language Value-added Performance On a par with average (0) Stanine (1-9) % of Students Analysed English Language Below average (-) Mathematics: Compulsory Part Above average (+) The table below displays the VA information of the user s school in each Key Learning Area (KLA). KLA Chinese Language Education English Language Education Mathematics Education No. of Subjects Offered No. of Subjects in Top 10% VA within KLA No. of Subjects in Top 50% VA within KLA It should be noted that in the table above, the number of subjects offered by the school within each KLA excludes any subjects with fewer than ten students available for VA analysis. In the SVAIS, three combinations under Combined Science (i.e. Biology/Chemistry, Biology/Physics and Chemistry/Physics) are counted as three individual subjects. Starting from 2015 HKDSE Examination, Accounting and Business Management are also counted as two individual subjects. 17

20 Appendix: Calculating VA Scores Multi-level Models The VA score refers to an estimate obtained using regression methods to make statistical adjustments to raw results to control for initial differences in the intake characteristics of students. Under the NAS, VA scores are calculated based on the HKDSE Examination results of students, their prior attainment (AAI) and other characteristics using multi-level models. To understand multi-level models, it is useful to begin with a simple (single-level) regression model. The formula for a simple regression model is given by equation (1) below: y i = β 0 + β 1 x i + e i (1) where subscript i takes values from 1 to n, where each value of i represents an individual student and n is the total number of students in all schools; y i is the actual examination score (say the normalised HKDSE Examination score) for student i; β 0 is the y-intercept, or the examination score when variable x i equals zero; β 1 is the slope of the regression line, or the coefficient of variable x i ; x i is the control variable (say student prior attainment); and e i is the error term. The expected score of student i, E(y i ) is given by equation (2) below: E(y i ) = β 0 + β 1 x i (2) If we subtract equation (2) from equation (1), y i E(y i ) = β 0 + β 1 x i + e i ( β 0 + β 1 x i ) y i E(y i ) = e i It is observed that the difference between student i s actual examination score and the expected score is the error term, e i. It is referred to as a residual because it is the part 18

21 of the score that is not predicted by the fixed part of the model represented by equation (2). It is typically assumed that the residuals follow a normal distribution with a mean of zero and common variance, i.e. e i ~N(0, σ e 2 ). Proceeding now to a simple multi-level model, we can rewrite equation (1) as y ij = β 0 + β 1 x ij + (u j + e ij ) (3) where a new subscript j takes values from 1 to m, where each value of j represents an individual school and m is the total number of schools; y ij is the actual examination score (say the normalised HKDSE Examination score) for student i in school j; x ij is the control variable (say student prior attainment) of student i in school j; and u j is the difference between school j s actual intercept and the overall mean value β 0. The critical feature of equation (3) is the u j term which is known as a level 2 residual. It is the existence of the two residuals - the level 2 school residual, u j and the level 1 student residual, e ij - which identifies equation (3) as a multi-level model. Furthermore, it is through the estimation of level 2 residuals u j that school VA estimates can be obtained. The VA Model The VA model adopted under the NAS is very similar to equation (3). The only difference is that it includes more control variables on top of student prior attainment. A full list of control variables currently used is given in Unit 3. The VA model is represented in equation (4) below: y ij = β 0 + β 1 x 1ij + β 2 x 2ij + + β n x nij + (u j + e ij ) (4) where y ij is the normalised HKDSE Examination score for student i in school j; β 0 is a constant term, indicating the overall mean examination score when all control variables (x 1,...,x n ) included in the model equals zero; β 1 x 1ij is the coefficient and the value of student prior attainment (AAI) for student i in school j; 19

22 β 2 x 2ij,, β n x nij are the coefficients and the values of other control variables for student i in school j; u j is the level 2 residual term indicating school j s effect on student performance; and e ij is the level 1 residual term for student i in school j, indicating that part of the score that could not be explained by factors already included in the model. The model assumes that the outcome variable (y ij ) and the residuals (u j, e ij ) are normally distributed. In addition, it is assumed that level 1 and level 2 residuals are not correlated. The distributional assumptions of the model may be summarised as: u j ~N(0, σ u 2 ); e ij ~N(0, σ e 2 ) In the model represented by Equation (4), a school s VA score is obtained by estimating the school residual term u j, which is then normalised and placed on a scale that ranges from -10 to +10. Contact Us For enquiries about school VA information, please contact the Indicators Section of the Quality Assurance & School-based Support Division at the EDB. Address: Room 1214, 12/F, Wu Chung House, 213 Queen s Road East, Wan Chai, Hong Kong Tel: (852) or (852) Fax: (852) indicators@edb.gov.hk 20