Version: 7 November Interpreting NAPLAN results and setting targets

Transcription

1 Version: 7 November 2011 Interpreting NAPLAN results and setting targets 1. EMSAD Draft at Nov 2011

2 INTRODUCTION The purpose of this document is to assist the principal and school self evaluation committee to interpret their NAPLAN results using SMART data analysis tools. It also offers suggestions on how to write achievable targets for literacy and numeracy based on NAPLAN data. 2. EMSAD Draft at Nov 2011

3 USING SMART DATA ANALYSIS Three SMART data analysis tools: o o o Percentage in Bands, Student Growth, and Means & Standard Deviations, will assist schools in developing targets based around improving the growth data for students. When setting targets it is important to remember that each year group is not retested until two years later. Each year group will be unique but the school improvement should be focused on improvement for individual student results. Percentage in Bands Trend Data Student Growth Means and standard deviation Growth data is used by different groups within the school: Executive/Teacher Principal How do our results compare with the state? How do our results compare with the region? What are our strengths in teaching? What areas do we need to focus on? Can we see the impact of changed teaching practice? Principal / School Self Evaluation team Where do we need to focus our teaching to improve student learning outcomes? What needs to be done to equal or better state growth? What professional learning and resources need to be allocated to address our targets? Are our standards and expectations high enough to address students from all backgrounds? Are we setting SMART targets around improving our growth? 3. EMSAD Draft at Nov 2011

4 Using Percentage in Bands for target setting The percentage in analysis provides detailed information on student groups and their performance relative to the state, region, and SEG in specific performance. There are three years worth of data displayed at once, which allows the school to see if there have been changes in the number of students achieving in the various performance. The information in the percentage in analysis can be of importance for schools that are performing below the state mean. The graphs may show improved school performance through the movement of students from lower into higher. As an example, the graph below shows increased percentages in the top two over three years (51% in 2008; 62% in 2009; 67% in Targets can be based around the percentage of student performance in. This is applicable in schools larger than ten students in the testing group. Schools can set targets around increasing student performance into higher. Where this is used, the increases should be a band or more as a target for Year 5 to Year 7 and Year 7 to Year 9 and two or more for Year 3 to Year 5. The following target was written after the school below considered the graph and the target setting tool: Increase the proportion of Year 3 students achieving in the proficiency for reading from 62% in 2010 to 65% in The target setting tool helped the school decide on a feasible target for the coming year. Note: Very small schools needs to be careful as a single student may be worth a high percentage. E.g. With four students each student is worth 25% (See section on small schools.) 4. EMSAD Draft at Nov 2011

5 Using Trend Data / Means and Standard Deviations for target setting Note: Both these analysis tools display information about the school mean. Because the mean is a measure of the middle, half of all schools in the state will be below the mean. In NSW there is approximately 1600 government primary and 400 secondary schools. If your school is below the mean it does not indicate it is a failure. Where schools are of concern is when they are below the state mean and when the average growth for students is not at state or better. It is important to note that schools can have improved growth above the state average growth for a year level. While this is an improvement for individual students as it indicates the school has taken them from lower band to a higher one, it could be that students are still operating well below what they are capable. When analysing mean, consider the size of the group. In small groups the mean can be affected by a number of either high or low achieving students. If your school has a small group of students, other data sources should be analysed to develop a better understanding of actual student and school achievement. The use of growth is more appropriate for small schools. A large standard deviation indicates a wide range of scores and a small standard deviation indicates a cluster around the mean score. When comparing means, what is considered an appreciable difference? 5. EMSAD Draft at Nov 2011

6 To answer this, look at how far the schools mean sits from the state mean, in terms of the proportion of the state standard deviation. As a rule, if the school s mean lies within 0.2 standard deviations of the state mean, then there is no appreciable difference. Beyond 0.2, the effect may be large enough to warrant an examination of contributing factors. In this numeracy example: The difference between the school and state mean is: = 17.7 When 17.7 is divided by the state mean 88.7 the result is Since this result is within 0.2 of the state standard deviation there is not a difference worthy of further consideration. This information should be considered in the school context and other school based assessment data. To easily and quickly analyse the means and standard deviations data, use the NAPLAN Analysis Spreadsheet which can be accessed from the EMSAD website a2f2f e e6e73772e e6175$$/directorates/schoi mpro/emd/npa/pubs/sse/naplan%20analysis%20tool.xlsx The Analysis spreadsheet will calculate and show your results using a range of five colours dark green / green (well above and above state average), yellow (within state average) and red / dark red (problem and severe problem). At a glance you will be able to determine which aspect needs further investigation. Using Student Growth for target setting Data from this section is particularly important to all schools. Schools that are not achieving the state mean may still be providing the environment for students to achieve the expected growth in reading and numeracy. Conversely a school that may be at or above the state mean may not have appropriate growth occurring for their students. Growth on the NAPLAN scales varies depending on prior scores and because of this, a measure based on the percentage of students achieving expected growth is more useful for diagnostic and school planning purposes than a measure based on average growth alone. In 6. EMSAD Draft at Nov 2011

7 general, there is a tendency for expected growth to be higher when you start with a low prior score than from a higher prior score. In using the growth data in SMART, it is important to recognise that: most students have a growth figure in a relatively small range around the state average the 2010 expected growth value thresholds should be considered interim until sufficient data are available over time to provide confidence in the measure. If the box with the displayed unmatched students is ticked, students who were not previously at the school for the last testing period will be indicated on the graph. Growth Charts There are corresponding growth charts that display student growth for the different aspects. These graphs provide a wealth of information on student performance and school performance. Further information is provided in a tabular form below the graph. Average Scaled Score Growth: There is a table at the bottom left that displays the average scaled score growth achieved by the students who were tested. It is separated into state, region and school growth and is also indicated for the whole group, and various selected groups. Schools need to be aware of their group size as small group results may not be valid, particularly if under 10 students. 7. EMSAD Draft at Nov 2011

8 The SSG is where a group of 5 or more selected schools have been used to compare results. Percentile Ranges: The expected minimum growth in scores that is expected for the testing interval (approximately two years apart) is the percentage of school students who did not reach the minimum state growth. Less than 25 th indicates the percentage of school matched students in the bottom 25% of the state for growth 25 th 75 th indicates the percentage of school students in the middle 50% of the state, this is also known as the inter-quartile range where half of all students score. 75 th and above indicates the percentage of the school students in the top 25% of the state. The example below indicates that too many students are in the lowest percentile, less in the middle and less in the top. Targets could be set to address this imbalance and to ensure the school performance matches the expected, or is better with a decrease in the number for the lowest percentile and higher in the middle and the top. Percentile Ranges School Result Less than 25 th 42.9 (Should be 25 or less) 25 th 75 th 35.7(Should be 50 or less) 75 th and above 21.4 (Should be 25 or higher) In the example below there are no students in the lowest percentile and more in the middle and the top indicating the students have been progressed. Percentile Ranges School Result < 25 th 0 25 th 75 th th and above 30 Expected Growth: The amount next to the blue wording is the percentage of school students who did not reach the expected state growth. The amount next to the green wording is the percentage of school students who reached or exceeded the state growth. Schools with negative growth Schools will often mention that students who are often operating at the top end in performance on the testing program can easily decrease on testing two years later if they make a few mistakes. This was often the case in the Basic Skills Test where a student at the top in Year 3 could easily show negative growth by making a few mistakes in Year 5. While this is true, it is no excuse for student underperformance for those students not at the top end where the majority of students are located. Schools need to look deeply into teaching performance, the learning environments and individual students where the data shows negative growth or there is an increase of students into the lower performance. 8. EMSAD Draft at Nov 2011

9 Small School Targets Small schools (less than ten students in the testing Year level) have to be very cautious in using the statistical data provided from the NAPLAN. Means and standard deviations are not necessarily relevant and can be greatly influenced by individual student performance. Growth is often more applicable for schools when setting improvement targets as it is more individual specific. Using percentages in the student performance in is problematic as an individual student is worth a large percentage. In a school with five students being tested, each student is worth 20%. Thus setting targets of increasing student performance by would have to be in 20% groups at least. For example having an increase in the top band by 5% is meaningless as it would have to be at least 20% to represent the student numbers for a school of five students. If one student leaves the school the target percentages would be no longer applicable. Targets that are more appropriate focus on students will be achieving the appropriate minimum growth for the Year group being tested. Thus targets such as; students will achieve growth equivalent or better to state level, are more appropriate to schools with less than 10 students. Because school plans are public domain documents and small schools have few students it may be possible for people to identify the individual performance of students. In these cases it is better not to specify the baseline in a document such as the school plan. Baseline data should be kept separately for discussions with the School Education Director. QUANTIFYING AND MONITORING TARGETS Extract from the School planning guidelines Monitoring of the school plan focuses on progress towards the intended outcomes of the three year plan through the achievement of annual targets School education directors can assist principals and school communities in identifying and setting targets. Principals also have access to a statistical tool that provides school staff with past student performance data and can support the development of literacy and numeracy targets based on this data. When deciding on how much to set as the goal for improvement it is useful to consider your baseline data. The following indicate graphical interpretations of achievable target trajectories from SMART data. It is reasonable to assume that state and regional trends will be relatively stable, thus in the examples below we have drawn state and regional trajectories on that basis. 9. EMSAD Draft at Nov 2011

10 Data reported in SMART Means Graphical example Improving the overall mean Assumed state and regional trajectory Targets for three years Percentage in Bands Improving the percentage in the top band Achievable trajectory for school Targets for three years Assumed state and regional trajectory Growth Improving the expected growth 100% 80% 60% 40% 20% 0% Achievable trajectory for school % less than expected growth % greater than or equal to expected growth The target setting tool uses the Percentage in Bands methodology above to provide advice about the NSW state targets for reading and numeracy. It also can be used in assisting schools to set targets for improved performance on National Assessment Program Literacy and Numeracy (NAPLAN). It should be used once the school has set its school identified priority areas and outcomes. A significant benefit of the target setting tool is that it allows target setting in term of movement of whole students across the achievement. As well, there is a means to allow schools to adjust for local knowledge of the student group s history and capacity to achieve the levels of performance that other student groups have achieved over time. The tool can be used to monitor achievement of targets over time. 10. EMSAD Draft at Nov 2011

11 The target setting tools for your school, preloaded with your schools NAPLAN data, are downloadable from the ASR Access tool 11. EMSAD Draft at Nov 2011

12 EXAMPLE 1 A large primary school in regional NSW (Demonstration school 13) Year 2010 Year 5 Aspect Numeracy Percentages in Band School State Region Did not meet minimum standard (Bottom band for Year Group) Above minimum standard (Middle three ) Proficient (Top two for Year Group) Focus Questions Results How is school performance in the bottom band? How is school performance in the middle? How is school performance in the top 2? How big is the tested group? A higher percentage of all students are below the national minimum standard (Band 3 - school 6%, state 4%) A higher percentage of students are in the middle (Bands 4, 5, 6 - school 78%, state 64%) A lower percentage of students are in the proficient (Band 7 & 8 - school 16%, state 32%) 12. EMSAD Draft at Nov 2011

13 9% of boys are performing below the national minimum standard in Band 3 compared with state - 4% 3% of girls are performing below the national minimum standard in Band 3 compared to state at 4% 73% of boys are performing in the middle - 4, 5, 6 compared with state at 59% 84% of girls are performing in the middle 4, 5, 6 compared with state at 68% 19% of boys are performing in the proficient compared with state at 35% 13% of girls are performing in the proficient compared with state at 28% Mean and Standard Deviation Data Aspect Numeracy Year 5 State Mean Standard Deviation Region Mean Standard Deviation School Mean Standard Deviation EMSAD Draft at Nov 2011

14 Focus Questions Is data within state parameters? Is data of concern? Is data of serious concern? How big is the tested group? [Use calculation spreadsheet to determine] Results From this mean data the school performance in this aspect is worthy of investigation. (The NAPLAN Analysis Tool indicates that there is a problem.) Data from Student Growth chart 14. EMSAD Draft at Nov 2011

15 Average Scaled Score Growth Category State Region School All Boys Girls Focus Questions Is school data above state average growth? Is any particular group doing better? How big is the tested group? Results The school has a lower average growth than the state but comparable to region. Boys growth is well below the state and region average. Girl s growth is just under state average and above region growth. Percentile Ranges Percentile Ranges School Results <25 th 31.25% 25 th 75 th 53.13% 75 th and above 15.63% Focus Questions Is school data 25, 50, 25? Is school data more towards and 75+? How big is the tested group? Result More students fall in the <25 th percentile range. Less in the 75 th or above range % of boys are in the less than 25 th percentile 31.25% of girls are in the less than 25 th percentile 48.78% of boys are in the 25 th 75 th percentile 53.13% of girls are in the 25 th 75 th percentile 12.2% of boys are in the 75 th and above 15.63% of girls are in the 75 th and above percentile Expected growth Less than expected growth 46.9% Greater than or equal to expected 53.1% growth 15. EMSAD Draft at Nov 2011

16 Focus Questions Is school data higher in >=? How big is the tested group? Result Slightly more than half of all students are achieving equal to or expected growth. Almost half of all students have not achieved expected growth. Boys with less than expected growth 53.7% Girls with less than expected growth 46.9% Boys with greater than or equal to expected growth 46.3% Girls with greater than or equal to expected growth 53.1% Target Setting Target based on Percentages in student Target is to reduce students in the below minimum band and increase in the proficient band School Mean and Standard Deviation From the calculation spreadsheet based on data being with 0.2 or 0.5 Standard Deviations, indicating if data is within state parameters, of concern or of serious concern Targets should set increase to move school from: serious concern to of concern of concern to within state parameters within state to equal or better than state School Average Growth Target should be equal to and higher than state average All groups should be performing at state average or better Percentile Ranges Target should be better than the 25, 50, 25. Targets should favour the middle or higher percentiles Minimum Growth Target is based on 100% achieving minimum growth Target should focus on more students at or above state minimum growth 16. EMSAD Draft at Nov 2011 Advice on writing target Target can be specified in number of students reduced in the below minimum band or increased in the proficient band. This can be against the region or state Target should show an increase, specified in marks, necessary to change from the various levels [use calculation spreadsheet to alter school mean] Eg: does the school need 8 marks to change from serious concern to of concern? Can be specified as a number but is better written as equal to or better state average growth Target can be specified in reduction of students in the lower percentiles and increase in the higher percentiles Target can be specified in more of the group at the minimum growth level

17 Targets From this data there are a number of possible targets: Decrease the percentage of students who do not meet national minimum standards in numeracy from 6% in 2010 to y% in Increase the percentage of students achieving in the proficiency for numeracy from 16% in 2010 to y% in Raise the schools average growth in numeracy (particularly boys ) to be equal to or above state average growth by By 2011, increase the percentage of students achieving in the 75 th or above percentile range by x% By 2011, reduce the percentage of students achieving in the less than the 25 th percentile by y% The target setting tool should be used to determine achievable data for EMSAD Draft at Nov 2011

18 EXAMPLE 2 A large, metropolitan high school (Demonstration school 26) Year 2010 Year 7 Aspect Writing Percentages of students in Band School State Region Did not meet minimum standard N/A (Bottom band for Year Group) Above minimum standard (Middle three ) N/A Proficient (Top two for Year Group) N/A Focus Questions How is school performance in the bottom band? How is school performance in the middle? How is school performance in the top 2? How big is the tested group? Results A higher percentage of students are performing in the bottom band compared to state (Band 4, school 16%, state 10%) A higher percentage of students are performing in the middle compared to state (Bands 5, 6, 7 School 71%, state 65%) A lower percentage of students are performing in the top two compared to state (Bands 8, 9 school 15%, state 24%) 20% of boys are performing in bottom band 18. EMSAD Draft at Nov 2011

19 10% of girls are performing in bottom band 71% of boys are performing in middle 69% of girls are performing in middle 10% of boys are performing in top 26% of girls are performing in top Mean and Standard Deviation Data Aspect Writing Year 7 State Mean Standard Deviation Region Mean Standard Deviation School Mean Standard Deviation EMSAD Draft at Nov 2011

20 Focus Questions Is data within state parameters? Is data of concern? Is data of serious concern? How big is the tested group? [Use calculation spreadsheet to determine] Results From this mean data the school performance in this aspect is worthy of investigation. (The NAPLAN Analysis Tool indicates that there is a problem.) Data from Growth chart Average Scaled Score Growth Category State Region School All Boys Girls Focus Questions Is school data above state average growth? Is any particular group doing better? How big is the tested group? Results The schools average growth is below state and region. Boys average growth is well below state and region. Girls average growth is close to state and region. 20. EMSAD Draft at Nov 2011

21 Percentile Ranges Percentile Ranges School Results <25 th 27.9% 25 th 75 th 54.68% 75 th and above 18.23% Focus Questions Is school data 25, 50, 25? Is school data more towards and 75+? How big is the tested group? Result Fewer students are performing in the top percentile range. More boys in the bottom percentile range (30.71%) Fewer boys in the top percentile range (13.57%) Fewer girls are in the bottom percentile range (19.05%) More girls are in the top percentile range (28.57%) Expected growth Less than expected growth 53.2% Greater than or equal to expected 46.8% growth Focus Questions Is school data higher in >=? How big is the tested group? Result 46.8% of students achieved expected growth. 43.6% of boys achieved expected growth. 54.0% of girls achieved expected growth. Target Setting Target based on Percentages in student Target is to reduce students in the below minimum band and increase in the proficient band Advice on writing target Target can be specified in number of students reduced in the below minimum band or increased in the proficient band. This can be against the region or state 21. EMSAD Draft at Nov 2011

22 School Mean and Standard Deviation From the calculation spreadsheet based on data being with 0.2 or 0.5 Standard Deviations, indicating if data is within state parameters, of concern or of serious concern Targets should set increase to move school from: serious concern to of concern of concern to within state parameters within state to equal or better than state School Average Growth Target should be equal to and higher than state average All groups should be performing at state average or better Percentile Ranges Target should be better than the 25, 50, 25. Targets should favour the middle or higher percentiles Minimum Growth Target is based on 100% achieving minimum growth Target should focus on more students at or above state minimum growth Target should show an increase, specified in marks, necessary to change from the various levels [use calculation spreadsheet to alter school mean] Eg: does the school need 8 marks to change from serious concern to of concern? Can be specified as a number but is better written as equal to or better state average growth Target can be specified in reduction of students in the lower percentiles and increase in the higher percentiles Target can be specified in more of the group at the minimum growth level Targets From this data there are a number of possible targets: Decrease the percentage of students performing in the bottom band for writing from 16% in 2010 to x% in Increase the percentage of students performing in the proficiency for writing from 15% to y% by Raise the schools average growth in numeracy (particularly boys ) to be equal to or above state average growth by By 2011, increase the percentage of students achieving in the 75 th or above percentile range from 18.23% to x%. By 2011, reduce the percentage of students achieving in the less than the 25 th percentile from 27.9% to y%. Increase the percentage of students achieving above expected growth from 46.8% to y by EMSAD Draft at Nov 2011

23 EXAMPLE 3 A small school in a semi-rural community on the metropolitan fringe (Demonstration school 7) Year 2010 Year 5 Aspect Reading Percentages of students in Band School State Region Did not meet minimum standard 17 7 N/A (Bottom band for Year Group) 23. EMSAD Draft at Nov 2011

24 Above minimum standard (Middle three ) Proficient (Top two for Year Group) N/A N/A Focus Questions How is school performance in the bottom band? How is school performance in the middle? How is school performance in the top 2? How big is the tested group? Results A higher percentage of students are below the national minimum standard (Band 3 - school 17%, state 7%) A lower percentage of students are in the middle (Bands 4, 5, 6 school 51%, state 59%) A similar percentage of students are in the proficient (Band 7 & 8 - school 34%, state 33%) However, the cohort consists of only 6 students. One student, below the national minimum standard resulted in the 17% result. Mean and Standard Deviation Data Aspect Reading Year 5 State Mean Standard 82.9 Deviation Region Mean Standard 81.4 Deviation School Mean Standard EMSAD Draft at Nov 2011

25 Deviation Focus Questions Is data within state parameters? Is data of concern? Is data of serious concern? How big is the tested group? [Use calculation spreadsheet to determine] Results From this mean data the school performance in this aspect is worthy of investigation. (The NAPLAN Analysis Spreadsheet indicates that there is a problem.) However, it is important to note that the mean is unreliable unless there are at least ten students. Data from Growth chart 25. EMSAD Draft at Nov 2011

26 Average Scaled Score Growth Category State Region School All Boys Girls Focus Questions Is school data above state average growth? Is any particular group doing better? How big is the tested group? Results The school has a higher average growth than the state. Boys growth is well above the state average. Percentile Ranges Percentile Ranges School Results <25 th 20% 25 th 75 th 80% 75 th and above 0 Focus Questions Is school data 25, 50, 25? Is school data more towards and 75+? How big is the tested group? Result The school has most students in the 25 th 75 th percentile. There are no students in the 75 th and above percentile. Expected growth Less than expected growth 20% Greater than or equal to expected 80% growth Focus Questions Is school data higher in >=? How big is the tested group? Result All boys show greater than or equal to expected growth. One girl has made less than expected growth. One girl was unmatched. Target Setting Target based on Advice on writing target 26. EMSAD Draft at Nov 2011

27 Percentages in student Target is to reduce students in the below minimum band and increase in the proficient band School Mean and Standard Deviation From the calculation spreadsheet based on data being with 0.2 or 0.5 Standard Deviations, indicating if data is within state parameters, of concern or of serious concern Targets should set increase to move school from: serious concern to of concern of concern to within state parameters within state to equal or better than state School Average Growth Target should be equal to and higher than state average All groups should be performing at state average or better Percentile Ranges Target should be better than the 25, 50, 25. Targets should favour the middle or higher percentiles Minimum Growth Target is based on 100% achieving minimum growth Target should focus on more students at or above state minimum growth Target can be specified in number of students reduced in the below minimum band or increased in the proficient band. This can be against the region or state Target should show an increase, specified in marks, necessary to change from the various levels [use calculation spreadsheet to alter school mean] Eg: does the school need 8 marks to change from serious concern to of concern? Can be specified as a number but is better written as equal to or better state average growth Target can be specified in reduction of students in the lower percentiles and increase in the higher percentiles Target can be specified in more of the group at the minimum growth level Targets From this data there are a number of possible targets: Increase the proportion of students achieving in the proficiency for reading by 2011 School trend data for reading shows positive increases from 2010 Reduce the proportion of students making less than expected growth by EMSAD Draft at Nov 2011

28 The following two examples the EMSAD model for writing small school targets EXAMPLE 4 A small school in a rural, regional centre Year 2010 Year 3, 5 Aspect Various aspects 28. EMSAD Draft at Nov 2011

29 2008 3/10 students in top three /17 students in top three /7 students in top three In 2011 there will be 10 students sitting NAPLAN (information from ASR). Target setting Numbers in top three % aggregated /10 11/17` 6/7 20/34=59% Reporting Numbers 11/17 6/7 7/10 in top three % aggregated 24/34=71% Target: Increase the three year aggregated percentage of Year 3 students achieving in the top three to 71% (from the average of 59%) Year 5 Writing 29. EMSAD Draft at Nov 2011

30 2008 4/8 students in top three /12 students in top three /7students in top three In 2011 there will be 14 students sitting NAPLAN. Target setting Numbers in top three % aggregated /8 4/12 1/7 9/27=33% Reporting Numbers in 4/12 1/7 12/14 top three % aggregated 17/33=52% 1n students sat the Year 3 writing. 14 achieved in the top three. A prediction of 11/14 is made for 2011 based on school based data, and knowledge of student background. Target: Increase the three year aggregated percentage of Year 5 students achieving in the top three to 52% (from the average of 33%) 30. EMSAD Draft at Nov 2011

31 Year 5 Writing 2008 No data 2009 No data % less than expected growth Target: Increase the percentage of students achieving greater than or equal to expected growth from 0% in 2010 to 33% in (The figure 33% was taken from the school s ASR target.) 31. EMSAD Draft at Nov 2011

32 Year 5 Numeracy In 2009, 17 Year 3 students sat the numeracy test. 11 students performed in the top 3. In 2011, 14 Year 5 students are expected to sit the NAPLAN numeracy test. 32. EMSAD Draft at Nov 2011

33 Target: Increase the percentage of the 2009 Year 3 cohort in the top 3 NAPLAN from 65% to 79% in Year (Aim for 11/14 students in top three ) EXAMPLE 5 A small, rural central school Year 2010 Years 3, 5, 7 Aspect Various aspects Year 3 Reading /9 students in top three /9 students in top three /8 students in top three In 2011 there will be approximately 10 students sitting NAPLAN (information from 2008 ASR). Target setting Numbers in top three % aggregated /9 5/9 3/8 13/26 = 50% Reporting Numbers 5/9 3/8 7/10 in top three % aggregated 15/27=56% 33. EMSAD Draft at Nov 2011

34 Target: Increase the three year aggregated percentage of Year 3 students achieving in the top three to 56% (from the average of 50%) Year 3 Writing /9 students in top three /8 students in top three /8 students in top three Anticipated number for 2011 is 10 students. Target setting Numbers in top three % aggregated /9 7/8 4/8 17/25=68% Reporting Numbers 7/8 4/8 7/10 in top three % aggregated 18/26=69% 1n students sat the Year 3 writing. 4 achieved in the top three. A 34. EMSAD Draft at Nov 2011

35 prediction of 7/10 is made for 2011 based on school based data, and knowledge of student background. Target: Increase the three year aggregated percentage of Year 3 students achieving in the top three to 69% (from the average of 68%) Year 3 Spelling /9 students in top three /8 students in top three /8 students in top three Target setting Numbers in top three % aggregated /9 6/8 4/8 13/25=52% Reporting Numbers in top three 6/8 4/8 7/ EMSAD Draft at Nov 2011

36 % aggregated 17/26=65% Target: Increase the three year aggregated percentage of Year 3 students achieving in the top three to 56% (from the average of 50%) Year 3 Grammar and Punctuation /9 students in top three /8 students in top three /8 students in top three Target setting Numbers in top three % aggregated /9 5/8 5/8 15/25=60% 36. EMSAD Draft at Nov 2011

37 Reporting Numbers 5/8 5/8 7/10 in top three % aggregated 17/26=65% Target: Increase the three year aggregated percentage of Year 3 students achieving in the top three to 65% (from the average of 60%) Year 3 Numeracy /9 students in top three /9 students in top three /8 students in top three Target setting Numbers in top three /9 5/9 6/8 37. EMSAD Draft at Nov 2011

38 % aggregated 18/26=69% Reporting Numbers 5/9 6/8 8/10 in top three % aggregated 19/27=70% Target: Increase the three year aggregated percentage of Year 3 students achieving in the top three to 70% (from the average of 69%) Year 5 Reading /10 students in top three /4 students in top three /8 students in top three Target setting Numbers in top three % aggregated /10 2/4 3/8 10/22=45% 38. EMSAD Draft at Nov 2011

39 Reporting Numbers 2/4 3/8 5/8 in top three % aggregated 10/20=50% In 2009, 9 Year 3 students sat the reading test. 5 students performed in the top 3. In 2011, 8 Year 5 students are expected to sit the NAPLAN reading test. Target: Increase the percentage of the 2009 Year 3 cohort in the top 3 NAPLAN from 56%% to 63% in Year (Aim for 5/8 students in top three ) Increase the percentage of students achieving greater than or equal to expected growth from 42.9% in 2010 to above 50% in Year 5 Writing /10 students in top three /4 students in top three /8 students in top three Target setting EMSAD Draft at Nov 2011

40 Numbers in top three % aggregated 5/10 2/4 7/8 14/22=64% Reporting Numbers 2/4 7/8 7/8 in top three % aggregated 16/20=80% In 2009, 8 Year 3 students sat the reading test. 7 students performed in the top 3. In 2011, 8 Year 5 students are expected to sit the NAPLAN writing test. Target: Increase the percentage of the 2009 Year 3 cohort in the top 3 NAPLAN from 88% to 86% in Year (Aim for 7/8 students in top three ) Increase the percentage of students achieving greater than or equal to expected growth from 42.9% in 2010 to above 50% in Year 5 Spelling /10 students in top three 40. EMSAD Draft at Nov 2011

41 2009 2/4 students in top three /8 students in top three Target setting Numbers in top three % aggregated /10 2/4 3/8 10/22=45% Reporting Numbers 2/4 3/8 5/8 in top three % aggregated 10/20=50% In 2009, 4 Year 3 students sat the reading test. 2 students performed in the top 3. In 2011, 8 Year 5 students are expected to sit the NAPLAN spelling test. Target: Increase the percentage of the 2009 Year 3 cohort in the top 3 NAPLAN from 50% to 63%% in Year (Aim for 5/8 students in top three ) Increase the percentage of students achieving greater than or equal to expected growth from 42.9% in 2010 to above 50% in Year 5 Grammar & Punctuation 41. EMSAD Draft at Nov 2011

42 2008 5/10 students in top three /4 students in top three /8 students in top three Target setting Numbers in top three % aggregated /10 2/4 2/8 9/22=41% Reporting Numbers 2/4 2/8 5/8 in top three % aggregated 9/20=45% In 2009, 4 Year 3 students sat the reading test. 2 students performed in the top 3. In 2011, 8 Year 5 students are expected to sit the NAPLAN grammar % punctuation test. Target: Increase the percentage of the 2009 Year 3 cohort in the top 3 NAPLAN from 50% to 63% in Year (Aim for 5/8 students in top three ) Increase the percentage of students achieving greater than or equal to expected growth from 28.6% in 2010 to X% in EMSAD Draft at Nov 2011

43 Year 5 Numeracy /10 students in top three /4 students in top three /8 students in top three Target setting Numbers in top three % aggregated /10 2/4 3/8 10/22=45% Reporting Numbers 2/4 3/8 5/8 in top three % aggregated 10/20=50% In 2009, 4 Year 3 students sat the reading test. 2 students performed in the top 3. In 2011, 8 Year 5 students are expected to sit the NAPLAN numeracy test. Target: Increase the percentage of the 2009 Year 3 cohort in the top 3 NAPLAN from 50% to 63% in Year (Aim for 5/8 students in top three ) Increase the percentage of students achieving greater than or equal to expected growth from 57.1% in 2010 to X% in EMSAD Draft at Nov 2011

44 Year 7 Reading /3 students in top three /9 students in top three /7 students in top three Target setting Numbers in top three % aggregated /3 3/9 5/7 11/19=58% Reporting Numbers 3/9 5/7 5/7 in top three % aggregated 13/23=57% In 2009, 3 Year 3 students sat the reading test. 2 students performed in the top 3. In 2011, 8 Year 5 students are expected to sit the NAPLAN reading test. Target: Increase the percentage of the 2009 Year 3 cohort in the top 3 NAPLAN from 50% to 63% in Year (Aim for 5/8 students in top three ) 44. EMSAD Draft at Nov 2011

45 Increase the percentage of students achieving greater than or equal to expected growth from 57.1% in 2010 to X% in INTERPRETING RESULTS AND SETTING TARGETS TARGET SETTING SCAFFOLD Year Aspect Cohort Percentages of students in Band School State Region Did not meet minimum standard (Bottom band for Year Group) Above minimum standard (Middle three ) Proficient (Top two for Year Group) Focus Questions How is school performance in the bottom band? How is school performance in the middle? How is school performance in the top 2? How big is the tested group? Mean and Standard Deviation Data Aspect State Mean School Mean Focus Questions Is data within state parameters? Is data of concern? Is data of serious concern? How big is the tested group? [Use calculation spreadsheet to determine] Results Year Standard Deviation Standard Deviation Results Data from Growth chart Average Scaled Score Growth Category State Region School All Boys 45. EMSAD Draft at Nov 2011

46 Girls Focus Questions Is school data above state average growth? Is any particular group doing better? How big is the tested group? Results Percentile Ranges Percentile Ranges <25 th 25 th 75 th 75 th and above Focus Questions Is school data 25, 50, 25? Is school data more towards and 75+? How big is the tested group? School Results Result Expected growth Less than expected growth Greater than or equal to expected growth Focus Questions Is school data higher in >=? How big is the tested group? Target Setting Target based on School Mean and Standard Deviation From the calculation spreadsheet based on data being with 0.2 or 0.5 Standard Deviations, indicating if data is within state parameters, of concern or of serious concern Targets should set increase to move school from: serious concern to of concern of concern to within state parameters within state to equal or better than state School Average Growth Target should be equal to and higher than state average All groups should be performing at state average or better Percentile Ranges Target should be better than the 25, 50, 25. Targets should favour the middle or higher percentiles Minimum Growth 46. EMSAD Draft at Nov 2011 Result Advice on writing target Target should show an increase, specified in marks, necessary to change from the various levels [use calculation spreadsheet to alter school mean] Eg: does the school need 8 marks to change from serious concern to of concern? Can be specified as a number but is better written as equal to or better state average growth Target can be specified in reduction of students in the lower percentiles and increase in the higher percentiles Target can be specified in more of the group

47 Target is based on 100% achieving minimum growth Target should focus on more students at or above state minimum growth Percentages in student Target is to reduce students in the below minimum band and increase in the proficient band at the minimum growth level Target can be specified in number of students reduced in the below minimum band or increased in the proficient band. This can be against the region or state Target Setting 47. EMSAD Draft at Nov 2011

48 Using Item Analysis to undertake a review of student performance Once your school has reviewed the means and standard deviations, percentages in, and school growth information for each year group, you will have a clear idea of how the group has performed compared to the state, region, SEG etc. Your school will have discussed where they need to focus their resources to bring about improvement in student performance. An integral part of this review, and particularly important to classroom teachers, is the information found in the Item Analysis. Teachers can use this information to closely examine each aspect. It will help them to determine what their students know, what needs to be taught or retaught, and access strategies to help teach skills / concepts. 48. EMSAD Draft at Nov 2011

49 Note: The information in the Item Analysis is important to all teachers, not just the teachers of the cohort tested. The following table shows where a stage could start their analysis Stage 1 - Year 3 data Stage 2 Year 5 data Stage 3 Year 7 data Stage 4 Year 9 data The NAPLAN tests are held early in the school year with students who have just exited a stage of learning. The performance of students can be a reflection of the teaching that took place in the previous stage. It is therefore important for teachers to analyse the types of questions that the students are proving difficult for their students. It may also be useful to check past years and see if there are any trends evident. To assist you with this process a template has been included that may help your team to analyse the questions causing difficulty in your school. Record the outcome, refer to the syllabus and spend some time examining the knowledge and skills required to competently achieve this outcome. 49. EMSAD Draft at Nov 2011

50 Record the skill band. A good idea would be to insert a copy of the NAPLAN question and resource material these can be accessed from Item Analysis Question details. Insert data from the Item Analysis tool for your school, state, region & SEG for this question. This section is very important! Have a discussion with colleagues about the responses your students gave. Use your syllabus to clarify the requirements of the outcome. Examine the questions. Can you determine why a considerable number of students may have given an incorrect answer? See below Once you have discussed the demands of the question and identified possible student misconceptions, discuss the strategies that could be implemented to improve student understanding. This section offers a checklist of areas to consider when analysing student performance. EXAMPLE NAPLAN Item analysis LITERACY QUESTION 50. EMSAD Draft at Nov 2011

51 Question Number: 19 Skill Band: RS2.6 Outcome: RS 2.6: Uses efficiently an integrated range of skills and strategies when reading and interpreting written texts. Infers a character's motivation in a fable. Regional % Correct: 74 State % Correct School % Correct Differen ce Year School State Difference 1 He wanted to punish his son. 12% 4% 8% 2 He was fighting with his son. 10% 4% 6% 3 He was too old to go himself. 27% 13% 14% 4 He wanted his son to learn.* 49% 77% -28% Question: Why did the father send his son into the forest? 51. EMSAD Draft at Nov 2011

52 Demands: Possible reasons for large difference: Students responded to story emotionally a dad sending a child out on his own, foreign concept to children, parents don t leave children alone this is mean. Students don t understand that a fable is a story with a moral. Children didn t ask the question what is this story trying to teach me? Lack of understanding of the word necessity. Perhaps read other stories where fathers are old and seek help from their younger, more-able children. Reasoning - why would a father send a son to do something he was able to do himself? Inability to read the text and guessed the answer. Statements of Learning: When students interpret texts they have the opportunity to draw inferences from directly-stated descriptions and actions. (p. 17) Teaching Strategies: Developing Characters in Narratives Check: Literacy demand (e.g. metalanguage; aspects of grammar) Word/sentence/text level Context; background knowledge Literal Vs inferential comprehension Visual literacy Distractors Teaching strategies, as accessed from the Question Details screen, are an important resource that may be a starting point in investigating strategies to support teaching and learning in the classroom. The teaching strategies have been specifically developed to address the skills and knowledge assessed and are linked to state syllabuses and the NSW Quality Teaching framework. They can be incorporated into school resources. A teaching strategy has been linked to every test item. Clicking on the Teaching Strategies button accesses a bank of literacy and numeracy strategies literacy strategies categorised by skill and numeracy strategies categorised by strand and sub-strand. There are strategies available for the different stages of learning. The bank of teaching strategies also include Supporting Aboriginal Students, Supporting ESL Students and Supporting Students with Learning Difficulties. 52. EMSAD Draft at Nov 2011