Smarter Schools National Partnership Interim Evaluation Summary Trim:12/334675
Trim:12/334675
Background National Partnerships are joint initiatives between the Australian and state governments to improve the quality of education across the country. Queensland is participating in the Smarter Schools National Partnership (SSNP) Agreement. The SSNP is aimed at addressing teacher quality, improving literacy and numeracy and providing support to students from disadvantaged areas. It encompasses three National Partnerships: - Low SES National Partnership (Low SES NP) - Literacy and Numeracy National Partnership (L&NNP) - Improving Teacher Quality National Partnership (ITQ NP) Report structure This report provides a summary of the key findings of an internal evaluation of the following selected reform initiatives: Turnaround Teams Four year School Strategic Plans Literacy and Numeracy Coaches. This report includes: an overview of the findings of a qualitative study 1 of the three NP initiatives listed above an overview of analysis of NAPLAN and PAT M and PAT R 2, student attendance and student retention data for the SSNP schools a précis of more specific findings for each of the NP initiatives general conclusion This interim evaluation can only provide insights into the success of the SSNP. Improvements in literacy and numeracy, measured by instruments such as NAPLAN and PAT-M, PAT-R will need to be measured in a longitudinal data series rather than the relatively short time span of 2010-2011. Data sources Data sources used in the evaluation include qualitative and quantitative data drawn from survey, interviews, observations, text analysis and document analysis, and case studies of 11 SSNP participating schools (see Attachment 2 for overview of source documents). This data has been supplemented by analysis of student performance in using NAPLAN and PAT-M PAT-R data for the SSNP schools (see Attachment 1). 1 Smarter School National Partnership: Queensland Interim Evaluation Report 2010-2011. The evaluation identifies the core features of the SSNP in Queensland schools, enablers and barriers to effective implementation, major outcomes and how effective reforms have been in lifting school performance and the sustainability of the initiatives. 2 Progressive Achievement Tests in Reading (PATR) and Maths (PAT M) have been taken twice year in schools involved in the LN NP program (once in February and once in October). PAT results provide information about the level of achievement attained by students in reading comprehension or mathematics. As students are tested twice in the same year, it provides a clearer picture of student progress within a set time frame. Smarter Schools NP Evaluation Summary - 3 -
Key features of each of the National Partnership Low SES NP The Low SES NP was designed to improve a broad range of outcomes for the target group of students from more disadvantaged, smaller and more geographically remote communities with a higher proportion of Indigenous students. Initiatives included: strategies to attract, develop and support highly qualified principals and teachers to schools in low socio-economic areas tailored and more flexible learning opportunities greater accountability of schools partnerships with parents, other schools, business and communities In Queensland, there are170 schools participating, comprising 131 state schools, 30 Catholic schools and nine independent schools. The 131 participating state schools extend right across Queensland, as far south as Logan, west to Cunnamulla, Mount Isa and Mornington Island, and north all the way to the Torres Strait. A full list of participating schools is available at: http://www.deewr.gov.au/schooling/programs/smarterschools/pages/default.aspx Strategies implemented in Queensland under the low SES National Partnership include: Principal Performance System School-based Strategies and Planning Four Year School Strategic Plan * Closing the Gap Education Strategy Foundation Learning priority area. An external evaluation of this initiative was conducted by Professor Bob Perry from Charles Sturt University. Go to http://deta.qld.gov.au/indigenous/to view the full evaluation report. Turnaround Teams Cape York Aboriginal Australian Academy (CYAAA). An independent external evaluation is due to commence September 2012 and conclude in April 2013. *Findings from internal evaluation included in this report. Under the NP, schools received additional funding to help achieve improvement across a range of areas including: closing the gap of achievement between Indigenous and non-indigenous students literacy, numeracy and science performance of students reporting and engagement of parents and the community student wellbeing and engagement student transition from school to work and further study teacher incentive and workforce planning strategies. Each National Partnership school published a four-year School Strategic Plan on their website outlining their key objectives and strategies for improvement. Smarter Schools NP Evaluation Summary - 4 -
Literacy and Numeracy National Partnership The Literacy and Numeracy Partnership Agreement (L&NNP) is arranged as actions under three reform initiatives: Strong leadership - effective teams High expectations - focused teaching Differentiated intervention - improved learning. The main aim of the Literacy and Numeracy NP was to raise the achievement levels of those students falling behind in NAPLAN testing, through a concerted focus on bringing more students up to or better than the national benchmark. In Queensland, 279 schools participated in the L&NNP with 34 schools also participating in the Low SES NP. The key strategies implemented included: Literacy & Numeracy Coaches* Summer Schools Teaching & Learning Audits Vacation Professional Development *Findings from the internal evaluation included in this report. These reform initiatives focus on building capacity over a number of years. While the focus is on literacy and numeracy, this National Partnership is also building the capacity of school communities to support continual improvement and the implementation of the Australian curriculum. Teacher Quality National Partnership The intent of the Improving Teacher Quality National Partnership was to improve the quality of teaching and leadership in schools, with a specific focus on principals, and maintain a quality teaching workforce. Strategies implemented include: Teacher Education Centres of Excellence Make a Difference Take the Lead Pathways to Teaching This NP is still in the early stages of implementation and useful evaluation data is not yet available. Therefore, no further information on these strategies is included in this paper. Smarter Schools NP Evaluation Summary - 5 -
Evaluation Findings After two years of SSNP activity in Queensland schools, the interim SSNP evaluation of the Turnaround Teams, Four year School Strategic Plans,Literacy and Numeracy Coaches initiatives and 11 case studies points to encouraging early achievements within the NP framework 3. The results from the case studies provide a rich picture of positive culture change that stakeholders attribute to SSNP resources and opportunities. The evaluation has found that the L&NNP and the Low SES NP in selected case study state schools has triggered an unprecedented level of diverse activity at the school level, predominantly through intervention programs targeting the individual student and school-wide changes. It also found evidence of growing acceptance within schools of the need for and focus on, culture change as way to achieve school improvement. In all 11 case study schools, the evaluation found evidence that: Focused quality leadership, appropriate to context, is critical. Quality leadership in NP schools has been the foundation for culture change. Continuing and focused professional development opportunities and positive learning relationships have increased for all stakeholders and reinvigorated enthusiasm for teaching and learning. NP activity has made a positive contribution to increased professional dialogue among stakeholders. After the introduction of other NP interventions such as Teaching and Learning Audits (all schools) and Turnaround Teams (L&NNP schools), most of the L&NNP schools moved into a new phase of interrogating NAPLAN data to identify individual student results rather than school results. There is an emphasis on student-centric learning and strategies to increase community participation and pride in schools. There is an increased capacity building through continuing reflection and refinement of school strategies as well as broader SSNP initiatives such as Literacy and Numeracy Coaches. There are positive signs that sustainability is being built into the strategic approach of more case study schools Most of the Low SES NP case study schools (including those that were also L&NNP) made promising improvements in the schooling experience for disadvantaged students, generally centred on developing innovative partnerships with parents, community, government and non-government agencies through early intervention programs and staffing. 3 The interim SSNP Evaluation Report 2010-2011 provides a summary of initial findings for each case study school. The report includes overall learning to date from the initiatives and case studies, the contextual school background and community information, the baseline positive aspects and challenges identified by school leadership teams at the first visit, and the analysis of the NP activity. As such, it presents formative findings for three of the 12 initiatives and the 11 case studies, and does not provide recommendations or conclusions. Smarter Schools NP Evaluation Summary - 6 -
Although the NPs have created the space for cultural change, there is evidence of a growing awareness in most of the case study schools that meaningful change is likely to take longer than they originally planned. Most stakeholders across the case study schools, and those participating in the evaluation from other NP schools, indicated their preference to continue with their school-based strategies. They indicated that one of the most important consequences of the NPs is their role in reinvigorating teaching and learning, and providing the resources for schools to make a difference to and impact on educational achievement. Teachers in NP schools have become more enthusiastic about their learning and teaching. The evidence suggests that schools that were both Low SES and L&NNP were the most active and that the increased flexibility and autonomy attached to the Low SES NP school funding arrangement encouraged a greater level of self-organising activity. The evaluation did find that there was varying degrees of effectiveness associated with some of the activity in these schools but overall most Low SES NP case study schools made promising improvements in the schooling experience for disadvantaged students. Principals expressed the request to continue the NP resourcing beyond the period of the NP as they attributed the funds with making positive improvement in their schools. Impact on student performance as measured by NAPLAN and Pat-M, PAT-R, Student Attendance and Retention data NAPLAN Results An analysis of student data shows that there has been little observable difference in student results for SSNP schools in regard to average performance on NAPLAN. A more detailed analysis has shown there has been greater movement in the results of NP schools in National Minimum Standard (NMS) 4 measures in both Reading and Numeracy 5. Some key aspects of this analysis are: Overall, improved NMS results are most evident for NP schools in Year 3 Numeracy, Year 7 Reading and Year 9 Reading. Declines in NMS performance evident for both Low SES and the LN NP schools in Year 5 across both Reading and Numeracy, and Year 9 Numeracy for Low SES NP schools. PAT-M PAT-R results Local measures such as PAT-R and PAT-M are considered to provide more sensitive or context relevant indicators of student improvement in literacy or numeracy. Analysis of PAT data of schools participating in the SSNP suggests that there have been signs of improvement. Overall, notable findings from PAT results in 2010 and 2011 reveal: 4 The NMS is a measure of the proportion of students achieving at or above the national minimum standard. 5 These findings should be regarded as indicative and preliminary and should be interpreted with caution. Smarter Schools NP Evaluation Summary - 7 -
An improvement in results on L&NNP student cohorts from Test 1 to Test 2 evident in both scale and stanine 6 scores with improvements particularly evident in the Year 3 cohort across both 2010 and 2011 in both Reading and Maths. Student attendance and retention Other available data also showed that: student attendance rates of the NP schools have remained relatively stable since 2008 with the exception of a slight increase in attendance rates of Year 9 students in Low SES NP schools. apparent progression rates (the measure of how many students move from year level to the next) in NP schools have increased slightly from 2008 to 2011 while student enrolments have decreased slightly from 2008 to 2011. It is too soon to make any assessment of the success or otherwise of the National Partnership initiative using student results as a measure alone. Good research and evaluation practice would suggest that at least five years of trend data is required before performance data of this nature can be reliably used to inform decisionmaking. 6 Scale score raw test scores are converted in scale score such that higher scales represent more difficult items. Stanines divide the total student distribution of abilities into nine categories with 1 represent in the lowest, 5 the midpoint and 9 the highest level of achievement. Smarter Schools NP Evaluation Summary - 8 -
Précis of Interim findings for Selected Initiatives The following section provides more information on the evaluation findings related to each of the three identified NP initiatives which were examined in the case studies. Four Year School Strategic Plan The four year School Strategic Plan initiative was designed to increase the capacity of schools to identify, through consultation with their local communities, explicit strategies and prioritisation of reform activities that could tackle disadvantage. School Strategic Plans were to focus on improved engagement and learning outcomes for various student cohort groups such as students with a disability, students with learning difficulties, Indigenous students, ESL students, refugee students and homeless students. The evaluation noted that evidence from the text analysis and the case studies showed, generally, that the overall concept of the four year school strategic plans has: succeeded in turning the focus of Low SES school communities to the task of proactive forward planning over the medium term (four years), and provided the initial boost to innovation and enthusiasm. been well received by stakeholders as the plans added the incentive of flexibility and extra resources to target and address the many challenges in Low SES NP schools. The evaluation also found that: the process had not been universally successful, as impact was mediated by many circumstances, including the nature of the communities involved and the strength, direction and particular qualities of the educational leadership. there was a strong commitment to community engagement in the strategies chosen for inclusion in the plans but whether or not that translated into meaningful engagement with communities would need to be investigated further. A longer term perspective may be useful for this purpose as it should give a clearer indication of how the process worked over the four year timeframe for all phases of the Low SES NP from 2009 2015. evidence gathered from other Low SES NP evaluation initiatives such as Turnaround Teams (TT) found that the strategic plans of the phase one and two schools which were the first schools to join were completed in a rush and this had an impact on other elements such as the TTs early effectiveness and capacity to assist schools. This situation was addressed with the later phases. In summary, the report noted although the implementation model may have been more effective with longer preparation time and notice particularly for schools in the first phase, school principals indicated they used the plans as a convenient organising strategy that was worthwhile despite the added workload they required. Smarter Schools NP Evaluation Summary - 9 -
Turnaround Teams Turnaround Teams provided targeted specialist advice to school leaders, teachers, other staff, students and parents to ensure whole school, innovative and flexible approaches to address issues such as literacy and numeracy, student wellbeing and data analysis. Each team included experienced teachers with expertise in areas such as curriculum planning and delivery, literacy and numeracy teaching, student wellbeing and data analysis. Turnaround Teams focused on developing school cultures that build collaborative team structures, the use of data to guide school improvement plans, and development of teacher capacity at school sites. The evaluation of the Turnaround Team (TT) initiative shows that: school-based participants have reported that they benefited from the availability and support of the TT particularly in planning, implementing and reflecting on data and the possibilities of evidence-based pedagogy in their particular school settings. the TT guidelines have supported leaders and teachers to undertake more advanced NP activities, such as the collection and meaningful analysis of data, than might have been the case without the TT involvement. some principals did not avail themselves early on of the expertise and assistance on offer from TTs as the strategy initially relied on TTs being invited into school. The TTs noted that these schools may have been in need of most assistance. school principals had a mixed response to TT. Those principals who had consistent and meaningful contact with TT members were satisfied with the service they offered. in schools where the leadership had not participated in the initiative, the success of the TT has rested on the personal and professional strengths of the educators involved. Literacy and Numeracy Coaches A key initiative of the L&NNP was the deployment of Literacy and Numeracy Coaches which resulted in the engagement and training of up to 91 literacy and numeracy coaches to work across 175 state schools. The coaches were employed to work directly with teachers and school teams to build their capacity to improve student performance in literacy and numeracy. The coaches assist teachers to build teaching skills and leadership through intensive classroom support and professional development. The introduction of coaches in Literacy and Numeracy Partnership schools focuses on whole-school improvement as well as on individual teachers. The evaluation evidence supports the literature that educational coaching provides a number of benefits for improving the teaching of literacy and numeracy, not only for improved test results (although longer-term outcomes are likely to be of more significance than any short-term gains or losses). There is evidence that: teachers with coaches are being encouraged to think more carefully through collaborative planning and reflective analysis about the craft of teaching to Smarter Schools NP Evaluation Summary - 10 -
improve the effectiveness and efficiency of teaching rather than rethinking the principles. coaches in schools have provided the means to build on the community of learning around literacy and numeracy, supported by a growing culture of collaborative sharing and teaching practice. Professional Development (PD) attached to the initiative worked to build the skills and the networks; it moved away from a model where PD occurs in isolation from the classroom to one where teachers are followed up and supported one-on-one in the classroom environment. It also filled an historical PD gap for teachers in primary schools now it is not just talking PD but real support. The evaluation report indicates that literacy and numeracy coaching has substantial appeal to teachers because it meets their professional development needs and reflects personal values for many teachers who wish to take their profession seriously. Whilst coaches used their role training to target key areas of pedagogy, it was essential firstly to gain and maintain the trust of teachers. Key outcomes indicated in the evaluation include: shifting the culture towards self-evaluation and inquiry in which teachers learn collaboratively improving the general experience of teachers, making it school based and classroom focused, but with important links to pedagogical knowledge, thus achieving research-informed practice improving teaching by providing feedback to teachers and allowing them to reflect intensively on the classroom evidence Finding related to student performance for schools with literacy coaches As previously stated, local measures such as PAT-R and PAT-M are considered to provide more sensitive or context relevant indicators of student improvement in literacy or numeracy. Analysis of PAT-R and PAT-M data of schools participating in the SSNP suggests that there have been signs of improvement even though NAPLAN data is yet to show improvement. In particular the analysis of PAT-R data for schools with literacy coaches has identified some noticeable improvements. Results showed that: students in reading focussed coaching schools improved by an average of one stanine across all year levels in both 2010 and 2011. 7 a reduction in the number of students in the lower stanines (stanines 1, 2, and 3) from Test 1 to Test 2 by between 16% (Year 7 2011) to 30% (for Year 3, 2010) an increase in the number of students in the upper stanines (stanines 7, 8, and 9) from Test 1 to Test 2 by between 10% (Year 3 & 5, 2010) to 13% (Year 3, 2011) 7 ACER recommends that only differences of two or more stanines should be regarded as indicating real difference in performance. Two stanines represent a change of one standard deviation (excluding Stanine 1 and 9) in performance. Based on this recommendation, improvements in performance based on average stanines may not necessarily indicate real change in performance. Smarter Schools NP Evaluation Summary - 11 -
In addition, a small cohort of schools working with literacy coaches showed a notable improvement in student outcomes in the PAT-R test. In both 2010 and 2011, six out of the reading coaching schools had more than 50% of students that moved up by two or more stanines from Test 1 to Test 2. Further analysis of student performance against 2007 norm data, suggest that students in 2010 for Years 3 and 5 had improved at a greater rate than the norm data by 2012. Additionally, 42% of students (Year 3 in 2012 to Year 5 2012) grew in excess of 0.5 standard deviations relative to the norm sample and 0.3% grew in of three standard deviations relative to the norm sample. These results point to some early of improvement on PAT-R for the students in the L&NNP cohort. Conclusion The SSNP has resulted in an unprecedented focus within participating schools on improving teaching quality, the literacy and numeracy development of young people, and the provision of specific support for students from disadvantaged areas. The NPs and the interim evaluation have provided a timely opportunity to gather and analyse data related to the implementation of the initiatives to support these outcomes. However, while the SSNP has created that space for cultural change, there is growing awareness in most the case study schools that meaningful change is likely to take longer than originally planned. This is particularly evident in the analysis of NAPLAN and PAT-M, PAT-R data relating to student performance which show early and tentative signs of improvement. However, qualitative evidence suggests that when other measures are taken into account, such as perceived growth in teacher/knowledge and professionalism, local reports of improvement in student attendance and enrolments in NP schools, there are early signs that some initiatives are starting to have a positive impact. There is also indications that schools are becoming more sophisticated in their thinking and the strategies they are using; and have indicated a preference for continuing with school based strategies. Most stakeholders across the participating schools have indicated that one of the key consequences of the NPs is their role in invigorating teaching and learning and providing resources for schools to make a difference and impact on education achievement. The Literacy and Numeracy Coaches are an example of an initiative that has strong support across participating schools. As previously stated, based on the data to hand, it is certainly too soon to make any assessment of the success or otherwise of the National Partnership initiative using student results as a measure alone. Good research and evaluation practice would suggest that at least five years of trend data is required before performance data of this nature can be reliably used to inform decision-making. Education Queensland will continue to work with the Commonwealth and participating schools over the remaining time of the SSNP in order to embed the practices already established that are showing signs of achieving the goals of the NP. Further work will also be done to determine if the self-organising model is working or whether a more supported structure around the reform effort is required. Smarter Schools NP Evaluation Summary - 12 -
ATTACHMENT 1: Student outcomes for SSNP Schools The following information 8 provides overview of trend information aggregated for Low SES National Partnership schools and the Literacy and Numeracy National Partnership schools for the period 2008 to 2011 using a range of student outcomes including NAPLAN results, Progressive Achievement Test (PAT) results, attendance, apparent progression rates and student enrolment numbers. There are a range of limitations associated with the use of aggregated broad measures for evaluation purposes. Whilst it will be difficult to establish whether any movement is a direct result of National Partnership activities it is important to document any trends that are potentially related. Similarly, the absence of any noticeable improvement in broad based measures establishes a basis for more refined analyses Due to the aggregated nature of the results presented in this report, a number of considerations must be kept in mind when drawing inferences from the results: - The interventions that are part of the L&NNP programs may be associated with a series of expected short-term outcomes. The outcomes may or may not be observable in NAPLAN or PAT results. Also, the expected short-term outcomes may not be the same in all schools. The anticipated short-term outcomes of the L&NNP programs have not been explicitly examined in this report. - NAPLAN and PAT results of L&NNP schools will have been impacted by a number of systemic, school-based and teacher-initiated changes and interventions. The impact of L&NNP may be difficult to isolate from the impact of other factors, particularly in the summary nature of the results presented in this report. This report describes the NAPLAN and PAT results of L&NNP schools. The results are not based on assessments of whether differences are statistically significant. Methodology - 131 state schools, 30 Catholic schools and nine Independent schools are involved in the Low SES NP. 175 state schools and 64 non-state schools are involved in the L&NNP. - The results presented below compared the average NAPLAN performance of the following groups of schools: o Low SES NP state schools o L&NNP state schools o Queensland schools o Schools in Australia as a whole - Results are presented for: o 2008 to 2011 o Year 3, 5, 7, and 9 o Proportion of students achieving National Minimum Standards (NMS) 8 See Attachment 2 for overview of internal departmental documents that data has been drawn from. Smarter Schools NP Evaluation Summary - 13 -
o Mean scale scores (MSS) o Reading and Numeracy strands RESULTS for NAPLAN Reading - Australia ranked the highest followed by QLD, L&NNP schools, with Low SES NP Schools performing below all groups. - The differences in performance across groups were greater in Years 3 and 5 compared to Years 7 and 9. - Overall, there were very little fluctuations in results from 2009 to 2011 for all groups. The following trends were apparent for each year level In Year 3 - there was a slight increase in performance from 2008 to 2011 with the steepest increase across all groups being from 2008 to 2009. In Year 5 - across all groups, there was a slight decline in performance from 2009 to 2011. In Year 7 - across all groups, performance generally remained stable from 2009 to 2011. In Year 9 - across all groups, performance declined slightly from 2009 to 2010 then increased slightly from 2010 to 2011. - Performance of L&NNP schools was generally at par with Low SES NP schools in 2009. However, in 2011, Low SES NP schools performed noticeably lower than L&NNP schools. Numeracy - Differences between groups were smaller in Numeracy performance compared to Reading performance. - As evident in Reading performance, differences between groups were larger in Years 3 and 5 compared to Years 7 and 9. - The following trends were apparent for each year level In Year 3 Across QLD, L&NNP schools, and Low SES NP schools, steady increases in performance were evident from 2008 to 2011. While performance of the nation has remained relatively stagnant since 2008. In Year 5 -Across all groups, performance has remained relatively stagnant since 2009. In Year 7 -Across all groups, performance generally remained stable from 2009 to 2011. In Year 9 -The performance of QLD, the nation and Low SES NP schools on MSS has declined slightly since 2009. - Performance of L&NNP schools has remained relatively stagnant since 2009. Performance of L&NNP schools was generally at par with Low SES NP schools in 2009. However, in 2011, Low SES NP schools performed noticeably lower than L&NNP schools. Smarter Schools NP Evaluation Summary - 14 -
Proportion of students achieving National Minimum Standards (NMS) Reading - Overall, across all year levels, the relative ranking of the groups based on the proportion of students achieving NMS in NAPLAN Reading from 2008 to 2011 did not change for all groups. - Overall, Australia ranked the highest followed by QLD, L&NNP schools, with Low SES NP Schools performing below all groups. - The differences in performance across groups were greater in Years 3 and 5 compared to Years 7 and 9. - The following trends were apparent for each year level - In Year 3 The steepest increase in NMS was from 2008 to 2009 across all groups. L&NNP schools and Low SES NP schools experienced slight increases in NMS from 2009 to 2010 with the steepest increase experienced by Low SES NP schools. The difference in NMS between QLD and Low SES NP schools was greater in 2009 than in 2011. - In Year 5 Performance of the nation and QLD has remained relatively stable since 2008. Both L&NNP and Low SES NP schools achieved steep increases in NMS from 2008 to 2009 but noticeable declines in performance from 2009 to 2010. These declines in performance were not evident in the nation or in QLD. Performance of L&NNP and Low SES NP schools on NMS has generally remained stable since 2010. Differences between both NP schools and QLD are the greatest in Year 5 compared to other year levels examined. In Year 7 Both QLD and the nation have performed similarly on NMS in Year 7 Reading. Performance of L&NNP and Low SES NP schools have generally increased since 2009 with L&NNP schools performing almost at par with QLD and the nation in 2011. Both L&NNP and Low SES NP schools have increased in performance since 2010 an increase that was not reflected in QLD or in the nation as a whole. In Year 9 Performance of QLD and the nation has remained relatively stable from 2009 to 2011. L&NNP schools have achieved relatively steady increases in performance since 2009 Low SES NP schools have increased in NMS performance since 2009 with the steepest increase being between 2010 and 2011. Overall, the differences in performance between both NP schools and QLD were smaller in 2011 than they were in 2009. Numeracy - The following trends were apparent for each year level - In Year 3 Smarter Schools NP Evaluation Summary - 15 -
- In Year 5 - In Year 7 - In Year 9 Both NP groups have experienced steeper increases in NMS than QLD or the nation since 2009. The differences in NMS between QLD and NP groups were much greater in 2009 than they were in 2011. Both NP groups experienced a decline in NMS from 2009 to 2010 a decline that was not evident in either the nation or in QLD. The performance of both NP groups has remained stable since 2010. The differences in NMS between QLD and NP schools were smaller in 2009 than they were in 2011. Performance on NMS across all groups has remained relatively stable since 2008. All groups have experienced a decline in performance since 2009 with Low SES NP schools experiencing the greatest decline from 2009 to 2010. RESULTS for PAT-M and PAT-R data - Methodology Since 2010 L&NNP schools have completed the Progressive Achievement Test (PAT) in Comprehension (R) and Mathematics (M) twice yearly (in February and October). The test is designed by the Australian Council for Education Research (ACER) to track student achievement levels in Reading and Maths and to monitor student progress. 137 of the schools completed the PAT-R test, while the remaining 34 schools completed the PAT-M test. In 2010, the number of students that took the both test 1 (in February) and test 2 (in October) at the same centre was 18,631 for PAT-R and 5,360 for PAT-M. In 2011 the count of students was 20,555 for PAT-R and 5,961 for PAT-M. Only students that undertook both test 1 and test 2 at the same school are included in analysis. As 79% of the L&NNP state schools focussed on literacy teaching strategies and 21% focussed on numeracy, the focus on the data analysis of student performance is on the Literacy (reading) outcomes. Results Analysis of the PAT-R data shows the average scale scores were higher in Test 2 (approximately eight months after taking Test 1) for students in Year 3, 5 and 7.Independent group s t-tests revealed that students across all year levels achieved statistically significant improvements in performance from Test 1 to Test 2. Results also showed that: students in reading focussed coaching schools improved by an average of one stanine across all year levels in both 2010 and 2011. 9 9ACER recommends that only differences of two or more stanines should be regarded as indicating real difference in performance. Two stanines represent a change of one standard deviation (excluding Stanine 1 and 9) in performance. Based on this recommendation, improvements in performance based on average stanines may not necessarily indicate real change in performance. Smarter Schools NP Evaluation Summary - 16 -
a reduction in the number of students in the lower stanines (stanines 1, 2, and 3) from Test 1 to Test 2 by between 16% (Year 7 2011) to 30% (for Year 3, 2010) an increase in the number of students in the upper stanines (stanines 7, 8, and 9) from Test 1 to Test 2 by between 10% (Year 3 & 5, 2010) to 13% (Year 3, 2011) Additionally, a small group of schools working with literacy coaches showed a notable improvement in student outcomes in the PAT R test. In both 2010 and 2011, 6 out of the reading coaching schools had more than 50% of students that moved up by 2 or more stanines from Test 1 to Test 2. Further investigation of these schools may be warranted to determine the factors that may have influenced these outcomes. A range of other variables such as change of school principal or literacy coach were considered but were not found to be statistically significant in relation to student outcomes. There is further evidence of improvement on PAT-R scores for the L&NNP cohort through analysis of gain in PAT-R scores relative to the 2007National Norm Sample10. This analysis suggests that (Using Time 1 PAT-R data only) that students in 2010 for Years 3 and 5 had improved at a greater rate than the norm data by 2012. See table below. In addition, 42.4% of students (Year 3 in 2010 to Year 5 in 2012) grew in excess of 0.5 standard deviations relative to the norm sample and 0.3% of students grew in excess of 3 standard deviations relative to the norm sample). These results paint a very encouraging picture of growth for our participating LNNP cohort. Proportion of students with z-score change Year Level Interval Proportion of students with change of >0.5 >1.0 >2.0 > 3.0 03-05 42.4% 18.4% 2.3% 0.3% 05-07 39.4% 13.8% 1.1% 0.1% 10 However, it is worth noting that students in the normed sample may have been higher scoring than the LNNP cohort of students and therefore it would make sense that students who started lower than the norm should, ultimately, grow at a faster rate over time. This bias has been demonstrated with the NAPLAN data. However, in this case we have compared using normed data and the largest possible cohort to attempt to limit the effect of the bias. Please note that Effect Sizes can be negative as well. Smarter Schools NP Evaluation Summary - 17 -
Attachment 2: Source documents 1. Smarter School National Partnership: Queensland Interim Evaluation Report 2010-2011 11. The interim evaluation report provides a comprehensive overview of four initiatives under the Low SES NP and the LN NP. It identifies the core features of the SSNP in Queensland schools, enablers and barriers to effective implementation, major outcomes and how effective the reforms have been in lifting school performance and the sustainability of the initiatives. The initiatives are: Turnaround Teams Foundations for Success (part of the Closing the Gap Strategy for Indigenous students) Four Year School Strategic Plans Literacy and Numeracy Coaches. (This is the final report on this initiative). The report provides case studies for 11 representative schools, quantitative and qualitative data drawn from surveys, interviews, observations, text analysis (computer) and document analysis. 2. Smarter Schools Evaluation Report- Low SES National Partnership and Literacy and Numeracy National Partnership Schools 2008-2011. This internal departmental report provides trend information aggregated for Low SES National Partnership schools and the Literacy and Numeracy National Partnership schools for the period 2008 to 2011 using a range of student outcomes including NAPLAN results, Progressive Achievement Test (PAT results, attendance rates, apparent progression rates and student enrolment numbers. 3. Smarter Schools Evaluation Report- Literacy and Numeracy Coaching Program 2011. This internal departmental report summarises the available information and statistical analysis relating to the Literacy and Numeracy Coaches program. It includes preliminary results summarising trend information aggregated for the period of 2008 to 2011. Smarter Schools NP Evaluation Summary - 18 -