Pilot. Matched design with process evaluation. Primary Outcome Maths and English

Transcription

1 Tutor Trust Primary NFER Liz Phillips Evaluation Summary Age range Primary and Secondary (Year 6 and 7) Number of pupils c. 252 Number of schools 13 Primary school and 33 Secondary schools Design Pilot. Matched design with process evaluation. Primary Outcome Maths and English Intervention The Tutor Trust aims to help close the attainment gap in the British education system by making a good quality professionally run tuition service available to a far broader group of students. The Trust has now developed an extension to its service, which focuses on transition. In the academic year the extended programme offered tuition to around 252 Year 6 pupils across 15 primary schools in Manchester, a subset of whom are within the scope of this evaluation. Around 200 of these pupils will receive further tutoring from the Tutor Trust in Year 7 in the academic year Significance Transfer to secondary schools is a time of particular vulnerability for pupils from disadvantaged backgrounds. Pupils experience a significant dip in their learning as they transfer to secondary schools, which contributes to the widening gap in performance between those from more and less advantaged backgrounds (Evangelou et al., 2008; Evans et al., 2010). Summer learning loss and transfer to secondary school have been identified as major contributory factors. The evaluation focuses on: the impact of the project on pupils attainment and successful transition to secondary school the effectiveness of the model and its implementation. Research Plan: quantitative outcome evaluation Research questions The research questions for the quantitative outcome evaluation are: Design How do the results of Year 6 and Year 7 pupils who have received tutoring compare with similar pupils who have not received tutoring? How does the impact of the tutoring differ for pupils who have received tutoring in Year 6 and Year 7 compared to pupils who have only received tutoring in Year 6?

2 Power In the quantitative strand of the evaluation we are evaluating the impact of the Tutor Trust tuition on children s attainment in the subject in which they have received tutoring (Maths and/or English). Our analysis aims to identify the progress that Tutor Trust pupils have made compared to that of a comparison group of similar pupils. Outcome measures The primary outcome is the attainment of the tutored pupils in English and Maths. We are measuring the attainment of the tutored pupils and comparing it with similar pupils who have not received tuition between: Year 5 optional tests and Key Stage (KS2) SATs KS2 SATs and PiE and PiM tests administered in Autumn 2013, when pupils have entered Year 7 Year 5 optional tests and PiE and PiM tests. Sample size calculations The chart below, which is based on some conservative assumptions of the expected sample sizes and other parameters, shows the probability of the analysis detecting different effect sizes. It is important to note that these calculations are based on estimated pupil numbers because it is not possible to accurately predict the numbers of pupils who will be receiving Tutor Trust tuition. It is also not possible to accurately predict the number of control pupils and primary schools. However, by considering conservative assumptions and parameter values in the calculations below it is possible to approximate a level of power that can only be improved by the actual data. 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Effect size The solid line represents the power given these conservative assumptions and shows power of 12% of detecting an effect size of 0.1; 35% of detecting an effect size of 0.2; 88% of 2

3 detecting an effect size of The assumption that the control and intervention groups are of the same size is a particularly conservative one. The intention is to collect information from all non-intervention Manchester primary schools: all else equal, an increase in the size of the control sample (in particular, primary schools rather than pupils) will tend to increase power. Analysis We will request NPD data for all Year 7 pupils who are in the core sample of secondary schools involved in the evaluation as this will allow NFER statisticians to analyse the Progress in English and Progress in Maths tests results. Requests will also be made for the 2013 Key Stage 2 results for those primary schools attended by Tutor Trust pupils. This will allow us to identify whether there is any significant difference between the attainment of Tutor Trust pupils and other, similar pupils, who were not part of the Tutor Trust programme. Both requests for NPD data will be linked to those pupils Key Stage 1 test results. We plan to carry out three analyses using the data collected during the quantitative strand. The primary analysis will compare the progress of pupils who have received Tutor Trust tuition with that of a comparison group of pupils who have not between Year 5 tests and KS2 SATs; KS2 SATs and PiM and PiE tests; and Year 5 tests and PiM and PiE tests. We will undertake the analysis using multi level modelling, which takes account of the inherent structure of the data, i.e. pupils within schools. This technique is well established as a reliable way of identifying intervention effects whilst controlling for other factors that may affect the variation in outcome. If these factors are not taken into consideration, the models may result in an over estimation of intervention effects. Analysis will be carried out separately for numeracy and literacy tests and will include background characteristics available from the NPD. Analysis will be able to identify whether there are any significant intervention effects and whether these effects are the same for boys and girls, or if they are dependent on whether a pupil is, or is not, eligible for free school meals. The comparison group will be constructed from the full sample pupils in Manchester that attended non-tutor Trust primary schools using propensity score matching (PSM). PSM will select those pupils most similar to the intervention sample according to their background characteristics, which will include a measure of prior attainment, gender, eligibility for FSM, and other relevant characteristics. 1 The calculation assumes: number of intervention primary schools = number of control primary schools = 15; number of intervention pupils = number of control pupils = 100; intra-school correlation = 0.15; confidence level of test = 95%; correlation between before and after test scores =

4 Research Plan: qualitative process evaluation Design We are visiting 6 primary schools and 5 secondary schools involved in the project to collect qualitative data on the quality and perceived impact of the tuition. During each school visit we plan to interview: the head or another member of the leadership team (likely to be the Head of Year 7 in secondary schools) a class teacher of the children receiving the tutoring a tutor two to five children who have received tutoring. Personnel NFER is directing and leading the evaluation. Helen Aston is directing the evaluation. Liz Phillips and Emily Buchanan (Lamont) are leading the process evaluation and Jack Worth is leading the outcome analysis. Andy Partington is the main point of contact at the Tutor Trust. Camilla Nevill is managing the evaluation at EEF, supported by Evaluation Officer Elena Rosa Brown. Each person has the following roles and responsibilities: Helen Aston: Oversight of project activities and QA of outputs. Liz Phillips and Emily Buchanan (Lamont): Recruiting schools to the evaluation. Process evaluation fieldwork. Qualitative analysis and reporting. Jack Worth: Quantitative analysis and reporting. Andy Partington: Gaining school s consent to participate in the evaluation and to provide pupil data. Gathering and collating pupil data from participating schools. Camilla Nevill: Oversight of the evaluation. Elena Rosa Brown: Leading operational aspects in the day-to-day running of the evaluation. Timeline The evaluation began in Spring 2013 and will conclude in early Summer Spring 2013 Gain primary schools consent to participate and begin primary school data collection Produce interview schedules Select qualitative fieldwork sample: primary schools Summer 2013 Conduct qualitative fieldwork: primary schools Gain secondary schools consent to participate Continue primary school data collection Assess viability of the quantitative strand Analyse qualitative data from primary school visits and provide short report 4

5 Present qualitative findings to EEF and Tutor Trust Autumn-Winter 2013 Continue primary school data collection Assess viability of the use of Year 5 data as a baseline measure depending on amount secured Conduct secondary school data collection Spring 2014 Conduct qualitative fieldwork: secondary schools Analyse qualitative data from secondary school visits Implement secondary school testing programme (dispatch, mark and analyse tests) Clean secondary school pupil data and issue NPD request Undertake impact analysis on test scores and KS2 data Conduct NPD data matching Produce draft final report Summer 2014 Submit final report (due End of May, 2014) Key findings meeting Provide feedback to secondary schools on pupils test results. Risks Risk Researchers lost to project due to sickness or absence The Tutor Trust do not provide the pupil data on time, due to delays in schools providing data Schools are unwilling to provide their data or engage with the evaluation Countermeasures and contingencies NFER has a large research department with numerous researchers experienced in evaluation who could be redeployed. NFER and the Tutor Trust will liaise closely to ensure that the pupil data is provided on time. Tutor Trust will encourage participation of Tutor Trust schools. NFER will use a reminding strategy to encourage non- Tutor Trust schools to participate. NFER is incentivising schools to administer the PiE and PiM tests by providing feedback. We will use Key Stage 1 data as a baseline measure if insufficient Year 5 data is provided. Insufficient sample size EEF will monitor the Tutor Trust s recruitment of schools. NFER will flag any sample size issues as they become apparent. Insufficient power The evaluation was designed with an effect size of 0.4 in mind, in line with the EEF-Sutton Trust Toolkit average effect size for small group tuition. Should the effect be lower than this, the probability of us 5

6 detecting the falls, as shown in Sample size calculations section. Data protection statement NFER s data protection policy is available at: 6