Conflict Resolution Uncut. CAYT Impact Study: REP14

Transcription

1 Conflict Resolution Uncut CAYT Impact Study: REP14

2 The Centre for Analysis of Youth Transitions (CAYT) is an independent research centre with funding from the Department for Education. It is a partnership between leading researchers from the Institute of Education, the Institute for Fiscal Studies, and the National Centre for Social Research.

3 Centre for Analysis of Youth AYT Transitions STUDY REFERENCE: REP14 Programme name: Conflict Resolution UnCut Contact details/links for further details: Working With Men, Unit K308, Tower Bridge Business Complex, 100 Clements Road, London, United Kingdom, SE16 4DG. Tel.: Programme description, aims and objectives: UnCut aims to prevent crime, particularly knife crime, by building boys conflict resolution skills and raising their awareness of the consequences of knife crime. The intervention consists of several programmes which include conflict management and community safety programmes for primary and secondary school students; a mock trial programme; assemblies; one-to-one sessions with selected students; sessions on transition to secondary school, co-delivered by peer mentors; managing conflict training for practitioners and working with parents. The programmes vary in length and in their targeted audience. For example, the managing conflict programme consists of 6-8 sessions with groups of eight pupils and addresses such topics as: examining communication in conflict-type scenarios, discussing weapons and violence, spotting and avoiding danger, examining the motives behind carrying knives and discussing aspirations and motivations. The mock trial programme has a similar structure but focuses on the consequences of knife crime, including legal aspects and the police perspective. Assemblies address whole year groups and explore medical and criminal justice issues. One-to-one sessions are delivered to students identified as requiring further support. Activities used in various programmes include conflict-resolution role plays, watching and discussing a mock trial DVD, contrasting earnings from immoral activities and from employment, a testimonial by a knife crime victim and expression of a topic through art and presentations. Target population: The intervention targets boys in primary and secondary schools (aged 10-16). Whole school years are included in assemblies and selected individuals take part in intensive programmes or one-to-one work. Additional support sessions are provided to Year 6 students in order to prepare them for the transition to secondary school. Expected outcomes: Improved conflict resolution skills, active listening skills, ability to solve disagreements, confidence, ability to control one s temper and to keep safe. Deterring and preventing knife crime by increasing boys awareness of its consequences and showing them practical ways of avoiding knife crime.

4 References: Bourne, V. (2011). Conflict Resolution Evaluation Report. Working With Men. Clear Plan (2010). Fear and Fashion: Programme Evaluation. Final Report. Working With Men (2012). UnCut Project : City Bridge Report. Study details: Conflict Resolution Evaluation Study The study uses a treatment and a control group to identify the impact of a conflict resolution intervention on boys conflict resolution skills, both self-reported and assessed independently based on observations by teachers. Wilberforce Study Presentations and Q&A sessions, co-delivered by peer mentors, were delivered to Year 6 pupils in Wilberforce Primary School, with the aim of preparing them for the transition to secondary school. A further set of workshops on conflict resolution and community safety was then delivered to 12 boys. The study presents data on students self-ratings (on a scale from 0 to 10) of conflict and safety-related questions asked before and after the conflict resolution workshops. Qualitative Data Between 2007 and 2010 over 2000 young people in North Westminster participated in UnCut activities. The majority were engaged through assemblies and over 350 young participants attended workshops or one-to-one sessions. Training was delivered to 68 practitioners (teaching staff, learning mentors and youth workers) and 129 parents were involved in the programme. A consultation process was conducted in 2011 with young people and practitioners, which provided qualitative data on outcomes. Study samples: Conflict Resolution Evaluation Study The sample consists of 54 boys. The intervention group includes 42 boys between 9 and 13 years old (mean age: 11 years) from 6 different schools. The control group consists of 12 boys between 10 and 13 years of age (mean age: 11.8 years) from 2 schools. The boys participating in the study had on average 0.7 exclusions, with more than a half of the sample (54%) having been excluded at least once. The sample consisted of representatives of nine different ethnic groups. Wilberforce Study The sample consists of 12 boys in Year 6 at Wilberforce Primary School who participated in a conflict resolution and community safety programme which consisted of six 1 hour weekly group sessions. Qualitative Data 35 young people (aged 14-18) and 18 practitioners who were involved in the UnCut project between 2007 and 2010 took part in consultations which provided qualitative information on the participants views of the intervention and its impact.

5 Methodology: Conflict Resolution Evaluation Study The study uses data collected at three points in time: before the intervention, immediately after the intervention and 6 weeks later, based on an intervention and a control group. Self-reported data and teachers observations are treated separately which increases the robustness of the results. In order to check whether the effectiveness of the intervention did not differ according to boys personal characteristics, additional estimates were performed, using data on the intervention group only, in which the following covariate factors were entered individually into the analysis: age, year group, school attended, ethnicity, exclusions. Wilberforce Study The analysis uses students self-reported ratings on 5 different skills and qualities, completed before and after the intervention. In each area students were given a scenario and asked how they would respond and why. Self-reported scores are compiled in overall averages. No formal statistical tests are used to detect statistical significance of the changes in scores. Qualitative Data The data were collected through one-to-one and small group (2-3) interviews conducted a year or more after the intervention. The interviewees were asked the same sets of questions. Responses to some questions were then compiled for the purposes of a numerical analysis.

6 Results and impact: Conflict Resolution Evaluation Study Before the intervention there was no statistically significant difference between the conflict resolution scores in the intervention and control groups when these skills were measured by self-report. Observer-rated scores of participants conflict resolution skills prior to the intervention were higher (better) in the control group and the difference approached statistical significance at the 5% level. Both the self-reported data and the observer rated scores indicated that there was a clear improvement in students conflict resolution skills over time in the intervention group, relative to the control group. This improvement in the skills of the intervention group was statistically significant. For both selfreported and observer data there was no statistically significant increase in conflict resolution skills in the control group. Additionally, the scores of pupils in the intervention group improved further by 6 weeks after the intervention and again this result was statistically significant. Additional analysis of other factors found no statistically significant impact from any of the following factors: age, year group, school attended, ethnicity, exclusions. Analysis of the observer rated data suggests that age and year group (which are closely related) have a statistically significant negative impact on pupils conflict resolution skills across the three time points, with younger pupils showing better conflict resolution skills. Further analysis suggests that the result may to some extent be driven by higher average scores assigned by teachers to Year 6 students as compared to students in Years 8 and 9, which may reflect an unintentional bias in the teachers ratings when applied to younger pupils. Wilberforce Study The study analysed students self-reported ratings of the following outcomes: listening skills, solving disagreements, level of confidence, controlling one s temper and keeping oneself safe. No formal statistical tests are used. The average scores are higher after the intervention than before for all outcomes measured and the increase is around 2 points (out of 10) for each outcome. The highest relative increase occurred in measures related to conflict management (controlling one s temper and solving disagreements) and keeping oneself safe, with relative improvements of 44-45% and 51% respectively. Student s Attitudes Qualitative Data A vast majority (91%) of respondents said that the programme had a positive impact on their attitude and behaviour. Students statements suggest that the intervention encouraged them to stay away from knife crime and provided practical ways to avoid it. It also increased students awareness of the consequences of knife crime. All (100%) respondents in the interviews expressed that the programme deterred them from carrying weapons. Most respondents (83%) agreed that the intervention should be targeted at those at risk of joining gangs and a majority (69%) expressed that the intervention should begin in primary school (students aged 10-11). Even though almost all students (94%) agreed that knife crime prevention work should be delivered in schools, most students (91%) suggested that the intervention should be delivered by individuals other than teachers, for example someone who used to be a gang member. Of the different parts of the programme the ones which stood out for students the most were testimonials by a knife crime victim, mock trial DVD and discussions, conflict resolution role-plays and comparing the impact on longterm earning potential from immoral earnings and from employment. Practitioners Feedback Qualitative Data The majority of respondents (83%) stated that the programme met their expectations. A selection of statements from the interviews also suggests that the programme was seen as useful and engaged young people. Improvement was noted in the way in which young people communicated with each other. The statements also suggest that the aspects of the materials found useful by practitioners included activities on negative images perceived due to dress codes or body

7 language; DVDs on such issues as conflict and stabbing; the conflict escalation exercise and an exercise which compared lifetime earnings from immoral sources and from employment. 100% of practitioners who took part in the interviews found the training they received useful. Statements by practitioners suggest that the training provided new tools and strategies and was a platform to hear from others involved in working with young people. Overall A formal statistical analysis shows that the conflict resolution intervention is associated with a statistically significantly higher increase in conflict resolution skills when compared to a control group, with gains measured up to six weeks after the intervention. No difference in the impact of the intervention was found according to the school attended, ethnicity or the number of exclusions. The difference according to one s age or year group is unclear and may be attributed to more generous rating awarded to younger students. A before-and-after analysis of data from a small sample of Year 6 students suggests that after the conflict resolution and community safety workshops there was an increase in the following outcomes: listening skills, solving disagreements, level of confidence, controlling one s temper and keeping oneself safe. Although it is unclear whether the differences are statistically significant, the relative increase in scores is substantial (between 29% and 51%). Qualitative data from interviews show that students felt that the programme had a positive impact on their attitudes and behaviour and that it deterred them from knife crime. Interviews also suggest that the programme met practitioners expectations and that the training they received was useful. Impact grade: 2 Costs: No figures quoted. UnCut consists of several programmes and the costs of the intervention depend on which programmes are adopted and on the needs of individual students (for example, transition workshops for Year 6 pupils or one-to-one sessions for students requiring further support). The costs also depend on whether the intervention is delivered only by professionals or whether it is co-delivered by peer mentors as well as on the length of the programme. To provide an indication of the intensity of the intervention (which is closely related to costs), the table gives details of the length of each programme and the usual numbers participating in each programme. Programme Length Number of participants Conflict Management Programme 4 sessions, 2hours each or 8 sessions, 1 hour each 8 Mock Trial Programme 4sessions, 1.5 hours each 8 Assemblies A single session Whole school year groups (150 students) One-to-one sessions Minimum 8 weeks 1 Managing conflict raining for practitioners 1 day 8-12 Peer Mentoring Training and Transition to Peer Mentoring Training: 3 1hr sessionstransition Presentation and 10 Peer Mentors

8 Secondary School Workshops Q&A: 1 session, 2 hours Approximately Quality of evaluation evidence: Conflict Resolution Evaluation Study uses a clearly defined control group in a mixed-design ANCOVA based on a sample of 54 observations. Outcomes are measured at three points in time and two separate measures of conflict resolution skills are used, increasing robustness. The sample is relatively small and the analysis of the impact of individual characteristics on the change in scores could be made more precise by including more observations. Wilberforce Study uses before-and-after analysis based on boys self-reported scores. Whilst a before-and-after analysis does not necessarily show a causal relationship, the relatively short length of the intervention (6 weekly sessions) suggests that little change can be expected independently of the intervention. However, a very small sample is used. A larger sample and, ideally, a matched control group, would increase the quality of evidence. Quality of evidence grade: 4 (Conflict Resolution Evaluation Study), 3 (Wilberforce Study), 1 (Qualitative Data) Appendix: details of impact grades and quality of evidence grades are set out below Impact grade Description 0 (none) No relationship between the youth service and the outcome in question. 1 (low) Provision of the youth service may be positively related to one but not all outcomes or just for subgroups of the target population. 2 (medium) The youth service has moderate impact on all outcomes and sub-groups or high impact on some outcomes and sub-groups. 3 (high) The youth service has high impact on all outcomes and sub-groups.

9 Score Type of study More Description Example of a study How to improve the quality of evidence 0 Basic Studies that describe the intervention and collect data on activity associated with it. A study that describes the intervention and states how much it cost or how many hours of services young people received. Collect some before and after data on the outcome of interest for those receiving the intervention. If it is too late for that, collect outcome after data for the group receiving the services and try to compare these outcomes with comparable youth using other sources of data. 1 Descriptive, anecdotal, expert opinion 2 Study where a statistical relationship (correlation) between the outcome and receiving services is established 3 Study which accounts for when the services were delivered by surveying before and after 4 Study where there is both a before and after evaluation strategy and a clear comparison between groups who do and do not receive the youth services 5 As above but in addition includes statistical modelling to produce better comparison groups and of outcomes to allow for other differences across groups 6 Study where youth services are provided on the basis of individuals being randomly assigned to either the treatment or Studies that ask respondents or experts about whether the intervention works. The correlation is observed at a single point in time, outcomes of those who receive the intervention are compared with those who do not get it. This approach compares outcomes before and after an intervention. These studies use comparison groups, also known as control groups. Study with a before and after evaluation strategy, statistically generated control groups and statistical modelling of outcomes. A study that uses focus groups or expert opinion or indeed surveys those who received the intervention after they received it. A study that conducts a survey only after the services have been delivered and concludes that youths who received the services responded more positively than those who did not. A study that conducts a survey before and after the programme. A study that matches two locations where both individuals and areas are comparable and surveys them before and after the programme e.g. pilot studies. A study that uses a statistical method, such as propensity score matching, to ensure that the group receiving the youth services is similar to the comparison group and a statistical model of outcomes (e.g. difference in difference). A study which conducts a Randomised Controlled Trial Collect some before and after data on the outcome of interest for those receiving the services. If it is too late for that, collect outcome after data for the group receiving the services and try to compare these outcomes with comparable youth using other sources of data. This evidence does not allow for the fact that prior to the intervention youths who received the service may have been different from those who did not. Collect some before and after data on the outcome of interest for those receiving the intervention. If it is too late to do that, see if you can compare outcomes for a clearly defined comparison or control group using other before data sources, such as administrative data. If you have before-after data you can measure the change in a particular outcome after the services were delivered. Try to determine whether you can compare this gain in the outcome for those who received the youth services to the gain for a similar group of youth who did not receive the services. You might use administrative data for this. You have most of the data you need. Contact an expert on statistics or econometrics and they will be able to apply various statistical methodologies to improve the robustness of your results e.g. matching methods to define a better control or comparison group. NOTE: this is the minimum level of evaluation quality applied by the Social Research Unit et al (2011), which also stipulates that any such study fulfil various quality criteria. Short of a random control trial, this methodology is the most robust. To improve confidence in the results try to collect additional data, perhaps from administrative sources, on the comparison group to determine any differences between them that may have pre dated the intervention. The gold standard. It is challenging to run a RCT, with cost, ethical and practical issues arising. Even with a RCT you have to think about how generalisable it is to other situations. If the RCT was only males, it cannot tell you about how well the youth service would do for females, for example.