Literature Review on the Role of External Contributors in School Drug, Alcohol and Tobacco Education

Transcription

1 RESEARCH Literature Review on the Role of External Contributors in School Drug, Alcohol and Tobacco Education David White, Emily Buckley, Judith Hassan Centre for Health Psychology Staffordshire University Research Report RR514

2 Research Report No 514 Literature Review on the Role of External Contributors in School Drug, Alcohol and Tobacco Education David White, Emily Buckley, Judith Hassan Centre for Health Psychology Staffordshire University The views expressed in this report are the authors' and do not necessarily reflect those of the Department for Education and Skills. Staffordshire University 2004 ISBN

3 Table of Contents EXECUTIVE SUMMARY, RECOMMENDATIONS AND CONCLUSIONS 2 TECHNICAL REPORT 7 1 INTRODUCTION 1.1 Aims Objectives Background External Contributors in Schools Methodological Issues METHODOLOGY 2.1 Overview Scoping Exercise and Development of Protocol Literature Review Inclusion and exclusion criteria Search strategy Searching the unpublished (grey) literature Hierarchy of evidence Review strategy and data extraction 15 Table 1: Hierarchy of Evidence 17 3 RESULTS 3.1 Locating Reports External Contributors That Have Been Examined 18 Table 2: Reports of Agency Involvement 19 Table 3: Outcomes Assessed 21 Table 4: Aspects of Education Assessed Training Provided for External Contributors External Contributions to Drug Education Effectiveness of External Contributors Outcome evaluations Stakeholder views and implementation issues 36 4 CONCLUSIONS AND REOMMENDATIONS 4.1 Observations Key Findings Critical Knowledge Gaps Areas of Future Research to Address Knowledge Gaps Recommendations 50 REFERENCES References to Included Studies 52 References to Excluded Studies 59 References to additional material in the report 61 APPENDICES (Available on the DfES Research Website at: Appendix 1 Data Extraction Sheet 63 Appendix 2: Responses to Calls for Papers 68 Appendix 3: Organisations Contacted Directly 71 Appendix 4: Tables of Included Studies 81 Appendix 5: Table of Excluded Studies 146 Appendix 6: Training Provided to External Visitors 152 Appendix 7: Additional Findings 157 1

4 Introduction EXECUTIVE SUMMARY, RECOMMENDATIONS AND CONCLUSIONS Responsibility for the delivery of drug, alcohol and tobacco education lies primarily with teachers, but frequently they are supported by external contributors drawn from a number of professions and external agencies. Approximately 80% of primary schools and a similar number of secondary schools use external contributors as part of their drugs education programme (Ofsted 2002). This review examined the available evidence on the role of external contributors in the delivery of drug, alcohol and tobacco education. There were three main aims of the research; 1. To identify which external contributors (i.e. police officers, nurses, Life Education Centres) are providing drug education within schools, 2. To review evidence of effectiveness of the drug education provided by external contributors (in terms of attitude, knowledge, intention and behaviour change, and also views of stakeholders such as children, parents and teachers), 3. To identify gaps in the research, and identify areas where further research is needed. The review examined research conducted since 1990 and published in English. One hundred and fourteen reports were identified as relevant to the review, in that they evaluated the contribution of external contributors (anyone other than a teacher or pupil from within the school) in providing school based drug education 1. Key Findings 1. The use of sixteen different external contributors has been evaluated 2 :- Police (30) Ex- and current drug users (3) Theatre (25) Youth Workers (1) Health Educators (11) Cartoon Animators (1) Peers from outside the school (10) Fitness Instructors (1) Life Education Centres (8) Professional basketball players (1) School Nurses (7) Health & Legal Experts (1) Drug Agency Workers (7) Parents (1) Researchers/ Psychologists (6) Singer/ Songwriters (1) 2. There is no evidence to suggest that any particular agency is more effective (in terms of being well received by pupils and teachers and/or leading to knowledge, attitude or behaviour change in the pupils) than any other in providing drug education. 3. When combined with class teachers, external contributors can bring specialist knowledge and novelty, leading to high involvement and enjoyment for children 3. 1 For full details of methodology see Chapter 2 in the full report. 2 Numbers in brackets refer to number of reports; See Chapter 3 in the report for full details. 3 Crosswaite & Tooby (2002), page 38 in full report. 2

5 4. Most of the British reports located were relatively small scale and failed to explore issues in great depth. Most focused on teachers perceptions of effectiveness. Whilst teachers views are important, they do not provide sufficient evidence to judge the effectiveness of particular programmes. 5. Children like receiving drug education from ex drug users and feel that they are better informed and recall more information as a result of this type of education Pupils receiving peer-delivered education believed that they had increased knowledge and they rated the sessions positively, as did their teachers. Young people report feeling able to talk freely in group sessions and approve of the lesson content provided by peers. Further, peer educators can become positive role models and a valuable resource for younger pupils Children enjoy and engage with Life Education Centres programmes, but there is not sufficient evidence to judge their effectiveness in terms of knowledge, attitudinal, intention or behavioural change Theatre can be an effective means for raising awareness of drug use issues, it can be highly effective in engaging the interest of pupils of all ages and it can be an effective trigger for discussion. It is a good activity for prompting interactive, participatory engagement of the sort advocated for effective drug education (See paragraph 14 for further details) The DARE programme (an intensive programme delivered by uniformed police officers) achieves some short-term changes in knowledge, attitudes and behaviour. But these effects decay rapidly, with no long term effects 8. This applies to both the DARE program implemented in the USA and that used in the UK. 10. Police officers can provide a valuable contribution (bringing specialist knowledge) to drug education when used in a supplementary role, but only when their expertise maps onto the aims of drug education within the school Programmes delivered by nurses have been shown to produce shortterm knowledge gains, and to have effects on knowledge and cannabis and alcohol use for 6 months. Longer term evaluations are needed to provide more evidence of effectiveness, current evaluations only 4 East Sussex LEA, page 42; O Connor et al (1999), pages 39 & Gloss (1995), page Conclusions are mainly based upon teachers perceptions. Kaplin (2003); Hawthorne (1993); page Doncaster Health Promotion (2002); Lisnov et al (1998), page Clayton (1991); Rosenbaum (1994); page Crosswaite & Tooby (2002); O Connor et al (2001), pages

6 measure either immediate outcomes or those after a period of just six months External contributors need to be supported by teachers, and their sessions must fit in with the curriculum, with preparatory and follow up work increasing the effectiveness of the sessions There is currently a great variety in the training received by external contributors specifically to prepare them to contribute to school drug education programmes. Such training may last from a couple of hours to two weeks. For many contributors, their training is not sufficient for them to enable children to develop the required skills (decision making, self esteem) or for the visitors to cope with the demands of a classroom full of children Research has found that there are certain factors that are present in more effective drug education programmes 4 (although the presence of these factors does not guarantee effectiveness): Social influence-based approach (teaching resistance skills and tackling perceptions of peer drug use) rather than fear-based approaches. Targeting of all forms of drug use. Interactive methods of delivery. Content/ delivery aimed to increase resistance skills, social competency and change attitudes. Content/ delivery aimed to enhance protective factors and reduce risk factors for substance use. Intensive and long-term delivery that should start early and continue throughout the school life. Age-specific and culturally sensitive, and adapted to address local conditions (i.e. drug use in particular areas). 10 Ingram (2002); Rye et al (2001), page Problems arising from the contributions of external individuals occur when their sessions do not fit in with the curriculum; O Connor et al (1999); Kaplin (2003); Crosswaite & Tooby (2003); pages Ofsted (2002); O Connor et al (2001); pages NIDA (1997); DH (2001); Cuijpers (2002); Page 26. 4

7 Gaps in the Available Evidence There are several questions which could not be answered by the available evidence, and these need to be addressed in future research:- 1. There is very little information available on the content and style of delivery used by any of the above-mentioned external contributors. 2. There is little evidence on how the contributions of external contributors can be effectively integrated into the school curriculum, and how much preparation or planning takes place before their visits. 3. There is little information available on the views of the external contributors, in terms of their perceptions of the programme, their training needs etc. 4. There is only limited information available on the effectiveness of Life Education Centres (LEC) programmes. The only UK evidence of the effectiveness of Life Education Centres is based upon teachers perceptions (which are not a satisfactory measure of effectiveness). 5. There is a general lack of evidence measuring effectiveness of programmes in terms of outcomes (e.g. attitude, intention or behaviour change). 6. There are no direct comparisons of curriculum delivery by different agencies and it is therefore not possible to conclude whether certain content is best delivered by particular individuals (i.e. legal content by police officers) There are no direct comparisons of the effectiveness of certain curriculum content or teaching styles on particular target groups (i.e. high risk individuals, different gender groups etc) 5. Recommendations 1. More research directly measuring outcomes is needed, particularly for Life Education Centres, nurse delivered and theatre in education programmes. The majority of research currently available on these agencies only measures short term effects, and tends to focus on teachers perceptions of effectiveness rather than objective measures from the children. 2. The police should be used in a supplementary role in drug education as an expert contributor in a manner reflecting their expertise, when that 14 Mellanby, Rees & Trip (2000) reviewed studies comparing peer delivered vs. teacher delivered curricula, but the peers were drawn from the same school as those they were teaching, page Although research has suggested that certain programmes may have differential effects on different populations, e.g. Hawthorne (1996) boys receiving LEC education increased alcohol use, and girls increased tobacco use compared to controls not receiving LEC education, page 31. 5

8 expertise maps onto the aims and content of the drug education planned by the school. 3. Theatre in education should be included as a part of the drug education planned by the school rather than as a stand-alone session. The play/production should be matched to the needs of the particular recipients and schools, and evaluations of alternative methods of delivering theatre performances should be encouraged. 4. Providing high quality training is essential for effective drug education. Training is required to enable external contributors to be flexible in their teaching methods to modify their approach in the light of pupil needs. Both teachers and external contributors require an assessment of their training needs and need to be offered appropriate training. 5. It is essential that there is a good fit between the external contributors input and the school drug education curriculum. Planning and coordinating activities of external contributors and ensuring that visitors are adequately supported by the school are also necessary to maximise their input. The following components are involved in these activities: Identification of pupil needs that cannot be met by existing staff and the identification of external contributors specially placed to meet those needs. Information on suitability of the agency and individuals for educating the relevant age group should be sought. Confirmation should be sought to ensure that the agency shares the values, objectives and approaches of the school. The external contributor should be provided with lesson plans for work already undertaken and broad term/year plans of aims and objectives. Meetings should take place between teachers and external contributors to plan the educational input and agree shared working practices and responsibilities. Preparatory and follow up work should be undertaken with pupils to maximise the input of the external contributors. The external contributor s input and delivery style should be monitored and evaluated to assess its appropriateness. All of these measures are necessary in order to maximise the effectiveness of drug education delivered by external contributors. 6

9 TECHNICAL REPORT 1 INTRODUCTION 1.1 Aims 1) To examine the role and expertise of external contributors and individuals in school drug, alcohol and tobacco education. 2) To identify the characteristics of external contributors/individuals who provide effective drug, alcohol and tobacco education. 3) To determine whether evidence exists on the effectiveness of different external contributors and individuals as educators for different sections of the school community. 4) To provide best practice guidance for the delivery of drug, alcohol and tobacco education in schools by external contributors/individuals. 5) To identify areas for future research. 1.2 Objectives Establish the range of external contributors and individuals who support or deliver schools-based drug, alcohol and tobacco education. Determine the extent to which the involvement of external contributors and individuals in drug, alcohol and tobacco education has been evaluated. Establish the effectiveness of education provided by external contributors including: 1) their impact on pupils knowledge and understanding, attitudes and skills development as well as their subsequent use of drugs, alcohol and tobacco; and 2) how the education was received by pupils, schools and the providers of the education. Explore how different stakeholders (e.g. schools, parents, pupils, teaching training agencies, Ofsted) perceive the use of different external contributors and individuals as drug, alcohol and tobacco educators. Establish the professional and personal experience/ training received by those who provide drug, alcohol and tobacco education in schools. Determine whether there are particular messages/content areas that are best delivered by particular people. What is the differential effectiveness of different drug, alcohol and tobacco education approaches/facilitators on different target audiences (vulnerable children)? 7

10 1.3 Background External Contributors in Schools The role of schools in health education has been boosted by a series of recent policy developments. These include the National Healthy Schools Standard (NHSS) and the primary schools primary care health links (PS/PC HL) which seek to increase pupils knowledge and understanding of a range of health issues as well as facilitate personal skills to promote good health. The National Healthy School Standard was proposed in the Green Paper 'Our Healthier Nation' in 1998 as part of the broader Healthy Schools Programme. In this paper, the government identified the school as a setting to improve the health and emotional well-being of all children and young people, and in particular those who are socially and economically disadvantaged. In the government's view, a healthy school is in a key position to improve children's health and educational achievement, with the support of other agencies. The NHSS serves as a vehicle to support delivery of PSHE and citizenship in schools. Projects developed through the Primary School/Primary Care Health Links (PS/PC HL) operate within the Healthy Schools Standard. These projects aim to establish links between primary and community health care professionals and primary schools to support teachers in delivering health education with the National Healthy Schools Programme including substance misuse issues. A range of health care professionals are involved including health visitors, practice nurses, school nurses and community nurses, as well as GPs. In parallel with the development of the National Healthy Schools Standard, the government s white paper Tackling Drugs to Build a Better Britain outlined a strategy, part of which was to prevent young people from developing drug misuse. Action was advocated to enable young people develop the skills to resist pressure to misuse drugs, which should start early at primary school. It recommended that primary schools should provide drug education which boosts self esteem and resistance skills. It was believed that partnership and cooperative working between a range of agencies would increase the probability of the strategy being successful. Support should be available for schools in developing their drugs education curriculum and policies from a Schools Drugs Advisor. The role of the Schools Drugs Adviser is to address gaps in the quality and coverage of drug education in schools, and to ensure that universal drug education is underpinned by appropriate prevention measures and individual support where it is needed. Within this context, drugs, alcohol and tobacco education increasingly begins at an early age, often beginning with medicines, but with increased curriculum time devoted to smoking, alcohol and tobacco education as pupils progress through their schooling. While the delivery of drug, alcohol and tobacco education lies primarily with teachers, increasingly they are supported by external visitors drawn from a number of professions and external contributors. The most recent report on drug education from Ofsted found that about 80% of primary schools used external visitors in classroom teaching about drugs (Ofsted 2002). The professionals supporting schools include police school involvement officers, community police officers, members of drugs and alcohol services, specialist 8

11 drug services for young people, community groups, school nurses and other health professionals, personal advisors from Connexions, theatre groups, youth services, parents of former drug users and national charities such as Life Education Centres. The recent Ofsted report found that 45% of primary schools and 66% of secondary schools used police officers to deliver some of their drug, alcohol and tobacco education, 43% of primary and 13% of secondary schools made use of school nurses, 27% of primary and 44% of secondary schools used theatre in education, 25% of secondary schools used youth workers from drug agencies and 11% of primary schools were visited by Life Education Centre mobile classrooms (Ofsted, 2002). Other providers of drug, alcohol and tobacco education include older peers (sometimes drawn from the same school, and sometimes from different schools as the children receiving the intervention) and drug users or ex-drug users (e.g. Allott et al, 1999). Although these activities are external the scope of the present review, in addition to their work directly with children in the classroom, external contributors may be involved in the development of resources for use in schools. These resources include interactive CD-ROMS, videos, games, leaflets and other printed material. Furthermore, external contributors may be involved in drug education delivered to pupils external of school, e.g. on field trips. They may also be involved in providing training for teachers on substance use issues. While external visitors are frequently involved with schools in delivering drug, alcohol and tobacco education, there is little evidence of how schools make use of such individuals/agencies, what they are doing, the quality of their contribution and the impact they have on pupils knowledge, skills and behaviour. However, more generally, there is evidence of what appears to work and what appears less effective in drug education. All drug educators, teachers and external visitors alike, should acknowledge this evidence base in developing their practice. In a recent review Lloyd et al (2000) concluded that: drug education should start in primary school and continue into secondary school; approaches based on life skills appear most effective; projects should seek to involve parents and communities; peer approaches are promising; interactive approaches are more effective than didactic ones; there are greater benefits to universal programmes than targeted approaches; long term, intensive programmes are most likely to be effective. The advocated life skills training includes decision making skills, self-esteem enhancement, resistance skills training, general life skills training, norm setting and assistance training in obtaining help from others. The most successful programmes have focused upon these interpersonal skills (e.g. Botvin et al, 2002). Research demonstrates that programmes based upon content designed to scare youth, provide information, plea to adolescent morality and change personality or character are highly ineffective. Instead social-influence-based approaches are more effective than traditional information-giving approaches (Cuijpers, 2002; Donaldson et al, 1996). Traditional information-giving approaches delivered by teachers in a non-interactive manner are less effective than approaches characterised by active methods, such as group discussion and role-play (e.g. Fletcher et al, 1997; Tobler et al, 2000). 9

12 Interviews of teachers have been undertaken to establish how teachers themselves view the role of external agencies in helping schools to deliver drug education (Devine, 1996). Teachers welcomed the opportunity of working with external visitors and delivering drug education together. They welcomed the support, knowledge and resources external contributors could offer. They particularly welcomed the staff development courses offered by some external contributors. However, there was less acceptance of the idea of external visitors working directly with pupils and particularly working with pupils without the classroom teacher present. If external visitors were to work directly with children, teachers believed that it was essential that they had the relevant experience to allow them to do so effectively. Relevant experience and expertise is particularly important when dealing with sensitive and possibly controversial issues. External visitors were often introduced to the school through personal contacts of school staff. How well teachers views on the utilisation of external visitors map on to the guidance for teachers on how to plan the involvement of external contributors and to ensure the effectiveness of the education delivered (e.g. DfEE Circular 4/95; DfEE, 1998) is explored in the review Methodological Issues Reports evaluating the role of external visitors focus on a range of issues. Some seek to assess the impact of the visitors input upon the pupils exposed to that input. Others seek to assess stakeholder views of the appropriateness and quality of the input. Yet others are concerned with implementation issues such as whether the educational input was delivered as planned. Different methods of enquiry will be appropriate depending upon the specific questions addressed. Many evaluations focus upon outcomes for pupils who have received the drug education. Ideally, outcomes should be assessed over a long period of time to determine whether the education has short or longer term impact. Outcomes of interest include knowledge, attitudes and intentions for future behaviour, as well as the acquisition of skills and measures of actual behaviour. A major problem with outcome evaluations is that most are methodologically poor (Foxcroft et al, 2002), with many not employing control or comparison groups, not using appropriate measures for assessing outcomes in terms of objectives or not taking baseline measures, so that change over time cannot be assessed. Gilvarry (2000) argued that some positive effects have been reported of universal preventive interventions, but that they too often lack thoroughness in implementation, data collection and follow-up, and have high rates of unexplained attrition. Authors are also inclined to be more enthusiastic about their results than the obtained results allow, such that results and conclusions must be examined carefully. This review explicitly examined the methodological soundness of studies to produce hierarchies of evidence to inform its recommendations. Since drug, alcohol and tobacco education should be focused upon a range of skills, attitudes and knowledge, all of these outcomes should be assessed. However, Ofsted (2002) reported that most schools measure the effectiveness of their drug, alcohol and tobacco education programmes in terms of acquired knowledge, and that the planning and teaching of the 10

13 programmes are not always sufficient to help pupils develop values, attitudes and personal skills that they need to make informed choices. Many interventions have been implemented, but have not been evaluated in terms of outcomes. Such process evaluations are useful since they can highlight the difficulties of implementing drug, alcohol and tobacco use prevention programmes, and can give indications of how programmes are received and so indicate their potential results. For instance, the main obstacles reported when implementing programmes have been that communities (including school staff) have been reluctant to acknowledge drug problems and have not seen prevention as a pertinent issue (Henderson, 1998). Furthermore, process evaluations can be particularly valuable when an outcome evaluation suggests that an intervention is ineffective. In these cases a good process evaluation can suggest whether the failure is due to a poor intervention or perhaps due to a poorly implemented intervention. There is widespread acceptance of the principle that user groups should have ownership of health promoting activities, including drug education. Therefore when developing and evaluating the quality of health promotion practices it is essential to obtain the views of service users and other stakeholders, especially those of pupils, their parents, teachers and the external visitors. Evidence of the value of interagency partnership approaches is limited, although there appears to be widespread support for the notion (e.g. Payne, 2000). Finally some programmes may have been implemented with no evaluation undertaken, but nevertheless some formative evidence may have been collected during the programme s development which may provide pointers to its likely success. Again evidence of this sort is included in the review. Although the literature on drug, alcohol and tobacco prevention is large, the evidence base is often weak. The review examined evidence relating to the main questions posed and bases its recommendations on that existing evidence. The review also identifies areas where additional/improved evidence is required. 11

14 2 METHODOLOGY 2.1 Overview A steering group was formed to oversee the development of the review and shape its direction. The full literature review was preceded by a scoping exercise in the form of a review of reviews. This was undertaken to clarify research questions which were answerable through the existing literature, identify effective search strategies etc. Secondly a protocol was developed identifying the procedures to be followed in the literature review. 2.2 Scoping exercise and development of the protocol A review of reviews was undertaken as a scoping exercise and to clarify research questions answerable through the existing literature, identify effective search strategies etc. Once completed, and following discussions with the steering group, some minor modifications were made to the intended scope of the review and the aims and objectives were clarified. The protocol used to guide the full literature review was developed. The protocol outlined: inclusion/exclusion criteria; the literature to be searched, including the grey literature; the informants to be approached; the search strategy to be adopted; the hierarchy of evidence to be adopted to inform the strength of recommendations that can be made on the basis of the evidence collected; and an outline of how the material would be analysed. An important feature of the protocol was the development of a data extraction sheet to be used in the literature review. The data extraction sheet identified the information that would be needed in order to answer the research questions posed by the review. The data extraction sheet developed is to be found in Appendix 1. Its use was intended to ensure the equivalent treatment of the evidence collected from all of the reports and so reduce subjective bias in the reviewing process and aid the interpretation of the obtained materials and synthesis of the available evidence. Discussions with the steering group identified additional informants, clarified the inclusion/exclusion criteria and assisted with the final design of the data extraction sheet. The protocol was then used to ensure that the review of reports was undertaken in a consistent and systematic manner. 2.3 Literature review The full literature review included academic research, LEA reports, Ofsted reports, guidance and policy documents and reports on consultations with key stakeholders (e.g. children, young people, parents and teachers) which relate to the role of external contributors in school drug, alcohol and tobacco education. The review included both published and unpublished literature. A criticism of some systematic reviews is that they do not give sufficient attention to the quality of the assessed treatment/intervention concentrating only on research quality (HEA, 1997). In assessing drug, alcohol and tobacco education programmes attention was given to both the quality of the programme and the strength of the evidence of effectiveness. In particular where the evidence suggested an intervention had been 12

15 unsuccessful, the evidence was examined to determine whether the intervention was inherently faulty or just badly delivered Inclusion and Exclusion Criteria Reports were included that met the following inclusion criteria: Focus on classroom based delivery but also targeted groups within the school context/curriculum Evaluate the contribution of external contributors, defined as anyone who is not a teacher employed by the school Evaluate interventions introduced into primary, secondary, special schools and pupil referral units Literature produced from 1990 onwards British literature and international literature from other English speaking nations and western Europe (conclusions will focus on findings both within and without the international context) Reports were excluded that met the following exclusion criteria: Literature related to counselling within schools or incident related lessons Use of peer educators from within the school Technologies and resources developed by external contributors intended for use by teachers Search Strategy Studies were located through electronic searches, hand searching and through contact with relevant individuals and organisations. Electronic literature searches were carried out on core databases in the fields of education, health and social sciences, including: Medline/ PubMed PsycInfo Web of Science BIDS ISI Current Contents ERIC & British Education Index SciSearch Social SciSearch Dissertation Abstracts Project Cork (Dartmouth Medical School database of information on substance abuse) Drugscope National Research Register NFER CERUK database The Cochrane Library CRD Databases, University of York Eppicentre, Social Sciences Research Unit, Institute of Education, London Addiction Abstracts Online Health Education Board Scotland database DEPIS database DPI database 13

16 DPAS database The National Research Register of ongoing and recently completed research projects funded by, or of interest to, the United Kingdom's National Health Service was consulted to identify relevant work as well as entries from the Medical Research Council's Clinical Trials Register. Preliminary reports were sought from study researchers. Electronic searches were supplemented by hand searches or online examinations of tables of contents from 1990 to March 2003 of the journals Addiction Druglink Health Education Journal Health Education Research The Journal of School Health The Journal of Drug Education School Psychology International The search strategy for the review was developed in a number of ways. An initial search was undertaken and the Thesaurus and MeSH terms associated with identified studies noted. The initial search consisted of: health AND education AND school AND (substance OR drug* OR alcohol OR cigarette* OR smoking) AND (intervention OR promotion) AND evaluation. This produced a large number of hits (PubMed 108, PsycInfo 53, Drugscope 0, Project Cork 22,000 when reduced to health AND education AND school AND evaluation, but reduced to 45 when the term young was added). Where applicable, limits on age of participants were set, or alternatively the following terms included in the search syntax: adolescent* OR boy* OR child* OR girl* OR junior* OR teenage* OR (young NEAR people) OR youth Directories of Thesaurus and MeSH terms were consulted to identify further search terms. Free-text searches were also undertaken. Due to variance in the design of databases, slightly different search strategies were used for each. The zero success on Drugscope reflects its different cataloguing structure to the other databases Searching the Unpublished (Grey) Literature A number of agencies were contacted to seek help in the identification of low circulation reports. Some were contacted by inserting a call for assistance in the regular mailings of the Department of Health and the Department for Education and Skills (e.g. mailings to PSHE network, DAT coordinators, Directors of Public Health at PCTs, Regional coordinators of NHSS). A list of individuals responding to these mailings is shown in Appendix 2: their assistance was invaluable. Additionally, a number of individuals and organisations were contacted directly. These include Connexions, Life Education Centres, the 42 Tobacco Alliances covering England, Action on Smoking and Health, Alcohol Concern, Institute of Alcohol Studies, The Health Development Agency (HDA), Department of 14

17 Health Tobacco Policy Team, The NHS Centre for Reviews and Dissemination, EppiCentre (Evidence for Policy and Practice Information and Co-ordinating Centre), the Cochrane Collaboration, Collaborative Review Groups on tobacco addiction and drugs and alcohol. In addition all DAT coordinators and HAZ substance misuse projects were contacted. A full list of those contacted is shown in Appendix 3. These mailings and contacts led to the identification of large numbers of reports Hierarchy of Evidence Table 1 on p22 provides the hierarchy of evidence adopted for quantitative reports. It should be noted that randomised control trials were not included in the hierarchy. The reason for their exclusion was that diffusion of information, new norms etc. can be expected when random allocation to groups is made at the class level, and if allocation is made at the school level a very large number of schools must be included to allow for randomisation within stratified groups of schools. Even the largest studies have too few schools included for randomisation to be an effective protection against bias. Tackling bias directly by incorporating blinding, attending to attrition and testing/correcting for baseline differences was therefore the preferred option. Criteria for assessing the quality of qualitative studies were derived from existing guidelines for evaluating qualitative research (e.g. Greenhalgh & Taylor, 1997; Yardley, 2000) and include: Are important problems addressed via clearly formulated questions? Was a qualitative approach appropriate? How were settings and subjects selected? Has the researcher s perspective been identified and taken into account? Are methods used for data collection described in sufficient detail? How are the data analysed? Are quality control measures implemented? Are results credible and clinically important? Are the conclusions justified by the results? Are the findings transferable to other settings? Review Strategy and Data Extraction Titles and abstracts of articles identified by the searches were read by two reviewers to select the set of reports relevant to the review themes. Initial judgements were made on study design, quality of evidence and relevance to the topic. Hard copies of studies identified for inclusion, together with any about which a decision on inclusion was not possible from these brief details, were obtained for more detailed appraisal. To ensure that the reviewing process was undertaken in a systematic manner each report was read and core data extracted onto data extraction sheets reflecting the hypotheses and questions posed by the review. Reviewers decided whether the report should be included in the final review, its place in the evidence hierarchy and recorded their judgement of the effectiveness of the education described. The reviewers thus agreed on the final composition of included studies and ensured that the reports met the Inclusion Criteria set. Not all studies provided sufficient information to 15

18 allow judgements of quality or conclusions on effectiveness to be made. In these cases authors were contacted and further information requested. The time required to locate potentially relevant reports and the resultant size of the literature located did not allow data from each report to be extracted separately by two independent reviewers in the time available. Instead a 10% sample of reports was selected based on random selection of reports by acquisition number and these were used to establish reliability data and provide a check on the validity of the review procedures. Following a detailed reading of each report a decision was taken of where in the evidence hierarchy it sat and whether it was of sufficient methodological strength to be used to generate reliable conclusions about programme effectiveness. Additional indicators of methodological soundness include the extent to which the aims and outcomes of the study were clearly defined, the replicability of the study, the type of controls employed, the mode of subject recruitment, the level of subject refusal and attrition, the sample size, and the type of analysis conducted. Methodologically sound studies should include information of all measures employed. The reviewers' judgements of the conclusions to be drawn from the study were compared to the assessment of the authors. Quality was ensured by strict adherence to a precise protocol, which includes clear guidelines on inclusion and exclusion criteria and by using a data extraction sheet to ensure the systematic reading of papers and extraction of data. 16

19 Table 1. Proposed Hierarchy of Evidence Level of Evidence Strength of Recommendation 1) Process evaluation. Can provide valuable evidence about likely impact of the education and therefore relevant evidence to inform recommendations. 2) Formative research. Can provide valuable evidence about likely impact of the education and therefore relevant evidence to inform recommendations. 3) Outcome measures of drug, alcohol and tobacco use Provides weak basis for recommendations. and/or attitudes, intentions and skills. A follow up period of less than one year. 4) Programme clearly specified. Outcome measures of drug, alcohol and tobacco use. A follow up period of less than one year. Inclusion of control or comparison data OR baseline data collected, but not both. 5) Programme clearly specified. Baseline data collected. Outcome measures of drug, alcohol and tobacco use. A follow up period of less than one year. Inclusion of control or comparison data. 6) Programme clearly specified. Baseline data collected. Outcome measures of drug, alcohol and tobacco use. A follow up period of at least one year. Inclusion of control or comparison data. Those assessing baseline and follow up data blind to condition. Attrition levels reported and analysis of effects of attrition. Analysis of group differences at baseline and treatment of any resulting confounds. No methodological flaws identified. A follow up period of at least one year. 7) Programme clearly specified. Baseline data collected. Outcome measures of drug, alcohol and tobacco use. A follow up period of at least one year. Inclusion of control or comparison data. Those assessing baseline and follow up data blind to condition. Attrition levels reported and analysis of effects of attrition. Analysis of group differences at baseline and treatment of any resulting confounds. No methodological flaws identified. An assessment of the fidelity of programme delivery. Provides modest basis for recommendations. Provides sound basis for recommendations. Provides basis for strong recommendations. Provides basis for strongest recommendations. 17

20 3 RESULTS 3.1 Locating Reports Identification of Published Reports 614 reports were identified as potentially relevant to the brief. Of these 452 were rejected from the reading of the abstract. Hard copies of 162 were obtained. 109 reports were excluded following data extraction, although 16 of these were of peripheral interest and are included in an excluded studies table. This left 53 articles included in the review. Identification of Unpublished/Low Circulation Reports Calls for information on evaluations in progress and requests for reports were posted to various mailing lists. Twenty six responses were received resulting in the identification of 24 reports. Many of the reports received were unclear about the nature of the intervention and who delivered it and so requests for further information were made. The responses received and help provided are outlined in Appendix 2. Direct approaches were made to 383 organisations, mostly in the UK, including all DAT coordinators in England, Scotland and Wales, but also covering USA, Canada, Australia, The Netherlands, Sweden, France and Italy. Details are presented in Appendix 3. By these various means 194 low circulation reports were identified. 89 were rejected from reading titles and abstracts/summaries and 105 hard copies were obtained and data extracted from them. Following data extraction 33 were excluded as irrelevant to the brief, 16 were identified as having peripheral interest and are included in the table of Excluded Studies. The remaining 61 reports are included in the review. Combining published and unpublished reports. 808 reports were identified of potential interest, 267 hard copies were obtained and 114 are included in the review with a further 32 included in an excluded table as having peripheral interest. Summaries of the included reports are provided in Appendix 4. The table is organised by the agency involved. Appendix 5 presents brief information on the reports excluded from the review, but judged to be of sufficient peripheral interest to the purpose of the review to justify drawing attention to their focus. While the focus is upon the included studies, some reference is also made to the excluded reports. 3.2 Which external agencies/individuals have been examined and what aspects of their role has been assessed? The review identified reports relating to the involvement in classroom drug, alcohol and tobacco education of 16 different agencies. These are shown in Table 2. It can be seen that there are a large number of reports of the operation of Life Education Centres, but also many reports of the involvement of the police and theatre in education. The remaining agencies have not been reported on frequently. The frequency with which agencies are involved in schools is not reflected in the frequency with which their 18

21 involvement has been assessed. For instance school nurses are very frequently involved in the classroom, but their involvement has been assessed rarely. Table 2. Reports of Agency Involvement Agency Involved Number of reports Number of Outcome Evaluations Number of Methodologically Sound Reports Police Theatre in Education School Nurses Drug agency Workers Life Education Centres 136* Peers from external the school Users or Ex-users Health Educators/Health Professionals Parents Fitness Instructors Youth Worker Cartoon Animator Songwriter/singer Basketball Team Unspecified Legal and Health Experts Researchers *The review included a single summary of 129 projects and a further 7 reports. 19

22 The foci of the included reports were principally outcome evaluation and process and formative evaluation. Over 90% of the reports (103 of 114) included some evaluation of outcomes of the drug education offered by external visitors in terms of changes in aspects of pupils functioning. Of these outcome evaluations 42 included pre- and post- education assessments and included comparison data. These reports were judged to be methodologically sound, or better. There was a good selection of methodologically sound studies of the role of the police and of health educators, but good quality evidence on other external visitors is sparse. A number of different outcomes were assessed in the reports. The outcomes measured, their frequency of use overall and their frequency of use in methodologically sound studies are shown in Table 3. 20

23 Table 3. Outcomes assessed Measures Number of Reports Number of Methodologically Sound Reports Pupils drug, alcohol and tobacco use Intentions for future use Attitudes to substance use Attitudes to the police 2 1 Attitudes to school 2 0 School attendance 2 0 Discipline problems at school 3 2 Knowledge Self esteem Self efficacy 6 3 Resistance/refusal skills 13 7 Decision making/problem solving skills 7 3 Communication skills 2 1 Assertiveness 4 4 Perceived peer substance use Perceived peer acceptance of substance use Perceived costs/benefits of substance use Association with deviant anti-social peers Pro-social values

24 Different reports included different combinations of outcome measures. The three most commonly used measures, substance use, knowledge and attitudes, were each employed in under half of the reports. Eight measures were employed only two or three times. Nevertheless, only attitudes to school and school attendance have not been explored in a methodologically sound study. Over half (66 of 114) of the reports include some process evaluation data, although this often consisted of no more than a question on whether pupils enjoyed the input of the external visitor. As with the outcome evaluations the type of data collected varied across reports, but included stakeholders perceptions of the educational input as well as implementation issues. The themes assessed are shown in Table 4 with the number of reports in which each theme appears. Table 4. Aspects of the Education Assessed Aspects Assessed Number of Reports Enjoyment of educational input 30 Perceived effectiveness of educational input 13 Recommend to other schools 1 Quality of educational material 13 Quality of the teaching 13 Teaching techniques adopted 8 Controlling disruptive children 3 Liaison with teachers/schools 6 Involving parents 4 Drug policy 2 Fit to curriculum 2 Facilities and resources 2 Credibility of educator 6 Training 4 Follow up work 3 Willingness to repeat experience 8 22

25 3.3 What training is provided for external visitors? Some information about the training received by those delivering the drug education was provided in 60 reports (53%). The provided information covered the topics included in the training for external visitors and the amount of time devoted to training. In no case were details of training methods provided. No details were provided about the training external visitors may have received previously. Details of the training provided on individual projects are shown in Appendix 6. Often information about the training external visitors receive was scant or not presented. This was particularly true of the reports of Theatre in Education. The amount of training offered varied greatly, with Life Education Centres providing the most intensive training (12 weeks), but many programmes providing much less, e.g. two or three days training. However, it is likely that different professionals would have had different training needs depending upon their prior training and experience. Few reports include an assessment of trainees views of the adequacy and quality of the training received. As can be seen from the Table 4 there was little comment in the reports on training with only four reports including comments on training. Two of the reports of the involvement of ex-users, and one of the peer education reports comment on the benefits to the ex-user of undergoing training and becoming involved as a peer educator (East Sussex LEA; Avis, 1998; Davidson, 1997). Successful outcomes include moving into paid work or into higher education. Personal development such as gaining confidence, communication skills, co-working skills, and personal learning were also identified as gains (Avis, 1998; Davidson, 1997; East Sussex LEA). One peer educator was quoted as saying I m proud of who I am, and of SHADE s powerful influence in making me recognise my own potential (Avis, 1998). They reported that undertaking training away from their usual environment helped them to challenge their own behaviour (Davidson, 1997). Also some dramatically reduced their own drug usage whilst others did not use any substances at all during the project. On-going support is needed to ensure that this progress is improved upon or sustained (Davidson, 1997). In all of these cases the external visitor took principal responsibility for the delivery of the drug education during the session. Where the external visitor is involved in supporting teachers in the delivery of drug education there may be different issues. The final report to comment on training identified the advantages of joint training of teachers and health professionals together, such as enhanced understanding and communication between the parties (Stark et al, 2003). 3.4 How external visitors contribute to classroom based drug education Police led drug education Police are involved in drug, alcohol and tobacco education either as principal educators or as support for teachers (Ofsted, 2002). The most widely researched programme is Project DARE, an intervention developed in America which involves uniformed police officers delivering structured classroom drug, alcohol and tobacco education. DARE was introduced in the 23