E-VALU-IT UPDATE. RoSPA s mission is to save lives and reduce injuries. RoSPA s mission is to save lives and reduce injuries

Transcription

1 E-VALU-IT UPDATE Lindsey Simkins, Road Safety Research and Evaluation Officer, RoSPA Elizabeth Knight, Road Safety Manager, London Borough of Hounslow Presented by: Lindsey Simkins (RoSPA) AND Elisabeth Knight (Hounslow) Road Safety Seminar 2012 Page 1 of 7

2 DFT HEALTH CHECK VISITS Difficult to know how to evaluate road safety Education, Training and Publicity (ETP) Big projects more likely to be evaluated How to evaluate lower-budget smaller scale interventions How to know which intervention is most effective and best value? NEEDS ASSESSMENT Evaluation seen as determining success or failure Little reference to internal evaluation findings as a source of data Reported absence of evaluation support and training Time, money, knowledge, and occupational culture barriers Desire for practical how-to evaluation help using real examples Preferred training approach a mix of face-to-face and selfstudy I am unsure about how best to evaluate. WEBSITE AND TOOLKIT AIMS To increase knowledge and skills of individual users in the design and execution of evaluation plans To increase evaluation capacity at an organisational level Road Safety Seminar 2012 Page 2 of 7

3 WEBSITE AND TOOLKIT OBJECTIVES 1. To observe a 20% increase in users self-ratings of their ability to plan evaluations of ETP interventions 2. To observe a 20% increase in users confidence to conduct evaluations of ETP interventions 3. To observe a 20% increase in users ability to advise others on evaluating ETP interventions WEBSITE AND TOOLKIT OBJECTIVES 4. To increase the percentage of internal evaluations conducted from 46% to 66% 5. To increase the percentage of evaluation reports completed from 79% to 95% 6. To reduce the percentage of evaluation reports only available upon request from 45% to 15% WHAT DOES THE TOOLKIT DO? Asks you questions about the INTERVENTION: What and why Aims and objectives Whom it intends to benefit (e.g. taxi drivers aged 25-54) Timescales and costs Prompts thinking with reality checks : Sufficient evidence to base intervention on? Sampling and access considerations who and how to contact? Matching outcome measures? Road Safety Seminar 2012 Page 3 of 7

4 EVALUATION RECOMMENDATIONS ASKS you about: The purpose of the evaluation improve and/or prove Monitoring and outcome measures TELLS you about: Types of evaluation e.g. formative and summative Evaluation designs e.g. RCT, post-then-pre Evaluation methods e.g. Questionnaire, In-depth interview HELPS you choose: Type of evaluation, design and method USER SURVEY APRIL 2010 (35 responses out of 160) In one quarter of cases no evaluation of ETP interventions Willingness to evaluate higher than perceived confidence or ability E-valu-it Toolkit and Report Template most helpful Toolkit could be simplified More guidance on writing questionnaires Time and funds to evaluate is key issue The biggest barriers are capacity, in terms of finance and staff time. Pressure is to deliver the same amount of service with less staff/budget... RECOMMENDATIONS IMPROVEMENTS TO CONTENT AND FUNCTIONS OF TOOLKIT INCREASE GUIDANCE ON WRITING SURVEY QUESTIONS OFFER BESPOKE TRAINING WORKSHOPS PROMOTE GREATER SHARING OF REPORTS ENCOURAGE PRACTITIONER DIALOGUE PUBLICISE AVAILABLE SUPPORT Road Safety Seminar 2012 Page 4 of 7

5 TRAINING DAYS 6 FURTHER TRAINING DAYS HELD ACROSS ENGLAND SINCE JAN EVERY WORKSHOP EVALUATED AGAINST OBJECTIVES PRACTICAL FOCUS ON REAL INTERVENTIONS WHY AND WHEN TO EVALUATE, AIMS AND OBJECTIVES, LOGIC MODELS, USING THE TOOLKIT, QUESTION WRITING AND RUNNING FOCUS GROUPS NEED MORE THAN ONE DAY! EVALUATION IS ABOUT LEARNING EXPERIENCES OF DOING EVALUATION The RSO experience of doing evaluation. Before Website: Poorly understood/unrealistic aims and objectives Evaluation typically in the form of: Numbers participating/passing courses Happy sheets After Website: Use of Logic Model to set realistic, measurable aims Better understanding of appropriate evaluation methods Importance of planning evaluation into projects from the beginning USER S TALES 1 I've used E-valu-it several times and whilst it does take time to get grips with it, I've found the most valuable part is it asks you lots of "why" questions... With Crash Course (an education intervention aimed at learners), by using the toolkit it made us find a comparison group to compare results. This showed the intervention worked. (Ruth Gore, Safer Roads Humber) Road Safety Seminar 2012 Page 5 of 7

6 USER S TALES 2 Whilst the number of questions can take time, the process of questioning why you are planning a certain intervention really focuses the mind. On several occasions the question-answering stage showed insufficient justification for a certain project and we went back to the drawing-board or abandoned the project before it began Be prepared knowing the scale of the problem; understanding casualty issues; examining different data sources; and knowing who your target audience is makes the questions quicker to answer Test questionnaires with colleagues/sample audiences early on so they can be tweaked in plenty of time (Tanya Fosdick, Road Safety Analysis Ltd) USER S TALES 3 The hardest thing with evaluation is getting started. It was much easier when we didn t have to evaluate, or we thought we were evaluating because we asked attendees if they enjoyed the course! Knowing we should evaluate and doing it is not an easy transition to make One of the most useful tools I have found is using a logic model as a planning exercise, it really makes me consider what I am doing and why, and if it s measurable. After the training I felt enthused to begin an evaluation which would hopefully prove and improve the initiative (June Howlett and Nigel Spencer Bucks County Council) ADVICE TO ANYONE STARTING E-VALU-IT Get started put something into the questionnaire even if it s only the name of the scheme! Use a logic model to help plan If it doesn t go to plan, don t get mad, get in touch with Lindsey and the team Make sure you save everything along the way Once reports are complete I send them to Lindsey for comment before I convert it to PDF Road Safety Seminar 2012 Page 6 of 7

7 ADVICE TO ANYONE STARTING E-VALU-IT Whatever puts you off using it, communicate with the E-valu-it team, they are very willing to help resolve issues and E-valu-it will only improve through us using it and feeding back on any problems we encounter If possible attend one of the training days or get together with others to arrange a training day specifically aimed at your needs Get started I think I may have mentioned this before, but it s crucial (June Howlett Buckinghamshire County Council) Road Safety Seminar 2012 Page 7 of 7