Leslie S Zun, MD, MBA, FAAEM Chairman and Professor Department of Emergency Medicine Professor Department of Psychiatry RFUMS/Chicago Medical School

Transcription

1 Leslie S Zun, MD, MBA, FAAEM Chairman and Professor Department of Emergency Medicine Professor Department of Psychiatry RFUMS/Chicago Medical School Mount Sinai Hospital Chicago, Illinois

2 Objectives Understand the role of the evaluator Pros and cons of outside evaluator Review of an example of a program with an outside evaluator

3 Framework for Program Evaluation US Public Health Service: Framework for Program Evaluation in Public Health. MMWR. 48; September 19,1999:1 40. Engage stakeholders Identify those affected by or will use the program Consider adding those opposed to the program Who are the end users Describe the program Mission and objectives of the program What is the need or gap for the program What effects are considered success What steps or activities are needed What resources are required

4 Framework for Program Evaluation Focus the evaluation design What is the evaluation purpose Relationship between programs activities and consequences Type of study design Gather credible evidence Well rounded picture of the program Meet with stakeholders How will results be used

5 Framework for Program Evaluation Justify conclusions The claims are made based on the data The conclusions are justified Ensure use and share lessons learned Provide feedback Follow up with the stakeholders Disseminate the lessons learned

6 Criteria in Selecting an Evaluator Individual or team Understanding of the program and its purpose Agree on measurement and outcome criteria Group facilitation level of involvement in study design, data entry, data analysis and conclusions How is the evaluation going to be used and reported Expertise and experience Cost How is the decision made

7 Inside or Outside Evaluator Inside member of the funded research team Outside becomes a ex officio member of the research team Evaluators Who funds them? What is their role in developing the program? What is role in the evaluation? Who do they report to?

8 American Evaluation Association Purpose to improve evaluation practices and methods Eight goals One of which is teach and learn evaluation in diverse professional settings Annual conference and interest groups

9 Sinai Health System and Boys & Girls Clubs of Chicago By Leslie S. Zun, MD and Jodi Rosen, MPH Funded by Joyce Foundation, Woods Fund, Open Society, Michael Reese Health Trust, Baxter International

10 Within Our Reach Introduction Who we are What we did What were our results Outside evaluator Funding Role in project design Role in project evaluation Duties and responsibilities

11 Introduction Sinai Health System 2 Acute Care Hospitals Rehab hospital Medical group Community institute Urban health institute Emergency Department Approaching 55,000 visits per year Level I Peds and Adult Trauma Primary teaching affiliation is Chicago Medical School Affiliated with the University of Chicago and Resurrection/St Francis Programs in Emergency Medicine 11/6/2013

12 Grant Funders Funded Evaluation - Joyce Foundation Requested outside evaluator Recommended evaluator Funded Project Woods Fund Open Society Michael Reese Health Trust Baxter International

13 Purpose of WOR To determine if addressing the psychosocial needs of at-risk youth can reduce recurrence of injury due to violence, with the ED as point of entry "Within Our Reach" Program

14 Structure of WOR Partnership between healthcare and social service providers Assessment and evaluation tools Enrollment Case management for 6 months "Within Our Reach" Program

15 WOR Basics year old victims of interpersonal violence Randomly assigned to treatment or control group Both groups evaluated at enrollment, 6 and 12 months Control group received brochure of services Treatment group received comprehensive assessment and case "Within Our Reach" Program management for 6 months

16 Within Our Reach Team Survey Tools Identifying and contact information Drug Use/Abuse Mental Health Gang Issues Education Employment Life Skills Pregnancy/Parenting Issues Medical Care Recreational Activities Gun Access Questionnaire

17 Evaluator s Tools Parenting Practices Questionnaire Family Relations Scale The Child Self Report of Delinquency CYDS Stress Measure Parents and Peers Peer Delinquency Future Expectations Attitudes and Beliefs about Achievement Aspirations/Values Social Competence

18 Program s Determinants of Success Enrollment in program Involvement in self improvements Self reported report of violence Return rate to ED Arrest/Incarceration

19 Evaluators Determinants of Success Social competency Values about achievement Values about responsibility Values about wealth Involvement with mother/father Emotional and social support from mother, father, peers Peer s delinquency level

20 Conclusion Program Successes Treatment group showed a significant reduction for selfreported victimization between the pre-test and posttest as compared to the control group Rate of recidivism for violent injuries was significantly lower in both WOR s treatment and control groups than ED historical use Treatment group (81.3%) more likely to use healthcare and social services than control group (9.8%) Evaluator assessment No change in behavioral assessment or attitudes as determined by the evaluator "Within Our Reach" Program

21 Lessons Learned Outside Evaluator Advantages Outside set of eyes Enhance study credibility and expertise Provides avenues of dissemination Disadvantages Reports to funder not PI May add survey tools, may not be validated May not support the study team

22 Contact Information Leslie Zun, MD Mount Sinai Hospital 1501 S California Chicago, IL zunl@sinai.org