Using Case Management Data to Improve Programs and Achieve Results

Transcription

1 Using Case Management Data to Improve Programs and Achieve Results

2 Agenda Presentation (30 minutes) Q&A and Discussion (15 minutes) Small Group Exercise (15 minutes) Sharing and Wrap-Up (15 minutes)

3 A theory of change is a formal rendering of the approach adopted by an organization (or collaboration of organizations) to change something about the world

4

5 A results framework is a rendering of the details of your theory of change (often in the form of a table) that assists the organization(s) to plan for and examine what does and does not work

6

7 A case management plan is a results framework at the individual-level

8

9 Performance measurement takes both data and program managers to yield meaningful results

10

11 As defined in the results framework: As collected in the case management system: Performance measurement questions: Needs Assessment & Segmentation Analysis: Total # household and children, by age, living in the Promise Neighborhood footprint Target Population Segments by age groups, targeted schools, other targeting criteria such as ethnicity or housing circumstances Participant Identification and Intake Address and School Enrollment Demographics Family Roster Performance Accountability: Are we enrolling children and families who can most benefit from our programs, per our TOC and Segmentation Analysis? Are referrals from partners producing the target populations we said we d reach? Have we identified the right partners and methods for enrolling these families? Population Accountability Are our enrollment numbers sufficient to reach the penetration rates we specified across solutions in order to move outcomes for the entire population?

12

13 As defined in the results framework: As collected in the case management system: Performance measurement questions: GPRA Performance Results and Target Setting: The specific conditions of well-being and target results (and related indicators) set for each target population A turn the curve graph presenting historic baseline and forecast for target indicators for each population Referral and Case Planning Referring agency Enrolling agency (if other than the Promise Neighborhood agency, itself) Baseline indicator measures for target results at intake Additional case notes on presenting issues at intake Additional baseline data from program assessments/pretests Performance Accountability Are our case managers and/or our partners referring children and families to appropriate solutions based on identified needs? Population Accountability Are we providing solutions that match the level and type of identified needs in the target populations?

14

15 As defined in the results framework: As collected in the case management system: Performance measurement questions: Continuum of Solutions and Evidence-Based Strategy: The type and combination of evidence-based or research-informed interventions chosen to achieve the results for each population type. The partners who can deliver these interventions and their enrollment targets. Service Provision Types of service Units or intensity of service At or below target intensity/units Attendance Program exits and reason for exit For Performance Accountability When we break down our case management data by partner, can we verify that each referral partner is providing the type of solution(s) we said they would? How much did each partner do? (Are partners consistently at target enrollment levels?) How well did each partner do it? (Is the quality of or fidelity to the intervention being provided by each partner appropriate? Are participants meeting the target intensity/dosage levels expected for each program? Are program attrition rates appropriate?)

16

17 As defined in the results framework: As collected in the case management system: Performance measurement questions: Annual Performance Reporting and Continuous Improvement (all of the above): Annual performance data for target results and re-evaluation of performance targets Annual Assessment and Case Review Address updates Demographic updates Annual indicator measures for target results Additional case notes on progress Additional followup data from program assessments/post -tests Performance Accountability For each partner or program, how many children served achieved the intended results? Do results of particular solutions vary by type, dose over time, or quality of the solution provided by each partner? Which partners are our high performers? Why? For Population Accountability If changes occurred, were they the result of our solutions the way our theory of change predicted, or was something else happening in the community that may have had a bigger impact on the indicator? Did some participants have better results than others and, if so, what explains this difference?

18

19 BOX 2 Performance Measurement Questions: THE WHO Are we enrolling children and families who can most benefit from our programs, per our TOC and Segmentation Analysis? Are referrals from partners producing the target populations we said we would reach? Have we identified the right partners and methods for enrolling these families? Are our enrollment numbers sufficient to reach the penetration rates we specified across solutions in order to move outcomes for the entire population?

20 Data

21 Program Implications Segmentation analysis identifies Latino children as high priority Partner B has clearly enrolled more Latino children than any other program Understand Partner B s success Replicate the strategy in other parts of the continuum through increased financial and staff resources

22 BOX 3 Performance Measurement Questions: THE WHAT Are our case managers and/or our partners referring children and families to appropriate solutions based on identified needs? Are we providing solutions that match the level and type of identified needs in the target populations? When we break down our case management data by partner, can we verify that each partner is providing the type of service(s) we said they would? How much did each partner do? Are partners consistently at target enrollment levels? How well did each partner do it? Are participants meeting the target intensity/dosage levels expected for each program? Are program attrition rates appropriate?

23 Data Students who improved attendance among chronically absent students the previous year Number of students Considerable improvement Some improvement Little improvement No improvement % % % % Characteristic Gender Male Female Promise Neighborhood Program Mentoring After school recreation Tutoring

24 Program Implications Current strategies seem to be working better for girls than boys almost a third of previously chronically absent girls showed considerable improvement over half of previously chronically absent boys showed no improvement Mentoring program served the highest share of students who showed considerable improvement in attendance, Tutoring program by far served the highest share of students who showed no improvement in attendance Further performance discussions with partners to understand how attendance solutions can better serve the types of students who still are still struggling which programs might be the best to do so

25 BOX 4 Performance Measurement Questions: THE SO WHAT? For each partner or program, how many children served achieved the intended results? Do results of particular solutions (e.g., home visiting) vary by type, dose over time, or quality of the solution provided by each partner? Which partners are high performers? Why? Should funding be shifted among partners and/or solutions? If changes occurred, were they the result of solutions in the way the theory of change predicted, or was something else happening in the community that may have had a bigger impact on the indicator? Did some participants have better results than others and, if so, what explains this difference? (For example, characteristics of the participants, mix of interventions, duration of interventions, intensity of interventions, etc.)

26 Data

27 Program Implications Aggregate performance graph shows that both attendance and academic achievement fell. Disaggregated data show the stable cohort of students improved in both attendance and academics. Discuss with partners that solutions may be working for students who are stably enrolled in school. Opportunity to focus on how to provide additional programs or targeted case management services to students who are moving in and out of the target school, who are more likely to have lower rates of attendance and achievement.

28 SMALL GROU P WORK: Performance Measurement Questions Available Individual-Level Data Program Implications

29 WRAP UP: Sarah Gillespie Mary Bogle Chris Hayes