Evaluation of Super Utilizer Innovations

Size: px

Start display at page:

Download "Evaluation of Super Utilizer Innovations"

Margaret Craig
10 years ago
Views:

1 Evaluation of Super Utilizer Innovations Marci Eads, Ph.D., Principal Mike Nardone, MPA, Managing Principal Health Management Associates November 18, 2014

2 Introduction Evaluation of Super Utilizer Innovations is hard. But it is also: feasible critical useful both in the short term and the long term 2

3 Like Many Things, Evaluation Is Hard Feasible & Balanced 3

4 Evaluation is Feasible: Real World Evaluation Nothing/ Poorly Designed Feasible & Balanced Ideal 4

5 Evaluation is Feasible: Getting Started What are the most important things you need to know about your program, who needs to know them, when, and what data can you convert to information to answer these questions? Who are you trying to impact and in what ways? Who are you going to serve? What are their needs? What and how much are you going to (can you) provide? What are your goals? Can you provide services that will meet their needs and your goals? What changes do you expect in the short, medium and long term? What do supporters and potential supporters care about? How will you know if it isn t working so you can fix it? What relevant and reliable data are available to you? What is the ROI? 5

What and how much are you going to (can you) provide? What are your goals? Can you provide services that will meet their needs and your goals?

6 Evaluation is Feasible: Getting Started 1. What information do providers and you need? 2. What information is actually available, and where is it? 3. Is the available information worth using? 4. What information is needed but not available, and is it worth getting? 5. Can you make effective use of the information? 6. Can providers make effective use of information? 7. How will information and system gaps be addressed? Critical Question: How will policy/program, data, and evaluation staff and vendors work together? 6

What information is needed but not available, and is it worth getting? 5. Can you make effective use of the information? 6.

7 Evaluation is Critical Evaluation Data Information Action Your Super Utilizer Initiative: Policies, Programs, Activities Other Initiatives 7

8 Evaluation is Critical Data Staff and Vendors Policy/ Program Staff Evaluation Staff Evaluation Data Information Action Your Super Utilizer Initiative: Policies, Programs, Activities Other Initiatives 8

Data Information Action Your Super Utilizer

9 Practical Tips: Planning the Evaluation Target population Who is involved in planning and implementation Who is involved in the evaluation and what their roles are Levels of evaluation Evaluation design options, threats to validity Data and measurement Uses of evaluation When information is available 9

their roles are Levels of evaluation Evaluation design options, threats to

10 Evaluation is Useful From the beginning, be thinking about how you will use your evaluation results Rapid cycle evaluation/program improvement Especially process measures, short term measures, early indicators Allows for immediate course corrections Allows for immediate targeting and shifting of priorities and services Sets up expectations of continuous attention to quality and cost Improves data quality Promote expansion/replication, generate/sustain buy in Especially via ROI Share lessons learned 10

corrections Allows for immediate targeting and shifting of priorities and services Sets up expectations of continuous

11 Practical Tips: Rapid Cycle Evaluation Think early about who needs to know what and when Link rapid cycle evaluation to: the needs you are trying to meet, your goals, and what you think you can impact Limit your data points to something manageable Make sure you give information tot the people who need it, in a way they can use it Track process challenges like attribution and enrollment, and use this information in a rapid cycle fashion (and in ROI calculations) 11

something manageable Make sure you give information tot the people who need it, in a way they can use it Track

12 Practical Tips: ROI Part of your overall evaluation all tips above apply Be sure to capture all costs Know that you will be missing some cost savings (i.e., in other systems) Know that rarely can you say definitively that your program caused this cost savings Many resources available - CHCS, Commonwealth Fund See study cited yesterday: 12

13 Practical Tips: Evaluation Design If you want to say something with credibility, you have to account for other explanations, be transparent about flaws. Careful design can help you with these common flaws: Threats to Validity Power Problems Lack of Findings 13

14 Back to Feasibility Determine your biggest priorities and goals, and design the most parsimonious evaluation you can to meet those goals. Make it simple, rigorous, useful, and understandable. 14