Caitlin M Cusack MD MPH Health IT Program Manager, NORC Lead, Value & Evaluation Team, AHRQ National Resource Center for Health IT

Transcription

1 Caitlin M Cusack MD MPH Health IT Program Manager, NORC Lead, Value & Evaluation Team, AHRQ National Resource Center for Health IT

2 Agenda AHRQ s National Resource Center for Health IT Why do we care about evaluating Health IT? Planning an evaluation, choosing measures Challenges of evaluation Available tools and resources for evaluation

3 Who is NORC? Affiliated with the University of Chicago, Chicago IL Not-for-profit established in 1941 Research conducted in the public interest Surveys, focus groups, observational studies Robust evaluation expertise Health services research Prime contractor for AHRQ s National Resource Center for Health IT

4 Agency for Healthcare Research and Quality Mission: To improve the quality, safety, efficiency and effectiveness of healthcare for all Americans Evidence through Research, Evaluation and Demonstration Convene Key Stakeholders Inform Policy 4

5 The AHRQ Health IT Portfolio Initial investments in Health IT began in 2004 AHRQ has now invested over $260 million These projects include: Grants Corporative Agreements Contracts

6 The AHRQ Health IT Portfolio The projects cross all populations, from large urban areas to small rural areas They cross all settings from large tertiary care centers to small physician practices The projects encompass all Health IT technologies Some support implementations, others support the impact of technology

7 National Resource Center Established a public website in June of 2005 Provides direct technical assistance to AHRQ s health IT projects Supports dissemination of lessons learned, both from within and outside of the health IT portfolio Acts as a resource for the latest tools, best practices and research results

8

9 Why Do We Care About Evaluating Health IT? Early adopters have implemented Health IT in order to: Improve clinical processes Improve quality of care Improve patient safety Improved efficiencies We have implemented because we have felt it was the right thing to do!!

10 Why Do We Care About Evaluating Health IT? Historically evaluation has ended up being secondary to implementation Implementations have proven to be difficult and costly Adoption has also proven to be difficult Focus has been on: Getting systems up and running Working to encourage the adoption of the system Evaluation is left as an after thought

11 Why Do We Care About Evaluating Health IT? With the number of very large implementations completed throughout the world, we should have robust, indisputable data on: Clinical impact Improvements in Safety and Quality Impact on efficiencies Return on investment True Costs Best practices The do s and don'ts of implementation

12 Chaudhry et al 257 studies reviewed 25% of the studies were from 4 institutions: Partners Healthcare, Boston MA Regenstrief Institute, Indianapolis IN Veteran s Health Administration Intermountain Healthcare, Salt Lake City Utah Only 9 studies evaluated multifunction commercial products Systematic Review: Impact of Health Information Technology on Quality, Efficiency, and Costs of Medical Care Chaudhry, et al Annals of Internal Medicine Volume 144 Number

13 Why Do We Care about Evaluating Health IT? We need a greater focus on evaluation in order to understand: What is the Value Proposition for health IT? What is the true clinical impact of these systems? What are the true costs of these systems? What has worked well? What hasn t worked? Is there an ideal way to roll these systems out?

14 Why Do We Care About Evaluating Health IT? What can an evaluation do for you locally? Address the interests of local stakeholders Increase your accountability and credibility Provide good publicity Provide data to help win over late adopters Provide data to inform future projects Provide data to plan for sustainability Provide lessons learned, barriers, facilitators for others

15 But Evaluation Can Be Tough! Evaluation can be challenging Easy to concentrate on implementation, putting evaluation aside Limited resources Balancing operational and evaluation needs can be difficult Limited literature to follow Tension between what one wishes to evaluate and the limitations of the IT implementation

16

17 The Evaluation Toolkit Created in response to a need, post-reviewing grantee evaluation plans Plans were disorganized, over scoped, not related to the technology, unrealistic The evaluation team conducted evaluation workshops, office visits, teleconferences, site visits, and created the toolkits and other tools

18 Evaluation Toolkit Project Description Project Goals Evaluation Goals Choose Evaluation Metrics Narrowing Metrics Draft a Plan Around Each Metric Write Your Plan Evaluation Resources Statistics Resources

19 Evaluation Planning Have a planning meeting with stakeholders, and touch base meetings thereafter The evaluation team Representation from those who are implementing the system Representation from the IT team Key stakeholder representation Invite others from your facility who are already collecting data As a group, work through the toolkit

20 Defining Goals Describe the implementation Define the goals of the implementation Define the goals of the evaluation

21 Choose Evaluation Measures Based on the goals you have outlined, choose metrics which will help you determine whether those goals have been met What is being measured should align with your goals If not, are the metrics or the goals incorrect? At the outset, don t consider what is feasible, but rather concentrate on everything that could be measured to demonstrate your success

22 Categories of Measures Clinical Outcomes Measure Clinical Processes Measures Provider Adoption and Attitudes Measures Patient Knowledge and Attitudes Measures Workflow Impact Measures Financial Impact Measures

23

24 Search for Other Easily Measured Measures Find out what data is already being measured in your organization Would any of these metrics add to your evaluation?

25 Narrow Down the Measures An effective evaluation focuses on a few measures done well Eliminate those measures not important to you and your stakeholders Consider which measures are feasible to accomplish

26 Consider the Project s Impact Impacts of a particular project vary based on where in the organization it is implemented and who will be using the new technology Ensure that the metrics you choose to measure will actually be impacted by your project

27 Determine Which are Important Determine which metrics are important to measure, taking your stakeholders into consideration It may be helpful to use a scale: 1 = Very Important 2 = Moderately Important 3 = Not Important This exercise will help your team to filter out those metrics unlikely to provide information of interest to your stakeholders

28 Determine Which Are Feasible Consider which metrics you ve choose are actually feasible to measure Consider your available resources: people, money, space, time Rather than abandon a project that turns out to have been ambitious, up-front focus on what is achievable Again, consider using a scale 1 = Feasible 2 = Feasible with Moderate Effort 3 = Not Feasible

29 Determine Sample Size Sample size is an important facet of feasibility Evaluation efforts can hinge on the number of observations planned or on the frequency of events to be observed. The less frequently the event occurs, the less feasible the planned metric becomes You will need enough observations to feel confident about the conclusions you want to draw from the data collected Estimating the number of observations needed for each metric may eliminate further metrics from consideration

30 Rank Your Choices Feasibility Scale 1-Feasible 2-Moderate Effort 3-Not Feasible Importance Scale 1-Very Important (1) (2) 2-Moderately Important (3) (4) 3-Not Important (5)

31 Choose Your Final Metrics You now have a list of metrics ranked by importance and feasibility Narrow the list down to four or five primary metrics Keep a list of secondary metrics that you can use if you have the time/people/financial resources to conduct

32 Evaluation Plan Should be tightly constructed For each measure consider the following: Data collection: What data is to be collected? What is the data source? Who will collect the data? Data analysis: How will the data be analyzed? Who will analyze the data? Budget: Is there enough money to complete the evaluation? Timing: When will you start? When will you finish? When will you complete your write up? Report: Who will write up the findings?

33 Toolkit Examples Pharmacy Barcoding Ambulatory CPOE Telemedicine-soon to come! Inpatient CPOE- soon to come!

34 Grantee Feedback kudos to the toolkit! It s been a very tangible way for me to herd the evaluation team With such great substance to it, it made my life much easier From a process point of view it was terrific Toolkit very valuable Served as check list so we wouldn t forget certain things Very good for me to stay on track and not forget things

35 Revised Toolkit July 2009 By measure, links to additional help Quick Reference Guides Canada Health Infoway s Benefits Evaluation Materials On-line Statistics Resources Additional measures in each category Revised data sources New Section on Study Design Cost considerations: impact of study design on cost New Section on Evaluation Resources New Examples

36 Initial Round of Grantee Plans Vastly over scoped Metrics unrelated to the implementation Little consideration to power Little consideration to resources: time, money, people Few details on timeline, people, budget

37 Second Round The grantees followed the toolkit outline Much better organized Far fewer, more relevant metrics Plans!

38 Data Exchange Toolkit Created out of a need: feedback from SRDs was that they were different from the grantees Basic format similar with the focus being on evaluation of projects involving data exchange Data Exchange Measures Data Exchange Examples

39 Measures Centered around data exchange between: Outpatient providers and laboratories Outpatient providers and pharmacies Providers and providers Providers and radiology centers Providers and public health departments

40

41 Challenges Delays in the implementation-beholden to issues occurring on the implementation side Delays in approval for the study (IRB?) Projects bogged down in developing a governance structure Lack of support from above and from stakeholders Changes in leadership mid-evaluation Lack of enough resources-including trained individuals Data collection unable to happen as planned

42 Challenges Integrating into the system what was to be evaluated proved more difficult than was conceived Pressure from stakeholders to measure something different Lack of response from vendors to fix issues Time delays due to upgrades Upgrades leading to the need to rebuild what was built for the evaluation Changes in the environment increased awareness, changes in policies

43 Challenging Metrics Anything which requires a manual chart review Anything which requires individuals with particular expertise Anything in which there is any potential controversy around definitions Anything in which new training has to occur prior to data collection Items where observations are occurring with low frequency Anything in which an expert is needed to review and validate

44 As soon as it becomes clear that what is being measured is not feasible the team should re-group and revise their plan to focus on what can be done Measuring a few things very well is far preferable than tackling numerous measures and not being able to finish them Where is the issue? How can that be mitigated? Does a discussion need to be held with your funders? Create a dialog Issues with completing an evaluation are valuable lessons learned

45

46

47

48 Compendium of Surveys The NRC received numerous requests for sample surveys For each request a search was made for surveys which could be used After a number of these were collected, the idea for the compendium was developed All surveys in the compendium are in the public domain Authors of surveys have given permission for their use

49

50

51

52

53

54

55

56

57

58 The Health IT Bibliography Purpose Information / knowledge resource Implementation-focused Vetted by experts Other key points Free access where possible Annotations from information specialist Primarily peer-reviewed articles

59 High quality implementationfocused literature on key informatics topics

60

61 Emerging Lessons from the Field Purpose Knowledge informed by practice Implementation-focused Other key points Topics include: EHR, CDS, CPOE, HIE A work-in-progress

62 Emerging Lessons Timely Organized Detailed with examples

63 A range of topics New topics each quarter New lessons every 6 mo.

64

65 What does the tool do? Purpose Measure ambulatory care workflow and workload Based on Overhage et al., 2001 Used in RCT Overview Institutions, clinics, physicians Activities major and minor Observations point in time for a physician

66 Database User s Guide Journal Article Examples

67

68 Quick Reference Guides Deep dive into measures in the toolkit Did not duplicate work done by Canada Health Infoway Prioritized those being done by current AHRQ Health IT grantees Further prioritized based on where there was literature

69 Quick Reference Guides Brief Description-why would you want to measure this? Measure Category-from the 6 toolkit domains Quality Domain-IOM quality domains Current Findings in the Literature Source of Data Methodology for Measurement Analysis Considerations Potential risks

70 Quick Reference Guides Percent of Orders Entered by Authorized Providers on CPOE Reduction in Length of Stay Percentage of Orders Ordered Verbally Percent of Alerts or Reminders that Resulted in Desired Action Prescribing Patterns of Cost-Effective Medications Prescribing Patterns of Preferred or Formulary Medications Medication Turn-Around Time in the Inpatient Setting

71

72 Useful Links National Resource Center for Health IT Evaluation Toolkits Compendium of Surveys Time and Motion Tool Health IT Bibliography