Better Quality Through Better Measurement Worksheets

Transcription

1 Better Quality Through Better Measurement Worksheets Asia Pacific Forum on Quality Improvement in Health Care Robert Lloyd, PhD, Mary Seddon, MD and Richard Hamblin Wednesday 19 September R. C. Lloyd & Associates, R. Scoville and the IHI

2 Faculty Robert Lloyd, PhD Executive Director Performance Improvement Institute for Healthcare Improvement Boston, Massachusetts, USA Mary Seddon, MD Clinical Director Centre for Quality Improvement, Ko Awatea Counties Manukau DHB, Auckland, NZ Richard Hamblin, Director of Health Quality Evaluation Health Quality & Safety Commission, Wellington, NZ 2008 Institute for Healthcare Improvement/R. Lloyd & R. Scoville

3 Exercise Quality Measurement Journey Self-Assessment This self-assessment is designed to help quality facilitators gain a better understanding of where they personally stand with respect to the milestones in the Quality Measurement Journey (QMJ). What would your reaction be if you had to explain why using a run or control chart is preferable to computing only the mean, the standard deviation or calculating a p-value? Can you construct a run chart or help a team decide which control is most appropriate for their data? You may not be asked to do all of the things listed below today or even next week. But, if you are facilitating a QI team or advising a manager on how to evaluate a process improvement effort, sooner or later these questions will be posed. How will you deal with them? The place to start is to be honest with yourself and see how much you know about the QMJ. Once you have had this period of self-reflection, you will be ready to develop a learning plan for self-improvement and advancement. Use the following Response Scale. Select the one response which best captures your opinion. 1 I could teach this topic to others! 2 I could do this by myself right now but would not want to teach it! 3 I could do this but I would have to study first! 4 I could do this with a little help from my friends! 5 I'm not sure I could do this! 6 I'd have to call in an outside expert! Source: R. Lloyd, Quality Health Care: A Guide to Developing and Using Indicators. Jones & Bartlett Publishers, 2004:

4 Quality Measurement Journey Self-Assessment Source: R. Lloyd, Quality Health Care: A Guide to Developing and Using Indicators. Jones & Bartlett Publishers, 2004: Measurement Topic or Skill Moving a team from concepts to set of specific quantifiable measures Building clear and unambiguous operational definitions Developing data collection plans (including frequency and duration of data collection) Helping a team figure out stratification strategies Explain and design probability and nonprobability sampling options Explain why plotting data over time is preferable to using aggregated data and summary statistics Describe the differences between common and special causes of variation Construct and interpret run charts (including the run chart rules) Decide which control chart is most appropriate for a particular measure Construct and interpret control charts(including the control chart rules) Link measurement efforts to PDSA cycles Build measurement plans into implementation and spread activities Response Scale

5 The Three Faces of Performance Measurement Aspect Improvement Accountability Research Aim Improvement of care (efficiency & effectiveness) Comparison, choice, reassurance, motivation for change New knowledge (efficacy) Methods: Test Observability Test observable No test, evaluate current performance Test blinded or controlled Bias Accept consistent bias Measure and adjust to reduce bias Design to eliminate bias Sample Size Just enough data, small sequential samples Obtain 100% of available, relevant data Just in case data Flexibility of Hypothesis Flexible hypotheses, changes as learning takes place No hypothesis Fixed hypothesis (null hypothesis) Testing Strategy Sequential tests No tests One large test Determining if a change is an improvement Run charts or Shewhart control charts (statistical process control) No change focus (maybe compute a percent change or rank order the results) Hypothesis, statistical tests (ttest, F-test, chi square), p-values Confidentiality of the data Data used only by those involved with improvement Data available for public consumption and review Research subjects identities protected

6 Dialogue #1 Why are you measuring? How much of your organization s energy is aimed at improvement, accountability and/or research? Does one form of performance measurement dominate your journey? Does your organization approach measurement from an enumerative or an analytic perspective?

7 Exercise: Developing a Set of Measures and Operational Definitions Identify a project you are currently working on or plan to address in the near future. Select 1-2 Process, 1-2 Outcome and 1 Balancing measure and develop a clear operational definition for each measure. Use the Measurement Plan and Operational Definitions Worksheet s to record your work.

8 Measurement Plan Worksheet Measure Name Type (Process, Outcome or Balancing) P P O O B Operational Definition Source: Lloyd & Scoville 2010

9 Operational Definition Worksheet Team name: Date: Contact person: WHAT PROCESS DID YOU SELECT? WHAT SPECIFIC MEASURE DID YOU SELECT FOR THIS PROCESS? OPERATIONAL DEFINITION Define the specific components of this measure. Specify the numerator and denominator if it is a percent or a rate. If it is an average, identify the calculation for deriving the average. Include any special equipment needed to capture the data. If it is a score (such as a patient satisfaction score) describe how the score is derived. When a measure reflects concepts such as accuracy, complete, timely, or an error, describe the criteria to be used to determine accuracy. Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.

10 Operational Definition Worksheet (cont d) Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, DATA COLLECTION PLAN Who is responsible for actually collecting the data? How often will the data be collected? (e.g., hourly, daily, weekly or monthly?) What are the data sources (be specific)? What is to be included or excluded (e.g., only inpatients are to be included in this measure or only stat lab requests should be tracked). How will these data be collected? Manually From a log From an automated system BASELINE MEASUREMENT What is the actual baseline number? What time period was used to collect the baseline? TARGET(S) OR GOAL(S) FOR THIS MEASURE Do you have target(s) or goal(s) for this measure? Yes No Specify the External target(s) or Goal(s) (specify the number, rate or volume, etc., as well as the source of the target/goal.) Specify the Internal target(s) or Goal(s) (specify the number, rate or volume, etc., as well as the source of the target/goal.)

11 Dashboard Worksheet Name of team: Date: Measure Name (Provide a specific name such as medication error rate) Operational Definition (Define the measure in very specific terms. Provide the numerator and the denominator if a percentage or rate. Indicate what is to be included and excluded. Be as clear and unambiguous as possible) Data Source(s) (Indicate the sources of the data. These could include medical records, logs, surveys, etc.) Data Collection: Schedule (daily, weekly, monthly or quarterly) Method (automated systems, manual, telephone, etc.) Baseline Period Value Goals Short term Long term Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.

12 Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, Measure Name (Provide a specific name such as medication error rate) NON-SPECIFIC CHEST PAIN PATHWAY MEASUREMENT PLAN Operational Definition (Define the measure in very specific terms. Provide the numerator and the denominator if a percentage or rate. Indicate what is to be included and excluded. Be as clear and unambiguous as possible) Data Source(s) (Indicate the sources of the data. These could include medical records, logs, surveys, etc.) Data Collection: Schedule (daily, weekly, monthly or quarterly) Method (automated systems, manual, telephone, etc.) Baseline Period Value Goals Short term Long term Percent of patients who have MI or Unstable Angina as diagnosis Numerator = Patients entered into the NSCP path who have Acute MI or Unstable Angina as the discharge diagnosis Denominator = All patients entered into the NSCP path 1.Medical Records 2.Midas 3.Variance Tracking Form 1.Discharge diagnosis will be identified for all patients entered into the NSCP pathway 2.QA-URwill retrospectively review charts of all patients entered into the NSCP pathway. Data will be entered into MIDAS system 1.Currently collecting baseline data. 2.Baseline will be completed by end of 1 st Q 2010 Since this is essentially a descriptive indicator of process volume, goals are not appropriate. Number of patients who are admitted to the hospital or seen in an ED due to chest pain within one week of when we discharged them Operational Definition: A patient that we saw in our ED reports during the call-back interview that they have been admitted or seen in an ED (ours or some other ED) for chest pain during the past week All patients who have been managed within the NSCP protocol throughout their hospital stay 1.Patients will be contacted by phone one week after discharge 2.Call-back interview will be the method 1.Currently collecting baseline data. 2.Baseline will be completed by end of 1 st Q 2010 Ultimately the goal is to have no patients admitted or seen in the ED within a week after discharge. The baseline will be used to help establish initial goals. Total hospital costs per one cardiac diagnosis Numerator = Total costs per quarter for hospital care of NSCP pathway patients Denominator = Number of patients per quarter entered into the NSCP pathway with a discharge diagnosis of MI or Unstable Angina 1.Finance 2.Chart Review Can be calculated every three months from financial and clinical data already being collected 1.Calendar year Will be computed in June 2010 The initial goal will be to reduce the baseline by 5%within the first six months of initiating the project.

13 Exercise: Data Collection Strategies (frequency, duration and sampling) This exercise has been designed to test your knowledge of and skill with developing a data collection plan. In the table on the next page is a list of eight measures. For each measure identify: The frequency and duration of data collection. Whether you would pull a sample or collect all the data on each measure. If you would pull a sample of data, indicate what specific type of sample you would pull. Spend a few minutes working on your own then compare your ideas with others at your table.

14 Exercise: Data Collection Strategies (frequency, duration and sampling) The need to know, the criticality of the measure and the amount of data required to make a conclusion should drive the frequency, duration and whether you need to sample decisions. Measure Vital signs for a patient connected to full telemetry in the ICU Blood pressure (systolic and diastolic) to determine if the newly prescribed medication and dosage are having the desired impact Percent compliance with a hand hygiene protocol Cholesterol levels (LDL, HDL, triglycerides) in a patient recently placed on new statin medication Patient satisfaction scores on the inpatient units Central line blood stream infection rate Percent of inpatients readmitted within 30 days for the same diagnosis Percent of surgical patients given prophylactic antibiotics within 1 hour prior to surgical incision Frequency and Duration Pull a sampling or take every occurrence?

15 Dialogue #2 Common and Special Causes of Variation Select several measures your organization tracks on a regular basis. Do you and the leaders of your organization evaluate these measures according the criteria for common and special causes of variation? Percent C-sections Percent of Cesarean Sections Performed Dec 95 - Jun / /96 4/96 6/96 8/96 10/96 12/96 2/97 4/97 6/97 8/97 10/97 month 12/97 2/98 4/98 6/98 Medication Error Rate 8/98 10/98 12/98 2/99 4/99 6/99 UCL= CL= LCL= If not, what criteria do you use to determine if data are improving or getting worse? Number of Medications Errors per 1000 Patient Days Week UCL= CL= LCL=

16 Run Chart Exercise Now it is your turn! Interpret the following run charts Is the median in the correct location? What do you learn by applying the run chart rules? 1. % of patients with Length of Stay shorter than six days 2. Average Length of Stay DRG Number of Acute Surgical Procedures Source: Peter Kammerlind, (Peter.Kammerlind@lj.se), Project Leader Jönköping County Council, Jonkoping, Sweden.

17 % of patients with Length of Stay shorter than six days Antal patienter med vårdtid < 6dygn i % vid primär elektiv knäplastik (operationsdag= dag1) Antal patienter i % Månad Source: Peter Kammerlind, (Peter.Kammerlind@lj.se), Project Leader Jönköping County Council, Jonkoping, Sweden. Mäta för att leda

18 3,5 Sekvensdiagram DRG 373 Average Length of Stay DRG ,5 Vårddygn 2 1,5 1 0, januari Febr Mars April Maj Juni Juli Augusti September Oktober November December januari Febr Mars April Maj Juni Juli Augusti September Tidsenhet Source: Peter Kammerlind, (Peter.Kammerlind@lj.se), Project Leader Jönköping County Council, Jonkoping, Sweden. Grundläggande statistik och analys

19 Number of Acute Surgical Procedures Akut kirurgi Antal operationer Månad Source: Peter Kammerlind, (Peter.Kammerlind@lj.se), Project Leader Jönköping County Council, Jonkoping, Sweden. Mäta för att leda

20 The Control Chart Decision Tree Source: Carey, R. and Lloyd, R. Measuring Quality Improvement in Healthcare: A Guide to Statistical Process Control Applications. ASQ Press, Milwaukee, WI, Variables Data Decide on the type of data Attributes Data Yes More than one observation per subgroup? No No Occurrences & Nonoccurrences? Yes < than 10 observations per subgroup? Is there an equal area of opportunity? Yes No Yes No No Are the subgroups of equal size? Yes X bar & R X bar & S XmR Average and Range Average and Standard Deviation Individual Measurement c-chart The number of Defects u-chart The Defect Rate p-chart The percent of Defective Units np-chart The number of Defective Units

21 How prepared is your Organization? Key Components* Will (to change) Ideas Execution Self-Assessment Low Medium High Low Medium High Low Medium High *All three components MUST be viewed together. Focusing on one or even two of the components will guarantee suboptimized performance. Systems thinking lies at the heart of CQI! 21

22 The Sequence of Improvement Test under a variety of conditions Make part of routine operations Implementing a change Sustaining and improvements and Spreading a changes to other locations Theory and Prediction Developing a change Testing a change Act Study Plan Do

23 Thanks for joining us! Please contact us if you have any questions: Robert Lloyd Mary Seddon Richard Hamblin