Presenter Disclosure Information

Transcription

1 Presenter Disclosure Information Mathew J. Reeves PhD, FAHA From Research to Reality: How YOUR Data Collection Efforts Impact Bedside Care and Improvements in Quality FINANCIAL DISCLOSURE: None UNLABELED/UNAPPROVED USES DISCLOSURE: None 1

2 From Research to Reality: How YOUR Data Collection Efforts Impact Bedside Care and Improvements in Quality Mathew J. Reeves, PhD, FAHA Department of Epidemiology Michigan State University

3 Data quality matters. The manuscript could be improved by more readily acknowledging issues with data quality throughout. This is a registry implemented by a very diverse group of users, some with limited training and all with biases to interpret data favorably from the institutions perspectives. Some data quality audits occur through JC site visits, but we all know (and the data demonstrate) that there are many limitations to data quality. This is demonstrated, for example, in figure 1, where.. 3

4 Data Quality Sure data quality is important but its just..... so boring!!! 4

5 First, let s understand who we are Understanding your audience.. 5

6 Q1. How are you involved in stroke care? (CHECK ONLY ONE) 1. I provide care (e.g., RN, MD, OT) 2. I both direct and provide care (e.g., MD/RN Champion) 3. I both provide and document care 4. I document care (e.g., abstractor) 5. I work on hospital systems and processes to improve care (e.g., QI department) 17% 27% 26% 7% 23%

7 Q2. Are you specifically responsible for data abstraction? (completing the GWTG-Stroke PMT) 1. Yes - regularly 2. Yes - but only occasionally 3. Never/rarely 4. Who me???!!! I am a doctor! 5. Who me???!!! I have an MBA! 36% 12% 38% 12% 2%

8 Q3. I think the quality (meaning accuracy) of the data we collect on stroke care in my hospital is.. 1. Exceptional 2. Good 3. Quite good 4. Fair/mediocre 5. Very Poor 32% 43% 13% 9% 3%

9 Q4. Which of the following statements best describes your attitude to this statement: The quality (accuracy) of hospital data collected on stroke care is critically important. 1. Yes it s critically important (its our number 1 priority) 2. Yes but it s important only to the extent that accurate data can be feasibly recorded 3. Yes - but it s limited by the system e.g., lack of EMR 4. Kinda but I can t change the system, so.. 5. No we have more important things to worry about 51% 32% 9% 6% 1%

10 Outline Part 1: Data quality work undertaken by Coverdell and GWTG-Stroke registries A. Case ascertainment audits B. Data reliability assessments Part 2: Does data documentation impact care? Evidence from the GWTG-Stroke registry Part 3: Practical challenges to maintaining data quality Complexity of data elements Missing data Census vs. sampling Quality Assurance - Staffing & Training Principles of Quality Control 10

11 The Importance of data quality.. We all want a registry that is. Representative (unbiased case sampling) Accurate/valid data elements (or at least reliable) But what is the goal of the registry? Epidemiologic research, public health surveillance, or quality improvement (QI)? Data quality standards should reflect the underlying purpose and use of the registry 11

12 Part 1: Data quality work undertaken by Coverdell and GWTG-Stroke registries A. Case Ascertainment Audits Must ensure that registry cases are representative of all eligible acute stroke admissions Don t include the wrong patients, and don t exclude the right patients! (= no selection bias) To do this we must have: A system to capture all potential acute stroke admissions A set of practical, standardized case definitions that can be implemented with accuracy by the hospital 12

13 A. Case Ascertainment Audits Relatively few studies MASCOTS case audit study Several studies assessing accuracy of ICD-9 codes Interest in comparing hot vs. cold pursuit mechanisms 13

14 hot vs. cold pursuit Prospective case ascertainment Hot pursuit (active case identification) by regular screening of ED census log, neurology consults/admissions, ward census logs Identify all admissions with signs and symptoms consistent with acute stoke (= suspect strokes) Apply clinical acute stroke case definitions (with final confirmation after discharge. May include confirmatory ICD-9 code?) 14

15 hot vs. cold pursuit Retrospective case ascertainment Cold pursuit or passive case ascertainment Identify stroke discharges with ICD-9 codes 430, 431, 432.9, listed as a primary or secondary code Review chart and determine whether acute stroke was the primary reason for admission 15

16 Q5. At my hospital we identify eligible stroke cases by. 1. Identifying them prospectively ( hot pursuit) 61% 2. Identifying them retrospectively ( cold pursuit) 3. A combination of both 4. Not sure 21% 10% 9%

17 MASCOTS Case Audit Study Assessment of a hot pursuit mechanism 17

18 MASCOTS Coverdell Registry ( ) Phase I Hospitals: 1 Harper University Hospital (Wayne Co.) 2 Detroit Receiving Hospital (Wayne Co.) 3 Ingham Regional Hospital (Ingham Co.) 4 Sparrow Hospital & Health System (Ingham Co.) 5 Borgess Medical Center (Kalamazoo Co.) 6 Bronson Methodist Hospital (Kalamazoo Co.) 7 St. Joseph Mercy - AA (Washtenaw Co.) 8 Univ. Of Michigan Hospital (Washtenaw Co.) Phase II Hospitals: 9 St. Mary s Hospital (Saginaw Co.) 10 Spectrum Health (Kent Co.) 11 St. Joseph Mercy (Macomb Co.) 12 Henry Ford Wyandotte Hospital (Oakland Co.) 13 Riverside Osteopathic Hospital (Wayne Co.) 14 Northern Michigan Reg. Health System (Emmet Co.) 15 St. Joseph Health System - Tawas (Iosco Co.) 16 Alpena General Hospital (Alpena Co.)

19 MASCOTS Case Audit Study Information on 8 stroke and 8 non-stroke cases were abstracted from each of 15 hospitals. Four stroke physicians independently assessed all 240 abstraction forms and classified each as: stroke, non-stroke or cannot be determined Consensus on physician assignments developed by a modified Delphi approach. Final physician designations regarded as the gold standard. Calculated Sensitivity, Specificity, and Predictive values of the hospital designation relative to the physician panel. 19

20 Agreement in designation of acute stroke between hospital team and independent physician panel Panel Decision Hospital Assignment Stroke Total Stroke Nonstroke Undetermined Nonstroke Sensitivity = 93/105 = 88.6% Specificity = 114/114 = 100% Predictive value positive = 93/93 = 100% Predictive value negative = 114/126= 90.5% 20

21 What is the accuracy of ICD-9 stroke codes ( )? Williams GR, Stroke (1999). Summarized the results of 4 prior studies. Codes are far from perfect with an inherent trade-off in terms of Se and Sp. 430, 431, 434, and 436 are the most accurate to identify acute stroke admissions. Code PPV % % % % % % 21

22 Accuracy of ICD-9 codes in MASCOTS Study Compared ICD-9 discharge codes for 2563 prospectively identified stroke admissions 85% (n= 2185) had a primary stroke code ( ) Of the 15% (n= 378) who did not: 31% had a primary code for another CVD condition 10% had a primary code for another Neurological condition 59% had a primary code for some other condition But 32% (n= 121) had a stroke code listed in a secondary position Hence, 2306 (90%) could have been identified by ICD-9 codes 22

23 Which approach is better? Prospective or retrospective? BASIC Project (Piriyawt, Am J Epi, 2002) Compared two approaches to identify strokes Of 666 confirmed hospital admissions Active system identified 91% Passive system identified 89% For active system, overall PPV of suspect cases was 11% For passive system, overall PPV of ICD-9 codes was 73% Concluded that active system was more accurate, more timely and led to more complete and accurate data Method of choice if resources are sufficient 23

24 Q6. At my hospital we abstract data from the charts of stroke cases. 1. Concurrently (during the hospital stay) 2. Retrospectively (after the patient is discharged) 3. A combination of both 4. Not sure 25% 52% 11% 11%

25 Part 1: Data quality work undertaken by Coverdell and GWTG-Stroke registries B. Data reliability assessments Key attributes of data quality are: Completeness extent to which all necessary data are documented Accuracy extent to which documented data conforms with the truth Proving accuracy or validity requires gold standard data, so we often have to revert to measuring reliability (repeatability) 25

26 Reliability (Repeatability) Wikipedia: In statistics, reliability refers to the consistency of a measure. A measure is said to have a high reliability if it produces consistent results under consistent conditions. Inter-rater reliability Intra-rater reliability 26

27 RTI audit of data completeness and agreement in eight PCNASR prototypes Reported by Yoon et al, 2006 Independent re-abstraction of a random sample 2,820 charts (38% of available subjects) from 91 randomly selected hospitals Compared completeness and agreement of data elements between prospective (CA, IL, MA, MI, NC) and retrospective (GA, OH, OR) sites Organized data elements into 12 categories. Completeness = data element documented Exact match = prototype abstractor and RTI auditor obtained the same value or agreed element was missing 27

28 Completeness Categories Prototype Audit Overall 79.1% 79.7% Demographics 97.7% 97.3% Pre-hospital EMS 64.6% 72.1% Emergency department 86.7% 83.9% Imaging 71.0% 70.0% Signs & symptoms of onset 63.5% 61.5% Thrombolytic treatment 53.5% 55.9% Reasons for non-rx thrombo. 57.1% 56.5% Medical history 91.5% 91.5% In-hospital Dx procedures 93.4% 95.0% In-hospital treatment 92.1% 92.6% Other hospital complications 99.6% 99.4% Discharge data 78.1% 80.6% 28

29 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Fig 1. Completeness of categorized data elements by data collection methods (I) Prospective Retrospective 29 Demographics Pre-Hospital EMS Data Emergency Department Imaging Sign & Symptoms Onset Thrombolytics

30 Fig 2. Completeness of categorized data elements by data collection methods (II) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Prospective Retrospective 30 Medical history data collection methods (II) In-hospital treatment In-hospital complications Discharge data Reasons for no thrombolytics

31 Fig 3. Exact match rate (%) of categorized data elements between prototypes and audit by data collection methods (I) 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Prospective Retrospective 31 Demographics Pre-Hospital EMS Data Emergency Department Imaging Sign & Symptoms Onset Thrombolytics

32 RTI audit - Conclusions Clearly defined, specific, and easy to understand data elements were more complete and had higher agreement (regardless of approach used) Data elements critical to the quality of stroke care were frequently incomplete and inaccurate: Date and time of stroke onset and ED arrival Use/non-use of rt-pa No significant differences between retrospective 32 or prospective approaches

33 MASCOTS Data Reliability Study Experienced stroke nurse visited each hospital (n= 15) and re-abstracted 8 stroke cases (n= 120) 3 cases were determined to be non-stroke, 13 charts were missing, final sample = 104 confirmed strokes Inter-rater reliability of original vs. re-abstracted data assessed. Calculated % agreement, Kappa (k), or intra-class correlation (ICC). K or ICC > 75% = excellent agreement. 33

34 Kappa Corrects for the agreement that occurs due to chance. Range -1 to +1 Interpretation: >0.75 = excellent agreement >0.40 = moderate agreement >0.20 = fair agreement <0.20 = poor agreement 34

35 MASCOTS Reliability Study BMC Neurology 2008; 8:19 35

36 MASCOTS Data Reliability Study - Results Excellent (K >0.75) Moderate (K >0.40) Fair/poor (K < 0.40) Age Ischemic stroke Str onset time (esti) Gender TIA Stroke team consult Black race White race Brain image time D/C medications Non-ambulance arrival D/C destination Hem. Stroke Hospital transfer Walk w. assistance D/C mrs Direct admit DVT prophylaxis Medicare Ambulance arrival Medicaid PMH stroke ED arrival time (min)* ED arrival time missing + Stroke onset time (min)* Str onset time missing + * When a time value was recorded. + Was time documented or not? 36

37 GWTG-Stroke Data Reliability Study Am J. Heart 2012; 163:392 37

38 GWTG-Stroke Data Reliability Study 223 GWTG-Stroke hospitals with >10 cases between Oct Sept 2009 were selected (66%, n= 147 responded) 3 charts per hospital randomly selected (at least one was a tpa eligible case). 438 charts were copied and submitted to Outcomes Sciences. Data were re-abstracted by trained auditors who compared values to what was previously recorded in GWTG-S database (an independent approach?) Calculated exact agreement (% accuracy), Kappa (k) and intra-class correlation (ICC). 38

39 GWTG-S Data Reliability Study - Results Excellent (K >0.75) Moderate (K >0.40) Age Brain imaging at your hospital Gender DVT prophylaxis by HD 2 Race IV tpa started at outside hosp Pre-admit medications D/C medications IV tpa started at your hosp ED arrival time (min)* Last known well time (min)* * Time accurate to within 15 mins High rate of missing data for hispanic race (46%), last known well time (19%), body weight (21%) 39

40 Audit Studies - Conclusions Reliability of individual variables varied greatly from excellent (e.g., age, gender) to fair (e.g., time related variables, stroke team consult). Results are highly dependent on audit methods (e.g., highly positive GWTG-S results) Clearly defined, specific, and easy to understand data elements result in higher quality data! 40

41 Part 2. Does data documentation impact care? Evidence from the GWTG-Stroke registry

42 Background A reviewer s comment to a prior publication concerning a QI study from a Michigan Stroke registry showing improvement in 5 of 16 care processes. Jt Comm J Qual Patient Saf. 2006;32(9): most of these changes probably represent better documentation rather than actual improvements in care Anon,

43 Temporal Trends in Acute Stroke and TIA Care % % Patients Treated. 60% 40% 20% 0% IV rt-pa 2 Hour Early AntiThrom DVT Proph DC Antithrom Anticog for AF LDL100 Smoking All-or-None Fonarow GC et al. Circ Cardiovasc Qual Outcomes epub Feb 22 Temporal trend P value is < for each measure 43

44 Factors that can influence performance measure compliance Change in the size of the target population (denominator) of a performance measure a smaller denominator increases compliance without an increase in treatment (numerator) How could this happen? Changes in the documentation of broad inclusion/ exclusion criteria. Changes in documentation of contraindications increase in contraindications a smaller denominator esp. relevant for complex measures e.g., IV-tPA, warfarin AF 44

45 Factors that can influence performance measure compliance Changes in documentation of missing data could exclude patients from numerator or denominator. could result in biased compliance rates if data not missing at random. Improved documentation of measure completion increase in the numerator causes an apparent increase in compliance esp. relevant for processes that are not well documented e.g., smoking cessation Gaming direct manipulation of either numerator or denominator 45

46 Factors that can influence performance measure compliance Better care!! A true increase in treatment rates More eligible patients get treated Numerator increases, denominator stays the same This could happen!! 46

47 Objectives To understand the relative contribution of the following mechanisms to increases in performance measure compliance : An increase in the proportion of eligible cases receiving treatment (i.e., improved care) An increase in the documentation of contraindications (i.e., improved documentation) A reduction in the prevalence of missing data (i.e., improved data quality) 47

48 Circ Cardiovasc Qual Outcomes 2011;4;

49 Methods Data on 569,883 ischemic stroke (IS) patients collected at 1,028 GWTG-Stroke hospitals between April 2003 and September Seven GWTG-Stroke performance measures (PM) examined: 1. IV tpa 2hr (arrived 2hrs of symptom onset). 2. Early anti-thrombotics ( 2 days). 3. DVT Prophylaxis (discharge 2 days, non-ambulatory). 4. Discharge anti-coagulants for Atrial Fibrillation (AF). 5. Discharge lipid therapy (LDL-C 100 mg/dl, or not documented, or on therapy prior to admission). 6. Discharge anti-thrombotics. 7. Smoking cessation 49

50 Methods Define the Target Population for each performance measure Target Population = the sub-group of acute ischemic stroke cases that were broadly eligible for each measure Defined according to the measure inclusion and exclusion criteria Examples: IV-tPA - ischemic stroke patients arriving within 2 hours of onset Smoking cessation - current smokers, and regular hospital discharge Examined trends in relative size of target populations across all years 50

51 Methods Define treated, non-treated, contraindications or missing data for each performance measure GWTG-Stroke definitions were applied to each PM to generate 4 mutually exclusive groups: 1. Proportion of subjects who were eligible and received treatment (Eligible TRT). 2. Proportion of subjects who were eligible but did not receive treatment (Eligible NO TRT). 3. Proportion of subjects who were ineligible due to documented contraindications. 4. Proportion of subjects whose eligibility/treatment status could not be determined due to missing data. 51

52 Figure 1. Trends in Size of Target Populations for 7 GWTG-Stroke Performance Measures % D/C anti-throm D/C lipid therapy Early anti-throm DVT Proph IV-tPA 2hrs D/C Anti-coag AF Smoking cessation Year % of all ischemic stroke admissions eligible for each measure 52

53 Figure 2 - IV-rtPA IV-tPA 2 hrs 100% % Subjects 80% 60% 40% 20% 0% Missing Contraindications No Treament Treatment Calendar Year 53

54 Figure 2 - Lipid therapy at discharge LDL >100 or ND % Subjects 100% 90% 80% 70% 60% 50% 40% 30% 20% Missing Contraindications No Treament Treatment 10% 0% Calendar year 54

55 Figure 2 - DVT prophylaxis DVT Prophylaxis % Subjects 100% 80% 60% 40% Missing Contraindications No Treatment Treatment 20% 0% Calendar Year 55

56 Figure 2 Early anti-thrombotics Early antithrom botics 100% % Subjects 80% 60% 40% 20% Missing Contraindications No Treament Treatment 0% Calendar year 56

57 Figure 2 Anti-thrombotic therapy at discharge Antithrombotics at discharge 100% 90% % % Subjects 70% 60% 50% 40% 30% Missing Contraindications No Treament Treatment 20% 10% 0% Calendar Year 57

58 Figure 2 Anti-smoking treatment at discharge Anti-Smoking Treatment at Discharge % subjects 100% 80% 60% 40% 20% Missing No Treatment Treatment 0% Calendar year 58

59 Figure 2 Anticoagulation for AF at discharge Anticoag. for atrial fibrilation % Subjects 100% 80% 60% 40% 20% Missing Contraindications No Treatment Treatment 0% Calendar year 59

60 Detailed examination of contraindications to Anticoag for AF PMT has a NC (= non contraindicated) checkbox followed by a series of checkboxes for specific contraindications. 60

61 Detailed examination of contraindications to Anticoag for AF Between 2003 and 2009 there was: A decrease in unspecified contraindications (NC checkbox) from 48% to 5% A slight increase in reporting of 3 specific contraindications: Bleeding risk increased from 7.9% to 13.7% Fall risk increased from 3.2% to 8.7% Patient refused increased from 0.8% to 2.9% 61

62 Conclusions These data suggest that the majority of performance improvement in GWTG- Stroke was associated with more patients receiving treatments for which they are eligible for, rather than changes to the underlying target population or in the documentation of contraindications or missing data. 62

63 Conclusions This study was not able to determine the role of improved documentation of measure completion (vs. better care) We still don t know if increase in smoking cessation was simply better documentation of care that had always been given. Regardless a major goal of any QI project is to improve data documentation. Not able to asses potential for gaming without independent validation of actual care 63

64 Part 3: Practical challenges in maintaining data quality in Stroke registries Complexity of data elements Missing data Census vs. sampling Quality Assurance - Staffing & Training Quality Control 64

65 Data quality is improved by Data elements that are easily understood Good and continued communication between investigators and collectors High quality training of motivated data collectors Institutional support for policy and systems change Structured data recording systems, like GWTG, greatly facilitate the collection of high quality data. 65

66 Complexity of data elements Tracking quality of care for stroke patients unfortunately requires the collection of complex or difficult to obtain data elements Date and time on stroke onset and ED arrival Date and time imaging results available Reasons for non-treatment with tpa Presence of contraindications Confirmation of stroke sub-type Severity of stroke (NIHSS or other) 66

67 Missing data Complexity of data elements contributes to the problem of missing data Need greater emphasis on the routine documentation of essential clinical data Documentation of date and time of stroke onset, ED arrival, stroke severity (NIHSS) should be a standard of care. Pay-for-reporting, meaningful use, and EMR will 67 all help?

68 Q7. The quality (meaning accuracy) of data collected on stroke care is/will be vastly improved by EMR.. 1. Strongly Agree 2. Agree 3. Disagree 4. Strongly Disagree 5. Not sure 68

69 Census vs. sampling Ideally care should be measured in all stroke admissions (= census), but this is impractical at large hospitals If case sampling is done it must be unbiased Obtain a true random sample Systematic sample (first 10 of the month?) More easily implemented using retrospective approach 69

70 Quality Assurance - Staffing & Training Complexity of eligibility criteria and data definitions requires that data collectors have sufficient knowledge and training Experience with stroke care very helpful Motivation is essential Need for several data collectors at each site Centralized training is essential especially at the onset Need for regular communication, frequent reminders, positive encouragement and 70 motivation, and periodic re-training

71 Principles of Quality Control Routine monitoring of submitted data with opportunity for data correction/clarification Check for completeness and out-of range Regular visual screening of summary data reports Site visits from coordinating center are essential especially early on Consider periodic audits of a sample of charts Ref: Arts DGT et al, JAMIA, 2002;9:

72 Perfect is the enemy of good Anon 72

73 Perfect is the enemy of good. but so is sloppy Reeves 73

74 FINAL QUIZ QUESTIONS!! 74

75 Q8. The ICD-9 codes for stroke (430 to 436) are all about equal in their ability to identify acute stroke cases 1. True 2. False 3. Not sure 75

76 Q9. If it is shown that two abstractors agree on the specific value of a clinical measure (e.g., BP, age, prior medications) then this proves Accuracy 2. Reliability 3. Both 76

77 Q10. If 2 people each tossed a coin how often would they agree (heads or tails) by chance alone? 1. 10% of the time 2. 25% of the time 3. 50% of the time 4. 75% of the time 5. Absolutely no idea 77

78 Answer: 50% of the time H, H = 0.25 H, T = 0.25 T, H = 0.25 T, T =

79 1. Age Q 11. Which of the following stroke variables have been shown consistently to be unreliable? 2. Gender 3. Past Medical history 4. Stroke team consult 5. DVT prophylaxis 79

80 Questions? Mathew Reeves Department of Epidemiology, MSU 80