Data Impact Challenge

Transcription

1

2

3

4 Data Impact Challenge Andre Mattman, MD FRCPC 1, John Andruschak 2, and Daniel T. Holmes, MD FRCPC 1 1 Department of Pathology and Laboratory Medicine, St. Paul s Hospital, Vancouver, BC, V6Z 1Y6, Canada 2 Lower Mainland Laboratory, Burrard Street Vancouver, BC V6X 2H3 Canada July 13, 2015 Contents 1 Question Interpretation of the Question Methods Data Custodian and Organization Data Extraction Inclusion and Exclusion Criteria Definitions Results 5 amattman@providencehealth.bc.ca jandruschak@phsa.ca dtholmes@mail.ubc.ca 1

5 4 Discussion Findings Analytical Scoring of Test Utilization: Challenging Study Weaknesses Study Strengths Possible Refinements Conclusions 10 6 References 10 1 Question In this report, we address the following Choosing Wisely Canada question: How frequently in the inpatient setting, is repeat CBC and chemistry testing conducted? 1.1 Interpretation of the Question In the acute inpatient setting, repeat testing is often necessarily frequent due to the significant metabolic abnormalities of the patients. In fact, for septic shock, diabetic ketoacidosis, and cardiac care, infrequent testing would be reckless. For this reason, we know and expect that intensive care unit (ICU) testing occurs at least once each shift (i.e. 3 daily). However, information like this this is uninformative when considering testing stewardship. We fear that the strict answer to this Choosing Wisely question, as stated, would simply be a numerical or graphical description of the distribution of time differences between testing events. However we don t feel that this addresses the spirit of the question. We understand the motivation of the question to be to answer: How often is repeat testing in inpatient settings of low clinical utility? It is this latter question we intend to answer as this is what we believe is the intended spirit. This question also allows the definition of a numerator and denominator as stipulated in the descriptions. 2

6 2 Methods 2.1 Data Custodian and Organization The data custodian of this project is the Lower Mainland Laboratory (LML) as represented by author Andruschak. LML is an organization of 14 diagnostic facilities geographically located ins Sechelt, Whistler, Squamish and Greater Vancouver, British Columbia. Among the LML labs are three quaternary care medical facilities: Vancouver General Hospital, St, Paul s Hospital, and BC Children s and Womens Hospital as well as a number of community hospitals with full services of Emergency Medicine, Internal Medicine, Surgery, Obstetrics and Intensive Care. 2.2 Data Extraction The LML clinical laboratory data are stored in Sunquest Laboratory Information System and duplicated in a data repository known as Sunset. Data was extracted from Sunset and anonymized with respect of subject, hospital ID and ordering physician. The extract start date was July 01, 2014 at 00:00:00 and the extract completion data was June 30, 2015 at 23:59:59. Representative tests from the classes of electrolytes and metabolism, renal chemistries, liver chemistries, and complete blood count (CBC) results were deleted as surrogates to explore patterns of ordering practice. Plasma sodium (Na), potassium (K) and fasting glucose (FG) were selected as representative electrolytes and metabolism, plasma creatinine (CRE) for renal function, alanine aminotransferase (ALT) for hepatic function, and hemoglobin (HB) and white blood cell count (WBC) for CBC. 2.3 Inclusion and Exclusion Criteria Data extraction was limited to inpatient collections which, in addition to all ward admitted patients, was taken to include Emergency Room (ER) patients at all facilities as ward-admitted patients frequently remain in the ER for a portion of their stay waiting for a ward bed to open up. 3

7 Data from facilities primarily functioning as community urgent care centres which don t technically admit patients (n=3 of 14 facilities) were excluded. 2.4 Definitions In order to estimate the utility of an ordered in the absence of clinical context, it is necessary to define simple criteria to define tests that appear to be of higher utility and test that do not appear to add new clinical information ( lower utility ). Because bioanalytical methodologies are non standard across the LML, normal results must be defined by institution and not by absolute numerical result. A test is defined as normal if it falls within the reference interval for the institution at which it was performed. Abnormal results are above or below the reference interval. For a normal result, a test ordering event is defined as higher utility if it is not preceded by a request for the same test that was likewise normal in a 30 day window preceding the order. Therefore higher utility tests either have no request for the same test in the previous 30 days or the last test within that time window was abnormal. This 30 day windowing is arbitrary. The time of order is defined by the time of order entry into the laboratory information system. Conversely, a normal test is considered of lower utility if it occurs within 30 days of a previous normal. We will refer to this as normal after normal. Abnormal results are defined as always appropriately ordered. While this definition has limitations, it is a necessary definition given the very large volume necessary testing from intensive care units, the results of which are almost always abnormal. Since classification of the test event depends on comparison to the first test in the time series, the first test itself cannot be classified. Likewise if a patient only had 1 of a given blood test in the 1 year timeframe, this result also remained unclassified. Unclassified test events were not used in calculation of fraction of lower utility test events (numerator) relative to the total number of classifiable test events (denominator). 4

8 All data analysis was performed in the R statistical programming language [1]. 3 Results There were test events in the 11 included hospitals in the 1 year time period analyzed. These data were acquired on unique patients. Absolute test volumes broken down by test were: Na, K, 9964 FG, CRE, ALT, HB, and WBC. A histogram of the number of tests performed on each patient in the study time frame is shown in figure 1. The median number of tests performed on each patient was 10 with a mean of 27.6 (interquartile range:[4-27], min=1, max=14518). 5

9 Histogram of Tests Performed per Patient Frequency Number of Tests in a Year Figure 1: "Distribution of the number of tests performed on an admitted patient" Summative results on test event utility broken down by test are shown in Table 1. Analyte Na K FGLU CRE ALT HB WBC N Normal (%) Lower Utility (%) Table 1: Absolute numbers of test (N) along with the percent of test classifying as lower utility as defined in section

10 4 Discussion 4.1 Findings Most inpatient tests are normal not surprisingly. However, the normalcy rate is highest in the electrolyte category (approx. 80%) while HB is the most frequently abnormal test among those considered. This suggests that electrolyte collections are providing the least probability of new diagnostic information, as defined here. This is borne out in the utility classification. Electrolytes are classed as lower utility test events nearly 70% of the time. ALT, WBC and CRE appear to be utilized more effectively being identified as lower utility only about 50% of the time while FGLU and HB are the least likely to be so identified. HB is, of course, the only test to appear more useful the more it is ordered since iatrogenic anemia can be induced by frequent testing. If this data were to be used to encourage more effective test utilization, electrolyte testing would seem worthy of consideration. 4.2 Analytical Scoring of Test Utilization: Challenging Any purely analytical approach to assess the utility of laboratory testing is hampered by the fact that it is very difficult to determine whether tests are ordered appropriately without knowledge of the clinical context. For this reason, any analytical tool one develops is limited to being metric of utilization and appropriateness rather than a true yard stick. It is important for data analysts to understand how nuanced the medical thought process is and how difficult it is to define inapprop. There are circumstances where an analytical approach can spot outlier behaviour. For example, if tests panels are ordered daily and are always normal through an entire admission, it is evident that they are not diagnostically useful. This however, is almost never the case since repeat testing on a normal subject will generate an abnormal result 5% of the time if reference intervals ( normal ranges ) are appropriately set. 7

11 4.3 Study Weaknesses In this case, we have essentially defined "normal after normal" to be of lower diagnostic utility, and abnormal to be of higher diagnostic utility, since the latter is by definition more clinically informative. However, this assumes that the clinician only responds to the binary status of normal vs abnormal and not to the continuous variable represented by the numerical result. While this approach might be common among medical students, as the clinicians understanding of disease becomes more sophisticated. In particular, the more experienced physician looks for all available information provided by the result: trending within the abnormal or normal range, whether the result is high normal or low normal, and the result within the context of other related tests. The approach employed in this analysis does not attempt to score physician ordering practice beyond assigning repeated normal results as less informative. In reviewing individual cases from the study, in particular a case of hypernatremia, the weakness of this approach was obvious: during the normalization of the Na, pairs of sequential results were frequently high normal, followed by an abnormal. The repeat normals would be classified by our approach as lower utility. Our approach also does not allow for acknowledging that repeat normal tests are appropriate if there is risk of catastrophic event. For example, if is known that a patient has very friable (easily broken) esophageal varices and has been admitted for gastrointestinal bleed and transfused into the normal range of Hb, it would be entirely appropriate (and a relief) to check hemoglobin repeatedly and obtain normal results. There is value in reassurance testing in a context like this. It is worth mentioning that many tests are performed in panels for which there is no financial penalty for measuring the whole panel. For example, at St. Paul s Hospital nearly all electrolytes from the emergency department are performed on the Radiometer ABL 800 Blood Gas Instrument. This instrument does all electrolytes, glucose, lactate, creatinine, Hb and full cooximetry on all samples at no additional cost than what would be incurred by a single test. For this reason, it would seem inappropriate to score normal-afternormal test events lower. 8

12 4.4 Study Strengths The approach employed is very simple both to understand and to code computationally. In this sense it is portable and if validated, could serve as a general quality metric to look for changes in clinical practice that might represent changes in testing stewardship. 4.5 Possible Refinements The most appropriate way to statically judge if a normal after normal testing event provided new clinical information, we could apply the concept of the reference change value (RCV) [2, 3] to determine whether a change in serial tests whose results lie within the normal range is large enough to be statically significant and not attributable to expected within-patient biological variation or within-instrument analytical variation. The formula for the RCV is: RCV = 2.77 CV 2 A + CV 2 I where CV A is the analytical coefficient of variation for the analyte of interest at the concentration of interest. It is important to note that CV A is a function of concentration. CV I is the within patient biological variation for the analyte and this is highly analyte dependent by virtue of human physiology. By way to example, however, we know that CV A is about 1% for sodium and that CV I is about 0.7%. This makes the RCV about 3.4% or about 4.8 mmol/l for a sodium around 140 mmol/l. We could therefore reclassify normal after normal results as higher utility if the RCV is exceeded since there is, by definition, bona fide diagnostic information contained therein. Applying this to our sodium data lowers the percent of lower utility tests modestly to 65.1% on a sample subset of 20,000 patients. An RCV approach could be easily be applied to all testing, not just normal tests, to assess the rate of availability of results that are statistically significant in their difference relative to the preceding test. However, this approach requires CV A data for all analytes, which is a function of the analyte concentration. Knowledge of test performance by site would be necessary for this approach. Use of this approach is out of scope at this time. 9

13 5 Conclusions We present a simple metric for classifying testing as lower or higher utility on a large mutlihospital inpatient cohort. 6 References References [1] The r project for statistical computing. [2] Callum G Fraser. Improved monitoring of differences in serial laboratory results. Clinical chemistry, 57(12): , [3] Callum G Fraser. Reference change values. Clinical Chemistry and Laboratory Medicine, 50(5): ,