A comparison of iatrogenic injury studies in Australia and the USA I: context, methods, casemix, population, patient and hospital characteristics

Transcription

1 International Journal for Quality in Health Care 2000; Volume 12, Number 5: pp A comparison of iatrogenic injury studies in Australia and the USA I: context, methods, casemix, population, patient and hospital characteristics ERIC J. THOMAS 1, DAVID M. STUDDERT 2, WILLIAM B. RUNCIMAN 3,4, ROBERT K. WEBB 3,4, ELIZABETH J. SEXTON 4, ROSS McL WILSON 5, ROBERT W. GIBBERD 6, BERNADETTE T. HARRISON 5 AND TROYEN A. BRENNAN 1,2 1 Division of General Medicine, Department of Medicine, Brigham and Women s Hospital and Harvard Medical School, and 2 Department of Health Care Policy and Management, Harvard School of Public Health, Boston, MA, USA, 3 Department of Anaesthesia and Intensive Care, University of Adelaide and Royal Adelaide Hospital, 4 Australian Patient Safety Foundation, Adelaide, SA, 5 Royal North Shore Hospital, North Sydney, NSW, and 6 University of Newcastle, Callaghan, NSW, Australia Abstract Objective. To better understand the differences between two iatrogenic injury studies of hospitalized patients in 1992 which used ostensibly similar methods and similar sample sizes, but had quite different findings. The Quality in Australian Health Care Study (QAHCS) reported that 16.6% of admissions were associated with adverse events (AE), whereas the Utah, Colorado Study (UTCOS) reported a rate of 2.9%. Setting. Hospitalized patients in Australia and the USA. Design. Investigators from both studies compared methods and characteristics and identified differences. QAHCS data were then analysed using UTCOS methods. Main outcome measures. Differences between the studies and the comparative AE rates when these had been accounted for. Results. Both studies used a two-stage chart review process (screening nurse review followed by confirmatory physician review) to detect AEs; five important methodological differences were found: (i) QAHCS nurse reviewers referred records that documented any link to a previous admission, whereas UTCOS imposed age-related time constraints; (ii) QAHCS used a lower confidence threshold for defining medical causation; (iii) QAHCS used two physician reviewers, whereas UTCOS used one; (iv) QAHCS counted all AEs associated with an index admission whereas UTCOS counted only those determining the annual incidence; and (v) QAHCS included some types of events not included in UTCOS. When the QAHCS data were analysed using UTCOS methods, the comparative rates became 10.6% and 3.2%, respectively. Conclusions. Five methodological differences accounted for some of the discrepancy between the two studies. Two explanations for the remaining three-fold disparity are that quality of care was worse in Australia and that medical record content and/or reviewer behaviour was different. Keywords: adverse events, complications, iatrogenic injury, medical record review, quality of care A troubling aspect of medical care is its capacity to cause an important marker for quality of care, and a number of not simply alleviate disability and disease. Wherever medical recent studies have attempted to quantify these rates and care is delivered, patients risk suffering injury as an unintended catalogue the kinds of injuries that occur [1 7]. consequence of that care. Iatrogenic injury rates have become Two of these studies [6,7], both based on the Harvard Address reprint requests to E. J. Thomas, University of Texas Houston Medical School, 6431 Fannin MSB 1.122, Houston, TX USA. eric.thomas@uth.tmc.edu 2000 International Society for Quality in Health Care and Oxford University Press 371

2 E. J. Thomas et al. Medical Practice Study [1,2], provide an opportunity to the mean for all reviewers. If [ 10% of the re-reviewed develop an international focus for understanding medical records were classified as AEs by the investigator, their reviews injury. The Quality in Australian Health Care Study (QAHCS) were substituted with re-reviews by a different reviewer who [6] reviewed admissions from two Australian states was masked to the purpose of the re-review. and found that adverse events (AE) were associated with There were three stages to the methods of the study 16.6% of hospital admissions in 1992 [6]; the Utah and reported here. Colorado Medical Practice Study (UTCOS) [7] reviewed Stage 1. Investigators from each country reviewed and discharges from the same year, and found rates of discussed each stage of their counterparts research process. 2.9% (3.2%, unweighted) in both of these states. UTCOS After this discussion, several methodological differences beresults were consistent with the Harvard Medical Practice tween the two studies were apparent. One-hundred randomly Study which estimated the AE rate in New York in 1984 to selected QAHCS events were jointly reviewed by an inbe 3.7%. All three studies defined an AE as unintended injury vestigator from each study to identify any additional methodor harm to a patient, caused by health care management ological differences that may have led to the higher QAHCS rather than a disease process, which led to hospitalization, AE rate. prolongation of hospital stay, morbidity at discharge or death. Stage 2. The Australian data were then treated as they would The disparity between AE rates in Australia and the USA have been had the study been conducted in the USA; all led to widespread commentary and debate, both in the QAHCS clinical summaries (not original medical records) medical community [8 13] and the popular press [14 17]. were reviewed by an UTCOS investigator and those that Questions were raised about the quality of health care in would not have met the UTCOS criteria for an AE were Australia. However, despite their shared methodological her- eliminated. We chose to use the UTCOS as the standard itage, there were doubts about whether the studies findings simply because it was more closely related to the Harvard were truly comparable. Medical Practice Study, the study upon which QAHCS and We therefore attempted to derive a valid basis for com- UTCOS were based. parison of AEs in QAHCS and UTCOS by identifying and Stage 3. Selected casemix, population, patient and hospital correcting for differences in methods. We also examined characteristics were obtained for the environment of each differences in context, casemix, population, patient and hos- study and the distribution of all variables found to be pital characteristics. univariate correlates of AEs in either study were tabulated International benchmarking of this kind would allow for comparison (see Tables 1 and 2). injury reduction efforts to be focused on particular areas of UTCOS had classified each 1992 US-Diagnosis Related concern, and would inform investigators in other countries Group (US-DRG, n=494) into one of four levels, according who are planning similar studies. Also, confirmation of the to the a priori clinical likelihood that it would be least (level QAHCS findings ( preventable iatrogenic injuries in 1) or most (level 4) likely to be associated with an AE. This 1992, of which were associated with death) is im- case-mix measure had shown strong predictive value for portant, as there is an emerging literature on how to prevent occurrence of AEs in univariate and multivariate analyses AEs [18 21]. conducted in UTCOS and in New York [1]. For the purposes of this study, QAHCS investigators classified Australian DRGs (AN-DRGs) from 1992 in the same way. DRG Methods information was available on all admissions in both studies and the AN-DRGs were converted to US-DRGs. Hence, The methods used in QAHCS and UTCOS were based on each sampled admission in QAHCS and UTCOS could be those developed for the Harvard Medical Practice Study [1, assigned a comparable DRG level. 2], which itself was based on the California Medical Insurance In addition, we compared the distribution of all variables Feasibility Study [22]. Trained nurses reviewed all records found to be univariate predictors of AEs in either study. The looking for screening criteria known to be associated with patient characteristics that met this requirement were age, an AE. Records screened positive were then reviewed by payer status, and race; the hospital characteristics were teach- physicians who used a data collection instrument with quesor ing status (major, minor, and non-teaching), location (urban tions designed to facilitate the reliable detection of AEs and rural), and size [small (< 8000 discharges per annum) their preventability or negligence. Physician reviewers then or large (> 8000 discharges per annum)]. Certain patient graded their confidence that an AE occurred using a 1 6 characteristics were collapsed to create comparable groups: confidence scale. (i) public insurance included Medicare from each country in Additional details of QAHCS [6] and UTCOS [7] are addition to Medicaid and free care programmes in the USA; reported elsewhere. Both studies had similar reliability for and (ii) non-white race (less than 5% of each study) included the detection of AEs by physician reviewers (κ =0.55 for Torres Strait Islanders and Aborigines from Australia, and QAHCS and κ =0.4 for UTCOS). As UTCOS reported a African American, Latinos, and Native Americans from the lower AE rate, it is also important to note that the UTCOS USA. We compared distributions of these characteristics investigators performed a quality control study. A study across the QAHCS and UTCOS study samples in order to investigator re-reviewed 50 randomly selected records of explore whether differences in case-mix may have contributed physician reviewers whose AE detection rate was 2 SD below to the disparity in injury rates observed. 372

3 Iatrogenic injury studies I: methods Table 1 Population characteristics, USA and Australia (1992 unless noted otherwise)... USA CO UT Australia NSW SA Population (in thousands) Median age (years) Female (%) Age (%) 0 24 years years years years and older Infant mortality (per 1000) live births Life expectancy at birth 2 Males Females data data. Results was 19.5% and in QAHCS was 43.7%, with the difference in rates not fully accounted for by the differing definitions Stage 1: methodological differences of criterion 1. The contexts of the studies Medical review UTCOS was undertaken in Colorado and Utah in order to Causation by health care management. Reviewers in both examine the feasibility of a no-fault insurance scheme. studies were asked to score the likelihood of causation by QAHCS was commissioned by the Australian Commonwealth medical management on a 1 6 scale (1=little or no evidence Government to determine the burden on Australian society of management causation; 2=slight to modest evidence; 3= each year of iatrogenic injury by estimating the prevalence not likely (less than 50/50 but close call); 4=more likely of AEs. In line with these objectives, USA reviewers made than not (> 50/50, but close call); 5=moderate to strong negligence judgements and estimated the annual incidence evidence; 6=virtually certain evidence). However, the threshand costs of AEs, whereas Australian reviewers judged pre- old to define an AE in QAHCS was an average score of 2 ventability and estimated the percentage of index admissions or greater for the two reviewers, whereas for UTCOS a score associated with an AE [6,7]. QAHCS and UTCOS used of 4 or greater was required, as in the previous studies. virtually identical instruction manuals, definitions and review Number of medical reviewers. In UTCOS, each medical forms. record forwarded by nurse reviewers was reviewed by one medical reviewer (a generalist ). UTCOS investigators sub- Selection of hospitals sequently checked to ensure that each met the criteria for an In UTCOS, hospitals were selected, and participated vol- AE, and those that did not were rejected. However, in untarily, because of their interest in taking part in a no-fault QAHCS, each forwarded medical record was reviewed by medical malpractice demonstration project. None of the two medical reviewers (senior specialists), and if there was invited hospitals refused to participate. In QAHCS, a stratified disagreement with respect to the presence of an AE, causation random sample of hospitals was studied, with each hospital s scores, or preventability scores, a third reviewer decided, in likelihood of selection being proportional to its number of consultation with the primary reviewers, what information inpatient separations in would go into the database. Timing of the AE in relation to its discovery and the Nurse review index admission. The index admission was that randomly In both studies, trained nurses reviewed randomly selected selected for review, although medical records prior to and medical records, and forwarded for medical review any record after that admission were also examined in each study. To which was positive for one or more of 18 criteria. However, estimate the annual incidence rate, UTCOS counted only criterion 1 (re-admission linked to a previous admission) was those AEs that caused the index admission or occurred interpreted differently in the two studies. In UTCOS it was during the admission, and were discovered during the index considered positive if the patient was re-admitted within 6 admission. QAHCS counted all index admissions associated months (if > 65 years old) or within 12 months (if < 65 years with an AE which occurred and were discovered at any time old). In QAHCS records were forwarded if there was a re- before or during the index admission, as well as those in admission regardless of when the AE linked to a previous which the AEs occurred during the index admission but were admission had taken place. The nurse referral rate in UTCOS discovered later. 373

4 E. J. Thomas et al. Table 2 Hospital and patient characteristics [n (%)] across QAHCS and UTCOS study samples and AEs Standardized Australia USA Australia Rate Australian discharges discharges AE Rate USA AE Rate ratio... 1 rate Study samples (100) (100) DRG categories (20) 3893 (26) (45) 4416 (30) (27) 4672 (32) (9) 1719 (12) Teaching status Major teaching 4996 (35) 1779 (12) Minor teaching 1547 (11) 5819 (40) Non-teaching/private 7636 (54) 7102 (48) Location Rural 2001 (14) 2166 (15) Urban (86) (85) Size Small (< (61) 3428 (23) discharges per annum) Large (> (39) (77) discharges per annum) Age (years) (14) 2902 (20) (20) 2718 (18) (18) 2859 (19) (20) 2546 (17) [ (28) 3673 (25) Missing data 2 (0) Payer category Private 4076 (29) 7703 (52) Public (60) 5990 (41) Uninsured 891 (6) Unknown 1563 (11) 116 (1) Race White (94) (74) Non-white (1) 3197 (22) Unknown 699 (5) 596 (4) All Australian AEs are significantly different from their USA counterparts at the P<0.05 level. 2 Public insurance included Medicare from each country in addition to Medicaid and other publicly funded programmes in the USA. 3 In the Australian study this category refers only to Aboriginal and Torres Straits Islanders. in the USA study it includes African Americans, Latinos and Native Americans. Stage 2: adjustment of QAHCS rate were eliminated by QAHCS investigators leaving 2353 h AEs originally reported in QAHCS. The results of correcting the Australian data to reflect what The subsequent six steps in the figure illustrate the it would have looked like had it been treated in the same effects of applying the UTCOS methods and definitions way as the USA data are presented in Figure 1, together with to the QAHCS AEs. Of the 2353 h AEs originally reported a description of the process. Alphabetical superscripts are in QAHCS 6, 973 would have been eliminated by the provided for cross-referencing between figure and text. The process outlined in Figure 1: 211 (of 430 i ) because of the first four steps show that QAHCS randomly sampled a higher causation threshold used by UTCOS; 172 (of 430 i ) hospitalizations and that 2822 f AEs were found by at least plus 32 v because of the narrower inclusion criteria with one of the two primary physician reviewers. Of these, 469 g respect to timing and/or discovery of the AE in relation 374

5 Iatrogenic injury studies I: methods Figure 1 The process and effects of treating QAHCS data using UTCOS methods. 375

6 E. J. Thomas et al. to the index admission used by UTCOS to calculate the causation threshold; 30% because of the different medical annual incidence rate; 79 (of 101 k ) because of different review process; 15% because of counting more than just the time constraints on criterion 1 at nurse review; 411 n (of events that determined the annual incidence rate; and 5% 587 m ) because UTCOS used one physician reviewer instead because of the inclusion of more AEs of certain types (see of two; 68 (48 r of 48 r, plus 3 of 109 s, plus 14 t of 14 t ) legend, Figure 1). because UTCOS excluded certain types of events from In any event, when the methodologies of the two studies their definition of AEs. were aligned, the QAHCS AE rate remained about three times greater than the USA rate (10.6 versus 3.2%, or 16.6% Stage 3 versus 5.4%, a ratio of 3.3:1). This ratio is fairly consistent amongst the casemix and patient characteristics, with the There were no substantial differences in the demographics exception of DRG category 1 and for children aged 0 14 (sex, age, infant mortality, and life expectancy) of the popuyears (Table 2). Standardization of the QAHCS rate for the lations among the USA (including Colorado and Utah) and distribution of each UTCOS patient and hospital characteristic Australia (including New South Wales and South Australia) caused insignificant changes in the QAHCS rate (Table 2). that could have contributed to this difference in AE rates The differences between the studies identified above are (Table 1). Selected patient and hospital characteristics are consistent with the contrasting goals of each study. QAHCS tabulated in Table 2 and reveal no differences that could sought to measure the impact of medical injury on the health have accounted for the different AE rates. This is illustrated care system by estimating the prevalence of patient injury by the small effect of standardizing the QAHCS AE rate for and total impact of AEs on admissions and costs. It also aimed each of the UTCOS patient and hospital characteristics. to generate information to support quality improvements Although more non-white patients appear to have been through the prevention of AEs. On the other hand, the included in the UTCOS sample, race was not really comprimary goal of UTCOS was to estimate the annual cost of parable because QAHCS included only Aboriginal and Torres no-fault medical malpractice compensation plans in Utah and Strait Islanders in this category. Colorado [23]. This necessitated the measurement of the annual incidence of AEs, taking particular note of AEs due to negligence. Theoretically, no-fault compensation schemes Discussion may also improve quality by facilitating increased reporting of AEs, although the development of quality improvement Our investigation compared two large studies conducted in strategies was not an explicit goal of UTCOS. These different Australia and the USA, both of which ostensibly used the goals could have biased the study results in a way that same medical record review method to detect AEs. We found increased the difference in AE rates. Because QAHCS was important differences in how the studies were conducted and designed to support quality improvement, there may have how the data were analysed which account for some, but not been an incentive to detect as many events as possible. In all, of the discrepancy between the Australian and USA AE contrast, the UTCOS reviewers may have been motivated to rates. detect fewer events because that would lead to more af- When the Australian data were treated in the same way as fordable estimates of no-fault medical malpractice insurance. the USA data, only 1499 (10.6%) of the QAHCS It is important to note that the vast majority of the admissions reviewed would have been counted as AEs using originally reported QAHCS AEs eliminated by the process UTCOS methods. This compares to the 2353 (16.6%) pub- outlined in Figure 1 satisfied that study s criteria for an AE. lished previously [1], and comprises the set to be compared The five differences outlined above reduced the disparity with the 475 UTCOS events (3.2% of admissions between UTCOS from greater than five-fold to three-fold. reviewed). There are a number of potential causes for the remaining This was due primarily to five methodological differences three-fold disparity in the rates between the two studies. between the two studies: QAHCS nurses referred records First, UTCOS used generalist reviewers whereas QAHCS for which there were any linked previous admissions, whereas used specialist reviewers. Although this could have biased in UTCOS there were age-related time constraints (Figure results in either direction, it seems likely that any such effect 1 k ); QAHCS used a lower confidence threshold for defining would have been small, as the tasks were fairly straightforward. medical causation for an AE (Figure 1 i ); QAHCS used two For example, generalists certainly may have missed errors physician reviewers compared to one for UTCOS (Figure related to a discipline of medicine with which they were 1 m,n,o ); QAHCS counted all AEs associated with the index unfamiliar. However, specialists may have also not detected admission instead of the smaller group that determined the AEs related to care provided by generalists, or to care annual incidence (Figure 1 i,v ); and QAHCS included some provided by other types of specialists. types of AEs which were not included in UTCOS (Figure Second, there may be different patterns of diseases or co- 1 q,r,s,t ). morbidities in the two countries (e.g. more osteoporosis, Hypothetically, had UTCOS reviewers used QAHCS hypertension or obesity), and medical practice may differ as methods, the 3.2% AE rate would have been increased by to how these are managed. However, any such differences 70% to 5.4% by: 6% because of more referrals due to a would be expected to manifest as differences in casemix. As positive nurse review criterion 1; 15% because of a lower discussed already, category 1 was associated with almost four 376

7 Iatrogenic injury studies I: methods times as many AEs as expected in QAHCS (QAHCS:UTCOS References ratio 11.5:1 versus 3.3:1), and there were also more AEs in the age group 0 14 years in QAHCS than UTCOS (ratio 1. Brennan TA, Leape LL, Laird NM et al. Incident of adverse 6.2:1). Thus, two groups which one might have expected to be events and negligent care in hospitalized patients. Results of least likely to be associated with an AE, had disproportionately Harvard Medical Practice Study I. N Engl J Med 1991; 324: more in QAHCS. This is the opposite pattern to that which might have been expected had casemix been a contributing 2. Leape LL, Brennan TA, Laird N et al. The nature of adverse factor to the higher QAHCS rate. events in hospitalized patients. Results of the Harvard Medical Third, there may be differences in what is recorded in the Practice Study II. N Engl J Med 1991; 324: medical record and how it is recorded. For example there may be less documentation of AEs in the USA due to the 3. Bates DW, Cullen DJ, Laird N et al. Incidence of adverse drug greater likelihood of malpractice lawsuits in the USA than in events and potential adverse drug events. Implications for prevention. ADE Study Group. J Am Med Assoc 1995; 274: Australia. We are not aware of any way to quantify, and thereby adjust for, underreporting of negligence in medical records. It is also possible that more events may have been 4. Andrews LB, Stocking C, Krizek T et al. An alternative strategy described in the USA in a manner in which it was not obvious for studying adverse events in medical care. Lancet 1997; 349: that they were caused by medical management rather than a disease process. In QAHCS the less complete the medical 5. Classen DC, Pestonik SL, Evans SR et al. Adverse drug events record the less frequently AEs were detected; 18.4% of in hospitalized patients. J Am Med Assoc 1997; 277: admissions with complete records were associated with an 6. Wilson RM, Runciman WB, Gibberd RW et al. The Quality in AE, whereas only 11.5% were when four or more essential Australian Health Care Study. MedJAust1995; 163: elements were missing from the record. Fourth, it is possible that the behaviour of reviewers might 7. Thomas EJ, Studdert DM, Burstin HR et al. Incidence and have differed between the two countries. AE judgements by types of adverse events and negligent care in Utah and Colorado. physician reviewers in UTCOS led to a judgement about Med Care 2000; 38: negligence; this may have led to a lower likelihood of judging 8. McNeil JJ, Leeder SR. How safe are Australian hospitals? Med that an AE had occurred in UTCOS. In contrast, the QAHCS J Aust 1995; 163: physician reviewers rated preventability a less controversial judgement, which might have led to the inclusion of more 9. Van Der Weyden MB. Politics and publishing: the Quality in Australian Health Care Study. MedJAust1995; 163: AEs with minor or transient outcomes. Also, simply using different reviewers may have had an effect; significant inter- 10. Baulderstone P. The politics of sick hospitals. Austral Med 1995; reviewer variation in adverse detection rates was found among 7: physician reviewers in the Harvard Medical Practice Study 11. Cordner SM. Australia s preventable hospital deaths. Lancet 1995; [24]. UTCOS reviewers may have missed AEs that QAHCS 345: reviewers would have found when reviewing the same records. Fifth, the standard of health care in Australia may be 12. Blomberg C. The Professional Indemnity Review: what did it lower, and that, across the board, many more AEs are caused accomplish? Med J Austral 1996; 164: by health care management in Australia than in the USA. 13. Tjiong RT. The professional indemnity review. A lost op- In this study we were limited by the use of case summaries portunity for reform. Med J Austral 1996; 164: of QAHCS AEs instead of the original medical records. Our 14. Reuter News Service. Australia says many die from hospital study is also limited by the use of only one investigator to mistakes. June 1, review the QAHCS case summaries. These limitations may affect the absolute numbers of events reported for the five 15. Boreham G, Painter J, Dow S. Hospital mistakes kill a methodological differences in Figure 1, but their relative year a report. The Age, June 2, importance is probably correct. 16. Editorial. Hospitals and health. The Age, June 5, Given the importance of patient injury internationally, a further study with a cross-over design is strongly recommended, as this is the only way to benchmark patient 18. Leape LL. A systems approach to medical error. J Eval Clin 17. Hogarth M. Killed in error. Sydney Morning Herald, May 11, injury using medical record review. Meanwhile, to gain further Pract 1997; 3: insight into the causes of the remaining three-fold disparity 19. Evans RS, Pestotnik SL, Classen DC et al. A computer-assisted between the studies, a qualitative comparison of the AEs in management program for antibiotics and other anti-infective UTCOS and QAHCS was carried out, and is described in a agents. N Engl J Med 1998; 338: companion paper [25]. 20. Bates DW, Leape LL, Cullen DJ et al. Effect of a computerised physician order entry and a team intervention on prevention of Acknowledgements serious medication errors. J Am Med Assoc 1998; 280: Rigby K, Clark RB, Runciman WB. Adverse events in health Funding was provided by the Australian Commonwealth care: Setting priorities based on economic evaluation. J Qual Government. Clin Practice 1999; 19:

8 E. J. Thomas et al. 22. Mills DH, Boyden JS, Rubamen DS. (eds) Report on the Medical in a retrospective chart review. Ann Intern Med 1996; 125: Insurance Study. San Francisco: Sutter Publications, Runciman WB, Webb RK, Helps SC et al.. A comparison of 23. Studdert DM, Thomas EJ, Zbar BIW et al. Can the United iatrogenic injury studies in Australia and the USA II: reviewer States afford a no-fault system of compensation for medical behaviour and quality of care. Int J Qual Health Care 2000; 12: injury? Law Contemporary Prob 1998; 60: Localio AR, Weaver SL, Landis JR et al. Identifying adverse events caused by medical care: degree of physician involvement Accepted for publication 4 August