Informatics and the American College of Surgeons National Surgical Quality Improvement Program: Automated Processes Could Replace Manual Record Review

Transcription

1 Informatics and the American College of Surgeons National Surgical Quality Improvement Program: Automated Processes Could Replace Manual Record Review David A Hanauer, MD, MS, Michael J Englesbe, MD, John A Cowan Jr, MD, Darrell A Campbell, MD, FACS BACKGROUND: STUDY DESIGN: RESULTS: CONCLUSIONS: The American College of Surgeons National Surgical Quality Improvement Program (ACS- NSQIP) provides reliable, risk-adjusted outcomes data using standardized definitions and end points. Collection of the data is time consuming, and the surgical clinical nurse reviewers (SCNRs) can sample only a subset of all surgical cases. We sought to test the feasibility of using an informatics tool to automatically identify postoperative complications stored as free-text documents in our electronic medical record. We used a locally developed electronic medical record search engine (EMERSE) to build sets of terminology that could accurately identify postoperative complications of both myocardial infarction (MI) and pulmonary embolism (PE) as defined by the ACS-NSQIP. All complications had been previously identified by our SCNRs and these were considered the gold standard. We used 5,894 cases from 2001 to 2004 from our institution s ACS-NSQIP dataset for building the terminology and 4,898 cases from 2005 to 2006 for validation. False-positive cases were then further reviewed manually. We achieved sensitivities of 100.0% and 92.8% for identifying postoperative myocardial infarction and pulmonary embolism, respectively, with somewhat lower specificities of 93.0% and 95.9%, respectively. These results compared favorably with results from the SCNRs, especially because our manual review uncovered cases previously missed. Informatics has the potential to improve the efficiency and accuracy of chart abstraction by SCNRs for the ACS-NSQIP. Using such tools may eventually allow all cases at an institution to be reviewed rather than a small subset. (J Am Coll Surg 2009;208: by the American College of Surgeons) Disclosure Information: The following disclosure has been reported by the authors: Dr Cowan is a consultant and receives salary and stock options from QCMetrix, Inc (which provides information technology and data support for the ACS-NSQIP). Received August 17, 2008; Accepted August 26, From the Departments of Pediatrics (Hanauer) and Surgery (Englesbe, Cowan, Campbell), the Center for Computational Medicine and Biology (Hanauer), and the Comprehensive Cancer Center (Hanauer), University of Michigan Medical School, Ann Arbor, MI. Correspondence address: David A Hanauer, MD, 5312 CC, SPC 5940, 1500 E Medical Center Dr, Ann Arbor, MI The National Surgical Quality Improvement Program (NSQIP) was initiated in the Department of Veterans Affairs (VA) Healthcare System 1 and was subsequently adopted by the American College of Surgeons (ACS) for use in private sector hospitals. 2 A defining feature of the ACS-NSQIP is the quality and reliability of the data it provides. Risk-adjusted outcomes are produced using standardized definitions and end points, and a meticulous data collection process is carried out by trained nurse reviewers, referred to as surgical clinical nurse reviewers (SCNRs). 3 Using the ACS-NSQIP system, various large studies have demonstrated a reduction in surgical morbidity and mortality. 4-6 Programs such as the ACS-NSQIP are becoming increasingly important, especially within the context of the growing focus on quality measurement and pay-forperformance initiatives. 7,8 The data collection process is time consuming and may be limiting the full potential of the ACS-NSQIP. For example, the labor-intensive data collection efforts require that a sampling methodology is used rather than examining every case. This limits cases studied to approximately 1,500 per institution per year. In addition, the ACS-NSQIP infrastructure has been slow to grow outside of general and vascular surgery. Finally, it is expensive to collect data using 2008 by the American College of Surgeons ISSN /09/$36.00 Published by Elsevier Inc. 37 doi: /j.jamcollsurg

2 38 Hanauer et al Informatics Can Improve Chart Abstraction J Am Coll Surg Abbreviations and Acronyms ACS American College of Surgeons EMERSE Electronic Medical Record Search Engine MedLEE Medical Language Extraction and Encoding MI myocardial infarction NLP natural language processing NSQIP National Surgical Quality Improvement Program PE pulmonary embolism SCNRs surgical clinical nurse reviewers VA Department of Veterans Affairs the current format, and this prevents many smaller hospitals from participating in the ACS-NSQIP. Because the majority of clinical patient notes are transcribed as electronic documents in the electronic medical record at our medical center, we sought to test the feasibility of electronically reviewing patient data for postoperative complications. Such an automated approach could reduce the burden on the clinical nurse and allow for more cases to be reviewed, with the ultimate goal of achieving a 100% review rate. METHODS After obtaining approval for this study from our institutional review board, we obtained a spreadsheet of all University of Michigan ACS-NSQIP cases from July 2001 through December This dataset contained the patient medical record number, operative date, and the coded values for the 28 postoperative complications of interest to the ACS-NSQIP. The data were split into a training set (all cases from 2001 through 2004) and a validation set (all cases in 2005 and 2006). In this study, we used an internally developed computer application known as the Electronic Medical Record Search Engine (EMERSE). 9 EMERSE was developed as an extension of an application originally created for identifying cases for our ACS tumor registry. 10 It provides an intuitive and simple interface for performing searches of the medical documents in our electronic medical record, CareWeb. Using EMERSE, bundles, or groups of search terms, can be created to perform standardized searches across patient lists. EMERSE supports the ability to search for or ignore phrases in a powerful manner and includes case-sensitive searches and wildcard matches. Document searches can be limited to specific time frames, which made using EMERSE ideal for searching only the 30-day postoperative period for each patient, the time frame for complications of interest to the ACS-NSQIP. Two postoperative complications were selected for automatic identification using our system. These were postoperative myocardial infarction (MI) and pulmonary embolism (PE). The definition for MI included the necessity for newonset Q-waves. Identification of MI was thought to be challenging, because the abbreviation can easily be confused with the state abbreviation for Michigan, which appears frequently in our clinical notes. PE was selected because it is a relatively uncommon event that might be easy to miss but should not easily be confused with other terms. Each of these complications had to occur within 30 days of the operative date for it to be counted. The postoperative complications coded by the SCNRs in the ACS-NSQIP dataset were considered the gold standards with which we would compare our methodology. We then imported the patient lists into EMERSE. Bundles of search terms were created by reading the clinical notes of patients with the relevant postoperative diagnoses in the training set and developing a list of relevant terminology. Other terms thought to be important for identifying patients were also included, even if we did not come across them. Bundles contained both words and phrases to highlight ( positive terms ) and words and phrases to ignore ( negative terms, also known as negation) during the search. Negation is very important to reduce the potential of false-positive hits. For example, when searching for the abbreviation MI, it is important to first ignore phrases such as negative for MI or Ann Arbor, MI. Through EMERSE we ran a total of 5,984 cases from the 2001 to 2004 training set using the appropriate Bundle (MI or PE) until we were able to identify all known positive cases and reduce the false-positive rate to a reasonable level by creating large lists of negative terms before conducting the search. We defined a positive case for NSQIP to be any patient whose dictated medical documents (operative notes, progress notes, discharge summaries, and so on) within a 30-day period of the operative date had at least 1 hit for the relevant terminology from the positive terms in the associated search Bundle. We included only the dictated documents and excluded other sections of our electronic medical record, including the problem summary list, pathology reports, and radiology reports. Once we were confident in the quality of the search Bundles, we ran the validation set through EMERSE to test the performance of the Bundles. The validation set included 4,898 cases from 2005 to 2006 not previously analyzed using EMERSE. All false positives, false negatives, true positives, and true negatives were recorded. Falsepositive and false-negative cases were then reviewed manually to determine the likely cause of the error to detect current weakness in, and direct future improvements to, the Bundles. We did not manually review the cases reported as negative by both the nurse reviewers and EMERSE.

3 Vol. 208, No. 1, January 2009 Hanauer et al Informatics Can Improve Chart Abstraction 39 Table 1. Comparison of the Electronic Medical Record Search Engine System with the Gold Standard Nurse Reviewer Variable True positive, n True negative, n False positive,* n False negative, n Sensitivity, % Specificity, % MI (EMERSE) 5 4, MI (nurse) 4 4, PE (EMERSE) 26 4, PE (nurse) 19 4, The final results in this table are based on manual review of all false-positive and false-negative cases. *We manually checked only the cases considered to be false positives from EMERSE. So our estimates of the false-positive rates for the nurses are based on the assumption that the nurses always correctly coded a complication. EMERSE, Electronic Medical Record Search Engine; MI, myocardial infarction; PE, pulmonary embolism. Table 2. Summary of False-Positive Errors Made in the Identification of Myocardial Infarction Cases in the Validation Set, 2005 to 2006 Total Error description Errors, n errors, % Patient had history of MI Q-waves mentioned in the notes but no postoperative MI was documented Contained text related to the state of Michigan (abbreviated MI ) MI mentioned in context of workup, rule out, negative for Non-Q-wave postoperative MI Miscellaneous MI mentioned in context of family history Total Data do not include the case of MI that was later determined to be a true positive based on medical record review. MI, myocardial infarction. RESULTS For myocardial infarction, we developed a Bundle containing 21 positive terms and 1,741 negative terms. The list of negative terms was very large, because we had to include every Michigan city and state combination such as Ann Arbor, MI so that the state abbreviation did not result in a false positive. Among the 5,984 cases reviewed in the training set there were 15 known cases of MI in the ACS- NSQIP database. Using our Bundle we found all 15 cases (100.0% sensitivity) and 108 false positives remained, resulting in a specificity of 98.2%. To determine how important the negative terms were in reducing the false-positive rate, we tested the dataset using only the positive terms. The sensitivity remained at 100.0%, but the specificity dropped to 38.3%. It was noted that the phrase Q-wave was not used in any of the documents that were used to correctly identify a case of postoperative MI, even though it was part of the ACS-NSQIP definition. We then tested the MI Bundle against the 4,898 cases in the 2005 to 2006 validation set against which EMERSE had not previously been run. This set contained four known cases of postoperative MI. Again, EMERSE correctly identified all 4 and had 343 false-positive cases. The sensitivity and specificity were 100.0% and 93.0%, respectively (Table 1). One of the false-positive cases was determined to have been a true Q-wave postoperative myocardial infarction that was missed by the SCNRs. This increased the total number of true cases in the validation set to 5 and decreased the final number of false-positive cases to 342, although there was no change in overall sensitivity or specificity. We also tested the importance of the negative terms in the validation set by rerunning the search without them. The specificity dropped to 16.1%. A summary of the falsepositive cases is shown in Table 2. Next we tested the Bundle created for PE. This Bundle contained 6 positive terms (pulmonary embolism, pulmonary embolus, pulmonary embol, pulmonic embol, pulmonary thromboembolism, PE) and 364 negative terms. Using the 2001 to 2004 dataset as the training set we were able to identify all 15 true-positive cases and had 62 false-positive cases, for final sensitivity and specificity of 100.0% and 99.0%, respectively. The Bundle was then tested against the 2005 to 2006 validation set. EMERSE correctly identified 17 of the 19 known PE cases and also identified an additional 208 false-positive PE cases, as summarized in Table 3. Table 3. Summary of False-Positive Errors Made in the Identification of Pulmonary Embolism Cases in the Validation Set, 2005 to 2006 Total Error description Errors, n errors, % PE was discussed as a possibility or a rule out was performed Patient had history of prior PE Contained text related to physical examination (abbreviated PE ) Miscellaneous Total Data do not include the nine cases of PE that were later determined to be true positives based on medical record review. PE, pulmonary embolism.

4 40 Hanauer et al Informatics Can Improve Chart Abstraction J Am Coll Surg The two true ACS-NSQIP PE cases that EMERSE missed were reviewed, and neither had any mention of PE during the 30-day postoperative interval. A PE was mentioned briefly in clinical notes 37 days and 44 days postoperatively, respectively, and in both cases, the PE was described as having occurred at an outside institution. The false-positive cases were also reviewed, and an additional nine cases of true postoperative PE were confirmed that were not included in the original gold standard ACS- NSQIP dataset. An additional postoperative PE case was confirmed, but it was still counted as a false positive because it was caused by septic emboli, so it did not meet the strict ACS-NSQIP definition of being caused by a blood clot. Using the corrected results based on our analysis of the errors, the final true-positive count increased to 26 and the false-positive count decreased to 199. The adjusted sensitivity and specificity were 92.8% and 95.9%, respectively (Table 1). DISCUSSION The results of our study demonstrated the feasibility of using automated processes to improve and extend the current manual review by SCNRs for identifying postoperative complications for the ACS-NSQIP. We designed the system to ensure a high sensitivity at the expense of a lower specificity so that we could be confident that we were not missing relevant cases. Not only did we find nearly all cases that were initially identified by the nurse, but we also found cases that had originally been missed. So even our gold standard had errors. Finding rare events that have just a brief mention in the clinical documentation is difficult for a person but easy for a computer. The existence of false positives means that the nurse will still have to review some cases. But by eliminating about 90% of the true-negative cases from review, this would allow the nurses to focus more on the cases most likely to yield results of interest. To date, we have attempted to identify only 2 of the 28 postoperative complications of interest to the ACS-NSQIP. Building the Bundles was relatively straightforward for both MI and PE. Other postoperative complications may pose more of a challenge, because their definitions are more complicated and terminology used to describe them more varied. This is especially true for surgical site infections. Use of sophisticated techniques such as natural language proccessing (NLP) may offer a more powerful approach to identifying cases of interest. A study similar to ours was conducted using a New York State adverse events database. 11 The authors used the Medical Language Extraction and Encoding (MedLEE) NLP system 12 to extract adverse events from discharge summaries. Among multiple adverse events, they looked at the terms new, acute pulmonary embolism, and perioperative/ periprocedural acute myocardial infarction. For both of these adverse events, their system had a lower rate of false positives but also missed a higher proportion of gold standard cases. Similar to our findings, the system also identified a substantial proportion of cases that had originally been missed by manual review. This is not surprising, because other studies looking at the interrater reliability of detecting adverse events through manual chart review have found much variability. 13,14 Another study compared the NLP MedLEE system with searches using a phrase-matching algorithm, a method similar to what was done using EMERSE, to identify adverse events related to central venous catheters. 15 The authors found that the phrase-matching algorithm was more sensitive but less specific than the NLP system, and this is consistent with our findings for the ACS-NSQIP data. Murff and colleagues 16 examined discharge summaries for adverse events using groups of trigger words and phrases much like the Bundles supported by EMERSE. Their word lists were relatively small and did not include negation, which may explain why they had a low sensitivity and an even lower specificity. Automated identification of negated concepts is improving and could help prevent such errors. 17 Forster and associates 18 applied a similar approach by searching for terms associated with adverse events using a commercially available search engine running on their desktop computer. Documents with a hit were then reviewed by a physician to determine whether the context of the term implied a true adverse event. They found a low sensitivity and a high specificity. The low sensitivity was attributed to language in the documents that was not included in their list of 104 search terms. A limitation of our study is that the Bundles we created may contain terms and phrases specific to our own institution, and even these are likely to change over time because of the yearly resident physician turnover. Nevertheless, the ease with which we were able to build our Bundles suggests that we could keep them up-to-date if the need arose. Another approach would be to encourage the use of standardized language, at least for complications known to be important for initiatives such as the ACS-NSQIP, because this could allow for easier extraction from the free-text notes. Development of a minimal standard terminology has occurred in the area of endoscopy, 19 but this has not yet been done for surgery. One institution has modified its electronic medical record system to better capture data specific for ACS-NSQIP. 20 No decision has yet been made about whether we should implement our system into the true workflow of the clini-

5 Vol. 208, No. 1, January 2009 Hanauer et al Informatics Can Improve Chart Abstraction 41 cal nurse reviewer. Such a change would significantly change the workflow. The current workflow is one in which the nurse reviews one patient for many complications simultaneously; our new workflow would require reviewing many patients simultaneously for a single complication. It is not clear how this might affect the efficiency of data abstraction. But we do believe that to attain the goal of reviewing 100% of cases, we will need to rely on either automated or semiautomated processes to increase efficiency. Redirecting the focus of the ACS-NSQIP away from arduous data collection and toward directed qualityimprovement initiatives requires significant advances in the efficiency of current data collection processes. Our findings suggest that even relatively simple approaches have the potential to improve efficiency and accuracy. Providing clinical nurses with tools such as ours might significantly reduce the time associated with collection of NSQIP data points. This would likely reduce costs of NSQIP participation and enable the nurse reviewer to focus on more directed quality-improvement efforts. Author Contributions Study conception and design: Hanauer, Cowan, Campbell Acquisition of data: Hanauer, Cowan, Campbell Analysis and interpretation of data: Hanauer, Englesbe, Cowan, Campbell Drafting of manuscript: Hanauer, Englesbe, Cowan, Campbell Critical revision: Hanauer, Englesbe, Cowan, Campbell Acknowledgment: We thank Suzanne Fleming, Laurel Phillips, and Jim Kubus for their help on this project. REFERENCES 1. Khuri SF, Daley J, Henderson W, et al. The Department of Veterans Affairs NSQIP: the first national, validated, outcomebased, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg 1998;228: Fink AS, Campbell DA Jr, Mentzer RM Jr, et al. The National Surgical Quality Improvement Program in non-veterans Administration hospitals: initial demonstration of feasibility. Ann Surg 2002;236: ; discussion Khuri SF. The NSQIP: a new frontier in surgery. Surgery 2005; 138: Khuri SF, Daley J, Henderson WG. The comparative assessment and improvement of quality of surgical care in the Department of Veterans Affairs. Arch Surg 2002;137: Neumayer L, Mastin M, Vanderhoof L, Hinson D. Using the Veterans Administration National Surgical Quality Improvement Program to improve patient outcomes. J Surg Res 2000; 88: Rowell KS, Turrentine FE, Hutter MM, et al. Use of national surgical quality improvement program data as a catalyst for quality improvement. J Am Coll Surg 2007;204: Jones RS, Brown C, Opelka F. Surgeon compensation: pay for performance, the American College of Surgeons National Surgical Quality Improvement Program, the Surgical Care Improvement Program, and other considerations. Surgery 2005; 138: Lindenauer PK, Remus D, Roman S, et al. Public reporting and pay for performance in hospital quality improvement. N Engl J Med 2007;356: Hanauer DA. EMERSE: the Electronic Medical Record Search Engine. AMIA Annu Symp Proc 2006; Hanauer DA, Miela G, Chinnaiyan AM, et al. The registry case finding engine: an automated tool to identify cancer cases from unstructured, free-text pathology reports and clinical notes. J Am Coll Surg 2007;205: Melton GB, Hripcsak G. Automated detection of adverse events using natural language processing of discharge summaries. J Am Med Inform Assoc 2005;12: Friedman C, Hripcsak G, Shagina L, Liu H. Representing information in patient reports using natural language processing and the extensible markup language. J Am Med Inform Assoc 1999;6: Neale G, Woloshynowych M. Retrospective case record review: a blunt instrument that needs sharpening. Qual Saf Health Care 2003;12: Thomas EJ, Lipsitz SR, Studdert DM, Brennan TA. The reliability of medical record review for estimating adverse event rates. Ann Intern Med 2002;136: Penz JF, Wilcox AB, Hurdle JF. Automated identification of adverse events related to central venous catheters. J Biomed Inform 2007;40: Murff HJ, Forster AJ, Peterson JF, et al. Electronically screening discharge summaries for adverse medical events. J Am Med Inform Assoc 2003;10: Huang Y, Lowe HJ. A novel hybrid approach to automated negation detection in clinical radiology reports. J Am Med Inform Assoc 2007;14: Forster AJ, Andrade J, van Walraven C. Validation of a discharge summary term search method to detect adverse events. J Am Med Inform Assoc 2005;12: Fujino MA, Bito S, Takei K, et al. Terminology and global standardization of endoscopic information: Minimal Standard Terminology (MST). Conf Proc IEEE Eng Med Biol Soc 2006;1: Hayden SP, Oppedisano R, Breudigam M. Leveraging electronic medical record (EMR) systems along with other health information systems (HIS) to improve data capture and reporting for a surgical quality improvement program at a tertiary care institution and integrated health system. AMIA Annu Symp Proc 2007;970.