Early Warning System Scores for Clinical Deterioration in Hospitalized Patients: A Systematic Review

Transcription

1 Early Warning System Scores for Clinical Deterioration in Hospitalized Patients: A Systematic Review M. E. Beth Smith 1,2, Joseph C. Chiovaro 1,6, Maya O Neil 2,4,7,8, Devan Kansagara 1,2,6,7, Ana R. Quiñones 3,7, Michele Freeman 7, Makalapua L. Motu apuaka 7, and Christopher G. Slatore 5,8,9 1 Department of Medicine, 2 Department of Medical Informatics and Clinical Epidemiology, 3 Department of Public Health and Preventive Medicine, 4 Department of Psychiatry, and 5 Division of Pulmonary & Critical Care Medicine, Department of Medicine, Oregon Health & Science University, Portland, Oregon; and 6 Division of Hospital and Specialty Medicine, 7 VA Evidence-based Synthesis Program, 8 Health Services Research & Development, and 9 Section of Pulmonary & Critical Care Medicine, Portland VA Medical Center, Portland, Oregon Abstract Rationale: Early warning system (EWS) scores are used by hospital care teams to recognize early signs of clinical deterioration and trigger more intensive care. Objective: To systematically review the evidence on the ability of early warning system scores to predict a patient s risk of clinical deterioration and the impact of early warning system implementation on health outcomes and resource utilization. Methods: We searched the MEDLINE, CINAHL, and Cochrane Central Register of Controlled Trials databases through May We included English-language studies of early warning system scores used with adults admitted to medical or surgical wards. We abstracted study characteristics, including population, setting, sample size, duration, and criteria used for early warning system scoring. For predictive ability, the primary outcomes were modeled for discrimination on 48-hour mortality, cardiac arrest, or pulmonary arrest. Outcomes for the impact of early warning system implementation included 30-day mortality, cardiovascular events, use of vasopressors, respiratory failure, days on ventilator, and resource utilization. We assessed study quality using a modified Quality in Prognosis Studies assessment tool where applicable. Measurements and Main Results: Of 11,183 citations studies reviewed, one controlled trial and 20 observational studies of 13 unique models met our inclusion criteria. In eight studies, researchers addressed the predictive ability of early warning system tools and found a strong predictive value for death (area under the receiver operating characteristic curve [AUROC], ) and cardiac arrest (AUROC, ) within 48 hours. In 13 studies (one controlled trial and 12 pre post observational studies), researchers addressed the impact on health outcomes and resource utilization and had mixed results. The one controlled trial was of good quality, and the researchers found no difference in mortality, transfers to the ICU, or length of hospital stay. The pre post designs of the remaining studies have significant methodological limitations, resulting in insufficient evidence to draw conclusions. Conclusions: Early warning system scores perform well for prediction of cardiac arrest and death within 48 hours, although the impact on health outcomes and resource utilization remains uncertain, owing to methodological limitations. Efforts to assess performance and effectiveness morerigorouslywillbeneededas early warning system use becomes more widespread. Keywords: clinical deterioration; early warning systems; systematic review (Received in original form March 7, 2014; accepted in final form August 12, 2014 ) Supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Quality Enhancement Research Initiative project ESP , and the Portland VA Medical Center. C.G.S. is a Core Investigator in Health Services Research and Development at the Portland VA Medical Center and was supported by a VA HSR&D Career Development Award. The Department of Veterans Affairs did not have a role in the design and conduct of the study; the collection, management, analysis, and interpretation of the data; the preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or the U.S. government. Author Contributions: M.E.B.S., J.C.C., M.O., D.K., A.R.Q., M.F., M.L.M., C.G.S.: contributed to the review of articles for inclusion, abstraction of data, and interpretation of the results. M.E.B.S., J.C.C., M.O., D.K., M.F., M.L.M., C.G.S.: contributed to preparation of the manuscript. Correspondence and requests for reprints should be addressed to M. E. Beth Smith, D.O., Portland VA Medical Center, R&D 71, 3710 SW US Veterans Hospital Road, Portland, OR smithbet@ohsu.edu The article has an online supplement, which is accessible from this issue s table of contents online at Ann Am Thorac Soc Vol 11, No 9, pp , Nov 2014 Copyright 2014 by the American Thoracic Society Originally Published in Press as DOI: /AnnalsATS OC on October 8, 2014 Internet address: AnnalsATS Volume 11 Number 9 November 2014

2 Observational studies suggest that patients often show signs of clinical deterioration up to 24 hours prior to a serious clinical event requiring intensive interventions (1). Early warning system (EWS) scores are used by hospital care teams to recognize early signs of clinical deterioration and trigger more intensive care, such as increased nursing attention, informing the care provider, or activating a rapid response team (RRT) or medical emergency team (2). The use of EWSs has increased since the Institute for Healthcare Improvement launched its 100,000 Lives Campaign in 2004, in which the use of RRTs was recommended to save lives (3). Many hospital systems have started to implement the use of EWS scores, and, although evidence shows that this leads to an increase in the use of RRTs and admissions to the intensive care unit (ICU), uncertainty remains as to their ability to predict an impending health event and whether they are useful in reducing cardiac arrest and death (1). In this study, we assessed the ability of EWS tools to predict cardiac arrest, pulmonary arrest, or death outcomes within 48 hours of data collection, as well as to determine the impact of EWS use on in-hospital health outcomes and utilization of resources. Materials and Methods We developed an analytic framework (Appendix A) with input from key informants (clinicians, nurses, hospital administration members, and patient advocates) to visually display the key questions and scope parameters of the review. We searched the MEDLINE, CINAHL,andtheCochraneCentral Register of Controlled Trials databases from their inception to May 2014 for English-language studies of EWS in adult medical and surgical ward populations (Appendix B). We obtained additional articles from systematic reviews and reference lists and by consulting experts. All citations were imported into an electronicdatabase(endnotex4; Thomson Reuters, New York, NY). Investigators reviewed titles and abstracts of citations identified through literature searches, and eligible full-text articles were independently assessed by two reviewers based on predefined eligibility criteria (Appendix C). Disagreements were resolved through group consensus. To assess the predictive ability of EWSs, we examined observational studies in which researchers reported associations between EWS scores and 48-hour mortality, cardiac arrest, or pulmonary arrest. To assess the impact of EWS implementation, we included studies in which researchers reported on 30 day mortality, cardiovascular events (cardiac arrest, acute coronary syndrome, and cardiogenic shock), use of vasopressors, number of days on a ventilator, respiratory failure, and utilization of resources, including ICU admission, use of RRT, and length of hospital stay. We used a best evidence approach to guide inclusion of studies (4). Data Abstraction, Quality Assessment, and Data Synthesis Data on population characteristics, study setting, number of subjects, number of subjects lost to full analysis, name and elements of scoring system, comparator, harms, implementation characteristics (e.g., staff training, pilot phase), and funding were abstracted from each study by one investigator and confirmed by a second investigator. For studies addressing the question of predictive ability, we abstracted data on model discrimination for outcomes of mortality, cardiac arrest, and pulmonary arrest within 48 hours. For all other studies, we abstracted patient health outcomes and resource utilization. For studies addressing predictive ability, we adapted criteria described in the Quality in Prognosis Studies assessment tool (5). In this article, we discuss the strengths and limitations of the studies and describe the overall potential for bias as applicable. For trials addressing health outcomes, we used criteria and methods defined in the methods guide developed by the Agency for Healthcare Research and Quality (6). We synthesized data qualitatively for the pre post observational studies because objective criteria are lacking to quantitatively rate or measure the quality in these types of studies. Results From among 11,183 titles and abstracts, 202 articles were selected for full-text review (Figure 1). Of these, we included 21 studies, eight providing primary data on predictive value of EWS scores and 13 pertaining to the impact of EWS interventions. We found 13 unique models ranging from four to 16 items with scores based on aggregate weighted systems (Table 1). Predictive Value of Early Warning Systems Eight observational studies (six prospective cohort and two case control) met our inclusion criteria (Table 2) (7 14). One study considered single predictors, and seven studies reported the predictive values of six distinct models of EWS scores for the outcomes of interest (death and cardiac arrest within 48 hours of measurement). No study reported on the predictive ability of EWS for respiratory arrest. Most of the models ranged from four to seven items, all of which included heart rate, respiratory rate, and blood pressure, and most included temperature and mental status. The studies were conducted in several countries (four in the United States, two in the United Kingdom, one in Canada, and one in Uganda); most were carried out in urban academic hospitals. Mortality Prediction. In general, the discriminative performance of EWS scores for predicting death within 48 hours of measurement was high (8, 9, 11, 13). In the largest study (n = 75,419), the researchers used the VitalPAC Early Warning Score (ViEWS), an abbreviated version of a previously derived EWS with vital signs recorded using VitalPAC software (8). Among medical ward patients, the abbreviated ViEWS score had an area under the receiver operator characteristic curve (AUROC) of 0.89 (95% confidence interval [CI], ). The AUROC values remained above 0.85 in subgroup analyses, including sex, year of admission, age, indication for admission, and specific diagnoses. Two other studies of the same cohort found EWS models performed similarly well for predicting death within 24 hours (ViEWS, [95% CI, ] and National Early Warning Score [NEWS], [95% CI, ]) (9, 11). In a study conducted in a resource-poor hospital in Uganda without an ICU, the researchers reported that the ViEWS had an AUROC of (95% CI, ) for death within 24 hours (13). Despite good discriminative ability, these studies also found clinically important trade-offs in sensitivity and specificity when using specific scores. Kellett and Kim found that low abbreviated ViEWS scores were associated with a very good prognosis: 0.02% of 49,077 patients with scores,3 (of Systematic Review 1455

3 Table 1. Parameters used in studies of the predictive ability and effectiveness of EWS scores for clinical deterioration in medical and surgical inpatients Study, country Parameters, name of scoring system Heart rate Respiratory rate SBP Temperature Decreased urinary output Parameters used in the scoring system O 2 sat Difficulty breathing Increase in supplemental O 2 Mental Status (LOC) Concern Other, specify Rothschild et al., 2010 (10), USA Churpek et al., 2012 (12), USA Maupin et al., 2009 (20), USA Jones et al., 2011 (19), UK Subbe et al., 2003 (26), UK Churpek et al., 2012 (7), USA O Dell et al., 2002 (23), UK De Meester et al., 2013 (17), Belgium Smith and Oakey, 2006 (25), UK Patel et al., 2011 (24), UK Kellett and Kim, 2012 (8), Canada Bailey et al., 2013 (15), USA Hammond et al., 2013 (27), Australia Mitchell et al., 2010 (21) Moon et al., 2011 (22), UK Green and Williams, 2006 (18), Australia Smith et al., 2013 (11), UK Opio et al., 2013 (13), Uganda Prytherch et al., 2010 (9), UK Albert and Huesman 2011 (16), USA Churpek et al., 2014 (14), USA Single items, not combined x x x x x x x x x DBP, seizures, uncontrolled bleeding, change in color 4-item CART x x DBP, age 5-item MEWS x x x x x Patientrack EWS x x x x x 5-item MEWS x x x x x 5-item MEWS x x x x x 5-item MEWS x x x x x 6-item MEWS x x x x x x 6-item EWS x x x x x x 6-item MEWS x x x x x x Indwelling Foley catheter 6-item VIEWS x x x x x x 7-item EWS x x x Shock index, DBP, anticoagulation use 7-item MEWS x x x x x x DBP 7-item MEWS x x x x x x x 7-item MEWS x x x x x x x 7-item clinical marker tool x x x x x x x 7-item NEWS x x x x x x x 7-item VIEWS x x x x x x x 7-item VIEWS x x x x x x x 12-item MEWS x x x x x x x x x x WBC count, new focal weakness 16-item EWS x x x x x x Time, prior ICU stay, age, DBP, BUN, potassium, anion gap, WBC, Hb, platelet count Definition of abbreviations: BUN = blood urea nitrogen; CART = Cardiac Arrest Risk Triage; DBP = diastolic blood pressure; EWS = early warning system; Hb = hemoglobin; ICU = intensive care unit; LOC = level of consciousness; MEWS = Modified Early Warning Score; O2 Sat = oxygen saturation; SBP = systolic blood pressure; ViEWS = VitalPAC Early Warning Score; WBC = white blood cell AnnalsATS Volume 11 Number 9 November 2014

4 Table 2. Observational studies of the predictive value of EWS scores Study, design, setting Population, mean age (yr), sex Comparison Number of patients Predictive measures of mortality occurring within 48 h of EWS data collection/analysis Predictive measures of cardiac arrest occurring within 48 h of EWS data collection/analysis Churpek et al., 2012 (12) Retrospective cohort study USA Academic tertiary care hospital, 500 beds Medical/surgical patients, including telemetry patients Patients without a cardiac arrest or ICU Txf Age n = 88 (cardiac arrest) 64 (cardiac arrest) 60 (ICU Txf) n = 2,820 (ICU Txf) n = 44,519 (control) 54 (control) Males 43% (cardiac arrest) 52% (ICU Txf) 43% (control) Churpek et al., 2012 (7) Medical/surgical patients Patients without cardiac arrest Total n = 47,427 NR AUROC: 0.84 (95% CI, NR) vs for MEWS Total n = 440 NR MEWS AUC (95% CI) Case control study, USA Age n = 88 (cases) 0.77 ( ) Academic tertiary care hospital, 500 beds Kellett and Kim 2012 (8) Retrospective cohort, Canada Urban academic hospital, 375 beds 64 (cases) n = 352 (controls) 58 (controls) Males 43% (cases) 49% (controls) Medical (non-icu)/surgical inpatients None Total n = 75, h mortality AUROC (95% CI) Age 63 n = 43,693 (medical) All: 0.93 ( ) 48.9% male n = 30,485 (surgical) Surgical: 0.89 ( ) Medical: 0.89 ( ) Opio et al., 2013 (13) Medical patients Patients without an event Total n = 844 Death within 24 h NR Retrospective cohort, Uganda Age 45.2 AUROC (95% CI): 0.89 ( ) 42% males Hospital, no ICU care, 330 beds Prytherch et al., 2010 (9) General medicine and emergency patients (consecutively admitted) Patients alive at 24 h following observation Total n = 35,585 patient episodes NR Death within 24 h NR Prospective cohort Age 67.7 AUROC (95% CI): study UK 0.89 ( ) Urban hospital 47.5% males Rothschild et al., 2010 (10) Medical and medicine subspecialty inpatients Control patients matched on day of admission Case control study, USA Age 61 n = 262 (cases) Urban academic medical 49.6% males n = 318 (controls) center, 745 beds Smith et al., 2013 (11) General medical patients Patients without an event Total n = 35,585 patient episodes Total n = 580 NR Among 26 patients with a cardiac arrest, 20 (76.9%, 95% CI %) did not have a preceding positive early warning sign. Death within 24 h Cardiac arrest within 24 h Prospective cohort, UK Age 67.7 AUROC: 0.89 AUROC (95% CI): Urban hospital 47.5% males 95% CI ) 0.86 ( ) Churpek et al., 2014 (14) Hospital ward patients Patients without events Total n = 59,301 NR Cardiac arrest within 24 h Age Retrospective cohort study, USA Academic tertiary care hospital, 500 beds 55 (controls) Novel EWS AUROC 64 (cardiac arrest) (95% CI): 0.88 ( ) ViEWS AUROC (95% CI): Males 0.74 ( ) 43% (controls) 41% (cardiac arrest) Definition of abbreviations: AUC = area under the curve; AUROC = area under the receiver operating characteristic curve; CI = confidence interval; EWS = early warning system; ICU = intensive care unit; ICU Txf = transfer to intensive care unit; MEWS = Modified Early Warning Score; NR = not reported; ViEWS = VitalPAC Early Warning Score. Systematic Review 1457

5 21 possible points) died within 48 hours. Although high scores were associated with a poorer prognosis (14% of 519 patients with scores.11 died within 48 hours), most patients (86%) with high scores survived (8). Prytherch and coworkers determined that the sensitivity of the ViEWS was approximately 67% at a specificity of 90% (9). Cardiac Arrest Prediction. Studies also found consistently good predictive performance of EWS scores for cardiac arrest (7, 11, 12, 14). The NEWS score had an AUROC of (95% CI, ) as a predictor of cardiac arrest within 24 hours (11). The Cardiac Arrest Risk Triage (CART) EWS, a four-item model, had relatively similar predictive ability for cardiac arrest within 48 hours (AUROC 0.84) and was superior to a more comprehensive Modified Early Warning Score (MEWS) model among patients admitted to medical and surgical units (MEWS, 0.78; P = 0.001) (12). Churpek and colleagues found similar results in a nested case control study (88 cases of cardiac arrest, 352 controls) of MEWS using data from the same cohort (AUROC, 0.77; 95% CI, ) (7). They also developed and cross-validated a novel 16-item EWS score using the electronic health records of all patients on the hospital wards over a 33- month period and found it to be superior to the ViEWS EWS score in predicting cardiac arrest within 24 hours (AUROC of 0.88 [95% CI, ] vs. AUROC of 0.74 [95% CI, ]) (14). As with studies of mortality, these studies reported high discriminative performance for predicting cardiac arrest, but potentially important trade-offs in sensitivity and specificity. At similar triggering scores, a specificity of 90% was associated with a sensitivity of 53% for CART and 48% for MEWS (12). Lowering the MEWS triggering score from 4 to 3 (of a possible 14) decreased both the specificity (87%) and the sensitivity (51%) (12). At a sensitivity of 60%, the comprehensive model developed by Churpek and coworkers had a specificity of 95%, whereas the ViEWS model had aspecificity of 85% (14). Single Vital Sign Prediction. Rothschild and colleagues conducted a case control study (n = 580) of at least one early warning criterion within 8 hours of a lifethreatening event among patients on the medical ward (10). Criteria most associated with a life-threatening event included respiratory rate.35 breaths/min (odds ratio [OR], 31.1; 95%, CI, ), need for supplemental oxygen to 100% or use of a non-rebreathing mask (OR, 13.7; 95% CI, ), and heart rate.140 beats/min (OR, 8; 95% CI, ). Multiple positive criteria were more common in cases than in controls (three positive criteria; 8.4% vs. 4.9%, P = ). However, among the 26 patients with a cardiac arrest, 20 (77%) did not have a preceding positive early warning sign within 8 hours of the event. Quality Assessment. Though the included studies on predictive ability were generally well conducted for their study design and provide important information, the body of evidence is limited by some risk of bias (Appendix D). The case control design of two of the studies (7, 10) has the potential for differential exposure assessment, as it is unclear whether vital sign measurement was different for cases compared with controls. In addition, although three studies were designed for validation of models (7, 8, 13), five were derivation studies (9 12, 14) and were not validated in separate populations of patients. Derivation studies are generally at risk for overfitting data to the population under study and may not be broadly applicable to populations of interest. Impact of Early Warning System Interventions We found one randomized controlled trial (15) and 12 pre post observational studies (16 27) that reported on the impact of EWS implementation on outcomes of 30-day mortality, cardiac arrest, and utilization of resources (Table 3). We found no studies reporting on the use of vasopressors, number of ventilator days, or respiratory failure. The EWS models ranged in scoring criteria from 5 to 12 items; all included heart rate, respiratory rate, and systolic blood pressure; and most included level of consciousness or mental status, temperature, and urinary output (Table 1). The studies were conducted in several countries (six in the United Kingdom, three in the United States, three in Australia, and one in Belgium), and most were carried out in urban academic hospitals. Mortality. Seven studies addressed the effects of EWS implementation on mortality and had mixed results (15, 17 19, 22, 24, 26). In one good-quality trial (15) and five pre post observational studies (18, 19, 22, 24, 26), no significant differences in overall mortality were observed. Only one study found a statistically significant decrease in mortality after EWS implementation (22). Moon and colleagues studied two hospitals and found that deaths per adult admission decreased from 1.4 to 1.2% (P, ) in one hospital and from 1.5 to 1.3% (P, ) in the other (22). They also found that patients who had undergone cardiopulmonary resuscitation had asignificant decrease in in-hospital mortality at the two hospitals, from 52 to 42% (P, 0.05) and from 70 to 40% (P, ), respectively (22). Notably, the hospitals underwent an expansion of critical care outreach services during this period, so the independent impact of EWS implementation is unknown. Cardiac Arrest. Mixed results were also found in the four studies evaluating the effects of EWS implementation on cardiac arrests (18, 19, 22, 26). Moon and colleagues studied the proportion of cardiac arrest calls per adult admission and found a decrease in the two hospitals involved in the study, from 0.4 to 0.2% at one site (P, ) and from 0.34 to 0.28% (P, ) at the other (22). Green and Williams found a significant decrease in the proportion of patients who had had a cardiac arrest (lack of pulse or respirations) at the time of the code blue call, from 52.1 to 35% (P = ) (18). Subbe and coworkers stratified the incidence of cardiac arrest by EWS score at time of admission and found no difference between low-risk (scores 0 2) and highrisk (scores 5 15) groups after EWS implementation, whereas the moderaterisk group (scores 3 or 4) demonstrated a significant increase in cardiac arrest (5% vs. 0%, P, 0.016) (26). No difference was found in the incidence of cardiac arrest after EWS implementation in the study by Jones and coworkers (19). Length of Hospital Stay Four studies evaluated length of hospital stay before and after EWS implementation, with inconsistent results (15, 18, 19, 21). In a good-quality trial (15) and a pre post observational study (18), no differences in length of hospital stay were detected 1 2 years after EWS implementation. A study with a shorter observation period (47 days before and 38 days after EWS implementation) found a significantly 1458 AnnalsATS Volume 11 Number 9 November 2014

6 Table 3. Studies of the impact of EWS interventions on patient outcomes and resource utilization Study, design, setting Observation period Population, mean age (yr), sex Score that triggers activation of response, description of response In-hospital outcomes: mortality (30-d), cardiovascular events Resource use: LOS, ICU Resource use: MET, RRT, critical care outreach teams, or nursing Albert and Huesman 2011 (16) 6 mo Telemetry ward n = 140 MEWS: 1 or 2; nurse alert to reassess patient in < 4h NR NR RRT calls: Retrospective cohort study, USA Age MEWS> 3 n = 78 (55.7%) Tertiary care academic hospital, 550 beds 65.8 (code) Mean MEWS for RRT mo after implementation: De Meester et al., 2013 (17) Pre post study, Belgium Tertiary care hospital, 14 medical and surgical wards Green and Williams, 2006 (18) Pre post study, Australia Tertiary care academic hospital surgical and medical wards, 323 beds Hammond et al., 2013 (27) Pre post study, Australia Tertiary care hospital 5-d period after ICU discharge 64.7 (RRT) Discuss with charge nurse and decision made on RRT activation based on stability n = 1, MEWS change score of 3 on one criterion; nurse felt patient unsafe Age 59 Increase frequency of patient observations and physician notified Patients who died without a DNAR or readmission to ICU (%), pre vs. post: 33% increase in code team calls and 50% increase in RRT calls NR Nursing compliance 53% Patient observation frequency per nursing shift increased from (95% CI ) to 1.07 (95% CI, ), P = vs. 3.5, NS Nighttime: no difference. 60% males Daytime: increase from (95% CI, ) to (95% CI, ), P = Pre: 1 yr n = 415 NR Mortality (%), pre Post: 2 yr vs. post: Demographics NR Activate ICU liaison team 33.9 vs. 34.5, NS Hospital: Pre vs. post: 19 d (pre) 16 d (post), NS Code blue calls for cardiac arrest: 52.1% vs. 35%, NS Code blue calls in patients who were still breathing and with a pulse: ICU: 47.9% vs. 64.6%, P = ICU liaison team visits: n = 153 (pre) including 111 admissions and 42 readmissions n = 412 (post) including 320 admissions and 92 readmits, NS Pre: 1 month Pre: n = 69 MEWS NR ICU readmissions (%): 13 vs. 26, NS Post: 1 month 24 h after being discharged from ICU or being admitted to ICU 630 patients (1,958 visits) vs. 1,889 patients (4,586 visits) LOS (median): ICU admissions with clinical Pre: 3.0 ( ) markers present Post: 2.6 ( ), NS,6 h: 58.8% vs. 75.4% >6 h: 41.2% vs. 24.5%, P, (P, ) Nursing, full vs. frequency: Discharged from ICU: LOS ICU: 3 vs 3, NS Within 24 h of ICU discharge: 210% increase (P = NR) Age 64 LOS (median), d Within 1 hr of ICU 58% males 13 (9 19) vs discharge: 32% 17 (10 27), NS increase, P = Unplanned admission to ICU: Age 60, 36% males Post: n = 70 (Continued ) Systematic Review 1459

7 Table 3. (Continued ) Study, design, setting Observation period Population, mean age (yr), sex Score that triggers activation of response, description of response In-hospital outcomes: mortality (30-d), cardiovascular events Resource use: LOS, ICU Resource use: MET, RRT, critical care outreach teams, or nursing Jones et al., 2011 (19) Discharged from ICU: Age 64 67% males Unplanned admission to ICU: Age 69 53% males Pre: 47 d Pre: n = 705 (7,820 Post: 38 d observations) EWS 3 5 (total 14): inform charge nurse and nurse intervention, recheck in 1 hr, if still > 3, call junior physician, if still > 3 in 1 hr, call senior physician and recheck 1 hr, if still > 3 call critical care medical team Pre vs. post: Pre vs. post: Pre vs. post: Pre post study, UK Age 70 (median) Mortality (%): Hospital (d): Clinical attendance (%) 52% males Academic hospital medical assessment unit Post: n = 776 (5,848 observations) Age 65 (median) 53% males with EWS 3 5: 29 vs. 78, P, Clinical attendance (%) with EWS. 5: 67 vs. 96.2, P = vs. 7.6, P = vs. 6.9, P, Clinical attendance (%) for EWS 3, 4, or 5 (automated alerts): Cardiac arrest (%): ICU (d): 29% vs. 78%, P, Maupin et al., 2009 (20) Pre post study, USA General hospital, 200 beds General medical and surgical wards Mitchell et al., 2010 (21) Pre post study, Australia 2 teaching hospitals Pre: 1 year n = NR EWS = 3: increased Post: 1 year VS frequency Pre: 4 mo Post: 4 mo EWS = 4: notify physician EWS = 5: call RRT (total 14) 0.4 vs. 0, P = vs. 26, P = 0.04 Clinical attendance (%) for EWS. 5: 67 vs. 95, P, EWS accuracy (%) with electronic calculation: NR NR RRT calls per 1,000 patient days increased from 7.8 to 16.4 after implementation Pre: n = 1,157 MEWS> 4 NR ICU admissions (%): 1.8 vs. 0.5 Age 58.6 >4: Contact intern RRR = % males Post: >6: Contact registrar (95% CI, ), n = 985 and intern to review P = Age 57.4 in 30 min LOS: 54.8% males Pre: 4.0 ( ) Post: 4.8 ( ), P =0.03 >8: Registrar and consultant notified, consider MET Moon et al., 2011 (22) Pre: 3 yr Pre: MEWS.1 (total 21) Pre vs. post: Pre vs. post: Hospital deaths per cardiac Pre post study, UK Post: 3 yr n = 213,117 (FH) arrest call (%), pre vs. post: Two academic tertiary care hospitals: Freeman Hospital (FH) Royal Victoria Infirmary (RVI) n = 248,260 (RVI) Post: n = 235,516 (FH) n = 281,831 (RVI) Graduated response for MEWS scores.1 Mortality of patients undergoing CPR (%): FH: 52 vs. 42, P = 0.05 % ICU admissions after undergoing in-hospital CPR: RVI: 70 vs. 40, P, NR LOS: NR FH: 26 vs. 21, P, FH: 3 vs. 2, P = Hospital deaths (%): RVI: 7 vs. 3, P, FH: 1.4 vs. 1.2, P, Total ICU admissions per year: RVI: 19.5 vs. 18.5, P = 0.12 (Continued ) 1460 AnnalsATS Volume 11 Number 9 November 2014

8 Table 3. (Continued ) Study, design, setting Observation period Population, mean age (yr), sex Score that triggers activation of response, description of response In-hospital outcomes: mortality (30-d), cardiovascular events Resource use: LOS, ICU Resource use: MET, RRT, critical care outreach teams, or nursing Demographics NR RVI: 1.5 vs. 1.3, P, Cardiac arrests (% of all admissions): FH: 857 vs (24.5% increase) FH: 0.36 vs. 0.25, RVI: 538 vs. 827 P, RVI: 0.29 vs. 0.24, (54% increase) P, Odell et al., 2002 (23) Pre: 7 mo n = NR MEWS = 3 (total 15) NR NR Calls to outreach increased Pre post study, UK Surgical wards Post: 3 month pilot Referral to patient s from 432 in 7 mo to 231 in 3 mo Patel et al., 2011 (24) Pre: 3 yr n = 32,149 Pre post study, UK Post: 3 yr (admissions) Trauma and orthopedic wards Smith and Oakey, 2006 (25) Retrospective cohort study with matched control, UK General hospital Ward patients with suspected Legionnaire s disease during an outbreak 21 d (during outbreak) medical team and the critical care outreach team Age NR Nursing to seek senior medical advice, referral to RRT (critical care outreach team) MEWS.4 (total 21) Decrease (%) in deaths per admissions, pre vs. post: Males: 0.4 (95% CI, ), P = % males Females 1.5% (95% CI, ), P = Total: 0.9 (95% CI, %), P = NR NR n = 89 MEWS = 3 (total 17) NR NR Median 4.9 observation sets/patient day Median age: Referral for critical Cases 64.7 care advice Controls 61.0 Subbe et al., Pre: 2 mo n = 1,695 (case) MEWS.4 (total 15) MEWS 0 2: Hospital: NR NR 2003 (26) Post: 2 mo Pre post study, UK n = 659 (control) Age: Doctors examined patients within 1 hr 6% (case) 6% (control), P = 0.97 ICU: General hospital medical admissions unit, 56 beds 64 (case) MEWS 3 4: LOS in ICU, 63 (control) 17% (case) 2 day (case) Males: 13% (control) 4 day (control), P =0.3 4% (case) P = % (control) MEWS 5 15: 28% (case) 20% (control) P = 0.25 Cardiopulmonary arrest, score of 3 or 4: 5% (case) Median 3.6 EWS/patient day; 2,036/3,739 (54.4%) observation sets contained a correct EWS Uo and LOC inconsistently recorded RR had highest scoring errors 264/2,757 errors (9.6%) vs. HR 5.4%, SBP 4.3% vs. Temp 3.9% 66/270 (24.4%) observation sets were underscored and should have triggered an intervention but did not proportion of incorrect scores higher in the cases (17%) vs. control (12%) for difference of 5% (95% CI, ), P = 0.02 (Continued ) Systematic Review 1461

9 Table 3. (Continued ) Resource use: MET, RRT, critical care outreach teams, or nursing Resource use: LOS, ICU In-hospital outcomes: mortality (30-d), cardiovascular events Score that triggers activation of response, description of response Population, mean age (yr), sex Observation period Study, design, setting 0% (control) P, NR ICU Txf: 16% in intervention group, 14% in control group, NS Mortality: 11% in intervention group, 10% in control group, NS 12 mo Trial group Real-time automated alerts score: NR Bailey et al., 2013 (15) Randomized controlled Length of stay, median: 7.07 vs. 6.92, NS crossover study, USA n = 2,353 Charge nurse assessed the patient and alerted covering physician Age 57 53% males 50% white, 48% black 1,250-bed tertiary care academic hospital, general medical ward Definition of abbreviations: CI = confidence interval; CPR = cardiopulmonary resuscitation; DNAR = do not attempt resuscitation; EWS = Early Warning System; FH = Freeman Hospital; HR = heart rate; ICU = intensive care unit; ICU Txf = transfer to intensive care unit; LOC = level of consciousness; LOS = length of (hospital) stay; MET = medical emergency team; MEWS = Modified Early Warning Score; NR = not reported; NS = not significant; RR = respiratory rate; RRR = Relative Risk Ratio; RRT = Rapid Response Team; RVI = Royal Victoria Infirmary; SBP = systolic arterial pressure; Uo = urine output; VS = vital signs. reduced length of stay (median [interquartile range (IQR)] 9.7 days [ ] vs. 6.9 days [ ]; P, 0.001) (19). An increase in length of hospital stay from 4.0 ( ) days to 4.8 ( ) days was observed in afourthstudycomparingdata4months before with those 4 months after EWS implementation (21). Variation in study populations (e.g., all patients on a ward vs. selected patients) and follow-up time make it difficult to assess the overall effect of EWS on length of stay across studies. Admissions to the ICU Five studies evaluating the impact of the EWS on ICU utilization also reported mixed results (15, 18, 19, 22, 26). Two studies found asignificant increase in the number of ICU admissions after implementing EWS and accounting for differences in overall hospital admission rates (18, 19), whereas a third study found no difference in the proportion of patients transferred from the general medicine wards to the ICUs (15). One study involving two hospitals found increases of 24.5% and 14% in the annual ICU admission rates, but a significant decrease in the proportion of patients admitted to the ICU after having undergone cardiopulmonary resuscitation (pre-ews 3% vs. post-ews 2%, P = 0.004; and pre-ews 6.65% vs. post-ews 2.63%, P, 0.001) (22). One study found the proportion of clinically unstable patients who were on the ward for 6 hours or longer had decreased from 41.2 to 24.5% after implementing EWS (18). Two studies found no difference in length of ICU stay (18, 26). Use of Rapid Response and Code Teams Four studies evaluated the impact of EWS on RRTs and code teams (16, 18, 20, 22). All of the studies found at least a 50% increase in the number of RRT or ICU liaison team calls. Code blue calls decreased by 6 33% in three studies (16, 18, 22). One study found that the number of code blue calls for a patient still breathing and with a pulse increased from 47.9 to 64.4%, suggesting that response teams were activated before the patient s condition deteriorated to the point of cardiac or respiratory arrest (18). Nursing Four studies examined the impact of EWS implementation on nursing (17, 19, 25, 27). Three studies measured the accuracy and compliance of scoring (17, 19, 25). Compliance was as low as 53% in one study 1462 AnnalsATS Volume 11 Number 9 November 2014

10 (17) and as high as % with the use of electronic calculations (19). Smith and Oakey found that the most inconsistently recorded data were urinary output and level of consciousness, with 45.6% missing values, and that respiratory rate had the highest errors (9.6%) (25). Hammond and coworkers found that urinary output was the only statistically significant improvement in individual vital sign recordings after EWS implementation (27% increase, P = 0.03) (27). Patient observations and clinical attention by nursing staff increased with EWS implementation, with greater attention given to patients with higher scores (19). One study found that the frequency of patient observations increased during daytimeshifts,butnotatnight(17). Two studies examined the period after discharge from the ICU, with one study finding a significant increase in full vital sign documentation during the first 24 hours post-icu discharge (P, 0.001) (27) and the other findinganincreasein the frequency of patient observations per nursing shift (17). Quality Assessment Although the one randomized controlled study was of good quality, all of the other included studies used an historically controlled study design, which has an inherently high risk of bias because of the presence of unmeasured confounding factors. Advances in medical care or other changes in practice cannot be ruled out. None of the included retrospective studies adjusted for preintervention trends in mortality rate or accounted for other secular changes in care that could simultaneously have affected mortality, and none followed changesinoutcomeratesovertime following implementation. Immortal time bias is another potential cofounder, in that patients without adverse events have more time available for vital sign measurement. The impact of EWS implementation on health outcomes and resource utilization is inconclusive because of these methodological limitations in the evidence. Discussion EWS tools, mostly using vital sign abnormalities, perform reasonably well in predicting cardiac arrest and death within 48 hours of measurement. The effects of EWS use on health outcomes and utilization of resources, however, remain uncertain. A priori, for predictive ability, we elected to include only studies that reported on critical events within 48 hours after the scores were intended to identify patients with more immediate critical needs so that interventions could be implemented before the actual event occurred. Death occurring beyond the acute time frame may actually reflect a subsequent change rather than the score that triggered the initial intervention. Kellett and Kim, in addition to reporting death within 48 hours, also reported on death at multiple time points up to 30 days and found good predictability for death over time (AUROCs: 0.93 at 48 hours, 0.87 at 5 days, and 0.81 at 30 days) (8). The ability of an EWS to predict both short- and long-term death may be attributed to the fact that patients with profound vital sign abnormalities are at high risk of death, regardless of specific interventions or their timeliness. Indeed, Opio and colleagues, in a study conducted in a resource-poor setting in Uganda without an ICU, found an AUROC for mortality similar to other EWS scores, suggesting that patients with severe vital sign abnormalities are at high risk of dying, regardless of their level of care (13). In this article, we report on the predictability of clinically meaningful events (cardiopulmonary arrest, mortality), rather than on the use of RRT or admission to ICU, because these are the intended consequences of elevated scores and may be associated with unintended negative effects. The impact of RRT implementation was recently studied in a large, historically controlled study in a U.S. hospital with an around-the-clock ICU consultation service and found an increased risk of hospital death in patients transferred to the ICU from the ward (OR, 1.23; 95% CI, ) (28). They found that RRT implementation did not improve the outcomes adjusted for severity of illness in patients transferred from the ward, and they expressed caution that RRT implementation may have unforeseen costs without obvious benefit (28). Given the limitations of the design of pre post studies, the impact of EWS implementation on in-hospital health outcomes and resource utilization has not been studied adequately to draw conclusions. With regard to scoring consistency, though RRT appeared to increase it and the use of programmed score calculators showed promise, all other outcomes were measured heterogeneously and had mixed results. Of note, the one randomized control study did not find any difference in mortality, transfers to ICU, or length of hospital stay (15). Additional studies of high quality are needed to determine if these findings are accurate in other populations. The current evidence is limited by the methodological flaws of the pre post study design: the widely varied characteristics in training and implementation, the lack of standardization of the score to trigger a response, and the heterogeneity of the response itself. In addition, the trade-offs in sensitivity and specificity for predicting mortality or cardiac arrest at specific cutpoints make it a challenge to directly inform clinical decision-making. This leaves the applicability of our findings to real-world clinical settings uncertain, but does provide a good foundation on which to build future studies. Previous systematic reviews identified similar shortcomings in the existing evidence (1, 29, 30). The most important limitation of our review is that, due to the paucity of available evidence, we included studies of low methodological quality to evaluate the impact of EWS implementation. Future research should include randomized trials as well as methodologically rigorous observational studies that include an active control arm with standardization of comparators. Clinically important health outcomes should include 30-day mortality, cardiopulmonary events, and quality of life at discharge when evaluating the impact of EWS implementation. Standardization of threshold scores and responses would help to improve the applicability of study findings. The use of electronic health records and software to calculate scores and alert staff may help to reduce heterogeneity. A recent study looking at vital sign changes within 24 hours of a cardiac arrest, ICU transfer, or death suggested that future studies consistently report on these three outcomes to allow for model comparison (31). We argue that ICU transfer is expected when a patient becomes unstable, which makes this a less meaningful outcome; however, we fully agree with the need for measuring only immediate critical outcomes (24- to Systematic Review 1463

11 48-hour cardiac or pulmonary arrest and mortality). EWSs are being used broadly without clear evidence of benefit in terms of patient outcomes. Recently, a working group of the American Thoracic Society recommended against implementing performance measures unless they demonstrate clear benefit to patients and [are] supported by strong evidence (32). The existing evidence on the impact of EWSs on patient outcomes is insufficient, and, without rigorously designed studies reporting clinically meaningful outcomes, there remains a risk of unintended consequences for resource utilization and cost. Conclusions Current early warning system tools perform well for predicting death and cardiac arrest within 48 hours, although the impact on in-hospital health outcomes and utilization of resources remains uncertain, owing to methodological limitations. Efforts to more accurately assess their effectiveness will be needed as use becomes more widespread. n Author disclosures are available with the text of this article at Acknowledgments: The authors thank Rose Relevo, M.L.I.S., M.S., for developing the search strategy and compiling the literature database for this report. References 1 McGaughey J, Alderdice F, Fowler R, Kapila A, Mayhew A, Moutray M. Outreach and Early Warning Systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev 2007;(3): CD Whittington J, White R, Haig KM, Slock M. Using an automated risk assessment report to identify patients at risk for clinical deterioration. Jt Comm J Qual Patient Saf 2007;33: Berwick DM, Calkins DR, McCannon CJ, Hackbarth AD. The 100,000 lives campaign: setting a goal and a deadline for improving health care quality. JAMA 2006;295: Treadwell JR, Reston JT, Singh S, Talati R, McPheeters M. A Framework for Best Evidence Approaches in Systematic Reviews. Methods Research Report. (Prepared by the ECRI Institute Evidence-based Practice Center under Contract No. HHSA I.) AHRQ Publication No. 11-EHC046-EF. Rockville, MD: Agency for Healthcare Research and Quality; June Available at: pdf/toc.pdf. 5 Hayden JA, van der Windt DA, Cartwright JL, Côté P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med 2013;158: Agency for Healthcare Research Quality, Effective Health Care Program. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication No. 10(14)-EHC063- EF. Rockville, MD: Agency for Healthcare Research and Quality; January 2014 [accessed 2014 Jun 23]. Available from: Methods-Guide pdf 7 Churpek MM, Yuen TC, Huber MT, Park SY, Hall JB, Edelson DP. Predicting cardiac arrest on the wards: a nested case-control study. Chest 2012;141: Kellett J, Kim A. Validation of an abbreviated VitalPAC Early Warning Score (ViEWS) in 75,419 consecutive admissions to a Canadian regional hospital. Resuscitation 2012;83: Prytherch DR, Smith GB, Schmidt PE, Featherstone PI. ViEWS towards a national early warning score for detecting adult inpatient deterioration. Resuscitation 2010;81: Rothschild JM, Gandara E, Woolf S, Williams DH, Bates DW. Singleparameter early warning criteria to predict life-threatening adverse events. J Patient Saf 2010;6: Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation 2013;84: Churpek MM, Yuen TC, Park SY, Meltzer DO, Hall JB, Edelson DP. Derivation of a cardiac arrest prediction model using ward vital signs. Crit Care Med 2012;40: Opio MO, Nansubuga G, Kellett J. Validation of the VitalPAC Early Warning Score (ViEWS) in acutely ill medical patients attending a resource-poor hospital in sub-saharan Africa. Resuscitation 2013; 84: Churpek MM, Yuen TC, Park SY, Gibbons R, Edelson DP. Using electronic health record data to develop and validate a prediction model for adverse outcomes in the wards*. Crit Care Med 2014;42: Bailey TC, Chen Y, Mao Y, Lu C, Hackmann G, Micek ST, Heard KM, Faulkner KM, Kollef MH. A trial of a real-time alert for clinical deterioration in patients hospitalized on general medical wards. J Hosp Med 2013;8: Albert BL, Huesman L. Development of a modified early warning score using the electronic medical record. Dimens Crit Care Nurs 2011;30: De Meester K, Das T, Hellemans K, Verbrugghe W, Jorens PG, Verpooten GA, Van Bogaert P. Impact of a standardized nurse observation protocol including MEWS after Intensive Care Unit discharge. Resuscitation 2013;84: Green AL, Williams A. An evaluation of an early warning clinical marker referral tool. Intensive Crit Care Nurs 2006;22: Jones S, Mullally M, Ingleby S, Buist M, Bailey M, Eddleston JM. Bedside electronic capture of clinical observations and automated clinical alerts to improve compliance with an Early Warning Score protocol. Crit Care Resusc 2011;13: Maupin JM, Roth DJ, Krapes JM. Use of the Modified Early Warning Score decreases code blue events. Jt Comm J Qual Patient Saf 2009;35: Mitchell IA, McKay H, Van Leuvan C, Berry R, McCutcheon C, Avard B, Slater N, Neeman T, Lamberth P. A prospective controlled trial of the effect of a multi-faceted intervention on early recognition and intervention in deteriorating hospital patients. Resuscitation 2010;81: Moon A, Cosgrove JF, Lea D, Fairs A, Cressey DM. An eight year audit before and after the introduction of modified early warning score (MEWS) charts, of patients admitted to a tertiary referral intensive care unit after CPR. Resuscitation 2011;82: Odell M, Forster A, Rudman K, Bass F. The critical care outreach service and the early warning system on surgical wards. Nurs Crit Care 2002;7: Patel MS, Jones MA, Jiggins M, Williams SC. Does the use of a track and trigger warning system reduce mortality in trauma patients? Injury 2011;42: Smith AF, Oakey RJ. Incidence and significance of errors in a patient track and trigger system during an epidemic of Legionnaires disease: retrospective casenote analysis. Anaesthesia 2006;61: Subbe CP, Davies RG, Williams E, Rutherford P, Gemmell L. Effect of introducing the Modified Early Warning score on clinical outcomes, cardio-pulmonary arrests and intensive care utilisation in acute medical admissions. Anaesthesia 2003;58: Hammond NE, Spooner AJ, Barnett AG, Corley A, Brown P, Fraser JF. The effect of implementing a modified early warning scoring (MEWS) system on the adequacy of vital sign documentation. Aust Crit Care 2013;26: Karpman C, Keegan MT, Jensen JB, Bauer PR, Brown DR, Afessa B. The impact of rapid response team on outcome of patients 1464 AnnalsATS Volume 11 Number 9 November 2014

12 transferred from the ward to the ICU: a single-center study. Crit Care Med 2013;41: McNeill G, Bryden D. Do either early warning systems or emergency response teams improve hospital patient survival? A systematic review. Resuscitation 2013;84: Gao H, McDonnell A, Harrison DA, Moore T, Adam S, Daly K, Esmonde L, Goldhill DR, Parry GJ, Rashidian A, et al. Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med 2007;33: Churpek MM, Yuen TC, Edelson DP. Predicting clinical deterioration in the hospital: the impact of outcome selection. Resuscitation 2013; 84: KahnJM,GouldMK,KrishnanJA,WilsonKC,AuDH,CookeCR, Douglas IS, Feemster LC, Mularski RA, Slatore CG, et al.; ATS Ad Hoc Committee on the Development of Performance Measures from ATS Guidelines. An official American Thoracic Society workshop report: developing performance measures from clinical practice guidelines. Ann Am Thorac Soc 2014;11: S186 S195. Systematic Review 1465