Validation of a Paediatric Early Warning Score: first results and implications of usage

Transcription

1 DOI /s ORIGINAL ARTICLE Validation of a Paediatric Early Warning Score: first results and implications of usage Joris Fuijkschot & Bastiaan Vernhout & Joris Lemson & Jos M. T. Draaisma & Jan L. C. M. Loeffen Received: 16 April 2014 /Revised: 27 May 2014 /Accepted: 29 May 2014 # Springer-Verlag Berlin Heidelberg 2014 Abstract Timely recognition of deterioration of hospitalised children is important to improve mortality. We developed a modified Paediatric Early Warning Score (PEWS) and studied the effects by performing three different cohort studies using different end points. Taking unplanned Paediatric Intensive Care Unit admission as end point and only using data until 2 h prior to end point, we found a sensitivity of 0.67 and specificity of 0.88 to timely recognise patients. This proves that earlier identification is possible without a loss of sensitivity compared to other PEWS systems. When determining the corresponding clinical condition in patients with an elevated PEWS dichotomously as sick or well, this resulted in a total of 27 % false-positive scores. This can cause motivational problems for caregivers to use the system but is a consequence of PEWS design to minimise false-negative rates because of high mortality associated with paediatric resuscitation. Using the need for emergency medical interventions as end point, sensitivity of PEWS is high and it seems, therefore, that it is also fit to alert health-care professionals that urgent interventions may be needed. Conclusion: These data show the effectiveness of a modified PEWS in identifying critically Communicated by Patrick Van Reempts J. Fuijkschot (*): B. Vernhout : J. M. T. Draaisma : J. L. C. M. Loeffen Department of Paediatrics, Radboudumc Amalia Children s Hospital, PO box 9101, 6500 HB Nijmegen, The Netherlands joris.fuijkschot@radboudumc.nl J. Lemson Department of Intensive Care, Radboudumc Amalia Children s Hospital, PO box 9101, 6500 HB Nijmegen, The Netherlands J. L. C. M. Loeffen Institute for Warranted Quality and Patient Safety, Radboudumc Amalia Children s Hospital, PO box 9101, 6500 HB Nijmegen, The Netherlands ill patients in an early phase making early interventions possible and hopefully reduce mortality. Keywords Paediatric Early Warning Score (PEWS). Rapid response systems. Early interventions. Situational awareness Abbreviations PEWS PICU SA SSE Introduction Paediatric Early Warning Score Paediatric Intensive Care Unit Situational awareness Serious safety event Vital parameters of hospitalised patients may deteriorate due to several reasons and subsequently require emergency interventions or intensive care treatment. Early detection of deterioration coupled to effective interventions will likely improve outcome. In the past decade, adult medicine has developed several early warning scoring systems and rapid response systems in order to improve recognition of clinical deterioration in hospitalised patients, thereby improving outcome. Validation studies have shown their effectiveness in identifying patients at risk for serious safety events (SSEs); however, its effect upon reducing overall mortality has yet to be determined [10, 14]. In paediatrics, early recognition of deterioration of vital parameters of hospitalised children is challenging because of the age-related range of reference vital parameters. Hence, the implementation of an unequivocal Paediatric Early Warning Score (PEWS) may improve early recognition and situational awareness amongst health-care professionals.

2 A frequently quoted clinically validated PEWS system has been designed by Parshuram and colleagues [11]. This scoring system which is based on seven vital parameters was prospectively validated in three Canadian and one UK children s hospital. Other, comparable PEWS systems validated in the West-European setting are referred to as the Brighton and Cardiff PEWS [1, 6]. Most systems have been validated retrospectively and suffer from missing data caused by incomplete scores. Still, when defining cardiopulmonary arrest or unplanned Paediatric Intensive Care Unit (PICU) admission as end points, these systems have shown to detect critically ill children at least minutes to 1 h prior to reaching these end points [1, 6, 13, 15]. These scoring systems therefore seem to be useful clinical tools. Most early warning systems are designed to predict the risk for cardiopulmonary arrest or unplanned PICU admission. However, warning systems that also identify the need for emergency medical interventions in an early phase enable health-care professionals to respond earlier and possibly reverse clinical deterioration, thereby preventing unplanned PICU admission. In the Netherlands, experience with regard to fully implemented PEWS systems was lacking. We decided to design and implement a PEWS system that was specified to our setting and which was constructed using latest insights from both paediatric and adult warning systems. Subsequently, we studied its effects upon several patient groups. Aims and objectives The aim of this study is to show the additional value of PEWS systems in clinical practice towards patient safety and healthcare quality. Objectives are to validate this PEWS system for its applicability in timely identification of sick patients (using different end points) and to study its capacity to identify the need for emergency medical interventions. Method Setting The Radboudumc Amalia Children s Hospital is a tertiary referral university hospital with three paediatric wards adding up to a total of 77 beds and is located in the Eastern part of the Netherlands. Yearly, it receives an average of 4,000 admissions at these wards where patients get highly specialised care from a broad spectrum of surgical and non-surgical hospital specialists. The same group of nurses with standardised paediatric qualifications and training cares for the patients. In case of clinical deterioration of patients, monitoring of vital norm parameters as well as various emergency medical interventions (such as a fluid challenge or supplemental oxygen, etc.) can be performed in the ward. In case of further deterioration towards a critically ill state, patients can be transferred to a 10-bed, full facility PICU. PEWS design and implementation From 2011, a modified PEWS system was implemented on all three paediatric wards. This scoring system was based on the data from the Parshuram study. Strikingly however, despite evidence that fever is an important factor in the prediction of paediatric sepsis, neither the Parshuram scoring system nor the Brighton PEWS includes body temperature [8, 16]. Data from adult warning systems show the importance of temperature as a key physiological parameter in predicting clinical deterioration [5]. We therefore decided to add temperature to our scoring system (addition of maximal 2 points to the total score of a patient) expecting to increase PEWS performance, especially in sepsis. Other minor adjustments were made to adapt the system to our setting and to improve user-friendliness. These included a simplified definition of work of breathing (normal or mildly, moderately, severely increased) and supplemental oxygen (room air, low-flow or high-flow supplemental oxygen). Operational proceedings for staff regarding usage of the scoring system in clinical practice were defined. This resulted in an eight-parameter-based bedside PEWS system with a possible scoring range of 0 28 points. All patients admitted to the paediatric wards are routinely scored every 8 h unless their clinical condition deteriorates in which case the frequency of scoring is intensified. A score of 0 3 indicates no specific actions from nursing staff. At a score of 4 7 points, the scoring frequency is automatically increased, and if the score exceeds 7, the nursing staff is instructed to immediately contact the medical team enabling it to promptly identify potential clinical deterioration and treat accordingly. The threshold was chosen based upon the data from the Parshuram study (threshold of 7 points to identify children at risk of cardiopulmonary arrest; corresponding sensitivity 0.64 and specificity 0.91) and the addition of an extra parameter (body temperature) to our scoring system. For each age category, a separate card was developed (0 3 months/3 months 1 year/1 4 years/4 12 years/ 12 years). An example of one card is given in Fig. 1. Study design and participants To study the performance of our warning system in general and in selected patient cohorts, we performed three different case cohort studies focusing on both the timely identification of sick patients (case cohort studies 1 and 2) and identification of patients in need for emergency medical interventions (case cohort study 3).

3 Fig. 1 PEWS card for age category 3 months 1 year Paediatric Early Warning Score Age: 3 months 1 year Score Respiratory rate < >90 (breaths/min) Respiratory effort* normal mildly moderate severely or apnoeic Pulse saturation in <91% 91-94% >94% room air Supplemental oxygen Room air Low flow NRB-mask oxygen Heart rate (bpm) < >190 Capillary refill time (sternum) < 3 seconds 3 seconds Systolic blood < >130 pressure (mmhg) Temperature ( C) < >38.5 * respiratory effort: nasal flaring or retrac ons Standard: Scoring frequency 1x per 8 hours PEWS 3x 0-2: reduce scoring frequency to 1x per day PEWS score 4 points or worried sign: increase PEWS frequency to 1x per 4 hours PEWS score 6 points: increase PEWS frequency to 1x per hour PEWS score 8 points: contact a ending physician within 10 minutes or call PMET PMET dial pager 2148 Resuscita on dial Identification of sick patients Identification of need for critical care type interventions Case cohort study 1 This study was performed to test the scoring system s general ability to identify sick patients. Correlation between warning score and severity of illness was studied by performing a retrospective database review of early warning scores in all patients admitted at the 20-bed paediatric oncology ward over a 3-month period. The study was performed on this ward because it was the very first at which PEWS was implemented in 2011 and also because of the clinical nature of paediatric oncology patients. Though their physiological responses may differ from other patients, it was expected that a higher number of sick patients could be encountered at this ward, thereby improving chances of successful validation. Focus was on the clinical condition of patients with alarming high scores ( 8). In addition, the effects of the added body temperature parameter upon the scores were studiedinbothpatientswithalarminghighscoresanda randomly selected control group consisting of one third of the patients with PEWS <4 at all times during admission. Case cohort study 2 To compare our modified PEWS with other existing scoring systems and validate PEWS performance in the general paediatric population, its performance was studied retrospectively over a 9-month period. A selected cohort of patients whose clinical course at any time during their admission at the general ward had deteriorated towards the commonly used end points cardiopulmonary arrest and unplanned PICU admission. Case cohort study 3 To study the capacity of our modified PEWS to identify patients in need for emergency medical interventions, we prospectively evaluated warning scores in all patients receiving emergency medical interventions (definition outlined in Table 1) at the paediatric wards over a 4- month period. Results Because of differences in methods and studied populations, we discuss our results for each cohort study separately. Case cohort study 1 In 118/199 (59 %) admissions to the paediatric oncology ward, the PEWS was correctly performed and could be used for inclusion in the study (adding to a total of 1,115 separate PEWS values). In 91/118 (77 %) admissions, the scores were <4 (baseline score) at all times during their stay (control Table 1 Definition of sick Any of the following criteria: - Clinical condition considered sick by the senior member of medical staff and documented as such - Critical care-type intervention (listed below) performed - Any involvement of paediatric intensive care staff including cardiopulmonary arrest

4 group). Even more, in 103/118 (87 %) cases, all the scores were <8 (threshold score). There were no cardiopulmonary arrests at the ward during the study period. Unplanned PICU admission was only seen one time. This patient scored a PEWS 8 pending from up to 4 h prior to admission. In the 81 (41 %) excluded admissions, neither cardiopulmonary arrest nor unplanned PICU admission was seen, excluding selection bias. Furthermore, PEWS 8 was scored 56 times in 15/118 admissions (13 %) resulting in a specificity of 0.88 when taking unplanned PICU admission as end point. Sensitivity was calculated at 1.00; however, due to only one unplanned PICU admission, this parameter is not reliable. The corresponding clinical condition of an elevated PEWS was dichotomously determined as sick (Table 2)or well.in 41/56 scores, the clinical condition of the patient was sick, resulting in a total of 15 (27 %) false-positive scores ( well patients with PEWS 8) and a positive predictive value of The clinical condition in the false-positive patients was dominated by PEWS-influencing factors such as pain (n=6), fever (n=1) or a combination of both (n=8). Whether or not patients with a PEWS below threshold score (<8) were actually in fact sick but managed without paediatric intensive care involvement could not be ascertained due to methodological difficulties. The volume of this group and corresponding amount of PEWS values was very large. Besides, the criteria of sick could only be derived from patients files manually, resulting in over a thousand separate queries. However, with regard to the commonly used end point unplanned PICU admission or cardiopulmonary arrest, we could ascertain that these scores contained no false negatives. Fever contributed to the score in 17/41 (41 %) cases in the sick patient group, indicating its importance as a Table 2 Definition of emergency medical interventions Airway/breathing interventions % supplemental oxygen by non-rebreathing mask - Airway opening manoeuvre or oropharyngeal/nasopharyngeal airway patency - Bag valve mask resuscitation - Adrenaline nebulisation Circulation interventions - Fluid challenge - Emergency transfusion of blood products - Adrenaline administration intramuscular/intravenous - Sepsis work-up (blood/urine/spinal fluid cultures followed by start of parenteral broad spectrum antibiotics) Other interventions - Any paediatric intensive care staff involvement - Cardiopulmonary resuscitation contributing factor. In the control group (PEWS <4 at all times during the admittance), fever had a much lower prevalence of 22 % (based upon a randomly chosen sample of 32/91 (35 %) cases). Case cohort study 2 No cardiopulmonary arrests occurred at the general paediatric wards during the study period. Out of 36 patients who had an unplanned admission to the PICU, 24 had sufficient data to retrospectively reconstruct the course of the PEWS in the hours prior to their PICU admission. Out of 24 patients, 16 scored PEWS of 8 at2 6 h prior to PICU admission. The overall median PEWS 2 6 h prior to PICU admission was 8.5 (range 2 15). When excluding patients with acute and unforeseen clinical deterioration in whom early detection is extremely difficult such as an anaphylactic reaction to parenteral administered drugs (n=1), epileptic seizures (n=1) and unexpected arterial haemorrhage (n=1), the median PEWS increased to 9(range 5 15). A detailed analysis of PEWS performance in subgroups is given in Table 3. The sensitivity of our PEWS in identifying patients 2 6 h prior to unplanned PICU admission (threshold score 8) is calculated at At time of their admission to the PICU, all of the included patients scored PEWS 8, indicating a further increase of sensitivity in the remaining time prior to reaching end point. Case cohort study 3 A total of 17 cases that received emergency medical interventions as previously defined at the paediatric wards were evaluated. It showed a median PEWS of 10 (range 8 15) in these patients at the time of the intervention. Therefore, with a threshold score of 8, no falsely negative warning scores could be detected in this study, indicating a high sensitivity in identifying these patients. Table 3 PEWS performance divided in subgroups; end point unplanned PICU admission Patient characteristics Number Median score PEWS [range] a All [2 15] <3months 6 7[5 9] 3months<1year 5 9[6 11] 1year<4years 5 9[2 15] 4 years<12 years 3 10 [8 10] 12 years 5 8 [2 14] Male 11 7 [2 15] Female 13 9 [2 14] a Two to 6 h prior to unplanned PICU admission

5 Discussion Key findings With this study, we show that our modified PEWS is capable of a timely identification of patients at risk for unplanned PICU admission, with comparable sensitivity (0.67 versus 0.64) and specificity (0.88 versus 0.91) towards the Parshuram PEWS. Sensitivity is lower than the Brighton PEWS (0.85), but validation studies of the Brighton PEWS include data up and until reaching the end point, thus increasing the apparent performance of this score [1]. Even more, by including data only up and until 2 h prior to reaching end point (instead of 1 h in the Parshuram study), we have shown that earlier identification is possible without a loss of sensitivity of the scoring system. A comparison between the original PEWS and the modified PEWS studied here is given in Table 4. It is possible that the parameter temperature has an additional value to this improved performance. Though this postulation cannot be backed up by data from this study, this is in line with findings from other studies [4, 5]. On the other hand, fever has direct and indirect (through elevation of heart and respiratory rates) effects upon the total PEWS. This may influence the scoring system negatively and result in an increased number of falsely positive PEWS values. Nonetheless, in the control group, fever never lead to an elevated PEWS, and in the sick-patient group, only one falsely fever-related elevated PEWS was seen. It therefore appears that fever as a single factor in otherwise clinically stable patients does not increase the scores above the alarming threshold. This study shows the capacity of warning systems in not only timely identifying the patients who are at risk for unplanned PICU admission but also towards identifying sick patients and the need for emergency medical interventions. This is relevant because the commonly used end point unplanned PICU admission is only a surrogate measure of a child s severity of illness and other end points (e.g. need for emergency medical interventions) may as well identify sick children, perhaps even in an earlier phase. Strengths and limitations Most studies in literature regarding paediatric warning systems use the same end points (cardiopulmonary arrest or unplanned PICU admission) when validating the system. This study adds insight in the effectiveness of such warning systems when also taking different end points (sick patients and need for critical care type intervention) in respect. However, some limitations are to be made. In both studies 1 and 2, there is a substantial rate of falsepositive scores. This is in line with findings from the Parshuram study. It could indicate that the cut-off point for alarming high PEWS at score 8 should be higher. However, a higher cut-off point would result in more false-negative, missed patients. In general, paediatric cardiopulmonary arrest or unplanned PICU admission is associated with high mortality and significant neurological disability [7]. This makes a higher rate of false-negative scores unacceptable. It is however crucial that caregivers realise the background of the chosen cut-off point and its consequences to keep them motivated to use the system despite the false-positive scores they encounter. Due to small patient numbers, considerations are to be made when interpreting our data. Also, the retrospective case cohort studies 1 and 2 suffered from missing data caused by incomplete scores, like many other studies on this subject [12]. In study 1, only 59 % of the admissions during the study period had sufficient data to be included in the study. Missing data to calculate PEWS or non-compliance to scoring protocol (e.g. PEWS scoring frequency below standard) was the main reason for exclusion. Nursing and medical staffs worked in different filing systems in our hospital during the time of this study. Because data from nursing files were often not secured Table 4 Comparison of original PEWS and modified PEWS Original PEWS (Parshuram et al.) Modified PEWS Threshold score PEWS 7 points 8 points (alarming score) Temperature scoring item No Yes End point in validation studies Cardiopulmonary Unplanned PICU admission resuscitation Unplanned PICU Need for emergency medical interventions admission Data inclusion in validation studies Ending 1 h before event Ending 2 h before event Specificity 91 % 88 % Sensitivity 64 % 67 %

6 after the discharge of the patient, this resulted in a loss of data causing a rather high number of exclusions in study 2. This all reflects on the difficulties systems like PEWS encounter upon implementation. In general, nursing staff welcomed the PEWS system and protocol. It provided an easy accessible data set (hence: knowledge) together with a straightforward protocol describing how to respond in critical situations. Adherence to the system proved however difficult because of lacking user-friendliness (e.g. additional paperwork) and scepticism mainly amongst medical staff towards the need of such a scoring system to identify sick children. Though not many patients were missed by the system (unplanned PICU admission without elevated PEWS in the hours preceding), these also demotivated staff to adhere to the PEWS protocol. In the past years, we have put in efforts to improve the system by improving user-friendliness by eliminating additional paperwork and creating a completely automated, electronic system. Also, nursing and medical staffs now work in the same electronic filing system and there is no data loss after discharge of the patient. The results given here provided insights in PEWS performance and were used to counter scepticism, especially regarding false-positive scores. PEWS patterns were used in the methodological analysis of serious safety events. All together, this resulted in a recent steep increase of compliance to the PEWS protocol. It is our belief that implementation of systems like PEWS can only be successful if performance data are continuously shared with the users improving insights in its (im)possibilities. In our hospital, cardiopulmonary arrests seldom occur at the wards most likely because of early interventions and referral to the PICU, indicating a rather safe environment. Yet, for our study, it resulted in the limitation that no data from patients with cardiopulmonary arrest could be included. In general, the usage of warning score systems is based upon the assumption that earlier identification and subsequently interventions performed earlier lead to fewer SSEs and reduction of morbidity and mortality. For this assumption however, compelling evidence is lacking in literature [9, 12]. Validation studies as given here only show that PEWS is, to a certain degree, capable of detecting sick patients (using different surrogates to define this) but potential direct effects upon a reduction of morbidity and mortality as well as SSEs are not measured. However, the survival of paediatric inhospital cardiopulmonary resuscitation has improved significantly over the past decades and relatively more resuscitations take place at PICU s than at normal paediatric wards[7]. This is a trend also seen in our own hospital. It is postulated that the use of early warning scores leads to earlier transfer to units with closer patient monitoring that allows for more prompt initiation of resuscitation efforts and hence PEWS indirectly improves outcome in resuscitation. If this is the case, then PEWS has added value in terms of cost efficiency and patient safety. However, studies to prove such a direct relationship are almost impossible to design and therefore lacking in literature. The same implies to potential effects of warning score systems on human factor competencies, especially on situational awareness (SA). Lessons learned from aviation and the nuclear power industries have shown the importance of situational awareness to the human decision maker in complex situations with high data flow and situational complexity [2, 16]. A recent study showed some additional benefits of early warning systems upon team communication and handling of critical care situations [3]. We believe that the implementation of a warning system alone is not enough to reduce SSEs and improve outcome in hospitals. This can only be accomplished when the warning system is part of a safety culture characterised by sufficient professional (team) training (focusing on both technical and non-technical skills) and that high PEWS values are promptly detected by rapid response teams and followed by interventions (emergency medical intervention or PICU admission). This should be considered when implementing such a system in a hospital. Implications of usage In our setting, we recently integrated PEWS with other risk factors of serious safety events (such as the worried sign and family concern), resulting in a Paediatric Risk Evaluation and Stratification System (PRESS). Each clinically admitted patient is stratified into a risk category (standard, medium or high) with safety rules and protocol matching accordingly. A software system that provides the clinician on call with real time data of warning scores and risk categories from admitted patients was developed. We expect that the system increases SA and forthwith improving team performance. Of course, we need data from pending research to back up these postulations. Conclusion This study shows that, by using a modified paediatric warning system, health-care professionals are able to identify sick patients in an earlier stage, thereby enabling them to possibly prevent further clinical deterioration and improve outcome. Conflict of interest References No conflict of interest is to be reported. 1. Akre M, Finkelstein M, Erickson M, Liu M, Vanderbilt L et al (2010) Sensitivity of the Pediatric Early Warning Score to identify patient deterioration. Pediatrics 125:e763 e769

7 2. Bleetman A, Sanusi S, Dale T, Brace S (2012) Human factors and error prevention in emergency medicine. Emerg Med J 29(5): Bonafide CP, Roberts KE, Weirich CM, Paciotti B, Tibbetts KM, Keren R, Barg FK, Holmes JH (2013) Beyond statistical prediction: qualitative evaluation of the mechanisms by which pediatric early warning scores impact patient safety. J Hosp Med 8(5): Burch VC, Tarr G, Morroni C (2008) Modified early warning score predicts the need for hospital admission and inhospital mortality. Emerg Med J 25(10): Cooksley T, Kitlowski E, Haji-Michael P (2012) Effectiveness of Modified Early Warning Score in predicting outcomes in oncology patients. QJM 105(11): Edwards ED, Powell CV, Mason BW, Oliver A (2009) Prospective cohort study to test the predictability of the Cardiff and Yale paediatric early warning system. Arch Dis Child 94(8): Girotra S, Spertus JA, Li Y, Berg RA, Nadkarni VM, Chan PS (2013) Survival trends in pediatric in-hospital cardiac arrests: an analysis from Get With the Guidelines-Resuscitation. Circ Cardiovasc Qual Outcomes 6(1): Isaacman DJ, Shults J, Gross TK, Davis PH, Harper M (2000) Predictors of bacteremia in febrile children 3 to 36 months of age. Pediatrics 106(5): McCabe A, Duncan H, Heward Y (2009) Paediatric early warning systems: where do we go from here? Paediatr Nurs 21(1): McGaughey J, Alderdice F, Fowler R, Kapila A, Mayhew A, Moutray M (2007) Outreach and Early Warning Systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev 3, CD Parshuram CS, Duncan HP, Joffe AR, Farrell CA, Lacroix JR, Middaugh KL, Hutchison JS, Wensley D, Blanchard N, Beyene J, Parkin PC (2011) Multicentre validation of the bedside paediatric early warning system score: a severity of illness score to detect evolving critical illness in hospitalized children. Crit Care 15(4): R Roland D (2012) Paediatric early warning scores: Holy Grail and Achilles heel. Arch Dis Child Educ Pract Ed 97(6): Solevåg AL, Eggen EH, Schröder J, Nakstad B (2013) Use of a modified pediatric early warning score in a department of pediatric and adolescent medicine. PLoS One 8(8):e Subbe CP, Kruger M, Rutherford P, Gemmel L (2001) Validation of a modified Early Warning Score in medical admissions. QJM 94(10): Tucker KM, Brewer TL, Baker RB, Demeritt B, Vossmeyer MT (2009) Prospective evaluation of a pediatric inpatient early warning scoring system. Spec Pediatr Nurs 14(2): Van den Bruel A, Haj-Hassan T, Thompson M, Buntinx F, Mant D, European Research Network on Recognising Serious Infection investigators (2010) Diagnostic value of clinical features at presentation to identify serious infection in children in developed countries: a systematic review. Lancet 375(9717):