Using Exercises to Measure Preparedness August 6th :00 AM Paul D. Biddinger, MD FACEP Elena Savoia, MD MPH

Transcription

1

2

3

4 Using Exercises to Measure Preparedness August 6th :00 AM Paul D. Biddinger, MD FACEP Elena Savoia, MD MPH 4

5 Acknowledgements We gratefully acknowledge funding support from the U.S. Centers for Disease Control and Prevention (CDC) supplement to grant number 5PO1TP (Linking Assessment and Measurement to Performance in Exercises and Drills - Preparedness and Emergency Response Research Center) and 1U9OTP Preparedness & Emergency Response Learning Center. The content of this lecture as well as the views and discussions expressed are solely those of the speakers and do not necessarily represent the views of any partner organizations, the CDC or the U.S. Department of Health and Human Services nor does mention of trade names, commercial practices, or organizations imply endorsement by the U.S. Government. 5

6 Paul Biddinger, MD FACEP Elena Savoia, MD MPH Foluso Agboola, MD MPH Dorothy Bernard, MPH

7 Learning Objectives 1. Describe the use of exercises to measure preparedness, 2. Understand the strengths and limitations of the measurement approach to exercise performance, 3. Become familiar with the use of an online tool, available to practitioners, to improve the quality of data collection and exercise evaluation. 7

8 Webinar structure PART A What does it mean to create an exercise for evaluation purposes? PART B What are the strengths and weaknesses in the measurement approach to exercise performance? PART C Walk trough our online exercise evaluation tool PART D - Results from field testing Q&A 8

9 Background Exercises that simulate emergency scenarios are widely accepted as a major component of every well-established preparedness program. Unfortunately, the variability in the quality of the exercises conducted and the lack of standardized processes to measure performance have limited the value of exercises in improving preparedness efforts. 9

10 How can we define Exercise Performance Measurement? The process of collecting, analyzing and reporting information regarding the performance of a public health or healthcare system and its components in responding to a public health threat during an exercise.

11 Our experience 11

12 PART A An exercise well-suited for evaluation purposes! 12

13 Elements of Exercise Evaluation Harvard T.H. Chan Conceptual Model: Key Elements of Exercise Performance Measurement The Purpose The Exercise The Unit The Instrument Accountability Quality Improvement Type of Exercise Quality of Exercise (Scenario, participants etc.) The Unit of Observation The Unit of Analysis The Instrument Structure The Measures Measurement Properties Reliability Validity Feasibility Savoia E, Agboola F, Biddinger PD. A conceptual framework to measure system s performance during emergency preparedness exercises. Int J Environ Res Public Health Sep 17;11(9): Utility 13

14 Key element 1: the purpose of the measurement Accountability vs Quality Improvement Are health systems prepared? Are levels of preparedness improving?

15 Quality Improvement What went well? What went wrong? Where was the plan inadequate?

16 Key element 2: the characteristics of the exercise Participants Timeframe Scenario

17 PART B 17

18 Key element 3: the unit of observation Public health or Healthcare of analysis

19 Key element 4: the instrument ±2 pounds

20 Measurement Properties Reliability and Validity

21 The Measures and the Evaluation Form Likert Scale Yes? No? Narrative

22 Inter-Rater Reliability Object or Phenomenon? 70% With our form =

23 Validity Face Content Criterion

24 PART C Let s take a look at our tool: The Online Exercise Evaluation Database 24

25 Online Exercise Evaluation Tool You have access to exercise performance measures tested for reliability and validity You have access to more than 500 different performance measures You can download the measures into an evaluation form Very soon you will be able to use the analytical feature to generate basic outputs based on your exercise data. 25

26 What Types of Measures are Available To You? A goalkeeper s performance: TASKS AND ACTIONS! A public health system performance: TASKS AND ACTIONS!

27 What Types of Measures Are Available To You? General Task Action Action Action 500 ACTIONS 150 GENERAL TASKS Task X Action 1 yes/no Action 2 yes/no Action 3 yes/no Action y

28 Exercise Evaluation Tool 28

29

30

31

32

33

34

35 PART D: Field testing 35

36 Field testing of the tool: January-September agencies (4 hospitals + 10 public health agencies) from 8 states Structured interviews were conducted with all users (exercise planners) to: 1) Assess the use of the tool during exercise evaluation planning efforts, 2) Compare the tool with other instruments of evaluation and 3) Determine how results can be used for after action reporting purposes. 36

37 Field testing Total number of exercises: 23 [1 workshop; 10 tabletop; 4 drills; 4 functional; 4 full scale] Average number of exercise players: 34 Average number of evaluators: 5 Top 3 capabilities tested: Emergency Operations Coordination; Emergency Public Information & Warning and; Information sharing Purpose of the exercise: 80% was for both performance assessment and training 37

38 Our end users background Average number of years in emergency preparedness: 13 Training in HSEEP: 70% (Yes) ; 30% (No) All users had experience in planning an exercise Average number of exercise planned in the last 3 years: 7 90% have experience in creating exercise evaluation form 85% have experience in evaluating an exercise 85% have experience in writing after action report 38

39 Interview Results 93% of users found the evaluation toolkit helpful in creating evaluation forms for their exercise(s) 93% of users would you use its again Reason(s) cited for liking the toolkit: Applicability of metrics Alignment with target capabilities (HPP and PHEP) Provided guidance on what was needed for evaluation Useful for exercise development Ability to customize Reason(s) cited for not liking the toolkit: Required too much customization 39

40 Interview results 43% of users used the toolkit for other purpose other than evaluation purpose Exercise development Training Evaluation of real world incident (One user plans to use it for this purpose) 72% of users liked the final evaluation form that was generated from the toolkit. 93% of users found the data collected with the evaluation form helpful in generating an AAR 40

41 Comparison with other instruments Which evaluation instrument method do you prefer? 72% HSPH; 21% HSEEP; 7% Other Which instrument do you think will best allow you to measure changes/improvement over time? 79% HSPH; 7% HSEEP; 14% Other Which instrument do you think best captured the root causes that positively and negatively influenced performance during the exercise? 79% HSPH; 7% HSEEP; 14% Other Which instrument do you think is easier to use? 65% HSPH; 21% HSEEP; 14% Other 41

42 If you are interested in using the toolkit: If you need assistance or are interested in participating in a cohort of users please contact: eprep@hsph.harvard.edu

43 43

44

45

46 Check the ASPPH Events webpage for announcements about future ASPPH Presents webinars