Seeking Your Contingency Plan: Are You. Panel Members: Johns Hopkins University & Health System

Transcription

1 Seeking Your Contingency Plan: Are You HOT, COLD,, or WARM? Panel Members: Bill Rider (Johns Johns Hopkins Hospital & University Hospital and University) Kathy Lee Patterson (Children s Hospital of Philadelphia) Anne Marie Turner (University of Rochester Medical Center) Kevin Chenoweth (Vanderbilt University Medical Center) Angela Devlen (Caritas Christi Health Care Christi Health Care) Agenda Overview of Five Institutions Demographics IT Divisions DR Theories Baseline Strategies for Determining HOT, COLD, or WARM Institution DR Practices Comparisons, Open Systems RTO and RPO Underlying factors: internal vs. external Testing Maintenance Are You HOT, COLD, or WARM: RPO and RTO Considerations Results Questions to Consider Overview: Healthcare BCP/DRP Benchmarks BCPWHO Website Questions & Answers Johns Hopkins University & Health System Acute Care Beds Licensed 1,467 Opened 1,327 % Of Occupancy 77.0 Discharges 77,962 Days 381,601 Outpatient Encounters 1,929,660 Emergency Visits 208,682 Operating Room Cases 68,070 Inpatient 27,968 Outpatient 40,102 (Johns Johns H opkins Hospital & University Hospital, Bayview Medical Center, Howard County General Hospital, Johns Johns H opkins Hospital & University Clinical Physicians, JHU School of Medicine) 1

2 JH Information Systems 1 IBM Processor 406 MIPS (DR is 206) 8 Meg Memory (DR is 4) 1.3 Terabytes Storage (DR is 1.3 Tb) Multiple DEC and AIX Midrange Platforms Over 450 Distributed Servers Multiple I.T. locations within 15 mile radius Organization segregates: I.T. Operations Network Services Enterprise Services Applications Support Strong Health Systems Number of Beds: Strong 750 beds, Highland 275=1025 Number of In-Patient Discharges per year: 55K (SMH and HH) Number of Out-Patient Discharges per year: Outpatient Visits (not discharges): SMH OP: 360K UR Medical Faculty Group: 660K SMH Emergency Dept 90K SMH Lab Specimens 500K (these are not actual patient visits but do result in registrations) Eastman Dental Center 50K HH Emergency Dept - 25K HH OP - 75K Total 1.7M Strong Memorial Hospital, Highland Hospital, School of Medicine & Dentistry, School of Nursing, Research, Mt. Hope Family Center, UR Medical Faculty Group, University Health Service, Primary Care Network, Eastman Dental Center, Long Term Care, Visiting Nurse Service Strong Health Information Systems IBM Processor Multiple AIX Midrange Platforms Over 280 Open System Medical Center Servers in Data Center Information Systems Division: Enterprise Operations Network Services Enterprise LAN Security Help Desk Support / Desktop Support Systems Interface Research Support Shared Services Organization Medical Informatics Clinical Systems Applications Project Management 2

3 Vanderbilt University Medical Center Serving Middle Tennessee, northern Alabama and southern Kentucky. Only Level 1 Trauma Center in the region Acute Care Beds 912 % Of Occupancy 88.0 Discharges 46, 402 Days 258,983 Ambulatory visits 998,536 Emergency Visits 91,550 Surgeries 38,573 VUMC Information Systems 2005 Top 100 Most Wired Hospitals and Health Systems Informatics (Enterprise) - Clinical, Education, Research Information Systems Information Technology Network Computing Services IBM Processor (Enterprise) Multiple AIX Midrange Platforms (Enterprise) EMC SAN (Enterprise) 96/53 Tb Storage (Primary data center/secondary data center) 800+ servers throughout the medical center The Children s s Hospital of Philadelphia Acute Care Beds Licensed 410 % Of Occupancy 85% ave. Discharges (In Patient) 23,251 Discharges (Out Patient) 931,667 Days Outpatient Encounters Emergency Visits Operating Room Cases Inpatient Outpatient 3

4 CHOP Information Systems No Multiple AIX Midrange Platforms Over 280 Open System Medical Center Servers in Data Center Information Services Organizations: Business Solutions Organization (BSO) Client Services Organization (CSO) Core Vendor Strategy Team Information Security Group (InfoSec) Integration Solutions Organization (ISO) Support Center: Help Desk & Desktop Support Technical Services Organization (TSO) Caritas Christi Second Largest Healthcare System in New England 6 Hospitals Employees, 12,000 Hospital beds, 1,552 Doctors, 2,305 Nurses, 1,880 Annual inpatient discharges,73,546 Annual emergency department visits, 238,551 Communities served, 55 Baseline Considerations RECOVERY TIME OBJECTIVE: (RTO) The period of time in which systems, applications, or I.S. functions must be recovered ered after an outage. RTO's are often used as the basis for the development of recovery strategies, and as a determinant as to whether or not to implement the recovery strategies during a disaster situation. RECOVERY POINT OBJECTIVE: (RPO) The point in time to which systems and data must be restored after an outage. RPO's are often used as the basis for the development of backup strategies, and as a determinant of the amount of data that may need to be recreated after the systems stems or functions have been recovered. I.T. RTO and RPO Need To Be Negotiated With The Business Units.In Order To Provide A Level Set For User Expectations Regarding The Current Version Of The Data, The Timely Restoration Of The Systems, And For The Business Units To Accurately Develop Their Business Contingency Plans 4

5 Baseline Considerations * $$ 900, , , , ,000 Comparison of $$ Impact To Cost To Recover Cost of Impact 400,000 Optimum Recovery Point (i.e., Cost vs. Risk) 300, , ,000 50,000 0 * min 1 hr to 3 hrs to 12 hrs - 24 hrs - 48 hrs - 72 hrs - mins to 1 hr 3 hrs 12 hrs 24 hrs 48 hrs 72 hrs 96 hrs Cost of Recovery * Numbers Above Are For Demonstration Purpose Only Accurate Impact numbers can be determined from the Business Impact Analysis Distributed Processing Open & Central Systems Print Server Application Server CITRIX Server Pharmacy Database EPR Database Database Server Clinweb PID Clindata Sybase Server Clinweb Clindata Sybase Server Inpatient Reg Consider the size of the environment(s) ) and the cost to back them up Recovery Strategy Needs to Recognize Diversity Print Server Application Server CITRIX Server Database Server Pharmacy Database EPR Database Clinweb PID Clindata Sybase Server Clinweb Internal Solution Clindata Sybase Server External Solution Inpatient Reg Open Systems DR Model Centralized DR Model 5

6 Evolution To A Different Set Of Standards File Recovery data restore from local or offsite backups High Availability data and hardware restore to local standby equipment (i.e., test, QA, Dev) Disaster Tolerance Failover to available equipment internal to the organization, but remote to the primary site (i.e., test, QA, Dev other campus location) Disaster Recovery Traditional DR model with hotsite and offsite storage locations Business Resumption Combination of any or all of the above Note: Regardless of the strategy employed..backed up data should still be copied to offline media and rotated to offsite storage Different Environments, Tactics, & Controls Print Server Application Server CITRIX Server Database Server Clinweb Clindata Pharmacy Database EPR Database PID Sybase Server Clinweb Clindata Inpatient Reg Sybase Server Open Systems DR Model -Data Replication To Local Storage -Failover and/or Quick Recovery -Local Connections For High Volume -Local AND Remote Recovery Centralized DR Model -Traditional Offsite Storage -Hotsite Location Approx 100 Miles From Primary Site -T1 Connection Between Hotsite and Local Internal Solution I.T. Operating Environments Hopkins Multiple Platforms /Midrange/Servers Dispersed Locations Hospital/University/Clinical Practices/Satellite JHU Campuses Campus Geography JHH and JHU within 10 miles Outlying Locations Urban and Rural URMC Multiple Platforms /Midrange/Serv ers Centralized Locations Primary Data Center/Backup Data Center/Hospitals Strong Health Geography SMH, HH, and Data Centers within 5 miles Outlying Locations Urban, Suburban, and Rural 6

7 I.T. Operating Environments CHOP Multiple Platforms Midrange/Servers SAN/VMWare VMWare/Blades Dispersed Locations Hospital/Clinical Practices Campus Geography CHOP and Remote Clinical Practices within 100 miles Remote Data Center (RDC) Outlying Locations Urban and Rural Multiple Platforms VUMC, Midrange, Wintel Servers Centralized medical center campus Adult, Children s, Rehab and Psych hospitals with clinics Remote clinics; Life-Flight also serving southern Kentucky and northern Alabama Primary data center Remote 2 nd data center Medical Center information systems and data center are distinct from University s I. T. Operating Environments Caritas Christi Multiple Platforms One primary data center w/back up location Campus Geography Supports 6 hospital widely dispersed throughout eastern MA and Clinical Practices within 100 miles RPO s, RTO s, SLA s Internal or External Solutions Drivers Hopkins Platform Size Driven and Midranges at Hotsite Servers In-House Recognition of Soft Costs Existing Network Multiple Facilities Current Capacity/Utilization URMC Criticality / BIA Driven: Patient Safety Patient Quality of Care Workforce Productivity Financial Legal/Regulatory Reputation Service Level Agreements Educational Research Programs Mission Critical: Dedicated DR systems & data in backup data center Offsite media Business Critical: Contract for shipment Offsite Media 7

8 RPO s, RTO s, SLA s Internal or External Solutions Drivers CHOP Criticality: Patient Safety Ideal Patient Experience Financial Legal/Regulatory Reputation Mission Critical: Tiered Recovery Strategy Redundant Data Center Offsite vaulting of media Business Critical: Workspace Recovery Offsite vaulting of media VUMC Combination hot-site, mobile, drop ship Most feasible at the time Estimated recovery time hrs Major data synchronization issues Recovery Time Objective 0 4 hrs, RPO zero data loss Staff willingness to travel at time of disaster questionable Dec. 03 Nov 04 (and beyond) Migrated to Internal co-location solution Lease 2 nd data center space and facilities (HVAC, power) Lease dual-path redundant fiber connection between primary data center and 2 nd data center Own and maintain redundant hardware Own and maintain internal network infrastructure Recovery time consistent with RTO & RPO RPO s, RTO s, SLA s Internal or External Solutions Drivers Caritas Christi Upcoming Review-Criticality Patient Safety Patient Quality of Care Workforce Productivity Financial Legal/Regulatory Reputation Service Level Agreements Educational Research Programs Underlying Factors for Internal vs. Outsourcing Solutions Hopkins RTO RPO Cost Of Recovery Impact Of Recovery URMC Mission Critical RTO 2 to 24 hours RPO 0 to minimal data loss BIA Downtime Impact vs. Time to Recover Costs Dedicated DR Hardware DR Contract for Shipment 8

9 Underlying Factors for Internal vs. Outsourcing Solutions CHOP Mission Critical RTO RPO 0-72 hours 0 to 36 hours Tier Recovery Strategy BIA confirmed Mirrored redundant equipment, applications and data VUMC RPO and RTO Critical systems RTO 0-4 hours RPO Viability Zero data loss Cost of Downtime (impact and $$$) vs. Cost to Recover Dedicated hardware Greater sense of Ownership Underlying Factors for Internal vs. Outsourcing Solutions Caritas Christi RTO RPO Cost Of Recovery Impact Of Recovery Testing: Hot, Warm, Cold Internal vs. Outsourcing Hopkins Combination Relocation to hotsite 2 per year Servers tested throughout the year validated during hotsite tests Customers test locally High volume applications ( , images, etc.) test locally URMC Mission Critical Systems 2 per year Business Critical Systems 1 per year Systems tested throughout the year: Hardware and Software Upgrades, project implementations, etc. Department Downtime Procedures Annual Testing 9

10 Testing: Hot, Warm, Cold Internal vs. Outsourcing CHOP Combination depending on Tier structure Tier 1 applications 2x a Yr.- redundant, mirrored at RDC (maintenance and testing performed) Tier 2 applications 1x a Yr. - hardware available, data on site at RDC Tier 3 applications TBD - data vaulted, restore on need Departmental Downtime Procedures tested at least annually VUMC Not as easy to find test window as originally thought Some downtime necessary Coordination is major headache Proposed: 2 x per year (stand-alone alone and integrated Incidents or maintenance failovers qualify as test End user involvement New systems to be tested prior to implementation (novel concept!!) Testing: Hot, Warm, Cold Internal vs. Outsourcing Caritas Christi Systems tested throughout the year Hardware and Software Upgrades, project implementations, etc. Department Downtime Procedures Annual Testing Maintenance: Hot, Warm, Cold Internal vs. Outsourcing Hopkins All maintenance issues validated during hotsite tests In-House & External Contracts Documentation Contact Lists Skill Sets Notification, Escalation, Declaration Procedures URMC Disaster Recovery System and Program Maintenance validated through: In-House DR system testing, IT and End-User testing Technical Documentation Audits Skill Set Evaluations Internal and External Auditors EOC and ISD Command Center Team Exercises Change Management Project Management Incident Debriefing Sessions 10

11 Maintenance: Hot, Warm, Cold Internal vs. Outsourcing CHOP Disaster Recovery and Program Maintenance validated by: In-House DR Testing RDC Testing Internal and External Audits Downtime Procedures with Clients VUMC Contingency Planning as core element in project/system evaluation, development user acceptance (quality) testing and implementation Testing Configuration Management Change Management Problem Management Incident Response Internal Audit Maintenance: Hot, Warm, Cold Internal vs. Outsourcing Caritas Christi Disaster Recovery Program currently undergoing being moved to internal solution RTO & RPO Considerations Negotiate The Service Level Agreement Between I.T. And Business Operations: Use Both The I.T. And Business RTO & RPO As The Basis Disaster Recovery Plan Test Results Quantify Timelines Business Contingency Plan Exercises Qualify Impact I.T. Capabilities Improve Timelines But At A Cost Business Contingencies Reduce Impact - But Require I.T. Capabilities Criticality Rankings Systems Recovery Sequencing Business Process Prioritization I.T. and Business Process Timelines Negotiated RTO and RPO 11

12 Results I.T. Better Understands the Customers Issues and Requirements Obtains A Clearly Documented Set of Customer Expectations for DRP s Clarify and Justify Budget Forecasts Establishes Specific Test Objectives Ensure Active Customer Involvement in Testing & Recovery Processes Business Units Better Understands the Role Of I.T. in the Contingency Process Obtains a Set of Parameters from which to Develop Their BCP s Workaround Procedures During Downtime Procedures For Capturing Lost Transactions From Downtime and During Recovery Restoration of Normal Environments Everyone works towards a common interest, that that of of ensuring that that the the business processes of of the the organization, its its mission, goals, and and objectives..and possibly the the community at at large are are protected Questions To Consider Was the original disaster recovery initiative driven by I.T., business units, or Sr Management? What are Sr. Management s s expectations with respect to continuity of service? Has a business impact analysis been done on some or all of the business units? Quantified Impact Quantified Cost of DRP vs. Impact of Risk Acceptable Downtime Criteria (services, workstations, etc.) What discussions have taken place between I.T. and critical business units? State of DRP State of BCP Quantified RTOs and RPOs Systems Development Life Cycles Questions To Consider What are the business units expectation with respect to current I.T. RTOs and RTOs? Are they driven by I.T. technologies or business requirements? Are there current SLAs? Service Center Problem/Change Control Network Outage Response Time Are regulatory compliance, industry certification, or audit issues creating more compelling reasons for addressing DRP and BCP? 12

13 Baseline Considerations Cost of Impact 900, , , , , , , , ,000 50, min 1 hr to 3 hrs to 12 hrs - 24 hrs - 48 hrs - 72 hrs - mins to 1 hr 3 hrs 12 hrs 24 hrs 48 hrs 72 hrs 96 hrs Without BCP Cost of Recovery 900, , , , , , , , ,000 50, min 1 hr to 3 hrs to 12 hrs - 24 hrs - 48 hrs - 72 hrs - mins to 1 hr 3 hrs 12 hrs 24 hrs 48 hrs 72 hrs 96 hrs Cost of Impact With BCP Cost of Recovery Numbers Are For Demonstration Purposes Only Seeking Your Contingency Plan Are You: HOT, COLD,, or WARM? Questions? 13