William Rider Manager Disaster Recovery & Data Security The Johns Hopkins Health System & University

Transcription

1 William Rider Manager Disaster Recovery & Data Security The Johns Hopkins Health System & University

2 Competitive Leadership- Twelve Principles For Success Brian Billick Chapter 3 Be Be Prepared The time to repair the roof is when the sun is shining John F. Kennedy Spectacular achievements are always preceded by unspectacular preparation Roger Staubach One of life s most painful moments comes when we must admit that we didn t do our homework, that we are not prepared Merlin Olsen Failure to prepare is preparing to fail John Wooden You must be able to respond to your circumstances as they exist not as you would like them to be Brian Billick

3 The Business Continuity Management Program The frequency of events, both manmade and natural are occurring at an alarmingly frequent rate.. And they are affecting more areas of the organization including I.T. Just as the impact of such events is being more severely felt in the I. T. Department, so it is also negatively affecting the business process of the organization, its mission, goals, and objectives..and possibly the community at large.

4 The interruption of fundamental business processes for any extended period of time could have a debilitating affect on our basic infrastructure.and our way of life E-Commerce Private and Business Online Trading Cash Advances at ATM Machines Personal and Commercial Online Banking Retail Purchases by Credit Cards Just In Time Inventories Communications Inpatient/Outpatient Hospital Registrations Pharmacy Refills Insurance Claims and Explanation of Benefits

5 It s not a Private Sector issue..or a Public Sector issue. The solution lies in the collaboration between Private and Public Sector to resolve these issues and the convergence to a single strategy with a consistent set of tactical solutions. Public Sector First Responders Emergency Medical Technicians Fire & Police Federal, State and Local Government Private Sector Healthcare Financial Manufacturing Retail Transportation

6 ERP DRP CMP BCP Working Definitions ERP Emergency Response Plan: Steps taken to immediately respond to an event, ensure personnel safety, minimize further impact to assets, and make proper notifications. DRP Disaster Recovery Plan: Steps taken to restore specified infrastructure requirements such as Information Systems, business equipment environments, internal and external network connections, and data structures utilizing alternate resources for hardware, software, data, and networks. CMP Crisis Management Plan: Steps taken to manage the event to ensure that order is maintained, proper information is being disseminated by appropriate representatives, action items are effectively escalated, and ongoing internal and external notifications are consistent. BCP Business Contingency Plan: Steps taken to restore alternate business processes in the event that automated processes or business infrastructures are unavailable, employing documented workaround and/or manual procedures and alternate resources.

7 Information Services DRP Disaster Recovery What To Do When The Computer Goes Down Those steps required to ensure recovery of critical systems Business Entities BCP Business Contingency What To Do While The Computer Is Down The steps required to ensure continued operations of critical processes

8 Considerations That Disaster Recovery Planning, Business Contingency Planning, Emergency Response Planning, and Crisis Management Planning are components of a much larger business strategy for a Continuity of Operations. That Business Contingency Planning is not and I.T. function. It is a function of business operations. A concept that must become a part of the corporate culture of an organization. That the attitudes and behaviors of those in the organization must reflect the importance of protecting people, process and assets as paramount to the organization. That a top-down commitment to Disaster Recovery Planning, Business Contingency Planning, Emergency Response Planning, and Crisis Management Planning must be demonstrated through ongoing training and awareness, clear and enforceable policies and standards, and operational efficiencies.

9 DRP DRP Disaster Recovery Plan: Steps taken to restore specified infrastructure requirements such as Information Systems, business equipment environments, internal and external network connections, and data structures utilizing alternate resources for hardware, software, data, and networks. - Hardware - System Software - Data and Data Structures - Applications - Networks - Desktop Services - Production Support

10 I.S. Recovery Time Requirements RECOVERY TIME OBJECTIVE: (RTO) The period of time in which systems, applications, or I.S. functions must be recovered after an outage. RTO's are often used as the basis for the development of recovery strategies, and as a determinant as to whether or not to implement the recovery strategies during a disaster situation. RECOVERY POINT OBJECTIVE: (RPO) The point in time to which systems and data must be restored after an outage. RPO's are often used as the basis for the development of backup strategies, and as a determinant of the amount of data that may need to be recreated after the systems or functions have been recovered.

11 Disaster Recovery Planning Operational Impact Analysis Applications Criticality Analysis Data Backups & Offsite Storage Recovery Teams & Recovery Times Objectives Testing & Maintenance

12 Disaster Recovery Planning Considerations Hardware Platforms Application Services Alternate Resources Network Connectivity Data Synchronization Systems Interoperability

13 DRP Approach Prioritize critical applications with consideration given to recovery time and recovery point objectives, while recognizing the business/operational impacts. Identify key infrastructure components to establish computing environment. Build recovery plans with testing performed at component and system level. Communicate RTOs and RPOs and validate User expectations.

14 I.S. Technology Recovery Four Phased Approach Emergency Management Coordinate Control Fund Approve Documentation Alternate Resources Available Data Response Recovery Resumption Restoration Immediate Actions -Personnel Safety -Damage Mitigation -Notifications Procedures -Hardware -Software -Data -Telecomm Systems Networks Interoperability Validation Refurbish Replace Construct Return

15 Interfaces & Dependencies Data Backup Synchronization 2:00am 3:00am 4:00am 5:00am 6:00am 7:00am 8:00am 9:00am 10:00am11:00am12:00am1:00pm Data Backup Synchronization Point Platform #1 Platform #2 Platform #1 Platform #2 Gateway Srvr #1 Gateway Srvr #2 Interface Engine Interface Engine DB2 DB2 App l Data App l Data Current Backups Proposed Snapshots Snapshot To Tape Physical Rotation To Offsite

16 * $$ 900, , , ,000 Comparison of $$ Impact To Cost To Recover Cost of Impact 500, ,000 Optimum Recovery Point (i.e., Cost vs. Risk) 300, , ,000 Cost of Recovery 50,000 * min 1 hr to 3 hrs to 12 hrs - 24 hrs - 48 hrs - 72 hrs - mins to 1 hr 3 hrs 12 hrs 24 hrs 48 hrs 72 hrs 96 hrs * Numbers Above Are For Demonstration Purpose Only Accurate Impact numbers can be determined from the Business Impact Analysis

17 BCP BCP Business Contingency Plan: Steps taken to restore alternate business processes in the event that automated processes or business infrastructures are unavailable, employing documented workaround and/or manual procedures and alternate resources. - Relocation of Personnel - Availability of remote support services and network connections - Contingency office space

18 Business Recovery Requirements RECOVERY TIME OBJECTIVE: (RTO) When do I.T. systems need to be available to the end users? RECOVERY POINT OBJECTIVE: (RPO) How current does the information have to be when systems and services are made available?

19 Business Contingency Planning Business Contingency Planning Business Impact Analysis Business Function Prioritization Manual Procedures Business Unit Contingency Planning Crisis Management Teams

20 Contingency Planning Considerations Business Cycles Personnel Skill Sets Alternate Personnel Alternate Resources Alternate Communications Human Resources Manual Procedures Alternate Locations Alternate Process Vendors

21 BCP Approach Prioritize Business Functions Identify Key Components Of Each Process Identify Risk Scenarios Loss Of I.T. Services Loss Of Network Connectivity Loss of Facility Loss of Personnel Loss of Supply Chain Loss of Inter/Intra Departmental Support Build Business Contingency Plans Exercise The Plans Validate BCP Test Results with I.T. RTOs and RPOs

22 Business Contingency Planning Crisis Management Emergency Response Evacuation Communication Four Phased Approach Emergency Management Coordinate Communicate Activate Resume Documentation Alternate Resources Available Data Reaction Relocation Resumption Restoration Logistics Location(s) Transportation Personnel Facilities Vendors Hardware/Software Communications Procedures Logistical Support Forms Contact Lists

23 2:00am 3:00am 4:00am 5:00am 6:00am 7:00am 8:00am 9:00am 10:00am 11:00am 12:00am 1:00pm Data Backup Synchronization Point Process #1 Process #2 Interfaces & Dependencies Data Backup Synchronization Process #1 Process #2 Interdependent Processes Interdependent Processes Supply Chain Supply Chain Work In Process Work In Process Input Transactions Input Transactions Current Backups Proposed Snapshots Snapshot To Tape Physical Rotation To Offsite

24 * $$ 900, , , , ,000 Comparison of $$ Impact To Cost To Recover Cost of Impact 400,000 Optimum Recovery Point (i.e., Cost vs. Risk) 300, , ,000 50,000 * min 1 hr to 3 hrs to 12 hrs - 24 hrs - 48 hrs - 72 hrs - Cost of Recovery mins to 1 hr 3 hrs 12 hrs 24 hrs 48 hrs 72 hrs 96 hrs * Numbers Above Are For Demonstration Purpose Only Accurate Impact numbers can be determined from the Business Impact Analysis

25 Window Of Exposure Without BCP 900, , , , , , , , ,000 50, min 1 hr to 3 hrs to 12 hrs - 24 hrs - 48 hrs - 72 hrs - mins to 1 hr 3 hrs 12 hrs 24 hrs 48 hrs 72 hrs 96 hrs Without BCP Cost of Impact Cost of Recovery Numbers Are For Demonstration Purposes Only

26 Window Of Exposure With BCP 900, , , , , , , , ,000 50, min 1 hr to 3 hrs to 12 hrs - 24 hrs - 48 hrs - 72 hrs - mins to 1 hr 3 hrs 12 hrs 24 hrs 48 hrs 72 hrs 96 hrs Cost of Impact With BCP Cost of Recovery Numbers Are For Demonstration Purposes Only

27 900, , , , , , , , ,000 50, min 1 hr to 3 hrs to 12 hrs - 24 hrs - 48 hrs - 72 hrs - mins to 1 hr 3 hrs 12 hrs 24 hrs 48 hrs 72 hrs 96 hrs Cost of Impact Without BCP Cost of Recovery 900, , , , , , , , ,000 50, min 1 hr to 3 hrs to 12 hrs - 24 hrs - 48 hrs - 72 hrs - mins to 1 hr 3 hrs 12 hrs 24 hrs 48 hrs 72 hrs 96 hrs Cost of Impact With BCP Cost of Recovery Numbers Are For Demonstration Purposes Only

28 I.S. Recovery Time Requirements RECOVERY TIME OBJECTIVE: (RTO) The period of time in which systems, applications, or I.S. functions must be recovered after an outage. RTO's are often used as the basis for the development of recovery strategies, and as a determinant as to whether or not to implement the recovery strategies during a disaster situation. RECOVERY POINT OBJECTIVE: (RPO) The point in time to which systems and data must be restored after an outage. RPO's are often used as the basis for the development of backup strategies, and as a determinant of the amount of data that may need to be recreated after the systems or functions have been recovered. Business Recovery Requirements RECOVERY TIME OBJECTIVE: (RTO) When do the I.T. Systems and Services have to be available? RECOVERY POINT OBJECTIVE: (RPO) How current does the information have to be when systems and services are made available?

29 Negotiate The Service Level Agreement Between I.T. And Business Operations Use Both The I.T. And Business RTO & RPO As The Basis Disaster Recovery Plan Test Results Quantify Timelines Business Contingency Plan Exercises Qualify Impact I.T. Capabilities Improve Timelines But At A Cost Business Contingencies Reduce Impact - But Require I.T. Capabilities Criticality Rankings Systems Recovery Sequencing Business Process Prioritization I.T. and Business Process Timelines Negotiated RTO and RPO

30 Results I.T. Better Understands The Customers Issues and Requirements I.T. Obtains A Clearly Documented Set Of Customer Expectations For DRP s - Clarify and Justify Budget Forecasts - Establishes Specific Test Objectives - Ensure Active Customer Involvement In Testing & Recovery Processes Business Units Better Understand The Role Of I.T. In The Contingency Process Business Units Obtain A Set Of Parameters From Which To Develop their BCP s - Workaround Procedures During Downtime - Procedures For Capturing Lost Transactions From Downtime and During Recovery - Restoration Of Normal Environments Everyone in in the the organization works towards a common interest, that that of of ensuring that that the the business processes of of the the organization, its its mission, goals, and and objectives..and possibly the the community at at large.are protected

31 The Business Continuity Management Program When the issues surrounding both I.T. Disaster Recovery Plans and Business Unit Business Contingency Plans come together what is at stake becomes much clearer, and each can understand the others objectives and expectations. Only then can a total Business Continuation Program be effective. And if the organization has an effective Business Continuation Program, not only can it assure that its goals and objectives will be met..but will also become a valued partner in the protection of the larger infrastructure..

32 Questions/Issues to consider: Was the original disaster recovery initiative driven by I.T., business units, or Sr Management? What are Sr. Management s expectations with respect to continuity of service? Has a business impact analysis been done on some or all of the business units? Quantified Impact Quantified Cost of DRP vs. Impact of Risk Acceptable Downtime Criteria (services, workstations, etc.) What discussions have taken place between I.T. and critical business units? State of DRP State of BCP Quantified RTOs and RPOs Systems Development Life Cycles What are the business units expectation with respect to current I.T. RTOs and RTOs? Are they driven by I.T. technologies or business requirements? Are there current SLAs? Service Center Problem/Change Control Network Outage Response Time Are regulatory compliance, industry certification, or audit issues creating more compelling reasons for addressing DRP and BCP?