White Paper. Lifecycle Disaster Recovery Costs

Transcription

1 White Paper Lifecycle Disaster Recovery Costs

2

3 Lifecycle Disaster Recovery Costs Do you really understand the costs to a financial institution for IT Disaster Recovery? Most professionals working in a bank or credit union environment know that implementing a disaster recovery solution for a financial institution is very different than doing so for an unregulated business. Financial institutions must work to guidelines and standards that are much more specific than general rules that are typically established for IT disaster recovery. In addition to being more specific, the guidelines and standards call for building and managing an end-to-end process that includes annual testing of a business continuity plan and recovery of IT systems. This white paper helps you identify the various elements that need to be put in place, and the associated costs of these elements, in order to have a compliant, functional and financially viable IT disaster recovery program. The Right Approach to a Disaster Recovery Program Most organizations think of IT disaster recovery the wrong way. IT professionals almost always start with hardware and software, with the goal of protecting data during a disruption. Financial institutions must take the opposite approach and ask what they need to do in order to restore normal operations for their banking customers or credit union members. The right approach requires a Business Continuity Plan (BCP) and Business Impact Analysis (BIA) that calls out what functions and operations within an institution must be restored, and at what timeframe following a disaster or disruption. The FDIC and NCUA (via the FFIEC) are very specific about calling out business operations or functions that must be recovered with a maximum allowable downtime (MAD) that falls into one of five categories 1 : Nonessential Normal Important Urgent Critical 30 days 7 days 72 hours 24 hours minutes to hours Only after the various business operations or functions of a financial institution have been categorized with a MAD can an effective disaster recovery program and set of technologies be put in place. As you can see applying a one size fits all approach with a single set of technologies is not sufficient from a compliance standpoint, nor is it best practice or cost effective. A Compliant Disaster Recovery Program The requirements for restoring normal bank and credit union operations are what drive disaster recovery lifecycle costs and compliance. Assuming you want to deliver a compliant disaster recovery program for your institution, there are several program elements that have nothing to do with hardware and software that must be developed, maintained and tested annually. These elements consist of people, processes and toolsets necessary to build, manage and maintain a compliant disaster recovery program. These are also things that need to be actively managed, maintained and exercised throughout the year in order for a financial institution and its regulators to be confident that the institution can recover operations after a disruption. 1 Source: FFIEC IT Examination Handbook, Business Continuity Planning, March 2008, Appendix F, p. F-3 1

4 The elements outside of hardware, software and facilities essential to a compliant disaster recovery program are: recovery plan development, testing (planning, preparation, execution and reporting), post test analysis, and disaster recovery program management. Recovery Plan Development Taking a BCP and BIA and turning them into a compliant and cost effective disaster recovery program is no simple task. Most institutions today have IT systems that support most if not all business operations or functions, yet are dependent on other systems or data to operate correctly. As a result having a plan that clearly articulates what systems must be recovered and in what order is critical if an organization is to resume normal operations within a specific timeframe. The System Recovery Plan starts with the institution s map of business processes and functions, and their supporting IT systems, that must be restored according to the maximum allowable downtime categories assigned in the institution s BIA. This map enables the organization responsible for disaster recovery (either the IT department or third party managed services organization like IT-Lifeline) to establish recovery point objectives (RPOs) and recovery time objectives (RTOs) for the identified systems which spell out how quickly these systems must be recovered in order to support the BCP and BIA. Once RPOs and RTOs are established for all IT systems, a recovery plan can be developed which spells out how the organization will restore normal operations after a disruption. A recovery plan includes all of the steps required to bring systems back on-line and reestablish functionality for system end-users. Recovery plans typically include processes and procedures for dealing with recovery of normal operations for branch office facilities, main office facilities, and hardware, software and communications networks. Recovery plan development is a complex and specialized activity. Most mid-sized banks and credit unions do not have the experience or expertise to develop and maintain a compliant and effective recovery plan without assistance from a third party. Third party disaster recovery experts develop and maintain recovery plans (in addition to doing all aspects of testing and recovery) for many banks and credit unions, and as a result understand the compliance requirements and best practices, and have templates available for use during recovery plan development. This third party experience is invaluable during plan development and can significantly reduce associated costs and time. Figure 1 Recovery plan development can be shortened and typically costs less when utilizing third party accumulative experience. Testing Recovery Plan Development Financial institutions should test recovery plans at least once per year. Examiners look for testing on a specific cadence and want to see the results of annual, progressive testing. However, in many cases these annual tests do not happen due to the resource burden placed on small IT staffs that are busy keeping things running. To test a recovery plan in a way that ensures compliance and the ability to restore operations involves several distinct phases: 1. Planning 2. Preparation 3. Execution 4. Reporting In-House Resources Total number of procedures 2-5 Time required to develop procedures (20 hours per procedure) FTE requirements Hours (On-going) Each of these phases has a specific set of activities and associated deliverables. And each phase should be documented in detail to both capture results and to show examiners that the institution is following compliance best practices. The Planning phase involves developing a test plan which outlines the specific business operations or functions to be restored, and the associated IT systems and personnel involved in the recovery. The Preparation phase involves scheduling the test including the resources, people and facilities required to support a successful recovery test. The Execution 2

5 phase is conducting the test, which typically takes place over the course of one or two days, depending on the business operations or functions to be recovered. And finally, the Reporting phase involves gathering all of the test results into a format that clearly spells out the areas that were tested, both the positive and negative results and outlines very specific remedial actions required to address any issues or failures associated with the test. There is a significant amount of work involved with a compliant testing program. Most mid-sized banks and credit unions have neither the personnel nor expertise to successfully deliver all four phases of a test. Disaster recovery specialists like IT-Lifeline can be used to leverage tools and resources for more complete testing at significantly lower costs. Figure 2 Testing phases can be shortened by leveraging IT-Lifeline s tools and resources for more complete testing. Testing Phases Post Test Analysis In-House Resources Time for test planning (hours) 50 Engineers required for test startup (per shift) Test team required for test Setup and teardown of environment (hours) 2-5 Intel Engineers, 2-5 Mid-range Engineers 5-7 resources away from home site for 3 days It is also best practice to take the results of an annual or biannual test and perform a detailed post test analysis. The analysis serves to identify areas of deficiency, and to come up with remediation plans to correct the deficiencies. This often comes in the form of missed RTOs with specific systems, or partial recovery of a system. Sometimes post test analysis also reveals deficiencies in alerting or personnel skill sets required to successfully recover from a disruption. These deficiencies are often corrected with implementing processes to regularly incorporate 80 FTE requirements 1.7 system updates and patches into recovery plans. The post test analysis also serves to bridge the gap between the prior test and upcoming test, and results of the post test analysis are often used as direct updates to a recovery plan. Working with a disaster recovery specialist like IT- Lifeline will greatly reduce the time and effort required to produce a post test analysis. Templates developed from hundreds of tests are used as a basis for documenting the test results and serve to streamline the analysis and reporting process. In addition, personnel that are dedicated testing and reporting resources serve to fill any skill set gaps that typically exist at mid-sized financial institutions. Figure 3 Post Test Analysis can be shortened by leveraging IT-Lifeline s tools and resources. Post Test Analysis Time required for post test reporting and analysis (hours) Program Management In-House Resources FTE requirements.06 One of the biggest changes in the regulatory environment relating to disaster recovery that has taken place over the past few years is the recognition that disaster recovery planning, including BCPs, BIAs, and recovery plans, and the resulting provisioning of IT equipment are no longer static. Examiners understand that technology is changing much more quickly than 10 years ago, and as a result business processes and organizational functions are changing. Customers are relying on IT systems for banking at home, at work and on mobile computers and phones. This increased rate of change is driving the need to continuously evaluate, update, monitor and test disaster recovery programs. This requires an actively managed lifecycle approach which means disaster recovery is an ongoing rather than static set of activities. Expertise in plan development, testing and post test analysis will greatly simplify the effort required to actively manage a disaster recovery program. In addition, the right toolset for making copies of data and systems greatly simplifies the need to deal with IT system update and patch management processes. 3

6 Working with a company like IT-Lifeline can greatly reduce the amount of time required to manage a compliant IT disaster recovery program. Figure 4 Program management can be shortened by leveraging IT-Lifeline s refined tools and deep expertise. monitor performance of backup and recovery hardware. Software Software Purchase: one time costs for acquisition of backup and recovery software. Maintenance: annual software maintenance fees. Program Management In-House Resources Communications Communications Link: provisioning costs for bandwidth required between primary and remote site. Time spent by program manager on program management Time spent by network, platform, DBA an app evaluating changes and making updates (hours per week) FTE requirements The Traditional Hardware, Software, and Facilities The easiest disaster recovery program costs to quantify are the things that IT personnel are the most comfortable with: the hardware and software. But as discussed above, hardware and software implemented in an environment that is governed by regulatory compliance requirements must be dictated by the organization s BIA, and the resulting needs to resume business operations within a certain time. Hardware and software implementation must also enable a bank s or credit union s ability to plan, test, analyze and manage a disaster recovery program. If the hardware and software selected can do these things, and the organization has a lifecycle management approach for its disaster recovery program, then the institution has a disaster recovery program that will ensure compliance requirements are met and work in times of crisis. Choices in this area are plentiful, with most major hardware OEMs providing storage, computers and software which will enable a financial institution to provision a disaster recovery platform. However, providers of disaster recovery services to banks and credit unions have proven that specializing in storing data and providing a shared resource pool for recovery operations offers an economy of scale that no mid-market bank or credit union can match. In addition, providers such as IT-Lifeline offer an additional layer of security and protection via specialized certifications such as an AT 101 SOC2 SM Type II attestation. Elements of hardware and software that must be included in any disaster recovery cost analysis are: Hardware Space: data center space required for servers and storage in a rack at a remote facility. Power: power required for the rack. Servers: server capacity required for remote facility. Storage: storage required for remote facility. Maintenance: equipment maintenance fees. Management & Reporting: toolset required to 4

7 Lifecycle Disaster Recovery Worksheet Use this basic worksheet to help determine the estimated total costs for managing and maintaining an in-house disaster recovery solution. Compare the cost against a third party vendor like IT-Lifeline who typically can provide a fully managed and compliant solution at a cost less than most can do themselves. Total Cost of Ownership In-House Costs Third Party Costs People Costs Application mapping (Non-recurring) Procedure development (Non-recurring) Test planning Post test analysis Program management Total non-recurring costs Total annual costs Putting it all together To really understand the costs associated with a compliant and functional disaster recovery program, a bank or credit union needs to account for all of the program elements. Annual Hardware Costs Space Power Servers Hardware maintenance Management and monitoring Total annual costs Vaulting Software Costs Vaulting software Vaulting appliances (Non-recurring) Management and monitoring Network Backup space and power Total non-recurring costs Total annual costs TOTAL NON-RECURRING COSTS TOTAL ANNUAL COSTS 5

8 Get protected ) Get protected by the trusted leader in cloud based disaster recovery and compliance services to community banks and credit unions.