Business Continuity Planning

Transcription

1 Business Continuity Planning Examples of Risk Analysis Workshop Outcomes Solution Summary and Case Studies By Dennis Wenk and Kathy Macchi

2 Executive Summary Evaluating business continuity plans requires expertise that few organizations have in-house. IT staff may lack the methodology for systematic analysis, sometimes letting emotional factors interfere with what should be business decisions. Even worse, although most organizations understand the immediate impact of a disaster, they may overlook more likely-to-occur events that can have a far more severe impact. Through the Risk Analysis Workshop, Hitachi Data Systems Global Solution Services makes its extensive experience available to organizations facing these complex business continuity issues. The Workshop s unique approach helps IT assess the resilience of their environment and prioritize the organization s plans for resolving gaps. Although every organization s issues are unique, common underlying issues are illustrated in the nine Workshop experiences summarized in this document.

3 Contents Introduction... 1 Risk Analysis Workshop Approach and Methodology... 2 The Objective... 2 The Team... 2 The Process... 2 Workshop Deliverables and Benefits... 2 Quantitative Operational Risk Assessment... 3 Workshop Examples... 3 Heterogeneous Platform Consistency... 3 Data Consistency... 4 Prioritization of Business Continuity Efforts... 4 Critical Resource Dependencies... 4 Minimizing Business Risk of Consolidating Data Centers... 5 Cost Justification... 5 Compliance... 6 Conclusion... 6

4 Business Continuity Planning Examples of Risk Analysis Workshop Outcomes Solution Summary and Case Studies By Dennis Wenk and Kathy Macchi Introduction In your business continuity planning, are you grappling with issues, such as: How much protection do I need to minimize risk? What is the payback for the investment required? How can I justify the expense to our executives? Are our current risk management plans adequate? Do our internal controls satisfy government regulations? Do we even have the expertise required to evaluate our plans rigorously? Are we overlooking a higher-level impact, especially in the area of compliance? How can we bypass political issues to make objective, business-based decisions? How can we pull the team together to move forward? Most organizations have unknowingly accepted much greater operational risk than they realize. Recent catastrophic incidents ranging from Hurricane Katrina to the Enron scandal show how severe the results can be. To protect investors, new government regulations require internal controls for managing risk. Yet IT management is often unsure how to approach this complex issue. Hitachi Data Systems Global Solution Services (GSS) offers the Risk Analysis Workshop to help organizations answer these questions and more. In the Workshop, Hitachi Data Systems consultants guide organizations through a systematic evaluation to identify gaps in their risk management. This methodical approach removes emotions from the discussion, helping organizations focus on the business issues and reach consensus on their priorities for going forward. Following the Workshop, an organization can also engage Hitachi Data Systems for the Quantitative Operational Risk Assessment service, during which GSS consultants design the optimal solution to close material gaps identified in the Workshop and prepare an in-depth cost/benefit analysis to justify the investment required. This program helps organizations make more informed decisions about their business continuity investment, and it can also help them improve the quality of service they provide to their organization. IT executives consistently report that the Risk Analysis Workshop s unique interactive approach yielded invaluable guidance for their organization s business continuity planning and helped them quickly reach consensus on a broad range of issues. 1

5 Risk Analysis Workshop Approach and Methodology The Objective Develop a gap analysis of the organization s environment where they are versus where they need to be. The Team Assemble a multidisciplinary group from across the organization. A variety of participants ensures that multiple perspectives are represented. Team members learn how the issues are interrelated and see their relative business impact. This helps the group focus on the core issues for the organization rather than individual department agendas. The Process The Workshop activities require two full days. The team develops a gap analysis by evaluating their organization s controls, processes, and procedures for risk management compared to a standardized approach. They compare the relative effectiveness and importance of their current state to their desired state for each of 34 topics (such as interruption impact, environmental issues, synchronization points, and offsite storage) within these five categories: Economic and regulatory Continuity background Infrastructure Data Recovery facilities The team also identifies the risks of not making any needed improvements and the obstacles to getting the job done. Workshop Deliverables and Benefits Within a week, Hitachi Data Systems delivers an executive briefing reflecting the Workshop findings: Diagnostic scorecard shows the ratings for the 34 areas, measuring them for status, quality, process, technology, and exposure. Grid analysis depicts the organization s strengths and deficiencies via Likert charts, ranking major limitations that need to be addressed. Recommendations describe options for getting to the desired state, including high-level designs and cost estimates. 2

6 In addition to the standard deliverables, the Workshop experience provides intangible benefits, which many organizations consider to be as valuable as the findings themselves: Insight and shared awareness of key issues and an appreciation of their interrelatedness Interdisciplinary consensus on priorities and next steps Help making decisions for the right business reasons Most important of all, the Workshop consistently helps organizations identify potential high-level considerations that they have overlooked. Compliance issues, in particular, are not well understood and could lead to serious consequences if left unaddressed. Quantitative Operational Risk Assessment Hitachi Data Systems and GSS encourage organizations to follow the Workshop with the Quantitative Operational Risk Assessment service. During a Quantitative Operational Risk Assessment engagement, GSS consultants propose alternatives, evaluating their effectiveness and analyzing their cost/benefits in-depth to determine the optimal solution for the organization. The Quantitative Operational Risk Assessment service is beneficial for: Making more informed decisions about business continuity investments Improving the quality of service IT provides Workshop Examples Although the complexity of IT environments makes each situation unique, the Risk Analysis Workshop addresses many common concerns. The experiences described below illustrate these key issues: Consistent practices across heterogeneous platforms Data consistency between sites Prioritization of business continuity investments Resource dependencies of critical business functions Balancing the risks and benefits of data center consolidation Investment cost justification Compliance Heterogeneous Platform Consistency Large Bank in Africa The key Workshop finding for the bank s participants was that they had placed their entire operation at risk by implementing inadequate protections for their open systems environment. Their mainframe business continuity plans were sufficient, but their remote procedures for the two environments were inconsistent. Key interdependencies between the two had not been fully identified, allowing the plans to be out of balance. Budgetary issues had taken priority over business continuity requirements, and, consequently, the whole environment was inadequately protected. 3

7 The bank s participants decided to obtain the technology necessary to resolve the open systems exposure. Data Consistency Large Retailer and E-commerce Site in Latin America A critical finding from the Workshop for the retailer s executives was that their business continuity plans were inadequate. Significant financial losses were likely to occur if they continued with the status quo. Such losses could, in turn, cost the executives their positions with the organization. The retailer s Workshop participants concluded that they must rethink their entire offsite strategy for data storage. They realized that their recovery site was inadequate and that there were data consistency issues between the sites due to the frequency of copying the data. The financial impact of an outage during Christmas shopping, their peak risk period, would be tremendous, so it was absolutely necessary that they get prepared to handle the seasonal spikes and surges. Following the Workshop the company improved overall recovery capability by acquiring redundant storage for remote copy/data replication and more compute power for the new recovery site. In addition to advising the retailer about their long-term business continuity plans, Hitachi Data Systems recommended temporary mitigating actions to minimize the possibility of interruptions during the upcoming Christmas season until the long-term recovery site enhancements were complete. Prioritization of Business Continuity Efforts Title Insurance Company The title insurance company hired a new executive to lead their business continuity planning department. He came to the Risk Analysis Workshop for assistance in prioritizing future efforts for the company s several data centers. Prior to the Workshop, the department s focus had been solely on major catastrophic failures, such as hurricanes (one data center is in Jacksonville, Florida). The new executive knew that it no longer takes a flood or hurricane to incur significant financial losses from an interruption of IT services. He knew they must figure out how to shift their focus from the potential losses from major catastrophes to expected losses from more ordinary failures. The team concluded they had been ignoring problems they were likely to encounter those causing a short but very costly period of downtime while overspending to protect against catastrophes that might never happen. In a very important way, the outcome of the Workshop for the customer went beyond identifying gaps. The process helped the team understand as a group that they had been focused on the wrong issues. Reaching this conclusion together significantly increased their support for the recommendations for going forward. For the company executive, this consensus was extremely valuable to him in his new role. Critical Resource Dependencies Aviation Technology Company During the Workshop, the company team realized that they did not know the impact of the loss of any single storage drive on their whole operation. This was the result of never having mapped their data to the infrastructure. They didn t know which of their resources were critical and what the dependencies were on these critical resources. They learned that for them to know how critical any given storage drive was to the organization, this mapping had to occur. 4

8 Because of the difficulty of going back to map the resources at this point, the company decided they had to proceed with replicating all of their data rather than just the critical data. Any further delay was considered too great a risk. Given the circumstances, mirroring the entire operation was considered cheaper than risking the wait until the mapping was complete. The company s executives expressed positive feedback on the Workshop. Initially, they had insisted on an abbreviated version to limit their time commitment. However, after they realized what valuable information and knowledge came out of the experience, they acknowledged it was a strategic error to schedule it for less than the prescribed two days. They recommend that other organizations not shorten the time allotted, given the value of their experience. Minimizing Business Risk of Consolidating Data Centers Bank in the United Kingdom After the merger of two banks, the new organization needed to consolidate its six data centers. The bank s objective for the Workshop was to understand the best way to balance the risks versus the optimizations possible with consolidating. Their sessions helped them answer questions such as: How many data centers were required to support the combined workload? Concentrating operations into fewer sites increases the exposure to risk. How much risk were they willing to take? If they reduced the data centers to two, how should they reduce the extent of the operational exposure? Following the Workshop, Hitachi Data Systems helped them evaluate their data centers for fit to purpose to handle the increased, consolidated workload and developed storage area network designs. Hitachi Data Systems also helped the bank determine the storage total cost of ownership before and after the consolidation to evaluate the cost savings. Cost Justification Latin American Bank The bank had taken some basic steps to address business continuity, such as offsite storage of tape backups, but local circumstances made operations very vulnerable. All bank operations for the entire country were in one city, exposing the bank to risks ranging from volcanoes to civil unrest. The bank s attempts to justify better risk management procedures to the parent company had not been successful. The bank needed help communicating its importance to the whole country s financial infrastructure and thus the magnitude of the risk of not addressing the bank s vulnerabilities. Utilizing the quantitative risk assessment developed after the Workshop by the Hitachi Data Systems team, the bank developed a concrete way to justify the investment. Two weeks after presenting it to its parent company, the bank received funding approval for a three-data-center solution. Privately held Financial Institution in the Midwest This financial institution wants to become more like a bank, but this change would make them subject to a number of government regulations, including the requirement to have a disaster recovery plan. All of the institution s computer operations are centralized at one site, supporting 1,000 stores around the country. If the computer center goes down, then all stores are out of operation. When the stores are down, the company not only loses business, but it is also subject to bad debt and a variety of other problems. 5

9 The management staff needed help quantifying the investment required and the benefits that would result from implementing a disaster recovery plan. To satisfy the bank regulations for adequate controls, they needed to set up a second site, but they did not want to spend more than necessary. The company was extremely pleased with the Workshop process and results. At the lengthy executive review session following the Workshop, the CFO repeatedly indicated that the analysis provided great value to them that they would not have obtained otherwise. The executive team approved the investment required for a second site based on the cost versus benefits justification developed with the assistance of Hitachi Data Systems. IT Services Company/Outsourcer This outsourcer was challenged by justifying the incremental cost of improving recovery time measures to their customers. As their customers database systems grow, they find it increasingly difficult to back up or to restore within the recovery time objective (RTO). This increases their customers exposure to catastrophic failure. Consequently the outsourcer needed better but more expensive alternatives such as remote replication to improve time to recovery but did not want to absorb all of the extra cost. The outsourcer engaged Hitachi Data Systems for the Workshop to develop a cost/benefit analysis, which they used to justify the added expense of the new technology to their customers. Compliance Latin American Cell Phone Carrier Due to shortcomings in backup procedures, the cell phone carrier was not able to recover in a timely fashion from an interrupted defragmentation process that left all systems in a disjointed state. Recovery took an entire week. During that time, prepaid phone cards did not decrement minutes from the customers balances, but the carrier was still charged for the service. Lost revenue amounted to US$15 million. The cell phone carrier signed up for the Risk Analysis Workshop following this episode. They learned that there is a critical business issue above and beyond the lost revenue they experienced with the interrupted process. That is, since their stock is traded on New York Stock Exchange, if this episode had occurred after Sarbanes-Oxley Act was implemented, they would have been required to disclose the situation to the public. In that case, the stock market would probably have reacted negatively, potentially costing their investors significantly. Going forward, this company must always factor compliance requirements into their business continuity planning. This realization provided the rationale the carrier needed to make the investment in a disaster recovery site. They purchased Hitachi TrueCopy Remote Replication software for remote data replication plus Hitachi storage. Conclusion Developing adequate business continuity/risk management plans is a complex challenge requiring expertise that most organizations do not have internally. Even more challenging may be justifying the required investments to senior management. Hitachi Data Systems Global Solution Services has the expertise to guide organizations through the process of evaluating their current plans and determining what remediation is required, if any. The methodical approach of the Risk Analysis Workshop has led many organizations to important realizations about the adequacy of their risk management plans and the reality of their readiness to handle business disruptions. 6

10 Hitachi Data Systems Corporation Corporate Headquarters 750 Central Expressway, Santa Clara, California USA Contact Information: / [email protected] Asia Pacific and Americas 750 Central Expressway, Santa Clara, California USA Contact Information: [email protected] Europe Headquarters Sefton Park, Stoke Poges, Buckinghamshire SL2 4HD United Kingdom Contact Information: + 44 (0) [email protected] Hitachi is a registered trademark of Hitachi, Ltd., and/or its affiliates in the United States and/or other countries. Hitachi Data Systems is registered with the U.S. Patent and Trademark Office as a trademark and service mark of Hitachi, Ltd. The Hitachi Data Systems logotype is a trademark and service mark of Hitachi, Ltd. TrueCopy is a registered trademark of Hitachi Data Systems Corporation. All other trademarks, service marks, company names, and logos are properties of their respective owners. Notice: This document is for informational purposes only, and does not set forth any warranty, express or implied, concerning any equipment or service offered or to be offered by Hitachi Data Systems. This document describes some capabilities that are conditioned on a maintenance contract with Hitachi Data Systems being in effect, and that may be configuration-dependent, and features that may not be currently available. Contact your local Hitachi Data Systems sales office for information on feature and product availability. Hitachi Data Systems sells and licenses its products subject to certain terms and conditions, including limited warranties. To see a copy of these terms and conditions prior to purchase or license, please go to or call your local sales representative to obtain a printed copy. If you purchase or license the product, you are deemed to have accepted these terms and conditions. Hitachi Data Systems Corporation All Rights Reserved. WHP KLD February 2007