White Paper Disaster Recovery: Act Locally. Think Globally and Holistically. Authored by Richard Dolewski, VP Business Development & Disaster Recovery
Table of Contents Superstorm Sandy Underscores Need for Modern Disaster Recovery Approach...3 Despite Regular Disaster Occurrences, Many Businesses Remain Unprepared...3 Assumptions that Sabotage Disaster Recovery Plans...4 Determining the Health of Your Current DR Plan...5 Framework for an Effective DR Plan...6 Key Attributes for Selecting a DR Service Provider: A Checklist...7 Prioritize Family First While Your Disaster Recovery Provider Focuses on Your Business...7 Appendix...8 Velocity Technology Solutions Velocity.cc 2
Superstorm Sandy Underscores Need for Modern Disaster Recovery Approach The Impact of Sandy Is Still Being Felt Hurricane Sandy clearly demonstrated to the business and IT worlds that disaster recovery planning cannot be approached as it has in the past. Businesses need to objectively review current disaster recovery (DR) preparedness by examining people, technology and geography as critical points-of-failure for delivering reliable DR performance in time of need. Superstorm Sandy was a worst-case scenario that became a real-life crisis. In such situations, traditional DR practices, disciplines and frameworks often fall short in reducing risk and ensuring business operations can resume quickly. This creates a potential risk that can cost a company dearly in terms of both reputation and the bottom line. Within days after Sandy hit, patience was already in short supply as consumers clamored for access to essentials like food, electricity and water. The need to deliver against strict service level expectations, while maintaining costs, does not go away even in the aftermath of a devastating natural disaster. Months have passed since Superstorm Sandy, and FEMA is still actively working to help businesses restore operations. No business is safe because natural disasters occur on a regular basis throughout the country ice storms, snow storms, tornadoes, earthquakes, flooding and hurricanes can all damage buildings and cause widespread power outages across hundreds of square miles. Superstorm Sandy, for instance, was felt as far west as Wisconsin. Traditional DR Approaches Often Fall Short While traditional backup solutions like tape may be satisfactory for non-critical business applications, they are insufficient for real-time, transaction-based operations and virtualized environments. Given rapidly growing volumes of corporate data, it is becoming increasingly critical for stakeholders to protect this data the most valuable business asset. IT infrastructure and networks can be replaced and facilities can be rebuilt or relocated. But data is priceless and cannot be replaced if lost. According to industry research, the average cost of data center downtime across all industries is approximately $5,600 per minute. On average, enterprises lose between $84,000 and $108,000 for every hour of downtime. People are assets too; you shouldn t take your employees for granted and assume they will be available to restore operations should a disaster strike. Many organizations strictly view their DR efforts as data-center-oriented. However, many single points-of-failure are human-response-based; that is, the risks lie within the human element. Can or will your staff be able to perform their duties as expected in the wake of a disaster or will your staff be unavailable? Businesses must consider the personal needs of their employees. $5,600 Average cost per minute of data center downtime across all industries This white paper from Velocity Technology Solutions examines the current state of DR preparedness. We also illustrate how traditional DR plans place businesses at a high level of risk in terms of their ability to recover in the event of a disaster. Despite Regular Disaster Occurrences, Many Businesses Remain Unprepared DR planning is rarely a favorite activity for businesses and their IT departments. Planning is labor-intensive and requires dedicated staff and organizational commitment. Many organizations invest in DR in the same way they see paying an insurance premium: grudgingly. DR is too often looked at as an expense, not an investment or even a competitive advantage relative to less resilient companies. The old way of thinking assumes, it will never happen to us. Even though the majority of companies claim to have a written plan, many fail to test the plan s effectiveness on a regular basis as a best practice. This paradigm needs to change because disasters requiring businesses to invoke their recovery plans occur more often than most people realize. According to Forrester Research, 60 percent of businesses have invoked their business continuity plans at least once in the past five years, while 28 percent have enacted their plan three or more times during the same time span. Furthermore, today s disasters aren t limited to natural events, but also include everyday human-initiated disasters like technological viruses, as well as cyber attacks. Velocity Technology Solutions Velocity.cc 3
People are assets too; you shouldn t take your employees for granted and assume they will be available to restore operations should a disaster strike. Despite these numbers, many businesses are not overly confident in their DR capabilities. The vast majority of IT executives that have DR plans in place (64 percent in fact) are concerned they may not recover complete operations or may not recover at all in the event of a disaster. IT executives understand implicitly that DR plans need to be tested regularly to ensure both systems and staff are ready should the recovery plan be invoked. But these executives also concede that planning and testing is currently not sufficient in frequency or completeness to assure success. 1 As shown by the figures below, research from IBM indicates a significant percentage of enterprise systems actually operate while relying on unrecoverable data due to no data backups at all, incomplete backups or unusable backups 2 : IBM Power Systems: Windows Environments: Oracle Databases: Virtual Infrastructures: 19% 15% 23% 32% Businesses fall short and lack confidence in their DR capabilities for many reasons: Incomplete plans that do not include all critical application integrations. Outdated plans that do not deliver a data protection strategy to meet business recovery needs. Delivery gaps in terms of IT staff training and capabilities. Plan testing that has not been recently conducted in its entirety and is unproven. Misunderstandings around recovery requirements among internal stakeholder groups. Preparing for a disaster is critical to the long-term health of every business. Customers will not always understand and remain patient when you experience a disaster that prevents your business from providing products and services. They expect your recovery environment to provide the same level of service as your regular production environment, meaning that transparency in the DR environment is critical. Assumptions that Sabotage Disaster Recovery Plans Many businesses create a DR strategy only to find it does not meet the business requirements for protection of real-time transaction processing data. Because organizations do not always have the necessary staff, skills or budget to invest in DR appropriately, IT finds itself struggling to reconcile what the company needs with what the budget will fund. The two sides often do not match up, because the primary business criteria for DR are rapidly becoming more challenging to meet: Recoverability: Increasingly aggressive Recovery Time Objective and Recovery Point Objective requirements. Diversity: The ever-expanding range of IT application environments that must be recovered. Affordability: Constrained budgets that limit the personnel, technology and off-site resources IT can work with. The success of any DR plan typically relies heavily on several flawed assumptions. For example, plans often assume that only the primary business site will be disabled by a disaster and that all other recovery facilities will be unaffected. Successful plans require accessibility to the off-site storage location and critical backup media. If the off-site data storage facility or the recovery site is located in the same FEMA region as your primary facility, a widespread natural disaster could make all three facilities unavailable. In addition, business leaders often mistakenly believe that all qualified personnel as identified in the recovery plan will be available to perform their responsibilities and that the recovery site IT infrastructure is accessible and available for use. Believing these assumptions will hold true could have major ramifications. Major storms are likely to prevent key personnel from executing the recovery plan. Businesses must remember that, for many 1 The Information Availability Institute (IAI): The State of Resilience 2010. Survey of 4,500 IT professionals. 2 IBM Resiliency Panel, 2011. Velocity Technology Solutions Velocity.cc 4
employees, family always comes first. You can t expect employees to travel to your recovery site when their own homes are in peril. Even if internal personnel are able to focus on restoring IT operations, they will most likely be distracted while monitoring how the disaster impacts their families. They could be under extreme duress to the point where they won t be able to perform at their normal level of excellence. Another critical consideration when it comes to widespread natural disasters is that many recovery hot site service providers take care of customers on a first-comefirst-serve basis. This is typically because of highly oversold facilities. The risk profile employed by these vendors is based on very few businesses requiring recovery assistance simultaneously versus the many requests that are likely to occur during a regional event. You could be turned away if too many customers of your provider declare a disaster, which is highly probable in a regional event. This unfortunate news was very common in Superstorm Sandy. Secondly, if your business declares a disaster, many vendors will force you to pay an unnecessary disaster declaration fee before you can start the recovery of your business. This cost can add up to thousands of U.S. dollars and cause delays while you try to access funding. Businesses should enforce multiple FEMA region separation because technologysmart solutions without geographic separations are at risk of failure. impossible to deliver, e.g., a significant number of servers that need to be recovered within a specified amount of time with insufficient staff. Businesses must ensure the DR plan aligns with recent IT integrations and deployments and is incorporated into the change-control process to account for new applications and servers. Evaluate Your DR Team Consider if IT has the right skill sets on hand to execute the plan. Every business has knowledgeable IT resources, but recovery requires a very specific skill set and training along with the ability to perform under pressure with 7x24x365 availability. Disasters do not go on holiday, so personnel need to be available throughout the year. Assigning DR to the right people with the right skills is not as easy as many businesses assume. Natural disasters also displace people. You may not be able to guarantee the availability of your staff during a disaster since family always come first and the business comes second. Your staff will take care of their families before focusing on the enterprise. The ideal plan characteristics should include a dedicated team that focuses on DR every day rather than once per year as a testing exercise. Recovery is only possible if your staff is available to put IT back together again, so ideally DR staff should be on-call at all times, without any other responsibilities, during the time of the disaster. Determining the Health of Your Current DR Plan An important first step towards improving business resiliency by enhancing your disaster recovery plan is to review the plan you currently have in place. The following four checkpoints serve as a guide to ensure you consider all the key factors: Assess Your Current Capabilities Is your business currently ready for a worst-case scenario? Businesses generally plan for this but often do not accurately measure their state-of-readiness. One common issue that causes recovery plans to fall short is an optimistic attitude during good times, resulting in poorly developed drafts that are never validated through testing. In other cases, plans are developed under pressure for audit purposes and include scenarios that are Choosing Your Recovery Site Traditional DR planning often assumes the availability of the recovery solution, which may include an offsite storage site, and the hot site that will host the IT infrastructure. Businesses must ensure geographic separation between their primary facility and recovery site. Ideally, businesses should enforce multiple FEMA region separation because technology-smart solutions without geographic separations are at risk of failure. Businesses should also consider using the cloud for business continuity. The distributed nature of the cloud addresses the ideal scenario for business continuity eliminating single points of failure. The ability to host data center assets off premise in remote, distributed data centers can protect data and applications from a disaster, even if it s a storm system spanning several hundred miles. Forrester s recent report, Master the Eight Disruptors that Will Transform Business Technology Velocity Technology Solutions Velocity.cc 5
Resiliency in 2013, highlights DR in the cloud solutions as a disruptive technology. In addition, Forrester claims that interest and adoption of these solutions is on the rise, with enterprises seeking out a cheaper, faster alternative to traditional services. Aligning DR with Business Requirements Measuring how well the deliverables of a DR plan meet business requirements is typically based on two primary components: Recovery Point Objectives (RPO): The farthest point in time you are willing to go back to for data recovery determined by when the last successful data backup was executed and confirmed. Recovery Time Objective (RTO): The maximum amount of time you are willing to wait before recovering data and resuming operations. RTO is determined by how quickly employees can access data and applications if your primary data center is incapacitated. Business leaders must be aware of the RPO and RTO delivery objectives established by IT to ensure they meet the business requirements. Traditional tape backup and recovery solutions are simply not feasible in a regional disaster. The risk lies with the accompanying latency between the time when the data was protected and the time when the loss occurs. Coupled with ever-increasing restore times driven by data growth, and the need for off-site storage for continuity protection (traditionally in the same FEMA region), businesses can no longer afford permanent, unrecoverable data loss attributable to tape backup. With a typical RPO of 24 hours, the likelihood of human errors and inherent weaknesses increases. The cost of permanently lost data is high and includes the cost of lost revenue, loss of business value and the cost to recreate the data. The Bureau of Labor reports that 93 percent of businesses that suffer a significant loss of data go out of business within five years. Businesses should thus set measurable goals for systems recovery (RTO) and data recovery (RPO) based on staff capabilities and currently deployed technologies. Maintaining continuous application availability is essential to overall business continuity efforts. Businesses can no longer afford permanent, unrecoverable data loss attributable to tape backup. There is a significant need for more flexible and costeffective technologies such as software replication including failover, and continuous data protection. It s imperative to vault servers and data outside of the primary business site s FEMA region so that they are available for data restoration and business resiliency. And, they must be available on an infrastructure that is powerful enough to support resumption of essential business operations. Otherwise, the company will fall short on its objectives. Framework for an Effective DR Plan Effective DR is an ongoing, recurring process. After initially creating and implementing the plan, businesses need to test the recovery process on a regular basis to ensure it actually recovers data and restores operations according to business requirements. The plan should also be reviewed at regular intervals to ensure it accounts for new applications and systems added to the IT infrastructure. If necessary, additional data backup and recovery site resources should be implemented. The Plan >>> Implement >>> Test process must become a repeating discipline within the organization and should address each of these components: Who will execute the plan? Where will your IT infrastructure be recovered? How will the data be protected, and how quickly can it be recovered? What is needed to resume operations? When will your business resume normal operations? Businesses and IT teams also need to evaluate DR solutions holistically and consider the total cost of ownership. To create a tight linkage between your production and recovery environments, engage with experts specializing in application-level recovery that offer advanced methods, such as cloud-based disaster recovery that can be provisioned from recovery sites across multiple FEMA regions. It s also helpful to know your service provider s capabilities in providing the specific resources you contracted for in the event that many customers declare a disaster at the same time. Leading providers balance the number of customers they service from each recovery site and do not oversubscribe their resources. Velocity Technology Solutions Velocity.cc 6
Key Attributes for Selecting a DR Service Provider: A Checklist Working with a managed cloud-based DR service provider frees up your staff to work on project initiatives instead of recovery planning and execution. This partnership enables your team to focus on customers and your business if a disaster strikes especially if you use a provider with a repeatable solution that handles electronic backups as well as the recovery. This type of relationship relieves you from having to handle the recovery logistics. As you compare potential disaster recovery partners, here s a checklist of key capabilities to consider: Allows customers to decide what constitutes a disaster, even if done so proactively ahead of a disaster. Performs application-level recovery on your behalf. Relies on a pricing structure with one monthly fee that covers the entire program. Does not charge disaster declaration fees. Allows ample access to the recovery infrastructure after a disaster declaration without assessing extra fees. Provides multiple failover sites with FEMA-region separation. Balances the number of customers from each FEMA region that access a recovery site to assure over-commitment-of-resources does not occur. Provides each customer with a dedicated infrastructure to assure equal or greater hardware performance as currently in use. Ensures low subscription ratios so you won t compete for recovery resources with other customers. Tests ability to deliver in a disaster on your behalf to verify solution completeness. Demonstrates ability to scale while still achieving near-zero RPO and RTO. By turning to a managed DR cloud provider that offers all of these attributes along with experienced experts that provide repeatable application-level and infrastructure-level recovery, businesses can virtually eliminate recovery risk. Prioritize Family First While Your Disaster Recovery Provider Focuses on Your Business As we have presented in this white paper, traditional DR approaches no longer satisfy today s business requirements and do not adapt well to the rapid changes that all businesses experience. IT must guarantee a quick, repeatable response in the event of a disaster to ensure the business resumes operations in near-zero time. The key to accomplishing this is to remove the people, technology and geography of the recovery site out of your home state and into a different FEMA region. Widespread storms can reap havoc on an entire region, and if your staff faces a choice of taking care of family versus helping your business recover, the family tends to take priority. To take on the DR challenge, we recommend seeking a partnership with a managed, cloud-based disaster recovery provider that offers these attributes: Specifically trained and highly skilled recovery experts. Leading data protection and recovery solutions. Committed access to recovery locations in multiple FEMA regions. Regular recovery testing. A pricing structure that makes disaster recovery more feasible to consume. The provider should also enable IT to audit and prove the DR solution is repeatable and meets the business requirements of internal stakeholders. Taking these steps essentially allows your personnel to focus on their families when a widespread disaster strikes, while the service provider focuses on restoring your business operations. For more information on how to create business resiliency through enhanced disaster recovery capabilities, contact Velocity Technology Solutions at 866.638.2779, or email us at info@velocity.cc. You can also visit our blog Views from the Cloud for the latest thoughts and use-cases on disaster recovery. We welcome your thoughts and want to hear from you. Velocity Technology Solutions Velocity.cc 7
Appendix Three Disaster Recovery Critical Points of Failure Each of these elements is critical in ensuring a DR plan delivers as promised to the business. The plan fails if any single element is not thoroughly considered and adequately addressed: 1. People: Who will be responsible for restoring the IT infrastructure and providing application access to employees? Do they possess the necessary skills? Will they physically be available and mentally ready to respond when disaster strikes or will circumstances involving their families prevent them from performing as needed for the business? In a best-case scenario during a major disaster, half of your staff is likely to be unavailable. In a worst case, you may find that no one is available. 2. Technology: How are corporate data, applications and the supporting infrastructure currently backed up, and is the data protection methodology recoverable on a consistent basis? If a disaster strikes, how will the data and applications be transferred over to the DR facility? Monolithic tape backups do not meet the requirements of today s businesses. Are you willing to accept data loss up to 24 hours and wait for recovery of systems in excess of 48 hours to restore operations back to normal? 3. Geography: Where are data and application backups stored in relation to the company offices? Where is the failover site located? Is there enough distance between these sites so that one major disaster won t take out all sites at once? If your primary production site exists within the same FEMA region as your data storage and recovery site, you are likely at risk that more than one site will become inoperable during a major natural disaster. Establishing FEMA region separation will ensure one event does not impact all three. Methodology to Guarantee Repeatable Disaster Recovery Capabilities 1. Perform an assessment of current backup and recovery implementation to ensure a complete data protection strategy is in place. You cannot recover what is not in your backups. 2. Capture server data and images, and then document critical servers in recovery scope. Complete the server documentation and application inventory so functionality can be restored as it was when the system went down. 3. Develop a recovery solution. 4. Implement the necessary technologies supporting recovery objectives in a separate FEMA region facility. 5. Test to ensure the plan is repeatable and satisfies business requirements. 6. Optimize components that fail to perform properly during the test phase. 7. Audit to demonstrate the recovery capabilities to key business leaders. About the Author Richard Dolewski is VP business development and disaster recovery at Velocity Technology Solutions. He is a certified disaster recovery planner with over 25 years of experience in information technology. He has extensive subject matter experience in disaster recovery planning, backup and recovery program design, high systems availability and infrastructure re-engineering. Richard is an award-winning speaker who has presented over 200 sessions at many user groups and conferences, including IBM, Infor, Oracle s JD Edwards and VMware. He is a frequent technical contributor to industry publications, such as iseries Magazine, Digital Journal, ITJungle, Midrange News and the author of System i Disaster Recovery Planning published by MC Press. Richard has held positions as president of local users groups, board member, advisory committee panelist for IBM Business Resiliency and speaker/subject matter expert at IBM COMMON, where he is a six-time winner of the Speaker Excellence Hall of Fame and the 2010 IBM COMMON Impact Award recipient. He has supported 20 computer room disasters and performed over 350 DR tests. Velocity Technology Solutions 850 Third Avenue, 10th Floor New York, NY 10022 Velocity and the Velocity logo are registered trademarks of Velocity Technology Solutions, Inc. All other trademarks are the property of their respective owners. 2013 Velocity Technology Solutions, Inc. All rights reserved. 131203WPDR1 866.638.2779 646.884.6600 Velocity.cc