Taking control of your finances.

Transcription

1 WHITEPAPER Taking control of your finances. Disaster Recovery Assurance in Private Clouds Unmatched Value for Your Virtual World

2 Disaster Recovery Assurance in Private Clouds EXECUTIVE SUMMARY The spectrum of corporate risk scenarios is constantly widening and challenging IT s ability to comply with corporate business continuity policies, sector-driven standards like ISO and disaster preparedness legislation. Society s ever increasing reliance on IT for its well-being makes it compulsory for organizations of all sizes to demonstrably be able to recover from disruptions and disasters in their IT infrastructure. Disaster Recovery (DR), is a C-level concern. While virtualization, private clouds and computing fabrics are transforming datacenters and enabling the delivery of IT as a Service (ITaaS), the rate of change is now accelerating, sometimes out of control. Hardware upgrades, software updates, middleware patches, malware protection mechanisms and new components are being introduced at an unprecedented rate, making it all but impossible to gauge the risk exposure of individual changes. In this whitepaper This white paper introduces the concept of Recovery Assurance across multiple clouds. DR Assurance enables a new paradigm where business continuity policies drive preparedness and testing processes. We will explore new techniques that give recovery procedures a level of granularity and automation impossible to obtain in physical datacenters, bridge the gap between physical and virtual resources, and assure the successful recovery of n-tier applications and IT services in case of disaster or disruption. This document is intended for: CIOs, CTOs IT Directors, Datacenter Managers Security and Risk Management Officers BC/DR Planners Datacenter Architects Application Owners In parallel to the more dynamic infrastructure, n-tier application complexity is increasing geometrically as the number of software components that collaborate in the provision of an IT Service continues to grow. Service Orientated Architectures (SOA) and SaaS are fueling interdependency and the risk that a component failure a snowball effect leading to severe service disruption. Assuring recovery can only be done through regular testing. Legacy mechanisms for DR testing are mostly manual, therefore very expensive and infrequent yearly, for most organizations. A new paradigm is called for, one where DR exercising is fully automated and iterative, with daily or hourly cycles.

3 DR & the Private Cloud Floods, fires and, in certain geographies, earthquakes and terrorism have long been the top-of-mind disaster scenarios. These large scale situations have brought about a tendency towards an all-hazards, all-inclusive approach to DR. This approach is so costly and complex that, for companies that can afford it, typical DR budgets run into the tens of millions of dollars. Yet DR tests are carried out perhaps once or twice per year, and recovery issues are revealed during most tests. The perception of recovery risk is generally on the increase. Sites who can afford to test are the lucky ones. For organizations who back up their systems but cannot afford to test DR preparedness, a real disaster situation can be a harrowing experience. Recovery processes fail to complete automatically and at best require manual intervention, at worst cause severe business disruption and significant exposure to punitive or legal action. Private clouds and virtualization enable new paradigms and approaches to DR. In clouds, workloads are independent of the physical hardware, and certain disruptions can be handled automatically or very quickly. For instance, a memory failure in a server may not generate a disruption if the hypervisor provides fault tolerance, and transfers the workload to the next available processor without loss of service. Because virtual assets are streams of bytes, managing them is akin to managing very large files. This property paves the way for the introduction of innovative DR practices where the scope of disruptions that can be handled automatically is significantly larger than in physical datacenters. In clouds that are built on a virtualization layer, the atomic unit of recovery is a virtual machine (VM). N-tier applications run across several VMs working in sync. Figure 1 depicts two typical IT services. Each circle represents a VM. Applications are run from web servers, application servers and an ERP backend. Each IT service is delivered through a collection of VMs, and two of them are shared across the two services. In a cloud, the precise location of VMs can change as hardware is constantly being repurposed and shared by a large number of IT services Firewall Firewall IT Service 1 SSO Server DNS Server Web Server Web Server IT Service 2 Comms Broker App Server Context Indexer Figure 1: IT Services and components ERP Backend Database ReliableDR consists of a single VM that runs on Windows This provides much better utilization of hardware and environmental resources. It also stretches legacy DR software and processes beyond what they were designed to do. Challenges in Assuring Recovery The advent of n-tier models, composite applications, SOA and new cloud-based models like SaaS, are rendering traditional backup-and-restore methods ineffective and in some cases obsolete. We can identify three key reasons why this is so. 1. Inconsistent Recovery: When individual components in an IT service are backed up independently, or replication ends abruptly during an incident, there is significant risk that the recovery brings up components out of sync with each other. In these circumstances, applications can behave inconsistently or unpredictably, or even fail to start and require extensive manual intervention to bring back into production.

4 2. Fragile processes: With huge increases in data volumes coupled with more 24x7 IT services, backup jobs become larger and more difficult to maintain while backup windows shrink. When jobs fail, there is rarely enough time available to restart them, and therefore the backup is incomplete or in an unusable state. 3. Disaster propagation: Backup or replication software copies data from one device to another and is totally service-unaware. Therefore, disasters of a logical nature such as database corruption, bad patches or malware get copied into the secondary (or DR) cloud, and its presence can go undetected until a test is made, or a disaster actually happens. In the latter case, as customer lore has it, you get two disasters for the price of one. In order to ensure service recoverability the above problems must be tackled: 1. Component interdependence: The components of applications and IT services must be safeguarded in a consistent state that can be guaranteed to be recovered. 2. Process errors: DR must have a closed-loop mechanism that supervises the successful completion of processes, and initiates immediate remediation in case of failure. 3. Disaster containment: Logical disasters must be detected, and infected or unusable data or components must be discarded immediately and replaced with healthy checkpoints. As we shall see, technology exists today that provides those solutions in private clouds. Replication & Virtualization The majority of enterprise datacenters use storage arrays with replication and snapshot capability. When applied to cloud environments, arrays can replicate VMs on the fly from primary to secondary clouds. Both storage arrays and hypervisors can collaborate and interface with applications to snapshot sets of VMs at the same time. These sets of VMs thus replicated represent an application or an IT service frozen in time. array replication typically requires similar hardware in both the primary and the secondary clouds. If the hardware is dissimilar, it is possible to use host-based (software) replication (HBR) across sites, as VMs are flat files. This gives datacenter architects more choices in the configuration of their sites, particularly if there are budgetary constraints that preclude the use of similar hardware in all sites. While HBR is less efficient than hardware replication, recent advances in hypervisor technology are enabling better performance levels using dissimilar hardware in primary and secondary clouds. This is giving more choices for datacenter architects to re-use older or lower- performing hardware for recovery sites, and therefore bring down DR costs without compromising Recovery Time Objectives (RTO). Recovery Compliance The first decade of the twenty first century has witnessed large-scale disruptive events that have underscored modern society s reliance on essential public and private services. The protection of these services is an area of broad legislative activity where governments are taking an active role in enforcing the deployment of BCM, like the Civil Contingencies Act in the UK. Government bodies like the European Central Bank and standards-seeking organizations are also actively issuing guidelines or preparing BCM programs, such as ISO22301ti2012 (BS25999) or PS-Prep or ISO/IEC In addition to complying with laws, regulations and standards, CIOs have to consider also how to facilitate e-discovery based on the litigation exposure of their business, as more and more events and transactions are recorded exclusively in electronic form. Virtualization enables a new paradigm to address these concerns. Let us consider the sets of VMs that are snapshot simultaneously, as mentioned in the previous section. These VMs represent an IT service frozen in time, digitally cryo-stored.

5 They are potentially restorable at a later point in time, and executable from that point forward, regardless of the evolution of the underlying hardware where they are stored and from which they execute. This paradigm enables cloud planners to provide long-term storage for an IT service, data and applications, with the assurance that they can be restarted in different hardware platforms a very long time from now. Compliance testing and e-discovery can be quickly and inexpensively facilitated in an private cloud environment. Backup or Replication Secondary Site Primary Site Hypervisor Primary Site Primary Point Catalog Hypervisor Secondary Site In such environments it is also possible to plug compliance processes after the verification that a replicated service is in a consistent state. This opens the door to storing the IT services in external sites, like storage clouds, having first certified their recoverability. Hybrid Services Virtualization has introduced another platform in the datacenter, not only separate from legacy platforms like mainframes and Unix systems, but also from physical x86 systems. It is very common for each platform to have separate DR tools, processes, storage media and skills. The multiplicity of assets involved in DR is another source of risk, and drives more complexity and cost into DR testing. Even among datacenters that standardize on x86 platforms, very few are 100% virtualized and many choose to keep database back ends running physical. This poses particularly difficult challenges to automate recovery testing and assuring policy compliance on Recovery Point and Recovery Time Objectives. Increasing IT Service Resiliency Each of the enabling technologies described in section 2.2 can be programmatically driven to introduce DR orchestration, and create innovative and 100% automated processes that increase IT service resiliency. Figure 2: DR orchestration in Private Clouds DR orchestration brings together the virtualization layers in hypervisors and storage arrays, along with replication or backup software, to manage consistent and recoverable copies of IT services. Preparing a DR Exercise Sets of VMs can be snapshot simultaneously, but are not necessarily in a consistent, recoverable state. For instance, they could take place in the middle of a transaction where the database backend considers the transaction finished, and the web server front end still has it in memory. Therefore, snapshots are candidate recovery points. DR orchestration can leverage the programmability of the hypervisor layer and create an isolated virtual datacenter with VMs where the snapshots can be moved and the IT service brought up to simulate a failover scenario using runbook automation. We will call this environment a test sandbox. Depending on how the service validation is carried out, test sandboxes can be configured either exactly like the primary site, reusing IP addresses within a fenced networking environment, or with different IP addresses and enabling some network traffic in and out of the test sandbox. It is also possible to include some service VMs that, as we will see later, can take the place of physical resources during a DR exercise. Upon building the test sandbox dynamically,

6 DR orchestration can validate the candidate recovery point. This can be achieved by starting the VMs in the sandbox using control flow policy to protect IT service consistency. The heartbeats of the VMs can be checked and individual Windows services and Linux daemons can be verified to be up and running. This process checks the health of each component VMs recovered from snapshot. Recovery Runbooks DR orchestration can drive the validation of candidate recovery points farther through runbooks that include the execution of queries, transactions or any other relevant workloads. Since the test sandbox is isolated from production, almost any kind of test can be run against the recovery point candidate, from the trivial to the most thorough of tests. DR testing can therefore be made into a corporate compliance process managed by a specialized team that develops service<specific recovery runbooks that must be executed in order to certify the recoverability of each IT Service within their responsibility. Recovery runbooks can validate each tier of an application and apply business rules to each recovery step to ensure policy compliance, for instance performing some component-level validation first, and later performing in-depth IT Service unit testing. The degree of recovery testing within a test sandbox is only limited by the amount of virtual resources available, and the period of time available for tests. There is no danger of contamination on the production site and when the tests are complete, the sandbox can be shut down and discarded. Upon applying recovery policy and verifying successful completion, the candidate recovery point can be certified and kept. The DR orchestrator timestamps and makes a catalog entry for the set of snapshots as a single certified recovery point, ready for use. Closed-Loop DR If the tests fail then the candidate recovery point is invalid and the DR orchestrator closes the loop: the snapshots are discarded, the test failure is logged and the entire procedure is restarted at once. If the test fails repeatedly, the DR orchestrator would raise a trouble ticket (SNMP trap and/or ) so that an operations analyst can be called to examine the logs and diagnose the reason for the recovery failure. Web services can drive automatic updates into customers IT governance systems and dashboards, so that compliance with business continuity policy can be monitored in real time, snapshotting all VMs at the same time and testing the recovery all together in the sandbox. This has the side effect that the RPO for the entire set of VMs has to be the same, and the amount of virtual resources needed for the sandbox can be quite large. There are cases where it is preferable to allow each service to be orchestrated separately, using a common CRP for the shared components. Service Dependency is a technique where the DR orchestrator loads an existing (baseline) CRP in the sandbox of a service that depends on it, and this service is snapshotted independently. For instance, there may be two applications sharing a database. Application A has an RPO of 8 hours, and application B s RPO is 24 hours. In this case, the database and application A are DR tested every eight hours. Every 24 hours, aner the corresponding test of application A, the DR orchestrator would snapshot application B separately, and load the available CRP from the last test of application A in order to certify the recovery using the same database. The dynamic nature of IT services, where changes are being made constantly to hardware, operating systems, middleware and applications, requires this closed-loop mechanism in order to assure recovery. Recovery Point Objectives are just that, objectives, unless the loop is closed and they are transformed into guarantees, while the candidate recovery points become certified recovery points (CRPs).

7 Primary Site Secondary Site Obtain Configuration Snapshot VMs Prepare WC Mount VMs Run Recovery Tests Catalog Valid RP >3 Failure Raise Alert Log Failure Discard Snapshot Disk Cloud Tape Figure 3: Closed-loop DR orchestration Service Dependecies It is common for different IT services to share several components, such as databases, identity and access managers, middleware servers and so on. In order to assure consistency in the recovery, all such IT services can be managed by the DR orchestrator as one set. Service Dependency increases resilience and speeds up both DR testing and recovery because failovers start from a common baseline, certified for multiple applications. Figure 4: Supporting different RPOs for services that have components in common ReliableDR: Modernizing DR ReliableDR is a next<generation, closed-loop DR orchestrator for private clouds. ReliableDR leverages virtualization and intelligent storage arrays to automate Recovery Assurance processes so that the RPO can be set by business policy, and RTO can be reduced to minutes and tracked against objectives with real tests. The core benefit of ReliableDR is its ability to align corporate BCP and DR. ReliableDR simulates failover scenarios across clouds as often as required, certifies that recovery points are ready for use in case of disaster or major disruption, and automates the failover processes to the point where service is restored. Components t0 t0+8 t0+16 t0+24 Database snapshot snapshot snapshot snapshot Applica,on/A snapshot snapshot snapshot snapshot Applica,on/B snapshot X X snapshot

8 Legacy, physically-based DR processes were built under the premise that the infrastructure had to be duplicated 1ti1 across the primary and secondary sites, and the storage was backed up or replicated periodically. Recovery is slow and labor<intensive. Hardware has to be verified, and operating systems have to be booted individually. Written procedures, when available, are often out of date. RTO is constantly on the increase because: Configuration drift is unavoidable with the very large rates of change at datacenters; therefore, the 1-to-1 mapping has to be sacrificed, and hardware configuration problems must be addressed manually during contingencies or DR tests. Recovery is multi-stage, typically requiring separate, labor-intensive and lengthy steps to start the hardware, the operating system and the applications; separate skills are normally needed for each stage of recovery Software has intricate dependencies, both with hardware and with other software components; startup procedures need to follow specific sequences so that certain applications or services are ready before others are launched. Stopping Configuration Drift ReliableDR provides configuration-aware orchestration. It has the intelligence to configure recovery VMs in real time, replicating the exact configuration of VMs in the production system at the time of the last certified recovery point. If the primary site s VMs change, for instance if memory is increased in one of them, the change is reflected immediately. With ReliableDR, configuration drift between primary and secondary sites is detected upon execution of the next testing cycle. RPO policy is enforced, and a trouble ticket is raised if an anomaly is detected. ReliableDR supports web services and can integrate its dashboard into customers governance systems, and provide real-time alerts when business continuity policy cannot be guaranteed due to malfunctions in the private cloud. Application Operating System Backup Local Shared Figure 5: Legacy DR Process Rapid Automated Recovery Application Operating System Restore Local Shared ReliableDR orchestrates storage hardware and hypervisor components and to produce CRPs in the secondary site, where the individual snapshots can be certified in a controlled and configured sequence. Snapshots generated by ReliableDR are consistent and complete, and contain all the necessary components required to restore the entire IT service, including data, operating systems, middleware, web front ends, and so on. ReliableDR allows DR planners to apply business policy to the DR processes of IT services individually, depending on criticality, laws, regulations or board directives. For the most important IT services, this can be as often as every hour or every 30 minutes, and ReliableDR will generate 24 or 48 CRPs per day. Because recovery is fast and automated, datacenters only need enough hypervisor licenses in the secondary site to certify the individual recovery points. In case of a real contingency, the licenses used in the production datacenter can be used for recovery.

9 RTO Accleration ReliableDR can configure, mount and start certified recovery points CRPs very quickly, usually in minutes. ReliableDR enables multiple IT Services to be restarted in parallel while respecting service interdependencies through flow control logic that enforces boot order priorities. This added intelligence reduces RTO, requires fewer skills, and less documentation, intervention and training on the part of system administrators in case of a real contingency. The improvements in RTO can be dramatic. Coupled with the ability to policy-drive the RPO generation, DR orchestration brings close alignment between IT Disaster Recovery processes and Business Continuity policy. Figure 6: RTO Acceleration Physical & Non-x86 Components IT services that are 100% virtualized can be fully automated. However, there might be services which have application components running just outside the cloud on physical x86 and/or non-x86 hardware. Since they fall outside the reach of the hypervisor, hybrid services must be orchestrated using different techniques. ReliableDR offers four different options that can be adopted, depending on the architecture and the configuration of the hybrid service. Option 1: Orchestrated P2V This option is available for physical x86 application components. ReliableDR orchestrates a P2V process to obtain a virtualized copy of the component dynamically. As a result, the complete service can be loaded as VMs in the test sandbox, and the recovery can be measured, predicted and assured. Option 2: x86 application component substitution This option is also available for physical x86 application components. Instead of having ReliableDR orchestrate the P2V process (Option 1), a copy of the physical application component will be manually prepared as a VM in the secondary site. For instance, a physical Oracle RAC can be substituted with a virtualized Linux Oracle. This task has to be done only once. As a result, the complete service can be measured, predicted, assured and used in case of a real failover with live data. Option 3: Non-x86 application component substitution This option is similar to option 2, where the production component is not x86 but has an x86 equivalent. For instance, an Solaris Oracle production database running on SPARC and a Linux Oracle (x86 VM) has been prepared in the secondary site for DR testing and failover purposes. If the application component has a persistence layer (eg. database files), the data files have to be converted and presented in the right format to the x86 hypervisor.

10 Option 4: Physical server duplication For this option, a physical server similar or identical to the production server is installed in the secondary site and is used to DR test and failover the complete service. The network for the sandbox must be configured so that the external physical server is accessible to the VMs being tested. ReliableDR guarantees that there is always a recovery point that is consistent and fully recoverable, and ready to provide service continuity. Figure 7: Comparing Legacy and Virtual DR Post-Certification Processes ReliableDR provides support for additional, customized processes to be executed using Microsoft PowerShell and Linux bash scripts. This capability can provide many advantages, such as 1. Move backup from the primary to the secondary cloud: for datacenters where backup is mandatory, the process can be brought to the DR site and effected on CRPs, thus removing overhead or the need for backup windows from production. 2. Additional compliance processing: organizations with more intricate regulatory or statutory compliance requirements can run them as ad hoc processes or extract recovery reports on the CRP at will without impacting production services. 3. Business analysis: LOBs can mine live data and extract ad hoc reports without affecting production performance. 4. Test and development: sandboxed replicas of complete and fully working IT services can be made available quickly for development and testing. Real Failover Upon declaring a real contingency, datacenter management can instruct ReliableDR to switch from testing mode to real failover mode. Instead of using sandboxes, ReliableDR generates VMs with production network connectivity. Using its catalog of CRPs, systems administrators choose the snapshot to recover from, and ReliableDR proceeds to start up the snapshots VMs using the control flow logic of each IT service restored. CASE STUDIES Santalucía Insurance Santalucía is an insurance company with a network of 365 agencies, 7.5m customers and 9,000 employees. Headquartered in Madrid, Santalucía provides home, life, health and accident insurance, as well as a wide portfolio of financial products for investment, pension and retirement plans. An early adopter of virtualization, Santalucía has primary (production) and secondary (backup) datacenters. Each has eight x86 farms hosting critical applications; storage is centralized in each datacenter using HP XP arrays. Using HP XP s native snapshot and replication capabilities, production virtual machines are hot<copied regularly to the secondary datacenter. ReliableDR orchestrates the VMware hypervisors and the HP XP storage arrays. In order to ensure consistency and immediate availability in case of disaster, ReliableDR verifies that the VMs are configured at the secondary datacenter like the primary site. This is done for over 100 VMs on a daily basis.

11 For a set of 60 VMs deemed to be mission-critical, ReliableDR runs further recovery point certification jobs every night between midnight and 4am. These jobs create test sandboxes in the secondary datacenter and bring up all 60 VMs. Upon successful execution and testing of each IT service, the snapshots are certified and cataloged. ReliableDR keeps seven generations of these certified recovery points, providing seven Certified Recovery Points going back from 24 hrs to 7 days. DR orchestration takes place out of the secondary datacenter, where ReliableDR runs as a virtual machine. The production datacenter is not impacted in any way by the DR orchestration. In case of a disaster, be it the entire primary datacenter or a subset, IT services can be brought up selectively and automatically by ReliableDR. mission-critical applications ReliableDR runs twelve DR tests per day and provides an iron-clad RPO of two hours. Out-of-the-box recovery certification functionality was applied to Microsoft SQL Server databases and Exchange. Corporate application servers were certified for recovery using ad hoc application tests agreed with the line of business. Van Lanschot now benefits from tiered and fully automated DR testing of their x86 applications pornolio. SLAs are constantly tracked and enforced. Paul Timmermans, Managing Director at Van Lanschot Belgium, said, Our Disaster Recovery strategy cannot fail to deliver and this is why we chose PHD Virtual Software s ReliableDR because it lets us run non-disruptive, frequent, scheduled Disaster Recovery tests to ensure that a successful service recovery is always assured. Van Lanschot Bank Van Lanschot, founded in 1742 and one of the oldest private banks in Europe, is an independent financial institution that offers private banking, asset management, business banking and corporate finance. It is headquartered in the Netherlands and has operations in Belgium, Luxembourg, Switzerland and Curaçao. Because the bank operates across several jurisdictions, it faced the challenge of demonstrating legal compliance from several countries, such as the UK Financial Services and Markets Act (FSMA), as well as being subject to regulations and audits from the National Bank of Belgium (NBB) and the Dutch National Bank (DNB). The bank runs two NetApp Filers in its production and DR datacenters and was using SnapMirror in order to generate recovery points. However, the existing toolset did not provide fully automated and iterative DR testing, which the bank needed in order to demonstrate compliance at all times. In particular, there was a need to reduce the existing RPO down from 24 hours. A search was made for advanced tools that could leverage the investment in the new, recently virtualized infrastructure. ReliableDR was deployed at Van Lanschot s DR site and integrated with SnapMirror to automate DR testing. Several RPO tiers were defined, and for CONCLUSION PHD Virtual ReliableDR is a unique solution that eliminates DR testing processes and guarantees that RPO and RTO for IT Services and business applications can always be met, whatever the scenario. Whether virtualizing IT infrastructures in order to reduce costs, improve disaster recovery capabilities or move towards an IT as a Service, private cloud model, PHD Virtual ReliableDR enables organizations to: Align business continuity policy with IT disaster recovery Deploy fully automated, non-disruptive and continuous DR testing Guarantee that all components of an IT service are always fully recoverable Report compliance deviations via dashboard, or problem management systems Measure and enforce RPO and RTO automatically Failover individual applications at the push of a button Support mixed physical and virtual environments Scale recovery from individual applications to large clouds

12 CONCLUSION PHD Virtual ReliableDR is a unique solution that eliminates DR testing processes and guarantees that RPO and RTO for IT Services and business applications can always be met, whatever the scenario. Whether virtualizing IT infrastructures in order to reduce costs, improve disaster recovery capabilities or move towards an IT as a Service, private cloud model, PHD Virtual ReliableDR enables organizations to: Align business continuity policy with IT disaster recovery Deploy fully automated, non-disruptive and continuous DR testing Guarantee that all components of an IT service are always fully recoverable About PHD Virtual Technologies PHD Virtual provides the absolute best value in virtual backup, disaster recovery assurance and monitoring for VMware, Citrix and cloud environments. More than 5,000 customers worldwide rely on our products because they are effective, easier to use and far more affordable than competitive alternatives. Delivering the highest performance and most scalable cross platform backup, disaster recovery, and monitoring solutions on the market and pioneer of Virtual Backup Appliances (VBAs), PHD Virtual Technologies has been transforming data protection for virtual IT environments since For more information, please visit: Report compliance deviations via dashboard, or problem management systems Measure and enforce RPO and RTO automatically Failover individual applications at the push of a button Support mixed physical and virtual environments Scale recovery from individual applications to large clouds PHD Virtual Technologies MAIN WEB info@phdvirtual.com North America Headquarters 1880 JFK Boulevard, Suite 1301 Philadelphia, PA 19103