EMC Cloud-Enabled Infrastructure for SAP - Business Continuity Series: Disaster Recovery Bundle

Transcription

1 White Paper EMC Cloud-Enabled Infrastructure for SAP - Business Continuity Series: EMC RecoverPoint, EMC Symmetrix VMAX, VMware vcenter Site Recovery Manager, VMware vcloud Suite, VMware PowerCLI Protection of mission-critical SAP landscapes in the cloud Cloud infrastructure resiliency Automated cloud infrastructure disaster recovery EMC Solutions Abstract This white paper focuses on the disaster recovery add-on bundle of the Cloudenabled Infrastructure for SAP. It explains the transformation of a single data center solution into a mission-critical business continuity solution, which protects data from full site disaster and increases resiliency of operations. The solution is enabled by EMC RecoverPoint, EMC Symmetrix VMAX, VMware vcloud Suite, VMware vcenter Site Recovery Manager, and VMware PowerCLI. November 2013

2 Copyright 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H

3 Table of contents Executive summary... 6 Business case... 6 Solution overview... 7 Key benefits... 7 Introduction... 8 Purpose... 8 Scope... 8 Audience... 8 Terminology... 8 Technology overview Introduction Physical architecture Hardware resources Software resources Disaster recovery for Cloud-Enabled Infrastructure Introduction Disaster recovery logical architecture Recovery modes DR operational cycle Failover process Workflow Failback process Workflow Setting up disaster recovery for CEI Virtual Data Center Introduction Configuring VMware vcloud Director Configuring the network Configuring infrastructure Chargeback Manager EMC storage infrastructure Introduction Configuring the Symmetrix VMAX storage EMC RecoverPoint Introduction Configuring RecoverPoint consistency groups VMware vcenter Site Recovery Manager

4 Introduction Integrating SRM with RecoverPoint Configuring VMware SRM SAP system architecture Introduction SAP ERP SUSE Linux Enterprise Server for SAP Applications SAP system architecture Key SAP design considerations Design best practices Data center RecoverPoint VMAX storage Cloud SAP Test and validation Overview Notes Test Scenario 1: Failover after disaster Test objectives Test procedure Results and analysis Test Scenario 2: Resynchronizing to Data Center A Test objectives Test procedure Results and analysis Test Scenario 3: Planned migration Test objectives Test procedure Results and analysis Test Scenario 4: Reprotection for Data Center A Test objectives Results and analysis Conclusion Summary Findings References

5 EMC documents VMware documents SAP documents Appendix Configuring consistency group Configuring VMware SRM Site pairing Array-based replication Resource mapping Protection groups Recovery plans DR design for three data centers

6 Executive summary Business case For most businesses, the focus on the customer s experience has led to more opportunities and competition. Businesses have expanded beyond the traditional geographical boundaries and markets, interacting directly with consumers. Capitalizing on this opportunity requires: Responsiveness Consumers have choices. If they are not satisfied with the service level they receive from one source, they go elsewhere to find what they need. Market and consumer intelligence Availability of the most current data and information is critical to understanding customer behavior and preference, which in turn helps to maintain a competitive advantage. Supply, demand, and order fulfillment management What may seem to be a simple transaction to an ecommerce consumer in reality necessitates the seamless availability of a federated landscape of interdependent business applications. As IT systems embark on their journey to the cloud, the importance of business continuity has become increasingly critical to the operations of an enterprise. Only one extended downtime may cause irrecoverable loss for many businesses. However, proper preparation typically involves a significant investment of time and cost to keep up with a company s recovery time objectives (RTOs) and recovery point objectives (RPOs) during a disaster. Designing disaster recovery (DR) solutions poses many challenges, including: Ensuring RTO and RPO for varying SAP applications in different organizations in a cloud environment Testing the recovery plan periodically Reconfiguration effort needed to prepare another site for DR Human intervention effort needed to carry out DR for hundreds of systems Data protection by using backup and restore is designed to protect the business data from isolated system failure. However, in a full data center disaster, backup strategies no longer meet these challenges. Besides, backup and restore strategies for restoring hundreds of SAP systems to DR within the restriction of RTOs expose system administrators to human error. This white paper, specifically designed to protect SAP mission-critical applications in a cloud environment, introduces an EMC solution that addresses all these challenges. 6

7 Solution overview This solution provides simplified and reliable DR for SAP systems and management infrastructures. We built the solution on top of the Cloud-enabled Infrastructure (CEI) for SAP and used EMC RecoverPoint and VMware vcenter Site Recovery Manager (SRM) as the primary enabling technologies. EMC RecoverPoint is a high-performing, reliable, and flexible replication solution designed for enterprise-class DR. VMware SRM, when integrated with RecoverPoint, can fully orchestrate the entire recovery cycle, from the failover until the SAP systems are ready for user access. This white paper demonstrates how the following technologies support the solution: EMC Symmetrix VMAX is a powerful, trusted, and smart storage platform for cloud environments running multiple mission-critical applications. VMAX provides array splitters designed for data replication. EMC RecoverPoint provides a robust replication mechanism between the VMAX platforms located in production and at the recovery data centers. Its data compression technology reduces bandwidth requirements to minimize overhead. VMware SRM integrates seamlessly with the RecoverPoint system and VMAX storage to orchestrate the DR process. VMware vcloud Director applies the principles of pooling, abstraction, and automation to all data center services, such as compute, storage, networking, and security. It enables the multitenancy business models for IT operation and provides the portal for self-service provisioning. Key benefits This solution adds DR functionality to the CEI for the SAP foundation bundle by providing: Increased resiliency during disasters by automating the failover and failback sequences for both management applications and SAP systems in the cloud environment Well-organized and highly coordinated recovery plans for a trouble-free failover The convenience of triggering the failover from any surviving site by a few clicks of a button Minimized network overhead during replication 7

8 Introduction Purpose This white paper describes a solution that provides DR for SAP systems running on the EMC CEI for the SAP foundation bundle. This solution provides automated DR with seamless integration of RecoverPoint and VMware SRM in a vcloud environment. This white paper: Introduces the key enabling technologies Describes the solution architecture and design Describes how the key components are configured Presents the test results demonstrating the increased availability of the environment Identifies the key business benefits of the solution Scope Audience Terminology The solution discussed in this paper focuses mainly on its integration with a standalone foundation bundle on a VMAX platform. However, the solution is fully compatible with any other CEI add-on bundles that may already be in place, such as the high availability (HA) and application mobility bundle and data protection bundle. This paper does not discuss additional configurations required to work with the HA bundle. This white paper is intended for SAP BASIS administrators, storage administrators, IT architects, and DR managers responsible for designing, creating, and managing business continuity, especially for customers who have implemented CEI for SAP. Understanding this paper requires a basic understanding of the concepts and workflows used by the CEI for SAP foundation bundle. We 1 highly encourage you to read the EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle White Paper first before continuing. This white paper includes the terminology in Table 1. Table 1. Terminology Term AAS ABAP ASCS Description SAP Additional Application Server (previously known as Dialog instances). SAP Advanced Business Application Programming. ABAP System Central Services. 1 In this white paper, "we" and our refers to the EMC Solutions engineering team that validated the solution. 8

9 Term Asynchronous replication Consistency group Copy Deduplication ERP IDES Journal volume NL-SAS OvDC PAS Protection Group PvDC Repository volume RPO RTO SAN Description The process of committing data writes when the data has been sent out for replication without waiting for acknowledgement from the receiving side. Asynchronous replication provides crash-consistent protection and recovery to specific points-in-time, with a small RPO. RecoverPoint configuration method that logically groups data sets so that they are consistent with one another. The consistency groups ensure that the updates to the production volumes are written to the replicas in a consistent write order and that the replicas can always be used to continue working or to restore the production source. A logical RecoverPoint entity that constitutes all volumes of a consistency group. A specialized data reduction technique that eliminates duplicate copies of repeating data. Enterprise Resource Planning. SAP Internet Demonstration and Evaluation System. An SAP system with preconfigured settings and sample data. A RecoverPoint volume that stores time-stamped application writes for later recovery to selected points-intime. Near-Line Serial Attached SCSI. Organization Virtual Data Center. A vdc abstraction, which is partitioned from a PvDC that maps resources to an organization. SAP Primary Application Server. A collection of virtual machines and templates that you protect together by using VMware SRM. Provider Virtual Data Center. A vdc abstraction that combines the computer and memory resources of a single vcenter Server resource pool with the storage resources of one or more datastores connected to that resource pool. A volume that stores configuration information about RPAs and consistency groups. This volume enables a surviving RPA to take over the replication activities of a failed RPA from the same cluster. Recovery point objective. The maximum tolerable period in which data might be lost from an IT service. Recovery time objective. The maximum amount of time allowed to resolve an issue. Storage area network. 9

10 Technology overview Introduction EMC, VMware, and SAP have joined forces to design a complete hardware and software solution that supports an on-premises private cloud. This initiative is now known as the CEI. Its architecture is divided into six different functionalities, delivered as separate bundles, as illustrated in Figure 1. Within the CEI, customers can choose the cloud functionalities they want to enable first without losing sight of their end goal an efficient and reliable private cloud. Figure 1. Cloud-enabled Infrastructure for SAP The CEI is based on a mandatory component called the foundation bundle. Table 2 outlines the functions and benefits of the CEI foundation bundle. EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle White Paper provides more information about the foundation bundle. Customers can then develop their own roadmaps for what to build next. Table 2. On-premise EMC Cloud-enabled Infrastructure for SAP solution: Foundation bundle Function Benefits Technology Virtual data center for SAP Infrastructure chargeback Autonomy of business units and application operations Service catalogs Service level agreements (SLAs) Management of vcloud tenants Resource pooling Cost measurement, analysis, and reporting of the use of compute, network, storage, and backup resources (in combination with data protection add-on bundle) VMware vcloud Suite Enterprise VMware vcenter Chargeback 10

11 Function Benefits Technology Integrated cloud management and performance analysis Storage tiering Cloud networking and security for SAP Manage availability, capacity, performance, and health in the SAP landscape Automatically get the right data to the right place at the right time Cloud-enabled Infrastructure security framework Authorization concepts Compliance and non-compliance tracking VMware vcenter Operations Manager EMC Storage Resource Management EMC Virtual Storage Integrator (VSI) EMC Symmetrix VMAX EMC FAST VP VMware vcloud networking and security Table 3 describes the add-on bundles that enable additional cloud functionality and data center enhancements. Table 3. EMC Cloud-enabled Infrastructure for SAP: Add-on bundles IT strategy Bundle Benefits Technology Business continuity HA and application mobility for SAP High availability within one data center and across two data centers with application awareness for the complete SAP landscape Non-disruptive movement of applications from one data center to another data center EMC VPLEX Metro vsphere HA VMware vsphere Distributed Resource Scheduler (DRS) Symantec ApplicationHA Improved resource utilization across data centers Minimize downtime for infrastructure maintenance DR for SAP Provide disaster recovery protection for cloud management applications and SAP systems VMware SRM EMC RecoverPoint Data protection for SAP Provide data protection to cover the cloud management applications and SAP systems Backup and recovery at all data centers with remote replication of backup sets for offsite protection EMC Avamar EMC Data Domain EMC Data Protection Advisor 11

12 IT strategy Bundle Benefits Technology Service support/ provisioning Automation and operations for SAP SAP application virtualization enables any service any time on any server Provision SAP systems on demand with automated endto-end process Reduce downtime window during maintenance leveraging mass operations SAP NetWeaver Landscape Virtualization Management Enhanced security Enhanced security and compliance for SAP Efficient, collaborative enterprise governance, risk and compliance (egrc) program across IT, finance, operations, and legal domains Data loss prevention Secure user to network authentication RSA Archer egrc RSA Data Loss Prevention RSA SecurID The stand-alone foundation bundle combined with all five add-on bundles form the fully realized CEI for SAP. Physical architecture The full physical architecture of the CEI for SAP consists of three data centers. Production Data Center A and Data Center B are co-located within approximately 100 kilometers (km), with cross-site IP/SAN connectivity and Layer 2 network. Data Center A and Data Center B together provide active/active access, cross-data-center high availability, application mobility, and maximum resource utilization. In addition, Data Center A and Data Center B are equipped with RecoverPoint to provide DR, while Data Center B is also equipped with Avamar and Data Domain to provide data protection. A third data center (Data Center C) is situated remotely in a different failure domain, serving as a standby recovery data center that contains replicated data stores and backup copies. Data Center C also houses a VPLEX witness cluster to monitor the status of the two production sites. EMC Cloud-Enabled Infrastructure for SAP Reference Architecture provides more details. In this white paper, we focus on a two-site scenario (Data Center A and Data Center C), where the DR bundle is implemented on a foundation bundle without a dedicated high-availability data center (Data Center B). The DR configuration for Data Center B is similar. See DR design for three data centers in the Appendix for the design overview for three data centers. In this solution, we built Data Center A for the cloud services upon which all the management and SAP systems run. We built Data Center C as a remote standby data center to add DR capabilities. Data Center C is located far enough from Data Center A to avoid a natural disaster from affecting both sites. The hardware resource and network topology in Data Center C and Data Center A are quite similar, as shown in Figure 2. 12

13 Figure 2. Physical architecture The ESXi servers and storage are totally reserved for DR. Proper sizing should be taken into account before providing DR capability. For example, if the total resource for all tenants subscribing to DR capability in Data Center A is 1000 GHZ CPU, 2048 GB memory, and 50 TB storage, the resource reserved in Data Center C should be no less than this capacity. Hardware resources Table 4 lists the hardware resources for this solution and includes the minimum requirements for validating the solution. Table 4. Solution hardware resources Purpose Quantity Configuration Storage (Data Center A) 1 EMC Symmetrix VMAX with flash/sas/ Near-Line Serial Attached SCSI (NL-SAS) storage pools under EMC Fully Automated Storage Tiering for Virtual Pools (FAST VP) control Storage (Data Center C) 1 EMC Symmetrix VMAX with flash/sas/nl- SAS storage pools under FAST VP control Storage replication 4 Two RecoverPoint Appliance (RPA) clusters: Two RPAs in Data Center A Two RPAs in Data Center C 13

14 Purpose Quantity Configuration VMware ESXi servers for the management cluster VMware ESXi servers for the resource cluster SAN-switching and networkswitching/routing platform 4 Two eight-core CPUs, 256 GB RAM: Two ESXi servers in Data Center A Two ESXi servers in Data Center C 4 Two eight-core CPUs, 256 GB RAM: Two ESXi servers in Data Center A Two ESXi servers in Data Center C 4 Backbone SAN switches 2 Backbone Layer 3 IP switches 2 Routers 4 10 GbE Layer 2 IP switches Software resources Table 5 details the software resources for the solution. Note: Other operating system and database combinations supported by the EMC Solutions Support Matrix (SSM) can also be used. Table 5. Solution software resources Software Version Purpose EMC Enginuity 5876 Operating environment for Symmetrix VMAX Active Directory Windows 2008 Central hostname and user management EMC PowerPath /VE 5.8 Multipathing software EMC Solution Enabler 7.5 Symmetrix command line interface EMC Unisphere (1.5.6) VMAX management software EMC RecoverPoint 4.0 P1 RecoverPoint software SAP ERP 6.0 EHP 5 IDES/Unicode Reference system Oracle database Used on all SAP systems VMware vcenter Chargeback Manager Chargeback software VMware vcenter Server 5.1 Management and resource clusters Microsoft SQL Server 2008 R2 Used by VMware SRM, VMware vcenter, VMware vcenter Chargeback Manager, and vcloud Director VMware vcloud Networking and Security 5.1 Networking and security software for vcloud Director VMware vcloud Director 5.1 Cloud management tool for provisioning virtual data center 14

15 Software Version Purpose SUSE Linux Enterprise 11 SP2 Operating system (OS) for SAP VMware vsphere 5.1 Hypervisor hosting all virtual machines VMware vsphere PowerCLI 5.1 Release 2 Windows PowerShell plug-in for managing vsphere VMware vsphere PowerCLI for Tenants 5.1 Release 2 Windows PowerShell plug-in for managing vcloud Director 15

16 Disaster recovery for Cloud-Enabled Infrastructure Introduction Disaster recovery logical architecture A DR solution is typically complex because a cloud landscape often involves multitenancy and large-scale environments. This solution integrates products from VMware and EMC to automate the DR process and reduce the complexity of the recovery plan. This section includes the following topics: DR logical architecture Recovery modes Operational cycle Failover process Failback process Figure 3 shows the logical architecture for DR. Figure 3. Disaster recovery logical architecture 16

17 In this solution, Data Center A is the production site while Data Center C is the remote recovery site where hardware resources are reserved for DR. We focus on an active/standby DR scenario where hosts at the recovery site are unused under normal conditions. We separated the DR process into two phases: Management group DR The virtual machines in the management group are not managed by vcloud Director. You can fully automate the DR process with native support of VMware SRM. VMware vcenter servers are deployed on both data centers. They manage their own ESXi hosts individually. VMware SRM servers are also deployed on both data centers. They continuously synchronize data with each other so that the failover and failback operations can be executed on any of the available data centers. The DNS server in Data Center C is configured to operate in slave mode. All DNS records updated in the master DNS server in Data Center A are replicated to the slave DNS server in Data Center C to ensure consistent DNS records across the entire infrastructure. Because of their unique replication technique, the DNS, VMware vcenter, and VMware SRM servers no longer need to be replicated at the storage level by EMC RecoverPoint. We highly recommend deploying a backup and restore strategy using the data protection bundle to protect these management applications against OS corruption or other logical errors. Resource group DR vcloud Director manages the SAP systems in the resource group. We used the RecoverPoint CLI and VMware PowerCLI to automate DR for this phase 2. The VMware SRM server executes the scripts after successful recovery of the management group. ESXi hosts in both Data Center A and Data Center C form a cross-data-center DRS cluster called the advanced cluster. This cluster is managed by a single vcenter server, as shown in Figure 3. ESXi hosts in Data Center C remain in maintenance mode so that the workload does not run in the recovery site under normal circumstances. When a failover occurs on virtual machines belonging to the resource group, the VMware PowerCLI script we developed shifts the ESXi servers from maintenance mode to active mode, ready for failover. We used the same VMware SRM to orchestrate both sequences. 2 vcenter Site Recovery Manager 5.1 (or earlier) does not support the protection of SAP workloads in vcloud Director (resource group) at this time. VMware vcloud Director Infrastructure Resiliency Case Study provides details about the integration between VMware SRM and vcloud Director. 17

18 Recovery modes This solution, through VMware SRM, allows two modes of recovery: Planned Migration and Disaster Recovery, as shown in Figure 4. Figure 4. Types of recovery in VMware SRM Both types attempt to replicate recent changes to the recovery site, but they differ in error tolerance. The Disaster Recovery option is used during the failover process. Real-life disasters often present uncontrollable factors and unforeseen consequences. However, the prime objective of DR is system availability; that is, to reinstate as many services as possible within the RTO and RPO. If the outage breaches the maximum tolerable downtime (MTD), the business is at critical risk of irrecoverable damage. In this case, the organization would rather resume the service first to relieve the operations from immediate danger, and deal with the smaller problems later. This is why the Disaster Recovery option is intended for actual site-wide impact. The Planned Migration option is used as a step in the failback process. Planned migrations only succeed when no errors are encountered during their execution. This guarantees a trouble-free failback normal operations are finally resumed. The following section demonstrates how to apply the recovery options in the operational cycle of DR. DR operational cycle All protected systems, from failover to failback, undergo a series of six states. Figure 5 shows transitions from one state to another. DC stands for data center and S1 through S6 represent the protection states at the data centers. 18

19 Figure 5. Operational states transitions 19

20 Table 6 describes the states during the failover and failback processes. Table 6. State S1 S2 S3 S4 S5 S6 Back to S1 Failover and failback transition states Description Both resource group and management group workloads run in Data Center A. Data Center C serves as the recovery site where data are replicated. A disaster causes Data Center A to fail. The entire infrastructure becomes unavailable, including the ESXi servers, network, storage, and RPA. IT administrator triggers a DR in VMware SRM at Data Center C. Data Center C becomes the production site and all virtual machines in both the resource group and the management group become active on Data Center C. Service is restored at Data Center C while Data Center A is being returned to an operational state. Until IT restores the original data center (Data Center A), remote protection is no longer possible. Data Center A recovers and becomes functional. The VMware SRM server and vcenter server in Data Center A are back online and resynchronized with Data Center C. Storage failback has not yet commenced. The effort required to rebuild and configure Data Center A depends on the disaster s impact on the site. At worst case, where all hardware devices are irrecoverably damaged and a complete data center rebuild is necessary, you need to reconfigure VMAX, RecoverPoint, VMware SRM, and VMware vcloud Suite. The detailed steps are described in the following sections: Virtual Data Center EMC storage infrastructure EMC RecoverPoint VMware vcenter Site Recovery Manager IT administrator triggers a reprotect operation in VMware SRM to reverse the original replication direction (that is, from C to A). After reprotection, RPA protects Data Center C and Data Center A becomes the DR data center. A downtime maintenance is scheduled. IT administrator triggers a planned migration. After migration, Data Center A becomes the production data center once again, but is yet to reestablish replication as a final post procedure. Finally, reprotection is triggered again to restore the replication from Data Center A to Data Center C. Afterward, Data Center A becomes protected once again and the environment is now considered to be fully restored to its normal state. 20

21 Table 7 summarizes the major elements of the six states during the failover and failback processes. Table 7. Failover and failback process states States Activity Role of Data Center A Role of Data Center C Replication Workloads run on S1 Normal activity Production Recovery A -> C Data Center A S2 S3 S4 S5 S6 S1 A full data center outage occurs after a disaster Workloads fail over to Data Center C IT rebuilds Data Center A Resynchronize replication Migrate workload back to Data Center A Resynchronize replication Outage Standby None None Outage Production None Data Center C Standby Production None Data Center C Recovery Production C -> A Data Center C Production Standby None Data Center A Production Recovery A > C Data Center A Failover process The failover operation consists of states S1 through S3 in Figure 5. The entire failover process includes: 1. Preparing the storage in Data Center C, which consists of the RPA operations 2. Failing over the management and resource groups: VMware SRM automates the management group failover. Custom PowerCLI scripts automate the resource group failover. Workflow After a successful management group failover, you can initiate the resource group failover. Figure 6 shows the key steps involved in failing over the resource group. 21

22 Figure 6. Resource group failover workflow To execute the failover process, perform the following steps as shown in Figure 6: 1. Enable image access for remote copy in Data Center C. This makes the volumes in Data Center C accessible to the ESXi hosts. 2. Fail over the volume to the remote copy. This turns the remote copy into the production copy while the original copy is not available. 3. Disable VMware High Availability (VMHA) for the DRS cluster. This ensures that VMHA does not restart the SAP virtual machines. We started vapps in Step 8 by using a script so that the correct startup sequence within the vapp will be followed. 4. Rescan the host bus adapter (HBA) and virtual machine file system (VMFS) volumes on the ESXi hosts in Data Center C. All virtual machines in the resource group appear Inactive in the vcenter server inventory after rescanning. Note: Resolving the unresolved volume issue arising from a universally unique identifier (UUID) conflict is essential. You must perform this process with caution to ensure that the mount process does not result in resignaturing volumes. Critical to the success of the approach described in this solution is that no MoRef or UUID changes occur. 5. Perform Exit maintenance mode for ESXi servers after you mount the datastores. 6. Register the virtual machines to the ESXi hosts because VMHA is disabled to prevent the virtual machines from automatically restarting. 7. Start the vshield Edge gateways before starting the SAP systems to ensure that the networking, security settings, and rules are ready. 8. Start all the vapps in vcloud Director using vcloud PowerCLI scripts. This ensures proper start order within the vapp. In this solution, ABAP System Central Services (ASCS) instance is started first, followed in order by the DB instance and then the application instances. 22

23 Failback process The failback operation includes states S4 through S6 in Figure 5. The failback process requires the following VMware SRM operations: Execute the reprotect process of Data Center C. This operation reestablishes replication with the direction reversed, that is, from Date Center C to Data Center A. Perform a planned migration from Data Center C to Data Center A. This operation requires downtime for both the management group and resource group. The required downtime depends on the number of ESXi servers, datastores, and virtual machines. Test Scenario 3: Planned migration demonstrates an estimation of downtime based on our environment. During a planned migration, all applications and storage data are reinstated to Data Center A. After this state, the business operations can be resumed, but Data Center A remains unprotected until storage replication is reestablished. Execute the reprotect process of Data Center A. This reestablishes the RecoverPoint replication from Data Center A to Data Center C, returning both data centers to their normal operating state. Data Center A then becomes protected against another disaster. Workflow Figure 7 shows the failback workflow. Figure 7. Resource group failback workflow 23

24 To execute the failback process, perform the following steps as shown in Figure 7: 1. Restart replication in RecoverPoint once Data Center A is fully functional. VMware SRM can restart the replication for the datastores used by the management group in the reprotect operation. You must execute the RPA script for the resource group on VMware SRM server. A successful reprotection ensures replication of datastores from Data Center C to Data Center A. Thus, the later planned migration can be executed with consistent data. 2. Shut down all the workloads including vshield Edge gateways resource group before planned migration. In this solution, the shutdown operations are included in the scripts. 3. Unmount the datastores from all ESXi hosts in Data Center C. 4. Enable image access for remote copy in Data Center A. This allows ESXi hosts access to volumes in Data Center A. 5. Fail over the volume to the remote copy. This turns the remote copy into the production copy while the original copy becomes replica. 6. Disable VMHA for the DRS cluster. This ensures that VMHA does not restart the SAP virtual machines. We started the vapps in Step 10 by using a script so that the correct startup sequence within the vapp will be followed. 7. Rescan the HBA and VMFS volumes on the ESXi hosts in Data Center A. 8. Enable Exit maintenance mode for the ESXi servers in Data Center A after mounting the datastores. 9. Register the virtual machines to the ESXi hosts, because VMHA is disabled to prevent the virtual machines from automatically restarting. 10. Start the vshield Edge gateways before the SAP systems to ensure that the networking, security settings, and rules are ready. 11. Start all the vapps in vcloud Director using vcloud PowerCLI scripts. This ensures the proper start order within the vapp. In this solution, we started the ASCS instance first, the DB instance second, and the application instances last. 12. Enable replication from Data Center A to Data Center C again. 24

25 Setting up disaster recovery for CEI Virtual Data Center Introduction Virtual Data Center is the key enabling technology introduced in the EMC Cloud- Enabled Infrastructure for SAP Foundation Bundle White Paper. This solution reuses the existing architecture in the foundation bundle and configures the DR capability as an add-on. This section describes how to configure the following components: VMware vcloud Director Networking VMware vcenter Chargeback Configuring VMware vcloud Director This section details the vcloud Director settings needed to fully integrate the DR bundle with the CEI for SAP. Figure 8 illustrates the cloud architecture with the DR add-on. Figure 8. Cloud architecture with the DR bundle 25

26 You must configure the following components to facilitate the DR capability: Storage Provider vdc Organization vdc Configuring storage We created new datastores together with storage DRS clusters and storage profiles to enable DR. Table 8 and Table 9 list the storage settings needed in vcloud Director, configured in both data centers. Table 8. Storage configuration in vcloud Director for Data Center A Consistency group Management group Storage DRS cluster Storage profile Datastore N/A N/A DS_MGMT_DR_Protected (2 TB) Yes DR MGMT SRM Infrastructure N/A DS_MGMT_DR_SRM (2 TB) No Replicated N/A N/A DS_MGMT_Placeholder (10 GB) No Resource group DR Gold Tier DR Gold DS_DR_Gold_001 (4 TB) DS_DR_Gold_002 (4 TB) Yes DR Silver Tier DR Silver DS_DR_Silver_001 (4 TB) Yes Table 9. Storage configuration in vcloud Director for Data Center C Consistency group Management group Storage DRS cluster DR MGMT SRM Infrastructure Storage profile Datastore N/A DS_MGMT_DR_SRM (2 TB) No Replicated N/A N/A DS_MGMT_Placeholder (10 GB) No For the management group, we created a 2 TB datastore in Data Center C to serve as the RecoverPoint replication target for the management virtual machines. We then placed the following virtual machines on this datastore: Active Directory (AD) server MSSQL server for vcenter, vcloud Director, and vchargeback vcenter server for the resource group vcloud Director vshield Manager vcenter Chargeback 26

27 For the resource group, we created two storage DRS clusters in Data Center A to group datastores with different performance levels. RecoverPoint replicates the following datastores: DR Gold Tier A storage DRS cluster for placing datastores set to the highest performance DR Silver Tier A storage DRS cluster for placing datastores set to high performance We assigned two 4 TB datastores to the DR Gold Tier and one 4 TB datastore to the DR Silver Tier, and then created two new storage profiles to characterize the datastores with DR capabilities: DR Gold DR Silver We then assigned storage profiles to the corresponding datastores in the storage DRS clusters and configured datastores in the DR Gold Tier with storage profile DR Gold and datastores in the Silver Tier with storage profile DR Silver. As listed in Table 8 and Table 9, two datastores are marked as not replicated by RecoverPoint in both data centers as follows: VMware SRM infrastructure DNS, VMware vcenter server for management group, VMware SRM server, and their MSSQL database server are deployed on this datastore. Since an independent VMware SRM server is already deployed in Data Center C, if a disaster occurs in Data Center A, we can trigger a failover from the VMware SRM server directly in Data Center C instead. Therefore, the containing datastore does not need to be replicated anymore. Placeholder Required for VMware SRM setup. Placeholder virtual machines consist of a few configuration files deployed on these datastores, including a.vmx file. However virtual machine disk (VMDK) files are not placed on these datastores. Therefore, these datastores can be configured with a smaller size. Configuring the provider vdc In the EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle White Paper, we defined two provider vdcs (PvDCs): Advanced and Standard. In this solution, we configured only the Advanced PvDC with the DR capability. We based this decision on the design consideration that the Standard PvDC is intended for development, test, sandbox, and POC systems. These may not require a fast and automated DR solution. Instead, we can recover these systems in Data Center C with the backup/restore solution. To add the DR capability to the Advanced PvDC, we must add the storage profiles DR Gold and DR Silver to the PvDC storage profile tab. Configuring the organization vdc After we add the DR storage profiles to PvDC, the DR storage profiles are also available in each existing organization vdc (OvDC) built on the Advanced PvDC. However, we recommend creating new OvDCs dedicated for DR storage profiles to group together the vapps with DR capabilities. This simplifies the management efforts during a failover or failback operation. 27

28 In this solution, we created new OvDCs in two organizations: OrgA_Dedicated_DR in OrgA and OrgB_Committed_DR in OrgB. Configuring the network The IP network environment varies in every cloud provider s data center. This solution introduces a network design that requires minimum change efforts during the failover and failback processes. Figure 9 illustrates the network design. Figure 9. Network design For the management group, VLAN 10 in Data Center A and VLAN 100 in Data Center C are designed to handle network activities of management virtual machines needed for the VMware SRM server. VLAN 11 is designed to handle the activities of management virtual machines needed to be protected. In case of a disaster, the protected virtual machines will be restarted in Data Center C on the same VLAN tag (VLAN 11). All the IPs of protected virtual machines remain unchanged so that no DNS records updates are required. For the resource group, VLAN 20 is designed to be an external network for tenants front-end access to SAP systems. Within vcloud Director, an organization network is created for each organization. A vshield Edge gateway is also created for each organization network to act as router and firewall. 28

29 We recommend establishing a secure WAN connection between two data centers. In this solution we used a site-to-site VPN to secure the connection. This solution has three types of cross-data center communications: Communication between the vcenter servers for the resource group in Data Center A and ESXi servers in Data Center C. Communication between the VMware SRM servers in both data centers. Communication between RPA clusters in both data centers. RecoverPoint replicates data over WAN. During the failover and failback processes, management virtual machines connected to VLAN 11 and SAP systems connected to VLAN 20 need to be transferred to another data center. Thus, the route table must be updated prior to a failover or failback action. This is a mandatory manual step and prerequisite on routes or any other network devices for a successful failover or failback. Table 10 summarizes the networks required in this solution. Table 10. Network configurations Network Purpose IP VLAN 10 (Data Center A only) VLAN 11 For DNS, VMware SRM and vcenter for management with their SQL Server For management virtual machines to be protected / /24 VLAN 20 External network in vcloud /24 VLAN 100 (Data Center C only) For DNS, VMware SRM and vcenter for management with their SQL Server /24 OrgNet_OrgA Organization network for OrgA in vcloud /24 OrgNet_OrgB Organization network for OrgB in vcloud /24 Configuring infrastructure Chargeback Manager vcenter Chargeback Manager provides the virtualized infrastructure metering functionality. It integrates with Cloud Director seamlessly, providing the service provider with the ability to chargeback the resource consumed by tenants, and generates cost and utilization reports periodically or on demand. 29

30 The EMC Cloud-Enabled Infrastructure for SAP Foundation Bundle White Paper defines several pricing models to differentiate OvDC with different allocation models and storage profiles with different storage performance levels. Several approaches are available to reflect the chargeback of DR capability. You can create and bind a pricing model to a new OvDC with DR capability, with reasonable charges for infrastructure including CPU, memory, and storage. In this way, you can accurately charge the infrastructure and account for the resource reserved in Data Center C. Another simple approach is to set a rate factor to the storage profiles DR Gold and DR Silver to reflect a charge for the DR capability. You only need to set it once and all the vapps deployed on the storage profiles DR Gold and DR Silver will be charged accordingly with base rate times rate factor. That simplifies the management efforts needed to create pricing models for each OvDC compared to the previously described approach. Figure 10 shows how to set the rate factor in vcenter Chargeback Manager. Figure 10. Setting the factor rate for DR storage profiles EMC storage infrastructure Introduction EMC Symmetrix VMAX is a high-end storage array optimized for the virtual data center. Built on the strategy of simple, intelligent, modular storage, the Symmetrix VMAX array incorporates a highly scalable Virtual Matrix Architecture that enables it to grow seamlessly and cost-effectively from an entry-level configuration into the world s largest storage system. The VMAX supports flash, SAS, and NL-SAS drives within a single array, and an extensive range of RAID types. EMC FAST VP automates tiered storage strategies. The EMC Enginuity operating environment provides the intelligence that controls all components in a VMAX array. The Symmetrix VMAX array-based splitter seamlessly integrates with EMC RecoverPoint, providing reliable replication. 30

31 Configuring the Symmetrix VMAX storage For this solution, we configured all storage into Symmetrix virtual provisioning pools. Thin devices (TDEVs) are bound to the Symmetrix pools and presented to the ESXi servers using masking views. Standard TDEV volume sizes of 2 TB and 4 TB are presented from the VMAX storage array to the ESXi servers. Both VMAX systems at Data Centers A and C have similar configurations. Table 11 details the storage layout for the solution. Table 11. Symmetrix VMAX storage layout Items Data Center A Data Center C Management group Resource group 2 x 2 TB TDEVs: 1 TDEV for management virtual machines that are not replicated to Data Center C, including VMware vcenter, VMware SRM, and its MSSQL database server 1 TDEV for management virtual machines that are replicated to Data Center C 3 x 4 TB TDEVs: 2 TDEVs associated with FAST policy Gold 1 TDEV associated with FAST policy Silver 2 x 2 TB TDEV: 1 TDEV for management virtual machines including VMware vcenter, VMware SRM, and its MSSQL database server 1 TDEV as the remote copy 3 x 4 TB TDEVs: 2 TDEVs associated with FAST policy Gold as the remote copy 1 TDEV associated with FAST policy Silver as the remote copy Journal volume 3 x 800 GB TDEVs 3 x 800 GB TDEVs Repository volume Placeholder volume 1 x 8 GB TDEV 1 x 8 GB TDEV 1 x 10 GB TDEV 1 x 10 GB TDEV 31

32 Choose Tag for RecoverPoint for the volumes used by RecoverPoint in Unisphere for VMAX or SymmCLI, including production copies, remote copies, journal volumes, and repository volumes. Figure 11 shows the configuration setting needed in Unisphere for VMAX. Figure 11. RecoverPoint tag in Unisphere for VMAX EMC RecoverPoint Introduction EMC RecoverPoint is an advanced enterprise-class DR solution designed with the performance, reliability, and flexibility required for enterprise applications. It provides integrated continuous data protection and continuous remote replication to recover applications to any point-in-time (PiT). RecoverPoint systems enable reliable replication locally with one RPA cluster to up to four remote RPA clusters, and over any distance. RecoverPoint systems support data replication for applications writing to SAN-attached storage using existing Fibre Channel (FC) and Internet SCSI (iscsi) infrastructures to integrate seamlessly with existing host applications and data storage subsystems. For remote replication, the systems use existing connections to send the replicated data over FC and WAN, and use FC and IP infrastructures to replicate data asynchronously or synchronously. The systems provide failover of operations to another cluster in the event of a disaster at the production cluster. RecoverPoint variants The RecoverPoint family consists of three variants: RecoverPoint/CL for EMC and non-emc storage platforms RecoverPoint/EX for EMC storage platforms (Block), licensed by storage array RecoverPoint/SE for VNX series, EMC CLARiiON CX3 and CX4 series, and EMC Celerra unified storage environments In this solution, we used array-based RecoverPoint/EX for EMC Symmetrix VMAX. 32

33 RecoverPoint 4.0 features RecoverPoint 4.0 offers the following new features and enhancements to significantly increase capacity and improve performance, helping to build a more flexible and reliable DR solution: Support for multisite replication RecoverPoint can replicate a primary site to up to four remote sites (1:4) for increased data availability. It can also protect data across branch offices with multisite replication from the branches to a central site (4:1). Any PiT recovery to VMware SRM operations Asynchronous throughput up to 400 MB/s using distributed consistency groups Replicated capacity from up to 2 PB per RecoverPoint cluster Support for synchronous replication over IP (versus FC-only in previous versions) A new, easy-to-use Unisphere interface for RecoverPoint management RecoverPoint Appliance The RPA is the RecoverPoint system's intelligent data protection appliance. RPAs manage all aspects of reliable data replication including physical hardware or virtual machines. RPAs are used to attain availability. If an RPA fails, another RPA assumes the workload. A group of two to eight physical (or virtual) RPAs at the same geographic location working together to replicate and protect data is called an RPA cluster. An RPA cluster supports immediate switchover to another appliance if one of the RPA nodes in the cluster fails. There can be up to five RPA clusters in a RecoverPoint system. To scale up and support a higher throughput rate, you can add up to eight RPAs to an RPA cluster. RPAs use powerful bandwidth reduction technologies, including deduplication and compression, to minimize the use of bandwidth and dramatically reduce the time lag between the source and target sites when writing data to storage. RecoverPoint splitter The RecoverPoint splitter is proprietary software that you can install on storage subsystems, and is built into the VNX, Symmetrix VMAX, and VPLEX storage systems. The RecoverPoint splitter is used to split the application writes so that they are sent first to the RPA and then to their normally designated storage volumes. Splitters perform this activity efficiently, with little impact on host performance because the RPA performs all CPU-intensive processing necessary for replication. This feature can also be upgraded without disrupting storage access. Unisphere for RecoverPoint The RecoverPoint 4.0 user interface was developed using Adobe Flex technology and promotes ease of use and a common user experience between the RecoverPoint, VNX, VPLEX, and Symmetrix products. 33

34 RecoverPoint includes a dashboard that provides an overview of the system s health, and all of the information displayed through Unisphere for RecoverPoint is consistently auto-refreshed. RecoverPoint replication options RecoverPoint supports three modes of replication: Continuous Local Replication (CLR) RecoverPoint continuously captures and stores data modifications locally, enabling local recovery from any PiT, with no data loss. CLR supports both synchronous and asynchronous replications. Continuous Remote Replication (CRR) RecoverPoint captures local data modifications and replicates data to remote sites through IP or FC connections. CRR supports both synchronous and asynchronous replications. Continuous Local and Remote Replication (CLRR) A combination of CLR and CRR that provides both local and remote data replication. CLR is normally used for PiT recovery when business data or application logic has been severely compromised. CRR is normally used for DR. CLRR thus implies that two replicas are made for every protected volume (local and remote) to protect the business from both scenarios. This solution demonstrates a CRR with asynchronous replication. Configuring RecoverPoint consistency groups RecoverPoint protects volumes using consistency groups. If two data sets are dependent on one another (such as a database and a database log), they should be part of the same consistency group. When creating a consistency group, define the following elements: Production copy The production copy consists of all source volumes for a consistency group replication, as well as the production journal volumes. Local copy A local copy consists of all target volumes for the replication in local storage for a specific consistency group and the copy journal volumes. Remote copy A remote copy consists of all target volumes of the replication in the remote storage for a specific consistency group and the copy journal volumes. Defining consistency groups A consistency group can contain a local copy, remote copy, or both, which facilitates CLR, CRR, and CLRR respectively. Besides containing a mandatory production copy, the consistency group can also contain a local copy, a remote copy, or both copies, depending on the replicating mode it facilitates. In this solution, we created two consistency groups with remote copy: The management group, which consists of all the volumes for managing the virtual machines The resource group, which consists of all the volumes for SAP systems from different tenants 34