EMC TRANSFORMS IT FOR SAP: FULLY AUTOMATED DISASTER RECOVERY

Transcription

1 White Paper EMC TRANSFORMS IT FOR SAP: FULLY AUTOMATED DISASTER RECOVERY EMC Symmetrix VMAX 10K, EMC VNX5300, EMC RecoverPoint, VMware vcenter Site Recovery Minimize SAP business processes downtime with a quick and automated recovery Simplify and accelerate nondisruptive disaster recovery testing Increase ROI by leveraging existing landscape investments in DR EMC Solutions Group Abstract This white paper demonstrates how EMC RecoverPoint can be integrated seamlessly with VMware vcenter Site Recovery to perform seamless recovery and storage migration between heterogeneous storage systems such as EMC VNX and EMC Symmetrix VMAX in SAP environments. This solution increases Return On Investment (ROI) by using existing investments in implementing a disaster recovery solution. January 2013

2 Copyright 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. All trademarks used herein are the property of their respective owners. Part Number H

3 Table of contents Executive summary... 5 Business case... 5 Solution overview... 6 Key results... 7 Introduction... 8 Purpose... 8 Scope... 8 Audience... 8 Terminology... 8 Technology introduction Overview EMC RecoverPoint EMC Symmetrix VMAX 10K Virtual Provisioning EMC VNX EMC Unisphere VMware vcenter Site Recovery SAP ERP Basics of RecoverPoint replication Replication modes Main components EMC RecoverPoint appliance Write splitters Replication sets and consistency groups Storage Replication Adapter Journal volumes Initial replication Solution architecture Overview Physical design Hardware resources Software resources Virtual machine resources SAP application environment

4 Solution configuration Overview Configuring replication Enabling bi-directional replication Customization specific to SAP Heterogeneous test environment Replication mode Startup priorities Network settings /etc/hosts file SAP shared folders SAP startup sequence SAP user logon Load-generation environment Bi-directional recovery testing System integrity and quality control Test and validation Overview Test scenario 1: Nondisruptive failover testing Test scenario 2: Disaster recovery Test scenario 3: Failback Test scenario 4: Point-in-time recovery Conclusion Summary Findings References White papers Product documentation SAP notes Appendix A: RecoverPoint replication modes Continuous data protection Continuous remote replication Concurrent local and remote replication Appendix B: Setting up replication Preparing VMAX 10K and VNX5300 for RecoverPoint connectivity Configuring RecoverPoint Configuring SRM

5 Executive summary Business case For most businesses, the growth in ecommerce and focus on consumers experience has led to more opportunity and competition. Simply stated, businesses have expanded beyond traditional geographic boundaries and markets, interacting with consumers directly. Capitalizing on this opportunity requires: Responsiveness: Consumers have choices. If they are not satisfied with the service level they receive, they go elsewhere to find what they need, leading to severe productivity loss. Market and customer intelligence: Availability of data and information is critical to any business for understanding customer behaviors and preferences to maintain competitive advantages. Supply, demand, and order fulfillment management: What may seem to be a simple transaction to an ecommerce consumer in reality necessitates the seamless availability of a federated landscape of dependent business applications. Ultimately, these business requirements translate into the challenges for SAP IT including: Scalability: Providing increased performance to handle increasing transactional processing volumes while maintaining access to increasing amounts of data. For example, customers who have started out with smaller legacy systems for SAP environments can opt to scale outward to more powerful storage solutions like EMC Symmetrix VMAX. Disaster Recovery: How to respond to increasing pressures to mitigate planned and unplanned downtimes as the growth in production operations gives rise to the need for new levels of protection against disasters or logical corruption. Additionally, from the standpoint of business continuity, Information Security Management Systems (ISMS) standards require regular testing of backup and recovery solutions to ensure effectiveness in the event of a disaster. Unfortunately, regular failover testing is rather difficult for businesses with very low downtime tolerance. Traditional disaster recovery (DR) solutions, such as an offsite tape restore, may no longer be adequate to address increasing recovery time objectives (RTO) and recovery point objectives (RPO). Storage replication solutions can satisfy these requirements. However, maintaining an idle data center for disaster recovery may not be suitable for some organizations either. These challenges demand a solution that offers effective, affordable, and efficient disaster recovery protection for these critical SAP business functions, without wasting any previous investments. 5

6 Solution overview This solution provides customers with a strategy that enables DR, using VMAX and VNX series interchangeably as production and DR storage for SAP applications. The solution team designed the architecture for central SAP installations, distributed implementations (database, central services, and application are on separate virtual machines), and heterogeneous EMC storage solutions such as VNX and VMAX families. We implemented this using a Symmetrix VMAX 10K storage array together with an existing VNX5300 storage array to form a bi-directional disaster recovery solution. We achieved this by replicating both storage devices to each other with EMC RecoverPoint and designing automated recovery plans using VMware vcenter Site Recovery (SRM). This solution demonstrates the following capabilities: Fast and seamless failover This solution allows quick and fully automated failover on multiple systems in the correct order, in a matter of minutes, and with a few clicks of a button. This allows enterprises to be resilient against disasters and to regain productivity sooner. Automated failback This solution facilitates automated reversal of the data centers roles after failover. After the original production site is operational, failback is just a matter of performing another failover. Point-in-time (PIT) recovery without database restore The solution can perform PIT recovery without the need to restore and roll forward the entire databases just before a logical corruption occurs. Nondisruptive failover testing This solution allows online testing by restoring a snapshot and encapsulating it in a private network during the test. This procedure effectively reduces the need for disruptive testing. Information security audit requirements are now easier to manage. 6

7 Key results This solution significantly improves business agility and scales out for many common needs for SAP production and nonproduction systems: The bi-directional replication strategy enabled by EMC RecoverPoint allows two sites to mutually replicate and serve as production sites and recovery sites of each other. This strategy reuses the existing storage systems VNX in this case, to effectively increase ROI. The integration of EMC RecoverPoint and VMware vcenter SRM provides a seamless end-to-end recovery strategy, which requires minimal downtime for failover and failback, and point-in-time recovery operations. This solution provides an environment for performing regular recovery testing without any downtime, to be compliant with the security audit requirements. Customers can leverage this solution to migrate their SAP environments between heterogeneous system environments. 7

8 Introduction Purpose This white paper demonstrates how EMC RecoverPoint can be integrated seamlessly with VMware vcenter SRM to perform seamless recovery and storage migration between EMC VNX and EMC Symmetrix VMAX systems. It describes the benefits in the following scenarios: After migrating business data from VNX to Symmetrix VMAX 10K, relocating the existing VNX5300 from the production site to the recovery site Configuring two active production data centers to mutually protect each other Scope Audience Terminology To achieve the stated purpose, the document covers the following: Components used and the key concepts involved in implementation The reference architecture of the bi-directional disaster recovery system using EMC Symmetrix VMAX 10K and VNX5300 Installation and configuration procedures for EMC RecoverPoint and VMware Site Recovery Use-case functionality test results This white paper is intended for SAP administrators, storage administrators, IT architects, and disaster recovery managers who: Have virtual SAP environments on existing storage and plans to migrate to a VMAX storage family, while keeping good use of the original storage after migration. Have existing virtual SAP environments on Symmetrix VMAX storage solutions and aim to find more economic strategies in implementing disaster recovery solutions. Table 1 includes the terminology used in this white paper. Table 1. Terminology Term Asynchronous replication CDP CLR Definition Process of committing data writes when the data has been sent out for replication without waiting for acknowledgement from the receiving side. Asynchronous replication provides crash-consistent protection and recovery to specific points-in-time, with a small RPO. Continuous data protection. Concurrent local and remote replication. 8

9 Term Consistency group CRR Deduplication Gatekeepers Journal volume KPI PIT Replica volume Replication set Repository volume RPA RPO RTO Synchronous replication Definition RecoverPoint configuration that logically groups replication sets that must be consistent with one another. The consistency groups ensure that updates to the production volumes are written to the replicas in a consistent write order and that the replicas can always be used to continue working from or to restore the production source. Continuous remote replication. A specialized data compression technique for eliminating duplicate copies of repeating data. Small, three-cylinder devices (approximately 3 MB) used by the solutions enabler to send control commands to the Symmetrix VMAX array. Timestamped application writes stored in RecoverPoint journals for later recovery to selected points-in-time. Key performance indicator. Point-in-time; usually refers to a system state that occurred at a specific date and time in the past. Volume that contains the replicated image from a production site. RecoverPoint configuration that defines an association between a production volume and the local and/or remote volumes to which it is replicating. Volume that stores configuration information about RecoverPoint appliances (RPAs) and consistency groups. This volume enables a surviving RPA to take over the replication activities of a failed RPA from the same cluster. RecoverPoint appliance. Recovery point objective. Recovery time objective. Process of committing data writes only after the receiving side acknowledges the data. This ensures that writes between two sites are always consistent. 9

10 Technology introduction Overview This solution uses the key components described in Table 2. Table 2. Key components Component EMC RecoverPoint EMC Symmetrix VMAX 10K EMC VNX5300 EMC Unisphere VMware vcenter Site Recovery SAP ERP 6.0 Role Storage replication device Production site storage Recovery site storage Storage management application DR failover coordination Enterprise application software Note While Symmetrix VMAX and VNX models were used in the solution testing, this solution is generally applicable to other models as well. EMC RecoverPoint EMC Symmetrix VMAX 10K EMC RecoverPoint is an advanced enterprise-class disaster recovery solution designed with the performance, reliability, and flexibility required for enterprise applications in heterogeneous storage and server environments. It provides bidirectional local and remote data replication, without distance limits. The Symmetrix VMAX 10K system is built on the highly scalable EMC Virtual Matrix Architecture, which enables the storage to grow seamlessly and cost-effectively from an entry-level, single-bay, single-engine configuration to a six-bay, four-engine system with 512 GB cache memory, 1,080 drives, and up to 1.5 PB usable capacity. Built for simplicity and ease-of-use, Symmetrix VMAX 10K is 100 percent virtually provisioned, with storage tiering automatically managed by EMC Fully Automated Storage Tiering for Virtual Pools (FAST VP). Symmetrix VMAX 10K supports Flash, Serial Attached SCSI (SAS), and Near-Line SAS (NL-SAS) drives with RAID 1, 5, and 6 protection options. Symmetrix VMAX 10K systems are preconfigured and ready for same-day installation and startup. They support all EMC Symmetrix monitoring and management tools, including the latest enhanced Symmetrix Management Console, which provides simpler installation and management with smart wizards. The integrated RecoverPoint splitter for Symmetrix VMAX 10K enables local and remote replication for flexible and efficient RPOs and RTOs. Virtual Provisioning EMC Virtual Provisioning simplifies storage configuration and management, improves capacity utilization, and enhances performance by enabling the creation of thin devices. Thin devices present more capacity to applications than what is physically allocated in the storage array. 10

11 Physical storage that supplies disk space to the thin devices comes from a thin pool. The thin pool is composed of data devices that provide the actual physical storage. You can add or remove data devices to grow or shrink thin pools nondisruptively. Virtual Provisioning supports local and remote replication with RecoverPoint on VMAX 10K. Virtual Provisioning is described in detail in EMC Solutions Enabler Symmetrix Array Controls CLI Version 7.3 Product Guide. EMC VNX5300 EMC VNX5300 storage array is a member of the VNX series storage platforms, which delivers innovation and enterprise capabilities for file, block, and object storage in a scalable, affordable, and easy-to-use solution. Designed to meet the highperformance, high-scalability requirements of midsize and large enterprises, the VNX series enables enterprises to grow, share, and cost-effectively manage multiprotocol environments. The VNX series is powered by quad-core processors, making it two to three times faster than its predecessor. The VNX processors support the demands of advanced storage capabilities such as Virtual Provisioning, compression, and deduplication. VNX arrays incorporate the RecoverPoint splitter, which supports unified file and block replication for local data protection and disaster recovery. EMC Unisphere VMware vcenter Site Recovery EMC Unisphere is the central management platform for EMC Symmetrix VMAX and VNX series, providing a single combined view of file and block systems, with all features and functions available through a common interface. Unisphere is optimized for virtual applications and provides industry-leading VMware integration. It automatically discovers virtual machines and VMware ESX servers and provides end-to-end, virtual-to-physical mapping. VMware vcenter SRM is a disaster recovery framework that integrates with EMC RecoverPoint to automate recovery of VMware datastores so that it becomes as simple as pressing a button. SRM is an extension to VMware vcenter that enables integration with array-based replication, discovery, and management of replicated datastores, and automated migration of inventory from one vcenter to another. SRM does not replicate any data, but, instead, leverages an external replication solution such as RecoverPoint. SRM servers coordinate the operations of the replicated storage arrays and vcenter servers on the production and recovery sites so that, as virtual machines on the production site are shut down, virtual machines on the recovery site start up and resume the responsibility of providing the same services, using the data replicated from the production site. Migration of protected inventory and services from one site to the other is controlled by a recovery plan that specifies the order in which virtual machines are shut down and started up, the computing resources that are allocated, and the networks they can access. Integrated with RecoverPoint, SRM also enables testing of recovery plans, using a temporary copy of the replicated data, in a way that does not disrupt ongoing operation on either site. 11

12 SAP ERP 6.0 SAP ERP 6.0, powered by the SAP NetWeaver technology platform, is a world-class, fully integrated enterprise resource planning (ERP) application that fulfills the core business needs of midsize and large enterprises across all industries and market sectors. SAP ERP 6.0 delivers a comprehensive set of integrated, cross-functional business processes and can serve as a solid business-process platform that supports continued growth, innovation, and operational excellence. 12

13 Basics of RecoverPoint replication Replication modes RecoverPoint offers different modes of replication, as shown in Figure 1. Continuous Data Protection (CDP) PRD Local replica Continuous Remote Replication (CRR) PRD Remote replica Concurrent Local and Remote Replication (CLR) PRD Local replica Remote replica *PRD Production volume Figure 1. RecoverPoint data replication options Types of data replication in RecoverPoint include: Continuous data protection (CDP) for local replication Continuous remote replication (CRR) for remote protection Concurrent local and remote replication (CLR) for combined local and remote protection Appendix A: RecoverPoint replication modes provides more details. Main components The following components enable this technology: RecoverPoint appliance (RPA) Write splitters Replication sets and consistency groups RecoverPoint Storage Replication Adapter (SRA) Journals 13

14 EMC RecoverPoint appliance RecoverPoint is available as a physical appliance. This enables support of large amounts of information stored across heterogeneous environments. This out-of-band approach enables RecoverPoint to deliver continuous replication. RPAs run the RecoverPoint software and manage all aspects of data replication. For local replication, a cluster configuration of two or more active RPAs is deployed. This supports immediate switchover to another appliance if one of the RPA nodes in a cluster goes down. For remote replication, a RecoverPoint cluster is deployed at both sites. RPAs use powerful deduplication, compression, and bandwidth reduction technologies to minimize the use of bandwidth and dramatically reduce the time lag between writing data to storage at the source and target sites. Write splitters RecoverPoint monitors writes using a technology called write splitters, which ensure that all host writes to a protected volumes are tracked and a copy is sent to local RecoverPoint appliances. RecoverPoint supports three types of write splitters in general: array write splitters, intelligent fabric-based write splitters, and host-based write splitters. In this solution, RecoverPoint uses an array-based write splitter that operates in each front-end adapter of the Symmetrix VMAX 10K array, as well as inside the storage processors of an EMC VNX storage array. Replication sets and consistency groups Regardless of the mode chosen, RecoverPoint allows you to define an association between a production volume and the local and/or remote volumes to which it is replicating, called a replication set. These replication sets, which must be consistent with one another, are logically grouped into consistency groups. The consistency groups ensure that updates to the production volumes are written to the replicas in a consistent write order and that the replicas can always be used to continue working from or to restore the production source. Storage Replication Adapter The EMC RecoverPoint SRA for VMware vcenter Site Recovery provides integration of SRM with RecoverPoint. The RecoverPoint SRA supports discovery of arrays attached to RecoverPoint and of consistency groups that are enabled for management by SRM. It also supports SRM functions such as failover and failover testing by mapping SRM recovery plans to the appropriate RecoverPoint actions. Journal volumes Journal volumes hold data that is waiting to be distributed to target replica volumes and retain copies of the data previously distributed to the target volumes. Journal volumes facilitate operational recovery to any point-in-time, as long as the relevant journal history is retained. 14

15 Each consistency group has its own set of journal volumes, which allows for differing retention periods across the consistency groups. Each consistency group has two or three journal volumes assigned to it: Local journal volumes, if configured Remote journal volumes, if configured Production journal volume, which stores information about the replication process (called marking information) that is used to synchronize the replication volumes at the two sites much more efficiently, when required Initial replication Regardless of the mode chosen, RecoverPoint allows you to define an association between a production volume and the local and/or remote volumes to which it is replicating to, called a replication set. These replication sets can be logically grouped into consistency groups. The consistency groups ensure that updates to the production volumes are written to the replicas in a consistent write order and that the replicas can always be used to continue working from or to restore the production source. Each consistency group is in one of the following states: Replicating: When the consistency group is enabled, the splitter is replicating to the RPAs, and the RPAs are transferring to the replica journal/journals. Marking: If the RPAs are unable to transfer to the replica journal, the location of the changes is stored in the RPAs as well as on the production journal volume. When contact with the remote site is reestablished, the remote replica is synchronized, but only at those locations that were marked as having changed. No marking/no replication: The splitter does not write to the RPAs. This can be caused by a manually disabled consistency group or by a disaster on the production site (no RPAs are available). 15

16 Solution architecture Overview Physical design This solution used a Symmetrix VMAX 10K storage array with an existing VNX5300 storage array to form a bi-directional disaster recovery environment. In this solution, we replicated both storage devices to each other with EMC RecoverPoint and designed automated recovery plans using SRM. The technical design adopted in this solution demonstrates an effective disaster recovery technique using CLR technology. In our test scenario, the production systems VM2 and VM3 were initially in a VNX storage solution on site C. Later, VM2 was migrated to VMAX storage on site A. Figure 2 depicts the final architecture of the solution environment after migration. Figure 2. Solution architecture 16

17 We used two data centers to host SAP systems VM2 and VM3: 1. We used two data centers: Site A: Contained VMAX 10K; production site of VM2 and recovery site of VM3. Site C: Contained VNX5300; production site of VM3 and recovery site of VM2. 2. We configured the two pairs of RPAs contained in each data center (RPA1 and RPA2 on site A, RPA3 and RPA4 on site C) as a cluster. 3. We partnered each RPA cluster with an SRM server. 4. We configured consistency groups on the RecoverPoint Management application so that the datastores of each SAP system can be replicated locally and remotely. 5. We installed two SAP systems: VM2 SAP IDES ERP, Oracle, Suse Linux, distributed SAP installation VM2 was initially installed on the VNX5300 array and then migrated to the VMAX 10K array. This DR solution is also used in migrating SAP systems in VNX to VMAX storage. VM3 SAP IDES ERP, SQL Server, Windows, central SAP system installation VM3 was independently installed on VMAX, and then migrated to the VNX array. This use case demonstrates bi-directional recovery. 6. We integrated SRM with RPA and configured it to allow mutual failovers between the two sites. Hardware resources Table 3 details the hardware resources for the solution. Table 3. Solution hardware environment Hardware Quantity Configuration Storage (Production site A) 1 EMC Symmetrix VMAX 10K with: Single engine 42 GB cache memory 66 x 400 GB, 15K FC drives, plus spares 13 x 200 GB Flash drives, plus spares 34 x 2TB SATA drives, plus spares 8 Gb FC connectivity 1 Gb network connectivity Enginuity

18 Hardware Quantity Configuration Storage (site C) 1 EMC VNX5300 with: 58 x 600 GB, SAS drives, plus spares 9 x 200 GB Flash drives, plus spares 30 x 2 TB SATA drives, plus spares 8 Gb FC connectivity 1 Gb network connectivity VNX Operating Environment Block software version: VMware ESXi server 2 2 x quad-core CPUs, 2.93 GHz, 64 GB RAM 4 x 8 Gb FC HBA 2 x 1 Gb Ethernet 1 4 x 10 cores CPUs, 2.00 GHz 400 GB RAM 4 x 8 Gb FC HBA 2 x 1 Gb Ethernet 1 2 x quad-core CPUs, 1.86 GHz 16 GB RAM 2 x 4 Gb FC HBA 2 x 1 Gb Ethernet Network switch 2 1 Gb Ethernet switches FC switch 4 8 Gb/s FC switches, 2 per site RecoverPoint appliance 4 Gen 4, 2 RPAs per site Software resources Table 4 details the software resources for the solution. Table 4. Solution software environment Software Version Purpose EMC Unisphere for VMAX Symmetrix VMAX configuration and management tool EMC Unisphere for VNX 1.1 VNX management software EMC Solutions Enabler 7.4 Symmetrix VMAX management software EMC RecoverPoint 3.5 EMC replication software, installed on each of the 4 RPAs VMware vsphere B Hypervisor hosting all virtual machines 18

19 Software Version Purpose VMware vcenter B Management of VMware vsphere VMware vcenter Site Recovery EMC RecoverPoint adapter for VMware vcenter Site Recovery SUSE Linux Enterprise Server B Managing failover and failback of virtual machines 2.0 EMC software for integrating RecoverPoint and SRM Server operating system for SAP VM2 Microsoft Windows Server 2008 R2 *64 Server operating system for SAP VM3 Microsoft SQL Server SQL Server 2008 R2 Database for SRM Virtual machine resources Table 5 details the virtual machine resources for the solution. Table 5. Virtual machine environment Role Purpose Quantity Configuration SAP VM2 ABAP SAP Central Services server (ASCS) Critical production 1 2 vcpus, 8 GB RAM, SLES 11, 2 vnics SAP VM2 database server Critical production 1 8 vcpus, 16 GB RAM, SLES 11, 1 vnic SAP VM2 Application servers Critical production 3 8 vcpus, 16 GB RAM, SLES 11, 2 vnics SAP VM3 Central System Non-critical production 1 16 vcpus, 24 GB RAM, Windows 2008 R2, 2 vnics SRM site A application SRM management system 1 2 vcpus, 4 GB RAM, Windows 2008 R2, 1 vnic SRM site A database SRM management system 1 2 vcpus, 8 GB RAM, Windows 2008 R2, 1 vnic SRM site C application SRM management system 1 4 vcpus, 8 GB RAM, Windows 2008 R2, 1 vnic SRM site C database SRM management system 1 2 vcpus, 8 GB RAM, Windows 2008 R2, 1 vnic Load runner controller Simulate user activity 1 Load runner generator Simulate user activity 1 2 vcpus, 16 GB RAM, Windows 2003 R2, 1 vnic 8 vcpus, 8 GB RAM, Windows 2003 R2, 1 vnic 19

20 SAP application environment Table 6 details the SAP application environment for the solution. Table 6. SAP application environment Specifications VM2 VM3 Number of datastores, type 2, VMFS 1, VMFS Operating system Database application Size SAP application Suse Linux Enterprise Server (SLES) 11 Oracle 11g R2 database server 1 TB SAP data, 1.4 TB allocated in 2 datastores SAP ERP 6.0 EhP 4 ABAP IDES Windows 2008 R2 SQL Server 2008 R2 630 GB SAP data, 1 TB allocated in 1 datastore SAP ERP 6.0 EhP 5 ABAP IDES Installation type Distributed Central 20

21 Solution configuration Overview Configuring replication Building this solution involves two major steps: Replication configuration SAP-specific customization The storage configuration procedure is as follows: 1. Prepare VMAX 10K and VNX5300 for RecoverPoint connectivity. a. Create LUNs and replica volumes. b. Provision RPA repository and gatekeepers. c. Tag all VMAX 10K devices to be protected by RecoverPoint. 2. Configure RecoverPoint. a. Install RecoverPoint. b. Configure RecoverPoint access to the splitters on the VMAX 10K and VNX arrays. c. Create consistency groups. d. Enable bi-directional replication. 3. Configure SRM. a. Installing SRM virtual machines and SRA for integration to RPA. b. Establish connections between protected and recovery sites. c. Configure RecoverPoint array managers. d. Configure protection groups. e. Configure inventory mappings. f. Customize virtual machine recovery options. g. Customize recovery site IP addresses. h. Create recovery plan. Appendix B: Setting up replication provides more detailed explanations of these steps. For a detailed discussion of all the steps, refer to Site Recovery Administration vcenter Site Recovery 5.1. Enabling bi-directional replication Adding a second consistency group with the roles of the data centers reversed (that is, production volumes on site C are replicated to replica volumes on site A) effectively enables bi-directional replication. You can use single set of paired SRM (and RecoverPoint) sites to protect virtual machines in both directions. Each site can serve as a production and recovery site simultaneously, but for a different set of virtual machines. Bi-directional operation is 21

22 not limited to array-based replication. However, whenever you use an array basedreplication as in this solution, each of the array s LUNs replicates in one direction only. Two LUNs in the same array can replicate in different directions from each other. Customization specific to SAP Some SRM customizations are valuable in automating recovery in an SAP environment. A distributed SAP system requires the following steps during the recovery startup: 1. Change IP addresses of the SAP instances. 2. Update hosts files to reflect internal IP settings. 3. Mount the /sapmnt/<sid>/sys/exe, /sapmnt/<sid>/sys/global, and /sapmnt/<sid>/sys/profile folders of the ABAP Central Services instance to the SAP application and database servers. 4. Start the applications in this order: a. SAP ABAP Central Services instance b. Oracle database instance c. All SAP ABAP Application instances This solution used the flexibility of the SRM recovery plan by inserting operating system commands that perform the previous tasks during a failover. This fully automates the startup process until all SAP systems are ready for login. To achieve the desired level of automation, the test environment was prepared as described in the following sections. Heterogeneous test environment We considered diversity when we built the test environment for a more extensive scope and more credible results. This includes SAP releases, types of databases and operating systems. We also tested the solution across components with non-uniform time zones. 22

23 Replication mode We set up CLR in both VM2 in the consistency group SiteA_PRD and VM3 in SiteC_TST, as shown in Figure 3 and Figure 4. Figure 3. RecoverPoint topology showing VM2 in CLR mode Figure 4. RecoverPoint topology showing VM3 in CLR mode 23

24 Startup priorities SRM allows virtual machines to be started up in a particular order by assigning a priority. Priority 1 virtual machines are started up first, followed by priority 2 virtual machines, and so on. We changed the priorities of the virtual machines as follows: Priority 1 ASCS; to make the SAP shared folders available for mounting Priority 2 Database instance; to ensure that the database is started before the SAP application instances are started Priority 3 All application instances Figure 5 shows the priority assignment. Figure 5. Virtual machine startup priorities Network settings We configured two static IP addresses for each virtual machine: An internal x.x IP address A service IP for log in (SAP virtual machines only) You can achieve this by configuring two network interface cards (NICs) for each virtual machine or by creating a script that binds the internal IP addresses as virtual. We used the former in this solution. /etc/hosts file SRM updates only the entry of the virtual machine's own hostname. Ensure that the internal IP addresses are used before and after recovery. Steps are as follows: 1. Create a hosts.i file as a template that maps the hostnames to the internal IP addresses. 2. Insert a script as a post-startup recovery step to reset the hosts files with contents from the hosts.i file to keep consistency. 24

25 SAP shared folders You can insert a mount command as a post-startup recovery step on every application instance or add the mount point entries in the /etc/fstab file. This solution used the latter technique to mount the /sapmnt/<sid>/sys/exe, /sapmnt/<sid>/sys/global, and /sapmnt/<sid>/sys/profile folders of the ASCS instance to the SAP application servers. SAP startup sequence We inserted the following scripts as a post-startup recovery step on the recovered virtual machine, with a 5-minute timeout. These scripts were assigned as follows: On all virtual machines: change the host script: cp /etc/hosts.i /etc/hosts On SAPVM2DB (Priority 2): assign the startdb script: mount -a su - oravm2 -c "lsnrctl start" su - vm2adm -c "startdb" ON SAP virtual machines: assign the startsap script: su - vm2adm -c "startsap" Figure 6 shows the scripts that were applied to the automated application startup after powering on. Figure 6. Scripts to automate application startup in post-power on steps 25

26 SAP user logon All SAP instances communicate with internal IPs. Thus, the hosts file only included internal IP addresses, as shown in Figure 7. Figure 7. Hosts file For SAP logon, we created two entries; one is for logging in to the production site A, the other for the recovery site C, as shown in Figure 8. Note that full IP addresses will not be shown in Figure 8 and Figure 9. Figure 8. SAP logon We created two logon groups using transaction code SMLG, which corresponds to the two sets of IP addresses. Users can just switch the logon group used in the SAP GUI after the failover, as shown in Figure 9. 26

27 Figure 9. Logon groups created in VM2 Load-generation environment We used LoadRunner for every test in this solution to run successive updates in SAP tables and to simulate a real-world environment. The timestamps on the table entries inserted by LoadRunner served as counterchecks of the timestamps recorded in SRM or RecoverPoint, but at the SAP level. Timestamps from LoadRunner updates provide more accurate application-level RTO and RPO key performance indicators (KPIs) for testing. The tests were considered successful when the results from SRM/RecoverPoint and SAP were consistent. Bi-directional recovery testing We performed the recovery tests on both VM2 and VM3 systems to validate bidirectional recovery. However, because the procedure was the same, for brevity this white paper shows only the screenshots of the more complex VM2 system. System integrity and quality control We took timestamps carefully to ensure consistency among all components. We performed SAP checks before and after testing for each scenario according to the best practices to ensure that the failed-over system was healthy technically and functionally. 27

28 We performed checks on the following transactions/commands: SICK System installation check SM51 Instance overview SM50 Work process overview, per instance SM21 System logs ST22 ABAP dumps DB02 Database management SM12 Display and delete locks SM13 Administrate (sic) Update Records df Ph 1 For Linux file systems Note These controls were observed, but the raw data and screenshots are not shown in this paper. 1 df Ph displays file systems, but in a POSIX human-readable format. 28

29 Test and validation Overview In the context of operational efficiency and effectiveness, this section aims to test and demonstrate the following capabilities: Nondisruptive failover testing This solution allows online testing by restoring a snapshot and encapsulating it in a private network during the test. This procedure effectively reduces the need for disruptive testing. Disaster recovery This solution allows the client to quickly automate the entire failover operation with a few clicks of a button. Failback testing Because site A may be geographically closer to users, or applications running previously on it are presumably more resource intensive, failback to the original site is recommended. This operation reinstates failed-over systems on VNX to their original servers as soon as possible. This design allows for rapid failback to reduce downtime for the operation. PIT recovery The solution aims to show how easy it is to perform PIT recovery, eradicating the need to restore and roll forward entire databases just before the logical corruption occurs. Test scenario 1: Nondisruptive failover testing Test scenario An organization originally has a central site C that contains SAP production systems VM2 and VM3 on VNX5300. A second auxiliary site A that contains VMAX 10K was built. VM2 will be failed over/migrated from site C (VNX5300) to site A (VMAX 10K) using this solution. To prepare the migration, we simulated disaster recovery failover on VM2 and VM3 on site C while both were in production. Test objectives The test objectives were: Performing failover testing on the recovery site without shutting down any virtual SAP system Testing bi-directional recovery Achieving an RTO of 30 minutes 29

30 Test procedure The test procedure was as follows: 1. Ran load on VM2. 2. Took timestamps and performed test recovery on VM2 and VM3. During testing, we did a health check on both systems to verify integrity. 3. Logged in to the SAP GUI, took the timestamp, and performed an SAP health check. 4. Checked the availability logs of VM2 and VM3 on site C to see if the production systems had gone down during testing. Findings Table 7 shows the findings of this test. Table 7. Nondisruptive failover test findings Success criteria Findings Result VM2 and VM3 stay online Availability logs showed that both systems were available during failover Passed Test failover is 100% successful Logs completed without errors Passed Test VM2 is operational Duration of test recovery falls within RTO System health status validated availability and integrity after completion Timestamp in SRM showed recovery time of 00:13:47 Passed Passed To summarize this test: Initiation of the testing required only three steps. Additional manual steps were not needed until completion, as shown in Figure

31 Figure 10. Test recovery execution The recovery was successful in SRM, as shown in Figure 11. Figure 11. Final status of recovery plan in SRM after testing Conclusion A fully consistent bi-directional SAP DR was easily made available for DR testing in less than 30 minutes while primary sites were available for production users with continued DR replication. 31

32 Test scenario 2: Disaster recovery Test scenario After a successful simulation in scenario 1, the same production SAP system VM2 on site C was failed over to site A for planned migration/disaster recovery. To simulate the disaster, we terminated the database processes and triggered disaster recovery from site C to site A. Test objectives The test objectives were: Performing failover on the recovery site Achieving an RTO of 30 minutes Test procedure The test procedure was as follows: 1. Ran load on VM2. 2. Took the timestamp and performed planned migration/disaster recovery on VM2. 3. Logged in to the SAP GUI, took the timestamp, and performed an SAP health check. 4. Checked the availability log of VM2 on the protected site. Findings Table 8 shows the findings of this test. Table 8. Disruptive failover test findings Success criteria Findings Result Test failover is 100% successful Logs completed without errors Passed Recovered VM2 is operational Duration of failover falls within RTO System health status validated availability and integrity after completion Timestamp in SRM showed recovery time of 00:15:06 Passed Passed To summarize this test: Initiation of the testing required only four steps. Additional manual steps were not needed until completion, as shown in Figure

33 Figure 12. Recovery execution using VMware SRM 33

34 The recovery was successful in SRM and validated in SAP, as shown in Figure 13. Figure 13. Final status of recovery plan executed in VMware SRM Conclusion The actual failover required only four steps. A fully consistent SAP DR was available in less than 30 minutes. Manual intervention was not needed after the recovery was initiated. This technique can be used to migrate systems from VNX to VMAX or vice versa. Test scenario 3: Failback Test scenario VM2 was on site A after recovery in scenario 2. Assume that site C is now fully operational after it has crashed. We then re-established protection between the two sites, but with the roles reversed for VM2: Site A was the protected site, replicating to recovery site C. VM3 remained unchanged. Test objectives The test objectives were: Performing failover on the recovery site Achieving an RTO of 30 minutes Test procedure The test procedure was as follows: 1. Issued reprotection on VM2 to reverse the direction of protection. 2. Took the timestamp and recovered VM2 to the original site. 3. Logged on to the SAP GUI, took the timestamp, and performed an SAP health check. 34

35 4. Checked the VM2 availability log in the protected site. 5. Failed over to site C again to verify if the reprotection was effective. Findings Table 9 shows the findings of this test. Table 9. Failback test findings Success criteria Findings Result Reprotection is 100% successful Logs completed without errors Passed Failover is 100% successful Logs completed without errors Passed Recovered VM2 is operational Duration of failback falls within RTO System health status validated availability and integrity after completion The timestamp in SRM showed: Reprotection time of 00:01:48 Recovery time of 00:16:48 Passed Passed To summarize this test: Initiation of reprotection required just four steps. Additional manual steps were not needed until completion, as shown in Figure 14. Then another four steps were needed for the failover Figure 14. Reprotection procedure 35

36 Figure 15 shows how reprotection reversed the direction of replication so that further post-processing was no longer necessary. Figure 15. SRM reprotection log showing successful reprotection in a reverse direction Figure 16 compares the replication status in the RecoverPoint application after reprotection was triggered. The replication source and target volumes were automatically changed by SRM as well. Arrows were added for reference. 36

37 After failover reprotection After failback reprotection Figure 16. RecoverPoint VM2 topologies after failover and failback Conclusion Reprotection needed only four steps and the actual failback needed another four steps. All DR changes to databases and file systems were reinstated to the original production site in less than 30 minutes. Test scenario 4: Point-in-time recovery Test scenario The system was corrupted because of an application logic error after a transport. We performed a point-in-time (PIT) recovery to restore the system before the transport. Test objectives The test objectives were: Performing PIT recovery without using a database restore Achieving an RTO of 30 minutes 37

38 Test procedure The test procedure was as follows: 1. Populated the table VBAK and captured a timestamped screenshot of the final entries. 2. Created a RecoverPoint bookmark. The screenshot taken previously represented the system state of the bookmark, as shown in Figure Populated the table VBAK further and captured a timestamped screenshot of the final entries. These entries represented the erroneous data that caused the PIT restore. 4. Used RecoverPoint to restore VM2 to a bookmark: a. Powered off VM2 in vcenter. b. Recovered production. c. Enabled image access of production. d. Resumed production. e. Powered on VM2 in vcenter. 5. Logged on to the SAP GUI, displayed the table VBAK, and captured a timestamped screenshot of the final entries. 38

39 Findings Table 10 shows the findings of this test. Table 10. Point-in-time recovery test findings Success criteria Findings Result VM2 is operational after the PIT restore PIT restore is successful Duration of restore falls within RTO SAP log in and health check verified application integrity Timestamps were consistent through the screenshots but the erroneous data was no longer found The entire process completed in 00:25:00 Passed Passed Passed Figure 17 shows the bookmark S1_T1 created in RecoverPoint. Figure 17. RecoverPoint journal bookmark Figure 18 shows that the final bookmarked entry was Sales Doc inserted at 23:12:15. Figure 18 shows the System Status window on the right-hand side. Figure 18. Table VBAK status during RecoverPoint bookmarking 39

40 Figure 19 shows that user activity resumed after bookmarking and the table was populated further. We captured this image approximately 6 minutes after capturing Figure 18. Figure 19. VBAK table after RecoverPoint bookmarking Figure 20 shows the RecoverPoint status in the middle of the restore process. We captured this image 10 minutes after capturing the previous image. Figure 20. RecoverPoint in the middle of the restore process 40

41 Figure 21 shows the status after the PIT recovery. Note that the result of the PIT recovery is correct and consistent, even if the time zone of the Management GUI (Figure 20, Windows time) is 8 hours ahead of the RecoverPoint server time and 12 hours ahead of the SAP GUI time. The latest entry was the same sales document, that is, Sales Doc Figure 21. VBAK status reverted to when the bookmark was created Conclusion Data recovery from a corruption can occur in minutes without the need of a full database restore or file system recovery. 41

42 Conclusion Summary Findings This solution offers effective, affordable, and efficient disaster recovery protection for critical business functions without wasting previous investments. This strategy significantly improves business agility and scales out for common SAP production system and test system needs, providing: Bi-directional replication of a central SAP environment and distributed SAP environments between heterogeneous arrays by using the integrated RecoverPoint splitters for Symmetrix VMAX 10K and VNX across two sites. Rapid and automated failover for planned and unplanned maintenance, enabled by integration of RecoverPoint and SRM. Nondisruptive disaster recovery testing using RecoverPoint and SRM. Point-in-time database recovery in a virtualized environment from bookmarked RecoverPoint images. Redeemed storage investment by reusing VNX as DR storage after migration to Symmetrix VMAX. The key findings of this proven solution are as follows: This solution is able to manage the varying DR complexities of SAP environments. Initiating the failover testing requires only three steps. A fully consistent bidirectional SAP DR is easily made available for DR testing in less than 30 minutes, while the production systems remain completely undisrupted. Additional manual steps are not required until completion. The actual failover requires only four steps to initiate. A fully consistent SAP DR is available in less than 30 minutes. Manual intervention is not required after the recovery is initiated. This technique can be used for VNX/VMAX migrations. Initiation of reprotection requires only four steps. The actual failback requires another four steps. All DR changes to databases and file systems are reinstated to the original production site in less than 30 minutes. Data recovery from a corruption can occur in minutes without the need of a full database or file system recovery using RecoverPoint point-in-time recovery. 42

43 The solution implements control measures to protect against human errors and technical limitations: We performed the tests using super user accounts for convenience, but the solution is capable of role-based access control (RBAC) to manage configuration and administrative activities in a more granular sense, if needed. For more information, refer to Site Recovery Administration vcenter Site Recovery 5.1. While the entire failover process is automated, the trigger itself is manual. This ensures that a failover can be initiated only after proper processes are followed, executive approvals have been granted, and a state of disaster (or resumption of normal operations) has been declared. The failover operation can be triggered with one or two steps. However, we put confirmation prompts in place to prevent inadvertent executions. 43

44 References White papers Product documentation SAP notes For additional information, see the following white papers: Business Continuity and Disaster Recovery for Oracle 11g Enabled by EMC Symmetrix VMAX 10K, EMC RecoverPoint, and VMware vcenter Site Recovery EMC Symmetrix VMAX RecoverPoint Splitter and Enhancements in Enginuity 5876 For additional information, see the following product documents: EMC Solutions Enabler Symmetrix Array Controls CLI Version 7.3 Product Guide Site Recovery Installation and Configuration vcenter Site Recovery 5.1 Site Recovery Administration vcenter Site Recovery 5.1 For additional information, see the following SAP notes: Note : Memory parameter recommendations for banking systems Note : Automated Oracle DB parameter check Note : Oracle : Database Parameter Settings 44

45 Appendix A: RecoverPoint replication modes Continuous data protection CDP continuously captures and stores data modifications locally, enabling local recovery from any PIT with no data loss. CDP supports both synchronous and asynchronous replications. Figure 22 illustrates the CDP workflow. Figure 22. CDP workflow The CDP workflow is follows: 1. The application server issues a write operation to a LUN protected by RecoverPoint. The write is intercepted by the RecoverPoint array splitter. 2. The splitter splits (duplicates and distributes the write request) to both the protected volume and RPA. 3. (3a) RPA acknowledges back to the splitter when receiving the write. (3b) At the same time, RPA moves the data into the local journal volume, along with the timestamp and any bookmarks for the write operation. 4. After the write request has been secured in the journal, it is then applied to the target replica volumes. The write order is preserved during distribution. 45

46 Continuous remote replication Continuous remote replication (CRR) supports synchronous and asynchronous replication between remote sites and a wide area network (WAN). CRR supports synchronous replication when the remote sites are connected through Fibre Channel (FC) and provides zero RPO. Asynchronous replication provides crash-consistent protection and recovery to specific points-in-time, with minimal RPO. Figure 23 illustrates the CRR workflow. Figure 23. CRR workflow The CRR workflow is follows: 1. The application server issues a write operation to a LUN protected by RecoverPoint. The write is intercepted by the RecoverPoint array splitter. 2. The splitter duplicates the write and sends it simultaneously to the protection volume and to its local RPA. 3. The local RPA receives the write operation: 3a. In asynchronous mode, acknowledgment is sent back from the local RPA to the splitter. 3b. In synchronous mode, acknowledgement is instead delayed until the write is received by the remote RPA. This request is bundled with other writes, deduplicated (that is, redundant blocks are removed), sequenced, 46

47 and timestamped. The bundle is then compressed and transmitted with a checksum for delivering to the remote RPA cluster. 3c. If the receiving RecoverPoint is not reachable, the write request is temporarily stored on the production journal volume until the remote RecoverPoint is accessible once again. The write entries are moved from the production journal to the target journal and normal replication is resumed. 4. When the package is received on a remote site, the receiving RPA verifies the checksum first to ensure integrity and then decompresses the data. 5. The remote RPA writes the data to the journal volume on the remote site. 6. The write is then distributed to the target replica volumes through an RPA FC connection after the data has been stored safely in the journal. The write order is preserved during distribution. Concurrent local and remote replication CLR is a combination of CRR and CDP and provides concurrent local and remote data replication. Figure 24 illustrates the CLR workflow. Figure 24. CLR workflow 47

48 The CLR workflow steps are as follows: 1. The application server issues a write operation to a LUN protected by RecoverPoint. The write is intercepted by the RecoverPoint array splitter. 2. The splitter duplicates the write and sends it simultaneously to the protection volume, and to its local RPA. 3. The local RPA receives the write operation: 3a. In asynchronous mode, acknowledgment is sent back from the local RPA to the splitter. 3b. The write request is then sent to the local journal and applied to the local replica volume. This completes the CDP sequence. 3c. In synchronous mode, acknowledgement is instead delayed until the write is received by the remote RPA. This request is bundled with other writes, deduplicated (that is, redundant blocks are removed), sequenced, and timestamped. The bundle is then compressed and transmitted with a checksum for delivery to the remote RPA cluster. 3d. If the receiving RecoverPoint is unreachable, the write request is temporarily stored on the production journal volume until the remote RecoverPoint is accessible once again. The write entries are moved from the production journal to the target journal and normal replication is resumed. 4. When the package is received on a remote site, the receiving RPA verifies the checksum first to ensure integrity, and then decompresses the data. 5. The remote RPA writes the data to the journal volume on the remote site. 6. The write is distributed to the target replica volumes through an RPA FC connection after the data has been stored safely in the journal. The write order is preserved during distribution. This completes the CRR sequence. CDP is normally used for PIT recovery where business data or application logic has been severely compromised. On the other hand, CRR is normally used for disaster recovery. CLR thus implies that two replicas are made for every volume protected (local and remote) to protect from both scenarios. 48

49 Appendix B: Setting up replication Preparing VMAX 10K and VNX5300 for RecoverPoint connectivity Configuring the RecoverPoint splitter on the storage array requires the following volumes to be provisioned beforehand. Step 1: Creating LUNs and replica volumes Create the following LUNs for RecoverPoint usage: One repository volume (at least 3 GB) for each RPA cluster. One replica volume each in the protected and recovery site for every consistency group. Three journal volumes (production journal, local journal, and remote journal volumes) for every site. Eight unique gatekeeper volumes each for RPA1 and RPA2. Figure 25 shows how we created the gatekeeper volumes using Unisphere for the VMAX GUI. Figure 25. Provisioning gatekeeper volumes using Unisphere Consider the following during capacity planning and risk assessment: The size of a replica volume must be equal to or greater than the size of the protected volume. For example, if the protected volume is 500 GB, CDP or CRR consumes an additional 500 GB and CLR consumes at least 1,000 GB, excluding the journal volumes. This means a CLR consumes twice as much as a CDP or CRR alone. The journal size depends on the RPO. Longer RPO requires larger journal sizes. Different consistency groups can be created in the same configuration. Production systems can be assigned to a consistency group in CLR mode while others can be assigned to a group in either CDP or CRR mode. 49

50 SAP systems falling under the same consistency group will failover as one bundle. Assigning a separate consistency group to each SAP system is highly recommended. The size of the journal volumes should reflect the RPO. You need to determine the expected peak change in the environment to calculate the minimum size of the journal volume. The formula is: ( ) ( ) ( ) Note Target side log size is 20 percent (0.2) according to the best practice and an additional 5 percent is allocated (1.05) for internal use. For example, to calculate the minimum journal size to satisfy a 48-hour rollback requirement (172,800 seconds) with an estimated 10 Mb new data writes per second: ( ) ( ) ( ) Step 2: Provisioning RPA repository and gatekeepers To provision RPA repository and gatekeepers, create masking views in VMAX and storage groups in VNX to present these volumes to the RPA cluster. We created the following masking views for this purpose: Recoverpoint_view: This presents the repository, journal, and replica volumes to all nodes in the RPA cluster. RPA1_GK_view: This presents the eight gatekeepers to RPA1. RPA2_GK_view: This presents the eight gatekeepers to RPA2. Step 3: Tagging all VMAX 10K devices to be protected by RecoverPoint As of RecoverPoint 3.4 (SP1 and later), production and replica LUNs must be tagged for RecoverPoint use in VMAX 10K splitter configurations. Otherwise, the devices will not be accessible to the splitter. Note that this step is not required for VNX storage. 50

51 Figure 26 shows how we tagged devices through Unisphere for VMAX. Figure 26. Tagging devices using RecoverPoint for VMAX Alternatively, the following command can be executed from the Solution Enabler for every device to be tagged: symconfigure sid <Symmetrix ID> -cmd set dev <device list or device range> attribute=rcvrpnt_tag; commit Figure 27 shows the output of this command. Figure 27. Tagging a RecoverPoint device from Solution Enabler To verify the device tag status, input the following command: symdev sid <Symmetrix ID> show <device list or device range> 51

52 Figure 28 shows the status with the device tagged as RecoverPoint. Figure 28. Verifying the RecoverPoint tag status It is a best practice to tag also the RecoverPoint gatekeepers, journal volumes, and repository volumes. Configuring RecoverPoint Step 1: Installing RecoverPoint As shown in Figure 29, the RecoverPoint Installer wizard guides you through the entire setup procedure including setting up the RPA cluster. Perform the steps described in the section Preparing VMAX 10K and VNX5300 for RecoverPoint connectivity before running the wizard. Figure 29. RecoverPoint Installer wizard Step 2: Configuring RecoverPoint access to the splitters on the VMAX 10K and VNX arrays You need to configure the VMAX 10K and VNX splitters in RecoverPoint to bridge the replication. We did this by using the New Splitter wizard in RecoverPoint through Splitter > Add New Splitter, as shown in Figure

53 Figure 30. Adding a new splitter in RecoverPoint Step 3: Creating a consistency group After successfully installing RecoverPoint, you can configure the consistency groups for the replication setup. The New Consistency Group wizard guides you during the entire setup. You can find the wizard in the RecoverPoint interface by right-clicking Consistency Groups and selecting Add Group, as shown in Figure 31. Figure 31. Adding a consistency group in RecoverPoint 53

54 Consider the following before launching the wizard: Replication policy settings for the consistency group Which site should be considered as the production site and which should be considered as the recovery site? Which type of replication should be used (CDP/CRR/CLR)? A one-to-one mapping of the production volumes to corresponding remote replica volumes Which volumes are allocated for local and remote journal Configuring SRM Installation prerequisites Before installing SRM, ensure that the following prerequisites are met: Each site must contain a vcenter server with at least one vsphere data center being managed. The recovery site must support array-based replication with the protected site. At least one virtual machine must be located on a datastore that is replicated by RecoverPoint on the protected site. A reliable IP network is required to connect the two sites. The recovery site must have access to the same public and private networks as the protected site, though not necessarily having access to the same range of network addresses. VMware tools must be installed on all virtual machines. Step 1: Installing SRM virtual machines and the SRA for integration to RPA Perform the following steps: 1. Install a database server on each site for SRM use. The databases will store the recovery plans and inventory information. 2. Install an SRM server on both sites. 54

55 3. Install a RecoverPoint SRA on the SRM server on both sites to enable integration. 4. Install the SRM plug-in on at least one vsphere client. SRM configurations are transparent to both sites, but it is highly recommended to install the SRM plug-in on both sites. For detailed steps, refer to VMware vcenter Site Recovery Installation and Configuration Center Site Recovery 5.1. Step 2: Establishing connections between protected and recovery sites Before configuring SRM, change the consistency group s management mode to Group is managed by SRM, RecoverPoint can only monitor. You can change this on the Policy tab after selecting the consistency group, as shown in Figure 32. Figure 32. Policy tab in a RecoverPoint consistency group When this is completed, the two SRM servers must be connected in SRM to establish reciprocity. You can do this through the Connect to Remote Site wizard, by specifying the IP/port details and authenticating credentials of the other (remote) SRM server, as shown in Figure

56 Figure 33. Connecting the remote site in SRM The new SRM interface contains an easy-to-follow guide tab that explains the steps throughout the configuration process. Step 3: Configuring RecoverPoint array managers After the protected and recovery sites have been paired, configure the RecoverPoint array managers (through the SRAs installed earlier) for SRM to discover replicated devices, compute datastore groups, and initiate storage operations. To do this, use the Add Array wizard, as shown in Figure

57 Figure 34. Adding storage arrays in SRM Typically, you need to perform this step only once after the sites are connected. Additional reconfiguration is not needed unless there is a change in connection information or credentials. After this step, all configurations can be done on just one SRM and will automatically be synchronized to the other SRM server. Step 4: Configuring protection groups You can create a protection group after configuring the array managers. A protection group is used to organize virtual machines that should be failed over together, such as in this solution where the SAP database, SAP Central Services instance, and three application instances are failed over as one entire set. You can create many protection groups, but all virtual machines assigned to a protection group must belong to the same datastore group (consistency group). The Create Protection Group wizard automatically detects the datastores protected by RecoverPoint and the virtual machines inside it. You need to specify which virtual machines among the list must be added to a protection group, as shown in Figure

58 Figure 35. Creating a protection group in SRM SRM then creates placeholder virtual machines automatically. Placeholders reserve the resources on the recovery site and act as a shadow standby before a recovery operation. The placeholders cannot be started independently of SRM and are removed altogether if the protection group is deleted. Figure 36 shows the actual virtual machines and the placeholders on the remote sites. Site A Site C Figure 36. Placeholders at each site In this solution, we created a separate protection group for VM2 and VM3 respectively, because both systems belong to different datastores and are cross-replicating toward each other s site. Step 5: Configuring inventory mappings and placeholder virtual machines As shown in Figure 37, the Resource mappings tab allows you to define which resources on the recovery site will serve as the counterparts to the ones on the protected site, such as network, resource pool, and virtual machines. SRM cannot 58

59 protect a virtual machine unless it has valid inventory mappings for key virtual machine resources. SRM derives the resource assignments of a newly created placeholder from the inventory mappings established on the protected site. EMC recommends that you group virtual machines into folders for easier management. Note A counterpart folder must exist in the recovery vcenter server as well Figure 37. Configuring resource mapping and placeholder in SRM Step 6: Customizing virtual machine recovery operations Especially in an SAP landscape, it is customary to start up applications according to the business priority (for example, SAP Production > Third Party Production > SAP Development and ERP > HR > BI), and in a particular order (such as DB server > SAP Central Services > App instances). These specifics can be customized to suit the recovery requirements of an organization to ensure that even complex recovery plans can be fully automated. This solution exploits this feature so that the SAP virtual machines are started up in the correct order, and a script is executed within the recovery sequence to start up the database, SCS, and all application instances, ready for login on handover. 59

60 Step 7: Customizing recovery site IP addresses One of the complicating factors in a typical failover is the readjustment of IP settings to adapt to the infrastructure differences after recovery. You may need to readjust IP settings on multiple virtual machines. SRM can also manage this tedious task. You can either do this individually by using the SRM plug-in, or by using a bulk IP customization utility (dr-ip-customizer.exe), which is especially useful for first-time configurations. This utility generates a comma-separated value (CSV) file containing the IP settings of all virtual machines configured for SRM failover. You can generate and edit the file as follows: 1. Log in to the command prompt or console of the vcenter server on the recovery site. 2. By using the dr-ip-customizer utility in the bin folder of the SRM installation path, generate the existing IP table and specify the file name and location where it will be saved. C:\Program Files (x86)\vmware\vmware vcenter Site Recovery \bin>dr-ip-customizer.exe cfg..\config\vmwaredr.xml cmd generate o <csv filename to generate> vc <VMware vcenter server hostname> 3. Edit the CSV file and customize the existing IP settings according to the IP settings on the recovery site. Figure 38 shows the CSV file opens in a typical spreadsheet application. Note that full IP addresses are not shown in this figure. Figure 38. CSV file where IP addresses can be changed 4. Run the dr-ip-customizer utility again to upload the edited CSV. C:\Program Files (x86)\vmware\vmware vcenter Site Recovery \bin>dr-ip-customizer.exe cfg..\config\vmwaredr.xml cmd apply csv <path to the csv file> vc <VMware vcenter server hostname> 60

61 Step 8: Creating a recovery plan A recovery plan is like an automated runbook. It controls every step of the recovery process, including the order in which SRM powers on and powers off virtual machines, the network addresses that recovered virtual machines use, and so on. When a recovery plan is created, it prescribes a series of recovery steps that SRM automates depending on the function executed. Figure 39 shows these functions. Figure 39. Functions in SRM GUI The following list explains each of the functions: Test: Online simulated recovery; SRM performs all recovery steps inside a test network so that the test can run without disrupting the virtual machines in the production environment. Cleanup: A function to undo the result of test recovery (such as unmounting recovered snapshots and returning placeholders) and to reset the recovery plan to a ready state. Recovery: The actual migration of virtual machines in the recovery plan to the recovery site. There are two types that differ only in error tolerance: Planned migration: If errors occur during migration, the execution is stopped and the errors must be resolved before resuming. Disaster recovery: If errors occur, SRM attempts to continue execution. Reprotect: An automated function that completely reverses the direction of protection after a failover. The original protected site becomes the new recovery site and the original recovery site becomes the new protected site, provided that the original site is operational. Cancel: Function to terminate an ongoing test at any time or an actual recovery during certain steps. 61