BUSINESS CONTINUITY AND DISASTER RECOVERY FOR ORACLE 11g ENABLED BY EMC SYMMETRIX VMAXe, EMC RECOVERPOINT, AND VMWARE vcenter SITE RECOVERY MANAGER

Transcription

1 White Paper BUSINESS CONTINUITY AND DISASTER RECOVERY FOR ORACLE 11g ENABLED BY EMC SYMMETRIX VMAXe, EMC RECOVERPOINT, AND VMWARE vcenter SITE RECOVERY MANAGER EMC SOLUTIONS GROUP Abstract This white paper describes a data protection and disaster recovery solution for virtualized Oracle Database 11g OLTP environments, enabled by EMC Symmetrix VMAXe with Enginuity for VMAXe, EMC RecoverPoint, and VMware vcenter Site Recovery Manager. It covers both local data protection and automated failover and failback between remote sites. July 2011

2 Copyright 2011 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. VMware, ESX, VMware vcenter, and VMware vsphere are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other trademarks used herein are the property of their respective owners. Part Number H8207 2

3 Table of contents Executive summary... 6 Introduction to EMC Symmetrix VMAXe... 6 Business case... 7 Solution overview... 7 Key results... 7 Introduction... 8 Purpose... 8 Scope... 8 Audience... 8 Terminology... 8 Key technology components Solution components Oracle application EMC Symmetrix VMAXe with Enginuity Overview Virtual Provisioning Symmetrix Management Console EMC VNX VMware vsphere EMC RecoverPoint Overview RecoverPoint data protection options RecoverPoint appliance RecoverPoint splitter RecoverPoint journals RecoverPoint consistency groups VMware vcenter Site Recovery Manager RecoverPoint and SRM integration RecoverPoint CDP and CRR how they work Continuous data protection Continuous remote replication Solution architecture and design Solution architecture Environment profile Hardware environment Software environment Symmetrix VMAXe storage allocation

4 Virtual machine resources Oracle database configuration Oracle database deployment on Symmetrix VMAXe Virtual Provisioning for Oracle ASM disk groups Database schema Configuring EMC RecoverPoint Preparing the VMAXe for RecoverPoint connectivity RPA repository and gatekeeper provisioning Enabling write protect bypass for RPA initiators RecoverPoint configuration for this solution Journal size Installing RecoverPoint Configuring CLR Overview Step 1: Present the storage to be replicated to the RPA clusters Step 2: Tag the protected devices for RecoverPoint use Step 3: Configure RecoverPoint access to the RecoverPoint splitters Step 4: Create the consistency group Configuring the consistency group for management by SRM Automating site recovery with VMware vcenter Site Recovery Manager Overview Prerequisites Step 1: Install and configure SRM Step 2: Connect the protected and recovery sites Step 3: Configure RecoverPoint array managers Step 4: Configure protection groups Step 5: Configure inventory mappings Step 6: Customize virtual machine recovery options Step 7: Customize recovery site IP addresses Step 8: Create the recovery plan Testing the Oracle Failover recovery plan Overview Testing the recovery plan Verifying the success of the recovery test The recovery test report Benefits of testing recovery plans RecoverPoint CRR site failover process

5 Overview Running the Oracle Failover recovery plan Restarting the Oracle database Verifying data integrity at the recovery site Benefits of RecoverPoint site failover with SRM RecoverPoint CRR site failback process Overview Housekeeping Configuring SRM for failback Testing the failback recovery plan Failing back to the production site Verifying failback to the production site Benefits of RecoverPoint site failback with SRM RecoverPoint CDP database recovery Overview Preparing the test scenario Recovering the database from the CDP image Benefits of database protection with RecoverPoint RecoverPoint operation when VMAXe is operating in a degraded mode Overview Setting up VMAXe degraded mode Testing degraded mode Conclusion Summary Findings References White papers Product documentation Other documentation

6 Executive summary Introduction to EMC Symmetrix VMAXe EMC Symmetrix is the world s most trusted storage platform and enterprise customers have been deploying large mission-critical applications with Symmetrix for over 20 years. Symmetrix VMAX is the world s most scalable storage array, with the richest feature set, best availability, and best performance in the industry. The Symmetrix VMAXe series is a new class of enterprise storage, which delivers an efficient and cost-effective hardware design combined with built-in software and simplified installation, configuration, and management. VMAXe bring high-end storage array capabilities to service providers, healthcare organizations, and others with demanding virtual computing environments but limited storage expertise and IT resources. The VMAXe employs an implementation of the Symmetrix Virtual Matrix Architecture that is optimized for rapid deployment and easier management. The system can scale from a single-bay, single-engine configuration to a sixbay, four-engine system with 960 drives and up to 1.3 PB of usable capacity. This enables customers to cost-effectively grow and upgrade their system to accommodate application and data growth. The Enginuity operating environment the intelligence that controls all VMAXe array components is delivered preconfigured with EMC Virtual Provisioning and EMC Fully Automated Storage Tiering for Virtual Pools (FAST VP). The 100 percent virtually provisioned environment improves storage utilization and reduces costs and administration time. FAST VP provides automatic and highly granular movement of sub-lun data between storage tiers. This optimizes performance and reduces cost, while radically simplifying management and increasing storage efficiency. The VMAXe uses 100 percent internal redundancy to deliver enterprise-class reliability, availability, and serviceability (RAS). EMC RecoverPoint is replication technology that uses sophisticated journaling techniques and write splitting to provide local and remote replication with DRVlike recovery to any point in time. The VMAXe has an integrated write splitter to support RecoverPoint replication. The VMAXe series is also designed for fast and efficient deployment and includes features that are particularly useful in small or crowded data centers: VMAXe systems are delivered preconfigured, 100 percent virtually provisioned, and ready for same-day installation and startup. With the VMAXe array dispersion capability, bays can be separated up to 10 meters apart, enabling very flexible deployments. High storage densities can also be achieved, with a single-rack, single-engine VMAXe capable of supporting 120 drives. 6

7 Business case Enterprise customers of all sizes and across different vertical segments experience similar challenges downtime, application and data growth, demanding SLAs, and budget restraints. Symmetrix VMAXe with Enginuity delivers multi-controller scale-out architecture, availability, and efficiency to enterprise customers to address these challenges. Data protection and disaster recovery are critical to all enterprises and are among the most important aspects of administration in Oracle database environments. EMC RecoverPoint is proven technology for high-availability Oracle environments, providing local and remote replication with no impact in asynchronous environments and very limited application degradation in synchronous environments: RecoverPoint s journaled replication architecture enables recovery to any point in time, which reduces the recovery point objectives (RPOs) for Oracle data. RecoverPoint maintains transaction-consistent images at the recovery site, which reduces the recovery time objectives (RTOs). The integrated RecoverPoint splitter for VMAXe simplifies database replication and supports operational and disaster recovery of virtualized environments. VMware vcenter Site Recovery Manager works with RecoverPoint to coordinate and automate the recovery process, and enables administrators to test their disaster recovery plans without impacting the production environment or ongoing replication. Solution overview This solution focuses on disaster recovery between heterogeneous arrays a Symmetrix VMAXe and an EMC VNX5700 enabled by RecoverPoint continuous remote replication (CRR) and the RecoverPoint splitter. Local point-in-time database recovery using RecoverPoint continuous data protection (CDP) is also documented. The solution demonstrates the data protection and disaster recovery capabilities of these technologies in the context of a virtualized Oracle Database 11g OLTP environment. The virtualization platform for the solution is enabled by VMware vsphere and VMware vcenter Site Recovery Manager (SRM). Integration of RecoverPoint CRR and VMWare SRM enables automated failover of the Oracle database from the production site to the recovery site and ensures that data replicated to the recovery site is available to the recovery site servers. Key results The solution offers the following key benefits: Simplified replication of Oracle OLTP database environments between heterogeneous arrays by using the integrated RecoverPoint splitter for VMAXe and the integrated RecoverPoint splitter for VNX Rapid recovery from unplanned disasters by using RecoverPoint in conjunction with Symmetrix VMAXe and EMC VNX arrays Automated failover and failback for planned and unplanned disasters, enabled by integration of RecoverPoint and VMware vcenter SRM Nondisruptive disaster recovery rehearsal using RecoverPoint and VMware vcenter SRM Point-in-time database recovery from bookmarked RecoverPoint images 7

8 Introduction Purpose This white paper describes a data protection and disaster recovery solution for virtualized Oracle Database 11g R2 OLTP environments, enabled by RecoverPoint, Symmetrix VMAXe and VNX RecoverPoint splitters, and VMware vcenter Site Recovery Manager (SRM). Scope The scope of this paper is to describe: The storage and virtualization infrastructure for the solution RecoverPoint and vcenter SRM configuration for automated failover and failback Nondisruptive disaster recovery rehearsal Failover to the recovery site Failback to the production site Local point-in-time recovery of the database RecoverPoint operation when VMAXe is operating in a degraded mode Audience This white paper is intended for Oracle database administrators, storage administrators, VMware administrators, EMC customers, and field personnel who want to understand the data protection and disaster recovery capabilities of Symmetrix VMAXe, RecoverPoint, and SRM in the context of virtualized Oracle OLTP databases. Terminology Table 1 defines terms used in this white paper. Term ASM CDP CLR CRR Table 1. Terminology Definition Automatic Storage Management. Oracle logical volume manager. Continuous data protection (see RecoverPoint data protection options). Concurrent local and remote replication (see RecoverPoint data protection options). Continuous remote replication (see RecoverPoint data protection options). Data device DR Enginuity Virtual Provisioning term for devices (not mapped to the host) that provide physical storage for thin devices. Data devices must be contained in a virtual pool before they can be used. Disaster recovery. The operating environment that provides the intelligence that controls all components in a VMAXe array. 8

9 Term FAST VP RDM RPA RPO RTO SMC SRM SYMCLI Thin device (TDev) Definition Fully Automated Storage Tiering for Virtual Pools. A feature of EMC Enginuity 5875 that provides automatic storage tiering at the sub-lun level. VP denotes virtual pools, which are Virtual Provisioning thin pools. Raw device mapping. A method of presenting a SAN/data device directly to a virtual machine. An alternative to using VMware VMFS. RecoverPoint appliance (see RecoverPoint appliance). Recovery point objective. The maximum acceptable time period between the last available consistent image and a disaster or failure. Recovery time objective. The maximum acceptable time to bring a system or application back to operational state after a failure or disaster. Symmetrix Management Console. A browser-based interface for managing EMC Symmetrix storage. VMware vcenter Site Recovery Manager. An extension to VMware vcenter that enables integration with array-based replication. Symmetrix Solutions Enabler command line interface. A cache-only device that is presented to a host. TDevs are pointers to units of physical storage contained in Virtual Provisioning thin pools. vcpu Virtual CPU. A processor within a virtual machine. VMware ESX 4.1 currently supports up to eight vcpus per virtual machine. Virtual pool VMFS A Virtual Provisioning thin pool, consisting of a collection of data devices that provides storage capacity for the thin devices that are bound to the pool. VMware Virtual Machine File System. 9

10 Key technology components Solution components This section provides an overview of the key components of the solution, as listed in Table 2. Table 2. Function Business application Solution components Components Oracle Database 11g R2 Enterprise Edition, with Oracle Grid Infrastructure and Oracle ASM Storage platforms Virtualization platform Replication and recovery Production site EMC Symmetrix VMAXe with Enginuity for VMAXe EMC Symmetrix Management Console Recovery site EMC VNX5700 EMC Unisphere VMware vsphere VMware vcenter EMC RecoverPoint VMware vcenter Site Recovery Manager EMC RecoverPoint Storage Replication Adapter for VMware vcenter Site Recovery Manager Oracle application The solution is designed to provide local protection and disaster recovery for consolidated Oracle Database 11g OLTP environments. It demonstrates these capabilities for a single-instance OLTP database deployed on a VMware ESX virtual machine. EMC Symmetrix VMAXe with Enginuity 5875 Overview The solution demonstrates the local protection and disaster recovery capabilities of Symmetrix VMAXe with Enginuity 5875, which provides the storage platform at the production site. The Symmetrix VMAXe system is built on the highly scalable EMC Virtual Matrix Architecture, which enables it to grow seamlessly and cost-effectively from an entrylevel, single-bay and single-engine configuration to a six-bay, four-engine system with 384 GB of cache memory, 960 drives, and up to 1.3 PB of usable capacity. Built with simplicity and ease-of-use in mind, the VMAXe is 100 percent virtually provisioned, with storage tiering managed automatically by EMC Fully Automated Storage Tiering for Virtual Pools (FAST VP). Flash, FC, and SATA drives are all supported, as well as RAID 1, 5, and 6 protection options. VMAXe systems are delivered preconfigured, ready for same-day installation and startup. They support all EMC Symmetrix monitoring and management tools, including the latest enhanced Symmetrix Management Console, which provides simpler installation and management with smart wizards. 10

11 The integrated RecoverPoint splitter for VMAXe enables local and/or remote replication for flexible and efficient RPOs and RTOs. Virtual Provisioning EMC Virtual Provisioning simplifies storage configuration and management, improves capacity utilization, and enhances performance by enabling the creation of thin devices that present applications with more capacity than is physically allocated in the storage array. The physical storage used to supply disk space to the thin devices comes from a thin pool, which is comprised of data devices that provide the actual physical storage. The thin pools can be grown or shrunk nondisruptively by adding or removing data devices. Virtual Provisioning supports local and remote replication with RecoverPoint on VMAXe. Virtual Provisioning is described in detail in the EMC Solutions Enabler Symmetrix Array Controls CLI Version 7.3 Product Guide. Symmetrix Management Console The Symmetrix Management Console (SMC) is a powerful, browser-based interface that simplifies management of EMC Symmetrix storage, from device creation to advanced Symmetrix features such as FAST VP, Virtual Provisioning, Autoprovisioning Groups, replication configuration, and monitoring. EMC VNX5700 For the solution, storage at the recovery site is provided by an EMC VNX5700 storage array, demonstrating RecoverPoint support for heterogeneous storage platforms. The VNX5700 is a member of the VNX series next-generation storage platform, powered by Intel quad-core Xeon 5600 series processors. It is designed to deliver maximum performance and scalability for midtier enterprises, enabling them to grow, share, and cost-effectively manage multiprotocol file and block systems. VNX arrays incorporate the RecoverPoint splitter, which supports unified file and block replication for local data protection and disaster recovery. EMC Unisphere is the central management platform for the EMC VNX series, providing a single combined view of file and block systems, with all features and functions available through a common interface. Unisphere is optimized for virtual applications and provides industry-leading VMware integration, automatically discovering virtual machines and ESX servers and providing end-to-end, virtual-to-physical mapping. VMware vsphere VMware vsphere provides the virtualization platform for the solution, with VMware ESX virtual machines hosting the database and management systems at both the production and recovery sites. VMware vsphere abstracts applications and information from the complexity of underlying infrastructure, through comprehensive virtualization of server, storage, and networking hardware. It is the industry s most complete and robust virtualization platform, virtualizing business-critical applications with dynamic resource pools for unprecedented flexibility and reliability. VMware vcenter provides the centralized management platform for vsphere environments, enabling control and visibility at every level of the virtual infrastructure, 11

12 including integration with RecoverPoint to enable discovery of the protection status for virtual machines in the environment. EMC RecoverPoint Overview RecoverPoint is an advanced enterprise-class disaster recovery solution designed with the performance, reliability, and flexibility required for enterprise applications in heterogeneous storage and server environments. It provides bi-directional local and remote data replication, without distance limits and with minimal performance degradation. EMC RecoverPoint/EX is a product variant that is optimized for the EMC VMAXe and VNX series and the CLARiiON CX3 and CX4 series of storage arrays. EMC RecoverPoint/EX is the product used in this solution. RecoverPoint data protection options RecoverPoint provides the following replication options for both physical and VMware virtualized environments: Figure 1. RecoverPoint replication options Continuous data protection (CDP): CDP continuously captures and stores data modifications locally, enabling local recovery from any point in time, with no data loss. Both synchronous and asynchronous replication are supported. Continuous remote replication (CRR): CRR supports synchronous and asynchronous replication between remote sites over FC and a WAN. Synchronous replication is supported when the remote sites are connected through FC and provides a zero RPO. Asynchronous replication provides crashconsistent protection and recovery to specific points in time, with a small RPO. 12

13 Concurrent local and remote replication (CLR): CLR is a combination of CRR and CDP and provides concurrent local and remote data protection. In RecoverPoint, CDP is normally used for operational recovery, while CRR is normally used for disaster recovery the solution demonstrates a RecoverPoint CLR configuration. RecoverPoint appliance RecoverPoint is appliance-based, which enables it to better support large amounts of information stored across heterogeneous environments. This out-of-band approach enables RecoverPoint to deliver continuous replication with minimal impact to an application s I/O operations. RecoverPoint appliances (RPAs) run the RecoverPoint software and manage all aspects of data replication. For local replication, a cluster configuration of two or more active RPAs is deployed this supports immediate switchover to another appliance if one of the RPA nodes in a cluster goes down. For remote replication, a RecoverPoint cluster is deployed at both sites. RPAs use powerful deduplication, compression, and bandwidth reduction technologies to minimize the use of bandwidth and dramatically reduce the time lag between writing data to storage at the source and target sites. RecoverPoint splitter RecoverPoint uses lightweight write splitting technology, on the application server, in the fabric, or in the array, to mirror application writes to the RecoverPoint cluster. VMAXe and VNX arrays have integrated RecoverPoint splitters that operate in each front-end adapter (FA) this ensures that the RPA receives a copy of each write. The array-based splitter is the most effective write splitter for VMware replication, enabling replication of VMFS and RDM volumes without the cost or complexity of additional hardware. The splitter supports both FC and iscsi volumes presented by the VMAXe or VNX arrays to any host, including to an ESX server. RecoverPoint journals RecoverPoint journals store timestamped application writes for later recovery to selected points in time. Three journals are provisioned for local and remote replication one production journal at the production site, and a history journal at both the production and recovery sites. For synchronous replication, every write is retained in the history journal for recovery to any point in time. For asynchronous replication, several writes are grouped before delivery to the history journal this supports recovery to significant points in time. Bookmarked points-in-time can be created automatically or manually to enable recovery to specific application or system events. RecoverPoint consistency groups The consistency and write-order fidelity of point-in-time images are assured by RecoverPoint s use of replication sets and consistency groups. A replication set defines an association between a production volume and the local and/or remote volumes to which it is replicating. A consistency group logically groups replication sets that must be consistent with one another. The consistency group ensures that 13

14 updates to the production volumes are written to the replicas in consistent write order and that the replicas can always be used to continue working from or to restore the production source. RecoverPoint replication is policy-driven. A replication policy, based on a particular business need, can be uniquely specified for each consistency group. This policy governs the replication parameters for the consistency group for example, the RPO and RTO for the consistency group, and its deduplication, data compression, and bandwidth reduction settings. VMware vcenter Site Recovery Manager VMware vcenter Site Recovery Manager (SRM) is a disaster recovery framework that integrates with EMC RecoverPoint to automate recovery of VMware datastores so that it becomes as simple as pressing a single button. SRM is an extension to VMware vcenter that enables integration with array-based replication, discovery and management of replicated datastores, and automated migration of inventory from one vcenter to another. SRM does not replicate any data. For this, it leverages an external replication solution such as RecoverPoint. SRM servers coordinate the operations of the replicated storage arrays and vcenter servers at the production and recovery sites so that, as virtual machines at the production site are shut down, virtual machines at the recovery site start up and assume responsibility for providing the same services, using the data replicated from the production site. Migration of protected inventory and services from one site to the other is controlled by a recovery plan that specifies the order in which virtual machines are shut down and started up, the compute resources that are allocated, and the networks they can access. SRM integrated with RecoverPoint also enables recovery plans to be tested, using a temporary copy of the replicated data, in a way that does not disrupt ongoing operations at either site. RecoverPoint and SRM integration Integration of SRM with RecoverPoint is provided by the EMC RecoverPoint Storage Replication Adapter (SRA) for VMware vcenter Site Recovery Manager. The RecoverPoint SRA supports discovery of arrays attached to RecoverPoint and of consistency groups that are enabled for management by SRM. It also supports SRM functions such as failover and failover testing by mapping SRM recovery plans to the appropriate RecoverPoint actions. 14

15 RecoverPoint CDP and CRR how they work Continuous data protection Local protection of the production environment is performed by RecoverPoint continuous data protection (CDP), using the integrated RecoverPoint splitter. Figure 2 illustrates the flow of a write in CDP. Figure 2. RecoverPoint CDP data flow 1. The application server issues a write to a LUN that is being protected by RecoverPoint. The write is intercepted by the RecoverPoint splitter. 2. The splitter splits the write and sends it to the production volume and to the RPA simultaneously. 3. When the write is received by the RPA, it is acknowledged back to the splitter. 4. In parallel, the RPA moves the data into the local journal volume, along with a timestamp and any application, event, or user-generated bookmarks for the write. 5. After the data is safely stored in the journal, it is distributed to the target local volumes through the RPA FC connections write order is preserved during distribution. 15

16 Continuous remote replication Replication of the production environment to the recovery environment is performed by RecoverPoint continuous remote replication (CRR) using the RecoverPoint splitter. Figure 3 illustrates the flow of a write in CRR. Figure 3. RecoverPoint CRR data flow 1. The application server issues a write to a LUN that is being protected by RecoverPoint. The write is intercepted by the RecoverPoint splitter. 2. The splitter splits the write and sends it to the production volume and to the local RPA simultaneously, the same as in a CDP deployment. 3. When the RPA receives the write, it immediately acknowledges it back to the splitter, unless synchronous remote replication is in effect. With synchronous replication, the ACK is delayed until the write has been received by the RPA at the remote site. 4. When the write is received by the local RPA, it is bundled with other writes, deduplicated to remove redundant blocks, sequenced, and timestamped. The package is then compressed and transmitted with a checksum for delivery to the remote RPA cluster. 5. When the package is received at the remote site, the remote RPA verifies the checksum, to ensure the package was not corrupted in transmission, and uncompresses the data. 6. The RPA then writes the data to the journal volume at the remote site. 7. After the data has been written to the journal volume, it is distributed to the remote replica volumes through the RPA FC connections write order is preserved during this distribution. 16

17 Solution architecture and design Solution architecture EMC solutions are designed to reflect and validate real-world deployments. Figure 4 depicts the physical architecture of the solution described in this white paper. Figure 4. Solution architecture The solution demonstrates the local protection and disaster recovery capabilities of the EMC Symmetrix VMAXe with Enginuity, which provides the storage platform at the production site. Storage at the recovery site is provided by an EMC VNX5700 array, demonstrating RecoverPoint support for heterogeneous storage platforms. VMware vsphere provides the virtualization platform for the solution. 17

18 Environment profile Table 3 details the environment profile for the solution. Table 3. Solution profile Profile characteristic Details Database type OLTP Database size 500 GB Number of databases 1 Workload profile Swingbench Order Entry (TPC-C-like) workload Database read/write ratio 60/40 Network connectivity FC 8 Gb and 10 GbE Hardware environment Table 4 details the hardware environment for the solution. Table 4. Solution hardware environment Purpose Quantity Configuration Storage (production site) Storage (recovery site) 1 EMC Symmetrix VMAXe with: Single engine 96 GB cache memory 88 x 450 GB, 15k FC drives, plus spare 16 x 2000 GB SATA drives, plus spare 4 x 200 GB Flash drives, plus spare Enginuity for VMAXe Q2SR 1 EMC VNX5700 with: 8 Gb FC connectivity 1 Gb network connectivity 15 x 200 GB SATA Flash 15 x 300 GB SAS drives 45 x 2 TB NL-SAS drives 2 x Data Movers 8 x GbE NICs 1 x Control Station VNX Operating Environment VMware ESX servers 2 2 x quad-core Xeon 5560 CPUs, 2.80 GHz, 98 GB RAM 2 x 10 GB CNA adapters Network switches 2 Gigabit Ethernet switches FC switches 4 8 GB/s FC switches, 2 per site RecoverPoint appliances 4 GEN 4, 2 RPAs per site Host adapters 4 10 GB CNA adapter (2 per physical server) 18

19 Software environment Table 5 details the software environment for the solution. Table 5. Solution software environment Software Version Purpose EMC Symmetrix Management Console 7.3 Symmetrix VMAXe configuration and management tool EMC Unisphere 1.1 VNX management software EMC Solutions Enabler 7.3 Symmetrix VMAXe management software EMC RecoverPoint EMC replication software, installed on each of the 4 RPAs EMC RecoverPoint Adapter for VMware vcenter Site Recovery Manager EMC software for integrating RecoverPoint and SRM VMware vsphere 4.1 GA B Hypervisor hosting all virtual machines VMware vcenter 4.1 GA B Management of VMware vsphere VMware vcenter Site Recovery Manager Managing failover and failback of virtual machines Oracle Database 11g R2 Enterprise Edition Oracle database software for grid computing Red Hat Enterprise Linux 5.5 Server OS for Oracle database server Microsoft Windows Server 2008 R2 x64 Server OS for Swingbench load generator Swingbench Database workload generation tool 19

20 Symmetrix VMAXe storage allocation The Symmetrix VMAXe array is factory-configured, based on information provided by the customer at the time of ordering. Storage pools are preconfigured based on the protection schemes requested by the customer. To provision storage to the ESX hosts, simply run the Add New Host And Provision Virtual Storage wizard from the SMC dashboard. This guides you through the steps involved, as shown in Figure 5. Figure 5. Add New Host and Provision Virtual Storage wizard Table 6 details the VMFS volumes provisioned for the solution from the Symmetrix VMAXe array. All these volumes were replicated with RecoverPoint. Table 6. Symmetrix VMAXe storage allocation VMFS volume Capacity Meta members Thin pools OracleVM_DataStore 100 GB 1 VM_ FC_3R5 DATA 800 GB 16 FC_3R5 REDO 100 GB 2 FC_3R5 TMP 200 GB 4 FC_3R5 FRA 1,440 GB 8 SATA_6R6 Oracle_Binaries 20 GB 1 FC_3R5 Note: Volumes replicated with RecoverPoint require a target volume of the same size on the local array and remote array. 20

21 Virtual machine resources Table 7 details the virtual allocation of hardware resources for the virtual machines at the production site. Table 7. Virtual allocation of hardware resources for virtual machines Virtual machine role Quantity Configuration Oracle database node 1 6 vcpus, 24 GB RAM, RHEL 5.5 VMware vcenter/srm server* 1 2 vcpus, 8 GB RAM, Windows Server 2008 R2 x64 EMC SMC/SPA server* 1 2 vcpus, 4 GB RAM, Windows Server 2008 R2 x64 Swingbench server* 1 4 vcpus, 8 GB RAM, Windows Server 2008 R2 x64 Application server 1 1 vcpu, 8 GB RAM, Windows Server 2008 R2 x64 * These virtual machines reside on an existing ESX server used only for management purposes. 21

22 Oracle database configuration Oracle database deployment on Symmetrix VMAXe Virtual Provisioning for Oracle ASM disk groups The Oracle database for the solution has separate ASM disk groups for data files, redo logs, fast recovery area (FRA), and temp files. Separate VMFS volumes are presented to the virtual machines on dedicated datastores for each ASM disk group. The +DATA, +TEMP, and +REDO ASM disk groups are deployed on thin provisioned devices bound to a virtual pool composed of FC devices with RAID 5 (3+1) protection. This pool provides the high performance required by these devices. The thin device provisioned for the +REDO logs was fully provisioned at the time of deployment that is, all physical storage was assigned up front. The VMFS datastore for the +REDO log file was also provisioned to request all storage from the outset. The +FRA device is bound to a virtual pool composed of high-capacity SATA devices with RAID 6 (6+2) protection. FRA devices typically do not have the same highperformance demands as data files and redo logs, so SATA devices provide an efficient deployment for this data. Figure 6 shows the relationships from the ASM disk groups through to the thin pools. +DATA +REDO +TMP +FRA ASM disk group Oracle_DATA Oracle_REDO Oracle_TMP Oracle_FRA VMFS datastore Thin device (TDEV) Thin pool FC Pool RAID5(3+1) SATA Pool RAID6 (6+2) Figure 6. Relationship between ASM disk groups and storage pools Database schema A single instance of the Swingbench Order Entry PL/SQL (SOE) schema was used to deliver the OLTP workloads required by the solution. The Swingbench SOE schema models a traditional OLTP database, with tables and indexes residing in separate tablespaces. 22

23 The schema for the solution contains the tables and indexes listed in Table 8. Table 8. Table name CUSTOMERS Schema tables and indexes Index CUSTOMERS_PK (UNIQUE), CUST_ACCOUNT_MANAGER_IX, CUST_ _IX, CUST_LNAME_IX, CUST_UPPER_NAME_IX INVENTORIES ORDERS ORDER_ITEMS PRODUCT_DESCRIPTIONS PRODUCT_INFORMATION WAREHOUSES INVENTORY_PK (UNIQUE), INV_PRODUCT_IX, INV_WAREHOUSE_IX ORDER_PK (UNIQUE), ORD_CUSTOMER_IX, ORD_ORDER_DATE_IX, ORD_SALES_REP_IX, ORD_STATUS_IX ORDER_ITEMS_PK (UNIQUE), ITEM_ORDER_IX, ITEM_PRODUCT_IX PRD_DESC_PK (UNIQUE), PROD_NAME_IX PRODUCT_INFORMATION_PK (UNIQUE), PROD_SUPPLIER_IX WAREHOUSES_PK (UNIQUE) LOGON To verify the virtual database environment, the Swingbench Order Entry workload was run against the database, with a user count of 100, as shown in Figure 7. Figure 7. Swingbench workload running against the database 23

24 Configuring EMC RecoverPoint Preparing the VMAXe for RecoverPoint connectivity RPA repository and gatekeeper provisioning Configuring the RecoverPoint splitter on the Symmetrix VMAXe array requires provisioning of the following volumes: A repository volume (3 GB minimum) for the RPA cluster. This stores configuration information about the RPAs and RecoverPoint consistency groups, which enables a properly functioning RPA to seamlessly assume the replication activities of a failing RPA from the same cluster. A repository volume of the same size was also provisioned on the VNX5700 for the RPA cluster at the recovery site. Eight unique gatekeeper volumes each for RPA1 and RPA2. These volumes are provisioned by creating Auto-provisioning masking views that present the volumes to the RPAs. For the solution, three masking views were created for the RecoverPoint cluster, as it consists of two RPAs (RPA1 and RPA2): RecoverpointConfig this presents the repository volume, the journal volumes, and the replica volumes to all nodes in the RPA cluster RPA1_GK this presents the relevant gatekeepers to RPA1 RPA2_GK this presents the relevant gatekeepers to RPA2 Figure 8 shows creation of the RecoverpointConfig masking view using the Masking View Management Create dialog box in Symmetrix Management Console. For further information on Auto-provisioning masking views, consult Deploying RecoverPoint with Symmetrix Technical Notes. 24

25 Figure 8. Configuring masking views for a RecoverPoint cluster 25

26 Enabling write protect bypass for RPA initiators The RecoverPoint splitter for VMAXe requires the RPA initiators to have special access that enables them to write to write-protected devices. To grant this access, the write protect bypass initiator flag WP_Bypass(WPBP) must be set for all RPA initiators. Figure 9 shows the SMC procedure for doing this Choose Enable Recover Point command for initiator group. 2. SMC confirms initiator group enabled for RecoverPoint. 3. Initiator properties show WPBP flag enabled. Figure 9. Enabling write protect bypass for RPAs 26

27 RecoverPoint configuration for this solution RecoverPoint was configured as follows for the solution: CLR was the replication method used for the solution. This combines both CDP and CRR. Two consistency groups were created: Oracle_SRM this encompasses all the replication sets for the Oracle database, Oracle binaries, and the datastore containing the running operating system for the virtual machine that hosts the Oracle software. App_SRM this contains the replication sets for the datastore containing the Swingbench and application server virtual machines (see Table 7). By separating replicated virtual machines into multiple consistency groups, it was possible to plan for different disaster recovery scenarios. For example, the Oracle environment could be failed over separately from the other virtual machines, or the entire production environment could be failed over as a single process. For each consistency group, three journals were set up, two at the production site to support the production volumes and their local replicas, and one at the recovery site to support the remote replica. All journals on the VMAXe were bound to the FC_3R5 pool. For further details on configuring RecoverPoint, consult the EMC RecoverPoint Release Administrator s Guide. Journal size The size of the journal volumes should reflect your RPO. Determining the journal size requires administrators to calculate the expected peak change rate in their environment. The following is the Journal Volume Sizing formula: Journal size = (new data writes per second) x (required rollback time in seconds) (1 target side log size) x 1.05 Twenty percent of the journal must be reserved for the target side log and 5 percent for internal system needs. For example, to support a 24-hour rollback requirement (86,400 seconds), with 5 Mb/s of new data writes to the replication volumes in a consistency group, the calculation would be as follows: (5 x 86400) (1 0.2) x 1.05 = Mb = GB (~70 GB) For the solution, all journals were sized to 500 GB. With a change rate of 5 Mb/s, this enables RecoverPoint to roll back at least seven days. It may be possible to increase the recover point by bookmark consolidation and journal compression. 27

28 Installing RecoverPoint The RecoverPoint Installer Wizard, which is part of the RecoverPoint Distribution Manager, guides you through the procedures for installing RecoverPoint and setting up the RPA clusters at the production and recovery sites. Before running the wizard, it is essential that the steps described in Preparing the VMAXe for RecoverPoint connectivity have been completed, as the wizard prompts you to identify the repository volumes created then. Figure 10 shows the Prerequisites screen from the wizard, listing the conditions that must be met before using the wizard. It also shows the Summary screen, which lists the tasks completed during installation. Figure 10. RecoverPoint Installer Wizard Configuring CLR Overview When RecoverPoint installation has successfully completed, you can configure the RecoverPoint consistency group(s) required for local and remote replication of your application data. To do this, perform the following steps for each consistency group: 1. Present the storage to be replicated to the RPA clusters at both sites. 2. Tag all VMAXe devices that are to be protected by RecoverPoint. 3. Configure RecoverPoint access to the RecoverPoint splitters on the VMAXe and VNX arrays. 4. Create the RecoverPoint consistency group. This section describes the configuration process for the solution s Oracle_SRM consistency group. 28

29 Step 1: Present the storage to be replicated to the RPA clusters Local array The application production devices that are to be protected by RecoverPoint must be presented to the RPA cluster. To do this: 1. Launch Symmetrix Management Console for the local array. 2. Open the initiator group for the existing auto-provisioning view that contains the devices to be protected. 3. Add the RPA initiator groups for all RPAs, as shown in Figure 11. Figure 11. Adding RPA initiator groups Remote array The remote journal and the target devices for the CRR copy must be presented to the remote RPA cluster. How this is accomplished depends on the target array type. The target array for the solution is a VNX5700. In this case, a storage group containing the journal and target devices was set up and presented to the remote RPAs. A second storage group was created and presented to the ESX host at the remote site to provide access to the replica storage. For more detailed information on configuring VNX storage for replication with RecoverPoint, consult EMC RecoverPoint Deploying with VNX/CLARiiON Arrays and Splitter Technical Notes. Step 2: Tag the protected devices for RecoverPoint use In RecoverPoint splitter for VMAXe configurations, it is necessary to tag devices for RecoverPoint use. This makes the devices accessible to the splitter and also makes it easy to identify which VMAXe volumes have been set up for RecoverPoint use. Figure 12 shows the Symmetrix Management Console procedure for tagging the devices in the storage group for the ESX server. Use the same procedure to tag the replica volumes in the RecoverpointConfig storage group. 29

30 Figure 12. Tagging devices for RecoverPoint use Note: You must repeat this step whenever you add devices to the configuration, if they are to be protected by RecoverPoint. Step 3: Configure RecoverPoint access to the RecoverPoint splitters To configure RecoverPoint access to the RecoverPoint splitters on the VMAXe and VNX arrays, use the New Splitter Wizard, as shown in Figure 13. Figure 13. Adding new splitters Step 4: Create the consistency group Creating a consistency groups involves: 1. Defining the replication policy settings for the consistency group, the production copy (that is, the data to be protected), and the local and remote replica copies. 2. Adding replication sets to the consistency group by selecting the production volumes to be replicated and assigning the corresponding replica volumes at the local and/or remote copies. 3. Selecting the journal volumes for the production and replica journals. 30

31 The New Consistency Group Wizard, in the RecoverPoint Management Application, guides you through the required steps, as shown in Figure 14. The replication policy settings are all optional as the default settings provide a practical configuration. Figure 14. New Consistency Group Wizard steps The consistency groups for the solution were created with the wizard and the default values were accepted for all optional settings. Figure 15 shows the final summary screen for the Oracle_SRM consistency group, which was configured with a local copy (Oracle_CDP) and a remote copy (Oracle_CRR) to support both local data protection and remote replication. The summary screen includes the replication sets defined for the consistency group. Figure 15. Summary of consistency group settings 31

32 Configuring the consistency group for management by SRM After the consistency group has been created and SRM has been installed, you need to configure the consistency group for management by SRM. You do this by using the Policy settings in the RecoverPoint Management Application, as shown in Figure 16. consistency group policy settings Figure 16. Configuring the consistency group for management by SRM 32

33 Automating site recovery with VMware vcenter Site Recovery Manager Overview Installing and configuring VMware vcenter Site Recovery Manager (SRM) for automated site recovery using RecoverPoint involves these steps: 1. Install and configure SRM. 2. Configure the connection between the protected and recovery sites. 3. Configure RecoverPoint array managers. 4. Configure protection groups. 5. Configure inventory mappings. 6. Customize virtual machine recovery options. 7. Customize recovery site IP addresses. 8. Create the recovery plan. Most of these steps are executed from the SRM interface in vcenter, where intelligent wizards support quick and easy configuration. For full details of these steps, consult the VMware vcenter Site Recovery Manager Administration Guide 4.1. It is possible to create multiple recovery plans to cater for many different recovery scenarios. In the example shown in Figure 17, three recovery plans have been created one to fail over the entire environment, and the other two to fail over the application and database layers independently. Figure 17. Multiple recovery plans All these recovery plans were successfully executed as part of the solution testing. For the purpose of demonstrating the failover process using SRM and RecoverPoint, this white paper documents failover and failback of the entire production environment. This section describes how SRM was configured for failover from the production site to the recovery site. The RecoverPoint CRR site failback process section describes how SRM was configured for failback from the recovery site to the production site. 33

34 Prerequisites SRM has several requirements for the vsphere configuration at each site: Each site must include a vcenter server containing at least one vsphere data center. The recovery site must support array based replication with the protected (production) site, and must have hardware and network resources that can support the same virtual machines and workloads as the protected site. At least one virtual machine must be located on a datastore that is replicated by RecoverPoint at the protected site. The protected and recovery sites must be connected by a reliable IP network. Storage arrays may have additional network requirements. The recovery site must have access to the same public and private networks as the protected site, though not necessarily access to the same range of network addresses. VMware tools must be installed on all virtual machines. Step 1: Install and configure SRM Installing and configuring SRM includes the following tasks: 1. Configure SRM databases at both sites these store the recovery plans, inventory information, and so on. 2. Install the SRM server at both sites. 3. Install the RecoverPoint Storage Replication Adapter on the SRM server at both sites this enables SRM and RecoverPoint integration. 4. Install the SRM client plug-in into one or more vsphere Clients at either or both sites. 34

35 Step 2: Connect the protected and recovery sites With the SRM database and SRM server installed at each site, you then need to configure the connection from the protected site to the recovery site. You do this by specifying the remote SRM server IP details in the Connect to Remote Site wizard at the protected site, as shown in Figure Figure 18. Connecting the SRM and vcenter servers at the protected and recovery sites Step 3: Configure RecoverPoint array managers For SRM to integrate with RecoverPoint, RecoverPoint array managers must be configured at both the protected and recovery sites. You do this by using the SRM Configure Array Managers wizard. The wizard discovers the replicated storage devices at the protected and recovery sites, and identifies the VMFS datastores that they support. When finished, it presents a list of replicated datastore groups. Figure 19 shows the wizard in progress, with RecoverPoint specified as the manager type and Production as the protected site. The connection information for RecoverPoint at the protected site is also specified. 35

36 Figure 19. Configuring the RecoverPoint array managers Step 4: Configure protection groups A protection group is a collection of virtual machines that all use the same datastore group (the same set of replicated LUNs) and that all fail over together. To create a protection group, you select the datastore group(s) to protect, and specify a datastore group at the recovery site where SRM can create placeholders for members of the protection group. You use the Create Protection Group wizard to do this. The wizard automatically detects the datastores currently protected by RecoverPoint and allows you to select the one(s) to include in the protection group. Two protection groups were created for the use case one for the Oracle_SRM consistency group and the other for the App_SRM consistency group. Figure 20 shows the Oracle protection group being created. Virtual machines supported by selected datastore group Datastore group selected for inclusion in protection group Figure 20. Creating a protection group 36