Business Continuity and Disaster Recovery for MEDITECH Enabled by EMC RecoverPoint and VMware vcenter Site Recovery Manager

Reference Architecture Business Continuity and Disaster Recovery for MEDITECH Enabled by EMC RecoverPoint and VMware vcenter Site Recovery Manager Abstract The document describes the reference architecture of a MEDITECH environment protected by an EMC RecoverPoint solution and VMware vcenter Site Recovery Manager. January 2014

Copyright 2014 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. VMware, ESX, vmotion, VMware vcenter, and VMware vsphere are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other trademarks used herein are the property of their respective owners. All trademarks used herein are the property of their respective owners. [VMware and/or Brocade may have substituted components of the original environment in this document with hardware of a similar (or higher) specification to the original equipment used in the EMC Proven Solution. The content contained in this document originated from a validated EMC Proven Solution. The modification introduced by [vendor(s)] may have caused changes in performance, functionality, or scalability of the original solution. Please refer to www.emc.com/solutions for further information on validated EMC Proven Solutions. Part Number h12764 2

Table of contents Reference architecture overview... 4 Document purpose... 4 Solution purpose... 4 Business challenge... 4 Technology solution... 4 Solution Architecture... 6 Architecture diagram... 6 Hardware resources... 7 Software resources... 8 Key components... 9 Introduction... 9 EMC RecoverPoint... 9 VMware vcenter Site Recovery Manager... 9 Brocade Ethernet/Fibre Channel technology... 9 Configuring EMC RecoverPoint... 11 Introduction... 11 Configuring RecoverPoint consistency groups... 11 RecoverPoint consistency group considerations... 12 Configuring the consistency group for management by SRM... 12 Configuring site recovery with VMware vcenter Site Recovery Manager... 13 Introduction... 13 Prerequisites... 13 Installing and configuring SRM... 13 Configuring Advanced SRM... 15 EMC VSI for VMware vsphere: EMC RecoverPoint Management... 17 Additional recommendations... 17 Testing the MEDITECH recovery plan... 18 Overview... 18 Testing the recovery plan... 18 Recovery test report... 20 Conclusion... 21 References... 22 EMC documentation... 22 VMware documentation... 22 3

Reference architecture overview Document purpose Solution purpose Business challenge Technology solution The document describes the reference architecture of a MEDITECH environment protected by EMC RecoverPoint solution and VMware vcenter Site Recovery Manager, which was tested and validated by the EMC E-Lab Verticals Solutions Group. This reference architecture validates the functionality of the EMC RecoverPoint solution and VMware vcenter Site Recovery Manager (SRM) and provides guidelines to build similar solutions. Companies that have deployed the MEDITECH EMR Suite depend heavily on the information, processes, and availability of their deployed application environments. In the event of an outage, either planned (testing, upgrades and updates, repairs) or unplanned disasters, the complexity and distributed nature of these applications make implementation and maintenance of traditional disaster recovery solutions lengthy, expensive, and complicated. Traditional recoveries follow a written playbook, full of technical details on Internet IP address changing, zoning, storage masking, the order of servers to bring up at another site, and much more. Each step must be done manually and flawlessly. Typically, experts in IP protocol, storage management and design, VMware, MS Windows, (to name some of the key technologies involved in a recovery), are required to be onsite. But, what are the chances that all these people will be on site to help immediately if a flood or earthquake strikes in the middle of the night? To compound these challenges, community hospitals and smaller health delivery sites are even less likely to have all the expertise on staff, let alone on site 7X24 immediately following disaster, or even a planned outage. EMC RecoverPoint and VMware virtualization technology helps healthcare delivery organizations gain key advantages: Maintains rapid access to patient data by restoring your environment quickly. Avoids loss by expediting failover and recovery from a major service disruption. Accelerates disaster recovery, minimizing its inherent risks and downtime by eliminating manual recovery steps. Delivers advanced capabilities for disaster recovery management, nondisruptive testing, and automated failover, especially with long-distance replication. Enables automated failover across geographically dispersed locations. Requires minimal administration and minimal additional hardware. The Business Continuity and Disaster Recovery for MEDITECH, enabled by EMC RecoverPoint and VMware vcenter Site Recovery Manager (SRM) solution, focuses on disaster recovery between two VNX7500 arrays enabled by RecoverPoint continuous remote replication (CRR). The virtualization platform for the solution is enabled by SRM and VMware vsphere. Integration of RecoverPoint CRR and VMware SRM enables automated failover of a 4

MEDITECH environment from the production site to the recovery site and ensures that data replicated to the recovery site is available to the recovery site servers. This solution incorporates the following components: EMC RecoverPoint Storage Replication Adapter (SRA) for VMware vcenter Site Recovery Manager (SRM) EMC VSI for VMware vsphere: EMC RecoverPoint Management 5

Solution Architecture Architecture diagram This solution was designed to reflect a multisite virtualized MEDITECH environment protected by (SRM). Figure 1 depicts the overall physical architecture of the solution. VMware ESXi 5.1 SRM Server Brocade VDX6720 LAN SRM Server VMware ESXi 5.1 Brocade DS6510 SAN Recoverpoint Recoverpoint Brocade DS6510 SAN Protected Site Recovery Site VMware Client with Vmware SRM & EMC VSI VMware Client with Vmware SRM & EMC VSI Figure 1. Solution architecture 6

Hardware resources Table 1 lists the hardware used in this solution. Table 1. Solution hardware Hardware Quantity Configuration Notes EMC VNX7500 2 DAEs configured with: 78 600 GB 10k rpm SAS disks 16 200 GB Flash drives VNX shared storage providing block storage EMC RecoverPoint 6 Gen 5 hardware 1 RPA cluster per site with 3 RPAs per cluster. Brocade 6510 SAN switch 4 8 GB/s FC switches 2 switches per site, dual FC fabric configuration Brocade VDX6720 Network Switch 2 10 or 1 Gigabit Ethernet switches Infrastructure Ethernet switch Intel server 6 4 octo-core Intel Xeon E7-4830 Family 128 GB RAM Converged network adapter Rackmount systems 3 servers per vsphere cluster. Two clusters, each hosting MEDITECH environment and Infrastructure virtual machines (VMware vcenter, VMware SRM, DNS, DHCP, Active Directory, MS SQL Server,) Windows 2008 Enterprise virtual machines. MEDITECH File Servers (FS) VM vcenter Server Appliance VM vcenter Site Recovery Manager Server VM 30 2 vcpu 4 GB Memory 40 GB VMDK 200 GB RDM (physical mode) 2 2 vcpu 8 GB Memory 25 GB VMDK 100 GB VMDK 2 2 vcpu 8 GB Memory 40 GB VMDK Generic File Server configuration representing MEDITECH FS application server. Windows 2008 R2 SP1 installed. SUSE Linux appliance for managing vsphere clusters and SRM operations. vcenter Site Recovery Manager application host SQL Server VM 2 2 vcpu 8 GB RAM 75 GB VMDK Database server to house SRM database. 7

Software resources Table 2 lists the software used in this solution. Table 2. Solution software Software Configuration Notes EMC VNX7500 VNX OE for block Release 32 Operating environment for the block EMC RecoverPoint Release 4.0 SP1 Operating environment for each RPA VMware vsphere ESX 5.5.0 Server hypervisor VMware vcenter 5.5.0 vcenter server appliance VMware vcenter Site Recovery Manager (SRM) EMC RecoverPoint Storage Replication Adapter (SRA) for VMware vcenter Site Recovery Manager EMC VSI for VMware vsphere: EMC RecoverPoint Management 5.5.0 vcenter SRM application 2.2.0 Facilitates SRM functions with EMC RecoverPoint 5.6.0 VSI plugin for vcenter to manage RecoverPoint point-in-time backups for SRM operations. EMC PowerPath Virtual Edition 5.9 Multipathing and load balancing for block access. Microsoft SQL Server 2008 Database that hosts the tables for VMware SRM. Microsoft Windows 2008 R2 Operating system for the server environment 8

Key components Introduction EMC RecoverPoint VMware vcenter Site Recovery Manager Brocade Ethernet/Fibre Channel technology This section briefly describes the key components used in this solution, including: EMC RecoverPoint VMware vcenter Site Recovery Manager (SRM) Brocade Ethernet/Fiber Channel Technology EMC RecoverPoint is an advanced enterprise-class disaster recovery solution supporting heterogeneous storage and server environments. RecoverPoint provides bi-directional local and remote data replication across any distance and utilizes continuous data protection technologies to provide consistent point-in-time recovery. VMware vcenter Site Recovery Manager (SRM) is the market-leading disaster recovery management product. It ensures the simplest and most reliable disaster protection for all virtualized applications. SRM leverages cost-efficient vsphere Replication and supports a broad set of high-performance storage-replication products to replicate virtual machines to a secondary site. SRM provides a simple interface for setting up recovery plans that are coordinated across all infrastructure layers, replacing traditional, error-prone run books. Recovery plans can be tested non-disruptively as frequently as required to ensure that they meet business objectives. At the time of a site failover or migration, SRM automates the failover and failback processes, ensuring fast and highly predictable recovery point objectives (RPOs) and recovery time objectives (RTOs). The Brocade VDX 6720 Switch is a high-performance, ultra-low latency wire-speed 10 Gigabit Ethernet (GbE) fixed configuration switch. It is specifically designed to improve network utilization, maximize application availability, increase scalability, and dramatically simplify network architecture in virtualized data centers. Brocade VDX switches feature Brocade VCS Fabric technology that enables organizations to build high-performance, cloud-optimized data centers while preserving existing network designs and cabling, and gaining active-active server connections. For scale-out fabric architectures, Brocade VCS Fabric technology allows organizations to flatten network designs, provides Virtual Machine (VM) mobility without network reconfiguration, and manages the entire fabric more efficiently. The Brocade VDX 6720 Switch offers: High performance and low latency: Provides high performance with 10 GbE ports and ultra-low latency through wire-speed ports with 600 nanosecond port-to-port latency and automated hardware-based Inter-Switch Link (ISL) trunking. Simplified architecture: Simplifies network architectures and enables cloud computing with Brocade VCS Fabric technology. Easy configuration: Enables organizations to manage an entire VCS fabric as a single switch with Brocade VCS Logical Chassis. Maximizes network utilization: Allows efficient, load-balanced multipathing at Layers 1, 2, and 3, with multiple Layer 3 gateways. 9

Advanced storage connectivity: Provides Ethernet storage connectivity for FCoE, iscsi, and NAS. Optimizes virtualization: Offers the automation needed to support highlyvirtualized server and storage environments while enabling the transition to cloud computing. The Brocade 6510 Switch meets the demands of hyper-scale, private cloud storage environments by delivering market-leading Gen 5 Fibre Channel technology and capabilities that support highly-virtualized environments. Designed to enable maximum flexibility and reliability, the Brocade 6510 is configurable in 24, 36, or 48 ports and supports 2, 4, 8, 10, or 16 Gbps speeds in an efficiently designed 1U package. A simplified deployment process and a point-and-click user interface make the Brocade 6510 both powerful and easy to use. The Brocade 6510 offers low-cost access to industry-leading Storage Area Network (SAN) technology while providing "pay-as-you-grow" scalability to meet the needs of an evolving storage environment. Additional highlights include: Provides exceptional price/performance value, combining flexibility, simplicity, and enterprise-class functionality in a 48-port switch for virtualized data centers and private cloud architectures Enables fast, easy, and cost-effective scaling from 24 to 48 ports using Ports on Demand (PoD) capabilities Simplifies management through Brocade FabricVision technology, reducing operational costs and improving application performance Simplifies deployment and supports high-performance fabrics by using Brocade ClearLink Diagnostic Ports (D_Ports) to identify optic and cable issues Helps maximize application uptime and performance while reducing overall operational expenses with ClearLink Diagnostic Ports (D_Ports) Simplifies and accelerates deployment with the Brocade EZSwitchSetup wizard Maximizes availability with redundant, hot-pluggable components and nondisruptive software upgrades 10

Configuring EMC RecoverPoint Introduction EMC RecoverPoint sits below the VMware infrastructure and is responsible for replicating all changes from the production LUNs to the remote replica LUNs at the disaster recovery site. For this solution, VMware vcenter Site Recovery Manager (SRM) leverages RecoverPoint continuous remote replication (CRR) to provide external replication between protected and recovery sites. Configuring RecoverPoint consistency groups Once RecoverPoint is installed and replication sites established, the next step in setting up and testing a disaster recovery plan using SRM with RecoverPoint is to configure RecoverPoint consistency groups for the VMware volumes that are to be protected and managed by SRM. The general procedure is as follows: 1. Create consistency groups. 2. Configure copies. 3. Add journals. 4. Add replication sets. 5. Enable group. 6. Start replication. This process is described in detail in the EMC RecoverPoint administrator s guide, located at http://support.emc.com. Figure 2 shows the consistency groups in the RecoverPoint Management console. Figure 2. Consistency Groups in the RecoverPoint Management console 11

For this solution, three consistency groups were created with 10 MEDITECH FS servers assigned to each group. For MEDITECH implementations using RecoverPoint, there can be more or less numbers of consistency groups configured; it depends on the RPO requirement the site is requiring. RecoverPoint consistency group considerations When creating RecoverPoint consistency groups for SRM management, the RecoverPoint consistency group has to contain the protected VM s boot drive and any data volume associated with it. The boot volume and data volume can be on different VMware datastores but cannot be in separate RecoverPoint consistency groups because when creating protection groups in SRM, SRM computes the set of virtual machines into a datastore group based on the RecoverPoint consistency group. SRM checks the datastore group to ensure it contains all the files of a protected virtual machine. If a volume is not present in the datastore group, the VM will not be protected by SRM. Configuring the consistency group for management by SRM After the consistency group has been created and SRM has been installed, the consistency group needs to be configured to be managed by SRM. This is done in the RecoverPoint management console. Select the consistency group that is to be protected by SRM and go to the Group Policy tab to adjust the settings. Figure 3 shows the external management of the consistency group to SRM in the RecoverPoint Management console. Figure 3. External Management for Medi_CG1 set for SRM management 12

Configuring site recovery with VMware vcenter Site Recovery Manager Introduction Prerequisites Installing and configuring SRM VMware vcenter Site Recovery Manager (SRM) is a plug-in to VMware vcenter, so disaster recovery tasks can be executed from the same centralized interface that is used to manage other virtual machine administrative tasks such as creation, migration, and deletion. However, SRM is not a built-in component of vcenter; it is a separate server process with its own separate database. The server processes for SRM and vcenter can coexist on the same server or reside on different servers. Similarly, both SRM and vcenter data repositories can be created in a single database or in separate databases. SRM has several requirements that need to be met in order to have a successful installation. Each site must include a vcenter server containing at least one vsphere data center. The recovery site must support array-based replication with the protected (production) site, and must have hardware and network resources that can support the same virtual machines and workloads as the protected site. At least one virtual machine must be located on a datastore that is replicated by RecoverPoint at the protected site. The protected and recovery sites must be connected by a reliable IP network. Storage arrays may have additional network requirements. The recovery site must have access to the same public and private networks as the protected site, though not necessarily the same range of network addresses. VMware tools must be installed on all virtual machines. Installing and configuring SRM with RecoverPoint includes the following tasks: 1. Configure SRM databases at both sites. 2. Install the SRM server at both sites. 3. Pair the Protected and Recovery site. 4. Set up Inventory Mappings. 5. Install EMC RecoverPoint Storage Replication Adapter (SRA) for VMware vcenter Site Recovery Manager (SRM). 6. Configure protection groups. 7. Create the recovery plan. 8. Install EMC VSI for VMware vsphere: EMC RecoverPoint Management. Details of the installation process can be referenced from the Site Recovery Manager Installation and Configuration Guide and the VSI for VMware vsphere: EMC RecoverPoint Management 5.6 Product Guide. 13

For this solution, a single protection group was created to failover the thirty MEDITECH servers. Multiple protection groups could be created as long as the protection groups match the number of RecoverPoint consistency groups that are managed by SRM. Explanation about array-based replication and protection groups can be found in the Site Recovery Manager Administration Guide, located on the VMware website. Figure 4 shows the protection group created. Figure 4. Protection group A single MEDITECH recovery plan was used in the solution. The recovery plan specifies how SRM recovers the virtual machines in the protection group. Just like the protection group, it is possible to have multiple recovery plans to recover mission critical, business critical, or business important systems independently. When using multiple recovery plans, SRM will execute one recovery plan at a time. Within the MEDITECH recovery plan the FS servers were divided into several priority groups. This allows granularity on which VMs go down first and come up first in a recovery scenarios. Within a priority group, dependencies can be created so that certain VMs in that priority group come up in a certain order. Figure 5 shows priority of the FS servers that are to start up during a recovery. FS002 has a dependency to FS001 so it will not power up until FS001 is up. 14

Figure 5. Recovery Plan with priority groups and dependencies defined Configuring Advanced SRM Using the Advanced Settings, you can view or change many custom settings for the SRM service. Advanced Settings provide a way for a user with adequate privileges to change default values that affect the operation of various SRM features. SRM applies the advanced settings to the virtual machines that you protect on a given site, and not to recovery plans. SRM applies advanced settings to a virtual machine at the moment that you configure protection on that virtual machine. If you change any of the advanced settings after you have configured the protection of a virtual machine, the new settings do not apply to that virtual machine. Modifications to advanced settings apply only to virtual machines that you protect after you changed the settings. This is by design, because if SRM were to apply changed advanced settings to virtual machines on which you have already configured protection, it could lead to unwanted changes in the protection of those virtual machines. For this solution, a few of the advanced settings were modified to improve the recovery process. Settings were applied on both the protected site and recovery site. These suggested values can be changed to meet the site needs. Recovery > Recovery.powerOffTimeout value changed from 300 to 600 sec. The Recovery.powerOffTimeout setting is used during recovery on the protected site to monitor the power state of the VM. If the VM does not shut down gracefully during this timeout period, vsphere will power the VM down forcefully. See Figure 6 for the setting. Recovery > Recovery.powerOnTimeout value changed from 300 to 600 sec. The Recovery.powerOn.Timeout setting is used during recovery on the recovery site to monitor the power state of the VM based on VMtools heartbeat. If the heartbeat does not respond during this timeout, the VM will be flagged as failed in the recovery plan and user intervention will be needed to troubleshoot. See Figure 6 for the setting. 15

Figure 6. Advanced settings for recovery timeouts on protected site storageprovider > storageprovider.fixrecovereddatastorenames check on. This setting was more for aesthetics in the viclient for when the datastores are on the recovery site. During a recovery, the replicated datastores are prefixed with a snap-xxx to identify that the replica datastore is being used. With the check mark selected, vcenter will rename the datastore without a prefix. See Figure 7 for the setting. Figure 7. Advanced settings for storageprovider on protected site. 16

EMC VSI for VMware vsphere: EMC RecoverPoint Management EMC RecoverPoint Management is a plugin of the EMC Virtual Storage Integrator (VSI) framework for VMware vsphere. EMC RecoverPoint Management VSI is designed to assist the SRM administrator in designating a specific point-in-time (PiT) copy to use for data protection and execution of a recovery plan when using SRM. During a recovery operation, if a point-in-time is selected using the VSI, VMs will be restored to that specific backup. If a PiT is not chosen, then SRM uses the latest PiT that it is aware of. Figure 8 shows the PiT available for VM FS001 in the EMC RecoverPoint Management VSI. MEDITECH_MBF_Backup PiT is selected for recovery. Figure 8. List of PiT copies for FS001 from the RecoverPoint Management VSI Additional recommendations Based on testing in the Vertical Solutions lab, other recommendations are as follows: Install the Site Recovery Manager database as close to the Site Recovery Manager server as possible. This reduces the round trip time (RTT) between them. The impact of round trips to the database server on recovery time performance is also reduced. A vsphere alarm should be created to announce the creation of new VMs on the datastore so that protection can be configured to include the new VM in the RecoverPoint consistency group and SRM protection group. 17

Testing the MEDITECH recovery plan Overview An SRM recovery plan can be tested at any time without disrupting replication or ongoing operations at the protected site. The non-disruptive test is carried out in an isolated environment at the recovery site, using a temporary copy of the replicated data. It runs all the steps in the recovery plan except powering down of the virtual machines at the protected site and assumption of control of replicated data by devices at the recovery site. This facility allows the recovery plan to be tested for disaster recovery compliance confirming timings and reliability. Another use case is having a temporary test/ development environment to troubleshoot issues or host/application patching validation. Testing the recovery plan This section describes a test recovery of the MEDITECH recovery plan. Initiate the test required by selecting the MEDITECH Recovery Plan and clicking Test. Figure 9 shows the initiating of the recovery of the MEDITECH Recovery Plan. Figure 9. Initiating a test recovery When running a test plan, SRM creates an isolated network on the recovery site and enables image access on the RecoverPoint consistency groups. The test VMs will be powered up and any customizations that need to be done to them will be applied. 18

Figure 10 shows the recovery steps performed during the MEDITECH Recovery Plan test. Figure 10. Recovery steps for test plan At this point, the environment is available on the recovery site and administrators can verify application functionality in the secure environment. Once the environment has been validated, click Cleanup to return the recovery plan back to a ready state. The cleanup will power down the VMs, unmount the storage, and disable image access on RecoverPoint. 19

Recovery test report After every recovery/reprotect or test recovery/cleanup operation, a recovery plan history report is generated. The history contains information about the result and the start and end times for the whole plan and for each step in the plan. The reports can be used to demonstrate timings of the recovery plan as VMs are protected or unprotected or for auditing purposes to meet compliance. Figure 11 shows a recovery plan history for the MEDITECH Recovery Plan. Figure 11. MEDITECH Recovery plan test recovery history report 20

Conclusion This document, using the example of a multi-tier MEDITECH application deployed on VMware virtual infrastructure, demonstrates how VMware vcenter Site Recovery Manager (SRM) with EMC RecoverPoint software enables the design of an effective and automated disaster recovery point-in-time solution. Traditional disaster recovery solutions are slow and prone to failures because they involve many manual and complex steps that are difficult to test and require expensive duplication of the production data center infrastructure to ensure reliable recovery. SRM is specifically designed to leverage the capabilities of vsphere, to simplify and automate the disaster recovery process so that you can reliably recover from data center outages in hours rather than days. SRM integrates tightly with vcenter Server to simplify disaster recovery management by automating the testing and orchestration of centralized recovery plans. MEDITECH users can replace traditional, manual runbooks with centralized recovery plans, reducing the time required for set up from weeks to minutes. When deployed in conjunction with EMC RecoverPoint continuous remote replication (CRR) in a virtualized MEDITECH environment, users can leverage the EMC VSI RecoverPoint management plugin to allow administrators to recover to any point in time across the virtualized MEDITECH universe during a failover. Using SRM and RecoverPoint for this solution lets you: Accelerate recovery for the virtual environment through automation. Recover to specific points in time, either the latest replicated crash consistent copy or from a MEDITECH Integrated Disaster Recovery (IDR) backup. Ensure reliable recovery by enabling non-disruptive testing. Test as frequently as you like. Simplify recovery by eliminating complex manual recovery steps and centralizing management of recovery plans. 21

References EMC documentation VMware documentation The following documents, located on the EMC online support website, provide additional and relevant information. Access to these documents depends on your login credentials. If you do not have access to a document, contact your EMC representative: RecoverPoint Installation and Deployment Guide EMC RecoverPoint 4.0 Administrator s Guide RecoverPoint Deploying with VNX/CLARiiON Arrays and Splitter Technical Notes RecoverPoint Replicating VMware Technical Notes VSI for VMware vsphere: EMC RecoverPoint Management 5.6 Product Guide VSI for VMware vsphere: EMC RecoverPoint Management 5.6 Release Notes The following VMware documents, located on the VMware website, also provide useful information: Site Recovery Manager Installation and Configuration Site Recovery Manager Administration 22