Cookbook Disaster Recovery (DR)

Transcription

1 Cookbook Disaster Recovery (DR) IBM Global Technology Services: Khoa Huynh Daniel De Araujo Bob Kellenberger VMware: Merlin Glynn 1

2 Summary This document is part of a series of VMware@SoftLayer cookbooks. They are intended to provide vsphere administrators with important information to deploy VMware vsphere environments within SoftLayer. VMware@SoftLayer refers to the deployment of VMware vsphere infrastructure in SoftLayer as a Hybrid Cloud. In this particular case, a hybrid cloud refers to the deployment of a private cloud within the SoftLayer environment. This document provides how vsphere administrators can set up Disaster Recovery (DR) solutions for vsphere infrastructure and applications which are either deployed in a VMware-based private (on-premise) cloud or in a VMware-based hybrid cloud within the SoftLayer environment. A separate cookbook will cover data backup, archival, and recovery (BURA) solutions for VMware-based hybrid cloud environments. This document is intended for experienced vsphere administrators with basic deployment skills to install and administer VMware vsphere ESX and vcenter products. It is not intended to provide information on enabling operating system tasks within VM guest operating systems. Additionally, it will not cover advanced network design concepts to recover guest network services. 2

3 Table of Contents 1 Disaster recovery concepts VMware Site Recovery Manager (SRM) Disaster recovery scenarios Fail-over from on-premise VMware installation to SoftLayer Creating recovery site at SoftLayer Setting up VPN connection Installing Site Recovery Manager (SRM) and plug-in Site pairing Inventory resource mapping Setting up storage replication Creating protection groups Creating a recovery plan Executing a recovery plan Reprotect and reverse recovery Disaster recovery between two SoftLayer sites Creating recovery site in Dallas 6 data center Setting firewall rules in Vyatta network appliances Installing the Site Recovery Manager (SRM) and Plug-in Site pairing Inventory resource mapping Setting up storage replication Creating protection groups Creating a recovery plan Executing a recovery plan Re-protect and reverse recovery Conclusion

4 1 Disaster recovery concepts A typical disaster recovery solution involves a protected site and a recovery site. These sites can be different parts of the same data center or they can be two separate data centers in different geographical locations. It is assumed that the protected virtual machines and their IT resources & services are currently up and running in the protected site. In the event of a disaster at the protected site, the virtual machines and all of their IT resources and services would then be restored on a pre-determined recovery site with minimal business downtime. The downtime should be part of the service level agreement (SLA), which dictates the recovery point objective (RPO) and recovery time objective (RTO) for each business process. These need to be determined, documented, and mapped to the underlying business applications and computing resources (physical and/or virtual) on which they operate. After the original site has been repaired and brought back on-line, the affected virtual machines and their resources / services could then be migrated from the recovery site back to the original site. This is called reverse recovery process. More specifically, a disaster recovery scenario would involve the following operations: Pre-synchronize storage resources between protected and recovery sites Shut down the virtual machines at protected site and prepare them for migration to the recovery site Synchronize storage resources between protected and recovery sites Suspend non-critical virtual machines at recovery site to make room for migrated virtual machines from the protected site Change storage at recovery site to writable Power on the virtual machines at recovery sites (these virtual machines can be powered on in a pre-determined priority order if necessary) It should be noted that the private network bandwidth within SoftLayer data centers are un-metered, so large amounts of data transfers over the SoftLayer's private networks during DR operations would not incur additional data charges. 2 VMware Site Recovery Manager (SRM) VMware VirtualCenter (vcenter) provides centralized management for VMware ESX hosts and all of the virtual machines (VMs) that run on them. Through vcenter, all functions and configuration of the ESX hosts, virtual machines, virtual networking, and storage layers can be managed through a single point of control. For example, we can use the vcenter interface to power on a group of virtual machines. Plug-ins, such as vsphere Replication, can be installed to extend the capabilities of vcenter to initiate advanced operations on virtual machines and/or their hosts. 4

5 VMware Site Recovery Manager (SRM) ( further extends the capabilities of VirtualCenter to facilitate business continuity and disaster recovery (BCDR) protection for virtual data centers. If a disaster occurs on a protected data center site, SRM can quickly restore affected virtual machines and their IT services on another undamaged part of the datacenter, or if the entire data center is negatively impacted, they can be restored on another recovery site in another geographic location. After the original datacenter has been repaired and brought back online, SRM can help move the virtual machines and their resources back to the original hosts. SRM can be set up on top of the existing IT infrastructure that vcenter manages, as shown in Figure 1. For the replication of storage (e.g. virtual disk images of the virtual machines or VMDK in the VMware environment), SRM can leverage vsphere Replication, which is embedded in vcenter via a plug-in, or support third-party storage replication solutions (e.g. array-based storage replication). Figure 1. VMware Site Recovery Manager (SRM). In order to carry out the recovery operations described in Section 1.1 in the event of a disaster, VMware SRM provides the facilities to set up the following: Connect the protected site to the recovery site ( pairing ) Set up inventory mappings (resource pool, folder, network) Specify a datastore to use for placeholder VMs Configure storage replication (either using vsphere Replication or setting up Storage Replication Adapters) 5

6 Create protection group(s) at the protected site to specify which VMs and resources should be migrated and recovered together Create a recovery plan to specify the actions that need to be performed during the recovery (some of which are mentioned above) Set up reprotect mode and reverse recovery process In addition, SRM also provides the capability of testing a recovery plan without impacting production VMs at the protected site. This would allow administrators to verify that the storage replication, recovery network design, and other aspects of the disaster recovery process work as planned beforehand and without the need to take down the protected VMs. SRM still leverages the traditional vsphere Client for interaction while configuring vsphere replication requires the vsphere Web Client, so for the BCDR scenarios considered in this cookbook, both tools are required. Figure 2 shows the Getting Started page for setting up the disaster recovery operation with the Site Recovery Manager. Figure 2. Getting started with Site Recovery Manager (SRM) 6

7 3 Disaster recovery scenarios In this cookbook, we will consider two different disaster recovery scenarios: The protected site is a private cloud at a customer s premise and the recovery site is in the SoftLayer environment Both the protected site and the recovery site are within the SoftLayer environment Let s first consider the fail-over / recovery scenario from on-premise VMware installation (a customer s VMware-based private cloud) to the SoftLayer environment. 3.1 Fail-over from on-premise VMware installation to SoftLayer Before we use the Site Recovery Manager (SRM) to set up the fail-over / recovery operation from the protected site, which is a customer s on-premise VMware installation, to the SoftLayer environment, as shown in Figure 3, we need to do the following: Create a recovery site at SoftLayer Set up a private network (VPN) connection between the customer s premise and SoftLayer Install the Site Recovery Manager (SRM) at both protected and recovery sites Install SRM plug-in for vcenter at both protected and recovery sites In Figure 3, the protected site is an on-premise VMware installation called VMware@Home and the recovery site is within the SoftLayer cloud environment called VMware@SoftLayer. Figure 3. Fail-over / recovery scenario between on-premise VMware site and SoftLayer. 7

8 3.1.1 Creating recovery site at SoftLayer Our first step is to create a recovery site for our on-premise VMware installation (VMware@Home) at the SoftLayer cloud (VMware@SoftLayer). For more information on how to create a VMware single site in the SoftLayer cloud environment, please refer to the vsphere Basic Site Reference Architecture cookbook Setting up VPN connection This step can accomplished by setting up DNAT and firewall rules for the vcenter instance or a customer-dedicated POP cross connect. However, a VPN tunnel will be used here for added security, lower cost, as well as simplicity. Please refer to Section III (vcloud Connector), Step 4 of the Migrating vsphere Workloads cookbook for more details on how to set up a simple IPSEC VPN connection with a Vyatta gateway appliance. At this point, our sample environment should look like what is depicted in Figure Installing Site Recovery Manager (SRM) and plug-in The following will be needed to set up SRM operations properly: vsphere Replication this is included with vsphere 5.5 licenses (Essentials Plus or higher) and can be installed via one or more vsphere Replication Appliances (VRAs). vsphere Replication can be used to replicate virtual machines from one site to another site. However, it does not provide a sophisticated recovery orchestration on its own. For our disaster recovery scenarios, we use vsphere Replication in conjunction with SRM. The VRAs must be deployed and enabled at both the protected site and the recovery site. Alternatively, they can be deployed via the SRM plug-in. Please refer to Section IV ( vsphere Replication ) of the Migrating vsphere Workloads cookbook for more details on how to install and set up vsphere Replication. Deploy Windows Server and install SRM at both protected site (VMware@Home ) and recovery site (VMware@SoftLayer). Please refer to for more information. License SRM using vsphere Web Client, assign a valid VMware license for SRM at both protected and recovery sites. Install the traditional vsphere Client plug-in this can be done at either or both of the protected and recovery sites because both sites can be managed from the vsphere Client on either site. 8

9 3.1.4 Site pairing Using the traditional vsphere Client, SRM can be invoked by navigating to the Solutions and Applications section in the Home view. Double-clicking on Site Recovery would yield the Getting started view in Figure 2. To perform site pairing, click on Configure Connection and enter the IP address and port number for the (remote) vcenter Server at the recovery site (SoftLayer). In our case, the remote vcenter Server in our SoftLayer environment is at and we use port 80 for this connection as shown in Figure 4. Figure 4. Specifying IP address and port number for the remote vcenter Server After authenticating to the remote vcenter Server (at the recovery site in SoftLayer), SRM will attempt to pair the protected site to the recovery site and vice versa, as shown in Figure 5. Click Finish to move on to the next step. 9

10 Figure 5. Connection between protected and recovery sites have been established Inventory resource mapping After pairing the protected and recovery sites, we need to determine which resources (e.g. resource pools, folders, and network) to fail over to the recovery site in the event of a disaster. The vsphere Web Client can be used to view all resources, as shown in Figure 6. In our example, we want to migrate 3 VMs in the Test VMs folder at the protected site: Joomla-web CPU = 1 (virtual) CPU Memory = 1GB Network = VM Network (via standard switch) Storage = 32GB (on mglab-lun-01 NFS share) Host = pesx-12.mg.lab Joomla-db CPU = 1 (virtual) CPU Memory = 1GB Network = VM Network (via standard switch) Storage = 32GB (on mglab-lun-01 NFS share) Host = pesx-11.mg.lab 10

11 Pinky CPU = 1 (virtual) CPU Memory = 2GB Network = VM network (via standard switch) Storage = 16GB (on mglab-lun-01 NFS share) Host = pesx-12.mg.lab In other words, the three VMs are residing on two different hosts, but share the same network and their storage resources are located on an NFS share. Once we determine which resources we should protect, we need to map them to the recovery site. Figure 6. Using vsphere Web Client to determine resources to protect. Figure 7 shows how we map the resource pool for the three VMs to the recovery site, which, in this case, is the capacity-01 cluster in the SoftLayer environment (Dallas 06). We can just select the resource pool that we want to map and click Configure mapping to select the resource pool at the recovery site for the recovered VMs. We need to do this for every resource pool that we want to fail over to the recovery site. In a similar manner, Figure 8 shows how we map the folder for the three VMs (MGLab Test VMs) to the recovery site, which is the Replicated VMs folder in the capacity-01 cluster at the recovery site (SoftLayer Dallas 06). Because all three VMs are on the VM Network at the protected site, we need to map the VM Network to another network (dvs-priv-vm-access) at the recovery site, as shown in Figure 9. Once recovered, the replicated VMs will run in the capacity-01 cluster on the dvs-priv-vm-access network at the recovery site in the SoftLayer environment. 11

12 Figure 7. Resource mapping. Figure 8. Folder mapping. 12

13 Figure 9. Network mapping Setting up storage replication As mentioned previously, VMware SRM can support storage replication either through vsphere Replication Appliance (VRA) or Storage Replication Adapters (SRAs). If SRAs are used, SRM would require direct access to the Storage Control Layer. However, at this time, SoftLayer storage services, such as iscsi and Quantastor servers, do not support SRM and its required SRAs yet. SoftLayer does allow customers the option to bring in their own storage appliances in which array-based storage replication can be deployed, but this would require SoftLayer Operations & Network Engineering approval (which is handled on an exception basis only). As a result, at this time, this cookbook only shows storage replication via vsphere Replication Appliance. VRAs must be deployed at both protected and recovery sites through the SRM plug-in. After the VRAs are deployed, we are now ready to configure vsphere Replication for each of the three VMs between the protected site (VMware@Home) and the recovery site (VMware@SoftLayer). From the vsphere Web Client at the protected site (VMware@Home), select a VM that we want to replicate, and then navigate to the Actions pull-down menu from the top. Select All vsphere Replication Actions Configure Replication (as shown in Figure 10). 13

14 Figure 10. Configure replication. On the Configure Replication pop-up, select the target site where the VM will be replicated. In our case, it s vc-1-01b in SoftLayer environment. Then click Next to select the replication server. For replication server, we can let VMware auto-assign the appropriate vsphere Replication server or manually select one of the replication servers listed. In our example, we chose the auto-assign option. Click Next to select the target storage device where the replicated files will be placed. In our example, we selected lclpesx-12b. Click Next to select the replication options. One of the replication options is the Guest OS quiescing option, which, if selected, may take several minutes and negatively impact recovery times. In our example, we chose None because our sample virtual machines are not configured to support quiescing methods. Click Next to select the recovery settings. There are two recovery settings: Recovery Point Objective (RPO) and Point-intime instances. Lower RPO times would reduce potential data loss, but would use more bandwidth and system resources. In our example, we chose the lowest RPO time (15 minutes). Point-in-time instances, if enabled, would be converted to snapshots during recovery. We could select the number of instances that will be kept per day and the number of days that those instances will be maintained. In our example, we chose not to keep any point-in-time instances. At this point, our recovery settings should look like those in Figure 11. Click Finish to let VMware configure the vsphere Replication for the VM. 14

15 If we go back to the traditional vsphere Client, and click on vsphere Replication tab at the lower right corner of the Home screen, you should see the replication process being executed for the VM that we just worked on, as shown in Figure 12. It should be noted again that the vsphere Replication process should be set up individually for each of the VMs that we want to fail over to the recovery site in the event of a disaster. Figure 11. Recovery settings. A couple of additional considerations when using vsphere Replication are: Replication is done on a per-running-vm basis, independent of the underlying storage type / vendor A single vcenter can support up to 10 vsphere Replication Appliances (VRAs) A single vcenter / vsphere replication deployment may not protect more than 500 VMs Replication RPOs cannot be set to less than 15 minutes with version 5.5 Replication time is greatly affected by available bandwidth between protected and recovery sites; in fact, vsphere Replication could benefit greatly from WAN optimization, such as the Riverbed Virtual Steelhead appliances, over long distances 15

16 Figure 12. vsphere replication in action Creating protection groups After the full initial storage synchronization through vsphere Replication for all protected VMs are completed, the next step is to create protection group(s). A protection group contains all virtual machines that are protected and recovered together. A protection group can have VMs that use either array-based storage replication or vsphere replication, but not both. An array-based protection group contains all the datastores that must be protected together while a vsphere-replication protection group contains individual VMs that must be protected together. In our example, we create a single protection group for all three VMs with vsphere Replication. To do this, we return to the traditional vsphere Client, and do the following: Click on Site Recovery under the Solutions and Applications tab in the Home view. Click on Protection Groups on the lower left corner, and then Create a Protection Group link on the Getting Started tab. On the resulting pop-up, select the protected site s vcenter IP address (which, in our case, is ) and then select vsphere Replication (VR) option. Click Next to go to the next page. Select all VMs that must be protected together, as shown in Figure 13, and then click Next. Enter a name and a description for the protection group. Click Next to go to the next page and verify all the settings, as shown in Figure 14. Click Finish if all details are correct. SRM will then configure the protection group. 16

17 Figure 13. Creating a protection group. Figure 14. Protection group details Creating a recovery plan After a protection group has been created, the next step is to create a recovery plan. A recovery plan contains all of the steps required to perform a fail-over / recovery should a 17

18 disaster occurs. Multiple protection groups can be specified in a single recovery plan. There can be multiple recovery plans; however, if we make changes to the recovery properties of a virtual machine in one recovery plan, those changes would be reflected in all recovery plans that contain that virtual machine. To create a recovery plan, we click on Recovery Plans at the lower left corner of the Home view in the traditional vsphere Client (just below the Protection Groups tab). Then we do the following: Click on Create Recovery Plan link in the Getting Started tab Select the recovery site in the resulting pop-up, as shown in Figure 15. Click Next. Select the protection group and then click Next. Select the test network to be used for testing the recovery plan. (One of the unique features of SRM is that it provides the capability to test a recovery plan to ensure that the fail-over / recovery operations work properly before disaster strikes without impacting the protected VMs which are running at the protected site.) In our example, we select Auto so that SRM can automatically create a new isolated network environment during each test. Figure 15. Creating a recovery plan. If the recovery plan is configured properly, clicking on the newly created recovery plan on the left pane should display the summary details as shown in Figure 16. We should review the different tabs for the recovery plan to ensure that the configuration is correct. For example, clicking on the Recovery Steps tab would display the current recovery steps (actions) in the recovery plan. We can add or remove a step in the recovery plan by 18

19 highlighting a specific step and then clicking Add Step or Delete Step in the action bar just above the list of the recovery steps. Figure 17 shows the available options for Add Step. Figure 16. Recovery plan summary details. Figure 17. Available options for Add Step. 19

20 For each specific step, we can insert either a command script (to be executed on the SRM server) or a prompt (to be dismissed by the user) before or after that step. If it is a command script that needs to be inserted, a time-out value can also be specified. As mentioned previously, one of the cool features of SRM is that it allows a recovery plan to be tested without impacting the running (protected) VMs. This can be done by simply clicking on Test at the upper right corner of the screen. The progress of the test can be monitored, as shown in Figure 18. Remember to do cleanup (i.e. clicking on Cleanup at the upper right corner) after each test. It should be noted that the recovery plans are stored at the recovery site. Figure 18. Testing a recovery plan Executing a recovery plan In the event of a disaster or a planned migration from the protected site to recovery site, a recovery plan can be executed by doing the following: At the recovery site (VMware@SoftLayer), log into the traditional vsphere Client, and then navigate to Site Recovery under the Solutions and Applications tab in the Home view Click on Recovery Plans tab at the bottom left corner Select the appropriate recovery plan in the left pane Verify that all recovery steps (actions) are correct under the Recovery Steps tab We are now ready to execute the selected recovery plan: 20

21 Click on Recovery button at the upper right corner of the screen A pop-up will be displayed. Select either planned migration or disaster recovery options. These two options are only different in case of storage replication errors. A planned migration would replicate the latest changes to the recovery site and the recovery operation would be cancelled if errors are encountered. A disaster recovery would attempt to replicate recent changes to the recovery site if the protected site is still functional enough for this to happen; otherwise, the recovery operation would still continue if errors are encountered (e.g. the protected site is down). Click Next to go to the next page. After reviewing the recovery plan details, click Start to start the recovery process. The progress of the recovery process is displayed under the Recovery Steps tab, as shown in Figure 19. After the recovery operation is completed, please be sure to check the Plan History tab to view any errors or warnings Reprotect and reverse recovery After the original protected site is operational again, we can click on Reprotect mode in the upper right corner of the screen to configure protection in the reverse direction, i.e. the recovery site will become the new protected site, and the original site will become the new recovery site. Figure 20 shows the details of the Reprotect mode; note that new protected and recovery sites are now the reverse of what we had previously. Figure 19. Execution of a recovery plan. Once the Reprotect process is executed, the storage replication will be initiated again, but this time, the replication is going in the reverse direction: from the new protected site 21

22 (original recovery site) to the new recovery site (original protected site). Of course, this assumes that both sites are fully operational. If we want to fall back the virtual machines to the original site, we need to execute the recovery plan after the Reprotect process is completed successfully. To do this, we need to do the following: Go to the original protected site (new recovery site), bring up the traditional vsphere Client, and navigate to Site Recovery under Solutions and Applications tab. Select the recovery plan by clicking on Recovery Plans in the lower left corner of the screen. Click on Recovery button in the upper right corner of the screen to start the fallback process to the original protected site (new recovery site). Again, the progress of this (reverse) recovery process can be monitored under the Recovery Steps tab. After this recovery process is completed successfully, all protected virtual machines should now be up and running at the original site. Click on Reprotect button in the upper right corner of the screen to set the protected site to be the original site and the recovery site is the other site. After this process is done, everything should be the same as what we had at the beginning, as shown in Figure 22. Figure 20. Configuring the reprotect mode. 22

23 Figure 21. vsphere replication in reverse direction. Figure 22. Back to the original state. 23

24 3.2 Disaster recovery between two SoftLayer sites In this section, we will look at how disaster recovery can be implemented between two SoftLayer sites (data centers). As an example, we will protect our current VMware environment in SoftLayer's Dallas 5 data center by setting up its recovery site in SoftLayer's Dallas 6 data center. Dallas 5 and Dallas 6 are actually separate SoftLayer data centers in Dallas, Texas. However, it would be rather simple to set up disaster recovery between two SoftLayer data centers because there are fast, secure, and unmetered private network links between all SoftLayer data centers. Before we use the Site Recovery Manager (SRM) to set up the fail-over / recovery operation between the protected site (VMware@Dallas 5) and the recovery site (VMware@Dallas 6), we need to do the following: Create a recovery site in Dallas 6 data center Modify the corresponding Vyatta network appliances in both the protected and recovery sites to allow private network traffic flowing between the two sites Ensure that the Site Recovery Manager (SRM) is installed at both protected and recovery sites. If a recovery site is used to protect multiple sites, then multiple instances of SRM must be installed at the recovery site one instance for each protected site. Install SRM plug-in for vcenter at both protected and recovery sites Creating recovery site in Dallas 6 data center Our first step is to create a recovery site in Dallas 6 data center for our protected environment in Dallas 5 data center. For more information on how to create a VMware single site in the SoftLayer cloud environment, please refer to the vsphere Basic Site Reference Architecture cookbook. A single recovery site can be set up for multiple protected sites Setting firewall rules in Vyatta network appliances In order for the SRMs and vcenters at both protected and recovery sites to communicate with each other, we need to set up the firewall rules in the Vyatta network appliances at both sites so that the private network traffic can flow between the two sites. For a list of ports and to understand source and target traffic, please visit the following VMware Knowledge Base: language=en_us&cmd=displaykc&externalid=

25 For example, at the recovery site 6), the Vyatta network appliance should have the firewall rules defined in such a way that it can accept the data traffic from the protected site in Dallas 5 data center. group { <snip> network-group SL_PRIVATE_NETWORKS { description "SoftLayer Source Nets for OOB mgmt Access" <snip> network /26 <snip> } network-group DAL { network /28 network /26 } <snip> } <snip> name SLSERVICES { default-action drop enable-default-log rule 10 { action accept source { group { network-group SL_PRIVATE_NETWORKS } } } rule 21 { action accept source { group { network-group DAL } } } } <snip> where the network group SL_PRIVATE_NETWORKS contains the IP address range for the management cluster (which includes vcenter, Site Recovery Manager, etc.) and the network group DAL contains the IP address range for the capacity cluster at the protected site (VMware@Dallas 5). In addition to the vyatta in Dallas 6, it is important that the vyatta in Dallas 5 allows traffic to flow from the VLANs of the hosts in the capacity cluster and the management VMs to SoftLayer's private network (in order to reach Dallas 6). This configuration 25

26 should have already been done when creating the VMware environment in Dallas 5. For further details, please see the vsphere Basic Site Reference Architecture cookbook Installing the Site Recovery Manager (SRM) and Plug-in The following will be needed to set up SRM operations properly: If a single recovery site is used for multiple protected sites, multiple instances of SRM must be installed at the recovery site one instance for each protected site. Each SRM instance must have its own ID. For more information on how to set up SRM in a shared recovery environment, please see vsphere Replication this is included with vsphere 5.5 licenses (Essentials Plus or higher) and can be installed via one or more vsphere Replication Appliances (VRAs). vsphere Replication can be used to replicate virtual machines from one site to another site. However, it does not provide a sophisticated recovery orchestration on its own. For our disaster recovery scenarios, we use vsphere Replication in conjunction with SRM. The VRAs must be deployed and enabled at both the protected site and the recovery site. Alternatively, they can be deployed via the SRM plug-in. Please refer to Section IV ( vsphere Replication ) of the Migrating vsphere Workloads cookbook for more details on how to install and set up vsphere Replication. Deploy Windows Server and install SRM at both protected site (VMware@Dallas 5) and recovery site (VMware@Dallas 6). Please refer to for more information. License SRM using vsphere Web Client, assign a valid VMware license for SRM at both protected and recovery sites. Install the traditional vsphere Client plug-in this can be done at either or both of the protected and recovery sites because both sites can be managed from the vsphere Client on either site Site pairing Using the traditional vsphere Client, SRM can be invoked by navigating to the Solutions and Applications section in the Home view. Double-clicking on Site Recovery would yield the Getting started view in Figure 23. To perform site pairing, click on Configure Connection and enter the IP address and port number for the (remote) vcenter Server at the recovery site (VMware@Dallas 6). In our case, the remote vcenter Server in Dallas 6 is at and we use port 80 for this connection as shown in Figure 24. The IP address of the (local) vcenter Server 26

27 at the protected site 5) is Note that all of these are private VLAN network addresses. After authenticating to the remote vcenter Server (at the recovery site in SoftLayer), SRM will attempt to pair the protected site to the recovery site and vice versa, as shown in Figure 25. Click Finish to move on to the next step Inventory resource mapping After pairing the protected and recovery sites, we need to determine which resources (e.g. resource pools, folders, and network) to fail over to the recovery site in the event of a disaster. Either the traditional vsphere Client or the vsphere Web Client in Home VMs and Templates view can be used to display all the virtual machines in the inventory at the protected site, as shown in Figure 26. From there, we can determine which VMs need to be protected. In our example, we want to migrate two VMs (appserver and appserver-db) in the Appserver VMs folder at the protected site in Dallas 5: appserver CPU = 1 (virtual) CPU Memory = 1GB Network = 1196 (standard port group) Storage = 17GB (on shared iscsi storage) Host = appserver-db CPU = 1 (virtual) CPU Memory = 1GB Network = 1196 (standard port group) Storage = 32GB (on shared iscsi storage) Host = In other words, the two VMs reside on the same host, share the same network, and their storage resources are located on iscsi storage. Once we determine which resources we should protect, we need to map them to the recovery site. 27

28 Figure 23. Getting started with SRM. 28

29 Figure 24. Site pairing. Figure 27 shows how we map the resource pool for the two VMs to the recovery site, which, in this case, is the SL-DAL-06-capacity-01 cluster in the SoftLayer environment (Dallas 06). We can just select the resource pool that we want to map and click Configure mapping to select the resource pool at the recovery site for the recovered VMs. We need to do this for every resource pool that we want to fail over to the recovery site. In a similar manner, Figure 28 shows how we map the folder for the two VMs to the recovery site, which is the Recovered VMs folder in the SL-DAL-06-capacity-01 cluster at the recovery site (SoftLayer Dallas 6 data center). 29

30 Figure 25. Site pairing completed. 30

31 Figure 26. Using traditional vsphere Client to view inventory resources. Because both VMs are on the VLAN 1196 network at the protected site, we need to map the VLAN 1196 to another network (dvs-priv-vm-access) at the recovery site, as shown in Figure 29. Once recovered, the replicated VMs will run in the SL-DAL-06-capacity01 cluster on the dvs-priv-vm-access network in the Recovered VMs folder at the recovery site in the SoftLayer Dallas 6 data center. 31

32 Figure 27. Resource mapping Setting up storage replication As mentioned previously, VMware SRM can support storage replication either through vsphere Replication Appliance (VRA) or Storage Replication Adapters (SRAs). If SRAs are used, SRM would require direct access to the Storage Control Layer. However, at this time, SoftLayer storage services, such as iscsi and Quantastor servers, do not support SRM and its required SRAs yet. SoftLayer does allow customers the option to bring in their own storage appliances in which array-based storage replication can be deployed, but this would require SoftLayer Operations & Network Engineering approval (which is handled on an exception basis only). As a result, at this time, this cookbook only shows storage replication via vsphere Replication Appliance. VRAs must be deployed at both protected and recovery sites through the SRM plug-in. After the VRAs are deployed, we are now ready to configure 32

33 vsphere Replication for each of the three VMs between the protected site 5) and the recovery site 6). From the vsphere Web Client at the protected site 5), select a VM that we want to replicate, and then navigate to the Actions pull-down menu from the top. Select All vsphere Replication Actions Configure Replication (as shown in Figure 30). On the Configure Replication pop-up, select the target site where the VM will be replicated. In our case, it s vc-1-01b in SoftLayer Dallas 6 site. Then click Next to select the replication server. For replication server, we can let VMware auto-assign the appropriate vsphere Replication server or manually select one of the replication servers listed. In our example, we chose the auto-assign option. Click Next to select the target storage device where the replicated files will be placed. In our example, we selected iscsi-capacity01-eql-lun0. Click Next to select the replication options. Figure 28. Folder mapping. 33

34 One of the replication options is the Guest OS quiescing option, which, if selected, may take several minutes and negatively impact recovery times. In our example, we chose None because our sample virtual machines are not configured to support quiescing methods. Click Next to select the recovery settings. Figure 29. Network mapping. There are two recovery settings: Recovery Point Objective (RPO) and Point-intime instances. Lower RPO times would reduce potential data loss, but would use more bandwidth and system resources. In our example, we chose the lowest RPO time (15 minutes). Point-in-time instances, if enabled, would be converted to snapshots during recovery. We could select the number of instances that will be kept per day and the number of days that those instances will be maintained. In our example, we chose not to keep any point-in-time instances. 34

35 Figure 30. Configure replication. At this point, our recovery settings should look like those in Figure 31. Click Finish to let VMware configure the vsphere Replication for the VM. If we go back to the traditional vsphere Client, and click on vsphere Replication tab at the lower right corner of the Home screen, you should see the replication process being executed for the VM that we just worked on, as shown in Figure 32. It should be noted again that the vsphere Replication process should be set up individually for each of the VMs that we want to fail over to the recovery site in the event of a disaster. 35

36 Figure 31. Recovery settings. A couple of additional considerations when using vsphere Replication are: Replication is done on a per-running-vm basis, independent of the underlying storage type / vendor A single vcenter can support up to 10 vsphere Replication Appliances (VRAs) A single vcenter / vsphere replication deployment may not protect more than 500 VMs Replication RPOs cannot be set to less than 15 minutes with vsphere version 5.5 Replication time is greatly affected by available bandwidth between protected and recovery sites; in fact, vsphere Replication could benefit greatly from WAN optimization, such as the Riverbed Virtual Steelhead appliances, over long distances 36

37 Figure 32. vsphere replication in action for appserver VM Creating protection groups After the full initial storage synchronization through vsphere Replication for all protected VMs are completed, the next step is to create protection group(s). A protection group contains all virtual machines that are protected and recovered together. A protection group can have VMs that use either array-based storage replication or vsphere replication, but not both. An array-based protection group contains all the datastores that must be protected together while a vsphere-replication protection group contains individual VMs that must be protected together. In our example, we create a single protection group for two VMs (appserver and appserver-db) with vsphere Replication. To do this, we return to the traditional vsphere Client, and do the following: Click on Site Recovery under the Solutions and Applications tab in the Home view. Click on Protection Groups on the lower left corner, and then Create a Protection Group link on the Getting Started tab. On the resulting pop-up, select the protected site s vcenter IP address (which, in our case, is ) and then select vsphere Replication (VR) option. Click Next to go to the next page. Select all VMs that must be protected together, as shown in Figure 33, and then click Next. Enter a name and a description for the protection group. Click Next to go to the next page and verify all the settings, as shown in Figure 34. Click Finish if all details are correct. SRM will then configure the protection group. 37

38 Figure 33. Creating a protection group. 38

39 Figure 34. Protection group details Creating a recovery plan After a protection group has been created, the next step is to create a recovery plan. A recovery plan contains all of the steps required to perform a fail-over / recovery should a disaster occurs. Multiple protection groups can be specified in a single recovery plan. There can be multiple recovery plans; however, if we make changes to the recovery properties of a virtual machine in one recovery plan, those changes would be reflected in all recovery plans that contain that virtual machine. To create a recovery plan, we click on Recovery Plans at the lower left corner of the Home view in the traditional vsphere Client (just below the Protection Groups tab). Then we do the following: Click on Create Recovery Plan link in the Getting Started tab Select the recovery site in the resulting pop-up, as shown in Figure 35. Note that the recovery site is the IP address of the vcenter server running in SoftLayer's Dallas 6 data center ( ). Click Next. Select the protection group and then click Next. 39

40 Select the test network to be used for testing the recovery plan. (One of the unique features of SRM is that it provides the capability to test a recovery plan to ensure that the fail-over / recovery operations work properly before disaster strikes without impacting the protected VMs which are running at the protected site.) In our example, we select Auto so that SRM can automatically create a new isolated network environment during each test. Figure 35. Creating a recovery plan. If the recovery plan is configured properly, clicking on the newly created recovery plan on the left pane should display the summary details. We should review the different tabs for the recovery plan to ensure that the configuration is correct. For example, clicking on the Recovery Steps tab would display the current recovery steps (actions) in the recovery plan, as shown in Figure 36. We can add or remove a step in the recovery plan by highlighting a specific step and then clicking Add Step or Delete Step in the action bar just above the list of the recovery steps. Figure 37 shows the available options for Add Step. 40

41 Figure 36. Recovery steps in the recovery plan. Figure 37. Available options for Add Step. 41

42 For each specific step, we can insert either a command script (to be executed on the SRM server) or a prompt (to be dismissed by the user) before or after that step. If it is a command script that needs to be inserted, a time-out value can also be specified. As mentioned previously, one of the cool features of SRM is that it allows a recovery plan to be tested without impacting the running (protected) VMs. This can be done by simply clicking on Test at the upper right corner of the screen. The progress of the test can be monitored in the Recovery Steps tab. Remember to do cleanup (i.e. clicking on Cleanup at the upper right corner) after each test. It should be noted that the recovery plans are stored at the recovery site Executing a recovery plan In the event of a disaster or a planned migration from the protected site to recovery site, a recovery plan can be executed by doing the following: At the recovery site (VMware@SoftLayer), log into the traditional vsphere Client, and then navigate to Site Recovery under the Solutions and Applications tab in the Home view Click on Recovery Plans tab at the bottom left corner Select the appropriate recovery plan in the left pane Verify that all recovery steps (actions) are correct under the Recovery Steps tab We are now ready to execute the selected recovery plan: Click on Recovery button at the upper right corner of the screen A pop-up will be displayed, as shown in Figure 39. Select either planned migration or disaster recovery options. These two options are only different in case of storage replication errors. A planned migration would replicate the latest changes to the recovery site and the recovery operation would be cancelled if errors are encountered. A disaster recovery would attempt to replicate recent changes to the recovery site if the protected site is still functional enough for this to happen; otherwise, the recovery operation would still continue if errors are encountered (e.g. the protected site is down). Click Next to go to the next page. After reviewing the recovery plan details, click Start to start the recovery process. The progress of the recovery process is displayed under the Recovery Steps tab, as shown in Figure 40. After the recovery operation is completed, please be sure to check the Plan History tab to view any errors or warnings Re-protect and reverse recovery After the original protected site is operational again, we can click on Reprotect mode in the upper right corner of the screen to configure protection in the reverse direction, i.e. 42

43 the recovery site will become the new protected site, and the original site will become the new recovery site. Figure 41 shows the details of the Reprotect mode; note that new protected and recovery sites are now the reverse of what we had previously. Figure 39. Starting the disaster recovery process. Once the Reprotect process is executed, the storage replication will be initiated again, but this time, the replication is going in the reverse direction: from the new protected site (original recovery site) to the new recovery site (original protected site). We can view the progress of this reverse storage replication by clicking on vsphere Replication in the lower left corner of the Home view provided by the traditional vsphere Client, as shown in Figure 42. Of course, this assumes that both sites are fully operational. If we want to fall back the virtual machines to the original site, we need to execute the recovery plan after the Reprotect process is completed successfully. To do this, we need to do the following: Go to the original protected site (new recovery site), bring up the traditional vsphere Client, and navigate to Site Recovery under Solutions and Applications tab. Select the recovery plan by clicking on Recovery Plans in the lower left corner of the screen. 43

44 Click on Recovery button in the upper right corner of the screen to start the fallback process to the original protected site (new recovery site). Again, the progress of this (reverse) recovery process can be monitored under the Recovery Steps tab. After this recovery process is completed successfully, all protected virtual machines should now be up and running at the original site. Click on Reprotect button in the upper right corner of the screen to set the protected site to be the original site and the recovery site is the other site. After this process is done, everything should be the same as what we had at the beginning, as shown in Figure 43. We now see that the protected site's vcenter server is at in Dallas 5 data center while the recovery site's vcenter server is at in Dallas 6 just like what we had at the beginning. Figure 40. Execution of recovery plan in progress. 44

45 Figure 41. Starting re-protect process. Figure 42. Reverse storage replication. 45

46 Figure 43. Back to original state. 4 Conclusion In this cookbook, we have demonstrated how VMware Site Recovery Manager (SRM) can be used to implement a Disaster Recovery (DR) solution for VMware-based hybrid cloud environments with SoftLayer: Protecting an on-premise, VMware-based data center with a disaster recovery site in a SoftLayer data center Protecting a VMware-based environment in a SoftLayer data center with a disaster recovery site in another SoftLayer data center The private network bandwidth within SoftLayer data centers are un-metered, so large amounts of data transfers over the SoftLayer's private networks during DR operations would not incur additional data charges. If additional information is needed on this topic, please contact Khoa Huynh (khoa@us.ibm.com). 46