BACKUP AND RECOVERY SOLUTION FOR VMWARE VSPHERE ON EMC ISILON STORAGE

Transcription

1 White Paper BACKUP AND RECOVERY SOLUTION FOR VMWARE VSPHERE ON EMC ISILON STORAGE Using Symantec NetBackup 7.5, VMware vsphere VADP and EMC Isilon SyncIQ Data Protection Software Abstract This white paper describes how a multi-tiered, multi-site backup solution can be deployed to protect data in a VMware vsphere environment using vsphere data protection APIs along with Symantec NetBackup 7.5 and EMC Isilon SyncIQ. This paper also outlines the setup, configuration, and functional testing of a disk-to-disk backup environment. July 2012

2 Copyright 2012 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice. The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. VMware, ESX, vmotion, VMware vcenter, and VMware vsphere are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other trademarks used herein are the property of their respective owners. Part Number H

3 Table of Contents Executive summary... 5 Business case... 5 Solution overview... 6 Key results and recommendations... 6 Introduction... 7 Purpose... 7 Scope... 8 Audience... 8 Terminology... 8 Assumptions Data protection strategies and use cases Data protection Use cases Technology overview Overview VMware VADP Symantec NetBackup Isilon SyncIQ vsphere Management Assistant Body of work Configuration Process overview Solution architecture Physical environment Validation Scenario #1: image-level backups Scenario #2: recovery of a VM from local backup Scenario #3: recovery of individual files from a VM backup image Scenario #4: recovery of a VM from remote backup Component hardware and software Storage configuration VMware ESXi host hardware configuration Environment resources and configurations Conclusion Summary Findings Next steps

4 References White papers Product documentation About EMC Isilon Contact EMC Isilon

5 Executive summary As enterprise server technology becomes increasingly powerful with each new generation of hardware, virtualization technologies and capabilities grow at the same pace. The end result for most organizations is an ever-increasing virtualized infrastructure. As more and larger workloads are migrated to a virtualized environment, the data sets associated with those workloads grow in both size and criticality. At the same time, organizations with 24x7 service level expectations and obligations are forced to consider strategies to keep services available at all times while ensuring that their data is protected. While a single backup solution is sufficient for many organizations, the ever-increasing complexity and criticality of an organization s data, coupled with the need to recover data against multiple failure scenarios, often requires organizations to consider a multiple layer approach to data backup and recovery. In the case of virtualized environments hosted on EMC Isilon storage, EMC offers both native features and interoperability with enterprise-class data protection solutions to assist customers in meeting their backup and recovery objectives. This white paper steps through several of those solutions to assist organizations in creating a comprehensive data protection strategy. By using EMC Isilon SyncIQ and SnapshotIQ data protection software, as well as by deploying virtualization-aware backup and recovery solutions, customers can maintain the continuous availability of virtualized workloads and services, while ensuring their ability to recover critical enterprise functionality and services in the event of data loss or corruption. Business case For most organizations, server virtualization has yielded both significant cost savings and service improvements. The downside in many cases is a proliferation of virtual servers and virtualized data. Often, organizations with a high percentage of their workloads virtualized find their traditional approach to backup and recovery is no longer adequate. Consider the following: Using traditional backup agents installed in each VM results in a significant amount of overhead on the virtualized infrastructure, as each VM runs nearly identical processes to back up nearly identical operating system data. As virtualized infrastructures continue to scale up, a traditional tape-backup system may not be able to scale at the same pace. Often, organizations find themselves constrained by traditional tape solutions that make full backups impossible to complete within the designated backup window. At the same time, as service demands and business operational requirements increase, those same organizations are seeing their backup windows continue to shrink. 5

6 Application consistency can be an issue if an organization s backup solution is not VM-aware. Using multiple backup solutions independently e.g. separate backups for operational recovery, application consistency and disaster recoverability often exceed the allotted backup window, and consume excessive backup media. In the event of a large-scale loss of data, such as multiple concurrent hardware failures leading to the loss of multiple LUNs a traditional backup approach would require a lengthy restore process to recover all VMs affected by the outage. Solution overview The solution described in this white paper features a multi-tiered approach to safeguard the integrity and the recoverability of critical organizational and system data. It takes advantage of data protection features from VMware, Symantec and EMC to ensure that, in the event of a system or site failure, organizations who adopt a backup and recovery strategy similar to the one enumerated in this paper can quickly and reliably recover impacted data and services. This solution is based on NetBackup 7.5 with deduplication from Symantec, combined with VMware vsphere APIs for Data Protection (VADP) feature and EMC Isilon SyncIQ. It provides a multi-layered strategy to protect vsphere data, including onsite, disk-based data, replication to an offsite disk cluster, and multi-tiered recovery strategies. Validation of the solution was achieved at multiple steps along the way as well. The steps of verifying the integrity of the overall solution included: Monitoring the success of the VADP-based backup policies as configured and scheduled in NetBackup 7.5 Checking the state of data deduplication during the backup process and monitoring the size of backup jobs relative to the original data footprint. Ensuring successful replication as scheduled between the primary Isilon storage cluster and the offsite disk cluster using Isilon SyncIQ. Monitoring replication size, duration and success of the SyncIQ daily synchronization. Verifying the successful recovery process of a VM using the locally-stored backup data. Performing a successful recovery of the same VM using the remote backup dataset as the basis for restoring the data. Key results and recommendations EMC recommends leveraging the latest developments in virtualization-aware backup technologies, and applying them to create a comprehensive data protection strategy that is designed to protect data at multiple levels and multiple locations. The intent is 6

7 to safeguard corporate and organizational data at multiple points, including file-level, VM-level, and site-level protection. Additionally, while the solution itself is a straightforward design, the criticality of the data being protected requires careful planning, implementation and testing prior to go-live. Confirmation of the solution s ability to deliver a complete backup solution, one that fits within operational and organizational constraints, is necessary to validate a stable backup environment that provides sufficient protection for the organization s data, and the services and processes that depend on it. Finally, in order to ensure the continued reliability and performance of the solution, regular monitoring of the jobs and processes that comprise the solution is essential, as is the practice of performing regular data restoration exercises. Doing so will validate both the data and the procedures necessary to successfully recover from data loss or corruption. Introduction Purpose The purpose of this white paper is to describe how a multi-tiered, multi-site backup solution can be deployed to protect data in a VMware vsphere environment using vsphere data protection APIs along with Symantec NetBackup 7.5 and EMC Isilon SyncIQ. This Guide outlines the setup, configuration, and functional testing of a diskto-disk backup environment for this solution. The guide features the latest generation of virtualization-aware backup technology, coupled with native snapshot and replication functionality from EMC Isilon. While the solution modeled in this guide specifically leverages Symantec NetBackup, this is only one example of an enterprise VM data protection platform. Although this is a specific approach to vsphere backups, EMC recognizes that the specific requirements that drove the solution in this guide are not universal. Many organizations have disparate industry-level and/or regulatory requirements for data protection, so every organization will have unique requirements and challenges. The data protection solution described in this paper, while designed to meet a specific set of protection criteria, may not meet every need for another organization. Rather, this guide should be used to illustrate a multi-tiered approach to data protection, which simplifies operations while minimizing the computing overhead frequently associated with traditional enterprise backup solutions. The solution presented here recognizes that enterprise backup strategies need to make use of the advancing technology that virtualization provides, while still needing to rely on more traditional methods like multiple layers of protection across multiple locations. A VADP-aware backup solution simplifies the backup platform by eliminating the need to install backup agents on every VM, and the change-block tracking feature means that only changed data needs to be backed up. At the same time, deduplication reduces the amount of data that SyncIQ needs to replicate via 7

8 WAN link to the offsite cluster. Finally, the use of disk-based backup as the primary offsite storage medium increases the speed at which data is transferred offsite. Taken together, the approach described in this paper results in simplified management, increased data protection, improved efficiency, and reduced backup windows for most organizations that adopt this strategy. Scope The scope of this white paper is to describe a backup solution for capturing, deduplicating and archiving enterprise VM data. It does not cover specific configuration steps of the entire solution, nor does it address the need for monitoring or verifying successful backup status. Every organization will have unique tools and requirements for those components Audience This white paper is intended for backup administrators, systems engineers and consultants to couple the next generation of virtualization-aware backup solutions with Isilon storage for their VMware environments. It describes how Symantec backup and deduplication tools combine with Isilon snapshot and replication software to simplify the backup and recovery of virtual machines and accompanying data. It assumes the reader is familiar and proficient with the following: Virtualization using VMware vsphere 5.0 EMC Isilon storage design and management EMC SnapshotIQ and SyncIQ data protection software Command-line syntax for administration of VMware vsphere and the EMC Isilon OneFS operating system. The automation of some tasks in this guide requires the use of the VMware vsphere Management Assistant and SSH connectivity to the Isilon storage cluster. Backup and recovery software and concepts Microsoft Windows Server 2003 or later, including hardware-level management, NTFS permissions, and Active Directory authentication Terminology This paper includes the following terminology. Table 1: Terminology Term Backup policy Definition The procedures and rules used to ensure adequate data backups can be made. As configured within most backup software, backup policies typically include parameters such as acceptable backup windows, the systems and data to be backed up, the type of backup to be performed, and the destination media. 8

9 Term Backup window Change Block Tracking Cumulative Incremental backup Data deduplication Differential Incremental backup Full backup Recovery Point Objective Recovery Time Objective Definition The time period required to complete a backup procedure. Backup procedures typically require the same computing and network resources as production workloads, so if a backup procedure exceeds its allotted backup window it can have detrimental effects on production system performance. Additionally, backups that fail to complete in the time available can put production data at risk if a system failure occurs before data can be successfully backed up. A VMware feature that helps perform incremental backups by tracking which disk sectors within a VMDK file have changed since the last backup. This feature can be leveraged by 3rd party backup and recovery software. A backup of all changes made since the last full backup. Compared to differential incremental backups, cumulative incremental backups require a longer backup window to complete successfully, but shorten the time necessary to restore data after a failure or loss. A data-reduction technique in which redundant data within a given dataset is reduced to a single instance, and subsequent occurrences of the same data are replaced with a link back to the first instance. This technique is used to improve storage utilization, or reduce the amount of data to be transferred via network link. A backup that contains only the data that has changed since the more recent backup. Compared to cumulative incremental backups, differential incremental backups require a shorter backup window, but lengthen the recovery time after a loss of data. A complete backup of all selected data on a given system. Abbreviated as RPO, the maximum amount of data loss, expressed in units of time, that an organization can afford to sustain. For example, a 12-hour RPO for a given organization means that the organization can afford up to lose up to 12 hours of data in the event of a major incident. In designing a data protection strategy, the RPO determines the frequency of the backup cycle that an organization must plan for, the frequency and size of the offsite synchronization process and the computing and network infrastructure necessary to support the designated RPO. The RTO is the duration of time allowed for system and data recovery in order to restore business functions after a service disruption. 9

10 Term Snapshot Synchronization Definition A point-in-time capture of the state of a file or file system. Snapshots streamline access to stored data and can speed up the process of data recovery. The process of making sure that two or more locations contain the same up-to-date files. Assumptions The following conditions were applied to the test setting. Results may vary if these same conditions are not in effect in the customer s environment. This guide assumes the use of Windows Server 2003 or later guest operating systems. Results may not be consistent with Linux or earlier versions of Windows. An Active Directory domain authentication source is necessary for the NetBackup Master and Media servers to communicate with one another. Symantec NetBackup 7.5, VMware vsphere 5 and EMC Isilon OneFS 6.5.x with Isilon SyncIQ 3.0 were used in this environment. Other versions of these platforms may yield different results. Specific components leveraged in this solution may require additional licensing to enable and use the full feature set of capabilities necessary for the multi-tiered data protection strategy outlined in this document. Additionally, the implementation and test procedures outlined in this document do not give exact instructions for each step. It is assumed that the audience is familiar enough with the tools and interfaces of the different products in this solution to perform them without explicit guidance. Data protection strategies and use cases Data protection As the name implies, data protection as a term refers to the safeguarding of important data against corruption, alteration, or loss. A well-planned and wellexecuted data protection strategy can protect critical organizational data against any or all of these scenarios. It can also serve as a defense against a disaster event, in which critical infrastructure and data is permanently inaccessible or destroyed. For many organizations, in a number of industries, an effective data protection strategy is required by law, and organizations must ensure that their overall approach is compliant with appropriate legal and regulatory requirements. The elements necessary for sufficiently thorough data protection are driven in part by the size and nature of the organization. It requires a multifaceted approach that includes a combination of technology, business processes and overall best practices. The specific mix of practices, processes and technologies necessary for an 10

11 appropriate strategy is driven in part by the size of the organization, and in part by its nature. For any organization, the objective of a data protection strategy is to minimize the risk of adverse business impact or loss resulting from inaccessible or unverifiable business data. The practices to consider in planning and implementing a data protection strategy include: Backup and recovery Backup integrity verification Offsite data replication Storage system security Data Lifecycle Management Backup and recovery refers to the protection of data by making one or more offline copies of the data to be restored in the event of disaster or data loss. Backup integrity verification requires the regular recovery of backup datasets to an alternate location to ensure the validity and accessibility of system and data backups. Offsite data replication is the real-time or near-real-time moving of data to a remote site, external to the primary storage system, to enable recoverability in the event of physical damage to systems and facilities. The two most common forms of this technique are remote copy and replication. These techniques duplicate data from one system to another, in a different location. Storage system security refers to taking appropriate measures to ensure the protection of the storage system itself against improper access. Minimally appropriate measures include the following: physical security of the facility in which the storage is housed, network access control to safeguard against improper data access, alteration, or theft, and antivirus software to protect against data corruption. Data Lifecycle Management is the automated movement of critical data to online and offline storage. Potential aspects of include moving data considered to be in a final state onto read-only storage, where it cannot be changed, and moving data to different tiers of storage based on its age. While this document focuses primarily on the technologies necessary to support a comprehensive data protection solution, it is not intended to serve as a universal guide for all organizations or all requirements. Use cases The solution described in this guide is designed to address the following data protection scenarios and objectives: Disaster recoverability Business continuity Disk-to-disk backup 11

12 Disk-to-tape backup Offsite data protection Disaster recoverability requires the implementation of processes and mechanisms to enable the recovery and continuation of technology infrastructure and critical business data in the event of a large-scale disaster. At a minimum, disaster recoverability requires a reliable, accessible offsite data repository located at a sufficient distance from the primary site to be unimpacted by the initial disaster event. Business continuity refers to the necessary steps to ensure that critical business functions will remain available in the event of a disaster. Disk-to-disk (D2D) backup is the backing up of data from one disk to another, rather than to removable media such as tape. Relative to tape-based backup, D2D offers higher backup speeds and capacities, which can reduce backup times. Because D2D provides nonlinear access to data, it can significantly decrease recovery times as well. Disk-to-tape (D2T) backup typically refers to the transfer of data from disk to removable media, which can then be disconnected and stored offline. Offsite data protection refers to the sending of critical data from the primary site as part of a disaster recovery plan. Data is either transported via tape or optical media, or replicated to an offsite storage array through a wide-area-network link. This ensures the recoverability of critical data in the event of a disaster or system failure. Technology overview Overview The solution described in this white paper leverages the following technologies as part of a comprehensive data protection strategy for a vsphere infrastructure. VMware vstorage APIs for Data Protection Symantec NetBackup 7.5 EMC Isilon SyncIQ VMware VADP Beginning with vsphere 4.0, VMware introduced the vstorage APIs for Data Protection (VADP) as a means for offloading backup processing from the host. Not only does VADP allow for easier access to a VM's virtual disk file, but it also contains features that can improve backup speeds, such as Changed Block Tracking (CBT). CBT keeps track of any blocks that have changed since the last backup. As a result, a backup application can simply query the VMkernel to identify the specific blocks needing to be backed up. This means the backup application no longer needs to track this, resulting in much quicker incremental backups. VADP provides a much more efficient mechanism for backing up when used with backup applications that take full advantage of them. 12

13 Symantec NetBackup 7.5 NetBackup for VMware provides backup and restore of the VMware virtual machines that run on vsphere servers. NetBackup leverages VADP to offload backup processes from the ESXi hosts to a designated vsphere backup server. It includes the following capabilities: Off-host backup of virtual machines. This reduces the backup processing load on the ESXi host(s). Increased backup speeds relative to standard file-order backup methods. Automatic creation of quiesced snapshots of Windows virtual machines using Windows Volume Shadow Copy Services (VSS). It can also create quiesced snapshots of Linux virtual machines if an appropriate utility (e.g. SYMCquiesce) is installed. Use of snapshot technology to keep virtual machines 100% available to users. Full and incremental backups, including block-level incremental backups, of virtual machines regardless of the VM s power state. VM- and file-level restore capabilities. NetBackup offers deduplication at a number of levels. Isilon SyncIQ 3.0 EMC Isilon SyncIQ replicates data in an Isilon storage environment from a primary site to a secondary, local or remote site, creating an offline copy of the data for disaster recovery, business continuance, disk-to-disk backup, or remote archiving purposes. SyncIQ delivers high-performance, asynchronous replication of data over both short (LAN) or long (WAN) distances, providing protection from site-specific and regional disasters. It includes a robust, policy-driven engine that allows for the customization of replication datasets to minimize system impact while still satisfying data protection requirements. vsphere Management Assistant The vsphere Management Assistant (vma) from VMware is a Linux-based virtual machine that includes prepackaged software such as the vsphere command-line interface, and the vsphere SDK for Perl. vma allows administrators to run scripts or agents that interact with ESXi hosts and the vcenter server through a single portal. Using the vma, administrators can manage vcenter services and hosts, as well as connect via SSH to other command-line administrative consoles, including the Isilon storage cluster, either for automating management tasks via script or for singleconsole access to multiple systems. 13

14 Body of work Configuration This section describes the solution as configured. Specific hardware and software configuration details are provided in Component Hardware and Software section of this guide. Process overview Backup of the vsphere infrastructure in this solution uses Symantec NetBackup Symantec and EMC Isilon SyncIQ according to the following processes: 1. The Master Server coordinates with the vsphere infrastructure, using VADP, to create snapshots of the VMs in the cluster for backup to one or more Media Servers. Specific backup settings are defined in one or more backup policies on the Master Server. Examples of settings defined in NetBackup policies may include: a. Backup schedule b. Scope of backup, e.g. all drives, specific volumes, exclusion of pagefiles, etc. c. Type(s) of backup, such as Full, Incremental, or Differential. d. Staging of backup data to the Media Server(s), including location, deduplication settings, etc. 2. Data from the snapshot(s) created in Step 1 is then streamed to the Media Server. 3. The Media Server stages the backup data to one or more storage repositories as defined in backup policies on the Master Server. In this solution, backup data is also deduplicated at the time of staging in order to reduce the storage footprint of the backup data. 4. Using SyncIQ, the deduplicated data is replicated to an offsite Isilon storage cluster for archival and potential disaster recovery purposes. 5. Additional offline archival takes place as data is backed up to tape. This can also be configured from the primary cluster. Solution architecture Figure 1 below illustrates the physical and process architecture of the backup solution described in this guide. 14

15 Figure 1: vsphere Infrastructure Solution Architecture with NetBackup Data Protection Physical environment This section describes the physical components of the environment Hardware configuration: At the primary site, the backup solution utilizes five vsphere ESXi hosts, running on a 9-node Isilon cluster composed of three Isilon S-Series nodes, three Isilon X- Series nodes, and three Isilon NL-Series nodes. At the secondary site, three vsphere ESXi hosts connect to five NL-Series nodes which replicate with the primary Isilon cluster using SyncIQ. The NetBackup 7.5 infrastructure in this solution consists of two Windows Server 2008 R2 VMs at the primary site, one configured as the Master Server, and the other set up as the Media Server. Software configuration The Master Server manages backup policy creation along with the creation, configuration, scheduling, and execution of backup jobs. It communicates with the vcenter server for VADP tasks, and with the Media Server on the actual streaming of VM data for backup. 15

16 The Media Server manages connections to the individual ESXi hosts in the vsphere infrastructure to receive VM data for backup and deduplication. It also provides connections to the storage pools that are used for staging backup data. Validation The following operational test cases were used as part of the validation process to verify the reliability and performance of the solution design as described in this document. These tests were intended to verify and document the ability of the solution to protect organizational data against multiple failure scenarios: Image-level backups Recovery of an individual VM from local backup Recovery of individual files from within a backed-up virtual machine Recovery of individual server from offsite copy This section describes the test procedures as executed against the design for all of the failure and recovery scenarios described above. Scenario #1: image-level backups Using VADP to connect to vsphere, NetBackup 7.5 creates a VMware-level snapshot of the virtual machine, and then presents it to the Media Server which backs up the VMDK files that make up the VM. Properly configured, this enables the option of either performing image-level recovery or file-level recovery from a single backup process. Test objectives The purpose of this test is to ensure the reliable backup of all VMs configured in the test backup policy according to the policy settings. These include a full weekly backup and a daily incremental backup per VM within the designated backup window. It also validates the deduplication of the VM data at the time of backup to the Media Server in order to minimize disk space used and the necessary bandwidth to replicate the backup data to the remote site. In this scenario, a policy was created in which the VMs on the test vsphere cluster were configured for daily backups. The policy was configured to perform full backups once per week, with incremental backups for each VM occurring on the other days. To minimize disk space, the NetBackup Media Server was configured to deduplicate backup data as it streamed from the individual VMs. This reduced both the amount of disk space necessary at the primary site to store the local backup data, as well as the amount of data that needed to replicate via WAN link to the secondary site. Test procedures 1. Create a 1TB VMDK file and attach it to the Media Server. From within the Media Server s guest operating system, mount the volume as E:\. Using the NetBackup Administration Console on the Master Server, create a storage unit on the Media 16

17 Server, assign it a name, set the storage unit type to Disk, and set the disk type to PureDisk. Configure disk deduplication from the NetBackup Administrative Console. 2. From the NetBackup Master Server, create a new VMware Backup Policy. Within the policy, configure the Policy Name, VMware backup host, backup options (e.g. block-level backups, exclusions, transport modes, and application protection settings as applicable). Select the virtual machine(s) to be backed up as part of the policy. Enable full backups and differential incremental backups as part of the policy, then set the full-backup frequency to weekly, and the incremental-backup frequency to daily, and the retention policy for two weeks. Set the backup window to run during off hours. Save the policy. 3. Allow the policy to run through two full weekly backup cycles to ensure full and differential incremental backups completed successfully. 4. Check Activity status on Master Server, check log reports to verify backup success. Test results Backup policies successfully executed against all in-scope VMs over a period of two weeks. Activity logs on Master Server showed backups occurring on a daily basis for all VMs configured in the test policy, with Full and Differential-Incremental backups happening as scheduled. Disk deduplication rates per server varied. Scenario #2: recovery of a VM from local backup This scenario is a validation of the ability of the solution to recover an entire failed VM. Situations in which this might become necessary include inadvertent VM deletion within vcenter, file-system corruption, or development environment reset. Test objectives Confirm the recoverability of individual VMs from the backup images. Ensure that VMs can be restored from any backup point over the retention cycle and that all VM data and settings are preserved in the restored copy. In this test, the target VM was deleted from within vcenter, and the NetBackup Backup, Archive, and Restore console was used to recover the VM back to its original configuration. Test procedures 1. Identify a target VM for the restore test in vcenter. Confirm successful backups of the test VM through the NetBackup Backup, Archive and Restore application. Confirm also that the backup history lists restore points for every day over the twoweek lifecycle of the retention policy. 2. Power down the target VM within vcenter. Delete the VM from disk. 3. Run the Restore process, choose a recovery point from the available backups, and recover the VM to its original location and retain the original settings. Configure 17

18 the restore job to power the VM up upon completion. Run the Pre-Recovery Check, and initiate the recovery process. Monitor recovery status through the Administration Console. 4. When restore process completes, log into recovered VM and verify network, disk and configuration settings. Test results Restore process completed successfully. Upon boot, VM displayed the Windows Error Recovery Screen indicating its last shutdown had not been successful. After logging into the guest operating system console, the Shutdown Event Tracker appeared. The VM's configuration, settings and files were otherwise completely recovered. Scenario #3: recovery of individual files from a VM backup image This scenario is a variation on the recovery of a full VM image from local backup. It extends the test in Scenario #2 with additional procedures designed to restore individual files from a VM's backup image. The test added the process for returning files to their original location on the VM's file system while leaving the rest of the VM intact, i.e., rather than a wholesale replacement of the entire VM to recover one or more individual files, this test validated a procedure for recovering files by extracting them from the backup image. Test objectives Verify that individual files can be recovered from the backup image and restored to their original location on the target VM. Test procedures 1. Identify a target VM for the restore test in vcenter. Confirm successful backups of the test VM through the NetBackup Backup, Archive and Restore application. Confirm also that the backup history lists restore points for every day over the twoweek lifecycle of the retention policy. 2. Identify target files on the VM for recovery. Note the files names, size, contents and path. Delete the files from within the guest operating system by moving them to the Windows Recycle Bin. 3. Identify another VM which can be used to stage the recovered virtual disk(s) for restoring the deleted files. This VM should be running the same guest operating system as the target VM, and should be a member of the same authenticating domain. 4. Launch the Backup, Archive and Restore application on the Master Server. Run the Restore process and choose a recovery point from the available backups. Configure the VM to be recovered to an alternate datastore, and note the new location. Enter an alternate name (e.g. <VM-name-Files> and an alternate datastore. To reduce the recovery time, choose Thin Provisioning for the restored virtual disks. Clear all network connections to avoid potential network and domain 18

19 conflicts when the VM powers on after recovery completes. Run the Pre-Recovery Check, and initiate the recovery process. Monitor recovery status through the Administration Console. 5. When restore process completes, power on the VM and verify recovery of target files as noted in step 2 above, including folder path, file name(s), size(s), and contents. If the file(s) to be recovered may have NTFS permissions different from those of the parent folder, note these as well. Power down the VM and remove it from the vcenter inventory, but do not delete its files. 6. From within vcenter, edit the settings of the staging VM identified in step 3 above. Add a hard disk to the staging VM and choose the Use an existing virtual disk option, then browse to the appropriate datastore selected in step 4 above and locate the appropriate VMDK file(s) to attach to the staging VM. 7. Log on to the staging VM's console. From Disk Manager, mount the newly attached disk and assign it a new drive letter. Open Windows Explorer and browse to the correct path within the newly-assigned drive letter to locate the file(s) to be recovered. 8. Copy the file(s) from the staging VM back to their original location on the target VM. If the NTFS permissions on the restore(d) files differ from the parent directory, these will need to be manually restored, since copying the files back to their original location will cause the restored file(s) to inherit permissions from the destination parent folder. 9. After verifying the restore back to the original location(s) on the target VM, unmount the drive from within the guest operating system of the staging VM. 10. From vcenter, detach the VMDK file from the restore guest. The VMDK file can either be left on the datastore or deleted at this time. Test results Restore process completed successfully. Individual files were completely recovered back to the original state at the time of backup. Note: While this process works for individual files, it can also be used to restore a complete virtual disk back to the target guest machine, without having to reset the entire VM back to its last backup state. Scenario #4: recovery of a VM from remote backup This final test is intended to demonstrate the ability of the solution design to replicate the entire backup dataset to an offsite location using SyncIQ, then recover individual or multiple servers to the offsite location. Test objectives Replicate the Master and Media servers to an alternate site, on a second Isilon cluster and accessible via a second vsphere cluster. Use the replicated NetBackup 19

20 environment to recover VMs to an alternate location and bring services and/or data online in the new site. Note: This procedure assumes that one or more domain controllers from the same domain are online at the recovery site. Using this process to backup and restore domain controllers may lead to domain-level corruption. Active Directory domain controllers should be backed up using one or more supported alternate Active Directory-aware methods. Test procedures 1. From the OneFS WebUI on the primary (source) Isilon cluster, create a new directory. Create an NFS export that references the new directory. This will be the source directory for SyncIQ replication that will be configured in Step 3 below. Note: For best results, create the new directory and NFS export using the procedures outlined in the Deployment Guide EMC Isilon Scale-Out Storage and VMware vsphere 5 white paper. A link to that document is included in the References section of this guide. 2. From the OneFS WebUI on the secondary (target) Isilon cluster, create a new directory and NFS export, using the same processes as in Step 1 above. This will be the target directory for SyncIQ replication that will be configured in Step 3 below. Note: As with Step 1, follow the procedures outlined in the Deployment Guide EMC Isilon Scale-Out Storage and VMware vsphere 5 white paper to complete this step. 3. From the OneFS WebUI on the primary (source) Isilon cluster, create a new SyncIQ policy. Give the new policy a name and description, and set the job to synchronize data between the Isilon clusters, and configure the job to run on a daily schedule. Set the source cluster s root directory to the directory created in Step 1 above. Set the target cluster s name or address, and set the target path to the directory created in Step 2 above. Save the new policy. Note: If the target cluster is specified using an IP address in this step, then SyncIQ will connect to only one node on the target cluster to replicate the source directory. Multiple nodes can be connected if a SmartConnect zone on the target cluster is configured in this step. 4. From the vma, mount a new datastore on all ESXi hosts in the primary-site vsphere cluster to the NFS export created in Step 1 above. This is necessary for proper vsphere HA and DRS functionality. 5. From the vma, connect to the ESXi hosts in the secondary site and mount a new datastore to the NFS export created in Step 2 above. 6. From the Activity Monitor of the NetBackup Administration Console, verify that all backup jobs have completed. Connect to the vcenter client and shut down the 20

21 Master Server and the Media Server. Migrate both NetBackup servers to the new NFS datastore created in Step 3 above. 7. When the migration of both NetBackup servers to the new datastore is complete, start the initial replication process by launching the SyncIQ job configured in Step 3 above. When the SyncIQ job has successfully launched, power both NetBackup VMs back on and continue running backup jobs as scheduled. The synchronization period will depend on the size of the NetBackup servers disks and the speed of the WAN connection. Continue to monitor the SyncIQ job from the WebUI of the source cluster until the directory has been entirely synchronized to the target Isilon cluster. 8. Allow the SyncIQ job to run as scheduled for one week before continuing with the next step. 9. Ensure that all backups have completed on the Master Server through the Activity Monitor of the NetBackup Administration Console. From within the guest operating system of the NetBackup servers, shut down both servers and ensure they are powered off before continuing. 10. Run the SyncIQ job manually. Replication should update only those blocks changed since the last synchronization job and will ensure full symmetry between source and destination data. Monitor the SyncIQ replication job until it completes. Note: The SyncIQ job will create a read-only directory on the target cluster. If viewing the target cluster s file system using the WebUI, a red padlock icon will appear on the target directory. 11. From the WebUI on the target cluster, break SyncIQ policy created in Step 3 above. This will enable read-write access on the target directory and allow the ESXi hosts at the secondary site to mount the NetBackup VMs replicated from the primary site. 12. Shut down both NetBackup servers at the primary site. From a vsphere client connected to the target vsphere cluster, remove both the Master Server and the Media Server from the vsphere inventory. Then, browse the datastore added in Step 5 to the ESXi hosts at the target site until the folders for the two NetBackup servers are located. Add both servers to the local vsphere cluster s inventory. 13. Ensure both servers have network connections to the virtual network on the target vsphere cluster. Boot both NetBackup servers and change the IP address to match the local subnet, and update the DNS server information as necessary. Verify connectivity between the restored NetBackup servers at the secondary site, and with the Active Directory domain. 14. Identify a target VM from the backup catalog to restore to the secondary site. Remove the target VM from vcenter. From the Backup, Archive, and Restore console on the Master Server at the target site, launch the VM restore process. Select an alternate location to recover the image. Change the vcenter server if necessary, as well as selecting an appropriate local ESXi host, Datacenter, Folder and datastore/datastore cluster. 21

22 15. Configure the restore job to power the VM up upon completion. Connect the VM to the appropriate local network. Run the Pre-Recovery Check, and start the recovery process. Monitor recovery status through the Administration Console. 16. When restore process completes, log into recovered VM. Change the IP address to match the local subnet settings and DNS services. Verify connectivity to the network and to Active Directory. Test results Replication of approximately 1TB of data across the WAN link between the primary and secondary sites took 23 hours. Subsequent daily synchronization jobs took roughly one hour each. Target VM restored to the new destination and successfully found the network and Active Directory upon reboot. Test notes Benchmark results are highly dependent upon workload, specific application requirements, and system design and implementation. Relative system performance will vary as a result of these and other factors. Therefore, this workload should not be used as a substitute for a specific customer application benchmark when critical capacity planning and/or product evaluation decisions are contemplated. All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute. Component hardware and software This section lists the hardware used to validate the solution. Storage configuration The storage configuration as described in this solution is intended to include a full representation of the available node types in a typical Isilon storage array. Table 2 describes the hardware and network specifications for each node type in the primarysite cluster, which consists of a total of nine nodes. 22

23 Table 2: Primary-Site Storage Node Hardware Specifications Node # Node Type Quantity Capacity per Node RAM Network Interfaces per Node NL 3 33,256 GB HDD 4GB 2x1Gb/s Ethernet 2x10Gb/s Ethernet 4-6 X GB SSD 9,626 GB HDD 7-9 S GB SSD 7,066 GB HDD 24GB 48GB 2x1Gb/s Ethernet 2x10Gb/s Ethernet 2x1Gb/s Ethernet 2x10Gb/s Ethernet The secondary-site storage cluster consists entirely of NL-series nodes, as shown in Table 3: Table 3: Secondary-Site Storage Node Hardware Specifications Node # Node Type Quantity Capacity per Node RAM Network Interfaces per Node NL 3 33,256 GB HDD 4GB 2x1Gb/s Ethernet 2x10Gb/s Ethernet VMware ESXi host hardware configuration The vsphere cluster consists of eight identically configured host servers, five in the primary site, and three additional hosts, running additional localized workloads at the secondary site. Table 4 describes the hardware configuration of each ESXi host server. The host count and configuration were selected based on the workload profile, and on the expected CPU and memory resource requirements. The host count and configuration are also based on the need to maintain sufficient vsphere cluster capacity in the event of a host hardware failure. 23

24 Table 4: Host Hardware Specifications Host Hardware Host Count Hardware Configuration 5 at primary site 3 at secondary site CPUs per host 2 Number of cores per CPU 8 Number of cores per host 16 CPU type CPU speed Memory per host Intel Xeon E GHz 48GB Network connections 1Gb/s Ethernet interfaces, per host 2 10Gb/s Ethernet interfaces, per host 2 Datastore connections NFS v3 24

25 Environment resources and configurations Table 5 lists the environment components, including component software, software versions, and configuration settings used in this solution. Table 5: Environment Components and Configurations Software Configuration Isilon Array OneFS operating system SyncIQ v at primary site v at secondary site v3 NetBackup 7.5 Servers Operating System Windows Server 2008 R2 Standard Edition, Service Pack 1 vcpu, per VM 4 RAM, per VM SCSI Controller VMware Tools for Windows 4GB LSI Logic SAS v8.6.0, build VMware Hardware Version 8 Symantec NetBackup v7.5 Master Server Disk capacity 1x40GB VMDK Media Server Disk Capacity vsphere Management Assistant appliance 1x40GB VMDK 1x1TB VMDK v

26 Conclusion Summary By combining Symantec s VADP-aware NetBackup 7.5 backup and deduplication software solution with EMC Isilon scale-out storage platform and SyncIQ data protection software, organizations can provide a comprehensive data protection and recovery strategy. NetBackup provides efficient off-host backup that integrates with vsphere to protect VMDK images and critical VM data against loss or damage. SyncIQ provides an efficient mechanism for replicating data between Isilon storage clusters across multiple sites. Finally, by deduplicating VM data at the time of backup with the NetBackup PureDisk capability, data consistency is preserved at all levels without overwhelming disk or network capacity resources. For organizations with workloads beyond those in a vsphere cluster, NetBackup and Isilon both offer simplicity and scale-out capabilities to ensure that organizations can expand their data footprint and backup capabilities to other platforms and services while maintaining the same level of simplicity that customers have come to expect from EMC s Isilon storage. Findings NetBackup 7.5, combined with Isilon SyncIQ, can be used for both local and offsite backups. If the vma is deployed in an organization, the process of shutting down the NetBackup VMs and launching a SyncIQ policy can be scripted and automated, eliminating much of the dependency on human administrators. The overall success of such a solution in any given organization will depend on multiple factors, including: The organization s designated Recover Point Objective How much data an organization needs to recover within the target Recovery Point Objective. The rate of data change per day The type and format of the organization s data. Some data types can be deduplicated at a much better rate than others The speed of the organization s WAN link between sites Finally, before deploying any backup solution into a production environment, a comprehensive testing and pilot program is should be undertaken to ensure both the adequacy and the cost-effectiveness of the desired backup solution. Next steps EMC can help accelerate the assessment, design and implementation of a vsphere backup solution on an Isilon storage platform while lowering the implementation risks. To learn more about this and other solutions, contact an EMC representative. 26