Protecting vsphere Environments with Hitachi Data Protection Suite

Transcription

1 1 Protecting vsphere Environments with Hitachi Data Protection Suite Reference Architecture Guide By Bob Ingram June 2011

2 Feedback Hitachi Data Systems welcomes your feedback. Please share your thoughts by sending an message to Be sure to include the title of this white paper in your message. 1

3 Table of Contents Solution Overview... 3 Key Solution Components... 5 Hitachi Compute Blade Hitachi Virtual Storage Platform... 6 Hitachi Adaptable Modular Storage 2000 Family... 7 Hitachi Data Protection Suite Powered by CommVault... 7 VMware vsphere VMware vstorage APIs for Array Integration with VADP... 9 Solution Design... 9 Hitachi Compute Blade 2000 Chassis Configuration SAN Connectivity Storage Configuration Network Configuration Infrastructure configuration Hitachi Data Protection Suite Configuration Engineering Validation Virtual Machine Backup and Restore Virtual machine Automatic Discovery Backup Scaling of Virtual Machines Microsoft SQL Application Backup and Restore SQL Auto-discovery of SQL Instances and Databases Auxiliary Copy to VTL Conclusion Scaling Virtual Machine Backup and Restore Summary

4 Protecting vsphere Environments with Hitachi Data Protection Suite Reference Architecture This documents the test results of a reference architecture solution for protecting a vsphere 4 environment, the virtual machines, infrastructure, and an application running on a virtual machine using Hitachi Data Protection Suite powered by CommVault in a VMware vsphere 4 environment. This architecture uses these complementary technologies in a backup solution, the VMware vsphere API for Data Protection and VMware vsphere API for Array Integration. To address the strain on backup infrastructure, VMware introduced the vstorage API for Data Protection (VADP). This API gives access to a VMkernel feature called Change Block Tracking (CBT) to enable high-speed incremental and differential backups at the block-level. CBT streamlines backups and addresses backup constraints. This enables the backup software to do the following: Monitor vmotion and Distributed Resource Scheduler (DRS) moves of VMs between datastores Perform full, differential, and incremental image backup and restore of virtual machines Perform file-level backup of virtual machines using supported Microsoft Windows and Linux operating systems Ensure data consistency by using Microsoft Volume Shadow Copy Services (VSS) for virtual machines running supported Microsoft Windows operating systems VADP with CBT provides you an opportunity to re-design backup infrastructure to potentially improve backup performance. Hitachi Data Protection Suite, certified by VMware, takes advantage of CBT. This paper s intended use is by IT administrators responsible for managing system backups. It assumes some familiarity with Hitachi Data Protection Suite, Hitachi Compute Blade 2000, Hitachi Virtual Storage Platform, and VMware vsphere. Solution Overview This scalable reference architecture is built on Hitachi s and VMware s hardware and software for protecting vsphere 4 environments. The Hitachi Compute Blade 2000 was configured with two distinct environments: Data Protection Suite functional virtual machines Protected virtual machines 3

5 This reference architecture provides the following capabilities: Complete virtual machine protection The VADP API protects virtual machines, all virtual disks, and data residing on the virtual machine. Administrators define the rules to base automatic discovery and addition of newly added virtual machines to the backup set. Restore of entire virtual machines, individual virtual disks, or individual files Backup administrators restore entire virtual machines in place on the same ESX host or out of place on another ESX host. Individual virtual disks, files, or folders can be restored. Protect applications residing on virtual machines Protect applications on standalone hosts or on virtual machines using the idata Agents in Data Protection Suite. There is automatic discovery of newly added databases and database instances. Restore database instances or individual databases Backup administrators restore database instances or individual databases to the same application host or to a different application host. Figure 1 shows an overview of the solution documented in this paper. Figure 1 4

6 To support this reference architecture, all infrastructure components were implemented in virtual machines on one ESX 4.1 host, including: Active Directory services Domain name services Data Protection Suite components vcenter server The protected virtual machines were implemented on a second ESX 4.1 host. Key Solution Components Table 1 and Table 2 list the hardware and software components used in this reference architecture. Following the tables are introductions to these components. Table 1. Hardware Componets Hardware Detail Description Version Quantity Hitachi Compute Blade 2000 chassis Hitachi Compute Blade 2000 X57A1 blades Hitachi Virtual Storage Platform Hitachi Adaptable Modular Storage blade chassis 2 8Gb/sec Fiber Channel switch modules 4 1/10Gb/sec network switch modules 2 management modules 8 cooling fan modules 4 power supply modules Full blade 2 8-Core Intel Xeon X7500 processors 256GB memory per blade 4 Microprocessor blades 4 Disk Adapters 2 16 port Fibre Channel Adapter pairs GB 10K RPM SAS drives 80GB Cache Memory Dual controllers 2 4GB Fibre Channel ports 8GB cache memory 20 1TB, 7200 RPM, SATA disks A0160-G / /B-M 1 Brocade 5300 Edge switches, 8Gb/sec 6.4.0c 6 5

7 Table 2. Software Components Software Version Hitachi Command Suite 7.01 VMware vcenter server build VMware Virtual Infrastructure Client VMware ESX build Microsoft Windows Server Hitachi Data Protection Suite Hitachi Compute Blade SP1 64 bit Enterprise 9.0 SP1 The Hitachi Compute Blade 2000 is an enterprise-class blade server platform. It features: A balanced system architecture that eliminates bottlenecks in performance and throughput Embedded logical partition virtualization (not utilized in this reference architecture) Configuration flexibility The Hitachi Compute Blade 2000 supports up to two management modules for redundancy. Each module is hot-swappable and supports the following: Live firmware updates without the need for shutting down the blades A LAN management interface that is independent from the data network for remote and secure management of the chassis and all blades A serial command line interface and a web interface SNMP and alerts For this reference architecture, the cold standby feature was not used. Instead, the failover used VMware vsphere clusters. Hitachi Virtual Storage Platform The Hitachi Virtual Storage Platform is a 3D scaling storage platform. With the ability to scale up, scale out, and scale deep at the same time in one storage system, the Virtual Storage Platform flexibly adapts for performance, capacity, connectivity, and virtualization. Scale Up Increase virtual server consolidation, improve utilization of resources, and reduce costs. Scale Out Add new physical or virtual servers to your environment to meet business demands. Scale Deep Extend the advanced functions of the Virtual Storage Platform to external multivendor storage. The trend in server virtualization is to consolidate the I/O workload of many servers onto a single storage system. The consolidation of more virtual machines onto a physical host requires storage systems to add more storage resources dynamically to keep up with I/O demand. The 3D scaling capability of the Hitachi Virtual Storage Platform meets that requirement. 6

8 The switch matrix architecture of the Hitachi Virtual Storage Platform makes all of this possible. It connects the basic components, front-end directors, back-end directors, global cache modules and virtual storage directors. As required, add redundant pairs of directors and cache modules without disruption to connected host servers. A global cache tightly couples all these resources that create a common pool of storage resources that can include external storage that is connected through frontend director initiator ports. You can double the scalability of the Hitachi Virtual Storage Platform using a dual-chassis system with up to six racks. The logic box in each chassis is the same, using the same types and numbers of logic boards. Any front-end port can access any back-end RAID group. No division within the storage system exists between the chassis. For more information about the Hitachi Virtual Storage Platform, see the Hitachi Data Systems web site. Hitachi Adaptable Modular Storage 2000 Family The Hitachi Adaptable Modular Storage 2000 family systems, with dynamic load balancing controllers, provide integrated, automated, hardware-based, front to back end I/O load balancing. This eliminates many complex and time-consuming tasks that storage administrators typically face. Both controllers dynamically and equally manage, balance, and share I/O traffic to back-end disk devices. The point-topoint back end design nearly eliminates I/O transfer delays and contention associated with Fibre Channel arbitration. This provides significantly higher bandwidth and I/O concurrency. The active-active Fibre Channel ports mean you do not need to be concerned with controller ownership. I/O is passed to the managing controller through cross-path communication. Any path can be used as a normal path. The dynamic load balancing controllers assist in balancing microprocessor load across the storage systems. In this solution, the use of Hitachi Adaptable Modular Storage 2000 family of products is for externally virtualized storage managed by the Hitachi Virtual Storage Platform. SATA disks are used as backup media. Hitachi Data Protection Suite Powered by CommVault The Hitachi Data Protection Suite powered by CommVault provides the foundational backup and recovery features to build a truly unified data management solution. These features support the following: Improved information asset management Speedier recovery of data Increased savings over loosely integrated legacy solutions The CommCell Management Group defines the scope of control for the CommServe host. All components under the licensing and control of the CommServe host are defined as being members of the CommCell group. The CommCell Console is the management graphical user interface (GUI) used for managing the CommCell. The media agent (MA) is responsible for the transfer of data from the client computer to the backup media, whether disk or tape. In the Hitachi Data Systems lab the MA was installed on a virtual machine. This reference architecture used a virtual tape library (VTL) supplied with the Data Protection Suite for testing purposes. A large number of UNIX and Windows platforms support the MA. 7

9 Using a non-virtual tape library requires using a physical host for connectivity to the library. In this case, the MA must be installed on a physical host. This reference architecture used three idataagents (ida) for communication to execute backups: Windows File System ida By default, this ida is installed along with every ida when detecting a Windows host. VMware ida The VMware ida, also called the Virtual Server Agent (VSA), is installed on a proxy host. This proxy host is a physical or virtual machine running Microsoft Windows. The Virtual Server Agent communicates with vcenter or the ESX host to discover virtual machines and execute backups using the vstorage API for Data Protection (VADP). Microsoft SQL ida This application ida is installed on the SQL host. It executes backups on the SQL server system databases and SQL server user databases. A complete list of application idas is available that the Hitachi Data Protection Suite documents web site listed below. Using the virtual server agent and the media agent reduce the impact of backup processes on production servers over traditional backup methods by offloading processing to a proxy host. Hitachi Data Protection Suite takes a snapshot of the VM, and then accesses the snapshot through the proxy server. The Hitachi Data Protection Suite can scale up to thousands of virtual machines and physical servers. Its configuration allows the automatic discovery and protection of newly added virtual machines to the environment. The de-duplication capabilities in Hitachi Data Protection Suite can reduce the amount of data being backed up from the source, shorten the backup window and significantly reduce bandwidth requirements. It can work with key data management operations, such as archiving, search, encryption, and direct recovery from de-duplicated tape. The Virtual Server Agent performs de-duplication at the proxy host, reducing the overall load to the production virtual machines and reducing backup traffic. The Hitachi Data Protection Suite is compatible with most servers, storage systems, applications, databases, and file systems. For more information and support see the CommVault website. For Hitachi Data Protection Suite documentation, go to CommVault Books on-line. VMware vsphere 4 VMware vsphere 4 is a highly efficient virtualization platform that provides a robust, scalable and reliable infrastructure for the data center. With features like Distributed Resource Scheduler (DRS), high availability, and fault tolerance, vsphere 4 provides an easy-to-manage platform. Use of vsphere 4 s round robin multipathing policy distributes load across multiple host bus adapters (HBAs) and multiple storage ports. Use of DRS with Hitachi Dynamic Provisioning automatically distributes loads on the ESX host and across the storage system. 8

10 With vsphere 4 architecture, backups and restores can be performed using the vstorage API for Data Protection. This new API set provides significant performance improvements over VCB, including the following: During backups, the Virtual Server Backup Agent creates a snapshot of the virtual machine data directly from the datastore. Then, the snapshot is moved directly to the backup media through the media agent without requiring any dedicated disk cache on the proxy server. Incremental backups with change block tracking (CBT) quickly identifies only the data blocks on the virtual machine that have changed since the last backup. Virtual machines or individual files are restored directly to the appropriate virtual disk through the Windows file system ida. Virtual machines with enabled fault tolerance are supported for protection and recovery. VMware vstorage APIs for Array Integration with VADP VMware vstorage APIs for Array Integration (VAAI) is not a part of VADP, but it offers advantages when making virtual machine backups. Making a backup of a virtual machine using VADP takes a snapshot of that virtual machine. The time it takes to open or close the snapshot can vary. Running vsphere 4.1 with VAAI-enabled storage systems makes snapshots at the hardware logical block address (LBA) level. This is instead of making a snapshot by software at the VMDK level. This significantly improves snapshot performance, including backup time. In addition, using VAAI removes the requirement to lock LUN using SCSI reservations by using the LBA, instead. More VM snapshots can be performed on a LUN at the same time without affecting performance. Solution Design This section describes the VMware vsphere and Hitachi Data Protection Suite solution architecture. The following is true for this reference architecture: The weekly and daily backups used SATA disk storage on the external Hitachi Adaptable Modular Storage The monthly archive used a virtual tape library on the external Hitachi Adaptable Modular Storage This solution utilized two Hitachi Compute Blade 2000 server blades: One server blade is dedicated to infrastructure virtual machines. One server blade is dedicated to the virtual machines being protected with Hitachi Data Protection Suite. 9

11 All storage is controlled by the Hitachi Virtual Storage Platform domain. The production storage is configured on internal SAS disks The backup storage is configured on external SATA disks. All storage is configured using Hitachi Dynamic Provisioning on the Hitachi Virtual Storage Platform. Hitachi Compute Blade 2000 Chassis Configuration This reference architecture uses the following: Two standard X57A1 blades Four 1/10 GigE LAN switch modules Two Fibre Channel switch modules Each blade has the following: Two on-board NICs 4 NICs provided by a mezzanine card Each of these NICs is connected to a LAN switch module. Each blade also has a 2 port Fibre Channel mezzanine card installed that is connected to both of the Fibre Channel switch modules. The host server architecture is a critical component of the virtualized infrastructure. The ability of the host servers to handle the workload of a large number of consolidation candidates increases the consolidation ratio and helps provide the desired cost benefits. SAN Connectivity Figure 2 shows the physical SAN connectivity to all hardware. 10

12 Figure 2 Table 3 shows the SAN zoning for the Hitachi Compute Blade 2000 servers. Table 3. SAN Zoning for Thie Hitachi Compute Blade 2000 servers Host Host HBA Number Director Zone Name Storage Port Storage Host Group BS2K-06-B4 HBA1_1 BS2K-06-B4-HBA1_1 CL1-C BS2K-06_B4 BS2K-06-B4 HBA1_2 BS2K-06-B4-HBA1_2 CL2-C BS2K-06_B4 BS2K-06-B5 HBA1_1 BS2K-06-B5-HBA1_1 CL3-C BS2K-06_B5 BS2K-06-B5 HBA1_2 BS2K-06-B5-HBA1_2 CL4-C BS2K-06_B5 Table 4 shows the SAN zoning for external storage connections between the Hitachi Virtual Storage Platform and the Hitachi Adaptable Modular Storage Table 4. SAN Zoning for the External Storage AMS2300 Port Director Zone Name VSP Port AMS2300 Storage Host Group 0A ASE45_37_1D_ASE_90_0A CL1-D 000-G001 1A ASE45_37_2D_ASE_90_1A CL2-D 000-G001 11

13 Storage Configuration For this reference architecture, Hitachi Dynamic Provisioning was used for all storage. Five dynamic provisioning pools were created on the Hitachi Virtual Storage Platform. The internal dynamic provisioning pools (pools 2, 3, and 4) were used for virtual machine boot and data devices. The external dynamic provisioning pool (pools 5 and 6) were used for backup devices. Pool 5 was used as a disk library. Pool 6 was configured to use the virtual tape library (VTL) supplied with the Hitachi Data Protection Suite for testing purposes. Note The VTL is not supported for use in production systems. Only use the VTL for testing. The dynamic provisioning pools are described in Figure 3. Figure 3 12

14 Table 5 shows the dynamic provisioning configuration, logical device (LDEV), LUN allocation, and the virtual machine file systems (VMFS). Table 5. Dynamic Provisioning Configuration Dynamic Provisioning Pool vsphere_boot Usage LDEVS VMFS Datastore ESX and Virtual Machine boot devices 02:00 ESX boot 02:01 ESX boot 02:02 VM boot 02:03 VM Boot Test 02:04 Data Protection Suite dedupe vsphere_data SQL Data 03:05 SQL data 03:07 SQL local backup vsphere_logs SQL Logs 03:06 SQL Logs SQL_Logs vsphere Backup vsphere_vtl Data Protection Suite Disk backup Virtual Tape Testing 03:00 Disk Backup 03:01 Disk Backup 03:04 vsphere restore staging 03:02 VTL 1 03:03 VTL 2 Datastore1 (BS2K-06-B4) Datastore1 (BS2K-06-B5) VM_Boot VM_Boot_Test HDPS_D SQL_Data SQL_Backup Disk_Backup Disk_Backup2 VMware_Proxy_BU VTL_Backup1 VTL_Backup2 Calculating the size of the backup storage pool requires a full understanding of a number of factors: Data retention policies Compression ratio De-duplication Data growth These factors are difficult to predict, even when there is careful planning. Hitachi Dynamic Provisioning can assist in several ways: Oversubscription of dynamic provisioning pools. Monitoring dynamic provisioning pool usage. Online expansion of dynamic provisioning pool. The sizing of backup disk pools for performance is difficult to predict, also. SATA disks in the Hitachi Virtual Storage Platform or AMS 2300 contain 3Gbps interfaces. Due to the large capacities, most SATA drives are used in RAID-6 configurations. This avoids the likelihood of a dual drive failure during the potentially long rebuild times of a failed drive onto a spare drive. Hitachi Data Systems recommends the use of RAID-6 (4D+2P) for SATA disks. Hitachi Dynamic Provisioning benefits backup performance by striping data across all disks in the dynamic provisioning pool. 13

15 Network Configuration The Hitachi Compute Blade 2000 hosts used in this reference architecture contained six network interfaces, with the configuration described in Table 6. Table 6. Networking Details NIC Speed Usage VLAN NIC0 100 Half Blade Management Console 168 NIC Full ESX Service Console 168 NIC Full Unused Unused NIC Full Distributed vswitch for virtual Machine connectivity 171 NIC Full vswitch 1 for backup and restore 170 NIC Full vswitch2 for VMkernel and vmotion 172 The VMwareProxy virtual machine and media agent were configured to utilize vswitch1 for backup and restore and the distributed vswitch for regular virtual machine connectivity. This segments the backup traffic so that VLAN 170 contains only backup traffic and VLAN 171 contains only virtual machine traffic. If the communication between virtual machines on the same ESX host is required, then data is transmitted through the ESX kernel. Infrastructure configuration To simulate a production data center, two virtual machines were used. vcenter server vcenter was used to manage the overall vsphere 4 environment. Active directory A Microsoft Active Directory (AD) server was created and all virtual machines joined to the domain. Hitachi Data Protection Suite Configuration For this reference architecture, the Hitachi Data Protection Suite infrastructure was installed on virtual machines. All virtual machines utilize Microsoft Windows 2008 SP2 64 bit Enterprise version. CommVault documentation has minimum resources required for the Hitachi Data Protection Suite infrastructure. These recommendations provide limited resources for a production environment. The Hitachi Data Systems labs implementation started with the minimum configuration. CPU and memory resources were monitored during backup and restore cycles. Resource utilization was monitored from the ESX hosts using ESX performance monitor and ESXtop, and monitored from the virtual machine through the use of Windows perfmon. When using virtual machines for the Data Protection Suite, do not oversubscribe ESX CPU and memory resources. This means that the total virtual CPU and virtual machine memory do not exceed the total physical available. 14

16 The virtual machines used in the Hitachi Data Protection Suite implementation are described in Table 9. It shows the minimum requirements and the actual resources used. Table 9 Virtual Machines Virtual Machine CommServe Media Agent VMwareProxy Usage vcpu Memory Ties together the Data Protection Suite CommCell components Transfers the data from the client computer to the backup storage idataagent interfaces with VADP Minimum required 2 32 bit or 64 bit Used 4 64 bit Minimum required 2 64 bit Used 6 64 bit Minimum required 2 32 or 64 bit Used 8 64 bit Minimum required 4GB Used 16GB Minimum required 4GB Used 16GB Minimum required 4GB Used 32GB Figure 4 shows the virtual machines used. Figure 4 15

17 Engineering Validation This section describes the tests performed in the Hitachi Data Systems lab. Seven use cases were tested: Virtual Machine Backup and Restore These tests used the VMware Data Protection Suite Virtual Server ida to back up and restore general purpose virtual machines, such as virtual desktops. Virtual machine Automatic Discovery These tests added virtual machines to the environment to test the automatic virtual machine discovery feature of the VMware Data Protection Suite. Virtual Machine Scaling These tests added virtual machines to scale out the protection environment to 300 virtual machines. Microsoft SQL Application Backup and Restore These tests installed the VMware Data Protection Suite ida for SQL on a virtual machine running Microsoft SQL used for database backup and restore. SQL Auto-discovery of SQL Instances and Databases These tests installed an additional SQL instance while adding more databases added to the first SQL instance to test the automatic discovery of SQL databases and instances. vsphere vcenter Backup and Restore These tests performed the vcenter backups using the VMware Virtual Server Agent and the VMware SQL ida. Auxiliary Copy to VTL These tests created an auxiliary copy of the virtual machine and SQL server backups to simulate the creation of a tape copy for off-site backups. Virtual Machine Backup and Restore In these tests, a general purpose virtual machine was created using Windows 2008 R2 Enterprise 64 bit and then converted to a template. Using this template, additional virtual machines were deployed. The VMware ida was used to connect to vcenter and discover the virtual machines for backup and restore operations. VAAI hardware-assisted locking provides a granular LUN locking method. This allows locking at the logical block address level without the use of SCSI reservations or the need to lock the entire LUN from access by other hosts. Since the Hitachi Virtual Storage Platform is VAAI enabled, the snapshot operations lock the logical block addresses of the LUN during creation and growing of the virtual disk snapshot. No SCSI reservations occur during the snapshot operations. Before beginning tests, VAAI was enabled to provide hardware assisted locking by configuring the following parameters. ESX Host, Configuration, Data Mover.HardwareAcceleratedMove=1 ESX Host, Configuration, Data Mover.HardwareAcceleratedInit=1 ESX Host, Configuration, VMFS3.HardwareAcceleratedLocking=1 Virtual Storage Platform Host Mode Option, 54 Support for the EXTENDED COPY Command=enable 16

18 The VMware Virtual Server ida was installed on the VMware Proxy virtual machine and the CommCell console auto-discovered the agent. Two agents were automatically added to the CommCell client computers: FileSystem This agent was used to back up the VMwareProxy host, system state, and all file systems. Included in all agents is defaultbackupset, which includes the sub-client default. The default backup set in the FileSystem sub-client was used to perform a full backup of the VMwareProxy host. VirtualServer This agent was used to connect to the vcenter server and discover all ESX hosts and virtual machines. The VirtualServer ida is the connection to the VMware VADP for handling virtual machine backups. The VADP allows Hitachi Data Protection Suite to use vcenter to create a snapshot of the virtual machine, and then mount the virtual disks to the proxy host through the vsphere hotadd feature. The VirtualServer ida also allows the backup of a virtual machine in suspended or power off state by accessing the virtual disk from the proxy virtual machine. Resources on the VirtualServer ida virtual or physical machine are important to back up and restore performance. Monitor the CPU, memory, network, and disk resources to know when to add necessary resources. In CommCell, under the sub-client VMwareProxy, an instance called vcenter was added. Then, a connection to the vcenter server was configured. Three backup sets were then configured using this agent: File level, configured to backup individual files from the virtual machines Disk level, configured to backup the virtual disks Volume level, configured to backup on the volume level All backup sets were configured to use the VMwareProxy. Because Hitachi Data Protection Suite is VMware aware; file level restore can be performed regardless of the backup level. Be sure the Enable Granular Recovery check box is selected (default) in Advanced Backup Options when performing the backup job if file level restore is required. Three restore types were tested. File Level Restore individual files and folders to the Virtual Server ida proxy host, or restore to the original location by adding the ida in restore mode to the virtual machine. Disk Level Restoration can be as a.vmdk file and then copied to the VMFS datastore, or restoration can be as a complete virtual machine. Volume Level Restoration can be as a physical volume, volume level as a VHD, or volume level as a vmdk file. 17

19 For consistency in the size of the data set, de duplication was turned off. Two virtual machines were used for these tests. Backup and restore times were recorded to compare backup types. All tests backed up the Windows folder on drive C. Table 7 shows the results of the backup tests. Table 7. Virtual Machine Backup Test Results Backup Type Compression ratio Backup Job Time (mm:ss) File Level 68.25% 22:22 Disk Level 63.9% 9:56 Volume Level 68.25% 13:17 The restore was performed to test restoring individual files as well as entire volumes and entire virtual machines. Since Hitachi Data Protection Suite is aware of the VMware virtual disk, file level restore is available with any backup type. All file level tests restored the Windows folder on drive C. When restoring individual files, they were placed in a staging area on a host with the Virtual Server ida installed. For these restores the VMwareProxy host was used for staging and a virtual disk was created in the VMFS datastore VMware_Proxy_BU for a temporary staging area. The restore job time was recorded when the files were restored to the temporary staging area. Table 8 shows file level restore times for the three backup types. All restore sets were equal size. Table 8. File Level Restore Test Results Backup Type Restore Job Time (mm:ss) File Level 7:55 Disk Level 11:43 Volume Level 18:07 Next the disk level restore was tested. Virtual disks are restored as.vmdk files to the Virtual Server ida proxy host. From there they were moved to the VMFS datastore through the vcenter datastore browser. Restore time is measured when the restore to the proxy is complete. Table 9 shows restore time from a disk level backup for the supported backup type. Table 9. Disk Level Restore Test Results Backup Type File Level Restore Job Time (mm:ss) Not supported Disk Level 11:17 Volume Level Not supported 18

20 Next, the complete virtual machine restoration was tested. If the backup was performed as a disk level backup, the entire virtual machine can be restored as one entity. The restore job time was recorded when the virtual machine was restored to the ESX host. Table 10 shows the time from the virtual machine level backup for the supported backup type. Table 10. Virtual Machine Level Restore Test Results Backup Type File Level Restore Job Time MM:SS Not Supported Disk Level 11:43 Volume Level Not Supported Volume level restore can be used to restore to a virtual hard disk (VHD) that can be mounted on another host. This includes a Microsoft Hyper-V host for data mining or a.vmdk file for mounting to a VMware vsphere virtual machine. These specific use cases were not tested. For more information on choosing backup and restore types, see the CommVault documentation on VMware. Virtual machine Automatic Discovery The ida for VMware automatically discovers new virtual machines based on rules set in the sub-client. The rules are based on the following criteria: Cluster Affinity Data Center Affinity Datastore Affinity ESX Affinity Match host names by regular expression OS Group ResourcePool Affinity VApp Affinity For the tests in the Hitachi Data Systems labs, the datastore affinity rule was used. By using datastore affinity, any virtual machines using or added to the datastore VM_Boot_Test as the OS boot hard disk is automatically added to the backup. The first phase of this backup was the discovery phase. Two virtual machines were restored from backup using the disk level backup and placed on the datastore VM_Boot_Test. After that, an incremental backup was run. 19

21 After the discovery phase, the backup set content shows the two additional virtual machines. These will be included in the next backup cycle. Since there has not been a backup performed on these virtual machines, a full backup will be performed on these virtual machines regardless of the type of backup selected for the next backup cycle. Backup Scaling of Virtual Machines Hitachi Data Systems recommends using a building block approach to scale the backup environment. A building block consists of a proxy virtual machine and a media agent host. If tape based backups are required, the media agent must be a physical host for connectivity to the tape library. If SAN based backups are required, the VMware ida can be installed on the media agent physical host. These tests were used to determine how the auto-discovery and VADP features will scale when large numbers of virtual machines are in a single backup set. To do this, a building block approach was used that added duplicate proxy virtual machines containing the virtualization ida and duplicate disk libraries. The key metric monitored was throughput in gigabytes per hour. Scaling of the backup was limited to the proxy host parameters and duplicate building blocks. Scaling of the backup LAN was not tested. Scale to 125 Virtual Machines For this test, 125 virtual machines were created from a template using the VMware powercli. Virtual machine names of Desktop1 to Desktop125 were used. In the sub-client, the discovery setting for Match guest host names by regular expression was set to Desktop*. Backups tested the following parameters: The number of SCSI adapters on the Proxy VM The number of CPUs and memory size for the proxy virtual machine Auto-discovery of backup clients By default, there is only a single virtual SCSI adapter on a virtual machine. This only allows a proxy virtual machine to mount 15 devices at a time. This means only 15 snapshots may be backed up at one time. Therefore, when starting the backup, 15 snapshots were created, mounted to the proxy virtual machine, and the backups are made. Next these snapshots were unmounted from the proxy virtual machine and deleted, and the next 15 snapshots are created, mounted, and saved. This process repeated until the completion of all 125 virtual machine backups. The backup throughput was approximately 213GB/hr. The CPU utilization on the proxy virtual machine was within an acceptable range of 70-80% with 6 virtual CPUs configured. The full backup completed in 10 hours 1 minute. For the next test, additional SCSI adapters were added to the proxy virtual machine by adding a small hard disk at SCSI 1:0, SCSI 2:0 and SCSI 3:0. This allowed the next backup to mount 24 disks to the proxy virtual machine (instead of 15) and increased the throughput to approximately 625GB/hr. The full backup completed in 2 hours 1 minute. CPU utilization on the proxy virtual machine was at 100%. Memory was configured at 16GB with the memory was at 100% utilization. Finally, two additional CPUs and 16GB of memory were added to the proxy virtual machine. This brought the configuration to the maximum of 8 virtual CPUs and 32GB of memory. The backup test increased the throughput to 715GB/hr with the full backup completed in 1 hour 46 minutes. CPU utilization returned to an acceptable range of approximately 90% with the memory at 95% utilization. 20

22 Figure 5 shows the throughput of the backup based on the number of SCSI busses configured on the proxy virtual machine. Figure 5 Figure 6 shows the throughput with additional virtual CPUs and memory. Figure 6 21

23 Scale to 300 Virtual Machines For this test, an additional 175 virtual machines were created from the template for a total of 300 virtual machines. No workload was applied to the virtual machines. To start, the added virtual machines used the same backup configuration as the original 125 virtual machines. The initial backup throughput was 640GB/hr. Then, a second proxy virtual machine was built. Two backup sets containing 150 virtual machines each were configured on the vcenter instance to use separate proxy virtual machines. The two backup jobs were executed at the same time. This increased the number of concurrent virtual machine backups from 24 to 48. Total backup job throughput was 640GB/hr and the backup completed in 7 hours 40 minutes. Figure 7 shows the throughput comparison with one proxy virtual machine and two proxy virtual machines. Figure 7 Microsoft SQL Application Backup and Restore This discusses the SQL backup and restore performed in the Hitachi Data Systems lab. A 10,000 user TPCE database was created. The Hitachi Data Protection Suite SQL ida and Windows file system ida agents were installed on the virtual machine hosting the SQL database. These idas were used to backup the SQL database and transaction logs and drive C. After installing the ida on the SQL host, Hitachi Data Protection Suite auto-discovered all databases configured in the SQL instance. After the backup was complete, the TPCE database was deleted using Microsoft SQL Management Studio. The properties of drive F, where the SQL_Data was stored, were examined to verify the SQL data removal. Finally, the database was restored from the backup. The restore was verified by examining the Windows file system space used and Hitachi SQL Server Management Studio. 22

24 SQL Auto-discovery of SQL Instances and Databases The Hitachi Data Protection Suite SQL ida automatically discovers any SQL instances added to the SQL host, if configured during installation of the SQL ida. By default, the discovery is scheduled every 24 hours or whenever restarting the communications service (GxCVD). Figure 8 shows the SQL Server Properties dialog box with Auto discover instances and Manage Content Automatically selected. Figure 8 In addition, the default sub-client within the SQL client will automatically backup any databases added to the SQL instance. To test the auto-discovery of SQL instances feature, a second SQL instance named HDPS_AUTO was installed with a database created called tpce_1000. All services on the SQL host were restarted using the Windows Server manager. Once the SQL services were restarted, the CommServe console was refreshed. The console showed the discovery of both SQL instances. Next, the auto discovery and backup of a database was tested. A database called tpce_5000 was added to the instance HDPS_AUTO and a backup run using the default sub-client. Once the backup was completed, Browse Backup Data was performed. All databases, including the recently added tpce_5000 database, were found in the backup set. vsphere vcenter Backup and Restore VMware vcenter is the configuration engine for the virtual infrastructure. If the vcenter server and database is lost, all the virtual machines will continue to run. However advanced features, such as Distributed Resource Scheduler and vmotion, will not be available. In addition, no environment-wide configuration changes can be made. If vcenter is installed in linked mode, multiple vcenter servers will contain the configuration. However, protecting vcenter is still of major importance. The vcenter installation can be on a virtual or physical machine. It can connect to an SQL or Oracle database running on the same or different host. 23

25 The first test was performed on a vcenter implementation which was installed on a virtual machine. The SQL Express, which is included with vcenter, was used as the database. A disk level backup of the virtual machine was performed. When restoring the vcenter, it was assumed that vcenter was not available. The Default Virtual Machine option was changed to install directly to the ESX host. After the virtual machine was restored, vcenter was examined and compared to the configuration of the original install. All objects were found to be in place, including vswitches, Distributed vswitches, Clusters, and licenses. For the next test, a vcenter instance was installed on a virtual machine and the SQL database was created on a separate virtual machine. The vcenter virtual machine was backed up using the Virtual Server Agent and a disk level backup. The SQL database was backed up using the SQL ida. Once the backups were complete, the vcenter and SQL virtual machines were deleted. Two new virtual machines were built using a template and SQL was installed on one. The SQL ida was pushed to the new SQL virtual machine and database backup was restored. The vcenter server certificate store located at %ALLUSERSPROFILE%\VMWare\VMware VirtualCenter was restored to the new vcenter virtual machine and vcenter was installed from the install DVD. During the vcenter install, when attaching to the SQL database, the Do not overwrite, leave my existing database in place option was clicked. Once vcenter was installed, it was examined and compared to the configuration of the original install. All objects were found to be in place, including vswitches, Distributed vswitches, Clusters, and licenses. Auxiliary Copy to VTL An auxiliary copy is used for secondary and off-site archives. The backup data is archived by copying from the backup device to the tape library. Only the MA hosts are utilized. For maximum efficiency and performance, the media agent hosts requires SAN and tape library access. With these in place, all auxiliary copies data movement will be in the SAN, not the LAN. Figure 9 illustrates the data flow for auxiliary copies using two SAN connected MA hosts. Figure 9 24

26 Auxiliary copies were scheduled to run once per day with one week retention. In the Hitachi Data Systems lab, a virtual tape library was used for the testing. During the engineering, validation in the Hitachi Data Systems lab the auxiliary copy was examined each day after completion and found to have run successfully. Conclusion This reference architecture guide documents a tested backup solution using Hitachi Data Protection Solution powered by CommVault for vsphere environments. This solution leverages the vsphere API for Data Protection and vsphere API for Array Integration, which are complementary technologies efficiency in a backup solution. Scaling During backup the number of concurrent backups of virtual machines is limited by the total number of SCSI busses available, the number of proxy hosts, and the CPU of the proxy host. Hitachi Data Systems recommends using the following virtual machine configuration: 8 virtual CPUs 32GB of memory Two SCSI buses on the proxy host The number of building blocks is dependent on the length of time allotted to protecting all virtual machines and the rate of change on the virtual machines. Virtual Machine Backup and Restore For the most efficiency and flexibility, Hitachi Data Systems recommends using disk level backups. While file level backups perform better than disk level or volume level during restore operations, the backups will be slower. File and volume level backups do not support full virtual machine restores. In sizing storage for restore operations, allow for all of the virtual disks on the virtual machine to be restored, as all virtual disks will be restored to the target VMFS datastore. Once the restore is completed, storage vmotion can be used to move the virtual disk to the original location. 25

27 Summary Some of the advantages of the Hitachi Data Protection Suite solution are shown in Table 11: Table 11. Hitachi Data Protection Suite Solution Advantages Feature Data Protection Suite integration with VADP Virtual Storage Platform integration with VAAI Virtual Machine Disk, volume or file level backups supported Data Protection Suite integration with common applications Advantages Integration with vcenter server and ESX to create snapshots of virtual machines. Offloads vsphere snapshot copy operation to the storage processor. Reduces SCSI reservation issues by locking at the logical block address instead of LUN level. External storage inherits VAAI support. All backup types support file level restores. Disk level backups support entire virtual machine restore. Integrates with the application to ensure consistent backup of the application data. For more information on the features of Hitachi Data Protection Suite powered by CommVault, see the datasheet. Hitachi Data Systems Global Services offers experienced storage consultants, proven methodologies, and a comprehensive services portfolio to assist you in implementing Hitachi products and solutions in your environment. For more information, see the Hitachi Data Systems Global Services web site. Live and recorded product demonstrations are available for many Hitachi Data Services products. To schedule a live demonstration, contact a sales representative. To view a recorded demonstration, see the Hitachi Data Systems Corporate Resources web site. Click the Product Demos tab for a list of available recorded demonstrations. Hitachi Data Systems Academy provides best-in-class training on Hitachi products, technology, solutions, and certifications. Hitachi Data Systems Academy delivers on-demand web-based training (WBT), classroom-based instructor-led training (ILT), and virtual instructor-led training (vilt) courses. For more information, see the Hitachi Data Systems Academy web site. For more information about Hitachi products and services, contact your sales representative or channel partner, or visit the Hitachi Data Systems web site. 26

28 Hitachi is a registered trademark of Hitachi, Ltd., in the United States and other countries. Hitachi Data Systems is a registered trademark and service mark of Hitachi, Ltd., in the United States and other countries. All other trademarks, service marks and company names mentioned in this document are properties of their respective owners. Notice: This document is for informational purposes only, and does not set forth any warranty, expressed or implied, concerning any equipment or service offered or to be offered by Hitachi Data Systems Corporation Hitachi Data Systems Corporation All Rights Reserved. AS June 2011 Corporate Headquarters Regional Contact Information 750 Central Expressway, Americas: or info@hds.com Santa Clara, California USA Europe, Middle East and Africa: +44 (0) or info.emea@hds.com Asia Pacific: or hds.marketing.apac@hds.com 27