VMware and Microsoft VSS: What You Need to Know

Commissioned by VMware and Microsoft VSS: What You Need to Know A Concentrated Technology SolutionSpace Analysis Greg Shields white paper / page 1

Introduction When it comes to Microsoft VSS, there s more to backing up virtual machines than meets the eye. You ve probably heard the somewhat contradictory statement that, Virtualization improves server backups while it at the same time complicates server backups. This statement rings true in part due to virtualization s new approaches in how backups can be captured. You know the classic stories. Once virtualized, a multitude of options present themselves for backing up a virtual server. On one hand, you can continue backing up servers just like in the physical world. With this method, you might install a backup agent into each virtual machine and back up files one by one to disk or tape. While this is an obvious tried-and-true method for backing up server data, it gains none of the benefits one desires out of virtualization. Restoring entire virtual machines isn t easy. The resource-heavy backup process itself impacts virtual machine performance. Backups still take long periods of time to complete. On the other hand is the host-based backup approach. Using this approach, VMs are backed up with assistance from their virtual host. Entire VM disk files.vmdk or.vhd can be captured at once, enabling easy restores of whole virtual machines in the case of failure. Smart vendors now integrate entire-vm backups with individual file restore, enabling files and folders to be restored with the same level of ease. Even smarter vendors go one step further, backing up server applications and their data with the same granularity and performance as individual files. Nearly every datacenter today places as much importance on application data as file data. Microsoft SQL Server, Exchange Server, Active Directory, Oracle, and others are all applications labeled with Tier 1 datacenter priority. That prioritization means that that your backup solution must handle application backups as richly as files, folders, and entire VMs. You might, however, be surprised to know that not all application-aware backups are created equal. While every backup solution is obviously different in how it performs its duties, you might not know that the on-the-server services used to gather application data can also be very different as well. The difference? Microsoft s onboard Volume Shadow Copy Service, or VSS. You probably know that this service is used by Windows to create volume snapshots for backup and recovery purposes. But there are more pieces to VSS than most people are aware of. Not understanding those pieces, and how they impact the success of backup and restore, can have a big impact on your data protection. Sound concerning? It is. Read on. white paper / page 2

What is VSS? Microsoft s Volume Shadow Copy Service, or VSS, is Windows built-in infrastructure for application backups. A native Windows service, VSS facilitates creating a consistent view of application data during the course of a backup. It relies on coordination between VSS requestors, writers, and providers to quiesce or quiet a disk volume so that a backup can be successfully obtained without data corruption. At least, that s the technical definition, the one with all the big words. In plainer English, VSS is a Windows service that interacts with installed applications to tell them when a backup is taking place. It also reports back to the server when the backup is complete, instructing the application and the server to perform important post-backup tasks such as truncating logs and other cleanup activities. Why is VSS necessary? One word: Coordination, specifically coordination between those applications, their data, and the activities being completed by your backup solution. This coordination is required to avoid the situation told in the sidebar story below. Backing up Exchange without VSS, a Bad Day Here s an example of how a backup job could work if VSS wasn t around to coordinate activities. It s not a situation you want to experience. One day you attempt to backup your Microsoft Exchange server named \\exchange01. At 10:00PM, your backup solution begins its backup job for this server and all its data. Being an Exchange server, \\exchange01 is host to a set of files which contain its Exchange database. As the process begins, the backup server transfers files, including the Exchange database files, from \\exchange01 to the backup storage device. At 10:05PM, just a few minutes after the backup job starts, Bob the Outlook user checks his mail. In doing so, he sends and receives a set of mail for the day. That process of sending and receiving mail changes the data inside the Exchange database. This presents a problem, because the database has at this point been partially backed up. Its file on disk is only partially transferred to the storage device. The data contained within the database files on \\exchange01 is now slightly different than the data that was captured by the backup solution. These two views of the database are no longer consistent. That s the start of a bad day, one that will eventually result in a corrupted database upon restore. white paper / page 3

VSS comes into play any time a transactional-based application is installed to a Windows server that requires backups. Those applications can be Microsoft Exchange, SQL Server, Active Directory, Oracle, or any of a number of applications which require open access to files on disk. As you ll find out in a minute, virtual machines themselves are also transactional-based items in a datacenter, requiring their own quiescence for proper backup. Finally, and most importantly, VSS also comes into play with individual files on disk, ensuring that open files are correctly captured during the backup process. Recall that the primary job of VSS is to quiet an application or file system just prior to a backup. This quieting action (called quiescence) creates a point in time from which backups are then sourced. You ll often hear this point in time referred to as a snapshot, although snapshots in this sense are very different than the virtual machine snapshots used by your favorite hypervisor. Creating that single point in time eliminates the sidebar s problem. It ensures that a common starting point for backups is shared by the server and the backup application, guaranteeing that each maintains that consistent view of the data. Figure 1: VSS components. VSS relies on the coordination of three different components to maintain this consistent view. You can see those three components in Figure 1. At its upper-left are a set of VSS Writers. Each VSS-aware application installed onto a server also installs its own VSS Writer. The VSS Writer s job is to coordinate backup activities with the application, instructing the application to quiesce at the appropriate time. VSS Requestors can be, among other things, the application you use for backups. The VSS Requestor s job is to coordinate VSS activities with those of the backup application. The VSS Requestor is also the component which actually requests that a volume shadow copy be taken. Once requested, the VSS Writer will instruct the application to perform whatever actions are required to create that volume shadow copy. The third component is the VSS Provider. Its job is to create and manage the shadow copies themselves. The VSS Provider can be either the operating system in combination with its file system, or it can be a hardware provider on an external storage array. Note You can use the command vssadmin list writers to list the VSS Writers which have been installed to a Windows computer. white paper / page 4

The Role of VSS with Virtualization While VSS has long been used for backing up running applications, it has become even more critical when paired with virtual environments. This added criticality arrives through the desire to back up entire VMs at once. Backing up an entire VM at once requires backing up that VM s disk file, again either a.vmdk file for vsphere or a.vhd file for Hyper-V. By backing up that VM s disk file as a pointin-time backup, it becomes possible to trivially and quickly restore that VM to that previous point in time. Getting there, as you can imagine, requires the same sorts of quiescence that applications require. Since a VM s file system is as interactive and always changing as an application s database, some mechanism to quiet the VM s file system is needed if a host- or externally-based backup solution is to gather the disk file and maintain a consistent view. What you might not know is that that mechanism isn t always the same, depending on your backup solution and your selected hypervisor. That said, some architectures don t provide some functions that are needed for true restores. Let s compare the approaches of three different solution sets. The differences here will give you some idea about how very different the simple task of backups can be. Solution Set #1: Native Hyper-V Data Protection A fully-native Hyper-V environment automatically enjoys all the benefits of VSS components. This is the case because a Hyper-V environment runs completely atop Microsoft Windows. Virtual machines in a Hyper-V environment are Windows (ignoring here Hyper-V s Linux capabilities), with Windows Server also being the operating system at the virtual host. Native Hyper-V uses the onboard Windows Server Backup as its backup application. Virtual Machine Microsoft Exchange Etc... VSS Writer Hyper-V Writer VSS Writer Volume Shadow Copy Service VSS Requestor Windows Server Backup VSS Provider Operating System Storage Array Disk Volume Figure 2: VSS in a Native Hyper-V Environment. white paper / page 5

As you can see in Figure 2, these elements map directly to the original three VSS components described earlier. Windows installs a Hyper-V VSS Writer with the installation of Hyper-V. The backup application Windows Server Backup serves as the VSS Requestor, with the operating system and/or storage array handling the VSS Provider role. In this configuration, the instance of Windows Server Backup on the virtual host requests the host s Hyper-V Writer to quiet the file systems of any running virtual machines so they can be backed up with a consistent view. But that isn t all. You should also recognize that each virtual machine has its own VSS components as well as the host. Each VM also has its own installed applications that require quiescence. Quieting those applications requires coordination between the host s backup activities and those going on inside the virtual machine. That s why Figure 2 also shows a VSS Writer inside the virtual machine. As a VSS snapshot is requested by Windows Server Backup, the Hyper-V VSS Writer on the virtual host integrates with any registered VSS Writers in the virtual machines (such as Microsoft Exchange in the case of Figure 2) to ensure that the VM s applications are properly quieted as well. This integration is accomplished through the use of the Hyper-V Integration Components, which are a separate but required installation to any Hyper-V virtual machine. As is obvious, there s an extra level of coordination involved to maintain that consistent view of data across host, VM, and applications. Solution Set #2: Native vsphere The situation gets slightly more complicated when virtual machines are run atop different hypervisors, such as VMware s vsphere. With either ESX or ESXi, there is no Microsoft Windows instance that operates as the virtual host. This means that there is no VSS at that layer in the stack to handle quiescence and snapshotting prior to a backup. These activities then must be handled by one of a range of different options, such as VMware Consolidated Backup for older ESX versions or the newer and more-capable vstorage API. vsphere added full support for VSS in version 4.1 for all guests including Windows Server 2008 with earlier Windows operating systems being supported in previous versions of vsphere. This VSS support was introduced into vspherehosted Windows virtual machines through an update to the VMware Tools. Just like Hyper-V s Integration Components, the VMware Tools are a separate but required installation into any vsphere-hosted virtual machine. white paper / page 6

Virtual Machine Exchange Server Etc. VSS Writer Volume Shadow Copy Service VSS Requestor VMware Tools VSS Provider Operating System Storage Array Disk Volume ESX Host Disk Volume Figure 3: VSS in a Native vsphere Environment. As you can see in Figure 3, a similar quiescence process occurs on a vsphere virtual machine as is experienced with Hyper-V. Here, however, the VMware Tools serve as the VSS requestor, instructing registered VSS Writers to perform preand-post backup actions as whatever backup solution on the ESX host begins a backup of the virtual machine. Not shown in Figure 3, however, is the actual backup solution used by vsphere. Native to the VMware solution set are two products which can be used to backup vsphere virtual machines. VMware Consolidated Backup, which is a now-deprecated solution that is no longer available in vsphere 4.1, as well as its replacement called vsphere Data Recovery. Both are relatively simple solutions that provide a basic level of backup and restore support for virtual machines and their data. Solution Set #3: Agent-Assisted Data Protection While the architecture that makes up Solution Set #2 will indeed work for backing up and restoring virtual machines, it does come with a set of concerning limitations. Those limitations have to do with the very applications which you are intending to protect with your backup infrastructure in the first place. However, the specific limitations have more to do with the recovery process than the actual backup process. Two restore use cases should be immediately obvious when looking at the architecture outlined in Figure 3. Those use cases deal with the restoration of Active Directory Domain Controllers and Microsoft Exchange servers. Let s take a look at both. white paper / page 7

First, as you already know, a successful Microsoft Exchange backup requires VSS for proper quiescence. The VSS process, as you already know, ensures that the database view remains consistent throughout its entire process of being backed up. What you might not know is that a restore of Microsoft Exchange also requires a VSS-aware restore as well. As part of that restore, a number of very important steps are required: Step 1: Boot the restored Exchange Server virtual machine with its mailbox stores dismounted. Step 2: Instruct the Exchange VSS Writer to perform a restore from the VSS snapshot. Step 3: Mount the mailbox stores. The most important of these steps occurs with Step 1. A fully-featured Exchange data protection solution needs to boot a restored Exchange Server with its mailbox stores dismounted in order to protect them from data corruption. This is also necessary so that Step 2 can be completed successfully. This situation is very similar to the second use case, Active Directory Domain Controllers (ADDCs), whose restore also requires special handling to assure data is not corrupted in the process. In the case of ADDCs, a restored server must be powered back on in non-authoritative mode. Non-authoritative mode ensures that the data on the ADDC is not inappropriately replicated to other ADDCs in the domain. Not doing this could create a situation known as Update Sequence Number (USN) Rollback, where Active Directory data between ADDCs is no longer consistent. While Windows Server 2003 SP1 and later operating systems include safeguards such as ADDC isolation that can prevent this scenario, these safeguards may not protect against it in every situation. Thus, it is exceptionally important that recovered ADDCs are powered back on in Directory Services Restore Mode. Virtual Machine Exchange Server Etc. VSS Writer Volume Shadow Copy Service VSS Requestor VMware Tools On-Demand Assistive Agent VSS Provider Operating System Storage Array Disk Volume ESX Host Third-Party Backup Host Disk Volume Figure 4: VSS in an Agent-assisted vsphere Environment. white paper / page 8

One solution to prevent these and other problematic situations is through the use of an on-demand agent installed to virtual machines during the backup process (see Figure 4). This agent is considered on-demand because it resides on the VM only during backups, and is late removed after the backup is complete. The presence of this agent facilitates the coordination between the vsphere VSS Requestor and the third-party backup host. Note While not depicted here, the same on-demand assistive agent could be used in a Hyper-V environment as well, with similar results. More importantly, recognize that an on-demand agent is one that is automatically available within the backed up virtual machine. This means that the same agent will be available after the virtual machine is later restored. Presence of this agent enables an immediate integration between the onboard agent and the thirdparty backup host and solution. Being present on the host as it is restored allows the agent to control postrestore actions such as un-mounting Exchange databases and bringing ADDC servers online in non-authoritative mode. These actions ensure that restored servers and their data have a greater guarantee of successful restoration with a minimum of accidental data destruction or corruption. And that s important when servers are down, stress levels are high, and the potential for mistakes is heightened. Agent-Assist and Transaction Log Handling There s another important facet to agent assistance that benefits data protection. The agent-assisted approach also enables greater support for handling application transaction logs both during and after a backup. Recall that a VSS snapshot creates that point in time that enables the backup solution and the application to maintain a consistent view of data throughout the backup. Maintaining this view as data changes in the real database requires logging changes to a transaction log. One significant limitation of some backup solutions is in recognizing when the backup has completed successfully. Application transaction logs, such as those used by Microsoft Exchange among others, are an important source of data reconstruction in the case of a failed backup; thus, it is important that a backup solution instruct the server to flush those logs only after the backup has been deemed successful. Some implementations, such as the VMware Tools implementation noted in Solution Set #2 above, are not equipped with the necessary instrumentation to know when a backup has completed successfully. Thus, they may either not prune transaction logs after the backup, or they may do so even if the backup has not completed successfully. One benefit of using an assistive agent in the virtual machine backup process is that this agent can be better aware of the success of the backup. That agent can then retry the backup in the case of a failure, or prune the logs once the backup has been deemed successful. Both of these situations prevent the situation where needed transaction logs are inappropriately discarded a situation which can prevent the server from being restored in the case of a failure. white paper / page 9

More to VSS than Meets the Eye Virtualization can indeed complicate backups as it improves their usability. Once virtualized, you can absolutely enjoy the ability to restore whole servers just as easy as files, folders, or application objects. But you can only get there if you implement solutions that really work. As you ve learned here, Microsoft s VSS is one solution that does work if it is integrated with a well-designed backup solution. Veeam s Approach to VSS Veeam Backup & Replication leverages VSS functionality to ensure consistent backups of applications within vsphere virtual machines. Veeam provides a complete implementation of VSS support, enabling proper restore of VSS-aware applications (e.g. Active Directory, SQL Server, Exchange) from backups in Veeam. white paper / page 10

About the Author Greg Shields, Microsoft MVP and VMware vexpert, is an independent author, speaker, and IT consultant, as well as a Partner and Principal Technologist with Concentrated Technology. With 15 years in information technology, Greg has developed extensive experience in systems administration, engineering, and architecture specializing in Microsoft OS, remote application, systems management, and virtualization technologies. About Veeam Software Veeam Software, an Elite VMware Technology Alliance Partner, develops innovative software to manage VMware vsphere. Veeam vpower provides advanced Virtualization-Powered Data Protection and is the underlying technology in Veeam Backup & Replication, the #1 virtualization backup solution. Veeam nworks extends enterprise monitoring to VMware and includes the nworks Management Pack for VMware management in Microsoft System Center and the nworks Smart Plug-in for VMware management in HP Operations Manager. Veeam ONE provides a single solution to optimize the performance, configuration and utilization of VMware environments and includes: Veeam Monitor for easy-to-deploy VMware monitoring; Veeam Reporter for VMware capacity planning, change management, and reporting and chargeback; and Veeam Business View for VMware business service management and categorization. Learn more about Veeam Software by visiting www.veeam.com. About Concentrated Technology, LLC Concentrated Technology was founded by IT industry experts Don Jones and Greg Shields to provide concise, accurate education in business technology topics. The company writes to a range of audiences from the C-level to the trenches, with a focus on practical technology solutions for today s business challenges. For more information, visit www.concentratedtech.com. white paper / page 11

100% Reliability Best RTOs Best RPOs VMware Backup SureBackup TM InstantRestore TM SmartCDP TM vpower TM Virtualization-Powered Data Protection TM Patents 5 Pending! VMware vsphere 5 Patents Pending! NEW Veeam Backup & Replication vpower enables these game-changing capabilities in Veeam Backup & Replication v5: Instant VM Recovery restore an entire virtual machine IN MINUTES by running it directly from a backup file U-AIR (Universal Application-Item Recovery) recover individual objects from ANY application, on ANY OS SureBackup Recovery Verification automatically verify the recoverability of EVERY backup, of EVERY virtual machine, EVERY time To learn more, visit www.veeam.com/vpower