Best Practices for Deploying System Center Virtual Machine Manager in Multiple Locations Author: Damian Flynn, MVP Published: 25 October, 2013 1
Even with the continuous evolution of Microsoft s Windows Server and System Center suites, some of the original architectural challenges we faced five years ago are still very relevant today and require special consideration. In this white paper, Savision and Microsoft MVP Damian Flynn outline the key concepts and latest developments of the Virtual Machine Manager (VMM) products along with some of the challenges of supporting multisite deployment. Key Concepts: VMM Host Groups and Libraries No matter if it is a small Proof of Concept implementation or a large live installation, any deployment scenario that requires supporting more than one physical location with virtualization hosts poses unique challenges. When Microsoft introduced the initial version of the Virtual Machine Manager (VMM) component of System Center, we were given the ability to organize our hypervisors into logical groups through a feature called host groups. Presented in the graphical user interface as a folder hierarchy with some special properties, the primary usage patterns included creating collections ranging from the very simple dedication of computer resources for roles such as development, staging, and production through to the concept of addresses of physical locations with host groups for Paris, Tokyo, and Seattle; each potentially subdivided into additional host groups representing a compute classification of, for example, gold, silver, and bronze. Although the host group feature of VMM has continued to mature with each new System Center release and additional functionality such as power management optimization has been added along the way, nothing of significance has really changed in this area in the latest release: System Center 2012 R2. Another artifact that has not really changed in VMM since its initial release is the library function. The library is a key configuration area within the product because if the library is not configured and managed properly then nothing really works in VMM: this is especially true given that we would not have objects in our library to deploy either virtual machines or services if not configured properly. While it is clear that the proper configuration of both host groups and the library is key in our successful definition and implementation of networks and policies, it is possible that the library might be even more critical and require more attention. Deployment Scenario: Multiple Physical Locations Consider the following deployment scenario: You are required to implement a new management infrastructure and VMM and associated hypervisors must be deployed to no less than two independent physical locations both of which you will be deploying virtual machines and services to, using your organization s standard operating system image. 2
At first glance, you would implement this in the same way that you have completed any number of your previous VMM installations. Create a File Share You would start by creating a file share on a Windows Server (Note: NAS devices are still not suitable for hosting libraries because the VMM agent still needs to be deployed to the hosting system) and then associate VMM with the share and allow it to perform any additional configuration. You would repeat these steps for any subsequent location and then establish a new file share, copy over the content from the primary location, and then enable the library for this location in VMM. You now have identical libraries in both physical locations but what happens when VMM indexes the objects in these file shares and presents them in the library? The contents of the shares (resources) are populated into the library and are ready for use in any of the VMM deployment scenarios and are repeated on each refresh cycle. At this point, you could consider restricting the new libraries to the hypervisors placed in specific physical locations through the use of host groups and then scoping the relevant share to each host group. This would ensure that as a user proceeded to deploy a new resource using an object from the library only objects in the local library would be available for selection. Create and Publish a Template In reality, we almost never create a new virtual machine on the fly. Instead, we use the template functionality to ensure that we deliver consistent and repeatable deployments. In our roles as administrators, we are responsible for the creation of the initial templates based on the new content hosted in our library and for presenting these back to our tenants ready for deployment. The template creation process is completely natural for us: building and publishing templates and selecting objects from the library to associate with our new template resources, for example a SysPreped Virtual Hard Disk (VHD). One small difference you might have observed as you applied this association was that the wizard actually displayed more than one copy of this object for you to select: this is due to the fact that we have the object physically in multiple indexed file shares. Issue: Deployments Failing After you publish the template, your tenants start deploying new solutions with success until you start to receive notifications that some deployments are failing. A quick investigation into the issue and we are able to determine the issue: users are able to deploy resources to one of the locations but all other locations are failing. This was caused by our decision to restrict the objects that were indexed in each physical location to only be available to hypervisors in the same location and then when we created the template we had to select one of the multiple choices of the same object. This means that the template is offered in all locations but the objects it requires for the deployment are actually restricted (as designed) so the failure is an expected one. As a quick fix, we could remove the restriction we placed but this will mean that the objects will be transferred over the WAN from the remote file share to the hypervisor, which defeats all of our work and will result in it taking hours to complete deployments. No user will be happy about that. Solution 1: Distributed File System Replication Services Our seemingly straightforward deployment scenario now has two problems that need to be addressed: 1. How are we going to ensure that once our environment is live we can keep all the file shares synchronized? It is imperative that resources added to any share will be also available to all other locations. 2. How are we going to resolve our template challenge? The initial theory is that we will require a version of the template for each physical location and need to reference the objects in the scoped library specific to the location and scope the templates per location. 3
If you adhere to the Microsoft support statements, this problem could turn out to be quite a challenge to resolve. Several years ago a note was posted to the SCVMM blog to indicate that using Distributed File Systems (DFS) for library servers would not be supported. The reason for this support restriction is that each library still requires its own VMM Agent that is responsible for indexing the content. It is easy to assume that this only really leaves us with solutions like RoboCopy to keep the libraries synchronized using file copies. However, there is a cleaner and more supportable approach that can be taken. We can configure our servers to use the Distributed File System Replication (DFSR) technology to keep folders synchronized at the block level. The result of this is that indexing will proceed as normal and if a file is being updated or added, it will be locked and, therefore, skipped by the VMM agent s indexer until its new refresh cycle. Using the DFSR services to keep the file share content synchronized would technically be external to VMM s agent activities and would not breach the support scenario. Solution 2: Equivalent Objects Solution 1 resolves the problem of keeping file shares synchronized but how do we address the template problem? Microsoft added a feature to the library to address this specific problem with the introduction of Virtual Machine Manager 2012. Although not perfect, this feature provides a seamless method for resolving the problem of multiple file shares and duplicate library objects. The feature is called equivalent objects and makes it possible for library owners to use a very simply tagging mechanism to group or associate objects in the combined library that are essentially the same. In our example deployment scenario, each copy of our VHD file would be tagged with a name and version to indicate that they are equivalent. Then we would create or update our templates to associate with the new tag rather than with the object in the scoped library and allow VMM to determine the correct object to use at deployment time. Although this is a very simple concept, it allows us to retain the single template approach but leverage the functionality of VMM to ensure that deployments will use objects from the library that are hosted on the file share local to the destination hypervisors. Another important point to consider when using the equivalent objects feature is that VMM does not double check the objects you are associating so you are responsible for ensuring that these are applied correctly. This has a major benefit if your locations are in different parts of the world because it is possible that virtual machines deployed using your updated template for the first location will reference a VHD using an English OS but when deployed to an alternate location this VHD might be configured with a Chinese OS. You are the one who decides what is equivalent. Single VMM Instance: Solutions for the Single Point of Failure Issue We have now resolved both of our library problems but we still have a few more challenges to consider before we can proceed with the implementation of a VMM installation for the management of multiple physical locations. The use of a single instance of VMM presents the problem of a single point of failure. Microsoft added support for failover clustering to SCVMM 2012 to address this specific problem. However, this implementation only offers an active/passive configuration. In the event of a failure, the passive node will assume the active role but this will terminate any active jobs during the transition. We will also have to scale the available resources on the associated VMM host when we scale the number of hypervisors we deploy in our facilities. 4
In our scenario, we also have to consider the challenges introduced when supporting multiple physical locations. If the interconnection between these facilities is lost, what should we expect from VMM? When there is no connectivity between the VMM agents in the now isolated location and our VMM server, we will be presented with a number of Hosts Not Responding issues and no view into the real health of the remote locations. We could place a VMM server in each location and configure the failover clustering, which now allows for at least 16 VMM servers to be positioned in our environment, but these all need to be configured to communicate with the VMM database on SQL. Things can get complicated quickly and even in this configuration an isolated facility that might be currently hosting the SQL databases would be enough to ensure that we are still badly exposed. We can continue to dig deeper and consider SQL replicas but VMM does not support these either. The answer to this problem might simply be that you should have redundant paths to your different physical locations: this could solve more than just VMM issues. Microsoft attempted to address the single point of failure issue with SCVMM 2012 SP1 by giving us the ability to implement a SQL 2012 Always On connection, which will enable us to float the same database to multiple locations. Essentially, this allows us to deploy a VMM server in each location using its active/passive cluster configuration. We are now capable of successfully resuming VMM services from a remote location but we continue to be exposed with the challenges of reaching the isolated remote location and, therefore, will continue to receive at least some Host Unreachable errors. However, the benefit of this configuration is that there is a minimum 50% chance that users will be able to continue deploying and managing resources: thanks in part to our host groups and libraries. The challenge, however, is if we lose access to the location that is hosting the majority of our production workload, we will not be able to monitor health or move active loads. Multiple VMM Instances: Revisiting the Higher TCO Issue An alternative approach is to introduce a new instance of VMM in the alternate location, which would be configured to manage the hypervisors and library in this location only. Taking this approach would clearly raise the Total Cost of Ownership (TCO) because not only would it require that we repeat all of our configuration settings on each instance but the complexity of ensuring everything remains synchronized increases as well. We are no longer going to be concerned with configuring our library resources as equivalent objects because now we are challenged with the ability to keep templates synchronized. We can do this with help from Orchestrator 2012 but not without some initial work configuring and validating the solution. Despite the apparent disadvantages, there are advantages for using multiple VMM instances: we retain the ability to manage all of our resources in a physical location and, with some initial work, we can customize each VMM instance to match the location s physical fabric and synchronization can be achieved with a little orchestration investment. With the current trend of virtualizing just about everything including the network; we can now implement Network Virtualization using Generic Routing Encapsulation (NVGRE) technologies in each location without the fear of serious repercussions when Host Unresponsive messages appear. In addition to this, we no longer need to fear the potential disastrous island effect in a multi-location cluster because of the possibility that more than one VMM server will remain active, especially when SQL Replication is used. As we start hosting virtual machines with multiple locations, we are architecting our implementations in a way that allows us to sustain highly available services, often by migrating workloads between physical locations or delivering resolutions for our disaster recovery scenarios. Although these implementations initially appear to be easier to achieve when using a single VMM instance to manage the respective locations, the launch of Windows 2012 Hyper-V by Microsoft introduced a new feature called Hyper-V replica that changed things. 5
The functionality of Hyper-V Replica has since been further enhanced by Windows 2012 R2: a near time replication of any virtual machine between hypervisors is now possible, even if they are situated in different physical locations (bandwidth permitting etc.). This provides a very efficient option for availability and recovery. To complement this solution, Microsoft also unveiled a hosted service referred to as Recovery Manager with the Windows 2012 R2 product release. This solution is a cloud service that connects to each of your independent VMM instances and displays the resources that are equivalent between instances. We can use this service to quickly ascertain the health and availability of VMM in each of our locations and orchestrate the failover of our resources to sustain business services with the click of a button. Considering these new architectural challenges and benefits, our TCO calculations need to be revisited. These calculations now begin to change drastically as we consider both RTO (Recover Time Objective) and RPO (Recover Point Objective) in our formula. 6
Service Provider Framework (SPF) Solution for Regular Services: Service Provider Foundation There is one final dilemma to consider before making a VMM design decision. Assuming the scenario in the previous section focused on virtualized line of business services, our next consideration must be to address both our regular tenants and their respective clouds. Do we have some tenants distributed to deploy resources to one location with others deploying to an alternate location without synchronizing these clouds or tenancies? This is an acceptable approach for the recovery manager scenario but is it suitable for regular services? With the launch of System Center 2012 SP1, delivered as part of the packaging of the Orchestrator component, a tiny yet extremely powerful new component named Service Provider Foundation (SPF) has been added to the suite of tools. Implemented as a Rest API front to the PowerShell interfaces of VMM and other components of System Center, this new endpoint offers a method of connecting with our Virtual Machine Manager instances. Initially designed to enable hosters to offer access to their cloud services over an HTTPS connection to their tenants with the companion product App- Controller, this component has also being adopted with the release of the Windows Azure Pack (WAP) for the integration of multiple VMM instances in a single unified portal. Windows Azure Pack App Controller REST API (Odata) PowerShell Web Service Authentiation Agregation PowerShell Scripts Orchestrator Runbooks By using the new WAP as the user interface for both our tenants and administrators, we can deliver a cohesive and rich user experience while masking the physical fabric implementations and distinct instances of VMM that might be deployed. Enhancing this service further with the integrated automation offered through the Service Management Automation technologies, also introduced with the Windows Server 2012 R2 release cycle, the configuration options available to support even more complex administrative requirements are a lot easier to achieve. Summary Virtual Machine Manager Instance A (Stamp 1) Virtual Machine Manager Instance B (Stamp 2) What might have originally been a very simple question especially based on numerous sample designs, screenshots, and guides presented in the multitude of published articles and books (including one by the author of this paper) related to implementing a VMM architecture, the correct design is no longer a clear-cut install and go approach. With a lot of complex decisions now requiring careful consideration, ranging from the host group through to the disaster recovery objectives, our choices must be carefully balanced with our business objectives and the ever increasing range of technological implementation options that we now have available to us in System Center 2012 R2. As we progress into the next wave of system management and evolve further with the cloud operating system, the architecture of our fabric management is paramount. With the many evolving challenges ahead, we must dedicate multiple hours to balance the design decisions while maintaining the fundamentals of managing resources simply and efficiently, leveraging the rich automation features to help us when possible. A good design is a design that will morph with the technology as the roadmaps to the future become clearer. It is now more important than ever to ensure that we know our history so that we can plan for the future. After all, clouds are not stationary. 7
About Damian Flynn Damian Flynn is the Infrastructure Architect on the corporate IT team of Lionbridge Technologies. His current focus is on Software Defined Networks (SDN) and Azure for Window server services, with perspective on orchestration of repeatable processes in Dev/Ops scenarios. He has a keen interest in cloud computing from both a business strategy and technical viewpoint and has presented sessions on building and managing private/hybrid clouds at a number of industry events. Damian authored the chapter on cloud computing in the book Microsoft Private Cloud Computing (Sybex) and in the Windows Server 2012 Hyper-V Installation and Configuration Guide (Sybex) and he is active in many Microsoft programs, is a Microsoft MVP, blogs at www.damianflynn.com, tweets at @damian_flynn, and has published a number of white papers and technical articles. About Lionbridge Lionbridge enables more than 800 world-leading brands to increase international market share, speed adoption of products and effectively engage their customers in local markets worldwide. Using our innovative cloud technology platforms and our global crowd of more than 100,000 professional cloud workers, we provide translation, online marketing, global content management and application testing solutions that ensure global brand consistency, local relevancy, and technical usability across all touch points of the customer lifecycle. Based in Waltham, Massachusetts, Lionbridge maintains solution centers in 26 countries. To learn more, visit http://www.lionbridge.com. About Savision Founded in 2006, Savision is headquartered in the Netherlands and is privately held. Savision s founders and executives bring years of enterprise systems and application management experience from large IT service companies. Savision s software is currently used by over 500 enterprises worldwide, including many Fortune 500 companies. Key in their choice were Savision s experience with, passion for, and knowledge of Microsoft System Center. At the Microsoft Management Summit 2013 in Las Vegas, Savision launched a free tuning and optimization recommendation solution for SCVMM, Cloud Advisor. You can download it here or go to For more information, please check out our blog links: http://www.damianflynn.com http://blog.savision.com 8