Capacity Planning 2.0 Application Awareness and Profiling Hyper9 Inc 9015 Mountain Ridge Drive Suite 220 Austin, TX 78759 sales@hyper9.com http://www.hyper9.com
Executive Summary Capacity Planning is the process of ensuring that the IT infrastructure can support agreed upon or target Service Levels in a cost effective and timely manner. Once thought of as nice to have, Capacity Planning has become a must have for virtual deployments where the sharing of underlying hardware resources (and the contention that inevitably arises) is built in by design. At its simplest level, Capacity Planning can be thought of as the task of balancing supply (CPU, Memory, Storage, I/O) with demand (applications/slas). The virtual environment brings a number of new twists on Capacity Planning however. Firstly, we have a different level of abstraction to deal with we primarily care about the supply side of capacity planning as pools or clusters of resources. With a certain degree of automation essentially built in to the virtualization platform, deploying a new application will frequently boil down to which cluster or pool should I deploy this application to? Secondly, whilst Capacity Planners have always been concerned about waste or how much oversupply or headroom to bake in waste takes on a whole new meaning in Virtual Environments. The ease which Virtual Machines (VMs) and applications can be created, cloned and moved has led to a proliferation or unplanned waste problem, often referred to as VM Sprawl. Depending on the environment, waste due to Sprawl could be considerable proper insight is key when adequately planning for capacity. Without proper insight into genuine consumption vs waste, we risk vastly over estimating the Virtual Infrastructure resources required. Finally, VMs don t exist in isolation. They run complex (possibly multi-tiered) applications that support different lines of business. Being able to profile how applications and business departments utilize (or potentially waste) the underlying physical resources is key. Even without the formal mantle of Chargeback, most companies need to understand how resources are being consumed by the business, so proper contributions can be calculated for budgetary planning and sensible use of resources can be encouraged on the demand side of the equation.
The Hyper9 Approach Hyper9 has built a powerful search based platform from which multiple Capacity Planning use cases can be satisfied. With search at its heart, multiple artifacts can be generated from a single unified solution such as reports, alerts and trends that give both a real time and historical perspective of the supply and demand side of Capacity Planning. Searching across Hyper9 s rich multi-faceted data set also provides the basis for different perspectives of the collected information - for example, a traditional infrastructure view of the underlying cluster capacity and consumption, but also additional views such as application aware usage profiles and mapping of Virtual Infrastructure usage (such as CPU, memory and storage consumption) to departments or other important business constructs. Virtual Infrastructure As covered earlier, managing capacity of a Virtual Infrastructure requires tracking of supply and demand at a more abstract level. For CPU and memory resources, this relates to the underlying clusters or pools of resources. For storage space (and generally I/O), this relates to the datastores or shared storage resources where the Virtual Machines are stored. In either case, Capacity Planning at the Infrastructure level typically distills to two fundamental questions: 1. When will I run out of capacity? 2. How many VMs (or applications) can I add? For both questions, it is desirable for a management solution to provide immediate answers, and refine those over time as more data becomes available. Hyper9 Performance Analyzer provides answers to the first question. A common question for example is to understand when clusters will run out of memory and CPU resources. Hyper9 Performance Analyzer - Cluster CPU Utilization with projected resource expiration
Performance Analyzer renders both a best fit trend line and a projection of 100% utilization over the timeframe of interest. A common follow up question that arises is who are the top resource consumers? Search provides an extremely powerful way for narrowing down the top consumers in an interactive fashion. For example, we can quickly find all of the ESX hosts that are part of the Corp HA cluster, understand their average CPU utilization, and plot them relative to the overall cluster utilization. Searching for top host consumers and overlaying against the parent cluster To answer the second over riding use case of How many more VMs can I add? we can leverage Hyper9 trends. Trends are simply a way of tracking the results of a search over time to give a historical context. Lets again take the cluster as the resource container we are working with we want to be able to track how many more VMs can be added to a cluster from the perspective of the overall memory (or CPU) resources available. This obviously depends on what kind of VMs we are adding (or the typical profile more on that later), but for now, lets extrapolate based on the average VM size across the cluster. The formula to do this is: SUM Hosts In Cluster {((100 Host Memory Utilization)/Host Memory Utilization) * #VMs per host} We have set the target utilization at 100%, but we can also set it to say 80% to allow for some headroom for example to account for the failure of a host in the cluster (assuming 5 equally powered hosts per cluster say) to ensure there is sufficient capacity even if we lose a host. It is also possible to set the target utilization greater than 100% to simulate adding capacity to the cluster as well.
Number of VMs that can fit in each cluster from a memory perspective What if we want to be proactively notified if the number of VMs that will fit is getting low? Hyper9 alerts like trends are simply based on an underlying search. We can take this capacity planning use case one step further and construct a proactive alert that will fire when the number of VMs that will fit on a cluster is below some threshold. In this example, we are setting up an alert that will e- mail (and/or send an SNMP trap) when the number of VMs that will fit on our production cluster from a memory perspective falls below 20. A note on VM Sprawl As mentioned earlier, genuine consumption and waste are really two sides of the same coin from the demand side of capacity planning. Consider the previous capacity calculations, what part of the predicted growth is coming from genuine consumption vs waste? Hyper9 contains a wealth of pre-built searches and reports to identify common VM Sprawl conditions. Some examples are: Sprawl Type Description Resources Wasted Stale VMs Zombie VMs VMs that have been powered off or suspended for extended periods VMs that are powered on but not actively being used for example, not being logged into in a dev/test environment. Disk Space CPU, memory, disk space, I/O Orphaned Files VMs (and other files) that have been removed Disk Space from inventory but not deleted from disk Old and Large Snapshots VMs with excessively large or old snapshots Disk Space Rogue VMs VMs that are powered on or off, that we do not expect to see in the environment CPU, memory, disk space, I/O
Remember, it should also be possible to tie both consumption and waste back to the applications and departments that are generating them. Application Awareness and Profiling VMs are not an island! They exist to serve applications and services back to the business. When planning for capacity, we are not just concerned about the average VM size within the virtual infrastructure. The pattern of resource consumption will vary greatly across workloads is this VM a web server, application server or database server for example? Is it a machine being used in dev/test/qa or in production? Hyper9 was designed from the outset to understand the complete stack of the virtual infrastructure and the workloads that run inside it. Again leveraging search, we can build up arbitrary profiles of usage of the virtual infrastructure and trend these over time. For example, we can search for all VMs running SQL server in our environment, or possibly only those being used in production, and leverage the Hyper9 Trend Preview feature to understand the average memory (or CPU, disk space etc ) consumed by a typical VM belonging to this profile. A memory profile of SQL Server VMs These application aware profiles can help to more accurately predict the resource needs of an application, whether we are moving an existing application from the physical world (and want to leverage the experience of prior conversions), or creating it from scratch in the virtual environment. Furthermore, most critical applications consist of multiple servers, broken into tiers such as presentation, application and database tiers, offering an overall application service back to the business (billing, inventory, ERP etc...). An application service could be thought of as a vapp [ref] in a purely virtual context the distinction is important because many companies are rolling out hybrid distributed application services that span both physical and virtual servers. Hyper9 provides a number of ways to define and track these abstract distributed applications and their corresponding resource requirements on the virtual infrastructure this will be covered in depth in a future paper.
Tying it back to the business We have talked about the need to tie consumption (and waste) back to the business departments / entities that are utilizing the virtual infrastructure. Companies are looking for transparency into the black box of virtualization, to justify existing investments, build the confidence to virtualize mission critical applications and ensure costs are equitably divided at budgeting time. Hyper9 provides open APIs and interfaces, both inbound and outbound, so that resource consumption by department / business unit can be reported on and shared with the business. Hyper9 Trend - Resource Utilization by department Once business mappings are imported, Hyper9 treats this as just another piece of data so that searches, reports, alerts and trends can be created specific to business departments, units, teams, projects and so on. These insights can be shared in a variety of ways such as embedding Hyper9 views into external portals so teams can get a handle on their consumption (and waste) patterns. Summary In this paper, we described a holistic approach to capacity planning that meets the needs of today s modern virtual deployments. Whilst understanding Capacity Planning from a virtual infrastructure perspective is necessary it is not sufficient we need to understand both consumption and waste as it pertains to the applications and business departments that utilize the underlying infrastructure. Only with this complete view can businesses appropriately plan and build up the experience necessary to virtualize the most demanding and mission critical applications.