WHITE PAPER Proactive and Predictive Virtualization Management Optimizes Datacenter Availability and Utilization Sponsored by: Dell Mary Johnston Turner June 2014 IDC OPINION The management needs of today's increasingly virtualized and dynamic enterprise datacenters are putting significant stress on management processes and tools that were designed for an era when system configurations changed slowly and virtualization densities were low. IDC's research shows that the average number of virtual machines (VMs) per virtualized server continues to rise each year while the emergence of virtual storage and network technologies along with cloud architectures is combining to create complex datacenter environments where maintaining consistent application performance depends on IT teams having deep visibility across resource allocations and configurations of both virtual and physical resources. The experiences of five Dell Foglight for Virtualization customers highlight the operational challenges faced in rapidly growing virtualized datacenters. Whether the customer had a need to be more proactive in planning for future physical capacity or needed to do a better job of reclaiming underutilized virtual resources, Foglight for Virtualization provided a cost-effective resource for improving the overall stability and performance of heavily virtualized datacenters. Customers reported a range of operational improvements resulting from their use of Foglight for Virtualization, including: As much as 50% faster problem diagnosis and root cause analysis Host utilization increases of 10 15% Higher levels of IT staff productivity Improved service levels due to more proactive identification and elimination of configuration errors and bottlenecks More proactive capacity planning and resource allocation planning June 2014, IDC #249480
IN THIS WHITE PAPER This IDC white paper describes important virtualized datacenter monitoring and management challenges facing IT managers today. It discusses how IT operations teams can improve end-to-end physical and virtual infrastructure performance and availability while becoming more proactive in terms of reclaiming underutilized resources and anticipating the need for additional physical capacity. Interviews with five Dell customers illustrate how Dell Foglight for Virtualization is helping a wide range of organizations achieve measurable long-term improvements to service levels, resource utilization, and IT staff productivity. SITUATION OVERVIEW IDC's research shows that enterprise IT environments are evolving rapidly in terms of scale, scope, and operational complexity as virtual servers are becoming the default platforms for most production workloads across enterprise and service provider datacenters. At the same time, the increasing availability of big data analytics, social, and mobile applications; continuous delivery development strategies; and cloud architectures are creating an environment where capacity requirements can be unpredictable and even small errors in resource allocation and configuration can impair application performance and needlessly increase the cost of infrastructure and operations. Management Challenges Facing Today's Virtualized Datacenters According to IDC, in 2014, the typical virtualized server will support about 10 virtual machines. By 2017, that number will be in excess of 12 VMs per server. In many organizations, the majority of production workloads are already virtualized and almost all new workloads are deployed onto virtualized infrastructure. The success of virtualization has proven that sharing resources across workloads and user groups is a great way to reduce costs and improve availability and performance, yet it also creates a new set of important management challenges that are not addressed by traditional monitoring processes and tools. Specifically, highly virtualized environments require IT teams to become much more sophisticated about capacity planning, configuration optimization, root cause analysis, and dependency mapping across virtualized and physical infrastructure. As more and more organizations begin to embrace cloud and software-defined datacenter architectures, the need for visibility across virtual and physical server, storage, and network resources will become even more important. In cloud environments, business analysts and application development teams expect to be able to quickly provision virtualized resources that will deliver high levels of performance and availability. Automated provisioning systems can quickly deploy and activate resources, but the policies and algorithms driving those resource allocation choices need to recognize the performance characteristics of each workload and ensure that VMs sharing physical hardware can coexist peacefully when it comes to consuming memory, CPU, storage, and network resources. Poorly configured environments will frustrate end users and negatively impact business performance. 2014 IDC #249480 2
Among IT decision makers whose organizations have already begun planning for or implementing production private and hybrid cloud architectures, 40% expect to purchase new capacity planning and virtual server management tools during the next three years in order to effectively manage and optimize their growing cloud and virtualized datacenter environments (see Figure 1). FIGURE 1 Top 5 New Management Software Purchases Needed Because of Cloud Over the Next Three Years Capacity planning and analytics Advanced virtual server management tools Application performance management and monitoring Workflow orchestration and task automation Public cloud service-level monitoring tools 40.0 40.0 34.0 33.3 32.0 0 10 20 30 40 50 (% of respondents) n = 150 Note: Multiple selections were permitted. Source: IDC's IT Management QuickPoll, 2013 IDC's research shows that the desire to proactively improve virtualization configuration, monitoring, and capacity planning is consistently a top priority for IT operations leaders. These decision makers are increasingly recognizing the fact that fragmented, disconnected configuration and monitoring tools make it difficult to correlate performance and configuration data and can slow problem resolution and root cause determination. As the use of virtualization and cloud scales up to support more and more mission-critical workloads, the potential business costs and risks associated with inefficient infrastructure management are very high. As a result, predictive, scalable, and flexible virtualization management tools that enable IT teams to become more proactive in the way that they manage their physical and virtual datacenter infrastructure will be vital to the long-term success of cloud and software-defined datacenter strategies. 2014 IDC #249480 3
Guidelines for Evaluating Virtualization Management Solutions In evaluating and implementing management solutions for large-scale virtualized datacenters, IT decision makers need to carefully consider how their requirements will evolve over time. IDC's research shows that more and more organizations are being called upon to support heterogeneous multi-hypervisor environments and new workload requirements such as VDI. OpenStack and related cloud system software solutions are enabling workload migration and portability between diverse on-premise and cloud-based infrastructure. As summarized in Figure 2, effective virtualization management solutions in the cloud and softwaredefined datacenter arenas will need to be able to provide proactive and predictive monitoring and analytics across heterogeneous physical and virtual environments. FIGURE 2 Virtual Datacenter Management Software Evaluation Checklist Heterogeneity Multiple hypervisors Virtual and physical server, storage, and network Scale Thousands of VMs Hundreds of clusters Multiple datacenters Visibility In-depth in-context drilldowns Out-of-the-box and custom reports and dashboards Predictive alerting and notifications Analytics Capacity planning Root cause and dependency assessment Identify and reclaim underutilized resources Source: IDC, 2014 2014 IDC #249480 4
Customers evaluating virtualization management tools should look for solutions that: Scale cost effectively to support thousands of VMs across multiple heterogeneous public and private datacenters. Provide integrated visibility and contextual drilldown across virtual and physical server, storage, and network resources to rapidly identify and resolve performance problems. Provide deep-dive insight into configuration errors and topological dependencies. Proactively alert on emerging performance impacts and capacity limitations. Streamline the process for identifying and reclaiming underutilized resources. Provide dynamic, predictive capacity planning based on historical trend analysis. Offer out-of-the-box reports that provide immediate value. Include the ability to create custom reports and dashboards as needed. Customers will want to evaluate alternatives and solutions in their own environments to assess how well various solutions meet their needs. Dell Foglight for Virtualization Delivers Proactive and Predictive Virtualization Monitoring, Reporting, and Analytics Dell Foglight for Virtualization is a scalable solution for monitoring physical and virtual server, storage, and network infrastructure. Foglight for Virtualization uses a federated architecture to be able to scale to monitor tens of thousands of VMs across hundreds of clusters and multiple datacenters. Although Foglight for Virtualization initially focused on providing deep visibility into VMware and Hyper- V environments, recent releases have added support for Citrix XenDesktop, VMware vcloud Director, virtual network switches, and a range of storage environments. Foglight for Virtualization uses a common graphical interface to provide detailed analysis of physical to virtual mapping, VM workload performance, configuration and resource utilization profiles, utilization analysis and predictive capacity planning, including scenario modeling. Foglight for Virtualization correlates virtual and physical configuration and performance data to quickly identify dozens of errors that can directly impact critical application performance. Graphic analysis and contextual deep-dive drilldowns allow IT staff to quickly detect, diagnose, and remediate problems, often before end users notice that there is an issue. Dashboards such as the one shown in Figure 3 provide a quick summary view of the entire datacenter environment and allow for deep inspection and click-through drilldowns. Additional analytic tools enable IT teams to identify and reclaim underutilized resources, migrate workloads as needed to optimize resource utilization, and proactively anticipate new capacity requirements. 2014 IDC #249480 5
FIGURE 3 Dell Foglight for Virtualization Dashboard Source: Dell, 2014 2014 IDC #249480 6
Customer Experiences: Foglight for Virtualization Helps Improve Service Levels and Increase Resource Utilization IDC interviewed five Foglight for Virtualization customers to learn more about how Foglight for Virtualization can improve virtualized datacenter performance and resource utilization. These customers represent a range of organizations in terms of scale and complexity. The benefits experienced as a result of Foglight for Virtualization's proactive and predictive monitoring capabilities include: As much as 50% faster problem diagnosis and root cause analysis Host utilization increases of 10 15% Higher levels of IT staff productivity Improved service levels due to more proactive identification and elimination of configuration errors and bottlenecks More proactive capacity planning and resource allocation planning More cost-effective launch and delivery of revenue-generating cloud services More detail on the customers' unique experiences with Foglight for Virtualization is provided in the sections that follow. Regional Power Distributor Cuts Problem Resolution Time in Half with Foglight for Virtualization A not-for-profit regional power distributor that runs 350 VMs across 30+ ESX hosts enables 38 independent electric utilities to collaborate and share the costs of back-office operations and electric distribution across the southern United States. The organization's virtualized datacenter supports a wide range of production applications, including electric grid management, billing, customer service portals, Web sites, and internal applications. With just two IT staff, the team needs to operate as efficiently as possible. Foglight for Virtualization was deployed after a competitive analysis and proof-of-concept evaluation that considered a total of five different vendors and product offerings. In situations where other products found 8 10 issues upon deployment, Foglight for Virtualization found more than 60. The team felt that Foglight for Virtualization was able to dig much deeper into virtual and physical infrastructure dependencies and performance than the other products it evaluated. Foglight for Virtualization was seen as providing significant value right out of the box using predefined reports. In addition, the team was able to efficiently create custom dashboards, allowing it to gain insight into a number of specific issues and conditions with just a few hours of staff effort. Foglight for Virtualization allows the organization to effectively address issues raised by the rapid growth in workload processing and data storage requirements. Capacity planning capabilities are becoming increasingly important. Foglight for Virtualization has had a direct role in improving the utilization of the organization's infrastructure by improving the organization's ability to better size initial deployments, reclaim underutilized assets, and proactively anticipate future capacity requirements. The result has been a 10 15% improvement in the utilization of the organization's infrastructure. 2014 IDC #249480 7
Simultaneous with improving resource utilization, the organization was able to substantially reduce the time required to move from problem diagnosis to resolution. Often, the time to resolve a problem was reduced by as much as 50% because of the ability of Foglight for Virtualization to automatically correlate alerts and prioritize notifications. This was a capability that the customer felt several other products lacked. The Web-based interface and graphical reports were well integrated and allowed for quick, deep drilldowns linked to the real-time operational context. The product's visibility and ease of use all contributed to significantly improved service levels and infrastructure stability. Global Financial Services Firm Sees Rapid Time to Value and Improved Service Availability Using Foglight for Virtualization A global financial services firm with two major datacenters running over 1,600 VMs relies on Foglight for Virtualization to provide daily insight into infrastructure performance and resource utilization. The datacenters support a wide range of workloads running on Windows and Linux platforms. The administrator team relies on Foglight for Virtualization to proactively identify capacity and performance issues and maintain expected service levels. End user specific reports and dashboards are provided to business groups that want insight into the performance of their enabling infrastructure. The company selected Foglight for Virtualization after completing a competitive assessment of several alternative solutions. The company selected Foglight for Virtualization because of the depth of insight it provides across both physical host infrastructure and virtualized systems and its clear root cause analytics. The evaluation team also noted that the capacity reporting and scenario modeling capabilities were important for their organization. The ability to monitor the company's emerging cloud infrastructure was another critical selection factor. As part of its virtualization initiatives, the firm has aggressively migrated and consolidated workloads to improve resource utilization without impacting performance and SLAs. Integrated monitoring alarms are credited with reducing VM misconfigurations and helping to identify partition and resource allocation errors. The product's ability to rightsize CPU and memory resources has been beneficial for eliminating bottlenecks that impact the performance of critical applications. Within 30 minutes of initial deployment, the product was delivering value. Proactive alerting and root cause analytics help the IT team prevent service-affecting incidents, which eliminates much of the organization's unplanned downtime. This, in turn, has noticeably reduced the number of incidents and tickets that need to be addressed by the group. Foglight for Virtualization is expected to contribute important automated problem detection and remediation capabilities across the firm's emerging automated, self-service private cloud environment, with the goal of minimizing the amount of time that system administrators need to spend on day-to-day configuration and support. Global Dev/Ops Lab Increases Infrastructure Utilization Global organizations often have to deploy software applications and middleware across widely heterogeneous infrastructure configurations and network topologies. In some parts of the world, the network infrastructure may be unstable or infrastructure environments may be anchored by legacy products. Application performance can be seriously degraded by a host of issues, including malware, network latency, and inconsistent patching and upgrades. 2014 IDC #249480 8
Against this backdrop, a major global organization has built an isolated, virtualized test lab capable of simulating a broad range of infrastructure environments and network topologies. The lab's mission is to test both commercial software and custom-developed code against a wide range of contingencies and architectures. With over 500 VMs, the datacenter's configuration is constantly changing based on the product being evaluated and the area of the world in which it is likely to be deployed. Coping with a tight budget that requires the lab to maximize utilization of its assets, the team has come to rely on Foglight for Virtualization as a critical tool for ensuring that storage and system resources are provisioned as efficiently as possible. Unused VMs are proactively reclaimed and underutilized resources are reconfigured as needed. The team credits Foglight for Virtualization with helping it to improve overall infrastructure utilization by 10 15%. National Healthcare Services Provider Improves Capacity Planning and Performance With more than 700 VMs located across two primary datacenters and a backup site, a national healthcare services provider implemented Foglight for Virtualization to improve capacity planning, resource configuration, and end-to-end application performance. Over the past four years, Foglight for Virtualization has become the datacenter operations team's go-to solution for providing a comprehensive view of configurations and utilization across physical and virtual systems and storage. Foglight for Virtualization's ability to efficiently collect, store, and analyze 18 months or more of detailed history has been particularly important since the organization's capacity planning and procurement cycles require significant lead time for gaining approval for new capital investments. The predictive analytics provided by Foglight for Virtualization's review of historical trends is critical to ensuring that the organization can stay ahead of the ever-increasing demands for server and storage resources. The customer finds that the Foglight for Virtualization database architecture is well suited for cost effectively handling its 200GB database. Prior to implementing Foglight for Virtualization, the organization found it was difficult to anticipate capacity crunches, which frequently resulted in last-minute scrambles to acquire needed hardware. Over time, as the team learned how to take full advantage of the product's root cause analytics and custom reporting features, it has gained important insights into how to more efficiently allocate resources between clusters to deliver higher levels of application performance. In one case, the organization was able to take the CPU Ready Time for its Citrix cluster down from more than 10% to the 1 2% range. This resulted in significant improvements to end-user performance and greatly reduced the number of complaints and trouble tickets that had to be managed. The speed with which Foglight for Virtualization can pinpoint detailed configuration and capacity issues has been particularly important to reduce the number of service-affecting incidents. Foglight for Virtualization also allows the datacenter team to continually identify and reclaim underutilized resources. Recently, a review of the entire environment helped free up several blades' worth of CPU and memory resources across 600 servers. 2014 IDC #249480 9
U.S. IT Solutions Provider Relies on Foglight for Virtualization to Monitor Private Cloud Infrastructure A full-service United States based IT solutions provider offers its customers a wide range of hardware and software support services, professional services, and managed infrastructure services, including private and hybrid cloud infrastructure monitoring and management. Foglight for Virtualization has quickly become one of the mission-critical private cloud monitoring tools of the organization because of the depth of insight and the level of end-to-end traceability it can provide across physical systems and storage infrastructure as well as hypervisors and virtual machines. The IT solutions provider embeds monitoring information provided by Foglight for Virtualization in each customer's private cloud monitoring dashboard. This provides customers with cost-effective real-time visibility and predictive analytics about the performance of virtual machines running in their private cloud environments. The monitoring and reporting insights enabled by Foglight for Virtualization have become key elements in the firm's private cloud customer demonstrations. The analytics and reports are frequently credited with helping close sales. On a day-to-day basis, the ability for customer administrators to view the health and status of the private cloud infrastructure supporting their VMs has helped reduce service calls and finger pointing by streamlining the process for determining whether performance problems are due to issues in customer VMs or in the service provider's private cloud infrastructure. From a cost/benefit point of view, the IT solutions firm reports that Foglight for Virtualization was 50% less costly than alternatives considered, which has allowed the organization to launch its private cloud service at a very competitive price point. Internally, the use of Foglight for Virtualization has improved the productivity of the IT solution provider's own IT staff. The IT solution provider reports that Foglight for Virtualization took less than a week to implement. Overall, the IT solutions provider believes this type of infrastructure-aware virtualization monitoring and reporting is an important differentiator that has allowed the firm to win many recent private cloud contracts. In fact, the company believes it would have been almost impossible for it to successfully offer its private cloud service without this level of visibility, reporting, and analytics. FUTURE OUTLOOK Over the past 10 15 years, thanks in large part to virtualization, and the more recent emergence of cloud architectures, datacenter capacity planning and configuration management needs have shifted from being hardware centric to software controlled. IDC expects that as the technologies mature, enterprise IT leaders will solidly embrace virtualized, software-defined datacenter architectures that enable workload portability and optimization without requiring hard-coded dependencies on specific vendor hardware, middleware, and service delivery platforms. These software-defined architectures will increasingly integrate multiple physical datacenters into a common software-defined logical environment and will require management solutions that can scale up rapidly while supporting heterogeneous infrastructure. 2014 IDC #249480 10
Cloud and software-defined datacenters will create dramatic shifts in IT management strategies. With regard to virtualization management, tools and processes will need to evolve to provide IT decision makers with a seamless view across not only on-premise resources but also public cloud infrastructure. Policies driving automated configuration and provisioning and workload migration will need to be tested and codified, and IT teams will need to aggressively optimize resources and proactively anticipate capacity requirements. IT operations analytics related to virtualization management will become more robust to rapidly capture, reduce, and store large volumes of data including performance metrics and event logs and will need to be extended to include capabilities to generate and execute complex queries on multiple data sources and include analysis capabilities to help users visualize and understand operational conditions and relationships. Predictive analytics capabilities including "machine learning" of baseline conditions, monitoring ongoing operational behavior, and recognition of potential anomalies or potential failures in time to take preventive actions will be expected. CHALLENGES/OPPORTUNITIES Dell's customers report that Foglight for Virtualization is a powerful, scalable tool that has historically been challenging to master. They applaud the product's ability to look across physical and virtual infrastructure and to provide deep drilldowns and custom reports as well as predictive analysis. However, some customers note that it can be time consuming to create custom reports and that initial system setup and configuration can be challenging. Dell recognizes this issue and is working to improve the custom reporting experience by adding more and more predefined dashboards with each new release. Most customers recommend that new customers work closely with Dell to tune the application. They also advise new customers to invest in training so they can make full use of the product's functionality. Foglight for Virtualization has demonstrated its ability to provide customers with a comprehensive view of dependencies across virtual and physical infrastructure and to improve resource utilization, capacity planning, and workload performance. To address the needs of tomorrow's datacenters, Dell will need to continue to evolve the product to address the management requirements of next-generation cloud and software-defined datacenters while continuing to simplify its customers' day-to-day experience. Customers that are considering Foglight for Virtualization should examine their organizations' mid- to long-term datacenter architecture plans and how Dell's product road map aligns with their emerging needs. CONCLUSION IDC expects that continued adoption of cloud and software-defined datacenter architectures will drive up virtual server densities and promote the use of more heterogeneous platforms. Effective management of these datacenters will require IT teams to become more proactive in assigning resources, maintaining configurations, and understanding physical and virtual resource interdependencies. Proactive monitoring and capacity planning will be needed to guarantee service levels as demand for resources intensifies. Virtualization management solutions built to handle the scale and complexity of these environments will deliver significant value in terms of cost savings, performance improvement, and overall IT staff productivity increases. 2014 IDC #249480 11
About IDC International Data Corporation (IDC) is the premier global provider of market intelligence, advisory services, and events for the information technology, telecommunications and consumer technology markets. IDC helps IT professionals, business executives, and the investment community make factbased decisions on technology purchases and business strategy. More than 1,100 IDC analysts provide global, regional, and local expertise on technology and industry opportunities and trends in over 110 countries worldwide. For 50 years, IDC has provided strategic insights to help our clients achieve their key business objectives. IDC is a subsidiary of IDG, the world's leading technology media, research, and events company. Global Headquarters 5 Speen Street Framingham, MA 01701 USA 508.872.8200 Twitter: @IDC idc-insights-community.com www.idc.com Copyright Notice External Publication of IDC Information and Data Any IDC information that is to be used in advertising, press releases, or promotional materials requires prior written approval from the appropriate IDC Vice President or Country Manager. A draft of the proposed document should accompany any such request. IDC reserves the right to deny approval of external usage for any reason. Copyright 2014 IDC. Reproduction without written permission is completely forbidden.