How to Avoid the Bankrupt Private Cloud Lessons learned from successful Virtual Data Center pioneers

Transcription

1 White Paper How to Avoid the Bankrupt Private Cloud Lessons learned from successful Virtual Data Center pioneers February 2011

2 Introduction What if you built a multi-million dollar virtualized private cloud, and all you had running on it was print servers? According to leading analysts, the journey to cloud computing has already begun for most enterprises. It starts with server virtualization initiatives and ends in a model where IT services are delivered like a utility, and resource provisioning is dynamic in a pay-as-you-go model. For most enterprise IT departments, however, this is uncharted territory. And without a road map developed by the pioneers, you may fall far short of your goals. Your new expensive infrastructure will still just be running print servers and not business-critical applications. So how do you prevent this from happening? Netuitive customers have built some of the world s largest deployments of virtualized infrastructure, with ambitious plans for cloud computing. Many lessons were learned as they struggled to refine their architectures with the goal of successfully migrating Tier 1 applications to their private clouds all while delivering even better, more proactive service management. How did they do it? In this paper, Netuitive shares some of their how to advice on: Common design mistakes to avoid Getting key Application Owners on board Delivering on performance promises by leveraging predictive analytics for IT Lesson 1: Why are we doing this again? When you embark on a potentially game-changing but difficult initiative, it s important to remember why you re doing it. One global banking customer which Netuitive is working with puts their cloud computing initiative under the strategic umbrella of Green IT because it is easy to explain to industry analysts, the investing community, and the public. We want to deliver the same services to our external and internal customers while consuming less energy, reducing our physical data center and carbon footprint, and saving money all at the same time. To achieve this end goal using a cloud computing model, IT management envisions the methodical development of a unit or container of computing power, based on virtualized resources available in private and eventually public clouds to minimize costs. Each unit would have an estimated operating cost as well as other performance parameters like power consumption and carbon footprint index. Once this computing unit is developed, the organization can meet with individual application owners to size up how many units their application needs on average and at peak loads to meet existing Service Level Agreements (SLAs). With a reliable range of monthly operational costs, this gives the application owners the flexibility to provision and configure how many units they want to fund for their application. This bank did a thorough analysis and mirrored what other leading enterprises told us about how they categorized the expected benefits of their cloud computing initiative: Page 2

3 Provisioning and resource reallocation new provisioning technologies for physical servers took the process from weeks to hours. In a virtual cloud, this can be reduced to minutes while adding the flexibility to dynamically size or provision virtual machines according to an infinite number of configurations to meet SLAs. Computing efficiency server consolidation means fewer idle resources, and flexible and dynamic allocation means that applications should consume less power, on average leading to better overall computing efficiency. Automated change control compared to similar technologies for physical environments, virtual cloud environments deliver server configuration changes in minutes vs. hours. The same goes for configuration rollback, which is important since many incidents are caused by problematic configuration changes. Disaster recovery the surest disaster recovery plans involve full hot backups of entire application infrastructures; however, these solutions are extremely expensive. Many companies believe that cloud based approaches to DR can deliver quick results at a much cheaper price tag. Peak load handling provisioning applications for peak loads is extremely costly and inefficient. To prevent performance problems due to capacity, cloud infrastructures handle planned and unplanned peak loads by dynamically provisioning additional resources from private or public clouds on a temporary basis. The lesson: it s not just all about server consolidation. Virtualization executives also note that, while server consolidation can save capital, it is often viewed as a tactical project within IT. The initiative becomes strategic when the vision is broadened to include running Tier 1, critical applications in your private cloud. This is where you need to remember the broad ranging benefits you will achieve for your enterprise to get you past the inevitable bumps along the way like dealing with virtualization sprawl. Lesson 2: The impact of virtual sprawl is worse than you think. The ironic thing about virtual machine sprawl is that it can happen right under your nose. The CTO of one large bank discovered the problem when his virtualization team was using Netuitive to optimize VM to host consolidation ratios. While looking for under-utilized physical assets, the team was surprised by a flood of alerts identifying idle or under-utilized virtual machines. The CTO states: Be careful, because while you may set out to eliminate under-utilized physical assets, you may end up with virtual server sprawl. And it can cost just as much or more to monitor a virtual server as a physical one. Also, we found that we were too conservative on VM density to ensure performance, and we ended up over-provisioning just as we did before. Once they eliminated these over-provisioned VMs (which were often running idle), the bank was able to confidently increase their consolidation ratios from 5 to 15 VMs per virtualization host for Tier 2 and Tier 3 applications. The resulting additional millions of dollars saved in hardware and storage funded expansion of the virtualization project into a full-blown cloud computing initiative. In addition to helping ensure maximum consolidation ratios, the VP at another bank reiterates why it s so important to prevent VM sprawl. Conventional wisdom is that managing a server running multiple VMs is similar to managing a single physical server. Not so: managing 20 VMs that share a server requires the same amount of work as 20 physical servers. 1 So eliminating VM sprawl not only minimizes hardware costs but significant operational costs as well. The lesson: fixing virtual sprawl is crucial. Your infrastructure will be more efficient, you will be able to optimize consolidation ratios, and you will also save on systems management labor. Page 3

4 Lesson 3: Avoid point tool proliferation. Do you have a bulldozer in your IT department? Because you may need one. Even as they start migrating Tier 2 and 3 applications to the virtualized cloud infrastructure, companies often face a new challenge. Before virtualization, IT Operations and Application Support teams were happy to live in their own IT silos. Fire drills ensued when performance problems arose, but once the problem was finally isolated to a specific area, they were quickly resolved. But the game has changed. As one enterprise IT executive states, Virtualization and cloud computing bulldozes the silo walls between operational teams currently monitoring various parts of the infrastructure and application environments operational issues that involve more than one team become far more common than issues that happen within a silo. 1 So what s the problem? IT Operations has specialized monitoring tools for servers, storage and network. Application Support teams also have their own tools for diagnostics in application, database, and web server tiers. Engineering teams will have their own tools for capacity planning of physical servers and network. Left to their own devices, each team will try to address cross-silo performance problems with more point tools with overlapping functionality (see diagram on right). This can prove to be a costly design error in your private cloud performance monitoring architecture. An isolated tool that only manages the capacity and performance of a virtualization environment will just create another IT silo. It will prevent you from mapping the footprint of your applications onto virtual machines when, months or years from now, you move to real-time provisioning in your private cloud. But it gets worse. There is also an operational penalty. The proliferation of point monitoring tools discourages, rather than encourages collaboration, making it virtually impossible to isolate performance problem root-causes. When you add the cost of training and replacement, you soon realize that the tool you choose now to scratch an itch will stay with you for many years. You have already invested in many tools over the years, so why look for another point tool? Instead, consider a solution that brings it all together, with out-of-the-box integrations with your existing tools, and an open-api to integrate with your proprietary data sources. This is the only way you are going to get a holistic, cross-platform view of your entire environment, which is critical to automatically isolate performance problems, or to plan capacity for applications in a private cloud. You will also need new IT analytics technologies (more on this later ) Page 4

5 One VP of Virtualization with one the world s largest deployments reveals, We realized this was an extremely complicated environment, with a lot of moving parts. More than any other time, the need for cross discipline insight is not just nice to have, but absolutely critical. Otherwise, if something breaks, you have eight guys and three vendors on the phone all pointing the finger at each other. The lesson: to understand and resolve performance issues in the cloud, you need cross-platform insight. And this won t come just from having point monitoring solutions in each silo and on each platform. The next lesson illustrates why this is so important. Lesson 4: Capacity does not equal performance. In virtual environments, application performance management requires equal insight into application behavior, infrastructure performance, and capacity. Before virtualization and cloud computing, the worlds of capacity and performance management were like two parallel lines that rarely intersected. Not unlike how seldom capacity planners actually talk to daily IT Operations staff. But in a virtualized private cloud, with all the excess capacity squeezed out of the infrastructure, capacity and performance become intrinsically intertwined. Having insight into both will be crucial to confidently migrating Tier 1 applications. You will need to figure out how to answer the following questions: 1. How can I dynamically allocate resources (capacity) without application visibility? 2. Do I have a mapping between my applications and each supporting VM? 3. How do I translate the Tier 1 application workload to resource allocations? 4. How am I going to make dynamic resource allocation decisions? 5. How often am I going to update my resource allocation policies? 6. How can I distinguish if a problem is performance or capacity related? 7. And how can I right-size capacity without over or under provisioning? Part of the solution is to have cross-platform visibility mentioned in the last Lesson. But there is a subtle trap that you also have to avoid. Capacity does not equal performance. In traditional capacity planning, resource bottlenecks are reflected in performance over an extended period of time and the problem is usually solved by adding capacity, in the form of storage, bandwidth, or computing power (servers). But in virtualized cloud environments, even when you find resource bottlenecks, you cannot fix performance issues simply by throwing more capacity at the problem. You need strong and deep insight into the behavior of your infrastructure and applications. In most cases, you may find that even though resource bottlenecks are a symptom, the root cause may be unrelated to capacity. The graph that follows on the next page illustrates an example. Performance Management Company, Page 5

6 Application owners commonly blame application performance issues on the lack of CPU resources and often request more virtual CPUs (vcpu) on the VM hosting their application. In reality, adding vcpus to a VM will make the problem worse if the real performance problem is caused by high CPU Ready Times due to physical CPU (pcpu) resource scheduling conflicts on the host. Without being able to identify the real issue you could end up with a virtual environment with a low VM to Host density, application performance issues that go unresolved, and a higher Total Cost of Ownership for your Virtual Data Center. The lesson: when managing performance and capacity in your private cloud, you need equal insight into their interrelated behavior. Because performance issues can happen no matter how much spare capacity you have. Lesson 5: Become strategic by winning over Tier 1 Application Owners. So you re confident in your cloud architecture. You ve got a solution that gives you cross-platform insight, including the complex relationship between performance and capacity. How do you get Tier 1 application owners on board? Tier 1 applications are the lifeblood of business. They are also usually the most complex and expensive applications to operate and maintain, with the greatest potential benefits to gain from cloud-based infrastructures. But championing this new infrastructure poses new challenges that are psychological as well technological. By nature, Tier 1 application owners are risk averse because the business is impacted if performance is degraded. As a result they prefer over-provisioning to ensure performance, whether you are talking about physical or virtual resources. Tier 1 app owners also typically want dedicated resources with minimal consolidation ratios (which goes against the grain of shared resource models). And while cost savings are important, don t expect them to be impressed by your server consolidation success: a few dollars saved on hardware is less important to them than NOT hearing from end-users about performance problems. So instead of just touting server consolidation, the best cloud computing champions will persuade application owners to migrate to the cloud infrastructure by stating, I can ensure the same or better performance for your application while saving you money on hardware and operational expenses at the same time. When asked how will you do this? they will reply: I can map your application workload to my virtual / cloud environment. I can right size provisioning to, based on your budget, meet your choice of Silver, Gold or Platinum SLAs. When your application has peak loads planned or unplanned I can dynamically reallocate or provision additional resources and you only pay for the usage as you go. I can analyze and forecast performance and easily add or reduce computing capacity. I can provide you with disaster recovery at a fraction of current cost, and I can get you back up just as quickly. The lesson: as you design your cloud architecture, make sure you have the technology to address the questions above. But keep in mind the psychology of your customer (Tier 1 application owners). To them it s almost always about performance, security and compliance first, and cost second. Page 6

7 Lesson 6: Dynamic resource allocation needs Behavior Learning Technology. To be more specific, you need predictive analytics powered by behavior learning technology. This is a new concept so we ll elaborate. The key question at this point is: have we covered all our bases to ensure performance for our Tier 1 application owners? One of the key capabilities in the architecture of private cloud infrastructures is to be able to dynamically right-size resources to proactively meet periods of peak demand for applications. This goes back to the roots of autonomic computing, marketed for years by the likes of IBM, etc. Virtualization technologies and private or hybrid cloud computing models have made it a closer reality due to how fast and dynamic changes can be made. A new class of software called Service Directors promises to orchestrate the dynamic resource allocation life cycle for cloud infrastructures. But how will these Service Directors know when and what type of allocation changes to make? Can you rely on manual methods similar to how VMware DRS (Dynamic Resource Scheduler) currently works? With VMware DRS, users define the rules for allocation of physical resources among virtual machines. The utility can be configured for manual or automatic control. 2 Bad resource allocation policies can put the performance of Tier 1 applications at risk in a dynamic environment with so many moving parts. And this type of manual approach is reactive rather than proactive. By the time your allocation rule kicks in, it may already be too late. As one industry analyst puts it the problem is that Many IT services are reaching a level of complexity where sophisticated mathematical algorithms and object models of the services are more precise and efficient than even your most talented engineers. 3 So what is the solution? As Gartner sees it, the answer may be in behavior learning technology. In an overview of the technology, Gartner states: Consider behavior learning technologies to gain a consolidated, holistic view of IT infrastructure and to gain an early warning of potential outages based on trends and patterns. 4 Behavior learning technology uses mathematics to provide the predictive analytics necessary to proactively manage performance in the cloud. The diagram below explains how all the ideas we ve discussed come together in a solution. First, existing monitoring solutions collect KPIs important in measuring performance and capacity. Then technology like Netuitive s Behavior Learning Engine aggregates and analyzes this data to understand and correlate behavior across IT silos. If performance problems are forecasted, the software makes a preemptive suggestion to a Service Director application which closes the control loop by triggering resource allocation changes after ensuring workflow and business policy compliance. Page 7

8 The diagram below is an example of closed-loop architecture for managing dynamic resource allocation in private cloud infrastructures. Behavior learning technology like Netuitive is uniquely suitable to play a central analytics role in the architecture. 1. Performance and capacity KPIs, and system attributes are collected into a centralized database (for example, Netuitive s Performance Management Database or PMDB.) 2. Integrations to service catalogs automatically group underlying components by service or application. 3. Key business KPIs, policies and Service Level Agreement information is integrated into the analytics engine. 4. Predictive analytics (with behavior learning technology) forecasts impending performance issues and sends Trusted Alarms to IT Operations consoles; if the problem is resource related, a Trusted Trigger is also sent to the Service Director application. 5. Management policies may require human operator confirmation of resource allocation decisions, which may be subject to various business constraints managed by the Service Director. 6. After ensuring that resource allocation requests conform to business policy and workflow requirements, the Service Director initiates action in resource controllers (such as VMware DRS or Microsoft SCVMM.) The final lesson: to automate accurate resource allocation in dynamic cloud infrastructures, you need predictive analytics powered by behavior learning technology. Page 8

9 Conclusion Enterprises typically share the same major goals when investigating virtualized cloud computing models, the most common of which are increased efficiencies through resource sharing, and improved agility to respond to fast-moving business needs. Cloud infrastructures have potential for improved change control through easier replication and rollback, as well as faster (and possibly cheaper) disaster recovery. But the ultimate sign of success for those leading cloud computing initiatives is to run their top applications on this new infrastructure. After overcoming early design challenges like eliminating virtual sprawl and monitoring point tool proliferation, you can win over Tier 1 application owners with a compelling and broad value proposition. But at this point, you may painfully discover that you are ill equipped to deal with the complexity of performance management in the new infrastructure. There are too many moving parts with complex dependencies, and tools that rely on manual, policy-based approaches to performance monitoring and dynamic resource allocation simply won t work. You risk letting application owners down on promised performance and flexibility. So in researching the problem, consider that Gartner believes that solutions like Netuitive move IT Operations to a more proactive state, where issues can be detected and addressed before affecting the business. 5 And for running Tier 1 applications in your virtualized private cloud, this may just be what the doctor ordered. About Netuitive Netuitive provides predictive analytics software for IT. Using its patented Behavior Learning Engine, Netuitive software replaces manual, rules-based methods for performance monitoring with automated statistical analysis that correlates and self-learns the operational behavior of IT systems and applications. It then forecasts issues before they impact performance and isolates root causes wherever a problem occurs. By integrating with nearly all of the industry's leading monitoring tools including VMware, HP, Microsoft, BMC, IBM, and CA, Netuitive is able to provide a holistic view across physical, virtual and cloud infrastructures enabling you to proactively manage performance and capacity. Netuitive takes all this telemetry out a massively virtualized infrastructure (the Cloud), learns what s normal, and fine tunes the monitoring accordingly to point out exceptions. This is highly compelling. - CTO, Top 10 Global Bank Page 9

10 References and Further Reading 1. Kotsovinos, Evangelos; Virtualization: Blessing or Curse? ; 22 November, 2010; ACM Queue White Paper; 2. TechTarget online definition; 3. O Donnell, Glenn; Dines, Rachel; Virtualization planning: 4 Systems Management Keys to Success ; May 20, 2010; 4. Williams, David; An Introduction to IT Operations Behavior Learning Tools ; 16 December, 2009; Gartner Research; ID Number G Williams, David; Behavior Learning Software Enables Proactive Management at one of World's Largest Telecom Companies ; 15 April 2009; Gartner Research; ID Number G Netuitive Technical Overview: Behavior Learning Technology and Predictive Analytics for IT ; December 2010; available at Page 10

11 Netuitive Predictive Analytics for IT Netuitive, Inc Sunrise Valley Drive Reston, VA USA Toll Free: Main: Netuitive Europe Ltd Chineham Business Park Crockford Lane Chineham, Basingstroke RG24 8AL United Kingdom Main: +44 (0) Web: Proactive. Predictive. Preventative. Netuitive provides predictive analytics software for IT. Netuitive replaces human guesswork with automated mathematics and analysis to forecast, identify and resolve IT performance issues before they impact quality of service. Hundreds of customers, including eight of the 10 largest banks, rely on Netuitive to proactively manage the performance and capacity of their IT infrastructures -- physical, virtual and cloud. Industry recognition includes the 2010 CODiE Award for "Best Systems Management Solution," the 2010 EMA Award for "Best Analytics," and the 2009 "Best of VMworld" Award for Virtualization Management. Contact Netuitive to learn more about our solutions for proactive IT performance management, or to schedule an on-site demonstration. Netuitive Inc., No material in this document can be reproduced without the expressed written consent of Netuitive, Inc. Page 11 Cloud Avoiding the Bankrupt Private