tm The Shortcut Guide To Data Center Energy Efficiency David Chernicoff
The road to virtualization now has a high speed lane. 7 1 2 6 3 Principles of InfraStruXure High Density-Ready Architecture... 1 Rack enclosures that are HD-Ready 2 Metered PDUs at the rack level 3 Temperature monitoring in the racks 4 Centralized monitoring software (not shown) 5 Operations software with predictive capacity management (not shown) 6 Efficient InRow cooling technology 7 UPS power that is flexible and scalable Virtualization is here to stay. And it s no wonder it saves space and energy while letting you maximize your IT resources. But smaller footprints can come at a cost. Virtualized servers, even at 50% capacity, require special attention to cooling, no matter their size or their location. 1. Heat Server consolidation creates higher densities and higher heat per rack, risking downtime and failure. 2. Inefficiency Perimeter cooling can t reach heat deep in the racks. And overcooling is expensive and ineffective. 3. Power Events Virtual loads move constantly, making it hard to predict available power and cooling, risking damage to your network. The right-sized way to virtualize. With the new HD-Ready InfraStruXure architecture, you can take on high-density by cooling the virtualized high-density row, controlling power at the rack level, and managing the system with advanced software and simulation. Though virtualizing saves energy, true effi ciency also depends on the relative effi ciencies of power, cooling, and servers. Right-sizing one and not the others (See Figure 1) leaves effi ciency savings on the table. To right-size, depend on the effi cient, modular HD-Ready InfraStruXure and neutralize heat at the source. Equipment will be safer and more effi cient running closer to 100% capacity. Don t agonize, virtualize. What are you waiting for? With HD-Ready InfraStruXure architecture anyone can virtualize anytime, anywhere. Just drop it in and go. Why do leading companies prefer InfraStruXure 6 to 1 over traditional data center designs? Find out at www.xcompatible.com You can deploy high-density racks right now... Deploy InfraStruXure as the foundation of your entire data center or server room, or overlay into an existing large data center. SCHEMATIC LEGEND: CRAC UNITS STANDARD DENSITY RACKS CENTRALIZED UPS INFRASTRUXURE HD-READY ZONES Figure 1 Efficiency and Virtualization Your servers are effi cient, but is your power and cooling? Pre-Server Virtualization Correct Server Utilization Correct-sized Power Correct-sized Cooling Post-Server Virtualization Correct Server Utilization Correct-sized Power Correct-sized Cooling The following have been tested and work best with InfraStruXure Solutions. Go to www.xcompatible.com to learn more. COOLING USAGE/CAPACITY SERVERS POWER USAGE/CAPACITY Big gains could be made with both server and power and cooling. 49 %* Efficiency Grossly oversized power and cooling cancels out potential gains made by virtualizing. 39 %* Efficiency Server Virtualization with Power and Cooling Right-sized power and cooling tip the balance back in your favor. Correct Server Utilization 62 %* Correct-sized Power Correct-sized Cooling Efficiency ALLIANCE PARTNER Virtualization: Optimized Power and Cooling to Maximize Benefits Download a FREE copy of APC White Paper #118: Virtualization: Optimized Power and Cooling to Maximize Benefits Visit www.apc.com/promo Key Code i205w Call 888-289-APCC x6032 Fax 401-788-2797 2009 American Power Conversion Corporation. All trademarks are owned by Schneider Electric Industries S.A.S., American Power Conversion Corporation, or their affiliated companies. e-mail: esupport@apc.com 132 Fairgrounds Road, West Kingston, RI 02892 USA 998-1792a Full details are available online.
Introduction to Realtime Publishers by Don Jones, Series Editor Introduction For several years now, Realtime has produced dozens and dozens of high quality books that just happen to be delivered in electronic format at no cost to you, the reader. We ve made this unique publishing model work through the generous support and cooperation of our sponsors, who agree to bear each book s production expenses for the benefit of our readers. Although we ve always offered our publications to you for free, don t think for a moment that quality is anything less than our top priority. My job is to make sure that our books are as good as and in most cases better than any printed book that would cost you $40 or more. Our electronic publishing model offers several advantages over printed books: You receive chapters literally as fast as our authors produce them (hence the realtime aspect of our model), and we can update chapters to reflect the latest changes in technology. I want to point out that our books are by no means paid advertisements or white papers. We re an independent publishing company, and an important aspect of my job is to make sure that our authors are free to voice their expertise and opinions without reservation or restriction. We maintain complete editorial control of our publications, and I m proud that we ve produced so many quality books over the past years. I want to extend an invitation to visit us at http://nexus.realtimepublishers.com, especially if you ve received this publication from a friend or colleague. We have a wide variety of additional books on a range of topics, and you re sure to find something that s of interest to you and it won t cost you a thing. We hope you ll continue to come to Realtime for your educational needs far into the future. Until then, enjoy. Don Jones i
Table of Contents Introduction to Realtime Publishers... i Chapter 1: Imp act of Virtualization on Data Center Power & Cooling... 1 Introduction... 1 Data Center Infrastructure Efficiency... 2 Understanding the Concepts Behind PUE and DCiE... 2 Using the PUE and DCiE Model in Your Environment... 3 Understanding the Benefits of Virtualization... 3 The Benefits of Virtualized Environments and Efficient Energy D esign Implementation... 4 Impact of Virtualizati on on Power Consumption and Efficiency... 5 Addressing the Challenges... 7 The Differences Between Physical an d Virtualized Environments... 8 Sizing Your Virtualized Environment... 8 Matching Power an d Cooling Delivery to the Virtualized Model... 10 Assuring Availability... 11 Designing to Address Potential Availability Issues... 12 Top Virtualization Issues Affecting Availability... 14 Summary... 15 Chapter 2: Best Practices for Data Center Design... 16 Key Issues in Building an Energy- Efficient Data Center... 16 Properly Sizin g Your Data Center... 17 Oversizing... 17 Issues & Drawbacks...... 18 Avoiding Oversizing... 19 Reducing Energy Consumption... 20 Data Center Physical Infrastructure Equipment... 23 Energy Consumption Reduction w ith Physical Infrastructure Equipment... 24 Energy-Efficient Systems Design... 24 ii
Table of Contents Understanding the Imp act of Humidity Control in Your Data Center... 26 Effects of Humidity... 27 Measuring Humidity... 27 Controlling the Environment... 28 Minimizing Internal IT Factors... 28 Understanding Cooling and Capacity Manag ement... 28 Designing to Support Growth and Cha nge... 29 Meeting Supply a nd Demand Goals... 29 Managing Capacity... 31 Monitoring... 32 Forecastin g... 32 Modeling... 32 Using the Row & Rack Cooling Methodologies... 32 Benefit s Over Traditional Room-Cooling Techniques... 33 Row... 33 Rack... 34 Conclusion... 34 Chapter 3: Free, or Nearly Free, T hings to Do in an Operating Data Center... 35 The Trad itional Cooling Model... 35 Auditing... 36 Find Out What s Rea lly Going On... 36 Checking Capacity... 37 Checking Hardw are... 38 Systems Testing... 40 Understanding the Impact of Temperature on Servers... 41 Finding the Sweet Spot for Minimizi ng Energy Consumption... 41 How Managing Airflow Reduces Costs... 41 iii
Table of Contents Understanding the Use of Blankin g Panels... 41 Minimal Cost / Maximum Gain... 42 Proper Pla cement of Vents and Tiles... 43 Containment... 44 Keeping the Cold Air, or Hot Air, Where You Need It... 44 Evaluatin g Cooling Architectures... 45 Conclusion... 49 Chapter 4: Strategies for Upgrading a Production Data Center... 50 Best Practices for Data Center Cooling and Power M igration... 50 Making Your Data Cen ter More Energy Efficient... 50 Defining the Projec t... 50 Defining the Goals... 51 The Costs of No t Upgrading... 51 Layout & Cooling... 51 Efficient Layout of the Data Center... 52 Implementing the High-Dens ity Zone... 54 The Benefits of Containment... 55 Hot-Aisle vs. Cold-A isle Containment... 56 Util izing the Pod Model... 57 Power... 58 Upgrading Existing Systems... 58 Introducing Scalable, Rack-Based Solutions... 59 Evaluating the Costs... 60 Management... 60 Capacity Management... 60 Hardware Management and Monitoring Tools... 62 Power and Cooling Management as a Component of IT Management Systems... 64 iv
Table of Contents Server Upgrades, Consolidation, and Virtualization... 64 How Implementing Newer Server Technologies Can Improve Your Overall Energy Efficiency... 64 Hardwar e... 65 Softwa re... 65 Conclusion... 65 v
Copyright Statement Copyright Statement 2009 Realtime Publishers. All rights reserved. This site contains materials that have been created, developed, or commissioned by, and published with the permission of, Realtime Publishers (the Materials ) and this site and any such Materials are protected by international copyright and trademark laws. THE MATERIALS ARE PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT. The Materials are subject to change without notice and do not represent a commitment on the part of Realtime Publishers or its web site sponsors. In no event shall Realtime Publishers or its web site sponsors be held liable for technical or editorial errors or omissions contained in the Materials, including without limitation, for any direct, indirect, incidental, special, exemplary or consequential damages whatsoever resulting from the use of any information contained in the Materials. The Materials (including but not limited to the text, images, audio, and/or video) may not be copied, reproduced, republished, uploaded, posted, transmitted, or distributed in any way, in whole or in part, except that one copy may be downloaded for your personal, noncommercial use on a single computer. In connection with such use, you may not modify or obscure any copyright or other proprietary notice. The Materials may contain trademarks, services marks and logos that are the property of third parties. You are not permitted to use these trademarks, services marks or logos without prior written consent of such third parties. Realtime Publishers and the Realtime Publishers logo are registered in the US Patent & Trademark Office. All other product or service names are the property of their respective owners. If you have any questions about these terms, or if you would like information about licensing materials from Realtime Publishers, please contact us via e-mail at info@realtimepublishers.com. vi
Chapter 1 [Editor's Note: This ebook was downloaded from Realtime Nexus The Digital Library for IT Professionals. All leading technology ebooks and guides from Realtime Publishers can be found at http://nexus.realtimepublishers.com.] Chapter 1: Impact of Virtualization on Data Center Power & Cooling Virtualization is one of the base technologies driving IT planning. With the clear benefits that virtualization technologies bring to the data center, the impact that virtualized environments have on the overall power consumption and cooling capacity currently present in most data centers is often overlooked. Optimizing the physical infrastructure that provides power and cooling to the data center, to work in concert with the deployment of virtualized environments, is critical to allowing the business to reap full benefit from virtualization in the data center. Introduction As virtualization becomes more widespread in your data center, you will likely find that the structure of your data center is changing. Although some servers may be removed, many servers will simply be repurposed, while additional, more powerful servers are installed. These new servers will have increased power and cooling needs when compared with lower powered servers that previously occupied the same space and will impact the efficiency of your data center power delivery and cooling. This is especially true if you are in a growing organization where the level of systems density maintained in the data center was fairly low. You may find that additional energy is now required by your data center, both as power and cooling. In an environment already heavily dependent on dense technologies such as blade servers, the trend towards virtualization may result in the opposite case: a surfeit of power and cooling capability. In both cases, the result is similar a poorly optimized energy delivery model that results in a decrease in efficiency. The second situation is the more common one, as early technology adopters are likely to have already built a high density data center environment. Properly optimizing power and cooling will often result in a significant reduction in the overall power consumption of the data center, with its attendant cost savings. Properly designed, the power and cooling systems deployed in your data center will provide the support needed to deploy your current virtualized environment in the most cost efficient fashion while allowing those same systems to scale to meet the needs of a growing business. Potentially, this may require changes to your power and cooling infrastructure. As one of the hallmarks of virtualization is the ability to quickly provision and scale your computing environment, it is only natural to deploy power and cooling systems that can meet the challenge presented by the non static data center environment. 1
Chapter 1 Data Center Infrastructure Efficiency The basic, primary requirement for implementing an energy efficient data center is to be able to measure the relative efficiency of the current, or planned, power and cooling systems deployment within that data center. As with any technology, the trick becomes finding an effective measuring technology that is vendor independent and applicable to a wide variety of conditions. To that end, The Green Grid (www.thegreengrid.org), an industry alliance made up of more than 100 member companies with products or technologies focused on the data center, has been launched to develop a standard set of measurements of data center efficiency to be used by the industry. To date, their first two benchmarks, Power Usage Efficiency (PUE) and Data Center infrastructure Efficiency (DCiE), have received relatively wide acceptance. These benchmarks can be used to build a model of your data center s energy efficiency that you can employ when determining which products and technologies, from a wide range of vendors, will be best utilized in your environment. Understanding the Concepts Behind PUE and DCiE The basic concept between PUE and DCiE is the ability to establish a metric for evaluating the efficiency of your data center. Making use of these metrics assists you in determining whether you can meet your needs with your existing systems, as well as whether those needs are being met efficiently. PUE and DCiE are complementary measurements. PUE is defined as: Total Facility Power/IT Equipment Power The reciprocal measurement, DCiE, is defined as: 1/PUE = IT Equipment Power/Total Facility Power x 100% Total Facility Power is the power measured at the utility meter. This can be an issue in a mixed use facility due to the need to establish the amount of power used solely by the data center facilities within the building without including power requirements of other organizations housed within. IT Equipment Power is the power delivered to devices within the data center used specifically for the management, processing, storage, or routing of data (servers, storage devices, network components, and so on). This measurement would most likely be taken as a value of the output of all power distribution units within the data center. 2
Chapter 1 Using the PUE and DCiE Model in Your Environment Clearly there is no high end limit to a PUE score, but a PUE value of 1 would indicate the utilization of 100% of the delivered power by IT equipment only. For comparison purposes, the ability to determine how efficiently power is being utilized within your data center gives you a useful metric for evaluating the impact of changes to the efficiency of your data center energy utilization. Reports on data center surveys provided by the Green Grid show common values in the 3.0 range, with the conclusion that, with proper redesign, those PUE values can be cut by as much as one third to one half. The DCiE measurement gives you the reciprocal information to that provided by the PUE measurement. For example, if the data center was seeing power delivery from the grid equaling 5000 watts and the measurement of the total IT power consumption was 2000 watts, the DCiE metric would indicate 40% efficiency. Both PUE and DCiE metrics are in use by vendors offering power efficiency services to business users; although the metrics present the same information, in different forms, you will likely find that PUE is in more common use. Determining your own PUE calculations is a valuable task before beginning any data center related projects. Understanding the Benefits of Virtualization One of the reasons that data centers have proliferated is due to the explosive increase in the number of servers used by large enterprises and the need to consolidate their location in order to improve management and support. But the cost of supporting individual physical servers, in terms of both IT support and facilities costs, has also continued to grow. This increase reduces the effectiveness of the management efficiencies that have been gained by consolidating the physical location of your enterprise servers. A large percentage of the servers in a data center spend the bulk of their time in an idle state. Although some applications place continual loads on the server, more often than not, the average data center server is simply sitting there, drawing power as much as 80% of the time, according to some analyst estimates. Even at idle, these servers are drawing approximately 30% of the power that they would draw at peak load, while they sit and wait for work. These costs have to be absorbed somewhere in the enterprise, and optimizations that can reduce these costs are important to IT budgets. An IDC report in 2007, Enterprise Class Virtualization 2.0, estimated that 50 cents of every dollar spent on servers was spent to provide power and cooling. The same report estimated that by 2010, that amount will have risen to 70 cents. With the potential lifetime costs of providing power and cooling to individual physical servers being 70% of the total expense, server consolidation using virtualization becomes the logical progression for upgrading any server running suitable applications. 3
Chapter 1 The changes in the data center wrought by the move to virtualized environments can be very obvious; a smaller number of more powerful physical servers replacing a large number of less powerful standalone servers. Thus, the overall power and cooling requirements of the data center are likely to go down, resulting in reduced expenditures for those needs, at least initially. This results in a degraded PUE value, as it increases due to the reduction in IT load while the physical infrastructure remains the same, unable to adjust for the drop in demand. However, there are a number of issues that will crop up, some of them unexpected, that provide an opportunity to more carefully optimize the energy efficiency of your data center and may impact how you manage your data center facility: The behavior of your servers will change Virtualized environments allow for the dynamic implementation of servers and applications. This means sudden changes on the load on the physical server hosting multiple virtual servers and the need to deliver efficient power and cooling to those physical servers in order to support the server state changes without over sizing the availability of resources provided to your servers. Server support priorities will change When a single server was hosting and supporting a single, low priority application, it received the level of support suitable for a server that wouldn t have a severe or immediate impact on the business process if it was offline temporarily. When you concatenate multiple servers with those types of applications to a single physical server, that host server suddenly acquires a much more mission critical role, as its health can now affect a much larger pool of users. Thus, the smaller number of physical servers that are the hosts for those low priority servers need to be considered high value servers and get the same level of power and cooling support offered to your mission critical servers. Matching the power and cooling model to the leaner IT model By providing an optimized power and cooling environment to match the newly streamlined server environment IT can see a much greater overall cost reduction than would be achieved by simply reducing the number of physical servers. The Benefits of Virtualized Environments and Efficient Energy Design Implementation Adding virtualization to your data center will not only allow you to enjoy the benefits that virtualization brings to your computing environment but also allow IT to reap the benefits of the knowledge that already exists, in house and in the industry, on the technologies and techniques for building, deploying, and managing power and cooling for high density rack mounted servers. Because the physical server consolidation that virtualization brings reduces the number of servers necessary to deploy in your data center, the most obvious benefit is the reduction in power required to maintain the same level of service to the business consumer of IT resources. But additional savings may be possible due to the reduced needs of the supporting power and cooling infrastructure. Maximizing the potential savings should be part of the overall migration plan for the data center virtualization implementation. 4
Chapter 1 The obvious way in which this would work would be the need for fewer power delivery systems and the reduction in cooling capacity due to the lower number of servers. However, it s not quite that simple; the layout or design of your data center may preclude major changes to the delivery of cooling services or fixed losses may make your reduction in power and cooling needs effectively less than might be expected. Impact of Virtualization on Power Consumption and Efficiency Seen from the outside, the implementation of a virtualized computing environment will always deliver better efficiency when seen in terms of computing power and optimization. What isn t as clear is that the conversion to virtual systems will always reduce the energy efficiency of the physical infrastructure of the data center due to the fixed losses inherent in the existing power and cooling systems. Not taking this opportunity to match the power and cooling delivery to the new conditions in the data center is effectively leaving money on the table; the savings potential in matching the power and cooling needs to the new reduced load is considerable. We ve already discussed the importance of understanding the PUE of your data center; grasping the total power aspects of the PUE will help you comprehend the impact of fixed losses on your data and cooling infrastructure. All power consumed in the data center, by both IT workloads and support workloads, is your total power consumption. The power consumed by those non IT workloads such as the cooling systems, power system inefficiencies, and other data center physical infrastructure systems is the loss we are addressing here. Figure 1.1: Power consumed by the physical infrastructure defines losses. 5
Chapter 1 Losses that are experienced are of two types; fixed and proportional. Some equipment will draw the same level of power regardless of load; others will vary based on workload, giving you the two types of infrastructure loss. Fixed loss This loss remains the same regardless of the workload. The system or device consumes a fixed amount of power regardless of the task at hand. When there is a high load present, as might have been the case before the data center virtualization was accomplished, these fixed losses are a small percentage of the overall power requirement. As the power need of these devices doesn t change as the overall load is reduced, these fixed losses, and their attendant cost, become a noticeable percentage of the overall power consumption of the data center. Proportional loss The amount of loss for a device is directly proportional to the workload on the device: increase the workload 50%, the loss increases 50%; decrease the workload 25%, the loss decreases 25%. The bottom line here is that fixed losses will limit the savings that can be achieved relative to power and cooling. The fixed costs must be reduced in order to see the maximum savings from a data center consolidation project. There are three options for reducing fixed costs: Elimination Elimination means removing some of the fixed loss devices from the infrastructure. An update of your infrastructure support systems might make certain devices redundant or unnecessary; removing them from the data center takes the fixed losses of these devices out of the equation. Reduction Reduction of fixed costs can be accomplished by switching to more efficient power delivery or backup systems. Looking for devices with reduced parasitic drain or more efficient power conversion to update existing systems will result in reduced fixed losses. Conversion Conversion means taking some of the fixed loss devices, such as cooling fans and pumping systems, and replacing the existing fixed speed fans and pumps with variable speed units, changing this portion of the overall power consumption from a fixed loss to a variable loss. By reducing the power consumed by fixed losses within the data center, IT is able to maximize the cost savings benefits that the virtualization process enables. Remember that your losses are the aggregate of all the devices within the data center that are not supporting the IT load directly. Reducing and converting necessary supporting loads from fixed to proportional loads will improve the overall efficiency of the data center. 6
Chapter 1 Figure 1.2: With the typical fixed loss averaging 35% pre consolidation, reducing or removing fixed consumption devices and optimizing for the virtualized environment results in significant savings. Remember that the move to virtualization and consolidation will always reduce the power consumption of the data center. Optimizing the energy efficiency of the data center requires changing the infrastructure, as necessary, to match the new load demands. Without making these changes, IT will simply be wasting power and reducing the overall data center efficiency. Making these changes is rarely a simple matter of removing devices that are no longer necessary. Reconfiguring your power delivery and the design and layout of your data center as well as adjusting the cooling infrastructure are all just pieces of the puzzle. It is almost always safe to say that the solutions that were optimized for a standalone server environment will be oversized and inefficient for your new virtualized data center. Addressing the Challenges Many institutions that are building data centers using server consolidation and virtualization look at the project and realize that power and cooling that is sufficient for their existing environment should be good enough for their new environment, with its smaller number of physical servers doing the same work. The problem with this attitude is that good enough rarely is, and if IT is going to the trouble of building a new data infrastructure, it is reasonable to build an optimized power and cooling infrastructure to maximize the return on investment (ROI) on the investment in virtualization. 7
Chapter 1 The proper way to approach these issues is to take the whole system approach; power and cooling are simply another component of the complete data center model and need to be fully integrated as part of the system to be deployed. This approach requires laying out the data center IT hardware in such a way that the dynamic issues of the environment can be addressed, implementing a scalable power and cooling infrastructure, and deploying the software tools to mange power and cooling as an integral part of the entire data center system. The Differences Between Physical and Virtualized Environments The primary difference between physical and virtualized environments is characterized by the flexibility of the virtualized model versus the traditional static model. In the traditional data center with individual servers, the role of those servers is defined when the server is installed. An estimate can be made of its planned workload, and appropriate power and cooling can be configured for the physical location of the server. That process has served IT well, but the dynamic nature of virtualized environments has rendered it obsolete. The virtualized environment can be changed at a moment s notice; servers can be deployed and provisioned as new virtual machines in minutes, running compute and powerintensive applications immediately. The reverse is equally true, and as the load on the virtual hosts changes to meet the demands of IT, the load on the power and cooling infrastructure needs to change with it to provide the most optimized and efficient environment. The very nature of a data center using virtualization to its maximum business benefit is one of change. Because of this, the power and cooling systems designed for the static model will be incredibly inefficient in providing services in the virtualized space. For example, the thermal characteristic of the data center server room was determined by the simple expedient of taking temperature readings in the operating data center. Once the hot spots were identified, the appropriate measures were taken to ensure that adequate cooling was available to the locations that needed it. Sizing Your Virtualized Environment With the virtual model, it s no longer that simple. Because the workload on the servers can shift dramatically, without any physical server changes, the thermal profile of the room will change with the dynamic software changes to the physical host servers. The hot spots in the server room will change as the workloads change, and the cooling system used to maintain the optimum temperature profile must change accordingly. Thus, cooling needs to be provided only when and where it is needed, ideally by a solution that is able to sense the changes in environment and react appropriately. In addition to this dynamic response to load changes, the solution needs to have a short air path between cooling and workload. 8
Chapter 1 This need has brought forth the row based cooling model. In this design, cooling units are located within the rows of server racks and are capable of responding to temperature changes they detect. In this way, cooling is delivered when and where it is needed, and the amount of cooling provided is matched to the needs of the environment. Placing the computer room air conditioners or air handlers (CRAC) directly in the rows that host the physical servers meets this need. Figure 1.3: Row based cooling systems are in the right place to most efficiently deal with hot spots. This short air path cooling model brings many advantages and benefits to data center cooling efficiencies and is especially effective when adding virtual server hosts into existing data centers. Cross Reference We will cover this technology in greater depth in Chapter 2. The implementation of a row based cooling methodology provides an additional technique when deploying high density servers into existing low density data centers: the highdensity zone. With this technique, an area of the existing data is designated as the highdensity zone where a set of racks containing these high density servers is effectively isolated from the rest of the data center. This isolation may be achieved by using a curtain system to define its limits; it can also be done strictly with row based cooling that is self contained in the high density zone and is effectively a neutral presence with no negative thermal impact on the rest of the data center. There is even a potential for positive impact, as additional cooling could be provided to devices outside the cooled rows. The high density zone can be cooled and managed independently of the rest of the data center. 9
Chapter 1 Matching Power and Cooling Delivery to the Virtualized Model One of the most significant issues related to power and cooling when bringing virtualization to existing data centers is the under loading of the power and cooling systems. Most data centers don t currently have a scalable power and cooling system. The primary selling point of such systems has been the ability to start small and grow with the data center. By doing so, a new data center didn t have to over invest in their power and cooling infrastructure, purchasing equipment that might never be used. Virtualization technology moving into existing data centers has reversed the problem. The goal in this situation is to be able to scale down to meet the new needs supporting the physical servers while retaining the ability to increase capacity to meet the growth needs of the organization. Matching need to demand provides the most efficient solution to the problem and avoids potential problems that can result from an over sized solution. Both issues focus on right sizing the power and cooling requirements to meet the existing needs of the organization while retaining the flexibility to scale to whatever the data center demands become. Failure to act on this issue takes us back to the problem of inefficient use of the power and cooling infrastructure due to the power that is used regardless of the load the fixed loss we discussed earlier. Figure 1.4: Comparing the efficiency of scalable power and cooling delivery to a legacy static over sized system. When looking at the two charts in Figure 1.4, you see that the overall capacity remains the same between the scalable and over sized examples. However, with the scalable model shown on the right, you see that the unavoidable inefficiencies in the data center that cause the losses are effectively proportional to the amount of resources being used. The converse is evident when the capacity exists at the same maximum level found in the over sized solution that results from leaving an existing data center power and cooling implementation in place after the move to virtualization. 10
Chapter 1 Remember that we are looking at the data center as a single entity, with multiple interlocking systems. In addition, the power and cooling infrastructure is, itself, made up of many different underlying systems, all of which will be affected by being severely oversized, if that turns out to be the case after the move to virtual machines. Everything from cooling systems to backup generators can be negatively affected by being underutilized. Systems designed for providing power and cooling to the data center are designed for a specific operational range, and if the load drops sufficiently due to virtualization, it is possible to fall below the lowest recommended operational range on your cooling equipment which can have results ranging from voided warranties to shortened compressor life due to repeated short cycling. It is important not to get caught up in the obvious when considering the effect of virtualization on your data center. You will see a reduction in overall electric costs. You won t see the increase in wasted energy because the percentage of power that is being used to drive the IT equipment significantly drops as the fixed losses take up more of the actual power and cooling expense. Utilizing a scalable architecture gives you some measure of future proofing your environment. If higher density virtualization is where the data center goes, you will be in a position to effectively deliver power and cooling services to a smaller, more efficient data center. If the data center needs to scale out the physical server presence, your scalable solution is right there with it. Flexibility and agility is critical to the delivery of costeffective IT services. Assuring Availability The arena of ensuring that power and cooling services are available falls into two categories: hardware and software. Hardware, such as backup generators, redundant cooling systems, and power protection systems, are not within the purview of this discussion. For this guide, we will be looking at the concepts of capacity management tools, which combine hardware instrumentation and software management to deliver the capability to provide dynamic management and capacity planning. Optimizing power and cooling delivery to the data center goes a long way to improving the delivery of IT services as it prevents potential problems with servers that are often the result of power and cooling problems. 11
Chapter 1 Designing to Address Potential Availability Issues There are three central challenges in capacity management that allow IT to address potential availability issues. Addressing these three issues gives IT the ability to quickly respond to changes in the data center and is crucial to getting optimal benefit in the virtualized data center. The need to know, and be able to manage, what is going on in real time is the driving force behind getting control of these three tasks: Power Cooling Physical Space An effective capacity management solution will have the ability to monitor power and cooling at the row, rack, and server level and allow for the remote management of the power and cooling supplies in order to ensure the most efficient operation of the data center. If any of the three challenges aren t met, the solution can t be delivered. Avoiding this situation is important to an effective deployment, but the prevention of stranded capacity is also an aspect of this problem. Stranded capacity is what you have when one or two of the three required resources are available but you have no way to deliver the remaining resources necessary for a successful deployment. These stranded resources are incredibly inefficient and can range from having over sized cooling at a specific location, to having racks with power and space available but insufficient cooling capability to allow the servers to be used, to having power and cooling available but not having the space to utilize it. This issue is more common than you might think and one of the goals of capacity management is to prevent this situation from happening. Anyone who has worked with network or IT systems management tools will find the concepts behind power and cooling capacity management tools familiar. What if analysis will play a large role in the selection, configuration, and administration of capacity management tools, especially with the use of virtual servers. By making use of data that is continually collected, your capacity management tool will be able to give IT the proper locations for adding physical servers in terms of space, power, and cooling requirements as well as define the impact of potential changes to the data center infrastructure. A quality tool will also allow for input from IT regarding issues such as the need to group servers with other specific servers to meet business or networking needs and whether the servers will require additional support for features such as redundancy. In addition, a tool will help IT understand the difference in impact on the data center environment between virtual server hosts and standalone servers. Planning isn t the only role of capacity management tools, however. The ongoing management of the delivery of power and cooling is central to the implementation of an efficient data center. Being able to respond to changes in the cooling requirements is only the start, albeit an important one. 12
Chapter 1 By building heuristic data from the management of the existing environment, the management tool will be able to predict where potential problems can occur and suggest or perform corrective actions to prevent problems from impacting operations. These technologies are old hat in the network and systems management world. Not taking this opportunity to apply them to the data center infrastructure means that a chance to improve the overall efficiency of the data center has been wasted. And don t think of it only as a way to make the power and cooling delivery more effective; consider also the issue of reducing the need for IT manpower by reducing the amount of time IT personnel are physically trying to resolve the problems we are addressing here in the data center. We ve talked a lot about efficiency of the data center; capacity management is a critical tool for attaining maximum efficiency (on any measurement, such as PUE or DCiE). As we discussed earlier, the benefits of measuring the efficiency of the data center will appear on the bottom line as an improvement in the ROI of your data center projects. Figure 1.5: Effective capacity management software is able to concatenate a large amount of data to provide an optimized data center power and cooling environment. Much, if not all, of the data that can be collected by the capacity management system is already being made available by devices in place in the data center. Actions are being taken manually, by IT personnel, to address problems as they occur, and the value of the information being provided is wasted due to the lack of a comprehensive proactive management system. With the implementation of virtual servers in the data center, the adoption of an effective capacity management and planning tool is critical to maximizing the success of the new virtual infrastructure. We have already discussed some of the changes that virtualization will make in your data center power and cooling needs, so let s take a look at how the implementation and deployment of virtualization within the data center will affect the demands on the capacity management tools. 13
Chapter 1 Top Virtualization Issues Affecting Availability The very nature of the virtualized data center has an impact on the availability and delivery of power and cooling services. To fully utilize a data center with a combination of standalone servers and virtual machine hosts, capacity management tools need to be able to adapt to the changing conditions, especially those introduced by the virtualization deployment. The following list will aid the IT reader in identifying the issues that will most be encountered when optimizing cooling and power solutions for the virtualized environment.: Changing density Compared with the state of the data center prior to the implementation of virtualization, there will often be an increase in density in one set of racks, where the servers acting as physical hosts to the virtual servers will be placed. Conversely, the overall density of the data center will go down due to the reduction in the number of physical servers. Thus, the thermal conditions of the overall data center will change and the power and cooling requirements will need to be reevaluated to meet the changing environment. Changing loads The workload that was once spread out over a large number of servers has now been confined to a much smaller number. As server loads migrate from standalone servers to virtual machines, the dynamics of the data center will continually change. Increased rate of change With one of the virtues of virtualization being the ability to quickly provision new servers and services, businesses will take advantage of this flexibility to test and adopt new server based technologies that might gain them a competitive edge. In the past, the issues involved in adding a server to the data center would have acted as a brake on rapid changes. With virtualized environments, these changes no longer have the negative business impact they once had. Thus, business units will jump on that process, forcing IT to make sure all the support pieces, including the power and cooling infrastructure for the virtual server hosts, is in place to allow these rapid changes. Unforeseen changes With virtualization, changes can be made to existing servers that significantly impact the cooling needs of those servers, and as the resulting hot spot is created, it will impact cooling on servers in the same physical area. As these changes can be made without any physical access to the server, IT needs reliable tools that monitor for hot spots and have the ability to dynamically adjust for these changing conditions. 14
Chapter 1 Interdependencies The data center is a complex organism. Changes made to the behavior of the data center such as moving servers, virtualization, changing cooling patterns, and modifying space requirements can have unexpected impacts due to the close relationships between the component entities that comprise the data center. Lean provisioning The move to a lean provisioning model for power and cooling allows maximum optimization of the data center infrastructure. However, unexpected changes can have unexpected results in the behavior of the power and cooling needs of the data center. Planning to address these issues as part of the data center migration to virtualized environments will give IT a hand up in implementing and deploying next generation technologies. Summary Virtualization is coming soon to your data center, if it isn t already there. And as the drive for server consolidation continues to increase, the performance of your data center in supporting high density, variable load systems will become critical. The optimal way to address these needs is to take the whole system approach to building, or re building, your data center power and cooling infrastructure. These issues need to be addressed while the changes in the data center are being planned, with the impact that virtualization will have on the data center infrastructure being considered as part of the implementation and deployment process. With this in mind, building a highly optimized and electrically efficient data center will not be an afterthought; the design and capabilities of the infrastructure can be tailored directly to the business needs of the organization, resulting in the most cost effective solution possible. 15
Chapter 2 Chapter 2: Best Practices for Data Center Design Building an energy efficient data center doesn t happen by accident even with the adoption of the latest hardware featuring energy efficient design, maintenance of a green perspective when making your purchasing decisions, and ongoing awareness of the economic benefits of energy efficiency. To develop a data center that maximizes the energy efficiency of the facility, from the ground up, an organization must consider design and implementation issues of the data center infrastructure as well as the components installed within. The only way to ensure that your data center is built with the highest possible energy efficiency is to make sure that the actual electrical cost of the data center is factored into the design parameters of the data center. This means taking the extra steps of utilizing the tools that are available for modeling the electrical costs of the data center, defining a model of the power utilization of the facility, and getting the necessary information to the responsible decision makers to enable them to properly factor in the electrical cost consequences to the overall efficiency of the data center. Key Issues in Building an Energy Efficient Data Center Building a data center is more than just making sure that you have space and power. Properly designing a data center that can provide a decade or more of reliable service to a growing company is a process that requires careful planning and the understanding and consideration of many issues: Optimizing your data center architecture Build the data center that you need to meet your current and future needs without oversizing or under committing resources to get the results your business needs Reducing power consumption: o Avoid oversizing o Rightsize your physical infrastructure devices o Invest in more efficient air conditioning o Invest in more efficient power delivery equipment o Virtualize servers o Implement improved data center cooling architectures o Take advantage of inexpensive changes that improve cooling efficiency 16
Chapter 2 Gaining the benefits of standardization Standardizing on power delivery systems allows for modularization of these systems, simplifying the process of growing or changing the data center infrastructure Understanding the economic benefits in saving kilowatt hours consumed Don t just accept that power costs money; make sure that power consumption avoidance is part of the design strategy Consideration of these issues will be your starting point in building an energy efficient data center that meets the needs of both IT and business. Properly Sizing Your Data Center Data centers are not static entities. Thus, sizing the power and cooling requirements of the data center requires consideration of the planned growth over the life cycle of the data center. Research has indicated that the design life of the average data center is 10 years. Although it can be difficult to accurately plan for the power and cooling needs of the data center more than a few years down the line, planning for properly sizing the data center energy needs begins with understanding the initial start up IT requirements, the minimum and maximum final IT requirements, and the timeframe for growth. With this information, you can create a growth profile that you can use as a basis for your physical infrastructure design. In most circumstances, the maximum expected load that is planned for is never reached. The criteria used for this number is often the worst case (or perhaps, best case) scenario that can be envisioned by the IT and business planners. This value may not match up with the business projections but rather is simply a reflection of the concerns of IT in growing the needs of the data center while minimizing the chance of requiring difficult and expensive rebuilding and reconfiguring of the data center to meet growing business needs. Oversizing The basic definition of oversizing is having significantly more cooling and power capacity than is necessary to efficiently run your data center. As power and cooling delivery has traditionally been a static process, oversizing in data centers is a very common occurrence and is one of the first issues identified when planning for an energy efficient data center. Oversizing is often a result of IT managers attempting to be proactive. That is, the managers attempt to future proof data center components when planning for the growth of the data center, and in doing so, spec out power and cooling facilities that far exceed the current or startup needs of the data center. 17
Chapter 2 Figure 2.1: Capacity over time compared with expected actual use. Issues & Drawbacks Many of the issues that IT attempts to address in the planning stages of data center design are those that logic would indicate will cause problems in the life cycle of the data center. Managers are attempting to head off problems by starting off their data centers with oversized power and cooling systems: Attempting to provide sufficient power and cooling capacity for the entire life cycle of the data center, not just the start up needs Reducing the chance that additional capacity will need to be added in the future, a very expensive process in an existing data center Providing sufficient capacity so that future growth does not require major changes in the data center that could cause downtime The problem here is that all the planning and design is to meet, in most cases, a need that cannot be accurately estimated over the life of the data center. Thus, the error of oversizing is not the fault of IT planners but rather the nature of the problem itself. Over the life of the data center, the power and cooling demands will be not be static; demand will change, both up and down, as the role of the data center and the equipment within changes to accommodate the needs of the business. 18
Chapter 2 Avoiding Oversizing Avoiding the problem of oversizing and its associated waste of already scarce funds is best done by approaching the problem with the goal of creating an adaptable power and cooling infrastructure. By doing so you are able to greatly reduce the costs and effort that are applied to the design and engineering of a traditional data center. A variety of techniques can be applied to allow for this flexibility in design and construction of your data center. The most important feature will be modularity, that is, the ability to implement and deploy building blocks of resources that don t require significant preengineering and work well within the confines of the data center as designed. This functionality gains the data center the following capabilities: No special site preparation required for changes in the data center power and cooling services delivery No need for raised floors or other similar data center architectural features Reduction or elimination of the need for configuration specific wiring or construction to deliver power and cooling services The ability to operate different parts of the data center with different redundancy configurations A major increase in flexibility and efficiency in the delivery of power and cooling without an increase in the overall cost in running the data center Reduced waste in terms of both energy and money, over the life of the data center Potential increase in the lifespan of the data center design as the modularity improves functionality and flexibility to adapt to changes in the technical and business environments 19
Chapter 2 Figure 2.2: Modular/flexible designs reduce the money wasted in the energy life of the data center by reducing the waste due to oversizing. Research has shown that the typical data center is built to support 300% of the required power and cooling capacity. Thus, the upfront costs are significantly higher than they need to be and ongoing maintenance costs are wasted on supporting unused capacity. By implementing an architecture that is able to deliver power and cooling services as and where needed, rather than a system that anticipates changes that may never happen, cost savings over the life of the data center are easily realized. Reducing Energy Consumption What does your power cost? It is a given that your data center is going to consume electricity, and it is easy to just shrug your shoulders and accept that there is a significant expense involved in getting that power. Power costs continue to increase and it shouldn t be expected that over the life of your data center, the basic cost for a kilowatt hour (kwh) will go down. Thus, it is important to build a facility that can make the best use of the power being delivered to it. 20
Chapter 2 To determine what your power is likely to cost, you need to understand the tariff structure and methodology of your power provider. A Web search for commercial general services tariff for your provider will usually get you a document that attempts to explain how you will be charged for service. A detailed explanation of electricity charges is beyond the scope of this guide, but the basics are as follows (terms may differ in your location): Customer charge The basic fixed amount charged by the provider for servicing the account Generation charge The cost for the production of the electricity used; there may be some flexibility here as many areas now offer the ability to negotiate between different suppliers for electricity Distribution charge The cost of delivering the power from the high voltage lines to the customer s premises There may be additional charges depending upon your provider. It s also likely that the service is not provided at a fixed rate but at a variable charge based on the amount of power used, type of power demand, peak service hours, seasonal demand, fuel surcharges, local taxes, and so on. So what does this mean to the IT data center staff? Basically, the issue is that the actual cost of power is a moving target. This doesn t mean that you should just accept that you will be paying some unknown amount each year for your data center s power needs. It means that your facilities team needs to negotiate with the power provider for the best possible rates for your needs, and that the design and implementation of the data center infrastructure should be as power miserly as possible. Waste is expensive, especially when factored in over the 10 year lifespan model for the data center. Most of the power going into the data center is used by the data center infrastructure. Studies have shown that in most data centers, less than half of the power delivered to the data center is used to run the actual IT load. Figure 2.3 demonstrates typical power utilization within the data center. 21
Chapter 2 Figure 2.3: Typical power utilization in the data center. This utilization pattern makes it clear that implementation of energy saving techniques and equipment in the data center infrastructure will result in the greatest energy savings. Ideally, we use the equipment that draws the least possible power to deliver the required services. This method isn t the same as using the most efficient equipment; energy efficiency measurements for devices are a somewhat nebulous metric and can rarely be applied across different types or brands of devices. Trying to select equipment for your data center by comparing energy efficiency ratings across vendors is a fairly useless exercise. Vendors need to provide information about energy consumption across a wide range of conditions in order to allow purchasers to make informed purchasing decisions. Looking at the actual power used to run devices and at ways to reduce that power demand is the best way to build an energy efficient infrastructure. In many ways, reducing the energy requirements of the IT load is the easiest part of the equation. Current technologies in server hardware and software design have made servers far more efficient in terms of energy use than designs even a few years old: Dual and quad core processor based servers can be used to replace multiple individual legacy servers N way servers using single core processors can be replaced with multiple core CPU servers for improved energy utilization and/or greater performance Blade servers using a variety of processor configurations allow for the tailoring of servers to applications and more efficient energy utilization 22
Chapter 2 Enterprise class hard drives have become more efficient, delivering higher capacities and greater performance without increasing power requirements Application consolidation allows for fewer servers to do the work of many earliergeneration servers Virtualization means that fewer physical servers are necessary. Each physical server removed means a savings of thousands of dollars in energy costs over the life of the data center, far beyond the cost of the physical server Data Center Physical Infrastructure Equipment Data center physical infrastructure systems are major consumers of the power being supplied to the data center. The more these devices consume, the less efficient the data center is. Specifically, these systems include: Switchgear UPSs Power distribution units Transformers Air conditioners Humidifiers Cooling pumps Heat rejection equipment (for example, chillers and towers) Problems with these devices such as under or oversizing, improper installation, and poor airflow control can cause the data center to draw 100% more power than is actually required for efficient operation. By properly matching the equipment to the need, or rightsizing, your infrastructure, savings in energy consumption can be realized. 23
Chapter 2 Energy Consumption Reduction with Physical Infrastructure Equipment Rightsizing is the optimal methodology for achieving maximum savings and energy consumption reduction with physical infrastructure equipment. Chapter 1 discussed the difference between proportional and fixed losses. Fixed losses are a significant portion of electrical consumption in typical data centers. In lightly loaded data centers, the energy expense for these fixed losses can actually exceed that of the IT load. If the data center energy infrastructure is oversized, these fixed losses unnecessarily become a significant fraction of the total electrical consumption. For this reason, rightsizing is critical. Studies have indicated that the potential savings with rightsized solutions approach 10 to 30%. Energy Efficient Systems Design Maximizing the efficient use of energy in the data center is not simply a process of investing in the equipment that advertises the highest energy efficiency. Data center efficiency models have been developed to estimate the energy efficiency of various data center architectures. Energy efficient data center system design is a result of an overall design plan where the proper equipment is selected, configured, deployed, and managed in such a way that its energy use is minimized while delivering the services that are required by the data center. Table 2.1 shows changes that can be applied to your data center design criteria that have the potential for the greatest savings in energy reduction. You will notice, however, that the benefits shown in the chart apply primarily to new data centers, as these concerns should be addressed in the basic design criteria of your data center. 24
Chapter 2 Table 2.1: Top design considerations for energy efficiency. Table 2.2 shows simple tasks that can be performed in existing data centers to achieve additional energy savings. 25
Chapter 2 Table 2.2: Changes that can be made to existing data centers to improve energy utilization. Understanding the Impact of Humidity Control in Your Data Center Humidity in the data center plays a major role in the efficient operation of the data center equipment, for both physical infrastructure and IT hardware. All equipment has an optimal environmental operational range and proper humidity plays a large role in defining that environment. 26
Chapter 2 Effects of Humidity Anyone who has lived in or visited an area where the humidity is very low has noticed that the static electricity phenomena are fairly commonplace. This is because the moisture in the air prevents this electrical discharge in more humid environments. Now consider the environment of a data center. If proper humidity isn t maintained, problems begin to occur. Too little humidity and the aforementioned static electric discharge can damage equipment; too much humidity and the possibility of moisture accumulating within electrical equipment can happen. Measuring Humidity Every piece of IT equipment that exists in the data center has a listed operating environment range defined as temperature and humidity. Generally, this is a very broad range of relative humidity (usually from 20% to 80%). Relative humidity is a measure of the percentage of the maximum water vapor that air can hold at a given temperature and pressure (our concerns are almost always at 1 atmosphere or 14.7 PSI). We are also concerned with the dew point, which is the point at which water becomes visible as condensation, having left the air (and its vapor form) and appearing as a liquid trickling down the sides of our equipment. With IT equipment, the humidity and temperature measurement is taken at the cooling air intake opening for the equipment. This is the condition of the air as it is being drawn into the equipment (exit temperature and humidity is not an issue). Figure 2.4: Measuring intake temperature. 27
Chapter 2 Controlling the Environment In order to control the humidity in the data center, you need to minimize the variable conditions that can cause changes in the air in the data center. A number of the steps that need to be taken deal with the actual construction of the facility, making use of features such as vapor barriers during this process. Location of the data center facility also needs to be factored in, as creating a data center in an existing structure means that you are able to factor in that the outside air is already being processed in some fashion; your data center may need to deal with normal office space air (in an existing structure with space that is being converted to a data center) and ensure that the existing office air conditioning does not conflict with the needs of the cooling equipment for the data center. Minimizing Internal IT Factors In general, humidity needs to be added to the air in a data center because the heat being generated by the equipment has the twofold effect of causing the air to dry out and to increase the vapor carrying capacity of the air by raising its temperature. In a properly designed air management system, humidity will be centrally provided by using one of the common types of humidification systems: Ultrasonic humidifiers vibrate water to create a mist that is introduced into the air circulation system Steam canister humidifiers use a set of powered electrodes to convert water to steam, which is then mixed with the air circulation system Infrared humidifiers use high intensity lights over open pools of water to release water vapor into the air, increasing the humidity Understanding Cooling and Capacity Management As technology needs grow, the loads placed on the data center continue to increase. As the load increases, the potential for failure increases, especially if the load growth is not managed or is managed improperly. Reducing TCO and maximizing availability are the most marketable benefits of a comprehensive management scheme. 28
Chapter 2 Designing to Support Growth and Change The flip side of building a modern data center that is capable of growing and changing to meet the needs of business is that the data center infrastructure needs to have a management strategy and technologies in place that can meet the demands that a rapidly changing business and technology environment can potentially place on the data center. There are a number of simple questions that have to be answered when making changes to the IT infrastructure within the data center. Ideally, the answers can be provided before the changes are made, rather than the traditional Let s make the changes and see what happens approach that is all too common: What percentage of my current total power and cooling capacity is being utilized? Can new technology be deployed without having a negative impact on the existing environment? Will installing new equipment affect my safety margins? Will I be able to maintain redundancy if new primary equipment is added to the environment? Where can I install new equipment to minimize the impact on my current cooling model? How much future growth can the existing infrastructure support? Where is the optimal location (rack) for my new IT equipment? With the current pace of change in the data center, the power requirements of new technologies, the high density servers and storage that have become commonplace, and the need for the ability to rapidly modify the conditions in the data center, a comprehensive management tool is an absolute requirement for the successful implementation of a stateof the art data center. Meeting Supply and Demand Goals The bottom line for being able to manage the power and cooling demands of the data center is the ability to answer questions, such as those previously posed, about the data center. The trick is to understand at what level of detail the various pieces of information need to be derived. 29
Chapter 2 Information about power and cooling at the data center/room level can be useful in answering general capacity questions. However, for accurate answers to questions about changes to the IT infrastructure, it is necessary to have information at the rack level. Having detailed information about power and cooling at the rack level provides the best, and most useful, information for capacity management. There are four key metrics that can be defined for rack level capacity management: As configured maximum demand The maximum level of power and cooling consumption that the rack can utilize when configured at its maximum density Current actual demand The real time measurement of the power and cooling services used by the rack at the current point in time As configured potential supply The amount of power and cooling that can actually be delivered to the rack by the data center infrastructure Current actual supply The power and cooling services that can currently be delivered to the rack as impacted by other factors in the data center that may impact the ability to deliver services to that particular rack Figure 2.5: Demand and supply demonstrated at the rack level. 30
Chapter 2 Managing Capacity Managing the power and cooling capacity of your data center ensures these systems are optimally deployed and utilized. The full spectrum of capacity management involves monitoring the existing environment, forecasting future needs, and modeling changes to the data center environment. Like any management system, the capacity management software needs to be able to present data on the capacity status of the current environment, set and manage the capacity plan, alert on conditions as set by the IT team, and allow for what if? planning for changes in the power and cooling infrastructure. IT personnel familiar with network and systems management tools should feel right at home, in terms of look and feel, when looking at the software capacity planning management console. Figure 2.6: A typical capacity management application management screen. 31
Chapter 2 Monitoring Monitoring falls into two categories: performance monitoring and workload monitoring. Watching these two areas allows for the proper management of the data center power and cooling infrastructure by providing management with a direct look at the behavior of their delivery systems in a near real time fashion. The capacity plan, which was established during the design phase of the data center, is monitored from this point. Any deviations from the plan trigger automated alerts, which are very important in maintaining the delivery of power and cooling services and keeping ahead of any potential problems. Forecasting Like monitoring, forecasting falls into two categories: supply and demand forecasting. Both of these forecasting models take advantage of the data collected by the capacity management software to give the administrator the ability to forecast future needs. Making use of historical data gives the software the ability to project the power and cooling requirements for potential growth within the data center. Modeling The capacity management system has access to a wealth of historical data, at a rack level, so the software can be used to analyze the effects of potential changes to the data center, both in terms of the power and cooling delivery systems and the addition or removal of racks and equipment. The software should be able to not only determine whether potential changes are possible, within the existing capacity management plan, but also add suggestions as to how changes should be made or where equipment should be located, using information drawn from its historical knowledge of the data center infrastructure. Using the Row & Rack Cooling Methodologies Room, row, and rack are the basic cooling methodologies available to the data center. Traditional, raised floor data centers are often referred to as room oriented cooling architecture. In this model, the computer room air conditioning (CRAC) units treat the room as a single entity. Thus, airflow differs greatly depending upon the layout of the room. Even in carefully designed data centers that employ this method, as the contents of the data center change, the way that the airflow behaves within the data center can change in unpredictable ways. It s even possible for the cooling needs of the room not to be met, even when the capacity is there, due to cooling being circulated through the room without getting to the systems that need it because of unexpected airflow restrictions and constraints. The inherent difficulties in building a large, flexible room oriented cooling architecture have led to the introduction of additional techniques for providing manageable cooling where it is needed: the row and rack architectures. 32
Chapter 2 Figure 2.7: The basic concepts of room, row, and rack cooling. The nature of the three architectures also allows for hybrid designs where any of the architectures, or even all three, can be successfully used in the same data center. Benefits Over Traditional Room Cooling Techniques Both the row and rack cooling architectures offer many benefits over the traditional roomoriented cooling architecture. Primarily, these benefits result in better control and delivery of cooling to the IT devices in the data center. Both architectural models offer a high level of flexibility and a reduced need for specific architectural designs to take advantage of the cooling capabilities. Row In the row oriented architecture, CRACS are associated directly with individual rows and installed in close proximity to their row (above, below, or within). This setup provides shorter airflow paths as well as much more predictable airflow, allowing all the cooling capacity of the CRACs to be used. The short airflow paths also mean increased efficiency for the CRACs by reducing the fan power required to deliver the cooling where needed. No raised floors are necessary, which removes the architectural requirements for their support. Rows can be configured to support specific types of applications; for example, a row with many high density racks running compute intensive applications might require a higher cooling capacity while the next row, running low density servers that might require much lower cooling support. This design flexibility can simplify the management of the data center cooling requirements. 33
Chapter 2 Rack With the rack oriented cooling architecture, the CRACs are directly attached to, or within, the racks that they are cooling, providing the maximum level of detailed control of the cooling capability and the most efficient cooling possible. This model also supports the highest rack densities and is unaffected by any installation variables, room, or row considerations. Each of the cooling models has their benefits and considerations, and a combination of any of the three architectures will result in a flexible cooling delivery system that can support any business data center needs. Conclusion The best data center design is a holistic process, taking into consideration the entire life cycle of the data center and the planned IT load. From the efficiency targets to delivering cooling to the IT load racks, the consideration of all aspects of your planned data center is the least expensive part of the data center design process and yet can result in the most significant savings over the life of the data center. Decisions such as standardizing on modular/flexible power and cooling systems, hybrid cooling architectures, and rightsizing the power and cooling equipment make the most of the funds that are invested in the data center project and offer long term cost savings in areas that traditionally caused significant expenditures over the life of the data center. Proper design considerations prior to the creation of the data center lead to a facility that is able to support the needs of the business for the foreseeable future. It is possible to have an energy efficient data center that gives significant agility to business processes. 34
Chapter 3 Chapter 3: Free, or Nearly Free, Things to Do in an Operating Data Center In an average data center, several steps can be taken to improve the cooling and operational efficiency of the data center. In the past, the data center was considered a single entity. This meant that cooling and energy delivery systems treated the data center as if conditions were equivalent throughout the space. The reality is that in most data centers, a number of different types of activities are being performed by the equipment that results in very specific sets of conditions. By grouping these different sets of activities by their need for power and cooling, it is possible to more efficiently deliver both services to the locations and systems within the data center that most need the support. By understanding the conditions within your data center, it is possible to find efficiencies in the delivery of power and cooling that can be achieved using the existing infrastructure and making organizational or minor physical changes to the data center itself. Doing so, you will be able to optimize the delivery of power and cooling services to devices within, on a limited budget. The Traditional Cooling Model In the traditional data center cooling model, cold air is delivered to the room in a manner designed to circulate the air across the devices that require cooling. This can be done via a raised floor delivery system (as shown in Figure 3.1) or, in smaller data centers, with placement of computer room air conditioners (CRACs) and returns as appropriate. Figure 3.1: Raised floor cooling in a traditional data center. 35
Chapter 3 Oftentimes, servers are mixed into the data center in no particular order, with high load and light load servers running side by side; there is no consistent methodology as to placement of racks and components, though ideally they are placed in locations that don t impede the air circulation within the facility. Due to the minimal expense in providing power and cooling services to the data center, older data centers often just threw oversized cooling systems at the potential problem; a solution that is no longer cost effective with the rising energy costs. Auditing Auditing is the process of identifying potential problem areas for cooling in the data center. At its most basic level, the purpose of the cooling audit is to make sure that cool air has an unobstructed path from the CRAC to the air inlet on the server and that hot air has an equally unobstructed return path. Failure to provide sufficient cooling in the right places can result in damaged equipment or temperature related electronic equipment failure. By auditing the cooling of the data center, these potential problems can be avoided. The first step in the auditing process is to perform a baseline audit; by doing so, you will be able to measure the success of any corrective actions that you determine are necessary to deliver proper cooling. If this is a new data center, you will be able to use this data to establish a baseline that is easily used to determine the impact of future changes. In an existing data center, this baseline will give you the starting point for establishing the cooling efficiency of your facility. When you complete your changes and have resolved that your data center cooling model is operating at peak efficiency, you will have established a new baseline model that future changes can be evaluated against. Find Out What s Really Going On Understanding how your cooling is actually working within your data center is critical for maintaining the long term efficiency of not just the cooling equipment but also the hardware that is being cooled. Once you have established your data center cooling baseline, you will be able to maintain the cooling efficiency of your environment. The next section of this chapter will provide basic guidelines for performing a cooling audit in your facility. We will look at the fundamental tasks necessary to properly establish the baseline information required for future cooling audits. 36
Chapter 3 Checking Capacity The first step in performing your cooling audit is the capacity check. This test determines whether you have sufficient cooling in place for your existing and planned environment. Remember that 1 W of power requires 1 W of cooling. Note Although there are many types of air conditioners, for the purposes of this chapter, we will use the generic term CRAC to refer to all types of computer room air conditioners. Remember that our goal is to match the cooling capacity to the cooling need. In the previous chapters, we discussed the issues involved in oversizing the cooling capacity of your data center and pointed out the problems and inefficiencies that result from this condition. By doing some research on the model of each CRAC unit, you should be able to find manufacturer s specifications that will establish the cooling capacity of each unit. The manufacturer will specify the optimal operating range of the cooling equipment and its efficiency based on the entering air temperature and the level of humidity. Remember that the air conditioning equipment is not solely the cooling component but also the external heat rejection equipment. In smaller environments, both air conditioning and heat rejection may be contained within the same unit. In larger environments, this will not be the case, though it s possible that both components may be acquired from the same vendor. In these cases, the cooling and heat rejection capabilities will almost always be matched. However, if you are using multiple vendors, these components may not be equally matched. In this case, use the lowest rated component for capacity calculations. Once you have the theoretical maximum cooling capability of your data center cooling equipment, you can use the worksheet shown in Table 3.1 to calculate the heat output of the equipment in the data center. Remember that our goal is to have the cooling capacity and the heat output match, which results in the most efficient operation for the data center. However, there are a number of factors that prevent the cooling equivalent from operating at maximum efficiency and achieving the maximum theoretical cooling capacity. 37
Chapter 3 Table 3.1: Estimated heat output calculation worksheet. Checking Hardware CRAC units generally have four modes of operation: cooling, heating, humidification, and dehumidification. We are looking for these units to operate in a coordinated mode. Although multiple conditions may exist simultaneously, for example cooling and dehumidification, it is important that all systems within the defined area be operating in the same mode. This defined area may be a rack, a series of racks, a row, or even a series of rows or a contained area within the data center. This allows us to prevent the condition known as demand fighting. When demand fighting is occurring, for example, one unit is humidifying while the unit next to it is dehumidifying; this situation not only increases the operating expenses by increasing energy use but also reduces the cooling efficiency and capacity of your data center. Studies have shown that failing to address the demand fighting problem can result in a 20 to 30% reduction in efficiency. The simplest way to avoid this condition is to ensure that the set points for temperature and humidity are consistent on all the CRAC units within the data center. These settings can be more narrowly focused based on the techniques being used to cool the data center, such as row and rack cooling. But in all cases, units within the same grouping should be set to match. To properly check hardware and to test the performance of the cooling system, it is necessary to measure both the return and supply temperatures. As Figure 3.2 shows, there are three points at which temperature should be monitored. 38
Chapter 3 Figure 3.2: Monitoring points. The temperature at the monitoring points starts at the supply air temperature. The temperature required at the server air inlet is the target temperature for the supply air temperature. The most common problem in delivering the appropriate temperature is short cycling. Short cycling is when the cool air supply leaves the CRAC and rather than flowing through the IT equipment at which it is targeted, it instead bypasses that equipment and flows directly into the air return duct for the cooling unit. The good news is there are inexpensive solutions to addressing air flow performance. Another very common problem that is easily dealt with is the issue of dirty filters in the air conditioning equipment. Simply making sure that clean air filters are in place goes a long way towards ensuring efficient operation of the cooling equipment. 39
Chapter 3 Systems Testing Testing cooling equipment for proper operation and to ensure that it s operating within its optimal design parameters requires specific knowledge of cooling equipment. This level of maintenance knowledge is unlikely to be present within an IT department. It is, however, available through a facilities department, a maintenance company, or HVAC contractor. There are a number of features that should be checked by the appropriate technician, including: Current status of the equipment Chilled water cooling circuit Condenser water circuit Air cooled refrigerant piping Although some aspects of the air conditioning equipment require specialized skills to test, the basic task of measuring temperature within the data center at specific locations, such as the aisles between the equipment racks, requires little more than temperature measuring equipment and a solid plan. Standards exist for the positioning of temperature sensors to allow for optimum testing. These standards are propagated by the American Society of Heating, Refrigeration, and Air conditioning Engineers, in Standard TC9.. Resource More details on the ASHRAE standards and their implementations can be found had to be www.ashrae.org. In general, recommended inlet temperature will be in the 64.4 to 80.6 F range (also part of the TC9.9 standard). Temperatures that exceed this range can result in a reduction in system performance, reduced lifespan, and unexpected shutdowns of the equipment. For general temperature measurement, the temperature should be measured at a point 5 feet off the floor, and the measurements should be carried out over a 48 hour period recording maximum and minimum temperature levels within that timeframe. Temperature should also be measured at the top, middle, and bottom inlet points of each rack, as very often air circulation is not equivalent at each point and equipment at the bottom of the rack is generating heat that is rising and can potentially cook equipment mounted above it. 40
Chapter 3 Understanding the Impact of Temperature on Servers The long held belief on data center server temperature has been that energy demands will decrease as server inlet temperatures increase. The trick, however, is finding the temperature that allows optimal server performance with minimal energy consumption. As erring to the wrong side (too warm) may cause hardware issues, it is important to properly balance server performance, energy demands, and cooling. Finding the Sweet Spot for Minimizing Energy Consumption The key to utilizing less energy is to look for techniques that will enable cool air to be delivered to the server air inlet with as little reduction in the temperature of the provided air as possible. Normally, the CRAC will deliver air at 55 degrees in order for the air to reach the server at 70 degrees. The temperature increase comes from air mixing between the CRAC and the server. Thus, making changes to the data center environment that reduce air mixing, implementing containment systems, and managing airflow will allow the source air to be delivered at a higher temperature and still be delivered to the server at the optimal temperature, reducing the energy costs associated with cooling the data center. How Managing Airflow Reduces Costs Unless the cooling capabilities are seriously undersized, there are a number of inexpensive technologies and tricks that can be used to aid in airflow management. By properly managing the airflow within the data center, there are significant potential costs savings that can be achieved by allowing equipment to operate in its optimal temperature range. Understanding the Use of Blanking Panels Rack mounting is the most common way that servers are deployed within the data center. Blanking panels are the pieces that are inserted into empty spaces in those racks. These inexpensive pieces are a critical component in managing airflow in the data center (see Figure 3.3). 41
Chapter 3 Figure 3.3: Airflow with and without blanking panels. As the figure illustrates, without the blanking panel in place, warm air from the server rack is drawn back into the server, requiring additional cooling to keep the incoming air at a temperature that the server will be happy with. The simple expedient of installing the blanking panels prevents this from happening. Minimal Cost / Maximum Gain So if blanking panels make such a significant difference with so little effort, why aren t they always used? The answer is a simple one: In most cases, data centers are used to having bolt on blanking panels on hand. This means that each blanking panel requires four screws and nuts to bolt it into place, which is a minor annoyance with a single panel, but a major effort with tens or hundreds of blank spaces to fill in the data center. Additionally, some blanking panels come in multiple sizes; the rack might need 4 U of space filled and the only panel available might be 3 U in size, making it a bit of a jigsaw puzzle to match up blanking panels with the spaces that need to be filled. To address this issue, third party vendors have started introducing standard size snap in modular blanking panels. Instead of needing to be screwed onto the rack, these modular panels snap in and are quickly and easily installed. They also only come in a single size (1U) so that the only combination required is having sufficient 1U panels available to fill the open spaces. Modular blanking panels provide a simple and inexpensive solution to this problem. Although blanking panels address the most common issues caused by improperly circulating air in your rack mounts, Table 3.2 highlights other common problems and offers simple and inexpensive solutions to make your server racks more effectively cooled. 42
Chapter 3 Table 3.2: Common cooling issues with rack mounted servers. Proper Placement of Vents and Tiles In a traditional raised floor data center, the placement of the floor tiles and vents is critical to maintaining proper cooling. In many older data centers, these tiles are not reconfigured when new or additional server racks are installed, resulting in a reduction in cooling efficiency due to the unplanned modification of the existing airflow paths. Properly reconfiguring the floor tiles and vents can address the changes made by the additional server racks and reduce the thermal load increase. 43
Chapter 3 Containment As data center professionals have been driven to reduce their operational costs, the energy management practices of an earlier generation have been identified as particularly wasteful, especially with the drive to green IT. A number of technologies have been developed to deal with more efficient cooling practices. Chapter 2 talked about one of the most successful row based cooling. In this chapter, we are looking at techniques that let us maximize energy efficiency in our existing environment. For that purpose, we need to consider the issue of containment. Keeping the Cold Air, or Hot Air, Where You Need It As we have previously discussed, one of the primary issues in data center cooling is the mixing of hot and cold air inappropriately, reducing the overall efficiency of the cooling process and requiring an increase in energy expenditure to provide the proper air temperature at the server inlet. This brings us to the concept of containment: keeping the hot and cold air separate, contained in an area that allows greater control over the environmental variables that most affect IT loads (temperature and humidity). Containment falls into two categories: Cold air containment Deployed in existing data centers, cold air containment systems (CACS) make use of the existing perimeter based CRACs and a containment system that delivers cold air to the computer rows and uses the bulk of the room as the hot air return plenum. Hot air containment A hot air containment system (HACS) makes use of in row cooling and an enclosed aisle to build a self contained system that can support highdensity IT loads. Although the systems are very different in operation and IT requirements as well as configuration and ease of implementation, there are a number of benefits that both types of containment systems have in common: Right sized physical infrastructure By matching the cooling capability to the cooling demand, data centers are able to avoid the problems associated with oversized solutions. Both containment systems allow IT to deliver the cooling capacity where it is needed. Better energy efficiency As we discussed earlier, the ability to operate cooling equipment at higher temperatures, yet deliver the same level of cooling to the server hardware, reduces the overall energy expenditure and reduces costs. Reduced humidity control costs In closed systems, such as CACS and HACS, little to no humidity changes are caused when the air is circulated through the system, reducing the need to expend energy to modify the data center humidity to maintain optimal environmental conditions. 44
Chapter 3 Evaluating Cooling Architectures Organizations with traditional raised floor computer rooms or those data centers simply using perimeter CRACs to maintain temperature and humidity can implement a CACS approach without a significant investment in new equipment. Figure 3.4: Air flow diagram for a cold air containment system. As Figure 3.4 shows, the CACS delivery system doesn t look much different from a standard raised floor data center. The difference is that the cooled air has been directed to a row (or rows) that have some form of containment system installed. This isn t a system that can be used for a small percentage of the data center. If this model is used, each row of rack servers must be contained within its own CACS as the overall temperature of the data center server room outside of the containment area will be well above the recommended 80.6 degree maximum inlet temperature for server operation. Other equipment within the data center that is not contained within the rows of servers, such as tape libraries and other supporting devices, will need to have arrangements made to deliver cooling. Air temperatures within the data center outside of contained areas can be expected to go as high as 100 degrees. These temperatures become the norm and workers within the data center need to become adjusted to the changes. Although there are formal systems for containing cold air, many data centers use the simple (and inexpensive) expedient of installing plastic curtains to surround the individual aisles that are being cooled. In a raised floor system, the floor of the curtained aisle gets perforated panels to deliver the cooled air to the front of the server racks, while the back of the racks are open to the room, which is used as the hot air return plenum. 45
Chapter 3 Figure 3.5: An example of a simple CACS setup. Keep in mind that techniques we ve already mentioned in this chapter concerning the proper use of blanking panels and floor tiles and vents are critical in the CACS model. The hot and cold air mixing needs to be controlled as much as possible in order to deliver cooled air with the least possible energy expenditure. There are two significant limitations to the CACS approach to cooling: Density limitations Without custom designed systems and significant modifications to the data center infrastructure, there is a practical limit to how dense the servers can be racked within a CACS. Adding fan powered floor tiles can increase potential density, as can moving to row based cooling systems, which removes the density limitation, from the perspective of cooling, entirely. Inefficiencies in air distribution The cooled air supply (that is, the CRAC) is at a distance from the CACS, and a significant portion of the overall energy requirement is needed to simply move the air to the location necessary. It is still more efficient, however, than the traditional whole room data center cooling model. The HACS avoids many of the pitfalls of the CACS approach, but it requires a significant investment in the cooling infrastructure if traditional perimeter cooling is in place. The requirements include in row cooling with variable speed fans, temperature controlled air supply, doors at either end of the row, and a roof. 46
Chapter 3 Figure 3.6: Example of a configured HACS. It is possible to build a HACS environment without in row cooling, but it would require significant custom ducting and design in order to deliver and circulate the proper amount of air. In addition, this configuration would negate many of the benefits of the HACS design. HACS is also effectively room neutral in terms of its impact on the data center environment. Because of its self contained nature, complete HACS configurations can be added to existing data centers without upsetting the environmental balance in the data center. This increases the design flexibility when adding additional HACS cooled rows to an existing data center. It becomes possible to not worry about environmental issues and just concentrate on finding an appropriate amount of space. HACS provide a higher level of operational efficiency due to a number of factors, from the ability to operate at higher temperatures (without impacting the rest of the room) to the ability to operate at higher IT load density for lower energy expenditure than a CACS solution. In an existing data center environment, implementing CACS can be done with minimal investment and modification of the current configuration. HACS offers the ability to build higher density data centers and can be added to an existing data center, but at a much higher price point than implementing CACS. For a point by point comparison of the two models, see Table 3.3. 47
Chapter 3 Table 3.3: Comparison of HACS and CACS benefits and concerns. 48
Chapter 3 Conclusion Although there are few truly free things that can be done to improve the energy efficiency of your data center, we ve given you the starting point to evaluate how your data center is utilizing energy as well as basic tips to improve the overall efficiency of the existing facility. Don t forget that there are always small things that can be done to improve overall efficiency that don t require major equipment configuration changes, such as the use of airflow management devices and using economy modes on cooling equipment. Understanding the energy needs and uses within your data center is the first step towards implementing a more energy efficient facility. Start with a basic cooling audit and evaluate the current data center cooling infrastructure, then move on with simple, inexpensive changes, such as procedural changes regarding ancillary power usage; blanking panels and efficient tile and panel configurations; and maintenance of unimpeded cooling airflow. These techniques are cost effective measures you can take to see noticeable improvements in cooling efficiency. 49
Chapter 3 Chapter 4: Strategies for Upgrading a Production Data Center With an existing data center, there is ongoing pressure to maintain an environment that is cost effective and productive. As technologies evolve and external factors affect the costs of doing business, IT executives have to stay on top of the technologies that can make their existing facilities as efficient as possible. Understanding these technologies and how they can impact the efficiencies of your data center provides a firm foundation for making choices about the future of your data center infrastructure. Upgrading an existing data center means making the right choices to make your data center more energy efficient. As we have covered in earlier chapters, there are both technological and organizational efforts that can be applied to achieve this end. The tasks involved in upgrading your data center fall into two general categories; those involved with the actual physical environment, such as layout, power, and cooling, and those that focus on the overall process, such as meeting best practice standards and management. Best Practices for Data Center Cooling and Power Migration Like any major business project, it is important to develop or adopt a set of best practices for the process of upgrading or migrating your data center power and cooling infrastructure. This guide has attempted to give you the basics you need to start your research and development process for your data center upgrade. Making Your Data Center More Energy Efficient This is the bottom line. When making your plans, evaluating technologies, and looking for hardware and software solutions, the goal for each item is to increase the efficiency of the data center. Don t forget the cumulative impact of changes and additions that you plan and be sure to look out for different people or groups who end up working at cross purposes in evaluating solutions and making purchasing decisions. To keep everyone on the same path, it is critical to have a plan. Standardizing the process will result in a successful project. Defining the Project There are a wide variety of tasks and sub projects that can be applied to your upgrade projects. It is important to make sure that all involved are on the same page and that the individual project milestones are well defined and lead toward the same goals. 50
Chapter 3 Defining the Goals Although the end goal is clear, the interim goals are incredibly important. The core issue of upgrading an existing data center is keeping existing services up and running. Thus, interim goals that allow effective staging of the upgrade process are critical. It is possible that energy consumption will increase at some points in the process as redundant systems are brought online before legacy systems are retired. But the end result of the process should be a more effective and efficient data center power and cooling architecture. The Costs of Not Upgrading As power costs continue to increase, there will be a hidden tax on inefficient data centers. This wasted money is difficult, in many cases, to quantify, but it is a problem nonetheless. With tight IT budgets the norm, wasting money because of a lack of progressive thinking in your data center is a waste that can be difficult to sustain. Beyond the simple cost issues, there are potential problems that can arise from failing to update your data center. The most significant is the inability to deliver the services and technologies that the business will demand in the near future to remain competitive or to create a competitive advantage. With the changes in IT technology resulting in much more dense data center IT loads, it is critical that IT be able to support those loads and have an agile and adaptable data center infrastructure. The inability to deliver the necessary power and cooling services in the data center will prevent the business from taking advantage of the technological edge that efficient IT can deliver. Layout & Cooling With your existing data center, it is most likely you currently have the traditional data center cooling model; a large room, possibly with a raised floor, where the temperature and humidity are maintained at a specific level throughout the space. The problem that we face is that when this model was originally implemented, the data center was a fairly static place. Changes were made slowly and gave the data center staff plenty of time and opportunity to make adjustments in the power and cooling for the space, as necessary. But older data centers are faced with two problems: the first is that they have changed over time and what was once considered state of the art is today a static environment that is unable to respond to the quickly changing technology environment that the modern data center has become. With rapid changes being made to the equipment in the data center, virtualization, on demand provisioning, and server consolidation, the demands on the power and cooling infrastructure have changed dramatically, and today s infrastructure needs to be flexible enough to respond to these ongoing changes. Second, older data centers often suffer from cooling and power problems that have been introduced by the slow change in IT loads over time. Delivery systems that were once running with excess capacity are now pressed to their limits and regularly operate out of their highest efficiency zones. The physical layout of the server racks that was initially appropriate now impedes the efficient cooling of the space, and the years have wrought changes in the infrastructure that simply don t allow for quick and easy fixes. 51
Chapter 3 Figure 4.1: Cooling patterns change as more demanding equipment is added to the data center. Changing the layout of an existing data center can be done in such ways as to address cooling problems and allow for more efficient utilization of existing cooling and power. At the same time, you can make room for upgraded equipment, for cooling and power and IT loads. Efficient Layout of the Data Center Legacy data centers are, in most cases, primarily making use of the room cooling architecture, as Figure 4.2 shows. More recently designed locations are more likely to be using the row and rack based cooling models that offer greater flexibility, allow for increased densities, and improve the efficiency of the data center. Figure 4.2: The three primary cooling architectures in use. Blue arrows indicate cooling supply paths. 52
Chapter 3 Let s look at the benefits of each model, from the perspective of a potential upgrade to an existing data center: The room cooling architecture The primary benefit of the room architecture is that in most circumstances it is already in place. By reorganizing the layout of the equipment within the data center, you can get greater efficiency if problems with the supply and return cooling paths were a significant issue in the past due to the equipment layout within the room. The row cooling architecture The cooling capacity can be targeted at the rows that need it, in the capacity that the row requires. There is no need for a raised floor installation, and different rows can have different amounts of cooling capacity applied to them based on the actual demand of that row. CRAC losses due to the need to cool large amounts of unrelated space are reduced, having the effect of reducing the amount of power consumed overall, improving overall energy efficiency. The rack cooling architecture As cooling and rack are completely integrated, the rack cooling architecture offers the greatest flexibility in deployment and load. The result is minimal loss in terms of CRAC utilization; power goes into cooling the specific rack, and the capacity of the rack mounted CRAC is completely dedicated to that role. Much higher density packaging is possible with this model than with either of the others. It is, of course, also possible to combine all three of these models in a single data center (see Figure 4.3). Changes in the data center may already have added row architecture cooling (in a dedicated area of the data center), in addition to the room architecture cooling already present. 53
Chapter 3 Figure 4.3: All three common cooling architectures in the same data center. This model is very likely to be an appropriate one for a large existing data center that wants to slowly migrate to the support of higher density IT loads. It allows for the existing infrastructure to be utilized while a more modern and efficient infrastructure is put into place. Implementing the High Density Zone With the upgrade and migration of the data center, support for new high density IT loads can be a major issue. This is why the combined cooling architecture model can be so effective. The key, however, to adding high density IT loads to an existing data center, without the expense of completely replacing the data center or upgrading the entire infrastructure, is the high density zone. 54
Chapter 3 Figure 4.4: The basic concept of the high density zone. High density zones have the advantage of concentrating the high density IT loads into a smaller area that can be controlled and monitored more closely than if the loads are spread out throughout the data center. To a large extent, the high density zone is treated as its own mini data center; it has its own cooling and power delivery and is kept separated from the rest of the data center environment. Ideally, it is completely thermally neutral, neither adding to nor easing the existing environmental load of the data center room. By using a row based cooling architecture, it is possible to utilize CRACs and modify the current room architecture to utilize a zone containment model to divert cooling to and waste heat from the high density zone. The Benefits of Containment Containment allows for hot and cold air streams to remain separate, increasing data center efficiency. When implementing a high density zone in an existing data center with existing perimeter air conditioning, containment is advised in order to maintain the environmental neutrality of the zone. 55
Chapter 3 Hot Aisle vs. Cold Aisle Containment Hot aisle and cold aisle containment models are the primary choices for building a highdensity zone in an existing data center. As we discussed in an earlier chapter, the hot aisle technique requires that the aisle be completely contained with a separate exhaust method for hot air so as not to contaminate the rest of the data center. Obviously, this solution (see Figure 4.5) requires a minimum of two rows of racks for the high density zone to allow the proper containment to be created. There are no special space requirements otherwise, as the space requirement for an even number of racks is the same as it would be for any pair of low density racks in the existing data center. Figure 4.5: Hot aisle containment for a high density zone. Rack containment has the benefit of not requiring complete pairs of rows for the containment model to work. It can be used for a single rack or single row of racks and is effective in that configuration. 56
Chapter 3 Figure 4.6: Rack containment for the high density zone. In the standard rack containment model, the hot air exhaust is contained to the rear of the rack with a front containment panel, and a rear panel or series of panels is used to channel the contained hot air. An optional configuration uses both front and rear containment to allow for a completely standalone high density rack. Utilizing the Pod Model If part of the data center migration plan includes expanding the data center, it is worth taking a look at the pod concept. In this model, the pod becomes the building block of the data center. Each pod is a complete entity containing: Power Cooling Servers Storage Networking 57
Chapter 3 This setup allows the capabilities of each component of the pod to be maximized, which makes it a very efficient model in terms of resources. Power and cooling is tailored specifically to the IT load of the pod, which results in minimal waste. Conceivably, different flavors of pods could be designed; servers could be one type of pod, while storage another, all with power cooling and networking components tailored to the specific needs of the pod. Growing the environment would mean adding pods of the appropriate nature. Pod technology has also proven successful for organizations that provide services, such as application and managed service providers. Some are looking to roll out containerized modular data centers for the ultimate in flexibility and efficiency as IT demand grows. Cold air containment systems (CACS) allow for the data center to make use of an existing room cooling architecture. They do so by utilizing techniques such as hanging plastic curtains to contain the cool air from a traditional perimeter cooling system where the cooled air is delivered via a raised floor plenum (vendors are now offering ceiling panels and aisle doors to deliver a much less home grown approach to implementing CACS). In this model, the delivery of the cooled air is controlled and the remainder of the room functions as the hot air return plenum. Although this solution is effective at delivering cool air where needed in an existing data center, it is extremely inefficient when compared with more modern and aggressive cooling techniques. There is a lower limit to the power consumption at each rack using this model, generally figured at 6kW/rack, and it can be very difficult to deliver adequate air pressure to the CACS due to issues of distance and potential airflow congestion in the raised floor due to cabling and power runs. Power The power question is one of being able to deliver sufficient power to the IT loads at the specified level of redundancy so that availability is assured. How this can be done in your data center is a matter of what your existing power infrastructure looks like. Upgrading Existing Systems In the initial design of the existing data center, there may have been provisions to scale the UPS systems. By scaling the existing system to meet the current needs, the power capabilities of the data center can be upgraded to a certain point. However, once the legacy system has reached its maximum power delivery capability, the problem of delivering power will once again need to be dealt with. If the choice is made at that point to completely replace the existing system, the expenditure to scale the system will likely have not yet been amortized. Remember that scaling the legacy UPS system is also a matter of scaling the redundant as well as the primary system. 58
Chapter 3 Introducing Scalable, Rack Based Solutions Scalable, rack based UPS systems will allow not just for the power delivery in that rack to grow but additional racks to be installed without impacting the existing systems power delivery. It is entirely possible that upgrading legacy systems will be a less expensive alternative in terms of capital outlay. The expense, however, isn t just budgetary; upgrading parallel bus UPS systems requires shutting down the systems, performing the upgrades, then testing and recommissioning the system before bringing the power to the data center back online. Thus, the data center will go offline for at least 24 hours in a normal upgrade process. Figure 4.7: Rack mounted UPS. Scalable rack based systems get around this problem completely by being self contained racks within the data center; the worst case scenario brings down a single rack while the UPS system within the rack has additional redundant modules added. In many cases, however, the IT load on the rack need never go down as part of this process. Simple UPS upgrades can be done during scheduled downtime or when the workloads are the lightest. 59
Chapter 3 Evaluating the Costs Hardware costs are a capital expense and can be budgeted for, but downtime is incredibly expensive due to the cascade effect that being offline has on business processes. Upgrading the legacy systems entails significant downtime for the data center. A simple 48 hours of downtime for a complete upgrade of the data center power supply could easily entail costs approaching half a million dollars; even more in a large enterprise business entity. With a scalable, rack based alternative, the cost of upgrading system capacity is likely to be less than 5% of the cost of upgrading legacy whole data center style UPS systems. This is a significant cost savings and represents a much more efficient utilization of IT resources. The same holds true for row based pods, as we discussed earlier. By adding a new zone to handle the new IT loads, disruption of the existing IT loads can be avoided. Management Management of the data center power and cooling infrastructure has traditionally been a somewhat haphazard affair. Although tools existed to manage specific devices, they rarely integrated with each other and almost never provided information of the ongoing behavior of the power and cooling systems relative to the impact of specific devices on environmental changes. Understanding what caused a specific problem was often just a series of best guess attempts; trying to evaluate the potential impact of adding highdensity racks to the data center at a particular location was basically impossible. Problems caused by changes in the data center infrastructure often led to unexpected problems with the power and cooling infrastructure as the data center staff lacked the ability to evaluate even the potential for problems after the IT loads had been moved, changed, or added. Adding the ability to reliably predict the behavior of the data center power and cooling infrastructure is a crucial component of building an energy efficient data center that is responsive to business needs and requirements. Capacity Management Capacity management for power and cooling has traditionally been done at a fairly high level. Data center managers knew what their total capacity was and took pains not to get too close to their maximum capacity based on the requirements of each piece of equipment within the data center. However, tools for managing the capacity of the data center at a more detailed level were lacking. Individual pieces of equipment often had vendor specific tools that allowed their equipment to be monitored, but the issue is one of being able to monitor the overall capacity of the data center, track environmental changes, and plan and evaluate the potential effect of different IT loads at different locations. 60
Chapter 3 Figure 4.8: Capacity management is a major component of service delivery. Capacity management and planning tools need to be able to provide data about ongoing operations of the data center as well as answer fairly specific questions about what is going on in th e data center and what the effect of changes would be: What are the best locations to install new equipment? Can new technology be deployed using the existing power and cooling infrastructure? Will relocating blade servers within the data center improve operational conditions? What impact will new equipment have on safety margins for power, cooling, and back up runtime? How close to the available limits of power and cooling is the existing configuration? At what point will additional capacity be required? Is the configuration able to maintain power and redundancy even in fault conditions? In order to determine the answers to these questions, it is necessary to monitor the data center at the appropriate level. For the most effective information necessary for accurate capacity planning and management, monitoring at the rack level provides the best choice. Monitoring the individual racks allows for detailed information about a very small area of the data center that is easily aggregated for an overview of the entire data center infrastructure and environment. 61
Chapter 3 Figure 4.9: Rack level management works best for capacity management and planning. In the past, data centers were often built with hugely over designed power and cooling delivery systems. This meant that there was very little need to have a detailed understanding of the data center operational environment. The existing IT technologies would be incapable of outgrowing the over sized systems in the data center. But as time went on, the data center IT technology and loads changed and what was once an over sized system gradually got to a point where it was being taxed to meet the needs of the IT loads. Conversely, many data centers also remained under utilized, operating far below their maximum capacity, yet because of the nature of their over sized supply systems, they cost far more to operate than they should. Both of these situations are unacceptable in today s business world. The rapid changes occurring in IT technology, which lead to fairly quick changes in IT equipment within the data center, means that the inability to monitor and plan for the changes within the data center quickly becomes a business liability, negatively affecting IT s ability to respond to business demands. By implementing rack level capacity management, it is possible to accurately monitor, manage, and evaluate the ongoing capacity issues within the data center. For building the most energy efficient data center possible, a comprehensive capacity planning solution is critical. Hardware Management and Monitoring Tools In addition to capacity management and planning software, physical infrastructure monitoring and management software should be considered a valuable addition to your management suite. These are tools that let you perform centralized management of the physical infrastructure of your data center. Ideally, these tools will be vendor neutral and allow simple plug and play with devices and equipment within the data center. Capabilities will range from real time device monitoring to video surveillance systems and physical security monitoring. 62
Chapter 3 Like any enterprise management application, the tools will be able to provide detailed reporting capabilities and allow the user to create alerts and automated processes regarding managed devices. Integration with a larger enterprise management system is usually accomplished by using SNMP, and the infrastructure management tools will also use SNMP to allow any appropriate device capable of using SNMP to integrate into the system. Figure 4.10: Infrastructure management tools give IT additional information about all aspects of the data center. Familiar consoles, like the one shown in Figure 4.9, allow IT staff to quickly get up to speed on the use of the infrastructure management tools. Direct integration with power and capacity planning tools is possible if software and hardware from the same vendor is selected. 63
Chapter 3 Power and Cooling Management as a Component of IT Management Systems No enterprise IT department is without an overall IT management tool or collection of tools. Depending upon how your IT management is configured and utilized, integration of the data center infrastructure tools may or may not make sense. At most, it is likely that you would want only a top level view of the data center delivered into a centralized IT management infrastructure. General status information and problem reports would tend to be the information that gets floated to the top level as detailed information on power and cooling status is of little use beyond the needs of the data center management team. Most IT management tools/consoles are able to accept information from other information gathering devices and applications. If your data center management tools are from different vendors than your overall IT management system, you ll need to deal with the vendor on the specifics of reporting information to another upstream tool. Server Upgrades, Consolidation, and Virtualization Throughout this guide, we have made regular mention of the rapidly changing technology that directly affects the data center. As we discussed in the first chapter, the technologies that have the most significant impact are consolidation, virtualization, and the server hardware technology that provides the greatest support for the software technologies. How Implementing Newer Server Technologies Can Improve Your Overall Energy Efficiency With the technology emphasis on greener computing, current generation server hardware, while not using any less power than its predecessors, uses that power more efficiently. The storage rack that was previously drawing n amount of power still draws the same amount of power but now supports two to three times the amount of storage. Multiprocessor servers still draw the same power, but now each CPU socket contains a multi core processor that significantly increases the processing power of the server. Server consolidation and virtualization takes better advantage of the hardware. The previous tendency of server hardware to spend most of its time sitting idle has been reduced to a large degree as the IT loads on that server have been increased due to the software technologies that allow the server to do more work. 64
Chapter 3 Hardware As IT departments have moved to higher density servers, the blade server has become commonplace in the data center. But as technology has advanced, the rack mounted servers and blade are now sporting multi core processors allowing the same space in the data center to do significantly more work. The downside of this increase in work is an increase in heat; hotspots where new technologies such as multi processor, multi core servers are being racked becomes commonplace. Although it is possible to build very efficient systems in terms of IT workload relative to power and cooling consumption, implementing these servers means that a detailed understanding of the cooling and power requirements of the data center must be maintained. Software Virtualization and consolidation have really become the hottest topics in IT. With virtualization becoming commonplace, the higher performance servers are far better utilized and IT is able to deliver improved services and be more flexible to the deployment and provisioning of new services. As this can be done now without adding servers to the data center, we go back to the basics of being able to support the IT load on the servers already in place. As the data center is provisioned to meet those demands and techniques such as right sizing are applied, the combination of server virtualization and the ability to match the power and cooling needs of these data center hotspots efficiently reduces the overall cost of data center operations. Conclusion The decision to upgrade your data center or migrate to new power and cooling technologies is a critical one that impacts the future of the business. Thus, the decision, process and planning stages all require detailed input from both the technology and business sides of the organization. Accurate planning for future growth and a dedication to doing it right will be important aspects of this process. Stop gap actions and a band aid approach to the issues of power and cooling in your existing data center are a recipe for disaster. Although there may be no specific point where systems stop working or are unavailable, the inability to quickly respond to a rapidly changing business environment has an ongoing negative effect on business. Upgrading the data center to allow this type of flexibility and efficiency without sacrificing availability provides the strongest possible base for future business development. 65
Chapter 3 Download Additional ebooks from Realtime Nexus! Realtime Nexus The Digital Library provides world class expert resources that IT professionals depend on to learn about the newest technologies. If you found this ebook to be informative, we encourage you to download more of our industry leading technology ebooks and video guides at Realtime Nexus. Please visit http://nexus.realtimepublishers.com. 66