Energy Efficient Thermal Management of Data Centers

Energy Efficient Thermal Management of Data Centers

Yogendra Joshi l Editors Pramod Kumar Energy Efficient Thermal Management of Data Centers

Editors Yogendra Joshi G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA, USA Pramod Kumar G.W. Woodruff School of Mechanical Engineering Georgia Institute of Technology Atlanta, GA, USA ISBN 978-1-4419-7123-4 e-isbn 978-1-4419-7124-1 DOI 10.1007/978-1-4419-7124-1 Springer New York Dordrecht Heidelberg London Library of Congress Control Number: 2012931944 # Springer Science+Business Media, LLC 2012 All rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York, NY 10013, USA), except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed is forbidden. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. Printed on acid-free paper Springer is part of Springer Science+Business Media (www.springer.com)

Preface Within the past decade, data centers have taken on an increasingly central role in the progress of two parallel areas of burgeoning global societal interest: Internet delivered information technology (IT) services and telecommunications. The terms data centers or server farms have been widely, and interchangeably, used to describe large facilities, sometimes over 40,000 m 2 of floor space, that house computing and storage equipment for IT, and switching equipment for telecommunications. Larger facilities require electrical power delivery in the range of tens of MW, which is converted into heat by the equipment. The need for keeping data centers operational with no downtime for many critical applications, such as banking, electronic commerce, stock transactions, and mobile communications, necessitates deploying significant redundancy in the power delivery and cooling associated with these mission critical facilities. Effective thermal management of data centers is thus essential for their proper operation and has been a key focus since the design of the earliest facilities. As computing capabilities advance, heat dissipations within IT cabinets have increased from a few kw to well over 20 kw within the past decade, particularly in high-performance server cabinets. To enable the successful deployment of this hardware, significant advances in cooling technologies, such as the use of indirect or direct liquid cooling, have also taken place. The increasing cost of energy has brought a new focus to the design and operation of data centers. A 2007 report to the US Congress showed that the use of electricity by data centers had doubled from 2000 to 2005 in the USA, and such increase was unsustainable. Benchmarking studies undertaken by the US Department of Energy in California s Silicon Valley data centers showed that as much as 30 50% of the overall electricity use of a data center was for cooling. A number of energy saving concepts, such as broadening the environmental temperature and humidity requirements envelope, and using outside air for cooling when appropriate, also termed free cooling, have since been incorporated in the designs of newer facilities, and recent estimates show a drop-off in the rate of increase in energy usage by data centers. v

vi Preface Several professional societies have been at the forefront of identifying issues relating to the energy management of data centers. Most notably, the American Society for Heating Refrigeration and Air-Conditioning (ASHRAE), in partnership with industry stakeholders, has focused on environmental guidelines for the proper operation of these facilities. Several revisions have been made to allow a broadening of inlet air temperature and humidity requirements, as they have a prominent impact on operational energy costs. Within the past 5 years, ASHRAE has also published a series of books identifying cooling technologies available to the data center designers and industry best practices for a disparate group of professionals, including architects, builders, mechanical engineers, electrical engineers, and IT professionals. Presently, data centers are managed based on accrued experience or best practices, which often lead to an overly conservative thermal management approach, at the cost of wasted cooling resources. Reducing energy consumption and carbon footprint of data centers, on the other hand, requires a fundamental principles based approach. The data center manager now has to supplement prior experience with conceptual understanding of heat transfer, fluid flow, thermodynamics, computational modeling, metrology, and data acquisition and processing. This book aims to provide such an in-depth understanding by drawing on the expertise of several researchers and industry practitioners. We focus on broader analysis and solution methodologies, rather than specific solutions. The book focuses on six themes. Understanding, monitoring, and controlling airflows and thermal fields in data centers are discussed in Chap. 2. Power delivery, distribution, and management within the data center are addressed in Chaps. 3 and 4. Energy-efficient operation of a data center requires numerous sensors. The acquisition, processing, and use of such data is an important activity. Quantification of energy efficiency also requires identification of suitable metrics. Another important thermodynamic consideration is that of exergy destruction, which can be minimized through careful design and operation. These concepts are addressed in Chaps. 5 7, 9, and 11. Computational modeling of airflow and temperature patterns within data centers through full field approaches, and rapid simulations are discussed in Chaps. 7, 8, and 10. A number of advances in cooling technologies are being incorporated in data centers to achieve both improved cooling capabilities and energy efficiency. For applications such as high-performance computing, it is clear that air cooling will not be sufficient. A transition to liquid cooling is already taking place, with the added benefit of the possibility of waste heat recovery, as discussed in Chap. 12. A number of other thermal and energy management advances are discussed in Chap. 13. The editors hope that this book will serve as a useful compilation of in-depth information in this area of growing interest. The book is aimed at the various stakeholders in the data center industry as well as the academic community. The former category includes critical facility designers, cooling, IT, and telecommunications equipment manufacturers, data center end users, and operators.

Preface vii We hope that established and new academic researchers involved in thermal design, power delivery, and cloud computing will also find the book useful. The Editors are grateful to all the chapter contributors for preparing authoritative compilations of the state of the art and recent progress in their selected topical areas. Finally, we acknowledge the help and guidance extended by the publisher through the editorial assistance of Steven Elliot and Merry Stuber. Atlanta, GA, USA Yogendra Joshi Pramod Kumar

Contents 1 Introduction to Data Center Energy Flow and Thermal Management... 1 Yogendra Joshi and Pramod Kumar 2 Fundamentals of Data Center Airflow Management... 39 Pramod Kumar and Yogendra Joshi 3 Peeling the Power Onion of Data Centers... 137 Sungkap Yeo and Hsien-Hsin S. Lee 4 Understanding and Managing IT Power Consumption: A Measurement-Based Approach... 169 Ada Gavrilovska, Karsten Schwan, Hrishikesh Amur, Bhavani Krishnan, Jhenkar Vidyashankar, Chengwei Wang, and Matt Wolf 5 Data Center Monitoring... 199 Prajesh Bhattacharya 6 Energy Efficiency Metrics... 237 Michael K. Patterson 7 Data Center Metrology and Measurement-Based Modeling Methods... 273 Hendrik F. Hamann and Vanessa López 8 Numerical Modeling of Data Center Clusters... 335 Bahgat Sammakia, Siddharth Bhopte, and Mahmoud Ibrahim 9 Exergy Analysis of Data Center Thermal Management Systems... 383 Amip J. Shah, Van P. Carey, Cullen E. Bash, Chandrakant D. Patel, and Ratnesh K. Sharma ix

x Contents 10 Reduced Order Modeling Based Energy Efficient and Adaptable Design... 447 Emad Samadiani 11 Statistical Methods for Data Center Thermal Management... 497 Ying Hung, Peter Z.G. Qian, and C.F. Jeff Wu 12 Two-Phase On-Chip Cooling Systems for Green Data Centers... 513 John R. Thome, Jackson B. Marcinichen, and Jonathan A. Olivier 13 Emerging Data Center Thermal Management and Energy Efficiency Technologies... 569 Yogendra Joshi and Pramod Kumar Index... 613

Contributors Hrishikesh Amur Center for Experimental Research in Computer Systems, School of Computer Science, College of Computing, Cullen E. Bash Hewlett Packard Laboratories, Palo Alto, CA, USA Prajesh Bhattacharya Lawrence Berkeley National Laboratory, Berkeley, CA, USA Siddharth Bhopte Small Scale System Integration & Packaging Center, Binghamton University State University of New York, Binghamton, NY, USA Van P. Carey Department of Mechanical Engineering, University of California, Berkeley, CA, USA Ada Gavrilovska Center for Experimental Research in Computer Systems, School of Computer Science, College of Computing, Hendrik F. Hamann IBM Thomas J. Watson Research Center, Yorktown Heights, NY, USA Ying Hung Department of Statistics and Biostatistics, Rutgers, The State University of New Jersey, Piscataway, NJ, USA Mahmoud Ibrahim Small Scale System Integration & Packaging Center, Binghamton University State University of New York, Binghamton, NY, USA Yogendra Joshi G.W. Woodruff School of Mechanical Engineering, xi

xii Contributors Bhavani Krishnan Center for Experimental Research in Computer Systems, School of Computer Science, College of Computing, Pramod Kumar G.W. Woodruff School of Mechanical Engineering, Hsien-Hsin S. Lee School of Electrical and Computer Engineering, Vanessa López IBM Thomas J. Watson Research Center, Yorktown Heights, NY, USA Jackson B. Marcinichen Laboratory of Heat and Mass Transfer (LTCM), École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland Jonathan A. Olivier Laboratory of Heat and Mass Transfer (LTCM), École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland Chandrakant D. Patel Hewlett Packard Laboratories, Palo Alto, CA, USA Michael K. Patterson Eco-Technology Program Office, Intel Corporation, Hillsboro, OR, USA Peter Z.G. Qian Department of Statistics, University of Wisconsin-Madison, Madison, WI, USA Emad Samadiani G.W. Woodruff School of Mechanical Engineering, Bahgat Sammakia Small Scale System Integration & Packaging Center, Binghamton University State University of New York, Binghamton, NY, USA Karsten Schwan Center for Experimental Research in Computer Systems, School of Computer Science, College of Computing, Amip J. Shah Hewlett Packard Laboratories, Palo Alto, CA, USA Ratnesh K. Sharma Hewlett Packard Laboratories, Palo Alto, CA, USA John R. Thome Laboratory of Heat and Mass Transfer (LTCM), École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland Jhenkar Vidyashankar Center for Experimental Research in Computer Systems, School of Computer Science, College of Computing, Chengwei Wang Center for Experimental Research in Computer Systems, School of Computer Science, College of Computing,

Contributors xiii Matt Wolf Center for Experimental Research in Computer Systems, School of Computer Science, College of Computing, C.F. Jeff Wu School of Industrial and Systems Engineering, Sungkap Yeo School of Electrical and Computer Engineering,