Imperial College 3 March 2009 Green Data Centers: An Overview

Transcription

1 Imperial College 3 March 2009 Green Data Centers: An Overview Updated: 26 Feb 09

2 LBNL s Data Center findings Benchmarking Efforts Energy Reduction Approaches Green Data Centers: An Overview 2

3 Scenarios of projected energy use from EPA report to Congress Green Data Centers: An Overview 3

4 Data Centers are INFORMATION FACTORIES Data centers are energy intensive facilities Server racks now designed for more than 25+ kw Surging demand for data storage Typical facility ~ 1MW, can be > 20 MW Nationally 1.5% of US Electricity consumption in 2006 Projected to double in next 5 years Significant data center building boom Power and cooling constraints in existing facilities Green Data Centers: An Overview 4

5 Energy issues abound Over the next five years, power failures and limits on power availability will halt data center operations at more than 90% of all companies (AFCOM Data Center Institute s Five Bold Predictions, 2006) By 2008, 50% of current data centers will have insufficient power and cooling capacity to meet the demands of highdensity equipment (Gartner press release, 2006) Survey of 100 data center operators: 40% reported running out of power, cooling capacity, and to a lesser extent - space without sufficient notice (Aperture Research Institute) Green Data Centers: An Overview 5

6 The rising cost of ownership From , computing performance increased 25x but energy efficiency only 8x Amount of power consumed per $1,000 of servers purchased has increased 4x Cost of electricity and supporting infrastructure now surpassing capital cost of IT equipment Perverse incentives -- IT and facilities costs separate Source: The Green Uptime Data Institute, Centers: 2007 An Overview 6

7 Data center definitions Server closet < 19 M 2 Server room < 47 M 2 Localized data center <93 M 2 Mid-tier data center <465 M 2 Enterprise class data center 465+ M 2 Today s training focuses on larger data centers however, most principles apply to any size center. Green Data Centers: An Overview 7

8 Energy efficiency opportunities are everywhere Load management Server innovation Virtualization Better air management Better environmental conditions Move to liquid cooling Optimized chilled-water plants Use of free cooling Data Center Power Conversion & Distribution Server Load/ Computing Operations Cooling Equipment High voltage distribution Highly efficient power conversion Use of DC power Efficient redundancy strategies Alternative Power Generation On-site generation Waste heat for cooling Use of renewable energy/fuel cells Green Data Centers: An Overview 8

9 Many areas for improvement Cooling Air Management Free Cooling air or water Environmental conditions Centralized Air Handlers Low Pressure Drop Systems Fan Efficiency Cooling Plant Optimization Direct Liquid Cooling Right sizing/redundancy Heat recovery Building envelope Electrical Power UPS and transformer efficiency High voltage distribution Premium efficiency motors Use of DC power Standby generation Right sizing/redundancy Lighting efficiency and controls On-site generation IT equipment Power supply efficiency Standby/sleep power modes IT equipment fans Virtualization Load shifting Green Data Centers: An Overview 9

10 Demonstrations and benchmarking make the case Cooling Air Management Fan energy savings of 75%; eliminate hot spots; raise temperature in data center Free Cooling Contamination levels shown to be low; no humidity control issues Right sizing/redundancy modular cooling can be very efficient Controlling computer room air handlers wireless solutions; use of IT equipment sensors Electrical power UPS efficiency Efficiency varies between mfgs and with loading Use of DC power Elimination of conversion losses IT equipment Power supply efficiency power supply efficiency varies IT equipment fans fans can be eliminated with innovative cooling Green Data Centers: An Overview 10

11 Root causes of electrical system inefficiency Physical infrastructure is typically OVERSIZED Power requirements are initially greatly OVERSTATED Legacy INEFFICIENT equipment is incorporated IT equipment is on and not doing anything Multiple POWER CONVERSIONS each conversion loses some power and creates heat Power conversion efficiency is not optimized Green Data Centers: An Overview 11

12 Electrical infrastructure opportunities - Design IT Load based on IT Nameplate plus future growth IT equipment load is typically much less than nameplate (say 25% of nameplate) - Data Center floor area based on design IT load / (5-100 W/SF) Today, electrical loads are densely concentrated - UPS system rating based on design IT Load + (20 50%) IT design load was already over estimated by factor of 4 - Standby generator sizing based on UPS system rating x 2 ++ Excessive capital expenditure - Lighting based on (32 W / M 2 ) x data center floor area Lighting could be 10 W/M 2 Green Data Centers: An Overview 12

13 Potential savings abound 20-40% savings are typically possible Aggressive strategies better than 50% savings Paybacks are short 1 to 3 years are common Potential to extend life and capacity of existing data center infrastructure Some opportunities need to be integrated with infrastructure upgrades But is my center good or bad? Green Data Centers: An Overview 13

14 Benchmarking Efforts Green Data Centers: An Overview 14

15 Benchmarking for energy performance improvement: Energy benchmarking helps identify better performing designs and strategies. As new strategies are implemented (e.g. liquid cooling), energy benchmarking enables performance comparisons. Green Data Centers: An Overview 15

16 Benchmarking: following energy end use Electricity Flows in Data Centers HVAC system local distribution lines lights, office space, etc. uninterruptible load to the building, 480 V UPS PDU computer racks computer equipment backup diesel generators UPS = Uninterruptible Power Supply PDU = Power Distribution Unit; Green Data Centers: An Overview 16

17 Data center cooling and power conversion performance varies DCiE (Data Center Infrastructure Efficiency) < 0.5 Power and cooling systems are far from optimized Currently, power conversion and cooling systems consume half or more of the electricity used in a data center: Less than half of the power is for the servers DCiE Data Center Infrastructure Efficiency DCiE = Energy for IT Equipment Total Energy for Data Center Cooling & Power Conversions Server Load /Computing Operations Cooling & Power Conversions Server Load /Computing Operations Cooling & Power Conversions Server Load /Computing Operations Typical Practice DCiE < 0.5 Better Practice DCiE = 0.7 Best Practice DCiE = 0.85 Green Data Centers: An Overview 17

18 High level metric IT/total Average.57 Higher is better Source: LBNL Green Data Centers: An Overview 18

19 Inverse metric total/it (PUE) Average 1.83 Lower is better Source: LBNL Green Data Centers: An Overview 19

20 HVAC system effectiveness We observed a wide variation in HVAC performance Source: LBNL Green Data Centers: An Overview 20

21 Energy Reduction Approaches Green Data Centers: An Overview 21

22 The good news: Industry is taking action IT manufacturers; SUN, IBM, DELL Infrastructure equipment manufacturers Industry Associations are active: ASHRAE Green Grid Uptime Institute Afcom Critical Facilities Roundtable 7 X 24 Exchange SVLG Data Center Pulse Utilities and governments initiating programs to help Green Data Centers: An Overview 22

23 Industry collaboration Some bold steps to improve energy efficiency (and save money) Optimize chip to atmosphere cooling Broaden recommended and allowable ranges of environmental conditions Debunk contamination and ESD fears Move to liquid cooling 80 degrees operate anywhere without chillers Optimize power conversion - Utility to end use Use of DC Power or AC all the way to the chip? Integrate renewable sources without conversion Integrate computing equipment and the building Great opportunity in containers (just bigger computers) Green Data Centers: An Overview 23

24 IT equipment load Predicting IT loads Over sizing, at least initially, is common Implement modular and scalable approaches IT loads can be controlled Power supply options Server efficiency Software efficiency (Virtualization, MAID, etc.) Redundancy and back-up power Low power modes Reducing IT load has a multiplier effect Green Data Centers: An Overview 24

25 High efficiency servers Energy savings and potential utility incentive for installation of three new High Efficiency Servers. Green Data Centers: An Overview 25

26 Server virtualization Green Data Centers: An Overview 26

27 Server virtualization Energy savings and potential utility incentive for Server Virtualization. Number of servers before virtualization: 50. Number of servers after virtualization: 30. Green Data Centers: An Overview 27

28 Power chain opportunities Roadmap to maximize energy efficiency: UPS load factor Transformers and PDUs loading Standby generation Lighting DC power distribution On-site generation Energy management and awareness Green Data Centers: An Overview 28

29 Managing UPS efficiency: UPS sizing and loading significantly affect UPS efficiency: Maximize UPS Load Capacity Specify UPS system that has higher efficiency at 10 40% load capacity Specify efficient UPS topology Consider modular UPS (an option to maximize UPS load capacity) Green Data Centers: An Overview 29

30 Transformers and PDUs Specify high efficiency transformers Install Low voltage (LV) transformers outside the raised floor area Reduce number of PDUs (with built-in transformer) inside the data center Green Data Centers: An Overview 30

31 Air system design overview Data center layout Airflow configurations Distribution: overhead or underfloor Control: constant or variable volume Airflow issues Green Data Centers: An Overview 31

32 Air cooling issues Limitations on the data densities served (~200 W/sf) Air delivery limitations Real estate Working conditions Hot aisles are (should be) uncomfortably hot Costly infrastructure High energy costs Management over time Reliability Loss of power recovery Particulates Green Data Centers: An Overview 32

33 Design conditions at the inlet to IT equipment: recently revised by ASHRAE 2004 Version 2008 Version Low End Temperature 20 C (68 F) 18 C (64.4 F) High End Temperature 25 C (77 F) 27 C (80.6 F) Low End Moisture 40% Relative Humidity 5.5 C Dew Point (41.9 F) High End Moisture 55% Relative Humidity 60% Relative Humidity & 15 C Dew Point (59 F Dew Point) Green Data Centers: An Overview 33

34 Design conditions for inlet air to IT equipment Revised New CL 1 and 2 Recommended Range 2005, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. ( Reprinted by permission from ASHRAE Design Considerations for Data and Communications Equipment Centers. This material may not be copied nor distributed in either paper or digital form without ASHRAE s permission. Green Data Centers: An Overview 34

35 Environmental conditions take aways Use the entire ASHRAE recommended range in data center operation. Provide the warmest supply temperatures that satisfy the equipment inlet conditions. Control to the widest humidity range. Humidification does not protect against ESD, consider grounding and personnel practices in lieu of humidification. There is no scientific evidence that high humidity causes equipment problems Isolate equipment that needs tighter humidity or temperature control. Green Data Centers: An Overview 35

36 Data center layout Server airflow front to back or front to back and top are recommended Cold Aisle Hot Aisle 2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. ( Reprinted by permission from ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form without ASHRAE s permission. Green Data Centers: An Overview 36

37 Underfloor supply Underfloor Supply Cold Aisle Hot Aisle Only 1 pressure zone for UF! 2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. ( Reprinted by permission from ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form without ASHRAE s permission. Green Data Centers: An Overview 37

38 Overhead supply You can incorporate VAV on each branch Overhead Supply Cold Aisle Hot Aisle 2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. ( Reprinted by permission from ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form without ASHRAE s permission. Green Data Centers: An Overview 38

39 Typical temperature profile with underfloor supply Too hot Too hot Just right Too cold Elevation at a cold aisle looking at racks There are numerous references in ASHRAE. See for example V. Sorell et al; Comparison of Overhead and Underfloor Air Delivery Systems in a Data Center Environment Using CFD Modeling ; ASHRAE Symposium Paper DE ; 2005 Green Data Centers: An Overview 39

40 Typical temperature profile with overhead supply Too warm Too warm Just right Elevation at a cold aisle looking at racks Green Data Centers: An Overview 40

41 Overhead vs. underfloor Green Data Centers: An Overview 41

42 Airflow design disjoint IT departments select servers and racks each having airflow requirements Engineers size the facility fans and cooling capacity What s missing in this picture?== Green Data Centers: An Overview 42

43 What happens when the HVAC systems have less airflow than the servers? HVAC _ Supply < V Servers V Hot spots Higher hot aisle temperature Possible equipment failure or degradation Green Data Centers: An Overview 43

44 What happens when the HVAC systems have more airflow than the servers? HVAC _ Supply > V Servers V Least hot spots Higher air velocities Higher fan energy Reduced economizer effectiveness (due to lower return temperatures) Green Data Centers: An Overview 44

45 In a perfect world, variable flow supply and server fans HVAC _ Supply = V Servers V Partial flow condition Best energy performance but tricky to control Works best with aisle containment Green Data Centers: An Overview 45

46 How do you balance airflow? Spreadsheet CFD Monitoring, infrared thermography or other site measurements Using aisle containment CFD image from TileFlow Used with permission from Innovative Research, Inc. Infrared thermographic image from LBNL Green Data Centers: An Overview 46

47 Isolating hot or cold aisles Energy intensive IT equipment needs good isolation of cold inlet and hot discharge. Computer room air conditioner airflow can be reduced if no mixing occurs. Overall temperature can be raised in the data center if air is delivered to equipment without mixing. DX coils and chillers are more efficient with higher temperature differences. Higher temperatures increase air- and watereconomizer effectiveness. Green Data Centers: An Overview 47

48 Cold aisle containment, underfloor supply 35-38º C 70-75º 21-24º C Green Data Centers: An Overview 48

49 Cold aisle containment, overhead supply Cold Aisle Caps Cold Aisle Hot Aisle 2004, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. ( Reprinted by permission from ASHRAE Thermal Guidelines for Data Processing Environments. This material may not be copied nor distributed in either paper or digital form without ASHRAE s permission. Green Data Centers: An Overview 49

50 Hot aisle containment with in-row cooling Hot aisle lid Rack End cap In row fan-coil APC reprinted with permission Green Data Centers: An Overview 50

51 Combined hot and cold aisle containment CRAC/CRAH/AHU Verari Systems, reprinted with permission Green Data Centers: An Overview 51

52 Ten Major Points raised-floor Data Centres 1.Mixing 2.Recirculation 3.Short circuiting 4.Leakage 5.Dehumidification 6.Under-floor obstructions 7.Under-floor vortex 8.Venturi effect 9.Poor return air path 10. Missing server rack blanking plates Green Data Centers: An Overview 52

53 Air-side economizer Green Data Centers: An Overview 53

54 Air-side economizer issues Hygroscopic dust Design humidity conditions Odors Pollen Green Data Centers: An Overview 54

55 Integrated water-side economizer (WSE) Twb 5C You can use either a control valve or pump 6.6C 6.6C 7.8C <15.5C 9.4C Heat Exchanger in series with chillers on CHW side 6.6C 15.5C Green Data Centers: An Overview 55

56 Free cooling with towers and plate-&-frame heat exchanger Use cooling towers and heat exchanger to produce chilled water Supplant or turn off chiller Green Data Centers: An Overview 56

57 Free cooling take aways Air- and water economizers can save significant energy if properly designed and controlled. Air- economizers can increase energy usage if you have humidity controls. Air-economizers do increase particulates but these can be addressed with standard filtration. Water economizers should be integrated by installing free cooling heat exchanger in series with the chillers. Green Data Centers: An Overview 57

58 Control issues Temperature Control Humidity Control Airflow Control Feedback and Diagnostics IT Integration Others Green Data Centers: An Overview 58

59 Temperature control Design Conditions Maintain inlet conditions at servers between 20 C and 25 C. 15 C to 32 C allowable. At ~25 C two speed and variable speed server fans speed up (wasting fan energy). Best practice Provide feedback from racks. Hardwired or wireless EMCS sensors. Network data exchange with server on-board sensors. Reset supply temperatures upward to keep most demanding rack satisfied (but below 25 C). Can have local temperature zones with distributed CRAC/H units. Green Data Centers: An Overview 59

60 Best practices for Cx Specify thorough commissioning for data centers Be specific on roles and responsibilities Have an experienced Cx Coordinator on the site Hold weekly Cx meetings with all trades represented Carefully coordinate electrical, mechanical and control system testing Green Data Centers: An Overview 60

61 Cx Take aways No right way to perform commissioning Comprehensive Cx involves many steps Various approaches contractor, engineer, 3 rd party, combo Commissioning plan must be in specifications Controls programming should be reviewed Prefunctional and functional tests verify system and component function Trend reviews while not as comprehensive can identify problems Coordination is important during commissioning Green Data Centers: An Overview 61

62 Resources Green Data Centers: An Overview 62

63 Industrial Technologies Program Tool suite & training Metrics & energy baselining Qualified specialists Case studies Certification of continual improvement Recognition of high energy savers Best practice information Best-in-Class guidelines R&D - technology development EPA Metrics Server performance rating & ENERGY STAR label Data center performance benchmarking Federal Energy Management Program Best practices showcased at Federal data centers Pilot adoption of Best-in-Class guidelines at Federal data centers Adoption of to-be-developed industry standard for Best-in-Class at newly constructed Federal data centers Industry Tools Metrics Training Best practice information Best-in-Class guidelines IT work productivity standard Green Data Centers: An Overview 63

64 EPA Energy Star programs Energy Star server specification Draft 2 specification released August 15, 2008 Definition broadened to cover full range of servers found in data centers Includes power supply efficiency and power factor requirements Proposes idle efficiency Green Data Centers: An Overview 64

65 ASHRAE guidelines six books published more in preparation ASHRAE, Thermal Guidelines for Data Processing Environments, 2004, Datacom Equipment Power Trends and Cooling Applications, 2005, Design Considerations for Datacom Equipment Centers, 2005, Liquid Cooling Guidelines for Datacom Equipment Centers, 2006, American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc., Order from Green Data Centers: An Overview 65

66 Save Energy Now DC Pro tool suite Tools to define baseline energy use and identify key energy-saving opportunities Determine overall performance Benchmark subsystems Assess energy savings potential Track energy intensity improvement Provide quantification of key metrics including cost ($), primary energy (Btu), and carbon Green Data Centers: An Overview 66

67 DC Pro on-line profiling tool INPUTS Description Utility bill data System information IT Cooling Power On-site gen OUTPUTS Overall efficiency (DCiE) End-use breakout Potential areas for energy efficiency improvement Overall energy use reduction potential Green Data Centers: An Overview 67

68 DC Pro energy assessment tools Data Center Assessment Tools More accurate overall energy performance (baseline) of data center Performance of systems (infrastructure & IT) compared to benchmarks Prioritized list of energy efficiency actions and their savings, in terms of energy cost ($), source energy (Btu), and carbon emissions (Mtons) IT HVAC Power Chain On-Site Gen Servers Storage & networking Software Air Mgmt CRAC/CRAH AHU Chillers Free cooling Power conversion Redundancy Standby losses Lighting Renewables Co-gen Green Data Centers: An Overview 68

69 Summary Green Data Centers: An Overview 69

70 Ten Major Points for 21 st Data Centres 1.Use most efficient servers: processor power, blade servers. 2.Specify high-efficiency server power-supplies. 3.Analyze power systems efficiency: UPSs and PDUs 4.Employ server power management features. 5.Deploy server virtualization software. 6.Optimize airflow pathways: hot isle-cold isle isolation; server rack blanking plates; under-floor system tiles. 7.Use free cooling: broaden operational temperature boundaries. 8.Install variable capacity cooling: e.g., modularized in-row liquid cooling. 9.Optimize central plant efficiency 10.Install advanced monitoring and control Green Data Centers: An Overview 70

71 Questions? websites: data_centers.html Green Data Centers: An Overview 71