An Employee-Owned Corporation Data Center Thermal Management BICSI South Central December 2007 2003 All rights reserved.
Two Strategies in One A high density cooling strategy An energy conservation and money saving strategy Containment isolation between cool supply air and warm return air
Hot Aisle/Cold Aisle Ideal Implementation HOT AISLE COLD AISLE HOT AISLE COLD AISLE Fronts face fronts; backs face backs Open floor tiles in cold aisle; no open floor tiles in hot aisle Seal all floor tile cable access cut-outs
Bypass Air Solutions Close all unnecessary access floor openings Areas around perimeter walls, columns, behind cooling units, behind PDU s. Ensure all sleeves and penetrations in walls are properly sealed Add 40% - 100% cooling capacity by reclaiming wasted bypass air
Bypass Air Solutions 94 90 87 83 79 75 72 68 64 61 57 Enlarged Scale
Bypass Air Solutions 79 77 75 72 70 68 66 64 61 59 57 Enlarged Scale
Hot Aisle/Cold Aisle Ideal Implementation HOT AISLE COLD AISLE HOT AISLE COLD AISLE Fronts face fronts; backs face backs Open floor tiles in cold aisle; no open floor tiles in hot aisle Seal all floor tile cable access cut-outs Air delivery satisfies air consumption
Cooling Capacity Planning Electrical load X 0.28 = Tons AC UPS kva X Power Factor 45 = Tons AC Space 500 = Tons AC Add for total cooling capacity requirement Add redundancy requirement Planning Procedure from Bick Group WARNING Whatever you do will be wrong!
Locating Cooling Units Cold Aisle C U Hot Aisle C U Cooling units or air handlers should be at ends of hot aisles
Improperly Located Cooling Units 85 82 79 77 74 71 68 65 63 60 57 Enlarged Scale
Improperly Located Cooling Units 26 25 24 22 21 20 19 18 16 15 14 Enlarged Scale
Locating Cooling Units
Locating Cooling Units 8 ft min C U Cold Aisle Hot Aisle C U Cooling units or air handlers should be at ends of hot aisles Don t locate open floor tiles too close to CU s
The Cause of Unbalanced Flow Distribution CRAC Velocity decreases, pressure increases
Effect of Plenum Height
Effect of Tile Open Area
Use of Variable Tile Open Area
Locating Cooling Units 8 ft. min C U Cold Aisle Hot Aisle C U Cooling units or air handlers should be at ends of hot aisles Don t locate open floor tiles too close to CU s Close openings in cabinet rows
Cabinets and Cable Pathway HOT AISLE COLD AISLE HOT AISLE COLD AISLE Cable Tray Electrical, etc. Cable Tray Electrical, etc. Data cable tray should run down hot aisle and be ¾ from bottom of floor tile Power cables and conduit should be below and in the cold aisle
High Density Air Cooling Paradigm Shift The cabinet is a box for housing servers NOT! The cabinet is an integral architectural feature of the data center space for securing the isolation between supply air and return air
Manage Air Inside the Cabinet Deliver chilled air to critical point(s) of use Prevent hot air re-circulation Use filler panels Seal off sides of equipment Use side panel(s) in bayed cabinets if required
Blocks Re-Circulation Through Cabinets 138 127 116 106 95. 84. 74. 63. 53 Enlarged Scale
Manage Air Inside the Cabinet Deliver chilled air to critical point(s) of use Prevent hot air re-circulation Use filler panels Seal off sides of equipment Use side panel(s) in bayed cabinets if required Remove hot air Rely on server fans and high flow doors or no doors Avoid cable congestion Beware fans
Top Mounted Fans Cause Hot Spots Dell White Paper: Rack Impacts on Cooling for High Density Servers The addition of rack fans or fan trays is not recommended. In some cases, additional top mounted rack fans have actually impeded server thermal performance.
Top Mounted Cabinet Fans Cause Hot Spots 138 127 116 106 95 84 74 63 53 Enlarged Scale
Top Mounted Cabinet Fans Cause Hot Spots 125 116 107 98 89 80 71 62 53 Enlarged Scale
Top Mounted Fans Cause Hot Spots 400-500 CFM: Typical cabinet roof fan 1350 CFM: Highest standard roof fan on the market 900 1000 CFM: Typical rear fan door ratings 1820 CFM: Four IBM eserver Blade Centers 2400 CFM: Six Dell PowerEdge 1955
Case 1: Temperature Distribution at Central Plane of Test Bed 138 127 116 106 95. 84 74 63 53 Enlarged Scale
Curing Hot Spots with Ducted Exhaust 90 87 84 80 77 74 71 68 64 61 58 Enlarged Scale
Curing Hot Spots with Ducted Exhaust 79 77 75 73 71 69 67 65 63 61 59 Enlarged Scale
Curing Hot Spots with Ducted Exhaust 30 28 27 26 24 22 21 19 18 17 15 Enlarged Scale
Curing Hot Spots with Ducted Exhaust 190 253 317 380 444 507 570 634 676 Enlarged Scale
Curing Hot Spots with Ducted Exhaust
Curing Hot Spots with ducted Exhaust 101 97 93 88 84 80 76 72 67 63 59 Enlarged Scale
The Problem Heat densities are exceeding the levels that can be cooled by air delivered through a perforated floor tile in front of a server cabinet, according to conventional wisdom
Cabinet Cooling Requirements Increase Blade Server IBM eserver Blade Center 14 servers, dual-processors, 9 RMU Sun Blade 8000 10 servers, quad-processors, 19 RMU Dell PowerEdge 1955 10 servers, dual-processors, 7 RMU HP BladeSystem c-class 16 servers, dual-processor, 10 RMU 42U Qty. Watts BTU/h Tons A/C M 3 h 4 20,068 68,492 5.7 0.86 2 15,000 51,195 4.3 ~ 6 21,570 73,618 6.1 1.13 4 20,056 68,451 5.7 ~
High Density Application Hot air is physically segregated from cold air in return air duct Rear door is sealed to prevent exhaust air from leaking into room Air Dams in front of cabinet prevent air recirculation inside cabinet Room A/C is easy to manage, with little regard to concentrated heat loads Cabinets can be placed and oriented any way desired Cold air delivery can be shared throughout room with fewer zone issues
Isolate Hot Air In The Cabinet Exploit the laws of physics from the Bernoulli Equation Inverse relationship between velocity and pressure
Isolate Hot Air In The Cabinet 73 70 68 65 63 60 58 55 53 Enlarged Scale
Data Source Third Party Test Servers (6) 7U IBM eserver BladeCenter chassis (14) dual-processor HS-20 blade servers (Intel Itanium 3.2 GHz) (4) 2000W (nameplate) power supplies (2) 350CFM nominal output variable-speed centrifugal blowers Power Source (2) 60A 208V 3-phase circuits Environment Active data center Average intake air temperature of 71 F (22 C)
Test Configuration INTERMEDIATE STRINGERS ADDED HOT AIR 22 1/2" x 22 1/2" Hot Air Duct V T dp 23" x 77" SOLID DOOR T T 24" x 48" Cabinet T T T T " " AISLE T P V TEMPERATURE STATIC PRESSURE VELOCITY HOT AIR T T T T T T COLD AIR COLD AIR COLD AIR T COLD AIR
Data Source Third Party Test Servers (6) 7U IBM eserver BladeCenter chassis (14) dual-processor HS-20 blade servers (Intel Itanium 3.2 GHz) (4) 2000W (nameplate) power supplies (2) 350CFM nominal output variable-speed centrifugal blowers Power Source (2) 60A 208V 3-phase circuits Environment Active data center Average intake air temperature of 71 F (22 C) 23kW Actual Measured Heat Load
Remove Hot Air From The Room 73 70 67 65 63 60 58 55 53 Enlarged Scale
Standard Data Center Temperature Variation 86 83 81 78 75 72 70 67 64 62 59 Enlarged Scale
No Temperature Variation 79 77 75 73 71 69 67 65 63 61 59 Enlarged Scale
High Return Air Temperatures Are GOOD Cooling Unit Supply Air Temperature Return Air Temperature Cooling Capacity Liebert FH200C 60 F 70 F 7.8 tons Liebert FH200C 60 F 90 F 15.5 tons Liebert FH200C 60 F 105 F 20.7 tons Liebert FH600C 60 F 70 F 23.0 tons Liebert FH600C 60 F 90 F 46.0 tons Liebert FH600C 60 F 105 F 61.3 tons
Ducted Exhaust with Ceiling Return Plenum All hot spots eliminated Total temperature variation in room is 6.7 C, so supply air can be raised from 11 CF to 18 C
New Construction Energy Saving Strategy Case Study Data Center: 800 kw actual heat load Free space return air Liebert FH422 (2 ea) Liebert FH529 (7 ea) Liebert FH600 (2 ea) Liebert FH740 (1 ea) TOTAL Isolated return air path Liebert FH529 (3 ea) Liebert FH600 (3 ea) Liebert FH740 (1 ea) TOTAL 40 tons 210 tons 60 tons 40 tons 350 Tons 90 tons 90 tons 40 tons 220 tons
Effect of Complete Isolation Best Practice CRAC w/ Economizer Evaporative Air Economizer Delivered 68-77 77 77 Supply 52-55 77 77 Water 42 65 65 Approach 37 60 77 Free Hours???
New Construction Results 5 Megawatt data center in San Francisco Bay Area 9 megawatts One year cooling energy use Chilled water CRAC units with water economizer and hot and cold aisles ($669,778) 2 megawatts One year cooling energy use Chilled water air handler with evaporative air economizer and isolation ($155,965) - Analysis by McKinstry Company
Conclusion Isolate return air from supply air De-couple point of air delivery from heat load for high densities Exploit economizer benefits Do your homework
Homework Air-Cooled High-Performance Data Centers: Case Studies and Best Methods, www.intel.com/it Utilizing Economizers Effectively in the Data Center, www.liebert.com Kyoto Cooling, www.uptimetechnology.nl Reducing Data Center Energy Consumption with Wet-Side Economizers, www.intel.com/it
An Employee-Owned Corporation Thank You Ian Seaton iseaton@chatsworth.com 2003 All rights reserved.