Liquid Cooling Solutions for DATA CENTERS - R.M.IYENGAR BLUESTAR LIMITED.
Presentation Goals & Outline Power Density Where we have been- where we are now - where we are going Limitations of Air Cooling Air Distribution, Effectiveness and Heat Removal What Does ASHRAE Say? Unified System Solutions Energy Efficiency
Common Data Center Scene Have you seen data centers that look like this?
Thermo photo Analysis
IT Equipment Trends This area is for the equipment footprint only! Power density is diluted when the entire data center area is added. *from The Uptime Institute (http://207.201.136.39/tuipages/whitepapers/tuiheat1.0.html)
Issues in the Data Center The rise of heat densities at the microprocessor level (watts)
Are We Hitting the Wall? The failure rate at the top third of the rack is three times greater than at the bottom. Mr. Kenneth Brill, Executive Director: The Uptime Institute, March 2003 DataCenter Dynamics Conference) The wall is at 3.5 KW/Cabinet (with raised floor) Without raised floor, 2.5 KW/Cabinet max. Steve Spinazzola, RTKL, Oct. 2004, 7x24
How Do We Remove the Heat? Water Water has 3500 times more heat capacity than air by volume Air Overhead Under floor
Overhead Air Distribution Limitations to capacity due to duct size / space constraints Higher cost for ductwork versus underfloor system
Underfloor Air Distribution Increased floor depth for higher flows Limitation of perforated tiles (400 cfm = +- 4 kw/rack) UF obstructions Air Misdistribution underfloor velocity/static pressure Air plumes
The Cause of Flow Misdistribution
Above floor Plumes Rack total airflow: 350cfm Max inlet temp: 16C/61F Rack total airflow: 700cfm Max inlet temp: 18C/64F Rack total airflow: 1400cfm Max inlet temp: 28C/82F
Underfloor Plumes Row Velocity vectors & pressure (inch wg ) under the raised floor Column 1 kw / Cabinet 24 Raised Floor
HVAC Effectiveness Bypass Airflow Air Distribution Efficiency 70% of cooling supply air is wasted and bypasses back to return system.
ASHRAE Standards Key Points of ASHRAE TC9.9 Equipment Environmental Specifications Equipment Room Airflow Standardize Measuring / Monitoring Points Equipment Airflow Protocol Syntax Equipment Manufacturer s Heat Reporting
Equipment Room Airflow ASHRAE Standards
The New Paradigm Extraction Versus Dilution
AIR COOLED EXTRACTION SYSTEM
POINTS TO CONSIDER Effective for standard racks up to 6 KW only No cabinet doors. Cable management behind servers in hot aisle. Limit air bypass use rack blanking panels and seal all floor cutouts. Limit mixing between hot and cold aisles. Partition panels above racks and doors at aisle ends. Sprinklers / FP piping in each aisle.
Rack Cooling Unit Options Down flow Units Centrifugal fan drive EC fan drive With DX system With chilled water
High Density Cooling Solution
Heat Density 25 20 Cabinet 15 10 5 0 Pre P3 Processor P3 Mobile Blade P4 P4 Blades/P4 Multi- Single Processor Processor 1U & 2U NextGen Ultra Dense Blade & 1U Server High-End Enterprise Servers
Cooling Hierarchy Cooled room, open racks Plenum distribution, server racks Forced air racks Water cooled racks Water cooled CPU 1 kw 1-3kW 3-6kW 6-20kW 20-35kW 35kW+
Background to the Need for Water Cooling Rising heat densities Limitations of air cooling Focus on heat removal at source Part load operating conditions Energy efficiency
COOLING SYSTEMS CLOSER TO THE LOAD. Mounting the cooling modules as close as possible to the source of heat such as placing them directly above, alongside, or within high-density racks reduces the distance that the fans must move air. This can provide up to 70% savings of the energy required to move the air.
OPEN COOLING ARCHITECTURE. The room s air volume is then used as thermal storage to ride through short power outages. Cool room air enters the front of the server rack cabinet and absorbs heat as it passes over the electronics. It then exits the cabinet, moves back into the room, and recirculated through CRAC.
Closed cooling architecture Closed architecture systems involve extraction of room air in server rack cabinets that are completely sealed from room air and environmental conditions. In a closed design, air is circulated through the electronics and passes through a liquid-toair heat exchanger located in the cabinet and back to them at room temperature.
Rack Cooling Unit Options CDU Cooling Distribution Unit Rear Door Heat Exchangers Chilled Water Cooling
CDU with Rear Door Heat exchanger Heat removal at source, rejected to water not room Rear door heat exchangers on back of high heat load cabinets without taking up any floor space High density racks can be added without need to upgrade CRAC units Easily retrofitted as high density racks added Eliminates low air temp setpoints to compensate for high density racks Only relatively small chilled water plant required CDU limits volume of chilled water in computer room Reduces number / capacity of CRAC Units required Reduces floor space required for CRAC Units
Data Racks & Rear Door Heat Exchangers Industry Standard Data Racks 600 wide 20kW Cooling 800 wide 30kW Cooling 6 Hot Swap Fans Retro-fittable to other Racks Without swap fans for >6 kw upto 10 kw
Typical Installations
System Concept Rear Door Heat Exchangers Cooling Distribution Cabinet 120-150kW External chiller Up to 6 racks (or 12 racks at lower loads)
Minimal Thermal Impact Server Implementation using CDU & Rear Door Heat Exchangers 20-25 C 20-25 C 20-25 C 20-25 C 20-25 C 20-25 C 20 kw 20 kw 20 kw 20 kw 20 kw 20 kw CDU 20-25 C 20-25 C 20-25 C 20-25 C 20-25 C 20-25 C
CDU Cooling Distribution Unit CDU120/150 Main Components 120 to 150kW (409,450 511,815 BTU/Hr) capacity. Serves 1 to 6 rear door heat exchangers (RDHx). 380-480v 50/60Hz global operation (+220v option). Run/Standby pumps. Run/Standby control valves. Energy saving pump speed control. 240L/M (63.4 GPM) flow capacity (40L/M per RDHx). Dew point control. Inbuilt manifold system with flow balancing. Quick release, self sealing connections for data rack hoses. Easy fill system. UL approved
Hose Connections Process (to RDHx) Secondary Circuit Connection: Quick release couplings for clean efficient installation Fool-proof input and output connections Tidy hose arrangement for accessibility. Primary Chilled Water Connections (1.1/2 Hose tails) Secondary Circuit Rack Return Connections (3/4 Hose tails) Secondary Circuit Rack Flow Connections (3/4 Hose tails)
Controller & Display Temp (T3) & Hum Sensor Air Eliminator Rack 1 Control Valves Heat Exchanger Speed Control Rack 2 Rack 3 Rack 4 Chilled Water Flow / Return T1 PS3 PS4 # Flowmeter Bypass Valve Air Eliminator Filter Distribution manifolds Flow Control 40L/M Quick Release Couplings Rack 5 Rack 6 Filter # Redundant 2-way valve Level Sensor. T2a T2b PS1 Pump P1 PS2 40L/M 40L/M 40L/M Rack 3 Rack 4 # # Redundant pump Pump # # P2 40L/M 40L/M Rack 5 Rack 6 Fill point Priming Pump P3 DATE 18/05/07 DR N DSM TITLE CDU PIPE SCHEMATIC DRG. No. A3NAD371 Iss. A
Modular unit of 20kW Can serve one RDHx or up to four 5kW horizontal spot cooling cassettes. Can be installed in isolation or in purpose cabinet (pictured) for up to 120kW. Purpose built cabinet includes chilled water manifold system & power distribution module. Easy fill system Dew point control Quick release, self sealing connections for data rack hoses Modular CDU
Typical Manifolds
Graphical Display and Keypad set point 18.0 C Power On indicator (red) Graphical display area dw o/r act. temp.0 C 18.4 C Level 1 alarm indicator (flashing amber) Level 2 alarm indicator (flashing red) ESC button -To return to System (default) screen Arrow buttons For menu navigation and value adjustment OK button To enter or accept the displayed value or command & to acknowledge alarms
Connectivity via Webserver, Modbus & Echelon BMS integration to facilities and connectivity to IT via IP & SNMP.