Pacific Northwest National Laboratory (PNNL)

Transcription

1 Energy Smart Data Center Phase I Andrés Márquez, Tom Seim, Kevin Regimbal, Steve Elbert, Kevin Fox, and Blanche Wood Pacific Northwest National Laboratory (PNNL)

2 Introduction Our current Environmental Molecular Science Laboratory (EMSL) HP cluster consumes 0.91MW power for a dual Itanium (1.5GHz) cluster delivering 11.8TF Linpack with 590KW compute power. Our estimates are: Cluster Thermal Management: Given a Coefficient Of Performance (COP) defined as the ratio of the cluster s power consumption vs. cooling infrastructure s power consumption results in COP=1.8 for 320kW for air-cooling Cluster Power Delivery (Rectifiers, UPS, Inverters, Feed, PDU, VR): Losses of +40% (240KW) of 590kW compute power 2

3 Phase I Objectives Adapt spray cooling to one rack of HP RX2600 servers Demonstrate globally spray cooled dual Opteron blade Develop physical feasibility demonstrator for 3-D memory module in package Operate both systems in PNNL s data center for one year Establish performance and reliability data for the RX2600 rack Use data and experiences to guide Phase II and III 3

4 Motivation for Spray Cooling Increase energy efficiency of large data centers and supercomputers Increase density of computer nodes in a data center (aka densification ) Increase reliability of the data center Better component reliability through lower junction temperatures Reduce temperature stresses by lowering temperature variations in the data center Eliminate some of the cooling infrastructure components 4

5 Why Spray Cooling is More Efficient Than Pool Cooling Critical Heat Flux: point at which heat transfer efficiency drops because a vapor barrier is formed Impacting spray droplets puncture and remove the vapor bubble Nucleate bubble size at puncture is smaller Vapor is entrained by droplets forming secondary nuclei ISR angular spray nozzles move vapor bubbles away from surface 5

6 Spray Cooling Possible Mechanisms Tan, S. et-al, Simulation of Spray Cooling Systems With Phase Change, U. Central Florida 6

7 Single-Phase Cooling Regime Rebound heated liquid droplets Nozzle Atomized droplets Heated Surface Heat Flux 7

8 Two-Phase Cooling Regime Mixture of liquid and gaseous droplets Nozzle Heated Surface Heat Flux 8

9 3-D D Spray Cooled Computer Boudreaux and Eden, Thermal analysis of spray cooled 3-D interconnected diamond substrate MCMs, IEEE TRANS Device and Mat. Rel., Vol. 4, No. 4, Dec

10 Ultimate Compute Density 10

11 Air Cooling: Rack All servers running High Performance Linpack Air-cooled racks are 8.2 kw thin node racks Floor tile air exit temp range of o C Best-case rack location in the facility Air-cooled rack is very sensitive to location in the facility Max recorded rack door temp of 39 o C Exhaust air up to 26 o C above floor tile air exit temp Large temp gradient over rack door Worst-case rack location in the facility Max recorded rack door temp of 49 o C Exhaust air up to 36 o C above floor tile air exit temp Large temp gradient over rack door 11

12 Facility Air Flow HP rx2600 rack Full board cooling SprayCool HP rx2600 rack needs, 2005 PNNL perf tiles, 2005 Source: Rasmussen, N., Cooling Strategies for Ultra-High Density Racks and Blade Servers, APC White Paper #46, 12

13 PNNL s s Thrust: Energy Smart Data Center (ESDC) Spot cool commodity processors (HP RX2600) Demonstrate densified SprayCool blades Launch life-cycle cost investigation (focus energy efficiency) Demonstrate small Global SprayCool cluster Phase I Increase ease-of-use (plug-nplay SprayCool Rack) 1U Server Conversion (HP RX1620) 1U Server Designed for SprayCool Global SprayCool blades with increased memory density Update to TCO Model Investigate Large-Scale Coherent Memory designs enabled by SprayCool Phase II Facility instrumented to measure power, cooling, noise HPC cluster of SprayCool racks w/spraycool Servers Assess free cooling, spraycooling, other cooling strategies Instrument and report power savings Develop physical feasability demonstrator for 3-d memory module in package Phase III 13

14 Thermal Management Unit ( Mustang Liberator ) 14

15 TMU Front View 15

16 Spray Spot Cooling: Server Conversion Standard fan heatsink SprayModule Kit SprayModule Retrofitted Server 16

17 Spray Spot Cooling: Rack Conversion Rack Manifold Fluid I/O SprayCool Rack at PNNL 17

18 Cooling System Block Diagram Microprocessor Controller Proportional Throttle Valve Solenoid Valve Building Water Pump Heat Water Leak Detector Pump Exchanger Legend: Water Fluorinert Electrical Supply Manifold Spray Modules (2/server) Return Manifold 18

19 Performance Testing Worst case location chosen in computer room Ran Linpack on both air-cooled and spray cooled configurations Ran Linpack, NASA Parallel Benchmark (NPB) and NWChem at PNNL Performed monthly robustness testing 19

20 Spray Spot Cooling: Rack All servers running High Performance Linpack Spray cooled rack is an 8.2 kw thin node rack Floor tile air exit temp range of o C Worst-case rack location in the facility SprayCool rack is relatively independent of location Requires less air flow - 2 fans removed per server Max rack door temp of 24 o C Exhaust air up to 11 o C higher than floor tile air exit temperature Smaller temperature gradient over rack door 20

21 Spray Spot Cooling: Server Floor tile air exit temp range of o C Server is running High Performance Linpack Server is open so air flow to motherboard components is compromised SprayModule Memory DIMMs Power supply boards 21

22 Spray Spot Cooling: Coolant Temp Range 80 RX2600 CPU Diode Temperatures vs. Coolant Temperature CPU Diode Temperature ( C) Node 1 CPU 0 Node 1 CPU 1 Node 2 CPU 0 Node 2 CPU 1 Node 3 CPU 0 Node 3 CPU 1 Linear (Node 3 CPU 0) Linear (Node 2 CPU 1) Coolant Temperature ( C) Dual Itanium II servers (130W TDP) Maximum diode temperature is 79.5 C when extrapolated to 50 C coolant Cooling tower water at PNNL should not exceed 30 C 22

23 Spray Cooled RX2600 Uptime Total operational time of 345 days (through June 10) Total number of down days was % overall system availability 23

24 Dual Opteron Blade With 2 Memory Modules Installed 24

25 Blade with Air Cooling 25

26 Tessera 3D Chip Scale Memory Module 26

27 Simplified 3-D 3 D Chip Scale Memory Construction DRAM die Micro via Substrate Solder ball Carrier Printed Circuit Board (PCB) Note: drawing not to scale 27

28 28

29 Coefficient of Performance (COP) COP = Compute Power Cooling Power Examples: COP Cooling Power Cooling Budget 1 = Compute Power 50% 0.5 = 2x Compute Power 67% 2 = One Half Compute Power 33% 3 = One Third Compute Power 25% 4 = One Fourth Compute Power 20% 29

30 Estimated Phase I COP Facility Analysis COP Air-Cooled (HP rx2600) 2.00 Spray Cooled CPUs (HP rx1620), bypass chiller 2.57 Spray Cooled Full Board, bypass chiller 3.36 Spray Cooled Full Board & Power Supplies, bypass chiller TBD 30

31 Spray Full Board: Coolant Temperature Range W dual Opteron Server 2 GB/CPU Stacked Memory 8 x CPU Burn, nbench Atomizing pressure of 20 psid Reject heat directly to cooling tower water Processor Temperatures (C) Coolant would be supplied at approx 20C in a data center MPB U1 Core MPB U2 Core Coolant Temperature (C) ("HOBO II CPU TEMP RESULTS.xls") PNNL cooling tower water should not exceed 30 C 31

32 COP Improvements Facility COP Baseline Air 84 racks total TFlops Baseline COP ~ 2.0 Spot Cooling SprayCool (1U servers) 69 racks total TFlops COP ~ 2.57 (no chillers) Spray cooled rx1620 Global SprayCool 48 racks total TFlops COP ~ 3.36 (no chillers) Facility Density 32

33 TOTAL COST OF OWNERSHIP Assumptions Impact to Model Payback Scenario 3 $1.2M annual facility upgrade Deferred construction Reliability costs Energy costs Maintenance costs.5 years Scenario 2* Increased floor space utilization Floor capacity Reliability costs Energy costs Maintenance costs 1.5 years Scenario 1* No facility gains included Reliability costs Energy costs Maintenance costs 2.8 years * Chillers in use 33

34 Spot Cooling Lessons Learned Issue Failure Solution Water Leak Hazard False Leak Alarm Any water fitting, valve, or plumbing component Sweating valve dripped water onto leak detector Add leak monitoring system with automatic power shutdown Add multi-zone leak detector, repack valve Thermal Interface Material (TIM) Improperly applied TIM caused CPU overheating and failure Additional training and procedures on application of TIM 34

35 Global Cooling Lessons Learned Issue Failure Solution Blade failure after 9 months of operation Blade would not boot due to faulty memory modules Material and fabrication process improvement in memory stack fabrication (i.e., solder reflow profile, package coplanarity, etc.) Water hammer Heat exchanger glue bonds leaked Replace heat exchanger with welded and brazed metal construction, install water hammer arrestor 35

36 Conclusions Successful adaptation of spray cool technology to mainstream data center servers Successful demonstration of global spray cool chassis and blades with 3-D chip scale memories 96% system availability after one year of operation (exceptional, given prototype status) Provided operational data for COP and TCO analysis 36