Role of the PI System in Research @ Georgia Tech Data Center Laboratory Presented by Vaibhav K. Arghode Prepared by Vikneshan Sundaralingam
Outline Data center laboratory general information Current use of PI System Future use of PI System Copyri g h t 2012 OSIso f t, LLC. i
3 Data Center Laboratory @ Georgia Tech A Platform for Transformative Research on Green Data Centers Director: Dr. Yogendra Joshi yogendra.joshi@me.gatech.edu http://ceetherm.gatech.edu/drupal/
4 8.75m Power distribution unit D O O R Downflow CRAC APC In row cooler HP rack HP rack IBM rack Center rack Mixed HP/ IBM rack CEETHERM Data Center Layout 3 5 7 9 11 13 1 2 4 6 8 10 12 14 Upflow CRAC APC In row cooler Mixed HP/ IBM rack HP rack IBM rack Center rack Server Simulator Downflow CRAC Upflow CRAC Dell Power edge 64 Node Rack 640 node rack Storage Center Center Center Center Storage Downflow CRAC Spare Center Center Center Center Center Center Network Downflow CRAC 6.4m 6.4m Experimental zone Computational a zone Center Details: 10 Racks 6 Servers / Rack 14 Blades / Server 4 Cores/ Blade Total: 3360 Cores Power distribution unit Power distribution unit Data Center Layout D O O R Equipment Donations courtesy: Data Acquisition: OSIsoft Building Management System: McKenney s Racks: APC Fan assist Chimney Racks: Wrightline Servers: Intel, Yahoo and OIT CRAC unit: Liebert Server Simulator: APC Branch circuit power metering: PDI corp Remote KVM switch: Minicom and Digi In row coolers: APC
CRAC (Down flow) CRAC (Down flow) CRAC (Up flow) CRAC (Up flow) CRAC (Down flow) CRAC (Down flow) 5 Dampers 6 inch thick Insulated collapsible partition Data Center- Air Distribution Cold air inlets Hot air exhaust Contained hot air roof return 2 ft. ducted supply Plenum 2 ft. ducted supply Plenum 3 ft Raised floor plenum Over head supply Under floor supply 2 ft. ducted supply Plenum 2 ft. ducted supply Plenum Economizer
6 Particle Image Velocimetry System (PIV) Fan speed and Heat setting dials PIV system One of the first PIV systems for rack level air flow mapping. 22 kw Server Simulator with adjustable fan speed and heater power settings to simulate a variety of air flow and heat loads. Server Simulator Perforated floor tiles with dampers
7 Particle Image Velocimetry Pulsed Laser t Cylindrical lens Measurement area and plane (0.60x1.9) m 2 Light Sheet Image map 1 pulse 1 Interrogation area Image map 2 pulse 2 Cross correlation function Vector map Data Analysis Velocity Stream lines Vorticity Turbulence CCD camera Flow seeded with particles x, y ( Particle Displacement) Noise filtering and random vector elimination x 5 5.2 6 y 5 5.4 6.8 u 2.3 0.8 0.32 0.3 Velocity vectors Δx u Δt v 0.8.1 Δy v Δt Vector Statistics Camera 2 (Image map B) Camera 1 (Image map A) Light Sheet Y X Perforated floor tile Rack (server simulator)
Height of the server simulator measured from the perforated floor tile (mm) 8 Tile/Rack Air Flow Studies using PIV 1950 1800 1650 1500 1350 1200 1050 900 750 600 450 300 150 Cold aisle 0 0 150 300 450 600 Distance measured from the centre of cold aisle (mm) 5.5 kw/rack (625 CFM) Rack air inlet 625 CFM Velocity (m/s) 3 2.8 2.6 2.4 2.2 2 1.8 1.6 1.4 1.2 1 0.8 0.6 0.4 0.2 Perforated tile 625 CFM Entrainment Cold aisle By-Pass Rack air inlet 2500 CFM 22 kw/rack (2500 CFM) Recirculation Perforated tile 2500 CFM [ref] Kumar, P., Joshi, Y., Experimental Investigations on the Effect of Perforated Tile Air Jet Velocity on Server Air Distribution in a High Density Data Center, Intersociety Conference on Thermal Phenomena (ITHERM), Jun 2-5, 2010, Las Vegas, USA.
Aisle center Rack Aisle center Rack Aisle center Rack Aisle center Rack 9 Pressure outlet Pressure inlet (K=10) Pressure inlet 22 kw/rack (2500 CFM) Aisle top Numerical Modeling Aisle top Aisle top Aisle top Grill Tile Gap Fan Mass flow inlet Pressure inlet Symmetry Body force 4.2million cells Velocity (m/s) Tile PIV (Experiments) Tile Geometrical Resolution Tile Modified Body Force (Developed) Modified body force with momentum source region of 22 width and 6 height suggests closer comparison with geometrical resolution and PIV results Tile Porous Jump (Generally Used) [mbf] Arghode, V. K., Kumar, P., Joshi, Y., Weiss, T., Meyer, G., Rack Level Modeling of Air Flow Through Perforated Tile in a Data Center, International Mechanical Engineering Congress and Exposition (IMECE), Nov 9-15, 2012, Houston, USA.
10 Grid Based Temperature Measurements T type thermocouples (wire gauge 28, 0.321 mm) 252 thermocouples (width = 4ft, depth = 2ft, height = 6ft 6inch) 3D mapping of the cold aisle 6 planes along the height, 5 planes along the depth and 10 planes along the width
11 Rack Air Flow and Heat Load Measurement Air Inlet Air Outlet T in RACK Rack Load Tester T out Thermistor (T) Thermal Anemometer (V) Array (15 3) of temperature (thermistor) and velocity (thermal anemometer) sensors Mass flow rate calculated based on the measured velocity Rack heat load calculated based on mass weighted outlet temperature and inlet temperatures and the mass flow rate Air mass flow Heat load m= A V Measured Velocity Flow Area Air density Q=m C p (T out -T in ) Air specific heat Measured Temperatures
12 Containment System with Active Fan Tiles Top Curtain Fan Assisted Tiles
Outline Data center laboratory general information Current use of PI System Future use of PI system Copyri g h t 2012 OSIso f t, LLC. ii
14 Temperature Thermal diodes (CPU temperatures) Sensor Types Resistance temperature detectors, thermistors, and thermocouples (air temperature) Power Branch Circuit Monitoring System (at Power Distribution Unit @ PDU level) Power Outlet Power Sensing (at Power Strips) Water Velocity/Flow : Dual turbine flow meter (Rear Door Heat Exchanger) Magnetic flow meter (Computer Room Air Conditioning) Air Velocity/Flow: Constant temperature hotwire anemometer (point velocity measurement) Particle image velocimetry (PIV) (velocity field) Flow hood (tile flow rate) Pressure Sensors Humidity Sensors
15 Interfaces MODBUS TCP/IP Communication protocol published by Modicon in 1979 for use with its programmable logic controllers (PLCs). Currently used as the interface with the building management system. SNMP (Simple Network Management Protocol) Internet-standard protocol for managing devices on IP networks. Used in facility to communicate with power strips and blade centers. IPMI (Intelligent Platform Management Interface) Used by system administrators to manage computers and gather data for monitoring purposes. Used in facility to communicate with server to collect server health data
16 Data Acquisition and Archiving Building Management User Machine 1 (e. g. CRAC) MODBUS Web Server Machine 2 (e. g. Power Strip) PI Server Software: Analyze and Visualize (e.g. Matlab, Labview) Machine 3 (e. g. Server) IPMI Headnode with Linux Platform
17 Rack Level Monitoring Example (SNMP)
18 Control System Design and Implementation Building management system (BMS) uses Niagra by Tridium to regulate cooling MATLAB script or code communicates with PI System using PI OLEDB, which sends commands through output tags to BMS. MATLAB built-in functions and tools used (e.g. SIMULINK) to design and implement controller. Benefits: A comprehensive library of tools in MATLAB for control system design No need for actual controller board Fail safe: no changes made to existing controllers in data center Suited for research purposes because of its flexibility and fast turnaround times PI Server MATLAB Building Management System
19 Controller Implementation Example Regulating maximum CPU temperatures for a rack by varying CRAC supply air temperature CPU Max Temp. Setpoint Idle to full load Integral action controller as an add-on to existing CRAC controls. Sampling period : 2 minutes Only one rack turned active in experimental zone (preliminary testing) Test case: Step increase in compute load [ref] Sundaralingam, V., Joshi, Y, Controller to regulate maximum server CPU temperatures in a rack by varying CRAC supply air temperatures, International Mechanical Engineering Congress and Exposition (IMECE), Nov 9-15, 2012, Houston, USA.
Outline Data center laboratory general information Current use of PI system Future use of PI system Copyri g h t 2012 OSIso f t, LLC. iii
21 Future Work Webserver Development Integrate Sharepoint Server with the PI System tools: PI WebParts, PI ProcessBook and etc. Webserver function Facilitate data extraction Visualize data Disseminate live and archived data to the public (login credentials will be provided for approved requests Tentative beta site availability (April 2013) Development of logic/physics based controllers for air cooled data center Contained aisles Active fan tiles Advanced air delivery systems
Brought to you by Copyri g h t 2012 OSIso f t, LLC.