High Density Data Centers Fraught with Peril. Richard A. Greco, Principal EYP Mission Critical Facilities, Inc.

Transcription

1 High Density Data Centers Fraught with Peril Richard A. Greco, Principal EYP Mission Critical Facilities, Inc.

2 Microprocessors Trends Reprinted with the permission of The Uptime Institute from a white paper titled Heat Density Trends in Data Processing, computer Systems and Telecommunications Equipment Version 1.1 2

3 Power Consumptions will Continue to Increase More transistors and higher clock speeds will result in more power consumption which in turn lead to greater production of heat. Projections of power consumption increases by 2X every four years. 3

4 Power Consumptions The fact is that the majority of servers available today have high power consumptions. Technology Impact on 42U Enclosure 5U 3U 2U 1U 3U Blade Total Load (kw) Albatros Server DS/5U 1.3 kw IBM X Series kw Compaq Proliant DL kw - - IBM X Series kw - Sun Fire B kw Number of Outlets

5 Cooling Microprocessors Within three to five years, researchers at Intel Corp., Hewlett-Packard Co. and IBM predict computer makers will have to move beyond fans and adopt cooling mechanisms such as radiators and liquid cooling systems to avoid potential overheating. 5

6 How Will These Trends Impact the Data Center? High density data centers Faster chips, more heat, more boxes Low to medium density data centers with hot spots Faster chips, more heat, but fewer total number of boxes Unknown effect 6

7 What is a High Density Data Center? A high compute density data center of today is characterized as one consisting of thousands of racks with multiple computing units The heat dissipation from a rack containing such computing units exceeds 10kW. Source: Thermal Considerations in Cooling Large Scale High Compute Density Data Center, Patel et al. 7

8 What EYP MCF Believes is the Practical Limit to a High Density Data Center A high density data center is one in which the power to the raised floor and the computing equipment exceeds 150 watts/sf over the entire raised floor. This is equivalent to approx. 3.5 kw/cabinet in an efficiently arranged data center. This is the point where traditional all air data centers begin to have significant limitations. 8

9 Cooling Strategies for the Conventional Data Center

10 Example of a Traditional All Air Data Center at 150 w/sf 15,000 sf, x36 cabinets, 3.5 kw/cabinet, 2170 kw cooling required Cabinets on hot aisle/cold aisle 16 ft aisle to aisle 23 or 24 CRACs (lay-out dependent) total at 110 kw each (375.3 MBH), 3 redundant 620 perforated tiles at 530 CFM each min. req. (24) 200 kw PDU s (2N configuration) 10

11 Two Approaches to Cooling the Data Center We use Computational Fluid Dynamics (CFD) to determine the preferred arrangement Option 1: CRACs on opposite walls Option 2: CRACs in middle and on one wall 11

12 Option 1: CRACs On Opposite Walls 150 w/sf Typical 15,000 sf data center kw/cabinet (24) kw CRAC units (24) 225 KVA PDUs 12

13 Front In, Back Out, Hot Aisle / Cold Aisle Configuration for Cabinet Air is drawn through the front of the cabinet (blue) and discharged through the back (red) Cold Hot Cold Aisle Aisle Aisle Supply Air is 55º F Return Air is 85º F 13

14 Option 1: CRACs On Opposite Walls, 3 Raised Floor 14

15 Option 1: CRACs On Opposite Walls, 3 Raised Floor 2 Units Failed 15

16 Option 2: CRACs in Middle and On One Wall 150 w/sf Typical 15,000 sf data center kw/cabinet (23) kw CRAC units (24) 225 KVA PDUs 16

17 Option 2: CRACs in Middle and On One Wall, 3 Raised Floor 17

18 Cooling Strategies for High Density Data Centers

19 High Density Data Center 240 w/sf Typical 15,000 sf 100 by kw/cabinet 3650 kw cooling 462 air 1200 cfm ea (40) kw CRAC units (32) 225 KVA PDUs (2N) 19

20 High Density Data Center, 4 Rows of CRACs, 5 Raised Floor 20

21 High Density Data Center, 4 Rows of CRACs, 5 Raised Floor 3 Units Failed 21

22 To Overcome Airflow Variations Use Fan Powered Cabinets Ensures airflow to cabinet regardless of pressure in floor and airflow through air grates, however re-circulation will occur if sufficient air from under floor is not delivered. Cabinet fans supplement server fans 22

23 Fan Powered Cabinets: Front In, Back Out, Footprint Vented, Fan Assisted Air is drawn through the front of the cabinet (blue) and discharged through the back (red) Cabinet Fan Discharge Air is 85º F Raised Floor Supply Air is 55º F 23

24 Fan Powered Cabinets: Front In, Back Out, Fan Assisted Air is drawn through the front of the cabinet (blue) and discharged through the back (red) Cabinet Fans Discharge Air is 85º F Raised Floor Supply Air is 55º F 24

25 Fan Powered Cabinets: Front In, Top Out, Fan Assisted Air is drawn through the front of the cabinet (blue) and discharged through duct work (red) Return Air Ceiling Plenum is 85º F Ceiling Cabinet Fans Hot Air Plenum Discharge Air is 85º F Raised Floor Supply Air is 55º F 25

26 Fan Powered Cabinets: Front In, Top Out, Footprint Vented, Fan Assisted Air is drawn through the bottom of the cabinet (blue) and discharged through the top (red) Discharge Air is 85º F Cabinet Fans Raised Floor Supply Air is 55º F 26

27 Fan Powered Cabinets: Front In, Top Out, Footprint Vented, Forced Ventilated Air is drawn through the bottom of the cabinet (blue) and discharged through duct work (red) Discharge Air is 85º F Ceiling Cabinet Fans Raised Floor Supply Air is 55º F 27

28 High Density Data Center without CRACs 325 w/sf Typical 11,200 sf high density data center kw/cabinet 462 cabinet coolers (32) 225 kva PDUs 28

29 Water Cooled Cabinet: Front In, Back Out Air is drawn through the front of the cabinet (blue) and discharged through the back (blue) Cooling Coil Cabinet Fans Discharge Air is 72º F Raised Floor Supply Return Piping 29

30 Water Cooled Cabinet: Footprint Vented Cool Air Cabinet Rear Cabinet Front Fan and Coil Location under Cabinet Raised Floor Supply Return Piping Building Floor 30

31 Refrigerant Cooled Cabinet Air is drawn through the front of the cabinet (blue) and discharged through the back (red) 45º F Water from Chilled Water Plant Heat Exchanger 60º F Refrigerant Pump Discharge Air is 85º F Raised Floor 31

32 Refrigerant Cooled Cabinet: Front In, Back Out Air is drawn through the front of the cabinet (blue) and discharged through the back (red) Piping to Heat Exchanger Discharge Air is 85º F Overhead Fan Coil Supply Air Supply Air is 55º F Raised Floor 32

33 Inherent Problems with High Densities Air Distribution Problems Inadequate raised floor height and excessive openings in perforated tiles can result in high velocity airflows. Bypassing of air within cabinets Re-circulation of air outside of cabinets Today users report a greater number of server failures in top 1/3 of cabinets 33

34 Inherent Problems with High Densities Cabinet fans are generally single power supply. Cabinet fan power should be from a separate PDU with redundancy. Identical cabinets may be required. UPS system must be larger or separated to incorporate fan power (up to 350 w/cabinet). For our example of 462 cabinets 162kW of fan power. 34

35 Inherent Problems with High Densities What about the rate-of-change at the microprocessor level during transitions from utility to generators. To my knowledge this has not be addressed by the industry. Is uninterruptible cooling mandatory? If so, what are the implications? Costs Reliability Redundancy Maintainability 35

36 Other Considerations Numerous additional monitoring points. Addition of cooling liquids in data center environment. Fire Zones. Noise. Seismic. Computational Fluid Dynamic modeling is required for design and recalculation when hardware changes. 36

37 Other Considerations It will be mandatory to have close coordination between facilities department and IT hardware planners. Does not exist today Generally higher risks with lower margins for errors. 37

38 Prognostications late 1990 s requirements of 100 watts per square foot of average power now exceed 200 watts per square foot, and will double again by META Report - August 2, 2001 by Rich Evans 38

39 Prognostications In fact, the watts-persquare-foot requirement will again double by 2008, so designers must work this into longer-range facilities plans so that space can be fully used, as opposed to 40 percent to 50 percent utilization because of insufficient power and cooling. META Report August 2, 2001 by Rich Evans 39

40 Prognostications Data centers utilize much less than 50% of the physical and power infrastructure. Source: Cost of Overbuilding by Domenic Alcaro 40

41 Prognostications Current estimates of data center power requirements are greatly overstated because they are based on criteria that incorporate oversized, redundant systems, and several safety factors. Source: Data Center Power Requirements: Measurements from Silicon Valley, Mitchell-Jackson, Koomey, Nordman, Blazek 41

42 Prognostications Furthermore most estimates assume data centers are filled to capacity. For the most part, these numbers are unsubstantiated. Source: Data Center Power Requirements: Measurements from Silicon Valley, Mitchell-Jackson, Koomey, Nordman, Blazek 42

43 So What s Real? Today s data centers incorporate significant amounts of legacy equipment, slow change out due to economy. Many data centers have significant amounts of available space on the raised floor. 43

44 Current Load Densities DATA CENTER TYPE WATTS/SF KW/CABINET Enterprise 20 to to 1.5 Internet/Co-location 40 to to 2.0 Managed Services 60 to to 2.5 High Density

45 My View Most projections of watts/square foot are overstated. Millions of dollars will be spent on unused infrastructure. There no longer are real drivers for compaction. Generally there is ample raised floor available. 45

46 My View High density data centers will not become a reality. Low/medium density data centers will have hot spots due to high density computing equipment. 46

47 My View of What to Do Manage your hardware installations Don t buy IT gear that is problematic Don t build yourself a problem Existing air delivery systems work Highly engineered systems are expensive to build and operate and significantly reduce reliability, redundancy and flexibility. 47

48 High Density Data Centers Richard A. Greco, Principal EYP Mission Critical Facilities, Inc.