Rate of Heating Analysis of Data Centers during Power Shutdown

2011. American Society of Heating, Refrigerating and Air-Conditioning Engineers, Inc. (www.ashrae.org). Published in ASHRAE Transactions, Volume 117, Part 1. For personal use only. Additional reproduction, distribution, or transmission in either print or digital form is not permitted without ASHRAE'S prior written permission. LV-11-C026 Rate of Heating Analysis of Data Centers during Power Shutdown Kishor Khankari, PhD Member ASHRAE Abstract During power outages servers in the data centers are generally powered by uninterruptible power supplies (UPS). At the same time the sources for active cooling such as CRACs, CRAHs, and chillers stop operating for a period until powered by alternate power sources. During this period servers continue to generate heat without any active cooling. This results in increase in the room air temperature within a short period that can be detrimental to the servers. This paper, with the help of a mathematical model, indicates that the rate of heating of a data center can start initially at a certain maximum level, and can then gradually reduce to a certain minimum level, which is the lowest possible rate of heating that a data center can attain. The rate of such exponential decay is a function of the time constant, which is the characteristic of a data center design and layout. The time constant depends on the heat capacity ratio and the specific surface area of racks in a data center. This paper analyzes various factors that affect these parameters and demonstrates how the time constant can be employed as a matrix to compare the thermal performance of data centers during the power outage period. INTRODUCTION Provision of continuous cooling to mission critical facilities is essential to maintain supply air temperatures to servers within the recommended range of 64.4 F (18 C) to 80.6 F (27 C) (as recommended by ASHRAE, 2008). During power outages servers continue to operate with the power provided by the uninterruptible power supply (UPS) units while the supply of cooling air is completely halted until alternate means of powering the cooling system are activated. During this time servers continue to generate heat and the server fans continue to circulate room air several times though the servers. This can result in sharp increase in the room air temperature to undesirable levels, which in turn can lead to automatic shutdown of servers, and in some cases can even cause thermal damage to servers. Provisions for making continuous cooling available during this period by alternate means are generally expensive and tedious. Data centers contain a large number of rack enclosures constructed out of rolled carbon steel. Before considering the other expensive options for continuous cooling it would be valuable to analyze whether rack thermal mass can help in reducing or eliminating the need for provisional cooling. Previous analysis by Khankari (2010) showed that thermal mass of rack enclosures can play a crucial role in attenuating the temperature rise of room air during the power shutdown period. However, availability of this thermal mass depends on the extent of the exposed surface area of the racks, which in turn, depends on the number of racks and number of rack rows in a data center. It was further demonstrated that data centers with low heat load densities are less likely to experience automatic server shut off due to increased air temperatures and can provide more time for starting alternate power systems. In addition to the temperature, the rate of heating of room air during the power outage period is also an important factor. Not only the temperature levels in the room be maintained within the recommended range but the rate of heating should also be kept at a minimum level. During a power outage the temperature of the room air as well as the rate of temperature rise vary with time. If the rate of heat generation from servers remains constant, the temperature levels in the room would Kishor Khankari is a Associate Partner at Syska Hennessy Group in Ann Arbor, Michigan. 212 ASHRAE Transactions

continue to rise and not reach a steady state during the power shutdown period. However, the rate of temperature rise can reach a steady state and attain a constant rate, which can also be the lowest rate of heating that a data center can attain. This lowest rate of heating and the rate at which it can be reduced from the maximum rate are the characteristics of the design and layout of a data center. This paper with the help of a heat transfer model evaluates this phenomenon and analyzes various parameters that can affect the thermal performance of a data center during the power shutdown period. DESCRIPTION OF HEAT TRANSFER MODEL The heat transfer analysis presented in this study is based on the heat transfer model developed during the previous study ( Khankari.2010). Unlike the previous heat transfer model, which was mainly related to study the variation of room air temperature with time, the present model is developed to study the variation of rate of air temperature rise or the rate of heat transfer between the air and rack mass with time. The previous model was based on the hypothesis that the total heat generated by servers during the off cooling period is primarily dissipated to the surrounding air through active recirculation induced by the server fans. Air then dissipates part of this heat to the surroundings through several pathways that includes rack enclosure mass, mass of the cold air trapped under the raised floor, and to the outside world through the building envelop. However, the previous analysis showed that about 98 percent of the total heat generated by the servers is absorbed by the room air and rack enclosures and less than 2 percent is dissipated to the other components. Therefore, in this analysis the heat transfer model is modified to assume that the total generated heat is dissipated among the room air and the rack enclosures only. This is described by the equation (1) (Table 1). The rate of heat transfer mechanism between the air and the rack enclosures, as shown in the equation (2b), depends on the heat transfer coefficients (U), exposed rack surface area (A rack ), and the temperature differences ( T) between the rack mass and the room air. It should be noted that the exposed rack surface area depends not only on the number of racks but also on the number of rack rows. The assumptions related to the previous analysis are still valid and are mentioned here for reference. The heat transfer model considered in this analysis is a zero dimensional model and assumes that all spatial variations within the data center are negligible. The air in the data center room is assumed to be well mixed, and hence, assumes a single mixed temperature. Since the air is rapidly moved by the server fans this assumption is quite reasonable. Also all the rack enclosure mass assumes a single temperature. The resistance to heat transfer within the rack mass is assumed to be small due to large thermal conductivity compared to the heat transfer coefficient on the surfaces. The rate of heat generation from servers is assumed to be constant during the power shut down period. Mathematical analysis of the heat transfer model and the development of various equations are presented in Table 1. RATE OF HEATING ANALYSIS After the active cooling completely stops, the air and rack temperature start rising due to the heat generated from the servers. Figure 1a shows a hypothetical trend of room air and rack temperature rise. Effect of various factors that can affect these trends is discussed in detail in Khankari (2010). Figure 1b shows the corresponding trend of rate of heating or rate of temperature rise with time. These trends reveal two important facts a) after an initial rapid rise, both the air and rack temperatures tend to vary at constant rate; and b) this rate is equal for both the air and the rack mass. As shown in Figure 1b the rate of heating of air starts at a certain maximum level (R max ), and then, gradually reduces to a certain minimum level (R min ), which is the lowest possible rate of heating that a data center can attain. Thus, R min can be an important characteristic of a data center design. The heat transfer analysis presented in Table 1 indicates that with increasing time, the difference between R max and R min gradually approaches zero while the room air temperature continues to increase at a constant rate of R min. For the purpose of this analysis a non-dimensional rate of heating (θ) is introduced. According to equation (6), initially when the rate of heating, R, is at R max, θ is equal to 1. Similarly, when R approaches the R min, and theoretically at time equal to infinity, θ will approach to zero. Thus, the non-dimensional rate of heating varies exponentially from 1 to 0 as shown in Figure 1c. This 2011 ASHRAE 213

parameter is helpful in analyzing the extent of the rate of heating (R) from reaching the potential minimum rate of heating (R min ) at any given time during the power outage period. As shown in equation (6), the rate of such exponential decay depends on the time constant (τ) of the data center. Air Rack R max Air Rack Temperature Rate of Temperature Rise (dt/dt) R min Time Time Figure 1 (a) Variation of temperature with time. Figure 1 (b) Variation of rate of heating with time. 1.0 0.9 Non-dimensional rate of heating (θ) 0.8 0.7 0.6 0.5 0.4 0.3 0.2 36.8 % line 0.1 0.0 Time Figure 1 (c) Variation of non-dimensional rate of heating (θ) with time. The time constant (τ) of the data center depends on the heat transfer coefficient (U), the specific heat of rack mass (C prack ), the specific surface area of the rack (A s ), and on the heat capacity ratio (C r ). Thus, the time constant is a characteristic of the design and layout of the data center. A good design of a data center should not only try to reduce the minimum rate of heating (R min ) to the lowest possible level but should also reduce the time constant (τ), which determines the rate at which the initial maximum rate of heating reduces to the minimum rate. MAXIMUM AND MINIMUM RATE OF HEATING It is assumed that initially (at time t=0), right after the power outage, room air and rack mass are in thermal equilibrium and both assume a certain average initial temperature. According to this assumption at time t=0, there is no exchange of heat between the air and the rack mass. This assumption leads to equation (4a) for the initial rate of heating, R max. This initial 214 ASHRAE Transactions

maximum rate of heating increases with the heat load in the data center and decreases with the room heat capacity. The heat capacity of the room depends on the volume of a data center. Thus, the maximum rate of heating decreases with an increase in the floor area and/or height of a data center room. In other words, data centers with larger floor area and/or larger height will have lower R max. This mathematical analysis also leads to another important conclusion, as indicated by equation (4b), that at steady state when the rate of heat transfer between the air and rack mass becomes constant, the rate change of air and rack temperature become equal (Figure 1b). This analysis leads to the equation (4c) for R min, which is the lowest possible rate of heating the room air can attain. Equation 4c indicates that the minimum rate of heating also varies directly with the heat load, but unlike the maximum rate, it decreases with the total heat capacity of the system, which is the sum of the room and rack mass heat capacities. As thermal mass increases, R min decreases. In other words, data center with larger thermal mass, for example, due to more number of racks will have a lower R min than an identical data center with a fewer racks. Similar to R max, the minimum rate of heating also decreases with an increase in the heat capacity of a data center room. Table 1: Mathematical Model (1) Therefore, from equation (1) and (2) From the equations (1) and (2) (2a) (2b) (3a) (3b) Initial and final conditions for the equation (3a) are, 0 (4), At time t = 0, T = 0, and therefore, from equation (1) and (2a) (4a) At steady state (time t = ), the equation (3b) reduces to (4b) From equation (4a) and (4c) 1 (4c) (4d) (5) With the initial and final conditions described above, the solution of the equation (3) is where, / (6) Time constant 1 and the specific surface area of rack 2 2 /N / (7) (8a) (8b) TIME CONSTANT As mentioned before the time constant (τ) of a data center is an important design parameter that determines its ability to reduce the initial maximum rate of heating, R max, to the final minimum rate of R min. Data centers with a large time constant will take relatively long time to reach the minimum level of heating compared to those with lower time constant. According to equation (6) and as shown in the Figure 1c, when the physical time, after the power outage, reaches the value of the time constant (t = τ), the difference (R R min ) reduces to 36.8 percent of its initial difference, (R max R min ). Thus, the time constant can be employed as a matrix to compare the thermal performance of the data centers. It should be noted that time constant is independent of the heat load in a data center. It mainly depends on the parameters related to the design and layout 2011 ASHRAE 215

of a data center. For constant values of heat transfer coefficient (U) and the specific heat of the rack material, according to equation (7), the time constant varies with the specific surface area of the racks (A s ) and the heat capacity ratio (C r ). While the specific surface area of racks determines the thermal conductance, the heat capacity ratio (C r ) determines the thermal capacitance of a data center. Larger values of thermal conductance and capacitance together determine the ability of a data center to keep the rate of heating to a minimum level by keeping the value of time constant low. Note the C r increases with the number of racks and decreases with the height of a data center. Figure 2a shows variation of time constant with the C r for various levels of specific surface area (A s ). According to this analysis, the time constant decreases with increase in the heat capacity ratio and specific surface area of racks. The effect of specific surface area of racks has a larger impact on the time constant at lower values of C r. This means that with fewer racks, the time constant can be reduced by increasing the available surface area of racks. This can be achieved, for example, by increasing the number of rack rows in a data center. On the other hand, at higher values of C r (higher rack density) the effect of specific surface area has a lower impact on the time constant. 350 300 250 Sp. Surface Area ft 2 /lb (m 2 /kg) 0.07 (0.014) 0.08 (0.016) 0.09 (0.018) Time Constant (τ), s 200 150 100 50 0 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 5.5 6.0 6.5 7.0 7.5 8.0 8.5 9.0 9.5 10.0 Heat Capacity Ratio (C r ) Figure 2a: Variation of time constant with heat capacity ratio at various levels of rack specific surface area. 160.0 140.0 Room Ht 10 ft (3 m) 15 ft (4.6 m) 120.0 20 ft (6.1 m) Time Constant (t), s 100.0 80.0 60.0 40.0 20.0 1 2 3 4 5 6 Number of racks per 100 ft 2 (9.3 m 2 ) floor area Figure 2b: Variation of time constant with number of rack rows at various levels of room height. 216 ASHRAE Transactions

50 45 Time Constant (t), s 40 35 30 25 Rack Weight 200 lb (90.7 kg) 300 lb (136 kg) 400 lb (181.4 kg) 20 15 10 1 5 9 13 17 21 25 Number of racks per row Figure 2c: Variation of time constant with number of rack rows at various levels of rack weight. An increase in the data center room height can increase the heat capacity of the room and decrease the heat capacity ratio. As a result, as shown in Figure 2b, the time constant decreases with increase in the room height at all levels of rack densities. It means that a data center with higher ceilings would take longer to reach the R min. However, it should be noted that the increased heat capacity of a room helps in maintaining low air temperature levels as shown in the previous study (Khankari, 2010). In order to keep the time constant low, the layout of data centers with higher ceilings can be modified by increasing the number of rack rows which can provide larger surface area. Figure 2c shows that the weight of racks has relatively a little impact on the time constant in comparison to the specific surface area of the racks. HEAT CAPACITY RATIO (C R ) The heat capacity ratio (C r ), a measure of thermal capacitance of a data center, is a ratio of heat capacities of rack mass and room air. According to equation (5), C r increases with increase in the rack density (number of racks per unit floor area) and weight of individual racks but decreases with the increase in the height of a data center room. It is convenient to express the rack density in terms of number of racks per 100 ft 2 (9.3 m 2 ) of a data center floor area. Figure 3 shows the variation of C r with the number of racks for various room heights. This analysis is developed for the rack weight of 250 lb (113.4 kg). With increase in the height the heat capacity of the room increases, and therefore, the heat capacity ratio decreases. A fully occupied data center, depending on the depth (D) of racks, can realistically accommodate maximum four to six racks per 100 ft 2 (9.3 m 2 ). As shown in Figure 3 data centers with room height between 10 to 20 feet, C r values can approximately range from 10 to 3, and therefore according to Figure 2a, the time constant values can range from 25 to 125 seconds. As indicated by equation (4d), the ratio of R max and R min varies with C r. It should be noted this ratio is independent of the rate of heat generation or the heat load in a data center. This relationship is useful in computing the potential R min of a data center from the initial R max and the C r of a data center. This ratio can be determined from the Figure 3 just by adding 1 to the values of C r on the y axis. Thus, for the most common data center designs with four to six racks per 100 ft 2 (9.3 m 2 ) floor area, depending on the height of a data center, R min can be reduced to 1/4th to 1/11th of the R max. 2011 ASHRAE 217

10.0 0.30 0.06 Rack Ht Heat Capacity Ratio (C r ) 9.0 8.0 7.0 6.0 5.0 4.0 3.0 Room Ht 10 ft (3 m) 15 ft (4.6 m) 20 ft (6.1 m) 30 ft (9.1 m) Specific Surface Area of Racks (ft 2 /lb) 0.25 0.20 0.15 0.10 0.05 0.04 0.03 0.02 m 2 /kg 7 ft (2.1 m) 5 ft (1.5 m) 3 ft (0.9 m) 2.0 1.0 0.0 1 2 3 4 5 6 Number of racks per 100 ft 2 (9.3 m 2 ) floor area 0.05 0.01 1 5 9 13 17 21 25 Number of racks per row Figure 3: Variation of heat capacity ratio (C r ) with number of racks. Figure 4: Variation of specific surface area (A s ) with number of racks Specific Surface Area The specific surface area affects the thermal conductance of a data center. The rate of heat transfer from the room air to the rack mass depends on the extent of the surface area of racks exposed to the surrounding moving air. The conductance (UA s ), and in turn, the time constant (τ), of a data center varies directly with the available surface area of racks. Data centers with larger available surface area can reach the minimum heating rate (R min ) in a shorter time. For most racks only the top and bottom surfaces are available for exchanging the heat. In addition, side panels of the end racks located at the end of each rack row are available for such exchange. Thus, the net available surface area of the racks varies with the size and number of racks, and number of rack rows. For this analysis the specific surface area is defined as the effective surface area of racks per unit weight of the rack. This can be computed by combining available surface areas of all racks and rack rows (equation 8a) and then dividing it by the total weight of the racks. If all racks in the data center are assumed to be identical, then the specific surface area can be computed by equation (8b). In this equation the total available surface area is expressed on a per rack basis and then divided by the weight of the individual rack. It should be noted that less number of racks per row (N rpr ) implies more number of rack rows. Figure 4 shows a variation of the specific surface area of racks with the number of racks per row for three different rack heights. With an increase in the number of racks per row (a fewer rack rows) the specific surface area of racks decreases. For example, for 7-feet high racks increasing the number of racks from five to ten per row, the specific surface area can reduce by about 18 percent. Increasing the number of racks beyond 15 per row has a minimum impact on the specific surface area. It means for the data center with fewer racks, the layout of the racks plays an important role in determining the time constant. Equation (8) shows specific surface area decreases with an increase in the individual rack weight; however, as discussed before that rack weight has of a less impact on the time constant. APPLICATIONS This section demonstrates the application of the above analysis for comparing the thermal performance of data centers during a power outage. Three different data centers were considered for this application. A list of various design parameters for these data centers is documented in Table 2. The thermal parameters are computed from the equations presented in Table 1. Data center B (DC B) is considered as a default case. Data center A (DC A) is similar to the Data center B in all respects except it is only half occupied, which is often the case on the first day of a newly-commissioned data center. Data center C (DC C) is also similar to DC B but has a ceiling height of 15 feet instead of 10 feet. In all cases the value of racks per row is 218 ASHRAE Transactions

kept the same, and hence, the specific surface area in all the cases remains the same as well. Table 2: List of parameters for the application examples Design Parameters Data Center A Data Center B Data Center C Floor area, ft 2 2500 (232.2 m 2 ) 2500 (232.2 m 2 ) 2500 (232.2 m 2 ) Height, ft 10 (3 m) 10 (3 m) 15 (4.6 m) Heat load, W/ft 2 50 (538 W/m 2 ) 100 (1076 W/m 2 ) 100 (1076 W/m 2 ) Initial temperature, F 65 (18.33 C) 65 (18.33 C) 65 (18.33 C) Number of racks per 100 ft 2 (9.3 m 2 ) 2 4 4 Number of racks per row 10 10 10 Weight of individual rack, lb 250 (113 kg) 250 (113 kg) 250 (113 kg) Thermal Parameters Max rate of heating (R max ), F/s 0.25 (0.14 C/s) 0.52 (0.29 C/s) 0.34 (0.19 C/s) Min rate of heating (R min ), F/s 0.06 ( 0.034 C/s) 0.07 (0.038 C/s) 0.065 (0.036 C/s) Specific surface area, ft 2 /lb 0.098 (0.02 m 2 /kg) 0.098 0.02 (m 2 /kg) 0.098 0.02 (m 2 /kg) Heat Capacity Ratio, Cr 3.27 6.54 4.36 Time constant (t), s 67.4 38.2 53.7 As shown in the Table 2, DC A has the highest value of time constant. In spite of the reduced heat load, as shown in Figure 5, it would take longer for DC A to reach its R min than DC B and DC C. With fewer racks, DC has a lower heat capacity ratio (C r ) than DC B, which in turn, results in a larger time constant. While it would take 67.4 seconds for DC A to reduce (R max R min ) to 36.8 percent level, DC B can achieve the same in 38.2 seconds. This situation can be improved by increasing the number of rack rows (reducing the number of racks per row parameter) which can help in increase the specific surface area. For example, if the number of racks per row in DC A is reduced to five, the time constant will reduce from 67.4 to 54.2 seconds. Like in the previous case, DC C also has lower heat capacity ratio than DC B, however, it is due to the higher ceiling height, which increases the heat capacity of the data center room. This results in larger time constant in the case of DC C than DC B. In spite of its lower R max value, as shown in Figure 5, it would take longer for DC C to reach its minimum rate of heating than DC B. Figure 6 shows a variation of room air temperature with time for all three data centers. It shows that DC A, due to lower heat load, has the lowest temperature levels. In spite of the same heat loads, the temperature levels in DC C are lower than DC B. This is due to the higher room height of DC C. In this case also if the number of racks per row is reduced to five, the time constant will reduce from 53.7 to 43.1 seconds. 1.00 0.90 0.80 DC A DC B DC C 0.70 0.60 Theta 0.50 0.40 36.8 % line 0.30 0.20 0.10 0.00 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 Time (s) Figure 5: Variation of non-dimensional rate of heating (θ) with time for the example cases in Table 2. 2011 ASHRAE 219

110 43.3 DC A DC B DC C 100 37.8 Temperature (F) 90 80 32.2 26.7 Temperature (C) 70 21.1 60 15.6 0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300 Time (s) Figure 6: Variation room air temperature with time for the example cases in Table 2. SUMMARY AND CONCLUSIONS A zero dimensional heat transfer model was developed to evaluate the rate of heating of room air during a power outage situation in a data center. Mathematical analysis indicates that the rate of heating of the room air can start initially at a maximum rate (R max ), and then, can exponentially reduce to a minimum rate (R min ), which is the lowest possible rate of heating that a data center can attain. The rate of this exponential decay depends on the time constant (τ), which is a characteristic of the design and layout of a data center. Data centers with larger time constant would take longer to reach the lowest level of heating, R min. A good design of a data center aims not only to keep the R min at low level but also tries to reduce the time constant (τ). The time constant (τ), which is independent of the heat load, depends on the room height, dimensions of racks, rack density, and on the number of racks per rack-row. These parameters are conveniently expressed in terms of the heat capacity ratio (C r ) and the specific surface area of racks (A s ) of data centers. This analysis demonstrates that increasing the heat capacity ratio and the specific surface area of a data center help in reducing the time constant. This can be achieved by increasing the rack density and number of rack rows, and by lowering the room height. However, lowering the room height can increase room air temperatures. In such situations it would be beneficial to arrange the layout of a data center with a higher number of rack rows without lowering the room height. Similarly the data centers with low rack densities can be arranged with a higher number of rack rows to reduce the time constant. The mathematical model developed in this paper demonstrates that the time constant can be employed as a matrix to compare and improve the thermal performance of data centers during a power outage. NOMENCLATURE Surface area of rack thermal mass (m 2 ) Specific surface area of racks area per unit weight of rack (m 2 /kg) Floor area of a data center (m 2 /kg) Heat capacity ratio of a data center Specific heat of rack mass (500 J/kg K) Depth of rack enclosures (m) Height of rack enclosures (m) 220 ASHRAE Transactions

Width of rack enclosures (m) Number of rack enclosures Number of rack rows Number of rack rows per rack per rack row Rate of heat generation from servers (W) t Rate of heat absorbed by room air (W) Rate of heat loss to rack enclosures (W) Rate of heating of a data center at time t (K/s) Maximum rate of heating of a data center (K/s) Minimum rate of heating of a data center (K/s) Temperature of room air (K) Temperature of rack thermal mass (K) Time (s) U Heat transfer coefficient of air over the rack surfaces (100 W/m 2 K) w Weight of an individual rack (kg) Heat capacity of room air (J/K) Heat capacity of rack mass (J/K) Thermal conductance between room air and rack mass (W/K) Temperature difference between room air and rack mass (K) Non-dimensional rate of heating of a data center Time constant of a data center (s) REFERENCES ASHRAE. 2008. ASHRAE Environmental Guidelines for Datacom Equipment - Expanding the Recommended Environmental Envelope. Atlanta: American Society of Heating Refrigeration and Air Conditioning Engineers, Inc. Khankari, K. 2010. Thermal mass availability for cooling data centers during power shutdown. ASHRAE Transactions, Vol. 116, Pt. 2. 2011 ASHRAE 221