Application of CFD modelling to the Design of Modern Data Centres White Paper March 2012 By Sam Wicks BEng CFD Applications Engineer Sudlows
March 14, 2012 Application of CFD modelling to the Design of Modern Data Centres By Sam Wicks Executive Summary By modelling the underlying physics of the data centre, Computational Fluid Dynamics (CFD) provides data centre operators with a thermal description of their facility at any running condition, providing the necessary depth of technical understanding upon which to base mission critical decisions. For the data centre designer, CFD produces efficiency predictions for facilities yet to be built and provides scientific validation for cooling architecture designs. When absorbed into the data centre design phase, CFD drives design, unlocking the most efficient data centre cooling designs possible. Due to the technical complexity inherent in understanding the physics and numerical methods behind a CFD analysis, CFD engineers are required to ensure modelling methods are accurate, while quantifying this accuracy with validation and verification. Notes & Resource 1. Introduction... 3 2. The Bottom-Up Approach... 4 3. CFD Applications... 5 4. CFD Limitations... 6 5. Conclusions... 8 6. References and Further Reading... 8 Computational Fluid Dynamics (CFD) is a powerful tool that data centre managers, designers and auditors have outsourced in the past to predict the behaviour of air flow within a data centre. Recent advances in computational power, CFD algorithms and software have enabled advanced, in-house capability within the data centre industry. The purpose of this paper is to provide end users, data centre designers and those with an interest in data centre design techniques with an introduction to CFD, the applicability of CFD within the industry and an overview of the new efficiency and cost saving possibilities available when this new technology is integrated with traditional design practices.
1. Introduction To build the most energy and cost efficient data centre cooling solution, it is necessary to be able to predict the cooling performance at the design stage. This allows the designer to explore the countless permutations of data centre layouts and cooling architectures in order to determine the best solution. There are many methods for predicting the behaviour of a data centre during the design phase. For an air-cooled data centre, current cooling design practices are at the scale of an entire room, with the heat outputs from rack equipment offset by the cooling capacity of the computer room air conditioners, coupled with volume flow rate recommendations. This method is illustrated by Figure 1. Total 30kW -40kW It is known that a volume of air exits the CRAC unit, per second, at a specified temperature. It is also known that a server has a volume of air passing through it per second, and that the server will transfer an amount of heat to this air. However, there is no guarantee that the air that leaves the CRAC ever reaches the rack. Instead, it is assumed that a sufficient volume of the cooled air reaches the server inlets, without knowledge of the exact volume. The placement of racks and CRACs is heuristically determined to reach this goal, with general rules of thumb guiding the design. There are further guidelines added atop this approach, increasing in complexity due to increased server densities, including the use of hot or cold aisle containment. By separating the hot exhaust air from cold inlet air, it is assumed Figure 1 Sizing of CRAC units from IT Load that racks will always draw cold air. The architecture is an improvement compared to uncontained solutions, but the design assumption ignores a myriad of interrelated, non-linear factors which conform to raise inlet temperatures by drawing air back between racks or lower efficiency by requiring excessive work from server fans. Traditional design methods fail to take into account the properties of cooling architecture in a way that produces quantifiable predictions. Consequently, to ensure the desired cooling conditions are met, conservative estimates of CRAC volume flow and temperature requirements drive design, hindering precision. As part of the cooling process, the CRAC unit fans impart momentum upon the air, raising the static pressure, thus driving the air flow. From the fan affinity laws, where power consumed rises with the cube of the speed of fan rotation, it can be seen that if a fan has a 15% higher volumetric flow rate than necessary, 39% of the power consumption is wasted. It is clear that by increasing the precision to which cooling architecture is designed, efficiency can be gained. This precision, required for the most efficient designs, may only be achieved through a different design approach, developed entirely from scientific, physical laws with a minimum of assumptions.
2. The Bottom-Up Approach At a fundamental level, as a by-product of performing computational tasks, the individual components within a data centre continuously generate heat. Without any methods of transporting this heat, the heat build-up would cause computing equipment temperatures to rise continuously, increasing the risk of failure, until the equipment overheats and fails catastrophically. By transferring the heat to a material, and transporting this material away from heat sensitive equipment, a safe, steady temperature can be achieved, controlled by the properties of the material and its interaction with the computing equipment. Air is the obvious and most utilised material for this purpose, owing to its availability, heat absorption properties and the ease with which it can be moved by fans and ducts. Other cooling architectures utilise direct to CPU water cooling, as water has a higher capacity to absorb heat than air and is also easily directed with pumps and tubing. It is this ability of fluid to convect heat which underpins data centre cooling. For an air-cooled data centre, it is therefore advantageous to possess a detailed knowledge of the air within all parts of the data centre during all aspects of operation, as this describes the location of heat and thus the locations and causes of any inefficiency. The scientific study of the movement of fluids is termed fluid dynamics, and fluid flows are described by the Navier Stokes equations, which are a set of non-linear, coupled partial differential equations. The mathematical complexity of these equations is such that no true, analytical solutions currently exist for many real flows. Their complete solution requires an understanding of one of the major unsolved problems in physics, and will earn a $1,000,000 prize from the Clay Mathematics Institute. A complete derivation of the Navier Stokes equations is beyond the scope of this introductory paper, but can be found in most introductory texts on the subject, see Anderson (1995). It is, however, possible to solve the Navier Stokes equations for complex, real flows, in a way that produces a solution precise enough for engineering applications, including: weather forecasting and the design of building HVAC systems, aircraft jet engines, hypersonic re-entry vehicles, and data centres. This method of solution transforms the complex mathematics into an approximate form which can be solved by computer processor, allowing the behaviour of the fluid to be predicted. Total 30kW. Empirical fan CFM Specified Intelligent Q [W]. Specified Fan Curves Figure 2 CFD Temperature Plot of Figure 1 Scenario The problem of fluid flow is now one of Numerical Analysis, a branch of Mathematics concerned with finding approximations to mathematical functions, and their associated error, instead of exact solutions. This is the approach of Computational Fluid Dynamics (CFD), a tool utilised within all industries concerned with the movement of fluids.
3. CFD Applications Figure 3 Plan view, In Row Cooling with Hot Aisle Containment Design As CFD deals directly with the underlying physics of cooling, it has wide applications for data centres. For design, CFD can be used to validate an existing design to prove the cooling architecture is fit for purpose, giving the rack inlet temperatures and CRAC running conditions, as well as illustrating any inefficiency and why it arises. Failure scenarios can be run in order to test the cooling ability of fewer CRAC units, or the interaction and mixing of standby CRAC unit flows with cooling flows from other sources, for use with Free Cooling. CFD can also be used earlier in the design phase to drive the design. If the cooling architecture s key requirements are outlined, multiple designs can be tested and any variable (such as CRAC type and running point, location, floor tile location, plenum depth) can be tuned such that the design is optimised to meet to cooling requirements in the most efficient or cost effective way. The accuracy of CFD also allows for rigorous design, with facility running conditions set to tighter tolerances, allowing for higher densities than previously possible. The heat distribution within the hot aisle shown in Figure 3 is an illustration of how CFD can be used to position and size in-row coolers. Even though they draw their air from the same hot aisle contained region, the CRAC inlet temperatures are not necessarily uniform. This is due to a number of interrelating variables, including the IT load distribution, server flow rates, CRAC fan set point and the heat distribution in the region exterior to the hot aisle. Each variable can be isolated and its dependencies tested, allowing optimisation of the design. Audit CFD is also useful for existing facilities, as it makes the invisible visible. After taking key measurements within the data centre, a CFD model provides a complete description of temperatures and flow rates at any location within the facility, allowing data centre operators to identify hot spots, inefficiencies and crucially their causes, as well as appraise their current capacity with a view to future requirements. The model can easily be adjusted to show the facility cooling behaviour with additional IT load, or changes to infrastructure and layout, without the logistics of relocating racks before the benefits are clear, with a full before-and-after description with efficiency gains and cost benefits shown by means of pressure, velocity and temperature contour plots, graphs, 3D visualisation and streamline analysis. The effects of a different CRAC running point can be tested safely, without incurring either high costs or risk to mission critical systems.
4. CFD Limitations A sound knowledge of the physical principles underpinning fluid flow and heat transfer within IT equipment and HVAC systems, as well as a general knowledge of fluid dynamics is required to ensure that CFD results are true to life. As outlined previously, CFD results are a numerical estimate of the air flow behaviour within the data centre and are not intrinsically accurate unless knowledge and care has been put into the methods used to obtain them. These methods are vast in both number and scope and vary depending on the flow being analysed, the required level of detail of the analysis and the computational power available to the CFD engineer running the analysis. The data centre has been divided into a number of points at which the CFD calculations are performed, which are termed nodes. Similarly to a digital image, where a larger amount of pixels increases the image detail, the amount of nodes affects the detail of the CFD model. Within data centre flows, the smallest air movements affect the largest. If there are insufficient nodes to capture these small movements, their effect on the entire air flow will be ignored, resulting in a CFD model which is not true to life. Similarly, as flows vary and develop over time, the number of time steps, and the size of the gaps between them, affects the CFD model accuracy. Low Node Count High Node Count Figure 4 Flow around Cylinders in a heat exchanger As an example, consider Figure 4, which depicts the flow in the wake of a cylinder within a heat exchanger, coloured by velocity where red is high and blue is low. The low node count CFD solution shows a small wake area, with flow velocity equal at the top and bottom of the cylinder. This is in contrast to the high node count solution, which accurately depicts the formation of unsteady vortices in the wake, which detach and head downstream. The low node count solution predicts neither eddies, nor the vibration they induce on the heat exchanger, which has the potential to catastrophically destroy the heat exchanger. Vast computational power and simulation times measured in years are required to achieve the level of detail required to sufficiently capture all aspects of complex flows. This is mainly due to the turbulent, chaotic variations and vortices common to most flows, which dissipate into increasingly smaller and faster eddies, the smallest and fastest of which have lengths thousands, if not tens of thousands times smaller than the length of the data centre. Figure 5 Rack exhaust temperature profiles under comfort cooling type AC units For the majority of engineering CFD analyses, instead of determining the miniscule details of fluid flow, only the time averaged, total effect of these turbulent eddies are considered, with their inner workings assumed. This
requires significantly fewer nodes and computational power within the capabilities of the modern workstation. The application of the correct assumptions, collectively known as turbulence models, varies on a case by case basis and requires significant engineering judgement. The method by which the individual components within a data centre are modelled is also of paramount importance for accuracy. Take for example a CRAC unit operating with an air volume output of 6000 m 3 / hr, as specified by the CRAC manufacturer. If no information is provided concerning the area through which the CRAC unit expels this air, the speed of the air as it exits the CRAC unit is also unknown. As an absurd case, this flow rate through a pipe with 2.5 inch diameter corresponds to a flow velocity in excess of Mach 1, which is unrealistic. As the air speed coming out of a CRAC unit relates to momentum the air possesses, which affects how the cold CRAC air mixes with the warmer, ambient air in the data centre, it is clear that even small deviations from the actual CRAC exhaust area will affect the prediction of the cooling air distribution within the entire data centre. Figure 6 shows an example of this effect. A CRAC unit is exhausting 6000 m 3 / hr of air through a perforated tile into the data centre. For Case 1, the CRAC exhaust has an area of 1.7m 2. For Case 2, the CRAC exhaust has a smaller, 1.1m 2 area. Compared to the first case, the air in the second case has moved further into the data centre before it slows to 0.2m/s, due to its higher momentum. This is visible from the larger volume contained within the iso-surface, a virtual surface where the air velocity is a certain value, for the purpose of analysis. Case 1: 1.1m 2 CRAC exhaust Case 2: 1.7m 2 CRAC exhaust Figure 6 Iso-surfaces of Velocity in vertical direction, through floor tiles This increased momentum affects the predicted pressure distribution, and in turn the predicted temperature distribution within the data centre, falsifying CFD results. Considering the example outlined above in conjunction with the assumptions required to model each server within a data centre, it is essential that the modelling techniques used, and any assumptions made, are based on sound physical reasoning. Thorough validation and verification, by means of a comparison with real world results from experiment and data acquisition, is the only means to determine the degree to which results are accurate. Although CFD can be sufficiently accurate to drive design in industries where precision is paramount, it is the CFD methods used which determine the accuracy. It is therefore important that the accuracy of a CFD model, in addition to an outline of these methods, is presented alongside results.
5. Conclusions As a tool, CFD has an unmatched capability to help data centre operators locate sources of inefficiency within their existing facilities. As the method deals directly with the underlying principles of data centre cooling, CFD also brings to light the causes of these inefficiencies, providing operators with a path toward efficiency gains. For the data centre designer, CFD allows temperature distributions to be predicted for facilities yet to be built, providing scientific validation for cooling architecture designs. When utilised early and regularly in the design phase, CFD can be used to drive design, unlocking the most efficient data centre cooling designs possible. Although CFD software is advancing rapidly, a thorough understanding of the underlying physics is still necessary on the part of the CFD engineer to ensure the modelling assumptions and initial data inputted into the simulation, as well as the results, are accurate. This process must be performed in tandem with validation and verification to ensure CFD results are fit for purpose. 6. References and Further Reading Anderson, J.D, 1995, Computational Fluid Dynamics, McGraw-Hill. Chen, Q., Srebric, J., 2001, How to Verify, Validate, and Report Indoor Environmental Modeling CFD Analyses, ASHRAE RP-1133.