Using CFD for Data Center Design and Analysis

Transcription

1 Applied Math Modeling White Paper Using CFD for Data Center Design and Analysis By Liz Marshall and Paul Bemis, Applied Math Modeling Inc., Concord, NH January, 2011 Introduction Computational fluid dynamics (CFD) is the numerical simulation of fluid flow. It can be used to predict fluid velocities, temperatures, and many other variables of interest for a wide variety of application areas. Over the years it has been used to simulate the flow of air over an airplane wing or water past a ship s hull; perform comparative drag analyses of automotive body shapes; predict the time needed to mix two or more liquids; and test strategies for reducing coal plant emissions. It is routinely used to model the flow Figure 1: CFD can be used to illustrate the flow of air, as in this example of hot aisle containment in a data center; flow pathlines are colored by temperature inside buildings of all sizes and rooms with specialized functions. This white paper focuses on how CFD is used for one such class of rooms: data centers (Figure 1). It begins with an overview of CFD basics and then describes how data center components can be represented using numerical methods. The chapter concludes with examples and suggested reading material. 1

2 Computational Fluid Dynamics The goal of a CFD calculation is to predict the highly detailed flow field in some region of interest. The flow field is not just the motion of a fluid: a liquid or a gas or even a collection of small particles, such as sand. CFD can also predict the distribution of temperature and chemicals. It can include other phenomena such as turbulence, timedependence, chemical reaction, evaporation, melting, or freezing. Despite the range of applications, the same general method is used. Conservation equations are solved in small cells that fill the region. The region must therefore be divided into a large number of cells and the equations must be rewritten so that they can be solved in each one. In the next few sections, these two processes are described along with an overview of how the solution is performed and how the results can be viewed and interpreted. The Computational Domain and Mesh A CFD simulation begins with the creation of the problem geometry. The region to be modeled is often called the computational domain, and the geometry consists of the boundaries of that domain and all of the objects contained within. While computer-aided design (CAD) software focuses on the solid objects, CFD focuses on the space inside or in between objects, where the fluid is likely to flow. Many application-specific CFD packages are available nowadays that make geometry creation straightforward. For data center applications, the CFD software has objects available for CRACs, racks, PDUs, perforated floor tiles and other common components, which can be positioned and sized easily within a room. Once the room and contained geometry are created, the entire space needs to be broken up into a large number of small cells where the calculations will be performed and the resulting information stored. The lines and planes that separate these cells are often collectively referred to as the mesh or grid. The cells can be of almost any shape; six-sided hexahedra, five-sided pyramids or prisms, and four-sided tetrahedra are the most common. The task of creating such a mesh can be time-consuming for the CFD analyst, but many automated or semi-automated tools are available in software today. Data center CFD software packages are among those that offer this feature. A typical mesh in a data center is shown in Figure 2. This mesh is non-uniform, which means that the size of the cells varies from region to region, depending on the detail required by the local geometry. Figure 2: The mesh on a planar slice through a data center 2

3 While the mesh in Figure 2 appears Cartesian, with lines at right angles, automatic mesh generators often distort the elements to fit around non-rectangular objects. One example is shown in Figure 3, where the mesh is distorted near the circular object and actually changes from Cartesian to a different style (called O-type) around some of the objects. Automatic meshing algorithms have instructions to deviate from a standard Cartesian format based on the needs imposed by the geometry. Often, a switch to another grid style will result in a reduced cell count in the domain, making the calculation run more quickly. Figure 3 : A Cartesian mesh is used for most of the region and around the two elements in the middle, while O-type grids are used for the rectangular and circular elements on the sides Conservation Equations If a small volume of fluid in motion is considered, two changes to its shape will usually take place. First, the fluid element will translate or rotate in space, and second, it will become distorted, either by a simple stretching along one or more axes, or by arbitrary twisting. The process of translation is referred to as convection, while the process of distortion is related to the presence of gradients in the velocity field and a process called diffusion. In the simplest case, these processes govern the evolution of the fluid from one state to another. Other factors can also contribute to changes in the fluid. For example, heat can cause a gas to expand and rise. Conservation equations include these effects and provide a generalized description of fluid motion. These equations track changes in the fluid that result from convection, diffusion, and sources of the conserved quantity. Conservation equations are coupled, meaning that changes in one variable (say, the temperature) can give rise to changes in other variables (say, the velocity). In most CFD products targeted toward a particular application, such as data center airflow, control of the equations during the solution process is shielded from the software user. Thus while it is important to have an appreciation of the equations behind the scenes, it is usually not necessary to understand the various techniques and controls for solving them. Continuity The equation of continuity is a statement of mass conservation. To understand its origin, consider the flow of a fluid of density through the six faces of a rectangular block, as shown in Figure 4. The block has sides of length x, y, and z and velocity compo- 3

4 Figure 4: A computational cell of volume x y z and velocity components u, v, and w in the x, y, and z directions, respectively nents u, v, and w in each of the three coordinate directions. To ensure conservation of mass, the sum of the mass flowing through all six faces must be zero. u out uin y z vout vin x z w w x y 0 out in Dividing through by ( x y z), the equation can be written as: u v w 0 x y z (1) (2) Momentum Similar logic can be used to construct equations for the conservation of momentum in each of the three coordinate directions. Momentum conservation is more complicated than mass conservation, however. Momentum sources such as gravity and pressure gradients play a role. In addition, a term representing the diffusion of momentum is included. This effect is attributed to gradients in the velocity field. Consider, for example, how a jet of water spreads (momentum diffusion) after being injected into a pool where the water is at rest on either side of the jet. Taking these effects into account, the momentum conservation equation in the x-direction for constantdensity fluids takes the form: u u u u u v w t x y z p u u u x i x y z g F x x (4) or, in differential form for the more generalized case where the density changes in time (t) and space: ( u) ( v) ( w) 0 t x y z (3) Here is the fluid viscosity, g is the acceleration due to gravity, and F represents any additional forces that might be present. The term multiplied by is the diffusion term, described briefly above as one that arises from gradients in the flow field. The three momentum equations (for u, v, and w components of velocity) along with the continuity equation are collectively called the Navier -Stokes equations. The four equations are used to solve for the four unknown variables: 4

5 three velocity components and pressure, even though the pressure appears only in the momentum equation. Numerical solution methods are used to link the equations through the density. Turbulence and Energy In most applications, more than just the Navier-Stokes equations are needed to describe the fluid behavior. If the Reynolds number is high enough, turbulence plays a role. The primary effect of turbulence is to increase diffusion. In the momentum equations, the viscosity effectively increases in the presence of turbulence. Thus water tends to act more like honey. Various approaches to include turbulence in the momentum equations are available. The simplest approach is to add a correction to the viscosity based on local flow and geometric parameters. More involved approaches require the solution of turbulence transport equations. Timedependent calculations can also be done to capture the random fluctuations that result from turbulent behavior. Many enclosed room applications, such as data centers, use the simpler approaches to include the effects of turbulence. In addition to turbulence, temperature can play a role in the flow field if the fluid density varies with temperature (as is the case if the ideal gas law is used). A conservation equation for thermal energy (analogous to that for mass and momentum) must also be solved. If the flow is turbulent, the transport of thermal energy is impacted through the conductivity of the fluid. Turbulence acts to increase the fluid s ability to conduct heat, or in other words, thermal transport through the fluid is more diffusive. Solving the Equations If each cell in the domain has a point at its center, the task of a CFD solver is to obtain a solution for all of the relevant conservation equations at every such point. If the domain is characterized by 100,000 cells and there are five conservation equations being solved, that means that 500,000 solutions must be obtained! These numbers are actually modest by today s standards, where models with tens of millions of cells are common and the physics requires at least seven separate equations to be solved. There are a number of numerical methods in common use by CFD solvers. In short, the differential equations are converted to algebraic equations. To compute how well those equations are balanced in each cell, the values of all variables at the cell faces must be computed. The manner in which those face values are computed differs for different solution approaches. The numerical method, the density of cells, the cell topology, and the nature of the flow field combine to impact the accuracy of the resulting solution. Residuals The set of coupled equations described above cannot be solved exactly. Instead, an iterative solution technique is applied. Roughly speaking, a guess is made for each variable at each point in the domain and the error in each conservation equation is tabulated. That is, the code takes stock of how closely the left side of each equation matches the right side. The cumulative error for each 5

6 equation is called the residual. The goal of the solver is to force the residuals to decrease with each step in the iterative solution process (Figure 5). Once the residuals have fallen below some preset value, the solution is considered converged. Only results from converged solutions are meaningful. Figure 5: Normalized residuals are shown to decrease during the iterative solution procedure continuity Displaying the Results Once a solution is converged, the results can be examined in a number of ways. Consider, however, the fact that there may be thousands or millions of points of data throughout the domain. How can one make sense of it all? One of the simplest ways to examine the results is to do so one plane at a time. Once created, a plane can be used to illustrate contours of a variable or velocity vectors. A plane can be created at some x, y, or z location or at the surface of an object, such as the inlet side of a rack row. An iso-surface, by momentum contrast, is a surface, usually not planar, where one variable has a constant value. For example, an iso-surface of constant temperature can be created and then contours of another variable, such as pressure, can be drawn on that surface. This type of display can convey more complex information than contours of a variable temperature plotted on a flat plane. Pathlines are another popular choice for examining results. These trace the paths of imaginary fluid elements that are released from one surface (say, where air enters a room). Pathlines can be colored by other problem variables, such as temperature, and they can usually be animated. In Figure 6, a raised-floor data center is shown that contains two rows of high density racks in the lower left. The racks are shaded by temperature contours and pathlines, also shaded by temperature, are released from the exhausts of these racks. The results show that one side of the high density racks runs hotter than the other side. As a result, the air returning to one of the CRACs is hotter than that returning to the other, a situation that places an uneven burden on the cooling system. An analysis such as this can illustrate imbalance in a data center that can result from equipment positioning or the presence of underfloor obstructions. 6

7 Overview of the Modeling Process One of the most important uses of CFD is as a planning tool. In this mode, an existing design is modeled and if the predictions are in good agreement with measurements, the assumptions are assumed reasonable and the model is altered to reflect one or more prospective changes. In the automotive industry, for example, such studies are done to test body shapes for aerodynamic characteristics before any metal is bent. The method is ideally suited for data centers where ongoing changes to equipment are routine. Using CFD, a number of scenarios can be studied before any equipment is moved, changed, or added. The success of this approach depends on the accuracy of the CFD model. To test for accuracy, it is a good idea to build a model and compare it against measurements (where Figure 6: A CFD model of a raised-floor data center showing temperatures on the surfaces of the rack rows and pathlines of return air, shaded by temperature available) first. Simple models are a good starting point. In a data center, a simplified model may use average heat loads in the racks, neglect small piping in the supply plenum, and make use of a coarse mesh, for example. A comparison of the simple model results with measurements will usually point to discrepancies, and these can be remedied by modifications to the model on the next pass. A refined model will therefore include more geometric detail and make use of a finer mesh. The choice of what detail to add can best be made by doing the first comparison with a simple model. For example, suppose the simple model neglects all of the small pipes in the supply plenum and treats all of the racks with average heat loads. If the flow distribution through the floor tiles is in good agreement with measurements but certain hot spots in the room are missed, 7

8 tain good agreement with data. In some cases there may not be good agreement with data after multiple refinements, and the root cause(s) should be investigated. Poor agreement could be the result of inaccurate assumptions about the heat loads and flow rates, especially if the fans in the equipment are variable speed fans and the model does not take that into account. Measurements can also be at fault, so analysts should always be aware of the error recorded. Figure 7: A flow chart showing the modeling process, starting with a simple model and refining it as needed before using it for analyzing prospective scenarios chances are that neglecting the small pipes was an acceptable approximation but neglecting the server-level detail in the racks was not. Once a refined model is in good agreement with data, that model can be used as the base case for modifications to the room. A flow chart illustrating this process is presented in Figure 7. Note that more than one refinement to the model may be necessary to ob- Techniques for Modeling Data Center Equipment CFD is used to simulate many different types of applications, and for each, the components that are common to that application are usually treated in a unique way. For example, a light source might be modeled as a volumetric heat source in the geometric shape of the light housing or bulb. A fan might be modeled as a planar area through which a certain flow of air is specified. In data centers, similar approaches are applied to model the components that are familiar to facilities engineers: CRACs, racks of equipment such as servers and switches, UPSs and PDUs, for example. In this section, methods for representing data center equipment are reviewed. 8

9 CRACs Computer room air conditioners (CRACs) or air handlers (CRAHs) are large rectangular objects that draw in warm exhaust air from the heat generating equipment in the room, cool and condition it, and then return it to the room for reuse. Under normal operation, they have a fixed flow rate and are capable of extracting a certain amount of heat from the air. A CFD model of a CRAC usually consists of a rectangular block with one or two fans used to move the air (Figure 8). In one scenario the volume inside the CRAC is not meshed so is not part of the solution domain. A return fan is used to draw ambient air out of the room while a second supply fan operating at the same flow rate is used to supply the cooled air back to the room. Some models of CRAC employ two or three supply fans and a single return fan. In another scenario, the volume inside the CRAC is meshed so that the inside air flow can be simulated. A heat sink is associated with the fluid inside the CRAC and a single fan is used at the supply side, modeled using a fan performance curve. the cold supply air in the room. For nonraised floor data centers, topflow or upflow CRACs are used to supply cold air from the top of the unit, with the return air passing through grilles on one side of the unit near the floor. Upflow CRACs are sometimes used along with downflow CRACs in raised floor data centers. Frontflow or in-row coolers are typically positioned between racks in a data center. They draw warm air from the rack exhausts, often the hot aisle in a hotaisle/cold-aisle arrangement, and produce cooled air out the opposite side, adjacent to the rack intakes. In-row CRACs may be used with or without other types of CRACs in the data center. Overhead cooling, often referred to as supplemental cooling, is available in overhead ceiling-mounted or rackmounted units. As for in-row coolers, these units act locally, circulating and cooling the air from nearby racks of equipment. Despite the many types of CRACs, they can all be modeled using a hollow block and pair of fans with matched flow rates. Fans Downflow, Upflow, In-Row, and Overhead The position of the return and supply fans depends upon the nature of the room and type of cooling application. Raised-floor data centers usually use downflow CRACs, with the supply fans directing the cooled air into the supply plenum beneath the floor and the return fans on the top of the unit. The cooled air enters the room through perforated tiles in the floor. If a ceiling plenum is used for the return air, these CRACs usually extend up to the ceiling plenum so that none of the warm return air has a chance to mix with Figure 8: A CRAC consists of a hollow block, a return fan, and one or more supply fans 9

10 Plug Fans While a great number of CRACs discharge cooled air in a direction normal to the CRAC boundary (straight down for downflow CRACs, for example), plug fans are gaining in popularity. For raised-floor data centers, plug fans extend below the lower surface of the CRAC into the supply plenum where they discharge air radially. Such fans can be modeled using a cylindrical fan in CFD. That is, the fan surface is not a plane but the wall of a cylinder. The volume inside the cylindrical fan is not meshed and is not part of the calculation domain. Cooling Methods While it is a simple matter to specify the temperature of CRAC supply air, most CRACs operate in a more sophisticated manner. In practice, the heat exchanger may cycle on and off so that a certain thermostat temperature is maintained at the CRAC return. In CFD, this can be accomplished using an iterative calculation of the following equation: Q mc T T ) (5) P ( R S Where Q is the amount of heat removed from the air (in SI units of W), m is the mass flow rate of air through the unit (kg/s), c P is the specific heat of air (J/kg-K), and T R and T S are the return and supply temperatures (in K), respectively. An iterative CFD calculation periodically adjusts the supply temperature (T S ) until the return air temperature matches the thermostat temperature. For many CRACs, the manufacturer supplies performance data for the cooling capacity (Q) as a function of return temperature. The relationship in Equation (5) is used, but the value for Q changes for different return temperature ranges. Racks Whereas CRACs extract heat from the air, racks and cabinets full of equipment add heat to the air that passes through them (Figure 9). Servers are rated by their heat load. Assuming a certain temperature rise across a server with a given power rating, Equation (5) can be used to compute the flow rate required to maintain that temperature difference. For example, to maintain a 20 F temperature rise across a server, about 150 CFM of cooling air is needed for every kilowatt of power. Racks can be modeled using their total heat load or by modeling the individual components with blanks and/or gaps in between. Whether a total heat load or equipment-level detail is used, the volume inside a rack need not be modeled. Instead, a matched set of flow boundary conditions can be used that ensures that the same flow rate enters and leaves the rack (or equipment) and that between those matched boundaries a certain temperature rise will occur. The temperature rise can be specified directly or can be the result of a specified heat load. Note that the volume inside a rack can be meshed and modeled, if desired, as in the case of the single-fan CRAC. This choice would be appropriate for analysts looking to model the temperature distribution inside individual servers for a given set of external conditions, for example. Racks can be modeled with air flow from front to rear, front to top, bottom to top, or 10

11 side to side. Flow entering the front can also be split to pass out the rear and top of the unit. Flow exiting from the top of a rack is an acceptable strategy if the rack is modeled with an average heat load. For a rack modeled with individual servers, it is not good modeling practice, because it is difficult to estimate how the air from each of the individual servers contributes to the top exhaust and the cooling of the other equipment. However, if a channel of exhaust air from the back of a rack of equipment is included in the model, the front-to-top flow strategy can be represented. In raised-floor data centers, underfloor cable trays that feed power to the racks do so through cutouts in the floor, which are usually under the racks and characterized by some percent open area. Openings in the floor allow supply air to mix with the air entering through the rack inlets. In CFD simulations, if cable cutouts are important, it is best to model the rack using an average heat load. Otherwise, the air drawn up from the plenum will mix with the cold air entering the server closest to the floor only. PDUs and UPSs In addition to racks, PDUs and UPSs are heat -generating components that require cooling. In raised-floor data centers, the cooling air is drawn through the floor into these units (Figure 9). Alternatively, cold air is drawn into the unit from side grilles. The heated exhaust air exits through the top of the unit. These components can be modeled like racks, using a matched set of openings with the same flow rate applied to both boundaries with heat addition between. They can also be meshed and given a volumetric heat load in the interior. For this modeling method, openings at the bottom and top will allow flow to be drawn up and through the device as a result of natural convection. Figure 9: Racks (right) and PDUs (left) consist of hollow blocks with openings (red or pink) for air flow and a heat gain Perforated Tiles and Ceiling Grilles Perforated tiles (Figure 10) allow the cold air in the supply plenum to enter a raised-floor data center where needed, usually in front of the rack rows. These tiles are characterized by some percent open area which causes a drop in pressure. They also serve to straighten the flow as it passes through. To accomplish these modeling goals, tiles are typically modeled using a thin porous region. The porous region (a volume) is assigned loss coefficients for 11

12 of air can be uniform across the diffuser or have some prescribed profile, such as high in the center and low at the edges. Figure 10: Perforated tiles (purple) are threedimensional objects that can straighten the flow as it passes through the three component directions. By making the loss coefficient much smaller in the direction normal to the floor, flow from the supply plenum will tend to align with that direction as it passes through the region. Cable cutouts on the floor of a data center not underneath racks - are usually modeled in the same manner. Ceiling grilles are partially open tiles in the ceiling. They cause a pressure drop for the flow as it enters the ceiling plenum, but they do not necessarily straighten the flow. Thus they can be modeled by a zero-thickness face or area that has a loss coefficient that gives rise to the appropriate pressure drop. Ducts and Diffusers In non-raised floor data centers, and in some raised-floor data centers, ductwork is used to transport cold (or warm) air as needed. The air is released to the room through diffusers, which can be set to direct the air at an angle or straight down into the room. The velocity To model a duct using CFD, the volume inside the duct can, but need not be meshed and included in the calculation. If the duct volume is not included, assumptions can be made to compute the properties of the air exiting from the diffusers. First, the air from the participating CRACs can be assumed to be well mixed, so the temperature at the diffusers can be computed using a weighted average of that produced by the CRACs. Second, the total mass flow produced by the CRACs can be divided among the diffusers and scaled to match the area of each. An assumption of equal flow rates is good if the diffusers are of similar size and not positioned in the middle of recirculation zones inside the ducts. Diffusers immediately following right-angle bends in the ductwork might not satisfy this requirement as well as those in the middle or at the end of a long section of ductwork. By including the duct volume in the calculation, the temperature and flow rate distributions for all participating diffusers in a duct system will be computed as a result of the flow inside the ducts. What to Include in a Data Center Model For the purpose of the CFD model, a data center is a closed environment with very little leakage of air or heat into or out of the room. This means that all heat produced by equipment in the room must be removed by the cooling units. All of the flow demands of the equipment should (but need not) be met by the cooling units as well. Thus when choosing what items to include in the model, 12

13 the most important are the major heat generating and removing ones, and any objects that contribute to the air circulation: all CRACs, CRAHs, racks of equipment, power generating stations, floor tiles, ceiling grilles, ducts and diffusers. The next level of importance consists of objects that impact either the flow or mixing of air in the room. This group includes baffles that are used for cold or hot aisle containment, turning vanes or other baffles in the plenum used to direct the supply air, plenum blockages and inert (non heat- or flowgenerating) objects in the room. While baffles can be set up to have some percent open area, impermeable baffles should be used with care. If baffles are used to section off an office or closet that does not contribute to the air flow in the room, the CFD solver may have difficulty, since the solution domain should be comprised of only a single connected volume of air. Side rooms that are not participating in the overall flow in the data center should be blocked off using solid objects such as blocks or slab-to-slab columns. Overhead and underfloor cable trays can impact the flow of air but may be deemed unimportant by the analyst, depending upon their size and position. Of less importance are small objects, especially in large data centers. Small pipes in the plenum (less than 3 in diameter, for example) fall into this category. Tables and chairs, filing cabinets and gas cylinders may be deemed too small or too few to make a difference in the flow field. Small heat generating objects, such as overhead lights or laptop-sized control stations may be contributors to the heat load in the room, but their role may be small compared to other equipment. If people are present most of the time in the data center (seated at a desk), their contribution to the heat may also be included. Finally, windows with minimal shading may also play an important role. Examples Two cases are presented in this section to demonstrate how CFD can be used to assess a data center. The first case is a raised-floor data center and the second has a non-raised floor and makes use of ducts and diffusers to distribute the supply air. Raised Floor Data Center The 1800 sq. ft. raised-floor data center shown in Figure 11 has an assortment of racks (light gray in the middle of the room), eight CRACs (dark gray along the walls), five PDUs (dark gray but smaller than the CRACs), and a ceiling plenum return. The downflow CRACs are each modeled with a cooling capacity and flow rate unique to the Figure 11: An isometric view of a raised-floor data with a ceiling return 13

14 unit, and each is set with a thermostat temperature of 75 F. Some of the racks are modeled using an average heat load while others cannot be approximated in this manner so are represented as a collection of servers, blanks and gaps. Each component in the racks is characterized by a heat load, an anticipated temperature rise, and a flow rate, controlled by a fan. The PDUs are assumed to draw air from the plenum and exhaust it out the top. Air is drawn up into the room through perforated floor tiles, all of which have a 25% open area. A summary of the heat loads and cooling capacity in the room suggests that there is 6% excess airflow and 3% excess cooling delivered by the CRACs. Despite the fact that the cooling capacity cuts very close to the heat load, the detailed CFD results indicate that all racks pass the recommended ASHRAE guidelines for inlet temperature falling below 80.6 F (Figure 12). There are a few reasons for this success. First, all perforated tiles are positioned directly in front of rack inlets. Second, there are no underfloor obstructions to divert the flow away from where it is needed. Third, the tiles offer uniform resistance to the flow because they have the same percent open area. The results support this finding: the average flow rate through the tiles is 681 CFM, with a maximum deviation of 14% with most tiles falling well within 8%. While the CRACs are not all the same, all are able to meet their target thermostat temperature to within a few degrees. Figure 12: Contours of temperature on rack and server inlets for the raised-floor data center The room does have some design flaws, however. First, the total cooling capacity of the CRACs is too close to the total heat load in the room. Thus if one of the CRACs were to fail, the others would have difficulty maintaining a 75 F thermostat temperature for the return air. This problem would need to be remedied by replacing one of the CRACs with a different unit. CFD could be used to determine the best choice of CRAC to replace. Second, the absence of underfloor obstructions gives rise to a large recirculation of air in the plenum, as is shown in Figure 13. Often, recirculation patterns such as this cause negative pressures in the eye and result in the flow of air from the room into the plenum. Had the power demands on the racks in the middle of the room been greater, enough supply air might not have been available. Baffles can be used to redirect supply air in a plenum so that recirculation patterns, especially large ones, don t develop and adversely impact the distribution of air through the perforated tiles. 14

15 Figure 13: Pathlines in the supply plenum, shaded by speed; the darker lines represent the more slowly moving supply air, most of which is shown after it enters the room Non-Raised Floor Data Center A non-raised floor data center with four CRACs in a middle aisle is shown in Figure 14. The data center is 800 sq. ft. with a heat density of about 81 W/sq. ft. The rack heat loads vary from 500 W to 4 kw. The CRAC returns are on alternating sides of the units; two face the right side of the room and two face the left side. Supply air mixes in a main duct above the CRACs and is dispensed through diffusers to cold aisles between the rack rows. A summary of the equipment in the room indicates that the CRACs provide 26% excess air and 27% excess cooling than the racks demand. While the room appears to have adequate cooling, the CFD results paint a very different picture. While all of the racks pass the allowable ASHRAE thermal guidelines (59 F to 90 F), none passes the recommended guidelines (64.4 F to 80.6 F). Despite the fact that all of the diffusers introduce cooling air at 60 F, the average maximum temperature delivered to the rack inlets is over 85 F. The reason this design fails is because the exhaust air from the racks circulates over and around the rows and mixes with the supply air on the opposite side. That is, even though there are cold aisles below the diffusers, the aisles are short and wide, allowing the rack exhaust air to infiltrate the cold zone and degrade the cooling air. Furthermore, the CRAC returns are visible to nearby supply ducts, a design that encourages the cooling air to short-circuit back to the CRAC returns before being fully Figure 14: A non-raised floor data center with a center aisle of top-flow CRACs and ducts and diffusers to deliver the supply air to the rack rows 15

16 utilized by the racks. In Figure 15, pathlines released from the rack outlets show how the exhaust air travels over the tops and around the sides of the rack rows so that it can be drawn in by the intake fans on the opposite side. The mixing of exhaust air with the cooler supply air at the inlets raises the temperature of the rack cooling air, especially near the tops and sides of the rack rows. It is interesting to note that a CRAC failure analysis done on this room shows that the performance of the CRACs is the same or better when each one of the CRACs is turned off. This unexpected result is due to the fact that a disabled CRAC will not short-circuit the supply air from the nearest diffuser directly to the CRAC return. This allows more cold air to be available for the pair of rack rows nearest the diffuser(s). Figure 15: Return pathlines, shaded by temperature, show some of the warmed exhaust air being entrained by the rack inlets, effectively raising the temperature of the cooling air Data Center Modeling Best Practices In recent years, several articles have been written on Best Practices for CFD modeling. These recommendations are typically targeted toward CFD analysts who have full control over the mesh generation and solver options. Even so, users of applicationspecific software should also be mindful of certain practices that ensure success. A short summary follows. Avoid putting too much detail into a model on the first pass. Get a general sense of the flow in the room before complicating the model with too many objects or details. Remember that a CFD model is different from a CAD model, where more detail may be better. Avoid small gaps. If a CRAC is 2 from a wall, there is probably no need to model the air between the CRAC and wall. Move the CRAC so that it is flush against the wall in such cases. Start with a coarse mesh solution and progress to a fine mesh only after confirming that the results make sense. Avoid separated regions of air. If adjacent rooms do not contribute to the overall flow in the data center, block them out of the model using a solid object. Understand how to use the software and its limits. Go through the 16

17 recommended training before starting to use the product. By adhering to simple guidelines such as these, most common modeling problems can be avoided. While early CFD software products were most often used by specially trained CFD analysts, today s applicationspecific products can be used successfully by engineers, designers, and technicians with a wide range of backgrounds. When choosing a product, a prospective user should consider ease of use, capabilities to suit a given set of needs, and availability of technical support. For example, if rack-level detail will be an important component of a planned analysis, a product should be chosen that includes this feature. Conclusions and Future Trends With data center managers and designers looking for innovative ways to cut energy costs, CFD is quickly becoming an essential tool of the trade. The flow field in a room can expose problem areas that don t show up in simple heat and flow balance calculations. In addition, trial layout modifications can be quickly evaluated using CFD prior to moving or replacing equipment. CFD is also useful for illustrating the division of labor among the cooling units in a data center and for illustrating the room conditions should one or more of them fail. centers as the loads vary. Dynamic CFD simulations will be able to identify the cooling distribution among CRACs and air handling units as server loads rise and fall with varying activity. In this manner, CFD will contribute to the control system that governs the overall facility cooling, activating certain units while deactivating or simply ramping down others. The potential feedback from such a system will not be instantaneous, but will be sufficient for the timescales typical of these fluctuations. Suggested Reading Two books provide excellent, readable summaries of CFD methodologies. Numerical Heat Transfer and Fluid Flow by Suhas V. Patankar (Hemisphere Publishing Corporation, 1980) is the early standard that has helped guide the growth of the field. An Introduction to Computational Fluid Dynamics, The Finite Volume Method, by H. K. Versteeg and W. Malalasekera (Prentice Hall, 1995, 2007) contains more recent examples and techniques that have been developed since Patankar s book was published. While Patankar focuses on Cartesian (or cylindrical) meshes in his examples, Versteeg and Malalasekera describe other mesh topologies as well. In addition, they provide discussions of some sophisticated physical models such as combustion and radiation. In the future, as computing speeds increase and hardware costs drop, CFD will become more integrated in data center monitoring systems. Rather than just be used for static assessments of data centers, CFD will be used in real time for dynamic models of data 17