White Paper Rack climate control in data centres

White Paper Rack climate control in data centres

Contents Contents...2 List of illustrations... 3 Executive summary...4 Introduction...5 Objectives and requirements...6 Room climate control with the CRAC system...7 The TopTherm Liquid Cooling Package...8 Room climate control with the LCP Passive...9 Row-based climate control with the TopTherm LCP Inline... 10 Rack-based climate control with the TopTherm LCP Rack... 12 Redundant rack cooling with the TopTherm LCP T3+... 13 LCP monitoring options... 14 Energy efficiency... 15 Climate control of small systems... 16 Summary... 17 References... 18 List of abbreviations... 19 2

List of illustrations Figure 1: Energy consumption in data centers... 6 Figure 2: CRAC system... 7 Figure 3: Functional principle of cold aisle containment... 7 Figure 4: Adjustment to meet the data center s requirements... 8 Figure 5: Function of the LCP Passive... 9 Figure 6: Functional principle of the Rittal TopTherm LCP Inline...10 Figure 7: Offset installation of the LCP Inline...11 Figure 8: Flush installation of the LCP Inline...11 Figure 9: LCP Rack...12 Figure 10: Function of the LCP Rack...12 Figure 11: Redundant supply to the LCP T3+...13 Figure 12: Supply channels...13 Figure 13: Optimisation of cooling performance with RiZone...14 Figure 14: Main display menu with alarm...14 Figure 15: Power consumption of EC and AC fans...15 Figure 16: Condenser...16 Figure 17: Layout of the LCP DX system...16 3

Executive summary The LCP from Rittal is ideal for dissipating high heat losses from racks in data centers. The different variants in the LCP family offer both row-based and rack-based solutions. An LCP uses cold water to cool down the air inside a circuit. The water heated during this process is cooled down again by a recooler and then returned to the circuit. The functions of the LCP can be continuously monitored with a PC or management software package. The controller reports any errors to a technician who is able to respond promptly and restore fault-free operation. By using a refrigerant to cool the air a small inexpensive condenser unit may be used to cool the hot water rather than a large overdimensioned recooler which saves space and money. This LCP variant is particularly well-suited to small systems and single enclosure applications. The design and control system of the Rittal TopTherm LCP range make it particularly energyefficient. High useful cooling outputs are achieved even at high inlet temperatures. This allows the proportion of indirect free cooling to be increased which in turn significantly reduces energy consumption enhances efficiency and helps to protect the environment. 4

Introduction These days data centers are at the heart of the IT system in nearly all medium-sized and large companies. This is where all the company s important data and information converge. In recent years data centers have grown steadily larger and their energy consumption levels have risen. A higher performance density also means that the servers produce more waste heat that needs to be dissipated. If heat is left to accumulate in the racks this can lead to a failure in the system; electrical components can even become overheated and cause a fire. A system failure will interrupt operations and data may be lost which in turn incurs high costs and loss of earnings for the company. To avoid such situations it is necessary to use a suitable cooling solution that is capable of dissipating excess heat completely keeping the climate inside the enclosure at a constant appropriate level. Rittal s Liquid Cooling Package (LCP) offers a cooling solution that is not only capable of dissipating up to 60 kw per rack but also has a modular design to accommodate growing cooling demands. An LCP is bayed to an enclosure and supplied with cold water via pipes in the raised floor. The LCP uses the cold water to cool the air inside the rack via a heat exchanger. In this way not only it is possible to achieve a high cooling output but also to achieve the required functional reliability by incorporating redundancies. 5

Objectives and requirements Modern data centers have a vast amount of computing capacity packed into the smallest possible space. As capacity rises however so too does the associated heat loss which must be dissipated in order to prevent overheating and damage to the electronic components. Without external cooling the air temperature inside a data center would rise to extreme levels within a very short space of time leading to failures. This requires an extremely powerful cooling solution capable of dissipating the heat loss in modern data centers completely and maintaining a constant air temperature. It is also important to ensure even distribution of the cool air. Because hot air is lighter and therefore rises so-called hot spots can form at a considerably higher temperature. In individual racks this could cause servers in the top rows to be cooled less effectively than those in the bottom rows. For this reason a cooling solution that distributes the cold air evenly is just as important. The infrastructure of a data center as shown in Figure 1 consumes the same amount of energy as the servers whereby cooling consumes the most energy after servers at 37%. In 2008 the electricity consumption of servers and data centers totalled 10.1 TWh Figure 1: Energy consumption in data centers equivalent to around 1.8% of Germanys total electricity consumption which means that almost four medium-sized coal-fired power plants are needed to supply Germany s servers and data centers 1. Out of consideration for rising energy costs as well as for the environment we need to find energy-efficient solutions are capable of minimising energy consumption while at the same time dissipating rising heat loss levels. 1 Cf. Umweltbundesamt (German Environment Agency) p. 8 6

Room climate control with the CRAC system Bei CRAC systems from Rittal are suitable for installation in data centers containing enclosures with low heat loss levels. These are positioned independently of the enclosure rows and cool the entire ambient air. The hot air is drawn in below the ceiling and cooled. This cold air is then routed to the enclosures in the raised floor and enters the perforated front doors via slotted plates. In this way the servers are able to draw in the cold air cool down their components and emit the heated air to the data center at the rear of the server enclosure. Cold water is used to cool the air in the climate control unit itself. The CRAC systems from Rittal shown in Figure 2 are connected to water pipes laid in the raised floor. One pipe supplies the equipment with cold water (or in the DX variant with refrigerant) while the heated water is transported via a second pipe. The pipes lead to a recooler (chiller) positioned outside the building which cools the water either mechanically or using ambient air. Rittal offers an aisle containment solution for installation in Figure 2: CRAC system data centers designed to improve efficiency. This is comprised of mechanical partitions that physically separate the space between two rows of racks. The enclosures can still be accessed via a special door. Figure 3 illustrates the principle of cold aisle containment. The cold air flows through the raised floor and the slotted plates into the cold aisle where it reaches the servers. The servers emit hot air into the environment where it is drawn in by the CRAC systems cooled down and transported back into the raised floor. The cold aisle offers the benefit of superior efficiency and hence a higher cooling output from the cooling systems. Without aisle containment the hot air would partially mix with cold air from the raised floor and raise the temperature of the air actually reaching the servers. Aisle containment keeps the hot and Figure 3: Functional principle of cold aisle containment cold air separate and stops them from mixing. The efficiency of the CRAC system is boosted and the reduced energy consumption helps to protect the environment. At the same time the performance of the CRAC system is also enhanced. 7

The TopTherm Liquid Cooling Package Rittal s TopTherm Liquid Cooling Package (LCP) offers a cooling solution capable of dissipating heat losses of up to 60 kw from an enclosure. The modular layout means that the data center is easily extended. As shown in Figure 4 the cooling output may be flexibly adapted to suit the current output requirements. 1 2 3 1: Installed excess capacity 2: Capacity adjustment 3: Current power requirements Figure 4: Adjustment to meet the data center s requirements Source: Rittal [1] Unlike a CRAC system an LCP cannot be freely positioned in the room but is instead bayed to the racks. This means that the full 42 U in a 2000 mm high enclosure is available for the server and equipment. Power and water connections are supplied by lines in the raised floor of the data center. Each LCP must be connected to two water pipes for cold and hot water. Hot water from the LCPs is cooled back down to the inlet temperature by a suitable chiller. The Rittal LCP is based on an air/water heat exchanger which uses cold water to cool the air inside the enclosure. Thanks to the division into rack and LCP the water circuit is physically separated from the sensitive electronics and heat is routed from the servers to the LCP's heat exchanger via an airflow. If a defect occurs in the water circuit the electronics inside the rack will not be damaged by escaping water. A leakage sensor allows the LCP to detect the defect and report it to a technician via a security system. One particular advantage of the LCP is the even distribution of cold air across the entire height of the rack. This means that all components are evenly supplied with cold air and no server is unnecessarily burdened with a higher inlet temperature which could shorten the service life of that server. The Liquid Cooling Packages are compatible with racks from Rittal. They are very quick and easy to install both as new installations and as retrofits. 8

Room climate control with the LCP Passive The Rittal TopTherm LCP Passive is fitted at the rear of a rack rather than on the door making it ideal for retrofitting because it eliminates the need to rearrange bayed enclosures when installing. However it is important to check first whether an LCP Passive is suitable. The LCP Passive can only be used if certain infrastructure and ambient conditions are met. Rittal s white paper on Row-based cooling with the LCP Passive outlines in greater detail the requirements and preconditions for use of the LCP Passive and considers the conditions in the data center. These usage restrictions are linked to the layout of the LCP Passive which comprises a special rear door with a built-in heat exchanger. The heat exchanger is also supplied with coolant via pipes in the raised floor but no fans are installed in the unit itself. Consequently the LCP Passive does not require any electrical power. Air flow is created by the server fans alone. However these fans must be capable of overcoming airside power loss by the heat exchanger in the LCP Passive something which should be taken into account and clarified at the general planning stage. Provided all these requirements have been met the LCP Passive may be installed. The cool ambient air is drawn in by the servers and heated. The server fans expel hot air at the rear passing directly through the LCP Passive as shown in Figure 5. The air is cooled down by the LCP Passive and flows back into the environment. This system allows a useful cooling output Figure 5: Function of the LCP Passive of up to 20 kw to be achieved while at the same time retaining 42 U of usable space inside the enclosure. As the LCP Passive does not have any fans and therefore does not require an integral controller it does not consume any electrical power. Very high levels of energy efficiency can be achieved with the LCP Passive. Provided the required infrastructure-related and ambient conditions in the data center are met the LCP offers an excellent alternative for the energy-efficient operation of an entire data center whilst minimising the amount of electricity consumed for cooling purposes. 9

Row-based climate control with the TopTherm LCP Inline The Rittal TopTherm LCP Inline is used for climate control of an entire row of racks. Unlike the CRAC system the LCP Inline is not positioned away from the rows of enclosures in the data center but instead is bayed to the racks. Figure 6 illustrates the functional principle of the LCP Inline. The hot air emitted at the rear of the servers is drawn in by the LCPs and cooled down. The cooled air is then expelled once again at the front. In conjunction with aisle containment from Rittal this ensures that the servers are supplied with cold air. An LCP Inline has a cooling output of up to 60 kw with a footprint of just 0.36 m². Compared with the CRAC system the LCP Inline has a much higher cooling output on a smaller footprint and is capable of dissipating significantly higher heat losses from an individual enclosure. Figure 6: Functional principle of the Rittal TopTherm LCP Inline 10

The LCP Inline may be integrated into a bayed enclosure suite in two different ways. As well as a flush variant (cf. Figure 8) the LCP Inline as shown in Figure 7 may also be installed projecting into the cold aisle. With this variant the fans are rotated through 90 and the air is no longer simply expelled into the cold aisle but is blasted directly in front of the servers. The advantage of this arrangement is that it eliminates deflection losses since the fans are able to expel air freely thereby minimising power consumption and boosting efficiency. In the projecting variant a curtain of cold air" forms in front of the server racks. As the air is expelled evenly in front of the servers this solution also achieves better air distribution. What is more when entering the cold aisle IT technicians no longer have to put up with air being blasted directly into their faces. Figure 7: Offset installation of the LCP Inline Figure 8: Flush installation of the LCP Inline 11

Rack-based climate control with the TopTherm LCP Rack The Rittal TopTherm LCP Rack (Figure 9) has a useful cooling output of up to 60 kw and is mounted on the side of a rack. Unlike the LCP Passive and LCP Inline a separate air circuit is used to cool the components inside the enclosure and the LCP rack. Figure 10 shows the air circuit inside a rack with an LCP. At the rear of the servers the hot air is drawn in at the sides by the LCP fans. The LCP uses the air/water heat exchanger to cool the air and blasts the cold air back out at the front of the servers. The servers use this cold air to lower their operating temperature. The servers are cooled independently of the ambient air inside the data center and cooling may therefore be flexibly adapted to the heat losses of an individual rack and extended in a modular fashion. Up to six fan modules may be installed in a modular configuration so as to adapt the cooling output in line with the actual requirements. This helps to avoid over-dimensioning and the associated higher energy consumption. Figure 9: LCP Rack It is also possible to use a single LCP in conjunction with two racks. The racks are connected to the LCP on the left and right and both are climate-controlled simultaneously. The cold air from the LCP is divided among the two racks. Consequently the sum total of heat loss from the two racks must not exceed the useful cooling output of the LCP used. Figure 10: Function of the LCP Rack 12

Redundant rack cooling with the TopTherm LCP T3+ Whereas the LCP variants Inline and Rack are each connected to a water circuit the Rittal TopTherm LCP T3+ offers a redundant cooling solution that is particularly fault-tolerant and therefore suitable for server racks that experience high thermal loads. The LCP T3+ has a useful cooling output of up to 25 kw with a redundant design. Redundancy is implemented in cases where it is necessary to permanently safeguard a vital function such as cooling of a server enclosure. It is defined as the duplication of functionally identical or equivalent resources in a technical system if these are not normally required during fault-free operation 2. To this end the affected component is equipped with two supply channels. If one supply channel should fail the second ensures full continued operation. Although a redundant configuration is possible with the LCP rack two LCPs are required for "2n" redundancy per server enclosure. The LCP T3+ supports 2n redundancy with just one device. To this end it is connected to Figure 11: Redundant supply to the LCP T3+ two supply channels for electricity and refrigerant. Figure 11 shows the layout of a LCP T3+ system with redundant supply channels. Whereas in normal operation electricity is supplied via one of the two connections only (and the system switches automatically to the second channel in case of an emergency) the supply of refrigerant to the LCP is divided among the two connected channels. As illustrated in Figure 12 each channel can assume full supply if the other channel should fail. In normal operation however the load is divided evenly among both supply channels. In the event of a failure the controller of the LCP T3+ automatically switches to back to Power Cooling standard dual-circuit operation as soon as an intact supply has been restored. Figure 12: Supply channels The controller of the LCP T3+ is extremely fault-tolerant. Cooling always has top priority and if there should be a failure in the control electronics the unit will automatically switch to emergency operation and to maximum cooling output. The intelligent control system means that the LCP T3+ is designed for highly efficient reliable operation. The redundant supply makes it suitable for high-mtbf applications and ensures consistent cooling even if one supply channel should fail. 2 Cf. Wikipedia 13

LCP monitoring options All system-relevant parameters (such as incoming and outgoing air temperature of the servers cooling output water throughput rate etc.) are continuously monitored by sensors in the LCPs and in the CRAC system. An Ethernet interface allows control of the climate control units to be integrated into a company and monitoring network. Administrators can use this interface to access the integral website of an LCP and poll readings or edit parameters and to monitor the LCP from their PC workstation. Figure 13: Optimisation of cooling performance with RiZone Source: Rittal [2] An LCP can also be incorporated into a higher-level network management system via SNMP and automatically monitored. One such management system is the data center management software RiZone from Rittal. This software visualises the current status readings in a data center automatically adjusts configuration values and in this way monitors and controls the individual components in the data center. One possible application is to adapt the cooling output of the LCP and other cooling components (such as recoolers CRAC systems etc.) to the actual load required. An intelligent control system allows the cooling efficiency as shown in Figure 13 to be significantly increased which in turn reduces energy consumption. The LCP is additionally available with an integral display. This can be operated via a touchscreen and displays log files alarms and other more detailed information in addition to current readings. It also allows parameters to be altered so that the LCP may be set directly on site as well as via the PC. The modern-looking interface and clear menu guidance system make it very easy to use. Figure 14 shows the main menu of the new interface displaying the cooling output temperature and any alarm or warning messages. Sub-menus allow current readings to be retrieved and parameters to be set. Figure 14: Main display menu with alarm 14

Energy efficiency Even without a superordinate management software the LCP has numerous pre-integrated functions which make it particularly energy-efficient. Rittal s LCPs use EC fans with a special technology which is more efficient than an AC fan and therefore reduces the electrical power consumption when controlling the airflow. The new controller specifically for IT applications is likewise designed to minimise consumption. Unlike AC fans EC fans do not operate at fixed speeds. They are infinitely controllable which means that they also have stepless power consumption (Figure 15). This can be precisely adapted to the cooling output to avoid under-cooling and prevent the fans from cutting in and out constantly. This also eliminates high recurring start-up currents needed to raise the fan output to a higher level which in turn minimises electrical power consumption. In the LCP the fans are installed in the cold air zone rather than in the hot air zone. In this zone the air has already been cooled down by the heat exchanger and is drawn in by the fans and passed in front of the servers. This arrangement puts less pressure on the Figure 15: Power consumption of EC and AC fans fans and service life is increased significantly so fans need to be replaced less frequently. One particular advantage of Rittal LCPs is that they achieve a high useful cooling output even at high inlet temperatures 3. This supports the use of free-cooling systems. Free-cooling systems use free ambient air to cool the medium and are extremely energy-saving compared with compressors and chillers 4. In countries with colder climatic conditions this indirect free cooling can be used even more frequently than in hotter countries and the chiller s operating hours can be cut to almost zero. The savings achieved with this type of operation are enormous and cooling efficiency is significantly enhanced. However when the chiller is in use the higher supported inlet temperatures ensure efficient operation. An LCP used in conjunction with a chiller has a significantly higher Energy Efficiency Ratio (EER) referring to the ratio between power input in the form of electricity consumption and power output in the form of cooling output. For example a ratio of 5 indicates that 1 kw of electrical power is needed to produce 5 kw of cooling output. In other words the higher this value the more efficient the cooling system. 3 Temperature of refrigerant cooled down by the recooler and passed to the LCP 4 Recoolers that use mechanical processes to cool the medium 15

Climate control of small systems The majority of LCP variants are used in larger data centers that generate high heat losses per enclosure. Due to the large number of racks several LCPs of a given variant may be required. These are generally connected to a large recooler and operated with water (with or without the addition of minimal refrigerant). For small applications with just one enclosure and a low heat loss the infrastructure required for recoolers would be too complex and expensive. For such applications we recommend the "DX" variant from the Rittal LCP product portfolio. This variant uses the refrigerant "R410a" instead of water. Unlike other alternatives this refrigerant boasts a very high level of energy efficiency 5 and supports the use of one condenser unit (Figure 16). Condensers have smaller dimensions than recoolers and place fewer demands on the infrastructure. The LCP-DX and the LCP-CW (Chilled Water using water as a refrigerant) operate in similar ways. The liquid refrigerant R410a reaches the Figure 16: Condenser LCP via a system of pipes where it is heated to Source: KFZtech above its boiling point. In a gaseous state it is then transported to the condenser unit which compresses the gas and returns it to liquid form. During this process the refrigerant emits heat and is thus cooled. The cooled liquid refrigerant can then be returned to the LCP. As condensers are smaller and less expensive an LCP DX is particularly well-suited to small applications with one or two enclosures and low levels of heat loss. Both the LCP Rack DX and the LCP Inline DX achieve useful cooling outputs of up to 10 kw. Both systems are connected to a condenser unit that is sited outdoors (Figure 17). The refrigerant makes the LCP DX very energy-efficient and particularly well-suited to smaller systems with low heat losses. The condenser unit is very small compared with the recoolers used in data centers and installation is quick and easy. 5 Cf. Lohr page 6 Figure 17: Layout of the LCP DX system 16

Summary The racks in modern data centers experience high losses in the form of heat that must be dissipated. When combined with aisle containment a CRAC system from Rittal is capable of dissipating low to medium heat losses. It is not suitable for high-capacity systems. In such environments the TopTherm LCP may be used. It is available in a number of different variants making it suitable for a wide range of applications. With useful cooling outputs of up to 60 kw it is also very powerful and capable of dissipating high heat losses from an enclosure. An LCP is connected to power and water lines in the raised floor and optionally to a data cable for connecting to a network. Cold water for cooling the hot air is provided by a recooler. An LCP may be remotely monitored and parameterised via a PC and the integral SNMP protocol means that it can also be incorporated into a management software package. An optional touchscreen displays the current status and allows parameters to be set directly on the unit. The TopTherm LCP Inline has the same function as a CRAC system. It cools the air from the environment and blasts it into the aisle between two rows of racks. By contrast the LCP Inline is positioned in series with the racks rather than being positioned freely in the room. The rack variant and the T3+ variant of the LCP are likewise bayed to the racks. However they only climate-control one or two enclosures rather than the entire ambient air. To this end a separate air circuit is maintained in the LCP and the rack which only cools the servers of one or two enclosures. Fail-safe operation is facilitated by use of the LCP T3+. Two independent circuits ensure the supply of cooling water and electrical power. Because an extensive cooling solution with large recoolers would be overdimensioned for very small systems and single enclosures the TopTherm LCP DX offers an excellent alternative for smaller applications. Once it has been used in the LCP (for air cooling) the coolant is cooled using a simple condenser unit thus eliminating the need for a large and expensive recooler. The TopTherm LCP family from Rittal is very flexible and its modular design allows flexible adaptation of the cooling output to suit the current output requirements. Data center expansions are quickly and easily achieved with LCPs. At high inlet temperatures an LCP still achieves a high useful cooling output whereby the proportion of indirect free cooling may be increased. This reduces the amount of energy consumed for cooling. The use of EC fans and a controller makes LCPs particularly energy-efficient. 17

References KFZtech: Climate control system Coolant circuit / components http://www.kfztech.de/kfztechnik/sicherheit/klima/klimazwei.htm 24.08.2011 Lohr Dipl.-Ing. Hans-Joachim: Development of an R410A heat pump http://www.lohrconsult.de/uploads/media/entw_r410a_01.pdf 24.08.2011 Rittal [1]: RimatriX5 system solution http://rimatrix5.de/rimatrix5/systemloesung/costs.html 19.08.2011 Rittal [2]: Rittal RiZone. Data Center Management Software http://www.rittal.de/downloads/printmedia/pm5/de/rizone.pdf 22.08.2011 Umweltbundesamt (Federal Environment Agency): Green IT: Zukünftige Herausforderungen und Chancen (Green IT: Future Challenges and Opportunities) http://www.umweltdaten.de/publikationen/fpdf-l/3726.pdf 18.08.2011 Wikipedia: Redundancy (technology) http://de.wikipedia.org/wiki/redundanz_%28technik%29 22.08.2011 18

List of abbreviations AC - Alternating current CW - Chilled water DX - Direct expansion EC - Electrical commutation EER - Energy Efficiency Ratio U - Height unit IT - Information technology kw - Kilowatt LCP - Liquid cooling package PC - Personal computer SNMP - Simple Network Management Protocol 19

20