Circuit Breakers in Data Centers: The Hidden Danger

Transcription

1 Circuit Breakers in Data Centers: The Hidden Danger How to efficiently protect critical IT equipment by choosing the most reliable overcurrent protection device for your Rack Power Distribution Unit. By Dr. Jim Anderson and Mike Jansma Contents Introduction... 1 How Are Servers Kept On All the Time... 1 Overcurrent Protection Devices... 1 The Problem with Fuses in a Data Center... 3 Circuit Breakers and PDUs: A Match Made In Heaven... 3 All PDU Circuit Breakers Are Not Created The Same... 4 Conclusion... 6 About the Authors... 7 About Enlogic... 7

2 Introduction How much does a modern data center cost? Millions? Hundreds of millions? The total cost of building a data center is increasing every day. Why do these buildings cost so much? The answer is simple: the job that they do is hard to do perfectly. A modern data center is designed to provide a secure, reliable home to hundreds (or even thousands) of servers servers that run mission critical software for companies all over the world. One part of a data center's job is to provide reliable, clean, power to each of the servers in the data center. Here in the 21 st century, that part of the job would be relatively simple, right? In our homes, when we flip switches things happen: lights turn on, the computer powers up, or the TV flickers on. However, think about all of the times that you've lost power or how often the lights flicker. A data center is designed to prevent these events. When it comes to the job of providing power to servers, data center designers have two primary tasks to accomplish: to provide reliable power and avoid unsafe operating conditions. But doing this is more difficult than you might imagine. How Are Servers Kept On All the Time? To ensure that servers always have power, data center designers go to great lengths. They often build two completely separate and redundant electrical power distribution systems along with multiple types of power backup solutions. These expensive and complex solutions ensure that if power to the data center is ever lost, backup systems can quickly provide the needed energy. Additionally, if there is ever a failure of one of the redundant power systems, the secondary system can provide the required electricity until the failed system is repaired. What makes all of the expensive design so fascinating is that it all comes down to a single component to ensure that all of the servers continue to receive the energy required in order to stay powered up: the rack power distribution unit (rack PDU). The rack PDU receives electricity from the data center's redundant power delivery systems and it is responsible for delivering that energy to the individual servers and IT equipment. If the rack PDU fails, then the attached servers will fail. The rack PDU is a data center operator's last line of defense against accidental circuit overloads and harmful faults. In many countries, the rack PDU includes design features that are required by electric code and mandatory industry standards requiring protection against these events. As such, rack PDU vendors incorporate solutions to for these occurrences. Overcurrent Protection Devices While most servers and IT equipment have become more energy efficient, the total energy demand required per unit has risen dramatically over the past 15 years from an average power density of 1-2 kw to an average of 4-6 kw per rack. Some high density IT racks have power loads of 20 kw and above. Page 1 of 7

3 These increasing rack power densities have led to dramatically larger input power feeds to the rack PDU devices. Rack PDUs are no longer built similarly to the power strip under your desk. Today, the most common power feed is a single phase, V input rated at 30-32A (depending on regional codes). Some have three-phase inputs with ratings as high as 23 kw and higher. At these high power densities, the amount of heat generated in the electrical conductor wires becomes substantial and dangerous if not limited to thermal design specification. These thermal limits are strained as the data center hot aisle temperatures become warmer due to higher power density. Thermal limits are additionally burdened with energy efficiency initiatives such as raising the operating temperature due to the new data center industry thermal standards (set about by ASHRAE). When electrical circuits are exposed to overcurrent or excessive temperatures, the conductor insulation melts and there is a real danger of smoke, fire, and electrical shorting faults, all of which create a danger to data center workers. Today s rack PDU requires Overcurrent Protective Devices (OCPD) to mitigate the overcurrent risks in the rack PDU and IT equipment. As more IT equipment is consolidated into fewer, more densely packed IT equipment racks, it has become critical to isolate potential electrical faults within one server rack in order to protect upstream power distribution equipment. There are 2 major types of OCPD used in today's rack PDU: supplemental protectors and branch-rated OCPD. Supplemental protective devices are devices that serve a single purpose: limit the total current output of a circuit. However, they are prone to inconsistent protection characteristics, highly sensitive to the surrounding thermal environment, and incapable of protecting circuits in failure situations. These types of highly disruptive faults pass by the supplemental protector (often the device fails) and usually result in upstream circuit breaker tripping to contain the fault with additional servers and IT equipment placed at risk for surge damage and dropped loads. These simple overcurrent protective devices might conform to and be labeled with industry standards including UL1077, UL489A, or other local standards. Conversely, branch rated circuit OCPD devices provide comprehensive protection against simple circuit current overload and isolation protection for harmful equipment circuit fault conditions. As the power density of IT racks climb, these fault protection devices become more critical to both reliability and safety. In many cases, protection is required by electric codes and industry standards. When a short circuit or ground fault failure occurs, the branch rated OCPD can normally respond by opening the circuit to isolate the fault to only the attached equipment. Often these OCPD devices are labeled to conform to branch rated standards such as UL. Figure 1 shows some typical examples of the OCPD used in IT rack PDUs. To further complicate the selection process, there are multiple technologies used in both supplemental and branch rated OCPD devices. Simple thermal fuses are the most basic technology, while resettable circuit breakers are the most common OCPD device utilized in most rack PDU devices. The most professional-grade rack PDUs will use non-thermal based, hydraulic-magnetic (hy-mag) circuit breakers to provide the most comprehensive overcurrent and fault protection within the rack PDU. Page 2 of 7

4 The Problem with Fuses in a Data Center How does a PDU keep the data center servers safe from dangerous overcurrent and fault conditions? There are many ways for a PDU to do this, but the least expensive method (in terms of cost to PDU vendors) is by the use of thermal fuses. Fuses are a special type of resistor that exists for one purpose: to fail when exposed to electrical currents beyond the designed capacity. A fuse works by flowing the electrical current through a piece of metal designed to melt when the current exceeds the rated limit. When an overcurrent condition occurs, the metal melts and the circuit is prevents electrical flow. The servers will lose power; however, if the fuse melts quickly enough the servers will not be damaged or destroyed. Because the fuse is designed to melt as a function of temperature, the use of fuses in thermally hot environments requires a derating of the current carrying capacity. In a data center, this means that a fuse rated for 16A or 20A might melt (or fail) at electrical currents lower than the designed. Failure to account for this derating results in a tripping of the fuse during normally safe operation. Fuses have other negative attributes for use in a data center. Sure, a fuse can step in and save the servers in an overcurrent event, however, it's what you have to do after such an event that makes fuses no longer an acceptable solution for data center application. Once a fuse has blown, it must be replaced. This replacement is dangerous for the user. As a result, the entire rack PDU must be powered down. Often this type of replacement must be performed by a licensed electrician. And the electrician must have the correct fuses on hand to replace the blown fuse. Considering that if a current overload condition occurs, you'll probably have fuses in all of the PDUs that are connected to that power source blow, you may be looking at an expensive and time consuming repair job. As long as those blown fuses are still installed in PDUs, all of the servers are running on a single backup power source. If a fault occurs to that power source, all of the attached severs will be lost. While fuses are inexpensive, recovering from a blown fuse can be devastating in terms of cost and downtime in the data center environment. Circuit Breakers and PDUs: A Match Made In Heaven The good news is that there is a much better solution: circuit breakers. Circuit breakers detect overcurrent (or fault) situations and break the electrical circuit providing power. How they do this and how well they do it is what distinguishes one PDU from another. There are several differences between fuses and circuit breakers. One of the most important differences is that unlike a fuse, when a circuit breaker trips because a circuit overload, no replacement is required it can be reset manually or automatically by data center operators. In a circuit breaker, an internal pilot device senses a fault current and trips the device. When a circuit overload condition occurs, contacts in the circuit breaker have to open to interrupt the circuit. To separate the metal contacts mechanically, stored energy within the breaker opens the circuit. Page 3 of 7

5 All PDU Circuit Breakers Are Not Created the Same So circuit breakers are better than fuses in a PDU, end of story, right? Well, no. There are different types of circuit breakers available. Different types of circuit breakers have different costs and physical size. When vendors build rack PDUs, they have to make a cost / benefit decision about which circuit breaker they will use. The type of circuit breaker has a significant impact on how well servers are protected against fault conditions and accidental tripping. Two different types of circuit breakers are often used in PDUs, and although these two different types of circuit breakers perform the same job, they do it differently based on performance characteristics Thermal-based circuit breakers Hydraulic-magnetic (hy-mag) circuit breakers Thermal Fuse Thermal Circuit Breaker Hydraulic Magnetic Circuit Breaker UL248 type UL489 Thermal type UL489 Hydraulic-Magnetic type Branch Rated OCPD Supplemental Protector Figure 1. Typical OCPD found in IT Rack PDUs. Thermal-based circuit breakers are also known as thermal circuit breakers or thermal-magnetic circuit breakers. These thermal-based circuit breakers are made using a bi-metallic strip. This strip has two different pieces of metal welded together; each meant to expand when in contact with the heat of an electric current. With two different kinds of metal, the expansion rate for each is different. The thermal based circuit breaker depends on the heat generated by the passing electrical current; the greater the heat, the more severely the metal strip bends. When there is an electric overload, the heat generated bends the strip enough to pull the circuit level to the off position, protecting the wiring from damage. When a thermal circuit breaker is reset, it is likely to trip again if the overload condition still exists. Page 4 of 7

6 Breaker Derating as % of Nameplate The problem with using a thermal circuit breaker in a PDU is that they are susceptible to unintended tripping when used in high temperature environments. Because the PDU is contained in a rack and is exposed to high temperatures, the PDU's thermal circuit breaker can accidentally trip and lose power. Figure 2 illustrates how the trip setting of a typical thermal circuit breaker changes due to ambient temperature; for many rack PDUs, the trip setting can vary by more than 15%. 130% 120% 110% 100% 90% 80% 70% Ambient Temperature, C Figure 2. Temperature Derating Curve for Typical Thermal Circuit Breaker. Another challenge when using a thermal circuit breaker is the tripping temperature changes by more than 15% over the rack PDU's operating temperature specification! Thermal circuit breaker vendors recommend a PDU load current at 80% or less of the rated current to account for the variation in temperature that the external rack temperature can cause a trip. This restricts total power output per rack, and results in unplanned downtime conditions. An alternative type of circuit breaker used in a PDU is the type used by Enlogic: the hydraulic-magnetic, or hy-mag, circuit breaker. The hy-mag breaker is independent of temperature as a tripping mechanism Hy-mag breaker's tripping mechanism does not use thermal elements (bimetal) and is not affected by ambient rack temperature. Rather, the current load is connected through a solenoid coil. At currents above the circuit breaker rating, the coil generates magnetic flux necessary to force a spring mechanism to open the electrical circuit and prevent current flow through the breaker. What this means is that the tripping current is related to the amount of current only; the hy-mag breaker will keep consistent tripping characteristics across the entire range of data center ambient temperatures. When a circuit breaker trips, the data center operator wants the affected servers powered up as quickly as possible. This is where another big difference between fuses and circuit breakers, and further between thermal and hydraulic-magnetic circuit breakers becomes clear. When a circuit breaker trips or fuse blows, users should explore why the event occurred and take action to prevent reoccurrence. However, in the case of a fuse: the entire rack PDU must be powered down and specially trained personnel must replace the blown fuse. In the case of the resettable circuit breaker: the circuit failure is often isolated to just one portion of the rack PDU output. Users can independently eliminate the fault condition and reset the circuit breaker Page 5 of 7

7 without powering down the remainder of the rack PDU. Further, circuit breakers don t require specially trained personnel to reset the device after failure. When a hydraulic-magnetic circuit breaker is tripped, it provides the ability to be instantaneously reset once you've corrected the cause of the circuit failure. This is not the case with thermal circuit breakers. With thermal circuit breakers, once the bimetal has been bent due to the tripping event, it does not immediately return to its original position. Instead, this circuit breaker remains tripped until the bimetal is cooled. The time from the circuit breaker tripping until it can be reset is called the reset time. Reset time can vary between 3 20 minutes depending on the breaker. Conclusion Currently, there are PDUs in data centers without any fuses or circuit breakers. These PDUs leave servers exposed and vulnerable to the fluctuating power system. With ever-growing rack power density, and increasingly variable power loads in the IT rack, the need for comprehensive circuit overload and fault protection is increasingly important. The two types of protection available include fuses and circuit breakers. Fuses are an older technology that requires problematic downtime and specialized effort to replace. Circuit breakers come in two primary types: thermal and hydraulic-magnetic circuit breakers. Thermal circuit breakers are susceptible to the ambient heat generated in a rack environment and are subject to accidental tripping. Hydraulic-magnetic circuit breakers, such as those used in Enlogic PDUs, provide comprehensive protection against circuit overload and isolation of faults. Unlike the thermal breakers, Enlogic s hydraulic-magnetic breakers are unaffected by heat and can be reset immediately after being tripped. Page 6 of 7

8 About the Authors Dr. Jim Anderson - Dr. Anderson is the Director of Product Line Management for Enlogic. He is responsible for ensuring that Enlogic creates the correct products with the correct features. Dr. Anderson received his Ph.D. in Computer Science from Florida Atlantic University and his MBA from the University of Texas at Dallas. He has worked as a Director of Product Line Management for a number of very large firms including Verizon, Alcatel, Siemens, and Boeing as well as working at a number of successful startups. Dr. Anderson is also a senior member of the Institute for Electrical and Electronics Engineers (IEEE) and holds a senior management position in the U.S. branch of the organization. Mike Jansma Mike Jansma is Co-founder and Chief Marketing Officer of Enlogic. As a technology innovation leader, Mike has an exceptional history of listening to customers, digging into complex problems, and providing innovative solutions for real customer pains in energy technology and power distribution fields. To date, Mike holds 7 technology patents with additional technology patents pending. Prior to founding Enlogic, Mike received his Bachelors and Masters degrees in Mechanical Engineering from The Illinois Institute of Technology and an MBA from Washington University in St. Louis. Mike began his career in Energy Technology at the US Department of Energy s Argonne National Laboratory. Later he held technical and marketing leadership positions at ABB and Schneider Electric/APC. About Enlogic Enlogic is a new company that makes an entire family of world class 4 th generation PDUs. Founded by the engineers who created the first three generations of PDUs that are on the market, Enlogic's PDUs are revolutionary in how reliable they are and how easy they are to both install and support. Enlogic's PDUs are amazingly slender in order to not interfere with any hot-swappable server components, they use hydraulic-magnetic circuit breakers and are temperature rated up to degree C in order to ensure proper operation in today's increasingly hot data center environment, and they feature Enlogic's novel field replaceable network control card that permits equipment changes without having to power down connected servers. You won't find these features in anyone else's PDUs, including any APC PDU. To find out more about what are arguably the world's best 4 th generation PDUs, contact your local Enlogic sales representative in order to request more information or even better, to set up an appointment to see what makes Enlogic's PDUs the right choice for your data center. Page 7 of 7