Fortune Oregon Data Center Increases Reliability with a High Resistance Grounding System

Transcription

1 Fortune Oregon Data Center Increases Reliability with a High Resistance Grounding System Cory David Smith, Project Manager, ECOM Engineering Inc., and David Lawrence Smith, Principal, ECOM Engineering Inc. Abstract A comparison of a High Resistance Grounding System and a Solidly Grounding system in a Data Center application. This document provides the pros and cons of using a high resistance ground system in a data center. It also looks at the common elements of a Data Center s power distribution system and explains the different setup between a High Resistance Grounded system and a Solidly Grounded System. I. NOMENCLATURE HRG: High Resistance Ground, PDU: Power Distribution Unit, UPS: Uninterruptible Power Supply, NRK: Neutral Reference Kit, Transient Voltage Surge Suppressor (TVSS)/Surge Protection Device (SPD), IT: Information Technology, NEC: National Electric Code T II. INTRODUCTION HIS document looks at the advantages and disadvantages of using a High Resistance Grounding system in a data center. The data center is an 12 Megawatt facility in Oregon built and operated by Fortune Data Centers. In this document a solidly grounded system and a high resistance grounding system fault current will be compared for the same system, the verified phase to ground fault for the high resistance ground will be measured and the charging current of the system will be verified for the grounding system. With the mitigation of damage during a single phase to ground fault and continuity of service during a single phase to ground fault, high resistance grounding is an ideal application for protecting critical loads in a data center. III. HIGH RESISTANCE GROUNDING IN A DATA CENTER The use of High Resistance Grounding or HRG at Fortune Oregon Data Center for critical loads allows for a more robust distribution system. The distribution system becomes more robust with the use of HRG by being able to continue service without interruption through a single phase to ground fault, location of the ground fault and then isolation and clearing of Fortune Oregon Data Center. Dave Smith, ECOM Engineering, Owners Representative, Jesse Smith, Nova Partners, Construction Manager, Travis Schumacher, DPR/FORTIS Mission Critical, MEP Coordinator. Fortune Oregon Data Center Engineer of Record Steve Emert PE, Rosedin Electric. Eaton PowerWare, Rosedin Electric. the ground fault with minimal damage to equipment and no interruption of critical power distributions systems. A HRG system has several advantages to implementation in a data center. The application of HRG eliminates the need for complex and expensive zone interlock ground fault breaker schemes. With the HRG system having a low phase to ground fault current, a restrained transient overvoltage and ability to continuously serve the critical load during a phase to ground fault condition, makes the HRG system an attractive solution to the data center community. The HRG system insures high availability and concurrent maintainability as demanded by IT managers served by data centers. By using a HRG system single phase to ground fault available energy is greatly reduced. The concern for groundfault protection is based on four factors: 1. The majority of electric faults involve ground. Even faults that are initially phase to phase spread quickly to any adjacent metallic housing, conduit, or tray that provides a return path to the system grounding point. Ungrounded systems are also subject to ground faults and require careful attention to ground detection and fault protection. 2. The ground-fault protective sensitivity can be relatively independent of continuous-load current values and therefore, have lower pickup settings than phase protective devices. 3. Because ground-fault currents are not transferred through system power transformers that are connected delta-wye or delta-delta, the ground-fault protection for each system voltage level is independent of the protection at other voltage levels. This configuration permits much faster relaying than can be afforded by phase-protective devices that require coordination using pickup values and time delays that extend from the load to the source generators and often result in considerable time delay at some points in the system. 4. Arcing ground faults that are not promptly detected and cleared can be destructive. [1] A. Types of Grounding in a Data Center Per the NEC only two types of grounding can be used in a low voltage system in the United States. A solidly grounded system and for three phase loads High Resistance Grounding can be implemented [2]. Both systems offer advantages and disadvantages based on application.

2 B. Advantages and Disadvantages of Solidly Grounded Systems in Data Centers A solidly grounded system offers several advantages: 1. During a phase to ground fault the system does not suffer from overvoltages. 2. The solidly grounded system has low step potentials between ground and the electrical system in normal operation. 3. Single phase to ground loads can be applied to the electrical system without the need for isolation transformers between the critical equipment and single phase loads for lighting and equipment. 4. Paralleling UPS units in a Solidly Grounded system is not as complicated as in a HRG system. Disadvantages of a solidly grounded system in a data center include: 1. High ground fault current and a voltage dip on the phase that has the ground fault. 2. Interruption of service during a phase to ground fault. 3. High ground fault current is available during arcing faults. 4. Phase to ground faults cause mechanical stresses in circuits and equipment caring the fault current. 5. Damage to electrical equipment caused by high phase to ground fault currents. 6. Complicated protection schemes to provide ground fault protection. A lack of selectivity is inherent in ground fault protection schemes. Per NEC, Section (A) the maximum ground fault sensor trip setting is restricted to 1200 amps. a. Thus data center critical power distribution systems larger than 1200 amps must rely on zone interlock schemes to coordinate ground fault protection. These schemes rely on breaker to breaker communication which is inherently a single point of failure for the critical power distribution system during a single phase to ground fault. b. Ground fault selectivity can also be achieved through a limited amount of time delay between breakers however, this allows a larger magnitude of fault current during a single phase to ground fault. c. The ground fault time delay available for coordination limits the number of breakers that can realistically be coordinated in series. C. Advantages and Disadvantages of High Resistance Grounding in Data Centers Advantages to high resistance grounding in a data center: 1. Reduces burning and melting effects in faulted electric equipment, such as switchgear, transformers, cables and rotating machines. 2. Reduces mechanical stresses in circuits and apparatus carrying fault currents. 3. Reduces electric-shock hazards to personnel caused by stray ground-fault currents in the ground return path. 4. Reduces the arc blast or flash hazard to personnel who may have accidentally caused or who happen to be in close proximity to the ground fault. 5. Reduces the momentary line-voltage dip occasioned by the occurrence and clearing of a ground fault. 6. To secure control of transient overvoltages while at the same time avoiding the shutdown of a faulty circuit on the occurrence of the first ground fault (high-resistance grounding).[1] 7. The load is maintained during a single phase to ground fault. 8. Allows time to respond, isolate and repair the single phase to ground fault event while maintaining the critical load. 9. HRG s reduces the available phase to ground arcing fault because the phase to ground fault current is limited to a maximum of 10 Amps by the HRG resistor. However there is still the danger of a three phase arcing fault which must be used to calculate the protection required to protect the technician working on the equipment. The limited phase to ground available current reduces the chance of a phase to ground arcing fault creating a three phase arcing fault. 10. During a Ground Fault Condition the ground fault can be left until an ideal time to clear the fault as long as the insulation of the cable is rated at 173% [3]. For a 480V system the conductors insulation is rated at 600V thus a 173% voltage of the phase to ground voltage brings the two un-faulted phase voltages in respect to ground to 480V. This does not exceed the insulation of the conductors. See Figures 1.A and 1.B for the normal operating characteristics and fault operating characteristics of the voltage with a HRG. Disadvantages of using a HRG system must be considered before implementing the system in the design of any building but especially a non-industrial application. Some of the following disadvantages must be addressed: 1. Transient overvoltages up to 250% of nominal system voltage can occur in the system. 2. Single phase loads cannot be connected to the HRG system. Life Safety loads must be separated by a transformer with a solidly grounded neutral. 3. Power Electronic equipment such as the UPS and TVSS must be selected so that they are able to operate with a HRG typically connected in a Delta configuration. 4. Switchgear control transformers cannot connected be phase to neutral and must be connected phase to phase. 5. Paralleling UPS modules with an HRG increases the complexity of the system. 6. During a ground fault event qualified personnel must go into energized equipment to meter feeders to find the location of the ground fault. With the Personal Protection Equipment (PPE) for Arc Flash based on the three phase arc flash energy. This condition can be mitigated by the installation of permanent amp meters on feeder circuits.

3 D. Data on the HRG System at Fortune Data Center Table 1: Unit Substation USG-1-4 Ground Fault Data Electrical Ground Fault Element Solid Name Ground Unit Substation USG , Amps UPS , Amps UPS-DP , Amps Input of 37, PDU Amps Ground Fault Resistance Ground Maximum Allowed Current Ground Fault Resistance Ground Measured Fig. 1.A: Voltage Characteristics of an HRG System Under Normal Operating Conditions. By utilizing a HRG system the phase to ground fault current of the system decreases by a factor of ten thousand or four orders of magnitude. The dramatic reduction of phase to ground fault current mitigates damage to the system during a phase to ground fault event. Based on IEEE Research Bus Duct ground faults are 2.3 times more likely to occur compared to phase to phase faults. Cable ground faults are 73 times more likely to occur compared to phase to phase faults and Cable Joints ground faults are 7.8 times more likely to occur compared to phase to phase faults [5]. Using this research as a guide the ability the HRG system has the ability to limit the amount of energy delivered by the most common type of fault. Fig. 1.B: Voltage Characteristics of an HRG System Under A Single Phase to Ground Operating Conditions. E. Challenges for the HRG System Charging current of the system must be less than the total ground fault current. For most 480V HRG systems the charging current is under 2A which can be measured or calculated [3]. While the overvoltage can be a problem to the insulation of the cable in a medium voltage system, in a low voltage 480V system where the phase to neutral voltage is 277V a 173% overvoltage brings the two un-faulted phases to a voltage of 480 Volts. The cable insulation is rated at 600 Volts which mitigates the danger of insulation break down avoiding a double phase to ground fault. Ground Faults should still be cleared as soon as possible as a second fault will create a phase to phase condition tripping upstream breakers. Intermittent ground faults may also cause transient overvoltages of 250% that damage the insulation of the cables and can overtime lead to a double phase to ground fault condition. PPE is based on the three phase arc flash condition in electrical panels. In an HRG system if any panel has been found in the Arc Flash study to be a non approach panel while energized. Design coordination measures such as permanently installed amp meters on feeders and or breaker trip units with a maintenance setting that reduce arc flash energy should be installed so the location of the fault can be found while keeping the operators safe.

4 F. UPS Grounding There are three choices for UPS grounding: four-wire source and four-wire load solidly grounded, three-wire source and three wire load solidly grounded and three wire source and three wire load high resistance grounded. See Figures 1, 2 and 3. Fig. 1: Four Wire Input/ Four Wire Output Solid Ground of a UPS [6]. In this configuration during a phase to ground fault the UPS will go to static bypass during the fault to be able to deliver enough current to clear the fault. Fig. 2: Three Wire Input/Three Wire Output Solid Ground of a UPS [6] In this configuration during a phase to ground fault the UPS will go to static bypass during the fault to be able to deliver enough current to clear the fault. Using a high resistance ground for the UPS that limits the phase to ground current to a maximum of 10 amps allows the UPS manufacture to disable the function on the UPS that when the UPS detects a ground fault to go to static bypass and instead have an alarm only. This is allowable because the UPS is able to supply the current to the system in the phase to ground fault condition. Thus the UPS stays online protecting the critical load. G. HRG System and Equations Type of High Resistance Grounding, there are two types of high resistance grounding. The first type of high resistance grounding is used with a relay or ground fault breaker that allows a certain amount of current through the resistor for up to 10 seconds and then trips the protection and continuity of service is broken. 1. The advantage to this is that the system is cheaper to install and gives the fault time to clear itself. This allows for more coordination with the ground fault protective scheme and makes coordinating the system easier. 2. The disadvantage of this system is that the resistor is only rated for a limited duration and the expensive breakers and relaying is still required and your system still loses continuity of service. Resulting in loss of critical load. The other type of High Resistance Grounding has a continuous duty rated resistor and can have a ground fault on the system for any amount of time. 1. With this configuration all advantages of High Resistance Grounding apply. For the HRG system to have stable operation some of the following conditions must be met. The ground fault current must be at least three times the charging current of the capacitance of the electrical system. Because of system reliability and the capacity to continue serving critical load a fully rated continuous duty resistor is the logical choice for a Data Center application. R NGR V LL Ohms (1) I G N X C R GR Ohms I G 3I CO Amperes (3) (2) P NGR I G R NRG Watts (4) Total Ground Current I G I R I C (5) Fig. 3: Three Wire Input/Three Wire Output High Resistance Ground of a UPS [6]. Capacitive Reactance X C ohms/phase (6) C Zero Sequence Capacitance C micro Farads ux /phase (7) Charging Current

5 3I C C E Amperes (8) Charging Current Where: f = Frequency in hertz 3I C. LE Amperes C 0 = Capacitance to Ground in µf Where: (11) E = Line to Line System Voltage E = Line to Line System Voltage Cable Capacitance Three-conductor Cable L = Line length in ft./1000 C. µf (9) Motors LOG D F 3I C 0.05 HP Amperes (12) RPM Measuri ng System Charging Current Where: C 0 = Capacitance to Ground in mf per 1000 feet. R MAX V LL = specific inductance of insulation Ohms (13) I C D 1 = d + 3c +b for three-conductor cable. d = diameter of conductor Where: c = thickness of insulation of conductor. 3I C0 = the estimated charging current b = thickness of belt insulation. V LL = the system line to line voltage [7] Charging Current Estimates Transformers C µF (10) Fig. 4: One Line Diagram for location of Phase to Ground Fault Measurement and Test at Fortune Oregon Data Center during commissioning

6 Fig. 5: Sample One-line Diagram for location of Ground Fault Location Circuitry Figure 4 shows the location for where the phase to ground faults where tested on site at Fortune Oregon Data Center during commissioning. In Figure 5 the location of phase to ground fault location circuitry is shown. In Figures 6 and 7 the circuit for measuring the fault current is shown. Fig. 7: UPS Output Switchboard Ground Fault Sensing Circuit. Fig. 6: Unit Substation Switchboard Ground Fault Sensing Circuit. IV. EQUIPMENT COORDINATION FOR SUCCESSFUL HIGH RESISTANCE GROUND OPERATION To insure a successful commissioning and operation of an HRG system on a highly reliable UPS critical power distribution system. The following items must be coordinated through the design, specification, submittal review and equipment start up. Starting with Electrical Switchboards components of the switchboards must be configured in to work with a HRG.

7 7 TVSS/SPD or Surge Protective Device must be a delta configuration. If a 4 wire solidly grounded SPD was installed the neutral wire would defeat the HRG of the system and provide the path of least resistance for the phase to ground fault. The surge arrester ratings must be specified so that neither the maximum continuous operating voltage nor the one-second temporary overvoltage capability is exceeded under system ground-fault conditions [1]. Control transformers in the electrical equipment must be phase to phase not phase to neutral as this will also defeat the HRG system. All control transformers and SPD s should be rated for line to line voltage, not line to ground. During testing the measured fault current of the system was 3.5 amps. Using the equation I G 3I CO Amperes (3) to find the charging current of the grounding system to be less than or equal to amps. See Figure 4, 5, 6 and 7 for Measurement of Charging Current diagram. The HRG itself must be configured for use in the system so there are no nuisance s alarms. In this project a PulserPlus Pro-Low Voltage High Resistance Grounding by Post Glover was used. Variables that needed adjustment for proper function included: adjustable pulse rate, adjustable time delays for fundamental and third harmonic settings to prevent nuisance s alarms. System monitoring is achieved through communications via RS232, Modbus or Ethernet protocols, this provides 7x24 monitoring via the BMS system. The HRG reduces the system ground fault current to a maximum of 10 Amps and for the installed system a measured 3.5 Amps during commissioning. Note the HRG system should be a continuous duty rated resistor. This is desired so that during a ground fault there is no interruption in service while the ground fault location is found and repaired. The Eaton 9395 UPS software must be enabled to allow the UPS to operate during a Ground Fault Condition without transferring to bypasss. On the Eaton 9395 a Neutral Reference Kit (NRK) must be installed. Battery monitoring systems should not use ground as a floating reference; they need an isolated ground reference for control circuits. For the Power Distribution Unit four items must be addressed. First the ground fault function on the primary side of the PDU must be turned off and have Annunciation only. Second Phase loss/ rotation shutdown must be turned off. The Over/Under voltage shutdown must be turned off. These three functions can be turned off during setup by the manufacture. Lastly on PDU the Auto-restart function must be enabled or the 480V main breaker will shunt trip on Undervoltage /Overvoltage phase loss/rotation and ground fault while used in conjuncture with a HRG during a single phase to ground fault condition. ground fault on the electrical system is reduced so that a single phase to ground fault is no longer a catastrophic failure of the electrical system requiring an immediate outage of service and potentially extensive repair. Finally the downstream customer does not notice the single phase to ground fault. The critical load is continuously fed thru the UPS during a single phase to ground fault. Allowing maintenance personal time to find, isolate and repair the fault while maintaining critical load. Using an HRG system in Fortune Oregon Data Center, increase the electrical system reliability for Fortune s customers through the implementation of a HRG system. VI. ACKNOWLEDGMENT The authors gratefully acknowledge the contributions of Brian Kane PE, Ed Spears and Steve Emert PE for their information and review of the paper. VII. REFERENCES [1] IEEE Recommended Practice for Grounding of Industrial and Commercial Power Systems, IEEE Standard , December [2] NFPA 70, National Electric Code [3] IEEE Recommended Practice for Protection and Coordination of Industrial and Commercial Power Systems, IEEE Standard , June [4] IEEE Recommended Practice for Electrical Power Distribution for Industrial Plants, IEEE Standard , December [5] IEEE Recommended Practice for Design of Reliable Industrial and Commercial Power Systems, IEEE Standard , February [6] Eaton Technical Brief, UPS Grounding: An Objective Overview, September 7, [7] Post Glover Application Guide, Ground Fault Protection on Ungrounded and High Resistance Grounded Systems, Copyright 2007 VIII. BIOGRAPHY Cory David Smith is a Project Manager at ECOM Engineering and is based in Sacramento, California. Cory has spent 8 years in various positions at ECOM Engineering specializing in design of Data Centers and Hospitals. Cory is a graduate of California State University Sacramento with a BSEE and a member of the IEEE. David Lawrence Smith is a Principal at ECOM Engineering and is based in Sacramento, California. Dave has 30 years of electrical design, project management, construction, and commissioning experience specializing in Data Centers, Hospitals and Critical Facilities. Dave is a member of the IEEE. V. CONCLUSION Using an HRG in a critical facility such as a data center makes sense for a number of reasons. At Fortune Oregon Data Center the single phase to ground fault current is limited to 3.5 amps at the Main Switchboard compared to the 44,500 amps of the same system with a solid ground. The reduction of the fault current by 4 orders of magnitude reduces the single phase to ground arc flash potential energy. The mechanical stress of a single phase to