Power Solutions for Data Centers Metering and protection architecture guide

Transcription

1 Power Solutions for Data Centers Metering and protection architecture guide The purpose of this guide is to provide technical guidelines to designers and field engineers who specify metering and protection architecture in order to maximize the performance of the Power Solutions for Data Centers offer. This guide also provides architecture recommendations and the reasoning associated with selecting specific devices, and how to calculate Power usage effectiveness (see Appendix A: How to calculate PUE on page 43). In this document Power Solutions for Data Centers Metering and protection architecture guide... 1 Hazard Categories and Special Symbols... 4 Safety precautions... 5 Introduction Schneider Electric. All rights reserved. StruxureWare, StruxureWare Power Monitoring, ION, ION Enterprise, MGE Galaxy 5000, Modbus, Modicon, BCPM, ConneXium, Compact, Masterpact, Powerpact, Power Measurement, PowerLogic, SEPAM, Micrologic, Schneider Electric, Symmetra MW, and System Manager, are either trademarks or registered trademarks of Schneider Electric in France, the USA and other countries. All other trademarks are property of their respective owners. Understanding customer value... 7 Real-time monitoring... 7 Power system monitoring... 8 Crisis management... 8 Power system alarms and events... 8 Power reliability... 8 Utility power quality... 8 Generator system testing... 9 Capacity management... 9 Generator system capacity management... 9 UPS system capacity management... 9 Power efficiency... 9 Power usage effectiveness (PUE) Power system losses Solution module metering requirements Metering and Protection Architecture Overview Architecture recommendations and reasoning Utility switchgear and MV transformer Recommendation summary Schneider Electric. All rights reserved. Page 1

2 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Key considerations Solution module considerations Generator and generator switchgear Recommendation summary Key considerations Solution module considerations Main switchboards (LV) Recommendation summary Key considerations Solution module considerations IT UPS input switchboards Recommendation summary Key considerations Solution module considerations IT UPS modules Recommendation summary Key considerations Solution module considerations IT UPS output switchboards Recommendation summary Key considerations Solution module considerations Power Distribution Units (PDUs) Recommendation summary Key considerations Solution module considerations Mechanical UPS switchboard (input and output) Recommendation summary Key considerations Solution module considerations Mechanical UPS modules Recommendation summary Key considerations Solution module considerations Mechanical switchboard Recommendation summary Key considerations Solution module considerations Page Schneider Electric. All rights reserved.

3 Auxiliary switchboard Recommendation summary Key considerations Solution module considerations Conclusion Appendix A: How to calculate PUE Power usage effectiveness calculations Total data center load IT load Factored PUE calculations Appendix B: Single line diagram of reference architecture References Terms and Definitions Additional information IT infrastructure architecture guide Communications architecture guide 2012 Schneider Electric. All rights reserved. Page 3

4 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Hazard Categories and Special Symbols Read these instructions carefully before trying to install, configure, or operate this software. The following special messages may appear throughout this bulletin or on the equipment to warn of potential hazards or to call attention to information that clarifies or simplifies a procedure. The addition of either symbol to a Danger or Warning safety label indicates that an electrical hazard exists which will result in personal injury if the instructions are not followed. This is the safety alert symbol. It is used to alert you to potential personal injury hazards. Obey all safety messages that follow this symbol to avoid possible injury or death. DANGER DANGER indicates an imminently hazardous situation which, if not avoided, will result in death or serious injury. WARNING WARNING indicates a potentially hazardous situation which, if not avoided, can result in death or serious injury. CAUTION CAUTION indicates a potentially hazardous situation which, if not avoided, can result in minor or moderate injury. CAUTION CAUTION used without the safety alert symbol indicates a potentially hazardous situation which, if not avoided, can result in property damage. PLEASE NOTE Electrical equipment should be installed, operated, serviced, and maintained only by qualified personnel. No responsibility is assumed by Schneider Electric for any consequences arising out of the use of this material. Do not base your maintenance or service actions solely on messages and information displayed by the software. Page Schneider Electric. All rights reserved.

5 Safety precautions Installation, wiring, testing and service must be performed in accordance with all local and national electrical codes. UNINTENDED EQUIPMENT OPERATION WARNING Do not use Power Solutions for Data Centers software and associated devices for critical control or protection applications where human or equipment safety relies on the operation of the control circuit. Failure to follow these instructions can result in death, serious injury, or equipment damage. INACCURATE DATA RESULTS WARNING Do not incorrectly configure Power Solutions for Data Centers software and its associated devices; this can lead to incorrect reports and/or data results. Do not rely solely on reports or data results to determine if Power Solutions for Data Centers and its associated devices are functioning correctly or meeting all applicable standards and compliances. Do not use reports or data results as substitutes for proper workplace practices or equipment maintenance; they are supplemental only. Failure to follow these instructions can result in death, serious injury, or equipment damage Schneider Electric. All rights reserved. Page 5

6 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Introduction This document is designed to facilitate specifying the metering and protection architecture that must be established to unlock the value in the Power Solutions for Data Center offer. The guide presents the reasoning behind the choices for each metering and protection recommendation and how they are related to each solution module in the Power Solutions for Data Centers framework. As part of the data centers reference tools, this guide explains why the metering and protection scheme was chosen in the reference architectures. By showing why meters and protection devices were placed in the different locations in the electrical distribution system, designers and field engineers will be able to adjust the design when different architectures are required. A reference architecture will be used in order to provide an example for how to determine the proper metering and protection scheme. By using the reference architecture example, the exact requirements for any specific application can be derived. Reference Architecture 34 is a data center design that was created as part of a set of architectures by the IT business unit in association with the Power business unit. Their goal was to create a tool that would help, when selecting from a number of data center architectures, to specify the requirements for new Data Center construction and to facilitate the project work required to build it. Note: See Figure 26, in Appendix B: Single line diagram of reference architecture 34, on page 45 for an image of the electrical distribution system. This guide focuses on applying best practices when choosing how to design a metering and protection architecture for a data center to meet specific customer requirements as defined by the following PowerStruxure for Data Centers Power Management modules: Real-Time Monitoring Crisis Management Power Reliability Capacity Management Power Efficiency Each solution module is focused on solving specific requirements and is used in association with the StruxureWare Power Monitoring 7.0 platform. For example, the Capacity Management module provides reports for managing the loading of both the backup generator system and the UPS system. These reports help the data center facility operations team to maximize the loading of these systems while ensuring that the intended redundancy design (For example, 2(N) is preserved.) Note: Enabling the value of each module requires very specific data from the metering and protection sub system. Therefore, before choosing which metering and protection devices to specify, designers and engineers must note the value that the end user is expecting. Specifying an inappropriate device could mean permanently preventing the end user from realizing the full value of these modules. Page Schneider Electric. All rights reserved.

7 Understanding customer value The Power Management solution is targeted at the facility operations team located at dedicated data center facilities. This management system solution is designed to help the operations team run the data center effectively by providing the following high level value propositions: Improve Reliability Improve Asset Performance Improve Financial Performance Each value proposition is addressed via the solution modules listed below: Figure 1: Power Solutions for Data Centers Solution Modules Specific applications must be enabled within each module to solve specific customer challenges. This requires the power management software to have access to specific data from downstream metering and protection devices. Real-time monitoring 2012 Schneider Electric. All rights reserved. Page 7

8 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Power system monitoring The facility operations team must be able to, at a glance, understand the state (For example, key data, status, and alarm points) of the power distribution system. This is of particular importance during a power system event, when conducting equipment maintenance, during commissioning activities, or during power system testing. An Uptime Institute analysis found that 73% of data center downtime events are caused by some form of human error. Therefore, it is critical that technicians have access to real-time system data before and after planned maintenance to reduce error and avoid a potential downtime event. Crisis management Power system alarms and events A power system outage, on average, costs a data center $505,000 per incident and the longer the outage the higher the cost. Therefore, monitoring key alarm parameters helps avoid unnecessary downtime (For example, % loading for power circuits) and quickly alerts the facility operations team to a power system event (For example, PDU A1 main breaker trip), thus facilitating a timely response. If a power system event does occur, it is critical that the appropriate people in the facility operations team know about the event immediately so that the process of recovering from the event can start right away (For example, minimizing the MTTR). This emergency process involves understanding what happened, what equipment was affected, what customers were affected, and what emergency maintenance must take place. Power reliability Utility power quality In some regions, the electrical utility is legally bound to adhere to a Power Quality (PQ) standard such as EN50160 or IEC In these areas the utility has an obligation to deliver power within defined parameters and any deviation gives the data center operator cause to question the quality of the power being delivered. In some cases, power quality events, such as repeated voltage sags, Page Schneider Electric. All rights reserved.

9 can trigger an unnecessary transfer to the backup power generation system, which is expensive, stressful, and risky. Monitoring and alarming PQ conditions on the facility can help increase the system availability and reliability, which in turn can save money. Generator system testing The only way to ensure that the backup power system operates as intended is to regularly test the full sequence of events associated with engaging the backup power system. Automatically documenting and tracking key operational data during these tests ensures that problems can be identified and corrected before they lead to a backup power system stoppage during a utility outage. Because it may not be possible to test each generator at full load, the generator system testing will predict that generators will provide backup power capability based on accurate interpretation of the measured parameters. Capacity management Generator system capacity management During a utility outage it is critical that the backup power system can handle the full data center load or at the very least, a subset of that load for mission critical activities such as servers and cooling. In addition to this basic requirement, it is also important that the redundancy design of the backup power system is maintained. This ensures that the stoppage of a generator(s) will not cause an overload and a data center shutdown. UPS system capacity management UPS systems are expensive, so maximizing the loading of the UPS systems (For example, sweating the assets ) helps minimize capital expenditures. However, this must take place while maintaining the redundancy design; otherwise any savings with capital expenditures will be at the expense of power system reliability. Power efficiency 2012 Schneider Electric. All rights reserved. Page 9

10 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Power usage effectiveness (PUE) Power usage effectiveness (PUE) is a key performance indicator (KPI) that is accepted throughout the industry. PUE defines the efficiency of power delivery to the critical IT load in an IT facility. For example, a PUE of 1.75 means that for every 1 kw of IT load power, 1.75 kw is required from the utility. Minimizing PUE and ensuring that it does not increase over time has a direct impact on the electric utility bill of the data center. For example, in a data center with a 1MW IT load and an electrical energy cost of 0.10 per kwh, going from a PUE of 1.75 to 1.60 equates to over 131,000 in annual savings (assuming 24 hour operation at 1MW IT load). Power system losses Inefficient power delivery in a data center can be attributed to three primary sources: 1) transformer losses, 2) UPS conversion losses, and 3) harmonics losses. Collectively, these losses equal ~ 10% of the electrical power usage in a typical data center. For example, in a data center with a 1MW IT load, an electrical energy cost of 0.10 per kwh, and a PUE of 1.75, 10% losses equates to over 150,000 in annual costs (assuming 24 hour operation at 1MW IT load). Solution module metering requirements This document outlines all of the details behind the metering and protection architecture. However, for simplicity, the required metering points per solution module have been summarized in Table 1. The X represents a required point in the electrical distribution system where some form of metering is required in order for the solution module to work. The details behind which meter or protection unit to use are summarized in Table 2. Page Schneider Electric. All rights reserved.

11 Table 1: Solution Module Metering Point Requirements Solution Module Metering Point Power Usage Effectiveness (PUE) Power System Losses Generator System Test Generator System Capacity Management UPS System Capacity Management Utility Interconnect Utility MV Switchgear x x x Generator Generator LV Switchboard Main LV Switchboard x x UPS UPS Output LV Switchboard x (Category 1) x x x x X Power Distribution Unit (PDU) IT Panelboard IT Busway x (Category 2) Mechanical LV Switchboard 2012 Schneider Electric. All rights reserved. Page 11

12 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Metering and Protection Architecture Overview The single line diagram for Reference Architecture 34 and a description of each section is shown in Figure 2. The various components of the electrical distribution architecture and each component or switchboard has its own monitoring and protection requirements. They are summarized in Table 2. The remainder of this document will outline the decision making process behind the metering recommendations for each section of the electrical distribution system. Figure 2: Reference Architecture 34 Single Line Diagram Page Schneider Electric. All rights reserved.

13 Table 2: Metering and Protection Devices for Reference Architecture 34 Power Equipment Device Type Device Models Location in system Utility Interconnect Digital Power Quality Meter Schneider Electric ION7650 Utility billing point Utility MV Switchgear Digital Power Quality Meter Digital Protective Relay Schneider Electric ION7650 Schneider Electric SEPAM S42/T42 MV Mains MV Feeders Generator Generator LV Switchboard Digital Power Quality Meter Digital Protective Relay Schneider Electric ION7650 Schneider Electric SEPAM G82/T42 MV Mains MV Mains and Feeders Digital Trip Unit (Power Quality) Schneider Electric Micrologic 6.0 H LV Mains/Transformer Main LV Switchboard Or Digital Power Quality Meter Or PM850 LV Mains/Transformer Digital Trip Unit (Current) Schneider Electric Micrologic 6.0 A UPS Digital Trip Unit (Power Quality) Schneider Electric Micrologic 6.0 H LV Mains UPS Output LV Switchboard Digital Trip Unit (Current) Or Digital Power Quality Meter Digital Trip Unit (Current) Schneider Electric Micrologic 6.3 A / 6.2 A Or PM820 Schneider Electric Micrologic 6.3 A / 6.2 A LV Feeders LV Mains LV Feeders Power Distribution Unit (PDU) Digital Power Meter Digital Branch Circuit Meter Schneider Electric PM5350 Schneider Electric BCPM LV Main LV Branch Circuits IT Panelboard Digital Power Meter Digital Branch Circuit Meter Schneider Electric PM5350 Schneider Electric BCPM LV Main LV Branch Circuits IT Busway Digital Power Meter Schneider Electric PM5350 Busway Feeder Plug-in Units Digital Trip Unit (Power/Energy) Schneider Electric Micrologic 6.0 P LV Mains Mechanical LV Switchboard Digital Trip Unit (Motor Protection) Digital Trip Unit (Motor Protection) Schneider Electric Micrologic 6.3 E-M Schneider Electric Micrologic 6.2 E-M Schneider Electric PM9C LV Feeders LV Feeders LV Feeders Digital Power Meter 2012 Schneider Electric. All rights reserved. Page 13

14 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Architecture recommendations and reasoning This section explains component by component the reasoning and recommendations for each part of the electrical distribution system. Additionally, where applicable, references to the solution modules that are impacted by the metering choice are highlighted. Utility switchgear and MV transformer The Utility MV Switchgear (A and B sides) is the top-level power distribution component for this reference design. MV Switchgear lineup has two incomers. This is to accommodate feeds from multiple Utilities. Although this is a possible scenario, it is not common to all data centers worldwide. Regardless of the feed topology, the primary purpose of this set of switchgear is to provide power to the MV Transformer. The MV Transformer converts the 20kV primary voltage to 410V/240V, which is the low voltage distribution level for this IEC-based design. The recommended metering and protection points as well as the respective devices are shown in Figure 3. Figure 3: Utility Switchgear and MV Transformer Page Schneider Electric. All rights reserved.

15 Recommendation summary Metering ION7650 with Power Quality framework Communications: TCP Protection SEPAM T42 Communications: TCP or Serial through EGX100 gateway Key considerations Metering Accurately monitoring the facilities energy consumption is required to ensure proper energy charges are being applied by the utility. When a revenue grade meter is combined with high accuracy instrument transformers (as per IEEE C57-13) utility energy accuracy can be verified. This verification ensures the data center facility is not only charged for what they consume. Note: Monitoring Power Quality disturbances according to the IEC and EN50160 standards is extremely important in order to ensure the utility is providing reliable and undistorted power. Additionally, by using an advanced power quality meter the data center facility manager can ensure that the harmonics generated by the facility are within the accepted limits with their utility. The ION7650 provides all the functionality required in order to accomplish these tasks and seamlessly integrates into the StruxureWare Power Monitoring software to provide rich, value-added reporting for the facility managers and operators. A summary of the features that are included with the ION7550/7650 digital power quality meter are shown in Table Schneider Electric. All rights reserved. Page 15

16 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Table 3: ION 7550/7650 Feature Summary Page Schneider Electric. All rights reserved.

17 Protection Critical Power entry points are susceptible to the utility grid voltage fluctuations, so the ability to monitor and control the lines is very important. Firstly a device capable of instantaneous over current and sustained over current (50/51) monitoring and control is required. Additionally, Power Quality events such as Voltage transients, sags and swells can affect the reliability and availability of the facility electrical distribution. The ability to take corrective action based on such events is also a key consideration. Finally, the MV circuit breakers are feeding the MV Transformers, so the protective device must have the applicable logic to deal with transformer faults. The SEPAM T42 has the minimum feature set in order meet these key requirements. If there is differential protection on the transformer, then the SEPAM T87 can be selected, however, this would increase the cost Schneider Electric. All rights reserved. Page 17

18 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Table 4: SEPAM Transformer Protection Summary Other A transformer running at, or below, the rated kva, rated winding temperature, and rated ambient temperature will last longer than a transformer that is not. To avoid the cost of replacing a transformer early or suffering a sudden stoppage, the power loading and temperatures of the transformer should be monitored using the upstream protective relay in the MV switchgear. A temperature sensor can be installed in the transformer and wired to the SEPAM that is upstream from it. Page Schneider Electric. All rights reserved.

19 Solution module considerations Power usage effectiveness (PUE) The utility input is a mandatory metering point for the Power Usage Effectiveness solution module. The calculation cannot be carried out according to the Uptime Institute standards if metering is not in place at this point in the electrical distribution system. For example, Figure 4 shows the simplest metering scheme required in a dedicated data center facility to enable the PUE Solution (outlined in red). Here, only a single meter on the utility feed coming into the building is needed. The utility meter data, combined with the energy data from the UPS units (the meter above it in the architecture), provides all the necessary monitoring points to calculate the PUE and understand the consumption characteristics of the facility. For more information on how to calculate the PUE, refer to Appendix A: How to calculate PUE on page Schneider Electric. All rights reserved. Page 19

20 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Figure 4: Dedicated data center facility single line diagram Page Schneider Electric. All rights reserved.

21 Power system losses There are two reasons metering at the MV level is required for the Power systems losses solution module. First, to calculate the losses due to the MV transformer, the module requires metering points on either side of the component. Placing a meter on the high-voltage side of the transformer meets part of this requirement. Second, to ensure that the harmonic-loss calculation can be carried out, a meter capable of logging the following measurements is required on the utility mains above the MV transformer: Average voltage phase to phase Power Factor Signed Phase A Current THD Phase B Current THD Phase C Current THD Generator and generator switchgear Backup generators are essential to ensure that power is available to the IT loads of a data center. The backup generators must be ready to pick up the load immediately. These generators must be have a reliable monitoring and control system to ensure safe and reliable operation. The control of the generators is managed by the generator controller, and the metering and protection is usually accomplished with an ION7650 and Micrologic trip unit. The recommended metering and protection points and devices are shown in Figure 5. Figure 5: Generator System 2012 Schneider Electric. All rights reserved. Page 21

22 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Recommendation summary Metering ION7650 with Power Quality & Generator System Testing framework Communications: TCP Protection Micrologic 6.0H (for designs without the ION7650) Micrologic 6.0A (for designs with the ION7650) Communications: TCP or Serial through EGX100 gateway Additional Generator Controller (Modbus) Communications: TCP or Serial through EGX100 gateway Key considerations Generators have operational limits, including power rating, power factor and harmonics. Therefore, power monitoring must be in place to identify how the system is operating and to ensure that it continues to operate within its designed limits. One form of monitoring must be in place either on the protection or a power meter. Following are two options: Metering Which meter is installed depends on what generator monitoring features the customer wants to enable. If customers want to enable the Generator System Test, then they should follow the guidelines based on the features they seek as outlined in Table 5. The ION7550 or ION7650 have all features of the Generator System Test, including the battery health monitor and fuel management. Protection The Micrologic 6.0H is a good candidate for the monitoring and control if there is no other power quality metering installed. This device combines the trip unit functionality with a basic power quality meter. However, if a more advanced power quality meter is selected, then a more basic Micrologic trip unit such as the Micrologic 6.0A can be selected. Page Schneider Electric. All rights reserved.

23 Generator controller integration The controller can be a PLC that has been programmed for that specific application. Typically the PLCs have a Modbus map that can be used for communication, so the devices can be integrated into the monitoring system. Generator operational data, such as status points (start, stop, etc), engine data, and electrical data, can be retrieved from the devices and brought into the system. This integration is vital to enabling customers to connect with all of the assets and get extra value out of the Power Solutions for Data Center system. Solution module considerations Generator system test There are various architectures that can be used to enable the Generator System Test solution module. The recommended architecture is to use ION 7550 or ION7650 on the output of each generator as shown in Figure 6. However if a customer has a limited budget, a PM850 or PM870 will work. These two meters, however will not provide some features, such as Battery Health Monitoring and Fuel Monitoring. The architecture in Figure outlines the recommended metering for the Generator System Test. Additionally; Table 5 summarizes the generator metering requirements necessary for the various features of the Generator System Test solution module. Figure 6: Recommended architecture for Generator System Test 2012 Schneider Electric. All rights reserved. Page 23

24 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Generator System Test - Feature Selection Guide Table 5: Generator System Test - Feature Selection Guide Monitoring & Reporting Levels (Note 1) EPSS -1 Essential EPSS -2 Enhanced EPSS -3 Comprehensive Features ATS ATS Status (Time/Date Stamp) ATS Monitoring / Logging / Trending Gen Set Gen Set Monitoring / Logging / Trending Gen Engine Monitoring / Logging / Trending (Note 2) Battery Health Monitoring Fuel Monitoring (Feeder Root) EPSS Test Reports (Note 3) EPSS TEST POWER MONITORING Pass/Fail EPSS Test Indication Web-based Graphical Screens for Remote Test monitoring (Note 4) Remote Initiation of EPSS Test Start from PC (Note 5) Full Supervisory Testing Controls (Note 6) Real-time Tables / Charts Event Capture / Alarming Event based paging Basic PQ Analysis-ATS with PM870 Advanced PQ Analysis-ATS with ION7650 Basic PQ Analysis-GEN Advanced PQ Analysis-GEN Disturbance Sag/Swell-ATS Disturbance Sag/Swell-GEN Transient Capture-GEN with ION7650 Utility Service Metering Breaker Status Monitoring (Note 7) Included Optional Page Schneider Electric. All rights reserved.

25 Main switchboards (LV) After the voltage has been transformed to the low voltage levels (< 600V) it can be distributed throughout the facility. The Main switchboards make up the beginning of the electrical distribution system and provide power to the IT UPS units, mechanical loads, and auxiliary loads. The size of the switchboards depends on the size of the facility and redundancy design. The main switchboard and the recommended metering for the 1MW design are shown in Figure 7. From Switchboard B Figure 7: Main LV Switchboard Recommendation summary Metering PM850 power meter on each feeder into the switchboard (secondary supply optional) Communications: TCP or *Serial through EGX100 gateway * Real-time data performance may suffer Protection Micrologic 6.0A for main feed circuit breaker and all feeder circuit breakers Communications: Serial through EGX100 gateway Key considerations Metering The purpose of metering the inputs of the main switchboard is to monitor proper power quality conditions on the secondary of the MV transformer, and to provide an accurate means to calculate the efficiency of the transformer for various load rates. Power Quality is important because non-linear currents due to the facility 2012 Schneider Electric. All rights reserved. Page 25

26 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide loads or malfunctioning equipment in the facility can create voltage distortions on the secondary of the MV transformer. These distortions are usually filtered by the UPSs but any loads that are not on a UPS, such as motors and other mechanical equipment, may be vulnerable to malfunction or damage. Finally if the recommended metering is chosen, the digital inputs of the meter can be used for status of the transfer scheme to be used with the Generator System Test solution. A PM870 power meter is recommended for each feed into the switchboard. The PM870 power meter can monitor all the key power system metrics, such as energy and demand, with ANSI Class 0.2S and IEC class 0.5S accuracy and monitor all the key power quality metrics, such as THD and harmonics (up to the 61 st ). Also, they can capture waveform traces for critical events in the facility. Their cost is lower than an ION7650 and their feature set meets all the monitoring needs at the switchboard level. Protection Protection units will be present on the circuit breakers in the main switchboard, and specifically for the 1MW reference design the main feeder breakers which have a nominal current (In) of 3615A. The Masterpact NW circuit breakers should be installed, such as the NW40 as outlined in Table 6. The NW40 has a rating of 4000A and a breaking capacity of 150kA. This capacity is well above the calculated short circuit current I CU of 65kA. The feeder circuit breakers follow the same process, but because they have lower current requirements, either Masterpact NT or NW may be suitable for the application depending on the circuit. Table 6: Masterpact NW Circuit Breaker Rating Selecting the right size circuit breaker and the right protection features for the application are important. Masterpact circuit breakers use Micrologic trip units in or to operate the circuit breaker and collect all the operational data. The Micrologic trip units have a variety of features that range from simple over current protection to selective protection combined with earth leakage detection as shown in Figure 8. Page Schneider Electric. All rights reserved.

27 Figure 8: Micrologic Protection Types The main switchboard of a data center allows high inrush currents from the mechanical load motors and inrush currents from various transformers in UPSs and throughout the facility. Some selectivity is required in order to coordinate the operation of the main circuit breaker with protection lower in the electrical distribution system. In the international markets, adding earth fault detection in accordance to IEC can give even more control on the protection scheme because the earth fault scenario has a specific time-current curve. The result is that the electrical distribution system is not reliant on the protection curve used 2012 Schneider Electric. All rights reserved. Page 27

28 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide for selectivity in order to deal with earth faults. For these reasons, a Micrologic 6 unit is recommended for protection through out the facility. After the protection features are selected, the metering features must be selected. If metering is already in place, then a Micrologic 6.0A unit would suffice If meter is not yet installed, then specify a Micrologic 6.0H unit to collect Power and Energy usage and Power Quality data. The feature summary for each different orderable option of the Micrologic trip unit is summarized in Figure. To ensure that all the power quality measurements are available on the trip unit, order the H option. Figure 9: Micrologic Trip Unit Feature Set Summary Adding communications to the trip unit, as illustrated in Figure 10, enables the data collected by the trip unit to be sent to monitoring software. This enables analysis of circuit breaker operations and energy consumption. Page Schneider Electric. All rights reserved.

29 Solution module considerations Figure 10: Micrologic Trip Unit Communications Architecture Table 7: Micrologic Trip Unit Accessories for Communications Item FDM121 display BCM for MasterPact NT/NW CB CCM for Masterpact NT/NW CB External power supply module (AC or DC, 3A) Generator System Test The Generator System Test solution module requires that the status points of the ATS shown in Figure 10 are captured by a meter in order to calculate the transfer time of the ATSs and evaluate them against pre-programmed pass/fail criteria. Without using PM870 power meters, this cannot be accomplished; the software requires that the status contacts are captured in the event log in the meter at the exact time they change. If the customer does not select PM870 power meters, an 2012 Schneider Electric. All rights reserved. Page 29

30 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide ION7550RTU can be specified instead. For further information, refer to the Generator System Test documentation. Power system losses IT UPS input switchboards In order to calculate the power losses due to the MV transformer, a metering point on the primary and secondary of the transformer must be present in the system. It is recommended to use the PM850 power meter because it has onboard logging and high-accuracy energy readings. The IT UPS input switchboard is required for distributing the UPS supply into the three feeds: Main UPS, Static bypass switch and manual bypass. Note that the switchboard does not contain any fault interruption means such as circuit breakers. The switchboard only uses disconnect switches, so there is no opportunity for monitoring the electrical circuits from a trip unit. It is recommended that the switchboard be metered because it enables the Power System Losses solution module. Metering is not required; however, adding meters is an option for the customer. The switchboard is shown in Figure 11. Recommendation summary Figure 11: IT UPS Input Switchboard Metering PM820 power meter on the main input Communications: TCP or *Serial through EGX100 gateway Page Schneider Electric. All rights reserved.

31 Key considerations The reason to include this metering point revolves around what kind of measurements the UPS is capable of providing (see IT UPS modules) and how often the bypass power transformer will be used. If the UPS provides all the necessary information, such as energy and demand and power quality to a satisfactory level and the customer is not concerned with the Power System Losses solution module, then this meter could be eliminated. Otherwise, it can help to cover all possible uses of the facility. Solution module considerations IT UPS modules Power system losses Recommendation summary If the UPS that is downstream from this switchboard is set to static bypass or manual bypass, then the LV transformer below it will be used to feed power to the IT UPS output switchboard. In this case, the only way to calculate the power losses due to this transformation is by monitoring both the primary and secondary of the transformer. By placing the meter at this point, the customer receives the added advantage of monitoring all operations of the UPS by a meter that provides energy and demand with ANSI Class 0.2S and IEC class 0.5S accuracy and monitor all the key power quality metrics, such as THD and harmonics (up to the 61 st ). These detailed measurements are not usually provided by the UPS manufacturer. UPS APC Symmetra MW Communications: Modbus RTU through EGX100 gateway Key considerations The Symmetra MW is a high-power UPS that is modular, fault-tolerant and available between kW. The Symmetra MW provides increased availability through internal N+1 configurability, predictive stoppage notification and multi-module paralleling features. Symmetra MW delivers best-in-class efficiency and a reduction in rating of electrical infrastructure, such as wires, transformers and generators. Slide-in/out power modules, manageable external batteries and self-diagnosing features greatly reduce the mean time to repair. Combined with a wide range of line-up and match options, Symmetra MW 2012 Schneider Electric. All rights reserved. Page 31

32 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide provides a customizable system in a standardized design for any large ondemand network-critical physical infrastructure. Solution module considerations Power usage effectiveness (PUE) The UPS is a mandatory metering point for the Power Usage Effectiveness solution module. Without metering in place at this point in the electrical distribution system, the calculation cannot be carried out according to the Green Grid standards. As shown in Figure 12, the UPSs are critical components to the calculation of PUE. If the UPSs are not monitored, it will be difficult and possibly inaccurate to calculate the PUE of the facility. For more information on how to calculate PUE, refer to Appendix A: How to calculate PUE on page 43. Power system losses IT UPS output switchboards By enabling communications to the UPS, the Power System Losses solution module can be easily configured to report all the efficiencies over time for the UPSs in the customer s facility. Most UPSs report both input and output power. When this metric is logged, the Power System Losses solution module has all it needs in order to report the efficiency of the UPS over time. After the power has been transformed and conditioned by the UPS or the UPS has been bypassed, it is fed to the IT UPS output switchboard as shown in Figure 12. The purpose of this switchboard is to distribute the UPS power to each of the IT loads. In this architecture, there are several possible sources of power, none of which are designed to feed the switchboard at the same time. The 1) main output of the UPS; 2) the static bypass of the UPS which is used when the UPS modules fault or a bypass command is issued to the UPS; and 3) the manual bypass which is used to completely bypass the UPS unit when it needs maintenance. Both the static and manual bypass feeds connect to the supply transformer. This transformer is used for grounding and helps to reduce the propagation of neutral currents higher in the electrical distribution system. Finally, 4) the secondary supply from network B in case maintenance is required higher in the electrical architecture. The switchboard has eight 3 phase-4 wire PDUs, and therefore, eight feeder circuit breakers. Typically, the number of circuits will vary depending on the size of the input feed and the size of the PDUs in the system. Page Schneider Electric. All rights reserved.

33 Recommendation summary Figure 12: IT UPS Output Switchboard Metering PM820 power meter on each feeder into the switchboard Communications: TCP or *Serial through EGX100 gateway * Real-time data performance may suffer Protection Micrologic 6.3A or 6.2A for all feeder circuit breakers Communications: Serial through EGX100 gateway Key considerations Metering Metering is recommended in the switchboard because the design only uses disconnect switches for the input feeds so getting the power consumption data from a trip unit is impossible. At minimum, the main UPS output should be metered with a power quality meter to ensure the UPS is operating within power quality standards. This is important because the PDUs feed sensitive IT equipment, so monitoring and reporting any power quality disturbances can help customers ensure proper IT equipment operation. A PM820 power meter is an ideal choice for this metering point because it provides energy and demand with ANSI Class 0.2S and IEC class 0.5S accuracy and monitors all the key power quality metrics, such as THD and harmonics (up to the 61 st ) and adheres to the EN50160 power quality standard. If the customer s budget allows, it is also recommended to monitor the secondary of the bypass transformer for 2012 Schneider Electric. All rights reserved. Page 33

34 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide the same reasons. Finally, if the customer wants every scenario monitored, then adding a meter to the secondary feed from network B can be recommended. Protection To determine the applicable protection unit, examine the size of the circuit breakers. The circuit breakers will be sized based on the conductor size that feeds the PDUs, which will be sized based on the nominal PDU load. In this case, seven of the circuit breakers are feeding 277kW loads, which at 410V LL results in approximately 390A at full load current. The protection setting (see the single line diagram) used (In) was 450A. Using the Compact NSX/Powerpact circuit breaker selection guide (except below in Table 8) the setting of 450A aligns with the Compact NSX 630 circuit breaker for IEC markets and the Powerpact 600A L-Frame for ANSI markets. The remaining PDU is a 138kW load, which at 410V results in approximately 195A full load current. A nominal setting of 230A was selected and aligns with the Compact NSX 250 circuit breaker for IEC markets and the Powerpact 250A J-Frame for ANSI markets. Table 8: Compact NSX/Powerpact Circuit Breaker Sizes The Micrologic trip units can have a variety of monitoring features as outlined in Table 9. If metering has been installed, then choose feature set A because the power monitoring is done at the feeder level. If metering is not in place, then choose feature set E. Page Schneider Electric. All rights reserved.

35 Table 9: Micrologic Trip Unit Power Metering Functions Solution module considerations Power usage effectiveness (PUE) The PUE of the facility can be calculated in many ways (refer to Recommendations for Measuring and Reporting Overall Data Center Efficiency available at If the customer requires the category 1 calculation, then the meter at the input of this switchboard should be used for its higher accuracy then the output measurements of the UPS Schneider Electric. All rights reserved. Page 35

36 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Power system losses Power Distribution Units (PDUs) Recommendation summary In order to report the power losses (efficiency) of the bypass transformer, a metering point on the primary and secondary of the transformer is required. Although this data would only be available when the UPS is in manual bypass, which is rare, the only way to calculate the metric is to add a meter or to use virtual summations of all the feeder circuit breakers. A second option would cost more money in engineering services and is not as easily understood by a customer. Metering PM5350 power meter for each PDU Communications: Serial through EGX100 gateway Key considerations Many PDUs have metering installed in the unit already, so metering the PDU with an external power meter may appear to be redundant, which it may be. If the customer is not concerned with the accuracy of the measurements and is content with an estimated figure, then metering is not required. If a customer s PDUs are not metered, or if the customer wants to ensure the readings are accurate for billing purposes or that the power quality measurements meet industry standards to ensure safe IT equipment operation, then a meter may be warranted. The recommended meter is the PM5350 power meter. This meter s energy accuracy is compliant with IEC class 0.5S and it also reports power quality measurements, such as THD and TDD. The meter includes Modbus communications, so it is easy to integrate into any power monitoring network. Solution module considerations Metering this location is not critical to any solution module in the Power Solutions for Data Centers offer. Mechanical UPS switchboard (input and output) The main purpose of having a mechanical UPS is to ensure the critical cooling loads, such as Computer Room Air Conditioners (CRACs) and Computer Room Air Handlers (CRAHs) stay powered during system events. These units provide precise temperature and humidity control for mission critical environments, so Page Schneider Electric. All rights reserved. Figure 1: Mechanical UPS Switchboard

37 any disturbance to their supply power could negatively impact the operation of the facility. The 1MW reference design calls for using in-row cooling units that are fed from the PDUs, so only the pumps that pressurize the cooling lines to the inrow units are fed from the mechanical UPS. The switchboard and the recommended metering design are shown in Figure 13. Figure 13: Mechanical UPS switchboard 2012 Schneider Electric. All rights reserved. Page 37

38 StruxureWare Power Monitoring /2012 Power Solutions for Data Centers Metering and protection architecture guide Recommendation summary Metering PM820 power meter on supply and output of UPS. Communications: TCP or *Serial through EGX100 gateway * Real-time data performance may suffer Key considerations A key consideration when including metering points is what kind of measurements the UPS is capable of providing. If the UPS provides all the necessary information, such as energy and demand and power quality to a satisfactory level and the customer is not concerned with the Power System Losses solution module then as a cost saving measure the meters on the input to the switchboard and the output of the USP could be eliminated. If a customer is interested in getting a detailed view of their mechanical system, the PM820 power meters are highly recommended. They provide monitoring of energy and demand and power quality which will ensure the UPS is operating normally. Also, PM9Cs can be added to the individual feeder branches of the switchboard to track individual equipment performance and energy consumption. They are IEC Class 0.5S and IEC Class 0.5 compliant; they report energy consumption with accuracy of 0.5% at 1.0 pf and 0.6% at 0.5pf. Solution module considerations Power system losses In order to report the power losses (efficiency) of the UPS a metering point on the primary and secondary of the unit is required. If the UPS has communications, the secondary meter is not mandatory. However, if there is not metering on board the UPS or more accurate metering is desired then the PM820 power meter is an ideal choice. Mechanical UPS modules Recommendation summary UPS APC MGE UPS Galaxy 5000 Communications: SNMP or *Serial through EGX100 gateway * Real-time data performance may suffer Page Schneider Electric. All rights reserved.