POWER EQUIPMENT AND DATA CENTER DESIGN

Transcription

1 WHITE PAPER #51 POWER EQUIPMENT AND DATA CENTER DESIGN EDITORS: David Loucks, Eaton Stephen McCluer, Schneider Electric CONTRIBUTORS: Lynn Simmons, Dell John Collins, Eaton George Navarro, Eaton Dusty Becker, Emerson Network Power Bill Campbell, Emerson Network Power Harry Handlin, GE Digital Energy Mark Szalkus, GE Digital Energy Brad Thrash, GE Digital Energy Pamela Lembke, IBM Kevin Bross, Intel Shaun Harris, Microsoft Jim Spitaels, Schneider Electric

2 PAGE 2 Table of Contents I. Introduction and scope... 3 II. Categories and definitions... 3 III. Power Component Requirements... 5 IV. About The Green Grid...51

3 PAGE 3 I. Introduction and scope This paper begins with a description of a power system diagram and a discussion of its importance. A complete power system diagram for a specific data center is essential for discussion on all aspects of designing an efficient power system. The paper then discusses various topics that impact efficiency, including power system configurations and how a data center is grown over time. Following this, the focus becomes more specific and different power system components are discussed in detail. A diagram showing measurement and control points is provided for each component as a point of reference. Timetables (current, 2014, and 2014+) for implementation of the core tenets of this guide are provided for each component and are intended to serve as a quick reference for data center designers. For example, someone planning on upgrading UPSs can refer to the topic on instrumentation in the timetable for UPSs and learn what specific features are recommended to be implemented in the year Wherever practical the text will distinguish between direct current (DC) and alternating current (AC) power distribution. However, when covering more general topics associated with power, the distinction may not be made and both DC and AC power solutions should be given consideration. II. Categories and definitions The following categories and components will be reviewed in their own sections of this paper. Each section category/component will contain an overview of the section, general information about efficiency, then more detail (including specific efficiency metrics). The End to End Power System The end to end power system includes all of the components that are a part of the power distribution path, that is, the path to move electricity from the building utility service to the IT load. UPS The uninterruptible power supply (UPS) converts unconditioned power to provide conditioned power to critical loads without interruption. It contains an energy storage system, such as a bank of batteries, which supply power to the load when utility power is unavailable. This discussion includes double conversion AC UPSs, line

4 PAGE 4 interactive AC UPSs, AC UPS eco-mode operation, and DC UPSs. Note that although a data center commonly has multiple UPSs, they are modeled as a single block. DC UPSs are also modeled as a single block, even though they may actually consist of multiple rectifier modules and separate batteries. Where appropriate, isolation is indicated in the block diagrams. Voltage-converting AC UPSs may alternatively incorporate an autotransformer in their bypass or output path. For more detail on UPS types, see IEC Transformers An isolation transformer is an electromagnetic device with multiple windings per phase that converts an AC voltage into another AC voltage and simultaneously provides galvanic isolation by transmitting energy between windings through magnetic inductive coupling (induction). The transformer winding size and type is selected to provide the required voltage transformation level and may also provide secondary benefits including improved safety ground performance, AC-DC (battery) isolation, harmonic cancellation through phase-shifting, and transient voltage reduction through grounding. An autotransformer is similar in material and function. However, it will be physically smaller and cost less than an equivalent isolation transformer. It changes voltage, but it does not have any additional power quality characteristics, such as galvanic isolation and harmonic cancellation, which are found in the isolation transformer. PDU A Power Distribution Unit (PDU) is an electrical distribution cabinet, free-standing or rack-mounted, whose main function is to provide a required point of power distribution. The PDU houses circuit breakers that are used to create multiple branch circuits from a single feeder circuit, and can also contain transformers, electrical panel boards, surge protection devices, and power monitoring/controls. ICTE PSU A typical power supply unit (PSU) for Information and Communication Technology Equipment (ICTE) is designed to convert (rectify) alternating current (ac) voltage from the mains supply to several direct current (dc) voltages, both positive and negative, typically + 12V,-12V,+5V,+5V standby and +3.3V. Switched mode power supplies (SMPS) are the predominant form. Some PSU models are available with DC inputs. 1

5 PAGE 5 Because the diverse collection of output voltages results in widely varying current draw requirements, most modern computer power supplies actually consist of several different switched mode power supplies. Each produces just one voltage component and each is able to vary its output based on component power requirements. All are linked together to shut down as a group in the event of a fault condition. Generator set A generator is made up of an engine (sized in kw) which spins an alternator (sized in kva) to produce AC electric power. Several generator sets may be connected to work together (in parallel) to provide the required power for the connected load. Generators may be portable or fixed and are available in a variety of sizes ranging from a few kw or kva to several MW or MVA each. Generators can be connected to data center power systems using either transfer switches, paralleling switchgear, or both, depending on the needs and design of the installation. III. Power Component Requirements The End to End Power System Creation of a detailed power system design specification is the beginning point for any discussion of the impact of a data center s power components on overall efficiency of the data center. A specification should address questions such as capacity constraints, space constraints, and both upstream and downstream power distribution. The data center power source capacity (current) and configuration (voltage), as well as the requirements of the expected compute loads (servers, storage, network, etc.), must be well defined and understood. The diagrams that follow below are for AC systems. The DC systems are not shown in the diagrams, but are similar. Typically the difference lies in removing the inverter. Static bypass and maintenance bypass would also be removed from the diagram and replaced with additional rectifiers. A common means of describing a data center power system is the power system diagram, often referred to as a one-line diagram in three-phase AC systems or DC systems. See Figure III-1 for a simplified example. Other input and output voltage ranges are also available, including medium voltage.

6 PAGE 6 208/120V Non-essential Loads (lighting, etc.) 480/277V Gen 480/277 VAC Maintenance Bypass PDU Critical Loads PSUs Gen ATS Static Bypass PDU PSUs UPS 34 kv fuse 480/277 VAC PDU PSUs Utility Utility Transformer Essential Loads (mechanical, cooling, etc.) Main Switchboard Figure III-1: Simplified Diagram 60 Hz Data Center Power System with Static UPS Each piece of power equipment is a fundamental building block in the larger power system design that a data center depends on for performance and features. It is the arrangement of these blocks into a functional system that matters the most to data center performance objectives and requirements. The power system diagram can be used to show the entire power system from end to end and in some cases can do so all on one concise sheet. At one end is the incoming power source. At the other end are the loads. Power system diagrams, though electrical in nature, can provide information about system and subsystem function and location. Functional details called out in the power system diagram can include system redundancy, the type of energy storage elements, emergency backup generators, maintenance bypass devices, protection devices (circuit breakers, breaker panels, and fusing), and metering points. A power system diagram can also include monitoring and measuring devices that are critical to data center efficiency. The diagram needs to be well defined and understood. This applies equally to both existing and new data centers. Building management or other system management software applications contain what amounts to a power system diagram with cooling and compute layouts added.

7 PAGE 7 Availability and Efficiency For the most critical loads, operational availability is the primary objective for the physical infrastructure. All data center power designs are based on an underlying assumption that power disturbances (such as transients, voltage fluctuations, outages, and other power quality abnormalities) are inevitable, whether originating from the electric utility or from within the facility. Any of these conditions can damage equipment and/or interrupt operations. Therefore equipment to mitigate such disturbances is built into every design. The addition of these devices increases energy consumption through added efficiency losses. Power system component layout and level of redundancy affect the losses within the power system. Built-in redundancy is perceived to be necessary to achieve maximum availability. To achieve greater levels of redundancy, more components are used to provide the same amount of backup power. These additional components, even when operating in standby mode, consume energy and therefore reduce the overall data center efficiency, as defined by PUE 2 or DCiE 3. Designers should be aware of the trade-off between redundancy/availability and efficiency. They should also consider solutions in which the power system can be reconfigured to provide the needed power and redundancy within the required time, while not needlessly operating standby equipment. Some common configurations are discussed here. A-B ( Dual Bus ) Systems This power system architecture is configured with two sides, A and B, each operating at less than 50 percent load. Each side can include multiple UPSs. Either side can handle 100 percent of the system load. If one of the sides has a problem, the load is connected or switched to the other operational side. The tie of the two sides requires a combination of switching devices and synchronization of all components downstream of the UPS. Switches can reside upstream of the UPS for maintenance isolation purposes. Figure 1-2 shows an example of AC system redundancy, designed to allow a high level of system availability even during maintenance or component failure. Extra redundancy at strategic areas inside your power system may lead to a higher availability. 2 Definition of PUE, Grid-Data-Center-Power-Efficiency-Metrics-PUE-and-DCiE.aspx 3 (see definition of PUE above)

8 PAGE 8 Side A Side B Utility Automatic transfer switch Generator Utility Automatic transfer switch Generator Distribution switchgear t e x t t e x t tie breaker Distribution switchgear Noncritical loads Essential mechanical loads UPS UPS t e x t e x t Maintenance bypass Noncritical loads Essential mechanical loads UPS UPS t e x t e x t Maintenance bypass 2-input critical load UPS Uninterruptible power supply PDU Power distribution unit PDU 2-input critical load Static transfer switch PDU 1-input critical load(s) Figure III-2: Simple Diagram - A-B Redundant power System The tie points can be of a continuous or momentary type with momentary being the more typical approach. A-B systems can be implemented either with a central bypass module or a distributed bypass. The centralized bypass module uses uninterruptible power supplies without internal bypass functionality. The distributed bypass has the bypass integrated with the individual parallel UPS. Distributed bypass is less common in current installations, but is coming of age and will be increasingly used. Note that even though this discussion describes a dual power path system, more than two paths are permissible. Isolated Redundant Systems Also known as a catcher system, this configuration can include any number of UPSs and includes a standby UPS prepared to back up the primary UPS. The redundancy is provided by virtue of a transfer to the catch side.

9 PAGE 9 One method is to use the standby UPS to power the static bypass of each connected UPS, as shown infigure III-3. Another method is to put a static transfer switches between the output of each UPS and its load, thereby introducing a single point of failure The standby UPS must be able to instantaneously pick up the load of one or more faulted UPS systems, which can result in a potential step load from zero to over 100 percent of rated capacity. In either scenario, the standby UPS runs at idle almost all of the time, consuming energy but providing no actual conditioning or protection until needed. Utility Generator Standby UPS (2N capacity) Static Bypass Automatic Transfer Switch Primary UPS 1 (N capacity) Static Bypass To PDU Group 1 Maintenance Bypass Static Bypass Primary UPS 2 (N capacity) To PDU Group 2 Maintenance Bypass

10 PAGE 10 Figure III-3: One-line representation of one of several isolated redundant (catcher) system configurations Emerging Power Systems Beyond the systems described above, other technologies are emerging that may provide more mainstream solutions in the future. These types of systems, most conducive to cloud technologies, move the redundancy out of the site power distribution line and into other areas, such as software or mirrored data centers. Reliability may be affected from a power system standpoint but is addressed elsewhere in the company s larger data center picture. Several examples of this exist today. At a basic level, ITE can be fed with two feeds, one on a UPS and one with utility only on the other. Battery backup function can be moved from the central UPS location to inside each server. Backup time can be reduced to less than 5 minutes. This can reduce overhead for a full UPS, but can lead to issues when dealing with batteries that need to be replaced prior to a server s end of life. A combination between the two is available: at the power supply level, the main AC input can come directly from the utility (the efficiency input ), with second PSU input of a DC battery backup (the availability input ), where batteries are racked near the ITE unit. Further outside the box come newer energy storage technologies. Lead acid batteries are the most common today, but other battery chemistries (such as lithium-ion) may become available that aid in more efficient, reliability, or economical storage. Other technologies like supercapacitors (or ultracapacitors) or flywheels could help with removal of batteries all together. Accuracy in measurement Any level of accuracy can be measured today. However, to measure with high accuracy is quite costly. Individual component built-in meters are typically not accurate enough for good PUE measurement (built in meters are around 95 to 98 percent accurate), but externally available bolt on type meters can be attached to obtain the desired level of accuracy. However, because of costs, this accuracy of metering is neither common practice on purchased powertrain components or at an installed data center facility. The accuracy of the measurement varies by use. If the measurements are used for tracking changes in performance within a facility, the built in meters are generally accurate enough. If certification or comparison to other facilities is required, more accurate supplemental meters should be installed at appropriate points in the power path.

11 PAGE 11 Core Tenets Timetable Instrumentation 2012: Typically, products are metered individually. Total facility power is measured manually at a meter or monitored via a utility bill. All products are be instrumented to measure volts and amps, enabling the calculation of watts, VA and apparent power factor. 2014: Metering should progress further down the powertrain (for example, from UPS output to PDU output to power supply input). Accuracy of measurements should improve to 1 percent : Metering should measure energy as well as power. Accuracy of measurements should improve to 0.5 percent if individual components in the power path will be measured in real time. Discoverability 2012: Modbus systems are configured on installation. System Network Mapping Protocol (SNMP) devices can be discovered, but are more often configured on installation. 2014: SNMP V3 will be more widely implemented : SNMP V3 will be used for discovering all components along the power chain. All components will be discoverable and auto announced, able to be integrated in with the smart grid. Scalability 2012: In general, power paths have more redundancy than required and so are not optimized for efficiency. 2014:

12 PAGE 12 All power chain components should be operating at the highest efficiency for the load range used. Equipment will become more automated and adapted to self-optimize its efficiency by scaling pieces or group of components : Provisioning power quality (that is, how and where the power is supplied from) should be based on the equipment type to reduce efficiency losses and cost where not required. Align rack power sizing with typical uses. Enhanced management infrastructure 2012: None 2014: None 2014+: Power equipment will use SNMP V3 as the de facto protocol to communicate between itself and the centralized management agent. Ability to be policy driven 2012: Components in the power chain Measured data from instrumented units is available or calculated by the system and displayed on the system. Each piece of equipment has its own timestamp, utility pricing and carbon awareness. This is statically known when available. 2014: Components in the power chain All equipment obtains a timestamp from one location. Accuracy of the timestamp improves over time. One second accuracy is required for efficiency purposes. More accuracy is required for fault analysis. All measured and calculated values are available to the centralized management console, including but not limited to volts, amps, watts, VA, apparent power factor. Energy (kwh) is calculated by the centralized management console.

13 PAGE 13 Utility pricing and carbon awareness Available in real time 2014+: Components in the power chain The centralized management console will calculate, report, and use the average, minimum and maximum values of volts, amps, watts, VA, power factor, energy. The centralized management console will take all inputs to optimize the overall efficiency of the power chain, specifically using PUE, predicting the most optimal path to efficiency. Utility pricing and carbon awareness Future forecasted; adjustments made to components of powertrain (shift workloads, shift to generator, etc.) based on forecast. Standardized metrics Metrics should match with Data Center Sustainability Model 2012: End to end efficiency for the critical power path (Building entrance to IT load) <90 percent for the data center usage pattern 2014: End to end efficiency for the critical power path of 90 percent Efficiency for the data center usage pattern 2014+: End to end efficiency for the critical power path of 92 percent Efficiency for the data center usage pattern Applicable Standards EPA Energy Star products for business and government:

14 PAGE 14 UPS Efficiency Topics Modes of operation UPS topology classifications and efficiency performance is guided by what s referred to as normal mode of operation. A double conversion UPS processes power through the rectifier (AC to DC) conversion and the inverter (DC to AC) conversion while a standby UPS largely operates its bypass circuit, passing somewhat unconditioned power from the input AC utility source to the output load. Recent development in UPS technology has introduced system architecture and controls that enable advanced UPS performance. Many UPSs that are fundamentally designed to perform as a double conversion can also operate in standby mode to raise the system efficiency, known generically as energy saving or economy mode ( eco-mode ) 4. Energy saving mode offers less power protection by nature, but is higher efficiency because it is not processing the same amount of power as a traditional double conversion UPS. There may be tradeoffs in levels of protection vs. efficiency that come with multimode operation so that it is necessary to choose the level that is appropriate for the data center requirement. Another example of more efficient modes of operation is the multi-module UPS. The UPS can place unneeded UPS modules in a low power standby mode. Another type of multi-mode operation does not cross between UPS classifications or types, but falls in the category of system provisioning. A UPS can be designed and configured to optimize its operation real-time to improve system efficiency. The two simple examples described have datacenter infrastructure equivalents: a) datacenters throttling back cooling in synchronization with heat load and b) server consolidation setting compute loads to operate at higher duty cycles and reduce overhead losses. The first example of UPS subsystem provisioning involves cooling. Advances in UPS controls have afforded the ability for the UPS to control its internal fan or cooling load as a function of operating conditions, output load and/or ambient temperature. Cooling can represent about 1 percent of a UPS s fixed losses, or power consumed that does not provide useful work to the critical output load. If fans can be throttled pack during lighter loads, system efficiency can be maintained at an optimal level under different operating conditions. 4

15 PAGE 15 The same applies to cooling load as a function of ambient temperature - UPSs designed to operate in ambient temperatures of 40 degrees C can made to operate cooling fans at slower speeds when the ambient that UPS operates in is well below 40 degrees C. System Airflow and Cooling Fan loads represent a big portion of a UPS s inefficiency. Ventilation is a particular design aspect of a given UPS and should take fan airflow into consideration in order to provide precise thermal management. Understanding a UPS fan arrangement and reasons for its design and construction can help identify system inefficiencies. Look for underlying thermal management that shares fan airflow across multiple power train sections. At times the number and size of fans relates to a design target for fan redundancy. The use of large fans that provide a broader airflow stream can provide degrees of redundancy similar to multiple smaller fans and reduce the system fan load. Environmental Conditions Ambient temperature - allow the measured or monitored environmental parameters to dictate levels of performance.

16 PAGE 16 Measurement and Control Diagram The following block diagram serves as a basis for illustrating different components of the CIM model for an AC UPS: Inputs: Outputs: Phase A (V, I, kw, kwh) Phase A (V, I, W) Phase B (V, I, kw, kwh) Phase C (V, I, kw, kwh) AC Bypass Input AC Output Phase B (V, I, W) Phase C (V, I, W) Phase A (V, I, kw, kwh) Phase B (V, I, kw, kwh) Phase C (V, I, kw, kwh) DC+ (Vdc, Idc) AC Utility Input AC UPS Status Mode (normal, energy saving, etc.) DC- (Vdc, Idc) DC Input Mode Inhibit Control Input Figure III-4: AC UPS Measurement and control points

17 PAGE 17 Core Tenets Timetable Instrumentation 2012: Nearly all new UPSs are fully instrumented to provide real time operating power and performance data. ENERGY STAR for data centers requires monitoring at the output of the UPS for PUE. Efficiency for most UPSs is statically known based on lab measurements, but might not be gathered in real time. Attribute AC Bypass Input AC Utility Input DC Input Notes Ingress power connection to the AC UPS from external ac bypass source Ingress power connection to the AC UPS from external ac utility source (can include a generator source) Ingress power connection to the AC UPS from external dc input source; could be a battery source or other type of DC source Control Input AC Output Egress power connection to the AC UPS to external ac output load bus Status Table III-1: Current AC UPS Connection Point Type Values

18 PAGE : (continued) Location Attribute Units AC Utility Input RMS Voltage Volts RMS Current Amperes Real Power Kilowatts Energy = Real Power Kilowatt-hrs * Time Frequency Interval Hz AC Bypass RMS Voltage Volts Input RMS Current Amperes Real Power Kilowatts Real Power * Time Kilowatt-hrs Frequency Interval Hz AC Output RMS Voltage Volts RMS Current Amperes Real Power Kilowatts Real Power * Time Kilowatt-hrs Frequency Interval Hz DC Input Average Voltage Volts DC Current Amperes Average Power Kilowatts Battery Time Minutes Table III-2: 2012 AC UPS Connection Point Metered Remaining Values 2014: Meters should be accurate to +/- 1 percent. Efficiency is measured in real time and accurate to +/- 1 percent. Instrumentation should be added for the following metered values: Location Attribute Units AC Output UPS Output Energy kw AC Utility Input Ambient Temperature Degrees C AC Utility Input Humidity % AC Output True Power Factor %

19 PAGE 19 Table III-3: Additional AC UPS Connection Point Metered Values for : Meters should be accurate to +/- 0.5 percent. Efficiency is measured in real time accurate to +/- 0.5 percent. Instrumentation should be added for display and alarm of the following metered values: Location Attribute Units AC Utility Input RMS Under voltage Volts AC Bypass Input AC Output RMS Over voltage RMS Under voltage RMS Over voltage RMS Under voltage Volts Volts Volts Volts RMS Over voltage Volts Overload Kilowatts System Efficiency % Availability Time Table III-4: Additional AC UPS Connection Point Metered Values for Discoverability 2012: UPS reports the items listed below: Attribute Definition Units Notes ID Name/Model Text String Table III-5: 2012 AC UPS Connection Point Status Values

20 PAGE : The UPS should additionally report the attributes below, and report all attributes in real time, at least every minute: Attribute Definition Units Notes Location Physical location Datacenter Security Sensitivity Floor # Room # Security Sensitivity Security Sensitivity Table III-6: Additional AC UPS Connection Point Status Values for : The UPS should report all attributes in real time, at least every second. Scalability 2012: There is currently no standard for UPS scalability. 2014: A UPS should be designed to add or remove capacity in real time for operation at the optimal efficiency point (without shutting down the load, hot scalable) or automatically self-configuring : UPS efficiency curves should be as flat as possible, operating efficiently at any load. This allows data centers to move away from modules. Efficiency should be 95 percent or higher when operating between percent load. Note that scalability affects reliability. For more information see IEEE 493, Gold Book. Enhanced management Infrastructure None specific to UPS.

21 PAGE 21 Ability to be policy driven 2012: Available operating modes for the UPS are statically known and controllable. Modes include double conversion, line interactive, eco-mode, bypass, offline hot standby, offline cold standby, and off. 2014: Above operating modes are pollable and controllable by the centralized management console. Efficiency is estimated at the UPS level by the unit and sent to a centralized management console : Operating modes are announced and controllable; Units communicate instantaneous monitoring and prediction of efficiency in each mode and can be adjusted based on this data. Standardized metrics For UPSs, metrics should harmonize with data center sustainability: 2012: No current efficiency standards are available for UPSs worldwide. UPS efficiency should be measured according to the standard IEC Annex I (2010). 2014: UPSs used in data centers should meet the efficiency requirements as outlined by the EU code of conduct or Energy Star for UPSs. Energy saving mode UPSs should be used if conducive to the business type : UPSs used in data centers should choose backup technologies based on TCO, materials and sustainability. Applicable Standards For efficiency: IEC 62040

22 PAGE 22 EPA Energy Star for UPSs, For reliability: IEEE 493 For design and safety: UL1778 For electromechanical design, specific to US: NEMA PE1 Transformers Efficiency Topics Transformers can appear in many places within the power train. For purposes of this discussion, the focus is on stand-alone transformers rather than transformers embedded in the equipment (such as UPSs or PDUs), as the performance of those transformers is factored into the performance of the device in which they are installed. Currently data centers have numerous isolation transformers because of the need to reduce the size and weight of conductors. Primarily for safety reasons, the loads within a data center presently tend to be operated at lower voltages ( Vac). As the number and size of the loads increases, the current draw increases. Higher currents require larger conductors. Larger conductors require more space and weigh more. Since electrical power is proportional to system voltage multiplied by current, at some point it becomes more cost effective to distribute power at higher voltages since the current levels (and therefore conductor sizes) will be smaller. In alternating current systems, a transformer is used to transform the voltage level. A more efficient practice would be to use fewer transformers or to use autotransformers throughout the data center. This is done by using end devices that will tolerate operating at higher voltages. However, this can pose a safety issue at the rack because of lower impedance and higher fault currents. Designers need to be aware of the tradeoffs when determining the voltage for their data center equipment.

23 PAGE 23 Measurement and Control diagram The following block diagram serves as a basis for illustrating different components of the CIM model for a transformer. Inputs: Outputs: Phase A (V, I, kw, kwh) Phase A (V, I, kw, kwh) Phase B (V, I, kw, kwh) Phase C (V, I, kw, kwh) Transformer Primary Transformer Secondary Phase B (V, I, kw, kwh) Phase C (V, I, kw, kwh) Transformer Temperature Pressure/Vacuum Liquid Level Figure III-5: Transformer Measurement and control points Core Tenets Timetable Instrumentation Transformers can be metered for many reasons. These include providing signals for protection, data for diagnostics, and data for energy management. The type of transformer instrumentation selected is typically the result of a trade-off between the cost of the instrumentation versus the importance of the transformer, or of the importance of the data or loads fed by the transformer. The instrumentation selected may include the monitoring of current, voltage, and the various signals derived from those two (power, energy, power factor, distortion, etc.), as well as temperature. In the case of dry-type transformers, the temperature monitoring may be done at multiple points on cores (a single transformer has one core per phase) or windings. In the case of liquid-filled transformers, temperature monitoring is usually only done at the top of the fluid. Typically liquid level and tank pressure or vacuum are also monitored in liquid-filled transformers. Transformers with forced cooling, either air and/or liquid, may have additional monitoring of the motor driving the fans and/or pumps. This fan/pump monitoring may include voltage, current, and values from those signals.

24 PAGE 24 Liquid-filled transformers are typically not found inside a data center and are typically larger than those used for individual PDUs. Large, critical transformers will typically also have sensitive transformer differential-current monitoring where the total current flowing into the transformer is compared to the total current flowing out of the transformer, adjusted by the turns ratio. This particular monitoring, called transformer differential protection, looks for small differences in the calculated value versus the actual value of the ratio of input to output current. Small differences can indicate a short within the transformer winding. This particular type of monitoring may not be cost efficient in a typical data center, however. 2012: Attribute Input Output Temperature Pressure or vacuum Liquid Level Notes Ingress power connection to the transformer from an external primaryside source (can include a generator source) Egress power connection from the transformer to an external secondary-side load Egress transformer system internal temperature reading Egress transformer system pressure reading Egress transformer system liquid level status Table III-7: 2012 Transformer Connection Point Type Values

25 PAGE : (continued) Attribute Definition Units Input RMS Voltage Volts Output RMS Current Real Power Energy = Real Power * Time Interval Frequency RMS Voltage Amperes Kilowatts Kilowatt-hrs Hz Volts RMS Current Amperes Real Power Kilowatts Real Power * Time Interval Kilowatt-hrs Frequency Hz System Temperature Degrees C Pressure PSI Liquid Level OK Logic High Table III-8: 2012 Transformer Connection Point Metered Values Typically, only input or output power is measured. High accuracy measurements at utility input should be more accurate than compute room level. Attribute Definition Units Notes System Liquid Level OK Logic High Table III-9: 2012 Transformer Connection Point Status Values

26 PAGE : Measure both input and output power to calculate efficiency. Efficiency measurements can not only monitor for savings, but can also monitor for degradation and harmonic currents. Measurement does not need to be done in real time, but should be measured periodically, either automatically or manually : None. Discoverability 2012: Transformers, at the basic core and coil level, are inherently non-reporting and do not announce energy or performance levels or location. However, some subset of the total installed base of transformers does have separate instrumentation installed that will report certain electrical and mechanical properties. This instrumentation typically is not manufactured by the same company as the transformer, but may be integrated into the transformer by the transformer OEM. The data reported from this instrumentation may be presented in the form of a display and keypad mounted on or near the transformer equipment or it could be in the form of mechanical gauges with mechanical contacts that open or close when certain limits are reached. These contacts can be supplied with a wetting voltage and the signal wired into a data center monitoring system through a digital input module. More advanced systems may use serial communications from the transformer monitor to the data center. For those systems that do have communications capability, the physical connections and protocols range from vendor proprietary to well documented open standards. As a result, the integration of transformer data almost always requires a custom integration effort. This effort varies depending on what combination of different vendor's electrical and mechanical monitoring systems are installed on any particular transformer. For a list of transformer attributes that might be reported, refer to the appendix.

27 PAGE : Attribute Definition Units Notes Location Physical location Datacenter Security Sensitivity Floor # Security Sensitivity Room # Security Sensitivity Table III-10: Proposed discoverability attributes 2014+: None Scalability Although transformers can be paralleled to increase available capacity as long as the turns ratios, phase shift (from primary to secondary), and phase rotation are matched between units, this is almost never done in practice. However, there is a particular type of distribution system called a network (either spot or grid) that does rely on paralleled transformers, but these are coupled with special types of protective devices called network protectors that monitor the direction, type (real vs. reactive), and magnitude of current flowing through each transformer. A system could be built with network protectors in anticipation of paralleling transformers. However, paralleling transformers increases available fault current and may over-duty the electrical distribution system unless it is installed with the thought that future upstream transformers may be paralleled. 2012: In the United States, low voltage (600 V and less on either winding) dry-type distribution transformers manufactured on or after January 1, 2007 are now required to meet NEMA TP1 efficiency levels. Likewise in the US, all transformers with rated primary voltage of between 601 and volts and a rated output voltage of 600 volts and less manufactured on or after January 1, 2010 are required to meet the efficiency standards outlined in 10 CFR Part

28 PAGE : The U.S. Department of Energy is examining higher efficiency transformers. Several Candidate Standards Levels (CSL) have been proposed. Currently interest is focusing on the CSL3 as a likely candidate for the future energy efficiency target, replacing the current TP-1 (CSL1) transformer, although the publish date of this document has not been decided. Fewer and higher efficiency transformers (NEMA TP1 or equivalent) should be used. Increasing the voltage within the datacenter increases this efficiency : Systems designed from the outset as a spot network may provide more scalability and higher reliability than conventional primary selective systems; because the system can be designed as N+1 (or more) without having to add any switching. Adding transformers increases capacity and redundancy, but it also adds available fault current. A distribution system designed with that in mind will be braced and rated for the higher fault currents. Efficiency can be increased by using fewer series isolation transformers or replacing them with autotransformers. Enhanced management Infrastructure 2012: As mentioned previously, transformers currently require external hardware to report into a management infrastructure. This hardware could be specified (but typically is not) to report into an IT monitoring system. 2014: None 2014+: None Ability to be policy driven This does not apply to transformers. Standardized metrics 2012:

29 PAGE 29 As a device that attempts to transport electrical energy as efficiently as possible, one design goal is to reduce transformer losses and thereby increase transformer efficiency. Transformer efficiency is defined as the ratio of the useful power leaving the transformer divided by total power entering the transformer; typically, the larger the transformer, the higher the efficiency. Today, transformers usually do not include high accuracy power monitoring of both the input (primary) and output (secondary) power, other than the transformer current differential protection mentioned previously. (Note that this differential monitoring neither calculates nor displays transformer efficiency for the operator to use.) 2014: None : Transformer efficiency should meet the appropriate CSL standard that will be defined by industry. Applicable Standards For efficiency: NEMA TP1

30 PAGE 30 PDU A Power Distribution Unit (PDU) is defined here as a free standing electrical distribution cabinet, whose main function is to provide a required point of power distribution. The PDU houses circuit breakers that are used to create multiple branch circuits from a single feeder circuit, and can also contain transformers, electrical panel boards, and surge protection devices. PDU transformers A secondary function of some PDUs is to convert voltage. The AC voltage converting PDUs contain either an isolation transformer or an autotransformer to step the AC distribution voltage down. See the Transformers section for more information. DC voltage converting PDUs contain DC/DC converters. For simplicity, multiple PDUs are modeled as a single block. Where appropriate, isolation is indicated in the block diagrams. Typical Power Distribution Unit Specifications: Rating: kVA Transformer type: Isolation transformer or Autotransformer Harmonic tolerance type: K4, K13, or K20 (optional based on need) High efficiency option: (TP1 or better) Input Voltages: 208,480, or 600VAC Output Voltages: 208/120VAC PDU Types A large data center typically contains a large number of PDUs. A single site can contain over 100 PDUs. The location and number of distribution points depends on the power system layout and design and can take on many more forms depending on their location and function in the overall power system, as shown in Figure 1-2.

31 PAGE 31 To IT Loads Input Metering Optional Transformer Output Metering To rack-mounted distribution To remote power panels Figure III-6: Representative single-line diagram of a floor-mounted PDU Examples of PDUs include Remote Power Panels (RPP), Power Distribution Racks (PDR), Rack Power Modules (RPM), and zero U electrical strips (epdu). The PDU naming convention used will vary depending on the equipment vendor. The following generic definitions serve as a guide in determining where a particular PDU type resides and what its purpose may be: Remote Power Panel (RPP) A remote power panel is a distribution panel that is subfed from an upstream panel, usually from a floor-mount PDU. This PDU maximizes use of datacenter space by locating the point of distribution of moderately high current capacity close to compute loads. The device is basically a transformerless PDU that meets a particular form factor requirement. Generally equipped with standard panelboards, this cabinet accommodates branch circuit breakers. Like its transformer- based relative, it can be configured with single or dual source feeds to accommodate an A-B bus and dual corded downstream equipment. Rack Power Distribution Unit (RPDU ) An RPDU should not be confused with a floor-standing PDU. Instead of using circuit breakers the RPDU consists of multiple receptacles into which the power cords of the ITE can plug. It is packaged in a standard

32 PAGE 32 ICT equipment rack form factor so that it can be located either in the same rack/cabinet or in-row close to the IT equipment racks that it is powering. RPDU s are typically cord-connected and available in a variety of form factors. Some are rack-mounted and take up 1-4U of shelf space. Others mount outside of the U area within a rack or cabinet (for example, mounted vertically in the rear of the rack or cabinet). RPDUs are also known by such names as rack power module, rack PDU, zero-u PDU, enclosure PDU, and cabinet distribution unit. Because it is the final connection point to the ITE, an RPDU can provide the finest level of power consumption granularity in the data center (except for what can be derived for the individual ITE itself). Similar to the larger PDU, the device can be highly configurable and may include a dual feed option, remote switching, rack-level power monitoring, or even individual cord circuit monitoring.. Efficiency Topics PDUs that contain transformers should follow the requirements laid out in the Transformers section of the document.

33 PAGE 33 Measurement and Control diagram Inputs: Outputs: Phase A (V, I, kw, kwh) Output 1 (V, I, W) Phase B (V, I, kw, kwh) Phase C (V, I, kw, kwh) AC Input 1 AC Outputs Output 2 (V, I, W) Output 3 (V, I, W) Phase A (V, I, kw, kwh) : : Circuit N (V, I, W) Phase B (V, I, kw, kwh) Phase C (V, I, kw, kwh) AC Input 2 (optional) Source 1 available (T/F) PDU Source 2 available (T/F) Output 1 enable (T/F) : : Output N enable (T/F) AC Input Select Override Control Inputs Status Output 1 enabled (T/F) : : Output N enabled (T/F) Figure III-7: PDU Measurement and control points

34 PAGE 34 Core Tenets Timetable Instrumentation For PDUs that contain transformers, instrumentation similar to that in the transformer section should be implemented. For PDUs that do not contain transformers, the inputs and outputs can be reduced to the power connections on the input and output locations; temperature, pressure and liquid level are no longer required. 2012: Attribute Input Output Temperature Pressure or vacuum Liquid Level Notes Ingress power connection to the transformer from an external primaryside source (can include a generator source) Egress power connection from the transformer to an external secondary-side load Egress transformer system temperature reading Egress transformer system pressure reading Egress transformer system liquid level status Table III-11: 2012 Transformer Connection Point Type Values

35 PAGE : (continued) Attribute Definition Units Input RMS Voltage Volts RMS Current Amperes Real Power Kilowatts Energy = Real Power Kilowatt-hrs * Time Interval Frequency Hz Output RMS Voltage Volts RMS Current Amperes Real Power Kilowatts Real Power * Time Kilowatt-hrs Interval Frequency Hz System Temperature Degrees C Pressure PSI Liquid Level OK Logic High Table III-12: 2012 Transformer Connection Point Metered Values Typically, only input or output power is measured. High accuracy measurements at utility input should be more accurate than compute room level. Attribute Definition Units Notes System Liquid Level OK Logic High Table III-13: 2012 Transformer Connection Point Status Values 2014:

36 PAGE 36 Measure both input and output power to calculate efficiency. Efficiency measurements can not only monitor for savings, but can also monitor for degradation and harmonic currents. Measurement does not need to be done in real time, but should be measured periodically, either automatically or manually : None. Discoverability 2012: Some models of all types of PDUs are discoverable on a network. This feature is typically available at an extra cost. 2014: All PDUs, regardless of their type or what they contain, should be discoverable Attribute Definition Units Notes Location Physical location Datacenter Security Sensitivity Floor # Security Sensitivity Room # Security Sensitivity Table III-14: PDU discoverability attributes 2014+: None Scalability 2012: None 2014: None 2014+: None

37 PAGE 37 Enhanced management Infrastructure 2012: DUs with transformers currently require external hardware to report into a management infrastructure. This hardware could be specified (but typically is not) to report into an IT monitoring system. 2014: None 2014+: None Ability to be policy driven 2012: Some models of rack power distribution equipment have the ability to be monitored, capped, or otherwise controlled. 2014: None 2014+: All models of PDUs should have the ability to be monitored and controlled at the output. Standardized metrics For PDUs with transformers, follow the information listed in the transformer section. Applicable Standards For efficiency: NEMA TP1

38 PAGE 38 Newer power distribution technologies under consideration Most data centers today use alternating current (AC) power systems, which distribute electricity somewhere between 100Vac and 600Vac throughout the facilities. However, a growing number of direct current (DC) advocates are promoting the use of DC power in the data center. Higher voltage direct current (HVDC) power distribution configurations (<600Vdc) have the potential to reduce energy consumption and increase efficiency in the data center. Higher voltage direct current power systems may involve fewer components, which can result in higher reliability and lower total cost of ownership when compared to AC power systems. Efficiency Efficiency was studied in The Green Grid via White Paper #16 Quantitative Analysis of Power Distribution Configurations for Data Centers 6. The more detail on the following information can be found in that white paper. A typical North American AC system drops the incoming AC voltage down through a series of conversions to 480Vac. At that point a double conversion uninterruptible power supply (UPS) converts AC to DC for battery charging, typically resulting in losses of 3% or greater. An inverter then converts the voltage back to AC, typically adding 1% to 3% losses for this second conversion. A power distribution unit (PDU) - which, if it has a transformer, is a third conversion, typically with 1% to 2% losses - distributes AC voltage to power supply units (PSUs) in each of the various IT equipment loads. The PSU finally converts the voltage to 12Vdc, currently with 6% to 10% losses. A typical HVDC system first uses a rectifier to convert the incoming 480 VAC to 380VDC. Then a PSU converts it directly to 12Vdc. Eliminating the extra conversions means that HVDC power distribution configurations generally need no inverters and fewer step-down converters or intermediate voltages. Reliability With fewer individual components, HVDC systems are less complex and likely to be more reliable than AC systems. In addition to conversion-related hardware, HVDC removes other elements, such as the sensing 6

39 PAGE 39 mechanisms that AC systems must use to ensure that equipment is in synchronization. Paralleling of sources in a DC system is simplified because synchronization is unnecessary. The reduction of complexity can also save space. Cost savings can also be realized due to fewer components in the UPS (and possibly in the PSU), reduced cable losses, and a reduced cable cross-sectional area, which requires less copper. All that said, AC represents far more of a known quantity. Additionally, other alternative AC solutions in development increase end to end efficiency. HVDC is a relatively new concept and, while well-understood, there are not nearly as many scaled-up, production-level examples of HVDC power distribution configurations in operation. However, many groups worldwide have launched HVDC demonstration sites. In Japan, verification tests were performed on an HVDC system in 2009 with the cooperation of domestic and international IT equipment manufacturers. The test results showed the HVDC system, when compared to the unmodified existing AC system, had higher availability and end to end efficiency. Please see The Green Grid s white paper #31 Issues relating to the adoption of higher voltage direct current power in the data center for further information and issues relating to the adoption of HVDC power system in the data center.

40 PAGE 40 Appendix Transformers An isolation transformer (Iso Xfmr) is an electromagnetic device with multiple windings per phase that converts an AC voltage into another AC voltage and simultaneously provides galvanic isolation by transmitting energy between windings through magnetic inductive coupling (induction). The transformer winding size and type is selected to provide the required voltage transformation level as well to provide secondary benefits including harmonic cancellation through phase-shifting and transient voltage reduction through grounding. Optional Transformer Optional Transformer Main Protective Device Main Protective Device Tie To IT Loads To IT Loads Figure III-8: One-line representation of Secondary Selective System Using Two Winding Transformer Transformer windings are available in a variety of connection types including zig-zig, wye, delta, and T (Scott). Each of the windings on an isolation transformer can be different connection types. For example, a commonly available transformer has a delta-connected input (called the primary winding) and a wye-connected output (secondary winding).

41 PAGE 41 Figure III-9: Delta/Wye 3-Phase Transformer (ANSI transformer symbols shown) Referring to Figure III-9: Delta/Wye 3-Phase Transformer (ANSI transformer symbols shown), notice that in addition to the three connections to the right side winding (secondary winding), that wye-connected winding has a fourth connection. The fourth connection, called the neutral, is typically grounded. A neutral connection provides additional transformation ratios since loads can be connected either phase-to-phase (e.g. A-B, B-C, etc.) or phase-to-neutral (e.g. A-N, B-N, etc.). When loads are connected to the transformer secondary phase to phase, again referring to Figure III-9: Delta/Wye 3-Phase Transformer (ANSI transformer symbols shown), the ratio of the secondary voltage to the primary voltage is defined by the ratio of the number of turns of wire in each winding, as follows: where: V V AB N N 1 Equation III-1 AB ' ' 2 VAB VA B N1/N2 is primary phase-to-phase voltage is secondary phase-to-phase voltage is turns ratio (ratio of number of turns on primary winding divided by the number of turns on the secondary winding When those same loads are connected phase-to-neutral, the voltage supplied will be equal to the phase-tophase voltage magnitude divided by the square root of 3. V AB ' ' VAN ' Equation III-2 3