Real-Time Energy Consumption Measurements in Data Centers. ASHRAE TC9.9 and The Green Grid

Transcription

1 Real-Time Energy Consumption Measurements in Data Centers ASHRAE TC9.9 and The Green Grid

2 First printing released at the ASHRAE Winter Annual Meeting, Jan 23 27, 2010 ASHRAE/TGG Real Time Energy Consumption Measurements in Data Centers. Atlanta: American Society of Heating, Refrigerating and Air Conditioning Engineers, Inc.; coauthored by The Green Grid, Beaverton, Oregon. 2

3 PURPOSE OF THE BOOK Gartner, IDC, Forrester and other analysts continue to rank Power & Cooling (energy efficiency) as the #1 concern of data center owners/operators TGG s PUE is becoming the proxy for data center energy efficiency The PUE cannot be reasonably determined if energy consumption in the data center cannot be measured EPA reports an average PUE of 2.04 (2009 data) The book is designed to Provide an overview of the state of energy consumption measurements in the data center Educate the data center owner/operator with respect to making real-time energy consumption measurements in the data center Demonstrate how to consolidate the energy consumption data into a single energy efficiency value, e.g., PUE Provide guidance with respect to determining energy efficiency in mixeduse facilities 3

4 TABLE OF CONTENTS 4 PART 1 BASICS Chapter 1 Introduction Chapter 2 How, What, & Where to Measure Chapter 3 Measurement Devices Chapter 4 Measurement Collection Systems Architecture and Software PART 2 COOLING SYSTEMS Air Measurements Chapter 5 Air Handler Units Chapter 6 Computer Room Units PART 3 COOLING SYSTEMS Hydronic Measurements Chapter 7 Pumps Chapter 8 Cooling Towers Chapter 9 Chillers Chapter 10 Heat Exchangers

5 TABLE OF CONTENTS PART 4 POWER SYSTEMS MEASUREMENTS Chapter 11 Introduction to Critical Power Distribution Chapter 12 Upstream Critical Power Distribution Chapter 13 Uninterruptible Power Supply Chapter 14 Computer Room Transformer and Power Distribution Unit PART 5 IT SYSTEMS MEASUREMENTS Chapter 15 Compute and Storage Systems Chapter 16 Networking Systems Appendix A Pumps Appendix B Chillers Appendix C Mixed-Use Facilities Appendix D Uninterruptible Power Supply Appendix E Onsite Power Generation and CCHP in Data Center Applications 5

6 KEY CONTRIBUTORS 6

7 KEY CONTRIBUTORS 7

8 PART 1 BASICS 8

9 MEASUREMENT DEVICES (Chapter 3) John Bean (APC), presenter 9

10 Part 1 Chapter 3 Sensors 3.1 Overview There are numerous attributes within the data center that require measurements Temperature Pressure Flow Rates Voltage Current Humidity Sensors come in a variety of connectivity styles: direct, analog (0 10vdc / 4 20ma) and serial communication (ModBus, BacNet, etc ) Introduce Terms: Span, Resolution, Accuracy 10

11 Part 1 Chapter 3 Sensors 3.2 Accuracy What may appear to be small inaccuracies frequently may be compounded to give very misleading data Example: Chiller with design temperature difference of 10 F Return chilled water sensor reading 1 F low Supply chilled water sensor reading 1 F high Combined error of 2 F Chiller reported capacity error of 20% 11

12 Part 1 Chapter 3 Sensors 3.3 Temperature There are several types of temperature sensors available Thermocouple dissimilar metals that produce a voltage the corresponds to junction temperature. Accuracy ± 1 to 5%. Care must be used when using hookup wire to avoid new dissimilar junctions with voltage potentials that could bias reading accuracy Thermistors semiconductor devices whose resistance varies in response to temperature change, typically are constructed of polymer or ceramic. Accuracy ± 0.1 to 2% RTD Resistance Temperature Detector, typically are wire wound or thin film devices whose resistance changes in response to an increase in temperature. Accuracy ± 0.01 to 1%. Are available in three and four wire version to compensate for lead wire resistance error All types may use a local transmitter to convert weak signal to more robust signal (0 10vdc or 4 20ma) to allow longer runs of wiring to measurement device 12

13 Part 1 Chapter 3 Sensors 3.4 Pressure There are several types of pressure sensors available Bourdon Tube Strain Gauge Compound (vacuum and positive pressure) Absolute Pressure Gauge Pressure Need to ensure wetted surface material is compatible with fluid to be measured All types may use a local transmitter to convert weak signal to more robust signal (0 10vdc or 4 20ma) to allow longer runs of wiring to measurement device 13

14 Part 1 Chapter 3 Sensors 3.5 Flow Liquid There are several types of liquid flow sensors available Paddle Wheel Turbine Wheel Shedding Vortices Venturi Ultrasonic Coriolis Variable Area Need to assure wetted surface material is compatible with fluid to be measured All types may use a local transmitter to convert weak signal to more robust signal (0 10vdc or 4 20ma) to allow longer runs of wiring to measurement device 14

15 Part 1 Chapter 3 Sensors 3.6 Flow Gas There are several types of gas flow sensors available Pitot Tube Hot Wire Anemometer Flow Nozzle Cup Anemometer All types may use a local transmitter to convert weak signal to more robust signal (0 10vdc or 4 20ma) to allow longer runs of wiring to measurement device 15

16 Part 1 Chapter 3 Sensors 3.7 Current There are several types of current sensors available Split Core Solid Core Resistor (voltage drop) Clamp-On Split Core Extra loops of conductor thru core act as current multiplier for current transformers All types may use a local transmitter to convert weak signal to more robust signal (0 10vdc or 4 20ma) to allow longer runs of wiring to measurement device 16

17 Part 1 Chapter 3 Sensors 3.8 Voltage There are several types of voltage sensors available Potential Transformer (high & medium voltage) Voltage Divider Voltage Transducer 17

18 Part 1 Chapter 3 Sensors 3.8 Power Monitor Typically use both current and voltage sensor and have necessary processing means to determine power in real time. Other common metrics from power meters may include KVA (apparent power) KVAR (reactive power) Power Factor Individual Leg Currents Individual Leg Voltages May or may not be revenue grade Typically may have local display along with serial communication interface to BMS or SCADA System 18

19 MEASUREMENT COLLECTION SYSTEMS Architecture and Software (Chapter 4) Kenneth Uhlman, Ph.D, P.E., presenter 19

20 Topics Business Questions Measurement Levels Scalable Hardware / Software Architecture 20

21 Business Questions Does this data metering and monitoring strategy support the organization s sustainability objectives? Will this project support current and future regulatory requirements domestically and abroad? Is there executive sponsorship to fund and support the project? How will this project be funded? What is the ROI and risks? Who currently pays the energy bills (IT or Facilities)? Is there a plan to charge back energy (and carbon) to the end users? Technology is not the limiting factor, but rather organizational structure and behavior 21

22 Measurement Levels Minimum Best State-of Item Practical Practical the-art Linked IT / Facility Business Goals Metrics Measure and log metrics Online data acquisition? Monitoring Separate views (IT & Facilities) Linked IT/Facility dashboards Linked Business / IT / Facility dashboards 22 Proactive Strategies Server managing servers Auto-discovery of IT / Facility devices Proactive IT/Facility Proactive IT/Facility/Smart Grid

23 Typical Data Center Meters Many data centers have plenty of meters for each sub-system Typically installed for capacity or component health, not enterprise efficiency FACILITY HEALTH CAPACITY EFFICIENCY May be able to aggregate and recast data to measure and log efficiency Solutions at various price points to meet business objectives M M M M M M M M M M M M M M M M M M M M M M M M M Data Center Meters M IT M Facilities M Utilities 23

24 Enterprise Energy Dashboard While a single-pane of glass may be an ultimate goal, there are many tactical activities for improvement The cost for enterprise views continues to decrease with the proliferation of networks and web-based equipment and gateways Initiate program with existing metering data where available 24

25 Sensing Hierarchy IT and Facilities equipment have different constraints bu are implemented similarly The application dictates realtime monitoring acquisition rates from seconds, minutes days, weeks or months - IT Applications -Networks - Power or cooling systems 25

26 Collecting Data Center Telemetry in an Enterprise Sites may be internal or external, domestic or international Network security is crucial Typically many vendors and vintages of Power, Building Automation, HVAC, Security, IT, etc. Local and central storage of data and acquisition rates 26

27 Managed server monitored by a manager server Proactive management of servers enables IT to match optimal energy rates, facility capacities and carbon footprint Virtualization enables IT loads to be shifted globally Facility systems must be linked to process 27

28 Data Collector Software Architecture Is the heart of a linked system Secure and customized screens for: - Executives -IT - Facilities Proactively optimize IT, Facilities and Utilities to meet the organization s business objectives 28

29 PART 2 COOLING SYSTEMS Air Measurements 29

30 AIR HANDLERS (Chapter 5) COMPUTER ROOM UNITS (Chapter 6) Daryn Cline (Evapco), presenter 30

31 Part 2 Cooling Systems Air Measurements Chapter 5 Air Handlers 5.1 Overview 5.2 Measurement Levels Chapter 6 Computer Room Units 6.1 Overview 6.2 Measurement Levels 31

32 Part 2 Cooling Systems- Air Measurements Chapter 5 Air Handlers 5.1 Overview 32

33 Part 2 Cooling Systems Air Measurements Chapter 5 Air Handlers 5.2 Measurement Levels 33

34 Part 2 Cooling Systems Air Measurements Chapter 6 Computer Room Units 6.1 Overview CRAC Computer Room Air Conditioners Direct Expansion (DX) Chilled Water Dual Source -Chilled Water Primary, DX Backup Downflow -raised floor Upflow -free discharge or ducted Ceiling units Environmental Control Humidity Control RH, Dew point Sensible Heat 34

35 Chapter 6 Computer Room Units M =Measuring Points for Power Consumption 35

36 Chapter 6 Computer Room Units 36

37 PART 3 COOLING SYSTEMS Hydronic Measurements 37

38 PUMPS (Chapter 7) COOLING TOWERS (Chapter 8) CHILLERS (Chapter 9) HEAT EXCHANGERS (Chapter 10) Daryn Cline (Evapco), presenter 38

39 Part 3 Cooling Systems Hydronic Measurements Chapter 7 Pumps 7.1 Overview 7.2 Measurement Levels-Electrical 7.3 Measurement Levels-Fluid Chapter 8 Cooling Towers 8.1 Overview 8.2 Measurement Levels Chapter 9 Chillers 9.1 Overview 9.2 Measurement Levels Chapter 10 Heat Exchangers 10.1 Overview 10.2 Measurement Levels 39

40 Part 3 Cooling Systems Hydronic Measurements Chapter 7 Pumps 7.1 Overview 40

41 Part 3 Cooling Systems Hydronic Measurements Chapter 7 Pumps 7.2 Measurement Levels-Electrical 41

42 Part 3 Cooling Systems Hydronic Measurements Chapter 7 Pumps 7.2 Measurement Levels-Electrical 42

43 Part 3 Cooling Systems Hydronic Measurements Chapter 7 Pumps 7.3 Measurement Levels-Fluid 43

44 Part 3 Cooling Systems Hydronic Measurements Chapter 8 Cooling Towers 8.1 Overview-Open Counter and Cross Flow Cooling Towers 8.1 Overview-Closed Counter and Combined Flow Circuit Cooling Towers 44

45 Part 3 Cooling Systems Hydronic Measurements Chapter 8 Cooling Towers 8.2 Measurement Levels 45

46 Part 3 Cooling Systems Hydronic Measurements Chapter 8 Cooling Towers 8.2 Measurement Levels Minimum Practical Measurement-True RMS Power Best Practical Measurement-Current Transducers Best Practical Measurement-State-of-the-Art 46

47 Part 3 Cooling Systems Hydronic Measurements Chapter 9 Chillers 9.1 Overview 47

48 Part 3 Cooling Systems Hydronic Measurements 9.2 Measurement Levels 48

49 PART 4 POWER SYSTEMS MEASUREMENTS 49

50 INTRODUCTION TO CRITICAL POWER DISTRIBUTION (Chapter 11) UPSTREAM CRITICAL POWER DISTRIBUTION (Chapter 12) UNINTERRUPTIBLE POWER SUPPLY (Chapter 13) COMPUTER ROOM TRANSFORMER & POWER DISTRIBUTION UNIT (Chapter 14) Steve McCluer (Schneider/APC), presenter 50

51 Chapter 11 - Critical Power Distribution Critical Power vs. Essential Power Focus is on the Critical Power Path Can be several meandering paths Supports those loads with zero tolerance for unplanned downtime Mostly (but not entirely) in the IT Equipment Room Essential Power Path can tolerate a brief disruption Highly precise meters are typically required for power quality and forensic analysis, overkill for routine efficiency studies. Part 4 describes a data center rated somewhere between 200 kw and 5000 kw 51

52 Generator Utility Switchgear & Transformers ATS Distribution Switchgear UPS Mechanical Switchboard Distribution Switchgear Distribution Switchgear PDU RPDU 52 IT Loads Mechanical Loads Critical Path Essential Path Non-Critical Loads Non-Data Center Loads Other Loads Path

53 Ch 12 Upstream Critical Power Distribution Service Entrance Metering Typically highly precise revenue grade Typically includes kw & kvar May be difficult to separate IT loads Automatic Transfer Switch (ATS) Typically less precise May capture variances between sources Primary Electrical Distribution Switchgear May or may not be available for each circuit Ideal place to capture power into IT equipment room 53

54 54

55 Chapter 13 Uninterruptible Power Supply (UPS) Centralized UPS Measured outside of IT Equipment Room Output of UPS may be ideal place to capture IT room critical power consumption Distributed UPS Measured close to the IT loads in the IT Room May be multiple UPSs May be highest power consumption in the critical power path Possible to capture power going to zones or rows Metering can be at power module &/or at the system level UPS room Distributed UPS 55

56 UPS Metering 56

57 CHAPTER 14 COMPUTER ROOM TRANSFORMER & POWER DISTRIBUTION UNIT (PDU) Transformers can appear in many places in the power path Efficiency lost at every conversion Typically no metering i.e., measurements taken by hand-held instruments PDUs With transformer, typically meter output May be best place to aggregate actual IT equipment energy consumption Without transformer, metering may be optional 57

58 58

59 PDU Figure 14.3 Power Flow Through a Power Distribution Unit 59

60 Rack-mounted PDU (RPDU) Measurements taken inside the equipment rack or cabinet Generally impractical for aggregating equipment power consumption Good for equipment measurement May be multiple units in one rack powering the same equipment Typically have lower accuracy than upstream devices Many levels of metering options 60

61 Appendix D - UPS Overview of UPS types Redundancy and availability Challenges for power measurement on multiple paths Rules of thumb for minimum practical level of UPS instrumentation Example for quick estimation of PUE / DCIE inside the computer room Sample case study: PUE and DCIE determination for the critical power path within the data center (Partial PUE) 61

62 PART 5 IT SYSTEMS MEASUREMENTS

63 COMPUTER AND STORAGE SYSTEMS (Chapter 15) NETWORKING SYSTEMS (Chapter 16) Mike Patterson, Ph.D., P.E. (Intel), presenter

64 Motivation IT equipment power is the denominator in the PUE Calculation Overview Workload Characteristics Issues Measurements Minimum Measurement Best Practical State of the Art 64

65 Power thru the IT gear (Server, Stor, & Comms) Room Rack rack power strip or PDU a PSU CPU, etc.. PDU Server b Measurement location (a) can give aggregate power (b) best location if actual data is avaiable 65

66 Process for Measurement IT Equipment Inventory Age & self reporting capability Location; remote storage, etc Data Center Workloads Power draw can have unexpected patterns; power vs. energy considerations Enterprise, HPC, Financial, Internet, , etc Time of day, weekly variations, batch processes Server to server variation due to workload 66

67 Issues and challenges Different organizations, different systems Different protocols IT Facility IPMI -BACnet DCMI -Modbus ASH -other Where is it brought together? Management console, spreadsheet, or back of an envelope? Redundant Power Supplies Must measure all feeds and attribute accordingly Power over Ethernet (POE) 67

68 68 Chapter 16 Network; essentially the same, see the book for details

69 Minimum Practical Measurements Equipment nameplate Better than doing nothing, but just barely See Green Grid White Paper #23, Proper Sizing of IT Power and Cooling Loads PDU Output Can give actual rack-level power (sometimes energy) Not always tracked on-line Single server/network/storage measured value Actual portable power meter Workload, timeframe, scalability questions 69

70 Best Practical Measurement Rack or row level PDU output Rack power aggregate No visibility to individual servers or workloads Easily tracked/trended in BMS system Majority of PDUs have this capability Ignores line losses This is a reasonable assumption 70

71 State-of-the Art Measurement On-line server power consumption New Energy Star for Servers requires this! Challenge is integrating server manageability protocols with BMS systems IPMI, DCMI, ASH, Vendor-specific Extensive server data available CPU utilization Front panel temperature sensors On-line network and storage power consumption Somewhat behind Server; Energy Star on the horizon Capabilities vary on self reporting Wireless remote sensors Wide range of data capture 71

72 Thank you for attending 2010 For more information, visit 72