Building an energy dashboard. Energy measurement and visualization in current HPC systems

Size: px

Start display at page:

Download "Building an energy dashboard. Energy measurement and visualization in current HPC systems"

Dina Osborne
8 years ago
Views:

1 Building an energy dashboard Energy measurement and visualization in current HPC systems Thomas Geenen 1/58

2 SURFsara The Dutch national HPC center 2H 2014 > 1PFlop GPGPU accelerators Grid HPC Cloud Hadoop Data Services 2/58

3 SURFsara 3/58

4 SURFsara 4/58

5 Energy consumption 5/58 Dongarra et al. Energy Footprint of Advanced Dense Numerical Linear Algebra using tile algorithms on multicore architecture PowerPack: Intel Xeon Sandy Bridge

6 Energy consumption 6/58 Dongarra et al. Energy Footprint of Advanced Dense Numerical Linear Algebra using tile algorithms on multicore architecture PowerPack: Intel Xeon Sandy Bridge

7 Energy consumption 7/58 Dongarra et al. Energy Footprint of Advanced Dense Numerical Linear Algebra using tile algorithms on multicore architecture PowerPack: Intel Xeon Sandy Bridge

8 Energy consumption 8/58 Dongarra et al. Energy Footprint of Advanced Dense Numerical Linear Algebra using tile algorithms on multicore architecture PowerPack: Intel Xeon Sandy Bridge

9 Energy consumption 9/58 Dongarra et al. Energy Footprint of Advanced Dense Numerical Linear Algebra using tile algorithms on multicore architecture PowerPack: Intel Xeon Sandy Bridge

10 Energy measurement Different sources Accuracy Sampling rate Overhead Processing measurements Postprocessing Visualization Interpretation 10/58

11 Node sensors 11/58 Running average power limit (RAPL) baseboard management controller (BMC), Intelligent Platform Management Interface (IPMI)

12 Node sensors Different sources Direct from the CPU Running average power limit (RAPL) Performance Application Programming Interface (PAPI) From component baseboard management controller (BMC) Intel node manager Intelligent Platform Management Interface (IPMI) 14/58 Running average power limit (RAPL) baseboard management controller (BMC), Intelligent Platform Management Interface (IPMI)

node manager Intelligent Platform Management Interface (IPMI) 14/58 thomas.geenen@surfsara.

13 RAPL RAPL is not an analog power meter! RAPL uses a software power model running on a helper controller Energy is estimated using hardware performance counters temperature, leakage models and I/O models The model is used for CPU throttling, turbo-boost Values are exposed to users model-specific register (MSR) 15/58 thomas.geenen@surfsara.nl Running average power limit (RAPL) baseboard management controller (BMC), Intelligent Platform Management Interface (IPMI)

counters temperature, leakage models and I/O models The model is used for CPU throttling, turbo-boost Values are

14 RAPL Intel Documentation indicates Energy readings are Updated roughly every millisecond (1 KHz) Rotem et al. show results match actual hardware * 16/58 thomas.geenen@surfsara.nl Rothem: Power-Management Architecture of the Intel Microarchitecture Code-Named Sandy Bridge, IEEE micro, 2012

show results match actual hardware * 16/58 thomas.geenen@surfsara.

15 RAPL More detailed study shows small deviations for different loads 17/58 Hackenberg et al.: Power measurement techniques on standard compute nodes: a quantitative approach, IEEE, 2013

16 PAPI performance application programming interface (PAPI) 18/58 MSRs can be accessed via /dev/cpu/*/msr

17 PAPI Performance application programming interface (PAPI) Read special registers (MSR) Performance counter hardware Intel, AMD, NVIDIA, ARM RAPL, APM, NVML, custom Measure energy and Flops, cycles Memory access, cache misses Ivy bridge 11 counters 19/58 MSRs can be accessed via /dev/cpu/*/msr

NVML, custom Measure energy and Flops, cycles Memory access, cache misses Ivy

18 Profiling applications Time Where is the time spend What is the application doing PAPI (hardware calls) MPI (communication between processes) OpenMP (communication between threads) Couple with energy consumption Same profile 20/58

between processes) OpenMP (communication between threads)

19 Profiling applications 21/58

20 Profiling applications 22/58

21 Profiling applications 23/58

22 Profiling applications 24/58

23 25/58

24 IPMI BMC 26/58

25 IPMI BMC 27/58

26 IPMI BMC 28/58

27 IPMI BMC Measure energy consumption of other components Baseboard Management Controller (BMC) IPMI Low sample rate 1 4 Hz Overhead Improves On chip averaging Higher sample rate Still low 29/58 thomas.geenen@surfsara.nl Baseboard management controller (BMC) Intelligent Platform Management Interface (IPMI)

28 Reporting What do we want to present to the end user Can use PAPI and tools for detailed analysis Misses part of the energy consumption Information on per-run level Energy consumption per run (total) More general view (total per component) Timeline Correlate with other data PAPI and BMC 30/58

29 SLURM Use the job scheduler to collect energy consumption data Typical situation on HPC systems Many users on the same system Share resources Have to schedule jobs Job is put in a queue Runs when resources are available SLURM Simple Linux Utility for Resource Management Open source 31/58 thomas.geenen@surfsara.nl

30 SLURM Use the job scheduler to collect energy consumption data Modular design Plugins for monitoring Energy consumption RAPL IPMI Grand total Timeline Uses additional threads to collect data (IPMI) 32/58

31 SLURM Use the job scheduler to collect energy consumption data 33/58

32 SLURM Use the job scheduler to collect energy consumption data Totals in database Timeseries in file HDF5 (XML) Scalable data format Individual sensors (IPMI) RAPL External sensor 34/58

33 Conclusions Many sensors available on current cluster hardware Different levels of detail Many profilers available Common api PAPI Combine with performance metrics Present totals to users Combine different measurements in one file (time series) Slurm tools 35/58

34 QUESTIONS? 36/58

Part I Courses Syllabus

Part I Courses Syllabus This document provides detailed information about the basic courses of the MHPC first part activities. The list of courses is the following 1.1 Scientific Programming Environment