Performance Tools for System Monitoring

Transcription

1 Center for Information Services and High Performance Computing (ZIH) Dresden Performance Tools for System Monitoring 1st CHANGES Workshop, Jülich Zellescher Weg 12 Tel September 5th, 2012 Wolfgang E. Nagel Dresden University of Technology! Founded in 1828! one of the oldest technical universities in Germany! 14 faculties and a number of specialized institutes! More than Students, about Employees, 500 professors! International courses of studies, bachelor, masters! One of the largest computer science faculties in Germany! 200 million Euro annual third party funding! One of the eleven Excellence Universities in Germany! 1

2 Current HPC Activities at TU Dresden! Dresden is just preparing the next HPC procurement (15 Mio. EUR in 2013/14, additional machine room ( about 5 MW), research focus on scalability, data intensive computing, energy awareness)! Work on the performance analysis environment for the next Oak Ridge system (Titan, based on Nvidia Kepler) Scalability (OS, information gathering, displays, ) Heterogeneity (Cluster, Multicore, GPUs) Integration of new functionalities/views I/O support (mapping local and global I/O requests) Energy measurement and mapping support! MPI correctness tools (MARMOT/MUST), application scalability (COSMO/SPECS code, FD4 library)! Running the Dresden CUDA Center of Excellence! Member of the IESP/EESI groups, co-coordinator of the German SPPEXA (DFG priority program), to prepare the software research roadmap for Exascale Scientific Application Areas Electrical Engineering 10% 26% Physics 41% Life Science 9% Chemistry 8% Material Science 3% Mechanical Engineering 3% Other PC Farm HPC Server 13% 34% Material Science Physics 42% Mechanical Engineering 6% Life Science Geology 2% 3% Other 2

3 HPC at ZIH (Procurement Ongoing) ZIH-Infrastruktur! Scalability! Data intensive computing Verbrauch Steuerung Optimierung Energieeffizienz- Analysen Login Batch-System Durchsatzkomponente HPC- Komponente HPC Block 1 Block 1 Block N Zugriffe E/A-Analysen Optimierung Steuerung Flexibles Agiles Speicher System HRSK-II Konzept! Energy efficiency Stromversorgung Challenges! HPC systems evolve; Tools need to adapt: MPI+ Accelerator+ Threads Heterogeneous/ Hybrid MPI+ PGAS? MPI+CUDA PGAS+CUDA MPI+Threads Paradigms #Cores Co-Array Fortran SHMEM TOP 500 Largest Machine 2001 Scalability Year Challenges 2011 Usability Fault Tolerance Towards Exascale Make all this accessible! 3

4 Scalability Dresden Zellescher Weg 12 Tel Wolfgang E. Nagel Meeting the Challenges Collaboration with ORNL! Goal: - Support hybrid system and software architecture at 10 Petascale - Show MPI and GPGPU programming - Do full system performance profiling and tracing! Facts: - Jaguar / Titan - > cores - 200,448 monitored MPI processes - >20 Tera-bytes of performance data - 21,515 VampirServer processes 4

5 Meeting the Challenges Collaboration with ORNL Timeline Meeting the Challenges Collaboration with ORNL 5

6 Meeting the Challenges Collaboration with ORNL Zoom into Details Load Imbalance in COSMO-SPECS Load imbalance of spectral micro-physics (green) leads to long waiting time (red) 6

7 Scalability of COSMO-SPECS Energy Dresden Zellescher Weg 12 Tel Wolfgang E. Nagel 7

8 Power Consumption Monitoring ZIH-Infrastruktur! High Precision! High Frequency Login HPC EnergieeffizienzAnalysen E/A-Analysen Durchsatzkomponente Flexibles Agiles Speicher System Verbrauch Block 1 Block 1 Block N! From complete system down to single s Optimierung Steuerung Zugriffe HPCKomponente Optimierung Steuerung Batch-System HRSK-II Konzept Stromversorgung Energy measurement Dataheap Server Test system TCP/IP Buffer DataheapPlugin TCP/IP Measurement environment Monitoring Server Application Power supply Power meter - Multiple sources - Combination of sources - No influence on test system - ZES LMG - NI PCI-6255 RS 232 8

9 Combined Performance und Energy Analysis Package energy consumption 12 V DC lane AC power supply cycles in C-State 6 High Frequency DC-Sampling 1 ms 0.1 ms! Measurements on Intel Sandy Bridge, 4 core, 12V! National Instruments PCI-6255 measurement card (16 Bit, 750 ks/s sampling, 80 analog inputs) 9

10 Energy Accounting! Atlas System: - 90 s with 4x AMD Interlagos - QDR-IB Interconnect! Megware ClustSaf - 1 Hz Sampling of all individual s - Integrated in ZIH software stack - Allows tracing and energy accounting! Sample Output: Run finished: Fri Apr 13 18:24: total runtime: s total energy : kj average power: W" I/O Analysis Dresden Zellescher Weg 12 Tel Wolfgang E. Nagel 10

11 File System Concept: FASS ZIH-Infrastruktur Login Batch-System HPC-Komponente Nutzer A Nutzer Z Nutzer A Nutzer Z Durchsatzkomponente Flexibles Agiles Speicher System Export Server/Dateisysteme Netz Storage SCRATCH Transaktion Server 1 Checkpoint. Server 2 Seriell Server N Switch 2 Switch 1 Optimierung und Steuerung SSD SAS SATA Zugriffs- Statistiken Zugriffs- Analyse Vampir I/O Analysis! recording of POSIX I/O and MPI I/O operations with VampirTrace! embedding of local data like InfiniBand statistics! on demand inclusion of external performance data from: I/O network storage controller file servers! within Vampir: counter timelines for the host based and the external data specialized I/O display to show: details for single I/O events grouping of events for the current portion of the timeline based on the filename, the type of the I/O record (read, write, ) and some more I/O request size statistics 11

12 Vampir: Combined Application + Lustre Server Analysis metadata rate metadata server load metadata server load detailed per file and summary statistics detailed per event data Target: Exploitation of all data sources relevant to I/O I/O network MDS OSS OSS SAN RAID RAID RAID utilization errors OSS utilization errors application events local events trace backend, cache throughput, IOPS 12

13 DataHeap: Application + Infrastructure Events application I/O + network VampirTrace Trace I/O network agent MDS OSS OSS OSS MDS OSS agent OSS OSS SAN agent SAN RAID RAID agent RAID RAID RAID RAID performance data database secondary counters Starting Point: 6 from 576 IOR processes 13

14 IOR: write phase IOR: Write Phase OVERHANG 650 vs. 800 MB/s per DDN port 14

15 IOR: Read Phase DIP Bad Disk Sector Outlook Dresden Zellescher Weg 12 Tel Wolfgang E. Nagel 15

16 Projects CRESTA eeclust H4H Programming Methods and Techniques HOPSA TIMaCS VI-HPS SILC ORNL-OLCF Software Tools CoolComputing SIOX Architecture and Performance Analysis HI-CFD SFB 609 HPC-FLiS SFB HAEC GASPI LMAC WisNetGrid MoSGrid Radieschen 100GBit/s ER-flow Mathematical Methods and Algorithms NGSgoesHPC SFB TR 79 SpaceSys Algorithms and Methods for biological Processes Virtuelle Leber GrowReg Grid-Computing/Cloud Computing LSDMA EMI DGSI GeneCloud New Machine Room: Architectural View Dr. Peter Fischer 16

17 CS Department and New Machine Room Planned Schedule Start: January 2013 Availability for HRSK Phase II: October 1st, 2014 Dr. Peter Fischer New Machine Room Infrastructure! PUE < 1.3! First building in university research mile! HPC room: 450 m^2 (12m x 36m, 140+ racks) 2.5 MVA Cold water: 20 C Warm water: C 250 kw air cooling capacity 17

18 18