How To Monitor Performance On A Microsoft Powerbook (Powerbook) On A Network (Powerbus) On An Uniden (Powergen) With A Microsatellite) On The Microsonde (Powerstation) On Your Computer (Power

Transcription

1 A Topology-Aware Performance Monitoring Tool for Shared Resource Management in Multicore Systems TADaaM Team - Nicolas Denoyelle - Brice Goglin - Emmanuel Jeannot August 24, 2015

2 1. Context/Motivations 2. Fast presentation of the tool 3. Demonstration 4. How does it works? 5. How is it made? 6. Features & Future Works TADaaM Team - Nicolas Denoyelle - Brice Goglin - Emmanuel Jeannot August 24, 2015

3 MOTIVATIONS Memory hierarchy is growing deeper and larger. No performance without a fair usage of the system topology Batch schedulers, runtimes, applications themeselves... are getting topology aware. Processing Unit NUMA Shared Memory Shared Cache IO Network Machine NUMA Shared Memory Shared Cache Topology Aware Performance Monitoring August 24,

4 MOTIVATIONS Memory hierarchy is growing deeper and larger. IO Network Hence, data management gives opportunities for performance improvements. NUMA Shared Memory Shared Cache Machine NUMA Shared Memory Shared Cache Processing Unit Topology Aware Performance Monitoring August 24,

5 MOTIVATIONS Memory hierarchy is growing deeper and larger. IO Network Hence, data management gives opportunities for performance improvements. NUMA Shared Memory Shared Cache Machine NUMA Shared Memory Shared Cache Processing Unit Topology Aware Performance Monitoring August 24,

6 MOTIVATIONS Memory hierarchy is growing deeper and larger. IO Network Hence, data management gives opportunities for performance improvements. NUMA Shared Memory Shared Cache Machine NUMA Shared Memory Shared Cache It is a multi-level and a multi-criteria problem. Processing Unit Topology Aware Performance Monitoring August 24,

7 MOTIVATIONS Need to match use cases, and relevant performance metrics for each level. Need to match performance and topology. Requires topology modeling skills. Requires adaptable performance monitoring. Topology Aware Performance Monitoring August 24,

8 Yet Another Tool to Monitor Applications Performance Focus on data presentation to link the results with topology informations. Relies on two cornerstones of topology modeling (hwloc) and performance counter abstraction (PAPI) to map the latter on the former Minimal configuration and software requirements. Can help finding and caracterizing localized bottlenecks. Topology Aware Performance Monitoring August 24,

9 Hardware Locality (hwloc) Portable abstraction of hierarchical architectures for high-performance computing Performs topology discovery and extracts hardware component information. Provides tools for memory and process binding. Many operating systems supported... lstopo utility to display the topology: Developped at Inria Bordeaux. Topology Aware Performance Monitoring August 24,

10 Hardware Locality (hwloc) Machine (31GB total) NUMANode P#0 (31GB) Package P#0 L3 (20MB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L2 (256KB) L1d L1d L1d L1d L1d L1d L1d L1d L1i L1i L1i L1i L1i L1i L1i L1i P#0 P#1 P#2 P#3 P#4 P#5 P#6 P#7 P#0 P#2 P#4 P#6 P#8 P#10 P#12 P#14 P#16 P#18 P#20 P#22 P#24 P#26 P#28 P#30 Topology Aware Performance Monitoring August 24,

11 Performance Application Programming Interface (PAPI) Consistent interface and methodology for use of the performance counter hardware. Real time relation between software performance and processor events. Many operating systems supported too. Reliable and actively supported. Used in a wide range of performance analysis applications. An abstraction layer to plug some other performance library is under development. Topology Aware Performance Monitoring August 24,

12 Dynamic Lstopo (example) Machine (16GB total) NUMANode P#0 (16GB) Package P#0 L3 9, e+01 (20MB) L2 2, e+02 (256KB) L2 2, e+02 (256KB) L2 2, e+02 (256KB) L2 2, e+02 (256KB) L2 2, e+02 (256KB) L2 5, e+02 (256KB) L2 2, e+02 (256KB) L2 3, e+02 (256KB) L1d 6, e+02 L1d 6, e+02 L1d 6, e+02 L1d 6, e+02 L1d 7, e+02 L1d 1, e+03 L1d 6, e+02 L1d 6, e+02 L1i 1, e+03 L1i 1, e+03 L1i 1, e+03 L1i 1, e+03 L1i 1, e+03 L1i 2, e+03 L1i 1, e+03 L1i 1, e+03 P#0 P#1 P#2 P#3 P#4 P#5 P#6 P#7 2,82176e+00 P#0 2,45916e+00 P#2 2,68772e+00 P#4 1,51689e+00 P#6 1,63417e+00 P#8 3,47249e+00 P#10 1,47808e+00 P#12 1,51514e+00 P#14 1,40868e+00 P#16 1,47441e+00 P#18 1,52142e+00 P#20 2,64271e+00 P#22 2,82472e+00 P#24 1,47165e+00 P#26 2,58947e+00 P#28 2,49344e+00 P#30 Sample of hardware performance counters mapped on a single socket of an Intel Xeon E C. Topology Aware Performance Monitoring August 24,

13 A Demonstration Worth Thousand Words L1 L2 L Accesses to a linked list of variable size. Topology Aware Performance Monitoring August 24,

14 A Demonstration Worth Thousand Words L1 L2 L Accesses to a linked list of variable size. Topology Aware Performance Monitoring August 24,

15 A Demonstration Worth Thousand Words L1 L2 L Accesses to a linked list of variable size. Topology Aware Performance Monitoring August 24,

16 A Demonstration Worth Thousand Words L3_MISS{ OBJ = L3; CTR = PAPI_L3_TCM; LOGSCALE = 1; } L2_MISS{ OBJ = L2; CTR = PAPI_L2_TCM; LOGSCALE = 1; } L1_MISS{ OBJ = L1d; CTR = PAPI_L1_DCM; LOGSCALE = 1; } SINGLE_L3_MISS{ OBJ = ; CTR = PAPI_L3_TCM; LOGSCALE = 1; } Topology Aware Performance Monitoring August 24,

17 Dynamic Lstopo (Usage) Machine (16GB total) Counters input: NUMANode P#0 (16GB) Package P#0 L3 2, e+03 (4096KB) L2 (256KB) L1d L1i L2 (256KB) L1d L1i SINGLE L3 MISS{ OBJ = L3 ; CTR = PAPI L2 TCM/PAPI L2 TCA ; LOGSCALE = 1 ; MAX= ; MIN=0; } P#0 P#1 P#0 P#1 P#2 P#3 Command line: lstopo perf-input counters input Topology Aware Performance Monitoring August 24,

18 Dynamic Lstopo (Theory) 1. Spawn one pthread per hardware thread (#0,..., #3). L2 L1 + L1 + + Topology Aware Performance Monitoring August 24,

19 Dynamic Lstopo (Theory) 1. Spawn one pthread per hardware thread (#0,..., #3). L2 2. For each timestamp, each thread collects a local set of performance counters. L1 + L1 + + Topology Aware Performance Monitoring August 24,

20 Dynamic Lstopo (Theory) 1. Spawn one pthread per hardware thread (#0,..., #3). L2 2. For each timestamp, each thread collects a local set of performance counters. L1 + L1 3. Counters are accumulated in each upper level. + + Topology Aware Performance Monitoring August 24,

21 Dynamic Lstopo (Theory) 1. Spawn one pthread per hardware thread (#0,..., #3). L2 % 2. For each timestamp, each thread collects a local set of performance counters. % L1 + L1 % 3. Counters are accumulated in each upper level. % + % % + % 4. For each level, a leaf computes an arithmetic expression of the performance counters in the set. Topology Aware Performance Monitoring August 24,

22 Dynamic Lstopo Software Architecture in Brief lstopo utility Monitors output Application monitors library hwloc library PAPI library Machine static topology Machine dynamic performance counters Topology Aware Performance Monitoring August 24,

23 Dynamic Lstopo Software Architecture in Brief lstopo utility Monitors output Application monitors library hwloc library PAPI library Machine static topology Machine dynamic performance counters Topology Aware Performance Monitoring August 24,

24 API m o n i t o r s = l o a d M o n i t o r s f r o m c o n f i g (NULL, m y p e r f f i l e, m y o u t p u t f i l e, 0) ; M o n i t o r s w a t c h p i d ( monitors, g e t p i d ( ) ) ; M o n i t o r s s t a r t ( m o n i t o r s ) ; /... / M o n i t o r s u p d a t e c o u n t e r s ( m o n i t o r s ) ; d e l e t e M o n i t o r s ( m o n i t o r s ) ; Topology Aware Performance Monitoring August 24,

25 Dynamic Lstopo (Output paje trace) % EventDef val 0 % Id int % Phase int % Time_us date % Value double % EndEventDef % EventDef container 1 % Id int % Level string % Sibling int % Name string % Logscale int % EndEventDef SINGLE_ L3_ MISS SINGLE_ L3_ MISS SINGLE_ L3_ MISS SINGLE_ L3_ MISS , , , , , , , ,00 Topology Aware Performance Monitoring August 24,

26 Features Record and/or Display live machine performance counters and match them with topology. Several settings: counters accumulation, sampling rate, attach to a process... Replay any trace with a topology file (for external display).. Sample specific parts of an application with the monitor library. Support legacy lstopo options (restrict topology, change display format... ). Topology Aware Performance Monitoring August 24,

27 Future works Match code and performance informations Accept user defined aggregation operator. Provide performance abstraction layer. Be able to delimit phases during execution. Find and give explicit hints on bottlenecks. Topology Aware Performance Monitoring August 24,

28 Conclusion Data locality becomes a main criterion for high performance. We built a tool based on a topology model and a performance library to help taking up the challenge. It maps performance values to machine objects. It is a visual tool, fast and easy to use. It is lightweight and causes less than 1% C overhead. Let you build topology aware performance models. Dynamic lstopo is into the process of beeing merged with hwloc project. Topology Aware Performance Monitoring August 24,

29 Now available from lstopo Thank you Topology Aware Performance Monitoring August 24,