Combining Instrumentation and Sampling for Trace-based Application Performance Analysis

Transcription

1 Center for Information Services and High Performance Computing (ZIH) Combining Instrumentation and Sampling for Trace-based Application Performance Analysis 8th International Parallel Tools Workshop Stuttgart, Germany, October 2, 2014 Thomas Ilsche Joseph Schuchart Robert Schöne Daniel Hackenberg

2 Introduction Looking at the landscape of performance analysis tools Identify established techniques Provide a structured overview Highlight strengths and weaknesses Identify novel combinations Combine strengths Mitigate weaknesses Look beyond the traditional fields of tools Thomas Ilsche 4

3 Classification of performance analysis techniques Data Presentation Profiles Timelines Data Recording Summarization Logging Data Acquisition Sampling Event-based Instrumentation Performance Analysis Layer Performance Analysis Technique Based on [10] Juckeland, G.: Trace-based Performance Analysis for Hardware Accelerators. Ph.D. thesis, TU Dresden (2012) Thomas Ilsche 5

4 Classification of performance analysis techniques Data Presentation Profiles Timelines Data Recording Summarization Logging Data Acquisition Sampling Event-based Instrumentation Performance Analysis Layer Performance Analysis Technique Based on [10] Juckeland, G.: Trace-based Performance Analysis for Hardware Accelerators. Ph.D. thesis, TU Dresden (2012) Thomas Ilsche 6

5 Data Acquisition: Event-based Instrumentation time bar foo main bar Measurement environment Event-based instrumentation; also: direct instrumentation, event trigger, probebased measurement or simply instrumentation. Modification of the application execution in order to record and present certain intrinsic events of the application execution, e.g., function entry and exit events. Thomas Ilsche 7

6 Data Acquisition: Event-based Instrumentation Overhead & perturbation depends on function call rate Hard to predict in complex applications Can be influenced by filtering function calls Preferably statically, not during runtime Complete information Accurate function call counts Message properties (semantics of function call arguments) Analysis tools may rely on completeness Thomas Ilsche 8

7 Data Acquisition: Event-based Instrumentation Various instrumentation methods available Compiler instrumentation * Library wrapping ** Source code transformation * Manual instrumentation * Binary instrumentation * Requires recompilation & separate performance measurement binary ** Requires relinking for statically linked binaries Thomas Ilsche 9

8 Data Acquisition: Sampling time 200us 400us 600us 800us bar foo main bar Measurement environment Sampling; also: statistical sampling or (ambiguously) profiling. Periodic interruption of a running program and inspection of its state. Thomas Ilsche 10

9 Data Acquisition: Sampling Overhead & perturbation depends on sampling rate Can be predicted Can be controlled Stack unwinding introduce uncertainty Easy to use (for end users) No recompilation or relinking necessary No filtering necessary Thomas Ilsche 11

10 Data Acquisition: Sampling Incomplete information No accurate function call counts No specific message properties or other semantics of function arguments Measurement has statistical value More reliable for longer running experiments Trade-off between accuracy and perturbation via sampling rate Thomas Ilsche 12

11 Classification of performance analysis techniques Data Presentation Profiles Timelines Data Recording Summarization Logging Data Acquisition Sampling Event-based Instrumentation Performance Analysis Layer Performance Analysis Technique Based on [10] Juckeland, G.: Trace-based Performance Analysis for Hardware Accelerators. Ph.D. thesis, TU Dresden (2012) Thomas Ilsche 13

12 Summarization vs Logging Defines how the recording during runtime is performed. Event-based Instrumentation Sampling Summarization Logging 0000us: Enter main 0050us: Enter foo 0100us: Enter bar 0300us: Leave bar 0650us: Leave foo 0700us: Enter bar 0900us: Leave bar 1000us: Leave main Thomas Ilsche 14

13 Summarization vs Logging Defines how the recording during runtime is performed. Event-based Instrumentation Sampling Summarization count[main]++ count[foo]++ count[bar]++ time[bar]+=200 time[foo]+=600 count[foo]++ time[bar]+=200 time[main]+=1000 Logging 0000us: Enter main 0050us: Enter foo 0100us: Enter bar 0300us: Leave bar 0650us: Leave foo 0700us: Enter bar 0900us: Leave bar 1000us: Leave main Thomas Ilsche 15

14 Summarization vs Logging Defines how the recording during runtime is performed. Event-based Instrumentation Sampling Summarization count[main]++ count[foo]++ count[bar]++ time[bar]+=200 time[foo]+=600 count[foo]++ time[bar]+=200 time[main]+=1000 Logging 0000us: Enter main 0050us: Enter foo 0100us: Enter bar 0300us: Leave bar 0650us: Leave foo 0700us: Enter bar 0900us: Leave bar 1000us: Leave main 200us: main foo bar 400us: main foo 600us: main foo 800us: main bar Thomas Ilsche 16

15 Summarization vs Logging Defines how the recording during runtime is performed. Event-based Instrumentation Summarization count[main]++ count[foo]++ count[bar]++ time[bar]+=200 time[foo]+=600 count[foo]++ time[bar]+=200 time[main]+=1000 Sampling time_ex[bar] += 200 time_ex[foo] += 200 time_ex[foo] += 200 time_ex[bar] += 200 Logging 0000us: Enter main 0050us: Enter foo 0100us: Enter bar 0300us: Leave bar 0650us: Leave foo 0700us: Enter bar 0900us: Leave bar 1000us: Leave main 200us: main foo bar 400us: main foo 600us: main foo 800us: main bar Thomas Ilsche 17

16 Summarization vs Logging Defines how the recording during runtime is performed. Event-based Instrumentation Summarization count[main]++ count[foo]++ count[bar]++ time[bar]+=200 time[foo]+=600 count[foo]++ time[bar]+=200 time[main]+=1000 Sampling time_ex[bar] += 200 time_ex[foo] += 200 time_ex[foo] += 200 time_ex[bar] += 200 Loses information Logging 0000us: Enter main 0050us: Enter foo 0100us: Enter bar 0300us: Leave bar 0650us: Leave foo 0700us: Enter bar 0900us: Leave bar 1000us: Leave main 200us: main foo bar 400us: main foo 600us: main foo 800us: main bar Requires memory at runtime Thomas Ilsche 18

17 Classification of performance analysis techniques Data Presentation Profiles Timelines Data Recording Summarization Logging Data Acquisition Sampling Event-based Instrumentation Performance Analysis Layer Performance Analysis Technique Based on [10] Juckeland, G.: Trace-based Performance Analysis for Hardware Accelerators. Ph.D. thesis, TU Dresden (2012) Thomas Ilsche 19

18 Data Presentation Each sample counts as 0.01 seconds. % cumulative self self total time seconds seconds calls ms/call ms/call name open offtime memccpy write Example profile (gprof) Can be generated by summarization, but also from logging Example timeline showing call-path and event annotations (Vampir) Needs logging during recording Thomas Ilsche 20

19 Classification of performance analysis techniques Profiling Tracing Data Presentation Profiles Timelines Data Recording Summarization Logging Data Acquisition Sampling Event-based Instrumentation Performance Analysis Layer Performance Analysis Technique Based on [10] Juckeland, G.: Trace-based Performance Analysis for Hardware Accelerators. Ph.D. thesis, TU Dresden (2012) Thomas Ilsche 21

20 Tools Profiling Scalasca TAU Tracing VampirTrace Event-based Instrumentation Sampling gprof Score-P perf HPCToolkit Extrae Example Concepts Allinea MAP Thomas Ilsche 25

21 Combining Performance Analysis Techniques (1) C++ Graph code INDDGO OpenMP, 4 Threads Uninstrumented :< 6 seconds Instrumented (profiling): 72 seconds 1100% overhead! A trace file would be ~3.8 GB with even more overhead Thomas Ilsche 26

22 Combining Performance Analysis Techniques (1) Estimated aggregate size of event trace: 3851MB Estimated requirements for largest trace buffer (max_buf): 3851MB Estimated memory requirements (SCOREP_TOTAL_MEMORY): 3860MB (hint: When tracing set SCOREP_TOTAL_MEMORY=3860MB to avoid intermediate flushes or reduce requirements using USR regions filters.) type max_buf[b] visits time[s] region ALL 4,038,048, ,849, ALL USR 4,038,047, ,849, USR OMP OMP COM COM 72 functions with > 1 million visits USR 365,389,440 14,053, Graph::lcgrand(int) USR 322,737,636 12,412, std::_list_iterator<int>::operator*() const USR 208,735,202 8,028, std::_list_iterator<int>::operator++() USR 201,389,266 7,745, std::_list_iterator<int>::_list_iterator USR 200,350,128 12,521, std::_list_iterator<int>::operator!= USR 1,040,000 40, Graph::Node* std:: addressof Thomas Ilsche 27

23 Combining Performance Analysis Techniques (1) MPI instrumentation For messages, complete information is very important during analysis MPI functions generally imply a certain minimum load Lower relative overhead expected compared to very short computation functions Call-path sampling For function execution statistical information may be sufficient Controlling the overhead of compiler instrumentation via filtering is not straightforward Tracing (Logging Timelines) Prototype in VampirTrace MPI: Traditional library interposition with PMPI Sampling: Performance counter based interrupt (e.g. every 1 million cycles) Thomas Ilsche 28

24 Combining Performance Analysis Techniques (1) Example with NPB-BT, 1 sample every 1 million cycles (~385 us) Thomas Ilsche 29

25 Combining Performance Analysis Techniques (1) MPI Instrumentation and call-path sampling Example with NPB-BT Runtime filter: matmul_sub, matvec_sub, binvrhs, binvcrhs, lhsinit, exact_solution Filter helps with overhead, but no more information about those functions ** Sampling rate of 2.6 ksa/s. Thomas Ilsche 30

26 Combining Performance Analysis Techniques (2) MPI and compiler instrumentation and performance counter sampling Tracing (Logging Timelines) Implemented as VampirTrace/Score-P metrics plugin MPI: Library interposition with PMPI Functions: Compiler instrumentation Hardware Counters: Monitoring thread wakes up in regular intervals and reads performance counter from application thread Thomas Ilsche 31

27 Combining Performance Analysis Techniques (2) NPB FT class B 16 procs 1ms sampling interval Thomas Ilsche 33

28 Combining Performance Analysis Techniques (2) Thomas Ilsche 34

29 Combining Performance Analysis Techniques (2) Thomas Ilsche 35

30 Combining Performance Analysis Techniques (2) Thomas Ilsche 36

31 Combining Performance Analysis Techniques (2) Thomas Ilsche 37

32 Combining Performance Analysis Techniques (2) Normalized trace sizes of NPB CLASS B, sampled (1 ksa/s). Baseline: trace without counters. Filtered functions: matmul_sub, matvec_sub, binvcrhs, exact_solution. Thomas Ilsche 38

33 Combining Performance Analysis Techniques Event-based instrumentation for: MPI, SHMEM, CUDA Manual instrumentation of longer program phases Function instrumentation together with sophisticated filtering Sampling for: Call-stack of programs where filtering is not feasible (often C++) Hardware counters External metrics (e.g. power consumption) Thomas Ilsche 39

34 Conclusion Combine strength and mitigate weaknesses by selecting the right technique for different aspects of performance analysis Sampling and instrumentation complement each other Many tools already cross the borders of single technique Use a clear terminology for techniques Separate the description for data acquisition, data recording and data presentation Event-based instrumentation vs sampling Tracing vs profiling Thomas Ilsche 40

35 Outlook Sampling in Score-P 1.4 planned for 2014 will be experimental New trace records for samples Analyzing & visualizing merged call-stack samples and region instrumentation Sample is valid for point in time Instrumentation covers time ranges Overlap? Thomas Ilsche 41