Tools for Analysis of Performance Dynamics of Parallel Applications

Size: px

Start display at page:

Download "Tools for Analysis of Performance Dynamics of Parallel Applications"

Earl Simpson
8 years ago
Views:

1 Tools for Analysis of Performance Dynamics of Parallel Applications Yury Oleynik Fourth International Workshop on Parallel Software Tools and Tool Infrastructures Technische Universität München Yury Oleynik, PSTI Lyon

Parallel Software Tools and Tool Infrastructures Technische

2 Outline Project Overview Performance Dynamics I. PAThWay II. Periscope Performance Dynamics Analysis Conclusion Yury Oleynik, PSTI Lyon 2

Simulation mbh, Braunschweig RWTH Aachen Technische Universität Dresden Technische Universität München

3 LMAC Leistungsdynamik massiv-paralleler Codes Performance Dynamics of Massively Parallel Codes German Research School for Simulation Sciences, Aachen Forschungszentrum Jülich Gesellschaft für numerische Simulation mbh, Braunschweig RWTH Aachen Technische Universität Dresden Technische Universität München University of Oregon, Eugene, USA (associated partner) Yury Oleynik, PSTI Lyon 3

4 Performance Dynamics Aspects Runtime performance dynamics Automatically search for relevant properties of runtime performance dynamics Cross-experiment performance dynamics Provide a tool for structuring, automation and tracking of performance dynamics spanning over the optimization process Yury Oleynik, oleynik@in.tum.de PSTI Lyon 4

dynamics Provide a tool for structuring, automation and tracking of performance

I. Cross-experiment performance dynamics PATHWAY 2.10.

5 I. Cross-experiment performance dynamics PATHWAY Yury Oleynik, PSTI Lyon 5

6 Problem statement Performance engineering is an iterative cycle How to organize the whole process? What tools to use, when and how? Have the performance improved in comparison to results of the original version from one month ago? Optimization spans over months What analysis have I done yesterday? Where have I stored the traces? Repetitive, manual processes How do I write LoadLeveler script? What arguments should I pass to Periscope? Verify Optimize problematic code sections Baseline Establish/Update Analyze Bottlenecks Execute Parallel application Monitor Performance Yury Oleynik, oleynik@in.tum.de PSTI Lyon 6

Optimization spans over months What analysis have I done yesterday? Where have I stored the traces?

7 PAThWay Eclipse plug-in for structured and methodical performance engineering using workflows Goals: Manage individual tasks as part of one workflow Keep track of the performance dynamics Automate performance engineering tasks, where possible Abstract complexity of the underlying software and hardware Yury Oleynik, PSTI Lyon 7

performance dynamics Automate performance engineering tasks, where possible Abstract

8 Yury Oleynik, PSTI Lyon 8

9 Workflow specification Formal specification of the optimization process using workflows Workflows specification using Business Process Model and Notation (BPMN) format Predefined components for performance engineering tasks Predefined workflows: Scalability analysis Cross-platform analysis New workflows can be added using graphical editor Once defined, workflows can be executed Live-tracking of workflow execution Persistence execution can be paused and continued later Yury Oleynik, PSTI Lyon 9

analysis Cross-platform analysis New workflows can be added using graphical editor Once defined, workflows can be executed

10 Workflow components Interactive tasks Simulation code and perf tools selection Runtime environment configuration Prompting user for documentation input for historical records Loops Starts all experiments at once Domain-Specific Items Execution management (creating batch scripts) Tools configuration and invocation Data postprocessing Yury Oleynik, PSTI Lyon 10

experiments at once Domain-Specific Items Execution management (creating batch scripts) Tools

11 Workflow Editor: Scalability Analysis Available workflow components Workflow editor Yury Oleynik, PSTI Lyon 11

12 Data and Workflow Repository Stores workflow related information Contains key properties of the supported tools: Characteristics (can profile/trace. etc) Startup arguments Meta data about the tool Experiments meta-data: Run date/time Location of result files Environment settings StdOut/Error Target machine Yury Oleynik, PSTI Lyon 12

etc) Startup arguments Meta data about the tool Experiments meta-data: Run date/time

13 Experiment Browser Database stores also properties of the tools Experiments view Standard output and environment configuration Experiments Meta-data Yury Oleynik, PSTI Lyon 13

environment configuration Experiments Meta-data 2.10.

14 Project Documentation Yes, you must still invest time in documentation but PAThWay prepares templates Documents each completed experiment Automatically inserts relevant experiment meta-data User can further extend documentation using Wikibased editor Yury Oleynik, PSTI Lyon 14

inserts relevant experiment meta-data User can further extend documentation

15 Project Doc List of Perf. Experiments Yury Oleynik, PSTI Lyon 15

16 Supportive Modules Parallel Tools Platform Module Starting interactive/batch jobs Monitoring execution & accessing data Keep snapshots of the sources Tracks the code evolution Based on Git Environment Detection Captures loaded modules Captures environment variables Yury Oleynik, PSTI Lyon 16

code evolution Based on Git Environment Detection Captures loaded modules

17 PAThWay Available as an Eclipse plugin from the update site: Installation guide: Yury Oleynik, PSTI Lyon 17

18 II. Performance Dynamics: runtime AUTOMATIC PERFORMANCE DYNAMICS ANALYSIS WITH PERISCOPE Yury Oleynik, PSTI Lyon 18

19 Motivating example [ Z. Szebenyi, B. Wylie, and F. Wolf. Scalasca parallel performance analyses of pepc. In Euro-Par 2008 Workshops-Parallel Processing, pages Springer, ] Yury Oleynik, oleynik@in.tum.de PSTI Lyon 19

In Euro-Par 2008 Workshops-Parallel Processing, pages 305 314.

20 Automatic Performance Dynamics Analysis with Periscope Motivation for Performance Dynamics Analysis Location and severity of performance bottlenecks is time-dependent Performance changes manifest themselves at various time scales Dimensionality of performance measurements makes manual investigation by the user tedious Analysis goals: Automatically detect changes in temporal performance behavior Quantify the negative impact of performance changes Reduce complexity and size of time-dependent measurements Simplify comprehension (no graphical visualization) Group entities with similar temporal performance behavior Yury Oleynik, PSTI Lyon 20

Automatically detect changes in temporal performance behavior Quantify the negative impact of performance changes Reduce complexity and size of time-dependent

21 Automatic Performance Dynamics Analysis with Periscope Helps to answer following typical questions: Does the performance degrade over time? What is the most relevant performance degradations? When is the degradation observed? Which process/location is impacted by the performance degradation? Are there similar degradations found in other processes or functions? Approach Multi-scale analysis Qualitative abstraction of time series with quantitative information sufficient to characterize impact Representation mimics human mental model of temporal behavior Automatic search for performance dynamics properties Yury Oleynik, PSTI Lyon 21

22 Automatic Performance Dynamics Analysis with Periscope: Analysis Steps 1. Measurement collection 2. Preprocessing 3. Qualitative Abstraction 4. Search for performance dynamics properties Yury Oleynik, PSTI Lyon 22

23 Automatic Performance Dynamics Analysis with Periscope: Measurement Periscope uses Score-P to collect dynamic profiles: Let C be the set of all profiled source code locations within the body of the progress loop, P set of all profiled application process, M set of all measured metrics and I a sequence of all profiled progress loop iterations, then the we can define time-series Dynamic Profile as a mapping: which maps a source code location c ϵ C, a process p ϵ P a metric m ϵ M and an iteration i ϵ I onto a measurement ʋ Yury Oleynik, oleynik@in.tum.de PSTI Lyon 23

24 Automatic Performance Dynamics Analysis with Periscope: Preprocessing a) perform scale-space filtering of time series with Gaussian filter b) detect extremas and inflexion points over scales c) connect extremas and inflexion points over scales b) c) a) Yury Oleynik, PSTI Lyon 24

25 Automatic Performance Dynamics Analysis with Periscope: Qualitative Abstraction a) track extrema and inflexion points from coarse to fine scales b) label intervals between extrema and inflexion points c) extract the level of the tree with the maximum lifetime b) D AB A D A C C BCD C C B CD B C CD BCD AB C B C B C C B C A - concave increase B - concave decrease C - convex decrease D - convex increase E - linear increase F - linear decrease G - constant a) c) DABCBCDABCDABCDABCDABC Yury Oleynik, oleynik@in.tum.de PSTI Lyon 25

26 Automatic Performance Dynamics Analysis with Periscope: Search for dynamics properties a. Search the maximum lifetime level for relevant temporal patterns Allows flexible specification of the performance dynamics patterns specified using the following parameters: Qualitative shape specification Perceptual salience Magnitude/Duration of the segment Any combination of parameters above Yury Oleynik, PSTI Lyon 26

27 Example Performance properties examples: Local maxima (peaks) in Execution time of the main loop body in process 0 are observed during iterations 6-24, 44-47, 57-63, 67-73, The most distinctive local minima (valley) in Execution time of the main loop body in process 0 is observed during iterations The tallest increase in Execution time of the main loop body in process 0 is observed during iterations Yury Oleynik, oleynik@in.tum.de PSTI Lyon 27

28 Conclusion Temporal performance dynamics is important There are at least two aspects to consider Runtime performance dynamics Cross-experiment performance dynamics Tools support is needed to help answer questions: What are the most relevant runtime performance degradations? How to organize performance optimization process and track performance evolution? Our solution Periscope Performance Dynamics Analysis automatically search for temporal performance degradations PAThWay organize and track performance optimization using workflows Yury Oleynik, PSTI Lyon 28

29 Thank you! Questions? Join us for the VI-HPS Extreme Scale Programming Tools SC13! Further info: Yury Oleynik, oleynik@in.tum.de PSTI Lyon 29

Recent Advances in Periscope for Performance Analysis and Tuning

Recent Advances in Periscope for Performance Analysis and Tuning Isaias Compres, Michael Firbach, Michael Gerndt Robert Mijakovic, Yury Oleynik, Ventsislav Petkov Technische Universität München Yury Oleynik,