A Framework for Online Performance Analysis and Visualization of Large-Scale Parallel Applications

Transcription

1 A Framework for Online Performance Analysis and Visualization of Large-Scale Parallel Applications Kai Li, Allen D. Malony, Robert Bell, and Sameer Shende University of Oregon, Eugene, OR USA, Abstract. We describe an online performance data access, analysis, and visualization framework that will form the basis of a large-scale performance interaction and steering system. The framework integrates the TAU parallel performance system with the SCIRun visualization environment to enable the flexible development of runtime tools for responsive performance assessment. A demonstration of the framework is given for a 500-processor parallel fire simulation. 1 Introduction Parallel performance tools offer the program developer insights into the execution behavior of an application and are a valuable component in the cycle of application development, deployment, and optimization. However, most tools do not work well with large-scale parallel applications where the performance data generated comes from upwards of thousands of processes. Not only can the data be difficult to manage and the analysis complex, but existing performance display tools are mostly restricted to two dimensions and lack the customization and display interaction to support full data investigation. In addition, it is increasingly important that performance tools be able to function online, making it possible to control and adapt long-running applications based on performance feedback. Again, large-scale parallelism complicates the online access and management of performance data, and it may be desirable to integrate performance analysis and visualization in existing computational steering infrastructures. The coupling of advanced three-dimensional visualization with online performance data analysis could enhance the methods used for large-scale, parallel application performance evaluation. However, it is important to start with a solid foundation of performance technology (e.g., robust instrumentation and measurement) upon which such support can be prototyped. This is particular true if the tools are to be portable across parallel computing platforms. The main challenge then is to develop a framework that focuses on higher-level analysis, visualization, and control issues, while hiding low-level performance data gathering and graphics rendering concerns. This also allows framework tool developers more freedom (and time) to create new functionality that can integrate with and extend framework capabilities. This paper presents a prototype framework for online performance analysis and visualization of large-scale parallel applications. The framework is built to work with the TAU parallel performance system [8, 19] as the foundation for the portable measurement of performance profiles. In addition, the framework leverages the capabilities of the SCIRun [13] computational problem solving environment. The motivation for this work comes from our ongoing experience building scalable performance analysis for the Uintah Computational Framework (UCF) [17, 18], but our broader research objective is to target a range of application in scalable parallel environments. We begin with a review of the online parallel performance analysis problem and related work in the area. The section following presents the framework design, discusses the tradeoffs made, and describes the framework s analysis and visualization modules in detail. We then briefly describe the TAU performance system and how it has been

2 applied for UCF performance analysis. The paper presents an example to demonstrate the framework s use on a 500-processor Uintah application. The paper is concluded with a discussion of future work. 2 Background and Related Work There has been a long time interest in the monitoring of parallel systems and aplications. This is due to the general hypothesis that by observing the runtime behavior or performance of the system or application, it is possible to identify aspects of parallel execution that may allow for improvement. What these aspects are depends on what is being monitored (e.g., systems resources, parallel activity, or communication). Several projects have developed techniques that allow parallel applications to be responsive to program behavior, available resources, or performance factors. The Falcon project [5] is an example of computational steering systems [20] that can observe the behavior of an application and provide hooks to alter application semantics. These actuators will then lead to changes in the ongoing execution. Because computational steering systems enable direct interaction with the application, they are often developed with sophisticated visualization frontends that provide graphical renderings of application state and objects for execution control. Active Harmony [6] follows in computational steering approach, but additionally provides for an adaptation controller that can automate decision making. It interacts with an algorithm adaptation layer that can coordinate resource control across multiple applications and libraries. Similarly, Autopilot [15] is a monitoring and adaptive control framework that uses application sensors to extract quantitative and qualitative performance for automated decision control, using fuzzy logic and hidden Markov models. Like Active Harmony, their actuators provide a programmatic mechanism to build adaptive applications. While both Active Harmony and Autopilot are oriented towards automated performance tuning, neither address the problem of scalable performance monitoring or provide capabilties to analyze or visualize large-volume performance information. Indeed, the difficulty of linking application embedded monitoring to data consumers will ultimately determine what amount of runtime information can be utilized. This involves a complicated tradeoff of instrumentation and measurement granularity versus the overhead of application / performance data transport versus the information requirements for desired analysis [9, 14]. Projects such as the Multicast Reduction Network (MRNet) [16] will help in providing efficient infrastructure for data communication and filtering. Similarly, the Peridot [4] project is attempting to develop a distributed application monitoring framework for shared-memory multiprocessor (SMP) clusters that can provide scalable trace data collection and online analysis. The system will have selective instrumentation and analysis control, helping to address node- and system-level monitoring requirements. A different approach to scalable monitoring is taken in [10]. Here, statistical sampling techniques are used to gain representative views of system performance characteristics and behavior. However, work is also needed on large-scale performance data analysis which can couple to systems such as Peridot and MRNet to provide runtime performance evaluation and control. The use of powerful parallel performance tools should not be dismissed merely because it is assumed they cannot operate in online modes, either because they process larger volumes of data or require more analysis power. It is reasonable to expect that classical offline tools could be brought online under the right system conditions. For instance, the Vampir [12] parallel trace analysis system offers sophisticated features for parallel performance visualization of message passing applications. Certain of its capabilities could function at runtime. Paradyn [11] is an example of a system that has demonstrated the ability to conduct complex analyses at runtime. In general, we believe the benefits seen in the application of online computation visualization and steering, itself requiring demanding monitoring support, could also be realized in the parallel performance domain. Our goal is to 2

3 Fig. 1. Online Performance Analysis and Visualization Architecture. consider the problem of performance monitoring, analysis, visualization, and steering as a whole, understanding the tradeoffs involved and designing a framework architecture to address them. 3 Framework Design Traditional parallel performance tools make measurments during program execution and then store performance data for analysis after the program terminates. Our work in this paper grew out of an interest to enable online access to performance data gathered by a portable, scalable parallel measurement system. At the same time, we wanted to offer interactive tools for interrogating, analyzing, and visualizing the parallel performance data retrieved from the running application. Unfortunately, large-scale parallelism complicates both the access to the measurements and the coordination of the analysis and visualization tools with the running program. As a result of our ongoing work with the University of Utah [18], we designed a performance analysis framework that couples an advanced visualization environment with our parallel performance system. This framework architecture is targeted for use in large-scale parallel applications where runtime benefits might be achieved through interactive performance steering. Issues such as what performance would be measured, what types of analyses would be performed, how performnace would be visualized, and how performance feedback would be enabled are ones that we attempted not to resolve directl in the framework design, but to leave open for specialization in the framework s use. Thus, we looked for opportunities where functionality could be programmed and extended, and the chosen systems and tools could be leveraged to best advantage. The framework architecture, shown in Figure 1, consists of four components. The performance data integrator component is responsible for interfacing with a performance monitoring system to merge parallel performance samples into a synchronous data stream for analysis. The performance data reader component reads the external 3

4 performance data into internal data structures of the analysis and visualization system. The performance analyzer component provides the analysis developer a programmable framework for constructing analysis modules that can be linked together for different functionality. The performance visualizer component can also be programmed to create different displays modules. As seen in Figure 1, our framework design incorporates the TAU performance system and the SCIRun [13] computational steering and visualization environment. The targeted functionality will allow parallel profile data, measured by TAU, to be accessed during execution, and moved into an analysis and visualization system running in SCIRun. Each of the components is discussed below. 3.1 Performance Data Integrator. The performance data integrator reads the performance profile files, generated for each profile sample for each thread, and merges the files into a single, synchronized profile sample dataset. Each profile sample file is assigned a sequence number and the whole dataset is sequenced and timestamped. A socket-based protocol is maintained with the performance data reader to inform it of the availability of new profile samples and to coordinate dataset transfer. 3.2 Profile Reader The performance profile reader, implemented as a SCIRun module, inputs the merged profile sample dataset sent by the data integrator and stores the dataset in an internal object-oriented structure. A profile sample dataset is organized in a tree-like manner according to TAU profile data hierarchy (see Section 4): node context thread profile data Each object in the profile tree has a set of attribute access methods and a set of offspring access methods, in order to support efficient data selection and quantification. It should be noted that the profile reader can maintain the entire set of performance profile samples generated since the start of the application. This is important for analyzing execution trends across sample sequences. 3.3 Profile Analysis Using the access methods on the profile tree object, all performance profile data, including cross-sample data, is available for analysis. SCIRun provides a programmable system for building and linking the analysis and visualization components. A library of performance analysis modules can be developed, some simple and others more sophisticated. We have initially implemented two generic profile analysis modules Gen2DField and Gen3DField that select data subsets and perform basic calculations. The modules provide interactive user control that allows them to be customized with respect to events, data values, number of samples, and filter options. Ultimately, the output of the analysis modules must be in a form that can be visualized. The Gen2DField and Gen3DField modules are so named because they produce 2D and 3D Field data, respectively. SCIRun has different geometric meshes available for Fields. We use an ImageMesh for 2D fields and a PointCloudMesh for 3D fields. 4

5 Sample 1 Sample i Sample k n:1 n:2 n:j c:1 c:2 c:k c:1 c:2 c:k t:1 t:2 t:m t:1 t:2 t:m p p p p p p Fig. 2. Internal Multi-Sample Profile Structure. 3.4 Performance Visualizer In our framework, the role of the performance visualizer component is to read output of the performance analysis modules (i.e., the Field objects) and show graphical representations of performance results. We have built four visualization modules to demonstrate the display of 2D and 3D data fields. The Terrain visualizer shows ImageMesh data as a surface. The user can select the resolution of the X and Y dimensions in the Terrain control panel. A TerrainDenotator module was developed to mark interesting points in the visualization. A different display of 2D field data is produced by the KiviatTube visualizer. Here a tube surface is created where the distance of points from the tube center axis is determined by metric values and the tube length correlates with the sample. The visualization of PointCloudMesh data is accomplished by the PointCloud visualizer module. This is useful to show sequenced profile performance results. The fourth module, the Scatterplot visualizer, shows relationships between performance metrics for different threads of execution in a three dimensional point plot where point coordinates are determined by metric values. These visualization modules are shown in examples below. 3.5 Performance Steering Figure 1 shows a performance steering component in dashed outline. While this is an important part of the framework design, its realization is presently waiting for new functionality to become available in the next generation of TAU and SCIRun. The steering opportunites we will have include both application control and control of the performance measurement. It is important to note that in the initial design, the performance data integrator can provide some support as a control point for data filtering and reduction. That is, it can offer a simple interface that allows selection of performance events of interest to be passed to the data reader. 3.6 Comments As initially conceived, the framework uses the file system on the parallel machine for off-loading performance data. This decision was made primarily for purposes of robustness and portability. However, we will want to support direct 5

6 network connection to the running application in the future. We believe there is flexibility in the framework design to accomodate this approach. Doing so will make it possible to leverage new performance monitoring infrastructure, such as MRNet, as well as to implement interface support for application control and steering. SCIRun is a dataflow execution environment. The formulation of the data reader, analysis, and visualization components of the framework in SCIRun translates into a system that reacts to the availability of new data. Thus, analysis and visualization are responsive during execution, updating their state and presentation to reflect current performance information. 4 TAU Performance System The TAU performance system [8, 19] is our integrated toolkit for performance instrumentation, measurement, analysis, and visualization of large-scale parallel applications. It targets a general computation model consisting of shared-memory nodes where contexts reside, each providing a virtual address space shared by multiple threads of execution. The model is general enough to apply to many high-performance scalable parallel systems and programming paradigms. Because TAU enables performance information to be captured at the node/context/thread levels, this information can be mapped to the particular parallel software and system execution platform under consideration. The TAU measurement library implements performance profiling and tracing support for events occurring at function, method, basic block, and statement levels. Performance experiments can be composed from different measurement modules (e.g., hardware performance monitors, such as PAPI [3]) and measurements can be collected with respect to user-defined performance groups. C, C++, Fortran 77/90, OpenMP, and Java languages are supported. Currently, the online performance monitoring support TAU provides consists of a profile dump routine that will write out the present parallel profile to the file system. This is what is used in the framework implementation at present. TAU has been ported to nearly all high-performance computing platforms and is being used extensively in the performance analysis of DOE applications. TAU is also being applied as the primary performance technology across a diverse set of code development projects, including Uintah [17], CCA [2], VTF [21], and SAMRAI [7]. 5 Framework Application We have tested our framework prototype using the Uintah computational framework [17] developed at the University of Utah. Uintah is a parallel, macro-level dataflow framework written in C++ and being used in large-scale simulations of accidental fires and explosions. During execution of a Uintah simulation, parallel profile data are sampled and written to profile files. The performance data integrator reads the performance profile files, generated for each profile sample for each thread of execution, and merges the files into a single, synchronized profile sample dataset. Each profile sample file is assigned a sequence number and the whole dataset is sequenced and timestamped. A socket-based protocol is maintained with the performance data reader to inform it of the availability of new profile samples and to coordinate dataset transfer. The SCIRun visual program in Figure 3 shows an example of how the data reader, analyzer, and visualizer modules are connected in a dataflow graph to process parallel profile samples from a Uintah application. The analysis and visualization modules on the left side of the graph are configured to extract exclusive execution time and display a terrain surface. The modules on the right side of the graph are being used to compare this metric for three different communication routines. The results are shown in scatter plot. 6

7 Fig. 3. SCIRun Program Graph for Online Profile Analysis and Resulting Performance Visualizations. The resulting visualizations produced by the SCIRun program are also shown in Figure 3. The example given is for a 500 processor run of a Uintah simulation. The top image shows the performance events (functions) along the left-to-right axis (labeled FUNCTION), the execution threads along the in-out axis (labeled THREADS), and the performance metric value along the up-down axis (in this case, EXCLUSIVE TIME). Highlighted lines are used to identify a particular function or thread in the terrain view. This performance view enables the user to quickly identify major performance contributors for the chosen metric of interest. The right image shows a point for every thread of execution in a three-dimensional space represented by three performance events. Here, Point coordinates are determined by the exclusive time values for these events: MPI Reduce, MPI Recv, and MPI Waitsome. The scatter plot shows how thread performance is clustered in the profile sample. 6 Conclusion and Future Work Although the framework is in the early stages, it demonstrates the significant tool advances possible through technology integration. As the Utah C-SAFE ASCI project moves towards Uintah computations with adaptive-mesh refinement capabilities, we expect the relevance of online performance analysis to increase in importance. Since 7

8 SCIRun is being positioned as a computational steering system for Uintah, the implementation of the online performance tool in SCIRun well positions it for use as a customizable performance steering tool. Further advances in the TAU performance system will enable dynamic control of measurements to better guide performance analysis. References 1. APART, IST Working Group on Automatic Performance Analysis: Real Tools. See 2. R. Armstrong, et al., Toward a Common Component Architecture for High-Performance Scientific Computing, High Performance Distributed Computing Conference, See 3. S. Browne, J. Dongarra, N. Garner, G. Ho, and P. Mucci, A Portable Programming Interface for Performance Evaluation on Modern Processors, International Journal of High Performance Computing Applications, 14(3): , Fall K. Fuerlinger and M. Gerndt, Distributed Application Monitoring for Clustered SMP Architectures, accepted to EuroPar 2003, Workshop on Performance Evaluation and Prediction, W Gu, G. Eisenhauer, E. Kraemer, K. Schwan, J. Stasko, J. Vetter, and N. Mallavarupu, Falcon: On-line Monitoring and Steering of Large-Scale Parallel Programs, Proceedings of the 5th Symposium of the Frontiers of Massively Parallel Computing, pp , J. Hollingsworth and P. Keleher, Prediction and Adaption in Active Harmony, Cluster Computing, 2(3): , R. Hornung and S. Kohn, Managing Application Complexity in the SAMRAI Object-Oriented Framework, Concurrency and Computation: Practice and Experience, special issue on Software Architectures for Scientific Applications, A. Malony and S. Shende, Performance Technology for Complex Parallel and Distributed Systems, in Distributed and Parallel Systems From Instruction Parallelism to Cluster Computing, G. Kotsis and P. Kacsuk (Eds.), Kluwer, pp , A. Malony, Tools for Parallel Computing: A Performance Evaluation Perspective, in Handbook on Parallel and Distributed Processing, J. Blazewicz, K. Ecker, B. Plateau, an D. Trystram (Eds.), 2000, Springer-Verlag, pp C. Mendes and D. Reed, Monitoring Large Systems via Statistical Sampling, LACSI Symposium, B. Miller, M. Callaghan, J. Cargille, J. Hollingsworth, R. Irvin, K. Karavanic, K. Kunchithapadam, and Tia Newhall, The Paradyn Parallel Performance Measurement Tool, IEEE Computer, 28(11):37-46, W.Nagel, A.Arnold, M.Weber, H.Hoppe, and K.Solchenbach, Vampir: Visualization and Analysis of MPI Resources, Supercomputing, 12(1):69 80, S. Parker, D. Weinstein, and C. Johnson, The SCIRun Computational Steering Software System, in Modern Software Tools in Scientific Computing, E. Arge, A. Bruaset, and H. Langtangen (Eds.), Birkhauser Press, pp. 1 44, D. Reed, C. Elford, T. Madhyastha, E. Smirni, and S. Lamm, The next frontier: Interactive and closed loop performance steering, Proceedings of the 25th Annual Conference of International Conference on Parallel Processing, R. Ribler, H. Simitci, and D. Reed, The Autopilot Performance-Directed Adaptive Control System, Future Generation Computer Systems, special issue (Performance Data Mining), 18(1): , P. Roth, D. Arnold, and B. Miller, MRNet: A Software-Based Multicast/Reduction Network for Scalable Tools, Technical report, University of Wisconsin, Madison, J. St. Germain, J. McCorquodale, S. Parker, and C. Johnson, Uintah: A Massively Parallel Problem Solving Environment, High Performance Distributed Computing Conference, pp , J. St. Germain, A. Morris, S. G. Parker, A. D. Malony, and S. Shende, Integrating Performance Analysis in the Uintah Software Development Cycle, International Symposium on High Performance Computing, pp , TAU (Tuning and Analysis Utilities). See J. Vetter, Computational Steering Annotated Bibliography, SIGPLAN Notices, 32(6), pp , VTF, Virtual Test Shock Facility, Center for Simulation of Dynamic Response of Materials. See 8