HOW TO VISUALIZE YOUR GPU-ACCELERATED SIMULATION RESULTS. Peter Messmer, NVIDIA

Transcription

1 HOW TO VISUALIZE YOUR GPU-ACCELERATED SIMULATION RESULTS Peter Messmer, NVIDIA

2 RANGE OF ANALYSIS AND VIZ TASKS Analysis: Focus quantitative Visualization: Focus qualitative Monitoring, Steering

3 TRADITIONAL HPC WORKFLOW Setup Workstation Analysis, Visualization Supercomputer Viz Cluster Dump, Checkpointing File System Visualization, Analysis

4 TRADITIONAL WORKFLOW: CHALLENGES Lack of interactivity prevents intuition Setup Workstation Analysis, Visualization High-end viz neglected due to workflow complexity Supercomputer Viz Cluster I/O becomes main Dump, simulation bottleneck Checkpointing Viz resources need to scale with simulation Visualization, Analysis File System

5 OUTLINE Visualization applications CUDA/OpenGL interop Remote viz Parallel viz In-Situ viz High-level overview. Some parts platform dependent. Check with your sysadmin.

6 VISUALIZATION APPLICATIONS

7 NON-REPRESENTATIVE VIZ TOOLS SURVEY OF 25 HPC SITES Surveyed sites: NERSC LLNL LLNL- -OCF SCF LANL ORNL- CCS DOD-ORC AFRL- DSCR AFRL ARL ERDC NAVY MHPCC ORS CCAC NASA- NAS NASA-NCCS TACC CHPC RZG HLRN Julich CSCS CSC Hector Curie

8 NON-REPRESENTATIVE VIZ TOOLS SURVEY OF 25 HPC SITES Surveyed sites: NERSC LLNL LLNL- -OCF SCF LANL ORNL- CCS DOD-ORC AFRL- DSCR AFRL ARL ERDC NAVY MHPCC ORS CCAC NASA- NAS NASA-NCCS TACC CHPC RZG HLRN Julich CSCS CSC Hector Curie

9 VISIT Scalar, vector and tensor field data features Plots: contour, curve, mesh, pseudo-color, volume,.. Operators: slice, iso-surface, threshold, binning,.. Quantitative and qualitative analysis/vis Derived fields, dimension reduction, line-outs Pick & query Scalable architecture Open source

10 VISIT Cross-platform Linux/Unix, OSX, Windows Wide range of data formats.vtk,.netcdf,.hdf5,.. Extensible Plugin architecture Embeddable Python scriptable

11 VISIT S SCALABLE ARCHITECTURE Client-server architecture Server MPI parallel Distributed filtering (multi-)gpu accelerated, parallel rendering* * requires X server on each node

12 PARAVIEW Scalar, vector and tensor field data features Plots: contour, curve, mesh, pseudocolor, volume,.. Operators: slice, iso-surface, threshold, binning,.. Quantitative and qualitative analysis/vis Derived fields, dimension reduction, line-outs Pick & query Scalable architecture Developed by Kitware, open source

13 PARAVIEW S SCALABLE ARCHITECTURE Client-server-server architecture Server MPI parallel Distributed filtering GPU accelerated, parallel rendering* * requires X server on each node

14 SOME OTHER TOOLS Wide range of visualization tools Often emerged from specialized application domain Tecplot, EnSight: structural analysis, CFD IndeX: seismic data processing & visualization IDL: image processing Early adopters of visual programming AVS/Express, OpenDX

15 FURTHER READING Paraview Tutorial: VisIt Manuals/Tutorials:

16 VTK - THE VISUALIZATION TOOLKIT

17 VISUALIZATION TOOLKIT Focus on visualization, not (only) rendering Provides more complex operations on data ( filtering ) Introduces visualization pipeline At the core of many high-level viz tools

18 VTK PIPELINE Source Raw data, shapes Filter Transform raw data Mapper Map data to geometry Renderer Render the geometry

19 OPENGL From the CUDA perspective OpenGL

20 OPENGL: API FOR GPU ACCELERATED RENDERING Primitives: points, lines, polygons Properties: colors, lighting, textures,.. View: camera position and perspective Shaders: Rendering to screen/framebuffer C-style functions, enums See e.g. What Every CUDA Programmer Should Know About OpenGL (

21 A SIMPLE OPENGL EXAMPLE glcolor3f(1.0f,0,0); State-based API (sticky attributes) glbegin(gl_quads); glvertex3f(-1.0f, -1.0f, 0.0f); // The bottom left corner glvertex3f(-1.0f, 1.0f, 0.0f); // The top left corner glvertex3f(1.0f, 1.0f, 0.0f); // The top right corner glvertex3f(1.0f, -1.0f, 0.0f); // The bottom right corner glend(); Drawing glflush(); Render to screen

22 glcolor3f(1.0f,0,0); glbegin(gl_quads); glvertex3f(-1.0f, -1.0f, 0.0f); glvertex3f(-1.0f, 1.0f, 0.0f); glvertex3f(1.0f, 1.0f, 0.0f); glvertex3f(1.0f, -1.0f, 0.0f); glend(); glflush();? float* vert={-1.0f, -1.0f,..}; float* d_vert; cudamalloc(&d_vert, n); cudamemcpy(d_vert, vert, n, cudamemcpyhosttodevice); renderquad<<<n/128, N>>>(d_vert); flushtoscreen<<<..>>>();

23 CUDA-OPENGL INTEROP: MAPPING MEMORY OpenGL: Opaque data buffer object Virtex Buffer Object (VBO) User has very limited control CUDA: C-style memory management User has full control CUDA-OpenGL Interop: Map/Unmap OpenGL buffers into CUDA memory space

24 display() init() RENDERING FROM A VBO Create VBO Initialize VBO Populate Render Display

25 display() init() RENDERING FROM A VBO WITH CUDA INTEROP Create VBO Initialize VBO Register VOB with CUDA Map VOB to CUDA Populate Unmap VBO from CUDA Render Display

26 MAIN OPENGL-CUDA INTEROP ROUTINES Register VBO for CUDA cudagraphicsglregisterbuffer(cuda_vbo, *vbo, flags); Mapping of VBO to CUDA cudagraphicsmapresources(1, cuda_vbo, 0); cudagraphicsresourcegetmappedpointer(&d_ptr, size, *cuda_vbo); Unmapping VBO from CUDA cudagraphicsunmapresources(1, cuda_vbo, 0);

27 CAN ALL GPUS SUPPORT OPENGL? GeForce : standard feature set including OpenGL 4.3/4.4 Quadro : + certain highly accelerated features (e.g. CAD) Tesla: If GPU is in All on operation mode * nvidia-smi -query-gpu=gom.current nvidia-smi q Graphics capabilities disabled *requires m or X class devices Graphics capabilities enabled

28 OPENGL SUMMARY + Relatively simple for basic viz + Existing/fixed/simple rendering pipeline - Low level - Triangles, points, rather than isosurfaces, height-fields - (currently) depends on Xserver for context creation - What about remote, parallel etc?

29 MORE OPENGL INFORMATION AT GTC S Multi-GPU Rendering, 03/24, 14:30, 210A S Tegra K1 Developer Tools for Android: Unleashing the Power of the Kepler GPU with NVIDIA's latest Developer Tools Suite, 03/24, 14:30, 210E S OpenGL 4.4 Scene Rendering Techniques, 3/25, 13:00, 210C S NVIDIA Path Rendering: Accelerating Vector Graphics for the Mobile Web, 3/25, 13:30, LL21C S OpenGL: 2014 and Beyond, 3/25, 14:00, 210C S Order Independent Transparency in OpenGL, 3/25, 210C

30 REMOTE VISUALIZATION

31 OPENGL CONTEXT State of an OpenGL instance Incl. viewable surface Interface to windowing system Context creation: platform specific Not part of OpenGL Handled by Xserver in Linux/Unix-like systems GLX: Interaction X<-> OpenGL

32 X11 Events Commands LOCAL RENDERING Application libgl Xlib GLX OpenGL 2D/3D X Server Driver GPU, monitor attached

33 X-FORWARDING: THE SIMPLEST FORM OF REMOTE RENDERING libgl Application Xlib X11 Commands X11 Events OpenGL/GLX On remote system: export DISPLAY= :0.0 2D/3D X Server Driver Network GPU, monitor attached

34 X-FORWARDING + Simple! + ssh X + No need to run X on remote machine - Lots of data crosses the network - All rendering performed on the local Xserver - Not useful for visualization of remote CUDA Interop apps (X-server doesn t see remote GPU)

35 SERVER-SIDE RENDERING + REMOTE VIZ APPLICATION Application libgl Xlib OpenGL GLX X11 Events 2D/3D X Server X11 Cmds Client Driver GPU Network Driver GPU, monitor attached

36 Scraper SERVER-SIDE RENDERING + SCRAPING Application libgl Xlib OpenGL GLX X11 Events 2D/3D X Server X11 Cmds Images Client Driver GPU Network Driver GPU, monitor attached

37 SERVER-SIDE RENDERING + SCRAPING + Full GPU acceleration + No X server on client Question: when to scrape? Xserver not informed about direct rendering Intercept glxswapbuffers() - Not multi-user => Occasionally used for remote desktop tools

38 GLX FORKING: OUT-OF-PROCESS Application OpenGL/ GLX OpenGL libgl Images X11 Events Proxy X Server GLX 3D X Server Xlib X11 Cmds Images Client App Driver GPU Network Driver GPU, monitor attached

39 GLX FORKING: OUT-OF-PROCESS + Full GPU acceleration + No X server on client + Multi-User - All traffic through Proxy X server => Occasionally used for remote desktop tools

40 GLX FORKING WITH INTERPOSER LIBRARY Application libgl VirtualGL Xlib GLX 3D X Server VGL transport (compressed) X11 Commands X11 Events OpenGL Images Images VirtualGL client 2D X Server Driver GPU Network Driver GPU, monitor attached

41 GLX FORKING WITH INTERPOSER LIBRARY Application libgl VirtualGL Xlib OpenGL Images GLX X11 Events Proxy X Server X11 Cmds Images 3D X Server Client App Driver GPU Network Driver GPU, monitor attached

42 VIRTUAL GL + TURBOVNC + Compressed image transport + Transparent to the application + Fully GPU accelerated OpenGL + Client with or without Xserver - Requires Xserver to access GPU

43 HOW TO SET UP VIRTUAL GL + TURBOVNC Requires Xserver running on server Root privileges for Xserver Requires installation of VirtualGL, TurboVNC on server Start VirtualGL-accelerated VNC server vncserver :3 TurboVNC viewer on client Linux, Windows, Javascript

44 CONNECTING CLIENT TO REMOTE SERVER

45 CONNECTING TO REMOTE VNC SERVER VIA A GATEWAY Establish tunnel on client ssh L 3333:node01.cluster.net:5903 login.cluster.net Connect client to localhost:3333 port 5903 port 3333 Gateway node01.cluster.net login.cluster.net client

46 LAUNCHING REMOTE CUDA/OPENGL APPLICATIONS VIA VGLRUN vglrun :3 glxgears export DISPLAY=:3 vglrun simplegl

47 REMOTE VIZ IN PARAVIEW/VISIT Approach 1: VirtualGL and VNC export DISPLAY=:3 vglrun paraview Paraview Client & Server under VirtualGL Approach 2: Local Client On remote server: pvserver On local workstation: paraview

48 REMOTE VISUALIZATION SUMMARY Multiple approaches to remote rendering X forwarding, remote viz app, scraping, interposer process/library Currently requires X server to generate context EGL will fix this, but requires application changes

49 PARALLEL VISUALIZATION Parallel Viz

50 PARALLEL VISUALIZATION Parallelism at multiple levels Filtering Rendering - Both supported by VisIt & Paraview - Heavy lifting already done! - Challenge: Setup in parallel environment - Both tools provide support for most common cases - Both VisIt & Paraview MPI parallel -> need custom build

51 BASIC STEPS TO PARALLEL VISUALIZATION Launch visualization server processes Most likely through queuing system Connect client to head node Tunnel from workstation Setting up virtualgl on remote node

52 DOMAIN DECOMPOSITION OF DATA - Visualization algorithms can work on decomposed data - May require ghost cells - May lead to load imbalance No ghost cells required Ghost cells required

53 PARALLEL COMPOSITING - Distributed geometry - Render with depth information - Composition using IceT

54 PARALLEL VISUALIZATION Particularly important for large datasets Visualization time often determined by filtering Rendering important if Highly complex visualization Complex visualization effects Low-power CPU Transparent support in Paraview, VisIt

55 IN-SITU VISUALIZATION & STEERING

56 BENEFIT OF IN-SITU VISUALIZATION Pipeline simulation cycle Visualization/analysis integral part of simulation Immediate feedback Reduce pressure on file system In some (future) cases: only way to analyze/visualize data

57 LIBSIM IN VISIT: VISIT SERVER AS A LIBRARY

58 CONNECTING TO A RUNNING APPLICATION

59 BUILD VISUALIZATION PIPELINE FOR RUNNING APPLICATION

60 BASIC INTERACTION/STEERING WITH RUNNING APPLICATION

61 INTERACTIVE VIZ APPLICATION WITH LIBSIM User implements callbacks for VisIt meta-data, mesh, variables and domains GetData: VisIt requests data for visualization Work directly on simulation data Transfer data from GPU Command server: Interaction VisIt front-end <-> simulation Steering

62 THE FUTURE The Future

63 VISUALIZATION DATA FLOW Filtering Rendering CPU Simulation GPU

64 VISUALIZATION WITH GPU ACCELERATED FILTER Filtering Rendering CPU Simulation Filtering GPU

65 VISUALIZATION WITH GPU ACCELERATED FILTER AND HARDWARE RENDERING CPU Filtering Simulation Filtering Rendering GPU

66 FULLY GPU ACCELERATED VISUALIZATION CPU Simulation Filtering Rendering GPU

67 SDAV SCIDAC-3 INSTITUTE ON VIZ/ANALYSIS AT SCALE ( ) Management, Analysis, Visualization In-situ analysis, indexing/compression I/O-, Viz frameworks Supporting application teams SDAV tools deployment: Paraview, VisIt, IceT,..

68 SDAV SCIDAC-3 INSTITUTE ON VIZ/ANALYSIS AT SCALE ( ) PISTON (LANL): Data Parallel Visualization Operators Isosurface, cut, threshold Built on top of Thrust Support of most Paraview operators Incorporation into VTK DAX (Sandia), DIY (Argonne), EVAL (ORNL) S Productive Programming with Descriptive Data: Efficient Mesh-Based Algorithm Development in EAVL DAX: A Massively Threaded Visualization and Analysis Toolkit for Extreme Scale

69 VIZ TALKS AT GTC2014 S Applications of GPU Computing to Mission Design and Satellite Operations at NASA's Goddard Space Flight Center Abel Brown ( Principal Systems Engineer, A.I. Solutions S Exploring the Earth in 3D: Multiple GPUs for Accelerating Inverse Imaging S Productive Programming with Descriptive Data: Efficient Mesh-Based Algorithm Development in EAVL S Dax: A Massively Threaded Visualization and Analysis Toolkit for Extreme Scale S Visualization and Analysis of Petascale Molecular Simulations with VMD S Now You See It: Unmasking Nuclear and Radiological Threats Around the World S Gesture-Based Interactive Visualization of Large-Scale Data using GPU and Latest Web Technologies S Scientific Data Visualization on GPU-Enabled, Hybrid HPC Systems S An Adventure in Porting: Adding GPU-Acceleration to Open-Source 3D Elastic Wave Modeling S Interactive Processing and Visualization of Geospatial Imagery S Live, Interactive, In-Situ, In-GPU Visualization of Plasma Simulations Running on GPU Supercomputers S Petascale Molecular Ray Tracing: Accelerating VMD/Tachyon with OptiX S Extreme Machine Learning with GPUs

70 SUMMARY Different methods of visualization at different levels OpenGL, VTK, VisIt & Paraview Remote visualization concepts X forwarding, Viz app, scraping, GLX forking Parallel rendering and compositing Handled transparently in key tools In-situ visualization concepts Expose simulation variables to visualization tool, interactive viz Future directions GPU accelerated filtering and rendering Thanks to Gilles Fourestey (CSCS), Nina Suvanphim (Cray), Jean Favre (CSCS), Adam DeConinck (NVIDIA), Robert Crovella (NVIDIA), Dale Southard (NVIDIA), Hank Childs (U Oregon), Jeremy Meredith (ORNL), Ian Williams (NVIDIA), Steve Parker (NVIDIA), Kitware, Paraview, VisIt and VirtualGL developers for their support!

71 ENJOY THE CONFERENCE AND SEE YOU AT GTC 2015!

72 ABSTRACT (FOR REFERENCE ONLY) Learn how to take advantage of GPUs to visualize results of your GPU-accelerated simulation! This session will cover a broad range of visualization and analysis techniques allowing you to investigate your data on the fly. Starting with some basic CUDA/OpenGL interoperability, we will introduce more sophisticated data models allowing you to take advantage of widely used tools like ParaView and VisIt to visualize your GPU resident data. Questions like parallel compositing, remote visualization and application steering will be addressed in order to allow you to take full advantage of the GPUs installed in your supercomputing system.