Scalable high performance visualization on Discovery Cluster.

Transcription

1 Scalable high performance visualization on Discovery Cluster. The session is structured as follows: Need for Parallelization in Rendering and Visualization The problem of large data sets Solutions Available software on Discovery for parallel distributed rendering Using GPU queue for scalable rendering and visualization Using general compute queue for scalable rendering and visualization Some example runs on the cluster GPU queue, general compute queue Questions Northeastern University Research Computing: Nilay K Roy, MS Computer Science, Ph.D Computational Physics

2 Why Parallelization in Rendering and Visualization?

3

4

5

6

7

8 Pipelining v/s Parallelism Pipelining: the problem is decomposed into individual steps that are mapped onto processors. Data travel through the processors and are transformed by each stage in the pipeline. Pipelining: offers only a limited amount of parallelism Pipelining: difficult to achieve good load-balancing amongst the processors in the pipeline as the different functions in the pipeline vary in computational complexity Pipelining: offers only a limited amount of parallelism Pipelining: for many problems like rendering pipeline, such a partitioning is natural Solution: To overcome such constraints pipelining is often augmented by replicating some or all pipeline stages Data are distributed amongst those processor and worked on in parallel If the algorithms executed by each of the processors are identical, the processors can perform their operation in lockstep, thus forming a SIMD (single-instruction, multipledata) engine If the algorithms contain too many data dependencies thus making SIMD operation inefficient, MIMD (multiple-instruction, multiple-data) architectures are more useful

9 Object Partitioning v/s Image Partitioning Object-space partitioning is commonly used for the geometry processing portion of the rendering pipeline, as its operation is intrinsically based on objects The data objects distributed amongst parallel processors belonged into object space, e.g. polygons, edges, or vertices Most parallelization strategies for rasterizers employ image-space partitioning Rasterisation: task of taking an image described in a vector graphics format (shapes) and converting it into a raster image (pixels or dots) for output on a video display or printer, or for storage in a bitmap file format Compared with other rendering techniques such as ray tracing, rasterisation is extremely fast. However, rasterization is simply the process of computing the mapping from scene geometry to pixels and does not prescribe a particular way to compute the color of those pixels. Shading, including programmable shading, may be based on physical light transport, or artistic intent.

10

11

12

13 Classification by Sorting a) Sort First b) Sort Middle c) Sort Last Considering the two main steps in rendering, i.e. geometry processing and rasterization, there are three principal locations for the sorting step: 1. Early during geometry processing (sortfirst) 2. Between geometry processing and rasterization (sort-middle) 3. And after rasterization (sort-last).

14 Load Balancing: Tasks are the basic units of work that can be assigned to a processor, e.g. objects, primitives, scanlines, pixels etc. Granularity quantifies the minimum number of tasks that are assigned to a processor, e.g. 10 scanlines per processor or 128x128 pixel regions. Coherence describes the similarity between neighboring elements like consecutive frames or neighboring scanlines. Coherence is exploited frequently in incremental calculations, e.g. during scan conversion. Parallelization may destroy coherence, if neighboring elements are distributed to different processors. Load balance describes how well tasks are distributed across different processor with respect to keeping all processors busy for all (or most) of the time

15 Workload Characterization: Clipping. Objects clipped by the screen boundaries incur more work than objects that are trivially accepted or rejected. It is difficult to predict whether an object will be clipped and load-imbalances can result as a consequence of one processor receiving a disproportionate number of objects requiring clipping. Tesselation. Some rendering APIs use higher order primitives, like NURBS, that are tesselated by the rendering subsystem. The degree of tesselation, i.e. the number of triangles per object, determines the amount of data expansion occurring during the rendering process. The degree of tesselation is often view-dependent and hence hard to predict a priori. Primitive distribution. In systems using image-space partitioning, the spatial distribution of objects across the screen decides how many objects must be processed by each processor. Primitive size. The performance of most rasterization algorithms increases for smaller primitives. (It takes less time to generate fewer pixels.) The mix of large and small primitives therefore determines the workload for the rasterizer.

16

17

18

19

20 Adaptive Load Balancing Algorithms: Usually performed in two steps: Load estimation (Count primitives per screen region, Estimate cost per primitive) Work distribution (Subdivide / tile screen to create regions with approx. equal load, or assign fixed-sized regions (cells) to processors to create approximate load balance. Some algorithms Roble s Method Whelan s Method Whitman s Method Muellers s Method (MAHD) Ellsworth Method Princeton Display Wall Method

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41 Rasterization v/s Ray tracing Real time issues GPU v/s FPGA PowerVR OpenRL ray tracing API Ray tracing's popularity stems from its basis in a realistic simulation of lighting over other rendering methods (such as scanline rendering or ray casting). Effects such as reflections and shadows, which are difficult to simulate using other algorithms, are a natural result of the ray tracing algorithm. The computational independence of each ray makes ray tracing amenable to parallelization. A serious disadvantage of ray tracing is performance. Scanline algorithms and other algorithms use data coherence to share computations between pixels, while ray tracing normally starts the process anew, treating each eye ray separately. However, this separation offers other advantages, such as the ability to shoot more rays as needed to perform spatial anti-aliasing and improve image quality where needed. True photorealism occurs when the rendering equation is closely approximated or fully implemented. Implementing the rendering equation gives true photorealism, as the equation describes every physical effect of light flow. However, this is usually infeasible given the computing resources required.

42 DISCOVERY CLUSTER Software on Discovery Cluster: 1) VisIt ( 2) ParaView ( 3) Vaa3D ( 4) Tachyon ( 5) Other options include parallel visualization and rendering features in commercial packages on Discovery Cluster like Matlab, Mathematica, SAS, Ansys-Fluent. Hardware on Discovery Cluster: 1) GPU Queue 2) Hadoop HDFS 50TB 3) Large Mem Queue 4) General Purpose Queues 10G and restriced IB

43 LETS FOCUS ON PARAVIEW for now and for runs on the DISCOVERY CLUSTER

44 Renato N. Elias, NACAD/COPPE/UFRJ, Rio de Janerio, Brazil Jerry Clarke, US Army Research Laboratory Swiss National Supercomputing Centre Bill Daughton, LANL Swiss National Supercomputing Centre

45 ParaView Application Architecture ParaView Client pvpython ParaWeb Catalyst Custom App UI (Qt Widgets, Python Wrappings) ParaView Server VTK OpenGL MPI IceT Etc.

46 ParaView Architecture Three tier Data Server Render Server Client

47 Standalone Client Data Server Render Server

48 Client-Server Data Server Render Server Client

49 Client-Render Server-Data Server Data Server Render Server Client

50 Connecting to a ParaView Server

51 Data Parallel Pipelines Duplicate pipelines run independently on different partitions of data.

52 Data Parallel Pipelines Duplicate pipelines run independently on different partitions of data.

53 Data Parallel Pipelines Some operations will work regardless. Example: Clipping.

54 Data Parallel Pipelines Some operations will work regardless. Example: Clipping.

55 Data Parallel Pipelines Some operations will work regardless. Example: Clipping.

56 Data Parallel Pipelines Some operations will have problems. Example: External Faces

57 Data Parallel Pipelines Some operations will have problems. Example: External Faces

58 Data Parallel Pipelines Ghost cells can solve most of these problems.

59 Data Parallel Pipelines Ghost cells can solve most of these problems.

60 Data Partitioning Partitions should be load balanced and spatially coherent.

61 Data Partitioning Partitions should be load balanced and spatially coherent.

62 Data Partitioning Partitions should be load balanced and spatially coherent.

63 Load Balancing/Ghost Cells Automatic for Structured Meshes. Partitioning/ghost cells for unstructured is manual. Use the D3 filter for unstructured (Filters Alphabetical D3)

64 Job Size Rules of Thumb Structured Data Try for max 20 M cell/processor. Shoot for 5 10 M cell/processor. Unstructured Data Try for max 1 M cell/processor. Shoot for K cell/processor.

65 Avoiding Data Explosion Pipeline may cause data to be copied, created, converted. This advice only for dealing with very large amounts of data. Remaining available memory is low.

66 Topology Changing, No Reduction Append Datasets Append Geometry Clean Clean to Grid Connectivity D3 Delaunay 2D/3D Extract Edges Linear Extrusion Loop Subdivision Reflect Rotational Extrusion Shrink Smooth Subdivide Tessellate Tetrahedralize Triangle Strips Triangulate

67 Topology Changing, Moderate Reduction Clip Decimate Extract Cells by Region Extract Selection Quadric Clustering Threshold Similar: Extract Subset

68 Topology Changing, Dimension Reduction Cell Centers Contour Extract CTH Fragments Extract CTH Parts Extract Surface Feature Edges Mask Points Outline (curvilinear) Slice Stream Tracer

69 Adds Field Data Block Scalars Calculator Cell Data to Point Data Compute Derivatives Curvature Elevation Generate Ids Gen. Surface Normals Gradient Level Scalars Median Mesh Quality Octree Depth Limit Octree Depth Scalars Point Data to Cell Data Process Id Scalars Random Vectors Resample with dataset Surface Flow Surface Vectors Texture Map to Transform Warp (scalar) Warp (vector)

70 Total Shallow Copy or Output Independent of Input Annotate Time Append Attributes Extract Block Extract Datasets Extract Level Glyph Group Datasets Histogram Integrate Variables Normal Glyphs Outline Outline Corners Plot Global Variables Over Time Plot Over Line Plot Selection Over Time Probe Location Temporal Shift Scale Temporal Snap-to-Time- Steps Temporal Statistics

71 Special Cases Temporal Filters Temporal Interpolator Particle Tracer Temporal Cache Programmable Filter

72 Culling Data Reduce dimensionality early. Contour and slice see inside volumes. Prefer data reduction over extraction. Slice instead of Clip. Contour instead of Threshold. Only extract when reducing an order of magnitude or more. Can still run into troubles.

73 Culling Data Experiment with subsampled data. Extract Subset Use caution. Subsampled data may be lacking. Use full data to draw final conclusions.

74 Memory Inspector

75 Rendering Modes Still Render Full detail render. Interactive Render Sacrifices detail for speed. Provides quick rendering rate. Used when interacting with 3D view.

76 Level of Detail (LOD) Geometric decimation Used only with Interactive Render Original Data Default Decimation Maximum Decimation

77 3D Rendering Parameters Edit Settings, Render View

78 Parallel Rendering

79 Parallel Rendering

80 Tiled Displays

81 Image Size LOD ParaView s parallel rendering overhead proportional to image size. Can use smaller images for interactive rendering. Original Data Subsample Rate: 2 pixels Subsample Rate: 4 pixels Subsample Rate: 8 pixels

82 Parallel Rendering Parameters Edit Settings, Render View

83 Parameters for Large Data Use display lists for GPU, off for CPU. Try outline instead of decimated geometry. Also try lowering decimation factor. Avoid shipping large data back to client. Turn on subsampling. Image Compression on.

84 Parameters for Low Bandwidth Try larger subsampling rates. Try Zlib compression and fewer bits.

85 Parameters for High Latency Turn up Remote Render Threshold. Allow more data to go to client. Play with the LOD Threshold and LOD Resolution to control geometry sent to client. Turn on Use outline for LOD rendering if decimated geometry too big for client. Try turning on interactive render delay.

86 Vector format image captures File->Export Scene->EPS Dumps image out in vector (scalable) format ideal for inclusion in technical papers File->Export Scene->WebGL Dumps out a web page that you can view in a browser and easily share.

87 LETS RUN SOME PARAVIEW EXAMPLES ON DISCOVERY CLUSTER

88 QUESTIONS? ABOVE: Asteroid Golevka measures about 500 x 600 x 700 meters. In a CTH shock physics simulation on GPU nodes, a 10 Megaton explosion was initiated at the center of mass. The simulation ran for about 15 hours on 7200 nodes of Red Storm (Sandia National Labs) and provided approximately 0.65 second of simulated time. The resolution was 1 meter, with a 1 cubic kilometer mesh that contained 1.1 billion cells. The remarkable resolution of this simulation provides realism in crack formation and propagation not seen in lower-resolution models.