10/23/12 Data analysis and visualization topics Sergei MAURITS, ARSC HPC Specialist maurits@arsc.edu Schedule - Visualization of 3-D data, visualization package Vis5D, conversion of user data to.v5d format (viewable with Vis5D and IDV) 101: Introduction to animation, making frames with Vis5D, basic animation (Linux) animation & composing review (Mac OSX) 3-D Data Visualization - isosurfaces (close to natural appearance) - cross-sections (3-D ==> 2-D) - probes/sounding (convenience tools) - vectors, streamlines, particles paths - volume rendering (mix of all above) - time dependence (animation) - multi-variable sets (clutter problem) - miscellaneous contexts (CFD objects, terrain, geographical maps, etc.) - discrete (molecules) vs. continuous fields ==> NO UNIVERSAL SOLUTION A Few Popular Visualization Packages - AVS & AVS Express - IDL & IDL itools - Matlab - TecPlot - NCAR Graphics + NCL (semi-free) - Paraview/VTK (free) - IDV (free) - Vis5D (free) 1
10/23/12 Vis5D - developed in mid-90es, NASA grant (freeware) by Bill Hibbard and team for meteo data - classic Vis5D has been developed up to v.5.2 (v.4.7 is particularly stable) - NCAR extensions for SGI (stereo, >2GB sets) - GNU team took over in 2001-2, brought Vis5D+ to v.1.2 (1.3 beta), Linux + OSX (some) support - new: autoconfig compilation, misc. extensions - still actively used package (10-20 downloads/month) - WWW pages, documentation (Vis5D & Vis5D+), mail list, fan clubs, pretty developed user base Vis5D (cont.) - Vis5D (3D+time+multi-variable = 5D) is a full suite of volumetric visualization tools, created originally for meteorological data (somewhat defines Vis5D context) - multi-scalars + two (at the time) vector field, + stream lines, trajectories, + customizable map & terrain - exists as a stand-alone GUI and/or API suite (advanced) - some customization (although, limited in terms of new graphics) is possible through Tcl-scripting (screen shots, spinning, etc.) and custom user functions - new variables: (array syntax) and user functions Vis5D (cont.) - data I/O: conversion Fortran or C codes to Vis5D-format data base.v5d, commented templates are available at ( /projects/classes/vis5d/linux/arsc_vis5d_help/ convert/ -.v5d - cross-platform, effectively compressed (up to 1 byte/node/variable) 3-D format of semi-standard status - upon conversion,.v5d files can be rendered immediately - /Vis5D/doc and /Vis5D/man - on-line documentation in PDF, SGML, HTML (in the ARSC Linux environment search, e.g. for INDEX.HTM at /usr/local/pkg/vis5d/current/doc/... Vis5D: Hands-on Tutorial 1. Find directory /projects/classes/vis5d/linux/arsc_vis5d_help 2. Review README-files (README.first -general, README.environment - your Vis5D environment at ARSC, README.tutorial hands-on step-by-step tutorial) 3. Copy the entire content of the directory above to local HD under unique name (/scratch/vis5d_youruid) (see README.tutorial for details) 4. Proceed with step-by-step tutorial described in Section 2 in README.tutorial 2
10/23/12 Schedule - Visualization of 3-D data, visualization package Vis5D, conversion of user data to.v5d format (viewable with Vis5D and IDV) 101: Introduction to animation, making frames with Vis5D, basic animation (Linux) animation & composing (OSX) - consists of pixels, usually squares - resolution number of pixels, N x M - color = R,G,B [+ Alpha] - generally you need N*M*3 [*4] numbers to store graphical info a lot! - in majority of applications space saving is desired (or highly desired) - savings are possible either - by limiting the number of colors (discretization of R,G,B-space and intro of the color tables: R'G'B' => color #n) or - by limiting the shapes and departing from simple but costly NxM model 1. Compression by limiting colors to 2,4,8,16,..., 128, 256 (pseudocolor) - GIF, (also PNG-8) small file size, limiting color space - WEB, science, supports transparency, animation 2. Compression by limiting shapes JPEG, JPEG2000 flexible compression level ==> variable file size, lossy compression loss of graphics quality,artefacts Widely used in photo, video 3. No compression TIFF, XWD, BMP,..., PNG-24 highest image quality, LARGE file sizes formats of choice in professional graphics (cont.) 4. Both types of compressions (color, shape) are irreversible, (keeping uncompressed archival copy is a reasonable approach 5. Worst case scenario second pass of compression of already compressed JPEG. This way compression artefacts will be preserved (stealing file size) and magnified (stealing quality). 6. Better way keep uncompressed archival copy and compress it to different levels of quality. 7. Surprise 30% of JPEG compression is not so bad, sometimes (photo for Web) 10% is sufficient. Photoshop SAVE_FOR_WEB utility is very useful for visual determination of compression level 3
10/23/12 Example of compression levels (using Save_for_Web in Photoshop) for scientific graph with uniform background of 1700x1000 resolu- tion (close to HDTV, 1080p), just 16 colors were used to draw it TIFF JPEG - 5 150 000 B or 5.15 MB - no compression 314 100 B (max = 100%), subjectively, no artefacts 129 700 B (mid = 50%), subjectively, minor artefacts 70 400 B (min = 10%), GIF - subjectively, a lot of artefacts 133 200 (256 colors) - no artefacts, adequate colors 113 000 (64 colors) - no artefacts, adequate colors 85 360 (16 colors) - no artefacts, adequate colors 76 760 ( 8 colors) - color distortion starts here (16) From Wikipedia: is the rapid display of a sequence of images of 2D or 3-D artwork or model positions in order to create an illusion of movement. It is an optical illusion of motion due to the phenomenon of persistence of vision, and can be created and demonstrated in a number of ways. Standard frequency of TV-based animation is 30 frames per second (or fps). All software interfaces adopt it as a base rate. By simple repetition of your frames, you can depart from this rate, but not dramatically - 8-10 fps is the practical limit. This means, you can repeat your frames 3-4 - or 5 (may be) times, but not much more. (Duration example: 45 frames x 3 = 135 135/30 = 4.5 sec) From the compression standpoint, the animation frequency of 30 fps means that your graphics compression problem is 30 times worse than in case of the static graphics From http://www.apple.com/quicktime/technologies/h264/ Full HD uncompressed (4:4:4): 1920W x1080h x24bit x30 fr = or 2MP x24bit x30 fr = Mbps 1424 4
10/23/12 The good news animated stream has a lot of redundancy, its compression can be dramatically more effective than compression of the static graphics. Earlier coding techniques used singly predicted (P) frames depended only on previous independently coded frames (I) and bipredicted (B) frames, which are depended on a past and a future I or P frames. The current advanced codecs (H.264) are much more flexible, which improves quality and decreases bitrate for the given resolution Compression ratio 1:200 or increases resolution for the given bitrate 5
10/23/12 Schedule - Visualization of 3-D data, visualization package Vis5D, conversion of user data to.v5d format (viewable with Vis5D and IDV) 101: Introduction to animation, making frames with Vis5D, basic animation (Linux) animation & composing (OSX) practical approach - Make frames in sufficient quantities (30 fps) with any viz package, use large fonts and thick lines, to make the sequence UNIX-friendly for scripting, number without leading zeros (f_0001.png vs. f_1001.png) - Linux utilities animate+convert can crop, change formats & quality fast results, see READMEs in directory.../movies) - Vi Vis5D, - Mac OS (ARSC-supported) QuickTime 7 (compression, timing,outputs) - Final Cut Pro (industry standard) - Shake advanced composing - Windows DviX Pro (was AVI, uses the same codec H.264) basically the same functionality as in QuickTime Pro, DviX free player A is available for Mac, QuickTime 7 is available for Windows 6