Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains

Transcription

1 Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains Marco Di Benedetto PhD Thesis MIMOS 2012 Award Finalist Supervisors Paolo Cignoni¹,², Fabio Ganovelli¹ ¹ Visual Computing Laboratory ISTI CNR ² Università di Pisa

2 About Me Researcher at the Visual Computing Lab ISTI CNR, Pisa Directed by Roberto Scopigno Received a Ph.D. in Computer Science from Università di Pisa in 2011 Paolo Cignoni and Fabio Ganovelli as Supervisors My work is mostly focused on Realtime Rendering Huge Models Processing & Visualization Parallel Algorithms & GPU-Assisted Computations Computer Graphics on Mobile Devices Interested both in Academic Research and Industrial Technology Transfer a CNR Spin-Off can be smelled in the air... Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 2

3 Motivations - Scenario Graphics Processing Unit (GPU) Low-cost 3D hardware accelerators Many cores Programmable Move complex tasks from CPU to GPU Terrain & Urban Data Shape and appearance Increasing availability Increasing detail captured Huge Datasets, Hard-to-Handle Information Remote Exploration & The WWW Widespread, fast Internet connections New web technologies Planet-scale content delivery, HW acceleration in Web Pages Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 3

4 Motivations - Problems Real-World Datasets (more memory than we can handle) Huge memory footprint out-of-core preprocessing & rendering Large data movements cache levels, compression Interactive Visualization Systems (smooth exploration, responsiveness) Rendering must be fast GPU-efficient data structures & algorithms and interruptible represent the original data at different levels of detail (multiresolution) to cope with lack of computational resources, cache misses,... Urban Models (hard to represent at different levels of detail) Many small, simple shapes (buildings) with lot of associated image data state-of-the-art simplification algorithms fail, new data representation needed Computer Graphics and the Web (lack of efficient CG tools) Close the gap between CG and Web developers need for web-friendly data structures, fast and configurable software with shallow learning curve Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 4

5 Contribution Compressed Batched Dynamic Adaptive Meshes State-of-the-art, out-of-core, wavelet-based compression scheme for terrain data, real-time decompression & rendering on the GPU Data Compression BlockMaps State-of-the-art, multiresolution data representation for urban models, GPU-based construction and real-time rendering GPU Representation SpiderGL Computer Graphics data structures & algorithms for the Web, multiresolution framework, performance-oriented software Web Compute & Deploy Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 5

6 Outline Multiresolution Idea Data Compression and Rendering for Terrain Models A Multiresolution Representation for Urban Models Computer Graphics for the Web Platform Conclusions Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 6

8 Multiresolution: Intuitive Idea (2D) 1024 x x 64 Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 8

9 Multiresolution: Intuitive Idea (3D) 50K Triangles 1K Triangles Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 9

10 Multiresolution: Intuitive Idea (3D Perspective) Perspective foreshortening effect multiple LODs in the same image Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 10

11 Multiresolution Rendering Why Multiresolution Memory & Computiational restrictions No need to have more information than one can show in few pixels: better performances, less aliasing At any time some representation of the entire dataset should be presented to the user Data is represented at several levels of detail DAG, k-ary Tree hierarchies At rendering time, the most appropriate LOD is selected Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 11

13 Data Compression and Rendering for Terrain Models Two main approaches 1) Adaptive coarse refinement from out-of-core multiresolution data structures: PRO: very efficient in approximating planar and spherical domains, fast GPU updates CON: large multiresolution structure require out-of-core data management 2) In-core rendering from compressed pyramidal structures PRO: exploits structure regularity to reach very high compression rates, often in-core CON: limited to planar domains, ignores local adaptivity Our Goal - To combine the generality and adaptivity of chunked multiresolution structures with the compression rates of nested regular grid techniques Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 13

14 C-BDAM Compressed Batched Dynamic Adaptive Meshes Based on a right-triangles hierarchy Each triangle is actually a triangle-shaped batch of thousands of small triangles Right-triangles are arranged in diamonds to form a hierarchy Leaf nodes are subregions of the input Internal nodes are simplified versions of their children Multiresolution is achieved by selecting a cut in the DAG that meets a tradeoff between projected on-screen error and computational cost Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 14

15 C-BDAM: Compression Wavelet-based compression Can manage several surface attributes (normals, colors,...) Compression is an out-of-core process Each diamond is processed independently Synchronization at each level completion Wavelet compression produces residual matrices from high-resolution diamonds Resulting matrices are sparse Integer quantization Matrices are intepreted as quadtrees and compressed Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 15

16 C-BDAM: Rendering A cut is selected based on error and available resources Regular connectivity of triangle batches is shared among all diamonds Finer diamonds geometry is generated from coarser ones and residual matrices Fast Decompression Executed in parallel on the GPU Smaller CPUGPU Transfers Transferred data is compressed Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 16

17 Real-Time Rendering with C-BDAM State-of-the art in terrain rendering Up to 57:1 compression rates, real-time ( Hz on commodity hardware) Combine the generality and adaptivity of batched dynamic adaptive meshes with the compression rates of nested regular grid techniques Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 17

19 A Multiresolution Representation for Urban Models Data availability Cadastral maps, ad-hoc 3D scanning Aerial, street-level photographs Hundreds to Thousands of Buildings Simple shapes (prism-like, few polygons) Hi-res façade images Simplification methods Rely on dense meshes (not appliable) Buildings risk to be joined or collapsed Impostors fail to reproduce parallax motion What We Want - Output-sensitive visualization framework - Capture salient shape and appearance information - Maximize global environment perception - Real-time on commodity hardware Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 19

20 The BlockMap Data Structure Key Idea: to compactly encode in a GPU-friendly way both shape and appearance of a generic urban model sub-block Textured Geometry BlockMap Texture GPU-Generated Image Requirements: Fast rendering (real-time interaction) Hierarchical representation (multiresolution) Error control (quality / performances) Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 20

21 Intuitive Idea: Build a BlockMap Input Textured Geometry BlockMap 1. Project a checkerboard 2. Lift tiles to mimic geometry 3. Finer grid more details captured 4. Take snapshots to acquire color Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 21

22 BlockMap Components Walls Normals HeighMap Roofs Color Walls Color Walls AO Roofs Normals Walls Strips OffsetMap Roofs AO Ray-Casting Rooftops Façades Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 22

23 Intuitive Idea: Draw a BlockMap Observer s Perspective Ray intersection cases: 1. Draw block bounding box 2. Lay down heightmap on bottom face 3. Cast rays from observer to box a. Nothing hit: don t write pixel b. Hit a roof: fetch data from roofmap c. Hit a wall: fetch data from wallsmap Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 23

24 Ambient Occlusion Surface attributes: we can extract them but also compute others Ambient Occlusion How many sunlight reaches a point on surface Enhances shape perception Approximates Global Illumination Most perceived effect in urban contexts Computed on the GPU Math formulation: integral of light coming from all direction in the exposed surface hemisphere Approximation During virtual scan, keep track of how many times a surface point is visible Divide by the number of scans Computed on the GPU with shadow mapping Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 24

25 Dataset Processing Build a Hierarchy: embed source dataset in a quadtree No assumption on source representation (textured polygons, implicit surfaces, ) as long as we can draw it Data Acquisition: Virtual Scan Acquire from several viewpoints Leaf nodes built directly from input Internal nodes generated from children BlockMaps multiresolution as a side effect... Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 25

26 The BlockMap Pipeline Preprocessing Hierarchical Subdivision BlockMaps Generation BlockMaps DB BlockMaps BlockMaps Streaming DB Real-Time Visualization Rendering Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 26

27 Huge Datasets: Out-of-Core Datasets size exceeds memory capacity BlockMaps DB On-demand data loading Multithreaded network client and cache system (disk, system RAM, video RAM) Network Server Net Cache policy Ensure some data exists from any node to the root Priority as a function of timestamp, projection error and distance from observer Can handle several GB of data Network Client Disk Cache Sys RAM Cache Gfx RAM Cache Renderer Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 27

28 Rendering the BlockMap QuadTree (binary tree shown for clarity) Sort by distance (nearest first) Ideal Cut Above Error Below Error Not Visible Candidate Cache Miss Backup Render Cut Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 28

29 Results in Urban Models Visualization State-of-the art in multiresolution urban models rendering Novel compact and efficient representation, handles ground, roofs & vertical walls Effective simplification results where other methods fail Runs in real-time ( Hz) on commodity hardware Image-based storage, web-ready Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 29

30 Technology behind BlockMap Rendering Each BlockMap can be represented as a standard image (e.g. RGBA PNG) The whole dataset needs few numbers per tree node Node bounding box, number of strips in the Walls Map A text file is enhough Shaders do not use special instructions Executable by graphics hardware since 2006 Emerging 3D technologies for the WWW have enhough expressiveness Can we try to explore cities directly within web browsers using BlockMaps? Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 30

32 Computer Graphics for the Web Platform Many attempts have been done so far Proprietary solutions, browser extension,... Standardization needed WebGL allows the use of the graphics hardware inside web pages (low-level) Most CG software libraries tend to be too high-level Developers are forced to comply to some specific paradigm Software users want to reuse as much as possible of their former knowledge CG experts often find limitations due to high-level abstraction Objective - Close the gap between CG and Web developers - Ease the use of WebGL without introducing unnecessary abstraction - Provide the most common CG data structures & algorithms - Asynchronous data fetches utilities & Multiresolution framework Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 32

33 SpiderGL: 3D Graphics for the WWW Lightweight, open source library ( ) Relies on WebGL for real-time rendering Goals: Efficiency (asymptotic bounds are not the only concern) Simplicity & short learning time Flexibility (help with common tasks without preventing low-level access) Leading design considerations Do not impose any design choice a priori Avoid over abstraction Allow seamless cooperation with native API Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 33

34 SpiderGL Architecture GL: rendering Mesh: editable or renderable 3D model Space: geometric s. & a. Async: asinchronous data requests managing UI: user and html interface Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 34

35 Geometry, Assets, Asynch Fetches, UI Support for CG-related linear algebra (vectors, matrices, quaterions,...) Geometric entities (planes, spheres, rays, boxes,...) Geometric transformation chain Mesh editing Asynchronous assets loading from several formats Dynamic priority queues for multiresolution data fetches User interface interactors Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 35

36 Rendering Large Meshes The provided mesh module allows rendering of large models, transparently overcoming WebGL limitations Automatic chunk partitioning Efficient rendering paths with run-time code generation More than 200 Hz for 1Mtri meshes Hundreds of millions of triangles at 60 Hz with Nexus technology Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 36

37 Large Datasets on the Web Reduce the CPU load by executing complex tasks on the GPU Reduce the impact on performances of interpreted languages Use asynchronous communication model typical of web applications And exploit built-in browser cache Terrain datasets composed of millions of triangles are explorable in real-time on common laptops Multiresolution for Gigapixels image browsing Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 37

38 Bring Real-Time CG to the Web SpiderGL is being used by major web browsers developers as a test harness platform and by several open source projects Efficient library for standard CG algorithms and multiresolution techniques Does not introduce measurable execution overhead, short learning times Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 38

40 Conclusions Data Compression and Rendering for Terrain Models C-BDAM: high compression rations, real-time rendering Combine the generality and adaptivity of batched dynamic adaptive meshes with the compression rates of nested regular grid techniques Multiresolution for Urban Models BlockMap: new data representation for city blocks Captures salient shape and appearance information Enhanced environment perception with ambient occlusion Image-based data structure, real-time rendering Computer Graphics for the Web Platform SpiderGL: a fast and flexible library to close the gap between CG and Web developers Multiresolution framework Used as test harness by major browsers developers and in open source projects Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 40

41 Publications C-BDAM - Compressed Batched Dynamic Adaptive Meshes for Terrain Rendering Enrico Gobbetti, Fabio Marton, Paolo Cignoni, Marco Di Benedetto, Fabio Ganovelli Computer Graphics Forum, Volume 25, Number Ray-Casted BlockMaps for Large Urban Models Streaming and Visualization Paolo Cignoni, Marco Di Benedetto, Fabio Ganovelli, Enrico Gobbetti, Fabio Marton, Roberto Scopigno Computer Graphics Forum, Volume 26, Number Interactive Remote Exploration of Massive Cityscapes Marco Di Benedetto, Paolo Cignoni, Fabio Ganovelli, Enrico Gobbetti, Fabio Marton, Roberto Scopigno The 10th International Symposium on Virtual Reality, Archaeology and Cultural Heritage SpiderGL: A JavaScript 3D Graphics Library for Next-Generation WWW Marco Di Benedetto, Federico Ponchio, Fabio Ganovelli, Roberto Scopigno Web3D 2010, 15th Conference on 3D Web Technology - ACM Visualization Methods for Molecular Studies on the Web Platform Marco Callieri, Raluca Mihaela Andrei, Marco Di Benedetto, Monica Zoppé, Roberto Scopigno Web3D 2010, 15th Conference on 3D Web Technology - ACM SpiderGL: A Graphics Library for 3D Web Applications Marco Di Benedetto, Massimiliano Corsini, Roberto Scopigno Proceedings of the 4th ISPRS International Workshop 3D-ARCH Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 41

42 Thank You! Questions? Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 42

43 Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 43

44 Building a BlockMap: Ray-Casting Structures HeightMap Approximate terrain and buildings shape Acquired by sampling depth from above OffsetMap Detect discontinuities on the heightmap Parametrizes façades Used to index walls colors/normals columns in the WallsMap Strips offset Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 44

45 Building a BlockMap: Shading Structures Rooftops Attributes Color [R, G, B] Ambient Occlusion Normal [Nx, Ny] Walls Strips Attributes Color [R, G, B] Ambient Occlusion Normal [Nx, Ny, Nz] Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 45

46 Building a BlockMap: Attributes Acquisition Render Targets 1. Draw Geometric, Material and Parametrization data to texture targets Color Target Parametrization Target. Superimpose the two targets and draw one point primitive per pixel 3. Scatter: redirect points to final destination & output (combine) fetched data WallsMap Stacked Walls: better W:H ratio Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 46

47 SpiderGL Performances JavaScript just-in-time compilers allow complex models to be visualized in real time For 3D-heavy applications, the bottleneck is the graphics hardware SpiderGL rendering process guarantees the minimum number of graphics API calls 3D drivers are optimized for that Performances Models with tens of thousands of triangles can be rendered at more than 1K Hz (strict realtime is 30 Hz!) More than 100 Hz for 1 millions of triangles Multiresolution terrain and BlockMaps visualization at 200 Hz Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 47

48 Future Work Investigating different compression strategies in C-BDAM Enhancing visual feedback in BlockMaps with precomputed radiance BlockMaps contruction completely delegated to GPU Extending SpiderGL with more complex structures for scene description Working on an automatic pipeline that allows to create multiresolution, web-friendly datasets for generic 3D models to be visualized on standard browsers Marco Di Benedetto - Multiresolution Techniques for Real-Time Visualization of Urban Environments and Terrains 48