3D Videocommunication

Size: px

Start display at page:

Download "3D Videocommunication"

Godwin Stafford
8 years ago
Views:

1 3D Videocommunication Algorithms, concepts and real-time systems in human centred communication EDITED BY Oliver Schreer Fraunhofer Institute for Telecommunications Heinrich-Hertz-Institut, Berlin, Germany Peter Kauff Fraunhofer Institute for Telecommunications Heinrich-Hertz-Institut, Berlin, Germany Thomas Sikora Technical University Berlin, Germany John Wiley & Sons, Ltd

Heinrich-Hertz-Institut, Berlin, Germany Peter Kauff Fraunhofer Institute for

2 Contents List of Contributors Symbols Abbreviations xin xix xxi Introduction Oliver Schreer, Peter Kauff and Thomas Sikora Section I Applications of 3D Videocommunication 5 1 History of Telepresence Wijnand A. Usselsteijn. 1 Introduction.2 The Art of Immersion: Barker's Panoramas.3 Cinerama and Sensorama.4 Virtual Environments.5 Teleoperation and Telerobotics.6 Telecommunications.7 Conclusion References 3D TV Broadcasting Christoph Fehn 2.1 Introduction 2.2 History of 3D TV Research 2.3 A Modern Approach to 3D TV A Comparison with a Stereoscopic Video Chain 2.4 Stereoscopic View Synthesis D Image Warping A 'Virtual' Stereo Camera The Djsocclusion Problem 2.5 Coding of 3D Imagery Human Factor Experiments

7 Conclusion References 3D TV Broadcasting Christoph Fehn 2.1 Introduction 2.2 History of 3D TV Research 2.3 A Modern Approach to 3D TV 2.3.1 A Comparison with a Stereoscopic Video Chain 2.

3 vi CONTENTS 2.6 Conclusions 36 Acknowledgements 37 References D in Content Creation and Post-production 39 Oliver Grau 3.1 Introduction Current Techniques for Integrating Real and Virtual Scene Content Generation of 3D Models of Dynamic Scenes Implementation of a Bidirectional Interface Between Real and Virtual Scenes Head Tracking View-dependent Rendering Mask Generation Texturing Collision Detection Conclusions 52 References 52 4 Free Viewpoint Systems 55 Masayuki Tanimoto 4.1 General Overview of Free Viewpoint Systems Image Domain System EyeVision D-TV Free Viewpoint Play Ray-space System FTV (Free Viewpoint TV) -^ Bird's-eye View System Light Field Video Camera System Surface Light Field System Model-based System D Room D Video Multi-texturing Integral Photography System NHK System D-II 3D Display System Summary 70 References 71 5 Immersive Videoconferencing 75 Peter Kauff and Oliver Schreer 5.1 Introduction The Meaning of Telepresence in Videoconferencing Multi-party Communication Using the Shared Table Concept 79

4 Implementation of a Bidirectional Interface Between Real and Virtual Scenes 46 3.4.1 Head Tracking 49 3.4.2 View-dependent Rendering 50 3.4.3 Mask Generation 50 3.4.4 Texturing 51 3.4.5 Collision Detection 52 3.

4 CONTENTS vii 5.4 Experimental Systems for Immersive Videoconferencing Perspective and Trends 87 Acknowledgements. 88 References 88 Section II 3D Data Representation and Processing 91 6 Fundamentals of Multiple-view Geometry 93 Spela Ivekovic, Andrea Fusiello and Emanuele Trucco 6.1 Introduction Pinhole Camera Geometry Two-view Geometry Introduction Epipolar Geometry Rectification D Reconstruction JV-view Geometry Trifocal Geometry The Trifocal Tensor Multiple-view Constraints Uncalibrated Reconstruction from N views Autocalibration Summary 112 References Stereo Analysis 115 Nicole Atzpadin and Jane Mulligan 7.1 Stereo Analysis Using Two Cameras Standard Area-based Stereo Analysis " Fast Real-time Approaches Post-processing Disparity From Three or More Cameras Two-camera versus Three-camera Disparity Correspondence Search with Three Views Post-processing Conclusion 130 References Reconstruction of Volumetric 3D Models 133 Peter Eisert 8.1 Introduction Shape-from-Silhouette Rendering of Volumetric Models Octree Representation of Voxel Volumes Camera Calibration from Silhouettes Space-carving 140

2 Pinhole Camera Geometry 94 6.3 Two-view Geometry 96 6.3.1 Introduction 96 6.3.2 Epipolar Geometry 97 6.3.3 Rectification 102 6.3.4 3D Reconstruction 104 6.4 JV-view Geometry 106 6.4.1 Trifocal Geometry 106 6.

5 viii CONTENTS 8.4 Epipolar Image Analysis Horizontal Camera Motion Image Cube Trajectory Analysis Conclusions 148 References View Synthesis and Rendering Methods 151 Reinhard Koch and Jan-Friso Evers-Senne 9.1 The Plenoptic Function Sampling the Plenoptic Function Recording of the Plenoptic Samples Categorization of Image-based View Synthesis Methods Parallax Effects in View Rendering Taxonomy of IBR Systems Rendering Without Geometry The Aspen Movie-Map ' Quicktime VR Central Perspective Panoramas Manifold Mosaicing Concentric Mosaics Cross-slit Panoramas Light Field Rendering Lumigraph Ray Space Related Techniques Rendering with Geometry Compensation Disparity-based Interpolation ' Image Transfer Methods -^ ' Depth-based Extrapolation Layered Depth Images Rendering from Approximate Geometry, Planar Scene Approximation View-dependent Geometry and Texture Recent Trends in Dynamic IBR ' 170 References D Audio Capture and Analysis 175 Markus Schwab and Peter Noll 10.1 Introduction Acoustic Echo Control Single-channel Echo Control Multi-channel Echo Control Sensor Placement Acoustic Source Localization Introduction Real-time System and Results 183

2 Categorization of Image-based View Synthesis Methods 154 9.2.1 Parallax Effects in View Rendering 154 9.2.2 Taxonomy of IBR Systems 156 9.3 Rendering Without Geometry 158 9.3.1 The Aspen Movie-Map ' 158 9.

6 CONTENTS ix 10.5 Speech Enhancement Multi-channel Speech Enhancement Single-channel Noise Reduction Conclusions 190 References Coding and Standardization. 193 Aljoscha Smolic and Thomas Sikora 11.1 Introduction Basic Strategies for Coding Images and Video Predictive Coding of Images Transform Domain Coding of Images and Video Predictive Coding of Video Hybrid MC/DCT Coding for Video Sequences Content-based Video Coding Coding Standards JPEG and JPEG Video Coding Standards MPEG-4 an Overview MPEG-4 Systems BIFS ' Natural Video Natural Audio SNHC AFX The MPEG 3DAV Activity Omnidirectional Video Free-viewpoint Video Conclusion ^- 214 References 214 Section III 3D Reproduction Human Factors of 3D Displays 219 Wijnand A. Usselsteijn, Pieter J.H. Seuntiens and Lydia M.J. Meesters 12.1 Introduction Human Depth Perception Binocular Disparity and Stereopsis Accommodation and Vergence Asymmetrical Binocular Combination Individual Differences Principles of Stereoscopic Image Production and Display Sources of Visual Discomfort in Viewing Stereoscopic Displays Keystone Distortion and Depth Plane Curvature Magnification and Miniaturization Effects - 228

2.3 Predictive Coding of Video 198 11.2.4 Hybrid MC/DCT Coding for Video Sequences 199 11.2.5 Content-based Video Coding 201 11.3 Coding Standards 202 11.3.1 JPEG and JPEG 2000 202 11.3.2 Video Coding Standards 202 11.

7 CONTENTS Shear Distortion Cross-talk ' :4.5 Picket Fence Effect and Image Flipping Understanding Stereoscopic Image Quality 230 References D Displays 235 Siegmund Pastoor 13.1 Introduction Spatial Vision Taxonomy of 3D Displays Aided-viewing 3D Display Technologies Colour-multiplexed (Anaglyph) Displays Polarization-multiplexed Displays Time-multiplexed Displays Location-multiplexed Displays Free-viewing 3D Display Technologies Electroholography Volumetric Displays Direction-multiplexed Displays Conclusions 258 References Mixed Reality Displays 261 Siegmund Pastoor and Christos Conomis 14.1 Introduction Challenges for MR Technologies Human Spatial Vision and MR Displays Visual Integration of Natural and Synthetic Worlds Free-form Surface-prism HMD Waveguide Holographic HMD Virtual Retinal Display Variable-accommodation HMD Occlusion Handling HMD Video See-through HMD Head-mounted Projective Display Towards Free-viewing MR Displays Examples of Desktop and Hand-held MR Systems Hybrid 2D/3D Desktop MR System with Multimodal Interaction Mobile MR Display with Markerless Video-based Tracking Conclusions 278 References 279

4 Aided-viewing 3D Display Technologies 238 13.4.1 Colour-multiplexed (Anaglyph) Displays 238 13.4.2 Polarization-multiplexed Displays 239 13.4.3 Time-multiplexed Displays 239 13.4.4 Location-multiplexed Displays 240 13.

8 CONTENTS xi 15 Spatialized Audio and 3D Audio Rendering. 281 Thomas Sporer and Sandra Brix 15.1 Introduction ', Basics of Spatial Audio Perception Perception of Direction Perception of Distance The Cocktail Party Effect ' Final Remarks Spatial Sound Reproduction Discrete Multi-channel Loudspeaker Reproduction Binaural Reproduction Multi-object Audio Reproduction Audiovisual Coherence Applications Summary and Outlook 293 References 293 Section IV 3D Data Sensors Sensor-based Depth Capturing Joao G.M. Goncalves and Vitor Sequeira 16.1 Introduction Triangulation-based Sensors Time-of-flight-based Sensors Pulsed Wave Continuous-wave-based Sensors Summary Focal Plane Arrays " Other Methods Application Examples The Way Ahead Summary 311 References Tracking and User Interface for Mixed Reality 315 Yousri Abdeljaoued, David Marimon i Sanjuan, and Touradj Ebrahimi Introduction Tracking Mechanical Tracking Acoustic Tracking \ Inertial Tracking Magnetic Tracking Optical Tracking Video-based Tracking Hybrid Tracking

287 15.4 Audiovisual Coherence 291 15.5 Applications 293 15.6 Summary and Outlook 293 References 293 Section IV 3D Data Sensors 297 16 Sensor-based Depth Capturing - 299 Joao G.M.

9 xii CONTENTS ] 7.3 User Interface Tangible User Interfaces Gesture-based Interfaces Applications Mobile Applications Collaborative Applications ' Industrial Applications Conclusions 331 References ' 331 Index 335

4.2 Collaborative Applications 329 17.4.3' Industrial Applications 329 17.

INDEX 337. Generalized sidelobe-canceller (GSC) 186 Gestures 76, 79 Ghosting see Cross-talk

INDEX 337. Generalized sidelobe-canceller (GSC) 186 Gestures 76, 79 Ghosting see Cross-talk Index 1D-II 70 2D morphing 165 2D-to-3D conversion 28 3D consortium 26 3DAV 26, 194, 204, 210, 211, 212, 214 3D display(s) aerial image 252 aided viewing 237, 238 applications 235 autostereoscopic 28 barrier-grid