Fraunhofer-Institut für Nachrichtentechnik Heinrich-Hertz-Institut Ralf Schäfer schaefer@hhi.de http://ip.hhi.de Immersive Medien und 3D-Video page 1
Outline Immersive Media Examples Interactive Media and Free Navigation Free Navigation in Video Scenes Examples Free Viewpoint Video Summary page 2
Immersive Media Costs vs. Sensation Costs Real Live Sensation IMAX American Football Cinema Theatre Rock Concert CD VoD VCR TV Interactive TV Immersive Media Sensation page 3
Immersive Display (UHDTV, D-Cinema) page 4
Example: Immersive Video Conferencing im.point Cameras 61 Plasma display Audio Speakers Semi-circular table Seamless transition between the real and virtual world Head motion parallax Life-sized upper body images representing gestures and body language 3D representation of the remote participants including provision of eye contact page 5
Example: Largest Panorama Worldwide Gasometer in Leipzig Printed panorama with 60 m diameter and 36 m hight (weight > 1 ton) page 6
page 7
Cylindrical Projection with HHI s CineCardultra Single level Multiple levels υ CineCardultra: Horizontal & vertical soft edge blending Warping for cylindrical and parabolic projection υ page 8
Interactive Media Allow to be active in some way instead of just being a passive consumer Enabled through convergence on telecommunications, consumer electronics and computer science Well-known: linking of multimedia data for interactive access Multimedia DVD, Internet presentations, Multimedia Home Platform (MHP) page 9
Free Navigation Arbitrary selection of viewpoint and direction within real world audio-visual scenes Well-known from computer graphics, virtual reality, computer games synthetic scenes, objects or static views of real objects or restricted to predefined viewpoints (e.g. Formula 1 on Premiere) page 10
Free Navigation in Real Scenes New technology allows generation of intermediate views in between real existing camera view Image-Based rendering, 3D reconstruction Allows e.g. viewing sports or theatre from arbitrary viewpoints New technologies for the whole chain: Acquisition, signal processing, modelling, representation, compression, transmission, rendering, display page 11
Scene Representations More geometry -based More image-based Static Texture, Single Geometry View-dep. Texture, Single Geometry Static Texture, View-dep. Geometry View-dep. Texture, View-dep. Geometry Layered - Depth images Light-field rendering Incarnations of 3D model Sprites Lumigraph Conventional graphics pipeline Warping Interpolation page 12
Omni-directional Video Systems with curved mirrors www.remotereality.com page 13
Omni-directional Video Example 1 Courtesy of Prof. Yokoya, Nara Institute of Science and Technology page 14
Omni-directional Video Example 2 page 15
Interactive Navigation in 3D Panoramas Patches page 16
Creation of 3D Panoramas By using several shots of a still camera and stitching techniques By using video and mosaiquing techniques page 17
Interactive Decoding and Rendering Tiles can be streamed as JPEG or JPEG2000 files Current view on screen Visibility sensor page 18
Free Viewpoint Video (FVV( FVV) Capturing a real dynamic scene with N cameras Transformation into a special data representation Interpolation of arbitrary intermediate views Allows free navigation within the scene (within practical limits) page 19
Example: Eye Vision using tracking cameras Virtual Camera Path X page 20
Eye Vision No reconstruction of 3D geometrie => many cameras No individual interaction possible Live broadcast event: Super Bowl 2001, 30 cameras Courtesy of Prof. Takeyo Kanade, Carnegie Mellon Univ. page 21
Visual Hull Reconstruction Synchronized multi-video recording Background subtraction Silhouette extraction Visual Hull reconstruction page 22
Example: Temple Acquisition by Photographs page 23
Processing Step: : 3D Mesh Extraction page 24
Processing Step: Texturing page 25
Multi camera video acquisition Courtesy of Stephan Wuermlin et al., ETH Zuerich, Switzerland page 26
Segmentation Courtesy of Stephan Wuermlin et al., ETH Zuerich, Switzerland page 27
Original sequence (1 camera) Source: ETH Zürich page 28
Free Viewpoint Video from 16 Cameras Results produced by HHI Same functionality as CG objects free navigation, can be viewed from any viewpoint/direction Integration into complete scenes (virtual/augmented/real) But: depict appearance, motion, deformation of real world objects page 29
Example: Depth-based Rendering Courtesy of Microsoft Research, Redmont, WA, USA page 30
Conclusion Immersive Media is a challengig and exciting field of research & development There are many approaches to allow interactivity and free navigation in video scenes Free Viewpoint Video (FVV) is a new concept for video entertainment which enables the user to navigate in real scenes in the same way as in virtual worlds page 31
Thank you Fraunhofer Heinrich-Hertz-Institut We put science into action page 32