Immersive Medien und 3D-Video

Transcription

1 Fraunhofer-Institut für Nachrichtentechnik Heinrich-Hertz-Institut Ralf Schäfer Immersive Medien und 3D-Video page 1

2 Outline Immersive Media Examples Interactive Media and Free Navigation Free Navigation in Video Scenes Examples Free Viewpoint Video Summary page 2

3 Immersive Media Costs vs. Sensation Costs Real Live Sensation IMAX American Football Cinema Theatre Rock Concert CD VoD VCR TV Interactive TV Immersive Media Sensation page 3

4 Immersive Display (UHDTV, D-Cinema) page 4

5 Example: Immersive Video Conferencing im.point Cameras 61 Plasma display Audio Speakers Semi-circular table Seamless transition between the real and virtual world Head motion parallax Life-sized upper body images representing gestures and body language 3D representation of the remote participants including provision of eye contact page 5

6 Example: Largest Panorama Worldwide Gasometer in Leipzig Printed panorama with 60 m diameter and 36 m hight (weight > 1 ton) page 6

7 page 7

8 Cylindrical Projection with HHI s CineCardultra Single level Multiple levels υ CineCardultra: Horizontal & vertical soft edge blending Warping for cylindrical and parabolic projection υ page 8

9 Interactive Media Allow to be active in some way instead of just being a passive consumer Enabled through convergence on telecommunications, consumer electronics and computer science Well-known: linking of multimedia data for interactive access Multimedia DVD, Internet presentations, Multimedia Home Platform (MHP) page 9

10 Free Navigation Arbitrary selection of viewpoint and direction within real world audio-visual scenes Well-known from computer graphics, virtual reality, computer games synthetic scenes, objects or static views of real objects or restricted to predefined viewpoints (e.g. Formula 1 on Premiere) page 10

11 Free Navigation in Real Scenes New technology allows generation of intermediate views in between real existing camera view Image-Based rendering, 3D reconstruction Allows e.g. viewing sports or theatre from arbitrary viewpoints New technologies for the whole chain: Acquisition, signal processing, modelling, representation, compression, transmission, rendering, display page 11

12 Scene Representations More geometry -based More image-based Static Texture, Single Geometry View-dep. Texture, Single Geometry Static Texture, View-dep. Geometry View-dep. Texture, View-dep. Geometry Layered - Depth images Light-field rendering Incarnations of 3D model Sprites Lumigraph Conventional graphics pipeline Warping Interpolation page 12

13 Omni-directional Video Systems with curved mirrors page 13

14 Omni-directional Video Example 1 Courtesy of Prof. Yokoya, Nara Institute of Science and Technology page 14

15 Omni-directional Video Example 2 page 15

16 Interactive Navigation in 3D Panoramas Patches page 16

17 Creation of 3D Panoramas By using several shots of a still camera and stitching techniques By using video and mosaiquing techniques page 17

18 Interactive Decoding and Rendering Tiles can be streamed as JPEG or JPEG2000 files Current view on screen Visibility sensor page 18

19 Free Viewpoint Video (FVV( FVV) Capturing a real dynamic scene with N cameras Transformation into a special data representation Interpolation of arbitrary intermediate views Allows free navigation within the scene (within practical limits) page 19

20 Example: Eye Vision using tracking cameras Virtual Camera Path X page 20

21 Eye Vision No reconstruction of 3D geometrie => many cameras No individual interaction possible Live broadcast event: Super Bowl 2001, 30 cameras Courtesy of Prof. Takeyo Kanade, Carnegie Mellon Univ. page 21

22 Visual Hull Reconstruction Synchronized multi-video recording Background subtraction Silhouette extraction Visual Hull reconstruction page 22

23 Example: Temple Acquisition by Photographs page 23

24 Processing Step: : 3D Mesh Extraction page 24

25 Processing Step: Texturing page 25

26 Multi camera video acquisition Courtesy of Stephan Wuermlin et al., ETH Zuerich, Switzerland page 26

27 Segmentation Courtesy of Stephan Wuermlin et al., ETH Zuerich, Switzerland page 27

28 Original sequence (1 camera) Source: ETH Zürich page 28

29 Free Viewpoint Video from 16 Cameras Results produced by HHI Same functionality as CG objects free navigation, can be viewed from any viewpoint/direction Integration into complete scenes (virtual/augmented/real) But: depict appearance, motion, deformation of real world objects page 29

30 Example: Depth-based Rendering Courtesy of Microsoft Research, Redmont, WA, USA page 30

31 Conclusion Immersive Media is a challengig and exciting field of research & development There are many approaches to allow interactivity and free navigation in video scenes Free Viewpoint Video (FVV) is a new concept for video entertainment which enables the user to navigate in real scenes in the same way as in virtual worlds page 31

32 Thank you Fraunhofer Heinrich-Hertz-Institut We put science into action page 32