A Framework for Realtime 3-D Reconstruction by Space Carving Using Graphics Hardware

Transcription

1 Media Christian Nitschke A Framework for Realtime 3-D Reconstruction by Space Carving Using Graphics Hardware Diploma Thesis

2 Bibliographic information published by the German National Library: The German National Library lists this publication in the National Bibliography; detailed bibliographic data are available on the Internet at This book is copyright material and must not be copied, reproduced, transferred, distributed, leased, licensed or publicly performed or used in any way except as specifically permitted in writing by the publishers, as allowed under the terms and conditions under which it was purchased or as strictly permitted by applicable copyright law. Any unauthorized distribution or use of this text may be a direct infringement of the author s and publisher s rights and those responsible may be liable in law accordingly. Copyright 2006 Diplomica Verlag GmbH ISBN:

3 Christian Nitschke A Framework for Realtime 3-D Reconstruction by Space Carving Using Graphics Hardware Diplom.de

4

5 Christian Nitschke A Framework for Realtime 3-D Reconstruction by Space Carving Using Graphics Hardware Diplomarbeit Bauhaus-Universität Weimar Fachbereich Medien Studiengang Mediensystemwissenschaft Dezember 2006 ID 10199

6 Christian Nitschke A Framework for Realtime 3-D Reconstruction by Space Carving Using Graphics Hardware ISBN: Druck Diplomica GmbH, Hamburg, 2007 Zugl. Bauhaus-Universität Weimar, Weimar, Deutschland, Diplomarbeit, 2006 Dieses Werk ist urheberrechtlich geschützt. Die dadurch begründeten Rechte, insbesondere die der Übersetzung, des Nachdrucks, des Vortrags, der Entnahme von Abbildungen und Tabellen, der Funksendung, der Mikroverfilmung oder der Vervielfältigung auf anderen Wegen und der Speicherung in Datenverarbeitungsanlagen, bleiben, auch bei nur auszugsweiser Verwertung, vorbehalten. Eine Vervielfältigung dieses Werkes oder von Teilen dieses Werkes ist auch im Einzelfall nur in den Grenzen der gesetzlichen Bestimmungen des Urheberrechtsgesetzes der Bundesrepublik Deutschland in der jeweils geltenden Fassung zulässig. Sie ist grundsätzlich vergütungspflichtig. Zuwiderhandlungen unterliegen den Strafbestimmungen des Urheberrechtes. Die Wiedergabe von Gebrauchsnamen, Handelsnamen, Warenbezeichnungen usw. in diesem Werk berechtigt auch ohne besondere Kennzeichnung nicht zu der Annahme, dass solche Namen im Sinne der Warenzeichen- und Markenschutz-Gesetzgebung als frei zu betrachten wären und daher von jedermann benutzt werden dürften. Die Informationen in diesem Werk wurden mit Sorgfalt erarbeitet. Dennoch können Fehler nicht vollständig ausgeschlossen werden, und die Diplomarbeiten Agentur, die Autoren oder Übersetzer übernehmen keine juristische Verantwortung oder irgendeine Haftung für evtl. verbliebene fehlerhafte Angaben und deren Folgen. Diplomica GmbH Hamburg 2007 Printed in Germany

7

8 To my Parents

9 I hereby declare that this submission is my own work and that, to the best of my knowledge and belief, it contains no material previously published or written by another person nor material which to a substantial extent has been accepted for the award of any other degree or diploma of the university or other institute of higher learning, except where due acknowledgment has been made in the text. Christian Nitschke, Weimar 12/06/2006

10 ABSTRACT Reconstruction of real-world scenes from a set of multiple images is a topic in Computer Vision and 3D Computer Graphics with many interesting applications. There exists a powerful algorithm for shape reconstruction from arbitrary viewpoints, called Space Carving. However, it is computationally expensive and hence can not be used with applications in the field of 3D video or CSCW as well as interactive 3D model creation. Attempts have been made to achieve real-time framerates using PC cluster systems. While these provide enough performance they are also expensive and less flexible. Approaches that use GPU hardware acceleration on single workstations achieve interactive framerates for novel-view synthesis, but do not provide an explicit volumetric representation of the whole scene. The proposed approach shows the efforts in developing a GPU hardware-accelerated framework for obtaining the volumetric photo hull of a dynamic 3D scene as seen from multiple calibrated cameras. High performance is achieved by employing a shape from silhouette technique in advance to obtain a tight initial volume for Space Carving. Also several speed-up techniques are presented to increase efficiency. Since the entire processing is done on a single PC the framework can be applied to mobile setups, enabling a wide range of further applications. The approach is explained using programmable vertex and fragment processors with current hardware and compared to highly optimized CPU implementations. It is shown that the new approach can outperform the latter by more than one magnitude.

11 ACKNOWLEDGMENTS At first, I want to thank my supervisor, Oliver Bimber for providing me with the opportunity to spend a one-year research program at Takemura Lab in Osaka University, Japan. This work would not exist in the current form without his great support. I also want to thank my second supervisor, Kiyoshi Kiyokawa for coming to Germany just to attend my defense, for all the good discussions and comments when I got stuck in the daily details, for the wonderful time in Japan and the efforts in extending my stay in the last minute. It is a good place to thank both for reading the thesis in one week. I know myself, a lot of pages came together. I would also like to thank Atsushi Nakazawa for directly supervising my work with interesting discussions and background knowledge. My parents and my family always supported me and and gave me a backing on my way. I want to thank you for all the support and that you never said no to my ideas.

12 The stone unhewn and cold becomes a living mould, The more the marble wastes the more the statue grows. Michelangelo Buonarroti

13 Contents CONTENTS 1 Introduction Application Classification Performance Contribution Overview Related Work Shape from Silhouette Image Segmentation Foundations Performance of View-Independent Reconstruction CPU GPU Acceleration Performance of View-Dependent Reconstruction CPU GPU Acceleration Conclusion Shape from Photo-Consistency Foundations Performance of View-Independent Reconstruction CPU GPU Acceleration Performance of View-Dependent Reconstruction CPU GPU Acceleration Conclusion Fundamentals Camera Geometry Pinhole Camera Model Camera Parameters Intrinsic Parameters Extrinsic Parameters Radial Lens Distortion Camera Calibration Light and Color Light in Space Radiance i -

14 Contents Light at a Surface Irradiance Radiosity Lambertian and Specular Surfaces Occlusion and Shadows Light at a Camera Color Color Representation Linear Color Spaces Non-linear Color Spaces Color Metric CCD Camera Color Imaging D Reconstruction from Multiple Views Visual Hull Reconstruction by Shape from Silhouette Shape from Silhouette Discussion The Visual Hull Silhouette-Equivalency Number of Viewpoints Conclusion Photo Hull Reconstruction by Shape from Photo-Consistency Shape from Photo-Consistency Discussion Photo-Consistency The Photo Hull The Space Carving Algorithm Voxel Visibility Conclusion Basic Algorithm Data Camera Parameters Image Data Reconstruction D Data Representation Volumetric Bounding Box Maximal Volume Intersection Visual Hull Approximation Photo-Consistent Surface Active Source Camera Test Photo Consistency Test ii -

15 Contents 5 Advanced Algorithm Overview Deployment Process Flow Texture Processing Lookup Table for Projection Coordinates Mapping Image Data into Textures Texture Upload and Processing Performance GPU Image Processing Destination Cameras Discussion Ray Casting vs. Multi Plane Sweeping Virtual vs. Natural Views Interleaved Depth Sampling Active Destination Cameras Source Camera Viewing Ray Intersection of Volume and Source Camera Viewing Ray Activity Decision Reconstruction Vertex Data Vertex Shader Visual Hull Approximation Decreasing the Sampling Error for Interleaved Sampling Early Ray Carving Fragment Shader Photo-Consistent Surface Filling Holes Modified Active Source Camera Decision Fragment Shader Color Blending Fragment Shader Render to Texture Postprocessing Extracting Texture Data Filling Interior Volume Data Ambiguities Performance Experiments System Setup Implementation Datasets Performance Abstract Data Performance Experiments CPU-GPU Texture Upload iii -

16 Contents Interleaved Sampling Early Ray Carving Fragment Shader CIELab-RGB Conversion Porting all Load to the Fragment Processor GPU-CPU Texture Read-back FBO Texture Size Impact of CPU-GPU Texture Upload on overall Performance Number of Source Cameras Number of Destination Cameras Concrete Data Performance Experiments Algorithmic Features Destination Cameras Volumetric Resolution Volumetric Bounding Box PCS Increments Conclusion Algorithmic Features Parameters GPU/CPU Comparison Quality Concrete Data Quality Experiments Volumetric Resolution Volumetric Bounding Box PCS Increments Visual Experiments Image Segmentation Interleaved Sampling and MV I Camera Viewing Cone Intersection Reconstruction of V HA and PCS Volumetric Resolution PCS Increments Geometrical Score for Active Source Camera Computation Range of Color Distances for Active Source Camera Computation Labeling of Interior Space Conclusion Image Segmentation BB, MV I and Viewing Cone Intersection V HA and PCS PCS Parameters Labeling of Interior Space iv -