The acquisition of appearance properties of real objects: state of the art, challenges, perspectives

Transcription

1 The acquisition of appearance properties of real objects: state of the art, challenges, perspectives Matteo Dellepiane Visual Computing Lab, ISTI-CNR Pisa 26th February 2015 The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/ ) under grant agreement n

2 The Big Issue Acquisition of surface appearance of real objects Real objects are complex (and spatially varying)! Accuracy vs Data acquisition / Complex setup / Costs We want to Acquire a real object!

3 Visual Appearance: Definition Reflectance Scattering Function (12D) (Light and view direction, Incident and outgoing surface point Wavelength, Time)

4 Visual Appearance: Definition

5 Acquisition of surface appearance Some approaches Acquisition of a Reflectance Function BTF (Schwartz 2011), SVBRDF from videos (Palma 2012) Modeling and fitting of Reflectance Function Material design (Wu 2011) Acquiring the appearance at a specific moment: RTI Imaging (Mudge 2010), Surface Light Field (Palma 2013) Mapping of photographic datasets Automatic texturing (Callieri 2003), Photo Blending (Callieri 2008), Point Clouds (Pintus 2008)

6 Acquisition of reflectance function University of Bonn: BTF (Schwartz 2011) Several tens of digital cameras and projectors... Fraunhofer IGD: 3D Mass digitization for Cultural Heritage (Santos, 2014) Impressive but transportable

7 Acquisition of reflectance function A Statistical Method for SVBRDF Approximation from Video Sequences in General Lighting Conditions (Palma, 2012) Simpler input: a 3D model, some video sequences, fixed light condition

8 Modeling and fitting of reflectance function Physically-Based Interactive Bi-Scale Material Design (Wu 2011) The model can be changed by hand, or better by example

9 Acquiring appearance at a specific moment RTI Imaging (Malzbender, 2001) - Fixed point of view - One directional light - Information encoded in the image Surface light field (Palma 2013) - Fixed light condition - Information encoded as a light field

10 Mapping of photographic datasets Main steps 1) Registration of the images on the geometry 2) Color projection Issues Lighting artifacts (shadows, highlights, uneven lighting) Aliasing/Ghosting (small misalignments) Solutions? Maybe

11 Lighting artifacts Use an acquisition device to estimate the lights in the scene. Stereo light probe Corsini et al Calibrate a light source to correct image artifacts before and during projection Flash lighting space sampling Dellepiane et al. 2009

12 Aliasing/ghosting 1) Use only one image for each portion of the surface Limitations: uneven lighting, misalignment in the borders, fragmentation 2) Further refinement of registration Limitations: input data, camera model (Dellepiane, 2013) 3) The same detail on the same surface Limitations (Dellepiane 2011): fragmentation, borders, strong processing, we warp the data 4) Put 2 and 3 things together (Gal, 2014)

13 Available solutions Can we obtain a decent acquired color on a 3D model? Multi-view stereo matching tools (aspect3d, Photoscan, MVE) Can do some of the work, still lighting independent MeshLab Complete pipeline for color projection, a few alternatives for color projection Commercial tools Usually for very simple parameterizations SpheronVR device A bit expensive

14 Moving to architectural data What about architectural (indoor and outdoor) data that could be acquired with optical approaches? THE GOOD Simpler geometry (usually) Simpler materials (usually) THE BAD Size! Lighting environment Hence we cannot acquire a lot of images, and we cannot control lighting

15 Acquiring reflectance? Projecting color? Relighting Objects from Image Collections (Haber 2009) Analyze images, estimate lighting environment, estimating the material

16 Acquiring reflectance? Projecting color? The Visual Turing Test for Scene Reconstruction (Shan 2013) Using images to get a quasi-realistic rendering

17 Acquiring reflectance? Projecting color? Removing shadows for color projection using sun position estimation (Dellepiane 2010) Using image, GPS position, time of the day to estimate sun position

18 Simplified acquisition? Using RTI imaging-like methods? Flat facades usually undergo a single directional illumination (the sun!) Check the presentation in Technical session 4 this afternoon...

19 Modeling and fitting? Architectural objects have some interesting characteristics, like quite simple materials, repetitive elements, plana surfaces... Why not using impainting techniques to remove lighting artifacts? Why not build a set of materials and use it?

20 Conclusions There s been quite a lot of research in the field But no strong reference solution has been provided The data to estimate are too complex Can we take advantage of the work from the crowds? No real chance, as long as we go optical? Multi sensor devices?

21 Thanks, Questions? How to contact us: Visual Computing Laboratory of ISTI CNR Pisa Acknowledgements: the Lab was founded and made big by Claudio Montani and Roberto Scopigno. Researchers currently working at VCG Lab: Paolo Cignoni, Fabio Ganovelli, Marco Callieri, Matteo Dellepiane, Massimiliano Corsini, Federico Ponchio, Nico Pietroni, Guido Ranzuglia, Luigi Malomo, Daniele Duranti, Eliana Siotto, Andrea Baldacci, Marco Potenziani, Gianpaolo Palma, Giorgio Marcias, Valeria Garro, Gaia Pavoni, Francesco Banterle, Marco Di Benedetto.