TURNING IMAGES INTO 3D AND 4D MODELS Fabio REMONDINO 3D Optical Metrology (3DOM) Bruno Kessler Foundation (FBK) Trento, Italy Email: remondino@fbk.eu http://3dom.fbk.eu Cultural Herritage: Monuments / sites / objects at risk (Multi-temporal) Documentation Digital preservation Digital restoration Machu Picchu (Peru): Too many tourists, landslides, etc. WHY 3D & 4D??? Pompeii (Italy): too many tourists, collapse, etc. Pava (Siena, Italy): Medieval site excavation Bamyan (Afghanistan) Buddhas destruction, 2001 2 1
WHY 3D & 4D??? Efficient energy management: Estimation of photovoltaic potential of building roofs Determination of critical areas along power lines Heating loss monitoring and mapping Governance & PA: Cartography updating City planning Hidden buildings identification GIS / BIM Building height extraction 3 WHY 3D & 4D??? Environmental monitoring: Risk analyses Biomass computation Change detection Deformation analyses Natural hazard assessment 4 2
3D RECONSTRUCTION CHAIN SENSORS & PLATFORMS SOFTWARE & ALGORITHMS VISUALIZATION, SEGMENTATION, SHARING, etc DATA RESULTS 5 TURNING IMAGES INTO 3D & 4D MODELS Contents: - Recording techniques - Photogrammetric sensors & platforms - Photogrammetric pipeline and products - Examples - Conclusions 6 3
Reality-based modeling Object / Scene Complexity [points/object] 6/30/2014 TECHNIQUES FOR 3D DATA RECORDING Image-based techniques (passive sensors) - photogrammetry, computer vision, etc. dense or sparse 3D points (point cloud) Range-based techniques (optical active sensors) Surveying methods - laser scanners, stripe projection systems, radar, etc. TOF or triangulation measurement principle dense point clouds - GNSS, Total stations, maps, etc. sparse points, low density, low resolution model, time consuming Procedural modeling: just architectural rules and generally no real measurements 7 Computer graphics (3DStudioMax, Maya, Sketch-up, etc.) - no real measurements TECHNIQUES FOR 3D DATA RECORDING 10 Mil 1 Mil 100 000 Close-range photogrammetry and UAV terrestrial laser scanners Aerial photogrammetry and LiDAR Satellite Remote Sensing 10 000 1 000 100 10 Total stations Tactile / CMM Hand measurements GNSS 1 0.1 m 1 m 10 m 100 m 1 km 10 km 100 km 1000 km Object / Scene Size after [Boehler, 2001] 4
PHOTOGRAMMETRY The technique to derive reliable & precise measurements by means of photographs / images The art, science and technology of obtaining reliable information about physical objects and the environment through the process of recording, measuring, and interpreting photographic images and pattern of electromagnetic radiant energy and other phenomena The art of turning images into 3D models Applications: from Mars to nanotechnologies, from small to large scale, from m to micron Photogrammetry: Analogue Analytical Digital Remote Sensing, Computer Vision INPUT: Satellite images Analog aerial-photos Digital airborne images Terrestrial images (digital/analog) GPS data OUTPUT: Digital Terrain/Surface Models (DTM/DSM) Ortho-images Maps (Textured) 3D models GIS layers 9 PHOTOGRAMMETRY how does it work? - Space intersection (triangulation measurement principle) of rays passing through homologous points in multiple images - At least 2 images - Generation of 3D coordinates (object) from 2D coordinates (image) -> ill-posed problem - Knowledge of GCPs or distance to get metric results (not like this for 3D scanning!!) - Example: aerial block 10 5
PHOTOGRAMMETRY how does it work? - 2D images need a mathematical model to convert the image measurements into 3D data - At least 2 images to derive 3D information - ill-posed problem - Collinearity principle - Camera calibration to correct systematic errors due to lens distortion Z Y object terrestrial example X Reference system O A O B 11 image A image B Photogrammetry and sisters Photogrammetry Accuracy and geometry as primary goal (photo grammetry) Automation not really necessary Geomatics background Application-driven (e.g. maps) Born and still strictly linked to the mapping field Computer Vision Automation as primary goal Informatics background Often trying to re-invent the wheel Technology-driven Driving force of the latest automation achievements Remote Sensing No geometric processing, mainly interpretation and classification RS meets Photogrammetry thanks to the latest very high-res satellite (30-50 cm resolution) with stereo capabilities 12 6
altitude 6/30/2014 TURNING IMAGES INTO 3D & 4D MODELS Contents: - Recording techniques - Photogrammetric sensors & platforms - Photogrammetric pipeline and products - Examples - Conclusions 13 PHOTOGRAMMETRY with + satellite (optical) images 600-800 km + airborne images + helicopter balloon UAV images 1-10 m + terrestrial images + underwater images 14 7
SENSORS & PLATFORMS FOR DATA CAPTURING Digital sensors & platforms Highlights & Trends Multiple platforms: satellite, airborne, UAV, mobile, terrestrial, underwater Increasing geometric resolution (giga-pixel images, mobile phones up to 40Mpx) New instruments with very fast acquisitions Abundance of data (larger sensors, higher frame rates, etc.) given very often troubles during the processing Off-line / post-processing Real-time, Static Dynamic (4D) Crowdsourcing / citizen sensors / DIY communities Integration (single sensor sensors integration sensor network) Return of oblique imagery, in particular for airborne acquisitions SENSORS & PLATFORMS FOR DATA CAPTURING Digital sensors & platforms - Unmanned / Remotely Piloted Vehicles Platform types: balloons, airship, kite, model helicopter (electric or ICE), fixed/rotary wing aircraft, solar powered, etc. Onboard sensors: navigation units (GPS/INS), digital camera or active sensors (laser scanner, Kinect, etc.) Terminology: Drone, Remotely Piloted Vehicle (RPV), Remotely Operated Aircraft (ROA), Micro Aerial Vehicles (MAV), Remote Controlled (RC) Helicopter, Model Helicopter, etc. Regulations under creation at EU level Point of interest for different communities: photogrammetry, surveying, robotics, computer vision, artificial intelligence, space domain, archaeology, geography, forestry, etc. Different open-source and low-cost solutions (HW and SW level) Problems and limitations are still existing: payload, endurance, instability, direct geo-referencing, etc. 16 8
SENSORS & PLATFORMS FOR DATA CAPTURING Large variety of UAV platforms for heritage applications: Swinglet-like Pteryx Aeromao Borjet Maja SmartPlanes 17 SenseFly Gatewing SENSORS & PLATFORMS FOR DATA CAPTURING Large variety of UAV platforms for heritage applications: Multirotor-like Heliprocam Droidworx NuvAero OktoKopter Aibotix Microdrones DraganFly ASCTEC GAUI Falcon 18 9
SENSORS & PLATFORMS FOR DATA CAPTURING Large variety of platforms for heritage applications: Model helicopter-like Helicam Autocopter SYMA Edmonton SurveyCopter Aeroscout 19 TURNING IMAGES INTO 3D & 4D MODELS Contents: - Recording techniques - Photogrammetric sensors & platforms - Photogrammetric pipeline and products - Examples - Conclusions 20 10
PHOTOGRAMMETRIC PIPELINE Image data acquisition Image pre-processing Calibration and orientation Measurements / Dense matching & 3D point cloud generation Surface generation & Texture Mapping Visualization, GIS products, replicas, inspection, virtual restoration, etc. Possibility to derive, according to the project requirements and scene, sparse or dense 3D point clouds, geometric features of textured 3D models, etc. 21 PHOTOGRAMMETRIC PIPELINE - Example 2. Compute the orientation parameters (e.g. with RPC or classical model) 3. Derive a surface model 1. Stereo pair 4. Texture mapping with orthophoto and visualization 22 11
PHOTOGRAMMETRIC PIPELINE - Example 1. Set of images 2. Compute the camera poses (camera was already calibrated in the lab) measuring different tie points in the images and running a bundle adjustment 5. Texture mapping for a photo-realistic 3D model 4. Convert the 3D point cloud in a surface 3. Manually measure many points to reconstruct the main geometric entities (planes, arches, columns, facades, etc) of the monument 23 PHOTOGRAMMETRIC PIPELINE - Example Object to be surveyed 5. Texture mapping for a photorealistic 3D model 2. Compute the camera poses measuring different tie points in the images and running a bundle adjustment 24 4. Convert the 3D point cloud in a surface 3. Measure many other points to reconstruct the entire object 12
PHOTOGRAMMETRIC PRODUCTS 25 Digital Terrain/Surface Models (DTM/DSM) Orthophotos Monoplotting 3D Models Cartography / Maps City Planning Industrial measurements Animations & Visualization 3D documentation and digital conservation Digital restoration Architectural drawings Physical reconstruction Virtual tourism Educational resources PHOTOGRAMMETRIC PRODUCTS - DEM 2.5 m SPOT5-HRG 15 m ASTER DSM DTM 26 13
PHOTOGRAMMETRIC PRODUCTS - Ortho Aerial image: Not possible to measure correct distances, due to perspective effects Orthophoto: Measured distances, areas and positions are like on a map 27 Aerial image: Overlapping of image and vector data is not consistent / correct Orthophoto: Correct overlap between image and vector data PHOTOGRAMMETRIC PRODUCTS - Ortho Image: A central perspective projection, generally tilted Orthophoto / Map: An orthogonal image Map Reference plane Orthophoto Shift - geometrically corrected (aerial / satellite) image - variations in scale and displacements (due to tilted acquisition and terrain relief) are removed ( image rectification ) 28 14
PHOTOGRAMMETRIC PRODUCTS - Ortho 29 PHOTOGRAMMETRIC PRODUCTS - Monoplotting Principle of height measurement (and metric information) from images If a Digital Elevation Model (DEM) is known, the X-Y location of T can be computed through ray intersection with the DEM. Fixing the X-Y location and measuring dr in the image leads to an estimate of dh. image DEM Possibility to use historical images to derive metric information of past landscape / events 30 15
TURNING IMAGES INTO 3D & 4D MODELS Contents: - Recording techniques - Photogrammetric sensors & platforms - Photogrammetric pipeline and products - Examples - Conclusions 31 Xochicalco, Mexico Pre-hispanic site, 700-800 AD Digitally reconstructed from archive aerial images (ETH project) Automated terrain reconstruction, interactive structure 3D modeling EXAMPLES 32 16
EXAMPLES The NASCA lines, Peru Lines and figures drawn in the desert (geoglyphs) Unknown motivation, many hypotheses (astronomy, religion, water) Project @ ETH: ca 400 aerial images, 1:7000 scale Mapping and archaeological analysis of the geoglyphs Digital preservation as of 1998 So far, ca 700 geoglyphs mapped / documented 33 EXAMPLES The NASCA lines, Peru 3D Modeling and Visualization Virtual flight over the geoglyphs 34 17
The NASCA lines, Peru GIS Visibility Analysis - Are geoglyphs really only visible from air? - Is intervisibility between geoglyphs higher than between arbitrary points? EXAMPLES - Visibility analysis - GREEN AREA = areas from where it s possible to see the RED DOT (e.g. the geoglyph) 35 EXAMPLES The NASCA lines, Peru Photogrammetric Products First complete map of the region including the geoglyphs Supports the archaeological fieldwork Physical 3D model, shown in museum of Palpa Virtual 3D model, used for archaeological analysis and prospection 36 18
EXAMPLES 3D modeling of a architectural monuments Lalibella, Ethiopia (ETH / Univ. Melbourne) Manual (interactive) reconstruction 37 EXAMPLES MEM3D project FBK Trento 38 19
TURNING IMAGES INTO 3D & 4D MODELS Contents: - Recording techniques - Photogrammetric sensors & platforms - Photogrammetric pipeline and products - Examples - Conclusions 39 Photogrammetry as valuable image-based approach for 3D / 4D data delivery Advantages of photogrammetry: cheap, easy to use, flexible, portable, archive images Disadvantages: great experience, scale Do not work alone, try to collaborate with the experts CONCLUSIONS 3D is important for its intrinsic added-value in many fields and applications 3D is not the end: segmentation, semantics, metadata, sharing, 4D, etc. 40 20