Facade Segmentation in a Multi-View Scenario

Transcription

1 Facade in a Multi-View Scenario Michal Recky, Andreas Wendel, and Franz Leberl {recky, wendel, leberl}@icg.tugraz.at Institute for Computer Graphics and Vision (ICG) Graz University of Technology, Austria 1

2 Input Shape Grammar Facade Result Road Result Image-based Procedural Modelling detailed views semantics (door, window etc.) quick transmission separate facades required! Image source: Simon et al. IJCV

3 Streetside Image Acquisition Multiple Cameras, Overlap, High-Resolution Separated Facade Segments Image source: Wendel et al. DAGM

4 Agenda Which existing algorithms could be employed? How can we improve the results for single views? How can we use the redundancy of the data in a multi-view scenario? Can we obtain accurate segmentations? 4

5 Related Work Facade Separation: Based on directional gradients [Hernandez et al. ICIP 2009] Based on height and appearance (boosted) [Zhao et al. CVPR 2010] Based on repetitive patterns [Wendel et al. DAGM 2010] Semantic of Street-Side Data: Based on 3D point clouds [Brostow et al. ECCV 2008] Based on 2D /3D superpixel properties (boosted) [Xiao et al. ICCV 2009] Based on 2D features of trained superpixels [Recky & Leberl OAGM 2009] 5

6 Related Work Separation based on analysis of repetitive patterns [Wendel et al. DAGM 2010] Issue: vegetation, cars, cables cause repetitive patterns Semantic street-side classification [Recky & Leberl OAGM 2009] Issue: no separation provided 6

7 Semantic Combine watershed superpixels to a meaningful over-segmentation based on a trained color model C 1,C 2 h h min f max s, s, f avg b b , 2 M. Recky and F. Leberl. Semantic of Street-Side Images. In Proc. OAGM,

8 Semantic Classification 9 semantic categories façade, sky, cloud, ground, roof, vegetation, grass, shadow, unidentified Trained classifier (unary potentials) color histograms, texture, position, size Spatial relations (pairwise potentials) bounding boxes and centers of regions Inference using a Discriminative Random Field (DRF) M. Recky and F. Leberl. Semantic of Street-Side Images. In Proc. OAGM,

9 Finding Repetitive Patterns Detection and description: Harris corners, color intensity profiles Benefit: matching is invariant to affine transformations! A. Wendel, M. Donoser, H. Bischof. Unsupervised Façade using Repetitive Patterns. In Proc. DAGM, LNCS 6376,

10 Facade Separation A. Wendel, M. Donoser, H. Bischof. Unsupervised Façade using Repetitive Patterns. In Proc. DAGM, LNCS 6376,

11 Our Single-View Approach 1) Consider segments classified as facade or unidentified as potential facade areas 2) Compute repetitive areas 3) Label potential facade areas overlapping the repetitive area as individual facades without prior with prior repetitive area over-segmentation facade segment 11

12 Our Multi-View Approach 1) Matching of segments in multiple views: transfer segment labels by projecting sparse 3D points (LiDAR/image-based) 2) Fusion of labels: Facade if... Semantically facade, part of separate facade in at least one image Semantically unidentified, part of separate facade in majority of images single-view multi-view 12

13 Evaluation Dataset: 250 images of a frontal-sideways tilted camera 9 separate facades in approx images each LiDAR data readily available Evaluation criteria: Precision/Recall, F 1 (effectiveness) Manually labelled ground truth Equivalent to [Wendel et al. DAGM 2010] for comparison 13

14 Evaluation - Separation Contributions to Separation: Prior knowledge from semantic segmentation +11.2% F 1 14

15 Evaluation - Contributions to : Prior knowledge from semantic segmentation Final segments based on trained over-segmentations +15.1% F % F 1 15

16 Evaluation - Contributions to : Prior knowledge from semantic segmentation Final segments based on trained over-segmentations Exploitation of redundant data in a multi-view scenario +14.9% F 1 16

17 Typical Results Single View Multi View 17

18 Conclusion Which existing algorithms could be employed? Facade separation combined with semantic segmentation. How can we improve the results for single views? By using semantic information as prior knowledge and by using trained over-segmentations. How can we use the redundancy of the data in a multi-view scenario? By transferring the segment labels using sparse 3D points (LiDAR/image-based). Can we obtain accurate segmentations? Yes, 96.6% segmentation effectiveness (F 1 ) in a multi-view setting for 250 images. 18

19 Thank you! The video can be found online at This work has been supported by the Austrian Research Promotion Agency (FFG) project FIT-IT CityFit (815971/14472-GLE/ROD) and by the Austrian Science Fund (FWF) under the doctoral program Confluence of Vision and Graphics W

20 References [Brostow 2008] [Xiao 2009] [Hernandez 2009] [Recky 2009] [Zhao 2010] [Wendel 2010] [Simon 2011] G. J. Brostow, J. Shotton, J. Fauqueur, R. Cipolla. and Recognition using Structure from Motion Point Clouds. In Proc. ECCV, J. Xiao, L. Quan. Multiple View Semantic for Street View Images. In Proc. ICCV, J. Hernandez and B. Marcotegui. Morphological of Building Façade Images. In Proc. ICIP, M. Recky and F. Leberl. Semantic of Street-Side Images. In Proc. OAGM Workshop, P. Zhao, T. Fang, J. Xiao, H. Zhang, Q. Zhao, L. Quan. Rectilinear Parsing of Architecture in Urban Environments. In Proc. CVPR, A. Wendel, M. Donoser, H. Bischof. Unsupervised Façade using Repetitive Patterns. In Proc. DAGM, LNCS 6376, L. Simon, O. Teboul, P. Koutsourakis, N. Paragios. Random Exploration of the Procedural Space for Single-View 3D Modeling of Buildings. International Journal of Computer Vision (IJCV), 93: ,