IMPLICIT SHAPE MODELS FOR OBJECT DETECTION IN 3D POINT CLOUDS Alexander Velizhev 1 (presenter) Roman Shapovalov 2 Konrad Schindler 3 1 Hexagon Technology Center, Heerbrugg, Switzerland 2 Graphics & Media Lab, Lomonosov Moscow State University, Russia 3 Institut f. Geodäsie u. Photogrammetrie, ETH Zurich, Switzerland
Problem domain *Courtesy of Optech Slide 2 of 40
Task Input Point clouds Classes of interest: Small objects Output Cars, traffic signs, lamp posts, Buildings, fences, Object centers and class labels Slide 3 of 40
Previous work Shape-based Recognition of 3D Point Clouds in Urban Environments (Golovinskiy et al., 2009) Let s study this part *Courtesy of (Golovinskiy et al., 2009) Slide 4 of 40
Localization 1. Remove ground points by detecting planes 1. Extract connected components (CCs) signs poles noise cars Ground it s important not to miss any relevant objects Slide 5 of 40
Localization Our tests: > 95% objects are retained Slide 6 of 40
Previous work Shape-based Recognition of 3D Point Clouds in Urban Environments (Golovinskiy et al., 2009) No real problems here! Let s study these parts Slide 7 of 40
Classification (Golovinskiy et al., 2009) Global descriptor per whole object hypothesis Simple geometric features Spin Images SVM classifier Trained on a big dataset 125 cars, 73 light poles, Slide 8 of 40
Previous work Shape-based Recognition of 3D Point Clouds in Urban Environments (Golovinskiy et al., 2009) Precision/Recall: Cars: 0.50/0.62 Light poles: 0.45/0.62 Slide 9 of 40
Previous work Shape-based Recognition of 3D Point Clouds in Urban Environments (Golovinskiy et al., 2009) Classification is a weak point Slide 10 of 40
Classification: key problems Significant shape variations Occlusions, back-faces Robust fitting is not applicable Global descriptors often fail Slide 11 of 40
Classification: one more problem * A typical parking case in Paris Slide 12 of 40
Our solution Implicit Shape Model Slide 13 of 40
Implicit Shape Model (ISM) A combination of visual dictionaries and the generalized Hough transform (Leibe, 2003; Knopp, 2010) Fully automatic, very robust against occlusions and noise Not applied to real-world point clouds before Slide 14 of 40
ISM: advantages Joint segmentation and classification Up to +24% to precision/recall Only few training objects required Slide 15 of 40
ISM Algorithm Training stage Training objects Classification stage Object hypothesis Extract keyponts, compute descriptors Extract keyponts, compute descriptors Construct dictionary Vote for object center Analyze voting space Slide 16 of 40
ISM: Simple 2D example
Training stage object center Training objects Extract keyponts, compute descriptors Slide 18 of 40
Training stage object center Training objects Extract keyponts, compute descriptors Slide 19 of 40
Keypoint information Descriptor Vector to object center Class Label Slide 20 of 40
Training stage object center Training objects Extract keyponts, compute descriptors Construct dictionary Clustering in descriptors space Slide 21 of 40
Classification stage Object hypothesis Unknown object Extract keyponts, compute descriptors Vote for object center Dictionary Analyze voting space Find peaks in voting space Slide 22 of 40
Voting object center! dictionary Slide 23 of 40
How does it work for outdoor urban point clouds?
Main differences Real data: noisy, frequent occlusions it is difficult to create reliable mesh Point density changes a lot We adapt to mobile mapping data: keypoint detector, descriptor voting space analysis Slide 25 of 40
Localization After localization we have many CCs to classify let s apply ISM! Slide 26 of 40
Visual dictionary - keypoints Keypoints: spatial sampling Slide 27 of 40
Visual dictionary - descriptor Spin Image Rotate grid & collect how many points are inside each cell Slide 28 of 40
Visual dictionary - descriptor Modifications: density normalization By dividing through the number of points in the local neighborhood normal mirroring Compute descriptor for direct and inverse axis directions Slide 29 of 40
Visual dictionary - geometric words K-means clustering in descriptors space Spin Image example Visual dictionary Slide 30 of 40
Classifying unknown objects Voting: independently for each class Individual votes are weighted to - correct imbalances between object classes - account for height distribution of geometric words Slide 31 of 40
Classifying unknown objects Center point extraction mean-shift to find density maxima in voting space inter-class non-maxima suppression can detect multiple objects (or no object at all) Slide 32 of 40
Experiments
Input data Ohio dataset urban scenes used in (Golovinskiy et al., 2009) Terrestrial & airborne laser scanning 15 tiles, 100x100 meters ~5 000 000 points per 100х100 tile Slide 34 of 40
Training and test parts Training 1 car 4 light poles 1 tree 1 wall background classes Test 15 tiles, 100x100 meters with 235 cars with 73 light poles 3935 CCs were extracted Slide 35 of 40
Results Cars Light poles Method Precision Recall Precision Recall (Golovinskiy) 0.50 0.62 0.45 0.62 Our method 0.66 (+0.16) 0.70 (+0.08) 0.69 (+0.24) 0.80 (+0.18) Slide 36 of 40
Spin Image modification Cars: +10-40% higher recall at same precision Light poles: +30-70% higher recall at same precision Slide 37 of 40
Implementation C++ language 5-10 minutes per tile 100x100 meters Intel Quad 2.4 GHz, 4 GB RAM, 1 Core Slide 38 of 40
Conclusions Fully automatic object detection in point clouds ISM improves precision/recall up to 24% over state-of-the-art Very few training objects are required Main limitation: objects with undefined shape and/or center Slide 39 of 40
IMPLICIT SHAPE MODELS FOR OBJECT DETECTION IN 3D POINT CLOUDS Alexander Velizhev Roman Shapovalov Konrad Schindler Thank you for your car detected inside a garage! attention!