Rapid Image Retrieval for Mobile Location Recognition

Transcription

1 Rapid Image Retrieval for Mobile Location Recognition G. Schroth, A. Al-Nuaimi, R. Huitl, F. Schweiger, E. Steinbach

2 Availability of GPS is limited to outdoor scenarios with few obstacles Low signal reception in Urban Canyons (multipath and occlusion effects) Hardly any positioning indoors (airports, train stations, museums, libraries, wholesale markets) Matching visual fingerprints of recorded images to a reference database (Google Street View) allows us to derive the pose in a very natural way Outdoor 6/21/2011 2

3 Challenges of mobile location recognition via BoF based image retrieval Only sparse reference data (inter panorama distance of 12.5m on average) Different lighting conditions, dynamic objects, 3-dimensional environment Very low retrieval times are essential due to the rapidly changing field of view Feature extraction - > RIFF features [1] require 27ms per 640*480 frame on Nexus One Network latency - > Transmission of visual word indices to the server [2] Feature quantization into visual words on the mobile device at low complexity [1] G. Takacs, V. Chandrasekhar, D. Chen, S. Tsai, R. Grzeszczuk,and B. Girod, Unified Real-Time Tracking and Recognition with Rotation-Invariant Fast Features, in CVPR, [2] D.M. Chen, S.S. Tsai, V. Chandrasekhar, G. Takacs, J. Singh, and B. Girod, Tree histogram coding for mobile image matching, in DCC /21/2011 3

4 Outline of the talk Related work Multiple Hypothesis Vocabulary Tree Quantization structure Adaptive clustering Visual word weighting Experimental evaluation Conclusion 6/21/2011 4

5 Related Work The robust and rapid quantization of features in visual words is of central importance Most renowned approaches are Hierarchical k-means (HKM) Greedy search in HKM Hamming embedding (HE) Approximate k-means (AKM) Comparison via application to location retrieval task 4 km² including 5000 panoramas composed of 12 rectified images Query images (640x480) are represented by 400 features on average 6/21/2011 5

6 Related Work HKM requires 60 L 2 checks Greedy search boosts performance of HKM at the cost of increased query time (510 L 2 checks) HE requires one third of comp. complexity but increases memory requirements AKM is set to perform 192 L 2 checks within 8 randomized kd-trees. Despite the inferior retrieval accuracy HKM is most suitable (25ms on 2.4 GHz desktop CPU) 6/21/2011 6

7 Multiple Hypothesis Vocabulary Tree (MHVT) Quantization structure Increase of branching factor ultimately leads to linear search and enormous computational efforts MHVT is limited to binary decisions Overlapping buffer is applied at the separating hyperplanes Database feature are assigned to both child nodes if the decision would be ambiguous Descriptors follow multiple hypothetical paths through the tree Robust against descriptor variations (e.g., wide baselines) 6/21/2011 7

8 Multiple Hypothesis Vocabulary Tree (MHVT) Adaptive clustering Large databases result in differently sized descriptor clusters Features have different occurrence frequency (e.g., windows) Overfitting can be avoided by stopping the separation once the descriptor distribution is close to a hypersphere Efficient approximation via the ratio of features assigned to the overlapping buffer 6/21/2011 8

9 Probabilistic feature weighting Use distance from separating hyperplane to estimate probability P f of quantization errors Estimate P f Probability of correct quantization is 6/21/2011 9

10 Experimental evaluation Combination of multiple hypothesis vocabulary and weighted scoring allows us to cope with continuous descriptor space while hardly increasing query time Experimental evaluation shows significant improvements w.r.t. retrieval performance, especially at wide baselines Overall time of 2.5ms per 1000 query descriptors on 2.6 GHz CPU 6/21/

11 Conclusion Facing the problem of feature quantization on the mobile device to facilitate mobile location recognition Multiple hypothesis vocabulary is generated to cope with ambiguous quantization steps Adaptive clustering reduces over fitting effects at differently sized feature clusters Integration of correct descriptor quantization probability into the distance calculation 10 fold speed up with respect to state-of-the-art resulting in 12ms for 1000 descriptors on a Nexus One Combination of Rotation Invariant Fast Features, tree histogram coding, and MHVT enables mobile real-time location recognition at 30 fps 6/21/

12 Backup: Probabilistic feature weighting Use P f to set weights a i Increases initial precision from 90.3% to 91.7% (left) and from 78.7% to 82.5% (right) No increase in query time 6/21/

13 Backup: Feature extraction and gnomonic projection 6/21/