Natural Feature Tracking on a Mobile Handheld Tablet

Transcription

1 Natural Feature Tracking on a Mobile Handheld Tablet Madjid Maidi #1, Marius Preda #2, Matthew N. Dailey *3, Sirisilp Kongsilp #*4 # Departement ARTMIS, Telecom SudParis / Institut Mines-Telecom 9, Rue Charles Fourier, 9111 vry Cedex, France 1madjid.maidi@telecom-sudparis.eu 2marius.preda@telecom-sudparis.eu 4sirisilp.kongsilp@telecom-sudparis.eu * AIT Vision and Graphics Lab., Asian Institute of Technology PO. Box 4, Klong Luang Pathumthani 1212, Thailand 3mdailey@ait.ac.th Abstract-This paper presents a natural feature tracking system for object recognition in real-life environments. The system is based on a local keypoint descriptor method optimized and adapted to extract salient regions within the image. ach object in the gallery is characterized by keypoints and corresponding local descriptors. The method first identifies gallery object features in new images using nearest neighbor classification. It then estimates camera pose and augments the image with registered synthetic graphics. We describe the optimizations necessary to enable realtime performance on a mobile tablet. An experimental evaluation of the system in real environments demonstrates that the method is accurate and robust. I. INTRODUCTION Mobile applications are mostly classified either as entertainment or visualization applications. Rather than displaying information statically, one can combine virtual graphics with the physical world using the mobile device's sensor data (gyroscope, GPS, and so on). Using virtual graphics, we can display extra information, creating a virtual world on top of the real one. This process of localization, registration, and synthesis is often called Augmented Reality CAR). While the most successful mobile AR systems use GPS or marker tracking approaches, only limited effort has focused on using image data and natural object tracking concepts [1]. Markerless tracking is a very complex task, as it uses image processing operators to detect natural features in the video stream [2]. Several markerless tracking approaches have been developed in recent years. Bleser and Stricker [3] developed a markerless tracking approach based on a 3D model of the scene. The system predicts the appearances of the features by rendering the model using the prediction data of a visualinertial fusion filter. Yoon et al. [4] present a model-based object tracking method to compute the camera's 3D pose. Their algorithm uses an xtended Kalman Filter (KF) to provide an incremental pose-update scheme in a predictionverification framework. Other approaches such as [5] combine different features such as edge, point, and texture information to compute the camera pose. Other authors were interested by the robustness aspect, thus, Stricker et al. [6] have presented an interactive AR application to solve occlusion problem. Occlusions are managed by locating the user hand and subtracting the background. This approach is feasible in the case of homogeneous background with the assumption of static camera. Naimark and Foxlin [7] implemented a hybrid vision-inertial self-tracker system which operates in various real-world lighting conditions. The aim is to extract coded fiducials in presence of very non-uniform lighting. In [8], the authors presented a technique based on active targets using amplitude modulation codes instead of binary codes. Such a system provides high precision with compact targets and operates in a wide range of viewing angles under various luminosity conditions. Maidi et al. [9] [1] [11] presented a robust fiducials tracking method for AR systems. A generic algorithm for object detection and feature points extraction is developed to identify targets in real-time. The authors proposed a tracking method based on RANSAC algorithm to deal with partial target occlusion. Moreover, a hybrid tracking architecture based on an inertial-vision system was presented to keep tracking in worst environment conditions such as motion blur and total occlusions of the markers [12]. Bleser and Stricker [3] presented a visual-inertial tracking device for augmented and virtual reality applications. The authors provided an evaluation of several markerless tracking approaches. Their solution relied on a 3D model of the scene to predict the appearances of the features by rendering the model using the prediction data of the sensor fusion filter. High stability and accuracy were demonstrated using the developed system. Recently, marker tracking based on keypoint descriptors such as SIFT [13] or SURF [14] has been proposed. Keypoint descriptor methods not only automatically detect points of interest, but also create invariant descriptors characterizing the local neighborhood of the keypoint. Such descriptors can be used to uniquely identify points of interest and to match them even under a variety of disturbing conditions such as noise and changes in scale, rotation, illumination, and viewpoint. 246

2 This invariance is an important criterion for mobile systems, for which environmental conditions are neither stable nor repeatable. Wagner et al. [I5] report on an optimized marker detection and tracking method based on SIFT, Ferns [16], and image patch tracking. The method achieves impressive tracking performance (up to 3 fps) on a 28-era smartphone. However, the method assumes only a single gallery image, and the performance evaluation was performed on raw images stored on the Windows mobile file system, not including image capture and application overhead. The work described in this paper focuses on real-time processing of images acquired directly from the camera device on an Android tablet in an OpenCV environment. At the time our experiments were performed, OpenCV for Android (Honeycomb) was in beta and camera capture overhead was very high, with maximum frame rates for the most trivial applications ranging from 5 to 1 fps. Despite this limitation, we are able to achieve real-time tracking, pose estimation, and graphics synthesis for a gallery of up to 12 objects. II. NATURAL 2D FATUR DTCTION AND TRACKING In recent years, there has been a growing interest in developing effective methods for searching images in large databases. Most approaches of database image retrieval focused on the search-by-query. These methods require that the users provide a request image. Then, the similar images to the query are retrieved in the image database. However, it is often difficult to find a good matching between query and trained images. Another approach of image retrieval is based on the browsing of the dataset environment that oflers an alternative to conventional search-by-query, but here also many issues have to be addressed. Indeed, the database needs to be organized logically and predictably so that the image query can be matched successfully. These approaches define a clustering system that classifies image features into groups, using quantitative and qualitative information of measurements and characteristics based on the training set in which clusters are already established (color segmentation, morphological parameters, geometrical primitives). Usually, the methodology consists in performing the class clustering to determine groups of features using techniques such as k-means, k-nn, etc. Afterwards, the classifiers are trained with learning algorithms (SVM, naive-bayes, Boosting,. ) and finally, a histogram of responses for each image is assigned for labeling the training dataset. Commonly, the methods developed for visual content analysis rely upon a multi-dimensional approach for huge database template matching and data retrieval demanding high computational resources. Thus, in our work we propose an effective search-by-query algorithm for descriptors matching based on a query array-image retrieval from a training database that we adapted to mobile application constraints. Moreover, the application is optimized in order to operate in an environment where memory storage and processing capabilities are, particularly, limited. The aim of this system is to enable robust feature-based recogmtlon and real-time tracking in unreliable performance-degrading conditions occurring in reallife tracking situations Our markerless tracking method relies upon the use of features already available in the scene, without adding specific patches or markers. Given a new image, the method computes SURF keypoints and keypoint descriptors and matches the descriptors to the stored descriptors for each gallery image using nearest neighbor search (figure I). Fig. 1. Pre-processing Keypoint extraction Desc ri ptor c I 1 Identification Image database The overall identification and tracking system diagram. First, the algorithm detects image keypoints, then, a descriptor array is computed for each keypoint in order to characterize the feature point and its surrounding pixel area. The keypoint extraction relies upon a Hessian detector that identifies the maximal salient regions. The detection is based on the Hessian matrix (q. 1) which operates in multiple scales and uses the spatial localization of invariant points [14]. H(x ") =, ( Lxx(x,O") Lxy(x, ") Once the keypoint is detected, the next step is to construct a circular region around the point and to compute the Haar wavelets in X and Y directions. This region is split into square sub-regions for which features at spaced sample points are computed. Hence, the descriptor array is built using the Haar wavelet response in horizontal direction Wx and vertical direction Wy [14]. Consequently, the descriptor array is represented by the following equation: (I) 247

3 Finally, the feature points represented with their descriptors are matched to reference data using a search-by-query strategy to retrieve the training descriptors corresponding to the closest descriptor set within a nearest neighbor ratio matching. The SURF- based algorithm presents several optimizations, the most significant one, is the improvement of the computing time using the integral image. Indeed, the image integral allows fast computation of the filters responses using a quick and efficient algorithm to sum up intensity values defined in subset grid of the image [14]. However, the difficulty in this technique is the effectiveness of the detection to recover relevant feature pairs by keeping only good matches while rejecting outliers. Indeed, these outliers will rise the probability that the system incorrectly matches the input pattern to a non-matching template in the database. The decision making process of the matching algorithm is based on a threshold which determines how close to a template the input needs to be located to be considered as a match. If the threshold is lower, there will be less false non-matches but more false rejects. Correspondingly, a higher threshold will increase the false accept rate. Practically, we estimate that a matching can be considered correct if the distance between two keypoints (distance between their corresponding descriptors) is 7% closer. III. POS STIMATION FOR AR APPLICATION estimation is the determination of the transformation that relates the object reference frame and the camera coordinate frames (figure 2). Before computing the pose parameters, we have to determine the 2D-3D matching pairs to establish a geometric constraint between the image and the 3D object reference systems [17]. To estimate camera pose, we first perform a camera calibration offline and store the camera's intrinsic parameters. At run time, when an object is detected, we use the set of 2D-3D correspondences with the analytical pose estimation method proposed by Zhang [18]. The pose estimation is an error minimization process enabling to define the function relating 3D points to 2D points by [19]: (3) F(p, P, Ix, R, T) = (X, Y, Z)T is a 3D point defined in the object ( u, v) T the projection of that point where P = coordinate frame and p = in the image. The camera model is defined by: sp = Ix (R T) P (4) s represents a scale factor, (R T) is the pose matrix: the rotation and the translation of the object coordinate frame according to the camera coordinate frame and I x is the intrinsic matrix of the camera. Assuming that the object is planar (Z = have: ), from q. 4 we (5) Camera reference frame x Fig. 2. Object reference frame y.x Reference frames used in pose estimation process. where Ti = (Tli T2i T3i) T is a column rotation vector. The projective matrix which relates the object coordinate frame to the camera reference frame can be estimated using at least 4 pairs of 2D-3D points. We denote this perspective transformation by ]vi and based on q. 4 and q. 5, we have: Finally, the resolution { of rotation and translation parameters is derived from the following equation system: Tl = >"I; l rnl T2 = >"I; l rn2 (7) T3 = Tl X T2 T = AI; l rn3 IV. IMPLMNTATION ON MOBIL TABLT We implemented a prototype AR application on an Android tablet running HoneyComb revision 7 and Java V6 update 31. We implemented the main image processing functions in native code with C++, OpenCV, and Android-ndk release 6. We modified the OpenCV SURF keypoint extractor to limit the number of feature points extracted. This simple change enabled a dramatic speedup in the keypoint detection and descriptor computation processes. The tracking system extracts local SURF features and classifies keypoints using k-nearest neighbors to find the best matching database image. The application enables continuous recognition of natural objects in live video captured by the tablet's camera. The user points the camera at a target and the system identifies the name of the object in the viewfinder in less than 25 rns. The boundary of the object is displayed and accurately tracked in real-time. The object is identified and its geometry is quickly retrieved and augmented from a database of object models. (6) 248

4 V. XPRIMNTS AND RSULTS A. Natural feature tracking results The objective of the application is to determine if the scene contains any pre-trained features. Initially, the system loads all objects and builds a database of images with keypoints and keypoint descriptors. J) Object recognition: We start by showing the obtained results from the object recognition test. Figure 3 shows how the object and its keypoints are characterized. SURF enables us to identifying objects with very few keypoints. However, we find that object texture affects the quality of the results; textured objects have more feature points and enable better recognition. the results. The number of matches tends to increase as the number of feature points increases. As pose estimation improves (to a limit) as the number of correspondences increases, it is important to find a sufficient number of matches. Match count Number of feature points Fig. 5. volution of number of pairs according to detected keypoints. Fig. 3. Feature detection and description. 2) Computational requirements: To characterize the method's resource requirements, we varied the number of keypoints and observed the impact of feature points on the frame rate. Figure 4 shows the results obtained from this test. We observe that the frame rate is inversely proportional to the number of detected points. This result is expected, since keypoint detection and descriptor extraction require computational resources (memory and CPU). 4 3 FPS 2 B. estimator evaluation J) xecution time: To evaluate the efficiency of the analytical pose algorithm in terms of computational resources, we compared the execution time of 3 pose estimators to highlight the most relevant method meeting the mobile realtime requirement. We computed 327 poses with the 3 pose estimators (figure 6). The mean time for determining a single pose was.1 ms using the analytical algorithm while it is about.6453 ms for the ICP algorithm (Iterative Closest Point [2]) and ms for the 1 method (Orthogonal Iteration [21]). These two iterative methods are, respectively, 6 and 16 times slower than the analytical algorithm. Obviously, the analytical algorithm presents the best performance, even though, the computation time required for pose estimation is negligible compared to that required for the keypoint detection and description steps. en. 2 CII 1.S j:: O.S -Analytical algorithm -ICP algorithm -Orthogonal Iteration 1 SOD 1 1S 2 2S 3 3S Fig. 4. o -----' r Number of feature points Variation of FPS according to the number of detected keypoints. 3) Matching pairs: In order to characterize the relationship between the number of feature points and number of matched points, we ran a third test with a single object. Figure 5 shows Fig. 6. Comparison of computation time using the analytical pose estimator versus iterative methods. 2) Reprojection error: To characterize pose estimation error, for each pose computation, we re-projected the target model onto the image and measured the deviation between the detected target corners and the reprojected corners. From figure 7, we notice that the algorithm is stable and accurate, with a mean error of.5 pixels. 249

5 Vi'.12 Qi ><.1 : :.8 s:::.6 :;::; (,J ::::I til s::: (,J :: >< I Fig. 7. Reprojection error. - algorithm -Real distance 3) Generalization error: This error measures the deviation of reprojecting other objects on the image according to the pose computed using only one fiducial object. To determine the generalization error, we use 2D-3D template pairs to estimate a reference pose. The difference resulting from the projection using the reference and the real pose constitutes the generalization error. The obtained results on generalization error are represented in figure 8. From the curve we notice that the overall error behavior of the algorithm presents some jittering but it doesn't affect the whole accuracy of the pose since this mean error is estimated to pixels only algorithm -Real distance 8,-----,-----,-----,-----,----,-----, In Qi >< 6c.... o 4",.,,I,.. 1\..... s::: o :;::; (II.!::! " s::: Fig. 8. Generalization error. 4) Real distance estimation: To evaluate the method's realworld pose estimation accuracy, we placed two targets in the environment and measured the real and estimated distance between them. The real distance between these two markers is: T = (-1,25,5) mm, this vector is the translation of the coordinate system of the second marker according to the coordinate system of the first marker. The results are illustrated in figure 9. The mean error in the translation vector is T = (3.884,3.719, ) mm. These errors are low except for the relatively large error in the Z translation direction, due to the use of a single camera. From table I, we notice that the pose estimator algorithm presents a relatively important mean error in Z direction with an important variance and standard deviation values. These errors have occurred due to the lack of information o 2 Fig. 9. Real distance estimation. J resulting from the use of a single camera which is unable to elaborate a depth function representing the scene. Indeed, accurate 3D localization approaches, generally, rely upon stereo-vision techniques which evaluate the distance using spatial disparity of a fiducial object in two images [1] [19]. The measurement system requires a pair of cameras, calibrated and oriented spatially, pointing the object of interest to defining a disparity function relating correspondent features in two views. Alternatively, a single camera can be used to estimate accurate 3D positions by capturing images from multiple viewpoints, whereas in our test, a single view monocular vision device is experimented which explains the inaccuracies in Z distance estimation. 5) AR tracking: The aim of this experiment is to informally evaluate the pose and the virtual overlay accuracy. Figure 1 shows the obtained results: the camera is moved freely around the target object. The algorithm detects and tracks the keypoints in each frame, and the pose estimator determines position and orientation of the camera. We notice that the virtual object is accurately superimposed on the real image over different camera pose. 25

6 Translation/Statistics Mean rror (mm) Variance (mm') Standard Deviation (mm) TABL I STATISTICS RLATD TO 3D LOCALIZATION USING TH POS STIMATOR. Fig. 1. estimation and virtual overlay results. VI. CONCLUSION In this paper, we present a markerless tracking system for mobile AR applications. We address the problem of tracking and augmenting the scene using only natural features present in the environment. The method extracts keypoints and local descriptors, matches keypoints in the probe image with those in the gallery, then performs pose estimation overlay of 3D graphics on the real image. We evaluate the performance of the application in terms of resources such as the processing time and memory utilization (number of keypoints and pairs). We analyze localization performance and find that the method is efficient and robust. Despite the limited support for image processing provided by the device's hardware and operating system, our experiments demonstrate robust real-time performance and prove the validity of vision-based tracking on mobile platforms using only image data. In future work, we aim to further improve the application's frame rate and image gallery size while retaining robust tracking. [2] M. Maidi, M. Preda, and V. H. Le, "Markerless tracking for mobile augmented reality," in I International Conference on Signal and Image Processing Applications (ICSIPA'211). Kuala Lumpur, Malaysia: I Signal Processing Society, November 211, pp [3] G. Bleser and D. Stricker, "Advanced tracking through efficient image processing and visual-inertial sensor fusion," in I Virtual Reality (VR), 28, pp [4] Y. Yoon, A. Kosaka, 1. Park, and A. Kak, "A new approach to the use of edge extremities for model-based object tracking," in Int. Con! on Robotics and Automation (ICRA), 25, pp [5] L. Vacchetti, V. Lepetit, and P. Fua, "Combining edge and texture information for real-time accurate 3d camera tracking," in Int. Symp. on Mixed and Augmented Reality (ISMAR), 24, pp [6] D. Stricker, G. Klinker, and D. Reiners, "A fast and robust line-based optical tracker for augmented reality applications," in Proc. First International Workshop on Augmented Reality (IWAR'98), San Francisco, USA, 1998, pp [7] L. Naimark and. Foxlin, "Circular data matrix fiducial system and robust image processing for a wearable vision-inertial self-tracker," in I International Symposium on Mixed and Augmented Reality (ISMAR'2), Darmstadt, Germany, 22, pp [8] --, "ncoded led system for optical trackers," in ACM and I International Symposium on Mixed and Augmented Reality (ISMAR'5), Vienna, Austria, October 25. [9] M. Maidi, F. Ababsa, and M. Mallem, "Robust fiducials tracking in augmented reality," in The 13th International Conference on Systems, Signals and Image Processing (IWSSIP 26), Budapest, Hungary, 26, pp [1] --, "Robust augmented reality tracking based visual pose estimation," in 3rd International Conference on Informatics in Control, Automation and Robotics (ICINCO'6), Setbal, Portugal, 26, pp [11] --, "Active contours motion based on optical flow for tracking in augmented reality," in 8th International Conference on Virtual Reality (VRIC'6), Laval, France, 26, pp [12] --, "Vision-inertial tracking system for robust fiducials registration in augmented reality," in I Symposium on Computational Intelligence for Multimedia Signal and Vision Processing (CIMSVP 29), Nashville (USA), March 3 April 2 29, pp [13] D. G. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Computer Vision, vol. 6, no. 2, pp , 24. [14] H. Bay, A. ss, T. Tuytelaars, and L. Van Gool, "Speeded-up robust features (SURF)," J. Computer Vision and Image Understanding, vol. 1l, no. 3, pp , 28. [15] D. Wagner, G. Reitmayr, A. Mulloni, T. Drummond, and D. Schmal stieg, "Real-time detection and tracking for augmented reality on mobile phones," I Trans. on Visualization and Computer Graphics, vol. 16, no. 3, pp , 21. [16] M. Ozuysal, P. Fua, and V. Lepetit, "Fast keypoint recognition in ten lines of code," in I Con! on Computer Vision and Pattern Recognition (CVPR), 27, pp [17] M. Maidi, 1.-Y. Didier, F. Ababsa, and M. Mallem, "A performance study for camera pose estimation using visual marker based tracking," Machine Vision and Applications, IAPR International Journal, Springer, vol. 21, no. 3, pp , 21. [18] Z. Zhang, "A flexible new technique for camera calibration;' I Trans. on Pattern Analysis and Machine Intelligence, vol. 22, no. II, pp ,2. [19] M. Maidi, M. Mallem, L. Benchikh, and S. Otmane, An valuation of Camera Methods for an Augmented Reality System: Application to Teaching Industrial Robots, ser. Lecture Notes in Computer Science. Springer Berlin Heidelberg, 213, vol. 742, ch. Transactions on Computational Science XVII, pp [2] P. Besl and N. McKay, "A method for registration of 3-d shapes," I Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 2, pp , [21] C. P. Lu, G. D. Hager, and. Mjolsness, "Fast and globally convergent pose estimation from video images," I Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 6, pp , 2. RFRNCS [1] M. Maidi, F. Ababsa, and M. Mallem, "Handling occlusions for augmented reality systems," urasip Journal on Image and Video Processing" vol. 21, pp. 1-12,