Visual Tracking. Frédéric Jurie LASMEA. CNRS / Université Blaise Pascal. France. jurie@lasmea.univ-bpclermoaddressnt.fr

Transcription

1 Visual Tracking Frédéric Jurie LASMEA CNRS / Université Blaise Pascal France jurie@lasmea.univ-bpclermoaddressnt.fr

2 What is it? Visual tracking is the problem of following moving targets through an image sequence. Tracking systems must address two basic problems: motion and matching. Motion problem: predict the location of an image element or a spatial attitude of the object being tracked in the image, using previous positions. Identify a limited search region in the image or in the pose space in which the element is expected to be found with high probability Matching problem: (also known as detection or location) identify the image element within the designated search region. Find correspondences (frame to frame, model to frame).

3 Introduction Favorite solutions for the motion problem are Kalman Filter [P.S. Mayback - Stochastic models, estimation and control, vol1/2. Academic Press, London, 1979], an optimal recursive estimator of the state of a system (under restrictive assumptions), or Particles Filters [S. Arulampalam, S. Maskell, N. Gordon, T. Clapp, A tutorial on particle filters, IEEE Trans. on signal processing, 2001] which can deal with non-gaussian estimations.

4 The matching problem requires a similarity metric to compare candidate pairs of image elements between: current frame / next frame current frame / object model This is closely related to the correspondence problem (stereovision / object recognition) where the same image elements must be detected and matched in two or more different images. A tracking specific problem is data association that is finding the true position of the moving target in presence of equally valid candidates (i.e. intersecting trajectories) [Y. Bar-Shalom and T.E. Fortman, Tracking and data association, Academic Press, London, 1988].

5 Performances / characteristics that are sought of a video tracker: Robustness to clutter: it should not be distracted by image elements resembling the target being tracked Robustness to occlusion: target should not be lost because of temporary target occlusion, but resumed correctly when the target re-appears False positive and negatives: only valid targets should be classified as such, and any other image elements ignored (the number of false alarms should be as small as possible) Agility: the tracker should follow target moving with significant speed and acceleration Stability: to lock and accuracy should be maintained indefinitely over time Computational cost: compatible with robotics (real time if possible)

6 How does it work? Tracking as been studied in computer vision and robotics for several decades. A huge variety of algorithms has been reported. We review only the most relevant literature. We concentrate on: 1. Motion estimation (a) Kalman filter (b) Particle filter (c) Examples

7 2. Matching problem (a) Template based approaches (b) Feature based approaches (c) Contour based approaches (d) Examples

8 PART 1 - Motion estimation To define the problem of tracking, consider the evolution of the state sequence of a target, given by is a possible non-linear function of the state is a process noise sequence and are dimensions of the state state and process noise vectors respectively

9 The objective of tracking is to recursively estimate from measurements where: is a possibly non-linear function is a process noise sequence andare dimensions of the measurement and measurement noise vector, respectively. based on the set of all available mea- up to time In particular, we seek filtered estimates of surements

10 From a Bayesian perspective, the tracking problem is to recursively calculate some degree of belief in the state, taking different values, given the data. up to time at time It requires to construct the pdf. It is assumed that the initial pdf is available. may be obtained recur- With these assumptions, the pdf sively in two stages : prediction and update.

11 Prediction stage Suppose the required pdf is is available at time The prediction stage involves using the system model to obtain the prior pdf of the state via the Chapman-Kolmogorov equation : It is assumed that the evolution of the state sequence describes a Markov process of order one; in this case:. The probabilistic model of state evolution is known.

12 becomes available, and this may Update At time step, a measurement be used to update the prior via Bayes s rules: where the normalizing constant the matching algorithm provides.

13 This recursive propagation of the posterior density is only a conceptual solution in that in general it cannot be determined analytically. Solutions do exist in a restrictive set of cases, including the Kalman filter and grid based filters. When the analytical solution is intractable, extended Kalman filter, approximate grid based filters and particle filters approximate the optimal Bayesian solution.

14 Kalman Filter The Kalman filter assumes that the posterior density at every time step is Gaussian and hence parametrised by a mean and a covariance. If Gaussian is Gaussian, it can be proved that is also are drawn from Gaussian distributions of known parameters and is known and is a linear function of and is a known linear function of and

15 and are known matrices defining the linear functions. The co- and are respectively and variance of consider the case when tically independent.. Here we and have zero means and are statis-

16 The Kalman algorithm can be viewed as the following recursive relationship: where

17 and covari- and where ance and: is a Gaussian density with argument mean are the covariance of the innovation term respectively. and the Kalman gain, This is the optimal solution to the tracking problem - if the highly restrictive assumptions hold.

18 Grid Based Methods if the state space is dis- Provide optimal recursion of the filtered density, crete and consists of a finite number of state. Suppose the state space at time consists of discrete states let the conditional probability of that state, given measure- be denoted by is that For each state ments up to time. Then the posterior pdf at k-1 can be written as where is the Dirac delta measure.

19 using previous equation, the prediction and update equations are, respectively: where:

20 Sub-Optimal Methods In many situations, the assumptions made above do not hold. Approximations are necessary Extended Kalman filter: based on the approximation that is approximated by a Gaussian and where now are non-linear functions, that are linearized using the first term in a Taylor expansion. and Approximate grid-based methods: if now the state is continuous, but can be decomposed into cells, then a gris-based method can be used to approximate the posterior density. Particle filters: the key idea is to represent the posterior density function by a set of random samples with associated weights and to compute estimates based on these samples and weights.

21 Particle Filters Let denotes a Random Measure that characterises the posterior pdf, where is a set of support points with associated weights. The weights are normalised such that Then the posterior density at can be approximated as The weight are chosen according to the principle of importance sampling.

22 Importance Sampling is a probability density from which it is difficult to draw sam- can be evaluated (and so up to proportionality). Suppose ples, but for which Let be samples that are easily generated from a proposal, call an importance density. A weighted approximation to the density is given by where is the normalized weight of the i-th particle.

23 Sequential case: one could have samples constituting an approximation to, and want to constitute an approximation to with a new set of samples. If the importance density is chosen to factorise such that and if then the importance density becomes only dependent on and In this scenarios, only need to be stored, and so one can discard the path and history observations,.

24 The modified weight is then and the posterior filtered density It can be shown that as posterior density the approximation approaches the true can be approximated as

25 Good choice of importance density The optimal importance density function to be. But sampling from is generally untractable. has been shown It is convenient - and very common - to chose the importance density to be the prior This yields finally: This choice is crucial.

26 Degeneracy and Resampling Degeneracy phenomenon is a common problem with particle filters: after a few iterations, all but one particles will have a negligible weight. The basic idea of resampling is to eliminate particles which have small weights and to concentrate on particle with large weights. The resampling step involves generating a new set by resampling (with replacement) times from an approximate discrete representation of given by: After the resampling, the weight are now reset to Systematic resampling is the generally preferred scheme

27 CONDENSATION

28 -

29 PART 2 - Data association Contributions can be structured depending on the kind of informations that are detected and associated in the images. Window tracking / template matching : the target is just a small image window, Feature tracking: targets are image elements with specific properties, called image features that are detectable parts of images. Feature tracking takes place by first locating features in two subsequent frames then matching each feature together of with a model.

30 local features: cover a limited area of an image (edges, corner points...) extended features: large part of the image (contour of basic shapes...) Planar rigid shapes : appearances of planar rigid shapes in motion are used to estimate the motion. Solid rigid 3D objects: the logical next step to the above with rigid 3D object. Deformable contours: the idea of projecting a CAD like model to predict the full target evolution often failed. Deformable regions are more flexible.

31 Feature tracking The main advantage is their greater robustness to clutter The price is more complex matching algorithm, as local features generally don t have discriminant attributes. Best studied features are contours, because they are meaningful features in structured scenes. Trackers can incorporate motion models and shape deformations constraining the possible deformations. very common approach: predict the state (position/orientation of the camera/objects) and use this prediction to predict the position of the feature in the image, find the correspondences between image and model feature in order to update the state (estimation). A very large number of papers in this area, (Magnus Anderson, Tracking Methods in computer vision, PHD, CVAP, KTH). The few following examples are taken as illustrations.

32 frame t frame t+1 frame t+1 Model feature Image feature a/ b/ c/ Magnus Anderson, Tracking Methods in computer vision, PHD, CVAP, KTH

33 D.G. Lowe, Robust model-based motion tracking through the integration of search and estimation, IJCV 8(2), 1992 : matching with minimal search

34 H. Kolling and H. Nagel, 3D pose estimation by directly matching polyhedral models to gray value gradients, IJCV 23(3), 1997 : use of grey levels values around edges

35 P. Wunsch and G. Hirzinger, Real-time visual tracking of 3D objects with dynamic handling of occlusions, ICRA 97: kalman + pose constraints derived from image features (pose optimisation) + feature selection

36 T. Drummond and R. Cipolla, real-time tracking of complex structures with on-line camera calibration, BMVC 99: model rendered, hidden line removal, matching, on-line calibration

37 K. Toyama, A. Blake, Probabilist tracking is a metric space, ICCV 01

38 Window tracking The simplest target possible. Can be tracked by correlation-like correspondence method [E. Trucco and A. Verri, Introductory techniques for 3D computer vision, Prentice Hall, 1998] Standard method explore all possible candidate windows within a given search region in the next frame and pick up the one optimising an image similarity metric. Typical metrics include SSD (sum of square difference) SAD (sum of absolute differences) or correlation The window tracked is assume either translating, or subject to linear or homographic deformations. As windows are unprocessed sub-image, these techniques can be applied at any image position, and can be used in stereo systems to produce dense surface reconstruction.

39 Lucas - Kanade tracker The major drawback of correlation like techniques is their computational cost With small inter-frame displacements, a window can be tracked by optimizing some matching criterion. B.D. Lucas, T. Kanade, An iterative image registration technique with an application to stereovision, IJCAI, 1981 : study the case of translations

40 Basic ideas : Taylor expansion (first order approximation) with by writing the tracking consists in finding the minimum of

41 At the end, we obtain the well known equation where and can be obtained by using this iterative procedure:

42 Shi - Tomasi tracker In CVPR 94, Shi and Tomasi, propose an extension of this framework to the case of affine motion. The displacement vectoris given by They obtain a similar equation:

43 -

44 Generalisation to complex motions Let be the brightness value at the location in an image acquired at time. Let the set at time. is a vector of the brightness values of the target region be the set of N image locations which define a target region. We refer to initial time ( ). as the reference template. It is the template which is to be tracked; is the parametric motion model denotes a set of parameters. where denotes an image location and time set of The, the position of the template is image locations, also denoted. is denoted. At

45 With these assumptions, tracking the object at time t means compute. such that We write the estimate of the ground truth value. The motion parameter vector of the target region squares function: can be estimated by minimizing the least Hager et al. propose a very straightforward and efficient computation of by writing: (1) where denotes the time between 2 successive images. This formulation makes real time implementations on standard workstations possible. can be obtained with small online computation, and If we write previous equation can be written: and, the (2)

46 Jacobian Images approach G.D. Hager and P.N. Belhumeur, Efficient region tracking with parametric models of geometry and illumination, PAMI 20(10), 1998 The previous equation shows clearly that Jacobian matrix. For this reason, we will record it as estimate of plays the role of a. The can be obtained by using the Jacobian image.. are small, it is possi- in a Taylor In order to simplify notations, we will denote If the magnitude of the components of ble to linearize the problem by expanding series about, and and by

47 at glected; is the Jacobian matrix of (3) whereare the high order terms of the expansion that can be newith respect to time, and is the derivative of I with respect to. By neglecting the and with the additional approximation, assuming the previous equation becomes

48 (5) By writing (4) denotes the transposi- we obtain a direct expression of tion of). (where By combining previous equations we obtain :

49 The straightforward computation of requires the computation of the image gradient with respect to the component of vector. Therefore depends on time-varying quantities and has to be completely recomputed at each new iteration. This is a computationally expensive procedure. Fortunately, it is possible to express as a function of the image gradient of the reference image, allowing to obtain with only few on-line computations [?]. Equation (5) involves the computation of the difference in intensity. It is possible to relate to the reference template given in the first image. If we assume that the pattern is correctly localized after the correction of the motion parameter, the image consistency assumption gives, leading to the relation

50 In this case, the previous equation links the difference between the template in the current region and the target template with a displacement aligning the region on the target. With these notations, the tracking consists in evaluating updating and consequently obtaining according to the equation:, and finally. -

51 -

52 can be seen as a set of Tracking by direct regression F. Jurie and M. Dhome,Hyperplane approximation for template matching, PAMI 24(7), 2002 We propose a different interpretation of the computation of matrix. Equation hyperplanes. In this section, matrix is written to distinguish it from. However it plays the same role. Let us write the elements of matrix (time is removed in order to simplify notations). The previous can be written:

53 , with Under this form, we can clearly observe that the coefficients ofhyperplanes that can be estimated by using a regression. are During the training stage, the region of interest is moved from the reference position (provided manually by selecting the region of interest in the first image) to. Once the region is moved, the vector is computed. This disturbance procedure is repeated with times,. we collect Finally and couples. It is then possible to obtain a matrix such that: is minimal.

54 can be com- By writing, we obtained an overdetermined system. puted from the transposition of. by where, and assuming denotes

55 EigenTracking M.J. Black and A.D. Jepson, EigenTracking: robust matching and tracking of articulated objects using a view-based representation, IJCV 26(1), Eigenspace approaches: given a set of images, eigenspace approaches construct a small set of basis images that characterize the majority of the variations in the training set. This representation is used to approximate the training set. For eachimage in the training set of scanning the image. images, we construct a 1D vector by

56 Each of these 1D vectors becomes a column in a matrix the number of training images is less than the number of pixels. We assume that is decomposed using SVD: where is an orthogonal matrix of the same size than representing the principal component directions in the training set. is a diagonal matrix with singular values sorted in decreasing order along the diagonal. is a orthogonal matrix encoding the coefficients to be used in expanding each column of in terms of the principal component direction. We can approximate some new vector as: where the and. are scalar values that can be computed by taking the dot product of

57 Robust Matching. Let The approximation e correspond to the least square estimates to the the This approximation does not work in case of occlusions. In that case, a robust error norm have to be used to compute This objective function can also be written: be:

58 Parametric Motion. Let represents an image transformation where represents the horizontal and vertical displacements at a pixel. The parameters have to be estimated. The goal is then to minimize: and They propose to use an optimization algorithm to find the

59 - -

60 Mean-shift Tracking D. Comaniciu, V. Ramesh and P. Meer, Real Time tracking using Mean Shift, CVPR2000 best paper. Given a set multivariate kernel density estimate with windows radius in the point x can be expressed as of points in the d-dimensional space the, computed where is a profile kernel, that commonly is

61 Assuming the derivative of for all exists, and by taking the estimate of the density gradient, the mean shift procedure which consists in computing iteratively where maximizes the density

62 Application to tracking: find the most probable target position. The dissimilarity between the target model and the target candidates is based on the comparison of color histograms. Restricted to 2D translations. Basic idea: evaluate the probability of each pixel in a windows, compute the mean of these probabilities and move the windows in this direction. Similar work: P.Perez, C. Hue, Vermaak, Gagnet, Color-based probabilistic tracking, ECCV02. Color histogram comparison using a sequential Monte Carlo Algorithm

63

64 Jepson/Fleet/Maraghi Tracker A. Jepson, D. Fleet, T.Maragui, Robust Online Appearance Models for visual tracking, CVPR01. A framework for learning robust, adaptative, appearance models to be used for motion based tracking. Mixture of stable image structure learned over long time course and 2-frames motion informations (features are steerable filters) An on line EM algorithm is used to adapt the appearance model parameters over time. The optimal motion is computed by using the likehood of the windows, knowing the appearance model as the objective function of an optimisation algorithm.