Performance evaluation of multi-camera visual tracking

Transcription

1 Performance evaluation of multi-camera visual tracking Lucio Marcenaro, Pietro Morerio, Mauricio Soto, Andrea Zunino, Carlo S. Regazzoni DITEN, University of Genova Via Opera Pia 11A Genoa - Italy s: {pmorerio}@ginevra.dibe.unige.it {lucio.marcenaro,carlo.regazzoni}@unige.it Abstract Main drawbacks in single-camera multi-target visual tracking can be partially removed by increasing the amount of information gathered on the scene, i.e. by adding cameras. By adopting such a multi-camera approach, multiple sensors cooperate for overall scene understanding. However, new issues arise such as data association and data fusion. This work addresses the issue of evaluating the performance of a multicamera tracking algorithm based on Rao-Blackwellized Monte Carlo data association (RBMCDA) on real data. For this purpose, a new metric based on three performance indexes is developed. I. INTRODUCTION Video-based tracking techniques are widely used in different fields related to ambient intelligence and scene understanding. Video surveillance is only one of the possible applications ranging from security to man-machine interaction; elderly people monitoring for sanitary assistance or rehabilitation is only one of the interesting fields where video-sensors and automatic processing algorithms can be used efficiently. Moreover, technological improvements in imaging hardware and related price cuts, enabled na even more pervasive usage of such sensors: IP based video cameras allowed easy hardware installation and remote monitoring and configuration while mega-pixel sensors and efficient video compression algorithms greatly improved quality and spatio-temporal resolution of acquired images. In this scenario, visual tracking is one of the principal techniques that are used for extracting high-level descriptors of the scene. The main purpose of tracking algorithms is to estimate moving objects trajectories by correctly assigning them a specific label that must be propagated over-time as the object moves on the scene. Each tracking technique starts from low-level data acquired directly from images, thus can be modelled as a state estimate problem starting from noisy observations. In this sense, tracking can be seen as a filtering process for cancelling observations noise and extracting the actual movement of each object in the scene. Several highlevel applications can be developed based of trajectories and detected movements, such as people counting, traffic monitoring, abandoned objects detection, automotive safety, etc.. The complexity of visual-tracking is very high and main problems that must be solved are related with unpredictable object movements, targets or scene appearance variations, nonrigid parts, occlusions (intra-, inter- or environmental) and sensor undesired movements. A multi-camera approach can be adopted in order to deal with these issues, although data fusion and association problems must be considered when multiple sensors have to cooperate for overall scene understanding. The work is organized as follows: Section II reviews the probabilistic approach to visual tracking and introduces the reader to particle filtering, Rao-blackwellized particle filtering and Rao-blackwellized data association (RBMCDA). Section III describes the experimental set-up and model used for performance evaluation. Section IV introduces the novel metric and performance indexes. In section V performance evaluation results are given. Conclusion are drawn in section VI. Future developments of the work are also proposed. II. PROBABILISTIC APPROACH Visual tracking problems can be mathematically modelled through probabilistic reasoning. Target positions can be stored in a state vector containing its descriptive features: target state must be estimated starting from noisy observations obtained from acquired images. Stochastic state and observation models are needed for this kind of approach. Bayesian theory can be used to estimate the state s a posteriori probability density function (pdf ): after the pdf is estimated, it can be used for evaluating the new state and for measuring the state estimate precision. The hidden state vector to be estimated can be expressed as x t R nx where n x is the state vector dimension and t N is associated to the considered temporal instant. The state vector x t can contain object position, speed, acceleration or shape related features (e.g., corners). Available observation vector z 1:t = {z 1 z t } at time instant t will be used for a posteriori state estimate. An on-line solution is needed for object tracking as one must be able to estimate the new state of each tracked object as new observations become available. The a posteriori state probability density function p(x t z 1:t ) is computed recursively through a two-steps prediction-updating procedure. If the previous step posterior pdf p(x t 1 z 1:t 1 ) is known, the pdf of the following state can be predicted by means of the state transition model p(x t x t 1 ), given by: p(x t z 1:t 1 ) = p(x t x t 1 )p(x t 1 z 1:t 1 )dx t 1 (1)

2 Moreover, by applying the Bayes theorem, it is possible to obtain the a posteriori state pdf p(x t z t ) through the following update equation that shows its dependency by observation probability p(z t z t 1 ) and the likelihood probability coming from the observation model p(z t x t ) p(x t z 1:t ) = p(z t x t )p(x t z 1:t 1 ) p(z t z 1:t 1 ) This formulation assumes that x t is a state of a Markov process with initial distribution p(x 0 ). Equations (1) and (2) have an analytical solution only in the case of linear-gaussian models. The Kalman filter provides the optimal solution. In the case of non-linear or non-gaussian models, a numerical approximation, like the one provided by applying a Particle Filter (PF), must be made [1]. A. Particle filtering The Particle Filter approximates the solution of the Bayesian filtering problem by representing the a posteriori pdf as a finite set of samples (particles) χ t = {x m t, wt m } Ns m=1 that are weighted by using the importance sampling technique. According to this algorithm, a set of N s samples { x m t } Ns m=1 are created starting from a reference distribution q(x 0:t 1 z 1:t ) instead of the a posteriori pdf p(x 0:t 1 z 1:t ) that, in general, cannot be expressed mathematically. The difference between the reference pdf and the a posteriori one is managed through a correction procedure, by using different weights. Under independence hypotheses, the reference distribution can be expressed as q(x t x t 1, z t ) and the weights can be computed by using the Sequential Importance Sampling (SIS) technique as w (m) t w (m) p(z t x t )p(x t x t 1 ) t 1 (3) q(x t x t 1, z t ) According to the SIS technique, N s samples x m t } can be generated starting from the importance distribution q(x t x t 1, z t ) and correspondent weights w (m) t can be computed by using Eq. (3). After a normalization step, the a posteriori pdf at time t can be approximated with a sum of N s weighted Dirac deltas p(x t z 1:t ) δ(x t x (m) t ). (4) w (m) t m=1:n s It has been demonstrated that this algorithm leads to a unstable behaviour due to the continuous increase of the weighting coefficients variance. This can be solved by using re-sampling procedures that try to redistribute particles in such a way that softly weighted ones are cancelled while those associated to higher weights are duplicated. The efficiency of particle filter strongly depends on the appropriateness of the importance distribution q(x t x t 1, z t ) that must be close to the a posteriori pdf. Within visual tracking, the definition of a reliable importance distribution is a complex task because of the high variability of movements and deformations that can be associated to a monitored target. Typically, several features are extracted and tracked from a single target: this problem can be addressed as a multiple (2) target tracking problem for each object. This is a challenging problem because of the uncertainty of the measurement origin (i.e., the centre of the object) and the cluttered and missed detections due to inaccuracies of the feature extractor and/or possible (self)occlusions. It can be demonstrated that data association uncertainty in tracking leads to a NP-hard problem, i.e. the association hypotheses grow exponentially with time as in the Multiple Hypotheses Tracker (MHT)[2]. As a result, approximations are required in order to make the problem feasible for a reasonable number of targets. In the past few years, great focus has been given to sequential Monte Carlo (SMC) strategies under the Bayesian framework because they apply a semi-soft-decision data association approach, i.e. hard-decision associations per particle while keeping multiple parallel hypotheses associations with all particles. Different strategies avoiding explicit enumeration of all possible hypotheses, thus making this approach to scale well with an increasing number of targets, are described in [3]. B. Rao-blackwellized particle filter The algorithm that will be considered hereinafter is an extended version of the Rao-Blackwellized Monte Carlo data association (RBMCDA) [4], which is basically an extended Rao- Blackwellized particle filtering based multiple target tracking algorithm [5]. The algorithm models as stochastic processes not only the state evolution of the target, but also its creation and suppression, i.e. the starting and ending time instant estimations for target tracking are not based on heuristics, but are accounted for in the probabilistic model. The basic idea behind the Rao-Blackwell algorithm is that, due to the conditioning of the data association and the targets creation or suppression, the a posteriori probability distribution of target states can be approximated by means of Gaussian pdf s (marginalization of the filtering or Rao-Blackwellization). Suppose the state x can be split into two substates r and s such that p(x t x t 1 ) = p(s t r t 1, s t 1 )p(r t r t 1 ), (5) and p(s 0:t z 1:t, r 0:t ) is analytically tractable. Decompose the posterior as p(r 0:t, s 0:t z 1:t ) = p(s 0:t z 1:t, r 0:t )p(r 0:t z 1:t ). (6) The term p(s 0:t z 1:t, r 0:t ) can be analytically estimated, while p(r 0:t z 1:t ) can be derived by means of the particle filter as p(r 0:t z 1:t ) = p(z t z 1:t 1, r 0:t )p(r t z t 1 )p(r 0:t 1 z 1:t 1 ) p(z t z 1:t 1 ) (7) This way, target states can be computed, while SMC particle filtering only need to be applied to the data associations and creation and deletion processes. This greatly reduces computational complexity, thus increasing the efficiency of the particle filter. Moreover, the Rao-Blackwell theorem states that the mean squared error of the RaoBlackwell estimator

3 does not exceed that of the original estimator (i.e. of the nonmarginalized particle filter). In this framework, a data association event can be modelled through a data association indicator c t, which takes T t + 1 values: c t = 0 for clutter and c t = j, j = 1,..., T t where T t is the total number of targets at time instant t A priori probabilities for data association can be modelled as a m-order Markov random chain with uniform priors over targets and clutter: p(c t c t 1,..., c t m, T t 1,..., T t m ) = T t 1. (8) Clutter measurement probabilities can be uniformly distributed in a measurement space of volume V, namely p(z t c t = 0) = 1/V. (9) Target measurements and target dynamics are basically modelled as in a Kalman Filter (linear dynamic model and Gaussian noise): p(z t x t,j, c t = j) = N(z t H t,j x t,j R t,j ), (10) p(x t,j x t 1,j ) = N(x t,j A t 1,j x t 1,j Q t 1,j ), (11) Fig. 1. Four different views of the scene where the measurement, covariance, transition and process noise covariance matrices H t,j, R t,j, A t,j and Q t,j can be different for each target. Extensions to non-linear models is straightforward (EKF, UKF) Target births only occur jointly with an association event: no association to a newborn target, no birth. Equivalently, the state prior is constant until a first measurement is associated, i.e. the target state does not follow the dynamic model before that moment. This very time instant is than identified as target birth. After an association is made, the lifetime of the target is given a probability density p(t d ). A Gamma distribution Γ(α, β) has been chosen for modelling such a probability. Increasing values of α shift the distribution to the right, thus resulting in lower probabilities for small lifetimes, while increasing values of β somehow flatten the distribution. Basically, by modelling deaths, targets which have not been assigned a measure for a certain amount of time are removed. If the last association with target j is made at time τ and at the previous time step t 1 the sampled hypothesis correspond to an alive target, then the probability of having a dead target at current time step t is p(death j ) = P (t d [t 1 τ, t τ] t d t 1 τ). (12) III. SETUP AND MODEL RBMCDA has been tested through a Matlab toolbox [6], which implements all the algorithm s functionalities and eventually provides simulations for testing. Tests have instead been run on real sequences. The dataset that has been considered can be found in [7], together with the ground-truth. It consists in four video sequences corresponding to four different views (i.e. different cameras) of a scene. Six people randomly access Fig. 2. Ground projections of tracked objects ( ) and ground-truth ( ) a 5,5 m x 5.5 m area in a room and walk around quite as much randomly for about 150 seconds at 25 fps. Together with the ground-truth data, homography parameters (coded in 3x3 matrices) for each cameras are supplied, to allow ground plane projections (or, equivalently, to allow a top-view shot). A screenshot of the four video framing is show in Fig. 1, while projections of the tracked objects on the world ground plane are given in Fig 2. Tracked objects are marked with different ( ) symbols (one colour for each camera view), while the ground-truth with a ( ). The model that has been employed is now briefly sketched: the considered state vector is x t = [ x y ẋ ẏ ] T, (13)

4 where x, y, ẋ and ẏ are the horizontal and vertical coordinates and their time variations respectively. The measurement vector is simply given by averaged position measurements z t = [ x y ] T. (14) The transition matrix for the state dynamic evolution is 1 0 t t A t = , (15) where t is simply 1 as only discrete unitary time steps are here considered. The observation matrix is [ ] H t =. (16) The noise R t is taken to be zero-mean Gaussian, with zero offdiagonal correlation terms, while the measurement covariance matrix is of the form Q t = (17) 0 0 σ σ where sigma is an appropriate constant. IV. PERFORMANCE EVALUATION To evaluate the algorithm performances, global traditional metrics, which exploit the information carried by all the particles, could not be employed. The key point is that this leads to an esteem that is corrupted by misassociations between particles and targets, thus being quite meaningless: a particle can in be associated to different targets. Even by considering the particle with higher weight, the association problem persists and its complexity grows exponentially with the number of targets. A novel metric has been therefore designed based on three performance indices which are discussed in the following, namely a fragmentation index I F ; a tracking index I T ; a visibility index I V. a) Fragmentation index: This index focuses on trajectory fragmentation and tries to measure its amount. It consists in the ratio of the difference between the actual number of trajectories T gt and the number of trajectories T generated by the algorithm I f := T T gt T gt. (18) Such an index approaches the optimal value of zero as the number of generated trajectories approaches the number of the real ones; it takes positive values when more trajectories are generated, i.e., roughly speaking, as fragmentation takes place. Negative values of I f basically result from losing a target without retrieving it. b) Tracking index: By means of this index, the number of frames in which target position estimations are effectively related to real data is accounted for (independently of trajectories fragmentation). Equivalently, the index measures the amount of time in which position estimations are supported by ground-truth data, as the fps rate is kept constant. Given the set of all the measured trajectories γ i : ( t) i R 2, let S be the sum of all the supports of γ i, S = i ( t) i and S gt be the same quantity computed for the ground-truth trajectories. The index is defined as I f := S gt S S gt. (19) It approaches zero (optimal value) as fast as the generated trajectories can hook up the when a birth occurs. In fact the former never overstay the latter, as in the case that has been considered real trajectories persist up to the end of the sequence. c) Visibility index: This performance index measures the difference between the actual number of people on the scene (N gt ) and those estimated by the algorithm (N), averaged over all the processed frames N f I v := 1 N f N f (N gt N) i (20) i=1 This index has an optimal value of zero, as the previous. However, its optimization turns out to be extremely hard, as it is strongly affected by frequent and unpredictable occlusions in the scene V. RESULTS Performances have been evaluated by tuning some of the parameters introduced in section II-B, namely P b, which models the birth probability density, and the two parameters α and β, which shape the Gamma function that models the lifetime probability of each target. Two 500-frame scenes with a different intrinsic complexity have been studied.the will be referred to as Scene 1 and 2 (simpler and more complex scene respectively). The three parameters have been varied in turn, while keeping the two other fixed. Table I summarizes some results for Scene 1. The ranges of α and β and P b have been explored systematically to localize local maxima for the tree performance indexes. Low values of α result in short lifetimes, and in fact in trajectories fragmentation, as one can infer from the positive values of I F (Table I(a)). On the other hand, the values of I T and I V do not show monotonicity; instead there seems to be a minima at α = 4. By varying β (Table I(b)), the fragmentation I F remains constant at the optimal null value, while the other two indexes behaviour is quite fuzzy. β = 1 seems to give the best fit. Eventually (Table I(c)), it can be noticed that better performances are obtained with the higher considered value of P b, i.e. P b = I V and I T improve as P b rises: this is basically due to the fact that there is less delay in hooking the target as the birth probability increases. Even higher values of

5 (a) Varying α, β = 1, P b = 0.01 α I F I T I V (b) Varying β, α = 4, P b = 0.01 β I F I T I V (c) Varying P b, α = 4, β = 1 P b I F I T I V TABLE I Scene 1 (a) Varying α, β = 4, P b = 0.1 α I F I T I V (b) Varying β, α = 8, P b = 0.1 β I F I T I V (c) Varying P b, α = 8, β = 5 P b I F I T I V TABLE II Scene 2 P b result in positive values of I F (as for P b = 0.1), as the algorithm generates too many targets. Table II summarizes some results for Scene 1. Here scene is complicated by more targets, resulting in more occlusions and misdetections. The optimal value for α has been found for α = 8 (Table II(a)). It can be noticed that the value of the three indexes show here a kind of local minimum: in particular, the algorithm estimates more trajectories than the real ones for α < 7 (I F > 0) and less for α > 8 (I F < 0). Table II(b) shows how β = 5 represents some kind of asymptote turning point for the values of all the indexes. in particular I V turns positive, meaning that too delayed target deaths result in too many people detected on the scene. From the analysis of Table II(c), it look like that than algorithm cannot avoid fragmenting trajectories in a more complex scene. For Scene 2 a deeper analysis was performed by jointly varying both the indexes α and β, while keeping P b fixed (Figure 3). The vertical coordinates of the plot is z = I F + I T + I V. It can be noticed that z clearly decreases as α and β increase. This is coherent with the above mentioned fact that real trajectories in the scene last until the end of the sequence, i.e. no target exits the scene before the very end. As a matter of fact, high values of α and β allow for long lifetimes. By exploiting the best values of α and β just recovered, the average distance error between real and estimated trajectories has been computed. Its dependence on the number of particles has been also evaluated. Results are given in Table III. It can be noticed that the lowest average error corresponds to n = 10 particles. However, as one can infer from the (negative) value of I F given in the table, this is due to the fact that less trajectories have been generated. Being the number of trajectories equal, better performance is reached as the number of particles increases. This is perfectly coherent with the theory, which states performance improvement on particle Fig. 3. P b = 0.01 increment, especially in noisy conditions. Number of particles Average error (pixels) I F TABLE III AVERAGE ERROR BETWEEN ESTIMATED AND REAL TRAJECTORIES (GROUND-PIXELS) VI. CONCLUSIONS A set-up for RBMCDA testing has been presented. A novel metric based on three indexes has been exploited in order to evaluate the algorithm performance, both qualitatively and quantitatively. Each index accounts for a typical problem in

6 visual tracking. Performance has been evaluated as α, β and P b, which model target birth and death probabilities, vary. Future developments of this work surely include comparative performance analysis with other trackers. Including simple particle filter (non-marginalized filter) evaluation could give evidence to the Rao-Blackwell theorem. REFERENCES [1] A. Doucet, S. Godsill, and C. Andrieu, On Sequential Monte Carlo Sampling Methods for Bayesian Filtering, Statistics and Computing, vol. 10, pp , [Online]. Available: [2] D. B. Reid, An algorithm for tracking multiple targets, IEEE Transactions on Automatic Control, vol. 24, pp , [3] J. Vermaak, S. J. Godsill, and P. Pérez, Monte carlo filtering for multitarget tracking and data association, IEEE Transactions on Aerospace and Electronic Systems, vol. 41, pp , [4] S. Särkkä, A. Vehtari, and J. Lampinen, Rao-blackwellized particle filter for multiple target tracking, Information Fusion Journal, vol. 8, p. 2007, [5] A. Doucet, N. De Freitas, K. Murphy, and S. Russell, Rao-Blackwellised particle filtering for dynamic Bayesian networks, in Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence, 2000, pp [6] LCE Models and Methods: RBMCDA Toolbox for Matlab - TKK - [7] CVLAB, EPFL -