Evaluating the Performance of Systems for Tracking Football Players and Ball

Transcription

1 Evaluating the Performance of Systems for Tracking Football Players and Ball Y. Li A. Dore J. Orwell School of Computing D.I.B.E. School of Computing Kingston University University of Genova Kingston University Kingston Upon Thames, U.K. Genova, Italy Kingston Upon Thames, U.K. Abstract In this paper, we discuss the different approaches used for evaluating the results of tracking algorithms, and in particular for analysis of football (soccer) tracking results. The focus of the study is on systems with multiple static cameras. The appropriate data representation and ground truth capture methods are discussed, and evaluation measures that indicate the performance of any given automatic tracker are presented. The evaluation method is demonstrated to compare results of an implemented multi-camera tracker. 1. Introduction Within the field of computer vision, comparison of algorithms requires an appropriate evaluation methodology. In this paper, we consider the different approaches that have been used to evaluate tracking algorithms, and apply this analysis to a relatively new domain: the tracking of football (soccer) players with multiple static cameras. Historically, tracking evaluation has used a diverse range of measures and procedures to establish a performance metric. The choice will inevitably depend on the target application, as the priorities will vary for different applications. Tracking of human motion has received significant attention over the last decade. Our impression of the performance evaluation of tracking techniques is that, to a large extent, rigorous quantitative evaluation has been neglected, or in many cases it has been performed informally on a few visual demonstrations. While this is adequate for demonstrating new techniques, it does not necessarily provide conclusions that are valid across domains. It also introduces a degree of subjectivity into the analysis of the results. As the field of visual surveillance matures, an evaluation of the complete tracking systems is increasingly important for several reasons: comparison of different algorithms and parameters therein, analysis of points of failure and comparison of a single algorithm across different data domains. However, specification and implementation of a full evaluation component for a tracking system can be a difficult task. In the football tracking domain, there are several important criteria and factors that determine the most appropriate evaluation methodology. The accuracy of camera calibration and synchronisation places practical limits on the accuracy with which the ground truth can be established. Innovative procedures are used to mark the 3D position of the ball, from single or multiple views. The intended use for the tracking data influences the most appropriate metric. For example, the identity of each player within a team is critical, if the data is to be used for coaching applications, but it is less so for spectator applications. We demonstrate how distance-based metrics are used to evaluate the relative performance of tracking algorithms or parameters therein. In the next section, we review the work relevant to the evaluation of tracking, followed by the description of the tracking implementation that is used to generate results for the evaluation process. In section 5 we present the steps we use to obtain the ground truth data. The procedure for evaluating the tracking results is described in Section 6 and the evaluation results are presented in Section Previous Work 2.1 Evaluation Methods For the visual surveillance systems, performance evaluation methods can be divided in two categories: evaluation with and without ground truth (GT) data set. This data set will contain information that is compared to the automatic tracker output (ATO), but is guaranteed to be correct up to some estimated tolerance. First, we discuss evaluation methods without access to a GT data source. This approach enables practical tracking evaluation on a large quantity of video data and can be useful in real time determination of tracking failure. There have been several techniques proposed for tracking performance evaluation [3, 17] where GT is not available, or it has been decided not to be collected. In [17], an algorithm for automatic performance evaluation that does not require GT data is presented. It defines several metrics based on the assumptions of direction and speed consistency, motion smoothness, and constancy of shape, area and appearance. Therefore measuring the de /05/$ IEEE 632

2 gree to which these qualities are present in the output from the tracking system, is a valid indicator of the performance of the system. One limitation is that it is only suitable for scenarios in which that assumption is correct, i.e. moving objects that will not change their direction and speed dramatically such as moving vehicles. In addition, it does not necessarily discriminate between successful and unsuccessful tracking, in circumstances where there are multiple moving objects which create occlusions. In [3], the authors propose a technique of using synthetically generated video sequences to evaluate tracking performance. The method constructs sequences containing complex motion scenes, by superimposing motion from isolated targets that have been successfully tracked. It allows the generation of a large variety of data sets representing different tracking scenarios and demonstrates a method to assess the sequences based on the occurrence and duration of dynamic occlusions which are most likely to cause the tracking algorithm to fail. The GT for these data sets is automatically generated from the procedure by which the synthetic sequences are created. A number of methodologies [15, 13, 4] have been defined for tracker performance evaluation provided with GT data. In [15], two approaches to measuring people tracking performance are presented. The first approach requires a full set of GT data as it compares the computed motion trajectories to the GT data, which enables a complete evaluation. The second approach presented is more pragmatic, as it detects specific trajectory events (such as line crossings), and the comparison is based on counting of the events rather than the positions of people, thus emphasising on recognition rather than tracking. In neglecting the tracking process, it fails to perform a full system evaluation. Needham [13] addresses the comparison of the ground truth trajectory and tracker output trajectory. It defines a set of positional tracking evaluation metrics corresponding to different types of trajectories based on human marked up GT data. This evaluation approach differs from the others in that it is not centered about a particular tracked position in space at a given time but on the overall path generated by successive tracking information. In [4], the evaluation of background subtraction and tracking including a track evaluation based on matching GT tracks to ATO tracks are discussed. 2.2 Evaluation of Football Tracking Results There is a significant body of research [8, 7, 12] concentrating on football tracking, among which the performance evaluation of the tracking is mainly based on a comparison between the GT data and tracking result. To evaluate the tracking performance from single and multiple camera views, Iwase and Saito use the spatial distance between GT and ATO as the evaluation measure [7]. In complex crowded scenes, very often moving objects occlude one another. It is however possible to recover from this problem by segmenting the blob representation of the player as described in [5]. In both papers, a component of the evaluation is the percentage of solved occlusions. In [8], the authors evaluate the performance using identity tracking as defined in Section Ground Truth Generation A number of systems are currently available for generating GT manually and/or evaluating tracking performance, of which a few are presented here. The Video Performance Evaluation Resource (ViPER) [2] provides a series of tools to generate GT data. This is represented in XML format, for evaluation and visualisation procedures. The tools operate on single camera view and thus only apply to 2D tracking. The Context Aware Vision using Image-based Active Recognition (CAVIAR) [1] provides benchmark video data sets, along with GT data obtained by hand-labeling the images and described in XML format. In addition to tracks of individuals, its innovation is the definition of groups of people, as a set of individuals reacting to one other [6]. A potential difficulty with this concept lies in establishing unequivocal criteria for interaction between two individuals. The Open Development for Video Surveillance (ODViS) [9] system provides an interactive framework allowing researchers to define GT data and analyse the performance of their tracking system. 3. Tracking Football Players Several researchers have presented results into tracking the positions of football players. Some results are for a single fixed camera [12], others are for a single moving camera [10, 11] and there are results for tracking through multiple cameras [8, 7, 14]. For multiple-camera systems, the architectural design of the tracking system reflects the need to extract the salient information from each data source prior to integration with the other sources. Here, we briefly describe the method presented in [18]. This is implemented and used to demonstrate the evaluation method. In particular, the effect of variation of a parameter (γ) controlling the probabilistic estimate of team category is investigated, to empirically determine the most suitable value. The system is composed of two stages. The first stage of processing extracts features from each camera. The second stage of processing integrates these features into a common representation. The interface between these two stages is a set of bounding boxes and associated properties per frame. These properties are the estimated ground-plane position, an estimate of the category and an indication of whether the object is being occluded

3 The first stage of processing uses a per-pixel Mixture of Gaussians adaptive background to extract the foreground regions. Morphological opening operations and adaptive masks are used to reduce the number of false positive observations. These regions are tracked using a Kalman Tracker, using the centroid position and its velocity and the region size as the state variables. This way, occlusions can be handled as partial observations that improve the tracking result. The category is determined by using the method of histogram intersection to calculate the relative overlap between the observation and each of the five histogram models (one for each category). For every observation, the values of the histogram intersection are normalised over the five categories to sum to one, and treated as a measure of the probability that the observation belongs to a category. To parameterise the relationship between the probability and normalised histogram intersection value, the latter is raised to some power, γ, and then renormalised. The second stage then integrates the features from each of the cameras. The features are projected onto the ground plane, and their error covariance is calculated. This is used as a validation gate on the process to fuse these observations into joint observations, which ideally form a one-toone mapping with all the objects in the scene. These observations are matched on a nearest-mahalanobis-neighbour basis to the Kalman state models that comprise the representation. The models can lose their association with the objects in the scene (false alarms, occlusions and rapid accelerations are typical causes of failure): a heuristic policy for the creation and deletion of these state models must be devised. For each element in the model, the category estimate is calculated as a linear sum of first stage category estimates, inversely weighted by their respective (spatial) covariances. In addition, the estimate is updated as a running average of recent observations, by adding a proportion α from the latest frame. 4. Output Representation In this section, we describe how the positions of the players, the ball and the events they compile, are represented in the prototype system. It is proposed to use the same format of representation for both ground truth (GT) and automatic tracking output (ATO) sources of data. As multiple cameras are used, it is suggested that ground plane co-ordinates are most suitable. The transformation to these co-ordinates is generated by Tsai s method [16] for co-planar calibration. We represent a player s position at time t as the single element i of the GT data set as x i (t). It has a 3D position x i IR 3, an identity (shirt number) p i IN and a category c i {c 1,...,c 6 } (two teams, the goalkeepers, the referees and the ball). Thus, the complete set of ground truth elements comprise the set X(t) ={x i (t) :i =1,...,m(t)}, where m(t) is the number of elements in the set at time t. The ATO at time t is represented by the set Y (t). Likewise, it contains the set of elements y j : j =1,...,n(t), where n(t) is the number of objects present in the ATO at t. Each of these elements includes the position y j, an identity label q j and a category estimate e j (t). Note that, except for the time-dependence of the category estimate, the form of GT and ATO representations are identical. 5. Method of Ground Truth Capture The GT data is generated by an operator using a custom graphical user interface. The operator specifies the realworld co-ordinates of players and ball by clicking on the appropriate view that shows any of the 8 camera outputs. Using the camera calibration co-efficients, the image-space points are transformed in the common ground co-ordinate system. In addition, the depth is required, but much of the activity can be approximated to lie in the ground-plane of the football field. The players are assumed to be situated on the ground plane (i.e. jumping up in the air is currently ignored). This position of each player is defined as the mid-point of the line connecting the two points vertically below his two feet. For cases where the calibration is inconsistent, the nearest camera is used to compute the position. When the ball is on the ground, it is simple to locate its position. However, when the ball is off the ground, locating the ball is not trivial. We have identified two different methods for providing a solution, described below. The first method uses two cameras to triangulate the position (shown in Figure 1), treating the two observations as wide-baseline stereo. The second method uses a trajectory model for the ball to interpolate its position from the two end-points of the path at which its 3D position can be more easily estimated. We use a parabolic model for the ball assuming there is negligible friction. The ball travels in a parabolic curve between t 1 and t 2, at which it bounces against the ground or a player. From knowledge of these two 3D positions and the time interval t 2 t 1, the 3D trajectory is completely determined. It is calculated in three stages as described below. However, first these two 3D points, p 1 and p 2 must be specified by the operator. This is straightforward for a ball on the ground. For a point in the air (such as when the ball is headed by a player), the operator uses the best camera view to select two image points: the ball s point of contact and the ground point vertically below. The first point defines the 3D line that contains p 1 = {p 1x,p 1y,p 1z }; the second point defines its position along that line. Given these points, the first step is to calculate the time t : the highest point in the parabola. This is calculated as: t = p 1z p 2z g(t 1 t 2 ) + t 1 + t 2 2 (1) 3 634

4 Figure 1: Ground plane showing views of view (top) for two cameras, for which subimages are shown (bottom). With two views, 3D points can be triangulated. where g is the gravitational acceleration. Thereupon, the initial vertical velocity v 1z is calculated. Hence, the position at each time-step thereafter is estimated by iteratively calculating the trajectory in finite time intervals, using these initial conditions. 6. Evaluating the Tracking Result We assume the existence of a GT data set and therefore the evaluation of the ATO data entails the comparison of these two data series. In this section, we discuss what factors are important contributors to the evaluation of tracking result, and define evaluation measures accordingly. We consider the specific scenario of the football match and then discuss how other scenarios differ. The primary goal is to assess the algorithm performance. The secondary goal is to identify specific errors and areas of weakness in the tracking performance, to aid the development process Player and Ball Positions One of the more challenging aspects to the problem of tracking football players is that all outfield members of the same team are identically dressed. Naturally, this makes the task of discriminating between these players more difficult. This could be attempted through continuity of position, facial appearance, height or shirt number. However, some applications only require knowledge about the position of the team without needing to identify the individual players within it. In this case, the output of the category of each player will be sufficient. Additionally, the recognition of the team category can be regarded as an preliminary step towards the classification of the individual players. Thus, for the football scenario, we regard there to be two levels of player tracking and recognition. The higher level aims to denote each player, referee and ball with a separate identity, that should be distinguished by the tracking system. This is referred to as identity tracking: each object on the pitch has its own label, that should be preserved throughout the course of the sequence. Identity tracking can be performed without knowledge of the names, because the labels can be arbitrarily assigned at the start of the sequence. In section 6.2 we define the performance evaluation measures P I for identity tracking. The lower level aims only to identify the category of every object tracked on the pitch. In normal play, each object is classified as one of the six categories e i (two teams, two goalkeepers, referees and ball). Two players of the same team are regarded as identical. In this paper, this is referred to as category tracking. In section 6.3 we define the performance evaluation measures P C for category tracking. At this point we introduce the proposed evaluation measures. The primary objective is to define a measure of the accuracy of the representation over the available sequence. We propose that this measure is defined over the two fundamental parameters of the representation, time and space: 1. At any given instant, the spatial accuracy of the ATO can be characterised as that proportion of players that are correctly represented to within Δd metres of the GT data. 2. Over a period of time t 0, the temporal accuracy of the ATO can be characterised as that proportion of its tracks for which its relationship with a GT track is maintained. The proposed evaluation measures will operate on the sets of GT and ATO data, {X(t)} and {Y (t)}, defined in section 4. The spatial accuracy of the tracking result is measured in the ground plane, i.e. there is a distance measure. In the following two sections, it is shown how the two parameters combine in different ways in the case of identity tracking and category tracking Evaluation of Identity Tracking In this section, a definition is presented for P I, the tracking performance evaluation across multiple cameras. Broadly speaking, it is the mean proportion of correctly 4 635

5 tracked objects over a number of frames. We now define a method for calculating that proportion, from a GT data set {X(t 0 ),X(t),...,X(T )} and ATO output {Y (t 0 ),Y(t),...,Y(T )} (See Section 4 for notation). To define what is meant by correct, we introduce the idea of a mapping between the set {p i (t)} of identification numbers of the GT, and the set {q j (t)} of identification numbers of the ATO, both at time t. The identity mapping function M I (t, p i ) evaluates to: the nearest ATO target from the set {q j }, or else the null token, ifnoatotarget is within Δd metres. The mapping is found by calculating m(t) n(t) distance matrix D, between the GT tracks X(t) and ATO tracks Y (t) at time t, such that: D ij = x i (t) y j (t) (2) The matrix D is then used to define the mapping for each GT element M(p i,t) as the closest ATO track q j, within Δd metres, that has not already been mapped to another GT track. The mapping is worked out in a closest-first order. The definition of whether a target i was successfully tracked between two times t 1 and t 2,isifM(t 1,p i ) = M(t 2,p i ), i.e. the same ATO track represent that GT track at these two times. An evaluation score f i (t 1,t 2 ) is assigned to this outcome, i.e. f i (t 1,t 2 )= { 1 if M(t 1,p i )=M(t 2,p i ) 0 otherwise and this is averaged over the available data, to define the performance evaluation of identity tracking, P I : P I (Δd, Δt) = T Δt m(t) t=t 0 T Δt m(t) t=t 0 i=0 i=0 f i(t, t +Δt) 6.3. Evaluation of Category Tracking In the previous section, it was shown how the two fundamental measures of tracking accuracy (time and space) are combined into a single evaluation measure, P I. For category tracking, it has not been possible to similarly combine these measures. Rather, it is shown below how these measures result in two separate evaluation criteria, P C and Q C : (3) (4) 1. At any given instant, the spatial accuracy of the ATO can be characterised as that proportion of the GT tracks that can be uniquely associated with an ATO track of the same category, to within a distance of Δd metres. This statistic can be evaluated over all tracks and the available time period to produce the first measure, P C. 2. The temporal accuracy of the ATO tracks is concerned with the constancy of the their category information over a period of time Δt. A measure, Q C, is designed to indicate the quality of the ATO data in this respect. The definition of the spatial measure of category tracking, P C, proceeds in a similar way to identity tracking measure, P I. However, we replace the identity mapping function M I (t, p i ) with a category mapping function M C (t, c i ) that maps the category of the GT track i against the category estimate e j of the nearest unassociated ATO track j (mapping closest matches first). An evaluation score h i (t) is assigned to this outcome, i.e. { 1 if M C (t, c i )=e j h i (t) = (5) 0 otherwise The spatial performance measure of category tracking is then defined as the proportion of tracks correctly tracked: P C (Δd) = T t=t 0 m(t) i=0 h i(t) T t=t 0 m(t) i=0 The definition of the temporal measure of accuracy, or categorisation discontinuity, does not require GT data set at all. It can be defined by comparing the category estimate for ATO track i at two different times, e i (t) and e i (t +Δt). A discontinuity has occurred if e i (t) e i (t +Δt) or there is no track i at either time. We need to count the number of categorisation discontinuities in the available time period. 7. Results and Analysis The evaluation method has been demonstrated on three different datasets. These illustrate the effect of varying two parameters in the method described in Section 3. The performance evaluation measure for identity tracking is shown in Figure 2(a). Here, P I (Δd, Δt) is plotted as a function of Δt, and keeping Δd fixed at 5 metres. Each value is calculated by averaging over all results in an 80 second window. In this case, it indicates how the result with γ =2.5, is a marginally more effective than a straightforward use of the histogram intersection output. 2(b) shows the evaluation measure for category tracking which is a plot of P C (Δd) as a function of Δd. 2(c) shows the discontinuity measure as a windowed average over an 80 second sequence. An inspection of the tracking results validate the evaluation result, in that the visual impression of the result correlates with the numerical values obtained. Nonetheless, there are several factors affecting the accuracy of the evaluation result, including the accuracy of the ground truth data (especially in highly crowded scenes) and the consistency of the calibration transformations. Even so, the results vindicate the overall methodology for evaluating different tracking results for a given video data set. There remains the more general problem of comparing tracking results across different sequences. Here, the amount of occlusion, image geometry and compression are all significant factors. (6) 5 636

6 proportion of objects correctly tracked result A result B timestamp (sec) (a) Spatio-temporal evaluation P I of Identity Tracking proportion of tracks within distance discontinuities / sec result A result B distance (m) (b) Spatial evaluation P C of Category Tracking result A result C timestamp (sec) (c) Temporal evaluation Q C of the Category Tracking Figure 2: Evaluation measures for result A (γ =1.0), B (γ =2.5) and C (γ =5.0) 8. Conclusion We have presented an analysis of evaluation methods for football tracking. We provide three performance measures appropriate for this context that can be used to assess the accuracy of tracking individuals and also categories of people whose members can be distinguished. Some data sets and performance evaluation service are now available for the research community at There is a strong motivation for generalising this work to other complex tracking problems. References [1] Caviar: Context aware vision using image-based active recognition. Web Resource - verified 29-04, [2] Viper: The video performance evaluation resource. Web Resource - verified 29-04, [3] J. Black, T. Ellis, and P. Rosin. A novel method for video tracking performance evaluation. In The Joint IEEE International Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, October [4] L. M. Brown, A. W. Senior, Y. Tian, J. Connell, A. Hampapur, H. M. C. Shu, and M. Lu. Performance evaluation of surveillance systems under varying conditions. In IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, January [5] P. Figueroa, N. Leite, R. M. L. Barros, I. Cohen, and G. G. Medioni. Tracking soccer players using the graph representation. In ICPR 2004, pages , [6] R. B. Fisher. Pets04 surveillance ground truth data set. In Proc. PETS04, May [7] S. Iwase and H. Saito. Tracking soccer player using multiple views. In APR Workshop on Machine Vision Applications (MVA02), [8] S. Iwase and H. Saito. Parallel tracking of all soccer players by integrating detected positions in multiple view images. In Proc. ICPR 2004, August [9] C. Jaynes, S. Webb, R. M. Steele, and Q. Xiong. An open development environment for evaluation of video surveillance systems. In Proc. PETS02, June [10] S. Lefevre, C. Fluck, B. Maillard, and N. Vincent. A fast snake-based method to track football players. In Proc. MVA, pages , November [11] S. Lefevre, J. Gerard, A. Piron, and N. Vincent. An extended snake model for real-time multiple object tracking. In Proc. ACIVS, pages , September [12] C. J. Needham and R. D. Boyle. Tracking multiple sports players through occlusion, congestion and scale. In BMVC, [13] C. J. Needham and R. D. Boyle. Performance evaluation metrics and statistics for positional tracker evaluation. In Computer Vision Systems, Third International Conference, ICVS 2003, April [14] Y. Ohno, J. Miura, and Y. Shirai. Tracking players and a ball in soccer games. In Int. Conf. on Multisensor Fusion and Integration for Intelligent Systems, pages , [15] S. Pingali and J. Segen. Performance evaluation of people tracking systems. In IEEE Workshop on Applicatons of Computer Vision, pages 33 38, November [16] R. Tsai. An efficient and accurate camera calibration technique for 3d machine vision. In Proc. CVPR, [17] H. Wu and Q. Zheng. Self-evaluation for video tracking systems. In 24th Army Science Conference, November [18] M. Xu, J. Orwell, L. Lowey, and D. Thirde. Architecture and algorithms for tracking football players with multiple cameras. IEE Proceedings on Vision, Image and Signal Processing, 152(2): , April