VEHICLE LOCALISATION AND CLASSIFICATION IN URBAN CCTV STREAMS

Transcription

1 VEHICLE LOCALISATION AND CLASSIFICATION IN URBAN CCTV STREAMS Norbert Buch 1, Mark Cracknell 2, James Orwell 1 and Sergio A. Velastin 1 1. Kingston University, Penrhyn Road, Kingston upon Thames, KT1 2EE, United Kingdom {norbert.buch, james.orwell, sergio.velastin}@kingston.ac.uk 2. Transport for London, Palestra, 197 Blackfriars Road, London, SE1 8NJ, United Kingdom ABSTRACT This paper presents an introduction to video analysis tools for urban traffic management. Based on a review of the limitation of current systems, a framework for localising and classifying cars in real-world coordinates is introduced as part of a project at Transport for London. Vehicle detection is performed using either motion silhouettes or 3DHOG (3D extended Histograms of Oriented Gradients). The latter is more robust in urban environments. Qualitative and quantitative evaluation of the proposed systems is provided with an outlook on further development potential. KEYWORDS vehicle detection, road user classification, pedestrian, urban traffic, visual surveillance, video analysis, computer vision INTRODUCTION Intelligent image detection systems are part of a centralised approach to modern day traffic management. This has arisen from the need for more cost effective and efficient monitoring of traffic. Traffic monitoring CCTV tends to be unique in that it includes high camera numbers, is in the public domain and contains long transmission paths (up to 4km). With 12 cameras and over 1 monitors it is not feasible to continuously monitor every CCTV camera installed within Transport for London s (TfL) network. In fact, it has been shown that manual monitoring over time significantly reduces the accuracy of detection. Therefore, the development of a technology that provides automatic and relevant real-time alerts to Traffic Co-ordinators can have an immediate and long term impact on traffic management through the implementation of responsive traffic strategies. In early 26, TfL launched the Image Recognition and Incident Detection (IRID) project. This project was tasked to review the current image processing market and see how it met TfL s detection requirements. Testing was carried out on the following criteria: Congestion, Stopped Vehicles, Banned turns, Vehicle counting, Subway Monitoring and Bus detection. [6,7]. Results from this testing show good performance in Congestion detection (8% precision), but poor performance in tracking based detection (~2% precision), clearly showing limitations in capability. This limitation led directly to the creation of a research relationship with Kingston University. The aims of this project are the localisation and subsequent 1

2 Buch et al. Vehicle Localisation and Classification in Urban CCTV Streams ITS World Congress 29 Detector frame GMM [11] Tracker foreground mask Classifier Closed Contours frame silhouettes Match Measure scores Maximum labels GP pos Kalman Filter tracks 3D Hypothesis GP positions 2D Projection [12] model data Models Figure 1 Framework for 3D localisation and classification and models used classification of vehicles and pedestrians in camera views specific to the urban environment of Transport for London. Estimating the vehicle position in real-world coordinates (on road maps) is also beneficial for traffic enforcement applications. The conventional concept of using background modelling for the generation of a motion mask is only used as baseline. Motion masks suffer from noise due to lighting changes, camera shake, rain, etc. and particularly from occlusion due to low camera angle. All these effects are inherent to urban traffic scenes. The project objective was to move beyond motion estimation and use other visual means to localise and classify cars. A texture based classifier (3DHOG) was introduced to overcome the mentioned problems. 3D FRAMEWORK FOR VEHICLE CLASSIFICATION AND LOCALISATION We developed a framework to localise vehicles either based on the motion mask or based on the texture in the image. The framework is shown in Figure 1. The detector uses background estimation with a Gaussian Mixture Model (GMM) [11] to model the static part of the scene. This model allows for several images to be considered static as shown in Figure 2, which enables rejection of moving objects in the background like trees. By considering the difference between the background and a new frame, an initial foreground mask is estimated. This mask is refined by a shadow removal algorithm, which considers areas as shadows, if they are slightly darker than the background but have the same colour. From the resulting binary foreground mask, closed contours are extracted. The contours provided by the detector are used to initialise hypotheses for vehicle locations on the ground plane. The hypotheses are verified by the classifier by comparing existing vehicle models with the input image to give a match measure. The models can either be the projected silhouettes or appearance data gathered during a training phase. Finding the maximum match in terms of location and vehicle type gives the final estimation. The detection of a single frame can be tracked over time using a Kalman filter, which will produce tracks (trajectories) for vehicles. Further details about the framework can be found in Buch et al. [2,4] and for the tracker in [5]. As visual output, the localised vehicles are marked up on the camera view and on a map of the area. Refer to Figure 2 for an example frame from TfL with corresponding map. The thin dark red boxes represent the regions of interest where vehicle detection is performed. This could be set to bus lane areas to detected unpermitted vehicles in those lanes. 2

3 Buch et al. Vehicle Localisation and Classification in Urban CCTV Streams Initial mask Frame Silhouettes Foreground mask Remove Shadow GMM ITS World Congress 29 Closed Contour Stable background Second background Foreground Detector a b Figure 2 a) Detector stage. b) Localised vehicles with superimposed wire frames and ground plane location. The coloured lines are the tracks of individual vehicles. RESULTS USING MOTION SILHOUETTES The localisation framework was first tested with a baseline approach using the overlap of motions silhouettes and model silhouettes as match measure. Figure 3 illustrates the operation where a score is defined as the ratio between intersection and union of both silhouettes. The 3D location of vehicles is found by generating a hypotheses grid (green crosses in Figure 3) around the back projected silhouette centroid (red cross in Figure 3). A score surface is generated for this grid and the hypothesis with the highest score is selected as location and class for the detected vehicle. Silhouettes Classifier Scores Overlap 1 ID 2 ID 3 Match measure [%] 8 Overlap Area ID 4 ID 5 6 ID 6 Maximum ID 7 4 ID 8 ID 9 2 3D Cros-section position [m] 2 Labels Hypothesis Model Silhouette Model 2D Project. Model s Ground plane map Figure 3 Classifier based on motion silhouette overlap with model 3

4 Buch et al. Vehicle Localisation and Classification in Urban CCTV Streams ITS World Congress 29 ground truth detection pedestrian bike car/taxi van bus/lorry FN count overlap pedestrian bike car/taxi van bus/lorry FP ground truth detection pedestrian bike car/taxi van bus/lorry count pedestrian bike car/taxi van bus/lorry Symbol Recall R Precision P Classifier P C Detector R D Detector P D GT Overlap Value 79.5% 83.9% 89.8% 88.6% 93.5% a b c Table 1 a) Confusion matrix of full system including detector. b) Confusion matrix of classifier only. C) Full performance for motion silhouettes based system..64 Classification performance The system was extensively tested with video footage from Transport for London and from the i-lids datasets [1]. The latter is a benchmarking dataset provided by the UK Home Office to imaging research institutions. In Table 1, we present classification results on 1 hour video footage from the parked car scenario. Good overall classification performance is demonstrated with some confusion between bikes and pedestrians. This is due the similar size of those two types of road users. The localisation performance is demonstrated by the bounding box overlap between the wire frame and the ground truth in the overlap row. The value of 64% overlap is good, considering the use of the wire frame rather than the motion silhouette for the bounding box estimation. Detailed explanation for performance measures can be found in [1] and its application in [2,4]. Tracking performance Tracking is performed on the ground plane of the scene, which simplifies behaviour analysis like bus lane monitoring. We use the standard formulation of the Kalman filter for a constant velocity model of vehicles. The object tracking performance is demonstrated by comparing our tracker with a baseline tracker (OpenCV blob tracker [1]). The OpenCV tracker uses an adaptive mixture of Gaussians for background estimation, connected component analysis for data association and Kalman filtering for tracking blob position and size. The data used is i- LIDS [1] as for the performance evaluation of classification. We propose a rich set of metrics such as Correct Detected Tracks, False Detected Tracks and Track Detection Failure to provide a general overview of the system s performance. Track Fragmentation shows whether the temporal and spatial coherence of tracks is established. ID Change is useful to test the data association module of the system. Latency indicates how quick the system can respond to an object entering the camera view, and Track Completeness how complete the object has been tracked. Metrics such as Track Distance Error and Closeness of Tracks indicate the accuracy of estimating the position, the spatial and the temporal extent of the objects respectively. More details about this evaluation framework can be found in [5] and Yin et al. [13]. The proposed system detected 94% of the ground truth tracks compared to 88% of the base line. Our system has half of the track detection failures compared to the base line. Please refer to Table 2 for a complete set of metrics and Figure 4 for visual tracking examples. 4

5 Buch et al. Vehicle Localisation and Classification in Urban CCTV Streams ITS World Congress 29 Metrics proposed Tracker OpenCV blob Tr. Number of Ground truth tracks 1 1 Number of system tracks Correct detected tracks Track detection failure 6 12 False detected tracks 27 9 Latency [frames] 5 5 Track fragmentation 8 18 Average track Completeness [time] 64% 55% ID change 1 3 Average track closeness [bbox overlap] 54% 35% Standard Deviation of closeness 2% 13% Average distance error [pixels] Standard Deviation of distance error Table 2 Tracking results of motion silhouette classifier Figure 4 Example tracking results from the i-lids data BEYOND MOTION: 3DHOG The limitations of motions silhouettes inspired the use of texture to detect vehicles by appearance. Good results have been reported elsewhere for patch based approaches in object recognition [9] and pedestrian detection with histograms of oriented gradients (HOG) [8]. We introduce a novel concept by applying the HOG descriptor to image patches defined in 3D model space. Full details on this 3DHOG framework can be found in Buch et al. [3]. This feature approach substitutes the overlap match measure in the block diagram with a training 5

6 Buch et al. Vehicle Localisation and Classification in Urban CCTV Streams ITS World Congress 29 a b c d Figure 5 a) 3D model with marked interest points. b) input camera image frame. c) extracted and normalised image patches displayed in 3D space. d) 3DHOG gradient features generated from the visible image patches based classification step. Figure 5 shows the model with patch centres defined with the model and the extracted patches. Affine transformations are used to generate those scale normalised patches. A descriptor is generated from every patch, which consists of either gradient histograms, a frequency spectrum or a simple image histogram. A data driven appearance model is learned, whereas a single Gaussian distribution is learned for every interest point descriptor. During system operation, new descriptors are generated for every hypothesis (2D projection block in Figure 1), whereas the distance between learned descriptors and newly seen descriptors define the match measure. The remainder of the framework remains identical to the earlier description. Performance comparison For evaluation on the same dataset as in the last section, a recall of 81% is achieved at a precision of 82%. Those results indicate similar performance to the motion based system, however providing more robustness against the usual noise sources in the urban environment. By using texture and appearance rather than motion, the 3DHOG classifier can deal with cases were the motion silhouette is significantly distorted or similar for different classes. There are illustrative examples in Figure 6. The case of oversized silhouettes due to shadows and lighting changes is rectified and a pedestrian in correctly detected. Another common problem is saturation of some areas in the camera. The example shows a very small silhouette for a white van, which was missed from the motion foreground due to the saturation in the same area. The 3DHOG classifier successfully detects and classifies the van. For objects of similar size like pedestrians and bicycles, the classifier can distinguish correctly based on the appearance. CONCLUSIONS We presented a review of commercial video analytics systems tested by Transport for London. The findings and the progress of the subsequent project with Kingston University is demonstrated. An improved computer vision system is demonstrated by introducing a 3D localisation framework for vehicles. Good results are demonstrated for motion silhouette based vehicle classification. An extension to texture based classification is given by moving beyond the concept of motion estimation. This extends the concept of HOG by introducing novel 3DHOG, which use a 3D surface window for vehicle classification. This classifier demonstrated superior performance for challenging cases where motions silhouettes are 6

7 Buch et al. Vehicle Localisation and Classification in Urban CCTV Streams ITS World Congress 29 3DHOG classifier Motion Silhouette classifier Figure 6 Comparison between 3DHOG and motion silhouette classifier. Noisy motion foreground (blue outline) is misclassified on the right. In contrast, the 3DHOG classifier correctly classifies the pedestrian inside the shadow and the van in the saturated area. incorrect. The frame to frame detection is used as input for a Kalman filter to generate trajectories of road users. Future work will focus on more evaluation of the proposed systems under diverse weather and operation conditions. ACKNOWLEDGEMENTS This work is sponsored and conducted in cooperation with the Directorate for Traffic Operations at Transport for London.. [12 [11] REFERENCES [1] Home Office Scientific Development Branch. Imagery library for intelligent detection systems i-lids. [accessed 19 December 28]. [2] Norbert Buch, James Orwell, and Sergio A. Velastin. Detection and classification of vehicles for urban traffic scenes. In International Conference on Visual Information Engineering VIE8, pages IET, July 28. [3] Norbert Buch, James Orwell, and Sergio A. Velastin. 3D extended histogram of oriented gradients (3DHOG) for classification of road users in urban scenes. In British Machine Vision Conference BMVC 29, London, September 29. [4] Norbert Buch, James Orwell, and Sergio A. Velastin. Urban road user detection and classification using 3d wire frame models. IET Computer Vision [accepted], 29. 7

8 Buch et al. Vehicle Localisation and Classification in Urban CCTV Streams ITS World Congress 29 [5] Norbert Buch, Fei Yin, James Orwell, Dimitrios Makris, and Sergio A. Velastin. Urban vehicle tracking using a combined 3d model detector and classifier. In 13th International Conference on Knowledge-Based and Intelligent Information & Engineering Systems KES29, Santiago, Chile, September 29. LNCS Springer. [6] Mark Cracknell. Image detection in the real world interactive session. In Intelligent Transportation Systems ITS 7 Aalborg, 27. [7] Mark Cracknell. Image detection in the real world a progress update. In Intelligent Transportation Systems World Congress ITS WC 28 New York, 28. [8] N. Dalal and B. Triggs. Histograms of oriented gradients for human detection. In Computer Vision and Pattern Recognition, 25. CVPR 25. IEEE Computer Society Conference on, volume 1, pages , 25. [9] Bastian Leibe, Ales Leonardis, and Bernt Schiele. Combined object categorization and segmentation with an implicit shape model. In ECCV 4 Workshop on Statistical Learning in Computer Vision, pages 17 32, May 24. [1] OpenCV. Open source computer vision library. [accessed 19 December 28]. [11] C. Stauffer and W.E.L. Grimson. Adaptive background mixture models for real-time tracking. In Computer Vision and Pattern Recognition, IEEE Computer Society Conference on., volume 2, pages , June [12] Roger Y. Tsai. An efficient and accurate camera calibration technique for 3d machine vision. In Proc. Int. Conf. on Computer Vision and Pattern Recognition (CVPR), (1986), pages , [13] Fei Yin, Dimitrios Makris, and Sergio A. Velastin. Performance evaluation of object tracking algorithms. In 1th IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, PETS'7, Rio de Janeiro, October 27. 8