Vehicle Vehicle Tracing by by a Motion a Motion History History Graph Graph motion analysis. Vehicle tracing has been extensively utilized in traffic scene analysis. Hadi Hadi Sadoghi Sadoghi Yazdi Yazdi There are Mojtaba two Mojtaba types Lotfizad Lotfizad of tracer: bottom-up and Engineering Engineering Department, Department, Tarbiat Tarbiat Moallem Moallem University University Department Department top-down. of Electrical of Electrical In the Engineering, first Engineering, group, Tarbiat tracing Tarbiat Modarres Modarres algorithm of Sabzevar, of Sabzevar, Sabzevar, Sabzevar, Iran Iran University, University, contains Tehran, Tehran, scene Iran, Iran, segmentation Lotfizad@modares.ac.ir and trajectory sadoghi@sttu.ac.ir prediction [5]-[8]. The second type is top-down or learning-based tracer. Ehsanollah Ehsanollah Kabir Kabir Mahmood Mahmood Fathy Fathy Department Department of Electrical of Electrical Engineering, Engineering, Tarbiat Tarbiat Faculty Faculty of Computer of Computer Engineering, Engineering, Iran Iran University University of Science of Science and and Modarres Modarres University, University, Tehran, Tehran, Technology, Technology, Tehran, Tehran, Iran, Iran, mahfathy@iust.ac.ir abir@modares.ac.ir Abstract: In this paper, a history-based vehicle tracing algorithm is presented which is a new top-down or learning-based vehicle tracer. History of trajectory is represented by a weighted directed graph (WDG), called motion history graph, MHG. This algorithm includes two phases; spatio-temporal trajectory is stored in a WDG in the learning. In the search phase the spatio-temporal database is employed to increase the performance of the predictor. The spatiotemporal database is also updated. The proposed technique is used for vehicle tracing in highways. It yields a decrease of up to 80% in prediction error relative to a conventional technique. Keywords: History-based tracer weighted directed graph, vehicle tracing, spatio-temporal data base, top-down tracer. 1 Introduction OTION analysis is a recent topic of interest M among the computer vision and video applications such as video parsing, indexing, browsing, searching, compression, coding, surveillance, monitoring, annotation and retrieval. A number of techniques have been proposed for motion-based video representation [1]-[4]. Tracing algorithm is an important tool for In this type, by determining a motion model a better result in tracing is obtained [9]-[19]. In this group, motion analysis can be studied from different aspects: a) Motion model The motion model is lie constant velocity, constant acceleration and vehicle turning. If motion model is determined suitable, tracing procedure gives better results [9]-[10] or if initial state of tracer is selected correctly according to inematical change in motion, tracer is converged rapidly [11]. Of course the motion model can be determined using learning procedure based on sample hypotheses or training sets [12]- [14]. b) Spatio-temporal representation of the model In motion analysis, spatio-temporal signal is obtained after object tracing which plays an important role in understanding the semantic content of surveillance videos. Representation of spatio-temporal signal can be given in two forms, short-time and long-time. Short-time representation In this ind of representation, short duration of x(t) and y(t) are usually used in information/event retrieval. For example, in [15], spatio-temporal relations between two trajectories or between a trajectory and a region are used to
define simple events. In [16] a motion trajectory is considered as a set of points. At each point, an object is represented by a displacement from the previous point and direction of motion. Also, at each point, the spatial relationships between objects are represented. The reference [17] presents a hierarchical approach to model videos at three levels, object level, frame level and shot level in video searching application. The model is captured from visual features of individual objects as relationships between objects. Visual spatiotemporal relationships between objects include relative color, size, position, speed, direction, and appearance/disappearance time. In [18] automatic traffic video indexing has been performed. The appearance sequence of the vehicle objects is coded based on the relative spatial locations of the vehicle objects in the sub-region in traffic video frame. Long-time representation In this ind of representation, long duration of x(t) and y (t) are used. For example, in [19] motion analysis is provided based on a feedforward TDNN 1, with x-y-t spatio-temporal input signal. This networ simultaneously manages to classify the shapes correctly as well as to estimate their speed and motion direction. The networ is learnt on synthetic image sequences. Also a similar approach is [20] that a neural networ is applied to predict future animal behavior. In [21] anti-face method has been presented for event detection, in both gray level and feature domains. For the gray-level domain, spatiotemporal templates are created by stacing the individual frames of video sequence, and the detection is performed on these templates. In order to recognize the motion of the features in a video sequence, the spatial locations of the features are modulated in time, thus creating a one-dimensional vector which represents the event in the detection process. In [22] a technique is presented for building coarse Motion History Image, MHI, and Motion Flow History, MFH, from the compressed video. The MHI gives temporal information of the motion at the image plane, whereas the MFH quantifies the motion at the image plane. The MHI and MFH that represent the presence of the human and directions of the actions are used for recognizing a set of seven human actions. The history of trajectory of the above wors is used in video event, indexing and recognition applications. In this paper, a history-based vehicle tracing algorithm is presented. In this algorithm a novel spatio-temporal database is used for better tracing. The proposed algorithm contains the prominent and new points as follows: a) Presentation of a new spatio-temporal database in vehicle tracing application. b) Applying the spatio-temporal data-base to improve searching and prediction performance. The basic worflow of the proposed algorithm is shown in Fig 1. The innovation of this wor lies in interfacing the spatio-temporal database with the search and prediction blocs. In section 2 of the paper, multi-object tracing system is presented. In section 3, a novel spatio-temporal database in the form of weighted directed graph, WDG, is proposed. Section 4 is devoted to the use of the motion history graph, MHG, to object tracing and the results obtained are presented in the final section. Fig.1: Bloc connections of tracing procedure in the proposed system 2 Multi-object tracing system The tracing algorithm has become a significant issue in traffic scene analysis. A large number of methods exist for tracing objects in outdoor scenes. Many types of object tracing methods are available whose standard types, include feature based [8], [26], region based [7], [27] and model based [5] tracing algorithms. The vision based tracing algorithm constitutes vehicle segmentation and state estimation which are reviewed in the following subsections. 2.1. Segmentation method A classic technique in the segmentation operation is the bacground subtraction. Static targets are attained using bacground subtraction according to (1). O = I B (1) 1 Time Delay Neural Networ
Where, I is the K th input frame and B is the bacground at K th frame. O is the static target. B is obtained by combining non-moving blobs according to (2). B ( i, j) = α B 1( i, j) + ( 1 α) NMB( i, j) (2) Where, NMB is a blob of the image which is chosen as the non-moving blob and i, j refer to the pixels of this area of the picture. Then each pixel in the bacground image is combined with the corresponding pixels of the received non-moving image blob. α is the coefficient of the effect of last bacground and 1- α is coefficient of the received non-moving blob. α is a number between 0 to 1 which is chosen here equal to 0.9. Fig 2 shows the results of the above segmentation method including main figure, result of bacground subtraction and binary image after using threshold, a mas image is generated (in this scene the threshold value is set to 55). for localization with this criteria bigger than 0.95. After doing corner detection, a grouping technique based on distance criteria that appeared in [8] is used for finding the vehicle boundaries. [8] use a common motion constraint; features that are seen rigidly moving together are grouped together. The grouper considers corner features in pairs. Two points that are less than a prespecified distance apart will be hypothesized to belong to the same vehicle. The distance, d, is measured in the world coordinates by multiplying the image distance with a depth scaling factor. Fig 4 shows the grouping technique for car localization, despite partial occlusion. Fig.3: Detected corners by normalized corner criterion bigger than 0.95 Fig.4: Car localization using feature grouping Fig.2: Segmentation method using adaptive bacground subtraction a) blob and mas detection in Frame #171 b) in Frame #173, blac car is lost by threshold 55. 2.2. Car localization After performing bacground subtraction and finding the existing objects in the scene, vehicle localization is used for determining the object boundaries. Localization helps to solve the partial occlusion problem which is done by finding feature points and grouping them. We use corners as features and extract these according to [28]. One approach to corner finding in [28] is to construct a covariance matrix of the gradient vector and performing a canonical correlation analysis in each small window. Fig 9 shows a sample of detected corners on segmented image 2.3. Car searching in consecutive frames After finding the vehicle's boundaries, a color histogram is built for the similarity measurement in consecutive frames. For each detected boundary, HSV color histogram is achieved and quantized to 18 6 6 bins such that a vector of length 648 is obtained. A matching operation is performed by measuring a Euclidean distance between each two vectors where it is smaller than 0.05. Minimum distance is given in search rectangular windows with length of 50 pixels. Then center of gravity of the found vehicle is applied to an estimator or predictor in order that after the convergence of the predictor for each vehicle can help for attribution of similar blobs and generating of smoothed trajectory. 2.4. Prediction of position using extended recursive least square algorithm Different types of noise are observed in car tracing that is an incentive for using adaptive
filtering for noise reduction. Several sources of noise is presented as follows, A) Membership of whole or part of a vehicle to another vehicle due to shadow and partial occlusion e.g. Fig 2. B) Detection of different parts of a vehicle because of type of segmentation method e.g. Fig 5. C) Variation of vehicle position because of partial occlusion e.g. Fig 4 and Fig 6. Fig.5: Detection of different parts of vehicle Fig.6: Noise in position due to partial occlusion Thus, we use an extended form of recursive least squares algorithm for filtering the vehicle position as in [25]. This filter has a better tracing capability compared to the conventional method [23]. We modify it based on the quantized input signal for decreasing the effect of noise in adaptation process. For better vehicle tracing we propose a novel technique that based on history of vehicles trajectory. In this algorithm firstly, history of trace of vehicles are stored in a novel spatio-temporal database, namely, motion history graph, MHG, in the form of weighted directed graph. The next section is devoted to explain the MHG. 3 A spatio-temporal database, motion history graph Spatio-temporal information is extracted for learning the trajectory after generation with multiobject tracing. In our previous wor [24], by suitable combination of spatial and temporal information that are obtained from the vehicles' trajectories, the spatio-temporal database was generated. For this purpose, the suitable combination of spatial and temporal information is obtained from the vehicles trajectory and a novel spatio-temporal data base from trajectories was obtained that includes spatial and temporal and directional information. The proposed spatio-temporal data base is based on weighted directed graph, which is obtained by the following steps: I. Trajectory of vehicles is stored in the x-y plane. II. These traces are clustered using fuzzy - mean into some 200 clusters. III. A WDG is constructed with some 200 nodes IV. Elements of WDG are completed using multi-object tracing algorithm. The above four steps are explained in detail as follows, The trajectories of vehicles are stored in the x-y plane on up to 8000 frames as shown in Fig 7. These trajectories are clustered by fuzzy K-means into 200 clusters, and then a graph with 200 nodes is built. We call this graph, motion history graph (MHG). The number of 200 is given based on this fact that in average 4 frames is required for moving vehicles between two nodes of the MHG. The weight of each lin of MHG expresses that how many vehicles so far are transferred from node ( i ) to node ( j ). So, for completion of weight of MHG, multi object tracing is done for other 9000 frames. Whenever, any vehicle is passed between any two nodes, the value of weight of its lin is incremented by 1. Part of the MHG is depicted in Fig 8, where the transition probabilities of 79.5% and 20.5% are obtained from c1 to c2 and from c1 to c3 nodes during initial observation of the scene (c1, c2, and c3 are three nodes of the MHG). These probabilities are obtained by counting the number of vehicles which pass from c1 node to other nodes. Therefore, if any blob lies at the vicinity of c1 node, then it will be attributed to c1 and is predicted to pass to c2 node with a probability of 79.5%, and pass to c3 node with a probability of 20.5%.
Fig.7: Vehicles trajectories in the x-y plane and 200 centers obtained with Fuzzy clustering Fig.8: Part of MHG The proposed MHG is defined as follows, MHG = { Ci, Pij i, j = 1,..., n} (3) Where C i is the i th node of the MHG that includes x-y position and the index of the node. Pij are weights of MHG or transition probability of node ( i ) to ( j ). These probabilities are obtained by counting blobs which have passed from each node to other nodes. The transition probability from node i to node j is, nij P ij = (4) Ni Where n ij is the number of passed blobs from node ( i ) to ( j ) and N i is the total passed blobs from node ( i ). A sample of the MHG is shown in Fig 9 with a transition probability bigger than 0.25. Fig. 9: Displaying those elements of the MHG along with connections (internodes dash line) having a transition probability more than 0.25 3.1. Updating the weights of WDG The weights of directed graph are updated gradually. This permits variations of the scene to be recorded. Different reasons are given for this variation, namely; Event occurrence, Road repairing, Road construction, Road blocing. Hence, P ij are updated and changed dynamically. 4 Feedbac from the MHG to the tracer, MHG technique In the presented system of Fig 1, a top-down relation is applied from the trajectory history to the tracer (searching and prediction moving object) whose details are given in the following subsection. 4.1. Using MHG in reducing the size of search window The search operations include searching similar blobs in two consecutive frames which are spatially close to each other. The MHG is used for reducing size of search window as shown in Fig 10. The MHG is used in reducing search window,
which reduce false alarm in detection of vehicles and gives a fast tracing algorithm. Details of this procedure are explained as follows, We appoint detected blob to node of MHG that has maximum self probability ( P ii) according to Fig 11, in this order, four nearest nodes from MHG which are in search area is found and ones is selected that has maximum self probability ( P ii). Search area is selected between these two nodes according to Fig 12. In this figure, primary search area is S1 before using MHG and using MHG S p1, S p2, S p3 are selected respectively if isn t found similar blob (Fig 8.a,b,c). Lin of Fig 12 is drawn as for P ij of MHG, chromatic lin shows bigger P ij. Fig.12: Selecting search area using MHG MHG can help to initializing of predictor with reducing salient least square or prediction error. This subject is followed in the next sub-section. Fig.10: Reducing size of search windows using MHG Fig.11: Appointment of detected blob to node of MHG 4.2. Using MHG in initializing of initial weight predictor Trajectory prediction for correctly tracing the moving object is required. Among the prediction methods, are the adaptive filtering ones [23]. Adaptive filters by predicting the trajectory or the path estimation from the past trajectory data, help in tracing. These algorithms do not exploit the trajectory learning; but instead, they act with regard to the vehicles trajectory information or a predetermined specific motion type (bottom-up tracer). We use MHG to initialize of predictor. In next sub-section we explain reason of using MHG in initializing of predictor and method of applying to the predictor. 4.2.1. Difficulties in tracing and searching in multi-object tracing algorithm
Some of the difficulties of using predictor are as follows: A) At the beginning of tracing of each vehicle, its RLS predictor has a large error and requires several frames to converge. B) Due to difficulties in target detection which cause partial detection, part of vehicles is detected (Fig 13); this gives a noisy situation to the RLS predictor and causes divergence of predictor. Fig.13: Partial or full vehicle is detected because of existing problems in detection algorithms or illumination condition Therefore, it is necessary to find a solution to fast convergence and reduction of error prediction. Thus, in the instant of divergence of the RLS algorithm, MHG gives the appropriate weights to the predictor to converge quicly. 4.2.2. Using MHG in RLS convergence and error reduction The RLS algorithm is applied to an FIR filter with two weights for predicting the positions x,y. the RLS algorithm has good convergence speed but its convergence requires observation of several samples of input frames (5-8 frames). In addition, this algorithm has not a suitable tracing capability in noisy environments [23]. Therefore, it is necessary to reduce the prediction error, when the RLS is diverged. The MHG can be used in fast convergence of the RLS algorithm and it can reduce prediction error. By attention to this fact that in average 4 frames is required to moving vehicles between two nodes of WGD, after membership of blob to each node, 4 next position of its is predicted using MHG. Thus initial weights of predictor are calculated before receiving second frames. The prediction error rate of the RLS algorithm after applying the MHG on moving vehicles was decreased by at least 80% at normal congestion in junction of highway to highway, highway to square. This shows the efficiency of the MHG algorithm in tracing moving targets. The learning curve of the RLS predictor before and after applying the MHG technique on moving vehicles is depicted in Fig 14. As is shown in the Fig 14, applying the MHG to the RLS estimator gives rise to a tangible decrease in prediction error. e + e is the sum of absolute along x and y. In addition we compare MHG-based predictor with conventional RLS upon 38 traced vehicles, which in Fig 15 sum of absolute prediction error in x,y direction is depicted. Increasing convergence speed and decreasing prediction error indicates that adding history of vehicles trajectory can help to better tracing. 4.3. Motion history graph based tracing, MHG, technique Structure of MHG technique is illustrated in bloc diagram of Fig 16 which it includes a feedbac from the trajectory history to detection and prediction. Fig 16 explains the way of applying MHG in search and prediction of moving blob. MHG technique is capable reduction of searching area and increasing to speed convergence of predictor. Fig.14: The learning algorithm of the RLS prediction before and after applying MHG Fig.15: Comparing MHG-based RLS and conventional RLS x y
Fig.16: Structure of MHG technique 5 Conclusion A history-based vehicle tracing algorithm was presented which is based on weighted directed graph spatio-temporal database. The proposed spatio-temporal data base is capable reduction of searching area and increasing to speed convergence of predictor. Generating of sptiotemporal data base is an important in video application, we proposed novel spatio-teporal data base based on weighted directed graph that can be used in object tracing. We applied the proposed scheme in vehicle tracing in different traffic scenarios. The results showed success of our idea. 6 References [1] S. Dagtas, W. A1-Khatib, A. Ghafoor, R. L. Kashyap, "Models for Motion-Based Video Indexing and Retrieval," IEEE Trans. on Image Processing, vol. 9, no. 1, pp. 88-101, Jan. 2000. [2] A-D. Bimbo, E. Vicario, D. Zingoni, "Symbolic Description and Visual Querying of Image Sequences Using Spatio-Temporal Logic," IEEE Trans. on Knowledge and Data Engineering, vol.7, no.4, pp.609 621, August 1995. [3] C. Cedras, M. Shah, "Motion-Based Recognition: A Survey," Image and Vision Computing, vol. 13, no. 2, pp. 129-155. Mar. 1995. [4] M. Shah, R. Jain, Motion Based Recognition, Kluwer, 1997. [5] M. Haag, H.-H. Nagel, "Tracing of Complex Driving Maneuvers in Traffic Image Sequences," Image and Computing, vol.16, pp.517-527, 1998. [6] D. Koller, K. Daniilidis, H.-H. Nagel, "Model- Based Object Tracing in Monocular Image Sequences of Road Traffic Scenes," International Journal of Computer Vision, vol.10, no.3, pp.257-281, 1993. [7] J. Badenas, J. M. Sanchiz, F. Pla, Motion- Based Segmentation and Region Tracing in Image Sequence, Pattern Recognition, vol.34, pp.661-670, 2001. [8] B. Coifman, D. Beymer, P. McLaunhlan, J. Mali, A Real-Time Computer System for Vehicle Tracing and Traffic Surveillance, Transportation Research Part C 6, pp.271-288, Mar.1998. [9] L. Zhao, C. Thorpe, "Qualitative and Quantitative Car Tracing from a Range Image Sequence, Proc. CVPR98, pp.496-501, June 1998. [10] R. Karlsson, "Simulation Based Methods for Target Tracing," PhD Thesis, Department of Electrical Engineering, Linopings University, Sweden, 2002. [11] C. Kervrann, F. Heitz, A Hierarchical Marov Modeling Approach for the Segmentation and Tracing of Deformable Shapes, Graphical Models and Image Processing, vol. 60, no. 3, pp. 173 195, 1998. [12] M. Isard, A. Blae, CONDENSATION- Conditional Density Propagation for Visual Tracing, International Journal of Computer Vision, vol.29, No.1, pp.5-28, 1998. [13] M. Isard, A. Blae, A Mixed-State Condensation Tracer with Automatic Model Switching, 5th Proceeding of European Conference on Computer Vision, vol.1, pp.893-908, 1998. [14] M. Isard, A. Blae, D. Raynard, Learning to Trac Curves in Motion, IEEE International conference on Decision Theory and Control, pp.3788-3793, 1994. [15] I. Ersoy, F. Bunya, S. R. Subramanya, "A Framewor for Trajectory Based Visual Event Retrieval," International Conference on Information Technology: Coding and Computing, ITCC 2004, Vol.2, pp.23-27, Apr.2004. [16] J. Z. Li, M. T. Ozsu, D. Szafron, "Modeling of Moving Objects in a Video Database," IEEE International Conference on Multimedia Computing and Systems '97, pp.336-343, June 1997. [17] Z. Aghbari, K. Kaneo, A. Mainouchi, "Content-Trajectory Approach for Searching Video Databases," IEEE Trans. on Multimedia, vol.5, no.4, pp.516-531, Dec.2003. [18] S-C. Chen, M-L. Shyu, S. Peeta, C. Zhang, "Learning-Based Spatio-Temporal Vehicle Tracing and Indexing for Transportation Multimedia Database Systems," IEEE Trans. on Intelligent Transportation Systems, vol.4, no.3, pp.154-167, Sept. 2003. [19] C. Wohlera, J. K. Anlaufb, "A Time Delay Neural Networ Algorithm for Estimating Image- Pattern Shape and Motion," Image and Vision Computing, vol.17, pp.281 294, 1999. [20] N. Sumpter, A. Bulpitta, "Learning Spatio- Temporal Patterns for Predicting Object
Behavior," Image and Vision Computing, vol.18, pp.697 704, 2000. [21] M. Osadchy, D. Keren, "A Rejection-Based Method for Event Detection in Video," IEEE Trans. on Circuits and Systems for Video Technology, vol.14, no.4, pp.534-541, Apr.2004. [22] R. V. Babua, K. R. Ramarishnanb, "Recognition of Human Actions Using Motion History Information Extracted from the Compressed Video," Image and Vision Computing, vol.22, pp.597 607, 2004. [23] S.Hayin, Adaptive Filter Theory,3rded,Printice Hall,1996. [24] H. Sadoghi. Yazdi, M. Lotfizad, E. Kabir, M. Fathy "Application of Trajectory Learning in Tracing Vehicles in the Traffic Scene" 9 th Iranian Computer Conf., vol.1, pp.180-187, Feb. 2004. [25] H. Sadoghi. Yazdi, M. Lotfizad, E. Kabir, M. Fathy, "Clipped Input Data RLS, CRLS, Applied to Vehicle Tracing," to be published in EURASIP Journal on Applied Signal Processing, Issue 8, pp.1221-1228, May 2005. [26] D. Chetveriov, J. Verestoy, Feature Point Tracing for Incomplete Trajectories, Digital Image Processing, vol.62, pp.321-338, 1999. [27] K. Karmann, A. Brandt, "Moving Object Recognition Using an Adaptive Bacground Memory," in: V. Cappellini (Ed.), Time-varying Image Processing and Moving Object Recognition, vol. 2, Elsevier, Amsterdam, pp. 297-307, 1990. [28] S. Ando, Image Field Categorization and Edge/Corner Detection from Gradient Covariance, IEEE Trans. Pattern Analysis and Machine Intelligence, vol.22, no.2, pp.179-180, Feb. 2000.