International Journal of Comuter Alications (0975 8887) Moving Objects Tracking in Video by Grah Cuts and Parameter Motion Model Khalid Housni, Driss Mammass IRF SIC laboratory, Faculty of sciences Agadir -Morocco Youssef Chahir GREYC-UMR CNRS 607, University of Caen Caen France ABSTRACT The tracking of moving objects in a video sequence is an imortant task in different domains such as video comression, video surveillance and object recognition. In this aer, we roose an aroach for integrated tracking and segmentation of moving objects from image sequences where the camera is in movement. This aroach is based on the calculation of minimal cost of a cut in a grah Grah Cuts and the D arametric motion models estimated between successive images. The algorithm takes advantage of smooth otical flow which is modeled by affine motion and grah cuts in order to reach maximum recision and overcome inherent roblems of conventional otical flow algorithms. Our method is simle to imlement and effective. Exerimental results show the good erformance and robustness of the roosed aroach. General Terms Tracking, Video Analysis, Comuter Grahic. Keywords Grah Cuts, Motion Models, Tracking, Video Analysis, Overlaing Objects. 1. INTRODUCTION Tracking of moving objects is one of the major areas of research, several aroaches have been develoed. We can classify these aroaches according to the rincile used in two great classes: Model-based aroach, these aroaches use the result obtained in the revious frame as a rediction rocess initialization in the current frame [1,, 3, 4, 5, 6]. Motion-based aroach (also called intensity based) [7, 8, 9, 10], the rediction of the tracked object is done by the motion model estimated between the revious and current frame. Whatever the method used, model-based or motion-based, we can classify these aroaches according to the object shae reresentations commonly emloyed for tracking into three categories (the reader is referred to [11] for details): Point: The object is reresented by a oint, that is, the centroid [1] or by a set of oints [13, 1]. Primitive geometric shaes: Object shae is reresented by a rectangle, ellise, etc [14]. Object silhouette and contour: the object is reresented by a contour [5] or by silhouette (region inside the contour) [6, 4]. The methods of the last category are often based on dynamic segmentation technique. These methods are used for tracking comlex non-rigid shaes and when we want to extract the silhouette of the object at any time, without rior knowledge of its new shae. To summarize, a robust aroach to tracking moving objects must solve three major classes of roblems. Problem of large dislacements: the methods based on the modification of contour (like [5]) or shae (like [4]) redicted in the revious ste cannot suort the great movement of objects. The simlest way to solve this roblem is to use the motion model of the object to track in the ste of redicting. Problem of toology change: Several aroaches in the literature namely [1,, 3] cannot handle toology change due to contour or silhouette used in the initialization or due to dominant motion model used that doesn t always cover all ixels of the object to track. This is the case for comlex non-rigid shaes (Namely the movement of the feet). Problem of overlaing objects: the classification of ixels of overlaing objects is the most difficult roblem and esecially when we want to extract the shae of objects at any time. In this aer, to manage the objects dislacement, we used a rediction based on motion model comuted between the current and revious frame. That will take into account the large dislacement. And we have used a dynamic segmentation ste based on otimal boundary diction by grah cuts to solve the roblem of toological change and the movement with a comlex model (such as when arts of an object have different models). This stage of refinement by grah cuts cannot classify the ixels of objects that are overlaed. We overcome this limitation through the use of a similarity measure based on the observed intensities. This aer is organized as follows: In the next section we resent the basic definition and terminology of grah cuts and how they are used for image segmentation. In sections 3 and 4, we introduce the algorithm of object extraction under hard constraints and we exlain how to use a grah cuts algorithm in order to refine a border of redicted objects, to obtain rich objects descrition and to overcome inherent roblems of conventional otical flow algorithms, and we give a solution for case of overlaing objects. In section 5, we give the results obtained. Lastly, we give a conclusion and future works.. GRAPH CUTS METHODS IN VISION Grah cuts have been used to efficiently solve a wide variety of vision roblems like stereo [15] image segmentation [16, 17], image restoration [15, 18], texture synthesis [19] and many others that can be formulated in terms of energy minimization. This energy is reresented in this form: 0
International Journal of Comuter Alications (0975 8887) E( L) E P data D E ( L smooth ) q 1.. n N V q ( L, L ) Where E is a iecewise smoothness term, smooth E is a data data term, P is the set of ixels in the image, V L, L ) is a q q ( q smoothness enalty function to imose a satial consistency, D(l P ) is a data enalty function and (L ) =1... n is a vector of variables in finite sace L L { s, t} (L for label). Every ixel is associated with a variable L. N designates the set of the neighbors of the ixel. The main idea of a grah cuts is to bring back the roblem of minimization of energy to a roblem of a minimal cut in a grah. Greig and al. have shown that the minimization (such as estimating the maximum a osterior Markov random field) can be achieved by the minimum cut of a grah with two secific nodes "source" and "sink" for the restoration of binary images [18]. Later, the aroach has been extended to non-binary roblems known as "S-T Grah Cuts" [15].V.Kolmogorov [0] have given a necessary and sufficient condition, known as submodularity condition for a function to be minimized by grah cuts. In the case of energy (1), V q (L, L q ) is a term of vicinity, which makes it ossible to check the condition, and the construction of grah is done as follows: As shown in figure 1, each ixel P ij in the image corresonds to a node (vertices) v ij in the grah. Two additional nodes are the source and sink, resectively the object and the background. Each node is linked to its neighbors by edges n-link, with connectivity chosen (in our imlementation we used 8- connected lattice as our neighborhood structure), and whose caacities deend on differences in intensity, this allows to encourage the segmentation (cut) which asses through regions where the image gradient is strong enough. Each node is also connected by edges, t-link to the terminals (source and sinks). A cut C of grah G (V, E) is a artition of the vertices V into two sets V 1 and V, such that : V V 1 1 V V V source V 1 and sink V In grah theory, the cost of a cut is usually defined as the sum of the weights corresonding to the edges it serves as is formulated in (). (1) The value of E (x) is equal to same constant lus the cost of the ST minimum cut C. We say that E is exactly reresented by G if the constant = 0. To calculate the minimum cut, several algorithms have been develoed: increasing ath [1], Pushing-relabel [] and the new algorithm of Boykov - Kolmogrov [3]. In our alication, we used the algorithm of Boykov-Kolmogrov for its seed. Sink Source Fig 1: Construction of grah for a 3x3 image with the connectivity N8. Each ixel corresonds to a node, and all ixels are connected to the source and the sink. 3. OUR APPROACH The simle way for tracking an object in movement is to detect motion by the subtraction of each two successive frames, but this necessitates that the background is stationary (fixed camera). In this aer, we resent an aroach of tracking moving objects in a dynamic frame (camera in movement). In the first ste, we use the grah cuts to select the objects to be tracked (section 4-1). The second ste consists of calculating the motion model for each object (section 4 -), which corresonds to the selected objects (or redicted objects), between the current frame and the following one. These models of movement will be used to redict each object in the next frame. Thereafter, we will use the grah cuts to obtain recise contours (section 4-3). Figure gives the rincial stes of our algorithm. To classify the ixels of overlaing objects we used the multi may cuts algorithm (section 4-4). 4. ALGORITHM IMPLEMENTATIONS In this section, we discuss the imlementation details of the roosed algorithm deicted in figure.the algorithm consists of three major stes : Selection of objects to track by Grah cuts (ste of initializatio, Prediction ste by visual motion model, and ste of borders objects refinement. C S qt (, q) C w(, q) () 1
International Journal of Comuter Alications (0975 8887) Selection of objects to track Prediction ste Refinement ste Estimation of motion model of each object between the frame i and frame i+1 Prediction of all objects in the frame i+1 Refining the objects border Automatic initialization of objects to track in the next frame Case of overlaing objects No Use of multi-way cuts to classify the ixels of the overlaing objects Number of objects refined == initial number of objects Yes Last frame of the video sequence? No Yes End Fig : General outline of our aroach. 4.1 Selection of Objects to Track To select the objects to track, we used the interactive segmentation method based on detection of the boundary by Grah Cuts [16]. The extraction is done by the exloitation of the rior information entered by the user (seed of the object and background) (see figure 4 -a). These ixels selected by the user are then connected to the terminals (source and sink) by the edges with infinite weight (4) (5). This will satisfy the constraints (3), while all other ixels (nodes) will have no links to the terminals (6) It is about using some toological constraints, known as hard constraints which can indicate that certain seeds of image a rior identified as a art of the Object or the Background All neighboring ixels are connected by edges, n-links, whose caacities deend on intensity differences I Iq (7). (see figure 3). O : L 1 B : L 0 The cost of the edges is given by: W (, S) if O W (, T) 0 W (, S) 0 if B W (, T) W(, S) W(, T) 0 if O B I I q W (, q) ex( * 1 ) dist(, q) : Standard deviation of the intensities of the seeds of object S: source, T: sink, O: seeds of object, B: seeds of background. Fig 3: A simle D otimal boundary for 3x3 image. Boundary conditions are given by object seeds O and background seeds B rovided by the user.
International Journal of Comuter Alications (0975 8887) (a) (a) (b) (b) (c) Fig 4: (a) First frame with hard constraints, (b) Objects selected by grah cuts- to be tracked, (c) identification of selected objects by algorithm of detection of connected comonent. 4. Visual Motion Model The whole key of the moving objects tracking, using motion based aroach, resides in the choice of the motion model for each object because this one must reresent the movement of all ixels in the object. Our aim is to control the dislacement of the objects to track with a rior knowledge of the objects and the background in the revious frame. As we have already introduced, a motion model is used to aroximate the objects in the courant frame. In the literature, several models of motion have been resented like rigid, homograhy, quadratic, etc. Affine Model (8) is a reasonable comromise between the low comlexity and fairly good aroximation of the non-comlex motions of objects at about the same deth. A u( i ) c1 a1 a xi ( i ) * v( i ) c a3 a4 yi In other words, affine motion model is a arametric modeling to six arameters (c 1,c, a 1, a, a 3, a 4 ) of the movement defining the transformation T: Ω t Ω t+1 which ermits to ass the ixel x to the ixel x'. Several methods have been roosed to estimate a arameter motion model. The most famous method in this field could be using multi-resolution aroach [4]. This method allows calculating a arameter motion model in real times between the two inut frames. In our alication, we used this algorithm for its seed. An imlementation of this algorithm between two successive images can be found in MotionD software, develoed at Irisa/INRIA Rennes [5]. (c) Fig 5: (a) First frame with hard constraints, (b) Object selected by grah cuts -to be tracked, (c), (d) The object redicted in the frames, 3 of the rond-oint sequence using motion model. 4.3 Refinement of Border Objects (d) We can clearly see that in the figure 5, that a art of the road background at summer covered by the redicted object and a art of the car object is not covered. Figure 6 & 8 shows this. Fig 6: This figure illustrates that a art of the rode at summer covered by the redicted object and a art of the car is not covered. To obtain a rich object descrition with the aim of accomlishing advanced scene interretation tasks we used the grah cuts aroach described in section 3.1. therefore, as we have already introduced, grah cuts method for segmentation requires the user to mark a few ixels as object (seeds of object) and few others as background (seeds of background) and to generate automatically this information, we used the mathematical morhology (see figure7): Seeds of object: Four-time erosion of object detected using motion model (figure 7-b). Seeds of background: Seven-time dilatation of the object detected minus five-time dilatation of the object detected (figure 7-c). 3
International Journal of Comuter Alications (0975 8887) (a) (c) (b) Fig 7: (a) object redicted using motion model (b), (c) hard constraint (seeds of object and background). The area between the blue (seeds of object) and the red line (seeds of background) is a band of incertitude around the borders. A grah is then built on this band to otimize the contours of the object. This otimizes the borders object with a recision of ± 9 ixels, thus the number of ixels an object aroaching the camera will increase and the reverse for the object away from camera. Fig 8: First image gives the result obtained using only motion model, second image give the result obtained after the refinement by grah cuts. 4.4 Case of Overlaing Objects Before calculating the motion model for each object, we use the algorithm of detection of a connected comonent to identify the selected objects (labeling). This algorithm will classify the image ixels according to their belonging to object_1, object_ object_n or to background. However, the image ixels classification is reliable when the objects to track are searated and consequently, multi-objects tracking may be ambiguous in some cases, like in the case of overlaing objects. We overcome this limitation through the introduction of a visual aearance constraint, where a classification according to the observed intensities is to be considered. To this end, we will use the multi-way cuts algorithm [15] to classify the ixels of overlaing objects. We define the cost functions for multi-way cuts as follows: (d) I objecti ( i1,... and if I' Ai ( ) objecti ( i1,... W ( Ai ( ), Li( i1,... ) 0 W ( Ai ( ), B) 0 I object i( i1,... and if I' Ai ( ) object i( i1,... W Ai ( ), L ) ex I I i ( i 1,... W ( Ai ( ), B) 0 if I' ( Ai ( ) W (, Li( i1,... n Background W (, B) ) ) 0 Where object i(i=1, is one of the objects to track, I is the image ixel in the frame I, ω Ai is affine motion model of object i, I ωai() is the match of ixel in the frame I using motion model, L i(i=1 is the terminal i (corresonds to the object i), B is the terminal corresonds to the background. (9), (10) and (11) are used to calculate the costs of the edges connecting vertices (ixels) with terminals. All neighboring vertices are connected by edges, n-links, whose caacities is calculated by (7). 5. EXPERIMENTATIONS AND RESULTS 5.1 Data The method has been tested on five sequences (taxit, rondoint, walk, croisement and Coastguarder). The first sequence has 41 frames of 56 x 191. The next sequence has 33 frames of 56 x 4. The third sequence has 17 frames of 300 x 00. The fourth sequence has 7 frames of 300 x 00.The last sequence has 80 frames of 300 x 40. In rond-oint, croisement and coastguarder sequence, the camera is in movement, and it is fixed in two other sequences. 5. Results and Comlexity The grah cuts aroach is often used to solve a labeling roblem. Knowing that, the comlexity to solve such a roblem deends on the number of ixels being matched. And in order to reduce a comlexity of grah cuts labeling, we only used the rectangle which contains the object to treat instead of using the entire image, this method allows us to reduce the size of the grah. By this reduction in data size, and by using the fast algorithm described in [3], the grah cut is made in a realtime. On figure 9, we can on the one hand see that the curve reresenting the number of ixels detected in the rediction ste is below the curve reresents the number of ixels of object refinement. On the other hand, the curve reresenting the number of ixels rejected by the refinement ste never touches the x-axis, which shows that the motion model (called dominant motio does not always cover all the ixels of the tracked object. Then the object redicted in the first ste, rediction ste, is only used as initialization to the next ste (refinement ste). 4
number of ixels numbre of ixels International Journal of Comuter Alications (0975 8887) Figure 10 shows the change in the size of the object between the first and the last frame. This change in size may be due to rarochement or being away of the object from the camera or due to the change in the toology. We have overcome these roblems, the roblem of the change in the size and the roblem of the motion model that does not cover all the ixels of the object, by the stage of refinement. We have imlemented our aroach in C++. And for all tests that we did, we used x GHz Intel Core Duo with GB in memory. Table 1: means execution time er frame (note that for reducing the comlexity of grah cuts labeling, we only used the rectangle which contains the object to refine instead of using the entire image that reduces the size of grah). Comutation of Solve a grah Build a grah motion model cut 0.013 s 0.005 s 0.01 s In case of the objects tracking that are not overlaing, total time er frame is 0.039 s. In this case, the tracking can be obtained in real time. In the other case, to this cost we must add the comutation time required to classify the ixels of overlaing objects (=number of objects * time to build and solve a grah cuts) for examle, for croisement sequence (two objects overla) means execution time er frame is 0.094 s. At first sight, we note that we have a great difference between the results (figure 8); the tracking resulting from combining Motion-model with Grah cuts is visually more satisfactory than those obtained by using only a arameter motion model modeled by affine motion. Figure 11 and 1 gives some examles of the exerimental results. We can see in the last frame of croisement sequence (figure 1) the reaearance of the object (black car) while the rogram could not detect it. This is normal because the object has disaeared in the revious frame and in the algorithm which we roosed the first ste uses the object selected by the user or the object detected in the revious ste. 4500 4000 3500 3000 500 000 1500 1000 500 0 statistical results obtained from rond-oint sequence Figure 9: statistical results obtained shows the number of true ixels detected in the first ste (tracking using only motion model), number of ixels added and rejected by ste of refining the objects border. 4500 4000 3500 3000 500 000 1500 1000 500 0 number of ixels of refined object number of true ixels detected by the first ste number of ixels rejected by the ste of refining number of ixels added by the ste of refining frames object size (in number of ixels) in the first and last frames first frame rond-oint walk taxi laste frame Fig 10: This figure shows the change in the size of the tracked objects. This change is taken into account in the ste of refinement of objects. 6. CONCLUSION AND FUTURE WORK In this aer we resented how we can track moving objects in video using the grah cuts aroach and otical flow modeled by affine motion. We showed how to refine the border of redicted objects. The combined aroaches roduce dense and accurate otical flow fields. It rovides a owerful and closed reresentation of the local brightness structure. Globally, the evaluation showed that the roosed aroach gives good results according to our test. Further research will concentrate on imroving the algorithm by defining criteria for identifying what roerties characterize objects and distinguish them from other objects and from the background that will allow us to find the disaeared object. We will also try using more sohisticated initialization of seeds (automatic initializatio. Fig 11: Result obtained in frames 7, 13, 18 and from rond-oint sequence 5
International Journal of Comuter Alications (0975 8887) Taxi sequence Walk sequence Coastguarder sequence Croisement sequence Fig 1: Exerimental results. We can see in the last frame of croisement sequence the reaearance of the object (black car) while the rogram could not detect it. This is normal because the object is disaeared in the revious frame and in the algorithm which we roosed the first ste uses the object selected by the user or the object detected in the revious ste 6
International Journal of Comuter Alications (0975 8887) 7. REFERENCES [1] D. Terzooulos, R. Szeliski. 1993. Tracking with kalman snakes. Active vision,.3 0. [] M. Isard, A. Blake. 1998. Condensation conditional density roagation for visual tracking. Int. J. Comuter Vision, 9(1) :5 8. [3] J. MacCormick, A. Blake. 000. A robabilistic exclusion rincile for tracking multile objects. Int. J. Comuter Vision, 39(1) :57 71. [4] C.R. Wren, A. Azarbayejani, T. Darrell, A.P. Pentland. 1997. Pfinder: real-time tracking of the human body. IEEE Trans. Pattern Anal.Mach. Intell. 19(7), 780 785. [5] N. Xu, N. Ahuja. 00. Object contour tracking using grah cuts based active contours. Proc. Int. Conf. Image Processing. [6] B. Rosenhahn, U. Kersting, S. Andrew, T. Brox, R. Klette and H-P. Seidel. 005. A silhouette based human motion tracking system Technical Reort (The University of Auckland, New Zealand: CITR) ISSN: 1178-3581. [7] B. K..P. Horn, B. G. Schunck. 1981. Determining Otical Flow. AI 17,. 185 03. [8] J. Barron, D. Fleet, S. Beauchemin. 1994. Performance of otical flow techniques. Int. J. of Comut. Vis. 1(1), 43 77. [9] A.R. Mansouri, J. Konrad. 003. Multile motion segmentation with level sets. IEEE Trans. Image Process. 1(), 01 0. [10] G. Adiv. 1985. Determining 3D motion and structure from otical flow generated by several moving objects. IEEE Trans. Pattern Anal. Mach. Intell. 7(4), 384 401. [11] A. Yilmaz, O. Javed, M. Shah. 006. Object tracking: A survey. ACM Comut.Surv. 38, 13. [1] C. Veenman, M. Reinders, and E. Backer. 001. Resolving motion corresondence for densely moving oints. IEEE Trans. Patt. Analy. Mach. Intell. 3, 1, 54 7. [13] D. Serby, S. Koller-Meier, and L. V. Gool. 004. Probabilistic object tracking using multile features. In IEEE International Conference of Pattern Recognition (ICPR). 184 187. [14] D. Comaniciu, V. Ramesh and P. Meer. 003. Kernelbased object tracking. IEEE Trans. Patt. Analy.Mach. Intell. 5, 564 575. [15] Y. Boykov, O. Veksler, and R. Zabih. 001. Fast aroximate energy minimization via grah cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 3(11):1 139. [16] Y. Boykov, M.P. July. 001. Interactive Grah Cuts for Otimal Boundary & Region Segmentation of Objects in N-D Images. In: roceedings of the International Conference on Comuter Vision, Vancouver Canada. [17] K. Housni, D. Mammass and Y. Chahir. 009. Interactive ROI Segmentation using Grah Cuts, GVIP-ICGST Journal,Volume 9, Issue 6, 1 6.. [18] D. Greig, B. Porteous, and A. Seheult. 1989. Exact maximum a osteriori estimation for binary images. Journal of the Royal Statistical Society, Series B, 51():71 79. [19] V. Kwatra, A. Schodl, I. Essa, and A. Bobick. 003. Grahcut textures: image and video synthesis using grah cuts. In Proceedings of SIGGRAPH. [0] V. Kolmogorov. 004. What Energy Functions Can Be Minimized via Grah Cuts?. IEEE transactions on attern analysis and machine intelligence, vol. 6, NO.. [1] L. Ford and D. Fulkerson. 196. Flows in Networks. Princeton University Press. [] A. Goldberg and R. Tarjan October. 1988. A new aroach to the maximum flow roblem. Journal of the Association for Comuting Machinery, 35(4):91 940. [3] Y. Boykov and V. Kolmogorov. 000. An exerimental comarison of min-cut/max-flow algorithms for energy minimization in vision. 3rd. International Worksho on EMMCVPR. Sringer-Verla, Setember. [4] J.-M. Odobez, P. Bouthemy. 1995. Robust multiresolution estimation of arametric motion models. Journal of Visual Communication and Image Reresentation, 6(4):348-365. [5] Vista team, htt://www.irisa.fr/vista 7