ROBUST REAL-TIME ON-BOARD VEHICLE TRACKING SYSTEM USING PARTICLES FILTER. Ecole des Mines de Paris, Paris, France

Transcription

1 ROBUST REAL-TIME ON-BOARD VEHICLE TRACKING SYSTEM USING PARTICLES FILTER Bruno Steux Yotam Abramson Ecole des Mines de Paris, Paris, France Abstract: We describe a system for detection and tracking of vehicles from a single on-board frontal camera, developed as a part of the European CAMELLIA project. Five image processing algorithms are used to detect target vehicles, classify them and maintain their exact localization. The fusion of the result of the algorithms is done using particle filtering. We assert that the particles filter forms the optimal mechanism to exploit the perceived data since it maintains the full probability density function based on all available algorithm in a given illumination and weather conditions. The algorithms are designed to exploit a set of low-level image processing operations, provided by a smart imaging core developed in the project. The result is that the system runs on 20 images/sec even on a regular pentium PC, and is design to run on real time using an ARM and the hardware core. The system was tested on many sequences and performs well even in hard conditions like rain and night. Keywords: Vehicle Detection, Adaptive Cruise Control, Stop&Go, Smart Imaging Core 1. INTRODUCTION Detection and tracking vehicles from a frontal onboard camera, installed on a moving vehicle, is a problem with several applications in driver assistance system. This paper describes an application for vehicle detection and tracking developed as a part of the European IST CAMEL- LIA project, which focuses on a platform-based development of a smart imaging core to be embedded in smart cameras. This imaging core provides low-level image processing operations like morphological and arithmetic operations, and the applications developed in this context are designed to take advantage of these operations. This paper will concentrate only on one of the applications of the project (namely low-speed obstacle detection). The interested reader is referred to (Jachalsky et al., 2003) or to the CAMELLIA project website at for information about other parts of the project. The application we present here (the high-level part) combines the output of several pedestriandetection medium-level algorithms in order to obtain an exact detection and localization of pedestrians. These medium-level algorithms do not access the image pixels directly but use the low-level operations mentioned above (a full description of the low-level operations, including the source code, is found online in The medium-level algorithms used are: (1) Shadow detection (2) Vertical edges detection

2 (3) Rear lights detection (4) Symmetry detection (5) Motion estimation A particle filtering framework is used to handle the analysis of these five algorithm and to maintain the knowledge about the existing targets. 2. THE PARTICLES FILTER FRAMEWORK Particle filtering is a useful method to track targets in non-linear, non-gaussian environment (Doucet et al., 2002), and the tracking of vehicles according to observation from five different algorithms is one such environment. In this application we use particle filtering to fuse the results of the five algorithms in order to perform accurate detection, localization and tracking of one or several vehicles. Particle filtering is a technique for implementing a recursive Bayesian filter by Monte Carlo simulations. The reader is referred to (Arulampalam et al., 2002) for a complete overview. We maintain, for each target, at each step k, a probability density function (PDF) p(x k z 1 k ) of the state x k of the target, according to a set of observations z 1 k = z 1,..., z k received at steps 1 to k. The PDF is represented using a set of N random samples with associated weights {x i k, wi k }N i=1, or particles. In our particular case, The state x i k is a triple X, Y, T representing a box-like vehicle whose bottom rear edge is centered around the ground point X, Y, 0. The parameter T can have one of the three values {MOT O, CAR, T RACK}, corresponding to motorcycle, car and truck, and the vehicle dimensions are determined by T using table??. Vehicle is assumed to aligned with the axes system (i.e. θ = 0, φ = 0, ψ = 0), as well as the ground, which is supposed to be flat in the area of interest. Vehicles with θ 0 (e.g. when driving on a turn) are treated in a later work. Observations z j k, j = are arriving at step k from the four algorithms. These observations are available through the likelihood functions l j (x k ) = p(z j k x k ) [0, 1] {DONT KNOW }, j = provided by the different algorithms. An algorithm returning DONT- KNOW is saying that it has no opinion about this particle, having not enough information available. This can happen, for example, when there is not enough illumination for shadow detection or when a car is entering from the left, still not exposed enough to perform symmetry analysis. 2.1 the likelihood function The final likelihood function which is used to update the weights of the particles is calculated as a product of all the likelihood values (except the ones equal to DONTKNOW), since they represent independent probabilities. A full description of the likelihood functions exceeds the scope of this paper; Naturally, each algorithm uses its special observation z k of the k th image (and, in the case of the motion estimator, also the k 1 th ) to estimate the probability p(z k x i k ) for each of the particles x 1 k... xn k of each existing target. Table 1 describes one iteration of the algorithm. The chosen particles filter version is the Sampling Importance Resampling (SIR) filter (see (Arulampalam et al., 2002), section B.1), which is efficient, simple and functions well when the distribution of the likelihood p(z k x k ) is not tighter than the prior density p(x k x k 1 ), as is the case here. The prior density - i.e. the evolution and noise model of the states - is calculated using the smoothed vehicle speed from the previous results. 2.2 Initialization of targets Targets are initialized by the shadow, the rearlights and the motion detectors. Each of the algorithm initialize a target anywhere it has an observation (a shadow, rear-lights or motion) which is not assigned to any existing targets. It is important to note that the algorithms are tune to prefer false initialization than missed target. This is because a falsely initialized target can be eliminated by low likelihood after it exists. A vehicle entering from the sides (so-called Cutin situation) is initialized as early as the time it appears on the screen, much before its rear part is visible. This is done using the motion detector, while using the shadow, and later the lights, as first likelihood values (see section for examples). Once appearing on the screen, the target is assumed to be in the nearest lane, with a lateral distance of 3.5 meters from the host vehicle. The true location is refined by the likelihood values. 3. THE ALGORITHMS When speaking about image processing algorithms in this context, we are referring to a type of algorithms which is being ran one time per every image of the video sequence. Typical algorithm will get as input the image, plus some additional information about already existing targets. For some algorithms this additional information is crucial because they are target-specific. For others this information is needed only to improve performance. After the algorithm is ran, it uses the results in two ways: create new targets and support/decline

3 Input: A target T already detected at step l of the algorithm (where l < k), represented by a set of particles {x i k 1, wi k 1 }N i=1 A likelihood function l(x) = p(z k x), provided using observations z k associated with the target T by the nearest neighbor method. A constant W 3, the speed calculation moving window width; V, a threshold for the variance of the speed; σnoise 2, the variance of the pedestrian movement noise; and N - number of particles. Calculate SS, the smoothed speed of the target: avr(s l+1,... s k 1 ) if k l > W and SS { var(s l+1,... s k 1 ) > V 0 otherwise N Where s k = i=1 (wi k xi k wi k 1 xi k 1 ) is the transient speed of the object at step k. Do for i = 1... N - Draw particle x i k normal(xi k 1 + SS, σ2 noise ) - Calculate w i k = l(xi k ) Calculate total weight t = N i=1 wi k Normalize particles weights: wk i wi k t, i = 1... N Resample using standard particles resampling algorithm (see (?)). Table 1. Step k of the particles filter on a given target theses (=particles) of existing targets. To create new targets, an algorithm provides us with a list of new targets, and for each target a list of hypotheses (=particles) where it believes the target is found. To support or decline the hypotheses of existing target, algorithms provide their likelihood function. We hereby briefly review the five algorithms used in our applications. These algorithms, except the motion detector, were used in (?), in the framework of a bayesian network. 3.1 vehicle shadow detection Shadow detection is based on the thresholding of images to find areas of darkness on the road. These areas are often shadows made by vehicles. The results are used both to generate new targets and to support/decline the existence of already discovered targets. The algorithm first selects a rectangle which is likely to fall entirely on the road, as demonstrated on Figure??. Then it computes the histogram of pixel values on this rectangle, as shown in Figure??. It threshold the entire image with a threshold value equal to 90% of the peek of the histogram (the threshold value is filtered in time, to prevent artifacts). Since this value is a little lower than the intensity of the road, the resulting blobs are the shadows. Shadows, which are located on reasonable locations (i.e. represent an approximate vehicle size when retroprojected to the world coordinates systems) are used to calculate likelihood values of existing targets - if they are close to one - or to initialize new ones otherwise. The system allows two separate shadows to match one vehicle, as shown in figure??. 3.2 rear-lights detection Lights detection is based on the detecting the rear lights and license plate of a car. The algorithm creates a histogram of the V image plane, and thresolds the images with a value equal to 95% of the maximum of the histogram, as shown in figure??. Then it searches for couples of blobs representing lights, or triples representing lights and plate. These values are used to initialize new targets or to provide likelihood value to existing targets particles. 3.3 symmetry detection Symmetry detection is based on the calculating the SAD (Sum of Absolute Differences) between some area of the image and its adjacent area in the mirror image, as seen in Figure??. If the two areas are two parts of a symmetry, the SAD will be zero. We repeat this algorithm on a set of adjacent pairs of regions to find the symmetry function as shown in Figure??. The results are used only to support/decline the existence of already discovered targets. Note that when the target is a vehicle, the symmetry function will typical have significantly low values in its middle and very high values in its sides, as seen in figure??. The low symmetry on the sides is as important as the symmetry at the middle. To understand this, observe that the sky have good symmetry in all the places. The algorithm does not initialize new targets. It provides the likelihood value by convultion of the result with the ideal function as shown in figure??. 3.4 vertical edges detection Vertical edges detection is based on looking for typical patterns of the vertical summing of the sobel image. The results are used only to support/decline the existence of already discovered

4 targets. Summing of the sobel typically gives a 3-peek function as shown in figure??. The 3 peeks become two when the vehicle is completely aligned ahead. Like in the symmetry algorithm, this algorithm does not initialize new targets, and provides the likelihood value by convultion of the result with the ideal function as shown in figure??. 3.5 motion segmentation are would like to add to the system is a learning This algorithm uses the motion detector of (Wittebroodalgorithm based on AdaBoost as done in (Viola and de Haan, 2001), which is highly efficient. The and Jones, 2001) for the detection of vehicles, with system computes motion models for each image the boosting values being the likelihood value. block, then groups blocks with similar motion, to form objects. These objects pass through a temporal filter that keeps only the motions which are consistent for several consecutive images. The algorithm is used to initalize targets and to provide likelihood values. The calculation is done simply by correlation with the detected moving objects. As will be seen in section 4, the likelihood value of this algorithm is quite coarse; it s important value is in initializing targets. We suggest that the particle filtering mechanism forms a good framework to fuse together any combination of algorithms to perform vehicle tracking. In another work being made as a part of the FADE2 Renault project, we combined laserradar information as just another algorithm which gives precise longitudinal information when a beam hits an object in front, and DONT- KNOW otherwise (see figure??). As expected, the mechanism performed well in exploiting this information. Other sources of information which Other extensions to the system to be done in the future include enlargement of the state space with θ 0 - that is, cars which are not directed straight ahead (e.g. when driving on a turn). This should be done carefully because merely adding dimensions to the state space might severely degrade performance. ACKNOWLEDGMENT 4. THE SYSTEM IN PRACTISE In this section we demonstrate, by showing reallife situations, that the particle filtering mechanism optimally exploits the analysis coming from the five algorithms. Figure 1 shows a typical daylight situation. A car is found about 30 meters in front of the host vehicle, and is initialized by the shadow detector. The target is quickly validated by all the algorithms except the motion detector (since the projected image of the car is not moving). In figure 2 we see a continuation of that sequence with a car entering from the right ( cut-in situation). The car is initialized very early by the shadow or motion detection (whichever detects first; usually in the second image after the appearance) and is maintained at the beginning mainly by the shadow detection, based on the assumptions outlines in section 2.2. The figure shows the scene a little bit later, after the rear of the car is beginning to appear. At this stage we can already see some initial contributions from the lights and vertical edges detection. As seen, motion detection is giving a high probability to the existence of the target but very little localization information. 5. CONCLUSION AND FUTURE WORK We have presented an application for vehicle detection and tracking based on particle filtering. This work was supported by the European IST CAMELLIA project and by the FADE- 2 Renault project. We would like to thank our partners including Renault research division, Philips Eindhoven, Phillips semiconductors Hamburg, and the universities of Hannover and Las- Palmas. REFERENCES Arulampalam, S., S. Maskell, N. Gordon and T. Clapp (2002). A tutorial on particle filters for on-line non-linear/non-gaussian bayesian tracking. IEEE Transactions on Signal Processing 50(2), Doucet, Arnaud, de Freitas, Nando and Gordon, Neil, Eds.) (2002). Sequential Monte Carlo Methods in Practice. Statistics for Engineering and Information Science. Springer-Verlag, New York Berlin Heidelberg. Jachalsky, J., M. Wahle, P. Pirsch, S. Capperon, W. Gehrke, W.M. Kruijtzer and A. Nuez (2003). A core for ambient and mobile intelligent imaging applications. In: Proceedings of the 2003 IEEE International Conference on Multimedia & Expo (ICME 2003). p. CDROM. Viola, Paul and Michael Jones (2001). Rapid object detection using a boosted cascade of simple features. In: Proceedings IEEE Conf. on Computer Vision and Pattern Recognition. Wittebrood, R.B. and G. de Haan (2001). Real-time recursive motion segmentation of

5 (a) (b) (c) (d) (e) (f) Fig. 1. A typical situation of a vehicle in front of our car. The frontal view (2(f)) and likelihood functions (from bird s eye view) of the shadow detection (2(a)), the vertical edges detector (2(b)), the symmetry detector (2(c)), the motion estimator (2(d)) and the rear lights detector (2(e)). Values are denoted from black (0) to white (1). (a) (b) (c) (d) (e) (f) Fig. 2. A vehicle entering from the left ( cut-in situation). The frontal view (2(f)) and likelihood functions (from bird s eye view) of the shadow detection (2(a)), the vertical edges detector (2(b)), the symmetry detector (2(c)), the motion estimator (2(d)) and the rear lights detector (2(e)). Values are denoted from black (0) to white (1).

6 video data on a programmable device. In: IEEE Transactions on Consumer Electronics. pp