FUSION OF LASERSCANNER AND VIDEO FOR ADVANCED DRIVER ASSISTANCE SYSTEMS

Transcription

1 FUSION OF LASERSCANNER AND VIDEO FOR ADVANCED DRIVER ASSISTANCE SYSTEMS Nico Kaempchen, Klaus C.J. Dietmayer University of Ulm, Department of Measurement, Control and Microtechnology Albert-Einstein-Allee 41, D Ulm, Germany Tel: Fax: {Nico.Kaempchen, Klaus.Dietmayer}@e-technik.uni-ulm.de ABSTRACT In established driver assistance systems each application relies on its own sensor, which observes the vehicles environment. Advanced driver assistance systems (ADAS) have increasing demand for several sensor systems. The described Laserscanner and video based sensor fusion approach serves as a general platform for multiple active safety and comfort applications. A large field of view is obtained and the certainty and precision of the estimates in the relevant regions is increased significantly. Vehicle position measurements and classification, lane estimation as well as pedestrian recognition enable a broad support for applications such as Lane Departure Warning, Automatic Emergency Breaking, PreCrash, Pedestrian Protection, ACC Stop&Go and Low Speed Following. INTRODUCTION Recent projects on driver assistance systems have focused on applications such as PreCrash (Chameleon) [1], ACC Stop&Go (Carsense) [2] and recognition of vulnerable road users (PROTECTOR). These advanced driver assistant systems (ADAS) have increasing demand for several sensor systems, which are mainly complementary but also redundant. Some research has therefore been focused on multi-sensor fusion [2,3,4,5,6]. The aim of such a step is to provide a fused description of the traffic scene surrounding the vehicle, which is relevant for ADAS, but not specific to a certain application. This fusion system incorporates the data of diverse sensors into a single description. Thus the field of view of a single sensor is enlarged, the certainty and precision of the estimates is increased and additionally this system design is economically efficient, as different applications share a set of sensors. SENSORS A Laserscanner and a monocular camera are combined in order to enable a reliable environment recognition in distances of up to 100 m and a wide angular field of view. LASERSCANNER Figure 1: ALASCA Laserscanner of the company IBEO AS integrated into a test vehicle. 1

2 VIDEO The monocular camera is mounted behind the windscreen beside the inner rear mirror. The camera is equipped with a ½ CCD chip which has a standard VGA resolution of 640x480 pixel. With a 8 mm lens a horizontal view of 44 degrees is realised at an average angular resolution of 0.07 degrees per pixel. SPATIO-TEMPORAL ALIGNMENT In order to synchronise the sensors, the camera is triggered when the rotating Laserscanner head is aligned with the direction of the optical axis of the camera [7]. The sensors are calibrated in order to enable a spatial alignment. By means of an accurate synchronisation and calibration, image regions can be associated directly with Laserscanner measurements (Fig. 2). Therefore it is possible to assign certain image parts a distance, which is a major advantage of this fusion approach. Figure 2: Left: Laserscanner measurement in the bird view. The different layers of the Multilayer Laserscanner are colour coded. In grey the field of view of the camera. Right: Laserscanner data projected into the image domain. SENSOR FUSION The Laserscanner and the video sensors are combined in the fusion system in a way which enables a many synergetic effects. The Laserscanner tracks and classifies vehicles, motorbikes and pedestrians in the environment in front of the ego vehicle. The image processing unit estimates the lane and the position of the ego vehicle in the lane. Another image processing module refines the lateral position and velocity estimation of the objects. High-Level Fusion Combination, Verification Lane Recognition Object Tracking and Classification Object Tracking ESP Laserscanner Video Figure 3: System architecture involving different levels of sensor fusion. 2

3 The high-level fusion combines the ego motion estimation, the lane recognition and the detected objects into a unified description. Based on this high-level fusion, applications could detect lane departures of the ego vehicle, obstacles in the driving path, determine objects of interest for ACC and Low Speed Following or detected potentially endangered pedestrians. The sensors are used mainly complementarily. Environment recognition in the near field (< 50 m) is based on the very accurate Laserscanners object tracking which covers a wide angular field of view. In the frontal far field the lateral offset estimate is refined by an image processing (Fig. 4). The highly reliable object detection of the laserscanner is used in order to control the attention of the video system. The distance measurements are combined with the viewing angle measurement of the video resulting in a precise width and position measurement of the object without any flat world assumption often used in monocular image processing approaches. Figure 4: Synergetic effect of sensor fusion in the far field: Accurate distance measurements of the Laserscanner (blue) and the viewing angle of the video (green) result in precise width and position measurement (red). The sensor fusion architecture also enables a data flow from the high-level modules to lower levels. For instance the Laserscanner and the video object tracking are provided with information about the ego-motion whereas the lane recognition module might incorporate the object positions. LASERSCANNER TRACKING AND CLASSIFICATION The distance measurements generated by one revolution of the rotating Laserscanner head (scan) are divided into segments, which are assumed to belong to the same object. The appearance depends on the orientation and distance of the object in the Laserscanner s coordinate system. Tracking is based on these segments using a Kalman-filter approach, where the state vector consists of the objects position, velocity, orientation, width and length [8]. The object is classified using the filtered width, length and velocity as features [9,10]. Possible object types are pedestrian, car, truck, (motor-) bike, small and big. VIDEO OBJECT TRACKING One advantage of the Laserscanner measurement principle is that only there where the laser beam is reflected by an object, a measurement is performed. The Laserscanner is therefore very reliable in object detection. Additionally the distance is measured very accurately. Both characteristics are weak points in monocular image processing approaches. On the other hand a video image exhibits a higher angular resolution than a scan. We therefore decided to take the advantage of both systems and design an video image processing module which is attention driven by the Laserscanner s object detection and which refines the lateral position and velocity as well as the width measurements. 3

4 ATTENTION CONTROL A region of interest (ROI) in the image is calculated from the objects width, measured by the Laserscanner and the type of the object, classified by the Laserscanners object recognition (Fig. 5). The ROI of each object is then searched for features which permit to recognise the presence of a vehicle. Depending on the classification of the Laserscanners object recognition into cars and trucks, different features are searched for. Using adaptable thresholds the applied image processing is widely independent of illumination changes. (a) (b) Figure 5: Attention control by the Laserscanners object detection PATTERN MATCHING FOR CARS Figure 6: Illumination and car type independent dark areas underneath cars. The appearance of cars depends on the type of the car, its colour and the illumination. There are however certain features which are independent of these characteristics (Fig. 6). The tyres, the lower part of the rear bumper and the shadow underneath the car are reliable features which appear in the image as dark regions and are widely independent of both the illumination and the cars brand and colour. In order to search for this pattern of dark area, a binary image is calculated from the ROI. The threshold is calculated from the histogram of the ROI. 10% of the darkest pixels are selected for further processing steps (Fig. 7). 10% Figure 7: Image in the ROI, histogram, adaptive threshold and binary. Possible shapes and sizes of the template s appearances in the image domain are motivated by the possible width of passenger cars as well as the real sizes of their wheels. The templates are scaled with respect to the distance of the vehicle as measured by the Laserscanner. 4

5 (a) (b) Figure 8: Wheel patterns with different tyre sizes (a) and of different width (b). The wheel pattern is searched for in every row of the ROI taking minimal and maximal possible parameters for the width of cars and their wheels into account (Fig. 8). Every match is taken as a hypothesis. In Fig. 9 (a) the green horizontal lines show the regions a matching wheel template was found. (a) (b) Figure 9: Matched wheel patterns (green), bumper patterns (red) and shadow patterns (blue). Then the ROI is searched for shaded regions which are above and below the hypothesised wheel patterns and stem from either the dark areas of the lower part of the bumper or the shadow underneath the car (Fig. 9 (b)). For all hypothesised combinations a probability is calculated. If the probability of the best hypothesis exceeds a chosen threshold it is taken as a match for the recognition of the wheels. From the position and width of the wheel pattern in the image domain together with the distance measured by the Laserscanner, a very accurate position and width of the object in the 3d ego-vehicles co-ordinate system can be calculated (Fig. 10). Figure 10: Resulting position and width measurements. PATTERN MATCHING FOR TRUCKS Trucks do not exhibit the same common patterns as cars as can be seen in Fig. 11. Even the often used symmetry operator performs poorly as there are often highly unsymmetrical writings or drawings and illuminations on the rear of lorries. Figure 11: Trucks exhibiting strong vertical edges. 5

6 The rear of most trucks is however characterised by strong vertical edges. The strongest edges are selected by analysing the sums of the edge magnitudes of each column in the ROI. Figure 12: Detection of strong vertical edges and the lower edge of the rear. The edges delimiting the rear of the truck are selected from the hypothesis by incorporating a priori knowledge about the viewing angle and the edge measurements of the Laserscanner (Fig. 12). The lower edge of the rear is determined by analysing the shadow underneath the truck in the binary image of the 10% darkest pixels in ROI. KORRELATION TRACKING In order to stabilise the estimates a feature tracking approach is applied to the vehicle in the ROI. Additionally this tracking approach delivers an estimate for the lateral velocity if scaled with the distance measured by the Laserscanner. LANE RECOGNITION The lane recognition based on image sequences aims in our approach at a determination of the ego vehicle in the lane by the means of lateral offset, yaw angle and lane width. Taking the speed of the ego vehicle into account, the time of lane departure can be predicted. Additionally, the linear extrapolation of the estimated lane can be used by the high-level fusion systems for a lane assignment of tracked preceding vehicles. Figure 13: Lane recognition based on lane markings. The lane recognition is based on multiple features which are bright patches of a certain size and strong edges (Fig. 13). The gradient direction of the edges is considered as well. This enables a precise and robust extraction of the lane markings. 6

7 RESULTS A major problem of the first generation of radar based ACC systems is the inaccuracy of the estimation of the lateral offset of preceding vehicles. Therefore lane changes of objects in front of the ego-vehicle are detected only when the vehicle is already completely on the target lane. Therefore in case of an object cutting into the ego-vehicles lane, the ACC system often reacts too late, so that the user is forced to take over. Figure 14: Distance x, the width and the lateral offset y of a preceding car. The proposed fusion was tested concerning the accuracy of the estimation of the lateral offset. Figure 14 shows the distance x of a preceding car over time. The car was detected in every instance at a distance range of between 50 m and slightly above 100 m. The second plot shows the width of the car estimated by the Laserscanner compared to the fused estimate and the third plot shows the estimated lateral offset y. No tracking was applied in this case, every parameter was derived from a single scan and image. The estimates of width and lateral offset of the Laserscanner exhibit a certain discretisation which originates from the angular resolution. The image processing algorithms are able to refine both estimates of width and lateral offset. CONCLUSION A sensor fusion system was presented which combines the object tracking of a Laserscanner and of a video system as well as lane recognition, performed by an image processing unit. The environment recognition in the near field (< 50 m) is mainly based on the very accurate Laserscanners object tracking which covers a wide angular field of view. In the frontal far 7

8 field the lateral offset estimate is refined by an image processing module which searches for illumination and vehicle appearance independent features. The fused estimate exhibits a high precision which is necessary for early lane change detection of preceding vehicles. The environment description is application independent and can be used by multiple automotive active safety and comfort applications. REFERENCES 1. Fuerstenberg, K.; Baraud, P.; Caporaletti G.; Citelli, S.; Eitan, Z.; Lages, U.; Lavergne, C.: Development of a Pre-Chrash sensorial system the CHAMELEON Project, in VDI Berichte 1653: Fahrzeugkonzepte für das 2. Jahrhundert Automobiltechnik 2001, Wolfsburg, Germany, 2001, pp Langheim, J; Buchanan, A.J.; Lages, U.; Wahl, M.: CARSENSE New environment sensing for advanced driver assistance systems, in Proceedings of the IEEE Intelligent Vehicle Symposium 2001, Tokyo, Japan, 2001, pp Stiller, C.; Hipp, J.; Rössig, C.; Ewald, A.: Multisensor obstacle detection and tracking, Image and Vision Computing, vol. 18, pp , Gruyer, D.; Royere, C.; Berge-Cherfaoui, V.: Credibilist multi-sensor fusion for the mapping of dynamic environment, in Fusion 2000, Paris, France, July Vukotich, A. and Kirchner, A., Sensor fusion for driver-assistance-systems, in Elektronik im Kraftfahrzeug, Baden-Baden, Germany, Dietmayer, K.C.J., Kirchner, A.; Kaempchen, N.: Fusionsarchitekturen zur Umfeldwahrnehmung für zukünftige Fahrerassistenzsysteme, in Fahrerassistenzsysteme, Springer Verlag, Germany, 2004, accepted 7. Kaempchen, N.; Dietmayer, K.C.J.: Data synchronization strategies for multi-sensor fusion, in Proceedings of the 10 th World Congress on Intelligent Transport Systems and Services, Madrid, Spain, 2003, No. T Fuerstenberg, K.Ch.; Dietmayer, K.C.J; Eisenlauer, S.; Willhoeft, V.: Multilayer Laserscanner for robust Object Tracking and Classification in Urban Traffic Scenes. Proceedings of ITS 2002, 9th World Congress on Intelligent Transport Systems, October 2002, ITS 2002 Chicago, Paper Fuerstenberg, K.Ch.; Dietmayer, K.C.J: Object Tracking and Classification for Multiple Active Safety and Comfort Applications using a Multilayer Laserscanner, in Proceedings of the IEEE Intelligent Vehicles Symposium 2004, Parma, Italy, Fuerstenberg, K.Ch.; Dietmayer, K.C.J.; Willhoeft, V.: Pedestrian Recognition in Urban Traffic using a vehicle based Multilayer Laserscanner, in Proceedings of the IEEE Intelligent Vehicles Symposium 2002, IV 2002 Versailles, Paper IV-80. 8