Vision In and Out of Vehicles: Integrated Driver and Road Scene Monitoring

Transcription

1 Vision In and Out of Vehicles: Integrated Driver and Road Scene Monitoring Nicholas Apostoloff and Alexander Zelinsky The Australian National University, Robotic Systems Laboratory, Research School of Information Sciences and Engineering, Canberra ACT 0200, Australia Abstract million people die in road crashes around the world each year. It is estimated that up to 30% of these fatalities are caused by fatigue and inattention. This paper presents preliminary results of an Intelligent Transport System (ITS) project that has fused visual lane tracking and driver monitoring technologies in the first step to closing the loop between vision inside and outside the vehicle. Experimental results of the active stereo-vision lane tracking system will be discussed focusing on the particle filter and cue fusion technology used. The results from the integration of the lane tracker and the driver monitoring system are presented with an analysis of the driver s visual behavior in several different scenarios. 1 Introduction It is estimated that around 30% of fatal car crashes can be attributed to driver inattention and fatigue [4] [11]. Numerous studies have been performed to analyse signs of driver fatigue through the measurement of the visual demand on the driver. This is often through frame-by-frame human-rater measurement or infrared cornealreflection technologies. While these studies produce valuable results, they are often time consuming and too unreliable for many research purposes [11]. An ITS project has recently been initiated at The Australian National University (ANU) which is focused on autonomous driver monitoring and autonomous vehicle control to aid the driver [3]. A major aim of this project is the development of a system of cooperating internal and external vehicle sensors for research into the visual behavior of the driver. This paper presents the first results from this study where a lane tracker was developed using particle filtering and visual cue fusion technology. This was integrated with a driver monitoring system (facelab [11]) to investigate the visual behavior of driver in a number of common driving scenarios. There are many benefits associated with the development of reliable lane tracking systems. Traditional uses include autonomous control of vehicles, Adaptive Cruise Control (ACC) and Lane Departure Warning (LDW) systems. In combination with the driver monitoring software we can detect when a driver is looking at and has their attention focused on the road. This is the first step in characterising the environment surrounding the driver for a complete analysis of what holds the drivers attention. There are numerous uses for this technology including: LDW systems and the reduction of false positives.

2 Vision In and Out of Vehicles: Integrated Driver and Road Scene Monitoring 609 Obstacle detection and warning systems. Automated driver visuo-attentional analysis systems. Fatigue and inattention warning systems 2 The Experimental Platform The testbed vehicle is a 1999 Toyota Landcruiser 4WD (see Fig. 1). Vision is the main form of sensing used on the vehicle, which has two different vision platforms installed (see Fig. 2). A passive set of cameras is mounted on the dashboard facing the driver and are part of the facelab system for driver monitoring. CeDAR, an active vision head designed at ANU [10], carries 4 cameras - one pair used for stereo vision in the near-field, and one pair for far-field stereo experiments and for mid-field to far-field scene coverage. Various other sensors (which were not used in this experiment) have been fitted to the vehicle including a Global Positioning System (GPS), Inertial Navigation Sensor (INS), and a Pulse FM Radar. Fig. 1. The testbed vehicle. 3 System Architecture The system architecture consists of the lane tracker, the driver monitor (facelab) and the high level data correlation systems.both the lane tracker and facelab run independently on different CPUs while the high level data correlation system produces the focus of attention of the driver with respect to the road. The driver s focus of attention is determined to be one of the regions defined in Fig. 3. The regions are divided into 3 different ranges - the near field, the far field and a horizon region. The horizon region is used to collect eye gaze data that is roughly parallel to the ground plane and that doesn t intersect with the other 2 ranges.

3 610 N. Apostoloff and A. Zelinsky Fig. 2. The vision platforms in the vehicle. Top: CeDAR active vision head. Right (above the steering wheel): facelab passive stereo cameras. Fig. 3. The regions of interest for obtaining the focus of attention of the driver.

4 Vision In and Out of Vehicles: Integrated Driver and Road Scene Monitoring 611 Each of these ranges is split into either 3 or 4 regions depending on the test being performed. On a two lane road 4 regions are defined - a region to the left of the road, the lane detected by the lane tracker, the right lane containing on-coming traffic and a region to the right of the road. If the test is on a one-way road such as a highway the right lane containing on-coming traffic is not used. The focus of attention region is obtained by calculating the intercept of the eye gaze vector of the driver with the regions defined in Fig. 3 which are found by the lane tracker. 4 Lane Tracking Despite many impressive results from lane trackers in the past [1] [2] [9] [12], it is clear that no single cue can perform reliably in all situations. The lane tracking system presented here dynamically allocates computational resources over a suite of cues to robustly track the road in a variety of situations. Bayesian theory is used to fuse the cues while a scheduler intelligently allocates computational resources to the individual cues based on their performance. A particle filter [5] is used to control hypothesis generation of the lane location. For a detailed description of the tracking system see [6]. Each cue is specifically developed to work independently from the other cues and are customised to perform well under different situations (i.e. edge based lane marker tracking, colour based road tracking, etc.). Cues are allocated CPU time based on their performance and the time they require. Two metrics, the Kullback-Leibler divergence and the Uncertainty Deviation [8], are used to evaluate the performance of each cue with respect to the fused result and the individual result of the cue respectively. Additionally, the framework of the lane tracker was designed to allow the cues to run at different frequencies enabling slow running (but valuable) cues to run in the background. A dual phase particle filter system is proposed to reduce the search space for the lane tracker. The first particle filter searches for the the road width, the lateral offset of the vehicle from the centerline of the road and the yaw of the vehicle with respect to the centerline of the road. The second particle filter captures the horizontal and vertical road curvature in the mid to far-field ranges using the state information captured by the first particle filter. In the work reported in this paper, the second phase particle filter of the system was not used and no road curvature was calculated. The state space for the particle filter is the lateral offset of the vehicle relative to the skeletal line of the road, the yaw of the vehicle with respect to the skeletal line and the road width (see Fig. 4). 4.1 Cues for Robust Lane Tracking The cues chosen for this experiment were designed to be simple and efficient while being suited to a different set of road scenarios. Individually, each of the cues would perform poorly, but when they are combined through the cue fusion process they

5 612 N. Apostoloff and A. Zelinsky produce a robust solution to lane tracking. Each cue listed below uses the road model shown in Fig. 4 to process the probability of each hypothesis from the particle filter. Fig. 4. Road model used for the first phase particle filter. The dark shaded region is used as the non road boundary in the colour cues while the light shaded region is the road region. Note that the figure is exaggerated for clarity. 1. Lane Marker Cue is designed for roads that have lane markings. A modified ternary correlation 1 to preprocess an intensity image of the road and the cue returns the average value of the pixels along the hypothesised road edges. 2. Road Edge Cue is suited to roads with lane markings or roads with defined edges. It uses a preprocessed edge map and returns the average value of the pixels along the hypothesised road edges. 3. Road Colour Cue is useful for any roads that have a different colour than their surroundings (both unmarked and marked roads). It returns the average pixel value in the hypothesised road region from a colour probability map that is dynamically generated each iteration using the estimated road parameters from the previous iteration. 4. Non Road Colour Boundary Cue is the opposite to the Road Colour Cue and returns the average road colour probability of the non-road regions. 5. Road Width Cue is particularly useful on multi-lane roads where is is possible for the other cues to see two or more lanes as one. It returns a value from a Gaussian function centered at a desired road width given the hypothesised road width. The desired road width used in this cue was 3.61m which was empirically determined from previous lane tracking experiments to be the average road width. 6. Elastic Lane Cue is used to move particles towards the lane that the vehicle is in. It returns 1 if the lateral offset of the vehicle is less than half of the road width and 0.5 otherwise. 1 The 1D ternary correlation function is modified to be two sided with a step from -1 to 1 and back to -1.

6 Vision In and Out of Vehicles: Integrated Driver and Road Scene Monitoring facelab facelab [11] is a driver monitoring system commercialised by Seeing Machines [7] based on research and development work between ANU and Volvo Technological Development Corporation. It uses a passive stereo pair of cameras mounted on the dashboard of the vehicle to capture 60Hz video images of the driver s head. These images are processed in real-time to determine the 3D position of matching features on the drivers face. The features are then used to calculate the 3D pose of the persons face ( mm, ) as well as the eye gaze direction ( ), blink rates and eye closure. 6 Integration of the Lane Tracker and facelab Combining the gaze direction captured by facelab and the parameterised lane model determined by the lane tracker, we can track the visual behavior of the driver relative to the road. Calculating the intercept of the drivers gaze vector with the regions of interest shown in Fig. 3 determines the driver s focus of attention. Additionally, the visual scan patterns of the driver can be recorded with emphasis on fixation points and saccade movements to different regions in the scene. 7 Experimental Results Results from the lane tracker and the integrated system are presented below. 7.1 Lane Tracking The lane tracker was tested in several different scenarios including: Highway driving with light traffic. Outer city driving with high curvature roads. Inner city driving with moderate levels of traffic. Figure 5 shows the output of the lane tracker in the above scenarios using the 6 different cues. The lane tracker was found to work robustly and solved the problems typically associated with lane tracking including: Dramatic lighting changes (see A in Fig. 5). Changes in road colour (see D-F in Fig. 5). Shadows across the road (see C,G-H in Fig. 5). Roads with miscellaneous lines that are not lane markings (see I in Fig. 5). Lane markings that disappear and reappear.

7 614 N. Apostoloff and A. Zelinsky Fig. 5. Results from the lane tracker. The boxes indicate the ends of the lines that define the lane being tracked. This can be attributed to the combination of particle filtering and cue fusion. Because of the particle filter, cues only have to validate a hypothesis and do not have to search for the road. This indirectly incorporates a number of a priori constraints into the system (such as road edges meeting at the vanishing point in image space and the edges lying in the road plane) which assist it in its detection task. Cue fusion was found to dramatically increase the robustness of the solution due to the variety of conditions the cues were suited to. The final configuration of cues in the system is a direct result of earlier experiments uncovering certain conditions in which the cues would fail. 7.2 Driver Monitoring and Lane Tracking The integrated driver monitoring and lane tracking system was tested both on the highway and on outer city high curvature roads. A comparison of the drivers focus of attention statistics is given in Fig. 7 while a comparison of the yaw of the vehicle (a good indicator in this case of the degree of curvature) and the drivers gaze yaw is given in Fig. 6. Figure 6 gives a good indication of the correlation between the driver monitoring system and the lane tracking system. The yaw of the driver s gaze closely follows the

8 Vision In and Out of Vehicles: Integrated Driver and Road Scene Monitoring 615 Fig. 6. Comparison between the vehicle yaw and driver s gaze yaw between two tests along the same piece of road but in different directions. The dashed line is the yaw of the vehicle while the solid line is the yaw of the driver s gaze. Note that the large peaks are generated by the driver through saccade movements to the left and right windows of the car. vehicle yaw (a good indication of the curvature of the road) in both graphs showing that the driver is following the road curvature with his gaze. Fig. 7. Comparison between the driver s focus of attention for 3 different tests along a high curvature road and the highway. In the 4 pie charts to the left, the top and bottom rows are the data from the two different tests along the same high curvature road while column 1 is for the South-West route and column 2 is the North-East route. The pie chart to the right is for the highway test

9 616 N. Apostoloff and A. Zelinsky Figure 7 shows the percentage of time the driver focused his attention on the different regions of interest in the scene. The Other field is representative of times which the drivers gaze did not intercept any of the fields of interest. These graphs clearly show the repeatability in the experiments as well as a certain characteristic of the road which these tests were carried out on. The road used for the 4 pie charts to the left has a predominantly right curvature in the North-East direction (column 2 of Fig. 7) shown by the greater proportion of time the driver spent focusing on the right lane as well as the right side of the road. Note that the Right Road regions in these pie charts indicates the lane that the on-coming traffic occupies. The amount of time the driver spent looking at the highway (right pie chart of Fig. 7) stayed approximately the same as with the high curvature roads even though it was actually a smaller region (no right lane). The time spent focusing on the left and right sides of the highway was approximately equal which is to be expected considering the low curvature of the road. 8 Conclusions An integrated visual driver monitoring and lane tracking system has been presented that was successfully used to close the loop between vision inside and outside the vehicle. It was found that the lane tracking system benefited greatly from the cue fusion and particle filtering technologies used and was shown to perform robustly in a number of situations. A strong correlation was found to exist between the eye gaze direction of the driver and the curvature of the road, while the repeatability of the experiments was high. The applications of such a system are numerous, possibly the most important being driver warning systems of the future. 9 Acknowledgments We would like to thank the staff of Seeing Machines for their help with facelab, particularly David Liebowitz who was kind enough to spend time helping with the analysis of the data collected. We would also like to acknowledge Gareth Loy and Luke Fletcher for their help with the tracking system used in the lane tracker. References 1. Parag H. Batavia, Dean A. Pomerleau, and Charles E. Thorpe. Overtaking vehicle detection using implicit optical flow. In Proc. IEEE Transport Systems Conference, Ernst D. Dickmanns. An expectation-based, multi-focal, saccadic (ems) vision system for vehicle guidance. In Proc. International Symposium on Robotics and Research, Salt Lake City, Utah, October Luke Fletcher, Nicholas Apostoloff, Jason Chen, and Alexander Zelinsky. Computer vision for vehicle monitoring and control. In Proc. Australian Conference on Robotics and Automation.

10 Vision In and Out of Vehicles: Integrated Driver and Road Scene Monitoring HoRSCoCTA. Inquiry into Managing Fatigue in Transport. The Parliament of the Commonwealth of Australia, M. Isard and A. Blake. Condensation conditional density propagation for visual tracking. International Journal of Computer Vision 92(1), Gareth Loy, Luke Fletcher, Nicholas Apostoloff, and Alexander Zelinsky. An adaptive fusion architecture for target tracking. In Proc. The 5th International Conference on Automatic Face and Gesture Recognition. 7. Seeing Machines. Facelab face and eye tracking system A. Soto and P.Khosla. Probabilistic adaptive agent based system for dynamic state estimation using multiple visual cues. In Proceedings of the International Symposium of Robotics Research (ISRR). 9. A. Suzuki, N. Yasui, N. Nakano, and M. Kaneko. Lane recognition system for guiding of autonomous vehicle. In Proceedings of the Intelligent Vehicles Symposium, 1992, pages , Harley Truong, Samir Abdallah, Sebastien Rougeaux, and Alexander Zelinsky. A novel mechanism for stereo active vision. In Proc. Australian Conference on Robotics and Automation. 11. Trent Victor, Olle Blomberg, and Alexander Zelinsky. Automating driver visual behaviour measurement. In Proc. Vision in Vehicles Todd Williamson and Charles Thorpe. A trinocular stereo system for highway obstacle detection. In Proc. International Conference on Robotics and Automation (ICRA99), 1999.