A Real-Time Driver Fatigue Detection System Based on Eye Tracking and Dynamic Template Matching

Transcription

1 Tamkang Journal of Science and Engineering, Vol. 11, No. 1, pp (28) 65 A Real-Time Driver Fatigue Detection System Based on Eye Tracking and Dynamic Template Matching Wen-Bing Horng* and Chih-Yuan Chen Department of Computer Science and Information Engineering, Tamkang University, Tamsui, Taiwan 251, R.O.C. Abstract This paper presents a vision-based real-time driver fatigue detection system for driving safety. Based on skin colors, the driver s face is located from a color video captured in a car. Then, edge detection is employed to locate the regions of the driver s eyes, which are used as the templates for eye tracking in subsequent frames. Finally, the tracked eyes images are used for fatigue detection in order to generate warning alarms for driving safety. The proposed system was tested on a Pentium M 1.4G notebook with 512 MB RAM. The experimental results seemed quite encouraging and promising. The system could reach more than 30 frames per second for eye tracking, and the average correct rate for eye location and tracking could achieve 96.0% on five test videos. Though the average precision rate of fatigue detection was 89.3%, the correct detection rate could achieve on the test videos. Key Words: Intelligent Transportation System, Driver Fatigue Detection, Eye Detection, Face Detection, Template Matching 1. Introduction *Corresponding author. [email protected] As the standard of living has improved in the past few decades, many families possess their own transportation vehicles. Therefore, people often drive their cars for business or trip. However, long distance driving usually makes drivers exhausted. Thus, the driver fatigue problem has been an important factor of traffic accidents. In order to decrease traffic accidents, many countries have begun to pay attention to the driving safety problem. Some psychologists [1,2] have investigated the drivers mental states relating to driving safety. Furthermore, many other researchers [3 6,10 19] have proposed various auxiliary mechanisms to improve driving safety. Cho et al. [3] designed a special car seat to improve the alertness of the driver by adjusting the temperature and humidity of the driver s thighs and hips to reduce the degree of drowsiness. Chieh et al. [4] monitored the grip force change of the driver on the steering wheel to detect fatigue. Wilson and Bracewell [5] analyzed EEG (Electroencephalogram) recordings by neural networks to monitor alertness. Similar research was also performed by Vuckovic et al. [6] to detect drowsiness. However, these methods need some special sensors adhering to the human body, which seems impractical during driving. Other researchers based on image processing [7] and computer vision [8,9] techniques to detect driver s fatigue from images captured by a camera to improve driving safety. Some of these researchers [10 14] utilized infrared CCD cameras to locate the driver s face and eyes positions easily. However, infrared cameras are very expensive, which makes them of limited usage. In addition, the influence of infrared light on human eyes is not known yet. To be practical, some other researchers [15 19] employed ordinary CCD cameras to perform fatigue detection. However, most of those driver fatigue detection methods suffer from the illumination change problem. In addition, they might not be suitable for real-time applications due to their complicated computations. In this paper, we propose a vision-based real-time driver fatigue detection system based on ordinary CCD cameras to cope with the above drawbacks for practical

2 66 Wen-Bing Horng and Chih-Yuan Chen usage to improve driving safety. For the illumination change problem, we adopt the HSI color model to represent face skin, not only because the model detaches the intensity from the hue of a color, but also because face colors have fixed distribution range on the hue component. Since eyes have the nature of sophisticated edges, we use the Sobel edge operator and the projection technique to locate eyes positions. The obtained eyes images are then used as the templates for eye tracking in subsequent frames, instead of using the static eye images in the database as used in [16], to enhance the reality and reliability of the templates to increase accuracy of eye tracking. Finally, the tracked eyes images are used to determine whether the eyes are open or closed, which is the basis of driver fatigue detection used in this paper. The rest of the paper is organized as follows. In Section 2, we briefly review the related driver fatigue detection methods based on ordinary CCD cameras. In Section 3, we present a new vision-based real-time driver fatigue detection system, including face detection, eye detection, eye tracking, and fatigue detection. The experimental results of the system are analyzed in Section 4. Finally, the paper is concluded in the last section. 2. Related Driver Fatigue Detection Methods Based on Ordinary CCDs In this section, we briefly review the related visionbased driver fatigue detection methods using ordinary CCD cameras. In 1997, Eriksson and Papanikolopoulos [15] proposed a method to detect driver fatigue from ordinary videos. In this method, the symmetric property of the face is used to detect the facial area of the driver on an image. Then, pixel difference is performed to find the edges on the facial region, and horizontal projection is used to locate the vertical position of the eyes. Finally, a concentric circle template is used to locate and track the exact eyes positions. Although face symmetry is an obvious feature for an upright face, it usually fails to locate the correct face position when the face tilts, rotates, or is shadowed. Instead of using symmetric central line method, Singh and Papanikolopoulos (1999) [16] later proposed another method to monitor driver fatigue. In this method, the Gaussian distribution of skin colors in the RGB color model is used to distinguish skin and non-skin pixels for face detection. Then, the Sobel vertical edge operator is used for eye detection on the driver s face. Besides, a set of eye images as templates in a database is used for eye detection and tracking. Although the Gaussian distribution of skin colors based on the RGB color model is used to predict skin quite well, the method cannot get rid of the factor of illumination changes. In addition, the eye images in the database as templates may be quite different from drivers eyes, which will reduce the accuracy for locating eye positions. In 23, Wang et al. [17] proposed a method for monitoring driver fatigue behavior. Like [16], the Gaussian distribution of skin colors in the RGB color model is used to locate a face region. Then, the face is binarized with a fixed threshold to locate the eyes positions with some geometric restrictions and to track the eyes by using the Hausdorff distance template matching. Finally, the features of the eyes are analyzed by the Gabor filters, and the states of driver s eyes are classified by neural networks. Since the eyes are detected with a fixed threshold, this system might not resist illumination changes. In 24, Wang et al. [18] proposed another method for driver fatigue detection by monitoring mouth movement. Similarly, the Gaussian distribution of skin colors is used to locate a face region. Then, a Fisher linear classifier is used to detect the mouth of the driver, and a Kalman filter is used to track the mouth movement. However, the mouth movement may not be a reliable feature for detecting drowsiness because the mouth shapes will also change when talking, eating, laughing, and so on. It is not easy to classify the mouth movement into a correct drowsy state. In 25, Dong and Wu [19] also presented a driver fatigue detection algorithm based on the distance of eyelids. Instead of using the RGB color model, this method utilizes the Gaussian distribution of skin colors based on the YCbCr color model to detect the face region. Then, horizontal projection is used to locate eyes positions, and the gray scale correlation is used for eye tracking. Finally, the distance of eyelids is calculated to detect fatigue. Although the method can resist the influence of illumination changes, it is not easy to distinguish the distance of eyelids of closed eyes or open eyes by small difference of eyelids distance, especially when persons have long eyelash. However, most of the above vision-based driver fa-

3 A Real-Time Driver Fatigue Detection System Based on Eye Tracking and Dynamic Template Matching 67 tigue detection methods using ordinary CCD cameras cannot resolve the illumination change problem. Besides, they might not be practical for real-time applications due to their complicated and intensive computations. 3. The Proposed Driver Fatigue Detection System The proposed system uses an ordinary color CCD camera mounted on the dash board of a car to capture the images of the driver for driver fatigue detection. The flow chart of the proposed fatigue detection system is depicted in Figure 1. The first image is used for initial face location and eye detection. If any one of these detection procedures fails, then go to the next frame and restart the above detection processes. Otherwise, the current eye images, as the dynamic templates, are used for eye tracking on subsequent images. If eye tracking fails, the processes of face location and eye detection restart on the present image. These procedures continue until there are no more frames. The detailed steps are described in the following subsections. 3.1 Face Detection Digital images usually adopt the RGB color space to represent colors. However, any color in the RGB space not only displays its hue but also contains its brightness. For two colors with the same hue but different intensities, they would be viewed as two different colors by the human visual system. In order to accurately distinguish skin and non-skin pixels so that they will not be affected by shadows or light changes, the brightness factor must be excluded from colors. Since in the HSI color model, hue is independent of brightness, this model is well suited for distinguishing skin and non-skin colors no matter whether the face is shadowed or not. Thus, in this paper it is used for face detection. The HSI color model has three components: hue, saturation, and intensity (or brightness). The detailed method for converting a color between the RGB space and the HSI model refers to [7]. The following Equations (1) to (3) show the results of converting a color from the RGB model to the HSI model. For any color with three elements, R, G, and B, normalized in the range between 0 and 1, the formulas for converting to its corresponding H, S, and I components are given as follows. Figure 1. Flow chart of the proposed driver fatigue detection system. (1) (2) (3) When (B/I)>(G/I), H =360 H. From the face images of the test videos, it is observed that the H values of skin colors fall in the range between 10 and 45. Using this condition, it is easy to separate the face region, as shown in Figure 2(b). By performing the vertical projection on skin pixels, the right

4 68 Wen-Bing Horng and Chih-Yuan Chen and left boundaries can be found when the projecting values are greater than some threshold. In order to eliminate the region of the ears on the both sides of the face, the value of threshold is defined as 50 pixels through experiments. Similarly, by performing the horizontal projection on skin pixels within the left and right boundaries, the top and bottom boundaries can also be found, as shown in Figure 2(c). According to the normal position of the eyes in a face, it is reasonable to assume that the possible location of eyes will be in the block of the upper two fifths of the face region, as shown in Figure 2(d). Figures 3(a) and 3(c) illustrate that even under the case of uneven illumination on the faces during driving, the proposed method based on the HSI color model can still correctly segment the face regions. 3.2 Eye Detection After locating the eye region, the original color eye region is converted into gray scale by Equation (4) as follows (refer to [7]). Y = 0.299R G B (4) where Y is the gray value of a color (R, G, B). Then, the Sobel edge operator is used to compute the edge magnitude in the eye region to find the vertical position of the eyes. In order to save computation time, an approximate calculation of the edge magnitude, G(x, y), is computed in Equation (5) as follows. G ( x, y)» S x + S y (5) where Sx and Sy are the horizontal and vertical gradient values obtained, respectively, from the Sobel horizontal and vertical edge operators, and Sx represents the absolute value of Sx. For detailed calculation, refer to [7]. After computing the edge magnitude of each pixel in the eye region, an edge map, E(x, y), is obtained by defining a pixel (x, y) being an edge pixel if its edge magnitude G(x, y) is greater than some threshold, T, as defined in Equation (6). (6) where black and white stand for the black pixel value and the white pixel value, respectively. The threshold T is set to 150 via several tests. The left part of Figure 4(b) illustrates the edge map of Figure 4(a). Performing horizontal projection on the edge map, the exact vertical position of the eyes can be located at the peak, as shown in the right part of Figure 4(b). Some- Figure 2. Result of face detection. Figure 3. Results of face detection under uneven illumination on the face during driving.

5 A Real-Time Driver Fatigue Detection System Based on Eye Tracking and Dynamic Template Matching 69 Figure 4. Results of edge detection and horizontal projection. times, it may need to smooth horizontal projections for locating peaks. After locating the vertical position of the eyes, as shown in Figure 4(b), the left and right eye positions can be located by finding the largest connected components from the center of the eye region along the horizontal line of the vertical position of the eyes. In order to avoid marking noise, the total number of pixels in a connected component must exceed some threshold so that it can be regarded as a reasonable eye. After finding the connected component of an eye, a bounding box for the connected component on the original image is used to enclose the eye image, which is used as the template for eye tracking in the subsequent frames. Figure 5 illustrates the bounding boxes of detected eye positions. 3.3 Eye Tracking After finding the eye templates, they are used for driver s eye tracking by template matching. The searching area for matching is the original bounding box of an eye by expanding some reasonable number, t, of pixels, say t = 10, in each of four directions. Consider an eye template g(x, y) locating in the position (a, b) of the face image f(x, y). Equation (7) is used for the eye template matching. (7) where w and h are the width and height of the eye template g(x, y), and p and q are offsets of the x-axis and y-axis in which (a t) p (a + t)and(b t) q (b + t). If M(p*, q*) is the minimum value within the search area, the point (p*, q*) is defined as the most matching position of the eye. Thus, (a, b) is update to be the new position (p*, q*). The newly updated eye templates must satisfy the following two constrains to confirm successful eye tracking: (i) The ratio of width to height of each new eye template is between 1 and 4. Figure 5. Results of eye detection with bounding boxes. (ii) Since the movement of the driver s head will not very big between each pair of consecutire frames (1/30 second), the distance between the newly obtained templates and the previous templates is less than 10 pixels. Otherwise, the eye tracking will be regarded as failure and the system will restart the face and eye location procedures. 3.4 Fatigue Detection At this stage, the colors of the eyeballs in the eye templates are used directly for fatigue detection. Since the property that the eyeball colors are much darker is a quite stable feature, the eye templates are inverted (negated) and then converted to the HSI color model. The original darker eyeballs become brighter ones in the inverted image. According to the observation, the saturation values of eyeball pixels normally fall in the range between 0. and This observation is used to distinguish whether a pixel in an eye template is viewed as an eyeball pixel, as shown in Figure 6. When the eyes are open, there are some eyeball pixels, as shown in Figures 6(a) and 6(b). When the eyes are closed, there are no eyeball pixels, as shown in Figures 6(c) and 6(d). By checking the eyeball pixels, it is easy to detect whether the eyes are open or closed. According to the experimental results as shown in Figure 7, the duration of a normal eye blink (the small peak) is usually less than 4 frames at the capture rate of 30 frames per second. If the duration for an eye-close is more than 5 frames, the distance of the car

6 70 Wen-Bing Horng and Chih-Yuan Chen Figure 6. Eyeball detection of open eye and closed eye. Figure 7. Analysis of eye blinks. will move more than 5.5 meters at the speed of 1 kilometers per hour in a high way. It will be very dangerous for the driver to ignore the status of the road when driving under such a condition. For this reason, in our system it is defined that when the eyes close over 5 consecutive frames, then the driver is regarded as dozing off. Of course, this parameter can be adjusted by users according to their needs. 4. Experimental Results The proposed driver fatigue detection system used a Sony PC115 color DV camera to capture driver s images. The system was tested under the environment of Pentium M 1.4G notebook with 512 MB RAM. The format of input video is true color. After starting the system, it took less than 40 milliseconds for initial face location and eye detection. Once the eye templates were found, eye tracking could achieve more than 30 frames per second. Recall that in this system, if the driver closes his/her eyes over 5 consecutive frames, then the driver is regarded as dozing off. Table 1 lists the results of eye tracking from five test videos, as shown in Figure 8. The first four videos were captured under different illumination conditions with different people and backgrounds. The last video, video 5, was captured when driving around a parking lot at nightfall with large illumination change. The field Total Frames means the total number of frames in each video. Tracking Failure is the count of eye tracking failure. Correct Rate of eye tracking is defined as in Equation (8), which is the ratio of (Total Frames Tracking Failure)toTotal Frames. (8) From Table 1, we can see that the correct rate of eye tracking is higher than 98.3% in the first four videos, and it still can reach 80.0% in the very strict environment of video 5. The average correct rate achieves 96.0% in the proposed system on these test videos. Table 2 shows the result of driver fatigue detection on the five test videos. The field Close Eyes represents the number eye closing and then opening of the driver in the video. Real Dozing is the number of dozing off determined by humans. Generate Warning is the number of fatigue detection that generates a warning alarm. False Table 1. Result of eye tracking Video1 Video2 Video3 Video4 Video5 Total Total Frames Tracking Failure Correct Rate 99.7% 99.6% 98.3% 98.4% 80.0% 96.0%

7 A Real-Time Driver Fatigue Detection System Based on Eye Tracking and Dynamic Template Matching 71 Figure 8. Snapshots during tracking with different people under different illumination. Table 2. Result of fatigue detection Close Eyes Real Dozing Generate Warning False Positive False Negative Correct Warning Correct Rate Precision Rate Video 1 Video 2 Video 3 Video 4 Video 5 Total % % Positive is the number of wrongly generated warning alarms. False Negative is the number of not detected fatigue. Correct Warning is the number of correctly detected fatigue, which is equal to (Generate Warning False Positive False Negative). Correct Rate of fatigue detection is defined as in Equation (9). posed system could still correctly detect and track the driver s eyes. In spite of this kind of interference and illumination changes, the average precision rate for fatigue detection for all videos could still achieve 89.3% in the proposed system. 5. Conclusion (9) Precision Rate for fatigue detection is defined as in Equation (10). (10) As can be seen from Table 2, the system could correctly detect all fatigue states in all the test videos. In video 3, three mistakes made by the system are due to the interference of the driver s hand, as shown in Figure 8(c). Even for the uneven illumination on the face in a moving car, as shown in Figures 8(e) to 8(h), the pro- This paper has presented a vision-based real-time driver fatigue detection system for driving safety. The system uses an ordinary CCD camera to capture the driver s images, which makes the method more practical. Tested on a notebook of Pentium M 1.4G CPU with 512 MB RAM, the system could reach 96.0% of average correct rate of eye detection on five videos of various illumination conditions, and could achieve of correct rate of fatigue detection with precision rate 89.3%. The speed of this system could reach more than 30 frames per second for eye tracking, with each color frame of size In summary, the proposed driver fatigue de-

8 72 Wen-Bing Horng and Chih-Yuan Chen tection system could cope with the illumination change problem and could be suitable for real-time applications to improve driving safety. Acknowledgement The authors would like to thank the anonymous reviewers for their valuable comments. This research was partially supported by the National Science Council, Taiwan, ROC., under contract no. NSC E-2-1. References [1] Shinar, D., Psychology on the Road, John Wiley & Sons, Danvers, MA, U.S.A. (1979). [2] Recarte, M. A. and Nunes, L. M., Effects of Verbal and Spatial-Imagery Tasks on Eye Fixations While Driving, Journal of Experimental Psychology: Applied, Vol. 6, pp (20). [3] Cho, K. J., Roy, B., Mascaro, S. and Asada, H. H., A Vast DOF Robotic Car Seat Using SMA Actuators with a Matrix Drive System, Proc. IEEE Robotics and Automation, New Orleans, LA, U.S.A., Vol. 4, pp (24). [4] Chieh, T. C., Mustafa, M. M., Hussain, A., Zahedi, E. and Majlis, B. Y., Driver Fatigue Detection Using Steering Grip Force, Proc. IEEE Student Conference on Research and Development, Putrajaya, Malaysia, pp (23). [5] Wilson, B. J. and Bracewell, T. D., Alertness Monitor Using Neural Networks for EEG Analysis, Proc. IEEE Signal Processing Society Workshop on Neural Networks for Signal Processing, Sydney, Australia, Vol. 2, pp (20). [6] Vuckovic, A., Popovic, D. and Radivojevic, V., Artificial Neural Network for Detecting Drowsiness from EEG Recordings, Proc. IEEE Seminar on Neural Network Applications in Electrical Engineering, Belgrade, Yugoslavia, pp (22). [7] Gonzalez, R. C. and Woods, R. E., Digital Image Processing, Second Edition, Prentice Hall, Upper Saddle River, NJ, U.S.A. (22). [8] Davies, E. R., Machine Vision: Theory, Algorithms, Practicalities, Third Edition, Elsevier, Oxford, UK (25). [9] Forsyth, D. A. and Ponce, J., Computer Vision: A Modern Approach, Prentice Hall, Upper Saddle River, NJ, U.S.A. (23). [10] Gu, H., Ji, Q. and Zhu, Z., Active Facial Tracking for Fatigue Detection, Proc. IEEE Workshop on Applications of Computer Vision, Orlando, FL., U.S.A., pp (22). [11] Ji, Q., Zhu, Z. and Lan, P., Real-Time Nonintrusive Monitoring and Prediction of Driver Fatigue, IEEE Transactions on Vehicular Technology, Vol. 53, pp (24). [12] Zhu, Z. and Ji, Q., Real-Time and Non-intrusive Driver Fatigue Monitoring, Proc. IEEE Intelligent Transportation Systems, Washington, D.C., U.S.A., pp (24). [13] Gu, H. and Ji, Q., An Automated Face Reader for Fatigue Detection, Proc. IEEE Automatic Face and Gesture Recognition, Seoul, Korea, pp (24). [14] Ito, T., Mita, S., Kozuka, K., Nakano, T. and Yamamoto, S., Driver Blink Measurement by the Motion Picture Processing and its Application to Drowsiness Detection, Proc. IEEE Intelligent Transportation Systems, Singapore, pp (22). [15] Eriksson, M. and Papanikolopoulos, N. P., Eye- Tracking for Detection of Driver Fatigue, Proc. IEEE Intelligent Transportation Systems, Boston, MA, U.S.A., pp (1997). [16] Singh, S. and Papanikolopoulos, N. P., Monitoring Driver Fatigue Using Facial Analysis Techniques, Proc. IEEE Intelligent Transportation Systems, Tokyo, Japan, pp (1999). [17] Wang, R., Guo, K., Shi, S. and Chu, J., A Monitoring Method of Driver Fatigue Behavior Based on Machine Vision, Proc. IEEE Intelligent Vehicles Symposium, Columbus, Ohio, U.S.A., pp (23). [18] Wang, R., Guo, L., Tong, B., and Jin, L., Monitoring Mouth Movement for Driver Fatigue or Distraction with One Camera, Proc. IEEE Intelligent Transportation Systems, Washington, D.C., U.S.A., pp (24). [19] Dong, W. and Wu, X., Driver Fatigue Detection Based on the Distance of Eyelid, Proc. IEEE VLSI Design and Video Technology, Suzhou, China, pp (25). Manuscript Received: Dec. 29, 26 Accepted: Jul. 25, 27