Real-time Detection of Nodding and Head-shaking by Directly Detecting and Tracking the Between-Eyes

Transcription

1 Real-time Detection of Nodding and Head-shaking by Directly Detecting and Tracking the Between-Eyes Shinjiro Kawato and Jun Ohya ATR Media Integration and Communications Research Laboratories Seika-cho, Soraku-gun, Kyoto , JAPAN {skawato, Abstract Among head gestures, nodding and head-shaking are very common and used often. Thus the detection of such gestures is basic to a visual understanding of human responses. However it is difficult to detect them in real-time, because nodding and head-shaking are fairly small and fast head movements. In this paper, we propose an approach for detecting nodding and head-shaking in real time from a single color video stream by directly detecting and tracking a point between the eyes, or what we call the between-eyes. Along a circle of a certain radius centered at the between-eyes, the pixel value has two cycles of bright parts (forehead and nose bridge) and dark parts (eyes and brows). The output of the proposed circle-frequency filter has a local maximum at these characteristic points. To distinguish the true between-eyes from similar characteristic points in other face parts, we do a confirmation with eye detection. Once the between-eyes is detected, a small area around it is copied as a template and the system enters the tracking mode. Combining with the circle-frequency filtering and the template, the tracking is done not by searching around but by selecting candidates using the template; the template is then updated. Due to this special tracking algorithm, the system can track the between-eyes stably and accurately. It runs at 13 frames/sec rate without special hardware. By analyzing the movement of the point, we can detect nodding and headshaking. Some experimental results are shown. 1. Introduction Gesture recognition plays an important role in the advancement of human-computer interaction since it provides a natural and efficient interface to computers. Among head gestures, nodding and head-shaking are very common and used often, so their detection is basic to a visual understanding of human responses. However it is difficult to detect them in real-time, because nodding and head-shaking are fairly small and fast head movements. Methods of estimating the 3D head pose with the regions of skin and hair have been reported in [1][3]. In these papers, they use area information rather than feature points, so the algorithm seems to be robust. However, the resolution is inadequate for detecting small movements like nodding or head-shaking. Feature-based tracking approach for a real-time 3D facial pose estimation have been reported in [2][10]. In these papers, they track multiple feature points by template matching using a special correlation processor and feed them to a Kalman filter, although they do not mention how to make the templates. A facial motion vector is calculated by averaging all feature motion vectors[10]. In an example plot of nodding gesture, we cannot read the motion speed and size. It is only qualitatively described. In this paper, we propose a more simple and straightforward approach to a detection of nodding and head-shaking in real time from a single color video stream by directly detecting and tracking a point between the eyes, or what we call the between-eyes. General face location problem in image sequences seems to have been solved using skin-color information[1][6][8]. We take this approach but do so in a very simple way. Most previous research has tried to find the eyes first to detect face orientation [2][4][9], because eyes are the most characteristic part of the face. We try to start with other parts than eyes. What is common to most people and easy to find for a wide range of face orientations? We claim that one possible candidate is the point between the eyes. We call it the between-eyes. The between-eyes has dark parts on both side as eyes and eyebrows, and it is comparably bright on the upper side (forehead) and the lower side (nose bridge). This characteristic seems to be common to most people, and it can be seen for a wide range of face orientations. Moreover, the pattern

2 around this point is fairly stable for any facial expression, while eyes, eyebrows, and the mouth are changeable. We propose an image filter, the circle-frequency filter, to detect the between-eyes. Although the between-eyes can be robustly extracted using the circle-frequency filter, many other similar characteristic points are also extracted. By evaluating other local features, we can limit them to one or several points. Unfortunately, we concluded from some experiments that we cannot recognize the between-eyes by only using local features. Thus, we use eyes to confirm the candidate point for the true between-eyes. Once the between-eyes is detected, we do not need eye detection for tracking. A small area around the between-eyes is copied as a template. Then the tracking is done using the template by selecting the between-eyes from the extracted points with the circle-frequency filter. Due to this special tracking algorithm, the system stably and accurately tracks the between-eyes and runs at 13 frames/sec rate without special hardware. By analyzing the movement of the point, we can detect nodding and head-shaking. Applying a special filter to detect feature points on a face is a similar technique to applying the separability filter described in [7]. However our approach will more directly detect the target point. In section 2, the details of the circle-frequency filter are described. In section 3, we describe a strategy for detecting and tracking the between-eyes using the circle-frequency filter, combined with the local features and eye detection. In section 4, a simple algorithm for nodding and head-shaking recognition is presented. In section 5, the implementation and some experimental results are described. Section 6 concludes the paper. 2. Circle-frequency filter Assume that f i (i =0; :::; N, 1) are pixel values along a circle centered at (x; y). Then we can calculate their discreet Fourier transform as X N,1 F n = k=0 f k e 2ikn=N : (1) The circle-frequency-n filter outputs jjfnjj 2, or the square of the spectral power of frequency n, at(x; y). Hereafter, n =2, and we call it simply the circle-frequency filter. Figure 1 shows an example of f i s at the between-eyes (the white dot at the center). The radius of the circle is 7 pixels, and the 36 pixel values along it are plotted on the right graph. The plot starts at the forehead and goes counter clockwise. There are two cycles of dark and bright. The output of the circle-frequency filter for this pattern becomes high, and this is known as a characteristic of the Fourier transform. Figure 2 shows an example of the input (left) and output (right) images of the circle-frequency filter, where the filter Figure 1. \Between-eyes" and pixel values along 7 pixel circle of radius Figure 2. Example of circle-frequency ltered image and local maximum points radius is 7 pixels. The cross marks on the left image are the local maximum points of the output image. Although there are many local maximum points other than the between-eyes, the between-eyes robustly appears as a local maximum point, and we can reduce the number of candidates with simple local feature criteria as described below. 3. Strategies for detection and tracking the between-eyes Figure 3 shows a flowchart of the system. We try to keep things simple to achieve a higher frame rate. The system starts with the detection mode. After getting a new color frame, a skin-color area is extracted (see section 3.1). In the tracking mode, instead of the skin-color area extraction, we select a small area around the previous point of the between-eyes for the search area, and the remaining processes are restricted to this area. This greatly reduces the processing time. The skin-color area or the search area is converted to a monochrome smoothed image. The circle-frequency filter is applied to this monochrome image,

3 and local maximum points of the output are extracted. We screen with local feature conditions, and a few candidates are left for the true between-eyes. Detection mode Start with detection mode Get a new frame Tracking mode In the detection mode, eyes are searched for each candidate. When only one candidate has a pair of eyes, the system assumes it is the between-eyes, and switches to the tracking mode after saving a small area around it as a template. Otherwise, it assumes that the detection failed and goes to the next frame. In the tracking mode, each candidate is compared with the template, and the best match is selected. If it satisfies a predefined matching criteria, the system assumes it is the between-eyes, and the template is updated. Otherwise, it assumes the between-eyes is lost, and switches to the detection mode. At the end of each process of the detection mode or the tracking mode, the system investigates if nodding or headshaking has happened or not. Extract skin-color area Search area prediction 3.1. Extraction of skin-color area Detection mode Search eyes Monochrom smoothed image of the area Circle-frequency filter Extract local maximum with some conditions Set detection mode NO Only YES one has a pair of eyes? Set tracking mode NO Tracking mode Select the between-eyes using the template Good match? YES Skin-color provides good information for extracting an area of the face[8][1][6]. Several different kinds of color space have been proposed, and Terrillon discussed which is best for modeling a Gaussian skin-color distribution[6]. To keep it simple, we use a look-up table instead of a Gaussian model. Video cameras use RGB representation. Chromatic colors (r, g), known as pure colors in the absence of brightness, are defined in a normalization process: r = R=(R + G + B); (2) g = G=(R + G + B): (3) We modify (r, g) space to (a, b) space as follows: G 1.0 a = r + g=2; (4) b = p 3=2g: (5) Save new template Save the coordinate of the between-eyes (R,G,B) (a,b) 1.0 R Nodding and headshaking detection O b a Figure 3. Flow chart 1.0 B Figure 4. Color space model

4 This means that a color (R, G, B) is projected on to (a, b) plane as shown in Fig. 4. Any color can be projected on the hatched regular triangle region, and the range of (a, b) is from (0, 0) to (1, p 3=2). We digitize the (a, b) space into squares of 0.01 size as a look-up table, and get a histogram for face images. We define the skin-color as colors when the histogram value is over a threshold because face images contain non-skin colors. The threshold value is determined empirically. Actually, we divide an image into blocks of 8 8 pixels. When more than half of the pixels of a block are skin-color, we assume it is a skin-color block. The remaining processes are done on skin-color block areas Extraction of candidate points We convert the skin-color block area (plus one peripheral block for correct filtering) to a monochrome and smooth 1/4 size image by sub-sampling and averaging 5 5 pixel values. We then apply a circle-frequency filter and extract the local maximum points. The color to monochrome conversion equation is v =0:299R +0:587G +0:144B: (6) To screen the candidates of the between-eyes, we establish two simple conditions. (1) The left and right pixel value of the between-eyes on the circle of the filter must be lower than that of the between-eyes itself. (2) The real part of F 2 in Eq. 1 must be positive, which means that the phase of signal f i begins with a brighter part. These conditions are reasonable when the face rotation in the image plane is restricted to less than 45 degrees. After applying these conditions, the candidates are sorted on the output of the circle-frequency filter. The top three are tested further by confirming eye detection in the detection mode and by confirming template matching in the tracking mode Conrmation by eye detection (in the detection mode) Eye detection is done in a pretty simple way. A similar technique is used in [5]. At first, rectangular areas on both sides of the between-eyes candidate point where eyes should be found are extracted. The area size is defined in advance. The next step is to find the threshold level for each area to binarize the image. This is done by searching the level from the lowest level with a connected component analysis. The threshold level is determined when the sum of the number of pixels of all the components, which is not on the border of the rectangular area, exceeds a certain value (25 in the experiment). We avoid the component on the border because it often belongs to hair. Sometimes the eyebrows have the same grey level as eyes. Thus, from the extracted components, we select one within a certain range of pixels (more than 7, less than 30) and with the lowest position. We applied this method to 344 cases of true betweeneyes candidates in a preliminary experiment. In 308 cases both eyes are found. We measured two parameters in these cases, the distance between the located eyes (D e ) and the angle (A e ) between the left and right eyes at the betweeneyes. Table 1 lists these parameters. Table 1: Statistics of eye parameters. (n = 308) av. min. max. D e (pixels) A e (degrees) D e varies proportionally with the image size, or the distance from the camera to the face. The image size does not change the angle A e, but it varies more than we anticipated when the face turns up, down, or sideways. It exceeds 180 degrees in five cases. We looked at their distribution and defined the threshold value for eye parameters as follows. 31 <D e < 42 (7) 115 <A e < 180 (8) When only one candidate has eyes with this condition, we assume it is the between-eyes, and the pixels around it is copied from the monochrome smoothed image as a template. Otherwise, the between-eyes is not determined Conrmation by a template (in the tracking mode) In the tracking mode, template matching is done for all the candidates and the best match is selected. The matching index is not a normalized cross correlation but simply a sum-up of the absolute values of the difference in corresponding pixels, where the template size is Evenif the best match is found, when the matching index is larger than a predefined threshold, the system assumes that the between-eyes is out of the search area and switches to the detection mode. The template is updated for every frame. Therefore, the system can follow the change in the appearance of the between-eyes, as long as the change is gradual and the circle-frequency filter picks it out as a candidate.

5 4. Detection of nodding and head-shaking As a result of the tracking of the between-eyes, we obtain its coordinate (x i ;y i ) for every frame i, where (x i ;y i ) is a pixel coordinate of the monochrome smoothed image. From a preliminary observation, we find that nodding and head-shaking are very similar movements, except that nodding appears in the y i movement, while head-shaking appears in the x i movement. We have to distinguish them from repeated look-right and look-left or repeated look-up and look-down. Figure 5 shows typical nodding and head-shaking patterns, which are measured by the between-eyes tracking system. The unit of time axis is 1=13 seconds. The head y (a) nodding t x (b)head-shaking Figure 5. Typical pattern of nodding and headshaking motion is not symmetric, and typically involves two or three cycles. The deviation is plus/minus 4 or 5 pixels at maximum, although it depends on the imaging size. The deviation also does not last long, typically seconds. The algorithms for nodding and head-shaking detection are identical except that the former processes the y- coordinate and the latter processes the x-coordinate. The head-shaking detection algorithm is described here. First, each frame is categorized to a stable, transient, or extreme state. If t In the case of look-left or look-right, there are stable states instead of extreme states between or after transient states, so that they can be distinguishable. 5. Implementation and experiment We implemented the system on a Silicon Graphics O2 workstation with a 175 MHz CPU. (Fig. 3). Figure 6 shows an example of the processing result. The lower image is the input color image with some overlaid graphics. The image size is A circle on the face means that the system assumes this point is the betweeneye. There is a horizontal plot and a vertical plot that show the movement of the y and x coordinate of the tracked between-eyes, respectively. Every dot is classified into either stable (yellow), transient (green), or extreme (red). These two plots shows a repeated nodding and head-shaking when all cases were detected correctly. When the system detects nodding, the lower left corner flashes green, and head-shaking flashes red. (see demo movie from ~skawato/) The upper left image is a monochrome smoothed image of the search area predicted from the previous point of the between-eyes. Sub-sampling creates a size that is 1/4 that of the lower image. The two cross marks are the candidates of the between-eyes. max(x i+n ), min(x i+n ) 2 (n =,2::: +2) (9) then frame i is in a stable state. If x i = max(x i+n ) (n =,2::: +2) (10) or x i = min(x i+n ) (n =,2::: +2) (11) then frame i is in an extreme state. Otherwise, the frame is in a transient state. The last frame and the previous one have no state. When the state changes from non-stable to stable, the nodding evaluation process is triggered, so the detection has a two frame delay. The evaluation is simple. If there are more than two extreme state between current stable state and the previous stable state, and all adjacent extreme states differ with more than two pixels in the x coordinate, then the system assumes that head-shaking has occurred. Figure 6. Example of the processing result There are two images in the upper right corner. In the middle is a result of the circle-frequency filter applied to the left image, although the pixel values are normalized to 0

6 255. A small image on the upper edge is a current template of the between-eyes. The system runs at 13 frames/second rate. 150 frame sequences of three people (450 frames total) are tested to determine the detection and tracking ability. The result is shown in Table 2. Each person looks up and down, left and right, and moves his neck. Give-up or false results occur when subjects look aside at extreme angles. Another case of give-up occurs when a subject looks in a certain direction and a glass reflection prevents a detection of the true candidate. In the case of subject C, the number of false results is large, because the wrong point is tracked for a long time. Regardless, when a subject looks forward in an ordinary ways, the system does not miss the between-eyes. Figure 7 shows the extreme cases for a subject that the system can follow in the tracking mode. Table 2: Experimental result of tracking for three persons. success give-up false A B C Figure 7. Limitation of face postures for tracking 6. Conclusion We have proposed a method of detecting nodding and head-shaking, combining the detection algorithm with tracking of the between-eyes. To efficiently detect and track the between-eyes, we proposed the circle-frequency filter. Local maximum points of the filter output are candidate points for the betweeneyes. The true between-eyes is robustly included in these candidates. We examined local features to select the true between-eyes, and we concluded that local features can be used to reduce the number of candidates. However, we cannot be sure which is the true between-eyes with local features alone, because there are similar patterns, so we use eye locations to confirm which is the true between-eyes. Once the true between-eyes is detected, there is no need for confirmation with eyes from the next frame or in the tracking mode. A small area around the between-eyes is copied as a template, and this copy is used to select the true between-eyes in the tracking mode. The template is updated after every success of tracking. To reduce the processing area, we extract a skin-color area in the detection mode, and predict a small search area from the previous point of the between-eyes in the tracking mode. This area restriction reduces the processing time significantly. We implemented the algorithm on a workstation, and it ran at 13 frames/second. The detection rate of nodding and head-shaking was satisfactory when subjects did not look extremely aside. In the future, we will research the detection of individual face parts such as the eyes, the eyebrows, the nose, and the mouth, and estimate the expression of the face or the gaze direction. References [1] Q. Chen, H. Wu, T. Fukumoto, and M. Yachida. 3D head pose estimation without feature tracking. Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, pp , [2] J. Heinzmann and A. Zelinsky. 3-D facial pose and gaze point estimation using a robust real-time tracking paradigm. Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, pp , [3] K. Mase, Y. Watanabe, and Y. Suenaga. Headreader: Realtime motion detection of human head from image sequence. Trans. IEICE, J74-D-II(3): , [4] E. Petajan. Robust face feature analysis for automatic speech reading and charactor animation. Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, pp , [5] R. Stiefelhagen, J. Yang, and A. Waibel. Tracking eyes and monitoring eye gaze. Proc. of the Workshop on Perceptual User Interfaces, pp , [6] J. C. Terrillon, M. David, and S. Akamatsu. Automatic detection of human faces in natural scene images by use of a skin color model and of invariant moments. Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, pp , [7] O. Yamaguchi, K. Fukui, and K. Maeda. Face recognition using temporal image sequence. Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, pp , [8] J. Yang, R. Stiefelhagen, U. Meier, and A. Waibel. A realtime face tracker. Proc. 3rd IEEE Workshop on Application of Computer Vision, pp , [9] J. Yang and A. Waibel. Real-time face and facial feature tracking and applications. Proc. Workshop on Audio-Visual Speech Processing, pp , [10] A. Zelinsky and J. Heinzmann. Real-time visial recognition of facial gestures for human-computer interaction. Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, pp , 1996.