Face Tracking with learning Half-Face Template

Transcription

1 Face Tracking with learning Half-Face Template Jun ya Matsuyama 1 and Kuniaki Uehara 1 Department of Computer and Systems Engineering, Kobe University, 1-1 Rokko-dai, Nada, Kobe , Japan junya@ai.cs.scitec.kobe-u.ac.jp, uehara@kobe-u.ac.jp Abstract. In this paper, we present an algorithm to detect and track both frontal and side faces in video clips. By means of learning Haar- Like features of human faces and boosting the learning accuracy in the InfoBoost algorithm, our algorithm can detect frontal faces in video clips. We map these Haar-Like features to a 3D model to create the classifier that can detect both frontal and side faces. Since it cost a huge time to detect and track faces using the 3D model, we project the Haar-Like features from the 3D model to a 2D space for generating various face orientations before the face tracking process begins. By using them, we can detect even side faces in real time and it is not necessary to learn frontal faces and side faces separately. 1 Introduction In recent years there has been a great deal of interest in detection and tracking of human faces in video clips. To recognize people in a video clip, we must locate their faces. To design a robot capable of interacting with humans, it is required that the robots can detect human faces within their sight and recognize whom they are interacting with. In this case, the speed of face detection is most important. Additionally when face detection is used in surveillance systems, the speed of face detection is important, but it is most important that the system can detect all faces in the video clips. Recently, Viola et al. propose a very fast approach for multiple face detection [7], which uses AdaBoost algorithm [2]. This algorithm learns fast strong classifier by applying AdaBoost to a weak learner. Then a classifier is applied to sub images of the target image, and this algorithm detects faces in the target image. However, their algorithm has two problems. Firstly, AdaBoost has two problems to detect human faces, which we will discuss in section 3.1 and 3.2. Secondly, this algorithm can detect only frontal faces. Some algorithms, which can detect non-frontal faces [4 6, 8], have already been proposed, but they cannot detect and track more than one face, and these algorithms need to be initialized with face parameters (location, direction, scale etc). In this paper, we substitute InfoBoost algorithm [1], which is based on information-theoretic approach, for AdaBoost algorithm, which is based on decisiontheoretic approach, to solve first problem and to improve the precision. Additionally, our algorithm does not learn the whole human face but half of it to

2 2 Jun ya Matsuyama et al. reduce learning cost and to improve the precision. We map these half-face templates to a 3D model, and reproduce the whole-face template from the 3D model. So we can even detect faces which are even rotated around the vertical axis, by applying the generated whole-face template. 2 HAAR-LIKE FEATURES In this paper, Haar-Like features are used to classify the images into face images and non-face images. Some samples of Haar-Like features are shown in Figure 1. Basically, the brightness pattern of eyes distinguishes the face from the background. (a) is an original facial image. (b) measures the difference in brightness between the region of the eyes and the region under them. As is shown in (b), the brightness of the eyes is usually darker than their underneath in a image. (c) measures the difference in brightness between the region of the eyes and the region between them. As is shown in (c), the eyes are usually darker than the regions between them in a image. Thus, an image can be classified as a facial or a non-facial image based on the brightness pattern described above. These features in (b) and (c) are generated from feature prototypes (Figure 2) by dilating these prototypes vertically and horizontally. The difference in the average brightness between the black region and the white region of features is used as a feature value, which is used as a parameter to classify images into faces or non-faces, and the threshold of this feature value is calculated from a training set. Viola et al. use feature prototype (a)-(d) shown in Figure 2, but we use four additional prototypes (e)-(f). From prototype (e), we can generate the feature, which measure the difference in brightness between the region of the mouth and its both ends. From prototype (f), (g), we can generate the feature, which will measure the difference in brightness between the region of the mouth or the eye and the top and bottom of it. From prototype (e), we can generate the feature, which will measures the difference in brightness between the region of the eye, the nose or the mouth, and the region around it. If we add these effective feature prototypes like (e)-(f), the learning time will be somewhat increased, but the precision of the classification will be improved. Also if we add some compound feature prototypes like (c)-(h), the time of the classification will be decreased. For example, feature prototype (e) can be considered as the prototype which is made by combining two prototypes (a) together. In the regions where the features generated from prototype (e) can be used, we need calculate twice for the value of the feature generated from the prototype (a). But now we need only calculate once for the value of the feature generated from the prototype (e). So the time complexity is decreased for these regions, and the detection time can be reduced.

3 Face Tracking with learning Half-Face Template 3 (a) (b) Fig. 1. Feature Example (c) (a) (b) (c) (d) (e) (f) (g) (h) Fig. 2. Feature Prototypes 3 APPLYING INFOBOOST Classification with Haar-Like features is fast, but it is too simple and the precision is not so good. So we apply Boosting algorithm to the classifier with Haar-Like features to generate a fast strong classifier. We use InfoBoost to solve these problems, because there are two problems in AdaBoost RELIABILITY OF WEAK CLASSIFIER RESULT FOR EACH SAMPLE Weak classifiers generated in the AdaBoost process do not regard the reliability of the classification results for each sample, so the classifiers which classify the sample with high reliability and the classifiers which classify the sample with low reliability are combined in the same ways at the last. For example, there are nine weak classifiers generated in AdaBoost process. If four of them classify a sample into face image with 95% accuracy, and five of them classify the same sample into non-face image with 55% accuracy, the image will be classified into non-face image. If we think about the reliability of these classification results, the sample must be classified into face image. We must think about the reliability of the classification results to classify images more accurately. 3.2 BASED ON DECISION-THEORETIC AdaBoost algorithm only thinks about if the classification is true or false, when the weights are updated. But actually there are four cases, a face image classified into a face image, a non-face image classified into face image, a face image

4 4 Jun ya Matsuyama et al. classified into a non-face image, a non-face image classified into a non-face image. The incidence and accuracy of these classifications are not always the same. For example, in a certain round of AdaBoost, if the number of face images which are classified into non-face images and the number of non-face images which are classified into face images are nearly equal, all we are required to do is to decrease the weights of correctly classified samples and to increase the weights of incorrectly classified samples. But, if the number of misclassified nonface images is larger than the number of misclassified face images, the sample images which are misclassified non-face images will be classified correctly in next round or later. But the misclassified face images may not be correctly classified, because AdaBoost does not take care of the reason of misclassification. We should not pay attention to only if the classification is true or false, but we must also consider the correct classes, and when we update the weights of each sample images we must regard these four cases carefully. 3.3 INTRODUCTION OF INFOBOOST In this paper, we consider these two problems, and use InfoBoost algorithm, which is a modification of AdaBoost. we propose three points where InfoBoost and AdaBoost differ. First, InfoBoost repeats the round T times (t = 1,, T ) in learning process. In round t, weak hypothesis h t is shown by h t : X R. This represents that the hypothesis is calculated as hypothesis with evaluation of reliability. In order to make it easy to handle, we restrict the region of hypotheses h t to [ 1, +1]. The sign of h t represents the class of prediction ( 1 or +1), and the absolute value of h t represents magnitude of reliability. For example, if h 1 (x 1 ) = 0.8, h 1 classifies the sample x 1 into the class 1 with the reliability value 0.8. Second, the accuracy of negative prediction α t [ 1] and the acculacy of positive prediction α t [+1] are calculated. These values reflect the magnitude of reliability and the precision of classification for each sample. These two parameters are treated as one parameter α t. If hypothesis h t classifies x i into the class 1, α t (h t (x i )) is α t [ 1] if hypothesis h t classifies x i into the class +1, α t (h t (x i )) is α t [+1]. Third, the distribution D t+1 is updated with α t (h t (x i )), label y i, hypothesis h t (x i ) with evaluation of reliability. Executing this operation, the weights of samples, which are classified correctly by hypothesis h t (x i ), are decreased, and the weights of samples, which is not classified incorrectly by hypothesis h t (x i ), are increased. We repeat these processes T times, and calculate the final hypothesis H with the resulting T hypothesis. Hypotheses h t with evaluations of reliability are voted with α t (h t (x i )) as weights. If the result of voting is a negative value, we output 1 as the final hypothesis, and if the result of voting is a positive value, we output +1 as the final hypothesis. When we apply InfoBoost to weak learners, it is necessary to show how correct the classification results of training samples are. We calculate the feature value for each training sample, and we calculate the difference d of the feature

5 Face Tracking with learning Half-Face Template 5 value f and the threshold t (d = f t). The signs of differences represent classification result. If the absolute value of d is large enough, the classification result is trusty, whereas if the value is too small, the classification result is not so trusty. Obtaining the differences of feature values and thresholds, we can express the classification result and the reliability of it for each sample. So we use the differences of the feature values and the thresholds as the classification result with evaluation of reliability. Because InfoBoost is based on information-theoretic approach, it has the benefit of not only the improvement of the precision of classification, but also it can provide some flexibility for the classifier. For example, in surveillance systems, we must detect all human faces, but it is allowed to detect few non-faces as faces. By increasing the weights of face samples classified into non-faces, we can adapt classifiers for surveillance systems. InfoBoost can bias the classification basis based on the importance of each of the four case (face classified into face, face classified into non-face, non-face classified into face, non-face classified into non-face) by biased the rules of updating weight. 4 PROJECTION TO 3D MODEL AND LEARNING HALF-FACE TEMPlATE If we use the classifier that is generated by the whole face images, we can detect only frontal faces. However, in video clips, the probability of complete frontal faces is not high. Most of them are side faces. Consequently, we map the classifier learned from facial features to a 3D model (Figure 3(a) (b)). The 3D model is consist of three area ((r),(c),(l) in Figure 3(b)). By using this 3D template, the classifier will be able to detect not only frontal faces but also side shots of faces. However, if we use 3D model in face detection, time complexity is increased and detection speed is decreased. Consequently, we create new classifiers by rotating the 3D model around the vertical axis and by projecting the 3D model classifier back to 2D again (Figure 3(c) (d)). If the rotation angle is θ (positive for clockwise rotation), the width w r of area (r) is modified to cos(θ π 4 ) cos w π r, and 4 the width w c of area (c) is modified to cos(θ)w c, and the width w l of area (l) is modified to cos(θ+ π 4 ) cos π 4 w l. We execute these processes in every 10 degrees of rotation angles, and use these classifiers in parallel to prevent the increase of the time complexity. But, there is some problem. When the 3D model is rotated enough, some feature rectangles in 3D model may be hidden and we cannot use these features. So we do not use the whole faces but use half-faces for learning samples, and because of symmetry of a face, we express the whole face by combining reversed facial features with the original ones (Figure 3(a)). In doing so, even if the 3D model is rotated, we can use all features of either right half-face or left halfface. Additionally because of learning half faces, we can free from the different illumination states between left and right faces. The classifier generated from boosting algorithm is a set of weak classifiers using Haar-Like features. Each weak classifier consists of three elements, coor-

6 6 Jun ya Matsuyama et al. Reverse Feature 4 4 Mapping (r) (c) (l) (a) Feature in 2D (r) (c) (l) Rotate (b) 3D-Model (r) (c) (l) Projection (c) Rotated Feature with 3D-Model (d) Rotated Feature in 2D Fig. 3. Feature Projection dinates of the reference points, width and height of the rectangles, and thresholds of Haar-Like features. So the features are easily mapped to the 3D model shown in Figure 3(b). Additionally we confine that a 3D model can be rotated only around the vertical axis, so the features is changed only in horizontal axis. Therefore the weak classifier, which generated by projecting the features from 3D model to 2D, is made by moving, expanding and shrinking the Haar-Like feature rectangles of the original weak classifier horizontally. We extend the configuration of the weak classifier cascade to use half-face templates with 3D model. Classification of images is very complicated. If we generate only one classifier by boosting algorithm, the total number of weak classifiers and the time complexity will be very huge. We generate the cascade of classifiers by dividing the set of weak classifiers to some sets to reduce the time complexity. When we use half-face templates, we evaluate them separately, and aggregate these evaluation results. So the classifier consists of two classifier

7 Face Tracking with learning Half-Face Template 7 cascades (Figure 4). In the case when the target sample is rejected in early stage on one of the classifier cascades, the calculation on the other of the classifier cascades is not needed. So we combine these classifier cascades as shown in Figure 5. By using this new cascade, unnecessary process is avoided. right half of part images True True True... True True False False False False part images split input data classified as a non-face False AND True input data classified as a face left half of part images False False False False True True True... True True Fig. 4. Two Classifier Cascades input data classified as a non-face right half of part images False False False False... part images split left half of part images True True True True... True True True... False False False False True input data classified as a face input data classified as a non-face Fig. 5. Combining Two Classifier Cascades 5 EXPERIMENTS We performed two experiments of the face detection. First, we compare the precision of classifier generated by InfoBoost to AdaBoost. We use Yale Face Database B [3] as positive samples. We use half of them as the training set, and use the rest as the test set. The result of our experiment is shown in Figure 6. The horizontal axis shows false positive rate and the vertical axis shows true positive rate. The horizontal axis is logarithmic axis to show the difference of the graphs clearly. This graph shows the precision of InfoBoost is higher than that of AdaBoost. Second, we perform the experiment of face tracking with 3D model. The snapshots of the tracking result are shown in Figure 7. The rectangles show where faces are detected by this algorithm on the image. The two lines in the

8 8 Jun ya Matsuyama et al Hit Rate AdaBoost InfoBoost False Alarm Rate 1e-05 Fig. 6. Precision of Classifier rectangles show the position of eyes. It shows that we can detect not only frontal faces but also side faces. We can detect faces in an image, which is 320 by 240 pixel size, in about 0.05 seconds on 3.2GHz Pentium 4. Fig. 7. Snapshots of Tracking Result

9 Face Tracking with learning Half-Face Template 9 6 CONCLUSIONS AND FUTURE WORK In this paper, we tried to improve the precision of classifier by using InfoBoost algorithm, and tried to detect not only frontal faces but also side faces by using 3D model and half-face templates. But we cannot detect faces rotated around horizontal axis, or the axis which is vertical for image. If we rotate the 3D model around these axes, the feature rectangles is deformed and become non-rectangle. We must think about the fast algorithm to calculate these feature values. In the process of extracting sub images from the target image, if we use skin colors to detect face candidate regions, the precision of classifier will be improved and time complexity will be reduced. References 1. J. A. Aslam. Improving Algorithms for Boosting. In Proc. of 13th Annual Conference on Computational Learning Theory (COLT 2000), pages , Y. Freund and R. E. Schapire. Experiments with a New Boosting Algorithm. In Proc. of 13th International Conference on Machine Learning (ICML 96), pages , A. Georghiades, P. Belhumeur, and D. Kriegman. From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose. IEEE Trans. Pattern Anal. Mach. Intelligence, 23(6): , R. Gross, I. Matthews, and S. Baker. Constructing and fitting Active Appearance Models with occlusion. In Conference on Computer Vision and Pattern Recognition (CVPR 04) Workshop on Face Processing in Video (FPIV 04), P. P. Pradeep and P. F. Whelan. Tracking of Facial Features Using Deformable Triangles. In OPTO-Ireland 2002: Optical Metrology, Imaging, and Machine Vision, Proc. SPIE, volume 4877, pages , D. A. Ross, J. Lim, and M.-H. Yang. Adaptive Probabilistic Visual Tracking with Incremental Subspace Update. In Proc. of Eighth European Conference on Computer Vision (ECCV 2004), volume 2, pages , P. Viola and M. Jones. Rapid Object Detection using a Boosted Cascade of Simple Features. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, pages , Z. Zhu and Q. Ji. Real Time 3D Face Pose Tracking From an Uncalibrated Camera. In Conference on Computer Vision and Pattern Recognition (CVPR 04) Workshop on Face Processing in Video (FPIV 04), 2004.