Face Haucination and Recognition Xiaogang Wang and Xiaoou Tang Department of Information Engineering, The Chinese University of Hong Kong {xgwang1, xtang}@ie.cuhk.edu.hk http://mmab.ie.cuhk.edu.hk Abstract. In video surveiance, the faces of interest are often of sma size. Image resoution is an important factor affecting face recognition by human and computer. In this paper, we study the face recognition performance using different image resoutions. For automatic face recognition, a ow resoution bound is found through experiments. We use an eigentransformation based haucination method to improve the image resoution. The haucinated face images are not ony much hepfu for recognition by human, but aso make the automatic recognition procedure easier, since they emphasize the face difference by adding some high frequency detais. 1 Introduction In video surveiance, the faces of interest are often of sma size because of the great distance between the camera and the objects. Image resoution is a potentia factor affecting face recognition performance. In the ow-resoution face images, many detaied facia features are ost and faces are indiscernibe to human. We aso notice that in many automatic face recognition systems, face images are down samped to sma size, and aso achieve satisfied performance. But how wi the image resoution affect recognition accuracy is sti open to discussion. Severa agorithms have been proposed to render a high-resoution face image from the ow-resoution one. This technique is caed haucination [4]. Since face images are we structured and have simiar appearance, they span a sma subset in the high dimensiona image space [3]. This impies that the high frequency detai can be inferred from the ow frequency components, utiizing the face structura simiarity. The simpest way to increase resoution is direct interpoation of input images with such agorithms as nearest neighbour, cubic spine. But its performance is poor if the image size is too sma. Baker and Kanade [4] deveop a haucination method based on the property of face image. It infers the high frequency component from a parent structure by recognizing the oca features from the training set. Liu et. a. [1] deveop a two-step statistica modeing approach integrating goba and oca parameter modes. Haucination has effectivey improved the resoution of face images thus makes it much easier for a human being to recognize a face. However, how much information has been extracted from the ow-resoution image by the haucination process and its contribution to automatic face recognition has not been studied in previous works.
g g + + B 1 Low frequency B 0 B K B 1 B K High frequency Figure 1. Muti-resoution anaysis in spatia domain. g is the smoothing function, and B 0,, B K, are different frequency bands In this paper, we study the face recognition performance using different image resoutions. We use a nove haucination method based on eigentransformation [6]. It is cosey reated to the work in [5], in which an eigentransformation approach was deveoped for sketch recognition. In our method, PCA is appied to the ow-resoution face image. In the PCA space, different frequency components are independent. By seecting the number of eigenfaces, we coud extract the maximum amount of facia information from the ow-resoution face image and remove the noise. The new haucinated face image is rendered by mapping between the ow- and high- resoution training pairs. We aso study the effect of haucination on automatic face recognition. Since haucination emphasizes the face difference by adding some high frequency detais, it may hep the automatic recognition process. Experiments are conducted on a database containing images of 188 peope and the XM2VTS face database [2]. 2 Mutiresoution Anaysis Viewing a 2D image as a vector, the process of getting a ow-resoution face image from the high-resoution face image can be formuated as I = HI h + n. (1) Here, I h and I represent the high- and ow-resoution face image vectors respectivey. H is the transformation matrix invoving burring and downsamping process, and n is the noise perturbation to the ow-resoution face image captured by camera. As shown in Figure 1, a process of iterative smoothing and downsamping decomposes the face image into different bands, B 0,, BK. In this decomposition, different frequency bands are not independent. Some components of the high-frequency bands, B 1,, B K, can be inferred from the ow frequency band B 0. This is a starting point for haucination. Many super-resoution agorithms assume the dependency as homogeneous Markov Random Fieds (MRFs), i.e. the pixe ony reies on the pixes in its neighborhood. This is an assumption for genera images. It is not optima for the face
e 1 e 2 e 3 e 5 Information on facia feature e 10 e 50 e 100 e 500 Figure 2. Eigenfaces sorted by eigenvaues. e i is the ith eigenface. Noise K Eigenfaces Figure 3. Extract facia information in the PCA space of ow-resoution face images. cass without considering face structura simiarity. A better way to address the dependency is using PCA, in which different frequency components are independent. 3 Haucination and Recognition Face image can be reconstructed from some eigenfaces in the PCA representation. PCA aso decomposes face image into different frequency components, but encoding facia information in a more compact way, since it takes into account of the face distribution. Our agorithm first empoys PCA to extract as much usefu information from ow-resoution face image as possibe, and then renders a high-resoution face image by eigentransformation. A detaied description for eigentransformation can be found in [5]. 3.1 Principe Component Anaysis We represent a set of face images by a N by M matrix, [, ] 1 M, where i is the image vector, N is the number of image pixe, and M is the number of the training sampes ( N >> M ). In PCA, a set of eigenvectors E = [ e1, h, e K ], aso caed eigenfaces, are computed from the ensembe covariance matrix, M T T C = ( i m )( i m ) = LL, (2) i=1 where m is the mean face computed from the sampe set, and L is the sampe matrix, L = [ 1 m1,, M mm ] = [ ' 1, ' M ]. (3) For a face image x, a weight vector is computed by projecting it onto eigenfaces, w = E. (4) T ( x m )
This is a face representation based on eigenfaces. A face can be reconstructed from the K eigenfaces, r = E w + m. (5) Figure. 2 shows some eigenfaces sorted by eigenvaues. Eigenfaces with arge eigenvaues are face-ike, and characterize ow frequency components. Eigenfaces with sma eigenvaues are noise-ike, and characterize high frequency detais. 3.2 Eigentransformation Given the ow-resoution sampe set L, according to singuar vaue decomposition theorem, E aso can be computed from, 1/ 2 E = LV Λ, (6) where V and Λ are the eigenvector and eigenvaue matrix for L T L. From (5) and (6), the reconstructed face image can be represented by 1/ 2 r = LV Λ w + m = Lc + m, (7) c = V Λ w = c1, c2,, c 1/ 2 where [ ] T M. Equation (7) can be rewritten as, M = Lc + m = i=1 r i i c ' + m. (8) This shows that the input ow-resoution face image can be reconstructed from the optima inear combination of the M ow-resoution training face images. Repacing each ow-resoution image ' i by its high-resoution sampe h' i, and repacing m with the high-resoution mean face m h, we get x h, which is expected to be an approximation to the rea high-resoution face image. 3.3 Recognition In our agorithm, the haucinated face image is synthesized by the inear combination of high-resoution training images and the coefficients come from the ow-resoution face images using the PCA method. Because of the structura simiarity among face images, in mutiresoution anaysis, there exists strong correation between the high frequency band and ow frequency band. For high-resoution face images, PCA can compact these correated information onto a sma number of principe components. Then, in the eigentransformation process, these principe components can be inferred from those of the ow-resoution face image by mapping between the high- and owresoution training pairs. Therefore, some information in the high frequency band bands are partiay recovered. In practice, the ow-resoution image is often disturbed by noise which has a fat distribution on a the axes. For ow-resoution face images, the energy on sma ei-
genvectors is sma, thus is overwhemed by noise. By seecting an optima eigenface number K, we can extract the facia information and remove the noise. The information on these noisy components (eigenfaces after K in Fig. 3) is ost, and cannot be recovered since the components on different eigenvectors are independent in PCA space. In this sense, our haucination method has extracted the maximum amount of facia information exists in the ow-resoution face images. Given the significant improvement of the face appearance by the haucination process, it is interesting to investigate whether the haucination heps automatic recognition. Since more high frequency detais are recovered, we expect the baucination process to hep the recognition performance. 4 Experiment 4.1 Haucination Experiment Our haucination experiment is conducted on a data set containing 188 individuas with one face image for each individua. Using the eave-one-out methodoogy, at each time, one image is seected for testing and the remaining are used for training. In preprocessing, the face images are aigned by the two eyes. The distance between the eye centers is fixed at 50 pixes, and the image size is fixed at 117 125. Images are burred by averaging neighbour pixes and down samped to ow-resoution images. Here, we use the eye center distance de to measure the face resoution. Some haucination resuts are shown in Fig. 4. The input face images are down samped to 23 25, with de equa to 10. Compared with the Cubic B-Spine interpoation resut, the haucinated face images have much cearer detai features. They are good approximation to the origina high-resoution images. Figure 5 reports the haucination performance for different input resoutions. The eye center distance is down samped to 20, 10, 7, and 5. Figure 6 repots the average RMS error per pixe in intensity for the 188 face images. Under a very ow resoution, the ow-resoution and direct interpoated face images are amost indiscernibe, and the RMS error of Cubic B-spine interpoation increases quicky. The performance of haucination by eigentransformation is much better. When de is down samped to 10, the resut of eigentransformation is sti satisfactory. For further ower resoutions, there are some distortions on the eyes and mouth. As discussed in Section 3, some high frequency detai is ost in the process of bur and downsamping, or is overwhemed by noise. Seecting the eigenface number in eigentransformation, we coud contro the detai eve by keeping maximum facia information whie removing the noise. This point can be iustrated in the experiment reported by Figure 7. We add zero mean, white Gaussian noise with five different standard deviations (σ ) to the ow-resoution face image, and then use different eigenface number (K) for haucination. The optima eigenface number decreases as the increase of noise. Using 180 eigenfaces, the haucinated face images are noisy and distorted for a the five eves of noise. When K is reduced to 100, face images under
(a) input 23 25 (b) Cubic B-Spine (c) Haucinated (d) Origina 117 125 Figure 4. Haucinated face images by eigentransformation. sma noise ( σ = 0.03,0. 05 ) are we haucinated. but resuts under more noise ( σ = 0.07,0.1,0. 12 ) have a arger distortion. Using 50 eigenfaces, a of the images show itte noise effect. So eigenface number can contro the detai eve to make the haucinated face images robust to noise. 4.2 Recognition Experiment We study the recognition performance using ow-resoution face images and haucinated face images. Two hundred and ninety five individuas from the XM2VTS face database are seected, with two face images in different sessions for each individua. One image is used as reference, and the other is used for testing. We use direct correation for recognition, which is perhaps the simpest face recognition agorithm. The recognition accuracies over different resoutions are potted in Figure 8. When de is reduced from 50 to 10, there is ony sight fuctuation on recognition accuracy using ow-resoution face images. When de is further reduced to 7 and 5, the recognition accuracy for ow-resoution face images drops greaty. Resoution with de equa to 10 is perhaps a ower bound for recognition. Beow this eve there may not be enough information for recognition. This is aso consistent with the haucination experiment in 4.1. Satisfactory haucination resuts can be obtained when de is arger than 10. We aso try to expore whether haucination can contribute to automatic face recognition. We expect haucination make the recognition procedure easier, since it emphasizes the face difference by adding some high frequency detais. In this experiment, the ow-resoution testing image is haucinated by reference face images, but the face image of the testing individua is excuded from the training set. As shown in Figure 8, the haucination improved the recognition accuracy when the input face images have very ow resoutions.
(a) Origina 50 ( 117 125 ) 20 ( 47 50 ) 10 ( 23 25 ) 7 ( 16 17 ) 5 ( 11 12 ) (b) The first row is the input face images, for which de is 20, 10, 7, 5 respectivey; the second row is the haucinated face images. Figure 5. Haucinated face images using input images of different resoutions. 5 Concusion Our haucination method based on eigentransformation coud extract the maximum facia information from the ow-resoution face images and render some high frequency facia feature to make the face image more discernibe. It aso makes the automatic face recognition more easier. We aso study the face recognition performance over different resoutions. A ow resoution bound for recognition is found in the experiment. This is ony a preiminary study. The resuts need to be further confirmed using more face recognition agorithms and data sets. Acknowedgement This work was supported by the Research Grants Counci of the Hong Kong SAR under Projects CUHK 4190/01E and AOE/E-01/99. Reference 1. C. Liu, H. Shum, and C. Zhang, " A Two-Step Approach to Haucinating Faces: Goba Parametric Mode and Loca Nonparametric Mode," Proc. of IEEE Internationa Conference on Computer Vision and Pattern Recognition, pp. 192-198, 2001. 2. K. Messer, J. Matas, J. Kitter, J. Luettin, and G. Maitre, XM2VTSDB: The Extended M2VTSDB, In the Second Internationa Conference on Audio and Video-Based Biometric Person Authentication, pp. 72-77, March 1999. 3. P. S. Penev, and L. Sirovich, The Goba Dimensionaity of Face Space, Proc. of IEEE Internationa Conference on Automatic Face and Gesture Recognition, pp. 264-270, 2000. 4. S. Baker, and T. Kanade, "Haucinating Faces," Proceedings IEEE Internationa Conference on Automatic Face and Gesture Recognition, pp. 83-88, 2000. 5. X. Tang, and X. Wang, Face Photo Recognition Using Sketch, Proc. of ICIP, 2002. 6. X. Wang and X. Tang, Haucinating Face by Eigentransformation, ICIP 2003.
Figure 6. RMS error per pixe in intensity using Cubic-spine interpoation and haucination by eigentransformation. The intensity is between 0 and 1. Figure 8. Recognition accuracy using owresoution face images and haucinated face images based on XM2VTS database. ( σ = 0. 03 ) ( σ = 0. 05 ) ( σ = 0. 07 ) ( σ = 0. 1) ( σ = 0. 12 ) (a) (K=50, σ = 0. 03 ) (K=50, σ = 0. 05 ) (K=50, σ = 0. 07 ) (K=50, σ = 0. 1 ) (K=50, σ = 0. 12 ) (K=100, σ = 0. 03 )(K=100, σ = 0. 05 )(K=100, σ = 0. 07 )(K=100, σ = 0. 1 )(K=100, σ = 0. 12 ) (K=180, σ = 0. 03 )(K=180, σ = 0. 05 )(K=180, σ = 0. 07 )(K=180, σ = 0. 1 ) (K=180, σ = 0. 12 ) (b) Figure 7. Haucinating face with additive Gaussian noise. (a): Low-resoution face images with noise, (b) Haucinated faces. K is the eigenface number, and σ is the standard deviation of Gaussian noise (Image intensity is between 0 and 1). The origina high-resoution face image is referred to Fig. 7 (a).