A Comparison of Photometric Normalisation Algorithms for Face Verification

Transcription

1 A Comparison of Photometric Normalisation Algorithms for Face Verification James Short, Josef Kittler and Kieron Messer Centre for Vision, Speech and Signal Processing University of Surrey Guildford, Surrey, GU2 7XH, UK Abstract The variation of illumination conditions of an object can produce large changes in the image plane, significantly impairing the performance of face verification algorithms. We present a comparison of five photometric normalisation algorithms for use in pre-processing face images for the purpose of verification. The algorithms are tested on various configurations of three contrasting databases, namely the Yale B database, the XM2VTS database and the BANCA database. 1. Introduction In general the variation between images of different faces is smaller than that of the same face taken in a variety of environments. External factors such as pose and illumination can cause significant changes in the image plane. It has been shown that illumination causes larger variation in face images than pose [2]. The importance of illumination is further illustrated by examination of the eigenface method [19]. Belhumeur improved the accuracy of a recognition system based on eigenfaces, by removing the first three principal components [4]. Several methods have been proposed to compensate for illumination changes. Kee used shape from shading methods to generate a map of surface normals and reflectances for each subject. These were then used to synthesize an image under the same illumination conditions as the probe image. The two images were then compared directly [13]. Belhumeur analytically investigated the subspace generated by varying the illumination of an object, showing that it formed a convex cone [5]. This method requires a large amount of training data, but Lee showed that the subspace could be generated using only nine images captured under a particular set of illumination conditions [15]. Recognition is carried out by finding the distance of the probe image to the illumination cone. These algorithms work well, but are computationally expensive. In this paper we compare five algorithms for photometric normalisation. A method based on principal component analysis, multiscale retinex [18], homomorphic filtering [10], a method using isotropic smoothing to estimate the luminance function and a method using anisotropic smoothing [11]. Three contrasting databases are used in the experiment. The Yale B database has only ten subjects but contains a large range of illumination conditions [9]. The images in the XM2VTS database were all captured under a controlled environment in which illumination variation is minimised [16]. The BANCA database contains much more realistic illumination conditions [3]. The methods were tested extensively on the three databases using numerous protocols. We show that the homomorphic filter and the anisotropic method yield the most consistent results across all three databases. In the next section, we outline each of the normalization algorithms. Section 3 details the experimental procedures used to compare the methods. The results of the experiment are presented in Section 4 and we conclude in Section Methods In this section we describe five methods of photometrically normalising images. Histogram equalization was used with each method Principal Component Analysis Method Principal component analysis is a popular method of reducing the dimensionality of a set of data. This is carried out using eigen analysis of the data covariance matrix to find an ordered set of orthogonal basis vectors that best define the directions of greatest variance. When applied to the task of face verification, these vectors are known as eigenfaces [19].

2 A face image Á µ can be represented as a set of coefficients ¾ µ where each coefficient corresponds to an eigenface ¾ µ so that the face image is equal to the weighted sum of the eigenfaces. Á µ Ô µ (1) Bischof and Leonardis [6] take advantage of the linearity of Equation 1 and convolve both sides with a kernel Ã. Ô Ã Á µµ Ã µ (2) where denotes convolution with a kernel Ã. This equation holds true, irrespective of the convolution kernel. As the nature of illumination variation is of low frequency, we can choose a kernel designed to remove low frequency information. The input image of a face verification system can be filtered and the resulting image decomposed into a set of coefficients corresponding to a similarly filtered set of eigenfaces. These coefficients can then be used with the original set of eigenfaces to reconstruct the image Multiscale Retinex Method An image acquired by a camera is the product of two components, reflectance Ê µ and illumination Ä µ [12] Á µ Ä µê µ (3) Illumination is the amount of light falling on the object due to the light source. Reflectance is the amount of light reflected from the surface of the object. Land [14] decomposed the image into reflectance and illumination, by estimating the illumination as a low pass version of the original image, thus finding the reflectance by dividing the image by the illuminance function. Rahman [18] improved upon Lands work by estimating illumination as a combination of images generated by low pass filtering the original image with Gaussians of varying widths. Ä µ Ë Û Á µ µµ (4) where Û is a weighting term and the Gaussian µ has a width at a scale as defined by µ Ã ¾ ¾ ¾ (5) 2.3. Homomorphic Filtering A homomorphic filter separates illumination and reflectance by taking the logarithm of Equation 3 [10]. ÐÒ Á µ ÐÒÄ µ ÐÒÊ µ (6) The Fourier transform of the result gives ÐÒ Á µ ÐÒ Ä µ ÐÒ Ê µ (7) which can be written as the sum of two functions in the frequency domain Ù Úµ Ä Ù Úµ Ê Ù Úµ (8) Ê is composed of mostly high frequency components and Ä of mostly low frequency components. can be convolved with a filter of transfer function À Ù Úµ that reduces the low frequencies and amplifies high frequencies, thus improving contrast and compressing dynamic range. À Ù Úµ Ù Úµ À Ù Úµ Ä Ù Úµ À Ù Úµ Ê Ù Úµ (9) The processed image can be found by inverse Fourier transforming Equation 9 and taking the exponential. Á ¼ µ À ÙÚµ ÙÚµ 2.4. Isotropic Smoothing (10) Illumination can be estimated as a blurred version of the original image. This blurring can be implemented by constructing an illumination function Ä that is similar to the original image Á but contains a smoothing constraint. The illumination function can be constructed by minimizing the following cost function. Â Äµ Ä Áµ ¾ Ä ¾ Ä¾ µ (11) where parameter controls the relative importance of the smoothness constraint. The problem in Equation 11 can be solved by a Euler-Lagrange diffusion process which can be discretized as follows Á µ Ä µ ÖÄ Æ µ ÖÄ Ë µ ÖÄ µ ÖÄ Ï µ (12) where ÖÄ refers to the derivative with respect to each of the four adjacent neighbouring pixels, i.e. the subscripts Æ, Ë, and Ï define the direction of the neighbouring pixel, so that ÖÁ Æ µ is the value of the µ pixel minus the value of the µ pixel. The amount of smoothing is controlled by the parameter c. This Equation can be quickly solved using multigrid methods [7].

3 2.5. Anisotropic Smoothing The cost function in Equation 11 can be generalized by incorporating a weight, µ, mimicking the perception gain [1, 11], in the term maximising goodness of fit of the solution to data, i.e. Â Äµ µ Ä Áµ ¾ Ä ¾ Ä¾ µ (13) Gross suggests that the coefficient should be a function of local image contrast, so as to enhance the image contrast in a manner which is insensitive to illumination. Using the reciprocal of Weber s contrast as the smoothing parameter, the equation becomes Á µ Ä µ where Æ µ ÖÄ Æ µ Ë µ ÖÄ Ë µ µ ÖÄ µ Ï µ ÖÄ Ï µ ¾ Á Á ÑÒ Á Á µ (14) (15) is known as Weber s contrast. As with isotropic diffusion, multigrid methods are used to solve the equation. 3. Comparison of the Methods For a thorough evaluation of the effectiveness of the photometric normalisation algorithms, we investigated their effect on the accuracy of a face verification system. The system was tested using the various normalisation algorithms on the Yale B database [9], the XM2VTS database [16] and the BANCA database [3]. A benchmark set of images, without illumination normalisation, denoted Geometric in the results, were also tested. The Yale B database contains images under widely varying illumination conditions and poses of ten subjects. Tests were carried out using the frontal pose set of images with varying illumination. Both the XM2VTS database and the BANCA database have detailed evaluation protocols. The face verification software is based on linear discriminant analysis. It uses three sets of data, namely training, evaluation and testing. For each claim, the system generates a score reflecting how well the probe image matches the claimed identity. These scores are generated for the evaluation set in order to find a threshold corresponding to the equal error rate i.e. the point at which the false acceptance rate (FAR) equals the false rejection rate (FRR). Figure 1. Examples of the YaleB (top), XM2VTS (middle) and BANCA (bottom) database images This threshold is then used for the test set, from which the FAR and FRR can be calculated. The final result is generated by averaging FAR and FRR, i.e. the half total error rate (HTER). The Yale B database contains 64 different illumination conditions for 10 subjects. The illumination conditions are a single light source, the position of which varies horizontally (from -130 Æ to 130 Æ ) and vertically (from 40 Æ to 65 Æ ). Due to the limited size of the Yale B database, the test was carried out with the face verification system having been trained on the XM2VTS database. The ROC curve was then generated from the resulting scores. The XM2VTS database contains images of 295 subjects, captured over 4 sessions in a controlled environment. The database uses a standard protocol. The Lausanne protocol splits the database randomly into training, evaluation and test groups [16]. The training group contains 200 subjects as clients, the evaluation group an additional 25 subjects as impostors and the testing group another 70 subjects as impostors. The BANCA database was captured over twelve sessions in three different scenarios and has a population of 52 subjects (26 male and 26 female). Sessions 1 4 were captured in a controlled scenario, sessions 5 8 were captured in a degraded scenario which was captured using a simple web cam and session 9 12 were captured in an adverse scenario. The BANCA database has seven configurations of training and testing data incorporating different permutations of data from the twelve sessions. The seven configurations are Matched Controlled (MC),

4 Matched Degraded (MD), Matched Adverse (MA), Unmatched Degraded (UD), Unmatched Adverse (UA), Pooled test (P), and Grand test (G). The content of each configuration is described by table 1. T represents clients for training, I impostors for testing and C represents clients for testing geometric homomorphic isotropic anisotropic retinex pca Session MC MD MA UD UA P G 1 TI T T TI TI 2 CI CI CI 3 CI CI CI 4 CI CI CI 5 TI I I TI 6 CI CI CI CI 7 CI CI CI CI 8 CI CI CI CI 9 TI I I TI 10 CI CI CI CI 11 CI CI CI CI 12 CI CI CI CI True Acceptance False Acceptance Figure 2. ROC curve for the Yale B database Table 1. How different sessions are used for the protocols of the BANCA database 4. Experimental Results geometric homomorphic isotropic anisotropic retinex pca This section presents a summary of the results of testing the various algorithms on the three databases. Firstly, the results of the Yale B database experiment are presented in Figure 2. The homomorphic filter shows a small improvement in performance. The anisotropic method achieves a much better performance and the value of the local contrast coefficients is illustrated by the huge improvement in accuracy over the isotropic method. By far the best performance is obtained by the retinex method. The second experiment was carried out using the two configurations of the XM2VTS database. The results obtained from configuration one of the database show the homomorphic filter achieving the highest accuracy, narrowly outperforming the anisotropic method. In contrast, the results from configuration two of the database show superior results for the anisotropic method. When applied to the XM2VTS database, where the illumination is controlled, the retinex method actually shows a considerable drop in performance. The filtering process therefore removes valuable discriminatory information in addition to the illumination information. The third experiment was carried out on the BANCA database. The results are summarised in table 3 which shows the half total error rates. The seven configurations of the BANCA database show differing results. The principal component analysis method True Acceptance False Acceptance Figure 3. ROC curve for configuration 1 of the XM2VTS database improves performance on the matched and grand configurations, but degrades performance on the unmatched and pooled configurations. The anisotropic method yields the best results by a large margin over all but the matched adverse and unmatched adverse configurations. In the case of the unmatched adverse configuration, the anisotropic method is narrowly better than the homomorphic filter and in the matched adverse case it performs significantly worse. The test sets in these configurations are formed exclusively from sessions 10, 11 and 12 which correspond to the adverse scenarios. The isotropic method outperforms

5 True Acceptance geometric homomorphic isotropic anisotropic retinex pca Protocol Method MC MD MA UD UA P G Geometric PCA Retinex Homomorphic Isotropic Anisotropic Table 3. Performance on all protocols of the BANCA database. Values shown are the HTER: half total error rate False Acceptance Figure 4. ROC curve for configuration 2 of the XM2VTS database Protocol Configuration 1 Configuration 2 Method FAR FRR HTER FAR FRR HTER Geometric PCA Retinex Homomorphic Isotropic Anisotropic Table 2. Performance on both protocols of the XM2VTS database. Values shown are the FAR: false acceptance rate, FRR: false rejection rate, and HTER: half total error rate the anisotropic method on the matched adverse configuration, i.e. the local contrast coefficients lead to a reduction in performance in this case. 5. Conclusions The performance of various photometric normalisation algorithms has been compared on three very different databases and over numerous configurations. The PCA method shows inconsistent improvements. The illumination variation in the Yale B database is vast and the retinex method proves to be superior, however as a photometric normalisation algorithm it performs badly on the realistically illuminated XM2VTS and BANCA databases. The XM2VTS database in contrast with the BANCA database has excellent illumination conditions. There are no examples of shadowing and all images are captured in the same environment. The difference between the homomorphic filter and the anisotropic methods demonstrated by the XM2VTS database is vary small and based entirely on the selection of training and testing data. However when illumination conditions are degraded, such as in the BANCA database, the anisotropic method is clearly shown to be superior. References [1] S. Acton, Multigrid Anisotropic Diffusion IEEE Trans. Image Processing, vol. 7, 1998 [2] Y. Adini, Y. Moses, and S. Ullman, Face recognition: the problem of compensating for illumination changes. IEEE Trans. Pattern Anal. Mach. Intelligence, vol. 19(7), pp , 1997 [3] E Bailly-Bailliere, et al., The BANCA Database and Evaluation Protocol AVBPA, 2003 [4] P. Belhumeur, J. Hespanha, D. Kriegman, Eigenfaces vs. Fisherfaces: Recognition using Class Specific Linear Projection IEEE Trans. Pattern Anal. Mach. Intelligence, vol. 19, pp , 1997 [5] P. Belhumeur, D. Kriegman, What is the Set of Images of an Object Under All Possible Lighting Conditions? IEEE Proc. Conf. Computer Vision and Pattern Recognition, 1996 [6] H. Bischof, A. Leonardis, Illumination Insensitive Eigenspaces Proc. ICCV, pp , 2001 [7] W. Briggs, V. Henson, S. McCormick, A Multigrid Tutorial Siam, Second ed. [8] B. Funt, et al., Luminance Based Multiscale Retinex Proc. AIC, 1997 [9] A. Georghiades, P. Belhumeur, D. Kriegman, From Few to Many: Illumination Cone Models for Face Recognition under Variable Lighting and Pose IEEE Trans. Pattern Anal. Mach. Intelligence, vol. 23, pp , 2001 [10] R. Gonzalez, R. Woods, Digital Image Processing Prentice Hall, Second ed. [11] R. Gross, V. Brajovic, An Image Preprocessing Algorithm AVBPA, pp10-18, 2003

6 [12] B. Horn, Robot Vision MIT Press, 1998 [13] S. Kee, K. Lee and S. Lee, Illumination Invariant Face Recognition Using Photometric Stereo IEICE Trans. Inf & Syst, Vol.E83-D, No.7, 2000 [14] E. Land, J. McCann, Lightness and Retinex Theory Journal of the Optical Society of America, vol. 61, pp1-11, 1971 [15] K. Lee, J. Ho, D. Kriegman, 9 Points of Light: Aquiring Subspaces for Face Recognition Under Variable Lighting IEEE Proc. Conf. Computer Vision and Pattern Recognition, 2001 [16] K. Messer, J. Matas, J. Kittler, XM2VTSDB: The extended M2VTS Database AVBPA, 1999 [17] P. Perona, J. Malik, Scale-Space and Edge Detection Using Anisotropic Diffusion IEEE Trans. Pattern Anal. Mach. Intelligence, vol.12(7), 1990 [18] Z. Rahman, G. Woodell, D. Jobson, A Comparison of the Multiscale Retinex with other Image Enhancement Techniques Proceedings of the IS&T 50th Anniversary Conference, 1997 [19] M. Turk, A. Pentland, Eigenfaces for Recognition J. Congitive Neuroscience, vol. 3, pp 71-86, 1991