Performance Comparison of Visual and Thermal Signatures for Face Recognition Besma Abidi The University of Tennessee The Biometric Consortium Conference 2003 September 22-24
OUTLINE Background Recognition Engine Test Database Information Fusion Data fusion Decision fusion Results Thermal Face Recognition Conclusions and Future work 2
Background Facial variations due to varying illumination, expression, and rotation degrade the performance of visually based face recognition systems. The use of thermal imagery has great advantages in poor illumination conditions, where visual face recognition systems often fail. Glasses block thermal emissions around the eyes and need to be taken into consideration. FaceIt, a face recognition commercial software package, highly ranked by the face recognition vendor test (FRVT), is used for the comparison of recognition rates using visual and thermal face images. 3
Recognition Engine FaceIt Create Gallery Select Subject Rank by Confidence Rates 4 Images are courtesy of Equinox Corporation
Test Database Database (Co-registered visual and LWIR images from Equinox) Total DB ( 1,622*2 = 3,244) - 90 individuals Eye alignment using FaceIt! Visual : Automatic(95%), manual (5%)! Thermal (LWIR): Eye coordinates from co-registered visual image! Fused image : Eye coordinates from visual image Size Glasses Lighting Expression Gallery 90 Off Overhead Neutral Probe 1 283 Off Overhead Varying Probe 2 370 Off Left Varying Probe 3 365 Off Right Varying Probe 4 177 On Overhead Varying Probe 5 172 On Left Varying Probe 6 165 On Right Database specifications with co-registered visual/thermal Varying 5
Information Fusion Data fusion (Low Level) Co-registered visual and thermal images Average of the 2 images used as a simple data fusion Weighted summation considered for a fully automated engine Decision Fusion (High Level) Visual and thermal recognition results from FaceIt Average and higher confidence rate used and compared 6
Sample Data Fusion 1 st row : Visual, 2 nd row: Thermal, and 3 rd row : Fused images Images courtesy of Equinox Corporation 7
Decision Fusion 79 16 52 22 88 65 33 05 01 10 33 19 25 45 49 21 60 05 70 15 33 05 79 19 25 16 52 45 60 22 Decision fusion based on average confidence rates 8
Comparison of Visual and Thermal Recognition (glasses off) Thermal images give better performance than 9 visual images in all cases
Comparison of Fusion Methods (glasses off) Decision fusion (average, higher confidence) 10 gives the best results
Comparison of Fusion Methods (glasses on) Using FaceIt, recognition of individuals wearing glasses in thermal images results in poor performance 11
Radiometric Calibration Radiometric calibration establishes a direct relationship between the gray value response at a pixel and the thermal emissions. Radiometric calibration standardizes all thermal IR data, when taken in different environments. Helpful in differentiating between running people, indoor/outdoor conditions, and possibly nervousness, Both absolute and relative differences of thermal signatures are needed for a comprehensive and accurate solution. 12
Face Detection in Thermal Images (a) Input Image (b) Applying threshold (a) (b) (c) (c) Morphological Operation (d) Connected Components (e) Fitting Ellipses (f) Cropped Face region (d) (e) (f) 13
Thermal Face Normalization Recognition algorithms developed for visual face recognition can be used for thermal face recognition. The eyes, which are usually used for the normalization of visual faces, cannot be easily detected in thermal images. The nose is usually the coldest area, but not always. Edge detection, ellipse fitting, and multithresholding approaches used for face scaling and orientation. Use of co-registered visual images helps in the normalization process. x 14 x
Detecting Glasses in Thermal Images (a) (b) (c) (a) Input Image, (b) Thresholding and Connected Components Analysis, (c) Ellipse Fitting Classification : Size comparison, location, and symmetry of the ellipses are used to recognize the presence of glasses. Weighted summation applied to implement data fusion. More or less weights are given to thermal component as function of presence or absence of glasses. 15
Conclusions Thermal images give better performance than visual images when individuals are not wearing glasses. Both fusion of visual and thermal images (Data Fusion) or recognition results (Decision Fusion) increase the overall performance of face recognition systems. 16
Future Work Fusion scheme generalized to the case of individuals wearing glasses using weighted summation and/or neural network decision schemes. Feature level fusion methods will be investigated by considering variations in visual and thermal images. 17