Face Recognition Using Line Edge Map

Transcription

1 764 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 6, JUNE 00 Face Recognition Using Line Edge Map Yongsheng Gao, Member, IEEE, and Mayor K.H. Leung, Member, IEEE AbstractÐThe automatic recognition of human faces presents a significant chaenge to the pattern recognition research community. Typicay, human faces are very simiar in structure with minor differences from person to person. They are actuay within one cass of ªhuman face.º Furthermore, ighting condition changes, facia expressions, and pose variations further compicate the face recognition task as one of the difficut probems in pattern anaysis. This paper proposed a nove concept, ªfaces can be recognized using ine edge map.º A compact face feature, Line Edge Map (LEM), is generated for face coding and recognition. A thorough investigation on the proposed concept is conducted which covers a aspects on human face recognition, i.e., face recognition, under ) controed/idea condition and size variation, ) varying ighting condition, 3) varying facia expression, and 4) varying pose. The system performances are aso compared with the eigenface method, one of the best face recognition techniques, and reported experimenta resuts of other methods. A face prefitering technique is proposed to speed up the searching process. It is a very encouraging finding that the proposed face recognition technique has performed superior to the eigenface method in most of the comparison experiments. This research demonstrates that LEM together with the proposed generic ine segment Hausdorff distance measure provide a new way for face coding and recognition. Index TermsÐFace recognition, ine edge map, ine segment Hausdorff distance, structura information. æ INTRODUCTION COMPUTERIZED human face recognition has been an active research area for the ast 0 years. It has many practica appications, such as bankcard identification, access contro, mug shots searching, security monitoring, and surveiance systems []. Face recognition is used to identify one or more persons from sti images or a video image sequence of a scene by comparing input images with faces stored in a database. It is a biometric system that empoys automated methods to verify or recognize the identity of a iving person based on his/her physioogica characteristic. In genera, a biometric identification system makes use of either physioogica characteristics (such as a fingerprint, iris pattern, or face) or behavior patterns (such as handwriting, voice, or key-stroke pattern) to identify a person. Because of human inherent protectiveness of his/her eyes, some peope are reuctant to use eye identification systems. Face recognition has the benefit of being a passive, nonintrusive system to verify persona identity in a ªnaturaº and friendy way. The appication of face recognition technoogy can be categorized into two main parts: aw enforcement appication and commercia appication. Face recognition technoogy is primariy used in aw enforcement appications, especiay mug shot abums (static matching) and video surveiance (rea-time matching by video image sequences). The commercia appications range from static matching of photographs on credit cards, ATM cards, passports, driver's icenses, and photo ID to rea-time. The authors are with the Schoo of Computer Engineering, Nanyang Technoogica University, Nanyang Ave., , Singapore. E-mai: {asysgao, asmkeung}@ntu.edu.sg. Manuscript received 5 Dec. 000; revised 7 Aug. 00; accepted 8 Nov. 00. Recommended for acceptance by Z. Zhang. For information on obtaining reprints of this artice, pease send e-mai to: tpami@computer.org, and reference IEEECS Log Number 33. matching with sti images or video image sequences for access contro. Each presents different constraints in terms of processing requirement. To the best of our knowedge, there is hardy any reported research work on face recognition using edge curves of faces except the outines of face profies. The ony most reated work was done by TakaÂcs [] using binary image metrics. The face coding and matching techniques presented in this paper are different from []. TakaÂcs used edge map face coding and pixe-wise Hausdorff distance tempate matching techniques, whie this paper proposed ine-based face coding and ine matching techniques to integrate geometrica and structura features in the tempate matching. A nove concept, ªfaces can be recognized using ine edge map,º is proposed here. A compact face feature, Line Edge Map (LEM), is extracted for face coding and recognition. A feasibiity investigation and evauation for face recognition based soey on face LEM is conductedwhich covers a conditions of human face recognition, i.e., face recognition under controed/idea condition, varying ighting condition, varying facia expression, and varying pose. The system performances are compared with the eigenface method, one of the best face recognition techniques, and reported experimenta resuts of other methods. It is a very encouraging finding that the proposed face recognition technique has performed consistenty superior to (or equay we as) the eigenface method in a the comparison experiments except under arge facia expression changes. A prefitering scheme (two-stage identification) is proposed to speed up the searching using a D prefitering vector derived from the face LEM. This research demonstrates that LEM together with the proposed generic Line Segment Hausdorff Distance measure provide a new way for face coding and recognition. Some of the ideas presented in this paper were initiay reported in [3]. In this paper, we report the fu and new formuation and extensive experimenta evauation of our /0/$7.00 ß 00 IEEE

2 GAO AND LEUNG: FACE RECOGNITION USING LINE EDGE MAP 765 techniques. In the foowing, a iterature review of face recognition techniques is given in Section. Most successfu approaches to fronta face recognition, namey, eigenface, neura network, dynamic ink architecture, hidden Markov mode, geometrica feature matching, and tempate matching are discussed. Section 3 describes the concepts of face recognition using ine edge map. A nove ine segment Hausdorff distance measure for human face recognition is proposed in Section 4. The advantages of the proposed approach are discussed and compared with existing pixe wise Modified Hausdorff Distance method. In Section 5, the system is extensivey examined on face recognition under controed/idea condition, size variation, varying ighting condition, varying expression, and varying pose. The storage and computationa requirements are anayzed, and the system performance is compared with existing approaches. Section 6 presents a two-stage face identification schemewhich speeds up the searching process by fitering out part of the unikey candidates. Finay, the paper concudes in Section 7. BACKGROUND This section overviews the major human face recognition techniques that appy mosty to fronta faces. The methods considered are eigenface (eigenfeature), neura network, dynamic ink architecture, hidden Markov mode, geometrica feature matching, and tempate matching. The approaches are anayzed in terms of the facia representations they used. Eigenface is one of the most thoroughy investigated approaches to face recognition. It is aso known as Karhunen- LoeÁve expansion, eigenpicture, eigenvector, and principa component. Sirovich and Kirby [5] and Kirby et a. [6] used principa component anaysis to efficienty represent pictures of faces. They argued that any face images coud be approximatey reconstructed by a sma coection of weights for each face and a standard face picture (eigenpicture). The weights describing each face are obtained by projecting the face image onto the eigenpicture. Turk and Pentand [7] used eigenfaces, which was motivated by the technique of Kirby and Sirovich, for face detection and identification. In mathematica terms, eigenfaces are the principa components of the distribution of faces, or the eigenvectors of the covariance matrix of the set of face images. The eigenvectors are ordered to represent different amounts of the variation, respectivey, among the faces. Each face can be represented exacty by a inear combination of the eigenfaces. It can aso be approximated using ony the ªbestº eigenvectors with the argest eigenvaues. The best M eigenfaces construct an M dimensiona space, i.e., the ªface space.º The authors reported 96 percent, 85 percent, and 64 percent correct cassifications averaged over ighting, orientation, and size variations, respectivey. Their database contained,500 images of 6 individuas. As the images incude a arge quantity of background area, the above resuts are infuenced by background. The authors expained the robust performance of the system under different ighting conditions by significant correation between images with changes in iumination. However, Grudin [6] showed that the correation between images of the whoe faces is not efficient for satisfactory recognition performance. An iumination normaization [6] is usuay necessary for the eigenface approach. Zhao and Yang [4] proposed a new method to compute the covariance matrix using three images each taken in different ighting conditions to account for arbitrary iumination effects, if the object is Lambertian. Pentand et a. [8] extended their eary work on eigenface to eigenfeatures corresponding to face components, such as eyes, nose, and mouth. They used a moduar eigenspace which was composed of the above eigenfeatures (i.e., eigeneyes, eigennose, and eigenmouth). This method woud be ess sensitive to appearance changes than the standard eigenface method. The system achieved a recognition rate of 95 percent on the FERET database of 7,56 images of approximatey 3,000 individuas. In summary, eigenface appears as a fast, simpe, and practica method. However, in genera, it does not provide invariance over changes in scae and ighting conditions. The attractiveness of using neura network coud be due to its noninearity in the network. Hence, the feature extraction step may be more efficient than the inear Karhunen-LoeÁve methods. One of the first artificia neura network (ANN) techniques used for face recognition is a singe ayer adaptive network caed WISARD which contains a separate network for each stored individua [9]. The way in constructing a neura network structure is crucia for successfu recognition. It is very much dependent on the intended appication. For face detection, mutiayer perceptron [0] and convoutiona neura network [] have been appied. For face verification, Cresceptron [] is a mutiresoution pyramid structure. Lawrence et a. [] proposed a hybrid neura network which combined oca image samping, a sef-organizing map (SOM) neura network, and a convoutiona neura network. The SOM provides a quantization of the image sampes into a topoogica space where inputs that are nearby in the origina space are aso nearby in the output space, thereby providing dimension reduction and invariance to minor changes in the image sampe. The convoutiona network extracts successivey arger features in a hierarchica set of ayers and provides partia invariance to transation, rotation, scae, and deformation. The authors reported 96. percent correct recognition on ORL database of 400 images of 40 individuas. The cassification time is ess than 0.5 second, but the training time is as ong as 4 hours. Lin et a. [3] used probabiistic decision-based neura network (PDBNN) which inherited the moduar structure from its predecessor, a decision based neura network (DBNN) [4]. The PDBNN can be appied effectivey to ) face detector: which finds the ocation of a human face in a cuttered image, ) eye ocaizer: which determines the positions of both eyes in order to generate meaningfu feature vectors, and 3) face recognizer. A hierarchica neura network structure with noninear basis functions and a competitive credit-assignment scheme was adopted. PDBNN-based biometric identification system has the merits of both neura networks and statistica approaches, and its distributed computing principe is reativey easy to impement on parae computer. In [3], it was reported that PDBNN face recognizer had the capabiity of recognizing up to 00 peope and coud achieve up to 96 percent correct recognition rate in approximatey second. However, when the number of persons increases, the computing expense wi

3 766 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 6, JUNE 00 become more demanding. In genera, neura network approaches encounter probems when the number of casses (i.e., individuas) increases. Moreover, they are not suitabe for a singe mode image recognition task because mutipe mode images per person are necessary in order for training the systems to ªoptimaº parameter setting. Graph matching is another approach to face recognition. Lades et a. [5] presented a dynamic ink structure for distortion invariant object recognition which empoyed eastic graph matching to find the cosest stored graph. Dynamic ink architecture is an extension to cassica artificia neura networks. Memorized objects are represented by sparse graphs, whose vertices are abeed with a mutiresoution description in terms of a oca power spectrum and whose edges are abeed with geometrica distance vectors. Object recognition can be formuated as eastic graph matching which is performed by stochastic optimization of a matching cost function. They reported good resuts on a database of 87 peope and a sma set of office items comprising different expressions with a rotation of 5 degrees. The matching process is computationay expensive, taking about 5 seconds to compare with 87 stored objects on a parae machine with 3 transputers. Wiskott and von der Masburg [6] extended the technique and matched human faces against a gaery of neutra fronta view faces. Probe images were distorted due to rotation in depth and changing facia expression. Encouraging resuts on faces with arge rotation anges were obtained. They reported recognition rates of 86.5 percent and 66.4 percent for the matching tests of faces of 5 degree rotation and 0 faces of 30 degree rotation to a gaery of neutra fronta views. In genera, dynamic ink architecture is superior to other face recognition techniques in terms of rotation invariant; however, the matching process is computationay expensive. Stochastic modeing of nonstationary vector time series based on hidden Markov modes (HMM) has been very successfu for speech appications. Samaria and Faside [7] appied this method to human face recognition. Faces were intuitivey divided into regions such as the eyes, nose, mouth, etc., which can be associated with the states of a hidden Markov mode. Since HMMs require a one-dimensiona observation sequence and images are two-dimensiona, the images shoud be converted into either D tempora sequences or D spatia sequences. In [8], a spatia observation sequence was extracted from a face image by using a band samping technique. Each face image was represented by a D vector series of pixe observation. Each observation vector is a bock of L ines and there is an M ines overap between successive observations. An unknown test image is first samped to an observation sequence. Then, it is matched against every HMMs in the mode face database (each HMM represents a different subject). The match with the highest ikeihood is considered the best match and the reevant mode reveas the identity of the test face. The recognition rate of HMM approach is 87 percent using ORL database consisting of 400 images of 40 individuas. A pseudo D HMM [8] was reported to achieve a 95 percent recognition rate in their preiminary experiments. Its cassification time and training time were not given (beieved to be very expensive). The choice of parameters had been based on subjective intuition. Geometrica feature matching techniques are based on the computation of a set of geometrica features from the picture of a face. The fact that face recognition is possibe even at coarse resoution as ow as 86 pixes [7] when the singe facia features are hardy reveaed in detai, impies that the overa geometrica configuration of the face features is sufficient for recognition. The overa configuration can be described by a vector representing the position and size of the main facia features, such as eyes and eyebrows, nose, mouth, and the shape of face outine. One of the pioneering works on automated face recognition by using geometrica features was done by Kanade [9] in 973. Their system achieved a peak performance of 75 percent recognition rate on a database of 0 peope using two images per person, one as the mode and the other as the test image. Godstein et a. [0] and Kaya and Kobayashi [8] showed that a face recognition program provided with features extracted manuay coud perform recognition apparenty with satisfactory resuts. Brunei and Poggio [] automaticay extracted a set of geometrica features from the picture of a face, such as nose width and ength, mouth position, and chin shape. There were 35 features extracted to form a 35 dimensiona vector. The recognition was then performed with a Bayes cassifier. They reported a recognition rate of 90 percent on a database of 47 peope. Cox et a. [] introduced a mixture-distance technique which achieved 95 percent recognition rate on a query database of 685 individuas. Each face was represented by 30 manuay extracted distances. Manjunath et a. [3] used Gabor waveet decomposition to detect feature points for each face image which greaty reduced the storage requirement for the database. Typicay, feature points per face were generated. The matching process utiized the information presented in a topoogica graphic representation of the feature points. After compensating for different centroid ocation, two cost vaues, the topoogica cost, and simiarity cost, were evauated. The recognition accuracy in terms of the best match to the right person was 86 percent and 94 percent of the correct person's face was in the top three candidate matches. In summary, geometrica feature matching based on precisey measured distances between features may be most usefu for finding possibe matches in a arge database such as a mug shot abum. However, it wi be dependent on the accuracy of the feature ocation agorithms. Current automated face feature ocation agorithms do not provide a high degree of accuracy and require considerabe computationa time. A simpe version of tempate matching is that a test image represented as a two-dimensiona array of intensity vaues is compared using a suitabe metric, such as the Eucidean distance, with a singe tempate representing the whoe face. There are severa other more sophisticated versions of tempate matching on face recognition. One can use more than one face tempate from different viewpoints to represent an individua's face. A face from a singe viewpoint can aso be represented by a set of mutipe distinctive smaer tempates [4], []. The face image of gray eves may aso

4 GAO AND LEUNG: FACE RECOGNITION USING LINE EDGE MAP 767 be propery processed before matching [5]. In [], Brunei and Poggio automaticay seected a set of four features tempates, i.e., the eyes, nose, mouth, and the whoe face, for a of the avaiabe faces. They compared the performance of their geometrica matching agorithm and tempate matching agorithm on the same database of faces which contains 88 images of 47 individuas. The tempate matching was superior in recognition (00 percent recognition rate) to geometrica matching (90 percent recognition rate) and was aso simper. Since the principa components (aso known as eigenfaces or eigenfeatures) are inear combinations of the tempates in the data basis, the technique cannot achieve better resuts than correation [], but it may be ess computationay expensive. One drawback of tempate matching is its computationa compexity. Another probem ies in the description of these tempates. Since the recognition system has to be toerant to certain discrepancies between the tempate and the test image, this toerance might average out the differences that make individua faces unique. In genera, tempate-based approaches compared to feature matching are a more ogica approach. In summary, no existing technique is free from imitations. Further efforts are required to improve the performances of face recognition techniques, especiay in the wide range of environments encountered in rea word. Edge information is a usefu object representation feature that is insensitive to iumination changes to certain extent. Though the edge map is widey used in various pattern recognition fieds, it has been negected in face recognition except in recent work reported in []. Face recognition empoying the spatia information of edge map associated with oca structura information remains an unexpored area. This paper proposed a nove approach that expoits such information. The eigenface technique provides a compact representation of the human face which is optima for face reconstruction. It is one of the most thoroughy investigated approaches and has demonstrated exceent performance. Hence, this technique is used, in this study, as a baseine for recognition performance comparison. 3 LINE EDGE MAP Cognitive psychoogica studies [48], [49] indicated that human beings recognize ine drawings as quicky and amost as accuratey as gray-eve pictures. These resuts might impy that edge images of objects coud be used for object recognition and to achieve simiar accuracy as gray-eve images. TakaÂcs [] made use of edge maps, which was motivated by the above finding, to measure the simiarity of face images. The faces were encoded into binary edge maps using Sobe edge detection agorithm. The Hausdorff distance was chosen to measure the simiarity of the two point sets, i.e., the edge maps of two faces, because the Hausdorff distance can be cacuated without an expicit pairing of points in their respective data sets. The modified Hausdorff distance in the formuation of h A; B ˆ N a PaA min bbjja bjj was used, as it is ess sensitive to noise than the maximum or kth ranked Hausdorff distance formuations. A 9 percent accuracy was achieved in their experiments. TakaÂcs argued that the process of face recognition might start at a much earier stage and edge images can be used for the recognition of faces without the Fig.. An iustration of a face LEM. invovement of high-eve cognitive functions. This is in accordance with the psychoogica reports of [48], [49]. However, the Hausdorff distance uses ony the spatia information of an edge map without considering the inherent oca structura characteristics inside such a map. Brunei and Poggio [] argued that successfu object recognition approaches might need to combine aspects of feature-based approaches with tempate matching method. This is a vauabe hint for us when proposing a Line Edge Map (LEM) approach which extracts ines from a face edge map as features. This approach can be considered as a combination of tempate matching and geometrica feature matching. The LEM approach not ony possesses the advantages of feature-based approaches, such as invariant to iumination and ow memory requirement, but aso has the advantage of high recognition performance of tempate matching. The above three reasons together with the fact that edges are reativey insensitive to iumination changes motivated this research. A nove face feature representation, Line Edge Map (LEM), is proposed here to integrate the structura information with spatia information of a face image by grouping pixes of face edge map to ine segments. After thinning the edge map, a poygona ine fitting process [4] is appied to generate the LEM of a face. An exampe of a human fronta face LEM is iustrated in Fig.. The LEM representation, which records ony the end points of ine segments on curves, further reduces the storage requirement. Efficient coding of faces is a very important aspect in a face recognition system. LEM is aso expected to be ess sensitive to iumination changes due to the fact that it is an intermediate-eve image representation derived from oweve edge map representation. The basic unit of LEM is the ine segment grouped from pixes of edge map. In this study, we expore the information of LEM and investigate the feasibiity and efficiency of human face recognition using LEM. A nove Line Segment Hausdorff Distance

5 768 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 6, JUNE 00 Fig.. Line dispacement measures. (LHD) measure is then proposed to match LEMs of faces. LHD has better distinctive power because it can make use of the additiona structura attributes of ine orientation, ine-point association, and number disparity in LEM, i.e., it is not encouraged to match two ines with arge orientation difference, and a the points on one ine have to match to points on another ine ony. 4 LINE SEGMENT HAUSDORFF DISTANCE A nove disparity measure, Line Segment Hausdorff Distance (LHD), is designed to measure the simiarity of face LEMs. The LHD is a shape comparison measure based on LEMs. It is a distance defined between two ine sets. Unike most shape comparison methods that buid a one-to-one correspondence between a mode and a test image, LHD can be cacuated without expicit ine correspondence to dea with the broken ine probem caused by segmentation error. The LHD for LEM matching is more toerant to perturbations in the ocations of ines than correation techniques since it measures proximity rather than exact superposition. Given two LEMs M ˆfm ;m ;...;m p g (representing a mode LEM in the database) and T ˆft ;t ;...;t q g (representing an input LEM), LHD is buit on the vector d * m i ;t j that represents the distance between two ine segments m i and t j. The vector is defined as 3 d m i ;t j * d m i ;t j ˆ d == m i ;t j ; d? m i ;t j where d m i ;t j, d == m i ;t j, d? m i ;t j are the orientation distance, parae distance, and perpendicuar distance, respectivey. A these three entries are independent and defined as d m i ;t j ˆ f m i ;t j : d == d? m i ;t j m i ;t j ˆ? : ˆ min == ; == : 3 Fig. 3. Choices of rotation. (a) Two ines to be measured. (b) Rotate the shorter ine. (c) Rotate the onger ine. (d) Rotate both ines haf of their ange difference in opposite direction. Soid ines represent ines before rotation. Dashed ines represent ines after rotation. The ine with arrows iustrates the ange difference of the two segments. m i ;t j computes the smaest intersecting ange between ines m i and t j. f is a noninear penaty function to map an ange to a scaar. It is desirabe to ignore sma ange variation but penaize heaviy on arge deviation. In this study, the quadratic function f x ˆx =W is used, where W is the weight to be determined by a training process. The designs of the parae and perpendicuar dispacements can be iustrated with a simpified exampe of two parae ines, m i and t j, as shown in Fig.. d == m i ;t j is defined as the minimum dispacement to aign either the eft end points or the right end points of the ines. d? m i ;t j is simpy the vertica distance between the two ines. In genera, m i and t j woud not be parae, but one can rotate the shorter ine with its midpoint as rotation center to the desirabe orientation before computing d == m i ;t j and d? m i ;t j. The shorter ine is seected to rotate because this woud cause ess distortion to the origina ine pair, as iustrated in Fig. 3. In order to cater for the effect of broken ines caused by segmentation error and aeviate the effect of adding, missing, and shifting of feature points (i.e., end points of ine segments) caused by inconsistency of feature point detection, the parae shifts == and == are reset to zero if one ine is within the range of the other, as shown in Fig. 4. Finay, the distance between two ine segments m i and t j is defined as r d m i ;t j ˆ m i ;t j d == m i ;t j d? m i ;t j : 4 d A primary ine segment Hausdorff distance (plhd) [3] is defined as H plhd M ;T ˆ max hm ;T ;h T ;M ; 5 where hm ;T ˆ P m i M m i X m i min d m m i t M j T i ;t j and m i is the ength of ine segment m i. In the cacuation of (6), the distance contribution from each ine is weighted by 6

6 GAO AND LEUNG: FACE RECOGNITION USING LINE EDGE MAP 769 a high confident ine.ahigh confident ine ratio R of an image is defined as the ratio of the number of high confident ine N hc to the tota ine number in the LEM N tota. R ˆ Nhc N tota : Hence, a compete version of LHD integrated with number disparity is defined as (8) by taking the effect of simiarity neighborhood into account. q H LHD M;T ˆ HpLHD M;T W nd n ; 8 7 Fig. 4. A cases with d == m; t ˆ0. its ength. In the definition of (), it can be found that the dispacement distance (Fig. ) depends on the smaer distance between the eft/right end points of the two ine segments to be matched, which means the measure ony refects the smaest shift of the two ine end points. If one of the ine end points is consistenty detected and the other one shifts, the dispacement distance is amost zero no matter how far the other end point shifts. This heps to aeviate the probem of shifting feature points. However, the design of plhd sti has the foowing weakness. Suppose T is the LEM of a test image to be matched and t i is a ine segment in T, M c is the corresponding identica mode of T in the database, and M n is a nonidentica mode of T in the database. If the corresponding ine of t i in M c is missing because of segmentation error, plhd wi take the nearest ine, m cn M c, as the corresponding ine of t i. And d m cn ;t i ˆmin mc M c d m c ;t i is used in the cacuation of plhd. Simiary, the nearest ine (m nn M n )oft i in M n is considered as the correspondent ine of t i, though M n and T are different objects. It is possibe to have d m cn ;t i >> d m nn ;t i for the matching of compicated and simiar objects such as faces. This kind of missing ines can cause arger disparity between T and M c than that between T and M n, though both m cn and m nn are actuay not the corresponding ine segment of t i and both d m cn ;t i and d m nn ;t i are reativey arge. This may cause mismatch. The number of corresponding ine pairs between the input and the mode is another measure of simiarity. The number of corresponding ine pairs between two identica images shoud be arger than that between two images of different objects. Hence, the probem mentioned above can be aeviated by introducing this number information into plhd measure. Assume that, for each ine t j in the test LEM T, its corresponding ine m i in mode M of the identica face shoud ocate near t j because the test image and the mode have been aigned and scae normaized by preprocessing before matching. Therefore, a position neighborhood N p and an ange neighborhood N a are introduced. Simiarity neighborhood N s is a combination of N p and N a as: N s ˆ N p \ N a : If at east one ine in mode M ocates within the simiarity neighborhood of a ine t j in test LEM T (that is, ocates in a given position neighborhood of t j and their ange difference is aso within the given ange neighborhood), it is most ikey for t j to find a correct corresponding ine among those ines inside the simiarity neighborhood. This ine t j is named as where H plhd M;T is the plhd defined in (5) and W n is the weight of number disparity D n. The number disparity is defined as the average ratio, the number of ines ocated outside the simiarity neighborhood to the tota ine number, of the two LEMs to be compared as D n ˆ R M R T ˆ R M R T ; 9 where R M and R T are the high-confident ine ratio of the mode and the test LEMs, respectivey. One way to determine the parameters W;W n ;N p ;N a in the LEM face recognition system using LHD is to seect the vaues with the smaest error rate of face matching using a typica database. We use simuated anneaing [9], [30] to perform the goba minimization of the error rate of face identification. Simuated anneaing is a we-known stochastic optimization technique where, during the initia stages of the search procedure, moves can be accepted which increase the objective function. The objective is to do enough exporation of the search space before resorting to greedy moves in order to avoid oca minima. Candidate moves are accepted according to probabiity p as p ˆ e Err t ; 0 where Err is the error rate of face identification and t is the temperature parameter which is adjusted according to certain cooing schedue. Exponentia cooing schedue [3] is adopted in this work. The face database [33] from the University of Bern is used as training data and W ˆ 30;W n ˆ 5;N p ˆ 6;N a ˆ 30 are obtained. Since the parameter seection process is conducted offine, it is worth spending a substantia effort to design the best LHD format (in the sense of minimizing error rate). This can resut in optima performance of the proposed face recognition system. 5 EXPERIMENTAL RESULTS A thorough system performance investigation, which covers a conditions of human face recognition, has been conducted. They are face recognition under. controed condition and size variation,. varying ighting condition, 3. varying facia expression, and 4. varying pose. The system performances are compared with the eigenface method [7], the edge map approach [], and reported experimenta resuts of other methods studied in [3]. In this study, three face databases were tested. The database from the University of Bern [33] (Fig. 5) was used

7 770 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 6, JUNE 00 Fig. 5. An exampe pair of faces from Bern University [33]. Fig. 6. An exampe pair of faces from the AR face database [34]. The two faces were taken with a two week interva. TABLE Face Recognition Resuts of Edge Map (EM) [], Eigenface (0-Eigenvectors), and LEM to examine the system performances under controed/idea condition and head pose variations. The database contains fronta views of 30 peope. Each person has 0 gray-eve images with different head pose variations (Two frontoparae pose, two ooking to the right, two ooking to the eft, two ooking downwards, and two ooking upwards). The AR face database [36] (Fig. 6) from Purdue University was used to evauate the system performances under controed/idea condition, size variation, varying ighting condition, and facia expression. The database contains coor images corresponding to 6 peope's faces (70 men and 56 women). However, some images were found ost or corrupted after downoading through Internet. There were sets of usabe images (6 men and 5 women). No restrictions on wear (cothes, gasses, etc.), make-up, hairstye, etc., were imposed to the participants. The Yae face database [35] was aso tested in this work in order to compare the proposed approach to methods studied in [3]. The database, constructed at the Yae Center for Computationa Vision and Contro, is composed of 65 images of 5 subjects. The images demonstrate variations in ighting condition (eft-ight, center-ight, right-ight), facia expression (norma, happy, sad, seepy, surprised, and wink), and with/without gasses. In a the experiments, preprocessing to ocate the faces was appied. Origina images were normaized (in scae and orientation) such that the two eyes were aigned roughy at the same position with a distance of 80 pixes. Then, the facia areas were cropped into the fina images for matching. Sampe cropped images can be found in Fig Face Recognition under Controed/Idea Condition and Size Variation The face images under controed condition in the database of Bern University and AR database were used to evauate the performance of the proposed approach. Two exampe pairs of the face images in the two databases are iustrated in Figs. 5 and 6. The recognition resuts are summarized in Tabe. It is found that the LEM approach performed better than the edge map and the eigenface methods. LEM and Eigenface achieved 00 percent accuracy for identifying faces in the database of Bern University. However, LEM significanty outperformed Eigenface on the AR face database. Detaied eigenface data were tabuated in Tabe.

8 GAO AND LEUNG: FACE RECOGNITION USING LINE EDGE MAP 77 TABLE Performance Comparison on the AR Database The performance of the eigenface method depends on the number of eigenvectors m. If this number is too sma, important information about the identity is ikey to be ost. If it is too high, the weights corresponding to sma eigenvaues might be noises. The number of eigenvectors m is imited by the rank of the training set matrix. One hundred and tweve is the upper bound of m in the experiment of the AR database and, thus, percent is the best performance that eigenface can achieve here. One way to interpret this is that the eigenface approach wi work we as ong as the test image is ªsimiarº to the ensembe of images used in the cacuation of eigenface [37]. And the training set shoud incude mutipe images for each person with some variations [7] to obtain a better performance. Here, ony one image per person was used for training and the other is used for testing. Another reason is the difference between two identica faces is arger in the AR database than that in the database of Bern University. In particuar, the iuminations of the input and the mode are sighty different (Fig. 6). This might be the major negative impact to the eigenface approach as compared with LEM since LEM is reativey insensitive to iumination changes. In the LEM method, the proposed LHD dissimiarity measure is aso evauated and compared with the plhd definition. The experimenta resuts are summarized in Fig. 7 with three cassification conditions (top, top3, and top5). In the top cassification, the correct match is ony counted when the best matched face from modes is the identica face of the input. In the top3 or top5 cassification, the correct match is counted when the identica face of the input is among the best 3 or 5 matched faces from modes, respectivey. It was found that for the top match, LEM with plhd performed better than edge map with MHD [] by 5.36 percent, and LEM with LHD coud further improve the performance of plhd by.68 percent, i.e., it correcty identified percent of the input faces. A sensitivity anaysis to size variation was conducted using the AR database. The size variation was generated by appying a random scaing factor, which was uniformy distributed within [-0 percent, +0 percent], to the test Fig. 7. Recognition resuts on the AR face database. images. Four faces with different sizes were generated from each test image. Thus, we had 448 test faces in tota with size variations ranging from -0 percent to +0 percent. The sizes of the mode images were not changed. The experimenta resuts are tabuated in Tabe 3. The resuts show that the edge map approach is sighty more sensitive to size variation than the eigenface approach. The LEM with plhd performs better than the eigenface approach in the top match. The proposed LEM with LHD, which outperformed the eigenface approach by an accuracy increment of.6 percent, is much more robust to size variations than a the others in the experiments. This is a very attractive property that can aeviate the difficuty of precisey ocating faces in the previous face detection stage. 5. Face Recognition under Varying Lighting Conditions Ideay, an object representation empoyed for recognition shoud be invariant to ighting variations. It has been shown theoreticay that, for the genera case, a function invariant to iumination does not exist [38]. Edge maps can serve as robust representations to iumination changes for some casses of objects with sharp edges, such as computers, tabes, etc. However, for other objects, such as faces, part of the edges can not be obtained consistenty. It can be shown theoreticay that edges on a smooth surface are not stabe with changes in the ighting direction [39]. The LEM is an intermediate-eve image representation derived from ow-eve edge map representation. The basic unit of LEM is the ine segment grouped from pixes of the edge map. It remains an open question whether LEM, edge map, and other possibe Fig. 8. Sampe cropped images of mode (eftmost) and test faces (under varying ighting and expression).

9 77 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 6, JUNE 00 TABLE 3 Recognition Resuts with Size Variations representations provide an iumination-insensitive representation for face recognition. The issue addressed in this section is whether the LEM representation is sufficient or how we it performs for recognizing faces under varying ighting condition. To answer this question, an empirica study was performed to evauate the sensitivity of the LEM, and the edge map and the eigenface representations to this appearance change through the performances of face recognition systems. The experiment was designed using face images taken under different ighting conditions from Purdue University (Fig. 8). The faces in neutra expression with background iumination (the eftmost image in Fig. 8) were used as singe modes of the subjects. The images under three different ighting conditions were used as test images. There are a tota of modes (representing individuas) and 336 test images. Note that the experiment was based on a singe mode view. The experimenta resuts with three different ighting conditions are iustrated in Tabe 4. These experiments revea a number of interesting points:. In a the three experiments, the LEM consistenty performed better than the edge map approach with an improvement of percent in recognition rate. The eigenface approach performed very bady in these conditions. For the eigenface method, it has been suggested that the first three principa components are the primary components responding sensiby to ighting variation. Thus, the system error rate can be reduced by discarding these three most significant principa components [3]. Though the accuracy of the eigenface approach increased without using the first three eigenvectors, the LEM sti significanty outperformed it.. The variations of ighting condition did affect the system performance. Nevertheess, the recognition rates of the LEM approach were sti high and acceptabe. Note that the recognition rates of LEM, when ony one ight was on, stayed as high as 9.86 and 9.07 percent, respectivey. In other words, the effect on recognition rates when one ight was on created ony 3.57 to 5.36 percent decreases in recognition TABLE 4 Recognition Resuts under Varying Lighting Conditions ªw/o st 3º stands for without the first three eigenvectors.

10 GAO AND LEUNG: FACE RECOGNITION USING LINE EDGE MAP 773 TABLE 5 Recognition Resuts under Different Facia Expressions accuracy. The arge number of subjects, compared with the database used in [3], [40], made it especiay interesting. These resuts indicate that the proposed LEM together with LHD provide a nove face recognition soution which is insensitive to varying ighting condition to certain extent. 3. It was found that the recognition rates with eft ight on were aways higher than that with right ight on. This coud be due to the fact that the iumination on faces from the right ight was sighty stronger than that from the eft ight. 4. When both ights were on, the error rates became much higher than that of ony one ight on. This evidence shows that LEM (and edge map) woud sti be affected by extreme ighting condition variations, such as overiumination, though it is insensitive to certain extent. The overiumination woud cause strong specuar refection on the face skin (it is no onger a Lambertian surface). Therefore, the shape information on faces woud have been suppressed or ost which coud resut in the increase of the error rate of cassification. 5.3 Face Recognition under Facia Expression Changes Simiar experiments were conducted to evauate the effects of different facia expressions (smie, anger, and scream) on the system performances. The face images of different facia expressions from Purdue University were used in the experiments (Fig. 8). The faces in neutra expression (the eftmost image in Fig. 8) were used as singe modes of the subjects. Totay, there were modes (representing individuas) and 336 test images. The experimenta resuts on faces with smie, anger and scream expressions were summarized in Tabe 5. The smie expression caused the recognition rate to drop by 7.86 percent as compared to neutra expression in Tabe, whie the anger expression caused ony a 3.57 percent drop of the rate. This is because the anger expression had produced ess physica variation from neutra expression than the expression of smie. The scream expression coud be the extreme case of deformation among various human facia expressions, i.e., most facia features had been distorted. The LEM performed better than the edge map approach in a three experiments. However, the eigenface approach is found the east sensitive to facia expression changes. Without using the first three significant eigenvectors, the method to reduce the negative effect of ighting condition variation, degraded the accuracy under facia expression changes. 5.4 View-Based Identification Experiment and Comparison with Existing Approaches In [3], Behumeur et a. designed tests to determine how different face recognition methods compared under a different range of conditions and presented the error rates of five different face recognition approaches (eigenfaces, eigenfaces without the three most significant principa components, correation, inear subspace, and fisherface) using the Yae face database [35]. Experiments were performed using a ªeaving-one-outº strategy: The modes were constructed with a images of a subject except one image that was used as input. The same strategy and database were adopted here in order to compare the LEM and the edge map approaches to methods studied in [3]. For comparison purposes, the images were simiary processed as stated by Behumeur et a. to generate cosey cropped images. Fig. 9 dispays sampes of cropped images of one subject from the database. The experimenta resuts together with the resuts conducted by Behumeur et a. were summarized in Tabe 6. A rough comparison of the experimenta resuts shows that LEM is superior to a the methods except the Fisherface method (Tabe 6). It is worth highighting that, though the performance of the edge map approach is the worst among a the methods in Tabe 6, LEM has greaty improved the system performance to ranks. Note that LEM is a more compact

11 774 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. 4, NO. 6, JUNE 00 Fig. 9. Sampe cropped faces used in our experiment. (ess storage and computationa requirement) representation than edge map. The resuts indicate that LEM performed much better than the eigenface method, the widey used baseine, and sighty better than Eigenface without the first three eigenvectors. Fisherface is specificay designed and ony vaid for appications of mutipe modes per person. By compicated computation, it maximizes the difference of the between-person variation and the within-person variation. The test and database (eave-one-out test on 5 individuas and images/person) are ªideaº for Fisherface, whereas a the other six methods that can be appied on singe mode recognition do not get any favor. 5.5 Face Recognition under Varying Poses The face database from Bern University is used to evauate the system performance on face images of different poses. One fronto-parae face per person was used as the mode. The system was tested using the eight poses ooking to the right, eft, up, and down for each person. There are 40 test images in tota. The recognition resuts are summarized in Tabe 7. It can be observed that pose variations degrade the recognition rate of a the three investigated methods, but LEM is the most robust to pose variation, whie the edge map approach performed the worst. Note that 30 is the maximum number of eigenvectors that can be used for the eigenface approach in this experiment. Thus, 65. percent is the best average performance that the eigenface approach can achieved here. 5.6 Storage and Computationa Compexity The storage requirement of LEM is anayzed and compared with edge map based on the fronta face database from Bern University. The data sizes in the experiment are isted in Tabe 8. On average, a face can be represented by an LEM with 93 feature points. Compared with the edge map, LEM requires ony 8.5 percent of its storage space. The computationa compexities of LHD and MHD [] are of the order O k m t and O k p m p t p, respectivey. m and t are the ine segment numbers of mode and test LEMs, whie m p and t p are pixe numbers of mode and test edge maps. k TABLE 6 ªLeave-One-Outº Test of Yae Face Database The vaues with * are from [3] TABLE 7 Face Recognition Resuts under Pose Different Variations

12 GAO AND LEUNG: FACE RECOGNITION USING LINE EDGE MAP 775 TABLE 8 Average Storage Requirement of Faces in the Experiments TABLE 0 Comparison of Storage Requirements and k p are the time to compute d m i ;t j in LHD and the Eucidean norm jjm p i tp jjj in MHD. Tabe 9 shows the rea average computationa time of MHD and LHD on faces of [33]. The experiments were conducted on a SGI Octane workstation with 300MHz CPU and 5MB RAM. The computationa time for LHD is ess than 50 percent of MHD. Since the cacuation of d m i ;t j spends much more time than a simpe Eucidean distance cacuation in MHD (that is, k is arger than k p ), O k m t O k p m p t p > O m t O m p t p : With some acceeration techniques (such as hardware acceeration, ook up tabe), the LHD computationa time can be further reduced by minimizing k. When k k p, the LHD computationa time coud be reduced to ony about 0 percent of MHD m t < 8:5% < 0% : m p t p This is the idea upper bound of computationa time decrement for LHD with respect to MHD. The storage requirement for the eigenface approach is Nm dm (Tabe 0), where N is the number of faces, m is the number of eigenvectors, and d is the dimensionaity of the image. The storage demand of LEM is Nn, where n is the average number of feature points. n is content-based, whereas m is fixed. Usuay, LEM demands more storage space than the eigenface approach for arge N as n is arger than m (not aways). In this experiment, n ˆ 93 and m ˆ. Each feature point is represented by two 8-bit integers and each eigenvaue is a 6-bit foat number. But, LEM does not need to store any vector of size d, whereas the eigenface approach must store the projection matrix (m vectors of dimension d). d is ˆ 5; 600 in this experiment. The computationa compexity incudes three aspects, i.e., matching time, training time, and updating time. The matching time is the most important one for arge database searching. LEM requires more matching time than the eigenface approach as n>m. However, the eigenface approach demands a substantia amount of training time TABLE 9 Average Computationa Time of MHD and LHD on [33] in the order of O N 3 N d to obtain a satisfactory projection matrix. When a new individua is added into the database, the projection matrix must be recomputed. This incrementa update retraining of O N 3 N d is another expensive operation that the LEM approach does not need. This retraining can be avoided by assuming that the new images do not have a significant impact on the eigenfaces, that is, just cacuate the weights for the new images using the od projection matrix. This is ony vaid if the system was initiay ªideayº trained. 6 FACE PREFILTERING In a face identification system, searching is the most computationay expensive operation due to the arge number of images avaiabe in the database. Therefore, efficient search agorithms are a prerequisite of identification systems. In most systems, face features are extracted/ coded offine from the origina images, and stored in the face feature database. In identification, the same features are extracted from the input face and the features of the input images are compared with the features of each mode image in the database. Apart from adopting a fast face matching agorithm, a prefitering operation can further speed up the search by reducing the number of candidates and the actua face feature matching, LEM matching in this work, shoud ony be carried out on a subset of target images. Due to the high simiarity of human faces, fast prefitering based on image features is a very difficut task and is usuay negected or avoided. This study is beieved to be the first piece of work on face image prefitering. Prefitering feature seection is critica because it significanty affects the remaining aspects of the system design and greaty determines the fitering capabiity of the system. For indexing visua features, a common approach is to obtain numerica vaues of n features and then represent the image or object as a point in the n-dimensiona space. In this work, a D vector derived from the LEM of a face is proposed as the face prefitering signature. This signature is proven to be abe to fiter out certain percent of candidates whie preserving a very ow fase negative rate though it is a very ow dimensiona vector and human faces are highy simiar. Experimenta resuts show that.0 percent of the candidates can be fitered out whie the fase rejection rate was 0.89 percent using the AR face database of Purdue University. The experiments on the face database of Bern University, which contains fewer variations between the two identica faces, demonstrated much better fitering rate. The system correcty fitered out 5.95 percent of candidates, whie the fase rejection rate was zero.