Recognition of Occluded Faces Using an Enhanced EBGM Algorithm.

Transcription

1 Recognition of Occluded Faces Using an Enhanced EBGM Algorithm Badr Mohammed Lahasan bmo12 Ibrahim Venkat Syaheerah Lebai Lutfi Abstract A new approach to recognize occluded faces is presented in this paper to enhance the conventional Elastic Bunch Graph Matching (EBGM) technique In the conventional EBGM approach, facial landmarks need to be chosen manually in the initial stage and a single graph per face had been modeled Our proposed approach intuitively fuses a Harmony search based optimization algorithm over the EBGM approach to automatically choose optimal land marks for a given face Further, instead of using a single graph, we deploy component level sub-graphs and locate optimal landmarks by maximizing the similarity between each of the sub-graphs Experimental results show that the proposed automatic method achieves an improved recognition rate when compared to the conventional EBGM approach Keywords: Face recognition, Elastic Bunch Graph Maching, Harmony Search, Similarity measure, Facial occlusion I INTRODUCTION Performance of face recognition systems are easily prone to facial occlusions such as sunglasses, hats, scarves, beards and so on [1] Face recognition systems form vital part of biometric oriented video surveillance systems and increasingly getting deployed in operational environments where facial occlusions are inevitable In order to enable such crucial automation based surveillance systems, modern face recognition approaches need to be fully automatic without imposing any human intervention such as manual selection of facial landmarks Though face recognition systems have crossed the experimental stage, to date the presence of facial occlusions degrades the performance of face recognition systems [2] This paper intends to propose a new approach to recognize occluded faces by enhancing the Elastic Bunch Graph Matching (EBGM), which is one of the state of the art face recognition techniques Specifically, this paper contributes to achieve the following goals : 1) Basically to transform the conventional semiautomatic EBGM approach into a fully automatic system 2) Incorporation of music inspired harmony search in order to intuitively determine optimal facial landmarks 3) Improving the recognition rate of the conventional EBGM approach by deploying a component level sub-graph mechanism /13/$3100 c 2014 IEEE In recent years, the problem of recognizing partially occluded faces has received considerable attention [3], [4], [5] Martinez [3] proposed a probabilistic face recognition approach that could recognize faces which are prone to uncertainty elements such as imprecise localization, partial occlusion and extreme expressions using a single training sample In his method, face images are analyzed locally in order to handle partial face occlusion among other variations Firstly, the face image is segmented into k local regions and an eigenspace has been constructed for each of these subregions Additionally, in order to gain robustness against expression variations, a weighting scheme associated with the emotional state of local regions has been considered According to the author the major limitation of the approach lies in the manual localization of local features In another approach, Wright et al [4] introduced a partition based Sparse Representation Classification (SRC) method which is inspired by the idea of compressed sensing In this method, a face is first partitioned into blocks and an independent sparse representation for each block is computed This is followed by a general classification algorithm and a voting method to identify face images More interestingly, Wiskott et al [6] introduced a method in which the recognition system is trained by using a single image of each of subjects in a large facial database While this method offers a way to minimize certain variance by extracting concise face descriptions in the form of image graphs, it also suffers from a major drawback whereby manual landmark selection is required Very recently, Venkat et al [5] proposed a Psychophysically Inspired SImilarity MApping (PISIMA) model based on local facial components rather than holistic processing which proved to be an efficient face recognition approach to recognize occluded faces Based on evidences gathered from the psychological domain, the authors proposed a Bayesian network model to capture the local features entailed in horizontal subregions of facial images However this approach requires more than one sample per subject From the above case studies, it has been observed that component-based approaches offer viable alternatives to holistic approaches Their recognition power stems from their ability to intuitively represent faces locally These approaches have been proven to be more efficient than holistic approaches in recognizing faces under uncontrolled conditions such as occlusions, expressions and so on Along this line of componentbased face recognition approaches, we also intend to formulate the face recognition task in our proposed approach through the use of local features

2 The paper is organized as follows: The following section provides the necessary background of the EBGM and harmonic search algorithms Section III moves on to the framework of the proposed approach Next, Section IV reports the experimental results and the discussions on the findings Finally, we conclude the work and briefly discuss our future direction in Section V II BACKGROUND 1 EBGM Elastic Bunch Graph Matching (EBGM) approach presented by Wiskott et al [6] is one of the graph-based algorithms that has been successfully applied for the task of face recognition In this approach, salient fiducial points on the face such as eyes, mouth, nose etc, are chosen manually and described by sets of Gabor response components known as jets (as in Fig 2) Basically, face images stored in face databases carry certain variances among the facial images due to the diversity caused by genders, races, opened/closed eyes and the use of facial accessories such as sunglasses Because of these variances, fiducial points on the face images have to be selected manually A combination of these fiducial entities structures out what is called a Face Bunch Graph (FBG) The average values of node positions of FBG aids to estimate the node positions on the new image to create a face graph The face graph includes twenty five landmarks (see Fig 1) and they are located around the eyes, nose, mouth and the edge of the face Each landmark represents a node and gets connected with the neighbor nodes by its edges which represent the distances between them Each node labeled with a jet intend to describe the local feature around that particular node Subsequently the similarity between these jets and the corresponding jets of the FBG is calculated to refine and displace the node location in the new image, where the maximum jet similarities represent the location of the nodes Fig 1: Fiducial points graph example The basic steps in EBGM algorithm [7] are as follows: 1) Jets are manually selected to represent facial points (as in Fig 2) 2) A bunch graph is created where each node of the bunch graph corresponds to a facial landmark and contains a bunch of model jets extracted from the model images (as in Fig 3) 3) The landmark for each image is located Firstly, the new jet is extracted from the new image and J 11 J 15 J 13 J 12 J 14 J 21 J 25 J 22 J 23 J 31 J 24 J 35 Fig 2: Manually selected jets Fig 3: Bunch Graph displacement from actual location is estimated by comparing it to the most similar model jet from the corresponding bunch 4) A face graph is created for each image by extracting a jet for each landmark The graph contains the locations of the landmarks and the value of the jets The original image can then be discarded 5) Face similarity is computed as a function of landmarks and jet values After creating the FBG, the subsequent task is to find the fiducial points for a given new image This is achieved by comparing the new image against the FBG and finding the fiducial points with the highest similarity There are a few ways to do this, as proposed in [6], [7] Basically the idea is summarized below: The initial position of a new node is estimated by calculating the average distance between the previously located node on the FBG The location of the nose bridge can be estimated by using the eye coordinates and then this location can be further refined by comparing the Gabor jet extracted from the estimated point to a model jet from the bunch graph Once the location of the three landmarks (eyes and nose bridge) are known, the location of the eyebrow can easily be estimated By iterating this process, all landmark locations can be found The corresponding jet is extracted from the estimated location of the new image node The similarity with each jet in the FBG is calculated for that node The aim is to maximize the similarity with the FBG in a neighborhood Finally, the coordinates where the maximum similarity has been obtained are saved This procedure is iterated for the remaining nodes of the graph 2 Harmony Search Algorithm The music inspired Harmony Search Algorithm which is considered to be part of the evolutionary computing domain was introduced by Geem et al [8] in 2001 Though this technique is widely studied by the optimization community, it has wide applications in diverse fields such as scheduling, medical imaging and so on In general, the Harmony Search Algorithm uses the following four main steps: J 33 J 32 J 34

3 1 Specifying the Harmony Memory (HM): Here a few Harmony parameters must be specified viz, the Harmony Memory Size (HM S), which is the number of solution vectors in HM, Harmony Memory Considering Rate (HMCR), where HMCR [0, 1], Pitch Adjusting Rate (P AR), where P AR [0, 1], stopping criteria (number of improvisation (N I)), the amount of maximum change in pitch adjustment (f w) Once these parameters are specified, the HM will be filled by a randomly generated solution vector as shown in Eq 1 Fig 4: Image Segmentation HM= x 1 1, x 1 n f(x 1 ) (1) x hms 1 x hms n f(x n ) After the image is segmented, its Gabor wavelet transform is calculated as: ω i (X) = K i 2 σ 2 e K i 2 x 2 2σ 2 [e j K i x e σ2 2 ] (3) 2 Improvise a New Harmony: The new vector is generated based on the following rules as in [9] : HM consideration, pitch adjustment, and random consideration In Harmony memory consideration the values of the decision variables are inherited form the stored harmony memory vectors, this is the case when the random number [0, 1] is within the probability of HM CR; otherwise, there is also a possibility that random consideration rule is guaranteed If the value of new x has been selected for HM, then, pitch adjusting should be applied with probability P AR 3 Update the Harmony Memory: After the generation of the New Harmony memory vector, the associated fitness function is computed After that, the new vector will replace the worst harmony vector according to the fitness value Where ω i is a plane wave distinguished by the vector k i enveloped by a Gaussian function, where σ = 2π is the standard deviation of this Gaussian Five different frequencies, indexed v=0,,4 and eight orientations, indexed µ = 0, 1,, 7 are used The center frequency of i th filter is given by the characteristic wave vector: ( ) ( ) Kix Kv cos θ K i = = µ K iy K v sin θ µ Where K v = 2 v2 2 π, θ µ = µ π 8 Then, when the Gabor wavelet is calculated, five images are chosen (Fig 5 (b)-(f)) as models to optimize the similarity between the sub-graphs of that image corresponding to the sub-graphs of the reference image (first image in the database (Fig 5 (a)) by harmony search as follows: (4) 4 Evaluation of Solutions: While the current iteration does not exceed the maximum allowed number of improvisations, steps 2 and 3 are repeated Otherwise the iteration process will be terminated III THE PROPOSED MODEL In this study, all images are enhanced by Median Filtering and Histogram Equalization to remove the noise and enhance the contrast levels Then each image is segmented into six segments (Fig 4 ) and each segment represents a sub-graph instead of one whole graph as used by the standard EBGM and it is carried out as follows: Consider that the given image is represented by I(x, y) whose height and width are represented as H and W respectively We segment I by: S g = ( H/3, W/2 ) (2) Fig 5: Samples Images from ORL database A sub-graph is created for each segment of the model image by finding the optimal landmark which represents the maximum similarity between each sub-graph of the corresponding reference image by harmony search as follows: Initialize harmony memory with x vectors of numbers specified by the variable HMS, each vector contains j numbers (in this case j = 4) generated randomly and represent the

4 coordinates of landmarks by: x i (j) = rand(0, 1) (x i (j) Maxrange x i (j) Minrange ) x i (j) Minrange(5) The range for each number is determined based on segment size By applying the basic permutation and combination principles, all possible landmarks coordinates of n elements can be computed using: x = n r (6) where x is the number of (r) permutations Hence, sixteen landmark coordinates are formulated where n=4,r=2 as : (x 1 (1), x 1 (1)) (x 1 (1), x 1 (2)) (x 1 (1), x 1 (4)) (x 1 (2), x 1 (1)) (x 1 (2), x 1 (2)) (x 1 (2), x 1 (4)) (x 1 (4), x 1 (1)) (x 1 (4), x 1 (2)) (x 1 (4), x 1 (4)) (7) Then the similarity of these landmarks (nodes) of corresponding nodes of the reference image is calculated by: S i (j, k) = l v i,k v i,j l v i,k 2 l v i,j 2 (8) of reference image is calculated using Eq (8) and the overall similarity of this sub-graph is calculated using: SG = S (9) Where N 2 is the number of landmarks (nodes) of the subgraph And the overall similarity of k sub-graphs is given by: k 1 OS = SG (10) k Finally to recognize test images, the overall similarity of test image with all reference images are being sorted and ranked, so that the image with the highest similarity value is addressed as rank 1 A face image is correctly recognized if it belongs to rank 1 IV N 2 1 EXPERIMENTAL RESULTS AND DISCUSSIONS In order to do an experimental validation of the proposed approach, we have used the Olivetti Research Laboratory (ORL) face database [10], which consists of 400 face images Basically there are 40 subjects (4 female, 36 male) each of them with 10 face samples containing different facial expressions, slight pose variations (tilting and rotation up to 20 degrees), varying illumination and scaling conditions The first image (Fig 5) for each individual is chosen as the reference image and four images are chosen for test (Fig 7) where k is a jet of the model image segment, j is a jet of the reference image and l = [1,, 40] Logically, the landmarks which maximizes Eq (8) will be optimal After those optimal nodes for each sub-graph (Fig6) at each model images are found, the average value of the optimal node positions of each sub-graph is calculated to find the average node position to be used for the new test image Fig 7: Examples Of Test Images Fig 6: Optimal Sub-Graph When an optimal sub-graph is created by average optimal node of model images, the similarity between nodes of the sub-graph of the test image and the nodes of the sub-graph Each of the test images are further occluded by a square of N N at a random location while the reference image remains without occlusion Initially the first test image is occluded with a minor occlusion by setting N = 10 Then to study the viability of the proposed approach against major occlusions N is further set to 80, 90 and 100 in the rest of the test images The recognition performances of standard EBGM algorithm and the proposed method are plotted in Fig 8 in terms of a Cumulative Match Characteristic (CMC) graph As depicted in Fig 9, the proposed method has considerably outperformed the standard EBGM algorithm The proposed enhanced EBGM yielded a recognition rate of 8187% within the first five ranks This is significantly better than the 55% recognition rate yielded by the standard EBGM For the case of recognition performance within the first 10 ranks,the proposed method achieved a recognition rate of 906% and the standard EBGM achieved only a recognition rate of 661 % Further the overall performance of the two algorithms

5 Fig 8: Comparison of Enhanced EBGM with Standard EBGM [6] L Wiskott, J-M Fellous, N Kuiger, and C Von der Malsburg, Face recognition by elastic bunch graph matching, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol 19, no 7, pp , Jul 1997 [7] D S Bolme, Thesis on elastic bunch graph matching, PhD dissertation, Colorado State University Fort Collins, 2003 [8] J H L G V Geem, Z W Kim, A new heuristic optimization algorithm: Harmony search, Simulation, vol 76, pp 60 68, 2001 [9] C L P Y Geem, Z W Tseng, Harmony search for generalized orienteering problem: Best touring in china, vol 3612, pp , 2005 [10] The ORL Database of Faces, developed by AT&T Laboratories Cambridge shown by the CMC graph in Fig 8 clearly justifies the significant improvement achieved by the proposed approach when compared to the standard EBGM V CONCLUSION In this contribution, we have presented a novel fully automatic approach to enhance the standard EBGM face recognition technique One of the potential advantages of the proposed technique is that it does not rely on manual ground truth data used to represent facial landmark locations such as eyes, nose and so on Optimal landmark locations are automatically identified in the proposed technique by means of a Harmony search algorithm The viability of the proposed approach has been demonstrated against minor to major occlusions and compared with the standard EBGM technique It has been observed that the proposed component level approach achieves a significant recognition rate when compared to the standard EBGM algorithm In the near future, we would investigate the performance of the proposed approach with large datasets Also instead of synthetic occlusions, we would study the behavior of the proposed approach with faces containing real occlusions ACKNOWLEDGMENT This research work is supported by an ERGS grant (Account# 203/PKOMP/ ) awarded by the Ministry Of Higher Education(MOHE), Malaysia REFERENCES [1] R Min, A Hadid, and J Dugelay, Improving the recognition of faces occluded by facial accessories, pp , March 2011 [2] H m Ekenel and R Stiefelhagen, Why is facial occlusion a challenging problem? vol 5558, pp , 2009 [3] A M Martinez, Recognizing imprecisely localized, partially occluded, and expression variant faces from a single sample per class, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol 24, no 6, pp , Jun 2002 [4] J Wright, A Yang, A Ganesh, S Sastry, and Y Ma, Robust face recognition via sparse representation, Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol 31, no 2, pp , Feb 2009 [5] I Venkat, A T Khader, K Subramanian, and P D Wilde, Recognizing occluded faces by exploiting psychophysically inspired similarity maps, Pattern Recognition Letters, vol 34, no 8, pp , 2013