Evaluation of wavelet based linear subspace techniques for face recognition

Transcription

1 Hima Deepthi Vankayalapati Evaluation of wavelet based linear subspace techniques for face recognition Masterarbeit zur Erlangung des akademischen Grades Diplom Ingenieurin Studium Information Technology Alpen-Adria-Universität Klagenfurt Fakultät für Technische Wissenschaften Begutachter: Univ.-Prof. Dr.-Ing. Kyandoghere Kyamakya Institute for Smart System-Technologies Transportation Informatics Klagenfurt, im October 2008

2 Eidesstattliche Erklärung Ich erkläre ehrenwörtlich, dass ich die vorliegende wissenschaftliche Arbeit selbstständig angefertigt und die mit ihr unmittelbar verbundenen Tätigkeiten selbst erbracht habe. Ich erkläre weiters, dass ich keine anderen als die angegebenen Hilfsmittel benutzt habe. Alle aus gedruckten, ungedruckten oder dem Internet im Wortlaut oder im wesentlichen Inhalt übernommenen Formulierungen und Konzepte sind gemäß den Regeln für wissenschaftliche Arbeiten zitiert und durch Fußnoten bzw. durch andere genaue Quellenangaben gekennzeichnet. Die während des Arbeitsvorganges gewährte Unterstützung einschließlich signifikanter Betreuungshinweise ist vollständig angegeben. Die wissenschaftliche Arbeit ist noch keiner anderen Prüfungsbehörde vorgelegt worden. Diese Arbeit wurde in gedruckter und elektronischer Form abgegeben. Ich bestätige, dass der Inhalt der digitalen Version vollständig mit dem der gedruckten Version übereinstimmt. Ich bin mir bewusst, dass eine falsche Erklärung rechtliche Folgen haben wird. (Unterschrift) (Ort, Datum) i

3 Abstract Face recognition is a complex visual classification task which plays an important role in computer vision, image processing and pattern recognition. Research concerning the face recognition started nearly in 1960s. The initial research directions in 1960 s are based on locating the features such as eyes, ears, nose and mouth on the photograph and calculating the distance between the reference image and the stored images. In 1990 s, researchers introduced linear subspace techniques, statistics related techniques, to face recognition problems. The introduction of the linear subspace techniques is a milestone in the face recognition concept. The main objective of the thesis is to improve the recognition rate of existing face recognition methods for large databases with varying pose, expression and environmental conditions (lighting, rainy etc). The human skill of identifying thousands of people even after so many years with different aging, different light conditions and viewing conditions excited many researchers to focus in face recognition systems. Researchers have developed various biometric techniques to identify or recognize persons with their physical characteristics like finger, voice, face etc. These biometric techniques have their own advantages and drawbacks as well. Among all the biometric techniques face recognition has a distinct advantage of collecting the required data or image without individual cooperation. In this work, we are attempting to answer the following research questions: Is the face recognition system invariant to face expressions? Is the face recognition system invariant to environmental conditions (like background, climate changes)? Is the face recognition system invariant to viewing conditions? Is the face recognition system giving higher performance or high recognition rate for large databases? Is the wavelet based linear subspace technique a good alternative to perform face recognition tasks? What is the efficiency of the wavelet based linear subspace technique when compared with its counter parts? In this thesis, different linear subspace techniques like principal component analysis (PCA), independent component analysis (ICA) and linear discriminant analysis (LDA) ii

4 are studied and implemented. Experiments are conducted to improve the performance of face recognition algorithms with varying side views, expression and illumination conditions. An emphasis is placed on the importance of testing the algorithms on a larger database in order to check their robustness. Among the three linear subspace techniques, LDA gives considerably better performance results on larger databases. However, the LDA recognition rate is not sufficient for real time applications, due to it s inability to extract the nonlinear features. In real time applications, we need to recognize the person among the large stored images with different background and environmental conditions. In this thesis, wavelets are proposed to extract the nonlinear features in images before the linear subspace techniques. Wavelets use nonlinear bases and give both the spatial and frequency information of the images at the same time. The results obtained show that the proposed wavelet based linear discriminant analysis (WLDA) performs better than the LDA in larger databases. Finally, the proposed WLDA technique is implemented in Simulink and on FPGA as a soft core with the help of DSP Builder from Altera. Keywords:Face Recognition, Linear subspace techniques, Wavelets. iii

5 Acknowledgments I would like to thank Univ.-Prof. Dr.-Ing. Kyandoghere Kyamakya for giving me an opportunity to work in the institute and for his constant encouragement and his constant support, advice throughout my work. It was a great learning experience working with the Institute for Smart System-Technologies team and especially with the other students working in the institute. My special thanks to Dr.-Ing Chedjou and Dr. Tuan for their valuable suggestions. Last but not least, I am deeply indebted to my husband for his constant moral support and valuable suggestions at the right time.. iv

6 Contents List of Figures List of Abbreviations Nomenclature viii x xi 1 Motivation Motivation Problem statement Major contributions of the thesis Thesis outline Background Introduction Digital image Image-Scalar operations Image-Image operations Image statistics Histogram of the image Mean of the image Variance of the image Image acquisition Feature selection and extraction Feature selection Feature extraction Face Recognition Introduction History of face recognition Face recognition basics Face detection and recognition Face recognition approaches Feature based face recognition Model based face recognition Template based face recognition Appearance based face recognition v

7 3.2.5 Face recognition applications Linear subspace techniques Introduction Linear subspace techniques Image representation Mean image from the face database Covariance matrix from the face database Principal Component Analysis Calculation of mean and covariance matrix from the given database Calculation of eigenvalues and eigenvectors Formation of a feature vector Derivation of a new data set PCA example Eigen face method Face recognition using eigen face technique Linear Discriminant Analysis LDA approaches Fisher s linear discriminant technique Subspace LDA Direct LDA for face recognition Independent Component Analysis Definitions of linear ICA Objective or contrast function of ICA Optimization algorithms of ICA ICA based face recognition Comparison of linear subspace techniques Summary Nonlinear Analysis Introduction Kernel based face recognition Kernel PCA Kernel PCA for face recognition Kernel FLD for face recognition Kernel ICA for face recognition Disadvantages of kernel based nonlinear analysis Summary Wavelet Transform Introduction Difference between fourier analysis and wavelet analysis Wavelet transform classification Continuous wavelet transform Discrete wavelet transform Daubechies wavelet transform vi

8 6.5 Wavelet based linear subspace techniques Summary Distance Measures Introduction Distance measures for face recognition Euclidean distance Standardized euclidean distance Mahalanobis distance City Block distance metric Minkowski distance metric Cosine distance metric Face recognition evaluation Watch list ROC Verification ROC Summary Database Introduction ORL database FERET face database AR face database CVL face database Essex face94 database Summary Field Programmable Gate Array Introduction Cyclone II FPGA DSP Builder tool Summary Experimental results Eigen vectors versus Recognition rate Comparison of linear subspace techniques Wavelets based linear subspace techniques Different wavelet functions versus recognition rate Different wavelet coefficients versus recognition rate Effect of database size on recognition Importance of test image on the recognition rate Impact of database quality on recognition rate FPGAs deployment Conclusion Future work Bibliography 97 vii

9 List of Figures 2.1 Representation of the rectangular shape digital image into pixel P (X, Y ) Gray scale representation of the example image Color storage representation in the color images R, G and B color channels of the example color image Image scalar operations (a) Image from essex face94 database (b) Image after scalar operation Image image operations (a) First Lenna image (b) Second Lenna image with different pose Blended version of two different pose Lenna images Gray scale histogram of the example gray scale image Color image histogram of the example color image Feature selection (a) Selection of wrong feature (b) Selection of correct feature Comparison of biometric techniques [18] Classification of face recognition techniques Example model of the labeled graphs D Morphable face model method Linear subspace technique general algorithm Matrix representation of N images present in the database Principal component analysis (a) Given original data and (b) After applying PCA PCA classification of given data (a) Worse classification of data (b) The best classification of data Dimensionality reduction of the given data after applying PCA First eight eigen faces obtained from essex face94 database by using PCA Mean image obtained from essex face94 database by using PCA Principal component analysis algorithm for face recognition Flow chart of linear discriminant analysis algorithm Fisher s linear discriminant algorithm for face recognition Mean face obtained from essex face94 database by using FLD method Highest fisher faces obtained for essex face94 database by using FLD method General linear discriminant analysis algorithm Four images of X1 class Four images of X2 class Mean images of two classes, X1 and X2, and total mean image viii

10 4.17 Mean image obtained from essex face94 database by using ICA method Independent component analysis algorithm ICA basis obtained from essex face94 database by using ICA Comparison between different linear subspace techniques Example kernel function of the nonlinear data set Discrete time fourier analysis function on signals Discrete time wavelet analysis function on signals Representation of the filter bank in wavelets Spliting the signal spectrum by using subband coding Discrete wavelet transformation decomposition tree Discrete wavelet transformation reconstruction tree Haar wavelet transform Scaling and shifting of the haar wavelets Usage of wavelet in the feature extraction Wavelet coefficients decomposition in discrete wavelet transform Recognition accuracy of the standardized euclidean distance and the euclidean distance [3] Recognition accuracy of the mahalanobis and the euclidean distance [71] Recognition accuracy of the city block distance and the euclidean distance [71] Recognition accuracy of the cosine distance and the euclidean distance [3] Watch list receiver operating characteristic curve [71] Verification receiver operating characteristic curve [72] Sample images of a single person in the ORL database Sample images of a single person in the FERET database Sample images of a single person in the AR database Sample images of a single person in the CVL database Screen shot of the DSP Builder tool box in Simulink Signal compiler block in the DSP Builder tool box Example Simulink model by placing the signal compiler block Cyclone II DSP Board Library in the DSP Builder tool Importance of eigen vectors in linear subspace techniques Number of eigen values versus corresponding eigen value Performance comparison of different linear subspace techniques (PCA, ICA and LDA) Performance comparison of different wavelet functions db1 and db Performance comparison of different wavelet based linear subspace techniques (WPCA, WICA and WLDA) Performance comparison of WLDA and LDA Mean images of the ORL and the FERET databases by using PCA technique Eigen faces of the ORL and the FERET databases by using PCA technique 95 ix

11 x List of Abbreviations PCA ICA LDA FLD KPCA KICA KLDA WPCA WICA WLDA ROC ORL FERET CVC DSP FPGA HDL VHDL CPLD ASIC PLL LAB LE IP NIST DARPA Principal Component Analysis Independent Component Analysis Linear Discriminant Analysis Fisher s Linear Discriminant Kernel based Principal Component Analysis Kernel based Independent Component Analysis Kernel based Linear Discriminant Analysis Wavelet based principal component analysis Wavelet based independent component analysis Wavelet based linear discriminant analysis Receiver Operating Characteristics Olivetti Research Laboratories Face Recognition Technology Computer Vision Center Digital Signal Processors Field Programmable Gate Array Hardware Description Language Very high speed integrated circuit Hardware Description Language Complex programmable Logic devices Application Specific Integrated Circuit phase locked loops Logic Array Blocks Logic Elements Intellectual property The National Institute of Standards and Technology The Defense Advanced Research Projects Agency

12 Nomenclature A C D b g H h I J m i n c P b S b S i S m S w T u W fld W opt W pca The mixing matrix The covariance matrix The whitening transform matrix The high pass filter Differential entropy The low pass filter The mutual information The negentropy Mean image of the class i in the database Random noise The centered images matrix of the database Between class scatter matrix Sum of all the class covariance matrices present in the database The overall scattering matrix of the database Within class scatter matrix Transpose operation Independent source signal The transformation matrix obtained by FLD method The optimal transformation matrix The transformation matrix obtained by PCA method xi

13 Chapter 1 Motivation 1.1 Motivation Face recognition is a complex visual classification task. Research concerning the face recognition started nearly from 1960s [1]. The face plays a primary role in identifying a person. It is the more easily rememberable part of the human. We can recognize thousands of faces in our lifetime. Human face is a complex model because it contains almost the same feature set (eyes, lips, nose, etc) in every face. Eventhough the feature set is the same in each face, its properties are quite different in each face. The skill of human recognition is very robust because it is possible to identify a person even after so many years with different aging, different light conditions and viewing conditions [2]. Face recognition plays an important role in various applications (e.g. computer vision, image processing and pattern recognition). The ideal computer vision models work more or less like humans vision. In computer vision related applications, face recognition is useful in taking the decision based on the information present in the images or video. They are many computer vision based applications on image analysis like recognizing and tracking humans in some public and private areas. Even in future driver assistance systems, driver face observation will play an important role. The person identification and verification is one of the main issues in many security related applications like banking, border check, etc. In these applications persons must be recognized or identified. Researchers have developed various biometric techniques to identify or recognize the persons with their physical characteristics like finger, voice, face etc. There are advantages and drawbacks of the biometric techniques. Among all the biometric techniques face recognition has a distinct advantage of collecting the required data, image, without individual cooperation. The face recognition system is very useful in criminal identification. In this application, the images of a criminal can be stored in the face recognition system database. In recognition algorithms based on matching methods, image acquisition is one of the important tasks. The image of a person must be taken directly from a digital camera or from the video sequence such that it contains maximum possible information about that person. The images must be taken quickly with a small resolution or size in order to speed the algorithms. If we take the high resolution images it takes much time to recognize the per- 1

14 CHAPTER 1. MOTIVATION 2 sons. Then the matching algorithms compare the acquired image with the images in the database to identify the criminal. In real time face recognition, the system must analyze the images and recognize the person very fast. The face recognition system only recognizes the person stored in the database. 1.2 Problem statement In person identification and verification, face recognition is playing a key role. Research on face recognition started early 1960s. In that time, the face is identified with the location of the features present in the images. These features are eyes, nose, mouth, etc. In 1970 s Goldstein, Harmon and Lesk used 21 specific characteristics of the face such as hair color, lip thickness for face recognition. The measurements performed by the methods developed till the end of 1970 s are manual. The real time face recognition system performance depends upon the given input data or image [3]. The general problems faced in real time face recognition are: Images vary in different environmental conditions like lighting conditions, background variation and climate changes. If we take the image in darkness (or less lighting) or with high lighting, we cannot extract the correct features; so the recognition rate is less Images vary in different face expressions like smiling, tearing etc Images vary in different poses. If we take image with one pose, the database contains that person image in other pose. Then it is difficult to recognize the person Due to the variations underlined, a new information is added to the face image when compared to stored data base images. So we cannot compare the features of the test image and stored data set images. There are some techniques to overcome these image variation problems. To overcome image variation problems, Kirby and Sirovich introduced (in 1988) the principal component analysis, which is a statistical technique [1]. The introduction of the principal component analysis is a milestone in the face recognition concept and this leads to the development of various different algorithms for the face recognition based on statistics related techniques. Linear subspace techniques are statistical related techniques which are used to reduce the dimensionality and classify the given data. Principal Component Analysis (PCA), Independent Component Analysis (ICA) and Linear Discriminant Analysis (LDA) are mostly used in linear subspace techniques. For recognition algorithms based on linear subspace techniques, we create the test and training set databases. A training set database contains Images of the person with different poses, expression and environmental conditions Many number of persons (the number depends on the data base)

15 CHAPTER 1. MOTIVATION 3 A class contains one person with different face variations. The test database contains different images of the persons present in the training database. The linear subspace techniques are successful to some extent but they are not effective in large databases [4, 5, 6]. Eigen vectors are calculated using the statistical measures like mean and covariance. The eigen vector gives the variation in the image PCA uses second order statistics. Second order statistics give information about amplitude spectrums but not their phase spectrums. This makes PCA not effective with pose variant test images PCA is not using information about the class [4, 5, 6]. It takes all persons images as one class. Because of that it extracts only the similar features in all images, making it difficult to recognize the person PCA is able to recognize the faces with varying face expressions but not with varying poses PCA is not effective to recognize the person among large databases. Because of the lack of class information ICA tries to maximize the statistical independency between the images [7]. It looks for statistically independent and non gaussian components in the images The recognition rate is better in pose images because of the higher statistics In ICA, the fixed point algorithm is used to calculate the weight matrix [8, 3, 9]. It is an iterative process. So it takes long time to calculate the uncorrelated features With ICA, it is difficult to recognize the person among large databases because of the lack of class information and time factor LDA only uses second order statistics. The second order statistics gives information about amplitude spectrums but not their phase spectrums. This is explained by the variation of LDA with viewing conditions LDA uses information about the class [10]. It tries to maximize the between class variance and minimize the within class variance [4, 5, 6]. In other words, it decreases the distance between same class images and increases the distance between different class images [11]. Because of that LDA easily recognizes the faces among large databases LDA is invariant to face expressions and light conditions The linear discriminant analysis performs considerably better when compared with PCA and ICA for large databases. LDA needs to be combined with other methods to perform much better on big databases.

16 CHAPTER 1. MOTIVATION Major contributions of the thesis The contributions of the thesis are: Presentation of the survey of the relevant literature on visual classification techniques Study of different linear subspace techniques PCA, ICA and LDA and non linear methods (KPCA, KLDA) Implementation of PCA, ICA and LDA techniques and comparison of the techniques using databases Use of the euclidean distance approach to calculate the best match Collection of different face databases like Essex face94, ORL database and FERET database to compare the results Proposal of a novel wavelet based linear discriminant analysis (WLDA) technique Wavelets extract the nonlinear features and form the feature vector. This feature vector is given as an input to the linear subspace technique. Wavelets reduce the complexity as well. Nonlinear features give the more important information present in the image. These nonlinear features improve the performance of the face recognition system. So wavelets based techniques are invariant to environmental conditions and viewing conditions. As part of this thesis, my focus is on the following aspects: Development of wavelets based linear subspace techniques which are invariant to different environmental conditions Comparison of linear subspace techniques with wavelets based linear subspace techniques Comparison of the effect of different wavelets functions and wavelets coefficients on the recognition rate Comparison of the WLDA recognition rate with the WPCA and WICA recognition rates This thesis also involves the soft core of the WLDA technique on FPGAs with the help of the DSP Builder tool from Altera. The Altera DSP Builder converts Matlab/Simulink models into the hardware description language based design files. 1.4 Thesis outline The thesis is organized as follows: Chapter 2 gives the introduction to digital images and various operation on images Chapter 3 gives the introduction, history and basics of face recognition

17 CHAPTER 1. MOTIVATION 5 Chapter 4 presents details about the linear subspace techniques and the comparison of these subspace techniques Chapter 5 presents some methods for the nonlinear analysis of data Chapter 6 gives the introduction to wavelets and wavelets based linear subspace techniques as well Chapter 7 presents different approaches to measure the distance (between the pixels belonging to different images) and explains each approach with its advantages and drawbacks Chapter 8 gives the details about various face databases and the difference between them Chapter 9 gives the details about field programmable gate arrays (FPGA) and the DSP Builder tool used to deploy the Simulink model to FPGAs Chapter 10 presents the results obtained by using different techniques with different databases

18 Chapter 2 Background 2.1 Introduction For all image processing applications, the input is a digital image and the output can be an image or image related data. In order to understand how different operations can be performed on image, we need to understand the basics of the digital image like What is meant by a digital image? How images are stored? What is the color representation of the image? in this chapter, the fundamentals required to understand an image are presented in detail. 2.2 Digital image Generally the image is assumed to be a rectangular shape matrix with X rows and Y columns. The resolution of the image is taken as XxY. The image is stored as a small squared regions or number of picture elements called pixels [12]. In image, P (0, 0) represents the top left corner pixel as shown in figure 2.1 and P (X 1, 0) represents the bottom left corner pixel of the image. In digital image, pixels contain color value and each pixel uses 8 bits (0 to 7 bits). Most commonly, image has two types of representation. One is gray scale image and another one is color image [12]. Gray scale image calculates the intensity of light and it contains 8 bits or one Byte or 256 values. 2 8 = 256 (2.1) These 256 values represent the gray scale image. Each pixel in the gray scale image represents one of these 0 to 255 values or one of the 256 gray shades. The value 0 represents the black and means very less brightness. The value 255 represents the white and means very high brightness [13]. 6

19 CHAPTER 2. BACKGROUND 7 P(0,0) P(X-1,0) P(X-1,Y-1) Figure 2.1: Representation of the rectangular shape digital image into pixel P (X, Y ) The remaining in between the boundary values (0 to 255 values) represents intermediate shades between black to white or gray shades as shown in figure 2.2. The images with only two colors (black and white) are different to these gray scale images [12]. Those two colors (black and white) images are called binary images. A binary representation of the images does not contains shades between black and white. Figure 2.2: Gray scale representation of the example image A given image contains some information. The amount of information depends upon the resolution of the image. This resolution depends on the camera properties. The memory and processing speed are also related to the image resolution. Gray scale images contain less information compared to color images.

20 CHAPTER 2. BACKGROUND 8 A given color image gives the intensity and chrominance of light. The color image contains 24 bits or 3 Bytes [14]. Each Byte represents one primary color (shade of that primary color). For first Byte primary color is red, second Byte primary color is green and Figure 2.3: Color storage representation in the color images third Byte primary color is blue. Each Byte has 256 values from 0 to 255. The pixel value 0 means none of the primary color is present. The pixel value 255 means the maximum amount of primary color shown in figure 2.3. The pixel value or color value can be stored in different color spaces such as HSV (Hue,Saturation,Value) etc. The example RGB color image is shown in figure 2.4. Figure 2.4: R, G and B color channels of the example color image

21 CHAPTER 2. BACKGROUND Image-Scalar operations The image supports different arithmetic operations like addition, subtraction, multiplication and division with the scalar value [12]. These are pixel wise operations. For example, take a scalar value c and choose addition operation. After that, take every pixel in the image and add with c. Then a new image is obtained with a high brightness than in the previous image as shown in figure 2.5. Figure 2.5: Image scalar operations (a) Image from essex face94 database (b) Image after scalar operation Image-Image operations In these operations, we take the image instead of the scalar value. The taken two images must contain the same resolution. These operations are done by taking two pixels with the same coordinate in different images (i.e. take first pixel in the first image and add this with the second image first pixel and so on) [12]. It supports basic operations like addition, subtraction, multiplication, division. Figure 2.6: Image image operations (a) First Lenna image (b) Second Lenna image with different pose

22 CHAPTER 2. BACKGROUND 10 For example, we wanted to generate blended image with these given two images g1 and g2 as shown in figure 2.6. We take a variable a with the value 0.5 in the function f as shown in equation 2.2. f = [a g1 + (1 a) g2] (2.2) Then the new image is formed by taking above operation on the two images. The blended version new image is shown in below figure 2.7 Figure 2.7: Blended version of two different pose Lenna images 2.3 Image statistics The arithmetic mean, the standard deviation and variance are statistics of the image. The histogram gives the statistical representation of the image data [12] Histogram of the image The histogram is very important and simple for analyzing the images. It calculates the count or how many times a particular color level is present (the values from 0 to 255 are presents) in the image. In other terms it gives the pixel intensity values. For gray scale images, there are 256 different pixel values. Figure 2.8 is the example gray scale image histogram graph. For color images, we got 3 histograms (the first histogram is for red color, the second histogram is for green and the third one is for blue color). This graph represents the color spread in the image. In the histogram, X-axis gives pixel

23 CHAPTER 2. BACKGROUND 11 Number of pixels with corresponding pixel value Pixel value Figure 2.8: Gray scale histogram of the example gray scale image values and Y-axis gives the frequency of each pixel. Figure 2.9 is the example color image histogram graph.

24 CHAPTER 2. BACKGROUND 12 Number of pixels with corresponding pixel value Pixel value Figure 2.9: Color image histogram of the example color image The histogram is useful to determine best contrast in the image and how intensity spreading in the image. And the disadvantage of histogram is, unable to give the information related to spacial relationships between the pixels Mean of the image The mean is the main issue in image statistics. The mean of the image represents the average pixel value in that image. In the gray scale image mean values give the average brightness or intensity value. The mean of the image is calculated from equation 2.4 u = E[f] (2.3) E[f] = 1 Y X Y 1 X 1 y=0 x=0 f(x, y) (2.4) Where f(x, y) is the image. E[f] is the expected value or the mean of the image. X stands for the number of rows and Y stands for the number of columns in the image Variance of the image The variance gives the spread of values or pixels around the image mean or the average of the squared distance of possible values from mean. V ar[f] gives the variance. Variance is calculated using the equation 2.6 or equation 2.8. The square root of the variance gives the standard deviation. This standard deviation gives the range of pixel values in the image. V ar[f] = E[(f u) 2 ] (2.5)

25 CHAPTER 2. BACKGROUND 13 V ar[f] = 1 Y X Y 1 X 1 (f(x, y) u) 2 (2.6) y=0 x=0 V ar[f] = E[f 2 ] [u 2 ] (2.7) V ar[f] = ( 1 Y X 2.4 Image acquisition Y 1 X 1 y=0 x=0 f(x, y) 2 ) ( 1 Y X Y 1 X 1 y=0 x=0 f(x, y)) 2 (2.8) Image acquisition is the processes of obtaining an image from the camera. This is the first step of any vision based systems [13]. In image acquisition, we take an image with system considerations. An acquisition can be from different environments : the face image from the real time electro optical camera to acquire a live picture of an object or image from the scanner or image from the magnetic disk or image capture by frame grabber or a video as a sequence of still images. 2.5 Feature selection and extraction Feature selection and extraction can be used in many image processing applications like edge detection, motion detection and so on. The feature contains piece of information and specifies property or characteristics of the object in an image. There are several types of features [15]. General feature: Features like color, texture and shape. They can be again divided into Pixel level features: Feature at every pixel. Example color, location. Local Features: Features after applying some image processing applications like segmentation and edge detection. Global Features: Features over entire image. Domain specific Features: Depends upon the application. Finally feature could be an edge, color, corner, blob (group of pixels) or any property of the image Feature selection In every image, many features are present. In those features, some are relevant and some are irrelevant features (characterized depending upon the application and also depending upon different light conditions). There are generally many low level features (irrelevant) and few high level features. Relevant features contains more useful information [16]. So we only take the relevant feature from all features. In feature selection, the first step is captures the image. The second step is, list all features in that image. Depending upon our application, classify the features in that image. next step is, list the most relevant features from all the features present in the image. The feature selection is very important step in image processing. Application performance

26 CHAPTER 2. BACKGROUND 14 depends upon our selected features. Relevant feature selection gives better and fast results [17]. Also improves the understanding of image data. But this feature selection needs exhaustive search to select the relevant features. If large number of features are available in our image, then it takes lot of time to select main features and also complexity increases. Figure 2.10 shows how to select good features. Figure 2.10: Feature selection (a) Selection of wrong feature (b) Selection of correct feature Feature extraction To classify an object from the image, we must extract features from the image [17]. So after the feature selection, we have to extract those relevant features from the image. Feature extraction is one of the dimensionality reduction method. In this step, we only take the selected features and discard the remaining features [15]. It looses very less information because it discards the irrelevant features. Even in case the amount of data is high, it only takes few useful data. So accuracy is almost the same. Best results are achieved, if we take more relevant features. Dimensionality reduction techniques are useful for feature extraction step. Linear subspace techniques are generally used for dimensionality reduction. These dimensionality reduction techniques or linear subspace techniques are discussed in Chapter 4.

27 Chapter 3 Face Recognition 3.1 Introduction Face recognition is one of the various biometric techniques used in identifying humans. Biometrics use physical and behavioral properties of the human. All biometric systems need some records in database against which they can search for matches [18]. Face plays a primary role in identifying the person and also face is the more easily rememberable part of the human. We can recognize thousands of faces in our lifetime. This skill is very robust because we identify a person even after so many years with different aging, different light conditions and viewing conditions [2]. The biometric techniques like finger print, iris and signature need some cooperation from the person for identification. The major advantage with face recognition is, it does not need any physical cooperation of the human at the time of recognition. Researchers are trying to implement robust face recognition systems, similar to the human recognition capabilities under various conditions, for security and multimedia information access applications where the identification of the object is necessary. The task of machine recognition of humans is becoming increasingly difficult and complex as the conditions/constrains and the sample size are growing. In this chapter, together with the history of the face recognition development, various face recognition methods are explained in detail History of face recognition The research dealing face recognition systems actively started in 1960 s. The initial research directions are based on the features such as eyes, ears, nose and mouth on the photograph; the distance from the reference data is calculated [19]. In 1970 s Goldstein, Harmon and Lesk used 21 specific characteristics of the face such as hair color, lip thickness for face recognition [19]. The measurements for the methods developed till the end of 1970 s are calculated manually. In 1988, Kirby and Sirovich introduced the principal component analysis, a statistics related technique, to face recognition problems [1]. The introduction of the principal component analysis is a milestone in the face recognition concept and this lead to the development of various different algorithms for the face recognition based on the statistics related techniques. 15

28 CHAPTER 3. FACE RECOGNITION Face recognition basics Face Recognition is one of the highly used biometric techniques [18] as shown in figure 3.1. Face recognition systems, irrespective of the method used to recognize, follow the principle of finger print matching methods. The images of the people under recognition are collected and placed in the data base. Images in the database are used to verify the person when the recognition of the person is needed. The major difference between face recognition and finger print matching is we need to take the finger print of the person during identification and match with one we have in our data base. In face recognition we can acquire this image without the direct knowledge of the person and the image to be identified irrespective of the quality of the image in database [20, 21]. This type of recognition without directly involving the person is highly useful in security and monitoring applications Weighted percentage Face Finger Hand Voice Eye Signature Biometric Technology Figure 3.1: Comparison of biometric techniques [18] The face recognition knowledge is also highly useful in the area of image analysis. Face recognition is relative to object recognition in many aspects. The difference between these two recognition techniques is that face recognition is only used to identify persons or individuals in a image. While object recognition is used to identify all objects present in the given image [22]. Many researchers also classify face recognition as a subtask in object recognition [22].

29 CHAPTER 3. FACE RECOGNITION Face detection and recognition Face recognition algorithms calculate the similarities and differences between the stored faces and given test face [23]. If we apply the face recognition technique to the images of persons, the system produces unexpected results. The important step in the face recognition process is detecting the face in a given image. Face detection or object detection is also one of the important application in computer vision. There exist many algorithms to detect the face or object depending upon the features. These algorithms determine the location and size of the face in the image i.e. detect human face features [24, 25]. If the given image contains more than one human face, these algorithms must detect all the faces present in that image. So sometimes face detection is also a part of face recognition system. In developing the face recognition system, we generally consider only face images of different persons (face detection is not required) [26]. Face recognition is used to recognize known faces in every day life. In face detection, storage is not a major issue as the faces are detected from the image and only the detected face is given to the recognition algorithm for the recognition process. But for face recognition, we must store the face images in different poses and under varying conditions in the database for matching purposes at testing time. This matching is based on the distance measures which is explained in Chapter[?]. 3.2 Face recognition approaches Almost all faces have same features such as eyes, nose and mouth arranged regularly in the same manner [27]. So feature selection and feature extraction are very important tasks. Several methods are proposed to extract good features. There are many difficulties in developing a real time face recognition system because of pose (frontal, profile), expressions, occlusion and light conditions etc. So researcher are putting more attention to the face recognition methods [3]. A number of face recognition algorithms have been proposed during last decades and still the research is going. The current face recognition techniques are categorized into Feature based Model based or Knowledge based Template based Appearance based or View based This classification is shown in figure Feature based face recognition In this technique, the selected set of features from the face image are considered for recognition. Face can be recognized by taking features like eyes, nose or mouth etc in detail. The algorithm first find small parts in the image and match these parts with the stored template image. The overall technique describes the position and size of each features (eye,

30 CHAPTER 3. FACE RECOGNITION 18 Graph Matching Feature based Elastic Bunch Graph Matching Model based (or) Knowledge based 3D Morphable Model Face Recognition Template based Linear PCA ICA LDA Others Appearance based (or) View based Nonlinear KPCA KICA Others Figure 3.2: Classification of face recognition techniques nose, mouth or face outline) [28]. The feature based technique is accurate even in small light and expression changes but it is sensitive to scaling and rotation of a face. In case of pose variation, it is difficult to identify the feature points. To overcome the difficulties in the pose variation graph matching techniques are used. The labeled graph is shown in figure 3.3 Figure 3.3: Example model of the labeled graphs 3D graph matching: 3D mesh is used to identify the feature points. High curvature points on face are identified and marked with point. This point and relation between these points are represented as a graph. Elastic bunch graph matching method: Almost all faces have same topological structure [29]. Faces are represented as a graph with main point like eyes, nose, etc are nodes

31 CHAPTER 3. FACE RECOGNITION 19 and edges are represented as a distance vector. Each node has a set of 40 gabor wavelet coefficients (including phase and magnitude) [29]. This labeled graph is a set of nodes connected by the edges. This method is called graph matching for 2D face recognition Model based face recognition In the model based face recognition technique, a model is generated based on the facial variations of the face image. The important information related to the image is used to construct this model. The model based approach has been mainly divided into three main steps [3]. The first step is to develop the face model based on the prior knowledge, second step is to try to fit this 2D projections to given 2D face image and the third step is to take the parameters of the fitting model. These parameters form the feature vector. Shape model is constructed by identifying the positions of the feature points. Texture model represents the gray intensities. After constructing the model, calculates the similarity between the given test face and the faces in the database. In this technique, recognition is an iterative process. The model parameters are calculated by using eigen face analysis. Figure 3.4: 3D Morphable face model method 3D Morphable model: Face is represented as a 3D model for handling different poses and illuminations. Individual faces are combined into a single morphable model by computing dense point to point correspondences to the reference face. The 3D model is obtained from best fit between its 2D projections to given 2D face image. The 3D morphable model is explained in figure 3.4. The 3D morphable model (a statistical model) combines the 3D shape and texture information of all example of individual faces into one vector space of faces [3]. After

32 CHAPTER 3. FACE RECOGNITION 20 forming the vector space of faces, it calculates the average face and individual faces are characterized by increasing the distance from the average. A manually labeled feature is marked on the face like tip of nose, eye corner etc. The number of features required for marking varies depending on the application. The correspondence of all these points or features gives the morphable model Template based face recognition The template based face recognition technique is one of the digital signal processing technique [30]. In this technique, we store some sample templates of the face image in the database. It involves the use of pixel bidirectional array of intensity values either original gray scale or only specific data [28]. The euclidean distance measure is used to calculate the matching between the stored template and the given face image. The performance depends on the resolution of the image. To improve the performance, the normalized cross correlation coefficient is used. pi T, T = E(I T T ) E(I T )E(T ) σ(i T )σ(t ) (3.1) Where I T is the random pixel values in the image I. T is random pixel values of the template. E(.) represents the expected value or mean. Brunelli and Poggio compared feature based face recognition and template based feature recognition techniques [28]. In their work, they were using feature based templates like eyes, nose and mouth in addition to the whole face template Appearance based face recognition Many approaches in computer graphics are related to the images directly not based on 3D models [27]. Faces are stored as two dimensional intensity matrix. The vector space contains different images and each point in the vector space represents an image. Almost all appearance based techniques use statistical properties like mean and covariance to analyze the image [3]. In this approach given a test image, the algorithm finds the similarity between test image and stored images using feature vector. Appearance based face recognition approach is again classified as Linear analysis Nonlinear analysis Linear (subspace) analysis The linear analysis explains the number of linear principal manifolds. Manifolds are important in mathematics and physics. The manifold explains the complex structure in terms of the important properties of the simpler space [31, 32]. In these methods, the average image of all the persons (also of each person) in the data base is calculated. Each image has been translated to average face by subtracting average image from each face image. The face space is a linear subspace of the image face. In the linear analysis, we have three classical statistical techniques.