Face Recognition in the Scrambled Domain via Salience-Aware Ensembles of Many Kernels

Transcription

1 Fae Reognton n the Srambled Doman va Salene-Aware Ensembles of Many Kernels Jang, R., Al-Maadeed, S., Bourdane, A., Crooes, D., & Celeb, M. E. (2016). Fae Reognton n the Srambled Doman va Salene-Aware Ensembles of Many Kernels. IEEE Transatons on Informaton Forenss and Seurty, 11(8), /TIFS Publshed n: IEEE Transatons on Informaton Forenss and Seurty Doument Verson: Peer revewed verson Queen's Unversty Belfast - Researh Portal: Ln to publaton reord n Queen's Unversty Belfast Researh Portal Publsher rghts () 2016 IEEE. Personal use of ths materal s permtted. Permsson from IEEE must be obtaned for all other users, nludng reprntng/ republshng ths materal for advertsng or promotonal purposes, reatng new olletve wors for resale or redstrbuton to servers or lsts, or reuse of any opyrghted omponents of ths wor n other wors. General rghts Copyrght for the publatons made aessble va the Queen's Unversty Belfast Researh Portal s retaned by the author(s) and / or other opyrght owners and t s a ondton of aessng these publatons that users reognse and abde by the legal requrements assoated wth these rghts. Tae down poly The Researh Portal s Queen's nsttutonal repostory that provdes aess to Queen's researh output. Every effort has been made to ensure that ontent n the Researh Portal does not nfrnge any person's rghts, or applable UK laws. If you dsover ontent n the Researh Portal that you beleve breahes opyrght or volates any law, please ontat openaess@qub.a.u. Download date:21. Jun. 2016

2 Fae Reognton n the Srambled Doman va Salene-Aware Ensembles of Many Kernels Rhard Jang, Somaya Al-Maadeed, Ahmed Bourdane, Danny Crooes IEEE, Senor Member and M. Emre Celeb IEEE, Senor Member Abstrat Wth the rapd development of nternet-of-thngs (IoT), fae sramblng has been proposed for prvay proteton durng IoT-targeted mage/vdeo dstrbuton. Consequently n these IoT applatons, bometr verfaton needs to be arred out n the srambled doman, presentng sgnfant hallenges n fae reognton. Sne fae models beome haot sgnals after sramblng/enrypton, a typal soluton s to utlze tradtonal data-drven fae reognton algorthms. Whle haot pattern reognton s stll a hallengng tas, n ths paper we propose a new ensemble approah Many-Kernel Random Dsrmnant Analyss (MK-RDA) to dsover dsrmnatve patterns from haot sgnals. We also norporate a salene-aware strategy nto the proposed ensemble method to handle haot faal patterns n the srambled doman, where random seletons of features are made on semant omponents va salene modellng. In our experments, the proposed MK-RDA was tested rgorously on three human fae datasets: the ORL fae dataset, the PIE fae dataset and the PUBFIG wld fae dataset. The expermental results suessfully demonstrate that the proposed sheme an effetvely handle haot sgnals and sgnfantly mprove the reognton auray, mang our method a promsng anddate for seure bometr verfaton n emergng IoT applatons. Index Terms Faal bometrs, fae sramblng, many manfolds, many ernels, random dsrmnant analyss, moble bometrs, Internet-of-Thngs, user prvay. W I. INTRODUCTION ITH rapd developments n Internet-of-Thngs (IoT) tehnology, fae reognton [1~4] has reently found a new use n web-based bometr verfaton, man-mahne nteraton, nternet medal dagnoss, vdeo onferenng, dstane learnng, vsual survellane, and psyhologal evaluaton. In the ontext of mass nternet tehnology, prvay [5~15] has beome an ssue of wde onern n web-based vdeo streamng. As a result, fae sramblng [5] s emergng as a pratal tehnque to protet The manusrpt was reeved, revsed. Rhard Jang and Ahmed Bourdane are wth Computer Sene & Dgtal Tehnologes, Northumbra Unversty, Newastle upon Tyne, UK. Somaya Al-Maadeed s wth Department of Computer Sene & Engneerng, Qatar Unversty, Doha, Qatar. Danny Crooes s wth ECIT Insttute, Shool of Eletrons, Eletral Engneerng & Computer Sene, Queen's Unversty Belfast, Belfast, UK. M. Emre Celeb s wth Department of Computer Sene, Unversty of Central Aransas, Conway, AR, USA. Correspondene e-mal: rhard.ang@am.org. Copyrght () 2015 IEEE. Fg.1. A deteted fae n vdeo srambled by usng the Arnold transform. Fg.2. Semant approahes suh as usng AAM [18]~[25] for faal emoton estmaton annot be appled n the srambled doman. prvay legally durng vdeo dstrbuton over the publ nternet. By sramblng faes deteted n prvate vdeos, the prvay of subets an be respeted, as shown n Fg.1. Compared wth full enrypton methods, fae sramblng s a ompromse hoe beause t does not really hde nformaton, sne unsramblng s usually ahevable by smple manual tres even though we do not now all the parameters. It avods exposng ndvdual bometr faes wthout really hdng anythng from survellane vdeo. As shown n Refs.[5~14], sramblng has reently beome popular n the researh feld of vsual survellane, where prvay proteton s needed as well as publ seurty. Another advantage of fae sramblng over enrypton s ts omputng effeny, and usually t s far smpler than omplated enrypton algorthms. In many busness ases suh as publ survellane, the purpose s lmted to only prvay proteton from unntentonal browsng of user data. Hene, full enrypton beomes unneessary n ths ontext. There are many ways to perform fae sramblng. For example, sramblng an be done smply by masng or artoonng [8]. However, ths nd of sramblng wll smply lose the faal nformaton, and hene subsequent fae reognton or verfaton beomes unsuessful n ths ase. Espeally for seurty reasons, t s obvously not a good hoe to really erase human faes from survellane vdeos. In 1

3 omparson, the Arnold transform [13, 14], as a bas step n many enrypton algorthms, s a nd of reoverable sramblng method. Srambled faes an be unsrambled by several manual tres. Hene, n ths wor, we have hosen Arnold transform based sramblng as our spef test platform. Fae reognton has been extensvely researhed n the past deade and sgnfant progress has been seen towards better reognton auray n reent reports [15~21]. These approahes usually explot semant fae models [22~23] where a fae s onsdered as an ntegraton of semant omponents (suh as eyes, nose and mouth), and hene semant related sparse features or loal bnary patterns (LBP) an be effetvely used to mprove the reognton auray. Beyond 2D faal modellng, 3D models [23] an also be exploted for better auray by tang advantage of 3D fae algnment. However, as shown n Fg.2, a srambled fae has a very dfferent appearane from ts orgnal faal mage. Whle we an easly math a 3D model to a normal faal mage, t beomes extremely hard to do so after the fae has been srambled. In the srambled doman, semant faal omponents smply beome haot patterns. In ths ontext, t beomes dffult to explot landmars or 3D models for better auray. As shown n Fg.2, whle fae models an be easly ftted wth a faal mage, t beomes mpossble after a fae s srambled nto haot patterns. As has been dsussed n [15], one straghtforward way s to use tradtonal data-drven approahes, where haot sgnals are treated smply as a set of data ponts spread over manfolds. Varous data-drven fae reognton algorthms have been developed over several deades. In the early days, lnear dmensonalty reduton [24~27] was used for ths hallenge, suh as prnpal omponent analyss (PCA) [24], ndependent omponent analyss (ICA) [24], and Fsher s lnear dsrmnant analyss (FLD) [25]. Wth ernel methods (KM) [26], these methods an be extended to a reprodung ernel Hlbert spae wth a non-lnear mappng, and extended as -PCA and -FLD. Reent progress on nonlnear manfold learnng [27~32] has produed a number of new methods for fae reognton, suh as Laplaanfae [30] and Tensor subspae [31]. These approahes have been suessfully used for data-drven fae reognton. However, for fae reognton n the srambled doman, we need a robust approah to handle haot sgnals n the srambled doman, whh appear random and beyond human perepton. In reent researh, mult-ernelzaton [32, 33] has been proposed to handle the omplexty of data struture, where t s beleved multple-vew dsrmnatve strutures [34, 35] need to be dsovered where a manfold may have dfferent geometr shapes n dfferent vews. Wth the hope of utlzng ths approah for haot sgnals, n ths paper we propose a new approah alled Many Kernel Random Dsrmnant Analyss (MK-RDA) to handle ths new hallenge of haot sgnal reognton n the srambled doman. We also propose a mehansm to norporate a salene model [36] nto MK-RDA for pattern dsovery from haot faal sgnals, sne t s beleved that semant features are usually salent and useful for faal pattern lassfaton. In the followng setons, faal mage sramblng usng the Arnold transform s ntrodued n seton II, and the semant mappng of faal omponents for robust feature extraton n the srambled doman s desrbed. In seton III, we ntrodue the baground and motvaton of our many ernel ensemble method, and present our many-ernel random dsrmnant analyss. In Seton IV, we present the framewor usng MK-RDA wth the salene model for haot faal pattern verfaton. Seton V gves the expermental results on three fae datasets, and onlusons are drawn n Seton VI. a) Faal omponents b) After one Arnold transform ) After 2 Arnold transforms b) After 3 Arnold transforms Fg.3. Fae sramblng by the Arnold transform. II. FACIAL COMPONENTS IN THE SCRAMBLED DOMAIN A. Fae Sramblng In many IoT applatons, t s not enouraged to hde any nformaton by enrypton; on the other hand, t s legally requred to protet prvay durng dstrbuton and browsng. As a result, sramblng beomes a ompromse hoe beause t doesn t really hde nformaton (unsramblng s usually ahevable by smple manual attempts), but t does avod exposng ndvdual faes durng transmsson over the nternet. Addtonally, sramblng usually has muh lower omputaton ost than enrypton, mang t sutable for smple networ-targeted applatons usng low power sensors. Among varous mage sramblng methods, the Arnold sramblng algorthm has the feature of smplty and perodty. The Arnold transform [11, 12] was proposed by V. I. Arnold n the researh of ergod theory; t s also alled at-mappng before t s appled to dgtal mages. It has been wdely used n vsual survellane systems where t s favored as a smple and effent sramblng method whh nevertheless retans some spatal oherene. In ths paper, we use ths sramblng method to set up the test envronment of our algorthm n the srambled fae doman. 2

4 a) Strutural salene mappng of semant features b) Summarzed semant map ) Srambled semant map Fg.4. Semant salene of faal mages In the Arnold transform, a pxel at pont (x, y) s shfted to another pont (x', y') by: x' 1 y' 1 1x mod N 2 y, (1) whh s alled two-dmensonal Arnold sramblng. Here, x and y are the oordnates of the orgnal pxel; N s the heght or wdth of the square mage proessed; x' and y' are the oordnates of the srambled pxel. The Arnold transform an be appled teratvely as follows: (2) P AP, P x, y 1 xy xy xy Here, the nput s the orgnal mage after the -th Arnold transform, and P xy +1 on the left s the output of the +1th Arnold transform. represents the number of teratons, where = 0, 1, 2 and so on. By the replaement of the dsrete latte for transplantaton, the Arnold transform produes a new mage after all pxels of the orgnal mage have been traversed. In addton, Arnold sramblng also has the property of beng yl and reversble. Fg.3-a) shows a fae wth ts faal omponents (.e., eyes, nose and mouth) rled by dfferent olors. Fg.3-b) shows the srambled fae after one operaton of the Arnold transform, where t an be seen that faal omponents have drast dsplaements. Fg.3-) and d) shows the srambled faes after two and three operatons of the Arnold transform. In omparson wth Fg.3-b), the srambled faes n Fg.3-) and d) are more dffult to dentfy by the human eye. In ths wor, we T use three operatons of the Arnold transform to sramble all faes. As we an see from Fg.3, before sramblng, faal omponents an easly be dentfed by the human eye. After sramblng, the mages beome haot sgnals, and t s hard to fgure out eyes and noses. Sne semant faal omponents are onsdered mportant ues for fae reognton, we need to fnd a way to norporate semant approahes nto the srambled doman to attan hgher mathng auray. In many IoT based applatons, t may not be allowed to unsramble deteted faes due to prvay-proteton poles. Moreover, unsramblng may nvolve parameters (suh as the ntal shft oordnates) that are usually unnown by the onlne software. Faal reognton n the srambled doman then beomes a neessty n these IoT applatons. B. Semant Faal Components Fundamentally a 2-D fae mage s the proeton of a real 3-D fae manfold. Ths vewpont leads to model-based fae reognton, where semant faal omponents (suh as eyes, nose, and lps) are modeled by ther parameters. A very frequently appled fae model s the atve appearane model (AAM) [20]~[23]. 3D faal nformaton s better for desrbng the semant faal omponents n the presene of llumnaton and pose hanges, where 2-D desrptors sometmes turn out to be less effetve. Hsu and Jan [23] have advoated that suh semant faal omponents onsttute the meanng of a fae and desvely form the bass of fae reognton. Along ths roadmap, template-based fae desrpton [21] has been onsdered to emphasze the mportane of semant faal omponents. In our human perepton system, onept-level semant features are more meanngful than pxel-level detals. A good emoton estmaton model usually reles on the mportane of semant features. Changes n a sngle pxel or sparse set of pxels should not dstort the fnal deson. Though semant approahes have attaned great suess n faal analyss, they need a robust sheme to map a 2D mage nto ts semant feature spae or 3D deformable model. Ths omputaton s not trval and usually annot be afforded by many real-world applatons suh as moble omputng platforms. Besdes, the deteton of semant features an be senstve to dfferent ondtons, and hene produes extra errors n fae lassfaton. To tae advantage of semant features wthout worryng about ts omputng omplexty, n ths paper we ntrodue a salene-aware method nto our faal analyss. C. Semant Salene Mappng of Faal Images Sne semant omponents are mportant ues to dentfy a spef fae, we need to fnd a way to ntrodue these fators n statst fae modellng. In ths paper, we propose to use salene learnng for semant faal mappng, and norporate the learned semant map nto a random forest method for fae reognton. As shown n Fg.4-a), faal omponents are usually salent features n a faal mage. In ths paper, we employ the Deep Salene model [39] for semat feature mappng. Unle other models based on olor salene usng pxel ontrast, ths deep salene model bases ts algorthm on strutural salene, and 3

5 hene an easly fnd the semant omponents as ts salent features, as shown n Fg.4-a). Ths fts well wth our purpose to explot semant omponents n a faal mage. We then apply a Gaussan mxture model to summarze the learned salene maps of the tranng dataset, where the salene dstrbuton s onsdered as a mxture of Gaussan funtons, (3) p x w g x, g x, where s the normalzed Gaussan dstrbuton wth mean µ and varane σ. In our wor, we use a two-lass GMM model and estmate the probablty of a pxel beng salent or non-salent. Learnng wth GMM mxtures an fnd optmzed Gaussan dstrbuton parameters n the GMM model, and onsequently produe a dstrbuton map S=p(x λ) from Eq.(2), whh s referred to as the semant mportane map n ths paper. Fg.4-b) shows the estmated semant mportane map learned from Fg.4-a), whh hghlghts semant features suh as eyes, nose and mouth. Ths mportane map represents the mportane of eah feature subspae n terms of ts relaton to semant features. Fg.4-) shows the srambled semant map. One we have the semant salene map of the tranng dataset, we an then use t to gude the feature samplng to favor semant features. III. ENSEMBLES OF MANY-KERNEL DISCRIMINANT ANALYSIS A. Baground on Mult-Kernel Approahes In many real world applatons suh as fae reognton and mage lassfaton, the data often has very hgh dmensonalty. Proedures that are omputatonally or analytally manageable n low-dmensonal spaes an beome ompletely mpratal n a spae havng several thousand dmensons. Ths has been well nown n mahne learnng as a notorous ssue --- the Curse of Dmensonalty [1~3]. To tale ths hallenge, varous tehnques [1~12] have been developed for redung the dmensonalty of the feature spae, n the hope of obtanng a more manageable problem. Dmensonalty reduton has beome an espeally mportant step for fae lassfaton. Varous algorthms have been developed for mage-based fae reognton. In ths paradgm, dmensonalty reduton [19] has always been a prmary onern. As mentoned prevously, methods developed for ths hallenge nlude prnpal omponent analyss (PCA) [24], ndependent omponent analyss (ICA) [24], and Fsher s lnear dsrmnant analyss (FLD) [25]. Wth ernel methods (KM) [26], these methods an be extended to a reprodung ernel Hlbert spae wth a non-lnear mappng, and extended as -PCA, -ICA and -FLD. Reent progress on nonlnear manfold learnng [27]~[31] has led to a number of new methods for fae reognton, suh as Laplaanfae [35], Tensor subspae [36], non-negatve matrx [37], and loal Fsher dsrmnant analyss (LFDA) [38,22]. These approahes usually assume there s an underlyng dsrmnatve struture to dsover, whh leads to the paradgm of manfold learnng. Reently, the mult-vew problem has been noted by the researh ommunty, where the same manfold an have dfferent shapes n dfferent subspaes, as shown n Fg.5-a). Foster et al. have employed anonal orrelaton analyss (CCA) [32] to derve the low dmensonal embeddng of two-vew data and to ompute the regresson funton based on the embeddng. Hedge et al [33] propose a multple proeton approah from the same manfold. Hou et al [34] used the parwse onstrants to derve embeddng n multple vews wth lnear transformaton. Xa et al [35] ombned spetral embeddng wth the mult-vew ssue. Han et al. [36] proposed a sparse unsupervsed dmensonalty reduton to obtan a sparse representaton for mult-vew data. Ln et al [37] proposed multple ernel learnng of a manfold, where varous ernel spaes are onstruted wth dfferent sets of parameters. Zen et al [38] onsdered multple ernels wth regards to mult-lass ases. In the mult-vew problem, as shown n Fg.5-a), although a manfold has dfferent forms n dfferent subspaes, these forms an always be unfed as the same manfold n a hgherdmensonal subspae. However, ths may not always be true. As shown n Fg.5-b), when the sequene of data ponts n the seond subspae s shuffled, the ombnaton of two submanfolds smply reates a nosy-le dstrbuton. Ths means two submanfolds annot be merged at all. In ths ase we have to treat t as a multple or even many manfold problem, where multple manfold strutures need to be dsovered. In our faal reognton n the srambled doman, faal mages beome haot sgnals, as shown n Fg.1 and Fg.2. In ths real-world ase, ts underlyng dsrmnatve strutures ould be more le the ase n Fg.5-b), where multple manfold strutures need to be dsovered. In ths paper, we nlude ths ase n our onsderaton and propose a new many-ernel approah to handle ts omplexty. Before we go further, we gve an ntroduton to ernel based analyss. B. Prelmnary on Kernel based Dsrmnant Analyss (KDA) For a set of data ponts {x }R N, we may selet a set of data ponts as the landmars {L } that an haraterze ths dataset. A data pont on the manfold then an be loated by ts ernel dstane to the landmars: x K x, L (4) a) Mult-Vew Problem b) Multple Manfold Problem Fg.5. Mult-vew dataset and mult-manfold dataset. When the sequene of data ponts n the seond subspae s shuffled, the two sub-manfolds beome ndependent of eah other, and annot be unfed n a hgher dmensonal subspae. Hene, eah data pont s represented n the onstruted ernel 4

6 Lst I. Random Generaton of Many Kernels Input: {x } Dataset; L K Number of ernels; Output: {κ } Construted many ernel representatons; Proess: Loop for L K tmes Generate random seleton {l m } Selet K L landmars from {x } Loop for eah data pont x Compute ts ernel representaton κ based on {l }. End Loop End Loop Return {κ }. a) Random feature seleton n srambled doman guded by the salene map n Fg.4-). ) Atual ht rates n srambled doman. spae as κ R M, where M s the number of seleted landmars. Followng ths, we then smply apply Fsher s lnear dsrmnant analyss n the ernel spae: T S B arg max (5) T S W where Φ s the proeton matrx, and S B s the between-lass ovarane matrx: K S B n (6) 1 and S w s the wthn-lass ovarane matrx: S W K n 1 1 (7) By optmzng over Eq.(7), we then have the Egen proeton b) The orresponded pxels on the orgnal faal mage. d) Unsramble the ht map ba to faal doman. Fg.6. Seletng ernel subspaes toward semant features. 5 matrx W, and eah data pont s then represented by ts new oordnates n the KDA spae: (8) y Here, Φ s an Egen matrx R D M, y R D, and D s usually a number smaller than M as well as smaller than the number of lasses n the tranng dataset {x }. C. Many Kernels for the Many Manfold Problem Though t has been assumed n many methods that there s only one underlyng manfold struture, t s obvous that there an often be multple manfolds underlyng many real-world datasets, as shown n Fg.5-b). However, the dsovery of the underlyng manfold strutures s an nverse engneerng problem that ould be very omplex, and often ntratable. For example, onsder seletng M dmensons out of the feature spae R N : there are K=N!/{M!(N-M)!} suh hoes that an be made, and wthn eah seleton an ndependent sub manfold may be dsovered. For example, when N=10 and M=5, K wll be 252. For a faal mage, there ould be 64 64=4096 dmensons, and M ould be any number. Hene, the estmaton of possble subspaes beomes an NP-hard problem that annot be handled exhaustvely n realst omputng tme. Hene, the dsovery of many manfolds beomes a maor hallenge that has not yet been fully appreated. In ths wor, to address the hallenge shown n Fg.5-b), we propose a randomzaton strategy to generate many ernels and try to over as many manfolds as possble n a gven dataset by hane, whh redues the omplexty of the many manfolds problem from ts exponental omputng tme to somethng manageable. D. Many Kernels from Random Feature Seleton If we have K data ponts {x }, then typally the random seleton of subspaes an be easly attaned by generatng a lst of random numbers l, and seletng K L features to onstrut the new datasets: (9) z ~ x ( l ) Here, {z }R KL. Then we an onstrut a ernel spae based on ths randomly seleted subspae: x K z, z (10) We an repettvely redo the above randomzaton proess, and as a result, we an easly onstrut as many ernels as we want. If we have L K ernels and eah ernel has K L dmensons, then for eah data pont x, we wll have the ernel representaton {κ } atually as an L K K L matrx. To guarantee the ernelzed dmensons are not too muh more than the orgnal data dmensons, we add a onstrant:, (11) L K K L ~ N whh means the many ernel proess wll not nrease or derease the dmensons. Ths proess s outlned n Lst I. E. Many-Kernel Random Dsrmnant Analyss The purpose of ths many-ernel strategy s to fnd the underlyng dsrmnatve strutures n eah subspae. After we obtan the many ernel based representaton κ, we an then apply dsrmnant analyss over eah ernel subspae and fnd

7 ts dsrmnatve proeton. For a set of tranng data and ts ernel representaton {κ }, we an alulate ts wthn-lass ovarane at ts -th ernel subspae as: S W K n 1 1 and ts between-lass ovarane matrx: K S B n 1 (12) (13) To fnd the most dsrmnatve features, we an maxmze ts between-lass ovarane over ts wthn-lass one by fndng a proeton matrx Φ : Φ ~ arg max Φ Φ Φ T T S S B W Φ Φ (14) By optmzng over Eq.(10), we then have the Egen proeton matrx Φ B D KL. For eah data pont κ, we an then have ts dsrmnant proeton n ts -th subspae: (15) y Φ For eah ernel subspae, we an obtan the ernel dsrmnant proeton for eah data pont. As a result, we wll have the L K proeton: (16) Y ~ where Y wll be a matrx B D LK. y IV. FACIAL SEMANTIC AWARE ENSEMBLES OF MANY KERNELS A. Salene-Based Feature Spae Reonstruton Unsurprsngly, salent features usually play an mportant role n fae lassfaton. Therefore, ratonally we an expet a mehansm to gve salent features more weght than others. In ths wor, we onsder a based strategy to reonstrut the feature spae to favor semant salent features. Consderng a srambled faal mage x as a vetor of faal features/sgnals {f 1, f 2, f, }, and a semant salene map S~{s 1, s 2, s, } learned from tranng (as shown n Fg.4-), we an then onstrut a new feature spae by replatng eah feature aordng to ts semant mportane. Assumng the maxmum multplatve fator as K s, the repetton of eah Fg.7 Overvew of the proposed salene-aware sheme 6 feature s then defned as: nt1 Ks s max s (17) Here, means how many tmes the -th feature/sgnal wll be repeated, and s s the salene value of the -th sgnal shown n Fg.4-). Consequently, we have a new set of features: new f,..., f,..., f,..., f, (18) Wth the above multplatve proess, salent features wll have a hgher lelhood to be hosen n the randomzed seleton proess n Eq.(9). We then an apply the random seleton to selet subspaes from the reonstruted feature spae χ new to form the many ernels for MK-RDA. Fg.6 shows the results of suh a salene-guded seleton usng the srambled salene map n Fg.4-). We an see that wth the salene gudng, semant faal features wll be more lely to be used to form our ernels subspaes. B. Salene-Aware MK-RDA After the feature spae s reonstruted, we an apply MK-RDA on the reonstruted datasets {χ } nstead of {x }, and we have: (19) z, z, Ψ Φ Y Ψ K ~ At the end, we wll have Y as a matrx B D LK. For any two data ponts x 1 and x 2, ther dstane n the proeted subspaes an be alulated as: d 2 y y, (20) Here denotes the Euldean norm. For data lassfaton, the lelhood of a data pont belongng to a lass an be estmated from ts dstanes to all tranng data ponts n the -th learned ernel subspae: P Φ Pˆ Φ Pˆ, Φ (21) Here, P( Φ ) denotes the estmated probablty n the -th ernel proeton Φ that an nput data pont x belongs to a lass ( = 1, 2,..., n ). For all ernels, the dsrmnant funton s defned as: 1 ~ m x P Φ (22) K t and the deson rule s to assgn x to lass for whh (x) s the maxmum. C. Overvew of the Salene-Aware Sheme Fg.7 gves an overvew of the proposed salene-aware sheme for srambled fae verfaton. Gven a tranng dataset, faes are forwarded to the tranng proedure. The offlne proedure then learns ts semant salene map. Followng ths, the database s srambled and the feature spae s reonstruted by multplyng salent features aordng to ther semant salene weghts. Random samplng s then appled to selet features sparsely to onstrut as many ernels as s allowed, and dsrmnant analyss s used to learn a ernel subspae for eah ernel. After a srambled faal mage s nput as a test, the nput s

8 a) A small fae dataset --- Yale dataset. b) Number of ernels n MK-RDA D. Dsusson of Salene-Aware MK-RDA Before we proeed to our benhmar experments, there are two questons that need to be answered. Frst, n the MK-RDA mehansm, what s the best L K to hoose? Namely, how many ernels are enough? Seond, n the above salene-aware mehansm, an suh a salene based mehansm really help attan better auray n fae reognton? Here, we desgn an experment to fnd out the answers to these two questons. For ths experment, we hose the Yale fae dataset [40] for our tests. In the Yale dataset, eah of the 15 subets has 11 sample faes wth dfferent expresson, llumnaton and glasses onfguraton. We only hoose 6 faes wth dfferent expressons for our test, as shown n Fg.8-a). Wth ths small dataset, we arred out the fae reognton tests by splttng the small dataset nto tranng and test datasets, where the tranng dataset has fve subets and test dataset has the rest. We then vared L K, the number of ernels, and K s, the max weght of salene map, n our experments. We then examned whh set of parameters gves the best error rates. Fg.8 shows the results of our experment. Fg.8-b) gves the experment results on the number of ernels. Gven K s as 1.5, the number of ernels vared from 5 to 60. We an see that the error rate s lowest when L K s around 32. Basally, more ernels mean more omputng tme. As long as we have a low error rate, usng fewer ernels s often preferable. It s also observed that ompared wth the baselne LDA, MK-RDA has attaned margnally better auray. We then ran an experment on K s. As shown n Eq.(17), K s=0 means no bas. The bgger K s s, the more based t s toward the salent features. Fg.8-) shows the expermental results. It an be seen that the error rate s lowest when K s s around 2.5. It s also observed that based samplng wth hgher K s smply worsens the auray beause t means some non-salent features may be abandoned n the random proess even though they may ontrbute to the reognton proess. ) Semant weght fator Ks Fg.8 Parameters n Salene-Aware MK-RDA proeted nto eah ernel subspae, and the dstane to eah tranng sample s omputed. The deson proedure s based on the ombnaton of all ernel subspaes va Eq.(22). It s noted that we an have unsrambled mages (manly for statst salene learnng) n the offlne tranng beause offlne tranng s arred out entrally wth authortes /busness supervsors permsson and wll not undermne users prvay. Prvay proteton s manly an ssue wth dstrbuton over the nternet. In ths sheme, the tranng proedure an be arred out offlne. The onlne verfaton then beomes purely a data-drven proess. In the test proedure, all test mages and semant maps are srambled for prvay proteton, and no orgnal fae wll be utlzed for reognton purposes. Hene, t s smlar to other data-drven approahes, and s smple and straghtforward. V. EXPERIMENTS To valdate our algorthm, we mplemented our fae reognton method n Matlab, and ran on a PC wth 2.5GHz dual-ore Intel CPU. Before runnng the benhmar on fae datasets, all mages n the datasets were srambled usng the (trple) Arnold transform [7~8]. Fg.11 shows seleted fae mages from the three datasets: ORL, PIE and PUBFIG. The ORL database has 40 subets, eah wth 10 faes at dfferent poses. In total, 400 faes are used for the test. The CMU PIE database [40] has 41,368 faes, omprsng 67 lasses wth about 170 faes per lass, nludng a wde spetrum of varatons n terms of pose, llumnaton, expresson and appearane. In our tests, we use 50 faes per subet, smlar to [30] and [31]. The PUBFIG database [42] ontans wld faes seleted from the nternet. It s very smlar to LFW [43] but t provdes standard ropped faes. As has been shown [43], baground textures n LFW an help attan a hgher auray. Sne we onsder fae reognton only, PUBFIG fts better wth our purpose. In many prevous reports [9], the leave-one-out test sheme 7

9 has been wdely used. However, ths test s too smple beause t leaves one mage out as the test mage and eeps all the rest n the tranng set. In our test sheme, we try to nrease the hallenge and adopt a test sheme alled leavng--out, where n eah test samples per ategory are left out as test samples. For example, we have N samples, and then we hoose all faes of (N-) samples as the tranng dataset, and use samples for the test. For a leavng out sheme, there are usually C N hoes. In our experment, we ust hose 3 sets of onseutve faes from N samples, startng at N/4, N/2 and 3N/4. As a result, we have 3 sets of tests n turn for a leave--out experment. The fnal auray s gven by the average of all three tests. It s noted that the onseutve splttng wll usually brng out the large dfferene between test and tranng datasets, beause the datasets have faes vared onseutvely and the frst faes are usually very dfferent from the last (N-) faes. Our benhmar tests am to verfy whether or not the proposed MK-RDA an enhane the auray on srambled fae reognton. Our approah s a pure data-drven fae lassfaton method. Hene, smlar to Ref.[15], we ompared our approah wth a number of typal data-drven methods, nludng Egenfae [25], Fsherfae [25], PCA[26], LDA[26], and Laplaanfae (LPP) [31], eah appled to faal mages n the srambled doman. In the evaluaton of the proposed sheme, we smply use the nearest neghbor lassfer beause any nvolvement of any other methods may blur the omparson and we then annot easly assert f the enhanement omes from our MK-RDA sheme or any other underlyng more omplated lassfers. A. Tests on the ORL Dataset The ORL database has 10 faes per subet. In our leave--out test, vares from 1 to 6. In total, eah -test has 3 subtests, wth dfferent seletons of query faes from 10 faes. The fnal auray s the average on all subtests. Fg.10-a) shows all leave--out tests, where vares from 1 to 6. We an see that the proposed MK-RDA attaned the best auray n all fve tests. Fg.10-b) lsts out the overall auray by averagng all tests. Here, we nluded PCA, LDA, PCA, LDA and LPP for omparson beause they are typal data-drven fae reognton methods based on dmensonalty reduton. We an see that our MK-RDA attaned the best auray over all -tests of around 95.7%. In omparson, LPP attaned 91.5%, LDA 93.3%, LDA 93.6%, and PCA and PCA attaned87.5%. B. Tests on the PIE Dataset In our experment, we used 50 faes per subet and n total 3350 faes were used n our leave--out experment. In ths test sheme, faes from N samples per subet are seleted as test samples, and the rest are used as tranng samples. Fg.11 gves the test results on the PIE dataset. Fg.11-a) shows all leave--out tests, where vares from 5 to 25. We an see that the proposed MK-RDA attaned the best auray n all tests. However, when s nreased, fewer samples are left for tranng and as a result the auray drops n all methods. Fg.11-b) lsts out the overall auray by averagng all tests. PCA and PCA attaned an average auray of around 76.0%, a) Samples n the ORL database and ther srambled mages b) Samples n the PIE database and ther srambled mages ) Wld faes n the PubFg dataset and ther srambled mages Fg.9. Faal mages n the ORL, PIE and PUBFIG datasets. a) Leave--out tests Method PCA PCA LDA LDA LPP MK-RDA Auray b) Over all auray of all tests Fg.10. Leave--out tests on ORL dataset. a) Leave--out tests Method PCA PCA LDA LDA LPP MK-RDA Auray b) Over all auray of all tests Fg.11. Leave--out tests on PIE dataset. 8

10 a) Ran-1 auray versus dmensonalty tranng faes. In total, we have =2.4 mllon pars for testng. Here we use two rtera to evaluate our experment. One s the ran-1 auray versus dmensonalty. The other s the true postve (TP) versus the false postve (FP). Fg.12-a) shows the auray versus dmensonalty. It s shown that the proposed MK-RDA attaned margnally better auray-dmensonalty performane, onsstently orroboratng the underlyng oneture that the proposed many ernels method may help apture the ntrns multple manfolds lyng under the gven dataset, as dsussed n Seton III. Fg.12-b) gves the results on TP-FP urves. Here, we obtaned a lelhood matrx of elements by omparng eah test sample aganst all tranng samples. Then we appled varyng thresholds on the lelhood matrx, and ounted how many pars lassfed as postve are false postve and true postve pars. From the results shown n Fg.12-b), t s observed that PCA has the worst performane, nearly no dfferent from random guessng. From the omparson, we an learly see that the proposed MK-RDA has learly better performane on the true/false postve tests, wth onsstently better true/postve rates (TPR) over other data-drven fae reognton methods. b) TP-FP urves Fg.12. Expermental results on PubFg wld faes. LDA attaned 80.0%, LDA got a better sore of 81.5%, and LPP has the seond best auray of 83.1%. In omparson, our MK-RDA attaned the best auray of 91.5, learly better than the other data-drven approahes. C. Tests on PUBFIG Dataset The PUBFIG dataset s desgned to ompare varous algorthms aganst the human vson system. Its typal benhmar test an have as many as 20,000 pars of faes for omparson. However, n IoT-targeted srambled doman, human perepton an barely reognze any srambled faes, mang t meanngless to arry out ths human-ompared test. On the other hand, n the senaros of IoT applatons, usually we have tranng datasets on the server sde, mang t most lely as a leave--out experment. For ths reason, we need to desgn a new evaluaton sheme. In our experment, we seleted 52 subets wth 60 faes eah, and splt t randomly nto test and tranng datasets, wth eah havng 30 52=1560 faes. We then test all data-drven methods by omparng eah test fae aganst all VI. CONCLUSION In onluson, we have dentfed a new hallenge n srambled fae reognton orgnated from the need for bometr verfaton n emergng IoT applatons, and developed a salene-aware fae reognton sheme that an wor wth haot patterns n the srambled doman. In our method, we onetured that srambled faal reognton ould generate a new problem n whh many manfolds need to be dsovered for dsrmnatng these haot sgnals, and we proposed a new ensemble approah Many-Kernel Random Dsrmnant Analyss (MK-RDA) for srambled fae reognton. We also norporated a salene-aware strategy nto the proposed ensemble method to handle haot faal patterns n the srambled doman, where random seleton of features s based towards semant omponents va salene modellng. In our experments, the proposed MK-RDA was tested rgorously on three standard human fae datasets. The expermental results suessfully valdated that the proposed sheme an effetvely handle haot sgnals and drastally mprove the reognton auray, mang our method a promsng anddate for emergng IoT applatons. REFERENCES [1] Sngh, A. ; Karanam, S. ; Kumar, D. "Construtve Learnng for Human-Robot Interaton", IEEE Potentals, Vol 32, Issue 4, 2013, Page(s): [2] Jayatlae, D. ; Iseza, T. ; Teramoto, Y. ; Eguh, K. ; Suzu, K. "Robot Asssted Physotherapy to Support Rehabltaton of Faal Paralyss", IEEE Trans Neural Systems and Rehabltaton Engneerng, Vol. 22, Issue 3, 9

11 2014, Page(s): [3] MDuff, D. ; Kalouby, R.E. ; Pard, R.W. "Crowdsourng Faal Responses to Onlne Vdeos", IEEE Trans Affetve Computng, Vol 3, Issue 4, 2012, Page(s): [4] Fle, S.; Strasser, W. "Smart Camera Based Montorng System and Its Applaton to Asssted Lvng", Proeedngs of the IEEE, On page(s): Volume: 96, Issue: 10, Ot [5] A. Melle, J.-L. Dugelay, Sramblng faes for prvay proteton usng baground self-smlartes, Pro IEEE Internatonal Conferene on Image Proessng (ICIP), 2014, pp [6] Z. Ern, M. Franz, J. Guaardo, S. Katzenbesser, I. Lagend, T. Toft, Prvay-Preservng Fae Reognton, Pro. Nnth Int l Symp. Prvay Enhanng Tehnologes (PETS 09), 2009, pp [7] T. Honda, Y. Muraam, Y. Yanaghara, T. Kuma, T. Funo, Herarhal mage-sramblng method wth sramble-level ontrollablty for prvay proteton, Pro. IEEE 56th Internatonal Mdwest Symposum on Cruts and Systems (MWSCAS), 2013, pp [8] A. Erdly, T. Bart, P. Valet, T. Wnler, B. Rnner, Adaptve Cartoonng for Prvay Proteton n Camera Networs. Pro. Internatonal Conferene on Advaned Vdeo and Sgnal Based Survellane, 2014, pp.6. [9] F. Dufaux, T. Ebrahm, Sramblng for Vdeo Survellane wth Prvay, Pro Conferene on Computer Vson and Pattern Reognton Worshop, Washngton, DC, USA, 2006, pp [10] F. Dufaux, Vdeo sramblng for prvay proteton n vdeo survellane: reent results and valdaton framewor, Proeedngs of SPIE, Vol. 8063, 2011, pp.14. [11] T. Wnler, B. Rnner, Seurty and Prvay Proteton n Vsual Sensor Networs: A Survey, ACM Computng Surveys, Vol.47, Issue 42, 2014, pp.1. [12] A. Erdly, T. Bart, P. Valet, T. Wnler, B. Rnner, Adaptve Cartoonng for Prvay Proteton n Camera Networs. Pro. Internatonal Conferene on Advaned Vdeo and Sgnal Based Survellane, 2014, pp.6. [13] Y. Wang, T. L, Study on Image Enrypton Algorthm Based on Arnold Transformaton and Chaot System, Pro Internatonal Conferene on Intellgent System Desgn & Engneerng Applaton, 2010, pp [14] Z. Tang, X. Zhang, Seure Image Enrypton wthout Sze Lmtaton Usng Arnold Transform and Random Strateges, Journal of Multmeda, Vol. 6, No. 2, Aprl 2011, pp [15] Y. Rahulamathavan, R. C.-W. Phan, J. A. Chambers, D. J. Parsh, Faal Expresson Reognton n the Enrypted Doman Based on Loal Fsher Dsrmnant Analyss, IEEE Tran. Affetve Computng, Vols. 4, Issue 1, 2013, [16] J. Wrght, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, Robust Fae Reognton va Sparse Representaton, IEEE Trans. Pattern Analyss and Mahne Intellgene, vol. 31, no. 2, pp , Feb Also see CVPR [17] C. Shan, S. Gong, and P. MOwan. Faal expresson reognton based on Loal Bnary Patterns: A omprehensve study. J. IVC, 27(6): , [18] Png Lu, Shzhong Han, Zbo Meng, Yan Tong, "Faal Expresson Reognton va a Boosted Deep Belef Networ", CVPR [19] Dong Chen, Xudong Cao, Fang Wen, Jan Sun. Blessng of Dmensonalty: Hgh-dmensonal Feature and Its Effent Compresson for Fae Verfaton. Computer Vson and Pattern Reognton (CVPR), [20] Peras, P. ; Passals, G. ; Theohars, T. ; Kaadars, I.A. "3D Faal Landmar Deteton under Large Yaw and Expresson Varatons", IEEE Trans Pattern Analyss and Mahne Intellgene, Vol 35, Issue 7, 2013, Page(s): [21] Taher, S.; Patel, V.M.; Chellappa, R., "Component-Based Reognton of Faes and Faal Expressons", IEEE Trans Affetve Computng, Vol 4, Issue 4, 2013, Page(s): [22] R. Hsu, A. Jan, Semant fae mathng, IEEE Internatonal Conferene on Multmeda and Expo, 2002, pp.145. [23] T. F. Cootes, G. J. Edwards, C. J. Taylor, Atve appearane models, IEEE T rans. Pattern Analyss & Mahne Intellgene, Vol. 23, No. 6, 2001, pp.681. [24] B. Draper, K. Bae, M. Bartlett, J. Beverdge, Reognzng faes wth PCA and ICA, Comput. Vson Image Understandng, Vol.91,Issue 1-2, 2003, pp.115. [25] A.M. Martnez, A.C. Ka, PCA versus LDA, IEEE Trans. Pattern Analyss & Mahne Intellgene, Vol. 23, No. 2, Feb. 2001, pp [26] M. H. Yang, Kernel Egenfaes vs. ernel Fsherfae: fae reognton usng ernel methods, Internatonal Conferene on Automat Fae and Gesture Reognton, 2002, pp.215. [27] B. Tenenbaum, V. Slva, J. Langford, A global geometr framewor for nonlnear dmensonalty, Sene, Vol.290, No.5500, 2000, pp [28] S. Rowes, L. Saul, Nonlnear dmensonalty reduton by loally lnear embeddng, Sene, Vol.290, No.5500, pp [29] M. Beln, P. Nyog, "Laplaan Egenmaps and Spetral Tehnques for Embeddng and Clusterng", Advanes n Neural Informaton Proessng Systems, 2001, pp.14. [30] X. He, S. Yan, Y. Hu, P. Nyog, H. J. Zhang, Fae Reognton Usng Laplaanfaes, IEEE Trans. Pattern Analyss & Mahne Intellgene, Vol. 27, No. 3, Mar. 2005, pp.1. [31] Xaofe He, Deng Ca and Partha Nyog, "Tensor Subspae Analyss", Advanes n Neural Informaton Proessng Systems 18 (NIPS), Vanouver, Canada, De [32] D. Foster, S. Kaade, T. Zhang, Mult-vew dmensonalty reduton va anonal orrelaton analyss, TTI-C Tehnal Report, TTI-TR , [33] Hedge et al., Random Proetons for Manfold Learnng, NIPS 2008 [34] Hou C, Zhang C, Wu Y, et al. Multple vew 10

12 sem-supervsed dmensonalty reduton. Pattern Reognton, 2010, 43(3): [35] T. Xa, D. Tao, T. Me, and Y. Zhang. Multvew spetral embeddng. IEEE Transatons on Systems, Man, and Cybernets, Part B: Cybernets, 40(6): , [36] Y. Han, F. Wu, D. Tao, J. Shao, Y. Zhuang, and J. Jang. Sparse unsupervsed dmensonalty reduton for multple vew data. IEEE Transatons on Cruts and Systems for Vdeo Tehnology, 22(10): , [37] Ln Y Y, Lu T L, Fuh C S. Multple ernel learnng for dmensonalty reduton. IEEE Transatons on Pattern Analyss and Mahne Intellgene, 2011, 33(6): [38] A. Zen and C. S. Ong. Multlass multple ernel learnng.in Internatonal Conferene on Mahne Learnng, [39] R. Jang, D. Crooes, "Deep Salene: Vsual Salene Modellng va Deep Belef Propagaton", AAAI 2014,, Quebe, Canada, July [40] T. Sm, S. Baer, M. Bsat, The CMU Pose, Illumnaton, and Expresson (PIE) Database, Pro. IEEE Int l Conf. Automat Fae and Gesture Reognton, May [41] M. Lyons, J. Budyne, and S. Aamatsu, Automat Classfaton of Sngle Faal Images, IEEE Trans. Pattern Analyss and Mahne Intellgene, vol. 21, no. 12, pp , De [42] Neera Kumar, Alexander C. Berg, Peter N. Belhumeur, and Shree K. Nayar,"Attrbute and Smle Classfers for Fae Verfaton," Internatonal Conferene on Computer Vson (ICCV), [43] Gary B. Huang, Vdt Jan, and Er Learned-Mller. "Unsupervsed ont algnment of omplex mages," Internatonal Conferene on Computer Vson (ICCV), [44] Png Lu, Shzhong Han, Zbo Meng, Yan Tong, "Faal Expresson Reognton va a Boosted Deep Belef Networ", CVPR Rhard Jang s urrently a Leturer n the department of Computer Sene and Dgtal Tehnologes, Northumbra Unversty, Newastle upon Tyne, Unted Kngdom. He reeved hs PhD n Computer Sene from Queen s Unversty Belfast, Belfast, UK, n July After hs PhD study, he has been worng n Brunel Unv., Loughborough Unv., Swansea Unv., Unv. of Bath and Unv. of Sheffeld. He oned Northumbra n May Hs researh nterests manly resde n the felds of Artfal Intellgene, Man-Mahne Interaton, Vsual Forenss, and Bomedal Image Analyss. Hs researh has been funded by EPSRC, BBSRC, TSB, EU FP, and ndustry funds, and he has authored and oauthored more than 40 publatons. Somaya Almaadeed reeved the Ph.D. degree n omputer sene from the Unversty of Nottngham, Nottngham, U.K., n She has been a Vstng Fellow wth Northumbra Unversty, Newastle upon Tyne, U.K., sne She s urrently wth the Department of Computer Sene and Engneerng, Qatar Unversty, Doha, Qatar, as an Assstant Professor, where she s nvolved n researh on bometrs, wrter dentfaton, mage proessng, and doument analyss. She has been awarded a number of grants, and has authored around 40 papers. Dr. Almaadeed s a member of dfferent nternatonal omputer sene ommttees. Her team reeved the Best Performane Award n the 2011 Internatonal Conferene on Doument Analyss and Reognton s Sgnature Verfaton Competton and Mus Sores Competton. Ahmed Bourdane reeved the Ingeneur d État degree n eletrons from Eole Natonale Polytehnque of Algers (ENPA), Algera, n 1982, the M.Phl. degree n eletral engneerng (VLSI desgn for sgnal proessng) from the Unversty of Newastle-Upon-Tyne, U.K., n 1988, and the Ph.D. degree n eletral engneerng (omputer vson) from the Unversty of Nottngham, U.K., n From 1992 to 1994, he wored as a Researh Developer n telesurvellane and aess ontrol applatons. In 1994, he oned Queen s Unversty Belfast, Belfast, U.K., ntally as Leturer n omputer arhteture and mage proessng and then as a Reader n omputer sene. He beame a Professor n Image Engneerng and Seurty at Northumbra Unversty at Newastle (U.K.) n Hs researh nterests are n magng for forenss and seurty, bometrs, homeland seurty, mage/vdeo watermarng and ryptography. He has authored and o-authored more than 200 publatons. Danny Crooes reeved the B.S. degree n Mathemats and Computer Sene n 1977, and the Ph.D. degree n Computer Sene n 1980, both from Queen s Unversty Belfast. He beame Professor of Computer Engneerng n 1993 at Queen s Unversty Belfast, Belfast, U.K., and was Head of Computer Sene from He s urrently Dretor of Researh for Speeh, Image and Vson Systems at the Insttute of Eletrons, Communatons and Informaton Tehnology, Queen s Unversty Belfast. Hs urrent researh nterests nlude the use of novel arhtetures (GPUs and FPGAs) for hgh performane speeh and mage proessng. Professor Crooes s urrently nvolved n proets n automat shoeprnt reognton, speeh separaton and enhanement, and medal magng. Professor Crooes has some 220 sentf papers n ournals and nternatonal onferenes. M. Emre Celeb reeved the B.S. degree n omputer engneerng from the Mddle East Tehnal Unversty, Anara, Turey, n 2002 and the M.S. and Ph.D. degrees n omputer sene and engneerng from The Unversty of Texas at Arlngton, Arlngton, TX, USA, n 2003 and 2006, respetvely. He s urrently a Professor wth the Department of Computer Sene, Unversty of Central Aransas, Conway, AR, USA. He has pursued researh n the feld of mage proessng and analyss. He has publshed more than 130 artles n ournals and onferene proeedngs. Hs reent researh s funded by grants from the Natonal Sene Foundaton. 11