Robus 3D Head Tracking by Online Feaure Regisraion Jun-Su Jang and Takeo Kanade Roboics Insiue, Carnegie Mellon Universiy, Pisburgh, PA 15213 {jsjang, k}@cs.cmu.edu Absrac This paper presens a robus mehod for racking he posiion and orienaion of a head in videos. The proposed mehod can overcome occlusions and divergence problems. We inroduce an online regisraion echnique o deec and regiser feaure poin of he head while racking. A se of poin feaures is regisered and updaed for each reference pose serving a muli-view head deecor. The online feaure regisraion recifies error accumulaion and provides fas recovery afer occlusion has ended, while prevening divergence problem which frequenly occurs in convenional frame-o-frame racking mehods. The robusness of he proposed racker is experimenally shown wih video sequences ha include occlusions and large pose variaions. 1. Inroducion 3D head racking is more han racking a face in video. I esimaes 3 roaion parameers and 3 ranslaion parameers of he head. Selecing a geomeric model o represen he head is imporan. The complexiy of he model affecs he working range of he racker, ease of iniializaion, and degree of compuaion. A planar model is simple, bu no covering large roaions [1 3], i works properly only when he head roaion was around he fronal view. To obain larger working ranges and more accurae moion, an ellipsoidal model [4], cylinder models [5 8] and more sophisicaed models [9, 1] have been sudied. Complicaed models can provide more accurae moion; however, hey generally require careful iniializaion as well as more compuaion. We have chosen a cylinder model o represen he 3D shape of a head. The cylinder model includes boh circular and ellipical cylinder. The simpliciy of he cylinder model provides robusness from iniializaion error compared o oher more sophisicaed models. A good racking mehod should be able o deal wih varying head poses, occlusions, illuminaion changes and facial expressions. Many mehods were proposed by using emplae updae and regisraion. Cascia e al. formulaed an image regisraion problem in he cylinder s exure map [5]. They used a linear combinaion of exure-warped emplaes and illuminaion emplaes o handle illuminaion changes in racking. Brown improved he exure-mapped cylinder approach by proposing adapive moion emplaes o enhance he moion beween successive frames and addiional emplaes o cover large head roaions [6]. Xiao e al. applied a dynamic emplae echnique in order o accommodae gradual changes in lighing and self-occlusion [8]. Some frames associaed cerain head poses were sored as references o preven error accumulaion due o he dynamic emplae. I is obvious ha wo conflicing sraegies, updaing emplaes and keeping reference emplaes, should be balanced. Updaing emplaes can cause error accumulaion and divergence in racking, while keeping reference emplaes canno accommodae appearance changes. Alhough emplae updae mehods are used in many sudies [3, 7, 8, 11], i is difficul o obain boh adapabiliy and sabiliy in racking performance. One easy way o avoid he divergence problem is racking-by-deecion [12,13]. A deecor can be applied o each individual frame o preven drif and divergence, which may occur in convenional frame-o-frame racking using he emplae updae echnique. However, in he 3D head racking problem, here is difficuly in making a universal deecor, which can cover appearance differences among individuals, wide ouof-plane roaion, illuminaion, ec. In his paper, we propose a cylinder model-based 3D head racker using he online feaure regisraion. The cylinder model covers a wide range of head moions and he online feaure regisraion deals wih he racking-by-deecion issue menioned above. To avoid making a generic head deecor, we only focus on he curren individual in racking sequence, since i makes deecion problems much easier. The online feaure regisraion echnique sores he feaure poins of a head for each reference pose while racking. The overall racking sysem is shown in Figure 1. An iniial esimaion of head pose uses Bayesian angen shape model (BTSM) face alignmen mehod [15]. The BTSM 978-1-4244-2154-1/8/$25. c 28 IEEE
frame Corresponding SIFT poins maching Moion esimaion beween frames Normalized correlaion μ μ = Iniial pose wih BTSM O Y Z X μ Δμ μ frame Kalman filer Corresponding Head poins Pose esimaion feaure wih head DB maching μ Poin on cylinder surface [ x, y, z] T Known pose a 1 Perspecive projecion μ pl Bes mached emplae Head Feaure DB Regisraion /Updae Figure 1. Overall archiecure of he 3D head racker p u = v correspondence u p = v Figure 2. Cylinder moion esimaion using known corresponding poin pairs face alignmen gives a se of facial poins, so ha we can esimae he 3D pose of he head. Even hough i only works properly for fronal face, i is more informaive o iniialize he 3D head pose han general face deecors which give a bounding box. Once a fronal head is iniialized, our racker works auomaically. The scale invarian feaure ransform (SIFT) [14] is used o exrac and mach feaure poins. The se of SIFT feaure poins forms a view-based head feaure daabase (DB), which provides robus performance in occlusions. Normalize correlaion mehod is used o find corresponding poins beween successive frames ogeher wih SIFT. Kalman filer is hen applied o combine he esimaed moion beween successive frames and he esimaed pose wih head feaure DB. 2. Cylinder Moion Esimaion 2.1. Rigid moion under perspecive projecion This secion presens a mehod ha esimaes a rigid cylinder moion, Δμ = [Δθ x, Δθ y, Δθ z, Δx, Δy, Δz] T, where Δθ x, Δθ y, Δθ z represen 3 roaions (pich, yaw, and roll) and Δx, Δy, Δz represen 3 ranslaions. Le a poin in an image a ime be p =[u,v ] T. Given a known pose μ of a cylinder a ime 1, we can calculae 3D poin X =[x,y,z ] T in he world coordinae by assuming ha he poin p is on he cylinder surface. The moion beween X and X can be represened by using wis represenaion [16]: x y z 1 = 1 Δθ z Δθ y Δ x Δθ z 1 Δθ x Δ y Δθ y Δθ x 1 Δ z 1 x y z 1 (1) An expeced projecion poin, p, is calculaed by using he 3D poin and moion vecor Δμ. [ ] p x y = Δθ z + z Δθ y +Δ x x Δθ z + y z Δθ x +Δ y f, x Δθ y + y Δθ x + z +Δ z where f is camera focal lengh. We assume ha he focal lengh is unknown. If he deph variaion of a cylinder is relaively smaller han he disance beween he cylinder and he camera, he unknown focal lengh does no cause a large error in pose esimaion [7]. The equaion (2) maps p o new locaion p. Assuming ha he corresponding poin pair, p and p, is found by image observaion, we can compue he moion vecor o minimize he sum of disance error, e, beween expeced and observed locaions of corresponding poin pairs: e = (2) N p i, p i,, (3) i=1 where N 3. Figure 2 shows cylinder moion esimaion mehod using known corresponding poin pairs.
We use weighed leas squares (WLS) esimaion o find moion vecor. The WLS deals wih ouliers which can be obained in he process of finding corresponding poin pairs. The weigh for each poin is updaed by using he disance error of he poin: w i, w i, exp( c e i, ), (4) where c is a posiive consan and e i, = p i, p i,. Every poin has a weigh value which indicaes how much he poin is consisen o he cylinder moion. The WLS is ieraively applied unil convergence. 2.2. Feaure maching To find he corresponding poin pairs beween wo images, wo kinds of feaure maching approaches are used. Firs, SIFT is used o obain disincive feaure poins in images. I is invarian under scaling, roaion and limied view change. Each feaure poin has a 128-dimensional descripor for maching. The advanage of SIFT feaure is ha i provides wide baseline maching wih low false posiive rae. Therefore, i is suiable o make up a head feaure DB (see secion 3), which is o deec head feaure poins regardless of he pose difference beween wo images. Second, we generae regularly placed feaure poins inside of he head region based on curren pose esimaion. Normalized correlaion mehod is applied o find maching poins. A recangle region cenered on a grid-ype poin is exraced o compue normalized correlaion o adjacen regions. A poin which has he maximum correlaion value (larger han proper hreshold) is chosen as a corresponding poin by searching over he adjacen regions of each poin. We consider his feaure as a complemenary feaure o SIFT. Because SIFT feaure poins may no uniformly appear inside of he head region and he number of SIFT feaure poins varies wih image qualiy. We can conrol he number and he locaion of grid-ype feaure poins. Normalized correlaion works when moion beween wo images is relaively small, so ha candidae search regions should be assigned properly. Boh kinds of feaures are used o find corresponding poins beween successive frames. SIFT feaures are also used o find corresponding poins beween curren frame and head feaure DB presened in he following secion. 3. Online Feaure Regisraion The proposed racker gahers head feaure poins from a pas sequence o improve is racking performance in he fuure sequence. 2D image observaion of he head region varies a lo when he head moves wih large roaion, especially ou-of-plane roaion. Use of an iniial reference emplae hroughou he whole racking sequence is no recommended. For example, ypical head racking sars wih (a) SIFT feaure poins (b) Regularly placed feaure poins Figure 3. Two kinds of feaure poins he fronal face region as a reference emplae. When he head roaes abou axis Y, one of he eyes becomes invisible and he profile area of he head appears. The reference emplae does no cover he new appearing region, which may conain useful feaures o help racking. We use SIFT feaures o make up a head feaure DB. Alhough we successfully obain he SIFT feaure poins which indicae he same 3D poins beween wo images, he descripor may be no mached well when ou-of-plane roaion exceeds some bounds. To make he head feaure DB cover a large range of ou-of-plane roaion, muli-view approach is considered. Basically, SIFT feaures and an associae head pose are sored when curren esimaion of head pose comes close o one of cerain reference poses. Reference poses are decided in ou-of-plane roaions (pich and yaw), because in-plane roaion (roll) is covered by SIFT feaures which is invarian o ha roaion. A head feaure DB consiss of many view-based emplaes, and each view emplae conains a se of SIFT feaures and head pose. Generaed head feaure DB is used o esimae poses in he remaining frames. When inpu frame comes ino racker, a emplae which has he mos number of mached feaure poins is seleced as he bes mached emplae. The head pose of inpu is esimaed by using he bes mached emplae in he same way as described in secion 2. An example of a head feaure DB obained in a real racking sequence is shown in Figure 4. The locaions of he feaure poins are displayed wih cerain views of he head. I is no necessary for feaure poins o have corresponding poins among emplaes. Each feaure poin in he DB has an accumulaed weigh as a confidence value. When a poin is mached beween he curren frame and head feaure DB, he accumulaion of weigh is w acc i, = w acc i, + w i,, (5) where w i, is a weigh from he WLS in (4). A poin wih high weigh means ha i was mached frequenly and moved consisenly wih head moion. The accumulaed
μ = μ + u + α, (6) Figure 4. Example of a head feaure DB where α represens he process noise. This equaion sands for he ransiion of sae vecor, μ. Many sudies for racking problems use dynamics based on smooh movemen, which makes he predicion sage fail when a sudden rapid movemen occurs. We do no assume a smooh head movemen, so ha he sae ransiion equaion conains he conrol inpu u. The noise α is assumed o be normally disribued as, α N(,Q ). The covariance marix Q is assumed by a diagonal marix whose elemens are se by using roo mean square error (RMSE) of disance errors e i, in (4). Le h be a head pose calculaed wih he head feaure DB. Then he observaion equaion is derived as weigh is used o assign he iniial weigh in he WLS ieraions, herefore, he WLS ges rid of ouliers a an earlier ieraion. Feaure poins in he newly generaed emplae inheri he accumulaed weighs from feaure poins in he exising neighbor view emplae, if hey are mached. Once he online feaure regisraion mehod makes a head feaure DB o cover a large range of view, he obained DB can be seen as a muli-view head deecor for he curren individual. Our approach is differen from he previous sudies [6, 8], ha sore view-based emplaes o cover a large range of roaion. In heir mehods, boh a candidae head region in he inpu frame and he closes emplae mus be seleced properly, which is difficul when he head pose in he curren frame and ha of he seleced emplae are quie differen. Especially, divergence in racking occurs frequenly when he candidae head region is seleced inaccuraely. To recover divergence in racking, heir mehods need o pick a saring frame of divergence and re-iniialize by using general deecors. On he oher hand, our individual-specific head deecor works regardless of he pose difference. SIFT feaure poins are mached in he whole inpu frame so ha a candidae head region is no needed. Therefore, i fundamenally avoids divergence in racking problems. 4. Tracking wih Kalman Filer There are wo ways of esimaing he curren pose of a head in our mehod. The firs is from accumulaing moions beween successive frames and he oher is from esimaing pose of he curren frame using head feaure DB. Kalman Filer is applied o combine he wo kinds of informaion. Le u be a pose difference beween successive frames; i is regarded as a conrol inpu in Kalman filer framework. The sae ransiion equaion is derived as h = μ + β, (7) where β represens he observaion noise wih β N(,R ). Similar o Q, R is se by using RMSE in he pose esimaion process from he curren frame and he head feaure DB. The oupus of Kalman filer are esimaed pose μ and covariance marix P as a confidence measure of curren esimaion. The head feaure DB is updaed using P. The curren view emplae replaces an exising view emplae, if he curren esimaed pose is close o he pose of he exising emplae and he covariance marices saisfy following equaion: P < P pl, (8) where P pl means covariance marix of he exising emplae. P is sored as P pl afer emplae replacemen. The accumulaed weighs of feaure poins are inheried from he old emplae for mached poins. 5. Experimenal Resuls We esed he proposed racking sysem in hree experimens. Throughou he experimens, an ellipical cylinder wih a radius raio of 1.3 was used o cover he side regions of he head, including ears. 5.1. Sequences wih ground ruh The firs experimen was done using Boson Universiy daase which provided he ground ruh of he 3D pose [5]. We compared he pich, yaw, and roll esimaed by our racker o he ground ruh. Figure 5 shows he roaion parameers for wo differen sequences. We esed 45 sequences in he daase; he average esimaion errors for pich, yaw, and roll were 3.7, 4.6 and 2.1, respecively.
Pich Yaw 5 4 3 2 1 5 4 3 2 1 5 4 3 2 1 5 1 15 2 5 4 3 2 1 5 1 15 2 5 5 1 15 2 5 5 1 15 2 4 3 2 4 3 2 (a) Texure-mapped cylinder racker Roll 1 1 5 1 15 2 5 1 15 2 Sequence 1 Sequence 2 Figure 5. Comparison wih he ground ruh. Each column shows 3 roaion parameers from a sequence. Blue sold lines indicae esimaed resuls, and black dashed lines indicae he ground ruh. 5.2. Comparison wih exure-mapped racker For he second experimen, we compared our mehod o a exure-mapped cylinder racker which incorporaed a dynamic emplae and muli-view emplae regisraion [8]. Opical flow mehod was used o rack cylinder surface regions. The dynamic emplae deal wih he appearance changes, however, i caused he divergence in racking. As shown in Figure 6(a), he racker chased a hand afer he hand occluded he head region. Their mehod needed reiniializaion by using a face/head deecor o recover he pose afer occlusion had ended, alhough muli-view emplaes were prepared. Because view-based exure emplaes were only available on he assumpion ha he curren candidae head region was exraced successfully. Our mehod overcame he occlusion and divergence problem as shown in Figure 6(b). Once he head region reappeared enough o mach inpu feaure poins from he regisered feaure DB, our racker immediaely recovered he pose. 5.3. Sequences wih large moion and occlusions As he hird experimen, he proposed racker was esed wih 4 real sequences ha conained large head roaions, parial occlusions and complee occlusions. Figure 7 shows some examples. In he firs frame 7(a), iniial head pose (b) Proposed racker Figure 6. Comparison beween he exure-mapped cylinder racker and he proposed racker was esimaed by using BTSM face alignmen. The racker covered wide ranges of roaions, 7(b), 7(c), 7(d), and he online feaure regisraion echnique generaed a head feaure DB o make an individual-specific head deecor. This deecor covered a large range of views observed in previous frames in he sequence. The racker showed robusness o parial occlusion in 7(e), 7(f). When he head was compleely occluded 7(g), he racker los he head and held he las successfully esimaed pose. In 7(h), he racker recovered he head pose immediaely when he head region sared o show parially. I should be noed ha he racker rapidly recovered he head pose, even hough he head reappeared wih a largely roaed view. The racker sared wih only fronal head informaion, afer ha, i learned he oher views of he head in racking sequence and consruced a muli-view head deecor o improve he racking perfor-
(a) (b) (c) (d) (e) (f) (g) (h) Figure 7. Tracking resuls under large roaions and occlusions. Blue poins indicae mached poins from he head feaure DB. mance. 6. Conclusions In his paper, we presened a robus 3D head racking sysem using online feaure regisraion. The proposed mehod incorporaes moion esimaion beween successive frames wih pose esimaion from he head feaure DB. The WLS mehod was used o rejec oulier feaure poins. Afer observing he curren individual s head movemen, our racker generaed an individual-specific head deecor. The obained deecor prevened racking error accumulaion and divergence, and recovered head pose rapidly when occlusion ended. For fuure work, we plan o research head racking wih non-rigid moion. Analyzing he muliple observaions of he same feaure poins is needed o discriminae he nonrigid poins from rigid poins. References [1] M. Black and Y. Yacoob, Recognizing facial expressions in image sequences using local parameerized models of image moion, IJCV, vol. 25, no. 1, pp. 23-48, 1997. [2] G.D. Hager and P.N. Belhumeur, Efficien region racking wih parameric models of geomery and illuminaion, IEEE Trans. PAMI, vol. 2, no. 1, pp. 12539, 1998. [3] Z. Zhu and Q. Ji, Real ime 3D face pose racking from an uncalibraed camera, in CVPRW, pp. 73, 24. [4] S. Basu, I. Essa and A. Penland, Moion regularizaion for model-based head racking, in ICPR, pp. 611-616, 1996. [5] M. La Cascia, S. Sclaroff and V. Ahisos, Fas, reliable head racking under varying illuminaion: An approach based on robus regisraion of exure-mapped 3D models, IEEE Trans. PAMI, 2. [6] L. Brown, 3D head racking using moion adapive exuremapping, in CVPR, 21. [7] G. Aggarwal, A. Veeraraghavan, and R. Chellappa. 3D facial pose racking in uncalibraed videos, in PRMI, pp. 515-52, 25. [8] J. Xiao, T. Kanade and J. Cohn, Robus full-moion recovery of head by dynamic emplaes and re-regisraion echniques, in FG, pp. 156-162, 22. [9] L. Lu, X.-T. Dai, G. Hager, A paricle filer wihou dynamics for robus 3D face racking, in CVPRW, pp. 7, 24 [1] M. Malciu and F. Preeux, A robus model-based approach for 3D head racking in video sequences, in FG, pp. 169-174, 2. [11] A.D. Jepson, D.J. Flee, T.F. El-Maraghi, Robus online appearance models for visual racking, IEEE Trans. PAMI, vol. 25, no. 1, pp. 1296-1311, 23. [12] M. Özuysal, V. Lepei, F. Fleure and P. Fua, Feaure harvesing for racking-by-deecion, in ECCV, pp. 592-65, 26 [13] M Grabner, H Grabner and H Bischof, Learning feaures for racking, in CVPR, 27. [14] D. G. Lowe, Disincive image feaures from scale-invarian keypoins, IJCV, vol. 2, no. 6, pp. 91-11, 24. [15] Y. Zhou, L. Gu and H.-J. Zhang, Bayesian angen shape model: Esimaing shape and pose parameers via bayesian inference, in CVPR, pp. 19-116, 23. [16] R.M. Murray, Z. Li, and S.S. Sasry, A Mahemaical inroducion o roboic manipulaion, CRC Press, 1994.