Liptracking and MPEG4 Animation with Feedback Control

Transcription

1 Lptrackng and MPEG4 Anmaton wth Feedback Control Brce Beaumesnl, Franck Luthon, Marc Chaumont To cte ths verson: Brce Beaumesnl, Franck Luthon, Marc Chaumont. Lptrackng and MPEG4 Anmaton wth Feedback Control. ICASSP 06: IEEE Internatonal Conference On Acoustcs, Speech, and Sgnal Processng, May 2006, Toulouse, France, France. 2, pp , <lrmm > HAL Id: lrmm Submtted on 9 Feb 2007 HAL s a mult-dscplnary open access archve for the depost and dssemnaton of scentfc research documents, whether they are publshed or not. The documents may come from teachng and research nsttutons n France or abroad, or from publc or prvate research centers. L archve ouverte plurdscplnare HAL, est destnée au dépôt et à la dffuson de documents scentfques de nveau recherche, publés ou non, émanant des établssements d ensegnement et de recherche franças ou étrangers, des laboratores publcs ou prvés.

2 LIPTRACKING AND MPEG4 ANIMATION WITH FEEDBACK CONTROL B. Beaumesnl, F. Luthon LUPPA IUT Informatque de Bayonne Château Neuf, Place Paul Bert Bayonne, France M. Chaumont LIRMM Computer Scence Department 161 rue Ada Montpeller, France ABSTRACT Ths artcle deals wth facal segmentaton and lptrackng wth feedback control by face model synthess. On ths topc, the search communty s dvded nto two parts : analyss and synthess. We want to use all the knowledge to create a global analyss/synthess chan where the mage analyss needs the 3D synthess and conversely. As t happens, applcatons lke face trackng or augmented realty need a rapd, robust and descrptve-enough soluton. Our soluton s based on a two step approach : the frst step s a real-tme facal segmentaton wth actve contour models and the second step recovers a 3D-face model n order to extract more precse parameters to adjust the frst step. The contrbuton of ths paper s to couple two research felds for creatng a real tme applcaton. The results obtaned show rapd and robust performances whch could be exploted n a more global real-tme face trackng applcaton. Fg. 1. Complete real-tme analyss/synthess framework. 1. INTRODUCTION We present a complete real-tme analyss/synthess framework allowng lp trackng for anmaton of a clone wth a sngle camera n unconstraned envronment (typ. webcam n the offce). The approach s based on lp segmentaton from a hue component computed wthn a non-lnear color space that s robust to lumnosty varatons. Internal and external actve contours are extracted, and then nterpreted to make real-tme realstc anmaton of a clone s mouth. To nterpret them, we use a feedback loop wth a 3D-face model. Ths model s shaped on the face and ts anmaton parameters gve a approxmaton of the mouth shape. Then we comparte t wth the poston of snake ponts to correct them. Fnally, robust snake ponts are sent to anmate a 3D-face model (typ. a MPEG4 avatar) (see Fg.1). Major methods don t mplement any feedback, and many of them gve us a average lp shape. They are often based on hgh level methods (AAM, ASM)[1], that requre large szed vdeo databases for off-lne learnng or use pattern matchng algorthm to track manual ponts of ntalsaton[2]. All of them return a mouth shape estmated by ther model. Our approach departs from the above mentoned methods n the sense that we want to use low-level algorthms that can adapt to any mouth shape but wthout a pror knowledge about the face. The mouth shape s more precse because we use lp gradents drectly on all mages. The results obtaned show real-tme and robust performances wthout any learnng stage, huge databases, or manual ntalsaton. 2. ANALYSIS : REAL TIME SEGMENTATION Our work assumes that we have at our dsposal an automatc tool for face detecton and trackng, so that an optmal framng of the speaker s face s at our dsposal (whch s requred n the case of vdeoconferencng for example). If ths s not the case, the only constrant s that the speaker should stay n front of the camera wth lttle moton (normal behavor n front of a webcam). In ths paper, we smply ask the speaker to seat drectly n front of the camera so that hs face covers the major part of the mage. Moreover, we ask hm to have the mouth closed (neutral poston) on the frst frame of the sequence, n order to ntalse the 3D-face model. The segmentaton algorthm s dvded nto four parts :

3 2.1. Color Converson For face and lp segmentaton, we use the LUX color space defned n [3]. Ths color space s non lnear wth respect to the RGB color components and s lttle senstve to lghtng varatons. It exhbts very dstnctly skn and lp hue areas. In ths color space, most of the color nformaton relatve to a human face s coded by the U component (red chromatcty) n the partcular case when R > L (L beng the lumnance Eq.1) (see Fg.2-a). L = (R + 1) 0.3 (G + 1) 0.6 (B + 1) (1) U = 256 L+1 R+1 f R > L, 255 otherwse or f L < δ. The threshold δ s ntroduced n order to detect very dark areas lke the nner sde of the mouth or nostrls. (2) 2.3. Regon Of Interest In order to ntalse actve contour models, we need to know where the mouth s n the current mage. For that, we use a lp tracker based on Lucas-Kanade algorthm appled on a few relevant ponts detected on the outer lp contour (see Fg.2-b). We can get mouth optmal boundng box by a horzontal mnmal enclosng area of all lp ponts. But, for the frst mage of the sequence, we cannot use t. Thus we ve mplemented a non lnear flter that fnds, n a sngle pass, the better canddate pont to be on the lp (named P ), usually at the lower left edge of the lps (see Fg.2-d). The flter s generated to detect the bgger horzontal reddsh form on the mage (Eq.3). A mouth optmal boundng box (named BB) can be obtaned by a smple scan of the connected component lp-hue area that ncludes P (pont gvng the maxmum value of M). 1 + j ν() a(j)m(j) f U() lps M() = 0 otherwse (3) 2.4. Actve Contour Model For estmatng the lps outer border, one actve contour [4] s ntalsed wth the BB detected as explaned above. Fg. 2. a) Hue U wth nonlnear flter mask ; b) Orgnal color mage, wth boundng-box and outer lp contour ponts ; c) Hue Hstogram ; d) 3-class map gven by the k-means algorthm, wth pxel P n whte 2.2. Clusterng Snce the hue dfference between face and lps s more contrasted n ths space, we can easly classfy pxels as lps or face (see Fg.2-c). For that purpose we use the k-means classfcaton algorthm. It works wth three classes : lp, face and background ; t explots two types of low-level nformaton : the mean value of hue n a gven neghborhood, and the maxmum devaton from ths mean value for any pxel n the neghborhood. Varous emprcal tests lead to set ad hoc parameter values that only depend on the mean-value of speaker s face hue n the whole mage. These parameters are used to ntalse the centers of the three classes (see Fg.2-d). After processng, ths technque gves us new precse centers of the three classes (because they are specfc of the mage). Fg. 3. Typcal snake convergence results for dfferent vsemes correspondng to French phonemes : top) vowels [ø], [e] plus neutral poston ; bottom) vowels [o], [a] and [] Ths snake s made of a fnte number of control ponts that are forced to undergo only vertcal dsplacements durng teratons. The ponts are ntalsed on cubc curves computed from the BB and from the lp map (to poston the lp corners). The forces used for snake convergence are the followng : Internal forces emanate from the shape of the snake (elastcty and stffness) External forces come from hgher level mage understandng processes (gradents) A contrant force that s specfc of the problem at hand (the snake s forced to converge towards the gravty center of the BB)

4 s.u s.v s = α u 0 u α v v } } ntrnsc parameters. r 11 r 12 r 13 t x r 21 r 22 r 23 t y r 31 r 32 r 33 t z } } extrnsc parameters. X Y Z 1 = T.M, wth αu = k u.f α v = k v.f. (4) After convergence of the outer snake, another snake (nner one) s ntalsed on the outer one, then shrnked by a non sotropc scalng w.r.t. the mouth center and takng nto account the actual thckness of lps. 3. USED OF AN A PRIORI INFORMATION : A 3D-FACE MODEL A contraro to the frst step of lp segmentaton (2), ths step s based on a pror knowledge : a face own a specfc 3D structure. In ths paper we use the 3D-face model named CANDIDE-3 [5] n order to model ths specfc 3D-face structure. The 3D-face model s extracted and then help us to lmt the face deformatons degrees and the mouth deformaton degrees. Thus, the 3D-face model allows to constran the prevous segmentaton soluton (obtaned thanks to actve contours) and then to keep the mouth poston n an acceptable soluton space. The 2D features ponts stemmng from the lp segmentaton may be nosy (2D postons are un-precse) and ther number s small. Our soluton to extract a 3D-face model (for more detals see [6]), wth the knowledge of 2D features ponts, takes care of those dffcult constrants and moreover s well-suted for real-tme applcatons. The soluton s dvded n two steps. The frst step recovers an approxmaton of the 3D-pose (secton 3.1), the second step recovers an approxmaton of the 3D-face model (secton 3.2) Pose approxmaton To extract the 3D-face model and the 3D-pose, we mnmze the dstance error E (see equaton 5) between the observed set of 2D mage ponts (u, v ) t } and the projected set of ponts (u, v )t }. The projected set of ponts (u, v )t } are obtaned by projectng all the correspondng 3D-face model vertex usng the T projecton (see equaton 4). E = (u u ) 2 + (v v ) 2. (5) The projecton of a set of ponts (u, v )t } belongng to the 3D model s the result of a shape dsplacement (S.σ), an anmaton dsplacement (A.α) and a projecton (T ) of an CANDIDE-3 average 3D-face model as expressed n the followng equaton : s.u s.v s = T.[M + S σ + A.α]. (6) } } M S and and A are respectvely the shape unt and the anmaton unt matrx, expressng the possble dsplacement of a vertex. The dsplacement ntensty s expressed by the weghtng vectors σ and α. More detals are gven n Ahlberg s report [5]. The computaton of projecton T (gven n equaton 5) s not an easy task ; projecton T should then be smplfed. Ths smplfcaton conssts n supposng that all the 3D vertex are n a same 3D plan. Ths s a realst hypothess when there are small depth dfferences between 3D ponts n comparson to the dstance between the camera and the face. By cancelng, from equaton 5, each E s partal dervatve n functon of T s parameters, we obtan 2 lnear systems (σ and α are set to zero). Those two systems are solved by usng classcal lnear algebra tools. Note that the matrx nvolved n that system s very small (matrx sze=4 4) ; ts computaton and ts nverson s very rapd Shape approxmaton Once T 2 4 projecton s computed, shape adaptaton s processed. The mnmzaton problem of equaton 5, s solved by fxng T 2 4 projecton. Equaton 5 s re-wrtten such that : E = [U N.S.σ] t.[u N.S.σ], (7) ( u wth U = v and N = T 2 3. ) ( M T. 1 ), We obtan a lnear system (equaton 8) by cancelng the partal dervatve E σ : ( S t.n t.n.s ).σ = S t.n t.u. (8) } } } } A B Soluton s such that σ = (A t A) 1 A t.b. The matrx nvolved n the system s small and sparse ; ts computaton and ts nverson are very rapd. The same reasonng may be done for α computaton.

5 4. FEEDBACK CONTROL 3D-face model extracton advantage s the rgdty compared wth snake convergence. Indeed the 3D-face model uses ther anmaton unts to fnd the mouth shape, thus the shape s more realstc than bad segmentaton. But t could be a dsadvantage compared wth a good convergence, because the shape descrbed by the model s an approxmaton. To use ths nformaton we have mplemented a feedback control consstng n three steps : error measurement (between some snake s pont and ts correspondng 3D pont model). backward transmsson of some ROIs that need to be processed agan (because of bad segmentaton) re-run of the segmentaton process on those ROIs 4.1. Error Measurement To establsh an error measurement we add two parameters to our algorthm : One to each snake pont. Ths parameter can have two values : 0 et 1. If t s 0 the pont can be moved to reduce snake energy, otherwse the pont s fxed. One to each lp s 3D model pont. It s used to stock error dstance wth ts correspondng snake pont. To compute error measurement we must fnd the correspondance table between snake ponts and 3D model ponts (just 18 snake ponts can be used, because the model s an MPEG4 3D model). Ths table s easy to compute, we must fnd the snake pont who are n the same row (or the nearest) that every lp s 3D model pont. After that, we can compute error by equaton 9 (where y s lne component) Error = 3Dmodel.y Snake f().y (9) The sgn of error value ndcates the poston of the snake compared to the poston of the 3D model Rescue treatment If the dstance between a snake control pont and the 3D mesh correspondng poston s clearly erroneous (see Fg.4- b) a rescue treatment s called to correct ts poston. Fg. 4. Regularsaton though the 3D model : a) Good match between segmentaton and the model ; b) Bad match : a control pont of the nner snake s clearly erroneous and s corrected by the 3D mesh correspondng poston. In ths case, all good snake ponts are fxed and we reposton the remander and choose a new method to treat them (typ. a new mage gradent to converge snake). Ths localrecomputaton can be made n a lttle area (a zoom-n n the neghborhood). After convergence of all snake ponts (wth a mnmum error crteron), we can anmate clone wth our MPEG4 ponts. To have a realstc anmaton, we use snake correspondng ponts to shape the model. 5. CONCLUSION Ths work demonstrates that one can buld a complete analyss-synthess chan that works n real-tme for 3D head anmaton, wthout usng sound nformaton nor face databases for learnng. The whole framework, mplemented n non optmsed C-code on an 386 processor at 1.4GHz, works n real-tme (.e. processng rate better than 30Hz). Another drecton of our current research s to segment not only the lps, but also other face features (namely nostrls, eyes, eyebrows and ears). Ths wll help us to adjust 3D pose (usng just the mouth ponts s a too poor nput data for 3D pose extracton) and may enable a better understandng of some spoken phonemes. Fnally, as we want to be able to anmate varous clones and propose a generc soluton, we are also workng on the use of MPEG4-complant 3D models (usng FDP and FAP, facal anmaton parameters) already partly present n our framework. 6. REFERENCES [1] I. Matthews and S. Baker, Actve appearance models revsted, Internatonal Journal of Computer Vson, vol. 60, no. 2, pp , November [2] F. Dornaka and F. Davone, Head and facal anmaton trackng usng appearance-adaptve models and partcle flters, n IEEE CVPR Workshop on Real-tme Vson for Human-Computer Interacton, Washngton DC, [3] Marc Levn and Franck Luthon, Nonlnear color space and spatotemporal mrf for herarchcal segmentaton of face features n vdeo, IEEE Trans. on Image Processng, vol. 13, pp , Jan [4] Andrew Wtkn Mcheal Kass and Demetr Terzopoulos, Snake : Actve contour models, Internatonal Journal of Computer Vson, vol. 1, pp , [5] J. Ahlberg, CANDIDE-3 - un updated parametersed face, Tech. Rep., Department of Electrcal Engneerng, Lnköpng Unversty, Jan [6] M. Chaumont and B. Beaumesnl, Robust and real-tme 3d-face model extracton, n IEEE Internatonal Conference on Image Processng, ICIP 2005, Sept. 2005, pp