Real-ime avaar animaion seered by live body moion Oliver Schreer, Ralf anger, Peer Eiser, Peer Kauff, Bernhard Kaspar, Roman Engler Fraunhofer Insiue for elecommunicaions/heinrich-herz-insiu, Einseinufer 7, 0587 Berlin, Germany {Oliver.Schreer, Ralf.anger, Peer.Eiser, Peer.Kauff}@fraunhofer.hhi.de hp://ip.hhi.de -Sysems Inernaional GmbH, Am Kavalleriesand, 6495 Darmsad, Germany Bernhard.Kaspar@-sysems.com Deusche elekom AG, Laboraories, Erns-Reuer-Plaz 7, 0587 Berlin, Germany roman.engler@elekom.de Absrac. he fuure cusomer service provided by call cenres will be changed due o new web-based ineracive mulimedia echnologies. echnical suppor will be offered in a compleely new way by using advanced image processing echnologies and naural represenaion of virual humans. We presen a prooype sysem of an animaed avaar, which is seered by live body moion of he operaor in a call cenre. he hand and head moion is ransferred direcly o he avaar a he cusomer side in order o suppor a more naural represenaion of he virual human. he sysem racks he operaors hands and he head moion quie robus in real-ime wihou specific iniializaion based on a monocular camera. Inroducion racking human bodies and faces has received a lo of aenion in compuer vision research in he las years. he reason is, ha a number of ineresing applicaions have been raised in he pas such as moion capure for enerainmen indusry or medical purposes, human-machine ineracion, auomaic surveillance sysems or ineracive web-based commercial applicaions. A lo of robus approaches have been developed, which are now going o be carried over o commercially available sysems. herefore, a lo of new challenges like robusness under differen lighning condiions, independency from differen users, eased use wihou sophisicaed iniializaion procedures urn ou. In his paper, a call cenre applicaion will be presened, where an operaor is represened o he cusomer via an animaed avaar. he head and body moion of he operaor is immediaely ransferred o he virual human by using robus skin colour segmenaion and facial feaure racking algorihms. he complee image processing is performed on monocular colour video images in real-ime on a convenional PC a full video frame rae. racking of human bodies and faces as well as gesure recogniion has been sudied for a long ime and many approaches can be found in he lieraure. A survey on human body racking is given in []. A real-ime body racking sysem using
srucured ligh wihou use of addiional markers is presened in []. his consrain is paricularly imporan in user-friendly applicaions. Hand gesure recogniion is reviewed in [] and a D hand gesure recogniion sysem is presened in [4]. racking he user s face and esimaing is pose from monocular camera views is anoher imporan issue. As he D informaion is los during perspecive projecion ono he image plane, some model assumpions have o be applied in order o esimae he D pose [5]. In [6], some specific face feaures are racked in order o recover he orienaion and posiion of he users head. Oher mehods use IR illuminaion, which simplifies racking of he eyes [7]. In he considered scenario of animaing a virual human, he accuracy of D posiions of head and hands does no play ha imporan role, bu he immediae ransfer of general live moion o he virual human is required such as waving hands, poining gesures or nicking he head. his allows some simplificaions in erms of accuracy, bu inroduces addiional challenges in erms of smoohness and reliabiliy of he animaed moion. In he nex secion, he sysem of a call cenre applicaion is presened. Alhough his sysem also includes speech analysis, he focus of his paper is on image processing. Hence, he skin-colour segmenaion and facial feaure exracion is repored briefly. Based on he specific aims of his applicaion, he reconsrucion of he hand and head posiion and he head orienaion is explained. Finally, resuls are shown and he aricle ends wih a conclusion. Sysem overview As shown in Fig., he considered applicaion provides for an operaor on he sender side, who is capured by a video camera mouned on op of he display. Based on he video informaion, he posiion of he hands and he head orienaion are regisered and convered o sandard facial and body animaion parameers as sandardised MPEG-4 (Par (Visual)). Video In Audio In Video Analysis Handsegmenaion and racking Feaure Poin racking Viseme Generaor Audio Grabber Operaor Avaar Animaion Parameer (FAP s, BAP s) Filer D Feaure Conversion Viseme Conversion Nework Inerface Nework Nework Inerface Cusomer Player Display Characer Speaker Fig.. Block diagram of he call cenre applicaion using an animaed avaar In addiion, he voice is capured and he audio signal is ransmied o he cusomer a he receiving side. he capured voice is analyzed and visemes (a visual represenaion of phonemes) are generaed o animae he lip shape corresponding o differen sounds. Based on hese visemes, a naural lip movemen of he avaar can be
Real-ime avaar animaion seered by live body moion reproduced. Beside he depiced modules in Fig., addiional modules are implemened in order o provide more complex and naural animaion. For insance, while loading predefined sequences of moion parameers from a library, he operaor can acivae specific high-level feaure animaions like opening sessions, leave-aking scenes or poining gesures. If racking and high-level feaure animaion are boh swiched off due o cerain reasons, he naural behaviour of he animaed avaar is improved by sligh random moion of he head and pars of he face (e.g. eye blinking). General facial expressions like friendliness, anger or fear, which are difficul o exrac from live video images can be chosen by an addiional expression generaor. All head and hand moions, facial expressions and lip movemen are described via body animaion parameers (BAP) and facial animaion parameers (FAP) according o he definiion in he MPEG-4 sandard. he complee se of animaion parameers and he audio signal are hen ransmied o he cusomer on he receiving side. As no video informaion is necessary, his approach is efficien in erms of he required bandwidh and herefore appropriae in web-based cusomer care applicaions. he cusomer is viewing a virual human represened by an avaar, which is seered by he live body moion and speech. In he nex wo secions, more deails will be presened regarding he image processing par of he sysem. he skincolour segmenaion algorihm ha is used for racking hand and head regions is explained firs. hen, he algorihm, which derives he D posiion of he hands from he corresponding segmens and which is used for seering he hand movemens of he avaar, is presened. Subsequenly, he algorihm for racking head feaures is described, and i is explained how i is used for animaing head roaion. Finally, some experimenal resuls are shown and a conclusion ends he aricle. Skin-colour segmenaion and racking he colour of human skin is a sriking feaure o rack and o robusly segmen he operaors hands and face. I is exploied, ha human skin colour is independen on he human race and on he wavelengh of he exposed ligh [8]. he same observaion can be made considering he ransformed colour in common video formas. Hence, he human skin-colour can be defined as a global skin-colour cloud in he colour space [9]. his is uilised successfully in a fas and robus region-growing based segmenaion algorihm [0]: he skin colour segmenaion is performed on predefined hresholds in he U,V-space of he video signal. hen, a blob recogniion idenifies he hands and he head in he sub-sampled image. Based on his informaion, a region growing approach segmens he complee skin-colour region of he hands and he head quie accuraely (see Fig. ). Blob deecion of hands and head on sub-sampled image racking on original image size Skin-colour segmenaion Iniialisaion racking Segmenaion Fig.. Block diagram of he segmenaion and racking mehod of he hands
4 he iniialisaion is performed and segmenaion and racking sar as soon as hree separaed skin-colour blobs are deeced. he blobs will be assigned o hands and he head supposing ha hands are iniially below he head, which holds for general poses. he approach achieves real-ime performance due o racking on sub-sampled images and skin-colour segmenaion limied o bounding boxes circumscribing he hands and he head. Resuls are presened in secion 6, Fig. 7 and Fig. 6. In his conex, i is a specific problem o resolve overlapping beween differen skin-coloured regions such as hands and he head. If a hand has conac wih he face, he following process is carried ou: In addiion o he hands, he head blob of he paricipan resuling from he iniialisaion phase is racked as well in he sub-sampled image, using a hird bounding box. If one of he hand boxes overlaps wih he head box, hen only he non-overlapping par of he hand box is considered for racking he cenre of graviy. More deails can be found in [0]. 4 Facial feaure exracion he aim of facial feaure racking is o obain sufficien informaion in order o derive a convincing and reliable roaion of he operaor s head. As a resul from he segmenaion and racking algorihm described in he previous secion, he bounding box of he operaor s head is used as a saring poin for facial feaure exracion. he skin-coloured pixels inside he bounding box are marked and a sandard feaure racker is applied o his limied face region. he feaure racker is based on a wosep approach. Firs, relevan feaures are seleced by using corner operaors such as Moravec or Harris deecors. Secondly, he seleced feaures are hen racked coninuously from frame o frame by using a feaure dissimilariy measure. his guaranees, ha feaures are discarded from furher racking in he case of occlusions. Even in he case of a roaing head some good feaures become disored due o perspecive changes or even become invisible and ge los. In Fig., markers of seleced feaures are shown in he face region in hree succeeding frames. he big cross assigns he median value of all skin coloured pixels. he considered skin colour region is marked by he line around he face. Due o he blond hairs of he es person, he hairs are recognized as well as skin. Fig.. Facial feaure racking resul of hree succeeding frames
Real-ime avaar animaion seered by live body moion 5 5 Reconsrucion of head orienaion and hand posiions he main goal of he applicaion from secion is o animae an avaar by human body moion capured from real-ime video. Hence, accuracy in erms of correc D posiions of he hands or precise nick and urn angles of he head are no required. However, reliable, convincing and smooh moions are imporan in order o suppor naural represenaion of a dynamic virual human. On one hand, his fac faciliaes he esimaion of animaion parameers in some way and can be exploied for simplificaions. On oher hand, he exraced parameers have o be filered and ouliers mus be discarded in order o provide smooh moion. he head orienaion is derived from he resuls provided by a facial feaure racker. Based on a few robusly racked facial feaures, he head orienaion can be analysed by comparing he relaive moion of facial feaures o he projeced D moion of he head. his D moion is calculaed by he mean change of posiion of all face pixels in succeeding frames. he ask is o disinguish beween head roaion and pure ranslaion. In he case of a pure ranslaion, he relaive moion of each feaure compared o he moion of he mean of all face pixel posiions should be zero. Jus he opposie holds in he case of he roaion. In his case, he moion of he mean of all face pixel posiions should be significanly smaller han he relaive moion of he facial feaures. his behaviour of facial feaure poins allows a simple approximaion of he head roaion in horizonal (urn angle) and verical direcion (nick angle). he median value of horizonal and verical coordinaes of facial feaure, whereas he mean of all face pixel posiions is denoed. he relaive change of facial feaure poins (horizonal/verical) is hen poins is assigned wih ( m, n ) by ( p, q ) calculaed by Equ. and he change of horizonal and verical roaion is approximaed by Equ.. A scale facor γ is inroduced o adop he pixel uni o angle. ( m m ) ( p p ) v = ( n n ) ( q q ) u =. (), π π ( γ u ), ϕ = ( γ v ) ϕu = sin. () sin 80 v As i is obviously no possible o calculae he absolue roaion from his mehod, drif effecs may occur. his can be avoided by coninuously weighing he curren urn (or nick) angle by some facor less han. As he cenral viewing direcion is he mos relevan, he animaed head will adop o his posiion afer a while. he posiions of he hands are derived from he resul of he skin-colour segmenaion and racking module. I provides reasonable and sable resuls of he moion of boh hands in he D image plane. o achieve naural avaar movemens, he D posiions have o be ransferred ono he D model of he avaar. Since wo degrees of freedom of he hand posiion are available, jus a simplified moion model can be implemened in his case. herefore he sysem is based on he assumpion, ha he hands of he animaed avaar mainly move wihin a D plane in he D space. hus, aking ino accoun some furher physical consrains such as he resriced range of elbow join and he proporions beween he upper arm and he forearm, he posiion of he avaar s hands can be compued from hese D racking resuls. Neverheless, a D o D reprojecion is necessary, which requires some knowledge 80
6 abou he imaging process. he general projecion equaion for a D poin M w in world coordinaes ino a D poin m in image coordinaes is as follows: ~ sm ~ = PM w wih p P = p p p p p 4 4 4 axr + u0r = a yr + v0r r a + u x x y y z 0 z a + v 0 z, r R = r r he marix P is called he general projecion marix conaining he inernal camera parameers (u 0, v 0, a u, a v ) and he exernal parameers (R, =( x, y, z )) relaing he world coordinae sysem o he camera coordinae sysem. For boh componens of he D image poin, we ge he following wo reconsrucion equaions: () ( u p ) M + p u p = 0, ( ) ( p v p ) M + p v p 0 w 4 4 w 4 4 = p (4) hese wo equaions have in general hree unknowns, he hree componens of he D poin, which can only be solved, if a second image of anoher camera is available, he so-called sereo case. As menioned previously, i is assumed, ha he hand posiion is fixed in a predeermined deph plane, which resuls in a fixed and known Z w coordinae. In his case, a reconsrucion becomes possible. Furhermore, we are able o avoid precise calibraion as jus a general ransformaion beween he D image plane and he D plane a fixed deph is required. he seup of he camera relaed o he world coordinae sysem is shown in Fig. 4. If we assume jus a horizonal roaion of he camera, a ranslaional shif in y- and z-direcion and a coinciding origin of he image coordinae sysem wih he principal poin, we ge he following simplified projecion marix (see Equ. 5). he reconsrucion equaion (Equ. 4) for he desired X w and Y w coordinaes becomes quie simple as shown in Equ. 6. a X b Y W ax 0 0 0 0 0 P ' 0 a yr ayr ay y, wih R = 0 r r, u0 = v0 = 0 r r 0 z r r W = x = + a Y W + b = 0 u b = a r v v a v r v b a = ( u r Z w + u z ) ( r v r ) Z w + av y v z + a = 0 wih a = a, a = u a r, (6) wih he resuling reconsruced poin in world coordinaes is finally: X W b ba ab, YW = b b, = = a (7) 0 () (5) camera Y c φ x Z c display Z w Y w origin Fig. 4. Simplified camera seup of he considered scenario
Real-ime avaar animaion seered by live body moion 7 he horizonal roaion and he ranslaion of he camera have been measured approximaely, whereas he horizonal and verical scale facor have been chosen experimenally. 6 Prooype resuls he presened approach is fully inegraed in a real-ime applicaion including ransmission of animaion parameers and display of he animaed avaar. In he following figures, several snap shos of a video sequence are presened, which show he acual pose of he user and he animaed pose of he virual characer. In he righ par of each example, he box around he head and he hand boxes including he segmened skin-colour pixels are shown. In Fig. 5 and Fig. 6 (righ), he ransformed head roaion is demonsraed resuling from he D video images of he operaor. In Fig. 6 (lef), a sequence of head nick and urn angles is presened. he sinusoidal behaviour resuling from up and down (lef and righ) moion, is clearly visible. Ineresingly, he nick angle changes since horizonal head urn was performed. he reason is caused by he mouning of he camera, which is looking from 0.4m above he head wih an angle of 0 degrees. In Fig. 7, he animaion of he avaar by moving hands is shown. Fig. 5. Example snap shos for a head urn FAP value head nick and urn angle 0.5 0.4 nick angle 0. urn angle 0. 0. 0-0. -0. -0. -0.4-0.5-0.6 frame Fig. 6. Sequence of regisered nick and urn angles described as FAP value (lef), example images for nicking he head (righ)
8 Fig. 7. Example snap shos for moving hands 6 Conclusion In his paper, we have presened a complee sysem which uses several modules in order o seer an avaar based on live moion of a person capured by a single camera. he approach is running in real-ime on a sandard PC. he algorihms are robus in erms of differen users, arbirary gesures and wih regard o he iniialisaion of he complee racking and segmenaion sysem. An auomaic iniialisaion prevens he user from difficul seup procedures or specific iniial gesures. his is paricularly imporan in consumer applicaions, where user friendliness and easy usage play a significan role. References..B. Moeslund and E. Granum (00) A survey of compuer vision-based human moion capure, Compuer Vision and Image Undersanding, vol. 8, no., -68... Jaeggli,.P. Koninckx and L. Van Gool (005) Model-based Sparse D Reconsrucions for Online Body racking. Proceedings of IS&/SPIE's 7h Annual Symposium on Elecronic Imaging - Videomerics VIII, vol.5665, San Jose, California, USA.. I. Pavlovic, R. Sharma,.S. Huang (997) Visual inerpreaion of hand gesures for human compuer ineracion: a review. IEEE rans. on PAMI, 9: 677 695. 4. A. Jus, S. Marcel and O. Bernier (004) HMM and IOHMM for he recogniion of Monoand Bi-manual D Hand Gesures. Briish Machine Vision Conf., Kingson Univ. London. 5. P. Eiser and B. Girod (998) Analyzing Facial Expressions for Virual Conferencing. IEEE Compuer Graphics & Applicaions: Special Issue: Compuer Animaion for Virual Humans, vol. 8, no. 5, pp. 70-78. 6.. Horpraser, Y. Yacoob, L.S. Davis (996) Compuing -D head orienaion from a monocular image sequence. nd In. Conf. on Auomaic Face and Gesure Recogn., p.4. 7. Z. Zhu, Q. Ji (004) D Face Pose racking from an Uncalibraed Monocular Camera. In. Conf. on Paern Recogniion, Workshop on Face Processing in Video, Washingon DC. 8. R. R. Anderson, J. Hu, and J. A. Parrish (98) Opical radiaion ransfer in he human skin and applicaions in in vivo remiance specroscopy. In R. Marks and P. A. Payne, ediors, Bioengineering and he Skin, MP Press Limied, chap. 8, pp. 5-65. 9. M. Sörring, H.J. Andersen, E. Granum, (999) Skin colour deecion under changing lighing condiions. Symp. on Inelligen Roboics Sysems, pp. 87-95. 0. S. Askar, Y. Kondrayuk, K. Elazouzi, P. Kauff, O. Schreer (004) Vision-based Skin- Colour Segmenaion of Moving Hands for Real-ime Applicaions. Proc. of s European Conference on Visual Media Producion (CVMP), London, Unied Kingdom.