Particle Filter with Analytical Inference for Human Body Tracking

Transcription

1 IEEE Workshop on Moion nd Video Compuing November 00, Florid Pricle Filer wih Anlyicl Inference for Humn Body Trcking Mun Wi Lee, Isc Cohen nd Soon Ki Jung Insiue for Roboics nd Inelligen Sysems Inegred Medi Sysems Cener Universiy of Souhern Cliforni Los Angeles, CA {munlee icohen Absrc This pper inroduces frmework h inegres nlyicl inference ino he pricle filering scheme for humn body rcking. The nlyicl inference is provided by body prs deecion, nd is used o upde subses of se prmeers represening he humn pose. This reduces he degree of rndomness nd decreses he required number of pricles. This new echnique is significn improvemen over he sndrd pricle filering wih he dvnges of performing uomic rck iniilizion, recovering from rcking filures, nd reducing he compuionl lod.. Inroducion Humn body moion rcking nd nlysis hs received significn moun of enion in he compuer vision reserch communiy in he ps decde. This hs been moived minly by he desire of undersnding humn pose nd gesures for building he ne generion user inerfce. Inspired from humn o humn inercions, such n inerfce will go beyond he mouse-keybord inercion, defining sysem h responds nurlly o user gesures. Nurlly oher pplicions reled o mrker-less cpure of he humn body moion cn be considered wihin he presened frmework refer o [] for survey). We presen in his pper he firs sep owrds such n inerfce by providing robus humn body rcking from se of synchronized nd geomericlly regisered video srems. The mehod relies on inferring n riculed body model from: he observed silhouees nd n nlyicl inference of humn body prs. An riculed humn body model sick-figure) is ofen used for deiled moion cpure. Indeed, i provides n effecive represenion of he physicl srucure nd consrins of he humn body. Fiing nd rcking he riculed body model becomes problem of esiming he se vecor describing he humn pose, where ech se prmeer represens one degree of freedom e.g. oin ngle) of he humn model. However, he lrge number of degrees of freedom ssocied wih he model cnno be ll nlyiclly inferred from imge feures. Vrious mehods hve been proposed o ddress his problem, which rely on inroducing ddiionl consrins o reduce he se spce, using lerned dynmic models [] or PCA-bsed dimensionl reducion [3]. However, hese mehods resric he pose spce nd re no suible for generl moion cpure pplicion. The pricle filer echnique [4][5][6] is promising mehod for humn body rcking [7] becuse i voids comple nlyicl compuions. Bsed on he Mone- Crlo simulion, pricle filer provides suible frmework for se esimion in nonliner, non- Gussin sysem [6]. However pricle filer requires n imprciclly lrge number of pricles o smple he high dimensionl se spce effecively; oherwise, i is esy o lose rck nd difficul o recover rcking filure becuse of smple depleion in he se spce. In ddiion, pricle filer requires n ccure model iniilizion. Ofen, iniilizion is done mnully, which is undesirble in mny pplicions. Recen works h used pricle filer for humn rcking hve focused on improving efficiency using vrince nlysis [8] nd simuled nneling Deprmen of Compuer Engineering Kyungpook Nionl Universiy 370 Snkyuk-dong Buk-gu Degu Souh Kore skung@knu.c.kr

2 IEEE Workshop on Moion nd Video Compuing November 00, Florid pproch [9]. Anoher work [0] uses D imge moion s feures o improve he likelihood mesure. In his pper, we propose novel mehod for humn body rcking which combines pricle filer wih nlyicl inference echniques. We propose mehods for deecing body prs such s he hed, he hnds nd orso. The resuls of hese deecions re used o mke inference on subse of se prmeers corresponding o he observed humn body pose. This ddiionl inference is used o improve he se esimion wihin he pricle filering frmework. The bsic modules of his proposed frmework re illusred in Figure, nd will be described in deils in Secion 4. There re severl dvnges of combining pricle filer wih nlyicl inference. Firsly, he inference helps o reduce he degree of freedom h is dependen on Mone-Crlo simulion during se esimion. This llows he use of smller number of pricles nd henceforh reduces he compuionl compleiy. Secondly, he nlysis is useful for uomic model iniilizion nd recovery of los rcks. Selecion. Resmple wih replcemen o produce he N pricles ~, from he se {. The { } i=,, N probbiliy of selecing pricle normlized weigh w. } i=,, N is proporionl o is Predicion. The smples re upded ccording o sochsic diffusion model, ) = ~ i +η, ) where η is vecor of sndrd norml rndom vribles. Upding. Given n observion y, he weighs re upded by he likelihood esimes, w p y ), nd re normlized. The min problem in pricle filering is o define he pproprie likelihood esime bsed on he vilble observions nd priors. Inpu Imge Body Prs Deecion y Figure : Pricle filering wih inference. The noions re described in Secion 4.. Pricle Filer Ariculed Model Fiing.. Pricle Filer Pricle filer, lso known s he Condension lgorihm [5] is robus online filering echnique, bsed on he Byesin frmework. This echnique provides suible bsic frmework for esiming he degrees of freedom of n riculed body model: Pricle filer esimes he ses by recursively upding smple pproimions of poserior disribuion. The poserior disribuion ime is represened by se of N pricles denoed by {, wih weighs { w } i=,, N Se Prior Esimion Pricle Filer) y nd upding. w,, } i =,, N Se Upding Wih Inference Likelihood Compuion. There re 3 bsic seps: selecion, predicion Figure : Ariculed humn body model... Humn Body Model Vrious riculed body models hve been proposed in he lierure ccording o he rgeed pplicion. Some hve very smll number of degrees of freedom nd focus on he limbs [] while ohers hve proposed model h conins hnd nd fingers oins. The riculed humn body model we use consiss of 0 oins nd 4 segmens, represening he hed, orso nd limbs Figure ). A pered 3D cone wih n ellipicl cross-secion represens ech segmen. The model hs 3 degrees of freedom h include he globl rnslion, roion nd scle, nd locl oin roions. Fiing nd rcking he riculed model o he deeced humns in he video srems requires he definiion of likelihood funcion llowing o mp he degree of freedom of he model ono imge properies. The likelihood compuion is bsed on wo componens: he foreground boundry of he moving person nd he deeced silhouee regions. Mching of Foreground Boundry. This involves mching he boundry of he foreground in he inpu imges o he prediced silhouee boundry of he humn body model.

3 IEEE Workshop on Moion nd Video Compuing November 00, Florid For ech pricle i) segmen of he humn model, S N s, we compue he posiion of ech = { s,, s }, where =4 is he number of segmens of he humn body model. For clriy, we omi he superscrip i) nd subscrip. Given foreground segmenion, se of conour poins long he boundry of foreground is erced, denoed by C = }, where N is he number of conour poins. Ech conour poin c is mched o he closes segmen { c =,, N c s k, such h k = rg min d c, s ), l where d c, s ) is he disnce of poin o he edge of segmen l s l proeced on he imge. The similriy mesure is given by, p c s ) = ep[ d c, s ) / σ ], k where σ is he vrince of model nd inpu edge dispriy. As he segmenion of moving obecs chrcerizing he foreground regions is no error-free we hve o ccommode for errors in he segmenion nd ccoun for ouliers mong he conour poins C. Denoing Poulier s he probbiliy of he poin c being n oulier l k c c N s, he likelihood mesure for conour poin c given S is p c S) = P ) p c s ) + P. oulier k oulier The vlue of P oulier depends on he quliy of foreground segmenion nd is derived empiriclly. Combining he likelihood of ll conour poins, he likelihood mesure for boundry mching is defined by: L N c Boundry = p C S) = p c S) =. The bove likelihood mesure is insufficien becuse i ofen leds o over-esimion of he humn size. We ugmen his wih he second likelihood componen described s follows. Mching of Prediced Silhouee Region. The humn body model, when proeced ono he D imges, should lie inside he erced foreground silhouee. The second likelihood componen penlizes ny pr of he body model h lies ouside he silhouee. Given prediced body pose, we compue he proecion of he humn body model on he D imges nd coun he number of piels, n, h re inside he proecion bu lie ouside he deeced foreground region. The likelihood is epressed s: L = α P ) nα α Region, where P α is probbiliy of flse negive errors in foreground ercion. The combined likelihood similriy for mching he riculed body model o he deeced silhouees is given by: L = L Boundry L Region. ) b) Figure 3: Likelihood compuion. In ), he foreground ercion is shown in ligh gry nd slighly misligned humn model is overlid on he imge for illusrion. The enclosed bo is enlrged nd shown in b), where he rrows illusre boundry errors s likelihood componen), nd he drk shded re he boom edge of he model s lower rm indices region mching errors nd likelihood componen). 3. Deecion of Humn Body Prs The likelihood mesure, defined in he previous secion, provides good esimion of he se vecor chrcerizing he riculed model when provided wih correc iniilizion nd here re sufficien pricles smpling he se vecor disribuion. However, requiring good iniilizion limis he use of such humn body rcking o se of specified body moion or posures nd prevens us from using his sysem for humn body moion cpure. We would like o eend he sysem cpbiliies such h i does no require mnul iniilizion sep nd cn overcome he problem of smples depleion during he rcking process. We propose n pproch h incorpores ddiionl cues in order o perform uomic iniilizion nd recover from body-prs rcking filures. In he following, we describe he use of hnds, hed nd orso locion for defining ddiionl cues in he pricle filer bsed rcking. 3.. Hnds Deecion Deecion. The hnds re deeced long he oulines of he foreground. Peks of conve curvure re erced long he silhouee boundry. We mch hese curvure peks in differen imges using epipolr consrins, nd reconsruc heir 3D posiions. Using prior esimion of he hnd posiions, bsed on humn body srucure nd rcking informion, we cn furher elimine unlikely hnd posiions.

4 IEEE Workshop on Moion nd Video Compuing November 00, Florid This requires h he hnd is visible in les wo cmer views. As his is no lwys he cse, we ssign prior probbiliy P h here is n occlusion nd n herefore no correc hnd mesuremen cn be obined. The bove mehod generes number of hypoheses of hnd posiion { }. These hypoheses re red y hnd using prior informion obined from rcking. The prior probbiliy disribuion of he hnd posiion is pproimed by mulivrie Gussin funcion: Pr y ) N, µ, Σ ), hnd y hnd where µ nd Σ re he men nd covrince of hnd hnd he hnd posiion bsed on weighed pricle smples. The probbiliies re normlized so h he sum of probbiliy is equl o - P ), where P is he probbiliy of no correc hnd mesuremen. n Upding. In he following we show how o inegre his prior ino he upding process of he pricle ses. For ech given pricle of se vecor, we use Mone Crlo mehod o selec he hnd posiion mesuremen y, bsed on he probbiliy P y hnd ). There is probbiliy h none of he mesuremens is used, nd he se P n prmeers re propged using sochsic dynmic model s in he sndrd pricle filer, s epressed in ) in Secion.. The mesuremen is used o upde he posiions of hnd nd elbow, while keeping he res of he body posiion unchnged. Given he posiions of he hnd nd shoulder, he elbow lies long circle. We choose he poin on his circle h is closes o he prior esime of he elbow posiion. This is equivlen o keeping he zimuh ngle he shoulder unchnged nd upding hree oher oin ngles one he elbow, wo he shoulder). n hnd hnd hnd silhouee o deec he hed. To chieve scle invrince, he conours re rescled wih respec o he esimed humn heigh. The chin code feures re normlized before comprison o chieve roion invrince. Mching error is bsed on chin code differencing. To obin hed posiion in 3D spce, he hed mus be deeced in les wo inpu imges. Epipolr consrin is used o remove flse mesuremens nd o chieve more ccure loclizion. In mos cses, he hed is ccurely loced. However, occlusions cn occur nd incomplee edge descripion of he hed cn be observed. We epress he probbiliy of flse mesuremen s zero men Gussin funcion of he mching error: e P = ep hed, πσ c σ c where e is he error in chin code differencing. The vrince is obined empiriclly. Wih his probbiliy, σ c Mone Crlo mehod is used o decide wheher o use he hed mesuremen during upding, using he sme pproch s in he hnd deecion described previously. Upding. A mesuremen of he hed posiion in he 3D spce provides hree consrins for upding he se prmeers. We choose o upde he hree degrees of freedom h re mos reled o he hed posiion: he orienion of he hed dofs) nd he posiion of neck long he body min is dof). Chnge in neck posiion necessrily generes chnge of he posiions of orso nd oher body prs such s he shoulders nd hips. Our sregy is o minimize chnges in he posiions of he end-effecors such s he hnds nd legs. We shif he orso only long is is nd compue he oin ngles he inermedie oins shoulders, elbows, hips nd knees) while keeping he hnds nd legs fied. Figure 4: Hnd deecion. Lef: Prior esimed pose wih deeced hnds shown s crosses. Righ: Upded pose. 3.. Hed Deecion Deecion. The hed deecion is performed using reference chin code represenion of hed-shoulder conour Figure 5) s emple for hed. We mch his emple long he conour boundry of he erced Figure 5: Hed deecion. Lef: Temple of hed. Righ: erced silhouee boundry, nd deeced hed Torso Deecion Deecion. A simple mehod is used o erc he min is of he orso. We firs erc he medil is of he D silhouees. The medil is poins in differen views

5 IEEE Workshop on Moion nd Video Compuing November 00, Florid re mched using epipolr consrin, nd he 3D posiions compued refer o [] for deils.) A line is hen fied o hese 3D poins using PCA nd RANSAC mehod Figure 6). This erced line provides mesuremen of he orso orienion, nd consrin h he orso mus ly long he line. This gives us four consrins, which re used o upde he se prmeers: Upding. The posiion dofs) nd orienion dofs) of he humn model re upded so h he orso is ligned o he erced medil is. Noe h he posiion of orso long he is, nd he roion round he is remin unchnged.) During his upde, we pply he sme sregy s in hed deecion o keep he posiions of hnds nd legs consn nd upde he ngles inermedie oins. Figure 6: Torso deecion. The medil is of he silhouee in wo views is erced, mched nd reconsruced in 3D. The orso is is found by fiing line o he medil is poins in 3D. 4. Pricle Filer wih Inference As discussed in Secion, Mone Crlo simulion is no pproprie for esiming high dimensionl se prmeers. The min ide of his pper is o use nlyicl compuion inferred from he deecion of body prs) o infer subse of he se prmeers. This will reduce he degree of dependence on he Mone Crlo simulion. In his secion, we presen generl frmework on how he nlysis resul is incorpored ino he priclefilering scheme. We se he rcking problem s Mrkovin se-spce model, where is he hidden se nd y is he observion. A ny ime, he poserior disribuion is given by Byes heorem: p y ) p y ) p ). 0: : : 0: 0: For firs order Mrkov process, he esimion cn be solved recursively [6], p y ) p y ) p y ), : : where p y ) is he likelihood disribuion. Under he pricle filering scheme, prior is consruced from smple drwn from he disribuion p y ). : Omiing he superscrip i) for clriy, he se esimion for ech pricle, fer he smple is drwn, becomes: p, y ) p y ) p ), where p ) is he rnsiion probbiliy disribuion. From here on, we consider only he esimion for one pricle. Suppose h he se vecor cn be decomposed ino wo prs, ) nd he observion y is ugmened by noher mesuremen h cn be used o esime y nlyiclly. In oher words, p y,,, ) cn be compued nlyiclly. Wih his decomposiion, we cn rewrie he esimion epression s, p,,, y, y ) p y, ) p,,, y ) ) p y, ) p y,,, ) p,, y If he decomposiion of se vecor =, ) is such h condiioned on, ) is relively independen on, hen he hird erm on he righ hnd side of ) cn y be pproimed by: p,, y ) k p, ), where k is consn. For emple, his pproimion is vlid when ) is hnd deecion mesuremen, re he se prmeers of he hnd nd elbow, nd re he se prmeers of oher prs of he body ecep he hnd nd elbow. In his cse, we cnno drw much informion on using. The esimion epression hen becomes: p,,, y, y ) p y, ) p y,,, ) p, ) The bove epression suggess h nlyicl inference, s represened by he erm p y,,, ), cn be used for se esimion. We cll his inference nlyicl becuse i involves he esimion of inermedie oin ngles using geomery nd inverse kinemics. This frmework is helpful becuse he predicion by nlyicl inference ofen hs much lower vrince compred o he iniil prior probbiliy. For discussion, we consider he simples cse, where we cn compue deerminisiclly by funcion f y,, =, ), such h: y y

6 IEEE Workshop on Moion nd Video Compuing November 00, Florid y,,, ), = f y,,, ), p = 0, oherwise. hen he esimion becomes: p,,, y, y ) p y, ) p, ), when = f y,,, 0, oherwise. This shows h he Mone Crlo compuion of he prior probbiliy is now effecively pplied only o he reduced se vecor. The simulion of ) p, is no required. This reduces he required number of pricles nd he compuionl lod. For he hnd deecion, we hve number of hypoheses for hnd posiion. If we ssume negligible loclizion error in hese hypoheses, he inference p y,, ) is non-zero for smll finie se of discree, vlues. For ech pricle, we smple from his finie se using Mone Crlo mehod. In oher words, he prior disribuion, which is used s he impornce smpling disribuion, hs collpsed from coninuous spce o few discree vlues. While he inference is no olly deerminisic, he degree of rndomness hs grely reduced. The se esimion process for pricle is shown schemiclly in Figure. In he figure, he shded bo represens he upding of se prmeers from body prs deecion. In our implemenion, i consiss of cscde of hree upding sges using inference resul from he deecion of orso, hed nd hnds respecively. 5. Eperimenl Resuls Eperimen Seup nd Trcking Iniilizion: Three clibred cmers re se o cpure sequences of single person moving in room. An empy scene bckground is firs lerned for deecion purposes. As he person eners he field of view of he cmers, he silhouees in he 3 views re erced using bckground subrcion. Once he person is wihin he field of views of he hree cmers, he pricle filer is iniilized using he inference mehods described erlier nd he following rules: he heigh of he person is esimed from he heigh of he silhouee, nd he orienion of he shoulders nd hips re inferred from he second principl is of he silhouee. The iniilizion is fully uomic nd does no require he person o snd in sndrd posure. Deils of his iniilizion sep cn be found in []. ) Depending on he pose, he firs iniilizion my no be ccure due o self-occlusions nd pose singulriies. Using body pr deecion wih pricle filer, he rcking lgorihm is ble o recover he pose subsequen frmes. Figure 7 shows sequence where he rcking srs when person is wlking owrds he cener of he room. I demonsres how he mehod is ble o recover from inccure iniilizion. ) b) c) d) Figure 7: Iniilizion nd rcking. ) The iniilizion of he model when he person eners he scene, he model is no mched properly. b) As rcking coninues, he legs nd hed re recovered. c) The hnds sr o pper s he person urns. d) The hnds re recovered. Comprison wih Sndrd Pricle Filer: As only hree cmers re used, pose mbiguiies will occur while observing person gesuring due o selfocclusion, moion singulriies nd bckground cluer. The rcking mehod should be robus gins hese problems. While he sndrd pricle filer uses muliple pricles o smple he poserior disribuion of he se spce, i suffers from he problem of high dimensionliy, which cuses smple depleion in mos of he se spce. As resul, when n mbiguiy occurs, i is esy o lose rck nd recovering los rcking is difficul. Our mehod uses body pr deecion o infer some of he se prmeers nd is ble o genere good se hypoheses, even in smple-depleed se spce region. This provides n venue o minin good rcking nd o recover he se prmeers fer los rcking. We compre he performnce of our mehod wih he sndrd pricle filer. Boh lgorihms were esed wih sequence h conins brief period of bou 0 seconds when he hnds of he person were hidden behind his bck nd were occluded from ll cmers Figure 8). Afer he hnds repper, he sndrd pricle filer is unble o recover from he los rck. In he proposed mehod, he hnds of he humn model he riculed body model re ble o dus o he correc hnd posiions fer he person s hnds repper. In his eperimen, 00 pricles re used for he new proposed mehod, while 400 pricles were required by he sndrd pricle filer lgorihm. In he new mehod, he reducion in compuion due o he smller number of

7 IEEE Workshop on Moion nd Video Compuing November 00, Florid pricles hs more hn offse he ddiionl compuion in body prs deecion. Boh mehods use he sme likelihood compuion, described in Secion. infer subses of se prmeers nd improve he se esimion wihin he pricle-filering scheme. This new mehod hs he dvnge of hndling rck iniilizion, recovering los rck, nd reducing he compuionl lod. 7. Acknowledgemen This reserch ws prilly funded by he Insiue for Creive Technologies. The uhors would like o hnk Rmkn Nevi for his guidnce, nd Hongi Li, Xuefeng Song nd To Zho for heir conribuion in his work. ) b) c) d) Figure 8: Trck recovery fer self-occlusion. Firs row shows he resul from sndrd pricle filer wihou inference. Second row shows he resul from improved pricle filer wih inference. For ech row: ) before occlusion, b) boh hnds re occluded behind he person body, c) one hnd reppers, d) boh hnds re visible. Eperimen wih Differen Person: Figure 9 presens resul wih video of person who is urning her body nd wving her hnds. Wih esimed poses, he humn model cn be rendered in rbirry views Figure 0). Figure 9. Sequence wih differen person. Figure 0. Rendered humn models wih esimed pose. 6. Conclusion There re wo min pproches o humn body rcking: nlyicl-bsed mehods nd synhesis-bsed mehods. This pper inroduces novel echnique h combines boh pproches by using nlyicl mehods o 8. References [] Thoms B. Moeslund, Erik Grnum, A survey of compuer vision-bsed humn moion cpure, Compuer Vision nd Imge Undersnding 8 00), [] V. Pvlovic, J. M. Rehg, T. J. chm, nd K. P. Murhy, A dynmic Byesin nework pproch o figure rcking using lerned dynmic models, ICCV 999, vol., [3] Sidenbldh, H., Blck, M. J., nd Sigl, L., Implici probbilisic models of humn moion for synhesis nd rcking, ECCV 00. vol., [4] N. Gordon, D. J. Slmond, A. F. M. Smih, Novel pproch o nonliner/non-gussin Byesin se esimion, Proc. Ins. Elec. Eng. F, v40, n, 993. [5] M. Isrd, nd A. Blke, Visul rcking by sochsic propgion of condiionl densiy. ECCV 996, [6] M. Sneev Arulmplm, S. Mskell, N. Gordon, T. Clpp, A Tuoril on Pricle Filers for Online Nonliner/Non- Gussin Byesin Trcking, IEEE Trns. Signl Processing, vol. 50, No., Feb 00, [7] J. Deuscher, A. Blke, B. Norh, nd B. Bscle, Trcking hrough singulriies nd disconinuiies by rndom smpling, ICCV 999, vol., [8] C. Sminchisecu, B. Triggs, Covrince Scled Smpling for Monoculr 3D Body Trcking, CVPR 00, vol, pp [9] J. Deuscher, A. Blke, I. Reid, Ariculed Body Moion Cpure by Anneled Pricle Filering, CVPR 000, vol, [0] Hedvig Sidenbldh, Michel J. Blck, D. J. Flee, Sochsic Trcking of 3D Humn Figures Using D Imge Moion ECCV 000, [] To Zho, Rm Nevi, Fengun Lv, Segmenion nd rcking of muliple humns in comple siuions, CVPR 00, vol, [] Isc Cohen, Mun Wi Lee, 3D Body Reconsrucion for Immersive Inercion, Second Inernionl Workshop on Ariculed Moion nd Deformble Obecs Plm de Mllorc, Spin, -3 November, 00.