Overview. Recognition and Tracking of Human Motion. Problem statement. Problem statement. Christian Vogler Rutgers University

Size: px

Start display at page:

Download "Overview. Recognition and Tracking of Human Motion. Problem statement. Problem statement. Christian Vogler Rutgers University"

Magnus Wilkinson
7 years ago
Views:

1 Reognition and Traking of Human otion Christian Vogler Rutgers University Overview Problem statement Reognition ase study: Sign language Phoneme-based modeling odeling simultaneous events Reognition with parallel hidden arkov models Traking ase study: The human fae 3D deformable model traking Statistial ue integration Conlusions and demos 2 Problem statement Problem statement Person Video image to omputer Extration of 3D features of arms, hands, body, and fae Video image Extration of 3D Person to omputer features of arms, Examples: Gesture sign language reognitionhands, body, Gait reognition and fae Faial expression reognition Examples: x, y, z position of hands veloity of the hands joint angles of the fingers Reognition of ativity from features Reognition of ativity from features 3 4

2 Why is the problem diffiult? Complexity: Potentially infinite number of possible human movements How an we model an infinite number of movements? Humans perform simultaneous movements Suh as: gesturing with both hands, faial expression while gesturing, et. How an we deal with the potentially huge ombination of possible simultaneous events? Overview Problem statement Reognition ase study: Sign language Phoneme-based modeling odeling simultaneous events Reognition with parallel hidden arkov models Traking ase study: The human fae 3D deformable model traking Statistial ue integration Conlusions and demos 5 6 Reognition ase study Use Amerian Sign Language (ASL) reognition as a ase study on how to solve these problems ASL has lots of struture Struture is well-researhed Can provide hints and prototypes for more general solutions This researh is based on joint work with Dimitris etaxas Phoneme-based modeling A defining harateristi of a language: Consists of small, limited number of basi units (= phonemes) Every word/sign an be built up from phonemes That is, we an build an unlimited set of signs from a limited set of basi units. 7 8

3 A primer on ASL phonology anual omponent of signs essentially onsists of four parts: Handshape Hand orientation Body loation ovement These parts an be put together into sequenes and parallel ombinations We follow the ovement-hold model Liddell Johnson (1989) 9 ovement-hold model Sequential aspets of ASL In a nutshell: Segments apture the sequential aspets of ASL Very easy to treat omputationally... bak forward bak H Forehead Use one hidden arkov model (H) per segment 11 12

4 ovement-hold model In a nutshell: Segments apture the sequential aspets of ASL Very easy to treat omputationally... Artiulatory features apture the simultaneous (parallel) aspets of ASL Very hard to treat omputationally! Why? 13 Simultaneous omplexity Unfortunately, no Far too many ombinations of simultaneous events Of ourse, not every ombination is valid But: Enumerating valid ASL ombinations is hard Even moreso in the ase of the general human motion reognition problem Solution: Simplify... Independent hannels Assume that all simultaneous events are independent from one another Division into independent hannels: Can be observed independently Can be ombined easily Can be trained independently 15 16

5 What does that buy us? Phoneme breakdown: We an build infinite number of signs from small, limited set of basi units Independent hannels: We do not need to onsider or train all possible ombinations of simultaneous events a priori We an put together new ombinations on the fly, at reognition time Overall: Enormous redution in omplexity 18 Hidden arkov models H example State-based statistial model System is always in some state At disrete time intervals, takes transition to another state Probabilisti transitions Eah state has output probability distribution Commonly a Gaussian density mixture Expresses probability that H generates this output in this state Probabilities are trained Y X 19 20

6 Continuous reognition Continuous reognition Chain Hs into network ath network to signal Find most likely state sequene through network X Front of Forehead bak forward bak H Forehead Father Try Book Father Try Book Woman Read Teah Woman Read Teah Continuous reognition Token passing algorithm bak forward bak H Forehead Father Woman Try Read Book Teah 23 24

7 !! # " Token passing Formally The token passing algorithm finds max Q P max Q 1,.., Q t O, Q Q 1 b Q1 O 1 i t 2 a Qi 1 Q i b Qi O = observation sequene, Q = state sequene, = H This is a produt of independent random variables O i Parallel Hs Extend to multiple, independent hannels aximize joint probability ultiply probabilities from eah hannel This extension formalizes PaHs Now maximize Q max 1,..,Q C 1 log P O, Q Probability ombination Essentially, searhes an H network in parallel for eah hannel When to multiply probabilities? Split up sequene into weighted ontributions from individual signs Q Q max 1,..,Q max 1,..,Q W C 1 log P C w 1 1 w O, Q log P O w, Q w 27 Probability ombination What does this mean? We an ombine the partial probabilities after eah sign (or eah phoneme) Helps onstrain the searh through the parallel networks Also: Paths from the hannels must be onsistent That is, they must touh the same sequene of signs Enfore through unique path identifiers 28

8 PaH token passing Experiments Woman Channel 1 Woman Try Like Try Teah Teah 3D data from otion Star 60 Hz Strong hand joint angles from Cyberglove 22-sign voabulary 400 training sentenes 99 test sentenes Speed: fps on a dual 2 Ghz, running Linux 2.4. Read Channel ultiple-hannel experiments Overview Reognition auray in Baseline: 1 hannel movement strong hand 2 hannels movement both hands 2 hannels movement strong hand handshape strong hand 3 hannels movement both hands handshape strong hand Problem statement Reognition ase study: Sign language Phoneme-based modeling odeling simultaneous events Reognition with parallel hidden arkov models Traking ase study: The human fae 3D deformable model traking Statistial ue integration Conlusions and demos 31 32

9 Traking ase study Trak the human fae from video sequenes Prerequisite for: Faial expression reognition Automated surveillane Identifiation Based on joint work with Siome Goldenstein and Dimitris etaxas Fae traking We want to reover 3D faial parameters from single amera Generally, a very diffiult problem One popular tehnique are 3D deformable models Surfae of model ontrolled by small set of welldefined parameters Redues the degree of freedom inherent in the task Deformable model example Jaw opening parameter How does traking work? Fit the model to human fae in the first frame Selet a set of interesting points on model Projet points into image spae Compute displaement of points from one image frame to the next image fores Convert to fores ating on model parameters generalized fores Integrate with Lagrangian dynamial system 35 36

10 Example fit of model Fores from multiple soures Point traker Edge traker? How to ombine this information? Optial flow Cue integration Information from multiple ues (point traker, edge traker, et.) is often ontraditory Different degrees of reliability over time and spae Statistial approah seems self-evident But: we don't know the probability distributions! We an, however, estimate regions of onfidene 39

11 $ Estimation of Gaussian Generalized fore is the result of a sum of a large number of image fores We know about the image fores: The support of their probabilities That they are uniformly bounded That they are independently distributed (Ensure by seleting points of interest suffiiently far from one another) Conditions for Lindeberg's Theorem hold Related to Central Limit Theorem Estimation of Gaussian Lindeberg's Theorem: Under these onditions, sum is approximated well by Gaussian probability distribution We an estimate generalized fores as Gaussian probabilities Even though exat distributions of image fores are unknown! Integrate Gaussians from different ues via maximum likelihood estimator Overview Problem statement Reognition ase study: Sign language Phoneme-based modeling odeling simultaneous events Reognition with parallel hidden arkov models Traking ase study: The human fae 3D deformable model traking Statistial ue integration Conlusions and Demos Demos Show movies 43 44

12 Conlusions Reognition: Independent hannels and parallel Hs keep omplexity of task down and improve reognition auray Traking: ethod based on region of onfidenes allows statistial estimation without having to know the detailed distributions Improves traking auray, espeially for lowquality image sequenes Contat info: WWW:

Improved Vehicle Classification in Long Traffic Video by Cooperating Tracker and Classifier Modules

Improved Vehicle Classification in Long Traffic Video by Cooperating Tracker and Classifier Modules Improved Vehile Classifiation in Long Traffi Video by Cooperating Traker and Classifier Modules Brendan Morris and Mohan Trivedi University of California, San Diego San Diego, CA 92093 {b1morris, trivedi}@usd.edu