Reovering Artiulated Motion with a Hierarhial Fatorization Method Hanning Zhou and Thomas S Huang University of Illinois at Urbana-Champaign, 405 North Mathews Avenue, Urbana, IL 680, USA {hzhou, huang}@ifpuiuedu Abstrat Reovering artiulated human motion is an important task in many appliations inluding surveillane and human-omputer interation In this paper, a hierarhial fatorization method is proposed for reovering artiulated human motion (suh as hand gesture) from a sequene of images aptured under weak perspetive projetion It is robust against missing feature points due to self-olusion, and various observation noises The auray of our algorithm is verified by experiments on syntheti data Introdution Hand gesture an be a more natural way for human to interat with omputers For instane, one an use his or her hands to manipulate virtual objets diretly in virtual environments However, apturing human hand motion is inherently diffiult due to its artiulation and variability One way to solve the problem is to deompose the hand into linked objets, trak them hierarhially and derive the onfiguration afterwards This kind of deomposition method was first used in human motion analysis by Webb et al[5] Holt and Huang [7] proposed a more general approah whih used perspetive (instead of orthogonal) projetion They found a losed form solution using an algebrai geometry method [6] In this paper, a hierarhial fatorization method is proposed for reovering artiulated hand motion from a image sequene Both global motion and loal artiulation are reovered simultaneously Setion 2 summaries previous work on the fatorization methods, espeially for multiple objet and non-rigid objet Setion 3 desribes the extension of the fatorization method to artiulated objet Setion 4 introdues the hierarhial fatorization method to handle olusion and feature point orrespondenes Setion 5 provides experimental results in both quantitative and visual forms Setion 6 onludes with appliable situations, the limitation of this approah and future diretions for extension and improvement The appendix gives some omputational details
2 Previous Work in Fatorization Methods Fatorization-based struture from motion of a single objet under orthographi projetion was introdued by Tomasi et al [4], and later extended to paraperspetive projetion model [2] A sequential version was proposed in [] Attempts were made to generalize the tehnique for full perspetive [3], but due to the inherent nonlinearity of amera projetion, some preproessing (espeially depth estimation) is neessary, whih leads to a sub-optimal solution Costeira et al proposed an algorithm for multi-body segmentation based on fatorization tehnique [2] Similar approahes were later developed for linearly moving objets [5] and for deformable objets [] Costeira [2] based their segmentation algorithm on a so-alled shape interation matrix Q (see below) If two features belong to two different objets, their orresponding element in Q should be zero; otherwise, the value should be non-zero They then grouped feature points into objets by thresholding and sorting Q Gear [4] formulated the task as a graph mathing problem by plaing appropriate weights on the graph edges Unfortunately, the performane of both tehniques degrades quikly when data points are orrupted with noise, beause the relationship between data noise and the oeffiients of Q (or weights of the graph edges) is so ompliated that it is hard to determine an appropriate threshold Ihimura [8] proposed an improved algorithm by applying a disriminant riterion for thresholding, but the disriminant analysis is still performed on the elements of Q, resulting a similar degradation with noise To avoid this problem, Kanatani [9] proposed to work in the original data spae by inorporating suh tehniques as dimension orretion (fitting a subspae to a group of points and replaing them with their projetions onto the fitted subspae) and model seletion (using a geometri information riterion to determine whether two subspaes an be merged) Wu and Huang in [6] proposed a new grouping tehnique based on orthogonal subspae deomposition, whih introdues the shape signal subspae distane matrix D, for shape spae grouping, based on a distane measure defined over the subspaes Robust segmentation results are ahieved Following the exploration of multi-body fatorization, it is natural to extend the multi-body fatorization method to artiulated objets, whih an be treated as a group of linked rigid bodies This paper mainly disusses the integration of kinemati onstraints into the multi-body fatorization method 3 Fatorization Method for Artiulated Objets Although the fatorization method here an be applied to arbitrary artiulated objets, we desribe the algorithm using human hand as an example The kinemati hand model has 2 degrees of freedom (DOF) for joint angles and 6 DOF for global pose Figure shows a kinemati hand model The distal interphalangeal (DIP) joint and proximal interphalangeal (PIP) joint eah of the four fingers has one DOF and the metaarpophalangeal(mcp) joint has two DOF due to flexion and abdution The thumb has a different struture from the
other four fingers and has five DOF, one for the interphalangeal (IP) joint and two for eah of the thumb MCP joint and trapeziometaarpal (TM) joint both due to flexion and abdution The palm is approximated by a rigid polygon Fig Kinemati hand model and joint notations 3 Rank onstraints for Artiulated Motion Aording to the kinemati model, we treat the hand as 6 linked objets (5 phalanxes and palm) Upon the F images of hand artiulation, we trak P (at least four non-oplanar) feature points on the -th objet Given the P = N = P feature points and their pixel oordinates (u f,p, v f,p ) in eah frame f, we ollet all the image measurements into matrix W 2F P = [ W W N ] () u, u,p u where W2F P = F, u F,P ζ and ζ is the foal length Applying the v, v,p vf, v F,P multi-body fatorization method [3] without onsidering the kinemati on-
straints, W an be fatorized as: W 2F P = [ M M N ] S 0 0 0 S 2 0 0 0 0 S N 0 0 0 S N (2) Where onsists of and M2F 4 = t i z, t i F z,f t z, j t j F z,f i x,f t x, t z, t x,f t z,f t y, t z, t y,f t z,f (3) i f = i y,f (4) i z,f j x,f j f = jy,f (5) jz,f whih are the axes of the amera oordinate frame expressed in the objet s frame t x,f, t y,f, t z,f represent the translation from the origin of the objet s frame to that of the amera frame x x P S4 P = y yp z zp (6) is the homogeneous 3D oordinates of the feature points in phalanx s frame It is obvious that W 2F P has at most rank 4N The motion estimation problem is formulated as finding the orresponding sequene of rotation-translation matries M f = [R f T f ] with respet to the initial pose 32 Solving Motion [R f T f ] by Fatorization Singular value deomposition is proved to be a robust method for reovering M matrix W = UΣV T (7)
Take the largest 4N singular values in Σ to form diagonal matrix ˆΣ We define ˆM = U ˆΣ 2 (8) Ŝ = ˆΣ 2 V T This fatorization is unique up to a nonsingular matrix A 4 4, sine MS = ( ˆMA)(A Ŝ) For a single rigid body, if we hoose the entroid of the objet as the origin of the world oordinate, we an solve A using the onstrains on M (as given in the appendix) However, in the ase of artiulated objet, different onstraints must be applied, whih will be disussed in Setion 4 Given M = ˆMA, the two axes of the amera frame at instant f in objet s frame are given by i f and j f And the third axis (9) k f = i f j f (0) The Full rotation matrix from objet s frame to the amera frame is 33 Reovering Loal and Global Motion R f 3 3 = [i f j f k f ] () Reovering of 20 joint angles of the hand model equivalent to solve the rotation matrix between adjaent phalanxes Assuming phalanxes, 2 are adjaent, we an solve the relative rotation at instant f as R f (, 2 ) = R f R 2 f, and reover the joint angles from R f (, 2 ) If we define the world oordinates as that of the palm (objet 0 ), the global hand pose is R f = R 0 f 4 Hierarhial Fatorization The ideal solution in Setion 3 has two diffiulties in pratie First it is very hard to trak all the feature points reliably and measure their loal oordinates, due to self-olusion and observation noise The feature points of different objets will be randomly mixed in matrix W There have been many methods to address this problem as listed in Setion 2 The seond diffiulty is more speifi for artiulated objet: sine we treated eah objet as independent objets, there is no guarantee that the ends of onseutive objets are linked in the reovered motion We propose a hierarhial fatorization method to solve this problem Given F images with P = N = P feature points and their pixel oordinates (u f,p, v f,p ) in eah frame f, we ollet all the image measurements into matrix u, u,p W 2F P = u F, u F,P ζ v, v,p (2) vf, v F,P
Using the multi-body fatorization method in [2], W 2F P an be fatorized as following Ŝ 0 0 W 2F P = [ ˆM N ˆM ] 0 Ŝ2 0 0 0 (3) 0 ŜN 0 0 0 Ŝ N Where and where A is a full rank 4 4 matrix In the matrix M 2 f = t 2 i 2 f z,f t 2 j 2 f z,f ˆM 2F 4 = M A (4) Ŝ 4 P = (A ) S (5) t 2 x,f t 2 z,f t 2 y,f t 2 z,f i 2 f, we have j2 f (6) i 2 f = (7) j 2 f = (8) Denote the 4 4 matrix A as the onatenation of two bloks A = [A R a t] (9) A R is the first 4 3 submatrix related to rotational omponent And a t is a 4 vetor related to translations From Equation (6)(7)(8), A R an be solved (the details are given in the appendix) To reover a t, [2] used the entroid of the feature points on objet, that is, the average of eah row of W : w P u,p [ ] s P = [ ˆMA R ˆMa t] (20) p= vf,p x p where s = P y p P p= zp is the 3D entroid of the objet The traditional method [2] assumes the entroid of the objet to be the origin of the objet, that is s = 0 Then w = M ˆ a t and a t is solved to be a t = ( M ˆT ˆM) Mˆ T w = Σ 2 U T w However, if we solve a t this way, the results will usually violate the linked-ends onstraints
Step(): Solving motion of the palm as the root objet Trak at least 4 feature points in general position on the palm; solving transformation matrix for palm M as desribed in Setion 3 Step(2): Solving motion of the five proximal phalanxes Based on M, we an solve the M matries for the five proximal phalanxes (the phalanxes attahed to the palm) Step(3): Solving the motion of the five medial phalanxes (the seond row) Step(4): Solving the motion of the five distal phalanxes (on the third row) Fig 2 The hierarhial fatorization algorithm To solve this problem, we propose the hierarhial fatorization algorithm for solving A ( = N) as shown in Figure 2 In eah step, the solution for A R is still the same as the multi-body fatorization method desribed in the appendix, but for a t, we introdue the translational onstraint as shown in Figure 3 Fig 3 An example of two linked phalanxes and 2 (the medial phalanx and the distal phalanx) and the translational relation between their loal oordinates Table lists the notations used in Figure 3 As M is solved from the previous step, we an express the translational onstraint due to the linked-ends of and 2 as: t x,f t2 x,f T f = t y,f t2 y,f t z,f t 2 z,f (2)
Table Notations used in Figure 3 notation physial meaning medial phalanx 2 distal phalanx T translation between the loal oordinates of and 2 in the amera frame T translation between the loal oordinates of and 2 in sub objet s frame where f = F Aording to the kinemati hand model T = 0 l (22) 0 where l is the length of T f = t y,f t z,f [ ] i f j f k f T (23) where i, j and k are defined in Equation (4), (5) and (0) Therefore, [t 2 satisfies t x,f [ ] = i f j f k 0 f l 0 x,f t 2 y,f t 2 z,f ]T (24) From Equation (24), we obtain the origin of objet s loal oordinates, whih is not the entroid of the feature points, but the joint where urrent objet is linked with the previous one We take the ratio between their x-y oordinates t x, t z, t x,f and the oordinates and stak them into vetor t t z,f =, whih is the fourth t y, t z, t y,f t z,f olumn of M Therefore ˆMa t = t and a t an be solved as: a t = ( M ˆT M ˆ ) M ˆ T t = Σ 2 U T t (25) 5 Experimental Results Figure 4 shows a syntheti sequene of hand motion rendered with OpenGL The joint angle data is olleted with CyberGlove T M
Fig 4 Syntheti image sequene of hand motion Fig 5 Hand motion reovered with the hierarhial fatorization method Figure 5 shows Hand motion reovered with hierarhial fatorization method, rendered from the same view point It an be seen that the metaarpophalangeal (MCP) abdution angle is a bit off, whih also shows in the first subplot in Figure 6 Finger 6 shows the omparison between the original joint angles and the reovered joint angles of the index finger The blue urves show the original joints angles over 38 frames The red urves show the reovered joint angles The 4 subplots orresponds to 4 angles, ie MCP Abdution (MCPA), MCP Flexion (MCPF), PIP Flexion (PIPF) and DIP Flexion (DIPF) respetively, at 3 joints of the index finger All the values shown in the plots are in degree angle Table 2 RMSE, std, range of motion(rom) and rrmse between the ground truth and reovered joint angles joint angle RMSE std ROM rrmse MCPA 286 7727 0635 34456% MCPF 25 3480 5544 2040% PIPF 60 2 8358 959% DIPF 0247 054 376 7956% In Table 2, RMSE denotes the rooted mean square error between the ground truth joint angles and the reovered ones, the mean is taken aross all the frames within the test sequene And std denotes the standard deviation of the absolute differene between the ground truth joint angles and the reovered ones, taken aross all the frames within the test sequene ROM denotes the range of motion, that is, the differene between the joint angle in the first frame and that in the last frame rrmse is the ratio between RMSE and ROM As the table shows,
MCP Abdution MCP Flexion PIP Flexion [deg] DIP Flexion 40 20 0 Index Finger: Original joint angles (blue urves), reoverd joint angles (red dashed urves) 0 5 0 5 20 25 30 35 40 80 60 40 20 0 0 5 0 5 20 25 30 35 40 40 30 20 0 5 0 5 20 25 30 35 40 5 0 5 0 5 0 5 20 25 30 35 40 time instant Fig 6 Original joint angles and joint angles reovered with hierarhial fatorization method the results for MCPF are the best, while the relative MSE for MCPA is the largest among all joint angles, whih shows that most of the noise is in MCPA and the noise is hanging dramatially with the motion This is due to the fat that most sample points are hosen at similar relative depth Nearly oplanar struture matrix will ause rank defiieny in singular value deomposition and thus ause majority of the noise in z diretion: the main diretion of MCPA As for PIPF, the MSE is 60, the standard deviation of the differene is very small (only 2), whih shows there is an almost onstant offset between the original angle and reovered angle This suggests that the noise is global and irrelevant to the partiular motion By inreasing the feature points on the medial and proximal phalanxes, this error an be redued For DIPF, both ROM and MSE are very small 6 Conlusions In this paper, we propose a hierarhial fatorization method for reovering the artiulated human motion The kinematial onstraints are utilized to grantee the ends of onseutive objets are linked The limitation for this algorithm is that at least 4 non-oplanar feature points on the palm must be traked reliably in order to reover the global pose and initiate the hierarhial algorithm After fatorization-based segmentation, labelling of eah objet is also nontrivial In the future we will onentrate on integrate more onstraints into the fatorization method For example, in eah proximal interphalangeal (PIP) joint
and eah distal interphalangeal (DIP) joint, there are only DOF in the rotation matrix R, while for MCP joints, there are only 2 DOF in R By projeting the orresponding rotation matrix the subspae of SO() and SO(2) [0], we an get more aurate reovering results Referenes C Bregler, A Hertzmann, and H Biermann Reovering non-rigid 3D shape from image streams In Pro IEEE Conf on Computer Vision and Pattern Reognition, pages 690 696, Hilton Head Island, SC, June 2000 2 J Costeira and T Kanade A multibody fatorization method for independently moving objets International Journal of Computer Vision, 29:59 79, 998 3 Joao Costeira and Takeo Kanade A multibody fatorization method for motion analysis Tehnial Report CMU-CS-TR-94-220, Computer Siene Department, Carnegie Mellon University, Pittsburgh, PA, September 994 4 C Gear Multibody grouping from motion images International Journal of Computer Vision, 29:33 50, 998 5 M Han and T Kanade Reonstrution of a sene with multiple linearly moving objets In Pro IEEE Conf on Computer Vision and Pattern Reognition, volume II, pages 542 549, Hilton Head Island, SC, June 2000 6 R J Holt and T S Huang Algebrai methods for image proessing and omputer vision IEEE Transations on Image Proessing, 5:976 986, 996 7 R J Holt, T S Huang, A N Netravali, and R J Qian Determining artiulated motion from perspetive views: A deomposition approah Pattern Reognition, 30:435 449, 997 8 N Ihimura Motion segmentation based on fatorization method and disriminant riterion In Pro IEEE International Conferene on Computer Vision, pages 600 605, Greee, Sept 999 9 K Kanatani Motion segmentation by subspae separation and model seletion In Pro IEEE International Conferene on Computer Vision, Vanouver, Canada, July 200 0 Yi Ma, Jana Koseka, Stefano Soatto, and Shankar Sastry An Invitation to 3-D Vision Springer-Verlag, New York, 2002 T Morita and T Kanade A sequential fatorization method for reovering shape and motion from image streams IEEE Transations on Pattern Analysis and Mahine Intelligene, 9:858 867, 997 2 C Poelman and T Kanade A paraperspetive fatorization method for shape and motion reovery IEEE Transations on Pattern Analysis and Mahine Intelligene, 9:206 28, 997 3 P Sturm and B Triggs A fatorization based algorithm for multi-image projetive struture and motion In Pro European Conferene on Computer Vision, volume II, pages 709 720, 996 4 C Tomasi and T Kanade Shape and motion from image streams under orthography a fatorized method International Journal of Computer Vision, 9:37 54, 992 5 J A Webb and J K Aggarwal Struture from motion of rigid and jointed objets Artifiial Intelligene, 9:07 30, 982 6 Y Wu, Z Zhang, T S Huang, and J Y Lin Multibody grouping via orthogonal subspae deomposition In IEEE Conf on Computer Vision and Pattern Reognition, volume II, pages 252 257, Kauai, Hawaii, De 200
Appendix: Solving A R The following equations hold for all, so the supersript are omitted Define l l 2 l 3 l 2 l 4 l 5 = L = A T R A R (26) l 3 l 5 l 6 m,f = [ ˆM(, ) f ˆM(, 2)f ˆM(, 3)f ] (27) m 2,f = [ ˆM(2, ) f ˆM(2, 2)f ˆM(2, 3)f ] (28) Aording to Equation (6) (7)(8), we have ˆm T,f L ˆm 2,f = 0 (29) ˆm T,f L ˆm,f = t z (30) ˆm T 2,f L ˆm 2,f = t z (3) Aording to [], for eah, there are 3 F equations and they an be rewritten as Gl = (32) where and g(m,, m, ) g(m,f, m,f ) g(m 2,, m 2, ) l G =, l = (33) g(m 2,F, m 2,F ) l 6 g(m,, m 2, ) g(m,f, m 2,F ) T = [ } {{ } 2F 0 } {{ 0 } ] T (34) F g(a, b) = [a b 2a b 2 2a b 3 a 2 b 2 2a 2 b 3 a 3 b 3 ] (35) Using Least Square Method, l an be solved l = (G T G) G T (36) The eigen-deomposition of matrix L gives matrix A R