Modeling the Constraints of Human Hand Motion John Lin, Ying Wu, Thomas S. Huang Bekman Institute University of Illinois at Urbana-Champaign Urbana, IL 68 {jy-lin, yingwu, huang}@ifp.uiu.edu Abstrat Hand motion apturing is one of the most important parts of gesture interfaes. Many urrent approahes to this task generally involve a formidable nonlinear optimization problem in a large searh spae. Motion apturing an be ahieved more ost-effiiently when onsidering the motion onstraints of a hand. Although some onstraints an be represented as equalities or inequalities, there exist many onstraints, whih annot be expliitly represented. In this paper, we propose a learning approah to model the hand onfiguration spae diretly. The redundany of the onfiguration spae an be eliminated by finding a lower-dimensional subspae of the original spae. Finger motion is modeled in this subspae based on the linear behavior observed in the real motion data olleted by a CyberGlove. Employing the onstrained motion model, we are able to effiiently apture finger motion from video inputs. Several experiments show that our proposed model is helpful for apturing artiulated motion. Introdution In reent years, there has been a signifiant effort devoted to gesture reognition and related work in body motion analysis due to interest in a more natural and immersive Human Computer Interation (HCI). As the ost for more powerful omputers dereases and PCs beome more popular, a more natural interfae is desired rather than the traditional input devies suh as mouse and keyboard. Using gestures, as one of the most natural ways humans ommuniate with eah other, thus beomes an apparent hoie for a more natural interfae. An effetive reognition of hand gestures will provide major advantages not only in virtual environments and other HCI appliations, but also in areas suh as teleonferening, surveillane, and human animation. Reognizing hand gestures, however, involves apturing the motion of a highly artiulated human hand with roughly 3 degrees of freedom (DoF). Hand motion apturing involves finding the global hand movement and loal finger motion suh that the hand posture an be reovered. One possible way to analyze hand motion is the appearane-based approah, whih emphasizes the analysis of hand shapes in images [5]. However, loal hand motion is very hard to estimate by this means. Another possible way is the model-based approah [3, 4, 6, 7, 9]. With a single alibrated amera, loal hand motion parameters an be estimated by fitting a 3D hand model to the observation images. One method of model-based approahes is to use gradient-based onstrained nonlinear programming tehniques to estimate the global and loal hand motion simultaneously [6]. The drawbak of this approah is that the optimization is often trapped in loal minima. Another idea is to model the surfae of the hand and estimate hand onfigurations using the analysis-by-synthesis approah [3]. Candidate 3D models are projeted to the image plane and the best math is found with respet to some similarity measurement. Essentially, it is a searh problem in a very high dimensional spae that makes this method omputational intensive. A deomposition method is also adopted to analyze artiulated hand motion by separating hand motion into its global motion and loal finger motions[9]. Although the 3D model-based approah makes motion apturing from monoular images possible, it also faes some hallenging diffiulties. Many urrent methods for hand posture estimation basially involve the problem of searhing for the optimal hand posture in a huge hand onfiguration spae, due to the high DoF in hand geometry. Suh a searh proess is omputationally expensive and the optimization is prone to loal minima. At the same time, many urrent approahes suffer from self-olusion. However, although the human hand is a highly artiulated objet, it is also highly onstrained. There are dependenies among fingers and joints. Applying the motion onstraints among fingers and finger joints an greatly redue the size or dimensions of the searh spae, whih in turn makes the estimation of hand postures more
ost-effiient. Another major advantage of applying hand motion onstraints is to be able to synthesize natural hand motion and produe realisti hand animation, whih would be very useful to synthesize sign languages. There has not been muh done regarding the study of hand onstraints other than the ommonly used ones. Even though onstraints would help redue the size of the searh spae, too many or too ompliated onstraints would also add to omputational omplexity. Whih onstraints to adopt beomes an important issue. Some onstraints have already been presented, studied, and used in many previous works [3, 4, 9]. The ommon ones inlude the onstraints of joints within the same finger, onstraints of joints between fingers, and the maximum range of finger motions. All these are presented as either equalities or inequalities. However, due to the large variation in finger motion, there are yet more onstraints that annot be expliitly represented by equations. In this paper we propose a learning approah to model the onstraints diretly from sampled data in the hand onfiguration spae (C-Spae). Eah point in this hand onfiguration spae orresponds to a set of joint angles of a hand state, whih is ommonly estimated in model-based approahes. Rather than studying the global hand motion, we will fous only on the analysis of loal finger motions and onstraints with the help of a CyberGlove developed by Virtual Tehnologies In. Moreover, we will study the onstraints of hand motions that are natural and feasible to everyone. In setion, a desription of a ommonly adopted hand kinematial model and the CyberGlove is given. Setion 3 desribes how we model the onfiguration spae and the observations of this model. Setion 4 shows the results of some preliminary examples of hand posture estimation taking advantage of this model. Setion 5 onludes our work and disusses some future diretions regarding modeling human motion onstraints. Hand skeleton model DIP PIP MCP Pinky Ring Middle Index Thumb IP MCP Figure : Kinematial struture and joint notations The human hand is highly artiulated. To model the artiulation of fingers, the kinematial struture of hand should be modeled. In our researh, the skeleton of a hand an be abstrated as stik figure with eah finger as a kinematial hain with base frame at the palm and eah fingertip as the end-effeter. Suh a hand kinematial model is shown in Figure with the names of eah joint. This kinematial model has 7 Degrees of Freedom (DoF). Eah of the four fingers has four DoF. The distal interphalangeal (DIP) joint and proximal interphalangeal (PIP) joint eah has one DoF and the metaarpophalangeal (MCP) joint has two DoF due to flexion and abdution. The thumb has a different struture from the other four fingers and has five degrees of freedom, one for the interphalangeal (IP) joint, and two for eah of the thumb MCP joint and trapeziometaarpal (TM) joint both due to flexion and abdution. The fingers together have DoF. The remaining 6 degrees of freedom are from the rotational and translational motion of the palm with 3 DoF eah. These 6 parameters are ignored sine we will only fous on the estimation of the loal finger motions rather than the global motion. Artiulated loal hand motion, i.e. finger motion, an be represented by a set of joint angles θ, or the hand state. In order to apture the hand motion, glove-based devies have been developed to diretly measure the joint angles and spatial positions by attahing a number of sensors to hand joints. CyberGlove is suh a devie. The goal of vision-based analysis of hand gesture is to estimate the hand joint angles or hand states without using suh physial devies but solely based on visual information. However, suh glove-based devie an help olleting ground truth data, whih enable the modeling and learning proesses in visual analysis. In our study, we employ a right-handed CyberGlove. The glove has four sensors for the thumb, a MCP and a PIP sensor for the PIP ( θ PIP ) and MCP ( θ MCP F ) flexion angles for eah of the four fingers, three more abdution sensors for the abdution/addution angles ( θ MCP AA ) between these four fingers. There are total of fifteen sensor readings of the finger joint angles; therefore we are able to haraterize the loal finger motion by 5 parameters. The glove an be alibrated to aurately measure the angle within 5 degrees, whih is aeptable for gesture reognition. For finger postures that are five degrees different would still appear to be the same posture. 3 Modeling the onstraints Modeling motion onstraints is ruial to effetive and effiient motion apturing. A omprehensive study of hand/finger motion onstraints and a learning approah to modeling the natural movement onstraints are given in this setion.
3. Constraints overview Hand/finger motion is onstrained so that hand annot make arbitrary gestures. There are many examples of suh onstraints. For instane, fingers annot bent bakward too muh and the pinky finger annot be bend without bending the ring finger. The natural movements of human hands are impliitly aused by suh motion onstraints. Some motion onstraints may have a losed form representation, suh that they are often employed in urrent researh of animation and visual motion apturing [3, 4, 9]. However, a large number of motion onstraints are very diffiult to be expressed in losed forms. How to model suh onstraints is still needs further investigation. Here we present some of the most ommonly used motion onstraints and justify the use of 5 parameters to represent the hand motion. Hand onstraints an be roughly divided into three types. Type I onstraints are the limits of finger motions as a result of hand anatomy whih is usually referred to as stati onstraints. Type II onstraints are the limits imposed on joints during motion, whih is usually referred to as dynami onstraints in previous work. Type III onstraints are applied in performing natural motion, whih has not yet been explored. Below we will desribe eah type in more detail. Type I onstraints. This type of onstraint refers to the limits of the range of finger motions as a result of hand anatomy. We will only onsider the range of motion of eah finger that an be ahieved without applying external fores suh as bending fingers bakward using the other hand. This type of onstraints is usually represented using the following inequalities: θ 9, MCP F PIP θ, DIP θ 9, and 5 θ MCP AA 5. () Another ommonly adopted onstraint states that middle finger displays little abdution/addution motion and the following approximation is made for middle finger: θ MCP AA =. () This will redue one DoF from the DoF model. Similarly, the TM joint also displays limited abdution motion and will be approximated by as well. θ TM AA =. (3) As a result, the thumb motion will be haraterized by 4 parameters instead of 5. Finally, the index, middle, ring, and little fingers are planar manipulators. i.e. the DIP, PIP and MCP joint of eah finger move in one plane sine DIP and PIP joints only has DoF for flexion. Type II onstraints. This type of onstraint refers to the limits imposed on joints during finger motions. These onstraints are often alled dynami onstraints and an be subdivided into intra-finger onstraints and inter-finger onstraints. The intra-finger onstraints are the onstraints between joints of the same finger. A ommonly used one based on hand anatomy states that in order to bend the DIP joints, the PIP joints must also be bend for the index, middle, ring and little fingers. The relations an be approximated as following: θ DIP = θ PIP. (4) 3 By ombining Eq -4, we are able to redue the model with DoF to one that is approximated by 5 DoF. Experiments in previous work have shown that postures an be estimated using these onstraints without severe degradation in performane. Inter-finger onstraints refer to the ones imposed on joints between fingers. For instane, when one bends his index finger at MCP joint, he would naturally have to bend the middle MCP joint as well. Many of suh Type II onstraints and related equations an be found in [, 4]. However, there are yet more onstraints that an not be expliitly represented in equations. Type III onstraints. These onstraints are imposed by the naturalness of hand motions and are more subtle to detet. Almost nothing has been done to aount for these onstraints in simulating a natural hand motion. Type III onstraints differs from Type II in that they have nothing to do with limitations imposed by hand anatomy, but rather are a result of ommon and natural movements. Even though the naturalness of hand motions is different from person to person, it is similar for everybody. For instane, the most natural way for every person to make a fist from an open hand would be to url all the fingers at the same time instead of urling one finger at a time. This type of onstraint also an not be expliitly represented by equations. 3. Modeling the onstraints in C-spae It is diffiult to expliitly represent the onstraints of natural hand motions in losed form. However, they an be learned from a large and representative set of training samples; therefore we propose to onstrut the onfiguration spae (i.e., joint angle spae) and learn the onstraints diretly from the empirial data using the approah desribed below. For notational onveniene, let us denote the feasible C-spae by Φ R with eah onfiguration denoted by φ = θ, θ, ). ( θ5. Loating base states ζ i in Φ. We will diretly loate the base states by fixing the hand in desired onfigurations and measure the 5 parameters assoiated with the orresponding state. Sine the sensors are very sensitive to finger movements, little variations in finger postures will also be reorded and will be onsidered as 5
the same state. As a result, we will use the entroid from the set of training data D = { x, j = N} as the loation i of the base state ζ i. Another alternative would be to ollet huge set of training samples x i from predefined motions and apply a lustering algorithm in order to loate the base states. However, sine we have full ontrol of how a hand must be onfigured to form the base state, we do not need to apply lustering algorithms to loate the base states in C-spae. ij Joint angles measured in radian.5.5.5 Making a fist.5.5 5 5 5 3 motion samples Figure 3: Joint angle measurements from the motion of making and opening a fist. Figure a: Some base hand states Figure b: Unfeasible onfigurations In our model, the hand gestures are roughly lassified into 3 disrete states by quantizing eah finger into one of the two states: fully extended or urled. The reason for hoosing these two states is that the entire motions of a finger falls roughly between these two states. Therefore the whole set of 3 states will roughly haraterize the entire hand motion (Figure ). However, sine not everyone is able to bend the pinky without bending the ring finger or with the help of thumb to hold the pinky, four of the states will not be ahievable by everyone without applying external fores. Therefore, these four states (Figure b) are not inluded in our set of base states in C-spae modeling. Finally, the onfigurations that are similar are onsidered as the same state. For instane, the ases with five fingers opening wide apart and with all fingers straightened but losed together are onsidered the same.. Motion modeling. With the set of base states ζ i established, we then ollet motion data for state transitions in order to model the onfigurations during natural hand motion. A large number of sets of motion data are olleted in order to observe the Type II and III onstraints of natural hand motions. An example of motion between making and opening a fist is shown in Figure 3. 3. Dimensionality redution. From Figure 3, we an learly observe some orrelations in the joint angle measurements. Therefore, together with the data olleted from stati states and the finger motions, we then perform Prinipal Component Analysis (PCA) to redue the dimension of the model and thus redue the searh spae while preserving the omponents with the highest energy. We note that 95% of the energy is ontained in the 7 dimensions that have the largest eigenvalues. We thus 5 7 perform the mapping R R on Φ by projeting the original model onto a lower dimensional subspae 7 Φ R with priniple diretions assoiated with these 7 largest eigenvalues. 4. Interpolation in ompressed C-spae. One a set of base states ζ i have been determined, the whole feasible onfiguration spae Φ an be approximated by these base states ζ i and an interpolation sheme. Our approah takes a linear interpolation in the lower-dimensional onfiguration subspae Φ. For eah onfiguration φ in Φ we will represent its parameters using a polynomial interpolation, i.e., 8 φ = α ζ, (5) i= i i in whih ζ i is the loation of base state i and α i is the parameters for φ. 3.3 Model harateristis Our model has three main advantages that will help redue the searh spae in gesture reognition. First, the
model is ompat due to the dimensionality redution using PCA. Seond, the motion onstraints are automatially inorporated into the model. Third, a linear behavior is observed in the state transitions in C-spae. The reason that motion onstraints are inorporated into this model is beause we sample diretly from natural hand motions. Configurations that are outside of permissible range limited by hand anatomy will not be ahievable in natural hand motions. Consequently, the inequalities and equalities inluding the intra-finger onstraints [3, 4], suh as θ MCP F = kθ PIP with k, and inter-finger onstraints [4] are automatially overed in this model..5.5.5.5.5.5.5.5.5.5 Figure 4: Motion transistions between four states between..5.5.5.5.5.5.5.5 3 Figure 5: Motion transistions between eight states. An interesting phenomenon regarding the Type III motion onstraints is observed from the motion data. We observe a nearly linear transition between states in C- spae. An example is shown for the ase of transitioning between four states from the result of moving only index and middle fingers (Figure 4). We have projeted the C- spae into R for observation in this ase. The four orners are the loations of the four disrete base states. A linear transition is learly observed from Figure 4. The middle lines are the path resulting from urling and extending both fingers together. This result reflets the high orrelation between fingers when performing natural movements. Although state transitions does not neessarily need to be performed in this manner and there exists infinitely many ways to move from one onfiguration to another, when the fingers are moving in their most natural way, it will take a nearly straight line path in C-spae. This observation will justify Eq 5 in estimating the hand onfigurations. Another example is 3 shown with three-finger motions projeted in R. The eight base states are roughly loated at the eight orners of a ube (Figure 5). 4 Experiments In order to evaluate the validity of this model, we perform some experiments using low-level visual features and estimate the postures onstituted by a subset of the 8 3 base states. segmented. 4. General approah The input images are assumed to be Using the result we observe from the linear behavior, we are able to approximate a onfiguration by taking the following steps:. In the training stage, first assoiate eah base state ζ i with a feature vetor ψ i.. Extrat features ψ input from the input D image, suh as edge, area, entroid, et. 3. Compute α i = h( ψ i, ψ input ), where h( ψ i, ψ input ) measures the loseness of ψ input to ψ i. 4. Based on the observation made from Type III motion onstraints, linearly interpolate the estimated onfiguration in ompressed spae Φ : estimate 8 φ = α ζ (6) i= 5. Reonstrut the estimated onfiguration state 5 φestimate R from φ estimate. 4. Experimental results i i Figure 6: Configuration estimations. (a) (b) () (d) Figure 7: Comparison of different tehnique. (a) original image. (b) estimation without Type II & III onstraints. () estimation without Type III onstraints. (d) estimation with Type I, II & III onstraints The results of the experiments are shown in Figure 6. The first row shows some input images and the seond
row shows the reonstruted 3D hand model based on estimation by our approah. The results are visually agreeable. Suh preliminary experiments show that the motion onstraints play an important role in hand posture estimation. More aurate and ost-effiient estimation an be obtained when a better motion onstraint model an be applied. Better result an be obtained with better feature extration methods, whih will be implemented in the future researh. A omparison of estimations using different types of onstraints is also shown in Figure 7. In Figure 7(b), estimation without applying Type II and III onstraints result in a feasible, yet unnatural onfiguration. In Figure 7(b), a loser approximation is obtained without applying Type III onstraints. The DIP and PIP joints should bend more to approximate a fist. Finally, by applying all three types of onstraints together produe the better result with a more natural approximation in Figure 7(). 5 Conlusion/Future Development A posture estimation problem generally involves a searh in high dimensional C-spae. Useful hand onstraints have been demonstrated to be able to greatly redue the searh spae, and thus improve gesture reognition results. Many onstraints an be represented in simple losed forms while many more an not and have not been found. In this paper, we presented a novel approah to model the hand onstraints. Our model has three harateristis. First, it is ompat by utilizing PCA tehnique. Seond, it inorporates onstraints that an and annot be represented by equations. Third, it displays a linear behavior in state transitioning as a result of natural motion. These properties together simplify onfiguration estimation in C- spae as shown in Eq 5 by a simple interpolation with linear polynomials. Some preliminary gesture estimation experiments are shown, taking advantage of this model. However, there is still muh to be done to improve this model. For instane, more states an be inluded to further refine the model. Deiding whih states to hoose will require more analysis of the C-spae. Furthermore, other onstraints might exist in the C-spae that haev not yet been observed. Finally, even though a nearly linear behavior is observed in state transition, it is not exatly linear. A more detailed study an better approximate the trajetories, whih in turn would help improve the onfiguration estimation. Nevertheless, suh modeling provides a different interpretation of hand motions and the urrent results look promising. Aknowledgement This work was supported in part by National Siene Foundation Alliane Program and Grant CDA 96-4396. Referenes [] C. Chang, W. Tsai, Model-Based Analysis of Hand Gestures From Single Images Without Using Marked Gloves Or Attahing Marks on Hands, ACCV, pp.93-93, [] C.S. Chua, H. Y. Guan and Y. K. Ho, Model-based Finger Posture Estimation, ACCV, pp.43-48,. [3] J. Kuh and Thomas S. Huang, Vision-Based Hand Modeling and Traking for Virtual Teleonferening and Teleollaboration, ICCV95, pp.666-67, 995. [4] J. Lee, T. Kunii, Model-based Analysis of Hand Posture, IEEE Computer Graphis and Appliations, Sept., pp.77-86, 995. [5] V. Pavlovi, R. Sharma, Thomas S. Huang, Visual Interpretation of Hand Gestures for Human-Computer Interation: A Review,, IEEE PAMI, Vol. 9, No. 7, July, pp.677-695, 997 [6] J. Rheg, T. Kanade, Model-Based Traking of Self- Oluding Artiulated Objets, IEEE Int l Conf. Computer Vision, pp.6-67. 995. [7] N. Shimada, et al., Hand Gesture Estimation and Model Refinement Using Monoular Camera- Ambiguity Limitatio by Inequalty Constraints, Pro. Of the 3 rd Conf. On Fae and Gesture Reognition, 998. [8] Y. Wu, Thomas S. Huang, Human Hand Modeling, Analysis and Animation in the Context of HCI, ICIP99, Japan, Ot., 999. [9] Y. Wu, Thomas S. Huang, Capturing Human Hand Motion: A Divide-and-Conquer Approah, IEEE Int l Conf. Computer Vision, Greee, 999.