Character Animation from a Motion Capture Database

Max-Planck-Institut für Informatik Computer Graphics Group Saarbrücken, Germany Character Animation from a Motion Capture Database Master Thesis in Computer Science Computer Science Department University of Saarland Edilson de Aguiar Supervisors: Dipl. Inf. Christian Theobalt Prof. Dr. Hans-Peter Seidel Max-Planck-Institut für Informatik Computer Graphics Group Saarbrücken, Germany Begin: June 1, End: November 26,

ii

Eidesstattliche Erklärung Hiermit erkläre ich an Eides statt, dass ich die vorliegende Mastersarbeit selbständig und ohne fremde Hilfe verfasst habe. Ich habe dazu keine weiteren als die angeführten Hilfsmittel benutzt und die aus anderen Quellen entnommenen Stellen als solche gekennzeichnet. Saarbrücken, den 26. November, 2003 Edilson de Aguiar

iv

Abstract Character Animation from a Motion Capture Database Edilson de Aguiar Master Thesis in Computer Science Computer Science Department University of Saarland This thesis discusses methods that use information contained in a motion capture database to assist in the creation of a realistic character animation. Starting with an animation sketch, where only a small number of keyframes for some degrees of freedom are set, the motion capture data is used to improve the initial motion quality. First, the multiresolution filtering technique is presented and it is shown how this method can be used as a building block for character animation. Then, the hierarchical fragment method is introduced, which uses signal processing techniques, the skeleton hierarchy information and a simple matching algorithm applied to data fragments to synthesize missing degrees of freedom in a character animation, from a motion capture database. In a third technique, a principal component model is fitted to the motion capture database and it is demonstrated that using the motion principle components a character animation can be edited and enhanced after it has been created. After comparing these methods, a hybrid approach combining the individual technique s advantages is proposed, which uses a pipeline in order to create the character animation in a simple and intuitive way. Finally, the methods and results are reviewed and approaches for future improvements are mentioned.

Acknowledgements First I want to thank my supervisors: Dipl. Inf. Christian Theobalt and Prof. Dr. Hans-Peter Seidel for their help and advice during the development of this thesis. In addition, I thank all my friends in the IMPRS and Kerstin Meyer-Ross, for her help here in Germany. For all my colleagues of the Computer Graphics group at MPI, thank you, specially Volker Blanz, for his help with the PCA theory and Thomas Annen and Grzegorz Krawczyk for helping me with the Latex stuff. I also wish to thank my family, who always supported me, encouraging me and making me never give up, despite the distance. Thank you Mom, Dad, Raquel, Rose and Enoc. Edilson de Aguiar

Contents 1 Introduction 1 1.1 Motivation.............................. 1 1.2 Goals................................ 2 1.3 Thesis Outline............................ 3 2 Fundamentals of Character Animation 5 2.1 Introduction............................. 5 2.2 Keyframing............................. 5 2.3 Physical Simulation......................... 6 2.4 Motion Capture........................... 8 2.5 Project Implementation Aspects.................. 10 2.5.1 Motion Capture Database................. 11 2.5.2 Skeleton Model....................... 12 3 Multiresolution Filtering Method 15 3.1 Introduction............................. 15 3.2 Related Work............................ 16 3.2.1 Signal Processing Methods................. 16 3.2.2 Multiresolution Methods.................. 16 3.3 Multiresolution Filtering Method.................. 16 3.4 Multiresolution filtering on motion data.............. 18 3.5 Application to Character Animation................ 19 3.6 Experiments............................. 20 3.7 Discussion.............................. 21 4 Fragment Based Methods 25 4.1 Introduction............................. 25 4.2 Related Work............................ 26 vii

CONTENTS viii 4.3 Overview.............................. 27 4.4 Motion Analysis........................... 31 4.4.1 Motion Phases....................... 31 4.4.2 Frequency Analysis..................... 33 4.4.3 Correlation......................... 33 4.5 Motion Synthesis and Texture................... 34 4.5.1 Fragmentation....................... 34 4.5.2 Matching.......................... 35 4.5.3 Joining........................... 36 4.5.4 Smoothing......................... 39 4.6 Experiments............................. 40 4.7 Hierarchical Fragment Method................... 41 4.7.1 Skeleton Hierarchy and Correlation............ 43 4.7.2 Method........................... 44 4.7.3 Experiments........................ 47 4.8 Discussion.............................. 48 5 Principal Component Analysis 53 5.1 Introduction............................. 53 5.2 Related Work............................ 54 5.3 Principal Component Analysis................... 55 5.3.1 Overview.......................... 55 5.3.2 PCA Theory........................ 55 5.3.3 Data Compression..................... 57 5.4 PCA for motion synthesis...................... 58 5.4.1 Overview.......................... 58 5.4.2 Motion Synthesis...................... 58 5.5 Experiments............................. 59 5.6 Discussion.............................. 62 6 Hybrid Approach 67 6.1 Introduction............................. 67 6.2 Hybrid Approach.......................... 68 6.3 Experiments............................. 69 6.4 Discussion.............................. 70 7 Conclusion and Future Work 77

List of Figures 2.1 Example of the motion capture session and equipments used to capture the motions used in the database. In (a) it is shown the camera setup and in (b) the subject performing the motion. Images used from http://www.e-motek.com................. 12 2.2 Example of the motion data in the database. The figure shows respectively the z-angle values for the pelvis, hip, clavicle, forearm and knee joints............................ 13 2.3 Joints and bones forming the skeleton model used in the project.. 14 2.4 The skeleton joint hierarchy. On the right side it shows the lower kinematic sub-chain and on the left the upper kinematic sub-chain. 14 3.1 Generation of the Gaussian pyramid. The value of each node in the next row,, is computed as a weighted average of a sub-array of nodes. In this example a sub-array of length five is used. Adapted from [BA83]........................ 17 3.2 Visualization of different frequency bands of a Gaussian pyramid (shown only for the first 40 frames). The band g0 corresponds to the original signal. The low-pass bands corresponding to the high frequency are g1 and g2, to the middle are g3 and g4, and to the low frequency are g5 and g6..................... 20 3.3 Visualization of different frequency bands of a Laplacian pyramid (shown only for the first 40 frames). The band-pass bands corresponding to the high frequency are l0 and l1, to the middle are l2 and l3, and to the low frequency are l4 and l5............ 21 3.4 Using multiresolution to increase the gain in the middle frequencies 23 3.5 Using multiresolution to decrease the gain in the middle frequencies 24 ix

LIST OF FIGURES x 4.1 The input for the general fragment based method: (a) keyframed and motion capture data are decomposed into frequency bands; (b) animators set the general method parameters. Driven and master joints are chosen to guide the method, and a particular frequency band of the master joint is chosen to guide the fragmentation step. Joints in black will be textured and in blue and red will be synthesized................................. 29 4.2 A fragment based method is composed by four steps: fragmentation (a), matching (b), joining (c) and smoothing. At the end, an original keyframed character animation is enhanced by synthesis and texturing (d)........................... 30 4.3 Example of a walking animation: (a) set of four phases during a human walking cycle; (b) the right hip z-angle values is plotted, where it is possible to see the respective phases........... 32 4.4 Plot of the pelvis joint angle against the hip joint angle for all examples in the database. The shape shown in red demonstrates a good correlation between these joints................ 33 4.5 Example of the fragmentation step: (a) original degree of freedom; (b) fragments created at locations where the first derivative changes its sign................................ 34 4.6 Considering one driven fragment (a), in the matching step all data fragments are compared with the driven fragments (b) being stretched or compressed properly (c). At the end, a number of good fragments are found (d).......................... 36 4.7 In the joining step the good fragments found in the matching step are concatenated or blended (a). Three different criteria were tested and compared: (b) best fragment; (c) cost matrix and (d) best animation................................ 38 4.8 In the smoothing step, the discontinuity magnitude (top left) is multiplied with a smoothing function (top right) and the result is added back to the original motion signal. In this way, the continuous version shown on the bottom left is generated. Adapted from [AF02]................................ 39

LIST OF FIGURES xi 4.9 Example of the joining approaches developed. In (a) all good fragments generated in the matching step for the z-angle of the pelvis joint are shown. The final sequence generated by the three approaches are then shown in (b), (c) and (d)............. 42 4.10 Example of possible joint correlations: (a) represents a bad correlation between joints that do not belong to the same kinematic sub-chain; (b) represents a good correlation between joints that belong to the same kinematic sub-chain................ 43 4.11 Example of the root generation stage: (a) driven joints (red) and root joint (green) are shown; (b) the parents of the driven joints are generated; (c) next parent joint is generated and (d) root joint is generated............................... 45 4.12 Example of the motion generation stage: (a) root joint; (b) root joint together with the original driven joints are used to texture or synthesis of its children; (c) next children are generated; (d) all joints are generated.......................... 46 4.13 Example of the Hierarchical Fragment method being applied to a human character: (a) shows some frames from the initial character animation where only lower body joints are keyframed; (b) shows the respective frames from the resulting character animation where upper body joints are synthesized and lower body joints are textured. 50 4.14 Another example of the final method being applied to a human character: (a) shows some frames from the initial character animation where only upper body joints are keyframed; (b) shows the respective frames from the resulting character animation where lower body joints are synthesized and upper body joints are textured. 51 4.15 Another example of the final method being applied to a human character: (a) shows some frames from the initial character animation where all left body side joints are keyframed; (b) shows the respective frames from the resulting character animation where all joints are synthesized......................... 52

LIST OF FIGURES xii 5.1 Three approaches are investigated in order to fit a PCA model to a keyframed and motion capture data (a). In (b) one PCA model is created for each missing DOF. In (c) one PCA model is created for each frequency band of a missing DOF. In (d) one PCA model is created for each sequence position of a particular frequency band of a missing DOF........................... 64 5.2 Applying the PCA method to a human character: (a) shows some frames from the initial animation where the left hip, knee, elbow and upper-arm joints are keyframed; (b) shows the respective frames of the resulting animation where new motion for the joints are generated from the database in order to match the keyframed DOFs.. 65 5.3 Use of the motion principle components in order to edit a motion: (a) arm positions are modified by altering the principle components for the right and left clavicle joints; (b) leg positions are also modified by altering the principle components for the right and left knee joints................................. 66 6.1 Pipeline showing the Hybrid approach............... 68 6.2 Example of the first stage: (a) shows some frames from the initial animation where left body side joints are keyframed; (b) shows the respective frames after applying the PCA method; (c) and (d) show possible editing capabilities of the PCA, where the influence of the principle components for some DOFs are modified......... 72 6.3 Example of the second stage: (a) shows some frames from the initial animation where left body side joints are keyframed; (b) shows the respective frames after applying the first stage (PCA); (c) shows the same frames after applying the Multiresolution Filtering method to decrease the low frequency bands, generating a smooth acceleration of the movement................ 73 6.4 Example of the last stage: (a) shows some frames from the initial animation where left body side joints are keyframed; (b) shows the respective frames after applying the first stage; (c) shows the frames after applying the second stage; (d) shows the same frames after applying the Hierarchical Fragment Method in order to improve its realistic appearance..................... 74

LIST OF FIGURES xiii 6.5 Example of the Hybrid approach: (a) shows some frames from the initial animation where left body side joints are keyframed; (b) shows the respective frames from the animation generated automatically by the hybrid method; (c) and (d) show the same frames from the resulting animations generated when motion principle components are used to change arm and leg positions by adding or subtracting constant values.................... 75

LIST OF FIGURES xiv

Chapter 1 Introduction 1.1 Motivation Generating realistic character animation remains one of the great challenges in computer graphics. Currently, there are three main methods by which this animation can be generated. Most commonly, keyframing is used, in which the animator specifies important key poses for the character at specific frames, and the computer calculates the frames in between by an interpolation technique. A second approach uses physical simulation in order to drive the character s motion. Although results seem to be promising, due to lack of control, difficulty of use, instabilities and high computation cost, this method has not been used with much success for characters. The last approach, motion capture, has been widely used to animate characters. The idea is to use sensors placed on subjects and collect the data that describes their motion while they are performing the desired motion. As the technology for motion capture has improved and the cost decreased, the interest in using this approach for character animation has also increased. The main challenge that an animator is confronted with is to create sufficient detail in order to generate an character animation with a realistic appearance. Achieving detail in a keyframed animation is extremely labor intensive. However, with motion capture the details are immediately present. In other words, the data contains the motion signature. The main problem with motion capture is the lack of flexibility. For instance, after collecting the data it is difficult to change it. Due to this problem, many animators have little interest in using motion capture data. Although keyframing is labor intensive, the animator can make a character do exactly what he wants it to. 1

CHAPTER 1. INTRODUCTION 2 Since it is often difficult to know exactly what motions are needed before entering a motion capture session, many techniques have been developed to edit motion capture data after it has been collected (see Sec. 2.4). In motion editing, two different aspects should be considered: first, it is important to take care not to alter the motion in such a way that the detail is lost. Second, the system should not just provide editing capabilities but it should capture the essence of the motion, since the animator may want a completely different action. Our intent in this project is to consider the case that an animator wants to use a number of existing motion sequences, for instance stored in a motion capture database, to generate new motions. The idea is to use the style and life-like qualities of the motions in the database to add details and a particular style to an initial keyframed animation. Then, a different approach to create a character animation is proposed: the animator starts the animation with keyframing, a method that he is familiar with, to create some degrees of freedom. After that, the motion capture database is used interactively to enhance the initial keyframed animation. At the end, using the strengths of keyframing and motion capture, the character performs realistic motion while preserving the keyframed style and incorporating the details of the motions in the database. As a result, the animator does not need to spend hours defining key poses and the expensive motion capture session can be kept at a minimum. 1.2 Goals As mentioned in the previous section, the main goal of this work is to combine the strengths of keyframing and motion capture in order to simplify the process of creating character animations. In recent years, different approaches have been used in order to achieve this goal. Liu and Popovic [LP02] presented a method for prototyping realistic character motion using a constraint detection method that automatically generates the constraints by analyzing the input motion. Tanco and Hilton [TH00] presented a system that synthesizes motion sequences from a database of motion capture examples using a statistical model created from the captured data. Pullen and Bregler [PB02] described a method to enhance an initial keyframed animation using motion capture data, by decomposing the data into frequency bands and fragments. In this project, we want to analyze important characteristics of these methods, by implementing and comparing two different approaches (chapters 4 and 5) and

CHAPTER 1. INTRODUCTION 3 verifying the importance of some aspects of human motion: skeleton hierarchy, joint correlation, motion frequency range and motion phases. In addition, we try to analyze and synthesize motion by fitting a principal component model. Another goal is to understand the details of the characteristics of human motion: variations in repetitive motions and differences in the same movement executed by different individuals. A detailed understanding of these data is important in fields like biomechanics and medicine, where many quantitative studies of human motion have been conducted for the purposes of treatment and prevention of injuries [AA00]. This understanding is also interesting for character animation, since the details of a motion usually reveal mood or personality. Using this understanding, in a long term we intend to develop a system able to describe and characterize the character motion in a high level way. For instance, using simple parameters and descriptions, the system will be able to create character animations reflecting aspects like gender, mood and personality. 1.3 Thesis Outline Chapter 2 gives a brief review of the three main animation methods. Their advantages and disadvantages are described in more detail with references to the related work. In addition, some implementation aspects of our project are mentioned: the skeleton model and the motion database that was used. Chapter 3 presents the multiresolution filtering technique. Possible motion editing capabilities of this technique are investigated and it is shown how this method can be used as a building block for character animation. In chapters 4 and 5 methods implemented to enhance an initial keyframed animation are described. Chapter 4 introduces the fragment based methods, showing that they can be successfully applied to character animation. After decomposing keyframed and captured data into frequency bands, motion phases are used to divide the motion capture data in small pieces, which are used to improve the original keyframed animation. In chapter 5 a principal component model is fitted to the motion capture database and it is shown that using motion principle components it is possible to create, edit and enhance a character animation. Using the techniques described in the previous chapters as components, chapter 6 introduces a pipeline solution to interactively control the creation of a character animation, which results in a better performance and more expressive results.

CHAPTER 1. INTRODUCTION 4 Finally, in chapter 7 all techniques and their respective results are briefly reviewed and approaches for future improvements are mentioned.

Chapter 2 Fundamentals of Character Animation 2.1 Introduction In this chapter the three main methods by which character animations are created will be briefly described: keyframe interpolation, physical simulation and motion capture. Each of these methods has its advantages and disadvantages and they are appropriate in different situations. In the following sections, characteristics of these animation methods will be reviewed in more detail on the basis of their relevant related work. In the last section, some important implementation aspects related to the project are mentioned: the skeleton model and the motion capture database that was used. 2.2 Keyframing In the traditional technique, an animator first draws the motion extremes, and then the intermediate frames using keyframes as a guide. Nowadays, using computers, an animator can specify keyframes by posing the model in a specific position. A computer then calculates the remaining frames by interpolating between keyframes in order to create the motion curves that drive the model action. The main problem with keyframing is that this technique is time-consuming and labor intensive. A typical articulated kinematic model, such as a humanoid character, usually has at least 50 degrees of freedom. In keyframing, an animator must animate all these DOFs, one at a time. To construct a more realistic model, its 5

CHAPTER 2. FUNDAMENTALS OF CHARACTER ANIMATION 6 complexity is increased and the animator must keyframe more degrees of freedom. Usually, constraints, like position of legs and arms at specific times, are always a problem because all DOFs should be specified at these positions in the character. One possible way to treat this problem is by using inverse kinematics to simplify keyframe definition. In this case, posing arm or leg, it is possible to calculate the position of parent joints - forearm, hip and knee, in order to permit the character to achieve this point. Another problem with this technique is the interpolation process. Usually interpolation is done with smooth splines which fails to model variations in high frequency that real human motion has. The number of keyframes set by the animator is also an important aspect. If too few keyframes are set, the motion may lack the details usually seen in live motion. To overcome this problem, trained animators achieve a high level of detail by setting more and more keyframes, but in this case, at the expense of more time and effort. Although keyframing is extensively used by animators, nowadays it is not a topic of much research. Recent works related to keyframing are trying to improve its quality using noise functions to describe variations in the motion signature. Bodenheimer et al. [BSH99] described how to introduce natural-looking variability into cyclic human motion animation using a noise function. Perlin and Goldberg [PG96] presented Improv, a system for scripting interactive actors in a virtual world using Perlin noise functions [Per85] to characterize personality and mood in human motion. 2.3 Physical Simulation This technique tries to reduce the animator s work by using physics to determine the motion in situations where it can be clearly specified. Although these methods have been successfully applied for animating cloth deformation (DeRose [DKT98]), rigid bodies (Baraff [Bar94] and Moore [MW88]) and to fluids (Foster and Fedkiw [FF01]), the application of physics based methods for the generation of character motion remains challenging. Most of the work in physical simulation was done by researchers seeking accurate models for use in biomechanical studies, as stated in Pandy [Pan01]. Such models were found to be very complex: more than one muscle can control one joint, muscles exert nonlinear forces on tendons and joints usually have complex kinematics, involving sliding and rotation about multiple axes, as mentioned in

CHAPTER 2. FUNDAMENTALS OF CHARACTER ANIMATION 7 Delp and Loan [DL00]. Clearly, this type of modeling is not practical for computer graphics, where an animator wants to quickly compose a wide variety of motions. In fact, an animator most certainly will not know the proper configuration of muscles and bones or the amount of energy required in a specific movement. An alternative way to create animations of articulated figures using physics was done by Witkin and Kass [WK88] in the method called spacetime constraints. Considering the entire animated motion as a numerical problem, in contrast to most previous methods that consider each frame independently, spacetime constraint methods allow motion editing while preserving the characteristics of the original motion and motion animating based on incomplete observations. The general approach is to specify some physical parameters of the character, like masses of each limb and spring constants of the joints. Then using constraints, like leg and arm positions at a specific time, an animator can control the character key positions. At the end, the motion is determined by solving a constrained optimization problem, where the character energy is minimized while the constraints are satisfied. However, normally it is difficult to model the complex human motion and correctly specify masses and joints spring constants. Another problem is the high computation cost for solving the constrained optimization problem. The most successful attempts at using physical simulation of human motion come from robotics research. For instance, Raibert [Rai02] introduced some form of feedback control in the robot movement instead of just solving the equations of motion and predicting the proper initial conditions. Applying the same principle to character animation, Hodgins et al. [HWBO95] developed a method to apply control systems to virtual humans. In other work, Raibert et al. [RH91] applied these controls to create animations of humans performing athletic events like running, biking and a gymnast vaulting. Although the characters vaguely performed the activity being simulated, the motions did not look realistic. Another problem is that each activity was treated separately, while it turned out that their method cannot be used in the same way for different activities. Due to lack of control, difficulty of use, instabilities and high computation cost, the use of physics simulation for animation of a complete character is not widely used yet. One possible way to include physics in character animation is by using spacetime constraints with other techniques, as described in the next section.

CHAPTER 2. FUNDAMENTALS OF CHARACTER ANIMATION 8 2.4 Motion Capture In this technique joint angles of a performing actor are recorded via sensors. These values are then used to create a character motion (Menache [Men99]). In the past, such data was extremely difficult to obtain, as the sensor technology was costly. However, recently the technology has improved, the costs decreased and, therefore, this technique is becoming more available for general use. Currently the most common techniques for obtaining motion capture data are mechanical, optical, magnetic and video-based systems. In a mechanical system, tracking is performed by having the subject wear a mechanical device, or exoskeleton. Then, angle measuring equipments are located at exoskeleton joint locations. By measuring the joint angles of the exoskeleton, subject limb orientation is obtained. The main advantage of this system is that it is accurate, cheap and allows the use of haptic devices in order to generate feedback reactions. However, it is not easy to perform fast and expressive motions due to the weight of the exoskeleton and the limited range of the angle measurement devices. In addition, problems due to shift positions of the exoskeleton cause errors in the motion capture process. In an optical system, retro-reflective markers are attached to the body of the subject. Then, a set of cameras surrounds the space where a subject moves and each camera sends out a beam of infrared light, which is reflected back from the markers. After the marker locations are recorded as 2D frames positions in the camera image planes, post-processing finds the 3D location of each marker at each time instant, and then solves for the joint configurations. The main advantages of optical systems are the very high rates of data collection and the possibility to create a great range of motion in a relatively large space. However, it requires intensive post-processing computations and presents problems related with occluded markers and with captured motions from more than one subject, where the markers can overlap each other. In a magnetic system, a known magnetic field is set up and a subject wears sensors that detect location and orientation of each limb based on the magnetic field. This technique allows real-time data collection and there are no problems with occlusion. On the other hand, this method is very sensitive to the area where the motion is performed. In addition, due to the wires that must be attached to sensors, many motions are awkward for the subject. Systems of the last category, video-based systems, try to get the motion data by merely using a couple of video cameras. Although it is an interesting technique,

CHAPTER 2. FUNDAMENTALS OF CHARACTER ANIMATION 9 it is complicated. Standard computer vision techniques do not work for an articulated figure, in which many of the motions cannot be defined by a simple affine transformation, but involve rotations about all the joints in the kinematic chain. Currently, much research is trying to make video-based motion acquisition more accurate and, therefore, useful for motion capturing. The interest in using motion capture for creating character animation is increasing. The main reason is that this technique can provide motion data for all degrees of freedom at a high level of detail. For instance, using motion capture all the details of a motion are inherent in the data, hence, coming for free. In addition, a transfer of data to a different character with different skeleton dimensions is also possible. On the other hand, this technique also has disadvantages. The main problem is the lack of flexibility. Since it is difficult to modify a captured motion after it has been collected, an animator should know exactly what he wants. In general this is not the case since the process of creating a character animation is normally evolutionary. Usually, an animator has only a coarse impression of the desired motion before he captures it, minor connections might be needed anytime thereafter. In addition, motion capture sessions are still costly and labor intensive, which makes the repetition process prohibitive. As a result, a great deal of research in recent years aimed at providing better ways of editing motion capture data after it is collected. A general approach is to adapt the motion to different constraints while preserving the style of the original motion. Wiley and Hahn [WH97] developed a method providing inverse kinematics capability by mixing motions from a database to create a new animation that matches a certain specification. Witkin and Popovic [WP95] developed a method in which the motion data is warped between keyframe-like constraints set by the animator. Warping is done by overlapping and blending motion clips. Rose et al. [RCB98] developed a method which uses radial basis functions and low order polynomials to interpolate between example motions while maintaining inverse kinematic constraints. As mentioned in the previous section, spacetime constraints can be used in order to include physics in character animation. Gleicher [Gle97] presented a method that allows an animator to start with an initial animation and to interactively repose the character. A spacetime constraint solver is then used to minimize the difference between the new and old motion, subject to constraints specified by the animator. A similar approach was also used by Gleicher [Gle98] to retarget motions to char-

CHAPTER 2. FUNDAMENTALS OF CHARACTER ANIMATION 10 acters of different dimensions. Lee and Shin [LS99] combined a hierarchical curve fitting technique with an inverse kinematics solver to adapt the motion. Popovic and Witkin [PW99] developed a method that uses a reduced dimensionality space and dynamics to perform the editing process. Rose et al. [RGBC96] described the generation of motion transitions using a combination of spacetime and inverse kinematics constraints in order to create seamless and dynamically plausible transitions between motion segments. A more general problem with motion capture is that it is not an intuitive way to start a character animation. Since many factors (such as the environment) influence the motion, the final motion sequence will not be known with all details right from the beginning. Animators are usually trained to use keyframes. They will often build an animation by first making an initial motion sketch with a few keyframes and add complexity and detail on top of that later. Therefore, combining the strengths of keyframing and motion capture, the process of animating a character is expected to be simplified. In our approach, an animator starts a character animation with keyframing, a method that he is familiar with, to animate some degrees of freedom. After that, he uses the motion capture database interactively to enhance the initial animation. 2.5 Project Implementation Aspects The methods described in the next chapters are implemented in our prototype system using the C++ programming language on the Linux platform. In order to facilitate the skeleton and animation manipulation, the free open-source character animation library CAL3D 1 is used in the project. The character animation library, CAL3D, is coded in C++ and uses the STL containers to store the data. It provides basic data structures for skeleton-based character animation: sequencing and blending of animations, handling of bones, skeletons, materials and meshes. The character motion is composed by two kinds of transformations: translation and rotation. Translations are represented as vectors and rotations as quaternions. In comparison with other rotation representations (e.g. explicit matrices or Euler angles), keyframe interpolation using quaternions is accurate and intuitive. 1 http://cal3d.sourceforge.net/

CHAPTER 2. FUNDAMENTALS OF CHARACTER ANIMATION 11 2.5.1 Motion Capture Database The motion capture data used in this work was obtained from MOTEK 2, a motion capture company that provides a set of motion sequences to the research community. The company uses a VICON 8 optical motion capture system with 8 to 24 cameras for data acquisition. Using cameras placed around the capture space to track the positions of markers attached to the body of performers, triangulation is used to compute the 3D position of a marker at any given sample, from an array of 2D information from every camera. For data processing the Diva software is used, which applies template matching software algorithms in order to solve occlusions and marker disappearance problems. Using markers attached to the body of a subject, cameras surrounding the subject send out a beam of infrared light, which is reflected back from the markers. After the marker positions are recorded in 2D from each camera position, postprocessing finds the 3D location of each marker at each point in time, and then solves for the joint configuration. At the end, the data is delivered as a set of translations and rotations of the joints of an articulated body which corresponds with the description of a human character. Figure 2.1 shows the motion capture session and equipments used to create the motions in the database. To evaluate the methods presented in this work, all the sequences of the database that represent variations of one particular action were considered. In our project we chose human walking, since it is the most common action that need to be animated in a realistic way. Starting with an initial database of 129 different motions, each of which of 40 frames long, we used the 3D Studio MAX version 5.1 3 to increase the length of all animations. Since walking can be considered as a cyclic motion, we created the final animation by repeating the original four times. At the end, each animation in the database has the length of 160 frames. Using the exporting software provided by the CAL3D library, these animations were exported to be used in our prototype as CAL3D animation files. Figure 2.2 shows the motion data plotted over time for different degrees of freedom (i.e. joint angles) for a particular animation sequence in the database. 2 http://www.e-motek.com 3 http://www.discreet.com/products/3dsmax/

CHAPTER 2. FUNDAMENTALS OF CHARACTER ANIMATION 12 (a) camera setup surrounding the space where a subject performs a motion (b) markers attached to the body of a subject performing the desired motion Figure 2.1: Example of the motion capture session and equipments used to capture the motions used in the database. In (a) it is shown the camera setup and in (b) the subject performing the motion. Images used from http://www.e-motek.com. 2.5.2 Skeleton Model In the skeleton-based approach, a skeleton model is a kinematic chain consisting of bones and connecting joints. Figure 2.3 shows the graph structure that specifies an articulated figure defining a character. Each bone in the skeleton, drawn in white, is a link in the graph and represents a limb. Joints, drawn in blue, connect the bones and represent rigid body transformations between them. Translations are represented by vectors and have three degrees of freedom. Rotations are represented by unit quaternions and also have three degrees of freedom. As shown in figure 2.4, a skeleton can be treated as a hierarchical model, where each joint has a parent and children. Joint rotations or translations of a parent affects all of its children joints. The paths from the skeleton root to the terminating joint in the skeleton, shown in the figure, can be treated as kinematic sub-chains, exhibiting different behaviors depending on the motion being performed. For instance, in a walking motion, the lower kinematic sub-chain, composed by pelvis, hip, knee, foot and toe, is more active, being responsible for the translational movement of the body. On the other hand, the upper kinematic sub-chain, composed by pelvis, spine, elbow, upper-arm, forearm, hand and finger, helps controlling the body equilibrium.

CHAPTER 2. FUNDAMENTALS OF CHARACTER ANIMATION 13 Angle values for some DOFs (rad) Keyframes (time) Figure 2.2: Example of the motion data in the database. The figure shows respectively the z-angle values for the pelvis, hip, clavicle, forearm and knee joints. The skeleton used during this research has 21 bones and 22 joints connecting them. Since the skeleton structure is fixed, only rotations are considered in our project, and then, each joint is represented by an unit quaternion, being described by three angular degrees of freedom. In total, the skeleton model representing the human character has 66 degrees of freedom. This skeleton is created in 3D Studio MAX using the biped tool from Character Studio version 4.0 4, in order to match the skeleton format of the motion capture files in the database. 4 http://www.discreet.com/products/cs/

CHAPTER 2. FUNDAMENTALS OF CHARACTER ANIMATION 14 HEAD UPPERARM FOREARM ELBOW FINGER NECK HAND PELVIS SPINE HIP KNEE FOOT TOE Figure 2.3: Joints and bones forming the skeleton model used in the project. PELVIS SPINE R HIP L HIP NECK R ELBOW L ELBOW R KNEE L KNEE HEAD R UPPERARM L UPPERARM R FOOT L FOOT R FOREARM L FOREARM R TOE L TOE R HAND L HAND R FINGER L FINGER Figure 2.4: The skeleton joint hierarchy. On the right side it shows the lower kinematic sub-chain and on the left the upper kinematic sub-chain.

Chapter 3 Multiresolution Filtering Method 3.1 Introduction In this chapter it is demonstrated that techniques from image and signal-processing can be a useful tool to generate and edit character animation. Using a skeletonbased approach to represent a character, the body configuration can be defined as joints parameters (e.g. rotations and translations). Over time, these parameters can be treated as signals and, therefore, analyzed with techniques from signalprocessing. In the motion capture data (see Sec. 2.5.1), each joint is represented by three degrees of freedom, where each DOF can be treated as a discrete sampled signal, corresponding to its values at each time instant, also known as frame. This chapter describes how a technique called multiresolution filtering can be applied to motion capture data. The idea is to use a range of filter-banks to pass the motion signal through a cascade of low-pass filters to produce a set of short-time band-pass and low-pass signal components. After filtering, the motion signal is represented as a collection of frequency bands where low frequency bands describe the global pattern of the motion signal, the middle frequency bands provide the motions details, and the high frequency bands usually contain noise like jitter and wiggling. Therefore, decomposing the motion data into frequency bands, existing motion data can be edited interactively by amplifying or attenuating a particular frequency band and new motions can be generated by band-wise blending of existing motions. In addition, this method can be used to concatenate and to compare motion sequences. It will be shown in the subsequent chapters that this functionality can be used as a building block for character animation. 15

CHAPTER 3. MULTIRESOLUTION FILTERING METHOD 16 3.2 Related Work A number of different approaches have been presented in the literature using signal processing analysis for editing motion data. Here they are conceptually categorized in signal processing methods and, more specifically, multiresolution methods. 3.2.1 Signal Processing Methods Unuma et al. [UAT95] used Fourier analysis techniques to interpolate and extrapolate motion data in the frequency domain. In this work, they could manipulate motion data and alter the style through interpolation, extrapolation and transitional tasks. Liu et al. [LGC94] reported that adaptive refinement with hierarchical wavelets provides a significant speed-up for spacetime optimization. Amaya et al. [ABC96] used Fourier analysis to generate emotional animation from neutral human motion. 3.2.2 Multiresolution Methods The notion of multiresolution analysis was invented by Burt and Adelson [BA83], who introduced a multiresolution image representation, the Gauss-Laplacian pyramid, to facilitate such operations as seamless merging of image mosaics and temporal dissolving between images. Lee and Shin [LS00] developed a multiresolution analysis method that guarantees coordinate-invariance without singularity, using hierarchical displacement mapping and motion filtering. In another work, Lee and Shin [LS01] presented multiresolution motion analysis as a unified framework to facilitate a variety of motion editing tasks. Bruderlin and Williams [BW95] adopted a digital filter-bank technique to address multiresolution analysis of discrete motion data. Their hierarchical representation of a motion with frequency bands allows level-by-level editing of motion characteristics. 3.3 Multiresolution Filtering Method Originally the method was developed for image mosaics [BA83]. Applying the multiresolution method to a two-dimensional image, the first step is to obtain the low-pass or Gaussian pyramid. The Gaussian pyramid can be obtained by repeatedly convolving a small weighting function (w(l, m)) with the image, while it is sub-sampled by a factor of 2 at each iteration (figure 3.1). This

CHAPTER 3. MULTIRESOLUTION FILTERING METHOD 17 Figure 3.1: Generation of the Gaussian pyramid. The value of each node in the next row,, is computed as a weighted average of a sub-array of nodes. In this example a sub-array of length five is used. Adapted from [BA83]. process is repeated until the image size is reduced to one, which is the average intensity, or constant value (DC). The pattern of weights w(l, m) used to generate each pyramid level from its predecessor is called the generating kernel. These weights are chosen subject to four constraints: For computational convenience, the generating kernel is separable: ŵ ŵ ; The function w(l,m) should be symmetric:, and ; The function w(l, m) should be normalized:, Each node of level l must contribute the same total weight to the nodes of level : In order to obtain the band-pass bands (also called Laplacian pyramid), each level of the pyramid is subtracted from the next lowest level. In this case, it is necessary to create new samples by interpolation, because these pixel grids differ in sample density. Finally, the low-pass and band-pass bands can be amplified or attenuated separately and the image can be reconstructed by adding up all the band-pass bands plus the constant value (DC).

CHAPTER 3. MULTIRESOLUTION FILTERING METHOD 18 3.4 Multiresolution filtering on motion data Following the algorithm described in Bruderlin and Williams [BW95], the principles of image multiresolution filtering will be applied to a particular degree of freedom of human motion, represented as a one-dimensional signal over time. Instead of constructing a pyramid of low-pass and band-pass sequences by reducing their lengths, the lengths are kept the same, but the filter kernel (w(m)) is expanded at each level by inserting zeros between their values. For example, using a initial kernel of length five (weights a, b and c), the following kernel sequence will be generated: = [c b a b c] = [c 0 b 0 a 0 b 0 c] = [c 0 0 0 b 0 0 0 a 0 0 0 b 0 0 0 c], etc The weights (a, b and c) can be found using the constraints described in section 3.3. Considering the kernel of length five the following values are found: and The length of the signal determines the number of frequency bands, which should be generated. In general for a signal of length m ( ), the number of computed bands (nb) is N. Then, the following steps are performed in order to create the Gaussian (G) and the Laplacian (L) pyramids: The Gaussian sequences are calculated by successively convolving the signal with the expanded kernels: (3.1) This can be calculated efficiently by keeping the kernel constant and skipping signal data points: (3.2) To deal with the case when the value lies outside the signal s domain, the signal is reflected on its boundaries.

CHAPTER 3. MULTIRESOLUTION FILTERING METHOD 19 The Laplacian sequences are calculated by subtracting two successive lowpass bands : (3.3) Finally the signal is reconstructed by summing up all band-pass bands and the constant value (DC): (3.4) During the reconstruction of the signal, each frequency band can be summed up with a different gain or weight ( ). Modification of these weights allows to increase or decrease the influence of each frequency band in the final reconstructed signal, generating slightly different signals and maintaining the basic information. 3.5 Application to Character Animation Looking at the motion data for a particular DOF as a one-dimensional signal over time, the algorithm described in the previous section is applied to the motion capture data motivated by the following intuition: low frequencies contain general and gross motion patterns, middle frequencies contain the signature of the motion and the high frequencies describe some details and possible noise. Then, treating each degree of freedom as a one-dimensional signal, it is possible to calculate its respective low-pass (G) and band-pass (L) pyramids. Using the Gaussian and Laplacian pyramids, the signal representing the degree of freedom can be reconstructed by summing up all band-pass bands plus the constant value (DC), as mentioned in section 3.3. By changing the reconstruction weights, it is possible to increase or decrease the effect of each frequency band on the final motion signal reconstruction. Different sets of weights will result in different motions, since each frequency band generally contains a particular information or detail of the overall motion. In this way, it is possible to perform some animation editing tasks. In addition, using the extracted frequency bands it is possible to create new motions by blending and concatenating frequency bands from two different existing motions. Examples of the Gaussian and Laplacian pyramids generated by the algorithm are shown in figures 3.2 and 3.3. Considering the pelvis z-angle as a sampled

CHAPTER 3. MULTIRESOLUTION FILTERING METHOD 20 Values for the Z-angle of the Pelvis (rad) 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 original g0 g1 g2 g3 g4 g5 g6 0.04 0 5 10 15 20 25 30 35 40 Keyframes (time) Figure 3.2: Visualization of different frequency bands of a Gaussian pyramid (shown only for the first 40 frames). The band g0 corresponds to the original signal. The low-pass bands corresponding to the high frequency are g1 and g2, to the middle are g3 and g4, and to the low frequency are g5 and g6. signal, figure 3.2 shows the generated Gaussian pyramid. The generated Laplacian pyramid is shown in figure 3.3 3.6 Experiments To demonstrate the frequency-based decomposition and motion creation power of the multiresolution method, the following experiments are performed. Using a human walking sequence with 160 frames as an example, six frequency bands are generated. Then, the multiresolution approach is simultaneously applied to all degrees of freedom of the skeleton model (see section 2.5.2). The same frequency band gains are used for all degrees of freedom and the resulting motion is generated at interactive rates. Figure 3.4 illustrates that it is possible to create a smoothed acceleration of the movement by increasing the middle frequencies (bands 3 and 4). In this figure it is also possible to see the comparison between the original signal and the reconstructed signal. Figure 3.5 shows an attenuated, constrained human walking with reduced joint movement when the middle frequencies are decreased. Similar re-

CHAPTER 3. MULTIRESOLUTION FILTERING METHOD 21 Values for the Z-angle of the Pelvis (rad) 0.2 0.15 0.1 0.05 0 original l0 l1 l2 l3 l4 l5-0.05 0 5 10 15 20 25 30 35 40 Keyframes (time) Figure 3.3: Visualization of different frequency bands of a Laplacian pyramid (shown only for the first 40 frames). The band-pass bands corresponding to the high frequency are l0 and l1, to the middle are l2 and l3, and to the low frequency are l4 and l5. sults can also be achieved by changing the low frequencies (bands 5, 6). Increasing the low frequencies attenuates the motion, and decreasing accelerates it. Since the high frequency bands generally represent noise, their changes resulted in a addition of a nervous twitch to the animation. 3.7 Discussion The multiresolution filtering technique provides a rapid interactive loop, and facilitate the reuse and the adaptation of motion data by amplifying or attenuating important frequencies of the motion data. This technique can provide a high-level motion control and facilitates to reuse and adapt existing motions of an articulated character. In the last instance, it can serve as building block for high-level character motion processing. Multiresolution filtering is not usable as a stand-alone technique for character animation. A drawback of this method is that some constraints such as joint limits or non-intersection with the floor can be violated in the filtering process. However, it can be solved by employing constraints or optimization after the definition of the

CHAPTER 3. MULTIRESOLUTION FILTERING METHOD 22 general motion. Nevertheless, the frequency decomposition can be used to extract certain features of a motion which can be used during character animation. Combining the multiresolution filtering with the techniques described in chapter 4, 5 and 6, a powerful tool has been developed which can be used in the following tasks: Concatenating motions: If two motion sequences should be concatenated, the multiresolution method gives control over the transition zone and the transition weight coefficient allowing concatenation band-by-band. Comparing motions: If two motion sequences should be compared, the generated low-pass and band-pass bands can be used in order to perform a comparison of only a particular frequency band, containing specific information. Blending motions: The multiresolution method allows blending of an animation band-by-band, where only important bands are added in order to generate a more realistic motion. In this case, it is possible to copy a particular information contained in a frequency band from one signal to another.

CHAPTER 3. MULTIRESOLUTION FILTERING METHOD 23 (a) original motion (b) reconstructed motion with increased gain 0.25 original i reconstructed i 0.2 Values for the Z-angle of the Hips (rad) 0.15 0.1 0.05 0-0.05 0 5 10 15 20 25 30 35 40 Keyframes (time) (c) signal comparison (only the first 40 frames) Figure 3.4: Using multiresolution to increase the gain in the middle frequencies

CHAPTER 3. MULTIRESOLUTION FILTERING METHOD 24 (a) original motion (b) reconstructed motion with decreased gain 0.2 0.18 original reconstructed Values for the Z-angle of the Hips (rad) 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0 5 10 15 20 25 30 35 40 Keyframes (time) (c) signal comparison (only the first 40 frames) Figure 3.5: Using multiresolution to decrease the gain in the middle frequencies

Chapter 4 Fragment Based Methods 4.1 Introduction In chapter 3 the editing and analysis power of the multiresolution filtering technique was described. In addition, it was mentioned how the multiresolution can be used as a building block for character animation, performing tasks like comparing, blending and concatenating motions. In this chapter, fragment based methods are described, which decompose motion into frequency bands and divide it into small pieces. A character animation is generated satisfying some guidelines given by an animator and using the details contained in fragments extracted from a motion capture database. The main goal is to create a character animation semi-automatically, following the same procedure that an animator would do manually. Animators are trained to use keyframing, and will often build a character animation by first making a rough animation with few keyframes to sketch out the motion. Complexity and details are added on top of that later. Using the same idea, a fragment based method combines the strengths of keyframing and motion capture. Starting from an initial character animation where only few degrees of freedom were keyframed, our method will use the information from the motion capture database to add details to the keyframed DOFs and to completely synthesize the remaining DOFs. At the end, an improved character animation will be generated, based on the animator s keyframed sketch and performing what he wanted. By combining the strengths of keyframing and motion capture, the creation time of an animation and hence the overall animation costs are reduced. As stated 25

CHAPTER 4. FRAGMENT BASED METHODS 26 in chapter 1, an animator does not need to spend hours defining key poses and expensive motion capture sessions can be kept at a minimum. In addition, our method can be used by someone who does not know the details about the motion he wants to create. Using a set of captured motions, for example downloaded from a database resource on the Internet, he can sketch the desired motion by keyframing some DOFs and then use this method to develop a enhanced version of it. 4.2 Related Work Considering a general animation, some work have addressed the problem of synthesizing motions or altering pre-existing motions to have a particular style. Chenney and Forsyth [CF00] used a markov chain monte carlo algorithm to sample multiple animations that satisfy constraints for the case of multi-body collisions of inanimate objects. Schodl et al. [SSSE00] used monte carlo techniques and developed the concept of a video texture, where an infinite amount of similar looking video sequences are generated from a starting short video clip. For character animation in particular, probabilistic methods were also used to represent the data. Bregler [Bre97] has used hidden markov models to recognize full body motions in video sequences. Brand and Hertzmann [BH00] have also used hidden markov models with an entropy minimization procedure to learn and synthesize motions with particular styles. A different approach was done by Chi at al. [DCB00], where they used the principles of laban movement analysis to create a method that allows the style enhancement of a pre-existing motion in an intuitive manner. Recently, some projects try to allow the creation of new character animations based on motion capture data. Li et al. [LWS02] divided the data into motion textons, where each of which could be modeled by a linear dynamic system. In this case the motions were synthesized considering the texton likelihood. Arikan and Forsyth [AF02] developed a method for automatic motion generation at interactive rates using a random search algorithm to find appropriate pieces of motion data. Kovar et al. [KGP02] defined the concept of a motion graph to enable the control of a character locomotion. This method allows a user to have high level control over the character motions. Lee et al. [LCR 02] presented a technique for controlling a character in real time using several possible interfaces. In this case the animations are created by searching through a motion database using a clustering algorithm. The methods described in this chapter are mainly based on the work of Pullen

CHAPTER 4. FRAGMENT BASED METHODS 27 and Bregler [PB02] [Pul02]. In these works they used signal processing techniques and a simple matching algorithm applied to data fragments in order to fetch missing degrees of freedom in a character animation using a motion capture database. Our approach described here tries to improve critical aspects of their work. We will demonstrate that the performance of the algorithm can be improved by using the skeleton hierarchy information to guide the search for good correlations between the joints. 4.3 Overview In order to use a fragment method, an animator should first create a sketch of the character motion by keyframing some degrees of freedom. Then, he should provide the following information: Which degrees of freedom should be used to drive the method. An animator can keyframe as many joints as he wants, but the method uses only some DOFs, which should contain the motion essence. For instance, considering a human walking, knee and hip joint motions are more important than toe and finger motions. These DOFs, considered more important for a specific action, should be set and are called driven joints. In addition, a particular driven joint should be chosen to drive the fragmentation step (see Sec. 4.5.1), being called the master joint. Which DOFs should be partially synthesized. Keyframed DOFs do not need to be completely synthesized since they already contain the motion essence. Small details in the motion, on the other hand, are not present. In this case, the details are extracted from the database and added to these DOFs in a process called texturing. Which DOFs should be completely synthesized. DOFs that are not keyframed are nevertheless important to correctly describe the desired character motion. The motion for these missing DOFs is completely generated in a process called synthesis. Which frequency bands are used for texturing. As mentioned in chapter 3, motion data can be treated as a signal over time, and therefore, it is possible to use the multiresolution method to decompose it into frequency

CHAPTER 4. FRAGMENT BASED METHODS 28 bands. Then, in the texturing process, only some bands are modified in order to generate the enhanced motion. Which frequency band is used to fragment the data. In order to compare driven joints and existing motions in the database, they need to be divided into small pieces, called fragments. Then, a particular frequency band from the master joint is used in the fragmentation step (see Sec. 4.5.1) and fragments are created taking into account its respective motion characteristics. How many matches should be kept during the matching step. Using the fragments generated, a matching step is performed where they are compared (see Sec. 4.5.2). Then, a certain number of candidates are stored and used to the synthesis and texturing. The main idea in a fragment based method is to consider correlation between joints during the character motion, and therefore, to generate an estimative of a joint motion based on the information of other joints. For instance, during a walking sequence the motions for arm, leg and hip joints tend to be correlated. Then given the arm and leg joint motions it is possible to estimate the hip motion. The method described here uses this fact. Character motion following the keyframe outline given by the animator is synthesized by concatenating small pieces of motion capture data from the database. The general algorithm is shown in figures 4.1 and 4.2. First, as shown in figure 4.1(a), keyframed and motion capture data are decomposed into frequency bands using the multiresolution method described in chapter 3. Then parameters for the method are set, as shown in figure 4.1(b), and the following four steps are performed: Fragmentation: One of the frequency bands of the master joint is chosen and the motion sequence is divided in fragments at specific points according to the motion behavior in that specific band (see Sec. 4.4.1). All existing examples in the database are fragmented at the same points in time. Thereby, driven fragments and data fragments are created depending on if the correspond joint is driven or not, as shown in figure 4.2(a). Matching: After generating data and driven fragments, each data fragment, at a particular sequence position, is compared with all driven fragments, in

CHAPTER 4. FRAGMENT BASED METHODS 29 (a) Keyframed and motion capture are decomposed into frequency bands (b) Animator set the method parameters Figure 4.1: The input for the general fragment based method: (a) keyframed and motion capture data are decomposed into frequency bands; (b) animators set the general method parameters. Driven and master joints are chosen to guide the method, and a particular frequency band of the master joint is chosen to guide the fragmentation step. Joints in black will be textured and in blue and red will be synthesized. all sequence positions, as shown in figure 4.2(b). The most similar data fragments for each sequence position are then stored, being called good fragments.

CHAPTER 4. FRAGMENT BASED METHODS 30 (a) Fragmentation: driven and data fragments are generated (b) Matching: data fragments are compared with driven fragments (c) Joining: best fragments are found for each sequence position (d) Enhanced character animation is generated by synthesis and texturing Figure 4.2: A fragment based method is composed by four steps: fragmentation (a), matching (b), joining (c) and smoothing. At the end, an original keyframed character animation is enhanced by synthesis and texturing (d).

CHAPTER 4. FRAGMENT BASED METHODS 31 Joining: After finding all good fragments for each sequence position, they are used to enhance the original keyframed animation. Using only the best fragments at each position, which are found based on heuristic evaluation criteria (see Sec. 4.5.3), they are concatenated and blended in order to compose the enhanced character animation, as shown in figure 4.2(c) and 4.2(d). Smoothing: The enhanced character animation can exhibit some jumping artifacts due to discontinuities where the best fragments were joined. In this case these discontinuities are reduced through a technique detailed in section 4.5.4. The result of the whole process is a character animation in which all DOFs are generated by texturing or synthesis. The final character animation presents the desired realistic appearance (examples can be found in Sec. 4.6). The rest of this chapter is structured as follows: Sec. 4.4 explains the fundamental basis used in the algorithm and Sec. 4.5 described the separate steps of the algorithm in detail. Then, in Sec. 4.6 results obtained with the general fragment method are described and its performance is demonstrated. After that, the general method is modified in order to better generate expressive motions and to capture more details from the database (Sec. 4.7). New experiments are made with this approach and it is shown that the performance improves (Sec. 4.7.3). The chapter concludes in Sec. 4.8 with a brief review and discussion. 4.4 Motion Analysis In order to develop a method where a character motion is created using examples from a database and following the guideline of some keyframed degrees of freedom, motion characteristics should be understood. In addition, it is important to find out which features of the original keyframed animation can be used to describe the details contained in its motion. Three different features are chosen to represent the motion characteristics: motion phases, frequency bands and correlation between joints. These features are described in the next sections. 4.4.1 Motion Phases As described in Pullen [Pul02], phase is considered a segment of time where a particular set of hard constraints are satisfied in a motion. Usually, the initial points

CHAPTER 4. FRAGMENT BASED METHODS 32 of these phases correspond to what traditional animators often use as keyframes in their animations. As seen in figure 4.3, for a walking cycle, there is a phase where the right foot is flat on the floor, another phase where the right heel lifts while the right toe stays on the floor, then the left heel touches the floor, and finally the left heel lifts while the left toe is touching the floor. (a) phases during a walking cycle 0.05 0 Values for the Z-angle of the Hips (rad) -0.05-0.1-0.15-0.2-0.25-0.3-0.35 0 5 10 15 20 25 30 Keyframes (time) (b) different phases in the motion signal Figure 4.3: Example of a walking animation: (a) set of four phases during a human walking cycle; (b) the right hip z-angle values is plotted, where it is possible to see the respective phases. The phases can be treated as a natural subdivision of the current motion being enhanced, and therefore, it is the ideal feature to be used in the fragmentation step to guide the generation of data and driven fragments (see Sec. 4.5.1).

CHAPTER 4. FRAGMENT BASED METHODS 33 4.4.2 Frequency Analysis All degrees of freedom of the human character are analyzed in the frequency domain, which simplifies the data format and allows the separation of different aspects of the motion. As mentioned in the Section 3.6, generally variations in the low frequency bands are associated with the overall motion, in the middle frequencies, with the motion signature and variations in the high frequency bands are perceived as jitter. These forms of fluctuation are present in real motion, and they should be preserved in order to capture all information from the original motion. 4.4.3 Correlation In human motion there are many correlations between joints during some actions. These correlations are especially clear for a repetitive action like walking. An example is shown in figure 4.3(a): as the right knee goes forward, the left arm swings forward, or when the right elbow angle has a certain value, the upper-arm angle is most likely to fall within a certain range. These correlations can be graphically seen in a plot such as that shown in figure 4.4. The fact that the plot has a specific shape (shown in red), indicates that there is a good relationship between the joints. Values for the Z-angle of the Knee (rad) -0.04-0.03-0.02-0.01 0 0.01 0.02 0.03 0.04 Values for the Z-angle of the Hips (rad) Figure 4.4: Plot of the pelvis joint angle against the hip joint angle for all examples in the database. The shape shown in red demonstrates a good correlation between these joints. Taking advantage of these joint relationships, it is possible to generate DOFs that have not been animated by using information from the keyframed joints. In addition, details can be added to a DOF that has been animated, by concatenating and blending the best fragments found from the analyze of the driven joints on selected frequency bands.

CHAPTER 4. FRAGMENT BASED METHODS 34 4.5 Motion Synthesis and Texture The general method starts with a sketch done by an animator. Suppose that an animator sketches out a human walking by animating only the legs and then he wants to generate the motion for the whole body. A good choice for the degrees of freedom to drive the method would be the hip z-angle and the knee z-angle, because the motion from both joints contain essential information about the walking action. Using the frequency analysis, keyframed and motion capture data are decomposed into frequency bands. For each DOF that has already been animated, only the middle frequency bands are created, leaving the overall motion intact. For each DOF that has not been animated, all frequency bands will be synthesized. The process to reach this goal consists of four main steps: fragmentation, matching, joining and smoothing. 4.5.1 Fragmentation In this step the driven and data fragments are generated. The first derivative of the master joint angle in the chosen frequency band is used in order to identify the motion phases. The time instants where the first derivative changes its sign is used to fragment the data [PB02], as shown for a particular DOF in figure 4.5. Using these same time instants, keyframed and motion capture DOFs in all frequency bands are broken into fragments, as shown in figure 4.2(a). Driven fragments are generated from the subdivision of the driven joints and data fragments are generated from the subdivision of all examples in the database. 0.03 0.02 0.01 Angle(rad) 0-0.01 Angle(rad) -0.02-0.03-0.04 0 20 40 60 80 100 120 140 160 Keyframe (time) 0 20 40 60 80 100 120 140 160 Keyframe (time) (a) original motion signal (b) fragmented motion signal Figure 4.5: Example of the fragmentation step: (a) original degree of freedom; (b) fragments created at locations where the first derivative changes its sign.

CHAPTER 4. FRAGMENT BASED METHODS 35 4.5.2 Matching After fragmenting all data, the matching step is performed. In this step data fragments are compared with driven fragments in order to find similar regions. The comparison is done in the frequency band chosen by the animator. Usually, one of the middle frequency bands is used since they contain the main motion characteristics. High frequencies do not reflect the overall motion and low frequencies contain a signal without a specific structure. The signal and its first derivative values are used to compare between driven and data fragments values. The first derivative is considered because it helps to choose fragments that are more closely matched not only in value but also in dynamics. The goal is to find, for each sequence position, a number of data fragments from the database which most closely matches the corresponding driven fragment. Before comparing the fragments, data fragments are stretched or compressed in time in order to have the same length as driven fragments by linearly re-sampling. To avoid creating fragments with non-natural poses, any data fragment that was originally larger or shorter than four times the driven fragment is not considered. For each sequence position, the comparison between driven and data fragments is done on the frequency band chosen by the animator. Each data fragment is compared to all driven fragments, in all sequence positions. Each comparison is assessed by a cost value which is stored. The cost value is a weighted sum of squared differences between driven and data fragment values, together with their first derivative values. In addition, in the cost value it is taken in account the relative importance of the different driven joints used, as stated in the following equation: (4.1) In the previous equation the indicates the cost to use the data fragment F at sequence position L. The value indicates the number of driven joints used and is the fragment length (i.e. the number of time frames at which the signal was sampled). The coefficient indicates the relative importance of each driven joint used and and the relative importance of the fragment values and their first derivative values in the summation. is the driven fragment value from DOF j at sequence position L and time frame i and is its respective first derivative value. is the data fragment F value from DOF j at time

CHAPTER 4. FRAGMENT BASED METHODS 36 frame i and is its respective first derivative value. After performing this comparison with all data fragments generated from the motion capture database, a number of closest matches for each sequence position is kept, according with their cost value, and the joining step is performed using these good fragments. Figure 4.6 reviews the whole matching step. Angle(rad) Angle(rad) 0 5 10 15 20 25 30 Keyframe (time) 0 5 10 15 20 25 30 35 Keyframe (time) (a) one driven fragment is shown (b) comparison of all data fragments with the driven fragment Angle(rad) Angle(rad) 0 5 10 15 20 25 30 Keyframe (time) 0 5 10 15 20 25 30 Keyframe (time) (c) all data fragments are compressed or stretched to have the same length of the driven fragment (d) the closest fragments are found Figure 4.6: Considering one driven fragment (a), in the matching step all data fragments are compared with the driven fragments (b) being stretched or compressed properly (c). At the end, a number of good fragments are found (d). 4.5.3 Joining As shown in figure 4.7(a), after the matching step a number of matching candidates (good fragments) are found for each sequence position. The next step is to find the best path through them in order to generate the complete sequence.

CHAPTER 4. FRAGMENT BASED METHODS 37 The joining step is based on a heuristic evaluation criteria. Three different criteria were tested and compared: best fragment, cost matrix and best animation. The first approach, the best fragment method, is only based on the final cost value of the good fragments, calculated in the matching step (see Sec. 4.5.2). Best fragments are considered good fragments that have the lowest cost value at each sequence position. As a result, this approach creates the final motion sequence joining all the best fragments, which corresponds to the best matches found, as shown in figure 4.7(b). Another approach tested, the cost matrix method, is based on the use of consecutive fragments, since in this case, the resulting animation is usually more pleasing, as stated in [PB02]. The idea is to consider the cost value together with the information about the fragment order in the motion capture sequences from the database. This approach considers the neighbors of each fragment, and it looks for sequences that maximize the use of consecutive fragments. For each link between fragments, a two-dimensional cost matrix is created, where the ith and jth components give the cost for joining good fragment i with good fragment j. A score of zero is given if both good fragments belong to the same animation and, in addition, they are consecutive in the original data. A score of one is given if they belong to the same animation but are not consecutive. Finally, a score of two is given if they do not belong to the same animation. The fragment sequence that is finally taken is the one which corresponds to a minimal cost path in the cost matrix. An example is shown in figure 4.7(c). Suppose we have four fragments positions to match, and we saved three good fragments. In the figure it is shown that for the fragment in the first position, the best matches are the fragments green-1, purple-3 and orange-1. For the fragment in the second position, the best matches are the fragments green-3, orange-2 and purple-1, and so on. The line draw indicates the path that produce the minimal cost. Therefore, this fragment sequence is used in the synthesis or texture of all DOFs. Generalizing the idea of using consecutive fragments, the best animation approach is also tested, where more emphasis is put on the use of consecutive fragments. In this approach, all matching fragments are taken from the sequence of the database that produces the best matching score, averaged over all time steps. This means that one animation is used to synthesize all time steps for one particular DOF, as shown in figure 4.7(d).

CHAPTER 4. FRAGMENT BASED METHODS 38 (a) good fragments to be joined (b) motion sequence is generated by joining the best fragment at each position (c) motion sequence is generated by the fragment sequence that minimizes the cost matrix (d) motion sequence is generated by joining fragments from one particular animation Figure 4.7: In the joining step the good fragments found in the matching step are concatenated or blended (a). Three different criteria were tested and compared: (b) best fragment; (c) cost matrix and (d) best animation.

CHAPTER 4. FRAGMENT BASED METHODS 39 4.5.4 Smoothing The fragments to be joined may not quite line up in cases where they were not consecutive in the original animation in the database. To smooth these discontinuities, the method described in Arikan and Forsyth [AF02] is used. Small discontinuities can be smoothed by distributing them using a window around the discontinuity. Then, the smoothing is achieved by multiplying the magnitude of the discontinuity with a smoothing function and adding the result back to the motion signal, as shown in figure 4.8. At the end, using this smoothing step, the discontinuities in the joined sequence are reduced. Figure 4.8: In the smoothing step, the discontinuity magnitude (top left) is multiplied with a smoothing function (top right) and the result is added back to the original motion signal. In this way, the continuous version shown on the bottom left is generated. Adapted from [AF02]. The smoothing function looks as follows: let d be the frame of the discontinuity and s the smoothing window size (for example 30 frames). Then, is a smoothing function that gives the amount of displacement that needs to be added to the original signal for every frame f. (4.2)

CHAPTER 4. FRAGMENT BASED METHODS 40 4.6 Experiments Using human walking as an example, the fragment method is evaluated using the three different joining approaches (see Sec. 4.5.3). Three different walking animations are used to verify the performance of the general method, each animation using a different set of keyframed joints. They have the same length, 160 frames, corresponding to four walking cycles. In the first example only the lower body joints for a human character are keyframed. Using the right hip, elbow, knee and foot z-angles as driven joints and the third frequency band of the right hip joint to drive the fragmentation step, five good matches are stored in the matching step for each fragment position. Upper body joints are synthesized and lower body joints textured. In the second example only the upper body joints for a human character are keyframed. A different set of driven joints is used: the right and left clavicle, the right upper-arm and the right forearm z-angles. The third frequency band of the right clavicle joint is used to drive the fragmentation step. Then, lower body joints are synthesized and upper body joints textured. In the last example only the joints for the left body side are keyframed. The driven joints used are the left upper-arm, elbow, hip and knee z-angles. The left hip is used as master joint. In this example, left side joints are textured and right side joints synthesized. Using the best fragment approach in the joining step, all three examples present considerable discontinuities in places where the best fragments are joined. After applying the smoothing step these discontinuities are reduced, but not totally since the initial discontinuities were large. As a result, the final animation exhibits some considerable jumping artifacts that are visually disturbing. This result shows that the use of originally consecutive fragments is important in the joining step, as mentioned in [PB02]. The best fragment approach fails because it analyses fragments only locally. It does not consider temporal coherence in the motion capture database. The cost matrix approach considers temporal coherence in the original data. With this approach better results are obtained for all three examples, since discontinuities are reduced using consecutive fragments in the joining step. Although the number of discontinuities decrease, they do not disappear totally as expected. Since animations in the database are created by repeating four times the same original animation, some data fragments created have the same cost values. Then, problems

CHAPTER 4. FRAGMENT BASED METHODS 41 during the path minimization occur, since many paths presenting the same final cost value are found. At the end, the final sequence contains some fragments that are originally not consecutive, and the character animation exhibits some jumping artifacts. The last joining approach, the best animation, is expected to solve these problems. In this case, all three examples present the expected performance: they are more pleasant, natural and without any jumping artifact. The use of the full existing motion sequence to describe a particular degree of freedom demonstrated two important characteristics: the fragments are already consecutive and they can be used to generate a degree of freedom without any loss of the motion information. Figure 4.9 shows the experiments as a graphical example. The sequence shown in black are the original DOF values and in red it is shown the sequence generated by the three joining approaches. As seen in figure 4.9(b) and 4.9(c), the final sequence generated by the best fragment and cost matrix approaches present considerable discontinuities in places where the best fragments are joined. On the other hand, in figure 4.9(d) there are not discontinuities in the sequence generated by the best animation approach. Using the best animation approach in the joining step, the final character animation shows a natural appearance. In the first example, where lower body joints were keyframed, the best animation criteria generates a character animation with an upper body motion that matches the initial keyframed motion. In addition, details in the lower body motion are increased. As a result, the final animation presents a natural and realistic appearance. Using the second example, where upper body joints were keyframed, the method generates a character animation with a lower body motion that corresponds to the initial keyframed motion, as well as improves the details in the keyframed upper body motion. Again, the final animation presents a natural appearance. For the last example, where left body joints were keyframed, the best animation approach also generates a final animation that presents a realistic appearance. The right body motion is correctly generated to match the initial keyframe motion. In addition, the motion details in the left body side are improved. 4.7 Hierarchical Fragment Method As demonstrated in the previous section, the general fragment based method is a valuable tool to create character animation. The final motion preserves the style

CHAPTER 4. FRAGMENT BASED METHODS 42 Angle(rad) Angle(rad) 0 20 40 60 80 100 120 140 160 Keyframe (time) 0 20 40 60 80 100 120 140 160 Keyframe (time) (a) good fragments are shown (b) sequence created by the best fragment approach Angle(rad) Angle(rad) 0 20 40 60 80 100 120 140 160 Keyframe (time) 0 20 40 60 80 100 120 140 160 Keyframe (time) (c) sequence created by the cost matrix approach (d) sequence created by the best animation approach Figure 4.9: Example of the joining approaches developed. In (a) all good fragments generated in the matching step for the z-angle of the pelvis joint are shown. The final sequence generated by the three approaches are then shown in (b), (c) and (d). of the initial keyframed animation and can be enhanced by the motion capture database to produce realistic motion for all joints of the body. In addition, motion details are added to increase the animation s realistic appearance. Although the general method presented a good performance, it can be improved. Currently, it cannot create animations that show all possible motion expressiveness and variations that are actually present in the database. In other words, the extrapolation power from the motion capture database is limited. For instance, using our most successful joining approach, the best animation, it is not possible to create a walking sequence showing all small differences in the left and right hand motion, as observed in a natural human walking. The motion created for the left

CHAPTER 4. FRAGMENT BASED METHODS 43 arm joint is the same as the right arm joint, since the fragments used to enhance the character animation come from the same existing motion capture sequence in the database, found by the analysis of just one master joint. In order to improve the general method performance, it is modified. A new algorithm is proposed, the hierarchical fragment method, where joint correlation information is used during the fragmentation and matching step. 4.7.1 Skeleton Hierarchy and Correlation In human motion there are many correlations between joints during normal movements. However, it is not the case that all joints are correlated in the same way. For instance, during a walking sequence one can expect a correlation between leg and knee joints, but not between leg and head joints. In general, joints that belong to the same kinematic sub-chain, like the hip, knee, leg and foot joints, will be correlated to each other since they work together to perform a particular action. However, the relationship between joints that do not belong to the same kinematic sub-chain (e.g. head, leg and spine) is difficult to be determined. Comparing the joint correlation in the motion capture database used in this project, joints that belong to the same kinematic sub-chain presented a good correlation, as expected. On the other hand, general correlations between joints, that do not belong to the same kinematic sub-chain, could not be found. 0.06 Values for the Z-angle of the Elbows (rad) 0.04 0.02 0-0.02-0.04 Values for the Z-angle of the Knee (rad) -0.06-0.04-0.03-0.02-0.01 0 0.01 0.02 0.03 0.04 Values for the Z-angle of the Hips (rad) -0.04-0.03-0.02-0.01 0 0.01 0.02 0.03 0.04 Values for the Z-angle of the Hips (rad) (a) the hip and elbow joints show a bad correlation (b) the hip and knee joints show a good correlation Figure 4.10: Example of possible joint correlations: (a) represents a bad correlation between joints that do not belong to the same kinematic sub-chain; (b) represents a good correlation between joints that belong to the same kinematic sub-chain.

CHAPTER 4. FRAGMENT BASED METHODS 44 Figure 4.10 shows a graphical example. In figure 4.10(a), two joints that do not belong to the same kinematic sub-chain are plotted. The figure is generated using the elbow joint angle against the hip joint angle for the human walking data. The lack of a defined shape demonstrates that there is no correlation between them. In figure 4.10(b), two joints that belong to the same kinematic sub-chain are plotted. Using the hip joint angle against the knee joint angle for the same data as previously, the figure is generated. Now, it is possible to see a defined shape, shown in red, which indicates a good correlation between these joints. In the method described in the following section, the skeleton hierarchy is used to find good correlations between the joints to be synthesized or textured. We will show that, in this way, the method s performance can be significantly improved. 4.7.2 Method The main goal of this new approach is to use the skeleton hierarchy information as a guideline to find good joint correlations. The method described here is an extension of the general fragment based method. Therefore, it uses the same four elements (fragmentation, matching, joining and smoothing), but in a different order. Starting by a sketch done by an animator, the method is divided in two stages: root generation and motion generation. In the first stage, root generation, all driven joints are analyzed and the corresponding skeleton root joint is found, which is the joint in the skeleton hierarchy that is the root of the kinematic sub-chains the driven joints lies on. The general idea is to successively use the information from a child joint to create motion for its parent joint. One example is shown in figure 4.11. Starting from the knee joint, first the hip joint is synthesized, using the four steps described in Sec. 4.5. Then, using the synthesized hip joint, the pelvis joint is synthesized, which is the root of the kinematic sub-chain. This process is performed iteratively for each driven joint and its parent joints until the root joint in the hierarchy is reached, which in our skeleton model is always the pelvis. Since the driven joints may lie on different kinematic sub-chains, they might produce different candidate motions for the root. One has to decide which root candidate will be used in the end. In order to find the final root joint sequence, the cost value from the matching step is used (see Sec. 4.5.2). For all driven joints, the cost value of the fragments at each sequence position is summed and the sequence that has the lowest cost value is considered the skeleton root joint,

CHAPTER 4. FRAGMENT BASED METHODS 45 being used in the next method stage. (a) initial configuration (b) first step (c) second step (d) final step: root generated Figure 4.11: Example of the root generation stage: (a) driven joints (red) and root joint (green) are shown; (b) the parents of the driven joints are generated; (c) next parent joint is generated and (d) root joint is generated. In the second stage, motion generation, the motion data generated for the root joint is propagated down in the skeleton hierarchy. The process is an iterative application of the principle of using the parent joint to create motion for subsequent children joints. As shown in figure 4.12, starting from the root (pelvis), first the hip and spine joints are created using the pelvis joint as master and driven joints. In each case the four steps are used to synthesize or texture the children joints (see Sec. 4.5). Once the children have been generated, they are used as master and driven joints to synthesize or texture their subsequent children. The additional benefit of this method is that it extends the expressive power of the general approach. By using the parent joints as master and driven joints

CHAPTER 4. FRAGMENT BASED METHODS 46 (a) the root joint generated (b) the root s children are generated (c) next children generated (d) all joints are generated Figure 4.12: Example of the motion generation stage: (a) root joint; (b) root joint together with the original driven joints are used to texture or synthesis of its children; (c) next children are generated; (d) all joints are generated. during the motion generation stage and by considering each kinematic sub-chain separately, it becomes possible to use parts of motion capture sequences during synthesis, that are very different. One can, for example, use a walking sequence to synthesize the lower body joints and data from a sequence that does not exhibit any lower body motion to synthesize the upper body joints. This would not been possible with the original fragment based method since it does not take in account knowledge about the skeleton hierarchy.

CHAPTER 4. FRAGMENT BASED METHODS 47 4.7.3 Experiments Using the same three human walking examples (see Sec. 4.6), the same parameters are set, as mentioned previously. The hierarchical fragment method is first applied to verify the performance of the joining approaches (see Sec. 4.5.3). Using the best fragment approach in the joining step, all three examples present an increasing number of discontinuities in places where the best fragments are joined. After applying the smoothing step these discontinuities are not totally reduced. As a result, the final animation exhibits many jumping artifacts that are disturbing and unpleasant. In comparison with the previous method, where the parent information is not taken in account, the number of discontinuities increased. This occurs because at each iteration a generated joint is used to generate its children joints. Then, discontinuities are propagated from the initial root joint, increasing at each iteration. In order to improve the hierarchical method performance, the cost matrix approach is used in the joining step. With this approach all three examples continue to exhibit jumping artifacts, generated by an increasing number of discontinuities during the joining step. Again, at each iteration, the number of discontinuities increased, being propagated from the parent to the children joints. As a result, the final animation is unpleasant and unrealistic. A better result is obtained using the best animation approach in the joining step. In this case, all three examples present the expected performance: they are pleasant and without jumping artifacts. In addition, the final motion is more expressive and natural than the resulting motions shown in section 4.6, presenting slight differences in the left and right arm motions, since to generate each joint the parent joint information is taken in account. Using the best animation approach, the final animation presents a natural appearance being enhanced by the motion capture database. In the first example, shown in figure 4.13, the method generates a character animation with an upper body motion that matches the initial keyframed motion and with increased details in the lower body motion. As a result, the final character animation presents a natural and realistic appearance. In the second example, shown in figure 4.14, the method generates a character animation with a lower body motion that corresponds to the initial keyframed motion, as well as improves the details in the keyframed upper body motion. Again, the final character animation presents a natural appearance. In the last example, shown in figure 4.15, the method also generates a final

CHAPTER 4. FRAGMENT BASED METHODS 48 character animation that presents a realistic appearance. The right body motion was correctly created to match the initial keyframed motion, presenting slight differences when compared with the left body side. In addition, the motion details in the left body side are improved. In general, the hierarchical fragment method presents the same good performance than the general fragment method. Comparing both methods, the hierarchical method s main advantage is the increase in the motion expressiveness, through the use of the skeleton hierarchy information. 4.8 Discussion In this chapter, two methods are presented combining the advantages of keyframing and motion capture. After creating an initial sketch of a character animation by keyframing, missing degrees of freedom and details are filled using the information from a database of motion capture files. Missing degrees of freedom are created by synthesis, and details are added to the keyframed DOFs by texturing. In both cases the same principle is used. The difference is that texturing only alters the middle frequency bands of the degree of freedom being enhanced. Both fragment methods have the same characteristics. They are computationally fast and present a good performance, being successfully applied to generate realistic character animation. The final motion preserves the style from the keyframed animation, which is enhanced by the motion capture database. In addition, motion details are added in order to increase its realistic appearance. As seen in the examples in the previous section, these methods can be used to improve the quality of a keyframed animation given by different initial joint configurations. They can correctly capture important aspects of the keyframed animation and allow the synthesis of an enhanced version of it. The major difference between them is that the hierarchical fragment method better extrapolates the examples in the database, resulting in an improvement in the character motion expressiveness. Using the parent joint, the motion essence from the original driven joints is preserved and the motion details of the particular joint being generated are included. In addition, this method can be applied to a more general database, containing more than one specific action, since it is possible to consider only the data from a particular kinematic sub-chain, independently of the rest of the data. Although both methods present a good performance, they have some disad-

CHAPTER 4. FRAGMENT BASED METHODS 49 vantages. These methods present a limited extrapolation capability. The database here plays an important role and the resulting character animation will be based on the data contained there. In our case, since the database does not present considerable motion variation for the joints belonging to the lower kinematic sub-chain, the resulting motion for these joints could not exhibit expressive motion variations. Another problem is related with the parameters to be set. Since the methods are not fully-automatic, the animator has control over the whole process and his choices guide the process development. However, sometimes the animator wants a simple way to control the whole process. In this case, the parameters in these methods can be quite hard to be controlled by animators used to work with keyframing. Small changes in the parameters can generate completely different animations. It is our goal to develop a method that is able to create realistic character animations, extrapolating new motions from the examples in the database and allowing the animator to have the control during the creating process, in a simple and efficient way, using intuitive aspects. Then, in order to improve some drawbacks of the fragment based methods, a new method is investigated where a principal component model is fitted to the motion capture database, as it is described in detail in chapter 5.

CHAPTER 4. FRAGMENT BASED METHODS 50 (a) initial character animation where only lower body joints are keyframed (b) enhanced animation where upper body joints are synthesized and lower body joints textured Figure 4.13: Example of the Hierarchical Fragment method being applied to a human character: (a) shows some frames from the initial character animation where only lower body joints are keyframed; (b) shows the respective frames from the resulting character animation where upper body joints are synthesized and lower body joints are textured.

CHAPTER 4. FRAGMENT BASED METHODS 51 (a) initial character animation where only upper body joints are keyframed (b) enhanced animation where lower body joints are synthesized and upper body joints textured Figure 4.14: Another example of the final method being applied to a human character: (a) shows some frames from the initial character animation where only upper body joints are keyframed; (b) shows the respective frames from the resulting character animation where lower body joints are synthesized and upper body joints are textured.

CHAPTER 4. FRAGMENT BASED METHODS 52 (a) initial character animation where all left body side joints are keyframed (b) enhanced character animation where all joints are synthesized Figure 4.15: Another example of the final method being applied to a human character: (a) shows some frames from the initial character animation where all left body side joints are keyframed; (b) shows the respective frames from the resulting character animation where all joints are synthesized.

Chapter 5 Principal Component Analysis 5.1 Introduction In chapter 3, the multiresolution filtering technique was investigated and successful applied to character animation, allowing some editing capabilities. Although the method works fine, it can be used only in a limited range of applications. For instance, it is not possible to edit a character animation if some degrees of freedom are missing. In chapter 4, fragment based methods are investigated and successfully applied in order to create a realistic character animation, starting from a keyframed animation, by synthesis and texturing. The main problem is that this method has a limited extrapolation capability. In this case, the resulting motion is strongly dependent of the examples contained in the database. In this chapter, a new method for creating character animation from a database is investigated enabling a better extrapolation from the database. Fitting a parametric model to the motions in the database, it is possible to create a description of the space of motions, which allows motion extrapolation. Among the different forms of parametric models that could be used, the Principal Component Analysis model is chose. PCA is a technique for reducing the dimensionality of data, by finding some linear transformation of the co-ordinate system. The transformed coordinate system is aligned to the data such that the variation along the new axis is suitably small. In this case, only the axis which describe a lot of variation are preserved, the others are canceled. Principal Component Analysis focuses on correlations between data instances. The main idea is to reduce the dimensionality of a data set in which there are a large number of interrelated variables, while retaining as much as possible of the 53

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 54 variation present in it. Since there are many correlations present in the motion data, PCA can perform feature extraction by removing redundancies in order to better represent the data. Using PCA, we aim at developing a method that show the same functionality present in the fragment based methods but with an increase in the overall performance due to the data extrapolation. In particular, using PCA we expect to derive parametric models for particular joint motions which we can use to interactively edit them, after the motion has been created. 5.2 Related Work Principal Component Analysis has been used in many and different fields for decades. Recently, the main areas in computer animation are recognition and classification, data representation and data compression. Applications in these fields can be seen respectively in the work of Blanz and Vetter [BV02], Bregler et al. [BLCD02] and Alexa and Müller [AM00]. In motion analysis, synthesis, classification and recognition, Principal Component Analysis is also extensively used, since it can identify particular characteristics of the motion data. Bevilacqua et al. [BRC02] used PCA to perform motion analysis in order to create animation from capture dance data, generating in parallel the sound-track to match the dancer s motion. Ormoneit et al. presented a method for modeling [OSBH01], learning and tracking [OSB 01] human video motion from video sequences using a large set of 3D periodic human motion data, where PCA was used to perform data analysis. Considering synthesis applications, Vasilescu [AV02] used PCA to extract motion elements and recombine them in novel ways. Blanz and Vetter [BV99] used a PCA model in order to create a morphable face model from a set of 3D face models by transforming the shape and texture of the examples into a vector space representation. Using PCA in order to perform motion classification, Troje [Tro02] described a two-mode PCA framework to linearly classify male and female walkers. Davis and Gao [JWD03] presented a framework that uses PCA to factorize action examples in order to label the effort of an action, corresponding to the perceived level of exertion by the performer. PCA is also used to recognize gestures and classify motion primitives. Jenkins and Matarić [JM02] presented a data-driven method for deriving perceptual-motor

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 55 action and behavior primitives from human motion capture data. Fod et al. [AFJ02] described a method for representing human arm motion compactly, in terms of a linear super-imposition of simpler movements termed primitives, derived from PCA. The work described in this chapter is more related to the work of Pullen [Pul02], where the PCA model is used to enhance an initial keyframed animation. Our approach described here tries to improve critical aspects of her work. It is demonstrated that choosing specific joints to be used in the PCA model, for instance taking into account skeleton hierarchy information, the performance of the method can be improved. In addition, it is shown that the PCA method can be used for editing motion through the use of the motion principle components. 5.3 Principal Component Analysis 5.3.1 Overview Principal Component Analysis is a technique for reducing the dimensionality of data by finding some linear transformation such that the variation of the data along some of the new dimensions is suitably small. If this is the case, the dimension which represent only small variation can be ignored. Analyzing the data, PCA first finds a direction along which the data varies as much as possible. Having such a direction, it finds another direction, orthogonal to the first, along which the data varies as much as possible. Then it finds a third direction, orthogonal to the first two and so on. At the end, having all dimensions, in other words all basis vectors, each initial vector in this data space can be reconstructed as a linear combination of these basis vectors. In addition, by ignoring the dimensions corresponding to suitably small variations, it is possible to compress the data. 5.3.2 PCA Theory The principle of PCA is as follows. Given a set of M data vectors, each of size N:

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 56 The first step in the PCA computation is to calculate the mean vector by averaging each co-ordinate separately: (5.1) Then, subtracting all initial vectors from the mean vector, the data matrix is created: (5.2) Each row of this matrix is a vector representing the difference between an original vector and the mean of them all. The next step is to construct the covariance matrix, which is a matrix defined as. The element values ( ) of this matrix are one measure of how well the i-th and j-th co-ordinates of all the points agree with respect to their mean values. In this case, if is positive, both dimensions increase together, if it is negative, they increase in inverse proportion. If the value is zero, they are independent. Since this correlation matrix C is symmetric, and real-valued, it has n eigenvectors and eigenvalues, where the eigenvectors are mutually orthogonal with respect to each other and, therefore, they can be used as a vector basis to this dimensional space [GHG89]. A simple way to calculate the eigenvectors and eigenvalues is by applying the SVD decomposition directly to the data matrix M, as described in Blanz and Vetter [Bla00], without creating the correlation matrix C. Using SVD decomposition, the data matrix M can be factorized as follows: (5.3) The columns of V are made from the eigenvectors of, the columns of U are made from the eigenvectors of and the eigenvalues of and are the square-rooted values of the columns of matrix S. By applying the SVD algorithm described in Press et al. [PFTV92] directly to the data matrix M, the eigenvectors are found from the matrix V and the eigenvalues from the matrix S, and thereby, the PCA model for the set of examples is created. The PCA model consists of the mean vector, eigenvectors and eigenvalues.

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 57 Using this PCA model, any original example can be reconstructed. Each original example in this data space can be described as a sum of the mean vector and a linear combination of the eigenvectors, as stated below: (5.4) The coefficients correspond to the dot product between the initial vector and the normalized eigenvector, as shown in eq. 5.5. They indicate the influence of each dimension (eigenvector) in order to construct an original example. (5.5) 5.3.3 Data Compression Using PCA it is possible to reduce the dimensionality of data sets. In a PCA model, the eigenvector corresponding to the largest eigenvalue is the direction along which the original points have their largest variance. The eigenvector corresponding to the second largest eigenvalue is the orthogonal direction along which the points have their second largest variation, and so on. Therefore, eigenvectors taken in the order of size of their eigenvalues are the most important dimensions, which can be used to describe the information contained in the data. Accordingly, data reduction and compression can be achieved by ignoring those eigenvectors corresponding to suitably small eigenvalues. Using only a limited number of dimensions to perform data compression, for example only the K most important, it is possible to write eq. 5.4 as: (5.6) Using this procedure, the approximation error can be controlled by the number of considered dimensions. Having a PCA model of motion capture data, it is possible to express motion cycles in terms of their most important dimensions, the principle components. The weights used during the reconstruction of the motion can be assigned higher level meanings and it can be changed in intuitive ways, such as in Blanz and Vetter [BV99]. This way, the principal components of human motion can be considered higher-level features which allow synthesis and editing of motion on a higher semantic level.

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 58 In the next section it is described how the PCA model is applied in the context of motion synthesis. 5.4 PCA for motion synthesis 5.4.1 Overview Our first goal is to use the PCA model to improve a given initial keyframed animation, where only few degrees of freedom were keyframed. In order to achieve this we need to fit a PCA model to the data and find the coefficients for this model which best reproduce the motion of the keyframed DOFs. Then, these coefficients can now be used to synthesize motion for the missing DOFs, if we also have a PCA model for them. There are different possible ways to achieve this. The first possible way would be to fit a simple PCA model to all DOFs of a motion capture database and to find the coefficients for the model that produce the lowest approximation error for all keyframed DOFs. However, as stated in Pullen [Pul02], the results with this approach are not satisfactory since the information contained in the keyframed DOFs is used in the same way to synthesize all remaining DOFs. PCA is a linear model, and as a result, it might not perfectly represent the motion data, which is presumably highly non-linear. The question to be answered is how we can properly use a linear model to represent non-linear data. In our case, instead of trying to synthesize all DOFs at once, we use the information contained in the keyframed DOFs to synthesize one DOF at a time. In other words, PCA models are learned for each missing DOF using the motion data from the database and from the keyframed DOFs. Learning many lower-dimensional PCA models containing only few DOFs, it is possible to represent the whole space of motions more accurately. In addition, each PCA model is specialized to a particular DOF and the reconstructed error is minimized. In this way, using a linear model the non-linearities are sufficiently well represented. 5.4.2 Motion Synthesis Starting by an initial animation where few DOFs were keyframed, for instance, an animator keyframes K DOFs out of N, a PCA model is created for each missing DOF using the database and the keyframed data. A set of data vectors is built for each missing DOF containing the keyframed DOFs, the parent DOF and

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 59 the missing DOF values at time step t for each animation a (keyframed and all examples in the database), as shown below: For each, the respective data matrix is built according to eq. 5.2. After creating the matrix, the PCA model is computed according to the procedure described in section 5.3. At the end, the respective eigenvectors and eigenvalues are found. For each PCA model, only a certain number of eigenvectors are kept. For instance, it is chose only K=4 eigenvectors to reconstruct a missing DOF, the same number of keyframed DOFs. Using these K eigenvectors it is possible to generate the values for the missing at each time point t using the PCA model, as stated in eq 5.7. In order to allow the user interactively control the influence of each principal component in the reconstruction process, weights are used to change the effect that each component will have on the reconstructed DOF. The coefficients correspond to the dot product between the initial vector and the normalized eigenvector, as shown in eq. 5.5. Then, the reconstruction process is done as a weighted sum, as stated in next equation: (5.7) In this case, after creating a character animation, an animator can interactively change it, modifying the influence that each principle component has in the synthesis of a particular degree of freedom. This way the high-non-linear space of motions is represented by a set of lower dimensional PCA models. 5.5 Experiments Three different human walking sequences are used to verify the general performance of the method. Each walking sequence uses a different set of keyframed DOFs, as mentioned in section 4.6. First, using directly the method described in section 5.4.2, data vectors from the keyframed animation and the motion capture files are created. An animator keyframes four DOFs (K=4) and all 62 missing DOFs are synthesized, one at a time. An example of a possible PCA model being created is shown in figure 5.1(b).

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 60 Keyframing the knee, foot, upper-arm and forearm joints, a PCA model for the hip joint is created using all DOFs shown in red (keyframed DOFs, pelvis and hip), taking in account all animation examples (keyframed and database). Constructing 62 PCA models, one for each missing DOF, the method fails to produce good results. The motion in all three cases are small in amplitude and vague, as if they had been averaged from the rest of the motion capture database. Since the simplification in the PCA model is too big, it is not possible to capture the motion details. In order to improve the performance, the Multiresolution Filtering method, described in chapter 3, is used to decompose the values of all DOFs into six frequency bands. For each frequency band, the PCA method described in section 5.4.2 is applied separately. Thereby, increasing the number of PCA models created. An example is shown in figure 5.1(c) where the information in a particular frequency band from all animation examples is used to created one of the six PCA models for synthesis of the hip joint. Keyframing four DOFs (K=4), the PCA method is applied to all 62 missing DOFs, one at a time. One PCA model is created for each frequency band and the reconstruction is done for each frequency band separately. 372 PCA models are created and the reconstructed motion from the six frequency bands are summed in order to created the final motion for a particular missing DOF. The created motion captures the essence and the details of the motion capture database, based in the keyframed DOFs information. As a result, it is not sufficient to learn separate PCA models for each DOF to be synthesized. The information from different frequency bands plays an important role and needs to be modeled separately. Using the idea of motion phases from section 4.4.1, a new approach is investigated, by which PCA is applied over local portions of data (fragments). Combining the modeling power on a higher semantic level given by PCA with the modeling power on a lower semantic level given by the fragments, we expect to correctly capture and synthesize all important details present in a motion. Applying the fragmentation method described in section 4.5.1, keyframed and motion capture DOFs are fragmented. For each frequency band of a particular missing DOF, fragments are created from the keyframed animation and from the files in the database. Then, in order to capture local information, a PCA model is built for each sequence position according to the method described in section 5.4.2. A graphical example of this approach is shown in figure 5.1(d), where a PCA model is created for a specific sequence position in one particular frequency band of the hip joint.

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 61 Keyframing four DOFs (K=4), a PCA model is fitted to all generated fragments at each sequence position, in a particular frequency band. After performing this process for all frequency bands, the results obtained from the six frequency bands are summed and the fragments are joined in order to create the final motion sequence for each missing DOF. Creating more than 20 PCA models to represent a particular missing DOF, it is really possible to capture much of the motion detail contained in the database. However, there are often large discontinuities in places where the fragments are joined, since each PCA model is created using only local information (fragments). These large discontinuities could not be eliminated with the smoothing technique described in section 4.5.4. As a result, the final motions created by the PCA models exhibit many jumping artifacts, originated from the transition gaps between the mean poses generated by the PCA models. Therefore, although the last approach can capture much more information about the motion, the approach considering only six PCA models to represent each missing DOF is found to be more useful, since it takes in account the full animation and generates more natural and pleasant results. Applying this approach to all three human walking sequences (see Sec. 4.6), the method s capacity to generate a realistic motion from different initial keyframed joint configurations is verified. In the first example, only lower body joints are keyframed. In this case the pelvis, right hip and knee and left hip are used as keyframed DOFs and all missing DOFs are synthesized. As a result, joints belonging to the lower kinematic subchains are correctly generated matching the keyframed motion. However, upper body joints cannot be correctly synthesized by the PCA method, presenting a vague motion which do not match the general human motion. Results for the second example, where upper body joints are keyframed, present the same problem. Using the right forearm, upper-arm, elbow and left elbow as keyframed DOFs, the PCA method is applied in the same way as before. Again, the PCA method can correctly synthesize motion for the joints belonging to the upper kinematic sub-chains. However, for the remaining joints (lower body), it cannot produce satisfactory results. Using joint information from only one kinematic sub-chain (upper or lower body), the PCA method cannot synthesize joints that belong to a different kinematic sub-chain. Creating data vectors with DOFs that are not correlated, the resulting motion becomes vague and small in amplitude. In general, the animation seems to be paused in a particular state, which is the mean pose for all animations

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 62 in the database. The results for the last example show an improvement in the performance of the method. Using the left hip, knee, elbow and upper-arm as keyframed DOFs, the method is applied in the same way. As seen in figure 5.2, in this case the motion generated presents the expected characteristics for all joints. Keyframing joints that belong to both main kinematics sub-chains, the resulting motion is improved. The motions are correctly extrapolated from the database examples and generated in order to match the keyframed DOFs. However, although the character performs the desired motion, it do not look realistic. The PCA model cannot synthesize all motion details as observed in real human motion, which makes the resulting motion unnatural. Although the general results for the PCA method do not generate realistic motion, the principle components show interesting properties. Using the synthesis process for the previous example, the motion principle components are interactively changed in order to verify their effects in the resulting motion. Modifying the influence of each principle component, an animator can edit the motion by adding or subtracting a constant value to each DOF, altering its low frequency band values. For instance, it is possible to alter the character s arm and leg position during an walking cycle without losing the general appearance of the animation, as shown in figure 5.3, 5.6 Discussion In this chapter, another method combining advantages of keyframing and motion capture is presented. An animator creates an initial sketch of an animation by keyframing some DOFs, and then, missing degrees of freedom are synthesized using the motion capture database. The Principal Component Analysis is used to extract information from a motion capture database by fitting a parametric model to the data in order to enhance the initial animation sketch. In general PCA cannot be used to improve the quality of a keyframed animation where some DOF are missing. Creating PCA models using keyframed DOFs from only one main kinematic sub-chain, it is not possible to correctly generate the motion for the joints belonging to a different kinematic sub-chain. Using keyframed DOFs from both main kinematic sub-chains, the motions for all missing DOFs are generated, however they do not look realistic. PCA cannot capture some important middle and high frequency details as observed in real human motion, which

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 63 makes the resulting motion unrealistic. As a result, final motions are small in amplitude and vague, being averaged from the rest of the database. Another problem is the increase in the computational time, compared with the Hierarchical Fragment method, since many PCA models should be calculated for the synthesis of a missing DOF. Although motions do not look natural, PCA can extrapolate new motions from the database examples, creating a correct mean pose for a character, which can be used as low frequency information. In addition, using the motion principle components, an animator can have control of the motion after it has been created. Using weights to modify the effects of the principle components, an animator can edit the motion, adding or subtracting a constant value to any degree of freedom, altering the low frequency bands. In this case, it is possible to alter the motion without loss of its human appearance. It is our intent to develop a method that is able to create realistic character animations, extrapolating the database examples and allowing the animator to have control during the whole creating process, in a simple and efficient way, using intuitive animation aspects. Separately, the three methods described in this thesis, multiresolution filtering, hierarchical fragment method and PCA, cannot integrally achieve this goal. However, since each of them has a desired characteristic, a new approach is explored using the three methods together in a pipeline solution, where the strengths of each individual method is combined to form a more powerful method, as detailed in chapter 6.

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 64 (a) Keyframed animation and motion capture data (b) A PCA model is created for the hip joint using all motion examples, shown in red (c) A PCA model is created for one particular frequency band of the hip joint, shown in green (d) A PCA model is created for a specific sequence position in one particular frequency band of the hip joint, shown in green Figure 5.1: Three approaches are investigated in order to fit a PCA model to a keyframed and motion capture data (a). In (b) one PCA model is created for each missing DOF. In (c) one PCA model is created for each frequency band of a missing DOF. In (d) one PCA model is created for each sequence position of a particular frequency band of a missing DOF.

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 65 (a) initial keyframed animation (b) improved resulting animation showing the performance of the PCA method Figure 5.2: Applying the PCA method to a human character: (a) shows some frames from the initial animation where the left hip, knee, elbow and upper-arm joints are keyframed; (b) shows the respective frames of the resulting animation where new motion for the joints are generated from the database in order to match the keyframed DOFs.

CHAPTER 5. PRINCIPAL COMPONENT ANALYSIS 66 (a) arm positions are modified by using the clavicle principle components (b) leg positions are modified by using the knee principle components Figure 5.3: Use of the motion principle components in order to edit a motion: (a) arm positions are modified by altering the principle components for the right and left clavicle joints; (b) leg positions are also modified by altering the principle components for the right and left knee joints.

Chapter 6 Hybrid Approach 6.1 Introduction In the previous chapters, several methods were described in order to generate character animation. Each of these methods has its advantages, however, when used separately they cannot fully satisfy our main goal, which is to use the strengths of Keyframing and Motion Capture to simplify the process of creating character animations. Multiresolution Filtering can be used to edit a motion considering its frequency bands, but it requires an input animation with all DOFs synthesized. Hierarchical Fragment method can correctly synthesize a realistic character animation from an initial keyframed sketch, however it cannot extrapolate a new motion from the examples in a database and it presents many control parameters. Principal Component method can extrapolate new motions from the examples in a database and it can also be used for editing purposes, however the resulting motion is not always realistic. Since each of them has a desired characteristic, a new approach is investigated by using the three methods together in a pipeline solution, in order to simplify the creation of a realistic character animation. Combining the strengths of all three methods, each method is used at a specific time and its benefits are used to improve the character animation, as described in the next sections. 67

CHAPTER 6. HYBRID APPROACH 68 6.2 Hybrid Approach To use the three methods together in one pipeline, their advantages and disadvantages should be compared in order to be applied the right method at a time. The first method discussed, Multiresolution Filtering, is not a method for creation of a character animation. However it presents some interesting editing capabilities to existing animations, being useful to alter the motion signature contained in a particular frequency band. The second method, Hierarchical Fragment method, is able to create a realistic character animation that matches an initial keyframed sketch. In addition, control parameters for this method can be simplified by using default parameters, in case where the initial keyframed animation presents many DOFs synthesized. On the other hand, the extrapolation capability is limited. The last method, Principle Component method, can improve an initial keyframed sketch by creating a mean pose, where low frequency information is present. Starting from the mean pose, we can create a new motion that extrapolates from the motions present in the database. In addition, the motion principle components provide editing capabilities by adding or subtracting a constant value to any DOF. Thereby, changing its low frequency information. However, it cannot always generate a realistic character animation and, depending of the initial keyframed DOFs, it cannot correctly generate the motion for all other degrees of freedom. Considering the characteristics of the three methods, we propose the following pipeline, shown in figure 6.1, which we expect to achieve our main goal. KEYFRAME ANIM. PCA MULT. FILT. HIER. FRAG FINAL ANIM. MOTION CAPTURE DATABASE Figure 6.1: Pipeline showing the Hybrid approach The different steps of the method are as follows. Starting by the initial sketch,

CHAPTER 6. HYBRID APPROACH 69 where only a small number of degrees of freedom are keyframed, the Principal Component method is applied. It is the best method to be used in the first stage since it can create new motions that extrapolate from the example in the database and it provides some editing capabilities through the use of the motion principal components. As a result, the motion generated in the first stage presents low frequency information for all DOFS, which can be interactively modified by changing the principal components. The motion presents information for all DOFs. Nevertheless, the resulting motion might not be realistic. Now, the Multiresolution Filtering method can be applied in order to modify a particular frequency band, for instance to give more expressiveness to the character animation. In a final step, the Hierarchical Fragment method is applied in order to improve the motion realistic appearance for the character. Using the result of the previous stages, instead of using the original keyframed sketch, missing DOFs can be textured and driven joints can be set in advance according with the motion capture database being used. The hybrid approach is a semi-automatic method. At each stage an animator can change the control parameters if he wants. In the first stage the influence of the motion components can be modified, in the second stage frequency bands for the motion can be altered, and in the last stage the driven joints can be changed. However, this form of interaction is not compulsory. The same keyframed DOFs from the initial sketch can be used in the PCA method as keyframed DOFs and in the last stage as driven joints. At the end, the resulting character motion is generated automatically, matching the initial keyframed sketch. 6.3 Experiments Using a walking sequence as an example, the hybrid method is applied to a human character (see Sec. 2.5.2) with 66 degrees of freedom. Considering an initial keyframed animation, where joints in the left body side are keyframed, the pipeline described in the previous section is applied in order to verify the method capacity to create a realistic character animation in a simple and intuitive way. The PCA method is applied first in order to synthesize low frequency information for all missing DOFs of an animation in which the left hip, knee, elbow and upper-arm are keyframed. After that, principle components are used to modify specific joint configurations, like arm and leg positions, by adding a constant value

CHAPTER 6. HYBRID APPROACH 70 to their DOFs. As seen in figure 6.2, PCA creates an initial motion for the nonkeyframed DOFs and allows the edition of an animation after it has been created. After that, the Multiresolution Filtering method is applied to the animation resulting from stage one. Decreasing the gain of the low frequency bands for all DOFs, figure 6.3 shows the resulting animation, where a smooth acceleration of the movement can be seen. At the end, the Hierarchical Fragment method is applied. Using the method s default parameters, the third frequency band is set for matching, since it can capture low and middle frequency information. As driven joints, it is found that the use of at least a pair of joints from each main kinematic sub-chain produces better results. Then, it is chosen the left hip, knee, elbow and upper-arm z-angle as driven joints. Since all DOFs have the low frequency information, they are textured. As a result, a character animation is generated showing an improved motion quality, as seen in figure 6.4. Since the hybrid approach is based in the previous three methods, the computation time increases, especially due to the PCA stage, and hard constraints are not incorporated. As a result, it cannot be applied directly to cases where the feet are meant to remain in contact with the floor, unless it is combined with an inverse kinematic solver. However, the hybrid approach proved to be useful to improve the quality of an initial animation, where only a small number of DOFs are keyframed. The method allows the automatic synthesis of a realistic character animation, matching the keyframed sketch, as well as allows motion editing at each step of the pipeline. Figure 6.5 shows two different resulting character animations from the same initial keyframed sketch, where modifications of the parameters in the PCA and Multiresolution methods are performed. 6.4 Discussion In this chapter an hybrid approach is presented, which uses the three methods described in chapters 3, 4 and 5 in order to develop a simple and intuitive method to create character animations. The sequence used in the pipeline can effectively combine the advantage of each individual method, since it is taken in account the advantages and drawbacks of each method. For instance, the PCA is applied first to create a new motion from the examples in the database and to be used as an editing tool. Then, the Multiresolution Filtering method can be used to alter a particular frequency band.

CHAPTER 6. HYBRID APPROACH 71 At the end, the Hierarchical Fragment method is applied in order to improve the animation realistic appearance, maintaining the essence of the character animation intact. As seen in the examples in the previous section, this method can be used to improve the quality of a keyframed animation, since each method acts in a specific way improving different aspects of the character animation. The database plays an important role, since PCA and Hierarchical Fragment methods extract information from it. In addition, the animator s choice can guide the method during all stages. Although the controllability increases, the method becomes simple and intuitive. During each stage an animator can interactively modify the motion by using the principle components (stage one), altering the frequency bands (stage two) or changing the control parameters (stage three). However, this procedure is not obligatory. The method allows the automatic synthesis of a realistic character animation, matching the keyframed sketch. Since the input for the Hierarchical Fragment method is an animation that has basic information for all DOFs, an animator does not need to worry about control parameters. In this case, default control parameters can be used, which have given good results. Although the Hybrid approach presents some problems with hard constraints, it satisfies our main goal, fulfilling our original requirements best. It combines the strengths of Keyframing and Motion Capture in a simple and intuitive way, in order to create character animations by using a pipeline that at each stage improves the initial animation in a particular manner.

CHAPTER 6. HYBRID APPROACH 72 (a) initial keyframed animation (b) resulting animation after applying the PCA method (c) using the principle components to change the arm position (d) using the principle components to change the leg position Figure 6.2: Example of the first stage: (a) shows some frames from the initial animation where left body side joints are keyframed; (b) shows the respective frames after applying the PCA method; (c) and (d) show possible editing capabilities of the PCA, where the influence of the principle components for some DOFs are modified.

CHAPTER 6. HYBRID APPROACH 73 (a) initial keyframed animation (b) resulting animation after applying the PCA method (c) using Multiresolution to generate a smooth acceleration of the movement Figure 6.3: Example of the second stage: (a) shows some frames from the initial animation where left body side joints are keyframed; (b) shows the respective frames after applying the first stage (PCA); (c) shows the same frames after applying the Multiresolution Filtering method to decrease the low frequency bands, generating a smooth acceleration of the movement.

CHAPTER 6. HYBRID APPROACH 74 (a) initial keyframed animation (b) animation after applying the PCA method (c) animation after applying the Multiresolution Filtering method (d) using the Hierarchical Fragment method to improve the realistic appearance of the animation Figure 6.4: Example of the last stage: (a) shows some frames from the initial animation where left body side joints are keyframed; (b) shows the respective frames after applying the first stage; (c) shows the frames after applying the second stage; (d) shows the same frames after applying the Hierarchical Fragment Method in order to improve its realistic appearance.

CHAPTER 6. HYBRID APPROACH 75 (a) initial keyframed animation (b) final enhanced character animation generated automatically (c) final enhanced character animation where the arm positions are changed (d) final enhanced character animation where the leg positions are changed Figure 6.5: Example of the Hybrid approach: (a) shows some frames from the initial animation where left body side joints are keyframed; (b) shows the respective frames from the animation generated automatically by the hybrid method; (c) and (d) show the same frames from the resulting animations generated when motion principle components are used to change arm and leg positions by adding or subtracting constant values.

CHAPTER 6. HYBRID APPROACH 76

Chapter 7 Conclusion and Future Work With the advent of photo-realism in Computer Graphics, life-like character animations that capture fine details and subtleties of a motion have become more important. Motion is a fundamental visual cue in human perception and slight inaccuracies are directly noticed. Hence, even if the rendering is perfect, the motion will still tell you that it is an artificial scene. Animators at a studio usually create animations for photo-realistic characters using keyframing. However, even for professional animators, the job of creating life-like animations for a character is challenging. Achieving realistic character animations by keyframing is labor intensive and time-consuming. One solution to this problem is to use motion capture data, since in them details are immediately present. However, motion capture data does not allow much flexibility, since the motions are fixed from the outset and it is hard to alter them afterwards. In addition, motion capture sessions are still cumbersome and expensive. The main goal of the work in this thesis is to combine the strengths of Keyframing and Motion Capture. Keyframing is ideal to quickly sketch a motion, but it is difficult to model all the details of life-like character animations. On the other hand, Motion Capture preserves all details of a motion, but it is not flexible. By combining them, we aim at designing a method that can simplify the process of creating realistic character animation, allowing an animator to have a high level of control over the whole process in a simple and intuitive way. In order to make that possible we needed to examine ways how to analyze the motions in the database and how to synthesize new motions. The features we have found more important to describe the motion characteris- 77

CHAPTER 7. CONCLUSION AND FUTURE WORK 78 tics are the frequency decomposition, skeleton hierarchy, joints correlation, motion phases and motion principle components. Frequency analysis, used for all methods described in this thesis, proved to be useful for giving control over different aspects of a motion. Low frequency bands describe the general motion, and are often what an animator creates as a first sketch. Middle frequency bands usually contain the motion signature, which is often the details and personality that an animator is interested in. Finally, high frequency bands present other details and jitters, which are important to characterize the motion. The use of correlation among joints is an important point of the methods we have developed. Considering correlation among joints, it is possible to generate an estimative of a joint motion, based in the information of other joints. These relationships are used to guide the synthesis of missing DOFs in an accurate way, respecting the true human movement. In addition, we have found a confirmation of the assumption that joints belonging to the same kinematic sub-chain in the skeleton hierarchy usually present good correlation between them. Therefore, this information is incorporated in the methods we developed in order to find good correlations among each particular joint. Motion phases are used to analyze and capture specific aspects of a motion contained in each particular state. The phases can be treated as a natural subdivision of a motion, being the ideal feature to fragment the motion data. The last feature, motion principle components, is used in the PCA method. The components present interesting editing capabilities, allowing an animator to add or subtract a constant value to any degree of freedom, modifying a motion without loss of its appearance. Using these features to assist in the creation of a realistic character animation, we have described four different methods, namely Multiresolution Filtering, Hierarchical Fragment method, Principle Component method and Hybrid approach. In the Multiresolution Filtering method (chapter 3), we analyze the motion using the frequency analysis as feature. Producing band-pass and low-pass bands for existing motion capture data, an animator can interactively synthesize new motions by amplifying or attenuating the gain of a particular frequency band. In addition, this technique is used as a building block for the other methods, allowing motion decomposition in frequency bands and addressing tasks like concatenating, comparing and blending motion band-by-band. In the Hierarchical Fragment method (chapter 4), more features are used in order to analyze the motion. Starting by a set of degrees of freedom keyframed by an animator, the Multiresolution method is used to decompose the motion contained

CHAPTER 7. CONCLUSION AND FUTURE WORK 79 in each DOF in frequency bands. Then, the motion phase feature is used to fragment the motion data properly. Taking in account skeleton hierarchy information in order to find good correlation between joints, driven fragments are used to search fragments on the database with similar characteristics. At the end, missing degrees of freedom are created by synthesis and details are added to driven DOFs by texturing. As seen in the experiments section (Sec. 4.7.3), the Hierarchical Fragment method can really capture the motion essence and details, being useful to generate a realistic character animation. However, this method cannot properly extrapolate new motions from the examples in the database and presents some control parameters that are complex to be set at the beginning of the process. In chapter 5 the Principle Component method is investigated. Fitting a PCA model to the motion capture database, the dimensionality of the motion space is reduced and patterns in the data are identified and expressed in such a way to highlight their similarities and differences. Starting by a set of DOFs keyframed by an animator, PCA is applied to extract the details of the motion capture database in order to match these keyframed DOFs, and therefore enhance the animation. This method can be used to improve the initial animation sketch, however it cannot generate a realistic character animation. PCA cannot capture some important middle and high frequency details as observed in real human motion. As a result, the motion is small in amplitude and vague, being averaged from the rest of the database. Although the motion does not look natural, PCA can extrapolate different motions from the database examples and can correctly create a mean pose for a character, which can be used as low frequency information. In addition, the motion principle components can be used to edit the motion, by adding or subtracting a constant value to any degree of freedom, altering the low frequency bands. Since with the previous methods we could not integrally achieve our goal, we decided to use the methods together, creating a pipeline solution, as described in chapter 6. Using each method at a time, their characteristics are used to enhance an character animation in a particular way. During each stage an animator can modify the motion by using the principle components, altering the frequency bands or changing the control parameters. In addition, the method allows the automatic synthesis of a realistic character animation, matching the keyframed sketch, with the use of default control parameters. As a result, the hybrid method is able to improve a character animation, given by some keyframed degrees of freedom, in a simple and intuitive way using the strengths of Keyframing and Motion Capture. One drawback of the methods implemented in this project is that they do not

CHAPTER 7. CONCLUSION AND FUTURE WORK 80 incorporate hard constraints. By now, the methods cannot be applied directly, for example, to cases where the feet are meant to remain in contact with the floor. Currently, we are trying to solve this problem by combining them with an inverse kinematic solver. The computational time of the hybrid approach can also be improved by using specific techniques in the PCA stage in order to solve the inverse matrix computation faster. In this case, it will be possible to make the method more interactive. All the methods in this thesis strongly depend on the data in the database. Expressive new motion can only be generated if a great variety of motion is present for each joint in the database. We found that the database we used might not be ideal with respect to these criteria. It does not present considerable motion variation for the joints belonging to the lower kinematic sub-chain. As a result, the motion generated for these joints cannot exhibit expressive motion variations. Using an improved motion capture database, with better examples, we expect that the performance of the methods can also be improved. Currently, the skeleton model used in the methods cannot be changed (see Sec. 2.5.2). Then, an interesting topic for future work would be to use the information contained in a motion capture database to generate new motions for characters that have different skeleton models. During the development of this project many aspects of the human motion were better understood. This understanding is important for character animation in computer graphics, since the details of a motion usually reflect aspects like gender, mood and personality. With the hybrid approach we developed a tool that fulfills many of the requirements of our original project goal. With this work, we gave a step further to our ultimate goal: to develop a system able to compose and create a character motion in a higher level way. We identified important features of human motion, such as frequency analysis and motion principle components, and showed how the information present in them can be used to generate new motion. Therefore, it is our intent in the future to investigate the use of these features in order to create a character animation to express mood and personality. In addition, we would like to incorporate the knowledge gained in this project in a motion capture system, in order to improve the quality of the motion capture data and also to generate better examples for new databases. We believe that in the near future a higher level system for creating realistic character animation, together with an accurate motion capture system will be able to generate augmented and virtual scenes to be used for entertainment, medical or educational purposes.

Bibliography [AA00] T. P. Andriacchi and E. J. Alexander. Studies of human locomotion: past, present and future. In J. Biomechanics, 33(10):1217-1224, 2000. [ABC96] Kenji Amaya, Armin Bruderlin, and Tom Calvert. Emotion from motion. In Wayne A. Davis and Richard Bartels, editors, Graphics Interface 96, pages 222 229. Canadian Information Processing Society, Canadian Human-Computer Communications Society, May 1996. ISBN 0-9695338-5-3. [AF02] Okan Arikan and D. A. Forsyth. Interactive motion generation from examples. In John Hughes, editor, SIGGRAPH 2002 Conference Proceedings, Annual Conference Series, pages 483 490. ACM Press/ACM SIGGRAPH, 2002. [AFJ02] [AM00] [AV02] [BA83] [Bar94] M. Matarić A. Fod and O. Jenkins. Automated derivation of primitives for movement classification. In Autonomous Robots, 12(1):39-54, 2002. Marc Alexa and Wolfgang Müller. Representing animations by principal components. Computer Graphics Forum, 19(3), August 2000. M. Alex and O. Vasilescu. Human motion signature: Analysis, synthesis, recognition. In Proceedings of the International Conference on Pattern Recognition (ICPR 02), 2002. Quebec City, Canada. P. J. Burt and E. H. Adelson. A multiresolution spline with application to image mosaics. ACM Transactions on Graphics, 2(4):217 236, October 1983. David Baraff. Fast contact force computation for nonpenetrating rigid bodies. In Andrew Glassner, editor, Proceedings of SIGGRAPH 94 81

BIBLIOGRAPHY 82 (Orlando, Florida, July 24 29, 1994), Computer Graphics Proceedings, Annual Conference Series, pages 23 34. ACM SIGGRAPH, ACM Press, July 1994. ISBN 0-89791-667-0. [BH00] [Bla00] [BLCD02] [BRC02] Matthew Brand and Aaron Hertzmann. Style machines. In Kurt Akeley, editor, Proceedings of SIGGRAPH 2000, pages 183 192. ACM Press / ACM SIGGRAPH / Addison Wesley Longman, 2000. V. Blanz. Automatische Rekonstruktion der dreidimensionalen Form von Gesichtern aus einem Einzelbild. In Dissertation Universitt Tbingen, 2000. Christoph Bregler, Lorie Loeb, Erika Chuang, and Hrishi Deshpande. Turning to the masters: Motion capturing cartoons. In John Hughes, editor, SIGGRAPH 2002 Conference Proceedings, Annual Conference Series, pages 399 407. ACM Press/ACM SIGGRAPH, 2002. Frédéric Bevilacqua, Jeff Ridenour, and David J. Cuccia. 3d motion capture data: motion analysis and mapping to music. In Proceedings of the Workshop/Symposium on Sensing and Input for Media-centric Systems, 2002. Santa Barbara, California, USA. [Bre97] Christoph Bregler. Learning and recognizing human dynamics in video sequences. In IEEE Conf. on Computer Vision and Pattern Recognition, 1997. [BSH99] [BV99] [BV02] Bobby Bodenheimer, Anna V. Shleyfman, and Jessica K. Hodgins. The effects of noise on the perception of animated human running. In N. Magnenat-Thalmann and D. Thalmann, editors, Proceedings of the Eurographics Workshop, Computer Animation and Simulation 99, pages 53 63. Springer, 1999. Milano, Italy, September 7 8, 1999. Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3D faces. In Alyn Rockwood, editor, Siggraph 1999, Computer Graphics Proceedings, Annual Conference Series, pages 187 194, Los Angeles, 1999. ACM Siggraph, Addison Wesley Longman. V. Blanz and Thomas Vetter. Face recognition based on 3d shape estimation from single images. In Computer Graphics Technical Report, No. 2, 2002.

BIBLIOGRAPHY 83 [BW95] [CF00] [DCB00] [DKT98] [DL00] [FF01] [GHG89] Armin Bruderlin and Lance Williams. Motion signal processing. In Robert Cook, editor, SIGGRAPH 95 Conference Proceedings, Annual Conference Series, pages 97 104. ACM SIGGRAPH, Addison Wesley, August 1995. Los Angeles, California, 06-11 August 1995. Stephen Chenney and D. A. Forsyth. Sampling plausible solutions to multi-body constraint problems. In Kurt Akeley, editor, Siggraph 2000, Computer Graphics Proceedings, Annual Conference Series, pages 219 228. ACM Press / ACM SIGGRAPH / Addison Wesley Longman, 2000. Liwei Zhao Diane Chi, Monica Costa and Norm Badler. The emote model for effort and shape. In Proceedings of SIGGRAPH 2000, ACM Computer Graphics Annual Conference, pages 173 182, 2000. Tony DeRose, Michael Kass, and Tien Truong. Subdivision surfaces in character animation. In Michael Cohen, editor, SIGGRAPH 98 Conference Proceedings, Annual Conference Series, pages 85 94. ACM SIGGRAPH, Addison Wesley, July 1998. ISBN 0-89791-999-8. S. L. Delp and J. P. Loan. A computational framework for simulation and analysis of human and animal movement. In IEEE Computing in Science and Engineering, vol. 2, n 5, pages 46 55, 2000. Nick Foster and Ronald Fedkiw. Practical animation of liquids. In ACM, editor, SIGGRAPH 2001 Conference Proceedings, August 12 17, 2001, Los Angeles, CA, pages 23 30, New York, NY 10036, USA, 2001. ACM Press. C. V. Loan G. H. Golub. Matrix computations. The Johns Hopkins University Press, Baltimore., 1989. [Gle97] Michael Gleicher. Motion editing with spacetime constraints. In Michael Cohen and David Zeltzer, editors, 1997 Symposium on Interactive 3D Graphics, pages 139 148. ACM SIGGRAPH, April 1997. ISBN 0-89791-884-3. [Gle98] Michael Gleicher. Retargeting motion to new characters. In Michael Cohen, editor, SIGGRAPH 98 Conference Proceedings, Annual Con-

BIBLIOGRAPHY 84 ference Series, pages 33 42. ACM SIGGRAPH, Addison Wesley, July 1998. ISBN 0-89791-999-8. [HWBO95] Jessica K. Hodgins, Wayne L. Wooten, David C. Brogan, and James F. O Brien. Animating human athletics. In Robert Cook, editor, SIGGRAPH 95 Conference Proceedings, Annual Conference Series, pages 71 78. ACM SIGGRAPH, Addison Wesley, August 1995. Los Angeles, California, 06-11 August 1995. [JM02] O. Chadwicke Jenkins and M. Matarić. Deriving action and behavior primitives from human motion data. In In Proceedings of 2002 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS-2002), pages 2551 2556, 2002. Lausanne, Switzerland. [JWD03] Hui Gao James W. Davis. Recognizing human action efforts: An adptive three-mode pca framework. In To Appear in International Conference on Computer Vision, Nice, France, Oct. 13-16, 2003. [KGP02] Lucas Kovar, Michael Gleicher, and Frédéric Pighin. Motion graphs. In John Hughes, editor, SIGGRAPH 2002 Conference Proceedings, Annual Conference Series, pages 473 482. ACM Press/ACM SIG- GRAPH, 2002. [LCR 02] J. Lee, J. Chai, P. S. A. Reitsma, J. K. Hodgins, and N. S. Pollard. Interactive control of avatars animated with human motion data. In Proceedings of SIGGRAPH 2002, 2002. [LGC94] [LP02] [LS99] Zicheng Liu, Steven J. Gortler, and Michael F. Cohen. Hierarchical spacetime control. In Andrew Glassner, editor, Proceedings of SIG- GRAPH 94 (Orlando, Florida, July 24 29, 1994), Computer Graphics Proceedings, Annual Conference Series, pages 35 42. ACM SIG- GRAPH, ACM Press, July 1994. ISBN 0-89791-667-0. C. Karen Liu and Zoran Popović. Synthesis of complex dynamic character motion from simple animations. ACM Transactions on Graphics, 21(3):408 416, July 2002. Jehee Lee and Sung Yong Shin. A hierarchical approach to interactive motion editing for human-like figures. Computer Graphics, 33(Annual Conference Series):39 48, 1999.

BIBLIOGRAPHY 85 [LS00] [LS01] [LWS02] [Men99] [MW88] Lee Jehee Lee and Sung Yong Shin. Multiresolution motion analysis with applications. In In Proc. International Workshop on Human Modeling and Animation, pages 131 143, 2000. Lee Jehee Lee and Sung Yong Shin. A coordinate-invariant approach to multiresolution motion analysis. In Graphics Models, v63:3, pages 1 19, 2001. Yan Li, Tianshu Wang, and Heung-Yeung Shum. Motion texture: A two-level statistical model for character motion synthesis. In John Hughes, editor, SIGGRAPH 2002 Conference Proceedings, Annual Conference Series, pages 465 471. ACM Press/ACM SIGGRAPH, 2002. Alberto Menache. Understanding motion capture for computer animation and video games. Morgan Kaufmann Publishers Inc., San Francisco, CA, 1999. Matthew Moore and Jane Wilhelms. Collision detection and response for computer animation. In John Dill, editor, Computer Graphics (SIGGRAPH 88 Proceedings), volume 22, pages 289 298, August 1988. [OSB 01] D. Ormoneit, H. Sidenbladh, M. J. Black, T. Hastie, and D. Fleet. Learning and tracking human motion using functional analysis. In In IEEE Workshop on Human Modeling, Analysis and Synthesis, 2001. [OSBH01] D. Ormoneit, H. Sidenbladh, M. J. Black, and T. Hastie. Learning and tracking cyclic human motion. In In Advances in Neural Information Processing Systems 13. The MIT Press, 2001. [Pan01] Marcus G. Pandy. Computer modeling and simulation of human movement. In Annu. Rev. Biomed. Eng. (3), pages 245 73, 2001. [PB02] Katherine Pullen and Christoph Bregler. Motion capture assisted animation: Texturing and synthesis. In John Hughes, editor, SIGGRAPH 2002 Conference Proceedings, Annual Conference Series, pages 501 508. ACM Press/ACM SIGGRAPH, 2002. [Per85] Ken Perlin. An image synthesizer. Computer Graphics, 19(3):287 296, July 1985.

BIBLIOGRAPHY 86 [PFTV92] [PG96] [Pul02] [PW99] W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling. Numerical recipes in c. In Cambridge Univ. Press, Cambridge, 1992. Ken Perlin and Athomas Goldberg. IMPROV: A system for scripting interactive actors in virtual worlds. In Holly Rushmeier, editor, SIGGRAPH 96 Conference Proceedings, Annual Conference Series, pages 205 216. ACM SIGGRAPH, Addison Wesley, August 1996. New Orleans, Louisiana, 04-09 August 1996. Kathy Pullen. Motion capture assited animation: Texturing and synthesis. In PhD Thesis, Stanford University, 2002. Zoran Popović and Andrew Witkin. Physically based motion transformation. Computer Graphics, 33(Annual Conference Series):11 20, 1999. [Rai02] Marc Raibert. Simulating humans. In In SimScience Workshop, 2002. [RCB98] Charles Rose, Michael F. Cohen, and Bobby Bodenheimer. Verbs and adverbs: Multidimensional motion interpolation. IEEE Computer Graphics & Applications, 18(5), September October 1998. ISSN 0272-1716. [RGBC96] Charles F. Rose, Brian Guenter, Bobby Bodenheimer, and Michael F. Cohen. Efficient generation of motion transitions using spacetime constraints. In Holly Rushmeier, editor, SIGGRAPH 96 Conference Proceedings, Annual Conference Series, pages 147 154. ACM SIG- GRAPH, Addison Wesley, August 1996. New Orleans, Louisiana, 04-09 August 1996. [RH91] Marc H. Raibert and Jessica K. Hodgins. Animation of dynamic legged locomotion. In Thomas W. Sederberg, editor, Computer Graphics (SIGGRAPH 91 Proceedings), volume 25, pages 349 358, July 1991. [SSSE00] Arno Schödl, Richard Szeliski, David H. Salesin, and Irfan Essa. Video textures. In Kurt Akeley, editor, Siggraph 2000, Computer Graphics Proceedings, Annual Conference Series, pages 489 498. ACM Press / ACM SIGGRAPH / Addison Wesley Longman, 2000.

BIBLIOGRAPHY 87 [TH00] [Tro02] [UAT95] [WH97] [WK88] [WP95] Luis Molina Tanco and Adrian Hilton. Realistic synthesis of novel human movements from a database of motion capture examples. In Proc. IEEE Workshop on Human Motion (HUMO 00), Austin, Texas, 2000. N. Troje. Decomposing biological motion: A framework for analysis and synthesis of human gait patterns. In J. of Vision, 2:371-387, 2002. Munetoshi Unuma, Ken Anjyo, and Ryozo Takeuchi. Fourier principles for emotion-based human figure animation. In Robert Cook, editor, SIGGRAPH 95 Conference Proceedings, Annual Conference Series, pages 91 96. ACM SIGGRAPH, Addison Wesley, August 1995. Los Angeles, California, 06-11 August 1995. Douglas J. Wiley and James K. Hahn. Interpolation synthesis of articulated figure motion. IEEE Computer Graphics and Applications, 17(6):39 45, November/December 1997. Andrew Witkin and Michael Kass. Spacetime constraints. Computer Graphics, 22(4):159 168, August 1988. Andrew Witkin and Zoran Popović. Motion warping. In Robert Cook, editor, SIGGRAPH 95 Conference Proceedings, Annual Conference Series, pages 105 108. ACM SIGGRAPH, Addison Wesley, August 1995. Los Angeles, California, 06-11 August 1995.