MOTION CAPTURE ASSISTED ANIMATION: TEXTURING AND SYNTHESIS

Transcription

1 MOTION CAPTURE ASSISTED ANIMATION: TEXTURING AND SYNTHESIS A DISSERTATION SUBMITTED TO THE DEPARTMENT OF PHYSICS AND THE COMMITTEE ON GRADUATE STUDIES OF STANFORD UNIVERSITY IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY Katherine Ann Pullen July 2002

2 c Copyright by Katherine Ann Pullen 2002 All Rights Reserved ii

3 I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Christoph Bregler (Principal Adviser) I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Patricia Burchat (Physics) I certify that I have read this dissertation and that, in my opinion, it is fully adequate in scope and quality as a dissertation for the degree of Doctor of Philosophy. Gene Alexandar (Mechanical Engineering) Approved for the University Committee on Graduate Studies: iii

4 Abstract This thesis discusses methods for using the information in motion capture data to assist in the creation of life-like animations. To address the problem of developing better methods for collecting motion capture data, we focus on video motion capture. A new factorization method is presented that allows one to solve for the model of the subject s skeleton from a series of video images. No markers or special suits are required. The rest of this thesis discusses new techniques for flexible use of motion capture data after it has been collected. For the case of cyclic motions such as walking, we demonstrate a technique for complete synthesis. It begins with an analysis phase, in which the data is divided into features such as frequency bands and correlations among joint angles, and is represented with multidimensional kernel-based probability distributions. These distributions are then sampled in a synthesis phase, and optimized to yield the final animation. We also demonstrate methods for performing texturing and synthesis of widely varying motions based on motion capture data. First we present results using principle components analysis as a basis for the texturing and synthesis. Second we discuss our most successful technique for motion capture assisted animation, in which a simple matching algorithm is used. These methods allow an animator to sketch an animation by setting a small number of keyframes on a fraction of the possible degrees of freedom. Motion capture data is then used to enhance the animation. Detail is added to degrees of freedom that were keyframed, a process we call texturing. Degrees of freedom that were not keyframed are synthesized. The methods take advantage of the fact that joint motions of an articulated figure are often correlated, so that given an incomplete data set, the missing degrees of freedom can be predicted from those that are present. Finally, we discuss the various techniques and results, and suggest approaches for future improvements. iv

5 Acknowledgements There are so many people to thank, people without without whom this project would not have been possible. First of all, there is the Stanford Movement group: Gene Alexander, Rich Bragg, Chris Bregler, Ajit Chaudhari, Erika Chuang, Hrishi Despande, James Davis, Lorie Loeb, Lorenzo Torresani, Kingsley Willis, and Danny Yang. These people have provided many wonderful discussions and insights about my work, and often believed that what I was working on was going somewhere even when I didn t think it was. My advisors Pat Burchat and Chris Bregler have been extremely supportive throughout my graduate career, giving sound advice but always respecting the path I chose even when it was unconventional. I am eternally grateful to the Stanford Dance Department, for completely altering the course of my life in a positive way. In particular I wish to thank the faculty members Kristine Elliot, Emilie Flink, Diane Frank, Tony Kramer, Theresa Maldenado, and Robert Moses. I owe my sanity to the many friends I have interacted with while at Stanford, especially Wendy Adams, Debbie Bielawski, Cindy Chen, Tasha Fairfield, Anna Friedlander, Leslie Ikemoto, Karyn Ishimoto, Susie Judd, Miriam Kaplan, Elizabeth Koenig, Nadya Mason, Amit Mehta, Georg Petschnigg, Allison Rockwell, Uma Sundaram, Doug and Melissa Thomas, Melissa Starovasnik, and Esther Yuh. I also wish to thank my family, who have always been there for me and encouraged me to pursue my dreams. It was great being in the Bay Area with the whole Tapia clan, spawned by my aunt Marjeen and Uncle Stino, who were always a comforting presence. My dear Grandma and Grandpa and Aunt Diane were always in my thoughts despite the long distances. Finally, I am especially grateful to my immediate family: Mom, Dad, and brother Walter. v

6 Contents Abstract Acknowledgements iv v 1 Introduction Project Motivation Application to Animation Motion Capture Methods Overview Optical Magnetic Video Animation Methods Keyframe Animation Physical Simulation Motion Capture Thesis Outline Video Motion Capture Motivation Background Theory and Previous Work Overview Video-based Tracking: Gradient Formulation Example Motion Model: Affine Tracking vi

7 2.2.4 Twist Motion Model Solving for Joint Locations Experiments Sampling Kernel-Based Probability Distributions Overview Related Work Sampling Probability Densities Signal Processing Methods Analysis Synthesis Experiments Discussion Principle Components Analysis Based Methods Overview Review of Principle Components Analysis Application to Motion Data Discussion Fragment Based Methods Overview Related Work Methods Frequency Analysis Matching Path finding Joining Experiments Walking Otter Character vii

8 5.4.3 Modern Dance Discussion Conclusions and Discussion 85 Bibliography 90 viii

9 List of Tables 2.1 Symbols used in derivations ix

10 List of Figures 1.1 Keyframe animation of a computer model. Here we illustrate the example of animating one of the spine angles of a human character to cause her to bend forward at the waist. The animator sets the upright start pose, shown at the left of the figure with the character colored red. The animator also sets the end pose, bent forward at the waist, shown to the right of the figure in red. The animator specifies how many frames should occur between these two poses, and the computer fills in the missing frames by interpolating between the two key poses. Another way to represent this process is with a graph, shown below the images of the character. The plot is of the spine angle as a function of time, and each point corresponds to one of the images at the top of the figure. Key positions are indicated with red dots, and positions interpolated by the computer are indicated with blue dots x

11 1.2 Comparison of keyframed data and motion capture data for root y translation for walking. (a) Keyframed data, with keyframes indicated by red stars. The points computed by the computer are indicated with blue dots. In this example, the keyframed data has been created by setting the minimum possible number of keys to describe the motion. (b) Motion capture data. Here all the points are specified by the data, and are shown with black dots. Notice that while the keyframed data is very smooth and sinusoidal, the motion capture data shows irregularities and variations. These natural fluctuations are inherent to live motion. A professional keyframe animator would achieve such detail by setting more keys Hopping Wallaby with acquired kinematic model overlayed Example of a set of 4 phases during a walk cycle. The phases are as follows (a) right foot flat on the floor; (b) right heel lifts, right toe still contacting floor; (c) left foot flat on the floor; (d) left heel lifts, left toe contacting floor. Note this is a simplified model, for example in reality there is a moment when the left toes are on the floor at the same time the right heel is touching the floor. However, we found this simplified model gave good results in the synthesis process Hip angle data with the phases marked. Right foot flat, green circle; right toe in contact, magenta triangle; left foot flat, blue star; left toe in contact, red square. Note how the data has a very particular structure within each phase Example of decomposing data into frequency bands. Shown is the left hip angle data, higher frequencies are at the top, lower at the bottom. A Laplacian pyramid decomposition was used for this plot Plot of the knee angle vs. the hip angle at each point in time for two walk styles. (a) normal walk (b) funky walk xi

12 3.5 Contour plot of a 2-D kernel-based probability distribution for the hip and knee angle, the same data as shown in the correlations plot in figure 3.4a. Four different sigmas were used, as a fraction of the standard deviation of the angle data. (a) 1/40 (b) 1/10 (c) 1/5 (d) 1/ Plots of hip angle data after sampling and optimization. Shown is the 5th lowest frequency band in a Laplacian Pyramid decomposition. (a) Motion capture data; (b) synthetic data after sampling; (c) the same synthetic data as in figure 3.6b after optimization Correlation plots of hip angle data after sampling and optimization. Shown is the 5th lowest frequency band in a Laplacian Pyramid decomposition, the same data as in figure 3.6. The plot is of each point at time t-2 versus the point at time t. (a) Motion capture data is shown with black circles, sampled data is shown with blue squares. Note how some of the blue squares fall outside the range one would expect them to be in based on the distribution of black circles. (b) Motion capture data is again shown with black circles, and the sampled data after being optimized is shown with magenta stars. Now the synthetic data falls within a range predicted by the real data Example frame from one of the output animations. Each of the characters was animated with a different set of synthetic data, note how they vary in their motions xii

13 5.1 Illustration of the difference between texturing and synthesis. All plots are of the lowest spine x angle. (a) A keyframed curve is shown with a dashed blue line. Here we did not illustrate the key positions in this figure, only the resulting curve after the computer interpolates between them. Note its smooth appearance, as is common with computer generated curves. The solid magenta line is the result after texturing. (b) In this plot, we consider a case in which the spine angle was not animated at all, as indicated by the dashed blue line which does not change with time. This degree of freedom was synthesized, and the result is shown with the solid magenta line. (c) A plot of motion capture data is shown here for comparison. Note that its overall appearance is similar to the textured and synthesized curves. Compare also figure Correlation between joint angles. Shown is the left knee x angle versus the left hip x angle for each point in time for human walking data. Data points are indicated with blue circles, and points that are consecutive in time are connected by black lines. The fact that this plot has a definite form demonstrates that the angles are related to each other. (Also see figure 3.4.) Choosing the matching angles from the keyframed data. Shown are plots of joint angle as a function of time for some of the degrees of freedom from a keyframed sketch of a humanoid character. (Not all of the degrees of freedom of this particular character are shown to save space). In this sketch, only the lower body degrees of freedom were animated, as can be seen by the fact that only the joint angles from the legs show any change with time. We choose some of the degrees of freedom that were animated to serve as the matching angles that will drive the rest of the animation. In this example we use the left hip x and knee x angles, as indicated with red dashed lines in the figure xiii

14 5.4 Matching angles in the motion capture data. In these plot we show the same degrees of freedom as in figure 5.3, but for the motion capture data. Here we can see that the motion for all degrees of freedom, including the upper body, is specified. The matching angles selected from the keyframed data are again indicated here by dashed red lines. These selected degrees of freedom will be compared to the keyframed data to find similar regions as described in the text Frequency analysis. Shown are bands 2-7 (where lower numbers refer to higher frequencies) of a Laplacian Pyramid decomposition of the left hip x angle for dance motions from both keyframing and motion capture. Higher frequency bands are shown at the top of the figure, lower frequency bands at the bottom. Adding all the bands together yields the original signal. One band, shown with a red dashed line, is chosen for the matching step Breaking data into fragments. The bands of the keyframed data and motion capture data shown with red dashed lines in figure 5.5 are broken into fragments where the sign of the first derivative changes. (a) keyframed data. (b) motion capture data. (c) keyframed data broken in to fragments. (d) motion capture data broken into fragments xiv

15 5.7 Matching. (a) A longer segment of the low frequency band of the hip x angle data (a matching angle) from figures 5.5 and 5.6 is shown here again in black, broken into fragments. To the left in blue is the first keyframed fragment from figure 5.6c. Note the position of this fragment is arbitrary here, it is shown only for purposes of comparison to the motion capture curve. We wish to find fragments of the motion capture data that are similar to it, and some possibilities are shown with dashed magenta lines. (b) The spine x angle motion capture data from the same locations in time is shown, broken into fragments at the same location as the matching angle data. If the animator wished to synthesize the spine angle data, the fragments of spine angle data from the same locations in time where the matching hip angle data was chosen would be saved, as indicated by dashed magenta lines. (c) If on the other hand the animator had already keyframed a sketch of the spine angle motion and wished to texture the result, only the high frequency bands of the spine angle data would be selected. Shown is a plot of the sum of bands 2 and 3 of a Laplacian pyramid decomposition of spine x angle motion capture data, and the chosen fragments after matching are again indicated by the dashed magenta lines Close-up of the matching process. Each keyframed fragment is compared to all of the motion capture fragments, and the K closest matches are kept. Shown is the process of matching the first fragment shown in figure 5.6c. (a) The keyframed fragment to be matched. (b) The keyframed fragment, shown in a thick blue line, compared to all of the motion capture fragments, shown in thin black lines. (c) Same as figure 5.8b, but the motion captured fragments have been stretched or compressed to be the same length as the keyframed fragment. (d) Same as figure 5.8c, but only the 5 closest matches are shown xv

16 5.9 Matching and synthesis. (a) The five closest matches for a series of fragments of keyframed data is shown. The keyframed data is shown with a thick blue line, the matching motion capture fragments are shown with thin black lines. (b) An example of one of the angles being synthesized is shown, the lowest spine joint angle rotation about the x axis. The five fragments for each section come from the spine motion capture data from the same location in time as the matching hip angle fragments shown in figure 5.9a. (c) An example of a possible path through the chosen spine angle fragments is shown with a thick red line Texturing. Shown are bands 2-7 of a Laplacian Pyramid decomposition of the lowest spine x angle for a keyframe animation of dance motion. On the left is the original keyframed data. On the right is the result after texturing, in which bands 2-3, shown in magenta, have been replaced by joined fragments of the corresponding bands of the motion capture data as described in the text Choosing a path by maximizing the instances of consecutive fragments. In the table we show a hypothetical example of a case where four keyframed fragments were matched, and the K = 3 closest matches of motion capture fragments were kept for each keyframed fragment. The matches at the tops of the columns are the closest of the 3 matches. Blue lines are drawn between fragments that were consecutive in the motion capture data, and the cost matricies between each set of possible matches are shown below xvi

17 5.12 Joining the ends of selected fragments. (a) Four fragments of spine angle data that were chosen in the matching step are shown. Note this graph is a close up view of the first part of the path illustrated in figure 5.9c. There are significant discontinuities between the first and second fragments, as well as between the third and fourth. (b) The original endpoints of the fragments are marked with black circles, the new endpoints are marked with blue stars. The second and third fragments were consecutive in the motion capture data, so the new and old endpoints are the same. (c) For each fragment, the line between the old endpoints (black dashes) and the line between the new endpoints (blue solid line) are shown. (d) For each fragment, the line between the old endpoints is subtracted, and the line between the new endpoints is added, to yield the curve of joined fragments. The new endpoints are again marked with blue stars Smoothing at the join point. A close up of the join between fragments 1 and 2 from figure 5.12 is shown with a red solid line. (a) The quadratic fit using the points on either side of the join point (as described in the text) is shown with a black dashed line. (b) The data after blending with the quadratic fit is shown with a blue dashed line Example frames from the walking animations. On the top row are some frames from the keyframed sketch, and on the bottom row are the corresponding frames after enhancement Example frames from animations of the otter character. On the top row are some frames from the original keyframed animation, while on the bottom are the corresponding frames after texturing Example frames from the dance animations. The blue character, on the left in each image, represents the keyframed sketch. The purple character, on the right in each image, shows the motion after enhancement xvii

18 xviii

19 Chapter 1 Introduction 1.1 Project Motivation The ultimate goal of this project is to gain a better understanding of nuance in human and animal movement. In particular, we are interested in characterizing (1) variations in repetitive motions in a given individual and (2) differences in the same movement executed by different individuals. We use the term motion texture (a term originally suggested by Ken Perlin, a professor at New York University) to describe both of these aspects of live movement. Just as a piece of cloth has a certain texture defined by its look and feel, so does an individual s way of moving. For example, often you can recognize the identity of a person from far away without being able to see his face, just by how he is walking. Also in analogy to the case of a cloth texture is the fact that an integral part of a motion texture is the presence of stochastic properties. Just as the piece of cloth has irregularities such as slight variations in the size of each stitch, a person s motion will have variations within it. These variations may take place over relatively long time scales. For example, when we walk not every step is identical. Some steps may be slightly shorter or longer, and the upper body will respond slightly differently to each step. The variations may also occur over shorter time scales, for example within one step of a walk cycle the motion is not perfectly smooth, but shows some natural high frequency fluctuation. There are a number of fields of study that require a detailed understanding of the motion texture exhibited by each individual. One of the biggest applications of this concept is in 1

20 2 CHAPTER 1. INTRODUCTION biomechanics and medicine. In recent years, as the technology has improved both for the tracking of motion (see section 1.3) and in computer modelling [35], there have been many quantitative studies of human motion for the purposes of treatment and prevention of injuries [2]. Upon being injured, people will often alter their patterns of motion. Initially they may do so to avoid pain, but sometimes long after the pain has dissipated the altered movement patterns will remain. For example, such a phenomenon has been observed in patients who have torn their anterior cruciate ligaments in their knees [8]. The changes are usually subtle, and can only be detected by careful analysis of the gait of each patient. Since every individual moves with his or her own texture, the changes must be understood in the context of that individual s way of moving. It is important to understand these changes, because they often are not best for the long term health of the patient. By being able to identify these alterations in motion, a therapist may be able to recognize and help the patient overcome them more quickly. Similarly, noting individual differences in how people move may lead to the ability to prevent certain injuries from occurring. For example, in patients with osteoarthritis of the knee, it has been shown that individual variations in the dynamic loading of the knee strongly affects the outcome of standard treatments [39]. Research is in progress to determine whether these variations also influence the tendency to develop osteoarthritis in the first place [2]. Again, these changes are subtle, and in diagnosis one must be able to distinguish a gait pattern that might lead to injury from the natural variations among the way in which different people walk. Another application which requires a detailed understanding of live motion is in computer vision. There has been a great deal of interest in recent years in creating a fully video-based tracking system for articulated motions (see section 1.3.4). In such a system, the input to the computer would be a series of video images of the subject, and the algorithm would automatically detect the overall position and limb configuration at each frame. This problem is a difficult one, especially in the case of faster motion where the difference in position between frames may be significant. It has been proposed [15] to make these techniques more efficient with an initial coarse search stage. Such a search could be made more efficient by including probabilistic information about the motion of the subject [47].

21 1.2. APPLICATION TO ANIMATION 3 In other words, if the algorithm could predict where the subject is likely to have moved to in the next frame based on the previous frames, it may be able to narrow down the search for the next position. As in the case of the biomechanics applications, such a search requires a detailed knowledge of the patterns of human motion. A final application for the understanding of the way in which humans and animals tend to move is in animation. Animators are often particularly concerned with the subtle detail of a character s motion, because the nuance often reveals mood or personality. As improvements in technology have made data of live motion more readily available, there has been an increased interest in using the information in such data to assist an animator in the creation of a character s motion. This thesis focusses on the development of methods for analyzing and manipulating motion capture data for use in creating more life-like animations. 1.2 Application to Animation Currently there are three main methods by which a computer animation of a character can be generated. Most commonly, keyframing is used, in which the animator specifies important key poses for the character at some frames, and the computer calculates what the frames between these keys should be with an interpolation technique. In a second method, physical simulation is used to drive the motion of the character. Due to the complexity of the required calculations, this method has not been used with much success for characters. Finally, in more recent years, motion capture has been used to animate characters. In motion capture, sensors are placed on a live person, and the data that describes his or her motion is collected and mapped onto the character. As the technology for motion capture has improved and the cost has come down, there has been increasing interest in using it for character animation. The benefits of motion capture are many. Often an animator is particularly concerned with the subtle detail of how the character moves. However, achieving detail in a keyframed animation is extremely labor intensive. With motion capture data, all of the detail is immediately present, along with the nuance that gives personality to that individual s way of moving. In other words, the data contains the texture of the motion. The goal of a skilled animator is to reveal the

22 4 CHAPTER 1. INTRODUCTION personality or mood of the character through its motion texture. One problem with motion capture data is that after it has been collected, it is difficult to change. In fact, partly because of this inflexibility, many animators have little interest in using motion capture data. Keyframing may be labor intensive, but one can make a character do exactly what one wants it to. It is often difficult to know exactly what motions are needed before entering a motion capture session, and afterwards when the animator sits down to create the scene, he or she may find that the data is not exactly what is needed. It could be a simple change, for example perhaps the subject in the motion capture session walked in a straight line, but in the scene being created the character must walk in a curved path and stop at a particular point. To address this problem, many techniques have been developed to help edit motion capture data after it has been collected [23, 28, 38, 45]. However these methods may not always be sufficient. First, in editing the motion, one must be careful to not alter it in such a way that the detail is lost. Second, often the animator may want a completely different action than was captured in the data, so that it is not a simple matter of editing a motion that is already there. It is this situation that we are especially interested in. The animator may not have the motions she wants in the data, but may have a number of other motions performed by the actor that have the style and life-like qualities that she wants for her animation. We ask the question, given the information that is present in the motion capture data, can the animator somehow generate other motions that have the texture of that motion? This question is, in fact, the central one addressed by this thesis. The ultimate goal of this work is not to create a fully automatic method for creating animations. Instead, we are seeking methods that enable an animator to use live data to assist with the creative process of developing a character animation. The ideal situation would be as follows. The animator gets an idea about a particular style of motion she would like, but which might be difficult to animate. She has a friend who moves in that style, and can easily collect some data of that friend moving. She then sits down to create her animation, starting with the keyframe method she is familiar with, and which allows her to control the character. However, now that she has the motion capture data, she can texture her animation with the style of her friend s motions. To achieve this dream scenario, there are several goals that must be met. First, to allow

23 1.3. MOTION CAPTURE METHODS 5 her to spontaneously collect data of her friend, the method of data collection must be simple and flexible. Current motion capture systems do not meet that criteria; they involve highly sophisticated, expensive equipment that one can only use in particular locations at prescheduled times. As a result, there is a great deal of interest in developing systems based totally on video data, which would be inexpensive and portable. Recently computer vision techniques have improved to the point that such a motion capture system may be possible. In this thesis we present another step toward achieving that goal. The second goal that must be met is to develop new techniques for using the data after it has been collected. The methods must be flexible enough to allow the animator to have control over the results, and yet maintain the texture of the original motion capture data. The bulk of this thesis is devoted to discussing such methods. In the remainder of this chapter, background information relevant to our work is provided. In section 1.3 we discuss currently available methods for motion capture, and the problems associated with each. In section 1.4 we discuss the advantages and disadvantages of the various methods of animation. Finally, in section 1.5 we provide a brief outline of the body of the thesis. 1.3 Motion Capture Methods Overview In general, the term motion capture refers to any method for obtaining data that describes the motion of a human or animal. Ultimately, to be useful for driving a computer generated character, this data must take the form of the angles of all the joints in the body being modelled, plus 6 more degrees of freedom for the overall rotations and translations of the body. However, the raw data may take different forms, depending on the method used for capturing the motion. Currently the two most common methods for obtaining motion capture data are optical and magnetic [32].

24 6 CHAPTER 1. INTRODUCTION Optical In an optical system, retro-reflective markers are attached to the body of the subject. A system of cameras (the number varies widely, anywhere from 6 to over 500, depending on the particular system) surrounds the space where the subject moves. Each camera sends out a beam of infrared light, which is reflected back from the markers. After the marker positions are recorded as 2D frames, post-processing finds the 3D location of each marker at each point in time, and then solves for the joint configurations. There are many advantages of using an optical system. The movement of the subject is relatively unencumbered, compared to other methods, and it is possible for the space in which the actions can take place to be relatively large. In addition, very high rates of data collection are possible, which is especially important for people in the biomechanics community doing detailed research on joint motions. On the other hand, there are some disadvantages of optical systems, mainly caused by the intensive post-processing required. Usually the data cannot be collected in real time, and in fact it may be several hours or days before the final result can be viewed, which can be problematic if the user has only one day scheduled at the motion capture studio. An even greater problem is that of occluded markers. The multiple camera set-up is designed to minimize this problem, so that at any given moment no matter what direction the subject is facing or what position he or she is in, the chances are good that each marker will be seen by at least some of the cameras. However in practice there are still often moments where markers are occluded from all of the cameras, usually by self-occlusion. For example, if the subject hunches forward too much, any markers on the front of the body will be covered. The software becomes confused when it loses sight of a marker, and a technician must spend time going through a data set fixing these moments by hand. A related problem is that it is difficult to capture more than one subject at a time, because when they get close together the markers of each overlap, and again the software becomes confused as to which marker belongs to which person.

25 1.3. MOTION CAPTURE METHODS Magnetic The advantages of magnetic systems address many of the flaws of optical systems. In a magnetic system, a known magnetic field is set up, and the actor wears sensors that detect the location and orientation of each limb based on that magnetic field. This method allows for real-time data collection, and there are no problems with occlusion. On the other hand, one big drawback of this method is that it is very sensitive to the area it is performed in. Metal objects must not be nearby, and usually a field of high enough quality for data collection can only be created in a relatively small space. In addition, wires must be attached to each sensor, which makes many motions awkward for the subject. In most cases the wires run from the sensors to an external interface. In higher-end models, the system is wireless in that the wires all connect to a unit worn by the subject as a backpack. Such a system allows for much greater freedom of motion than having wires run to an external location, but still may encumber the motion of the subject, much more so than in an optical system in which the only objects attached to the person are the small reflectors Video The technology surrounding both optical and magnetic systems is continuing to improve, and the motion capture data generated by such systems is becoming more readily available to animators interested in using it. In fact, there are numerous motion capture studios where one can have custom data collected. However, it is still a cumbersome process. The fees may be large, and a particular day must be scheduled ahead of time to collect the data. If the animator then finds the data is not exactly what is needed, another day must be scheduled, and another fee paid. An even greater problem may be that the data must be collected in a special studio. Even for an optical system, in practice the space is usually quite limited. The most dynamic motions are likely to be found in a different environment, for example an athlete in the midst of a game on the field or a dancer performing on stage. If you take the athletes off the field, have them wear special suits, stand around and be calibrated, and then ask them to perform their activity, the results will not be nearly as dynamic as if they were actually in competition.

26 8 CHAPTER 1. INTRODUCTION As a result, it would be extremely useful to be able to get motion data by merely using a couple of video cameras. However, this technique is a difficult one. Standard computer vision tracking techniques do not work for an articulated figure, in which many of the motions cannot be defined by a simple affine transformation, but involve rotations about all the joints in a kinematic chain. Work has been done to address this problem by Bregler [14], in which a kinematic chain tracker was developed that can simultaneously extract the global translation and rotations of an articulated figure as well as the joint angles. A problem with this method is that an accurate model of a skeleton is required, but that information may not be known before beginning the experiments. In this thesis the tracking method is extended to allow one to solve for the joint positions, which provides the necessary information to begin tracking. 1.4 Animation Methods There are three main methods by which computer animations are created: (1) key frame interpolation; (2) physical simulation; and (3) motion capture. Each of these methods has its advantages and disadvantages and is appropriate in different situations. In this thesis we are mainly concerned with using motion capture data, but to put the work in context it is useful to consider a brief review of each of the animation methods. In the following sections, we discuss and review the advantages and disadvantages of the various methods of animation in more detail Keyframe Animation Keyframe animation has been used by traditional animators (animators who draw the frames of the animation by hand) long before the advent of computers. In traditional animation, one normally draws the extremes, or important landmarks in the motion, called keyframes, and then draws the intermediate frames using the keyframes as a guide. With the advent of computers and 3D graphics, people began using the computer as a tool to assist in creating an animation. A 3D model of a character is created in the computer, and the animator again specifies keyframes, this time not by drawing, but by posing the model in the computer. In

27 1.4. ANIMATION METHODS 9 this case the animator does not have to create the intermediate frames. Instead, the computer calculates them based on the keyframes, usually by interpolating between the key positions to create the motion curves that drive the action of the modelled character. This process is illustrated in figure 1.1. On the surface, this use of the computer may appear to be a great savings in labor. However, in reality the use of computer models has its own set of difficulties associated with it. As a result, creating a computer animation can be just as labor intensive as making a traditional animation. One main reason for the high labor cost of a computer animation is that a typical articulated figure model such as a humanoid character usually has at least 50 degrees of freedom. For example, a minimal model of a human may have a left and right hip, knee, ankle, ball of foot, shoulder, elbow, and wrist, as well as 5 joints for the spinal column. Each of these joints has 3 degrees of freedom (rotation about the x, y, and z axis). In addition we must include the 6 degrees of freedom for the root translations and rotations, for a total of 63 degrees of freedom. Making the model more realistic by adding hand and finger joints or more spine joints would further increase the complexity. The animator must then painstakingly animate each of these degrees of freedom, one at a time. Another problem with keyframe animation as used on a computer is the interpolation process. If too few keyframes are set, the motion may be lacking in the detail we are used to seeing in live motion (figure 1.2). The curves that are generated between key poses by computer are usually smooth splines or other forms of interpolation, which do not represent the way a live human or animal moves. Live motion contains variations at high frequencies that splines do not. An animator may achieve a high level of detail by setting more and more keyframes, even to the point of specifying the position at every time, but at the expense of more time and effort. In fact, many other researchers before us have made the observation that part of what gives a texture its distinctive look, be it in cloth or in motion, are variations within the texture. These variations are often referred to as noise, and one of the earliest papers to address this topic was in image texture synthesis, where random variability was added to textures with the Perlin-noise function [36]. These ideas were later applied to animations [37]. Other researchers have created motion of humans running using dynamical simulations [25] and applied hand crafted noise functions [10].

28 10 CHAPTER 1. INTRODUCTION Figure 1.1: Keyframe animation of a computer model. Here we illustrate the example of animating one of the spine angles of a human character to cause her to bend forward at the waist. The animator sets the upright start pose, shown at the left of the figure with the character colored red. The animator also sets the end pose, bent forward at the waist, shown to the right of the figure in red. The animator specifies how many frames should occur between these two poses, and the computer fills in the missing frames by interpolating between the two key poses. Another way to represent this process is with a graph, shown below the images of the character. The plot is of the spine angle as a function of time, and each point corresponds to one of the images at the top of the figure. Key positions are indicated with red dots, and positions interpolated by the computer are indicated with blue dots.

29 1.4. ANIMATION METHODS (a) Keyframed Data translation in inches (b) Motion Capture Data time in seconds Figure 1.2: Comparison of keyframed data and motion capture data for root y translation for walking. (a) Keyframed data, with keyframes indicated by red stars. The points computed by the computer are indicated with blue dots. In this example, the keyframed data has been created by setting the minimum possible number of keys to describe the motion. (b) Motion capture data. Here all the points are specified by the data, and are shown with black dots. Notice that while the keyframed data is very smooth and sinusoidal, the motion capture data shows irregularities and variations. These natural fluctuations are inherent to live motion. A professional keyframe animator would achieve such detail by setting more keys.

30 12 CHAPTER 1. INTRODUCTION The disadvantage of these methods is in that they require one to (1) decide on a noise function; (2) tune the function to optimize the look of each animation; and (3) the resulting animations, while improved from the original, still may not look correct, or truly life-like. Our work differs in that we extract a motion texture, which inherently contains variations, from live data rather than trying to develop an artificial noise function that must be tuned before being added to the animation Physical Simulation In order to reduce the burden on the animator, there has been a large amount of research to develop techniques based on physical simulation. These methods have been most useful for animating cloth deformations [7, 20], rigid objects [5, 6, 33], or fluids [21, 22] where the physics of the situation that determines the motion can clearly be specified. However, the problem of creating a complete physics-based model of an articulated figure for the purposes of artistic animation has not yet been solved. Most work in physical simulation so far has taken place by researchers seeking highly accurate models for use in biomechanical studies. Such models require one to account for complexities such as the fact that more than one muscle usually controls each joint; muscles exert forces on tendons, which may have non-linear properties; and joints usually are not simple hinges, but may have complex kinematics, involving sliding as well as rotating about multiple axis [19]. This type of modelling is not practical for animation, in which one wants to be able to quickly compose a wide variety of motions. It is extremely unlikely that the animator will know the proper configuration of muscles and bones or the internal energy required to move them to create the desired motion. In fact the animator probably will not know even basic starting points such as the masses of the limbs of the character, which may or may not even resemble a human if it is a fantasy creature. However, a number of researchers have developed clever methods to make use of physics in their work. Some of the most successful such animations of articulated figures using physics based methods were done using the method of spacetime constraints [49]. The animator specifies the physical parameters of the character, for example the masses of each limb and the spring constants of the joints. High level controls such as key positions and hard constraints at contact points are

31 1.4. ANIMATION METHODS 13 also specified. The motion is determined by solving a constrained optimization problem over all of the time points at once. The solver seeks to minimize the energy of the system while maintaining the constraints set by the animator. In practice the most complicated character this technique was applied to was the Luxo Lamp (from Pixar Studios). This character had just four joints, each of which had one degree of freedom, plus two translational degrees of freedom. In other words, the motion was restricted to a plane. The resulting animations were quite life-like and appealing, but extending this method to a human model is not practical. A simple animation of the lamp making a jump took just under 10 minutes to compute. When one considers the far more complex motion of a human, for example the presence of two legs and the weight shifting back and forth between them, or the difficulty of even correctly specifying the masses and spring constants for all of the joints, it is clear that the computations would be excessively long if they could be performed at all. Because of these difficulties, the most successful attempts at using rigorous physical simulation of human motion have been modelled after work in robotics research [42]. The problem of making a physical robot walk is analogous to generating a simulation of a walk. The legs move with input of energy from a power source, but it is difficult to know exactly how much energy to put into each leg at what time points to have the robot walk forward in a coordinated manner without falling down. The problem is complicated by the fact that it is likely that the robot is being controlled at a high level by the user, for example with a remote control that allows for the specification of direction and velocity. Solving the equations of motion and predicting the proper initial conditions to move such a system in real time would be extremely difficult if not impossible. As a result, most robots incorporate some form of feedback control into their movement. A sensor detects whether the robot is falling, moving too slowly, etc., and then responds by increasing or decreasing the energy input to the appropriate joints. The same principle can be applied to animation. Now instead of a physical robot, we have a simulated model of a human in the computer. Jessica Hodgins and her colleagues have developed a method of applying control systems to virtual humans [43] and applied it to create animations of humans performing athletic events such as running, biking and a gymnast vaulting [25]. The animations were successful in that the characters clearly

32 14 CHAPTER 1. INTRODUCTION performed the activity being simulated. However, they are clearly simulated; one can see immediately that the motions are not life-like. In addition, each specific motion had to be treated differently. For example, for running a state machine was used; the dynamics are different depending on whether the character is in flight or has one foot in contact with the ground, which in turn differs from the gymnast who vaults off of her hands or the cyclist who is constantly in contact with the bike seat. In fact this problem of being unable to generalize the model is inherent to rigorous physical simulation; it will always be difficult to find a model and control system applicable to any motion an animator might dream up. As a result, we felt that in developing a method for creating life-like animations, it would be more effective to use a statistical analysis of live data to generate the motion curves. Correlations among various features in the data can be modelled with probability distributions. For example, we create multidimensional probability distributions based on correlations among various features of the data that we can sample from to create new motions (chapter 3). In other work, we break the real and keyframed data into fragments, and seek the closet matches for use in texturing and synthesis (chapter 5) Motion Capture The use of live data for animations has a long history. Traditional animators (animators who draw each frame by hand) often carefully study motion by looking at movie frames in slow motion to see exactly how a live person executes a motion. Taking this idea even further, they may revert to rotoscoping, in which case a film of a live actor is actually traced frame-by-frame. In this case they then often go back and accentuate the motion beyond what was originally there to make it more extreme and give even more personality to the characters. With the advent of computer and 3D models of articulated figures, animators have another tool at their disposal in the form of motion capture data. In motion capture data, the joint configuration of a live person are detected by sensors, and these angles are then read into the computer model to create the animation (see section 1.3 for more details). Originally such data was extremely difficult to obtain, as the sensor technology is costly and few animators had access to it. However in recent years the technology has improved