Robust and Automatic Optical Motion Tracking

Transcription

1 Robust and Automatic Optical Motion Tracking Alexander Hornung, Leif Kobbelt Lehrstuhl für Informatik VIII RWTH Aachen Aachen Tel.: +49 (0) Fax: +49 (0) Abstract: Marker-based optical motion tracking is an established technique to capture and reconstruct the skeleton and motion of a subject. However, several practical problems still arise, mostly due to ambiguities caused by occluded or wrongly identified markers which often makes post-processing by a human user inevitable. We present techniques to make marker-based optical motion tracking an automatic and robust process. The aim of our framework is a self-calibrating system, which automatically identifies rigid cliques of markers and recovers the skeleton topology and geometry of a tracked subject without any auxiliary information about the tracking setup. The gathered information is used to make the actual motion recording phase robust to marker occlusions by reconstructing missing limbs or joints of the subject using inverse kinematic methods. The resulting techniques provide a simple, general framework to perform optical motion tracking which minimizes the need for complex and manual post-processing. Keywords: Optical Motion Tracking, Self-calibration, Retargetting, Animation 1 Introduction Capturing a real actor s motion plays an important role in computer animation as well as in motion analysis for medicine or sport science. Tracking the position and orientation of the subject s limbs allows the realistic reproduction and transfer of this motion to virtual characters with the same skeleton topology. Several approaches to track motion exist, like contour finding [CTMS03], or marker-based methods, where the trajectory of markers Figure 1: A subject s attached to the subject s limbs are tracked magnetically [OBBH00] or optically [HFP + 00] (Fig. 1). Although optical tracking is in general arm equipped with optical markers. the most reliable system in terms of accuracy and robustness to external influences, it suffers from two fundamental problems. On the one hand optical markers are visually indistinguishable. Therefore we need appropriate methods to identify markers based on other

2 criteria in order to associate detected markers with the respective limbs. The second fundamental problem is that of occlusion. To reconstruct the three dimensional position of a marker, it has to be seen by at least two cameras. This cannot be ensured for a freely moving actor. Hence we need methods to compensate for missing markers, so that we can still reconstruct the position and orientation of the actor s limbs, even if a significant number of markers is occluded. Existing research and commercial solutions like our ART tracking system [ART] often need a considerable amount of time for a manual calibration to allow for reliable and robust marker recognition and tracking. In contrast, this project focuses on methods to make optical motion tracking a completely automated pipeline which minimizes the necessity for intervention. Beginning with a self-calibrating system initialization, which uniquely identifies rigid cliques of markers, we automatically compute the underlying skeleton geometry and topology of a tracked subject. In particular, we do not constrain the degrees of freedom of the underlying model in any way, so that we are able to track arbitrary articulated bodies. Moreover, we reconstruct the complete position and orientation of limbs in contrast to other methods, which often have open degrees of freedom concerning the orientation of limbs, or which have to constrain the skeleton topology beforehand. During the actual motion recording phase we take advantage of this information to compensate for occluded markers, and to reconstruct the position and orientation of limbs and joints in an accurate and robust way. 2 Related Work Several partial solutions increasing robustness and automation in optical motion tracking have been proposed. [RL02] present an automatic method to identify marker cliques. However, they need an explicitly occlusion-free training sequence which is processed offline to determine marker cliques and model parameters like the skeleton structure. Our method does not impose constraints on the initial training sequence and provides permanent feedback since it is computed online. Our work on marker tracking and the dynamic identification of rigid marker cliques by formulating them as instances of a generic correspondence estimation problem is based on [SLH91]. They present an elegant algorithm to associate the features in two images for applications in computer vision. The transfer of their method to the domain of optical motion tracking allows us to solve several tracking-related problems in a unified manner. [OBBH00] show how to estimate the structure and geometry of an unknown skeleton model. They describe a least squares fit of input motion data of individual limbs to a rotary joint model. Other methods like [SPB + 98] compute joints by estimating the rotation center of markers and their associated limbs. We use the technique of [OBBH00] since it results in higher accuracy and robustness concerning noise. Approaches to make the actual recording of motion data more robust range from predicting future marker positions using a Kalman filter [DU03] or search space reductions based on other prediction quality measures [vlvr03] to resolving occlusions based on the skeletal model of the tracked person as described in [HFP + 00]. Our method does not try to identify or reconstruct

3 markers based on predictions of future states but focuses on their robust recognition based on generated marker-signatures. This ensures a reliable identification even after occlusions during several frames, where prediction models possibly fail due to unconstrained movements of the tracked subject. We improve the actual tracking quality in the case of missing markers by applying methods of inverse kinematics to the computed skeleton as presented in [TGB99]. They show how to reconstruct missing inner limbs of a skeleton up to one degree of freedom based on adjacent limbs in real-time. We extend their solution to determine the remaining degree of freedom if only one additional marker on the lost limb is known. In a recent work [ZH03] use a force-based forward dynamic model to map optical motion tracking data to a body model. Their technique fails to estimate the skeleton of the tracked subject and is also running offline. However they explicitly mention benefits of a real-time system for motion capture. Commercial systems like [Vic] provide software tools for all phases of the tracking pipeline. However, such systems focus on setups with single markers attached to limbs, resulting in the above mentioned restrictions. [ART], which is a commonly used system for VR-applications, provides only low-level tracking and marker recognition without methods for automatic calibration, skeleton estimation or robust tracking of articulated bodies. 3 Self-Calibration To track an unknown subject s motion, it is equipped with a set of spherical optical markers m 1,..., m k (Fig. 1). As the subject moves, the tracking system reconstructs the 3D position of a marker at time t when it is seen by at least two cameras. Due to indistinguishable markers and occlusions, our input data consists of an unstructured set of detected markers M t = {m t 1,..., m t k(t) } given by their corresponding 3D positions P t = {p t 1,..., p t k(t) }, at Frame F t with k(t) k. To calibrate and prepare the motion tracking system for the actual motion recording phase, we have to solve the above mentioned two fundamental problems of marker distinction (map m t i to m j ) and temporal marker occlusion (recognize m t+n i as m t j). We present a method to solve both problems as instances of a general correspondence estimation problem, resulting in a simple, coherent general framework.! #" Figure 2: In this figure, eight markers are part of two distinct cliques, while one marker is tracked isolated. Within each clique, the inter-marker distances are constant. Between two sets M t 1 and M t three classes of markers have to be distinguished during continuous tracking: Lost markers (empty dashed boxes), tracked markers, and new markers. We solve this problem by finding corresponding elements in M t 1 and M t. Every time a new marker is found it is assigned to an unused global marker identity m i.

4 Our methods for correspondence estimation are motivated by the work of [SLH91]. They presented an elegant approach to find a partial mapping between two sets of objects which minimizes the overall squared sum of some inter-object measurements based on a singular value decomposition of a proximity matrix. [SLH91] used this method to find an assignment between feature points in two images. However, using their method in a more general sense by formulating each tracking-subproblem as a matching problem between two partially corresponding sets of objects yields a simple framework to solve tracking-related tasks. We extended their method to work better with sets of objects, where the actual number of corresponding elements can be arbitrary. The first low level tracking task where we apply our correspondence based method is the continuous tracking of markers between successive frames F t 1 and F t. As mentioned above, our input data consists of unstructured sets of markers M t 1 and M t and their corresponding 3D positions P t 1 and P t. Due to occlusion some of the markers in M t 1 will be lost in M t, some will be trackable through both frames with slightly different positions, and some markers will be new in M t (Fig. 2). Our modified correspondence estimation algorithm identifies these classes of markers by assigning corresponding positions between the two sets P t 1 and P t. In particular, our algorithm finds a partial mapping of subsets of markers within the two sets M t 1 and M t, allowing for vanishing or newly appearing markers. As depicted in Fig. 2 markers can be temporally occluded for the tracking system and are therefore lost during the continuous tracking approach. Since markers cannot be distinguished the system does not know, whether a new marker was already tracked before and thus should be associated with the former identity. To resolve these ambiguities, one attaches not only one but several markers to every limb of the tracked person. Markers located on the same limb form rigid cliques with characteristic invariant inter-marker distances, while distances to markers on other limbs will change over time. We can think of attaching a string between each pair of markers. As we go from Figure 3: Ripping strings. Initially all markers are connected to each other. By moving the respective cliques around, edges between different cliques are destroyed and only the final rigid cliques remain. frame to frame, we record the length variation of each string. If a string is stretched too much it rips (Fig. 3). In the end only the strings between the rigid cliques remain. These constant distances to other markers within the same clique form a unique distance pattern or signature Sig i for every marker m i. This makes it possible to identify a currently unknown marker m t j. Suppose this marker was already seen by the system before, then there exists some marker identity m i with a unique distance pattern Sig i corresponding to m t j. So after computing the set of distances of m t j to all other markers found in the same frame F t, we can identify m t j as m i by finding the distance

5 pattern Sig i within this set. This is in particular difficult since signatures are often only partially available because of marker occlusions. Furthermore, signatures are often partially equal since the range of possible marker distances within one rigid clique is restricted by marker sizes, the precision of the tracking system, and of course the wearability of a clique for the tracked subject. This problem of identifying temporally occluded markers based on signatures is also solved by our correspondence estimation algorithm. During the run of our algorithm we continuously track markers, dynamically create these signatures to identify single markers and rigid cliques, and resolve ambiguities caused by temporally occluded markers. As soon as a target number of rigid cliques is found, we automatically compute the position and orientation of each corresponding limb by embedding a local coordinate system into the corresponding clique of markers. At this point, the basic calibration of the tracking system is accomplished. We can reliably track a subject equipped with several sets of markers. In contrast to systems like [ART], this initialization is done completely automatic in real-time with a permanent feedback to the user, which allows a maximum efficiency and flexibility. In the following step, the underlying skeleton model is automatically reconstructed for higher-level tracking tasks. 4 Skeleton Reconstruction Each of the identified rigid cliques corresponds to a limb of the tracked subject. Several methods were proposed to automatically reconstruct the underlying skeleton structure. Under the assumption of a skeleton model with rigid bones and rotational joints, [OBBH00] show how to robustly compute precise joint positions by solving a least squares system of motion measurements. Each limb l i is associated by a time-varying local coordinate system. Thus there is a transform L t i = [R t i t t i] which maps from l i s current local coordinates to world coordinates. The joint between two limbs l i and l j has the property that it has constant local coordinates c i with respect to l i and constant local coordinates c j with respect to l j. The coordinates c i and c j are related to each other by the fact that they map to the same position in world coordinates, i.e., L t ic i = L t jc j for every frame F t. For every possible pair of limbs and measurements in n frames this leads to an overdetermined system that we can solve for the local joint coordinates in the least squares sense. R 0 i. R n 1 i R 0 j. R n 1 j [ c i c j ] = t n 1 j t 0 j t 0 i. t n 1 i (1) The skeleton structure can be computed by a minimum spanning tree connecting the limbs. Since every joint is defined by the two respective positions L t ic i and L t jc j, it can be computed during the actual tracking phase in the following section by averaging the two positions. Even if one limb is completely lost, all joint positions are still explicitly defined. The geometry of the bones is given by distance between adjacent joints. The computed joint positions allow us to calculate a further type of signature to identify markers, since the distance between markers and joints associated with the same limb remain constant. We will exploit these additional signatures

6 in the following section. The extracted skeleton allows us to retarget the captured motion data to arbitrary objects with the same skeleton structure. Moreover, the computed model helps us to make the actual motion capturing phase more robust in cases of occluded markers. 5 Robust Motion Capture During the actual motion capturing phase we are able to use several methods to make the tracking robust. We have a robust method to identify formerly lost markers, redundant orientation and position information for every limb, and a skeleton which reduces the degrees of freedom for every limb by imposing constraints from neighboring limbs. In this section we provide an example of how to exploit the skeleton structure to reconstruct lost limbs in cases, where complete cliques of markers could not be tracked due to occlusions. Consider the case where two inner limbs like the forearm and upper arm of a tracked human body, a so called human arm like chain (HAL-chain) are lost. In this case the position of the missing inner joint can still be computed up to one degree of freedom. [TGB99] show that it has to lie on a circle defined by the intersection of two spheres (Fig. 4) given by the outer joint positions j 1 and j 2 (wrist and shoulder) and both inner limb lengths l 1 and l 2. In practice it is very unlikely that both cliques of the lost limbs are completely occluded. In most cases there will be at least one additional marker position p available. This marker can be identified by its rigid distance to its corresponding joint. Knowing its constant distance d to the missing inner joint position, it is possible to define a third sphere centered at the marker s position p with radius d. The lost inner joint position j has to be the intersection of these three spheres. Figure 4: This figure shows a situation, where the upper and lower arm are lost during tracking. The lost inner joint can be reconstructed by intersecting three spheres as described below. The orange circle visualizes the intersection of two of them. The third sphere intersects this circle in two points, yielding the lost inner joint. j S(j 1, l 1 ) S(j 2, l 2 ) S(p, d). Usually this yields two possible solutions. Additional markers constrain the left ambiguities in a similar way, until the the correct joint position can be reconstructed exactly. Otherwise one can choose the most plausible of the left solutions, according to continuity assumptions or other heuristics.

7 6 Results In our current setup we use four ARTTrack1 cameras [ART], placed in the four upper corners of a rectangular room. While this setting allows us to track an unconstrained moving person, it also results in very frequent marker occlusions. For example, while tracking the HAL-chain in Figure 4 with 17 markers in 4 cliques, we had an average of 21% of markers lost between two successive frames at a tracking rate of approximately 50 frames per second. In spite of this high percentage of lost markers it is still possible to capture the motion of a subject reliably since reappearing markers and partially visible cliques are generally identified within a few frames. To compare the quality of the inverse kinematic technique for inner joints to the actual joint position, we measured the deviation (Fig. 5) between both estimates for the elbow position (Fig. 4). The average deviation of the reconstructed joint from the actual joint position is only about 8 mm with a standard deviation from this value of 7 mm. This is very close to the actual precision of the tracking system for the used marker size. The high peaks result from wrongly identified single markers, in which case the sphere S(p, d) is of wrong size and the computed circle intersections result in wrong marker positions. However, such errors can be identified easily by assuming a continuously moving subject. The wrong positions can be compensated by enforcing physically plausible movements of the joints. Figure 5: The deviation between the computed joint position using the inverse kinematic method and the actual joint position for the elbow. The peaks correspond to falsely computed joint position due to wrongly identified markers. However, they can be easily detected and compensated. The necessary length of the initial phase for learning characteristic signatures depends on the presentation of cliques to the system. In an optimal setting, the self-calibration phase can be accomplished with all marker cliques already attached to the subject. In the case of a high loss rate like the above mentioned 21% for our system, it can be more efficient to calibrate the marker cliques in such a way, that a minimal number of occlusions is ensured, e.g. by attaching the marker cliques to the subject after the initial clique-calibration step. For continuously tracked markers without frequent occlusions, the self-calibration is finished as soon as the last clique gets visible. In cases where all markers are visible from the beginning, the calibration is completed after just a few frames. Since the system is running in real-time, one has direct feedback about the tracking quality, during the initialization as well as during the final motion capture. The computation of the skeleton geometry and topology is also a matter of seconds. For the skeleton shown in Figure 6 with markers attached to 12 limbs, we generally consider motion sequences between 20 and 60 seconds duration. The actual model estimation takes approximately 1 second for 60 seconds of recorded motion at 50 fps. For a good model approximation it is of

8 primary importance that the tracked subject exercises all degrees of freedom for each joint. If the initial phase was already performed with the subject equipped with markers, these measurements can already be used for the skeleton reconstruction. Figure 6 shows a few frames from a captured sequence using our system. Despite the fact that only 79% of markers could be tracked between two successive frames, the marker identification and inverse kinematic methods allowed us to reliably record the motion of a person equipped with 12 cliques of markers. 7 Conclusions We presented different methods to make optical motion tracking a robust and automatic process. Automation was achieved by creating a real-time capable self-calibration method, which learns characteristic marker signatures to identify rigid cliques of markers corresponding to limbs of the tracked subject. One key aspect of our method was the mapping of several crucial subproblems to different instances of one generic correspondence matching problem, resulting in a simple and robust algorithm. From the rigid cliques of markers the system automatically extracts the geometry and topology of the subject s skeleton. The robustness of the tracking process is improved by the reliable identification of markers even after long and frequent occlusions. Based on the computed skeleton we showed how to apply inverse kinematic methods to reconstruct limb or joint po- sequence using our system. Figure 6: A few frames from a captured motion sitions in cases, where an explicit clique-based computation is impossible due to an insufficient number of tracked markers. In the future we will integrate more sophisticated prediction filters for the movement of markers, which will improve the tracking during those periods, where insufficient information is available to apply the presented inverse kinematic methods as in the case of lost outer limbs. References [ART] ARTtrack1 & DTrack, A.R.T. advanced realtime tracking GmbH,

9 [CTMS03] Joel Carranza, Christian Theobalt, Marcus A. Magnor, and Hans-Peter Seidel. Freeviewpoint video of human actors. In ACM Transactions on Graphics, volume 22, pages , [DU03] [HFP + 00] Klaus Dorfmüller-Ulhaas. Robust optical user motion tracking using a kalman filter. In 10th ACM Symposium on Virtual Reality Software and Technology, Lorna Herda, Pascal Fua, Ralf Plänkers, Ronan Boulic, and Daniel Thalmann. Skeleton-based motion capture for robust reconstruction of human motion. In Proc. Computer Animation, [OBBH00] James F. O Brien, Robert E. Bodenheimer, Gabriel J. Brostow, and Jessica K. Hodgins. Automatic joint parameter estimation from magnetic motion capture data. In Graphics Interface, [RL02] [SLH91] [SPB + 98] [TGB99] [Vic] [vlvr03] [ZH03] Maurice Ringer and Joan Lasenby. A procedure for automatically estimating model parameters in optical motion capture. In British Machine Vision Conference, pages , Guy L. Scott and H. Christopher Longuet-Higgins. An algorithm for associating the features of two images. In Proc. R. Soc. London, volume 244, pages 21 26, Marius-Calin Silaghi, Ralf Plänkers, Ronan Boulic, Pascal Fua, and Daniel Thalmann. Local and global skeleton fitting techniques for optical motion capture. Lecture Notes in Computer Science, 1537:26 40, Deepak Tolani, Ambarish Goswami, and Norman I. Badler. Real-time inverse kinematics techniques for anthropomorphic limbs. Graphical Models, (62): , Vicon iq, Vicon Motion System Ltd, Robert van Liere and Arjen van Rhijn. Search space reduction in optical tracking. In A. Kunz and J. Deisinger, editors, Ninth Eurographics Workshop on Virtual Environments, number 9, Victor B. Zordan and Nicholas C. Van Der Horst. Mapping optical motion capture data to skeletal motion using a physical model. In D. Breen and M. Lin, editors, Eurographics/SIGGRAPH Symposium on Computer Animation, 2003.