Vision-based Walking Parameter Estimation for Biped Locomotion Imitation

Vision-based Walking Parameter Estimation for Biped Locomotion Imitation Juan Pedro Bandera Rubio 1, Changjiu Zhou 2 and Francisco Sandoval Hernández 1 1 Dpto. Tecnología Electrónica, E.T.S.I. Telecomunicación Universidad de Málaga, 29071-Málaga, Spain, 2 School of Electrical and Electronic Engineering, Singapore Polytechnic, 500 Dover Road, Singapore 139651 Abstract. This paper proposes a new vision-based system that can extract walking parameters from human demonstration. The system uses only a non-calibrated USB webcam connected to a standard PC, and the human is only required to put three color patches on one of his legs and walk roughly in a perpendicular plane with respect to camera orientation. The walking parameters are then extracted in real time, using a local tracking system to follow the markers and a fast decision layer to detect the main features of the leg movement. As only one leg can be tracked properly using only one camera, we assume symmetric movement for left and right legs. Once extracted, the parameters have been successfully tested by generating walking sequences for both simulated and real Robo-Erectus humanoid robots. 1 Introduction Biped Locomotion has become a key topic in robotic research over the last decade. Both hardware improvements and new software architectures have allowed the implementation of impressive humanoid robots such as Honda ASIMO, Sony QRIO or Fujitsu HOAP-2. Many universities have begun research in humanoid robotics, and some very interesting prototypes have appeared over the last years. Most of them use one of the two main approaches to biped locomotion: static or dynamic. Both approaches will be reviewed in Section 2. The main objective, in any case, is to produce a gait as natural and stable as possible. One of the most popular and efficient methods to generate these natural gaits is imitation. As humanoid robots are inspired by the properties of a biological system -humans- it is not far-fetched to consider biologically inspired computation methods for them. The idea is that the movements of a humanoid robot will be more natural when produced through imitation of a human teacher. This process of imitation can be simplified by extracting suitable characteristics of the human motion [1]. If we focus in the walking process, a common assumption is to consider it as a periodic movement. The postures at the beginning and the end of each step have to be identical to have a continuous and repeatable gait [3]. This requires the

selection of specific initial conditions, constraint functions and their associated gait parameters. However, this gait will not usually be human-like. A common solution to this issue is the use of Human Motion Capture Data (HMCD) to drive the robot [5]. Nevertheless, some researchers show that the HMCD cannot be applied directly to humanoid robots due to kinematic and dynamic inconsistencies between the human subject and the humanoid. Kinematic corrections are usually required while calculating the joint angle trajectory [2]. Adaptation of HMCD for a humanoid robot is commonly made with periodic joint motion corrections at selected joints to approximately match the desired Zero Moment Point (ZMP) trajectory. Constrained optimization methods can be used to maximize the dynamic stability against sliding during walking. In any case, it is very complex to adapt the trajectories from human to humanoid. The main problems arise from differences between human and humanoid bodies, stability, and noise during the perception stage. In order to get as good reference trajectories as possible, commercial HMCD systems can be used. However, most of them are very expensive and intrusive. They usually require special suits or markers, fine calibration and specific environments. This paper introduces a new approach to the adaptation problem. The proposed method relies on parameter extraction to facilitate the gait conversion from human to humanoid. This means that perceived trajectories will not be used directly to create synthetic gait. Instead of this, the system analyzes the perceived motion and extracts the main walking parameters from it. These parameters are then used to create motion for the humanoid. If the set of extracted parameters is sufficient to characterize the gait, the generated movement will imitate correctly the perceived one [3][6]. The proposed method does not require a complex perception system to extract the base parameters. The HMCD used for our system was collected in the ARICC Lab at Singapore Polytechnic, using a non-calibrated USB webcam. The human performer only need to wear three color patches roughly placed on hip, knee and ankle. 2 Biped Locomotion and Key Walking Parameters Biped motion is generally divided into a Single Support Phase (SSP), when only one foot is on the ground, and a Double Support Phase (DSP), when both feet are on the ground. In ordinary human gait, the DSP lasts for approximately 20% of the step cycle. There are two types of biped gait: static and dynamic. Static walkers rely on the static equilibrium condition: maintain the Center of Gravity (CG) on the convex hull of the contact area with the ground. This approach denies inertial forces and therefore can be applied only if robot movements are very slow. Dynamic walkers achieve fast and natural walking motion following the principle of dynamic equilibrium: they use ZMP instead of CG, so that inertia components and gravity are considered. This is the approach adopted in this paper. The process of gait generation can be simplified if we just generate a synthetic gait from key walking parameters. These parameters can be seen in Fig. 1.a and

are deeply explained in [3]. There are four time-related parameters, related to step period, SSP and DSP intervals, and time of maximum ankle height. The detection of these key moments in the walking process will be one of the main objectives of the perception stage. The rest of parameters are related to lengths and distances. Fig. 1. a) Walking parameters and coordinates, b) Joint positions and angles 3 System Overview Our system uses a tracking stage to estimate joint positions (Fig. 1.b) in realtime. The outputs of this stage are the centroids of the detected markers. This information is used as input for a decision algorithm that determines the walking phase, and detects transitions between phases. The moments in which these transitions occur, and the joint positions at these moments, are used to extract the walking parameters. Finally, these parameters are tested by generating simulated and real robot walking sequences and comparing them with the perceived ones. 4 Perception System Our perception system relies on a fast and efficient color-based tracking algorithm to extract the movement of a human leg. We instrument the human with three color patches located in the hip, knee and ankle of one of his legs. The tracked human is also supposed to walk more or less in a perpendicular straight line with respect to camera orientation. The perception system is divided into two stages. The first stage tracks the color markers, and reduces drastically the amount of information the system is

using, from a video sequence to just the (x, z) positions of the three markers in each frame. The second stage uses these positions to determine the phase of the movement. 4.1 Tracking Algorithm The algorithm we are using to track the color markers was proposed in [4], and it relies on a hierarchical representation of the regions of interest to track these regions in real time. The basis of the algorithm is to use a weighted template to follow each object. This template changes not only its position, but also its shape so that it can successfully handle partial occlusions, and perspective and physical deformations. The method represents both target and template using a Bounded Irregular Pyramid (BIP). This structure is a hierarchical approach to tracking that reduces very much the computational cost and allows the system to track the three leg markers in real time. The only difference between the original tracking algorithm and our implementation is that we have used Hue-Saturation-Intensity (HSI) color space, more suitable to track saturated colors. See [4] for further details. 4.2 Decision Algorithm The positions of hip, knee and ankle are used to extract walking parameters from the HMCD. As these parameters are related to walking phases, the proposed system must be able to detect these phases and, specially, the transitions between them. Each step can be roughly divided into three stages: the first is the Double Support Phase (DSP). The second and third phases are contained in the Single Support Phase (SSP). The decision stage will determine when the leg is changing from one phase, or state, to another (Fig. 2). More concretely, the decision will be related to ankle movement. A previous average filter is applied before decision to reduce noise. Alternative approaches were tried such as computing joint angle trajectory minima, and other measurements on relative joint distances. However these have appeared to be more sensitive to noise, as the inertia of markers and noise affects more the distance and angle information. Fig. 2. Decision algorithm states

5 Extraction of Walking Parameters Our approach to walking process considers it as a symmetric, periodic and smooth motion. The hip, knee and ankle trajectories, both in Cartesian and Joint spaces, can be considered as smooth curves characterized by a set of key points [3]. These key points should be identified in the transition moments where the state of the leg is changing. So changes detected in the decision stage become the key moments in which walking parameters can be extracted. These parameters, already introduced in Sect. 2, can be divided into two main groups: time parameters and position parameters. 5.1 Time Parameters Time parameters are directly obtained from the decision stage. We consider that each step begins with a DSP followed by a SSP. Being T (kab) the time instant, in step k, in which the leg moves from state a to state b, time parameters are obtained as detailed below: T s.thessp period is obtained using this equation: T s = T (k12) T (k20) (1) T d. The system is tracking only one leg. This means that there is no direct way to obtain the DSP period as it is defined as the time between the SSPs of both legs. If we assume a symmetric walking pattern, then T d can be obtained from the moment in which the tracked ankle stops: T d =(T (k20) T ((k 1)12) T s )/2 (2) T m. This is the moment in which the ankle reaches its maximum height, measured from the beginning of the previous DSP. This is easily obtained from the transition between states 0 and 1: T c. This is the step period, T c = T d + T s. T m = T (k01) T (k20) + T d (3) 5.2 Position Parameters These parameters (Fig. 1.a) are obtained in the same way as time parameters, as the main information about joints motion is also related to transition moments: S l. The step length is the horizontal motion of the tracked ankle during states 0 and 1: S l = x a (k12) x a (k20) (4) where x a (kab) denotes the horizontal (x) value for the ankle position in step k, in the transition from state a to state b.

Ha0 and La0. Both parameters are related to transition between states 0 and 1: Ha0= z a (k01) z a (k20) ; La0 = x a (k01) x a (k20) (5) Xsd and Xed. These parameters are distances between hip and ankle in key instants. The system extracts these parameters at the beginning and the end of the SSP: Xsd = x a (k20) x h (k20) ; Xed = x a (k12) x h (k12) (6) Hm max and Hm min. The extraction of these parameters is slightly different, as they are not related to transition moments [3]. Hm max and Hm min are only required to produce a more natural motion in the simulator and real robot. The perception stage will obtain them as the maximum and minimum vertical hip positions during the human demonstration. 6 Experiments and Results The proposed system has been tested in an indoor environment, using pink color patches as markers. The algorithm runs on a standard PC and the images are captured using a Ranger VCAM365 commercial webcam. Our approach does not require the camera to be calibrated. Besides, no specific illumination conditions nor specific background have been used. The HMCD has been used to generate walking sequences over a Simulator developed at ARICC [6]. These sequences have been also successfully adapted to a real Robo-Erectus robot [7], also developed and built at ARICC, in Singapore Polytechnic. Table 1 shows the walking parameters extracted from a perceived walking motion. Time parameters are in seconds, position parameters are in pixels. After perception, all position parameters are normalized with respect to perceived shank length. When these parameters are translated to the robot simulator or the real robot, they are multiplied by the value of shank length that simulator or robot are using. Table 1. Walking parameters extracted from human motion Parameter Value Normalized Value Parameter Value Normalized Value T c 2.1325 2.1325 Ha0 3 0.069767 T s 0.9690 0.9690 La0 12 0.279070 T d 1.1635 1.1635 Xsd 12 0.279070 T m 1.6165 1.6165 Xed 8 0.186047 Shank 43 1 Hm max 71 1.651163 S l 26 0.604651 Hm min 66 1.534884 The parameters presented in Table 1 were used to generate a simulated walking sequence. The horizontal motion of hip and ankles in the simulator is shown

in Fig. 3.a. By extracting only the relevant parameters and reconstructing the trajectories the system is able to synthesize a low noise smooth and stable motion. On the other hand, Fig. 3.b presents the trajectories in joint space for the previous movement. We can see here how it would be much more difficult to extract the walking parameters in this space, because angle trajectories are not as clearly associated with walking parameters as joint displacements. Fig. 4 shows the perceived walking cycle. Although there are evident similarities between real and generated cycles (so that real and artificial movements will be very similar), the perceived walking sequence is much more noisy, and non periodical. Besides, we must also consider the possibility of losing markers. All this justifies the extraction of parameters as opposed to direct trajectory imitation. Fig. 3. Simulated walking cycle: a) Horizontal displacements of the hip and both ankles, and b) Joint angle trajectories 7 Conclusions and Future Work This paper has presented a novel approach to extraction of walking parameters from a perceived video sequence. The chosen parameterization allows to apply the extracted parameters on the simulator or real robot. The results show that generated sequences are rather similar to real ones. The proposed method requires only a standard, uncalibrated digital camera and three color patches. Future work will address on-line balance compensation and imitation. We will also collect human walking data under various behavioral conditions. Parameter estimation techniques will be tested to extract more effectively human walking parameters for humanoid control and imitation. We will also conduct research on the use of periodic joint motion corrections at selected joints to dynamically match the desired ZMP trajectory.

Fig. 4. Perceived walking cycle: a) Horizontal displacements of the hip and right ankle, b) Joint angle trajectories 8 Acknowledgement This work has been partially supported by the Spanish Ministerio de Educación, Cultura y Deporte (MECD), project number TIN2004-05961. References 1. Aggarwal, J.K., Slotin, J.: Human motion analysis: a review. Computer Vision and Image Understanding, 73, 3, (1999) 295-304 2. Dasgupta, A., Nakamura, Y.: Making feasible walking motion of humanoid robots from human motion capture data. In Proc. IEEE Intl. Conf. on Robotics & Automation, Detroit, Michigan, (1999) 1044-1049 3. Huang, Q., Yokoi, K., Kajita, S., Kaneko, K., Arai, H. Koyachi, N., Tanie, K.: Planning walking patterns for a biped robot. IEEE Trans. Robot. Automat., 17, (2001) 280-289 4. Marfil, R., Rodriquez, J.A., Bandera, A., Sandoval, F.: Bounded irregular pyramid: a new structure for color image segmentation. Pattern Recognition, 37, 3, (2004) 623-626 5. Safonova, A., Pollard, N., Hodgins, J. K.: Optimizing Human Motion for the Control of a Humanoid Robot. In Proc. 2nd International Symposium on Adaptive Motion of Animals and Machines (AMAM2003), Kyoto, Japan, (2003) 6. Tang, Z., Zhou, C., Sun, Z.: Gait planning for soccer playing humanoid robots. Lecture Notes in Control and Information Sciences, Springer-Verlag, 299, (2004) 241-262 7. Zhou, C., Yue, P.K.: Robo-Erectus: a low cost autonomous humanoid soccer robot. Advanced Robotics, 18, 7, (2004) 717-720