Evolutionary Behavior Acquisition for Humanoid Robots

Transcription

1 Evolutionary Behavior Acquisition for Humanoid Robots Deniz Aydemir 1, Hitoshi Iba 1 1 Graduate School of Frontier Sciences, Department of Frontier Informatics Tokyo University, Tokyo, Japan {deniz, iba}@iba.k.u-tokyo.ac.jp Abstract. This paper describes and analyzes a series of experiments to develop a general evolutionary behavior acquisition technique for humanoid robots. The robot s behavior is defined by joint controllers evolved concurrently. Each joint controller consists of a series of primitive actions defined by a chromosome. By using genetic algorithms with specifically designed genetic operators and novel representations, complex behaviors are evolved from the primitive actions defined. Representations are specifically tailored to be useful in trajectory generation for humanoid robots. The effectiveness of the method is demonstrated by two experiments: a handstand and a limbo dance behavioral tasks (leaning the body backwards so as to pass under a fixed height bar). 1 Introduction The recent remarkable progression of robotics research makes highly precise and advanced robots available today. Despite the availability of sophisticated robots, acquisition of behavioral tasks remains as a big hurdle in the field. Currently, several approaches are prominent in evolutionary behavior acquisition. [1], [2], [3] investigate appropriate neural network architectures using genetic algorithms for the adjustment of network parameters. Authors of these papers try to evolve behavioral tasks mainly based on navigation in a constructed environment for the wheeled robot Khepera. Although the results from these experiments are promising in terms of conceptual findings, there exist very few applications of the neuro-evolutionary techniques for more complex and high mobility robots such as humanoid robots. Another approach in evolutionary behavior acquisition is evolutionary gait optimization undertaken by the authors [4], [5]. These experiments involve optimization of a readily available controller for a previously specified behavior, such as quadruple walking. Main drawback here is the assumption that a hand designed controller is readily available. In this paper, we take a slightly different approach than the techniques discussed above. Rather than optimizing a hand designed controller or trying to evolve primitive behaviors conceptualized with neural networks, we consider the behavior acquisition task as a combinatorial optimization task where the task at hand is decomposable into primitive actions, and the goal is to find the

2 optimum sequence (behavioral sequence) of those primitive actions which constitute the desired complex behavior. In this regard, evolving feedback controllers is beyond the scope and aim of this paper. Before delving into details of the devised GA architecture we would like to discuss the difficulties and restrictions regarding the humanoid robots. 2 Peculiarities of Humanoid Robots Balancing requirements for biped humanoid robots are governed by complex equations and are mostly specific to the generated motion patterns. One general approach in controlling the balance of a walking biped humanoid robot is Zero Moment Point (ZMP) [7]. ZMP computation requires the precise knowledge of robot dynamics, center of mass location and inertia of each link involved in the motion pattern. Another approach which requires relatively limited knowledge of robot dynamics is Inverted Pendulum Model (IPM). However, IPM is inapplicable in cases where the foot must be placed in specified locations during the phase of a motion (Fig.1a.) In order to resolve this issue, hybrid approaches, combining ZMP and IPM methods are also proposed [8]. The main difficulties with these approaches are the need for the precise knowledge of robot dynamics which is not available all the time, and the customization or in some cases redesign of the dynamics models for each individual motion. Moreover, ZMP based approaches are not directly applicable in situations where the robot s feet have no contact with the ground as demonstrated in the handstand task (Fig. 1b) or in case of interacting with a third object such as kicking a ball (Fig. 1c) (a) (b) (c) Fig. 1. Typical examples where traditional methods fall short. (a) Walking on specified locations (b) Handstand task (c) Ball kicking There exist also motion generation techniques based on Interactive Evolutionary Computation (IEC) for people who have no specialized knowledge of humanoid robots [9]. However, IEC methods suffer from the subjective evaluation criteria incurred by the human factor involved and still require the developer to account for the inverse kinematics equations governing the balancing issues between the key frames of a motion [10]. Addressing these issues, we devise a general learning scheme based on genetic algorithms which requires only minimum knowledge of humanoid robot dynamics and the balancing requirements of a particular motion pattern. With the proper selection of a primitive action, the genetic algorithm

3 implicitly accounts for the balancing requirements and the acquisition of desired behavior, where traditional methods would require custom balancing methods based on ZMP and/or IMP along with complicated forward and inverse kinematics computation. The proposed method has the following advantages over the traditional methods. Alleviates the need for ZMP, IMP based balancing methods Does not require inverse and forward kinematics calculations Does not require a precise knowledge of the robot dynamics and joint interactions Originality and creativity in behaviors achieved through evolutionary process In the upcoming section we present our approach for finding an effective and compact representation to be used in the evolutionary algorithm for the humanoid robot joint controllers. 3 GA Architecture 3.1 Problem Representation The representation of a joint controller makes use of two significant trajectory path characteristics. Trajectory paths are continuous and mostly sinusoidal in nature. Considering these two properties, a joint controller is represented as a sequence of allele pairs denoting the transition point and type of motions in terms of primitive actions. In physical terms, transition points correspond to places where the derivation of a trajectory curve changes, i.e. the point where the direction of a motion alternates or stops. When composed of primitive actions, a transition point is defined as a change in the primitive action type belonging to a behavioral sequence. For example, for the chromosome in Fig. 2, gene locations 3 rd, 6 th, 9 th positions represent the transition points for a 10 unit time motion of a particular trajectory. (0,1) (3,0) (6,1) (9,0) Fig. 2. A chromosome using absolute timings for transition points in joint trajectories, and a binary field to denote the type of transition Based on this definition, transition points of primitive actions in the behavioral sequence and the type of transition (positive, negative or still) is represented as a gene in the form of an allele pair (t τ, p i ). This defines a set of primitive actions starting from time t τ bounded by the next gene pair s transition time field t τ+1 in the chromosome. Parameter p i denotes the type of transition based on the previous p i-1 value except for the initial p 0. The first transition type value p 0 exceptionally defines the type of the first primitive action, regardless of the previous transition type value. Since a binary field is used for the transition type field, each joint involved in a behavior must initiate a rotation either with a positive or negative slope, i.e. initially

4 no joint is allowed to stay still. The main advantage of this representation is the fact that using absolute timings for transition points alleviates the need to keep the genes in sorted order. So the order of genes is irrelevant of their interpretation and the representation is not susceptible to fixed locus assumption [12]. Another advantage is the relatively easy handling of boundary conditions for the primitive action timings. As the transition timing is bounded by the experiment time, there is no processing necessary to adjust the transition points. Finally, restricting the problem representation to only sinusoidal nature motions effectively reduces the search space explored by the evolutionary algorithm. 3.2 Mutation Operator The mutation operator is the customized version of the single point mutation which is applicable to allele pair locations. With a given probability p mut, a specific transition point is perturbed with discrete values ranging from a negative lower bound to a positive upper bound. The pseudo code given below describes this variation process for the transition time field of a gene. N denotes the maximum value the transition time field can take. As for boundary conditions, each chromosome is thought to be circular. A modulus operation is applied after the perturbation of the gene location containing the transition point which removes the edge effects. Pseudocode for range mutation operator. Range_mut( offspring, pmut, lower, upper ) L = length( offspring ) for( i = 0 to L do ) prob = random( 0, 1 ) if( prob > pmut ) var = random_int( lower, upper ) offspring[i].time = ( offspring[i].time + var + N ) mod N endif done 3.3 Postprocessing of a chromosome Since chromosomes accommodate absolute timings for trajectory transition points, the very first gene defining the rotation of a joint for the initial time step may disappear through the evolution process due to the genetic operations applied. Similarly, duplicate genes can occur inside a chromosome due to the same reason. The strategy employed for ensuring the existence of the initial gene is to find the gene with a minimum transition time field inside the chromosome and reset the field to zero. To eliminate the duplicate genes, simply repeat the genetic operation causing duplicates until we get distinct transition time fields for each allele pair inside the chromosome.

5 3.4 Fitness Evaluation Scoring of individuals is done using these components as shown in eq.(1) and (2). In eq.(1), waist, chest and ankle altitudes are simply summed up to evaluate the fitness of individuals for the handstand behavior. Following this calculation, individuals having a waist altitude greater than their ankle altitudes are given a penalty proportional to the waist altitude gained. This penalty is used to eliminate the individuals trying to gain fitness by only raising their chest and waist altitudes in the early generations. Table 1. Components of the fitness evaluation for the handstand task Variable Name waist_alt chest_alt ankle_alt Value Waist Altitude(cm) Chest Altitude(cm) Ankle Altitude(cm) Fitness = waist_alt + chest_alt + ankle_alt. (1) Fitness = Fitness - waist_alt. (if waist_alt > ankle_alt ). (2) For the limbo dance behavior, components used in the fitness evaluations are listed in Table 2. First of all, a target point in the three dimensional space is chosen just behind the humanoid robot specified by the coordinates target_x, target_y and target_z. The Euclidian distance between this target point and the center of the arm ankle segment of the humanoid robot is calculated as given in eq.(3). Accommodating this calculated distance and a constant k1, a minimization procedure of the Euclidian distance between the robots arm and the target point is undertaken in eq.(4). Constant k2 in eq.(4) is used such that, k2 > elapsed * k3, to separate fall situations from the stable ones. Table 2. Components of the fitness evaluation for the limbo dance task Variable Name arm_pos_x arm_pos_y arm_pos_z target_x target_y target_z elapsed Expression X coordinate of arm ankle Y coordinate of arm ankle Z coordinate of arm ankle X coordinate of target position Y coordinate of target position Z coordinate of target position Elapsed simulation time Euclid_dist = Distance( arm_pos, target ). (3) Fitness = ( k1 Euclid_dist ) + k2. (4) Fitness = elapsed * k3 ( if fall ). (5)

6 4 Experiments and Results We performed two experiments in order to show the applicability and generality of our approach. The first behavior attempted is a handstand behavior and the second is limbo dance behavior. 4.1 Robot Simulation Environment We use the simulation environment available from Open Architecture Humanoid Robotics Platform (OpenHRP) which consists of a simulator and motion control library of humanoid robots [13]. Humanoid robot HOAP-1 manufactured by Fujitsu Automaton Limited is used for the experiments. HOAP-1 has 20 degrees of freedom (DOF). Robot is about 6kgs in weight and 48cms in height. 4.2 Handstand Behavior The main goal in the handstand task is to properly raise the legs while balancing the whole body on the hands and optionally the forehead. Joints evolved for this behavior are waist, arm and knees. Main parameters and experiment settings are shown in Table 3. Since the trajectories are long and require less precision, the primitive action is allowed to have a coarse granularity. A chromosome is designed for each degree of freedom for all joint types as given in Fig.1. However, instead of designing a pair of chromosomes for symmetrical joints, one joint is represented by a single chromosome and the symmetrical reflection is taken for the symmetric joint on the other side of the body. This effectively reduces the required number of chromosomes by half. Table 3. Genetic algorithm parameters and experiment settings for the handstand task Parameter Value Population size 100 Generation size 50 Chromosome Length 30 Selection Roulette wheel Range mutation probability 0.1 Crossover probability 0.9 Primitive action 0.27radian/s (open loop)

7 Fig. 3. Significant moments from the handstand behavior experiment of the best individual evolved. Initial position and screenshots from the 4 th, 8 th, 12 th, 16 th, 20 th seconds are displayed The acquired handstand behavior can be said to have several human like properties. First of all, the robot wide opens its arms to properly balance itself starting from the middle top frame in Fig. 3. Next, in the down left frame, again using the arm joint, the robot attains more height by closing the arm joints. Lastly, the robot bends its knees while raising its legs, possibly not to fall, and then finally stretches its knees to attain more altitude, in the down right frame. 4.3 Limbo Dance Posture The main objective of the dance is to lower the upper body along with hips and knees to walk under a fixed height horizontal bar. For the humanoid robot, the task is simplified to achieving the necessary posture to pass under the bar. (a) Simulation

8 Fig. 4. Simulation and real experiment results for the limbo dance task. For the limbo dance behavior, a counter intuitive and an unpredictable result arose. One would expect the humanoid robot to initially bend its upper body backwards to achieve the necessary limbo posture. However, as can be seen in top middle frame in Fig. 4, the robot initially bends its body forward, evidently to compensate for the improper balancing caused by the rotation of knee joints. This behavior may as well be attributed to the fact that the battery pack attached to the back of the humanoid robot forbids an initial backward lean and initially mandates a forward lean to shift the center of mass forward in order to keep the balance in the following steps of this behavioral sequence. Despite this unexpected constraint, the humanoid robot successfully learns the complex behavior governing the interactions among the knee, ankle and waist joints, using this exceptional balancing strategy to achieve the necessary limbo posture in a stable manner. 5 Discussion Empirical results regarding the handstand and limbo dance behavior suggest that both the evolutionary architecture and in particular the problem representation look promising as a possible solution to address problems in complex behavior acquisition for humanoid robots. Experiments showed that the method is applicable to different behavior acquisition tasks with minor changes. Moreover, the behaviors acquired by the humanoid robot surprisingly bears resemblance to human designed controllers and at some points surpasses the ideas and the predictions existent in a hand designed program. This became especially apparent in the limbo dance behavior when the robot unexpectedly started with a forward body lean to keep its balance although the objective of the behavior is to attain a backward lean.

9 (a) (b) (c) Fig. 5. Best fitness values for (a) handstand task and (b) limbo dance task. Fall rates for (c) limbo dance task and (d) handstand task Comparative results are also provided with a next ascent hill climbing algorithm in Fig. 5. Solutions obtained with the hill climbing algorithm tend to have low fall rates when compared to the genetic algorithm used as shown in Fig. 5c and 5d. However, results in Fig. 5a show that next ascent hill climbing method is incompetent for the handstand task and the solutions improve only up to 3 rd generation, finally ending up with a stable but premature posture. As for the limbo dance task, hill climbing algorithm shows better performance until the 20 th generation with a low fall rate. However, after this point the genetic algorithm generates better solutions than the hill climbing algorithm, with the best individual having 5cm higher fitness value as shown in Fig. 5b. (d) 6 Conclusion and Future Research In this paper we presented a general approach in evolutionary behavior acquisition for high mobility robots. The empirical results show clearly that evolutionary algorithms with problem specific representations possess the potential of achieving more than a satisfactory level of success in high level behavior acquisition tasks for humanoid

10 robots. The experiments have also demonstrated that genetic algorithms can be used to evolve complex behaviors without the need for understanding the detailed dynamics and physics of the humanoid robot and the desired behavior. Another significant result is the observed creativity and the originality in the behaviors which are comparable to human designed controllers. For future work, we are planning to include the action granularity and dynamics parameters of the joints into the learning process to evolve more complex behaviors. Furthermore, we would like to conduct experiments for planning tasks such as [14] using the behaviors learned here as primitives and transferring the simulation results on the real robot for the handstand task as well. References 1. I. Harvey, P. Husbands, and D. Cliff, Issues in Evolutionary Robotics, Proceedings of the Second International Conference on Simulation of Adaptive Behavior (SAB92), MIT Press, Cambridge, MA (1993) I. Harvey, P. Husbands, D. Cliff, A. Thompson, and N. Jakobi, Evolutionary robotics: The Sussex approach, Robotics and Autonomous Systems (1997) S. Nolfi, Evolving non-trivial Behaviors on Real Robots: A garbage collecting robot, Robotics and Autonomous Systems (1997) S. Chernova, M. Veloso, An Evolutionary Approach To Gait Learning For Four-Legged Robots, Robots, In Proceedings of IROS, Sendai Japan (2004) 5. T. Rofer, Evolutionary Gait-Optimization Using a Fitness Function Based on Proprioception, Lecture Notes in Artificial Intelligence, Springer (2005) S. Nolfi, Evolutionary Robotics: Exploiting the full power of self-organization, Connection Science, (1998) 10(3 4) 7. K. Nishiwaki, S. Kagami, Y. Kuniyoshi, M. Inaba, and H. Inoue, Online generation of humanoid walking motion based on a fast generation method of motion pattern that follows desired zmp, Proceedings of the 2002 IEEE/RSJ Intl. Conference on Intelligent Robots and Systems, EPFL, Lausanne, Switzerland (2002) S. Kajita, F. Kanehim, and K. Kaneko, "Biped Walking Pattern Generation by using Preview Control of Zero-Moment Point, Proc. on the ICR.4 (2003) H. Takagi Interactive Evolutionary Computation: System optimization based on human subjective evaluation, IEEE Int l Conf. on Intelligent Engineering System (1998) T. Yanase, and H. Iba, Evolutionary Motion Design for Humanoid Robots, In Proc. of the Genetic and Evolutionary Computation Conference (2006) (To appear) 11. G. P. Wagner and L. Altenberg, Complex adaptations and the evolution of evolvability, Evolution (1996), vol. 50, no. 3, D. E. Goldberg, K. Deb, H. Kargupta, and G. Harik, Rapid, accurate optimization of difficult problems using fast messy genetic algorithms, in Proc. 5th Int. Conf. on Genetic Algorithms. San Mateo, CA: Morgan Kaufmann (1993) F. Kanehiro, S. Kajita, and H. Hirukawa "OpenHRP: Open Architecture Humanoid Robotics Platform," The International Journal of Robotics Research (2004) vol. 23, No S. Kamio and H. Iba, "Random Sampling Algorithm for Multi-agent Cooperation Planning," Proc. of IEEE/RSJ International Conference on Intelligent Robotics and Systems (2005)