Can Body Language Shape Body Image?

Transcription

1 Can Body Language Shape Body Image? Luc Steels,, Michael Spranger University of Brussels (VUB AI LAB) Sony Computer Science Laboratory Paris Abstract One of the central themes in autonomous robot research concerns the question how visual images of body movements by others can be interpreted and related to one s own body movements and to language describing these body movements. The discovery of mirror neurons has shown that there are brain circuits which become active both in the perception and the re-enactment of bodily gestures, although it is so far unclear how these circuits can form, i.e. how neurons become mirror neurons. We report here further progress with our robot experiments in which a group of autonomous robots play language games in order to coordinate their visual, motor and cognitive body image. We have shown that the right kind of semiotic dynamics can lead to the self-organisation of a successful communication system with which robots can ask each other to perform certain actions. The main contribution of this paper is to show that if the robot has the capacity to imagine the behavior of his own body through selfsimulation, he is better able to guess what action corresponds to a visual image produced by another robot and thus guess the meaning of an unknown word. This leads to a significant speed-up in the way individual agents are able to coordinate visual categories, motor behaviors and language. Introduction Starting with work published in an artificial life context more than a decade ago (Steels, 99, 00), we now have quite solid mechanisms that show how a vocabulary may self-organise in a population of embodied agents that lexicalises perceptually grounded categories, such as colors (Steels and Belpaeme, 00) or spatial relations (Steels and Loetzsch, 008). In these experiments, the formation of a language is not just an after-thought, something that takes place after solid concepts are formed. Rather, the formation of categories takes place in intimate co-evolution with the formation of language and both mutually influence each other. The question we address here is whether this approach is relevant to understand the formation of a body-image as well. Body image refers to a collection of representations that embodied agents must maintain in order to move about in the world, plan and execute action, perceive and interpret the behaviors of others, build and use an episodic memory, and understand or produce language about action, for example commands. The body image is not static but dynamically changing over time in concordance with own body movements or body movements of others. The subject of body image has received wide attention, particularly in the neurological literature because of puzzling phenomena such as phantom limbs, mirror box experiments, unusual pain, out of body experiences, etc. (Ramachandran and Hirstein, 998; Rosenfield, 988; Blanke and Castillo, 007) and in the neurobiological literature because of the discovery of mirror neurons (Rizzolatti et al., 996; Rizzolatti and Arbib, 998). These disorders, experiments and neurobiological observations make it quite clear that body image is not a simple, fixed innately given internal representation but a dense network of representations that forms in development and continues to change and be adapted throughout life. The relation between visual representations and recognition of bodily action on the one hand and the bodily action itself has also been intensely studied in robotics research, particularly in research on imitation (Billard, 00; Demiris and Johnson, 00; Mataric, 00). It is moreover a key topic in embodied artificial intelligence (Pfeifer et al., 007; Nabeshima et al., 006) which emphasises the grounding of cognition in bodily activities. In a recent paper (Steels and Spranger, 008) we have already reported experiments with QRIO humanoid robots (Gutmann et al., 00) set up to understand the relation between the visual and motor body image through a particular language game which we call the Action Game. Two agents are randomly chosen from a population and take on the roles of speaker and hearer. They are downloaded in a robot body in order to play a situated embodied language game. The speaker asks the hearer to do a physical action and the game is a success if the hearer indeed performs the requested action as judged by the speaker. If the game fails, the speaker repairs the communication by performing the action himself. Obviously this game can only be played successfully when the agents are able to categorise bodily gestures performed by others based on visual input and relate them to their own motor behaviors that would produce these same gestures. In Artificial Life XI

2 Figure : A humanoid robot stands before a mirror and performs various motor behaviors thus observing what visual body-images these behaviors generate. these experiments, agents start without any prior set of image schemata nor words for bodily actions, and they do not know the mapping from the visual domain to the motor domain. If we observe an increase in communicative success, starting from scratch, then this means that agents have not only self-organised a lexicon for naming possible actions and the image schemata they generate, but that they have learned a bidirectional mapping between the visual and motor domain as well. We will see that this is indeed possible. In the first experiment reported earlier, called the mirror experiment (Steels and Spranger, 008), robots learn the bi-directional mapping between visual body-image and motor behavior by standing before a mirror, executing actions, and observing the visual body-images that they generate. Once all agents in the group have each learned this mapping, they play language games settling on names for these actions. In the second experiment (called the body language experiment), robots do not learn the bi-directional mapping between image schemata and motor body-image through a mirror but through the language game itself. Both experiments are briefly summarised in the next sections of the paper. Then we focus on the role of imagination through self-simulation. As advocated by several researchers, selfsimulation can be used to enhance understanding of the movements of others (Rizzolatti and Arbib, 998; Jeannerod, 00), and thus play a role in language understanding and learning (Feldman, 006; Feldman and Narayanan, 00). In the experiment to be discussed, robots maintain a motor body image of themselves which they update through proprioception. Additionally we equipped the robots with a kinematic model used for simulating the execution of their Figure : Network linking sensory experiences, image schemata of postures, nodes for postures acting as mirror neurons, and nodes triggering the motor behavior that achieves the posture. Nodes for words (shown as squares) are associated with the posture nodes. own actions. By adding a component that is able to inspect this simulated body image, it becomes possible for the robot to imagine to some extent what a particular action looks like and this helps to guess the meaning of unknown words and thus speed up the coordination between visual body image, motor behaviors and language. The Mirror Experiment In the mirror experiment, each robot stands before a mirror in order to acquire the relation between his own motor body-image and (a mirror image) of his own visual bodyimage (see figure ). Our experiments have so far focused on static gestures (postures) which require motor behaviors that each involve typically about 0 motor commands with associated proprioceptive feedback. Because all robots have exactly the same body, a robot can use a visual body image of himself in order to categorise the body image of another robot, after perspective reversal (Steels and Loetzsch, 008). And so once each robot has learned the relation between visual body-image and motor body-image for himself, they are quickly able to settle on a shared vocabulary by playing an Action Game (see figure 7). We approach the problem of body image here as a coordination problem. Agents maintain a semiotic network linking nodes for image schemata for postures with the motor behaviors that they generate, mediated by a posture node which is functioning as a mirror neuron (see figure ). Similar to research by Triesch et al. (007) we assume that there is nothing special about mirror neurons but that neurons become mirror neurons because they take on a particular role in networks. The vision system of the robot performs foreground/background segmentation and feature extraction in Artificial Life XI

3 Figure : Aspects of visual processing. From left to right we see the source image, the foreground/background distinction, the result of object segmentation (focusing on the upper torso), and the feature signature for successive frames of this posture based on centralised moments. terms of centralised moments (Mukundan and Ramakrishnan, 998, see figure ). Values for each of these features are combined into a feature vector that constitutes the sensory experience of a perceived body image at a particular moment in time. The specific visual feature vectors are then classified using image schemata. An image schema is a feature vector with the same dimensions as the sensory experience and with a typical point as well as maximum accepted deviations from this point. The best matching prototype is found by distance computation and then the prototype is adjusted to better assimilate the new experience. When no prototype is matching or a new action is selected and executed, a new one is created, with the sensory experience being the first seed. Each image schema is linked to a posture node which is also linked to a motor behavior which can achieve that particular posture when executed. When one robot is watching another robot, he performs a perspective reversal operation on the visual image, in the sense that the robot computes the position of the other robot and then perform a geometric transformation, so that the position of left and right arm now matches with his own. For example, if one robot stands in front of another one, this means a 80 degree transformation. The inventory of image schemata and their relation to sensory features is acquired using a prototype based approach and the links between posture nodes and motor behaviors through kinesthetic teaching. When a robot stands before the mirror, he selects a posture and activates the corresponding motor behavior. This motor behavior generates a sensory image which is categorised with a particular image schema. And so through standard Hebbian learning, which enforces the connection between nodes that are simultaneously active (Hebb, 99; Bishop, 99), the link between the image schema and the posture gets established and progressively enforced. Once this network is established, the development of a shared lexicon in a population of agent is straightforward, based on the well known lateral inhibition dynamics of the Naming Game (Steels, 99). Results of this experiment for a population of 0 agents are shown in figure. The graphs show the global behavior of the population, after each individual has coordinated motor behavior and visual body-image through the mirror. Figure : Results of the Action Game played by a population of 0 agents for 0 postures. Robots have first coordinated their visual body-images and motor behaviors by standing before a mirror and observing their visual appearance in relation to certain motor behaviors. The x-axis plots the number of language games. The running average of communicative success and average lexicon size are shown as well as invention and adoption frequency. 00 % success is reached easily after about 000 games. Already after 900 games there is more than 90 % success. The graph shows the typical overshoot of the lexicon in the early stage as new words are invented in a distributed fashion followed by a phase of alignment as the agents converge on an optimal lexicon. Figures and 6 show snapshots of the semiotic dynamics for the first 00 games. Figures shows the communicative success, lexicon-size, and invention and adoption frequency. Figure 6 we show the average score in the population for different words competing for the same meaning. We see clearly that a winner-take-all situation arises after about 00 games, with one word dominating for naming this particular posture. Coordination without mirrors We have seen that once coherent links exist between the image schema for a posture and the motor behavior that generates this posture (here mediated by the posture nodes acting as mirror neurons), it is straightforward for a group of agents to self-organise a lexicon of names which can be used both for describing a particular posture or as commands for asking another robot to achieve it, and thus for playing the Action Game. We now report a second earlier experiment where robots coordinate visual body-image and motor behaviors through language without using a mirror first and without using posture nodes (see Steels and Spranger, 008 for more details). The game is similar to the mirror experiment except that now the two different robots stand immediately in front of each other (see figure 7), without any prior Artificial Life XI

4 Figure : The same graphs as shown in the previous figure, but now zooming in on the first 00 language games. The agents quickly communicate successfully every second game. The population starts to communicate 00% successfully as soon as the number of word drops to the optimal number, by eliminating unsuccessful hypothesis. exposure to mirrors. The speaker asks the hearer to perform an action and there is communicative success if the speaker agrees that the right action has been performed. Moreover the speaker also performs the motor-behavior that is linked to the word. More precisely, the interaction pattern is as follows:. The speaker randomly chooses an image-schema from the known schemas as the topic of the conversation.. He looks up a word associated with the schema. If there is none he invents one.. He looks up the motor-behavior associated with the word. If there is no associated motor-behavior he picks one randomly.. The speaker utters the word and performs the motorbehavior picked.. The hearer parses the uttered word and looks up the image-schema and motor-behavior associated with the word. 6. If there are no image-schemas associated with the word, he classifies the image-schema resulting from the motorbehavior performed by the speaker and associates it with the word. 7. The hearer performs the motor-behavior associated with the word. If there is none he picks one of the known motor-behaviors randomly. 8. Both agents determine the success of the interaction. The speaker determines if the hearer performed the intended Figure 6: The graph shows the average score for all the words competing for the same meaning (the raise both arms action in this case). For every agent of the population the score of all words (scoring higher than 0) is averaged. The data stems from an experiment equal to the experiments depicted in figures and. The winner-take-all dynamics that coordinates the lexicon among the agents is clearly visible. The agents create words for the action, these words become coordinated, with some of dying out and one surviving the competition. behavior. The hearer determines whether the imageschema associated with the uttered word corresponds to the motor-behavior performed by the speaker. That is he compares the visual features created by the action of the speaker with his expected image-schema. Upon determination of success the lexicon of the agents taking part in the interaction is updated. Successful links between network nodes are enforced, unsuccessful links or links to nodes which were not activated are punished (lateral inhibition). Results of this second experiment for a population of agents and postures are shown in figure 8. The graphs show the global behavior of the population focusing on the first 000 language games. 00 % success is reached after about 000 games and stays stable (unless new postures get introduced and then the networks of all agents expand to cope with this). Already after 000 games there is more than 90 % communicative success. The graph shows the typical overshoot of the lexicon in the early stage as new words are invented and a phase of alignment as the agents converge on an optimal lexicon size, which is words to name each of the postures. The frequency of invention and adoption is also shown and dies down as the lexicon stabilises. So clearly agents are able to self-organise a lexicon of commands even if no prior lexicon exists and even if they are not pre-programmed nor learned mappings from motor behaviors to visual image schemata before the lexical process Artificial Life XI

5 Figure 7: Two humanoid robots face each other to play an Action Game. They ask each other to achieve a certain posture, like raise the left arm, or stretch out both arms. The middle image shows the internal motor-body-image obtained by combining proprioceptive streams with the body model. The right image shows two example postures as seen through the camera of another robot. starts. Communicative success does not necessarily mean that agents relate all visual prototypes correctly to the corresponding motor behaviors. However figure 9 shows that the right correspondences progressively emerge. The figure plots the aggregated strength (brightness, black: strongly linked, white no link) of the relation between visual prototypes (y-axis) and motor behaviors (x-axis) in this population over time. Which links are established by agents and which links are consistently established by all agents is shown with two diagrams. The first one (bottom row) shows the evolution of agreement among the agents (computed by taking the product of the strength over all different agents and all experiments) and the second one (bottom row) shows whether agents make any link at all (computed by taking the average strength of all relations between visual prototypes and motor behaviors for all agents for a series of experiments) If there is a single column between a particular visual prototype p i and its corresponding motor behavior m i with strength.0, then this means that all agents have this particular mapping (bottom row) and agree on it (top row). Using Imagining through Self-Simulation We now address the question whether it is possible to improve the efficiency of the overall system. The first step is to realise that agents are dealing with a search problem. When the speaker does not know which motor behavior corresponds to a posture he would like to see achieved, he has to make a guess (step ) and when the hearer encounters an unknown word, he has to choose a motor behavior that could be associated with the visual image he is perceiving (step 7). In the interaction pattern used in the previous experiment, these choices are made randomly, which implies that the choice is correct in only A of the cases, with A equal to the number of possible actions, explaining why we see a worsening of performance as the number of actions increases. We now show how this choice can be improved dramatically by us- Figure 8: Results of the Action Game played by a population of agents for postures (the results of 00 experiments are averaged). Robots have NOT coordinated their visual body-images and motor behaviors by using a mirror but only through language. The x-axis plots the number of language games. The running average of communicative success and average lexicon size as well as invention and adoption frequency are shown on the y-axis. Figure 9: Relation between visual prototypes and motor behaviors for a population of five agents negotiating names for postures. The diagram shows the aggregated strength of the relations over time. Top row: aggregation by product thus showing where they completely agree. Bottom row: aggregation by sum showing which relations have been made. Left column: after 00 interactions. Middle column: after 00 interactions. Right column: after 00 interactions. The diagram shows that agents progressively achieve the correct mappings and agree on them. Artificial Life XI 008 8

6 Figure : Left: The image shows a motor control data stream for creating a waving motion. The information can be used to create an expected motor body image by combining the information the robot has about its body, limb lengths and dimensions, as well as the position of the motors. Given even more information stemming from proprioceptive sensors, a similar representation called the experienced motor body image can be constructed. Given the knowledge about the position of the sensors as well as the aforementioned configuration of the body, representations like those to the right can be computed. The two images on the right show two stages in the simulation of the raise both arms action creating an expected motor body image, as well as an experienced motor body image. Figure 0: Visuo-motor correspondence results from 00 experiments ( agents, actions), after 0000 interactions. Left: aggregation by sum showing which associations have been made overall. Right: aggregation by product showing which associations agents agree upon. In all these experiments agents find always the right mappings and establish the correct visuo-motor connections. ing a simulation-based approach (Feldman, 006; Feldman and Narayanan, 00). The basic idea is straightforward. Each robot maintains an experienced motor body image (see figure 7, middle image) which is based on his own body model (containing information about body parts, their size and shape, how they are attached to each other) and recorded proprioceptive motor streams. In addition to the experienced motor body image, robots can also construct an expected motor body image for a particular action using his own body model, the initial conditions before the action starts, a model of the body kinematics (see figure right) and recorded motor control streams, such as the one shown in figure left, with quantities like RightShoulderPitch, RightShoulderRoll, RightShoulderYaw, RightElbowPitch, etc. This predictive model is already employed in the adaptive control systems of the robot and is now put to a new usage. First of all, the expected motor body image can be decoupled entirely from ongoing behavior so that the robot can simulate a complete action. Of course the simulation will deviate from reality depending on the complexity of the action and the amount of interaction with the environment that is needed. Second, a simulated visual body image can be generated from the simulated motor body image. For example, the motor body image contains information about the angles of the different joints which can be used to internally visualise the position of the joints with respect to the body. Note that this visual imagining is from the robot s own frame of reference. In order to create an expectation of what the same behavior performed by another robot would look like, a third step is needed: The robot has to perform a perspective reversal by performing a geometric transform on this visual image, based on knowing the position of the other robot (Steels and Loetzsch, 008). Finally, similar visual processing and categorisation can be carried out on this simulated visual body image as on the real experienced visual body image of the other robot, specifically centralised moments can be computed again to extract the features needed for gesture categorisation. Given this competence in visual imagination, robots can now improve drastically the quality of their guesses in steps and 7 of the interaction pattern. Speakers can use the simulated visual body image of the other robot to make a better guess of the correct motor-behavior given a visual prototype of the posture they want to see adopted by the other robot. This is done by a hill-climbing process that starts from a random choice of action from the action repertoire, simulates Artificial Life XI 008 8

7 Figure : Influence of simulation on the performance of the agents in the Action Game ( agents, 0 actions). The figure shows the communicative success averaged over 00 different experimental runs, each for a single parameter. 0.0 is the case were the speaker guesses the correct motor-behavior with the base line probability 0..0 means that speaker and hearer guess correctly every time they have to choose a motor-command to match with a desired visual prototype (as speaker) or interpret an unknown word (as hearer). that choice to construct a visual body image of the other robot, and compares this to the desired posture. In case of a failure, a new action is chosen until a reasonable match is found. The hearer goes through a similar procedure if he has to guess the meaning of an unknown word. As mentioned earlier, simulated behavior will always deviate from actual behavior and so this process does not give an absolutely certain guess. Figure shows the important impact on performance of this simulation-based approach. The base-line case (marked 0.0) is the one used in earlier experiments, i.e. where the agent has a A chance to choose the correct action. It shows the slowest rise towards communicative success. The best case (marked.0) is one where the agents manage to always guess correctly both as speaker and as hearer what motor behavior achieves a visual prototype of a posture based on self-simulation, imagination, perspective reversal, and visual categorisation. This case approaches the performance in the mirror experiment where agents first learned the mapping between visual body image and motor body image by standing in front of a mirror. In the intermediary cases we see that that the higher the probably of correct guessing the faster the population reaches full communicative success. Figure shows the semiotic dynamics for the vocabulary of the agents for the same series of experiments. There is always a phase of invention and spreading with a typical overshoot in the size of the vocabulary, which then settles down on an optimal vocabulary as agents align their word meanings. We see that the overshoot of words is much smaller when the quality of guessing improves, which means that Figure : Study of the influence of simulation on the performance of the agents in the game ( agents, 0 actions). The graph depicts the number of words (average lexicon size across all agents) averaged over 00 experiments for a given parameter. 0.0 is the case where the speaker guesses the correct motor-behavior with the base line probability 0..0 means that the speaker guesses correct every time he guesses a motor-command. time to convergence is also much shorter. Conclusions This paper examined the role of body language in the formation of body image, specifically whether Action Games in which agents ask each other to perform actions, can lead to a coordination of image schemata and the motor behaviors that generate them. We have seen that this is indeed the case, even if there is no prior coordination of image schemata and motor behaviors (through a mirror for example) or even if no prior lexicon is given to the agents but they have to selforganise one from scratch. This paper focused on whether overall performance could be improved by allowing agents to imagine what their own bodily movements by simulating. Even if self-simulation is not perfect and only partial information can be gleaned from the motor body image, we have seen that agents have now a better way to guess the meaning of unknown words and hence they can faster zoom in on a system that coordinates different body images of the self and those of others. Acknowledgements This research has been carried out at the Sony Computer Science Laboratory in Paris with partial support from the ECAgents project, funded by the EU Future and Emerging Technologies program (IST-FET) as IST-90, and the ALEAR project, funded by the EU Cognitive Systems program. We are extremely grateful to Masahiro Fujita, Hideki Shimomura and their team at the Sony Intelligent Systems Research Lab in Tokyo for giving the opportunity and support Artificial Life XI 008 8

8 to perform the QRIO experiments, and to Martin Loetzsch for his help in setting up the experiments. The information provided here is the sole responsibility of the authors and does not reflect the EU Commission s opinion. The Commission is not responsible for any use that may be made of data appearing in this publication. References Billard, A. (00). Imitation: a means to enhance learning of a synthetic proto-language in an autonomous robot. In Dautenhahn, K. and Nehaniv, C. L., editors, Imitation in Animals and Artifacts, pages 8. MIT Press. Bishop, C. (99). Neural Networks for Pattern Recognition. Oxford University Press, USA. Blanke, O. and Castillo, V. (007). Clinical neuroimaging in epileptic patients with autoscopic hallucinations and out-ofbody experiences.case report and review of the literature. Epileptologie, : Demiris, Y. and Johnson, M. (00). Distributed, predictive perception of actions: a biologically inspired robotics architecture for imitation and learning. Connection Science Journal, ():. Feldman, J. (006). From Molecule to Metaphor: A Neural Theory of Language. MIT Press. Rizzolatti, G., Fadiga, L., Gallese, V., and Fogassi, L. (996). Premotor Cortex and the Recognition of Motor Actions. Cognitive Brain Research, ():. Rosenfield, I. (988). The invention of memory: a new view of the brain. Basic Books, New York. Steels, L. (99). A Self-Organizing Spatial Vocabulary. Artificial Life, ():9. Steels, L. (00). Evolving grounded communication for robots. Trends in Cognitive Sciences, 7(7):08. Steels, L. and Belpaeme, T. (00). Coordinating perceptually grounded categories through language: A case study for colour. Behavioral and Brain Sciences, 8(): Steels, L. and Loetzsch, M. (008). Perspective alignment in spatial language. In Coventry, K. R., Tenbrink, T., and Bateman, J. A., editors, Spatial Language and Dialogue. Oxford University Press. To appear. Steels, L. and Spranger, M. (008). The robot in the mirror. Connection Science, 0(). Triesch, J., Jasso, H., and Deak, G. (007). Emergence of Mirror Neurons in a Model of Gaze Following. Adaptive Behavior, ():9 6. Feldman, J. and Narayanan, S. (00). Embodied meaning in a neural theory of language. Brain and Language, 89():8 9. Gutmann, J., Fukuchi, M., and Fujita, M. (00). Real-Time Path Planning for Humanoid Robot Navigation. In Int. Joint Conference on Artificial Intelligence (IJCAI), Edinburgh, Scotland. Hebb, D. (99). The Organization of Behavior: A Neuropsychological Theory. John Wiley & Sons. Jeannerod, M. (00). Neural Simulation of Action: A Unifying Mechanism for Motor Cognition. Neuroimage, ():0 09. Mataric, M. (00). Sensory-Motor Primitives as a Basis for Imitation; Linking Perception to Action and Biology to Robotics. In Dautenhahn, K. and Nehaniv, C. L., editors, Imitation in Animals and Artifacts, pages 9. MIT Press. Mukundan, R. and Ramakrishnan, K. (998). Moment Functions in Image Analysis: Theory and Applications. World Scientific. Nabeshima, C., Kuniyoshi, Y., and Lungarella, M. (006). Adaptive body schema for robotic tool-use. Advanced Robotics, 0(0):0 6. Pfeifer, R., Bongard, J., and Grand, S. (007). How the body shapes the way we think: a new view of intelligence. Cambridge, Mass.: MIT Press. Ramachandran, V. and Hirstein, W. (998). The Perception of Phantom Limbs. Brain, (9): Rizzolatti, G. and Arbib, M. (998). Language within our Grasp. Trends in Neurosciences, ():88 9. Artificial Life XI 008 8