1 MOVING EYE INTERACTIVE TELEPRESENCE OVER INTERNET WITH A BALL SHAPED MOBILE ROBOT Panu Harmo, Aarne Halme, Hannu Pitkänen, Petri Virekoski, Matias Halinen,, Jussi Suomela Automation Technology Laoratory, Helsinki University of Technology, Konemiehentie, 0150 Espoo, Finland. Tel. +358 (0)9 4513313, fax +358 (0)9 4513308, e-mail: Panu.Harmo@hut.fi, http://www.automation.hut.fi/iecat/ Astract: This paper descries a rootic and automation system presently under development for studying home automation and indoor rootic technologies as well as new service scenarios ased on these technologies. The system consists of remote users, a home automation server, home automation equipment and a all shaped moile root called Rollo. A remote user connects to the server from the Internet to interact with the root y using an augmented reality user interface. The Rollo root, its ehaviour and functions in the system are discussed. Utilizing the system for oth educational purposes and to develop a remote home aid system is discussed. Copyright 000 IFAC Keywords: telecommunication, rootics, virtual reality, multimedia, remote control 1 INTRODUCTION Moile roots can e useful in many ways in homes and offices monitoring and measuring the environment. They can e used as moving communications platforms or they can e simply toys or companions. More specialized roots can perform services such as cleaning and transportation of small ojects. Indoors roots can est e utilized in an intelligent environment, which comprises of home automation networks, connected sensors, actuators and other devices. Indoor service roots use the local network to exchange information and controls with the other connected devices. A remote user can connect to the local network to retrieve data and to control the root and other devices in the network. In this type environment effective remote interaction with the service root can e implemented y utilizing virtual models and augmented reality. TCP/IP networks use packet-ased transmission and don t guarantee any defined delivery times. The speed of the network depends on the transmission speed and the total traffic in the network. This means that closed loop teleoperation over the Internet is impossile ecause of long and unpredicted time delays. In practice the only choice is so called move and wait principle [Sheridan, 199] which was first used in space teleoperation applications. Vehicle is operated y open loop commands without immediate feedack. After one or more commands have een executed operator waits for the confirmation and the feedack. In virtual presence the operator feels to e present in an environment, which has een artificially generated y computer. Pure virtual environments are usually used only in simulators and games. In teleoperation virtual reality is used usually to augment telepresence; virtual ojects are overlaid on top of real-time video. This is called augmented telepresence (or augmented reality). Augmented reality can e used, for example, for prediction and planning of operations in cases where long time delays distur teleoperation. In prediction mode the estimated (simulated) outcome of the operator s actions are shown virtually. The real actions are shown in the same display after the delay. This way the operator can do direct teleoperation tasks despite of the time delay. The developments in home (and small office) automation are paving the way to home rootics. During the last few years the market for commercial home automation networks and home network gateways has grown, ut no one technical solution has een universally accepted. Lack of standards is hindering the growth. The home server and gateway technologies are important, ecause home roots will communicate with the outside world through these gateways.
SYSTEM OBJECTIVES The oject of our project is to develop the use of augmented telepresence operator interfaces and other relevant technologies for monitoring and interactive telepresence in indoor environments using moile roots and home automation. The practical goal is to uild two systems for two different uses. One system is used for education of university students in mechatronics and rootics. The second system is a remote assistance system for people living at home, ut needing occasional outside assistance. Technologically oth systems have common features although the applications are different. Aside from developing the general system concept, key technologies are identified, evaluated, studied and utilized. The main features of the system are: Two-way communications system etween people and equipment at two distant locations over Internet Video and voice communications Augmented reality user interface Use of virtual models in navigation and path planning Teleoperation of cameras, roots and other devices over networks Access and sharing of documents and data Data collection, storage, management and distriution The central technologies needed to develop a functional indoor service root system are the following: Video conferencing (video and audio codecs and coding standards) Home automation and home automation networks Emerging standard home gateway platforms Moile indoor service rootics and home roots Technologies for virtual models and augmented reality systems In order for us to uild this type of a system the following tasks must e accomplished Development of the root mechanisms and controls Study of its kinematics and controls for remote operation Use of the standard internet technologies for uilding up the system Emedding standard videoconferencing techniques in the user interfaces. Develop and test positioning systems for moving roots, people and equipment. Develop the service scheme, i.e. the application environment 3.1 The system 3 EQUIPMENT The Moving Eye system consists of remote user stations, the Internet, server/gateway computer, servo controlled camera, a all root called "Rollo", local network and local devices. See figure 1. Depending on the application the system structure varies. Ball root - Rollo Remote User Station Wide area telecommunications network (Internet) Local Server / Gateway servohead and camera Local Network Sensors, actuators and devices Figure 1. The Moving Eye general system structure 3. Ball shaped root Rollo Figure. Rollo Rollo has a spherical transparent cover (Fig.). It is moved with two electrical motors. The energy source is a NiCd attery, which provides autonomy of several hours. For visual and audio perception the root is equipped with a camera and a microphone and a video link. The camera can e tilted +-100 deg. When it points upwards it is used for detecting visual eacons on the ceiling. The communication to the control station is done with a radio modem. The root is equipped with a micro controller oard (Phytec MiniModul-167 using Siemens SAB C167 CR-LM micro controller). The root has sensors for temperature, pan, tilt and heading of the inner
3 mechanics, and pulse encoders for motor rotation measurement. The software is written in C language. The local server transmits controls to the root using commands that are kinematics invariant (i.e. uses the work environment variales only). The commands include heading, speed and running time/distance. Commands can e given one y one or as a list. The system has also an automatic localisation command, which causes the root to stop, wait for some time to smooth out oscillations, turn the camera to vertical position, find the visile eacons and automatically calculate the position, which is returned to the control station. Mechanical Structure. The plastic covers of Rollo (Fig. 3) are connected together with an equatorial rim. The inner mechanics (internal drive unit) of the all is hanging from the rim. The rim can e rotated around two axes of the inner drive unit (IDU). When rim is rotated around the X-axis the rim and the all rotates to the direction of the inner mechanics (in the fig.3 either towards to or away from the reader). When rim is rotated along the Z-axis (the contact points of the inner mechanics and rim change) and the rim plane is in horizontal position the heading of the inner mechanics change assuming that there is some friction etween the all cover and ground. Z Plastic cover Rim Internal drive unit Figure 3. Functional principle of Rollo Mathematical Model. A dynamic model of Rollo has e presented y Halme et al. [Halme et al., 1996 and ] and Spitzmüller [Spitzmüller, 1998]. To simplify the model the all root is reduced to point masses on a thin plate. See the following list, figure 4 and equations 1 4. This model is of course an over simplification of the real 3-D dynamics of the all, ut it provides a ase for developing controls for the root. The turning of the inner mechanism is modelled to affect to the all heading only. r e r r a r w θ X external radius of the all radius to IDU s centre of gravity (M) radius of a ig rack wheel (fix) radius of a small rack wheel angle of inclination of IDU rolling angle of the all rolling angle of the IDU γ O C G A H H i j H H λη, H slope of the ground origin: fixed on ground point where the all touches the ground center of the all center of mass of the IDU, reference unity vectors unity vectors for inclined ground λ d m m I I g k K t K R m G V motor i θ = + j unity vector in the direction GA mass of the all mass of the IDU inertia of the all aout G inertia of the IDU aout A gravitational acceleration friction coefficient torque constant of the motor voltage constant of the motor terminal resistance of the motor reduction ratio of the motor's gearhead voltage for motor O γ G ri d u C A θ re λd η λ Reference line attached to the all Figure 4 shows the notations in the system equations. Eq. 1 rag Gra KtVmotor KtK md gr r w rw Rm Id + r m θ mdrre sinθ md gr sin( θ + γ ) + ( m + m ) gre sinγ re k v = I + I + m r + m ( r + r r r e Eq.. rag Gr a KtVmotor KtK r w rw Rm [ Id md ( r rre cos )] + θ md gr sin( θ + γ ) v = I + r m d d e Eq.3. and 4. = v, = v e sin( θ + γ ) cosθ )
4 3.3 Position estimation using ceiling eacons The on-oard camera can e turned to vertical direction to look at the ceiling. There are coded eacons on the ceiling so that in each location of the environment the camera can see at least one of them. The positions and orientations of the eacons are known. It is assumed that the floor is flat and its distance from the ceiling is known. produce the augmented reality in this software. This software also contains user interface to simulation model of Rollo as well as controls for controlling Rollo s movements. ceiling O a A B α c z image plane lens Figure 6: The user interface is used for root controls. The view to the process can e video, virtual model or augmented reality. Here real-time video of Rollo and its surroundings are shown. B' α Figure 5. Using a simple camera model when image plane is tilted (one-dimensional case). a: Distance of the all s position O and the point A where the normal from the lens of the camera meets the ceiling. : Distance A to the known marker B on the ceiling. c: Normal projection of the distance. α: Inclination angle in one-dimensional case. The root position can e calculated using the simple camera model (pinhole model) from the known marker B position and direction, when the room height (z) and focal length of the camera (f) are known and when the inclination angle α, the root heading and the position B are measured. Figure 7: The virtual model of the remote environment shows the virtual root as well. 3.4 Local Server The local server can e a normal PC or a special server PC running with Windows NT, Linux or Unix operation systems. Its functions are to provide the user access to the root, camera controls, video, audio, data and environment controls. The server is used for downloading the user interface program to the user stations as well. The local equipment can e connected straight to the server or through a local area network. In real applications the server is remotely maintained and it does not need to have a keyoard or display. 3.5 User Stations User software contains virtual model of the remote environment. Video and virtual model are overlaid to Figure 8. Augmented view of the corridor contains live video with an overlaid virtual model, which displays additional information
5 4 APPLICATIONS 4.1 Educational system The educational system has een developed for virtual laoratory exercises, which university students can do over Internet. The exercises are part of the IECAT (Innovative Educational Concepts for Autonomous and Teleoperated Systems) project. http://ars-sun1.ars.fh-weingarten.de/iecat/iecat.html The laoratory experiment environment utilizes the Moving Eye system in an office environment. The mission of Moving Eye is to inspect and monitor the office and some of its equipment. The root can e programmed as an autonomous device or it can e teleoperated via Internet. Because of quite a nasty dynamics of the root and network delays direct teleoperation is not working well. Instead, teleoperation is uilt on a model-ased concept where the operator uses the augmented reality to plan the route of the root efore giving motion commands. The overall experimentation system includes versatile possiilities to set up interactive laoratory exercises from an elementary level to more advanced levels. Topics include mechatronics, root kinematics and dynamics, localisation and navigation, augmented VR-techniques, communication systems and Internet ased control of devices. User 1 Ball root - Rollo INTERNET User Laoratory Server (Keeper Software) servohead and camera Laoratory Ethernet Sensors and actuators Figure 9. The educational telerootic system The educational system consists of 5 experiments. The first experiment includes series of pictures, video, and simulation experimentation, which explain the technical components and structure of the all shaped root. The idea of the second experiment is to learn the mission environment and to ecome acquainted with the augmented telepresense system. The office corridor 3D-model is shown to the user. He can move in this virtual environment with the virtual Rollo and look it trough the root s virtual camera. From the wall real camera he can see the overall real picture as well as the augmented view. The kinematics of the Rollo is non-trivial. The dynamics is still far more complex. However, it is necessary to know oth at least to some extent to get an idea how the root moves. The third experiment introduces the main mechanical equations. The students are asked to simulate these equations with Matla and to get an idea how the real root moves when commanding the motors. The existing command lirary of the root is introduced. In experiment 4 the position of Rollo is otained using the on-oard camera and eacons fixed on the ceiling of the corridor. The last experiment is called the Moving Eye mission. Here Rollo monitors and guards an office during non-working hours. In the mission the root travels around the office and transmits images from given targets, which are defined using the virtual model. 4. Home Helper System with Rollo The Home Helper system intends to commercialise the use of the Moving Eye concept for home environments. The system will e used for people, e.g. senior citizens, to manage to live etter and safer at home. User Station 1 Home automation network 1 Wireless Home Network INTERNET ISDN User Station Internet Service Provider (Home server connections and server management) INTERNET ISDN, ADSL Indoor Service Root Rollo Home Server (Gateway) Servohead and camera Home automation network Home Equipment Figure 10. The Home Helper system layout consists of Internet service providers external Internet servers, home gateways, local networks, wired and wireless local devices and home roots. The Home Helper system provides a moile multimedia platform for communications etween home and outside assisters. The system is connected to various networked devices at home. The devices provide possiilities for remote security surveillance, teleoperation of the devices, and interactive assistance to people living at home.
6 5 RESULTS A demonstration system for testing the system functions is operating at the Automation Technology Laoratory of Helsinki University of Technology. The following functions and features have een demonstrated and their applicaility in the home service root system has een tested: P Measurements Simulation error Position. error Est. % x y d head x y d x y d head 100 731 484 774 36.6 79-484 559-8 6 10 6.6 100 797 73 810 17.4 13-73 346-0 1 3 13.5 100 810 153 814-0.3 00-153 5-10 -17 0 15.4 100 808 91 809-5.7 0-91 -14-9 17 11.1 100 755 144 759 -.3 55-144 93-46 -9 47 10.5 75 34 305 36 60.9-8 -305 316-38 -36 5 31. 75 378 0 378 37. -118-0 10-37 -7 46 9. 75 86 343 31 74.6-6 -343 344-11 54 56 5.9 75 380 75 381 61.9-10 -75 14-70 -6 75 55.6 75 35 1 35 59.3-65 -1 68 55-39 67 55.7 Tale 1 shows results of two experiments where Rollo is driven straight ahead 6 seconds with 100% and 75% power. Rollo s real position is measured with a measuring tape and the results are compared to simulated results and to position estimates derived from the eacon positioning. The measured heading is from the magnetic compass on-oard. Positioning heading estimate is calculated from the eacon images. Simulation results show significant deviations from the measured positions. On the other hand root position determination using the ceiling ased eacons produces good position estimates. They are with a 75 mm radius from the real position. The mesured and estimated headings differ greatly from each other and from the anticipated 0 degrees, which is given y the simulations. This deviation is caused y the pendulous motions the root does after turning of the drive motor. Augmented reality provides an efficient medium for communications etween a remote user and a local system. The user can navigate in the virtual model and susequently use it as an operator interface. A wealth of information from real time measurements, and history data to third party Internet pages can e accessed this way. Internet- and local area networks produce unpredictale communications delays in receiving the video and in sending the commands. However, the virtual model will react to the camera control commands immediately. This way the user can interact with the model. A after a time delay he will see the effects of his commands on video as well. The wait and see mode of teleoperation work well, ut it makes the system controlling very slow. 6 CONCLUSIONS The presented concept utilizing the Rollo root for developing home services and education is a working solution. There are several growing technologies that support the development of indoor service rootics: Home servers Home automation networks Wireless communications Faster communication lines to homes Virtual model is a good interface to data and documents. Using a completely transparent model plain video image can e an interface to machine data and documents. Technology is ready for the real use of this type of systems that are using augmented reality. Standardisation has progressed in key technologies such as: video conferencing, technical documentation, and telecommunications. The performance / prize ratio has increased significantly in areas like computer graphics, and computing power. The use of 3-D modelling in design and documentation is constantly increasing and video conferencing is ecoming a standard way of communication. All these factors make the rapid deployment of these types of systems feasile. REFERENCES Figure 11. The user interface utilizes augmented reality. Here the results of a simulated position of Rollo, the measured position of Rollo and the actual position of Rollo as seen on the video can e seen as three separate roots and their positions can e compared. Halme A., Suomela J., Schönerg T., Wang Y., A Spherical Moile Micro-Root for Scientific Applications, ASTRA96, ESTEC, Noordwijk, The Netherlands, 6.-7.,1996 Sheridan T. B. Telerootics, Automation, and Human Supervisory Control, The MIT Press 199 Spitzmüller, S., Microcontroller Based Control System for a Rolling Miniroot, Diploma Thesis, Helsinki University of Technology, 1998