Teleoperation Teleoperation and applications
Human Robot Interface Closed loop control system Human typically closes the loop in the highest level In many cases realtime human interaction is needed direct teleoperation Actuator contol loops are closed by human in real-time Jussi Suomela
Development of H R interaction Jussi Suomela
Notes You always need telesystem or human intervention as a backup at some point a human will need to take control embed in your design Roboticists automate what is easy and leave the rest to the human - Don Norman The user interface is absolutely critical User interface make up 60% of commercial code Useful= is the program purpose useful? usually given to designer via specifications and requirements Usable= can a human use it efficiently? designer must conduct usability studies avoid if I can use it, some one else will
Definition Teleoperation means simply to operate a vehicle or a system over a distance Distance can vary from tens of centimeters (micro manipulation) to millions of kilometers (space applications)
Teloperation Robotics HRI The 1 st mobile machines without human onboard were teleoperators the first prestage of a robot How to control a vehicle over a distance? Phased development from mechanical manipulation to high level supervisory control Today both closed loop teleoperation and high level communication are needed
Robot teleoperation Basicly most of the mobile robots are teleoperated machines and most of the teleoperated machines are mobile robots, only level of autonomy varies HRI = teleoperation interface Autonomous robots??
Development manipulation vehicle (submarine) control space Semiautonomous vehicles Autonomous vehicles (no closed-loop teleoperation needed)
Terminology Teleoperation: to operate a vehicle or a system over a distance Operator : human operator is the person who monitors the operated machine and makes the needed control actions Teleoperator is the teleoperated machine. A sophisticated teleoperator can also be called as telerobot In supervisory control remarkable part of the control is delegated to the teleoperator end (compare coordinated teleoperation)
Definitions (way to control) Mechanical manipulation The control commands are transmitted mechanically or hydraulically to the teleoperator. Visual feedback can be straight or via monitor. This is typical for manipulation of dangerous materials as well as micro manipulation
Definitions (way to control) Remote operation/control: The operator has most of the time straight visual contact to the controlled target. Control commands are send electrically by wire or radio
Definitions (way to control) Normal or standard teleoperation Wireless control and visual feedback via camera monitor system
Telepresence When sufficient amount of sensor information (vision, sound, force) is brought from the teleoperator site to the operator he or she feels physically present in the teleoperator site Called also tele-existence Important information is transferred and dangerous/noise is filtered
Virtual presence (reality) Like tele-presence except the sensor infromation is generated artificially by computer(s) Simulators Games Models (design etc.)
Augmented reality Real information (usually image data) is mixed with additional virtual information Numerical information, realtime models, etc.
Definitions (way to control) semi- or full autonomy robot is tele-controlled when needed Visual feedback through the robot s eyes (cameramonitor) ex. Sojouner with different modes, Automine human doesn t have to do everything Supervisory control
Definitions Closed loop control (Direct teleoperation): The operator controls the actuators of the teleoperator by direct (analog) signals and gets real-time feedback. This is possible only when the delays in the control loop are minimal. Coordinated teleoperation: The operator again controls the actuators, but now there is some internal control - remote loop - included. However, there is no autonomy included in the remote end. The remote loops are used only to close those control loops that the operator is unable to control because of the delay.
Definitions In supervisory control [Sheridan, 1992], the remarkable part of the control is to be found in the teleoperator end (compare coordinated teleoperation). The teleoperator can now perform part of the tasks more or less autonomously, while the operator mainly monitors and gives high-level commands. The term task based teleoperation is sometimes used here, but it is more limited than "supervisory control".
Closed loop/supervisory control
History
History The first modern master - slave teleoperators were mechanical pantographs. These manipulators were developed by the group of R. Goertz in the late 1940s at the Argonne National Laboratory where Enrico Fermi developed the first nuclear reactor
History The mechanical manipulators were soon replaced by electro mechanical servos In 1954 Goertz s team developed the first electro mechanical manipulator with feedback servo control. After this the teleoperation of manipulators and vehicles spread out rapidly to new branches where advantages of teleoperation techniques could be utilized.
Applications Submarines (ROV)
Applications Space Perfect for teleoperation: safety and costs problem very long delay => move and wait
Space: Lunokhod 2 Lunokhod 2 (1973) operated for about 4 months, covered 37 km (23 miles) of terrain, including hilly upland areas and rills Lunokhod 2 was equipped with three slow-scan television cameras, one mounted high on the rover for navigation, which could return high resolution images at different rates 3.2, 5.7, 10.9 or 21.1 seconds per frame (not frames per second). These images were used by a five-man team of controllers on Earth who sent driving commands to the rover in real time.
Applications Military underwater ground air semiautonomous / closed loop control Anti terrorist typically closed loop control
Applications Medical Endoscopic surgery ~ (micro) mechanical manipulation (is this teleoperation??) minimal damage, smaller risks Surgery robots, scaled manipulation Telesurgery Specialists can operate over distances
Applications Industrial Mining Resque operations Mining in unsafe areas
Industrial applications Why to teleoperate if there is no danger?? Semiautonomous work machines with part time teleoperation or even crowd sourcing!! All possible work is done autonomously Difficult tasks and exceptions are teleoperated or crowd sourced This will be the future
Automine Autonomous hauling and dumping Teleoperated loading Professional operators on-ground Less waste time full 24/7 operation An operator / 2-3 machines Autonomous driving is more machine friendly less maintenance
Components [Uttal 89] Local display Local control device Communication Remote Local Display Communication Remote Sensor Mobility sensor Control mobility effector Effector power Power
Technology Mechanical manipulators, (good feedback) Electrical servos TV, (closed loop) Radio control Video, (closed loop, supervisory)
Delay There are always delays in a teleoperation loop Every part of the system has some delays Digital systems increased the delay feedback delay teleoperator delay transmission delay Control delay
DELAY signal speed max. ~ 300 000km/s How big the delay can be In practise (mobile machines) < 0,1s => perfect, < 0,3-0,5s = > easy In long delay systems move and wait task based teleopration Can be estimated if the system frequency is known!
Long delay teleoperation No possibilities for closed loop control because of the mission it s difficult to increase the autonomy => move and wait teleoperation models of robot and environment, operator loop in control
Problems cognitive fatigue communications dropout communications bandwidth communications lag too many people to run one robot (hidden cost) Predator: 7:1 Human robot ratio 4 people to control it (52-56 weeks of training) one for flying two for instruments one for landing/takeoff plus maintenance, sensor processing and routing lack of self-awareness in Kosovo, come along side in helicopter and shoot down
Teleoperation best suited for the tasks are unstructured and not repetitive key portions of the task require dexterous manipulation, especially hand-eye coordination, but not continuously key portions of the task require object recognition or situational awareness the needs of the display technology do not exceed the limitations of the communication link (bandwidth, time delays) the availability of trained personnel is not an issue
Situational awareness Is typically limited in teleoperation configurations Lack of SA generates problems in performance and safety Reasonable cost/performance ratio to be found Telepresence systems increase the cost&complexity rapidly
Sense of presence Based on tele(virtual)-presence Sheridan s definition of telepresence
Sense of presence Extent of sensory information has a much greater impact than the other two combined. These three factors however cannot describe presence alone. Task variables, such as task difficulty and degree of automation, also are important to presence.
Telepresence Already camera monitor combination creates some level of presence more sophisticated system is called for in order to call it telepresence To provide a perfect telepresence, all human senses should be transmitted from the teleoperator site to the operator site vision, hearing and sense are relatively easy smell and taste are more complicated Presence is perfect when human can t make a difference between the virtual/tele and real presence
Vision Humans get 90% of their perception To see is to believe eyes are very complex opto-mechanical systems (7 DoF with head) FoV is (H)180 deg x (V)120 deg Focused area only few degrees Movements from whole area Extremely difficult system to be imitated
Vision Head tracking Head following cameras (2-3 DoF) HMD => relatively good feeling of presence
Hearing Human area 16 20000Hz Important in telepresence In case of mobile machine control the noise can be filtered and the important sounds transferred with reasonable volume
Touch The most important human sense Human touch sensors mechanoreceptors are activated by touch, i.e. by pressure on the tissues Three basic classes tactile information ( touch ) kinesthetic information ( force, angle ) Vestibular information ( pose )
Tactile referring to the sense of contact with the object, mediated by the responses of low-threshold mechanoreceptors innervating the skin (say, the finger pad) within and around the contact region
Kinesthetic referring to the sense of position and motion of limbs along with the associated forces conveyed by the sensory receptors in the skin around the joints, joint capsules, tendons, and muscles, together with neural signals derived from motor commands
Force feedback (kinesthetic) force generated by the teleoperator, usually a manipulator, is fed back to the operator in order to generate a real response in gripping and manipulation tasks Also in virtual environments Inbuilt in mechanical manipulators
Haptic feedback (tactile) haptic feedback, the tactile skin sensors have the main role. tactile sensing of the robot manipulator is fed back to the fingers of the operator Other possibilities also
Vestibular sensors inside the inner ear angular acceleration and thus rotation linear acceleration in the horizontal and vertical plane, i.e. to gravity => position and movements of the head to be detected Important in dynamic driving tasks
Vestibular feedback not usually used in teleoperation not needed and expensive to implement usually in simulators to create presence If vision and vestibular sensors mismatch => simulator sickness (=seasickness)
Simulator Sickness similar to motion sickness difference is that SS can occur without any actual motion of the operator Symptoms: apathy, general discomfort, headache, stomach awareness, nausea, etc. encountered especially when HMD type displays are used
Simulator Sickness The most typical reason of SS is the cue conflict In cue conflict different nerves get different information from the environment Typically conflict between visual and vestibular inputs especially when HMD is used and the time lags in vision and control => + Moving teloperator
Future worksite concept
Operation
Video