Human Interaction with Robots Working in Complex and Hazardous Environments Bill Hamel, Professor & Head IEEE Fellow RAS Vice President for Publication Activities Mechanical, Aerospace, & Biomedical Engineering Department University of Tennessee
Human Interaction with Robots Working in Complex and Hazardous Environments Preliminaries Complex and Hazardous? Robots? Human Interaction? Needs for Performance Improvements? Modern Telerobot? System Concept & Architecture Where should research foci be? Current Research Telerobotics Experimental System Results Future Directions Closing Remarks Napoli 18 December 2006 2
Complex and Hazardous? Remote operations in hazardous and uncertain environments unstructured Nuclear, Space, Underwater, Medical Surgery Remote work systems that involve: Remote sensing Manipulators Mobility Use of tools Human-in-the-loop operations/control Napoli 18 December 2006 3
Robot? Telerobotics Telerobots? Telerobot = Teleoperator U Robot tr = Robot U Teleoperator Tr = Teleoperator U Robot tr Tr Manual control U Autonomous operations Human interaction Computer assisted teleoperation Selective autonomy Napoli 18 December 2006 4
Human Interaction Tasks telepresence Sensors Manipulators Signals Human/Machine Interface Mobile Transporter Power teleaction Tools Physical/Temporal/Hazards Barriers Napoli 18 December 2006 5
Decades of Refinements Napoli 18 December 2006 6
Needs Reality: Too expensive Too slow Too complicated Improve remote work efficiency Best: direct/teleaction 10 Worst: direct/teleaction ~ 100 s Open engineering Power supply and transmission; cable handling 3D remote viewing and sensing Napoli 18 December 2006 7
Telerobot Functional Architecture RTSA TR Control HMI Autonomous Operations Interactive Task Planner In Situ Model Builder Manipulation Tooling HMI Teleoperations Cooperative Assists Planner Control Mobility Task Space Fault Detection & Recovery Computer assisted teleops Napoli 18 December 2006 8
Critical Performance Issues HUMAN-MACHINE COOPERATIVE TELEROBOTIC PROCESS Select and configure assistance strategy Execute manually with assistance Seamless Transfer Essential Manual teleoperation REMOTE OPERATIONS TASK SEQUENCE Subtask i manual Subtask i+1 Subtask i+2 automatic Subtask i+3 Subtask i+4 manual Manual teleoperation Build in situ 3D model Plan subtask Execute automatically Verify results ROBOT TASK SCENE ANALYZER PROCESS Napoli 18 December 2006 9
Telerobot Control Structure ESTOP System ON IDLE ERROR OK MANUAL_TELEOP ERROR OK AUTOEXEC Cartesian Controller + +! -! + K m B m +! - d dt +! Ju Ju -1 +! +! + - B s -1 K s B s -1 Trajectory Generator Task Planner & Assist Functions RTSA 3D Task Geometry & Action Points Napoli 18 December 2006 10
Research Test Bed Schilling Titan II manipulator master controller pan-tilt stereo sensor head D&D mockup manipulator communication hardware camera control hardware laser range camera host computer (LINUX) development computer (Windows NT) image processing computer (Windows NT) LOCAL AREA NETWORK Dual remote manipulators Napoli 18 December 2006 11
Human Interactive Task Space Modeling and Planning Max use of a priori knowledge Parallel Execution Manual Foreground. Autonomous Background. Operator Control Where, what, and how. Acceptance of results. Simplified GUI 5 pop-up window menus Color overlays AUTOSCAN (BACKGROUND) OOIs OOIs OOIs PANORAMIC VIEW ROI4 ROI2 ROI1 Confirm window User points camera ROI2 ROI4 ROI3 Queue ROI5 No Okay? Yes ROI1 Select ROI MANUAL (FOREGROUND) Place part Napoli 18 December 2006 12
Laser pointing designator ± 10 mm in robot workspace Napoli 18 December 2006 13
REMOTE OPERATIONS TASK SEQUENCE SUBTASKi: Manual SUBTASKi+1: Auto SUBTASKi+2: Manual SUBTASKi+3: Auto SUBTASKi+4: Auto TELEROBOTIC SUBTASK SEQUENCE i+3 RTSA BUILD IN SITU 3D Model PLAN SUB- TASK EXE- CUTE 8-10 min 4 min 18 min VERIFY RESULTS 32 min versus 45-90 min Teleop Napoli 18 December 2006 14
Recent activities 7 dof haptic controller Barrett Technologies WAM Large-scale multifingered end effector Barrett Technologies Wraptor Reduced tool specializations Controller integration Tooling:behavior-based grasping Test & evaluation Napoli 18 December 2006 15
Continuing Research RTSA HMI Autonomous Operations Interactive Task Planner In Situ Model Builder TR Control Manipulation Tool-based Control Tooling HMI Teleoperations Cooperative Assists Planner Control Mobility Task Space Re-Plan Sequence Fault Detection & Recovery Start re-plan Computer assisted teleops Retrace/restart original Abort Plan Sequence Operational Space Fault Detection Original Plan Sequence subtasks Drift state/step Fault state/step Return to original Intelligent/interactive State Management Napoli 18 December 2006 16
Higher-fidelity telepresence and teleaction needs, directions Perception/Sensing 3D viewing 3D graphics augmentation Natural kinesthetic/tactile feedback Intelligence/Control Human relationships Interactivity reduced through increasing autonomy More cooperative Humans move up in hierarchy Machine learning By observation symbiosis Action/Manipulation Multi-fingered end effectors Human Machine Interface Algorithmic complexity VR-based interfaces Napoli 18 December 2006 17
Summary True Telerobots are being realized. There is hope for improving the effectiveness of remote operations in complex environments. Ongoing research will continue to drive this trend. Napoli 18 December 2006 18
Thank you for your attention!