Human-oriented Telerobotic Control over the Internet

Transcription

1 1 Human-oriented Telerobotic Control over the Internet Sandra Hirche Ass. Inst. for Information-Oriented Control Dept of Electrical Engineering and Information Technology Technische Universität München Seminar GIPSA-Lab CNRS, Grenoble 18/01/2010 Thanks to T. Matiakis, I. Vittorias, M. Rank

2 Research Interests 2 control communication networked control systems cooperative control control human telerobotics physical interaction communication network A1 auditory feedback visual feedback commanded velocity stereo camera head microfon array A2 C P AN communication time delay packet loss A3 Ai haptic feedback environment force force sensor human system interface (HSI) communication network e.g. Internet teleoperator (TO) environment Today s Talk haptic telepresence

3 Multimodal Telepresence Applications 3 visual auditory haptic olfactory multimodal whole-body sensor-actuator-suit gustatory humanoid teleoperator tele-maintenance and -diagnosis in manufacturing facilities tele-assembly in nano/makro environments, tele-surgery fly-, drive-by-wire, rapid prototyping Introduction Stability Transparency

4 Networked Haptic Telepresence Challenges 4 control loop closed over communication network stability human should feel like directly interacting transparency auditory feedback visual feedback commanded velocity stereo camera head microfon array communication time delay packet loss haptic feedback environment force force sensor human system interface (HSI) communication network e.g. Internet teleoperator (TO) environment

5 Haptic Telepresence Fundamental Research 5 control - stability human - transparency uncertain subsystems communication: time delay, packet loss, information rate, human & environment human perception human perceived performance, communication effects, human-oriented design,...

6 State-of-the-Art Bilateral Control Approaches 6 passivity-based (scattering, Llewelyn, time-domain) ISS robust control (µ-synthesis, parameter space) adaptive, switching predictive methods General question: model-based vs. model-free depends on assumptions/knowledge of human/environment trade-off transparency vs. robust stability certificates This talk: passive human/environment - passivity-based methods

7 Overview 7 Stability with communication unreliabilities Stability with constant time delay Extensions for varying time delay and packet loss Transparency - human-oriented analysis and design Time delay and packet loss Haptic data compression

8 Instability with Time Delay 8 Closed control loop with constant time delay T G(s) e st Open loop: G OL (s) = G(s)e st G OL [db] φ [ ] G(s) = 10 s T = 500 ms unstable T = 100 ms stable Theorem (linear time invariant system) Stability if gain margin A < 1 (A = G OL at φ = 180 ). stability for arbitrary constant time delay iff G < 1, ω > 0

9 Stable Haptic Telepresence with Constant Time Delay 9 Theorem (Spong et al. 1989/Niemeyer et al. 1991) Stability for arbitrary large constant time delay with (i) velocity-force-architecture, and (ii) scattering transformation: u = (2b) 1 2 (f + bẋ); with wave impedance b > 0. v = (2b) 1 2 (f bẋ), human HSI ẋ u l u r h scattering T 1 transfor mation f d h v l T 2 v r scattering transfor mation ẋ t f e TO environ ment

10 Background Small Gain Approach 10 System Σ: ẋ = f(x, u), y = h(x, u) with x(0) = 0 is passive if t 0 ut (τ)y(τ) dτ V (x(t)) t is L 2 -stable if γ < s.t. y t γ u t u, t (γ is L 2 -gain, u t 2 = t 0 ut u dτ L 2 -norm) has scattering operator (y u) = S(y + u) Theorem (passivity small gain) Σ passive γ S 1 Consider systems Σ 1 and Σ 2 in feedback interconnection Theorem (small gain) γ 1 γ 2 < 1 L 2 -stability (external input output).

11 Small Gain Interpretation 11 assumption: HSI/human, TO/environment (strictly) passive (ẋ-f-architecture, local control) scattering operator (with scaling) e.g. TO/env.: Σ r : [(2b) 1 2 (f e + bẋ d t )] [(2b) 1 2 (f e bẋ d t )] γ r γ l γ T1 γ T2 = γ r γ l < 1 stability for arbitrarily large constant time delay Σ l small gain loop Σ r human HSI ẋ u l u r h ẋt scattering T 1 scattering transfor transfor mation T 2 mation fh d v l v r f e TO environ ment

12 Application of the Scattering Transformation 12 experimentally most successful approach many theoretical extensions HSI (Munich) Internet Teleoperator (Berlin) UDP socket PC RTLinux PC RTLinux 7 actuated joints I Matlab/ Simulink Matlab/ Simulink II force control Sensoray S626 IO DAC ADC counter position/ velocity control Sensoray S626 IO DAC ADC counter motor encoder II force/torque sensor screw I

13 Related Work and Extensions 13 Challenge Contributions time delay Spong+ - varying gains, passive position (varying) Niemeyer+ - wave integral transmission Munir+ - prediction based Hirche+ - generalized wave variables packet loss Yokokohji+ - energy control Stramigioli+ - sampled data, port Hamiltonian Spong+ - passive interpolation Hirche+ - passive extrapolation data compression Hirche+ - passive deadband control all based on passivity/ scattering transformation

14 Approximate Human/Environment/Robot Models 14 Idea [Hirche+ 2010, submitted] use approximate knowledge of damping characteristics for less conservative design (QSR-dissipative system) Example Σ : Mẍ(t)+D(x, ẋ, t)+kx(t) = f(t) lower bound D(x, ẋ, t) d min 0 excess of passivity Σ is IF P (δ = d min ) 1 for Σ : ẋ f and OF P (ɛ = d min ) 2 for Σ : f ẋ 1 IFP: input-feedforward passive 2 OFP: output-feedback passive

15 Background Dissipative Systems 15 Definition [Willems1972] Σ : ẋ = f(x, u), y = h(x, u), x R n, u R p, y R q dissipative if V (x(t)) V (x(0)) [ ] [ ] t [ ] T u Q S 0 u y P dτ, P = y S T. R Special case IF-OFP systems: Q = δi, R = ɛi, S = 1 2I, δ, ɛ R feedback interconnection of IF-OFP systems is IF-OFP Example: Σ 1 : OF P (ɛ 1 ) Σ 2 : IF P (δ 2 ) Σ 1 : OFP(δ 1 ) Σ 2 : IFP(ɛ 2 ) } Σ : OF P (ɛ 1 + δ 2 ) } Σ : IF P ( ) δ 1 + ɛ δ1 2 + ( ɛ 22 ) 2

16 Generalized Wave Variable Transformation» cos θi sin θi transformation M = BR, scaling B, R = sin θi cos θi 16» ul v» l ur v r = M = M» ẋm fm» ẋ s f s Theorem (Hirche+ 2009) Delay-independent finite L 2 -gain stability if θ [θ l, θ r ] (i) sin(θ l )cos(θ l ) ɛ l sin 2 (θ l ) > 0, cot2θ l = ɛ l (ii) sin(θ r )cos(θ r ) δ r cos 2 (θ r ) > 0, cot2θ r = δ r extensions to time-varying delay and packet loss exist

17 17 Comparison with Standard Wave Variable Transformation Magnitude (db) Phase (deg) standard scattering transformation environment generalized scattering transformation environment generalized scattering transformation standard scattering transformation Frequency (rad/sec) Result Transparency improved compared to standard WV Spring-damper environment Z e (s) = 300 s + 60 IFP with δ r = δ e = d min = 60 stable if θ [45, 89 ], choose θ = 89 T 1 + T 2 = 100ms

18 Varying Time Delay: Violation of Small Gain Condition Fact T 1 (t) instead of constant T 1 (for T 2 analogous) assumption T 1 T 1,max < 1 (causality) γ 2 1 = T 1 1 T 1,max > 1 (if there exist time intervals with T 1 > 0) stability condition violated! Proof. u r,t 2 = = t u 2 l (τ T 1(τ))dτ 0 t T 1 (t) 1 T 1 (0) T u l,t 2 1,max t T 1 (σ) u2 l (σ)dσ u l, t mit σ = τ T 1 (τ) T 1 (σ) u2 l (σ)dσ 18

19 Varying Time Delay: Stability by Time-Varying Gain 19 Introduce time-varying gain u l T 1 (t) f 1 (t) u r Theorem (Spong et al. 2002) Stability for varying time delay with f 1 (t) = Proof. u r,t 2 = = t f 2 1 (τ) u2 l (τ T 1(τ)) dτ 0 t T 1 (t) 1 T 1 1 T 1 (0) u l,t 2 1 T 1 (t). mit σ = τ T 1 (τ) t T1(t) u 2 T 1 l (σ) dσ = u 2 l (σ)dσ u l, t γ f1 T 1 = 1 stability by small gain theorem 0

20 Communication with Packet Loss Forward path only (backward path analogous): packet loss data reconstruction time delay scattering transf. P 1 u l u r T 1 reconstruction 20 Theorem Stability if k Θ u T r (k)u r (k) u T l (k T 1)u l (k T 1 ) k Θ with Θ the set of indices k of lost packets (1 sample per packet) Proof. u T r u r = u T r u r + u T r u r k k Θ k / Θ u T l (k T 1)u l (k T 1 ) + u T l (k T 1)u l (k T 1 ) k Θ k / Θ k ut l u l communication line time delay and packet loss: γ P 1 1

21 Data Reconstruction: Analysis and Synthesis Hold Last Sample or any other estimator (HLS) γ P 1 > 1 stability not guaranteed signal original signal reconstructed signal with HLS interval with packet loss energy generated 21 Zeroing (Z) γ P 1 < 1 stability guaranteed not transparent signal original signal reconstructed signal with Z interval with packet loss energy dissipated HLS/Z with small gain observer (related to time-domain passivity) HLS if N 1 u 2 l (k) N u 2 r(k) > 0 k=0 k=0 Z otherwise γ P 1 = 1 stability guaranteed better transparency than zeroing signal original signal reconstr. signal intervals with packet loss energy dissipated energy generated time

22 Overview 22 Stability with communication unreliabilities Stability with constant time delay Extensions for varying time delay and packet loss Transparency - human-oriented analysis and design Time delay and packet loss Haptic data compression

23 Networked Haptic Telepresence Challenges 23 control loop closed over communication network stability human should feel like directly interacting transparency auditory feedback visual feedback commanded velocity stereo camera head microfon array communication time delay packet loss haptic feedback environment force force sensor human system interface (HSI) communication network e.g. Internet teleoperator (TO) environment

24 Perceived Transparency 24 Transparency (Lawrence 1993) if displayed impedance Z h = environment impedance Z e (mechanical impedance Z : velocity force) with time delay & packet loss not achievable idea: consider human perceptual limits in analysis and control design Perceived transparency if Z h (Z e, Z e + ), with determined by perception threshold

25 Psychophysics 25 Human cannot perceive arbitrarily small stimulus differences. Weber s law I = constant = JND I where I stimulus intensity, I just noticeable absolute difference JND s determined in psychophysical experiments e.g. for parameters of mechanical impedance: JND stiffness = (23 ± 3)% [Jones 1990] JND viscosity = (34 ± 5)% [Jones 1993] JND inertia = (21 ± 3.5)% [Beauregard 1995]

26 Transparency with Constant Time Delay 26 Assumptions: constant time delay, scattering transf., Z e linear time-invariant Result: Z h with Padé approximation time delay, T = T 1 + T 2 Z h (s) = b Z e(s) + b + (Z e (s) b) e st Z e (s) + b (Z e (s) b) e st b2z e(s) + bt s 2b + T sz e (s) Summary results inertia displayed in free space mit T stiff wall displayed softer with T maximum displayable stiffness mit T displayed stiffness difference mit T

27 Perception-Oriented Design 27 Example stiff wall (spring characteristics): Z e = k e /s, k e > 0 Z h k h /s mit 1/k h = 1/k e + T/2b µ k h s Ì ¾¼¼Ñ ω[s 1 ] ½¼¼Æ»Ñ Transparency: k h = k e b not realizable Perceived transparency: k h k e (1 JND, 1 + JND) b > 1 JND T k e JND 2 realizable, validated in user studies

28 Transparency with Packet Loss 28 Zeroing - no stiffness displayed Energy supervised HLS/zeroing - reduced stiffness displayed Zeroing spring damper environment ¾¼¼Æ»Ñ, ½Æ»Ñ ÈÐ ¾± ÈÐ ¾¼± ÀÞ Amplitude response for perceived and environment impedance Energy supervised HLS/zeroing ÈÐ ¼± ÀÞ Ì ½Ñ ÈÐ ¾¼± packet rate 1000Hz Method: Monte Carlo simulations for different packet loss probabilitiesp l, mean frequency response for perceived impedance from cross correlation

29 Perceived Transparency with Packet Loss 29 transparency degradation not perceivable for quite large packet loss probabilities bound not invariant to env. properties (and sampling rate) k h [N/m] Energy supervised HLS/zeroing 23% (JND for stiffness) Loss not perceivable P l [%]

30 Transparent Design for Random Delay & Packet Loss Assumptions: Time delay with probability density function p(t ), communication network induced packet loss with probability P komm l 30 Dejitter buffer: Trade-off between higher constant delay T and additional packet Ô Ìµ loss P dejitter l through discarding packets ÌÑ Ò Ì Ì È ØØ Ö Ð Goal Displayed impedance environment impedance

31 Æ»Ñ Ì Ñ 31 Transparent Design for Random Delay & Packet Loss (2) k h [N/m] Example: Environment = stiff wall Z e = ke s Ì Ñ T [ms] (P komm l = 0) Displayed stiffness with time delay and packet loss Dejitter buffer: T = 90ms packet loss P dejitter l = 48% ¾¼¼Æ»Ñ ½Æ»Ñ Energy supervised HLS/zeroing Environment: ÈÐ ± P l [%] mapèð Ì µ for Æ»Ñ Poisson distribution ÌÑ Ò ¼Ñ Ô Ìµ with 250 Optimum with respect maximal perceivable stiffness 100 ½¼Æ»Ñ 200 ¼Æ»Ñ ¾¼Æ»Ñ ÈÐ ± P [%] l validated in objective experiments and human user studies T [ms]

32 Haptic Data Compression 32 Typically packet rate = sampling rate of local control loops Ex.: 6 DoF teleoperator, sampling rate 1000Hz 500kbit/s High packet rate/ load difficult under harsh communication conditions as e.g. underwater, space Goal: Network traffic reduction without transparency degradation Network traffic reduction Data load compression low efficiency due to low load/header ratio Reduction transmitted packet number Multi rate system Deadband control

33 Deadband Control 33 Transmit data only if difference between the current and the most recently transmitted value exceeds certain threshold asynchronous sampling strategy Exploit human perceptual limits: humans not able to detect arbitrarily small differences in motion/force signals (JND) signal to transmit sent a packet deadband value standard deadband sample index k

34 Stability with Deadband Control 34 Assumptions: Constant delay and no packet loss Deadband control on scattering variables Stability by small gain observer for better performance zeroing replaced by most conservative value within deadband ẋ h ẋ h Dead Human HSI Scattering Transformation u l v l band Control Forward path Data Recon struction Data Recon struction Backward path Dead band Control u r v r Scattering Transformation f e TO ẋ t Environment f e

35 Transparent Reduction of Network Traffic Load Detection threshold for deadband value by psychophysical experiments (3 interval forced choice paradigm, 1DoF telepresence system, 11 test persons, 100ms roundtrip delay) Number of transmitted packets measured during experiments 35 Transmitted packets [%] Detection threshold 20 determined in psychophysical experiments % Deadband Deadband value ÔÏ Result Network traffic reduction by 96% without perceivable transparency degradation

36 Intercontinental Telecooperation Experiment 36

37 Discussion Haptic Telepresence 37 fundamental issues: stability and transparency in this talk stability with communication unreliabilities constant and varying time delay, packet loss passivity based control human-oriented analysis and design consider human perception thresholds Current research less conservative human model (dissipativity, parameter pdfs) multi-modal transparency

38 Acknowledgements 38 Persons: M. Fujita, S. Hara, M. Buss, E. Steinbach, P. Hinterseer, A. Peer, B. Stanczyk, M. Kuschel, T. Matiakis, C.-C. Chen, I. Vittorias, M. Rank, A. Bauer, K. Klank, P. Kremer, R. Schuster,... Sponsors: DFG, SFB453 Japanese Society for the Promotion of Science (JSPS) Siemens AG Technische Universität München

39 Selected Publications S. Hirche and M. Buss, Transparent Data Reduction in Networked Telepresence and Teleaction Systems Part II: Time-Delayed Communication, PRESENCE, S. Hirche, P. Hinterseer, E. Steinbach, and M. Buss, Transparent Data Reduction in Networked TPTA Systems Part I: Communication without Time Delay, PRESENCE, S. Hirche and M. Buss, Insights on Human Adapted Control of Networked Telepresence and Teleaction systems, International Journal of Assistive Robotics and Mechatronics, vol. 7, pp , March S. Hirche and M. Buss, Transparenz haptischer Telepräsenzsysteme mit konstanter Zeitverzögerung, at Automatisierungstechnik, vol. 54, pp , February S. Hirche and M. Buss, Time Delay Issues in Haptic Telerobotics, in Adv. in Telerobotics, Springer, 2007, to appear. S. Hirche, M. Ferre, J. Barrio, C. Melchiorri, M. Buss, Bilateral Control Architectures for Telerobotics, in Advances in Telerobotics, Springer STAR series, R. Aracil, M. Buss, A. Peer, S. Cobos, S. Hirche, and M. Kuschel, The Human Role in Telerobotics, in Advances in Telerobotics, Springer STAR series, 2007, to appear. S. Hirche, A Study on Networked Dissipative Systems, in SICE Symp. on Control Theory, (Osaka, Japan), S. Hirche, C.-C. Chen, and M. Buss, Performance Oriented Control over Networks: A Switched Time Delay System Approach, in Proc. of the IEEE Conf. on Decision and Control, (San Diego, US), T. Matiakis, S. Hirche, and M. Buss, A Novel Input-Output-Transformation Method to Stabilize Networked Control Systems Independent of Delay, in Mathematical Theory of Networks and Systems, (Kyoto, Japan), S. Hirche, Haptic Telepresence in Packet Switched Communication Networks, Nr.1082 in Fortschrittsberichte VDI, VDI-Verlag, Düsseldorf, Germany, 2005, PhD Thesis, Rohde & Schwarz Award. S. Hirche, P. Hinterseer, E. Steinbach, and M. Buss, Towards Deadband Control in Networked Teleoperation Systems, in Proc. IFAC World Congress, (Prague, Czech Republic), 2005, IFAC World Congress Best Poster Award. S. Hirche and M. Buss, Packet Loss Effects in Passive Telepresence Systems, in Proc. of the 43rd IEEE Conference on Decision and Control, (Paradise Island, Bahamas), pp , S. Hirche and M. Buss, Passive Position Controlled Telepresence System with Time Delay, in Proc. of the American Control Conference, (Denver (CO), US), pp , 2003, Best Session Paper. N. Chopra, M. Spong, S. Hirche, and M. Buss, Bilateral Teleoperation over Internet: the Time Varying Delay Problem, in Proc. of the American Control Conference, (Denver (CO), US), pp ,