From extreme learning machines to reservoir computing: random projections and dynamics for motion learning

Transcription

1 From extreme learning machines to reservoir computing: random projections and dynamics for motion learning u y q Jochen Steil 2011, Tutorial at CBIC

2 Biology Physics Bielefeld Environment is part of: Intelligent Bielefeld University one of 4 strategic profile areas two central scientific institutes (CoR-Lab, CITEC) 5 of 13 departments cooperate 2 special research units and 3 interdisciplinary graduate programs industrial partners (Honda, Miele, Bertelsmann, OWL MASCHINENBAU,...) > 350 researchers, ~ 10 Mill EUR funding/year EU projects: italk, ROBOTDOC, HUMAVIPS, MONARCA, EMiCab, AMARSi, Echord Research Institute for Cognition and Robotics (CoR-Lab) Excellence Cluster Cognitive Interaction Technology (CITEC) Linguistics Psychology & Sports Technology Chemistry Economics History, Philosophy Law Mathematics Education Public Health Sociology

3 Reinhart, Steil, Humanoids 2009 Rolf, Steil, ICDL 2009 Neumann, Rolf, Steil, SAB 2010 Rolf, Steil, Trans. Autonomous Mental Development, 2010 Steil, Neural Networks, 2007 Reinhart, Steil, 2010: Differential Equations & Dynamical Systems Lemme, Steil, ESANN, 2010 Reservoir Dynamic Networks Neural Dynamic Movement Primitives Stability Theory of Recurrent Networks DAAD fellowship Russia Steil, Dissertation, 1999, Steil, Neurocomputing, 2002 Steil, Neurocomputing, 2006 Dynamics & Learning Visual Online Learning Robot Learning Architecture Neural Perceptual Grouping a) Input image b) Edge features Speech recognition Active vision + exploration Interaction Robot arm control Robot grasping c) CLM grouping d) Potts spin grouping Steil, Götting, Wersing, Körner Neurocomputing, 2002 Denecke, Wersing, Steil, Körner, Neurocomputing, 2007 Denecke et al., ESANN, 2010 Gesture recognition Object referencing Object recognition and pose Shared Attention Hand tracking Steil, Roethling, Haschke, Ritter Robotics & Autonomous Systems, 2004 Wischnewski et al, Cognitive Computation, 2010 Wersing, Steil, Ritter Neural Computation, 2001 Weng, Wersing, Steil, Ritter, Trans. Neural Networks, 2006

4 the multi-layer perceptron (MLP) u q

5 The standard neural learning approach for multi-layer perceptron (MLP): Minimize (quadratic) error function

6 Compute gradient w E by backpropagation u q adapt weights adapt weights local error output error

7 u q adapt weights task specific hidden representation

8 challenge: task specific hidden representation? propagation of errors? novel approach: learning & hidden state separate: linear output regression high-dim. random projection

9 u q random projection linear output regression

10

11 Outline Extreme Learning Machine (ELM) & Intrinsic Plasticity learn inverse kinematics ELM + Recurrence = Echo State, Reservoir Computing (RC) Associative Neural Learning (for kinematics) inverse kinematics + trajectory generation Programming Dynamics (shape attractor dynamics) learn and select multiple inverse kinematics solutions Learning velocity fields (movement primitives) Learning sequences of movement primitives

12

13 1990 s: Random projections for data processing dimension reduction separation properties are preserved only linear projections used/analyzed very powerful tool, see e.g Now: random projections dimension expansion (like for kernels) non-linear transformations universal feature set obtained

14 ELM - Extreme Learning Machine (Huang 2006) high dimensional random neural feedforward network, random u q learning: only by regression (+ weight decay regularization)

15 training: collect states for all tr training inputs (state harvesting) collect targets ˆQ = ( ˆq1,..., ˆq tr ) T do linear regression W out = ( H T H + ɛi ) 1 H T ˆQ regularization through epsilon -> model selection! ( not critical/considered in Huang 2006 (and other literature), because data sets were very large)

16 u q some other issues: input scaling, overfitting, initialize bias, weights in correct range model selection is important

17 u q idea: input specific tuning of features by adaptation of the nonlinear function Optimizing Extreme Learning Machines via Ridge Regression and Batch Intrinsic Plasticity, Neumann, K., and J.J. Steil, Neurocomputing, to appear.

18 Intrinsic Plasticity: optimize parameters of single neuron! use parametrized Fermi: h(x, a, b) =1/ (1 + exp ( ax b)) adjust parameters online: also batch algorithm available: Neumann & Steil, ICANN 2011 IP Learning

19 VIDEO Neumann, Emmerich, Steil, in preparation

20 Intrinsic Plasticity input/task specific scaling set of features is input specific set of features is less diverse IP acts as input regularizer reduces dependence on #nodes, initialization parameters! (similar to the Gaussian kernel with in SVM) Neumann, Emmerich, Steil, in preparation

21 ELM + Intrinsic Plasticity learn inverse kinematics (static, simple, here well defined) input position, output joint angles Neumann, K., and J.J. Steil, Neurocomputing, to appear. ELM + IP

22 ELM + Intrinsic Plasticity standard UCI tasks we (almost) always use IP! Neumann, K., and J.J. Steil, Neurocomputing, to appear

23 IP regularization through reduced feature complexity -- too much IP produces degenerated features VIDEO 0

24 Outline Extreme Learning Machine (ELM) & Intrinsic Plasticity learn inverse kinematics ELM + Recurrence = Echo State, Reservoir Computing (RC) Associative Neural Learning (for kinematics) kinematics + trajectory generation Programming Dynamics (shape attractor dynamics) learn and select multiple inverse kinematics solutions Learning velocity fields (movement primitives) Learning sequences of movement primitives

25 ELM + Recurrency

26 ELM & Recurrence? staged processing from left to right, network used for feature generation with recurrence: converge to attractor u q u q Extreme Learning Machine Attractor based ELM = Echo state network for static mappings ELM vs Echo State

27 What does recurrence add? non-linear mixtures of features counteracts the regularization effect of IP ELM + Recurrency

28 What does recurrence add? non-linear mixtures of features counteracts the regularization effect of IP VIDEO ELM + Recurrency

29 What does recurrence add? VIDEO ELM + Recurrency

30 What does recurrence add? counteracts the regularization effect of IP combination useful reduces sensitivity to model selection parameters Abalone task from UCI repository test error ELM IP + ESN hidden input epsilon scaling neurons ELM + Recurrency

31 Digression: ELM+ Intrinsic Plasticity + Recurrence = Reservoir Learning with IP (Steil@HRI, 2006) still works despite recurrence empirically very useful strong regularizer w.r. to #nodes, connectivity pattern, sparseness... achieves both lifetime and spatial sparseness! % of time steps Mackey Glass % of time steps Random without IP epoch 20 epoch without IP epoch 20 epoch neuron output neuron output Steil, 2007, Neural Networks Reservoir, IP, sparseness

32 VIDEO Steil, 2007, Neural Networks Reservoir, IP, sparseness

33 Application: tool manipulation for ASIMO.+ 933&:$4; <==8!$ &'( 3""#$%!" / 933&:$4; +384$4358 &'(!""#$%!" / *+,- )#$%!!" $"657&!8 Rolf & Steil, LAB-RS, 2010, best paper award

34 Application: manipulating a stick Neumann, Rolf & Steil, SAB 2010 TUM/DLR,

35 Positions of the hand center points while holding a stick Positions of the hand center points while holding a stick Y-Axis X-Axis Y-Axis X-Axis Neumann, Rolf & Steil, SAB 2010 TUM/DLR,

36 Outline Extreme Learning Machine (ELM) & Intrinsic Plasticity learn inverse kinematics ELM + Recurrence = Echo State, Reservoir Computing (RC) Associative Neural Learning (for kinematics) kinematics + trajectory generation Programming Dynamics (shape attractor dynamics) learn and select multiple inverse kinematics solutions Learning velocity fields (movement primitives) Learning sequences of movement primitives

37 Associative Neural Reservoir Learning

38 Associative Neural Recurrent Learning ANRL from staged processing to dynamics finally add feedback ELM based: u q Associative Neural Reservoir Learning

39 Associative Neural Recurrent Learning ANRL reservoir based: h Associative Neural Reservoir Learning

40 Echo State Network vs Recurrent Dynamics Text Echo State (staged) - layered update (single pass)! - it takes one time step from u - q: u h q update: u(k) - h(k+1) & readout: q(k+1) = W h(k+1) ANRL (recurrent) - all nodes one RNN! - sychronous update - it takes two time steps from u - q: u h q u(k) - h(k+1) - q(k+2) - needs always prediction!

41 Associative Neural Recurrent Learning ANRL h idea: data pairs (u,q) from some relation (function) "Recurrent neural associative learning of forward and inverse kinematics for movement generation of the redundant PA-10 robot", Reinhart & Steil, LAB-RS, 2008, best paper award

42 Associative Neural Reservoir Learning basic ideas: (u,q) become inputs and outputs store data (u,q) in attractors efficient learning possible (regression/online as before) û(k) ˆq(k) h y(k) u(k) q(k)

43 Associative Neural Reservoir Learning basic ideas: generalization by associative attractor completion choose between multiple solutions through feedback trajectory generation by means of transients x u ˆx h(x, ˆq) ˆq ˆq z 1

44 Our application: Kinematics forward kinematics FK: u=ik(q) uniquely defined (many-to-one) u (hand position) q (arm angles)

45 Kinematics: inverse kinematics IK: q=ik(u) not uniquely defined (one-to-many) q (arm angles) elbow up elbow down u (position) selection of solution = redundancy resolution

46 Example: redundancy resolution: elbow down elbow up

47 association is based on data (not functions) G ={(u^i,q^i)}, i = 1...N read out forward kinematics read out inverse kinematics q G u

48 inverse kinematics (A example-arm-manifolds.mpeg) Rolf, Steil, Gienger, IEEE TAMD, 2010, Goal Babbling permits direct learning of inverse kinematics

49 association by storing pairs (u,q) via SOM u= Fingertip position q= sampled on grip 10x10x10 Walter & Ritter,

50 - (u,q) association by concatenation - train (P)SOM network - read out both ways and get FK and IK! Walter & Ritter,

51 Barhen, Gulati, Zak, Intelligent robots and computer vision, 1989

52 ANRL for kinematics (icub Arm) include forward-model simultaneous learning of both models inverse kinematics 0.1 training data network response h q joint angle [rad] u 2 [m] u 1 [m] time step k forward kinematics, sensory prediction

53 online learning: train from trajectories state in reservoir is useful for temporal integration û(k) ˆq(k) hy(k) u(k) q(k) limit case: attractor u y h q u = FK(q ) q = IK(u )

54 Task: Move Hands Trajectory Representation Inverse Kinematics Robot Control Our approach: learn learn forward and inverse kinematics in associative reservoir network learn Generalization: a) set new target - compute joint angles

55 Evaluation & Generalization (icub 7 DOF arm) joint angle [rad] E[m] time step k projection to y-z (along spiral) regularization by learning joint angle [rad] number of movements time step k probability that E < value u 3 [m] u 2 [m] excellent generalization & graceful degradation continuous shift of attractor possible error E [m] error E [m] Reinhart & Steil, IEEE Conf. Humanoids, 2009

56 Interactive learning of redundancy resolution by ANRL 2010, (Lemme, Rüther, Nordmann, Wrede, Steil, Weirich, Johannfunke)

57 VIDEO

58 Task: Move Hands Trajectory Representation Inverse Kinematics Robot Control Our approach: learn learn forward and inverse kinematics in associative reservoir network Trajectory Generation Generalization: a) set new target - compute joint angles b) movement generation by iteration toward target attractor

59 cope with feedback: internal simulation, sensory prediction Reinhart & Steil, IEEE Conf. Humanoids, 2009

60 Movement generation by attractor dynamics controller setting autonomous operation by sensory prediction Reinhart & Steil, IEEE Conf. Humanoids, 2009

61 Movement generation by attractor dynamics (icub arm) start targets generated movements target start generated movements 3 0 u 3 [m] u 2 [m] u 1 [m] u 2 [m] u 1 [m] home-to-target generalization start-to-home generalization Reinhart & Steil, IEEE Conf. Humanoids, 2009

62 Robustness by attractor dynamics start target generated movements perturbation perturbation! 0.02 u 2 [m] Reinhart & Steil, IEEE Conf. Humanoids, 2009 u 1 [m]

63 Analysis: Speed profiles w.r. to t 3.5 x t = 0.02 t = 0.04 t = 0.08 velocity [m/time step] time step k

64 maximal velocity [m/iteration] t = 0.02 t = 0.04 t = 0.06 t = 0.08 t = target distance [m]

65 Outline Extreme Learning Machine (ELM) & Intrinsic Plasticity learn inverse kinematics ELM + Recurrence = Echo State, Reservoir Computing (RC) Associative Neural Learning (for kinematics) kinematics + trajectory generation Programming Dynamics (shape attractor dynamics) learn and select multiple inverse kinematics solutions Learning velocity fields (movement primitives) Learning sequences of movement primitives

66 Output feedback & programming dynamics problem: how to shape feedback loop? more general: how to imprint arbitrary dynamics? how to generalize and stabelize? x u ˆx h(x, ˆq) ˆq ˆq z 1

67 Programming Dynamics: Example 2-D network + 1-D input to parametrize attractor Reinhart & Steil, ICANN 2011

68 Programming dynamics: directly shape network transients to attractor ELM+feedback or fully recurrent reservoir batch learning (improve by reservoir regularization, see references below) approach generate sequence of desired states get weights through linear regression use smart sampling of state trajectories Reinhart & Steil, ICANN, 2011 Reinhart & Steil, Differential Equations and Dynamical Systems, 2010 Reinhart & Steil, Humanoids 2011, Reinhart PhD-thesis, 2011

69 Programming multiple redundancy resolutions: generate trajectories toward attractor points q (x 1, q 1 ) (x 3, q 3 ) (x 2, q 2 ) q 1 4 (t) q2 4 (t) x 1 =x 2 x 3 =x 4 x Reinhart & Steil, Humanoids 2011

70 Example icub right arm positioning: network setup: - input: 4 DOF joint angles, 3 wrist coordinates - training data: systematic sample (including redundancies!) of icub arm/hand positions 3 wrist x W inp x h(x, q) W out x coordinates uu u ˆx 4 DOF q W inp q W out q ˆq

71 Example icub right arm positioning: associative completion through dynamics xu ˆx u h(xu, ˆq) ˆq ˆq z 1

72 Example icub right arm positioning: in the control loop! xu ˆx u q h(x, q) u τ ˆq

73 Example icub right arm positioning: dynamical selection of solutions q q 3 [deg] q q 1 [deg] 2 [deg] 4 [deg] time step k perturbation

74 Example icub right arm positioning: also mixed constraints possible!

75 Outline Extreme Learning Machine (ELM) & Intrinsic Plasticity learn inverse kinematics ELM + Recurrence = Echo State, Reservoir Computing (RC) Associative Neural Learning (for kinematics) kinematics + trajectory generation Programming Dynamics (shape attractor dynamics) learn and select multiple inverse kinematics solutions Learning velocity fields (movement primitives) Learning sequences of movement primitives

76 Learning velocity fields (movement primitives) confidence output Movement Primitives

77 Learning velocity fields (movement primitives) target demonstrations reproductions Simulation Results ẋ( m/s) ẏ( m/s) x( m) x( m) Training Data: Human Movements y( m) Movement Primitives

78 Learning velocity fields (movement primitives) y (dm) 0 Confidence y (dm) Training Data Default Data Reproduction x (dm) x (dm) Movement Primitives

79 Generalization with confidence Movement Primitives

80 Learning velocity fields (movement primitives) VIDEO Movement Primitives

81 outlook: neural dynamic motion primitives integration du/dt standard motion generation via DMP allows to learn trajectory shapes (AMARSi D4.1 update)

82 Outline Extreme Learning Machine (ELM) & Intrinsic Plasticity learn inverse kinematics ELM + Recurrence = Echo State, Reservoir Computing (RC) Associative Neural Learning (for kinematics) kinematics + trajectory generation Programming Dynamics (shape attractor dynamics) learn and select multiple inverse kinematics solutions Learning velocity fields (movement primitives) Learning sequences of movement primitives Learning sequences of movement primitives

83 Sequencing of movement primitives train for constant value of sequencer relative start coordinates type of movement Lemme, work in progress, unpublished train for zero value of velocity switch feedback off while running Movement Primitives

84 Sequencing of movement primitives Movement Primitives

85 Sequencing of movement primitives Movement Primitives

86 Summary: Extreme Learning Machine (ELM) & Intrinsic Plasticity learn inverse kinematics ELM + Recurrence = Echo State, Reservoir Computing (RC) Associative Neural Learning (for kinematics) kinematics + trajectory generation Programming Dynamics (shape attractor dynamics) learn and select multiple inverse kinematics solutions Learning velocity fields (movement primitives) Learning sequences of movement primitives Movement Primitives

87 Current work: associate visual feedback audio-visuo-motor patterns combine trajectory generation and inverse kinematics provide training data in autonomous exploration use platforms you can not model! VIDEO

88 R. F. Reinhart, Associative Learning, Programming Dynamics, icub, PhD Student M. Rolf, Motion Learning on ASIMO, Goal Babbling, PhD Student K. Neumann, Bimanual Motion Learning on ASIMO, Intrinsic Plasticity A. Lemme, Sequencing of Motion Primitives FlexIRob: A. Nordmann A. Lemme S. Rüther A. Weirich M. Johannfunke Dr. S. Wrede S. Krüger M. Götting Cognitive Systems Engineering: System Integration, ASIMO, icub, Kuka support

89 Publications Regularization and stability in reservoir networks with output feedback. R. F. Reinhart and J.J. Steil, Neurocomputing, conditionally accepted Optimizing Extreme Learning Machines via Ridge Regression and Batch Intrinsic Plasticity Neumann, K., and J.J. Steil, Neurocomputing, conditionally accepted Batch intrinsic plasticity for extreme learning machines. K. Neumann and J.J. Steil. ICANN, pages , State prediction: A constructive method to program recurrent neural networks. R. F. Reinhart and J.J. Steil. ICANN, pages , Neural learning and dynamical selection of redundant solutions for inverse kinematic control. R. F. Reinhart and J.J. Steil, IEEE Humanoids, 2011 A constrained regularization approach for input-driven recurrent neural networks. R. F. Reinhart and J.J. Steil. Differential Equations and Dynamical Systems, 19:27 46, Reservoir regularization stabilizes learning of Echo State Networks with output feedback. R. Felix Reinhart and J.J. Steil, ESANN, pp , 2011 Teaching and Learning Redundancy Resolution for Autonomous Generation of Flexible Robot Movements. S. Wrede, M. Johannfunke, A. Lemme, A. Nordmann, S. Rüther, A. Weirich, J.J. Steil, Workshop Computational Intelligence, GMA-FA 5.14, Dortmund, 2010 Learning Flexible Full Body Kinematics for Humanoid Tool Use. M. Rolf, J.J. Steil and M. Gienger, Int. Symp. Learning and Adaptive Behavior in Robotic Systems, 2010 Learning Inverse Kinematics for Pose-Constraint Bi-Manual Movements. K. Neumann, M. Rolf, J.J. Steil and M. Gienger, Int. Conf. Simulation of Adaptive Behavior, 2010 Recurrence enhances the spatial encoding of static inputs in reservoir networks. C. Emmerich, F. R. Reinhart, and J. J. Steil. ICANN, pp , Attractor-based computation with reservoirs for online learning of inverse kinematics. R. F. Reinhart, and J.J. Steil, Proc. ESANN, pp , 2009

90 Efficient exploration and learning of whole body kinematics. M. Rolf, J.J. Steil, and M. Gienger, Proc. Int. Conf. Developmental Learning, 2009 Reaching movement generation with a recurrent neural network based on learning inverse kinematics. R. F. Reinhart and J.J. Steil. IEEE Conf. Humanoid Robotics, pages , Recurrent neural associative learning of forward and inverse kinematics for movement generation of the redundant PA-10 robot. R. F. Reinhart and J.J. Steil. Learning Adaptive Behavior in Robotic Systems, pp , Improving reservoirs using intrinsic plasticity. Schrauwen B., Wardermann M., Verstraeten D., Steil J.J., Stroobandt D., Neurocomputing, pp , 2008 Online reservoir adaptation by intrinsic plasticity for backpropagation-decorrelation and echo state learning. J.J. Steil. Neural Networks, 20(3): , Online Stability of backpropagation-decorrelation recurrent learning. J.J. Steil, Neurocomputing, vol. 69(7-9), pp , 2006 (some, few) related publications: A. Lemme, R. F. Reinhart, and J. J. Steil. Efficient online learning of a non-negative sparse autoencoder. ESANN, pages 1 6, Butko and Triesch. Exploring the Role of Intrinsic Plasticity for the Learning of Sensory Representations. Neurocomputing, 70(7-9): , Lukoševičius and Jaeger, Reservoir computing approaches to recurrent neural network training, Jaeger, The "echo state" approach to analyzing and training recurrent neural networks, Baraniuk, R, Wakin, M., Random Projections of Smooth Manifolds, Foundations of Computational Mathematics, vol(9), no. 1, 51-77, 2009 HG.-B. Huang, Q.-Y. Zhu, and C.-K. Siew, Extreme learning machine: Theory and applications, Neurocomputing, vol. 70, no. 1-3, pp , G.-B. Huang, L. Chen, and C.-K. Siew, Universal approximation using incremental constructive feedforward networks with random hidden nodes, IEEE Transactions on Neural Networks, vol. 17, no. 4, pp. 879{892, 2006.

91 Thank you for your attention! x q h(x, q) τ ˆx ˆq more information: