Active Audition and Sensorimotor Integration for Sound Source Localization

Transcription

1 Active Audition and Sensorimotor Integration for Sound Source Localization Mathieu Bernard 25 novembre /27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

2 Introduction CIFRE thesis. Co-direction : Patrick Pirim, Brain Vision Systems Bruno Gas, ISIR, UPMC Alain de Cheveigné, LPP, Paris Descartes Outline Artificial auditory system for sound localization, Audio-tactile model for texture recognition, Active audition and sensorimotor integration. 2/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

3 Autonomous robotics Psikharpax, the robot rat 3/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

4 Artificial auditory system Bioinspired sound localization Outer ear Outer ear Inner ear Inner ear Binaural localiation cues ITD - ILD Auditory model Implementation robot sound capture core library real time - c++ standalone programs wav files robotic simulation simulated sound sources 4/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

5 Outer ear model Pinna Auditory fovea Microphone Oriented toward the fovea Support Servomotor Outer ear = Pinna, microphone and software capture. Spectral and directional cues. 5/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

6 Inner ear model (1/2) Inner ear = cochlear model and pulse train generation. Cochlear model Gammatone filterbank Usually 30 channels from 300Hz to 8kHz at 20kHz 6/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

7 Inner ear model (2/2) Inner ear = cochlear model and pulse train generation. Pulse train generation Cochlear channel output Pulse train Discrete representation, noise suppression. A pulse = temporal and amplitude information. 7/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

8 ILD computation (1/2) Energy for each pulse p(t), the energy is : E(t) = u=t u=t T p(u) 2 (1) 8/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

9 ILD computation (2/2) ILD = Interaural Level Difference For each channel i : ILD i (t) = 2E left,i (t) E left,i (t)+e right,i (t) 1 9/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

10 ITD computation (1/3) ITD = Onset extraction and delay lines. Onset extraction from a pulse train Comparison with a dynamic threashold, 2 parameters. 10/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

11 ITD computation (2/3) Delay lines model ITD = Onset extraction and delay lines. 11/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

12 ITD computation (3/3) ITD = Onset extraction and delay lines. 12/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

13 Auditory and tactile model for texture perception (1/3) Similar model for transduction and processing Audition and touch support fine texture discrimination skills. Rat vibrissae and cochlea transduction based on resonance. Strong interaction for auditory/tactile spectral processing. 13/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

14 Auditory and tactile model for texture perception (2/3) Whiskers filterbank adapted from gammatone cochlear model : Feature extraction = Instantaneous Mean Power : 14/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

15 Auditory and tactile model for texture perception (3/3) Texture classification : 8 textures, 3 layer perceptron Response to a pure tone (whisker at 630 Hz) Influence of the number of channels 15/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

16 Auditory Evoked Behaviors (1/2) Outer ear Inner ear Energy ILD Outer ear Inner ear Energy Auditory model Motor control Neck and body Motor control Neck ILD minimization ILD < 0 turn right, ILD > 0 turn left. Video! Wheels Follow neck orientation Constant speed, Smooth rotations. Video! 16/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

17 Auditory Evoked Behaviors (2/2) Phonotaxis trajectories. Front-back disambiguation. 17/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

18 Sensorimotor Approach (1/3) Environment state e E and motor state m M, Sensory state s S, we have s = Φ(m, e), Φ is called a sensorimotor law. 18/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

19 Sensorimotor Approach (2/3) Quand on dit [...] que nous localisons tel objet en tel point de l espace [...] cela signifie simplement que nous nous représentons les mouvements qu il faut faire pour atteindre cet objet. [...] Nous nous représentons les sensations musculaires qui accompagnent [ces mouvements] et qui n ont aucun caractère géométrique, qui par conséquent n impliquent nullement la préexistance de la notion d espace. H. Poincaré, L espace et la géométrie, Proposed formalization for localization Find the motor state m such as m = argmin Φ(m, e) Φ(m 0, e 0 ). m M 19/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

20 Sensorimotor Approach (3/3) Sound source localization Two ways for m estimation Orienting behavior After completion we have m = m end s end = Φ(m end, e) = Φ(m 0 + δm, e 0 + δe), Manifold learning Sensory space S lies on a low-dim manifold R, Dimension reduction technique : S R, Same topology as the embodying space (at least locally). 20/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

21 Manifold learning (1/3) CAMIL database, ILD vectors : 21/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

22 Manifold learning (2/3) CAMIL database, ITD vectors : d=690 d=3 22/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

23 Manifold learning (3/3) 2 outer ears models : HRTF and directive filters Front-back disambiguation with spectral cues : 23/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

24 Sensorimotor integration (1/2) Learning algorithm Iterative learning of R, Self-supervised. 24/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

25 Sensorimotor integration (2/2) Simulation results over 400 experiments 25/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

26 Intrinsic Dimension Estimation Simulation (ILD, azimtuh 180 & 360) CAMIL dataset (ILD & ITD, azimtuh 360, elevation [-30, 30]) 26/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration

27 Perspectives SensoriMOTOR 27/27 Mathieu Bernard - ISIR - BVS Active Audition and Sensorimotor Integration