A mixed structural modeling approach to personalized binaural sound rendering

Size: px

Start display at page:

Download "A mixed structural modeling approach to personalized binaural sound rendering"

Scarlett Brooks
8 years ago
Views:

1 A mixed structural modeling approach to personalized binaural sound rendering Federico Avanzini Michele Geronazzo Simone Spagnol Sound and Music Computing Group Dept. of Information Engineering University of Padova

Geronazzo Simone Spagnol Sound and Music Computing

2 Outline Introduction Sound and Music Computing in Padova Motivations Binaural listening Individualized virtual auditory space Binaural technologies Customized binaural audio delivery Mixed structural models Pinna models Extraction of pinna features Image-based HRTF selection Applications Auditory feedback for exploration tasks Orientation and mobility aids

delivery Mixed structural models Pinna models Extraction of pinna features Image-based

3 Sound and Music Computing in Padova 1960s-1970s: the CSC, Centro di Sonologia Computazionale Electronic music at the Conservatory of Padova Research on speech synthesis 1980s: Real-time sound synthesis and processing 1990s: Expressive information processing 2000s: Multimedia and multimodality Today: SMC Group, Dept. of Information Engineering Technologies: physical modeling, binaural 3D audio, multimodal interaction, audio processing/restoration Application domains: musical instruments/interfaces, virtual rehabilitation, technology augmented learning, interfaces for the visually impaired, musical/audio archives

4 Individualized 3D auditory displays Everywhere Mobile devices Headphones

5 Individualized 3D auditory displays Everytime GP-GPU Virtual scenario

6 Individualized 3D auditory displays Everyone

7 Auditory localization cues Acoustic Waves Inner Ear Outer Ear Transformations Body reflections Head diffraction Room Middle Ear Static and dynamic localization cues

$diffraction Room Middle Ear Static and$

8 Head Related Transfer Functions (HRTFs) Highly-dependent on individual anthropometric data

9 HRIR/HRTF databases and formats

10 Mixed structural models

11 Mixed Structural Models (MSMs) Each physical effect is processed separately Structural components Structural models (Brown&Duda 1998) Synthetic pole/zero models series expansions acoustic simulations Measured Extrapolated Partial Head-Related Impulse Responses (phrirs) Acoustic information related to specific body parts

models series expansions acoustic simulations Measured Extrapolated Partial

12 MSM definition Given a subject i: Selection Individual Model

MSM research workflow MSM goals: Progressively remove individual partial components (I = 0) Provide techniques to phrtf modeling/selection Evaluate

13 MSM research workflow MSM goals: Progressively remove individual partial components (I = 0) Provide techniques to phrtf modeling/selection Evaluate their combinations Given a set of subjects and a set of models: Find best combination of modeled and selected components, including balance between S and M

their combinations Given a set of subjects and a set of models: Find best

14 Pinna models Pinna-related cues (high-frequency) associated to elevation perception Ad-hoc designed algorithm for separating resonant and reflective components of Pinna-RTFs (PRTFs) Multinotch-multiresonance filter

algorithm for separating resonant and reflective

15 Pinna models: anthropometric fitting Relation between PRTF notches and main pinna contours: Optimal ray-tracing on a set of subjects Contour extraction

16 Example: Pinnaless KEMAR HRTFs Mixed Structural model with N=2, M=S=1, I=0 Head: spherical model with optimal radius Torso: CIPIC subject with closest shoulder width

17 Example: full KEMAR HRTF (small pinna) Mixed Structural Model with N=2, M=I=1, S=0 Head+torso: pinnaless KEMAR Pinna: pinna model with anthropometric fitting

18 Example: anthropometry-based selection Mixed Structural Model with N=1, M=I=0, S=1 Non-individual HRTF selection on CIPIC db Localization task: C1-based selection Increases average elevation performances by 17% Enhanches externalization Reduces up/down confusions

Localization task: C1-based selection Increases average elevation

19 Applications

20 Orientation & Mobility Learning Provide users with offline information about a given ennvironment Haptic mouse (the TAMO) and 3D audio 1. Reaching 2. Object reconstruction 3. Location/size estimation

Reaching Experimental results Location/size estimation Location: Improvement in performance TAMO 2Daudio+TAMO: 28% 2Daudio 2Daudio+TAMO: 27% localization errors significantly

21 Reaching Experimental results Location/size estimation Location: Improvement in performance TAMO 2Daudio+TAMO: 28% 2Daudio 2Daudio+TAMO: 27% localization errors significantly different between TAMO and TAMO + Audio3D, p = 0.02 Size: Pearson's Chi-Square test of independence of incorrect/correct, p = 0.05, reduced size errors: -35% with TAMO control condition

22 Ongoing/future work Mixed structural models Estimation of phrtfs from full HRTFs Increased number of structural components (e.g., ear canal, headphones) Individualized headphone compensation Evaluation Psychophysical experiments Computational auditory models Usability tests Realizations Robust automatic pinna contour estimation In-place sensing and processing on a mobile device Augmented headphones (head tracking, integrated camera,...) Need for ground truth (true subjects' HRTFs /phrtfs) Recordings BEM simulations

23 Grazie dell'ascolto

Convention Paper Presented at the 137th Convention 2014 October 9 12 Los Angeles, USA

Audio Engineering Society Convention Paper Presented at the 137th Convention 2014 October 9 12 Los Angeles, USA 9126 This Convention paper was selected based on a submitted abstract and 750-word precis