Sound and Music Computing Research: Trends and Challenges. Xavier Serra Music Technology Group Universitat Pompeu Fabra, Barcelona

Transcription

1 Sound and Music Computing Research: Trends and Challenges Xavier Serra Music Technology Group Universitat Pompeu Fabra, Barcelona

2 Introduction

3 Music communication framework Musical Knowledge base Composer Listener Perception Cognition Symbolic representation Temporal controls Sound field Room Performer Source sound Physical Knowledge base Instrument (Moorer, 1990)

4 SMC research disciplines Music Theory Performance Composition Artificial Intelligence Computer Science Programming Psychomusicology Psychology Cognition SMC Digital hardware Psychoacoustics Digital signal processing Acoustics Device design Engineering Physics (Moorer, 1990)

5 New context for SMC research CREATION (production tools) content processing CONTENT (databases) MUSIC 3.0 collaborative creativity content sharing COMMUNITY (social tools)

6 Basic Research Challenges

7 Understanding sounding objects String vibration Membrane vibration

8 Understanding perception Diagram of inner ear

9 Understanding cognition research (Peretz and Coltheart, 2003)

10 Current Trends and Challenges

11 Sound synthesis Current trends: Audio content processing Physical modeling based on measurements Traditional approach to sound synthesis Gesture-sound modeling

12 Content processing Vocaloid (Yamaha & MTG-UPF, 2005)

13 Physical modeling from measurements Various sensors on a violin Measuring impulse responses from a violin

14 Gesture-sound modeling Performed scores Audio Bow position Bow velocity Bow acceleration Bow pressing force Bow- bridge distance String Playing state estimator Finger position Violin synthesis Pachelbel Canon

15 Sound/Music description Current trends: Search and retrieval Combining bottom-up and top-down approaches (Lesaffre et alt., 2003)

16 Search and retrieval Harmonic analysis (Gomez, 2006) MusicSurfer

17 Bottom-up and top-down Music recommendation system (Celma, 2006)

18 Interaction Current trends: Understanding performance expression New musical interfaces Mathews with the Radio Drum

19 Understanding performance obtain encode High-quality recordings Machine representation analyze Expressive aspects of recordings Structure of pieces Machine learning Models Analyze extract Symbolic description Synthesized score Automatic expression

20 New interfaces Reactable (Jordà, 2006)

21 Broad Challenges

22 Bridging the semantic gap Human Knowledge memories understanding personal identity opinions emotions Content Objects genre source harmony dynamics Signal features melody loudness time timbre spectrum intensity duration pitch frequency semantic gap music scores similarity rhythm expectations semantic features labels sentences tags shot rhythm signs motions scenes adjectives verbs contrasts textures articles numbers graphic style nouns colors shapes Audio Text Image (music recordings) (lyrics, editorial text, press releases, ) (video clips, CD covers, printed scores, ) Music information plane

23 Bridging the semantic gap Human Knowledge memories understanding personal identity opinions emotions Content Objects Statistical dynamics modeling Signal features genre source melody Machine learning harmonymusic theory loudness time timbre Signal spectrum processing intensity duration pitch frequency semantic gap music scores similarity rhythm expectations semantic features labels sentences Web mining tags shot rhythm signs motions scenes adjectives verbs contrasts textures articles numbers graphic style nouns colors shapes Audio Text Image (music recordings) (lyrics, editorial text, press releases, ) (video clips, CD covers, printed scores, ) Music information plane

24 Bridging the semantic gap Human Knowledge Signal features personal identity opinions emotions Computational neuroscience Content Objects memories understanding Music Computational cognitionsimilarity musicology expectations music Reasoning scores genre rhythm graphic Text rules semantic style source melody understanding labels shotmultimodal Machine features processing learning harmony Music Ontologies rhythm signs tags Statistical theory sentences dynamics Web motions modeling mining loudness time timbre Signal spectrum processing intensity duration pitch frequency scenes adjectives verbs contrasts textures articles numbers semantic gap nouns colors shapes Audio Text Image (music recordings) (lyrics, editorial text, press releases, ) (video clips, CD covers, printed scores, ) Music information plane

25 Understanding music making (Leman, 2007)

26 Understanding communication Score A Score B Score C Score D COMPOSITION Performer A Performer B Performer C Performer D Instrument A Instrument B Instrument C Instrument D PERFORMANCE Compositional channel: musical message + role in the performance (solo, accompanist, etc ) Visual channel AUDIENCE Audience indiv. 1 Audience indiv. 2 Audience indiv. 3 Sonic channel Instrumental channel: soundproducing and modifying movements / actions, haptic feedback

27 Understanding social interaction