Perspektivenvorlesung Information Retrieval Music and Motion Bonn University Prof. Dr. Michael Clausen PD Dr. Frank Kurth Dipl.-Inform. Christian Fremerey Dipl.-Inform. David Damm Dipl.-Inform. Sebastian Ewert Dr. Tido Röder Habilitation Meinard Müller Saarland University and MPI Informatik meinard@mpi-inf.mpg.de Dec. 2007 PhD students Winter Term 2008/2009 Dipl.-Inform. Andreas Baak Dipl.-Math. Verena Konz Dipl.-Ing. Peter Grosche Dipl.-Inform. Thomas Helten (DFG) (MMCI) (MMCI) (DFG) Music Data Music Data Various interpretations Beethoven s Fifth Bernstein Karajan Scherbakov (piano) MIDI (piano) General Goals Music Synchronization: Audio-Audio Beethoven s Fifth Automated organization of complex and inhomogeneous music collections Karajan Generation of annotations and cross-links Tools and methods for multimodal search, navigation and interaction Scherbakov Music Information Retrieval (MIR)
Music Synchronization: Audio-Audio Beethoven s Fifth Music Synchronization: Audio-Audio Feature extraction: chroma features Karajan Karajan Scherbakov C C# 1 0.9 C C# 1 0.9 D 0.8 D 0.8 D# 0.7 D# 0.7 Scherbakov E F F# G G# A 0.6 0.5 0.4 0.3 0.2 E F F# G G# A 0.6 0.5 0.4 0.3 0.2 A# B 2 4 6 8 10 12 14 16 18 0.1 0 A# B 5 10 15 20 0.1 0 Synchronization: Karajan Scherbakov Music Synchronization: Audio-Audio Cost matrix Music Synchronization: Audio-Audio Cost-minimizing warping path 1 1 18 0.9 18 0.9 16 0.8 16 0.8 14 0.7 14 0.7 Karajan 12 10 8 0.6 0.5 0.4 Karajan 12 10 8 0.6 0.5 0.4 6 0.3 6 0.3 4 0.2 4 0.2 2 0.1 2 0.1 2 4 6 8 10 12 14 16 18 20 0 2 4 6 8 10 12 14 16 18 20 0 Scherbakov Scherbakov System: SyncPlayer/AudioSwitcher Music Synchronization: MIDI-Audio
Music Synchronization: MIDI-Audio Music Synchronization: Scan-Audio MIDI = meta data Automated annotation Audio recording Sonification of annotations Music Synchronization: Scan-Audio Music Synchronization: Scan-Audio Scanned Sheet Music Scanned Sheet Music Symbolic Note Events OMR Correspondence Correspondence Audio Recording Audio Recording Music Synchronization: Scan-Audio Music Synchronization: Scan-Audio Scanned Sheet Music Symbolic Note Events Scanned Sheet Music Symbolic Note Events OMR High Qualtity OMR Dirty but hidden Correspondence Correspondence High Qualtity Audio Recording Audio Recording
System: SyncPlayer/SheetMusic Music Synchronization: Lyrics-Audio Difficult task! Music Synchronization: Lyrics-Audio Lyrics-Audio Lyrics-MIDI + MIDI-Audio Music Synchronization Turetsky/Ellis (ISMIR 2003) Soulez/Rodet/Schwarz (ISMIR 2003) Arifi/Clausen/Kurth/Müller (ISMIR 2003) Hu/Dannenberg/Tzanetakis (WASPAA 2003) Müller/Kurth/Röder (ISMIR 2004) Raphael (ISMIR 2004) Dixon/Widmer (ISMIR 2005) Müller/Mattes/Kurth (ISMIR 2006) Dannenberg /Raphael (Special Issue ACM 2006) Fujihara/Goto (ICASSP 2008) Wang/Iskandar/New/Shenoy (IEEE T-ASLP 2008)
Similarity cluster
Global structure
Global structure Global structure System: SyncPlayer/AudioStructure Dannenberg/Hu (ISMIR 2002) Peeters/Burthe/Rodet (ISMIR 2002) Cooper/Foote (ISMIR 2002) Goto (ICASSP 2003) Chai/Vercoe (ACM Multimedia 2003) Lu/Wang/Zhang (ACM Multimedia 2004) Bartsch/Wakefield (IEEE Trans. Multimedia 2005) Goto (IEEE Trans. Audio 2006) Müller/Kurth (EURASIP 2007) Music Information Retrieval Multimodal Computing and Interaction Sheet Music (Image) MIDI CD / MP3 (Audio) Music Synchronization Audio Matching Music
Multimodal Computing and Interaction Motion Capture Data Sheet Music (Image) MIDI CD / MP3 (Audio) Digital 3D representations of motions MusicXML (Text) Music Singing / Voice (Audio) Computer animation Sports Music Literature (Text) Music Film (Video) Dance / Motion (Mocap) Gait analysis Motion Capture Data Motion Capture Data Motion Retrieval Motion Retrieval = MoCap database = query motion clip Goal: find all motion clips in similar to
Notion of Similarity Relational Features Numerical similarity vs. logical similarity Logically related motions may exhibit significant spatiotemporal variations Relational Features Relational Features Induced feature sequence: Relational Features Motion Retrieval
Relational Features Motion Templates Spatio-temporal invariance Indexing Efficient retrieval & preselection Problem: feature design & selection Motion Templates Motion Templates Motion Templates Motion Templates
MT-based Motion Retrieval MT-based Motion Retrieval: Jumping Jack MT-based Motion Retrieval: Jumping Jack MT-based Motion Retrieval: Elbow-To-Knee τ MT-based Motion Retrieval: Elbow-To-Knee MT-based Motion Retrieval τ
MT-based Motion Retrieval Conclusions Automated data organization Exploiting multimodality Synchronization Handling object deformations Indexing and efficiency Conclusions Lecture Music Processing Textbook Summer Term 2009 Thursdays 16-18 MPI, Room 024 3 Credit Points Website Meinard Müller Information Retrieval for Music and Motion 2007, XVI. 318 pages 136 illus. 39 in Color, Hardcover ISBN: 978-3-540-74047-6 www.springer.com/978-3-540-74047-6/ 69,50 EUR Selected Publications M. Müller (2007): Information Retrieval for Music and Motion. Monograph, Springer, 318 pages F. Kurth, M. Müller (2008): Efficient Index-Based Audio Matching. IEEE Trans. Audio, Speech & Language Processing, Vol. 16, No. 2, 382-395. M. Müller, D. Appelt (2008): Path-constrained Partial Music Synchronization. Proc. International Conf. on Acoustics, Speech, and Signal Processing (ICASSP 2008) M. Müller, F. Kurth (2006): Enhancing Similarity Matrices for Music Audio Analysis. Proc. International Conf. on Acoustics, Speech, and Signal Processing (ICASSP 2006) M. Müller, T. Röder (2006): Motion Templates for Automatic Classification and Retrieval of Mocap Data. Proc. ACM SIGGRAPH / Eurographics Symposium on Computer Animation (SCA 2006) M. Müller, T. Röder, M. Clausen (2005): Efficient Content-Based Retrieval of Motion Capture Data. ACM Trans. Graph. 24 (SIGGRAPH 2005)