Methods for Analysing Large-Scale Resources and Big Music Data

Transcription

1 Methods for Analysing Large-Scale Resources and Big Music Data Tillman Weyde Department of Computer Science Music Informatics Research Group City University London

2 Overview! Large-Scale Music Analysis! Big Music Data! Available technologies and methods! The Digital Music Lab! Architecture! Data! Example! Chord recognition! Chord sequence analysis! Visualisations! Hands-on tasks! Exploring British Library content 2 The DML Project: Objectives and Methodology

3 Large Scale Music Analysis! Music has gone digital on a large scale.! What about musicology?! What is different about large scale data! How can we analyse large scale music data 3 The Digital Music Lab Project

4 Large Scale Music Analysis! Big Music Data Collections become increasingly available digitally, e.g.! itunes and Spotify (and others) offer over 30 m tracks each! British Library holds ~ 3 millions of audio recordings (~10% are digitized), 1.5 million music prints, 100k manuscripts! Internet archive: 2.5 m audio tracks 4 The DML Project: Objectives and Methodology

5 Large Scale Music Analysis! Big Music Data in Research! Systematic Musicology has developed as "data oriented empirical research" Parncutt, R Systematic musicology and the history and future of Western musical scholarship. Journal of Interdisciplinary Music Studies, 1, 1-32.!... working with larger data sets will open up new areas of musicology. N. Cook (2005). The Compleat Musicologist. Keynote speech at ISMR. 5 The DML Project: Objectives and Methodology

6 Big Data Analysis Workflow! Acquisition, Storage and Management! Large Hardware & Software Systems! Exploration, Hypothesis development! Query & Search, Visualization! Modelling & Testing! Statistical tools 6

7 Big Data Analysis Technology! Parallel processing on large arrays! Cheap unreliable hardware! Software Architecture! Map/Reduce, Computation Graphs (Hadoop, Spark)! Algorithms! Failure-tolerant, efficient, simple! Visualisations 7

8 Big Music Data Applications (1)! Popular:! Animated! Recommendation (network based) Narrative 2.0 Liveplasma 2.0 8

9 Big Music Data Applications (2)! Not many: e.g. music history graph by Google (since 1950, not classical music )! 9

10 Digital Transformations in Musicology! Challenges! Gap between musicology and music technology (music information retrieval)! Large heterogeneous data collections! Need for software infrastructure! Audio and symbolic music processing! Connecting resources (semantic web, linked data)! Tools and visual interfaces! Methods for gaining musical insight from data 10 The Digital Music Lab Project

11 Open Questions (partly)! How can music research use audio transcription and analysis on large data collections?! How can we provide an infrastructure that enables researchers to make use of large data collections and create reusable open datasets?! How can computational tools be made usable for music researchers, musicians and other users (who are not necessarily computer scientists)? 11 The Digital Music Lab Project

12 Musicological Questions From 2014 Workshop! Analysing styles, trends over time! Work across different heterogenuous collections! Utilise external metadata and annotations 12

13 Infrastructure needed! Feature Extraction! Vamp and other plug-ins! Parallelisation! Middleware! Semantic Web! Music Ontology! Aggregation and collection level analysis 13 13/03/2015

14 Large Scale Music Analysis! Plan for this session! Show Digital Music Lab and our approach to large scale music analysis! Workflow and technologies! Present features and interfaces! Hands-on data exploration! Methods for further analysis! Discussion 14

15 The Digital Music Lab 15

16 The Digital Music Lab project! January March 2015 small follow-up project running now! City University (Dpt of Computer Science, Dpt of Music)! Tillman Weyde, Stephen Cottrell, Jason Dykes, Emmanouil Benetos, Daniel Wolff, Dan Tidhar, Alex Kachkaev! Queen Mary UoL (Centre for Digital Music)! Mark Plubmley, Simon Dixon, Mathieu Barthet, Steven Hargreaves! University College London (Dpt of Computer Science, Centre for Digital Humanities)! Nicolas Gold, Samer Abdallah! British Library (BL Labs)! Aquiles Alencar-Brayner, Mahendra Mahey, Adam Tovell 16

17 Goals! Develop a networked infrastructure to bring computation to the data! Avoid copyright problems by design! Integrate audio feature extraction and transcription! Development of analysis tools! Interactive visual interfaces! Musicological applications 17 13/03/2015

18 Outputs! Curated datasets and derived data (>4 Terabytes)! Web service with visual interfaces for data exploration! Publications (more to come)! Redistributable virtual machine images (in preparation) 18 13/03/2015

19 The Digital Music Lab! Overview 19

20 The DML System Provides...! Access: Systematic exploration of heterogenuous and large music libraries! Control: Interfacing with complex automatic music analysis tools! Analysis: Gain summarised knowledge on large numbers of recordings! Sharing: Experiments reproducible with same data, clear provenance of analysis results. 20

21 The Technical Perspective! Access to data! Audio access restricted by physical location! Metadata unification of different formats! Control via web interface to large-scale analysis! Interactive UI for overview and exploration! Scalable analysis is available on collection-level and recording-level! Share the well-defined and derived data! Re-use of existing software and published code for analysis 21

22 Software Ecosystem! Distributed system! Virtual machines (VirtualBox)! Open Source OS (Ubuntu)! Parallelised existing analysis tools! Python (NumPy)! Vamp Plugins! Big-Data map-reduce (Spark)! Computation management! Built on semantic architecture! Interactive user interface for exploration and analysis! Built using state-of-the-art web technologies 22

23 Data-Flow for Computational Analysis User Interface Provide Web Server Database: Results & Metadata Analysis Management: Cliopatria Audio, Transcriptions and Feature Storage Access Audio and Features Computing Server 23

24 Physical Locations Matter: Content Access! Two computing servers, located at BL and ILM! Allow for in-place access to restricted data! Dedicated server at City for web access 24

25 Sustainability! Preference on Open Source! Basic infrastructure (Ubuntu, Spark, Vamp...)! Soundsoftware repository for! Publishing versioned code of newly developed software! Backup and sharing: Open data / features / results! Open and reproducible method! Enables similar set-up in further institutions 25

26 Results Implemented in the DML System! Conceptual framework (including implementation) for collection-level analysis! Collection in focus as object of analysis! Data-flow for interactive retrieval of results! Secure, responsive and redundant network structure! Distributed computation ressources! Open-source software ecosystem for large-scale music analysis! Parallelised feature extraction and results management! Collection-level analysis, interface and visualisation 26

27 Feature Extraction 27

28 Audio Descriptors List 1. Spectrogram 2. MFCCs 3. Chroma 4. Onsets 5. Speech/Music Segmentation 6. Chords 7. Beats/Tempo 8. Key 9. Melody 10. Note Transcription 28

29 Raw Audio 1. B Sample from CHARM: JS Bach, Chorale Prelude - Beloved Jesus, Cohen, Harriet (piano), Columbia,

30 Spectrogram 2 versions:! STFT magnitude spectrogram! Constant-Q Transform magnitude spectrogram 30

31 MFCCs! Stand for: Mel-Frequency Cepstral Coefficients! Extracted using QM Vamp Plugin Set 31

32 Chroma! Spectrum projected onto 12 bins (representing semitones of an octave)! Extracted using: QM Chromagram and NNLS Chroma Vamp plugins 32

33 Onsets! Onset: the beginning of a musical note or another sound! Extracted using QM Onset Vamp plugin 33

34 Speech/Music Segmentation! Useful for ethnographic recordings/radio broadcasts! Extracted using BBC Speech/Music Segmentation Vamp Plugin 34

35 Chords! Extracted using Chordino Vamp Plugin 35

36 Beats! Beat locations labelled with metrical position! Extracted using Beatroot, Marsyas, Tempotracker Vamp Plugins 36

37 Tempo! Estimated based on onset/beat information! Extracted using Tempotracker and Tempogram Vamp plugins 37

38 Key! Extracted using QM Key Vamp plugin (supports major/minor keys) 38

39 Melody! Or more precisely: Sequence of fundamental frequency (F0) values corresponding to the perceived pitch of the main melody.! Extracted using MELODIA Vamp plugin 39

40 Note Transcription Semitone Resolution! Multiple-pitch detection (onset/offset/pitch/velocity)! Extracted using Silvet Vamp plugin! Synthesized transcription example: 40

41 Note Transcription High Pitch Resolution! Multiple-pitch detection on a 20-cent resolution useful for tuning/ temperament analysis and analysis of non-western music! Extracted using Silvet Vamp plugin 41

42 Data! British Library! CHARM! I Like Music 42

43 Data - British Library Music Dataset! Currently identifying, organising, and curating available music data collections from the BL Sound Archive! Over 3M digital audio recordings, in a variety of formats! Copyright-cleared material will be made available to the public! Copyright-restricted material will be accessible to BL users 43

44 Data I Like Music Dataset! I Like Music: digital music service provider to companies who hold public performance licences! Sole provider of online music to the BBC! Holds a commercial music library of 1.2M tracks and a production music library of 400k tracks 44

45 Data CHARM and Mazurka Dataset! CHARM: AHRC Research Centre for the History and Analysis of Recorded Music ( )! CHARM Dataset: 5k copyright-free historical recordings ( ) + metadata! Mazurka Dataset 3k recordings + metadata! Ideal for musicological analysis using computational methods 45

46 Extracted features available today! ILM, BL, CHARM datasets ~ tracks! Transcriptions MIDI and high resolution! Beats and tempo curves! Chroma! Chord and key 46

47 Infrastructure Feature Extraction Vamp plug-ins Spark and other techniques for parallelisation Middleware Semantic Web server (RDF with Prolog using ClioPatria) Music Ontology Manages aggregation and collection level analysis Provides SPARQL endpoint 47

48 Infrastructure Derived data from 2 collections! Accessible via the web 48

49 Interfaces and Visualisations Audio collections Chord sequence patterns Tag crowd-sourcing 49

50 Studies! Temperament! Chord progressions 0.5 ET Just Vallotti fcmtge scmtfd Kellner Lehman

51 Examples! Chord Sequence Analysis 51

52 Large-Scale Analysis of Chord Sequences! Extracted chord sequences (e.g. Am7/E7/Gmaj7, etc...)! On ILM's commercial music collection (1m tracks )! Parallel processing of multiple music clips! Chordino Vamp plugin (Queen Mary University of London)! 6 weeks on 8 core virtual machine! Retrieve most frequent chord patterns using Sequential Pattern Mining (SPM)! In specific genre subsets (classical, folk, jazz, blues, rock, reggae)! Chord pattern graphs visualised with open source graphviz 52 Barthet, M. et al. Big Chord Data Extraction and Mining. In: CIM 2014

53 Audio-based Automatic Chord Recognition Chordino Vamp plugin [Mauch and Dixon, ISMIR 2010]. Uses chromagram obtained with a non-negative least squares (NNLS) procedure for approximate note transcription. Accuracy of 80% when assessed with MIREX 2009 s dataset (Popular songs from the Beatles and Queen). Excerpt from Buddy Guy s Mary Had A Little Lamb 53

54 Chord Sequence Patterns! Segmentation of audio recordings into chord sequences! Representation of no detection (N)! Most frequency of (non-contiguous) chord sequences are patterns 54

55 Chord Pattern Length Bell-shaped curve for all genres Jazz and Classical have more patterns: greater harmonic diversity? Folk has more long patterns? 55

56 Frequent Chord Patterns: Classical vs. Blues Classical Blues Classical patterns more distributed Blues connects dom7 with dom7, classical doesn t (dom7 is typically resolved to major or minor tonic) 56

57 Visualisation of Chord Sequence Patterns! Most prominent chord sequences! Compare two collections or visualisations 57 Kachkaev, A. et al. Visualising Chord Progressions in Music Collections. In: CIM 2014

58 Circular Grid (circle of fifths, straight and twisted) 58

59 Parallel coordinates (chord types, circle of fifths) 59

60 Transition matrix (chord types, roots) 60

61 Tonnetz 61

62 Folk vs. Jazz 62

63 Demo 63

64 Practical exercises! Open (similarity) or (tempo curve)! Follow the worksheet or explore freely! Pitch (class) distribution! Similartiy! Tuning! Tempo curves! Take notes of results, ideas and hypotheses! You can explore the chord sequences here: 64

65 Practical exercises! Tasks 1! Schoenberg has relatively flat pitch profile 65

66 Practical exercises! Tasks 2! Mozart sonatas seem more homogeneous 66

67 Practical exercises! Tasks 3! Tuning was stable between these periods 67

68 Practical exercises! Tasks 4! Tempo smoothes out, some final ritardando 68

69 Where to go from here! Found something interesting?! Results need careful interpretation wrt! Noisy data! Data collection! Significance! Cross-validation! Meaning! Can be challenging, needs expertise in music and data 69

70 Where to go from here (ctd.)! Follow up ideas and hypothesis by close examination of the data! Extract data with SPARQL interface (more on Semantic Web and SPARQL tomorrow)! Use programming languages and statistical tools, e.g. Python, Pandas, SciPy, R, Matlab, SPSS, 70

71 The end! Open discussion! Thank you 71