Music instrument categorization using multilayer perceptron network Ivana Andjelkovic PHY 171, Winter 2011

Transcription

1 Music instrument categorization using multilayer perceptron network Ivana Andjelkovic PHY 171, Winter 2011 Abstract Audio content description is one of the key components to multimedia search, classification and source identification. This project examines precision of audio content description based on spectral information only on one hand, and spectral and temporal on the other. Multilayer neural network with varied parameters is used to classify musical instruments based on obtained descriptors. 1. Introduction 1.1. Motivation Research of music instrument recognition methods has received much attention and its numerous applications range from automatic music cataloguing, transcription, audio indexing to identification of performers based on their instrument playing style. The goal of this project is to gain practical knowledge of a particular classification method, namely Multilayer Perceptron and apply it towards classifying musical instruments based on two different sound analysis approaches. Music instrument recognition most often consists of two phases: (1) Audio content analysis musical features can be temporal, spectral and perceptual, to name a few, and some of the commonly used computational feature extraction methods include Mel-frequency cepstral coefficients (MFCC), Short Time Fourier Transform (STFT) and Wavelet transforms. (2) Classification various learning methods that analyze data and recognize patterns, such as neural networks training and support vector machines algorithms, 1

2 have been used independently with different success rates. The combination of one or more different methods may be applied towards solving a classification problem, which can yield better results [1]. This project utilizes FFT and Wavelet transform techniques to analyze audio content, and multilayer neural network to classify it Sound analysis It has been shown by Brown et al. [5], Dubnov et al. [6], and others that spectral information is sufficient to distinguish between instruments from different families. Most commonly used algorithm to obtain the spectrum of audio signal is Fast Fourier Transform (FFT). However, sounds that musical instruments produce vary in how their features, such as loudness and spectral content change over time. For instance, "attack" and "decay" (Figure 1) of a sound have a great effect on the instrument's sonic character (Figure 2). Figure 1: Sound envelope; Attack time is the time taken for initial run-up of level from nil to peak, beginning when the key is first pressed. Decay time is the time taken for the subsequent run down from the attack level to the designated sustain level. Sustain level is the level during the main sequence of the sound's duration, until the key is released. Release time is the time taken for the level to decay from the sustain level to zero after the key is released. 2

3 1500 Strings x Piano 1 x x 10 4 x 10 4 Figure 2: Spectrum obtained using FFT and plotted in Matlab (left), and sound wave (right) of two second isolated note played on orchestral strings (top) and piano (bottom). Notice the difference between almost instantaneous attack of piano sound and gradual attack for strings. There exist numerous techniques for observing how spectrum of a signal changes over time. Perhaps the best known among them is Short-time Fourier Transform (STFT). The data to be transform is broken up into windows of desired size, and FFT is applied to each one of them. The drawback of STFT is that it has fixed resolution all windows are the same size. The width of the windowing function relates to how the signal is represented. It determines whether there is good frequency resolution (frequency components close together can be separated) or good time resolution (the time at which frequencies change). A wide window gives better frequency resolution but poor time resolution. A narrower window gives good time resolution but poor frequency resolution. The Wavelet Transform overcomes the fixed size windowing problem and gives good time resolution for high-frequency events, and good frequency resolution for lowfrequency events. Thus it yields better results in examining most real signals. 3

4 2. Experiment 2.1. Problem description The experiment examines the precision of classifying music instruments from different families and within a same family based on spectral information only and based on spectral and temporal information. (Figure 3) Figure 3 As stated in the introduction, numerous studies showed that spectral information can be sufficient to categorize instruments into different families. In particular, study by Hassan [4] confirms this finding by examining isolated notes played on different musical instruments. This study served as a guide for designing the experiment. Several studies addressed usage of wavelet transform and neural networks to classify musical instruments [1],[2],[3]. For instance, it has been shown that wavelet coefficients used to train a neural network yield 78% success rate in identifying unknown complex classical movements [2] and can be successfully used to identify dominant instrument in a short clip of music [3]. Based on the successful classification results of complex audio data using wavelet transform and neural networks, I hypothesized the same methods would work well in classification of isolated notes played on instruments in the same family Training data processing Due to time constrains, the data was obtained by recording two second isolated notes, played on software instruments in Mac OS application Garage Band. Although the sounds are realistic, they are synthesized rather than obtained by recording real instruments. In addition, the sustain and release rates of different instruments may not be proportional across different instruments. Four instruments belonging to different families are flute, piano, guitar and orchestral strings. Three instruments belonging to wind family are flute, saxophone and trumpet. Selection of the instruments was solely based on their availability. Four octaves, totaling 48 notes were obtained for each instrument, at Hz sampling rate. 4

5 Results of both FFT and Wavelet analysis were relatively large and too cumbersome for training a neural network. Therefore, an important step is to reduce the size of data while maintaining the unique representation of each sample. In the case of FFT analysis, 1024-point FFT was applied to each note. Next, second half of result was discarded because of the symmetry. The result was a vector of 512 elements for each note. In the case of Wavelet transform, a technique Wavelet Rank Dispersion Vectors (WRDV) was used to reduce the data size. WRDV is defined as a histogram of the rank orders obtained by the wavelet coefficients of a given wavelet scale among all the coefficients. First, continuous wavelet transform using Meyer wavelets was applied to the original data. Second, a large number of resulting coefficients were represented as rank vectors containing 530 elements Classification method Neural Networks Matlab toolbox was used to train the multilayer network using various parameters and easily plot results of experiment. Among several available methods and learning algorithms, the one used to address classification problem was Neural network pattern recognition tool. In the case of classification of instruments from different families, there were either 530 (WRDV) or 512 (FFT) input neurons, and 4 output neurons representing 4 instrument. There were 3 instruments belonging to the same family, and so the number of output neurons in such case was 3. Number of neurons in the hidden layer was varied throughout the experiment. Following are the default settings for some of the training functions that were not changed during the experiment: (1) Transfer function: tan-sigmoid for hidden layers, and linear for output layer. (2) Backpropagation network training function: Scaled conjugate gradient backpropagation (3) Backpropagation weight/bias learning function: Gradient descent with momentum weight and bias learning function The pattern recognition tool by default randomly divides the input data set such that 70% of data is used for training, 15% for validation and remaining 15% of data for testing. 5

6 2.4 Results Different families of instruments The analysis of results was not conducted in great detail. However, it was sufficient to make general conclusion about efficiency of different approaches. As expected, classification of instruments from different families based on spectral information only was successful, but inferior to classification based on wavelet coefficients. Figure 4 shows Receiver Operating Characteristic 1 plot, and confusion matrices 2 for three different training sessions based on FFT data. The number of hidden neurons is 40, which yielded the best classification results. The average Mean Square Error for all three sessions was Further analysis may answer the question which instrument was least or most often misclassified. Figure 4: ROC plots and Confusion matrices for three different sessions on training with FFT data. Number of hidden neurons is True positive rate vs. false positive rate for a binary classifier system 2 Each column of the matrix represents the instances in a predicted class, while each row represents the instances in an actual class 6

7 Figure 5 shows the ROC and confusion matrix of the most successful training session using WRDV, where hidden layer contains 30 neurons. Average Mean Squared Error or two different sessions was Figure 5: The most successful session based on WRDV data Wind family of instruments Distinguishing among instruments within the same family was not satisfactory based on FFT data but, as hypothesized, it was successful based on WRDV data. Figure 6 shows ROC plots and Confusion matrices for three different training sessions based on FFT data only, where number of neurons in the hidden layer was 30. Average MSE for all sessions was Neither increasing the number of hidden neurons, nor changing the division of data set such that 80% is used for training, significantly improved the results. 7

8 Figure 6: Results of three training sessions based on FFT data Figure 7 shows results of the most successful training session based on WRDV data, where 50 hidden neurons were employed. Mean Squared Error was Although the results of other training sessions are not available at the moment, they were overall satisfactory. Figure 7: Results of the most successful training session based on WRDV, utilizing 50 hidden neurons 8

9 3. Conclusion Results of the experiment show that Wavelet Rank Dispersion Vector measure used to train multilayer neural network can be successfully employed to classify solo instruments. The experiment was designed and conducted within limited amount of time, and it could be greatly improved for further studies. Nevertheless, learning experience was invaluable and obtained results align with the findings in the field. First, the loudness and release time of sound envelopes should be normalized across samples for all instruments. Second, different degrees of similarity should be noted between instruments within the same family. This experiment examined three instruments within wind family, but the classification method can be, for instance, tested for woodwind or brass instruments. Third, the choice of different backpropagation algorithm may influence the success rate of classification. Finally, method can be compared against others in classification of more complex musical excerpts. References: 9

10 [1] Identifying the classical music composition of an unknown performance with wavelet dispersion vector and neural nets, Stephan Rein; Martin Reisslein [2] Audio content description with wavelets and neural nets, Stephan Rein, Martin Reisslein, and Thomas Sikora [3] Musical Instrument Identification Using Wavelets and Neural Networks, Jeffrey Livingston, Nathan Shepard [4] Instruments recognition using neural networks and spectral information, Ezzaidi Hassan [5] Feature dependence in the automatic identification of musical woodwind instruments, Brown J. C., Houix O. and McAdams S., [6] Polyspectra as measures of sound texture and timbre, Dubnov S., Tishby N. and Cohen D 10