Due Wednesday, December 2. You can submit your answers to the analytical questions via either

Transcription

1 ELEN 4810 Homework 6 Due Wednesday, December 2. You can submit your answers to the analytical questions via either - Hardcopy submission at the beginning of class on Wednesday, December 2, or - Electronic submission (in pdf form) on Courseworks. Please name it in the format of yl3027 hw6 writeup.pdf. Thanks. Analytical Questions Please complete problems 5.22, 5.34, 5.36 in Oppenheim and Schafer (3rd Edition). Justify your answers! Computational Questions For this homework we will focus on the properties and potential applications of the short-time Fourier transform (STFT). This is a useful tool for analyzing signals with spectral characteristics that change over time, and is particularly important for applications involving time series signals such as music or video. We expect that the computational section will not have a heavy workload, and the emphasis is mostly on developing intuition and familiarization with employing the STFT. Therefore, although we do not require students to submit their observations, it will be particularly beneficial for students to go through them on this assignment. 0.1 Uncertainty Principle Introduction. Here we will visualize the STFT of a signal via its spectrogram. The STFT can be expressed via the following equation, s k,m STFT (x) [k, m] DFT N (D m N0 x w) [k], (1) where w is a window with supp (w) = N. The window is used to obtain a small chunk of the signal x near the sample m N 0, and the DFT analyzes the frequency content of the signal at that chunk. One important insight from Fourier analysis regarding the STFT is the uncertainty principle, which suggests that the spectral resolution of the STFT trades off with its temporal resolution. Enlarging the size of the window provides better accuracy w.r.t. frequency information, but precision

2 w.r.t. time of occurrence is lost. On the other hand, shrinking the size of the window gives you better temporal precision, but resolution of the frequency content is lost. A common technique is to apply an overlapping window by choosing N 0 < N. This allows you to get frequency information more smoothly w.r.t. time when using a large window. Of course there is no free lunch, as computational costs increase proportionally with overlap. Preparation. In a script file specexercise.m, sample over the following signals over t [0, 2] seconds at a sampling frequency of f s = 1kHz { cos (2π 250 t), t 1 x 1 (t) = sin (2π 250 t), t > 1, x 2 (t) = chirp (t, 0, 1, 100, Quadratic ), to produce the arrays x1 and x2. The first signal x 1 is a pure tone of 250 Hz with a discontinuity at t = 1s, and x 2 is a quadratic chirp - a signal whose frequency increases quadratically over time. As preparation, please briefly go through the essential usage of the chirp() in the MATLAB documentation, and play back the tones using sound(). We will use the spectrogram() function on x1 and x2 to explore the uncertainty principle w.r.t. the STFT. Tasks [3 pts]. Complete the script specexercise.m to produce a separate figure for x1 and x2, with 2 2 subplots. Each subplot should contain a spectrogram created by spectrogram(x,window,noverlap,nfft,fs) (please look up its usage!) using FFT size 512, plus the following parameters: 1. Window size: 128 points. Overlap: 0 points. 2. Window size: 128 points. Overlap: 120 points. 3. Window size: 512 points. Overlap: 0 points. 4. Window size: 512 points. Overlap: 500 points. For each subplot, append an appropriate title, e.g. Quadratic chirp pt window - No overlap. Observations. What happens to the frequency localization of the STFT on x1, outside of the discontinuity, as the window size is increased? Now look at the discontinuity, what happens to the time localization as the window size is increased? What about the effect of chaging the window size on x2? How does overlapping change the STFTs with the different window sizes for x1 and x2?

3 0.2 Visualization and Onset Detection Introduction We will now use the STFT get a feel of how frequency characteristics can vary over time as a result of tonal, percussive and instrumental components in a piece of music. The file excerpt.wav contains a 16s segment of the song Cups by Anna Kendrick; we will produce a visualization of the frequencies components of the this segment over time. Next we will attempt a simple technique for extracting percussive onsets from the segment, which you can listen to in isolation from cupsonly.wav. Tasks. This problem is split into four small subparts. Each of these parts - except Part C - should take a very short amount of time to complete. For Part C, you will need a simple video editor for attaching audio files to video clips. The video editor should also allow you to change quality settings to reduce the output video size for submission. Any number of such video editors are available online for free. For Windows, Windows Movie Maker will do nicely Part A - Producing the STFT At the beginning of the script mkvid.m, we apply audioread() on excerpt.wav to read the stereo signal x and sampling frequency fs. The totaltime of the segment is also calculated. Apply spectrogram() to each channel of x to obtain s1 and s2. For submission, please use the following settings: Window size: 1024 pts. Overlap 512 pts. FFT size Alternatively, if you are seriously constrained on computing power, you may use Window size: 2048 pts. Overlap 1024 pts. FFT size 1024, However we heavily suggest using the first setting to get enough resolution from your STFT. Furthermore, you should retain the frequencies and times as fspec and tspec from one of your spectrogram() calls. The frequencies and times are of course the same across both Part B - Producing the visualization plots [2 pt] Now we will write a small function visualize() to modify the output from spectrogram() so that it is more interpretable to the human eye. The function is defined as follows [ A ] = visualize( s, alpha, fsmooth, tsmooth ), where s is some output from spectrogram(), notice that it has size numel(fspec) numel(tspec), and is complex. The function should produce a real, nonnegative array A of the same size by doing the following. 1. Retaining only the magnitudes and taking the log: A (1) k,n = log 10 (1 + alpha s k,m ), where s k,m is the STFT produced via Eqn. (1). 2. Next, if fsmooth is nonempty - check using isempty():

4 smooth A (1) w.r.t. frequency by applying the function smooth() to each column of A (1) k, k [fspec], with the parameter fsmooth. Lets call this A (2). smooth() averages out the signal more agressively as fsmooth increases. if fsmooth is empty then make no change to A (1) 3. Finally, as above, get A by smoothing A (2) w.r.t. time, with the parameter tsmooth if it is nonempty. For interpretation, it is usually not necessary to show too much data or be too precise, but it is important to retain smoothness. Considering the simplicity of the smooth() function, this becomes particularly useful if we are constrained in computing power or memory Part C - Compiling the visualization [2 pts] Although it would be amazing to implement a system that visualizes audio tracks in real-time, this requires dealing with significantly more lower-level details that are beyond the scope of this course. For now we will simply take A and compile a video clip, then append excerpt.wav to the clip. To do this, we will construct a series of video frames: 1. In mkvid.m, under Part B, call visualize() on s1 and s2, we will refer to the outputs as A1 and A2. For submission, set fsmooth = [] and tsmooth = [] empty or, if using the alternative spectrogram settings, fsmooth = 5, tsmooth = We will take frames across t [0, totaltime] at the framerate fps. By default fps is set to 60, but again if necessary you may lower this down to To capture the frames, produce a 2 1 subplot figure over each time index : (a) First find the time index idx of the spectrogram from tspec that is closest to t(i), i.e. idx = arg min j tspec(j) t(i) (2) (b) For the first subplot, plot the frequency-slice of A1 at time index idx. Set the limits of the frequency and magnitude axes by typing xlim([0 fspec(end)]); ylim([0 max([a1(:);a2(:)])*1.1]); and append an appropriate title, e.g. Left Channel. (c) For the second subplot, do the same thing for A2. Please use a different color for the plot to distinguish from the previous one. 4. The final part of the loop consists of forcing MATLAB to update the plots as the loop runs, and capturing the frame from each loop. This part is done for you. Now run mkvid.m to collect the frames. You may want to enlarge or maximize your figure before capturing the frames to get as much detail for playback as possible. The final section of mkvid.m compiles the frames into a video output nosound.mp4.

5 Unfortunately MATLAB does not have the functionality to append audio, so please use a video editor to do this. For submission, save the final video as output.mp4, and use the settings to ensure the final file is below 10MB! Once you have done this play the video to see the visualization. If using Windows Movie Maker, just use Add videos and photos and Add music to attach music to the video clip. Then progressive lower the settings under Save as until the output size goes below 10MB. Observations. Feel free to try this on a number of different spectrogram() and smoothing settings. If you have enough computational resources, you should definitely use the recommended spectrogram(), fsmooth, tsmooth and fps settings, which will give you very good spectral and temporal resolution Part D - Onset detection [3 pts] Being able to extract occurence times and pitches of percussive events can tell us a lot about a piece of music. The task of extracting such information is called onset detection. For this problem, the song in our excerpt is given its namesake from the rhythm generated by playing the cup game [1]. We would like to extract this rhythm from the music, which you can listen to in isolation from cupsonly.wav. The idea behind STFT approaches towards onset detection is that the occurence of certain events - such as when the singer draws out a note - is accompanied by a significant increase in the magnitude of the corresponding frequencies. You may be able to observe this from the visualization, especially if you use our recommended parameters Figure 1: Signal from left channel of cupsonly.wav plotted against time (s). Here the rhythm happens in isolation, so the onset times are simply the occurence times for each of these events plotted. The challenge is to try and recover this from excerpt.wav, when the rhythm is mixed with other sources.

6 Writing the detection function. The Hainsworth onset detection function DF (m) applies this principle on the STFT coefficients {s k,m } by doing the following: ( ) sk,m d m (k) = log 2, (3) s k,m 1 DF (m) = k K max (d m (k), 0), (4) where K is a chosen subset of frequencies. Thus, we are summing the positive log differences over a subset of frequencies to produce a detection function DF (m) over the time indices m {1,..., numel(tspec)}. The goal is for large peaks to occur in DF (m) when m indexes the onset of a percussive onset. Complete the function hainsworth() defined as follows [ DF ] = hainsworth( s, fspec, sbints ), where s and fspec are some collection of STFT coefficients and frequencies from spectrogram(), respectively. The 2-column array sbints is a collection of frequency pairs {α i, β i } i representing intervals (α i, β i ), where α i < β i are given in Hz. To produce the length numel(tspec) 1 column array DF, 1. Get K = i {k αi,..., k βk } by concatentate the index ranges. Find the corresponding index k f for a frequency f (in Hz) by searching for the index k f = arg min j fspec(j) f. (5) 2. Apply equations (3) and (4). The fuction diff() may come in handy. Testing the results. Now return to mkvid.m, and apply hainsworth() to s1 to get the detection function DF. If you have used the recommended parameters for the visualization and played it with the music, you may notice that the tonal component of the segment concentrates in the spectrum around the frequency ranges [0, 5kHz] [9kHz, 14kHz]. Our interest is in detecting percussive onsets, so provide the argument to sbints to take frequencies from the range [5kHz, 9kHz] [14kHz, fspec(end)hz], i.e. outside of the range of tonal frequencies. In Section C in the code, replace the 2 1 subplot figure with a 3 1 subplot figure. The top two subplots stay the same as Part C. For the final subplot, do the following: 1. Plot DF over tspec(2:end), 2. At tspec(idx) - the time referenced by the specific frame, stem() a red line of height twice the maximum value of DF. Remember to use hold on! 3. Fix the axis limits xlim([0 tspec(end)]); ylim([0 max(df)]*1.1); 4. Append a suitable title, e.g. Hainsworth DF.

7 Once compiled, this produces a red line that scans over the detection function as the music plays so you can listen to the results. Compile the frames again and append the track as in part C. If you have the capability to do so, slow down the playback of the output to better listen to the details. The VLC media player has this functionality (under Playback) and is freely available for all major operating systems. Observations. Is the rhythm in the segment captured by the peaks in DF? How well does the function work? How often and in which particular instances does it fail? Do you know, or can you think of any ways to do better onset detection given such a complex music sequence? If you have any ideas, please do let me (Yenson) know! Those who are interested may refer to this tutorial about onset detection methods and Hainsworth s detection function from [2]. Video Requirements. Produce a video of the 3 1 subplot figure, along with audio, as output.mp4 in MPEG-4 format - please use one of settings listed. Remember to lower the quality settings on your video editor when saving output.mp4 so that it is less than 10MB - marks will be docked if your video is too large! 1 Submission Submission Instructions. Please upload and submit the completed functions and scripts specexercise.m, visualize.m, mkvid.m and hainsworth.m. In addition submit the video output.mp4 from Problem 2 (remember to follow video requirements!). Please submit directly (no.zip file) to Courseworks before the beginning of class on Wednesday, Dec. 2nd. No need to submit observations. References [1] Wikipedia. Cup game wikipedia, the free encyclopedia, [Online; accessed 18-November- 2015]. [2] Nick Collins. A comparison of sound onset detection algorithms with emphasis on psychoacoustically motivated detection functions. In Audio Engineering Society Convention 118. Audio Engineering Society, 2005.