MASSEY UNIVERSITY SCHOOL OF ENGINEERING AND ADVANCED TECHNOLOGY. Engineering Project. Submitted as part requirement for B.Eng. (Hons).

Transcription

1 MASSEY UNIVERSITY SCHOOL OF ENGINEERING AND ADVANCED TECHNOLOGY Engineering Project. Submitted as part requirement for B.Eng. (Hons). Investigation into Audio and Surround Techniques via a Short Film William Borman ( ) SUPERVISOR Wyatt Page

2 2 Summary This project involved the creation of audio for a short animated film of approximately nine minutes in duration entitled The Paintbrush, and encompassed all music and sound effects for the film. This included composition of an original soundtrack with subsequent recording, mixing and mastering into surround sound. Sound effects were recorded in four-channel surround sound, and all audio was mixed and mastered into surround (5.1) and stereo (2.0) tracks. The project explored the possible improvement of realism, immersion and quality of surround sound in film and music applications. Through recording in four channels to match the surround speaker output, greater sense of 'being-there' was achieved for the audience, as well as more sense of space and clarity in audio by spreading sound effects and music more evenly across all speaker channels. The possible use of convolution, a mathematical process useful for adding room sound to audio, was explored by writing a convolution application in MATLAB and recording impulse responses. Careful use of this successfully added the desired ambience to audio in the film. Misconceptions and lack of standardisation in surround sound were explored in detail, and recommendations for future use have been made. Student Anna Tisch created the film's animated video content. Some delays in video content creation caused delays in subsequent audio creation for those sections, but overall the collaboration was largely successful.

3 3 Table of Contents 1 Introduction The Process Conception and planning Storyboarding stage Music composition...6 Details...6 Notes on composition Animatic stage Music recording...7 Details...7 Recording...8 Quadraphonic instruments...9 Sampled strings and Bidule...9 Other instruments Rough animation and onward sound effects...11 Details...11 Recording Time management Surround Sound Background Introduction Recording and mixing methods mix...16 Music mixing - what people want...18 Stereo mix Standards and implementations...19 Standards...19 Studio reality...21 Cinema reality...21 Consumer reality...23 Bass Management Results of The Paintbrush in both mixing styles Issues with the Massey University Wellington surround room...26 Main Speakers...26

4 4 Secondary Speakers...27 Acoustics Convolution What and why...29 An explanation of convolution...29 Process...30 Equipment Convy a convolution processor in MATLAB...32 Remaining issues with Convy Use in The Paintbrush...34 Locations recorded...34 Use of convolution Potential in film Discussion Conclusions and Recommendations References Appendices Appendix A: Included resources Appendix B: High resolution images...43 Screenshots...43 Waveforms...43 Software...45 Photos...50 Old Piano Recording...50 Other Instrument Recording...52 Sheet Music Appendix C: Convolution Appendix D: Timelines Appendix E: A few notable sounds Glossary...62

5 5 1 Introduction Multichannel audio for film has developed from simple mono beginnings into a multitude of different formats for cinema and home theatre speaker systems. Recording techniques and audio mixing methods have not altogether followed these developments on the playback side of the process. Today, there is a lack of standardisation between surround formats, and decoders for any format do not necessary conform to specifications (Blackburn,2000). Recommended speaker arrangements and mixing styles are supplied by format creators such as Dolby (2005a,2005b), but are often not followed well by audio engineers. Recording remains overwhelmingly in mono or stereo, despite the need for conversion into surround sound. However, technologies such as simple, portable multichannel recorders are now readily available, and digital multichannel formats are higher quality than previous generations. Aside from substantial sound quality loss in some older systems, older formats such as Dolby Pro Logic, which used matrix encoding, also lost data on the panning of sounds into different speaker channels (Elen, 2001). Today, the possibility of creating better surround mixes is far more feasible than it has been in the past. Due to a previous background in audio work, but a lack of film audio experience and a need for film content to make up a work portfolio, it was beneficial to myself to attempt this project in Through the creation of the film The Paintbrush, Anna Tisch and I have created an original film while also investigating possible improvements to the surround sound format and applying multimedia engineering principles. Tisch originally expressed interest in creating an animated film in late 2008, for which I suggested I could create the audio and soundtrack. Much of this project follows experience gained through the Bachelor of Engineering degree at Massey University majoring in Multimedia Systems Engineering. This includes programming and use of MATLAB software, use of Digidesign hardware and software for audio work, mathematics in the form of convolution, and use of Apple Final Cut Pro for combining and editing Tisch's film sequences. The final result serves in its own right as an original film, but also as a practical example for anyone interested in greater realism and better utilisation of surround sound. Section 2 details the process of the project itself, while section 3 covers surround sound in general and its use in The Paintbrush. Section 4 describes convolution, its use in the film, and possible use in regards to film in general. Discussion, conclusions and recommendations are provided in sections 5 and 6 based on the previously discussed material. Appendix A details the contents of the two DVDs intended to be included with this report. Please see the glossary (section 9) for definitions of potentially unfamiliar terms.

6 6 2 The Process 2.1 Conception and planning Plot and style for a film were discussed between Tisch and myself, with the story and concept of The Paintbrush decided over the summer of Once university recommenced in late February 2009, basic research began into surround sound and audio in film. Tisch researched the animation aspects.1 Since the aim of creating audio for the film was improving the current standard of surround sound recording and mixing in film, the first step was naturally to investigate the status quo. The current state of surround sound in film is detailed in section Storyboarding stage Music composition Details Tisch created storyboards (see Figure 2.1) of the basic film structure which were complete by mid-march. At that point the plot and style of the film was developed enough that music ideas and composition could begin. Figure 2.1 Example of a page of the storyboard for the film 1 Appendix D, figure D1.0 shows a complete timeline of the project, while figure D1.1 shows an ideal timeline as projected at the time of the first interim report (April 2009).

7 7 The Paintbrush has a fairly fluid story with similar themes throughout there are generally no greatly tense or emotional moments which need a big change in style of music. The film also switches between two main (and only) characters throughout. For this reason it seemed a good idea to give each character a distinct aural theme which would help tie their respective sections together, but also keep things similar enough that the music would merge smoothly between scenes. Notes on composition The music has two main themes, one for the girl character and one for the boy. These tend to play during their respective scenes. See the Sheet Music section of Appendix B for images of the music and chord structure. The short section with lyrics at the end creates a hopefully memorable change in style. The chord structure of the end section also plays during the earlier outdoor section, albeit in a much different style. The girl's theme is in B minor, and D major (the relative major) for more light-hearted sections. It features a slightly unusual chord structure and builds from tentative, childlike beginnings to lush instrumentation mirroring her imagination in the fantastical sequences. The boy's theme is in E major and features more sparse instrumentation and a prevalence of major 7th chords. The slightly unusual sound of the melody comes from the use of a minor 5th chord. The slow build up from around the end of the cupboard scene was designed to accommodate any scenes that need to be cut due to time constraints. Without going into any great detail, the same two chord sequence of two bars (approximately 17 seconds) repeats several times. Cutting other sections of music may be hard to reconcile back together, but here a simple cut and move of this section of music is almost enough. In the final movie, one of these 17 second sections has been removed. 2.3 Animatic stage Music recording Details After storyboards were complete, Tisch began work on a simple animatic of the film (see figure 2.2) a full realisation of the film in very simple animation.

8 8 Figure 2.2 Example of a frame from the animatic The main issue with the audio creation process was that in traditional film, most or all of the video content will be complete before the audio work begins. With The Paintbrush, we couldn't afford to use this method as both projects needed to span the whole year. I could afford a slight delay until I began, but available time was too short to wait very long so the collaboration work had to be strongly interleaved. The animatic was intended (and this largely eventuated as planned) to mirror the final film very closely in terms of scenes and their timing, if nothing else. This would mean that I could compose the music based on the animatic, though sound effects would have to wait until some degree of final animation was complete. Recording The animatic was completed in mid-may, at which point the music recording and mixing process began in earnest.2 The only pieces recorded before this time were the sections with the old piano, which were recorded using an old, out-of-tune (but not terribly so) over damper piano from the 1890s, which required travelling to Masterton. For the animated part of this section, where the girl is shown playing the old piano at the start of the film (before it turns into the imagined grand piano), it was deemed to be easier to match animation to sound than sound to animation. Therefore, this section could be recorded before the animatic was complete. Some cutting and editing was needed to fit the audio smoothly into the final film; the transition from real to imagined piano can still be a little jarring. I videotaped the performance to aid in animation (see figure 2.3). Figure 2.3 Real piano footage versus animated piano. Although complete realism was the intention for the sound effects, the music was to first and foremost sound good, whilst also having elements of realism. All tracks were first recorded in stereo at my home studio3 using a Digidesign MBox 2 with Pro Tools 7.4 software. Once a basic mix of the tracks was complete, any tracks using samples or effects which Massey University did not have were bounced down4 to simple 2 See the Appendix B Photos section for images of recording methods described herein. 3 See 4 This term is one of several explained in the glossary.

9 9 audio tracks. At Massey, the surround room with Digidesign 003 unit and Pro Tools 8.0 software was used to convert tracks to 5.1 and complete the mix. After that, the music was tested on different systems and tweaked for some time before reaching the final version. Quadraphonic instruments The acoustic guitar5 and a section of violin at the end of the film are recorded using a quadraphonic (four-channel) arrangement of four Røde NT5 small-diaphragm condenser microphones6, as shown in figure 2.4. The rear microphones are spaced further apart than those at the front to approximate a surround sound set up. The old piano was recorded in a simpler 3-channel arrangement, since the second pair of NT5s was not yet available. It uses an AKG C4000 large-diaphragm condenser in the rear to pick up room sound. Figure 2.4 A typical 4-channel recording system used, with identical Røde NT5 microphones. Angles are the same as the Zoom H2: Massey's matched pair (with Massey stickers) provides a 90 degree angle at the front, while my own matched pair provides 120 degrees at the rear. This particular set up was for recording violin. The main piano requires some careful microphone placement to sound its best. The standard set up as in Figure 3.4 also introduces too much room noise into the recording. A drier7 sound is easier to mix and will stand out more without having to boost the volume so much. Reverb can still be added later, including convolution (see section 4) for improved realism. The final recording has the two rear microphones in a figure 2.4 type configuration at a little below head height, and front microphones low down on the soundboard, with the front of the piano open. This gave a good, dry sound for the front speakers while including room sound at the rear. 5 This is the only instrument not played by myself. I was not happy with the original acoustic guitar part, and the short acoustic guitar sections are now performed by Robyn Murray. 6 Two matched pairs, where each one of a pair should be as identical to the other as possible. 7 Less echo.

10 10 Sampled strings and Bidule The violin and viola are samples8 from the Vienna Symphonic Library, with many different sounds and variations in musical expression triggered by MIDI sequences. The most powerful control over the Vienna samples comes from using the packaged Vienna Ensemble software, but this is not yet supported by Pro Tools in Windows. Using this software within Pro Tools was highly desirable for creation of high-quality string instrumentation in the soundtrack, so possible options were explored. To interface Vienna Ensemble with Pro Tools, software was needed which could run VST plug-ins (of which Vienna Ensemble is one) while taking MIDI data directly from Pro Tools in real time. Furthermore, the sound of the strings then needed to be fed back into a Pro Tools track. Any other option would not satisfy the control required to create complex, realistic string parts in the film score. The solution turned out to be Bidule, an extremely powerful modular, real-time application made by Canadian company Plogue. Bidule is not widely known, but the purchase was made after hearing rave reviews. The software may be run as a plug-in in Pro Tools, or on its own, and allows low-level control of signal routing between applications, addition of effects, and running of other plug-ins and software. One has essentially very low-level control of signal paths while the software remains fairly simple and intuitive due to the use of a modular synthesizer-type paradigm. Figure 2.5 shows a typical example of a Bidule routing set up. Figure 2.5 Bidule with several modules connected. Bidule allowed multiple instances of Vienna Ensemble, each carrying a different instrument, to interface with MIDI tracks in Pro Tools in real-time and send audio back to be recorded. Mixed opinion has been given on whether the final string tracks sound convincingly realistic. 8 Pre-recorded sounds.

11 11 Certainly, my own violin playing (as heard behind the short song at the end of the film) would not have been up to standard for the main score, so a compromise had to be reached. Other instruments Instruments not listed so far are mostly played through direct inputs, such as the bass guitar, and hence comprise only a mono track which is panned as required in the surround mix. The electric guitar is also a direct input, running through a software-based guitar amplifier emulator. 2.4 Rough animation and onward sound effects Details As Tisch completed the 3D backdrops and their camera movements, she began work on the rough animation that would then be modified and coloured to become part of the final film. The rough animation was enough to begin recording sound effects, beginning around mid August. Almost all sound effects are recorded using Massey University's Zoom H2 recorder, a small portable 4-channel recorder with built-in microphones that pick up sound with a high level of accuracy and store it in a high quality uncompressed format. Since even higher sound accuracy and quality was desired for the musical score, the NT5 array was used instead, but the Zoom H2 provides near the same quality without portability issues. A few sound effects that were outside my means to reproduce were created electronically or retrieved from other sources, though these have been kept to a minimum. Recording The idea was to record everything in 4-channel sound, to be placed to the L (Left), R (Right), Ls (Left surround), and Rs (Right surround) speakers in a typical 5.1 sound system, with recordings of sound placed accurately in 3D space. The benefit of using this system is that it all but eliminates the need for panning and placement of sounds in the mix. The negative being that every sound effect has to be recorded individually to fit the scene it is being used in. For instance, if a character walks toward the screen, footsteps must be recorded walking toward the screen. The desired benefit being that this will create a convincing illusion that what is on screen is really happening in 3D space. A standard mono recording of footsteps can be used over and over in different applications, but the sound may not be particularly convincing. Effects and panning need to be added to make it sound as if a character is walking from one place to another. 2.5 Time management As illustrated by the differing timelines in Appendix D, the project fell further behind schedule

12 12 as the year progressed. In a standard film project, the audio work (especially the soundtrack) can be carried out after the film is largely complete. In our case, both projects needed as much of the year available as possible to have a chance at completion, necessitating the system used where the animatic followed the timing of the eventual film. Tisch still had to complete key sections of the film before sound could be created. The late start on the rough animation meant a late start on sound effects in particular, and occasionally changes had to be made which could have been done correctly from the beginning if audio work could have been started after more of the film was complete. Rendering of the 3D scenes took longer than expected, which caused the greatest lapse in schedule. By the final presentations, the music was complete, and sound effects were complete for sections which Tisch had also completed, except for a couple of scenes added at the last minute. However, much of the film was still to be completed, and hence much of the audio as well. Audio work seemed to overall take less time than the equivalent film work, and scenes which needed few sound effects could be completed easily enough within a day to a decent standard. This meant audio work could usually 'catch up' to film work fairly quickly if necessary. I handled the combining and timing of scenes in Final Cut Pro, and the majority of the poster design, to help even out the workload. Appendix B, Figure B2.6 shows the process of combining clips in Final Cut as more scenes were completed or modified.

13 13 3 Surround Sound 3.1 Background In 1940, Disney's Fantasia was the first film to use any form of surround sound, using three discrete sound channels. The growth in the popularity of television spurred the first commercially successful multichannel sound formats in the 1950s, accompanying a move to wide-screen format, in an attempt to compete (Dolby, 1999 & Holman, 2000). The development of surround sound since the first commercial implementations has been fairly eclectic. The formats that have been pushed and advertised the most by their creators have tended to succeed witness the prevalence of Dolby formats in both home and cinema audio even if these are not necessarily the 'best.9' Early cinema surround formats were all at least 4-channel with everything recorded onto magnetic tape. A good basic system, but without a high degree of sound quality. Home stereo audio premiered in 1958, but was reduced to two channels because of the limitations of the vinyl record format, rather than listening preference (Dolby,1999). Current cinema surround formats may have several front speaker channels, but only one or two channels accounting for the signal to all speakers on the sides and rear of the cinema. This partly reflects the development of the format. The rear channel has continually expanded to create a more diffuse soundfield, but without taking on more actual discrete channels of audio. Current home theatre formats are largely based off cinema formats, which works well in the sense that the surround sound for a cinema film can be easily converted to surround for a DVD or Blu-ray. However, this has also created some redundancy - the LFE (Low Frequency Effects) channel was created in the analogue cinema era so bass frequencies could be included without overloading the main channels, but is essentially redundant in the modern digital era, especially in home formats. In home theatre systems, many formats have come and gone. The 5.1 format has succeeded as the first popular format with more than two channels, ostensibly due to forceful marketing (Elen, 2001), pushing out competitors such as Quadraphonic and Ambisonic. Dolby formats dominate cinema, but others do exist. Discrete channel numbers in both home and cinema systems are increasing but formats are slow to catch up with what speaker systems can play, and the use of more and more channels puts more workload onto the mixing engineer to utilise everything well. Mono and stereo sound have been mixed by engineers with maximum utilisation of all available channels since the beginning, with careful panning and equalisation10 of audio to create the best possible soundscape. Theoretically, surround formats should simply expand 9 Of course 'best' is extremely subjective. Most accurate reproduction of sound, most pleasing to listen to, highest quality format, cheapest or simplest format? 10 Modifying the frequency components of audio.

14 14 on these formats, introducing the ability to create an immersive experience of literally being surrounded by sound, with more space to include sounds without them vying to be heard in the same speaker, and the option of making the listener feel as if they're really 'there' surely an ultimate goal of the film experience. In practice, good use of 5.1 in the film industry is variable at best. Recording of sound is usually mono or stereo (although simple surround recording solutions do exist), with judicious use of panning and fake ambient effects taking the place of true surround recordings. The simplistic nature of surround channels in cinema, covering a large area of the rear and side of the cinema with a single channel, limits options for rear sound placement further, and with many films there may be only the occasional sound behind you with almost everything coming from the front. The quality of audio formats has certainly improved. Old analogue formats often relied on downscaling to fewer channels in encoding, and some information would be lost when upscaling11 to all channels again for playback. Modern sound in both home and cinema formats is digital, and while quality may still be lost through the use of data compression12, channels are stored completely separately and what you record is essentially what you get back out. In short, this means more adventurous panning and sound placement can be implemented. Furthermore, modern computers are now powerful enough to run many audio channels at once, and simplify audio work so that complex mixing can be done in a much shorter time frame. There is also enough processing power to calculate mathematical models that were previously inaccessible due to the huge processing time needed. For instance, this report explores the use of convolution through mathematics software to place sounds in a location in which they were not recorded. This process allows one to record a real location, and later apply the sound of that location to any other audio. The development of surround sound formats and new technologies has left the current state of film audio in something of an uncertain place. This project builds on development that has already occurred, exploring techniques which may assist surround sound to move smoothly into a more standardised and immersive future. 3.2 Introduction A fundamental idea of this film project was to implement surround sound in a more convincing and technically thorough way than is the current norm. Dolby, as the main purveyor of surround systems provides several excellent white papers on mixing in surround, which are available for free. This report references the Dolby Surround Mixing Manual13 and 5.1 Channel Music Production Guidelines14, as well as the short Dolby 11 This is done by processing within the decoder itself, converting the channels in the signal into more channels for playback. 12 For instance, all DVD audio is compressed to a maximum data rate of 448kb/s, meaning that some audio quality is lost in an effort to reduce file size. With a small amount of good compression, this loss should be inaudible. 13 Issue 2 14 Issue 3

15 15 Digital Guidelines document. There are many routes which may be taken in mixing 5.1 surround sound, the most common for film being to keep most sound and music in the front left and right speakers, dialogue in the centre speaker and a few ambient sounds and extra effects in the surround speakers (see figure 3.1). Music production in 5.1 generally contains more in the surround channels, but this varies. Dolby themselves give some general guidelines. As a guideline, the image should direct attention to the front of the sound field and it should be noticeable when the Surround channel is removed from the mix. Attention should not be drawn directly to the Surround channel when it is returned to the mix. Surround channel effects should complement, not distract from on-screen action. (Dolby, 2005b, p. 47) Obviously the aim is to keep the audience mainly focused on the screen in front of them, so having the loudest sound coming from that direction is logical. However, having the loudest sound coming from the front does not have to be at the expense of surround channel content. We can perceive a sound as being in front of us while other sounds, and a large component of the main sound, are also coming from behind us. Dolby themselves admit that there are many options and little standardisation. Use of the centre and rear speakers continues to be a topic of debate among artists, producers, engineers, and other serious listeners. It is expected that genres of perspectives will develop over time and that mixes may someday be grouped into categories such as in the band, in the audience, or other real or imaginary listening perspectives. (Dolby, 2005a, p. 36) Needless to say, these categories have not yet eventuated. There seems to be a lack of knowledge in the industry about how to mix standard 5.1 surround well, and how to set up and use speakers correctly. This isn't helped by a lack of standardisation. Surround in film is generally based on panning mono or stereo recordings around the sound-field as necessary, and even true surround recording devices do not usually match the angles of speaker placement standards. In The Paintbrush, the aim was to explore recording as much of the sound as possible in four channels (two front, two back), which from early tests seemed as though it could give a much better sense of being on location. This could then be compared with standard methods fairly easily by removing the rear component of tracks. Films also generally use stereo music tracks, mostly placed in the front left and right speakers as a simple stereo (2.0) set up would be. Since the soundtrack15 for The Paintbrush was recorded specifically for the film and mixed in surround from the outset, the music could continue to present itself in all speakers. Since the rear surround channels are used little in most films, it seems logical to place some of the music content there. 15 Soundtrack can mean the music or all the audio of a film depending on the context. Please note that in this report, the word soundtrack always refers only to the music. When referring to all audio in the film, the words sound or audio may simply be used.

16 Recording and mixing methods 5.1 mix Figure 3.1 Traditional movie surround versus The Paintbrush Most audio, for music or film, is still recorded in mono or stereo using traditional methods. Many 5.1 music mixes are based on the same tracks as the stereo mix, just with additional channels available for panning. This is mainly useful in the sense that having more speakers means less masking of multiple sounds coming from one location, and less degradation of sound from the speaker having to play several sounds at once. Being surrounded by music can also be more pleasing than simple stereo, sounding more 'open' or immersive. Note that immersive doesn't mean realistic. Panning is dictated by artistic choice rather than recording method. Granted, the music in The Paintbrush is largely done the same way; the focus on true surround is more aimed at the ambience and sound effects, while the feature of the music here is that it's mixed in surround at all. However, if I could record instruments into four channels I did so, as described in section 2.3. Sound effects for film are almost always recorded in mono or stereo, as will be apparent from looking at any Foley sound set. Mono and stereo are easier to record, take up less storage space, and are easy to pan into the desired location. Have a mono recording of a car you want to be behind the listener? Pan it to the back. However, say you record a car in quad (four-channel) from four different directions, with the car behind the microphones in the position it's intended to be in the film. This is the way every sound effect in The Paintbrush is recorded. This should (and does, from listening tests) give the listener a much better sense of 'being there', since the sound is distributed correctly into all speakers, as it would be if you were on location Reproduction of sound from all directions through a standard 5.1 system is not perfect, even with perfect speakers. A system called Ambisonics may be of interest, as it attempts to alleviate many of the sound localisation issues with discrete channel surround. In Ambisonics, sounds are positioned in the sound field and played through each speaker as necessary, rather than sending to specific channels. At the centre point, sounds are reconstructed exactly. The system was invented by Michael Gerson in the 1970s, and is touted as being superior to Dolby-style surround in several ways, but has never been widely marketed.

17 17 17 The advent of not only stereo, but quadraphonic and even 5-channel microphones built into one unit are making surround recordings much easier. For the sound effects in The Paintbrush, Massey University's Zoom H2 recorder (figure 3.2) was used exclusively, which is a small, portable quadraphonic recorder that works extremely well. The microphones have a fairly flat frequency response that pick up sound clearly and accurately. The microphones are not as high-quality as the NT5 array used for the music, but very high quality is a little less important for effects. The problem with surround recording is it's very tailored to a specific scenario. The recordings for The Paintbrush are perfect for the movie because I recorded them to match position as closely as possible with what's happening on screen. This means the sounds don't even need to be panned into location the four channels just need to be placed in their respective surround speakers L, R, Rs and Ls. However, these recordings would be difficult to use in another situation. Switching a sound from the front to the back would involve simply swapping the four channels around, but not everything is so easy. Figure 3.2 Zoom H2 on the left, with some props used for recording the (incomplete) scene on the right. The side of the Zoom H2 with the screen is the 'back', recording at 120, while the front records at 90. It recording in four-channel is worth it? I think so, at least in some situations. If you're recording most sounds specifically for a certain film, and you have the time and equipment to do things properly, why not utilise the surround speakers which much of your audience will have? The sound is demonstrably better. The Zoom H2 and similar recorders eliminate the difficulty of recording to more than one or two channels at once, and file size is much less of an issue as digital storage continues to increase in size and value. One area of film where true surround recordings are sometimes already used is in ambience. Ambient recordings of background location sounds often appear in all speakers, including rear surrounds; in contrast to the great multitude of sound effects which appear exclusively at the front. However, ambience is still often recorded in mono or stereo18. A mono recording can give a fairly good sense of being on location when played through all speakers, but all sounds will appear to come from all directions (essentially from the centre), where the listener is, and 17 See, for instance, the Holophone H2-Pro microphone array. 18 I would go so far as to say most of the time, since small location recorders are often used for ambience, which have only a single microphone. However, I have been unable to source concrete data on the subject.

18 18 any moving objects will appear stationary, mysteriously getting louder and softer (with a little Doppler effect) but not actually coming or going. Using the Zoom H2 near a road instantly shows the benefit of more channels, as when played back, cars sweep past in an extremely convincing manner. Generally, the centre channel is reserved predominantly for dialogue. Giving dialogue its own speaker helps isolate it and bring it more clearly to the audience's ears. Since there is no dialogue in The Paintbrush, instruments in the soundtrack, and some sound effects can be placed there, as well as blending in some audio from the left and right front speakers to help produce a stable front sound image. When the left and right speakers are too far apart (see earlier speaker placement discussion), with nothing in the centre channel,...sounds are drawn into the speakers: localization is poor at the front... (Elen, 2001). With all sound effects and ambience recorded in 4 channels, and music in all speakers, The Paintbrush explores a more natural, immersive and realistic approach to surround mixing that is not often seen in any other films. Music mixing - what people want Music in The Paintbrush is mixed to take full advantage of every available speaker, though the front left and right speakers continue to contain the loudest and most important instruments. However, Holman (2000) references a Gallup poll showing that two thirds of listeners preferred a best seat in the house perspective rather than in the band. The preferred option may still be mixing all instruments to the front three speakers, with only reverb and crowd noise (if a live performance) in the rear. One may wonder, though, whether the general listening population is aware of the increased clarity and space associated with the in the band arrangement. When answering this poll, it is highly doubtful that examples of both styles were actually provided; choice may be based on perceived preference rather than true experience. Stereo mix A stereo mix of The Paintbrush is provided for compatibility with systems not equipped for surround sound. The simplest, and common way of creating a stereo mix can be done automatically from within a surround decoder19. Simply combine the Ls and Rs surround channels into L and R respectively (often at -3 decibels reduced volume), and the centre channel into both L and R equally (also at -3dB). Reduce overall volume and/or apply compression as required to keep levels under 100%. Some films provide a separate stereo mix along with a 5.1 mix, which should be automatically selected by the listener's decoder when appropriate. The Paintbrush currently provides a separate stereo mix, but one which is created via a simple combining of tracks as previously described. If there is enough time, a better stereo mix may be created where sounds are actually mixed properly into stereo. For instance, extra room sounds from the rear speakers should be removed, and EQ should be applied to instruments differently to fit them together more amicably. With fewer speakers, some instruments may be better off 19 Called downmixing.

19 19 removed altogether to retain space in the mix. Simply combining tracks can muddy the sound as more instruments and effects come from the same location. This is more difficult for the speaker to produce, and more difficult for the human ear to hear, as sounds in the same frequency band tend to mask each other. If the same sound originally came through multiple speakers, it may now be doubled-up a big problem if the two were slightly out of time. Issues like these are worse than usual in The Paintbrush, where far more of the audio (especially the music, which is often only stereo anyway) is in the surround speakers. Creating a dedicated, separate stereo mix of the audio (especially the soundtrack) will always produce a more desirable result. The DVD format in particular is set up to easily switch between different mixes automatically depending on the user's number of speakers, so that is not an issue. The real issue is that a simple combining of channels is almost instantly done, while a true mix in stereo may take a long time. 3.4 Standards and implementations Standards DVD audio supports a multitude of different formats, from 1.0 (mono) to All support up to 24-bit/96KHz bit rates, well ahead of the 16-bit/44.1KHz format of a standard Red Book audio CD. Additionally, mono or stereo tracks may use up to a 192KHz sample rate. Since multiple audio tracks may be included on one DVD, multiple formats are feasible. For The Paintbrush, most recording was done via an Mbox 2 or using the Zoom H2 recorder, both of which only support up to 24-bit/48KHz, so those settings were used throughout. An upmix to a higher sample rate would have had a negligible effect, therefore I cannot comment on the benefit of including, say, a 192KHz stereo track instead of taking the automatic downmix from the 5.1 track. Note that while consumers and cinemas may have more than 5(.1) speakers, DVD doesn't support these formats. Amplifiers that support greater numbers of speakers simply upmix the 5.1 (or less) content into more speakers. Blu-ray discs now support greater numbers of speakers and higher quality audio.22 Dolby specifies recommended surround speaker layouts for several different rooms, and a basic layout for 5.1 as shown in figure 3.3. Subwoofer placement is left up to experimentation, with the main goal being that the subwoofer's location in the room is not apparent to the listener. Asymmetrical placement is recommended so as not to create symmetrical standing waves in a symmetrical room (Dolby, 2005a). Note how the front left and right speakers are only 30 degrees either side of centre. 20 Technically DVD also supports SDDS Sony Dynamic Digital Sound which has up to 8 channels, but no current home theatre systems support it. 21 See the Glossary for explanations of terms used in this section. 22 Not in terms of bit depth or sample rate, which is the same as DVD, but due to the fact that DVD audio also uses lossy compression (as in data, not as in amplitude) up to maximum bit rate of 448kb/s, while Blu-ray can go so far as to use completely lossless compression.

20 20 Figure 3.3 A a chart of Dolby 5.1 recommended speaker layout. See Dolby's 5.1-Channel Music Production Guidelines, Figure 3-3 for a more thorough diagram the document is free, but the original figure is copyrighted. Every channel except the LFE is full-range, while the LFE carries only sound between 3Hz and 120Hz. However, it is a popular misconception that the LFE channel is the 'subwoofer channel' (Elen, 2002). Not so. If any other speakers in the system are not full-range (or at least, cannot go as low as a connected subwoofer), bass management is intended to send the low frequencies to the sub(woofer). Therefore, the subwoofer will carry LFE channel content (sounds up to 120Hz), and any sounds from other speakers below the chosen crossover frequency, as long as bass management is used correctly. It is vitally important not to treat the LFE channel as the source of all bass, most importantly because the LFE content is completely discarded by Dolby Digital decoders when downmixing (Taylor et al., 2008). Strangely, the angles of the recommended speaker configuration don't match the angles of most recording devices. For instance, the Zoom H2 has a 90 degree angle at the front although this sets it up at an angle between quad and 5.1, so it could be used for either. Therefore if people have their speakers set up correctly, the sound field won't actually match what I've recorded as sounds for The Paintbrush. There isn't really anything that can be done about this except using a custom microphone array or possibly some clever panning. Studio reality In an ideal situation, the studio will have speakers arranged as in figure 3.3, with one or more subwoofers taking the LFE channel and any bass that falls below the crossover frequency of other speakers. In reality, not all studios have correct bass management. Digidesign Pro Tools systems only provide discrete outputs any sending of bass to different channels has to be done after the output from the Pro Tools system itself. Other software and hardware systems are not always any better.

21 21 Pro Tools does provide an extra fader on every 5.1 track for sending audio to the LFE channel, but this isn't managed bass, as the level going to the subwoofer is set by the user independently of the volume of the track itself. Furthermore, bouncing-down tracks to 5.1 will include all the extra LFE content in the LFE track, which isn't what we want. This would cause bass levels to be doubled up, as adding to the LFE doesn't cut from the main track. If the user has full-range speakers, bass will now come from the subwoofer (due to the LFE track) as well as the channels the bass track was originally in. If the listener doesn't have full-range speakers, bass management will send the same bass to the subwoofer twice. It is possible to obtain pseudo-bass management in Pro Tools by setting up your I/O appropriately, applying specific EQ, and sending tracks to multiple outputs (see Bass Management, 2007), but this requires extra work and there is no official support. Outputs really need to be run through an actual decoder which handles bass management itself. If not, each of the surround speakers needs to be full range (or have its own subwoofer). Cinema reality Even very small cinemas these days usually have at least Dolby Stereo playback, which is (contrary to what the name might suggest) four tracks L, R, C, and Surround encoded down into two, and matrix decoded back into four for playback. Better cinemas will have Dolby Digital23 in at least 5.1, and/or competing formats such as SDDS24 or, more commonly, DTS. All surround speakers on the left side and back of the cinema will take the left surround signal, and the same for the right. Front L and R are placed at either extreme of the cinema screen (Dolby, 2005a). In most films, the surround channels are used fairly sparsely, so the system works well except that people near the front of the cinema may end up with much louder sound than those at the back, since most sound is coming from the front. On my own travels I noticed particularly that the Paramount Theatre in Wellington seems to be using loud and not very accurate PA-type speakers at the front, with a boosted mid-range so everything sounds harsh and sibilance in speech is unnaturally loud. Not an ideal setting to showcase your film, where listener's ears quickly become fatigued. Other theatres like Reading Cinemas in Wellington have excellent speakers but are playing movies too loud indeed, above regulatory limits, with a chance of permanent hearing damage. In a way, the current standard of keeping most audio in the front speakers is a way of playing it safe, in the sense that following the status quo means your soundtrack will almost certainly sound okay as cinemas will be set up for it. The problem with something that uses a lot of rear surround content like The Paintbrush, is that there are less parts of the theatre where everything is going to sound well balanced. I fear that by making a mix better, you are also making it worse for anyone not in the middle of 23 Individual cinemas boast about their sound systems, but there doesn't seem to be any concrete data on what percentage of cinemas have which system. Certainly, a large percentage of cinemas in New Zealand are now using Dolby Digital sound. 24 Sony Dynamic Digital Sound. This is, however, a dying system.

22 22 the cinema (see figure 3.4). Figure 3.4 A movie theatre may have a relatively small 'sweet spot' the optimal listening area. Being too close to any edge will make the respective speaker channels disproportionately loud.25 If there is very little content in the surround speakers, the only real problem is that sitting further back in the cinema will be quieter overall. However, you're also essentially wasting two channels and sometimes many speakers on the sides and back of the cinema just to be safe and traditional. If there is a lot of content in the surround speakers, as in The Paintbrush, there should be a much more immersive effect when sitting near the centre of the cinema, but I am concerned that there could be major volume balance issues when sitting too close to the sides, especially near the back. Unfortunately, it's not possible for me to test the film in a real cinema. Consumer reality Consumers playing a DVD at home may possess a multitude of different speaker systems. A TV with stereo output, or worse, only mono. Some will have two speakers with an additional subwoofer (a so-called 2.1 system), or a full 5.1 setup. Less common systems include a multitude of standards including quadraphonic (4.0) and anything up to speakers and two subwoofers, with 7.1 becoming popular due to the Dolby Digital IIx upmix decoder becoming standard in many home theatre receivers. Actual survey data seems impossible to find, but the most common by far are almost certainly 2.0 and 5.1, owing to the proliferation of systems sold in these configurations. DVD itself supports audio from mono to 5.1, and more channels will always be able to automatically downmix into less. Consumer systems obviously aren't usually going to be up to the standard of a cinema, but the upside is that if things are positioned fairly well, it alleviates the aforementioned problem with cinema where you need to be sitting near the middle. If you place your speakers around 25 Paramount Theatre image courtesy of Venue Web, retrieved September 21, 2009 from

23 23 the room and sit on a couch in the middle, everything should be fairly good for everyone on the couch. Of course, many people do not set things up correctly, or have inferior speakers. The aim, as always, is to mix and master the audio in a way that's going to sound as good as possible in everyone's system. A lot of time was spent with The Paintbrush refining the mix, testing it on various speaker set ups: 5.1 Massey University's main set up, with Yamaha satellites and subwoofers. Quite accurate sound reproduction, powerful, some bass management issues. 5.1 Massey's small Logitech consumer set up. Less accurate, most bass from subwoofer (small satellite speakers). 2.1 My Energy set up at home. Quite accurate general hi-fi system. Main speakers are large little use of subwoofer. 2.0 My small Genelec studio monitors. Very accurate apart from lack on low-frequency bass reproduction. 5.1/7.1 Wyatt Page's home set up. Accurate good quality system. Ability to upmix to 7.1. I think the final mix is fairly well balanced despite the wide variation in different systems, as illustrated in figure 3.5. This won't apply, however, if people have their systems set up incorrectly, and again The Paintbrush is a little more susceptible to that than traditionally mixed music or film.

24 24 Figure 3.5 Frequency responses of two home theatre 5.1 surround systems. The general aim is to have a flat line as frequency increases on the x-axis, thus providing accurate sound reproduction. Left, a good quality Canton home theatre system (Pohlmann, 2009). Right, a similarly priced Bose Lifestyle 15 system. Bose is a very popular brand unfortunately known for providing style over substance or quality. Note how none of the speakers can really produce audio at around 300Hz. (Intellexual.net, 2004). Bass Management Bass management is so widely misunderstood that I feel it deserves its own section. First, and most importantly, the Lfe channel, the.1 in 5.1, is not the subwoofer channel. Any decoder should also include bass management, where if any of the other speakers are not full-range, and a subwoofer is available, bass will automatically be redirected to it. The user should be able to manually set the crossover frequency where this happens. The LFE channel itself is a relic of analogue cinema where bass frequencies could overpower the rest of the mix, and so were relegated to their own channel (Elen, 2002). The Lfe channel remains in modern digital consumer systems but is effectively redundant. The only remaining real reason to use it appears to be if you desired bass to come from the user's subwoofer even if they had full-range speakers on all other channels. Various sources also claim that when the user doesn't have a subwoofer, the Lfe is discarded entirely, although Elen appears to claim that it is rerouted to any other large speakers. The true explanation seems to be that different receivers each tend to implement bass management in different ways (Blackburn, 2000, gives a wealth of information on this). Correct bass management is specified in the Dolby Digital specifications, so one may expect all Dolby Digital decoders to work the same way. Not so. Dolby Labs admitted that many (I would say most) products are not managing bass properly. Dolby Labs indicated that they will explain to the manufacturer what the problems are with bass management in each product Dolby Labs evaluates, but Dolby Labs will not stop the company from selling the product as is, nor will Dolby Labs withhold permission to apply the Dolby Digital logo (Blackburn,2000). Elen goes so far as to say If you are mixing music, don t put anything in the LFE (2002). Blackburn indicates that many films put extra bass in the Lfe channel for a multitude of minor sounds (indeed, he uses this as a sign of a good quality mix).

25 25 Figure 3.6 Complete waveform the 5.1 SACD mix of Pink Floyd's Dark Side of the Moon (L, R, C, Lfe, Ls, Rs). Figure 3.6 shows a waveform of a contemporary SACD music mix in 5.1. There is some content in the LFE channel but the volume is very low. I've personally listened to this mix, and the Lfe channel really only contains a quiet rumble. It wouldn't be a huge loss if the Lfe channel was discarded in a subwoofer-less system, but the drop in bass level may be noticeable. I started mixing the soundtrack to the film with a fair amount of material in the Lfe. After I read up on correct bass management practise I reduced this to what you see in figure B1.3 in Appendix B. After some testing I decided the most logical course of action was to move all bass to the main channels as Elen recommends. This would have the added bonus of correctly directional bass if the user had full-range speakers. The final mix is as seen in figure B1.1; the only remaining Lfe content being some extra low-end on the couple of bass glissandos. It's worth stating again that this doesn't mean there is any less bass. As for using the Lfe in film (as opposed to music), it's hard to say what the best option is. I used a little Lfe on low sound effects but mainly I kept bass in the main channels (see figure B1.4). I suspect that a lot of the Lfe content in films is based on the use of the Lfe channel in cinemas, if a separate 5.1 mix is not done for the consumer market. And by the sound of things, a separate mix should be done, but it may come down to time and budget constraints. You don't want to put too much in the Lfe, in case it gets discarded entirely, but what if the user has their system set up with incorrect bass management, so they don't get bass from the main speakers? Blackburn states that many decoders simply discard the bass from small rear channel speakers, and the user could also accidentally set their front speakers to 'large,' disabling bass management to the subwoofer when the speakers can't handle bass either. In that case the Lfe channel may be the only bass they get. In the end, one really can only hope that whoever's listening has things set up correctly, as making bass frequencies work for all people in the mess of standards and ignoring of standards is effectively impossible. 3.5 Results of The Paintbrush in both mixing styles I experimented with mixing part of The Paintbrush in a more conventional style, for comparison.

26 26 Changes made were as follows: Soundtrack switched from 5.1 mix to stereo mix, panned to L and R speakers Ambient noise converted to a single mono file, playing through all speakers Sound effects converted to mono or stereo, and almost entirely placed in front L and R speakers, unless something was obviously right behind the listener. The audio may sound roughly similar to the true version on casual listening. However there is a distinct muddying of the sound from putting much more through the front speakers, and a certain loss of immersion from the lack of audio in the rear. As with all audio applications, quality of sound is largely subjective; it's hard to gauge exactly how much better or worse each version is. It does seem that using the true quadraphonic recordings, and putting more into the surround speakers, creates a greater sense of realism. It also makes the film more enjoyable to listen to. The version with most of the sound from the front is somewhat flat and boring; the true version feels more 'right.' 3.6 Issues with the Massey University Wellington surround room The Massey surround room (3D31, Wellington campus) has decent speakers and equipment, but it is currently impossible to set up the main system completely correctly, and it is difficult to use the secondary system at all (as at September 2009).26 Main Speakers The problem with the main speakers is the fact that Pro Tools has no bass management. For details see section 3.4, Studio Reality. As mentioned earlier, it is possible to have bass management in Pro Tools but only through inexact and unofficial methods. Figure 3.7 shows some possible configurations using the standard Pro Tools 5.1 channel panner. Assume that a track has some low bass content. With correct bass management: The first option would have bass coming from the font speakers if they were large enough, otherwise bass would be routed to the subwoofer. The second option would have bass coming from front and rear speakers if they were large enough, otherwise bass (from any that were small) would be routed to the subwoofer. If all four were small, the same bass would be sent to the subwoofer four times, but it should work acceptably well. The third option would be the same as the second, except the Lfe is also sending bass from the track into the subwoofer, so the bass will be doubled up even more. One may want to reduce the bass in the main tracks to compensate. However, Pro Tools has no bass management so if the surround speakers cannot play low 26 Edit, 18 October 2009: It has been brought to my attention that the Yamaha DM1000 mixing console in the room can process bass management. Set up and testing is still needed but this is likely to be the best and simplest solution, requiring no extra purchasing.

27 27 bass, you won't hear any. One might then be tempted to take option C and add to the subwoofer (connected directly to the Lfe), but when someone plays your mix on their Dolby system, they'll be getting twice as much bass as you're hearing. A B C Figure 3.7 Managing bass using the Pro Tools 5.1 panner. A will put the bass in the front left and right speakers only. B will put the bass in all speakers (except any connected only to Lfe). C will put the bass in all speakers and in the Lfe channel. Massey has four subwoofers (and 8 mid-size speakers). I propose that the best current option is how the system was set up on delivery a sub on L, R, Ls and Rs, just the small speaker for C and nothing for the Lfe channel. This means that when not using the Lfe (and the only Massey paper that currently uses the surround room is a music mixing paper, so they shouldn't be), things will sound pretty much how they will in the end-listener's system. The subs will need to be rearranged to hear the Lfe, sacrificing low bass on another speaker. Secondary Speakers The smaller problem with the secondary speakers is that it is impossible to easily A/B27 between them and the main speakers, and a greater problem is that the right signal cannot be sent to them at all. The Logitech system can take six direct inputs, or it can take a single S/PDIF digital input. The S/PDIF input should be compatible with the S/PDIF on the 003, but for some reason it is not. The 003 appears to be using a slightly different format. The 6-channel direct input system works but distorts a lot at high volume since the 003 line outputs are professional +4dBu level, while the Logitech system is expecting -10dBV. The best current option is using the direct connection and simply turning the volume far down in Pro Tools so it doesn't clip. This still requires unplugging all 8 plugs for the main speakers and switching them for the 8 plugs for the Logitech system, in the correct order. 27 Switch between speakers for comparison purposes.

28 28 Acoustics The room is expected to move at some point so any current issues may be alleviated. Actual acoustics and reflections in the room don't seem too bad. Attenuation of outside noise is very good. The main issue is that the light fixtures rattle loudly at a bass frequency which I have isolated to be approximately 50Hz.

29 29 4 Convolution 4.1 What and why This section explores the use of convolution in audio. For a quick mathematical explanation of convolution, please first see Appendix C. An explanation of convolution Convolution is used in audio to record the acoustics of a room. Through the mathematical process of convolution one may isolate only the acoustics (mostly the echo/reverberation) of the location, and then apply that reverb28 to any other sound. Many hardware and software convolution units and plug-ins exist for processing audio, usually in real time (thanks to the speed of the fast Fourier transform). Most of them are also very expensive. There are a few small general purpose convolution programs available but these are usually geared to the more technical user. I contemplated attempting to create a simple, free convolution plug-in for Pro Tools that could then be provided as a free alternative for other people to use. I had already worked with convolution in other university papers, using MATLAB29. However I soon discovered that Digidesign, the creator of Pro Tools, requires that a fee be paid to gain access to develop plug-ins in their format30. Massey suggested that they may be able to pay this fee, but I decided to develop a convolution utility with a Graphical User Interface (GUI) in MATLAB, being already knowledgeable of the process. We had explored the use of convolution with audio in a previous audio paper and the results seemed to be very good. We had taken pre-recorded room responses and written MATLAB code to convolve and deconvolve them with other sounds. I was interested in exploring this further, and convolution could also provide some effects that were needed in the soundtrack. Namely: No reverb plug-ins that Massey nor I had could create reverb effects in true surround. The best possible effect would be using two stereo reverbs and adding a slight delay to the rear effect. If I recorded the sound of an appropriate room using convolution, I could record in quadraphonic using the Zoom H2, and hopefully place the user more believably in a location. This method can be applied to mono or stereo as well as quadraphonic sounds. I had recorded the old piano section early on, and it later transpired that in one part, the sound of the piano needed to be coming from another room. It would have been difficult to make the trip to go back and record again, and recording in another room would likely be low level and contain a lot of background noise. I wondered if convolution would be able to recreate the sound of being in another room convolution on audio is usually only used for reverb effects. 28 Reverb is the common term for echo effects on audio. 29 MATLAB is a software program which allows use of a high-level programming language to easily perform many complex mathematical operations. 30 RTAS Real Time AudioSuite.

30 30 The source for convolution may be recorded using an impulse or a sweep. An ideal impulse is impossible to create in the real world; a perfect impulse is an infinitely short pulse that contains all frequencies. However, a short, sharp sound like a gunshot may suffice. The other common, and better option, is to use a sine wave sweep; a sound which sweeps through a large frequency band31 at a constant volume. The recorded sound can be deconvolved with the original sweep sound to create an impulse response. This is what you would have got directly if using the impulse method. However, using a sweep is less susceptible to distortion and can give a cleaner, more balanced result (Kemmler, 2006), especially since you're now making sure to include all frequencies equally. Figure 4.1 Various waveforms showing the convolution process. A: A sweep file. B: A after being played in a room and recorded again. C: Pure impulse response obtained from deconvolution of B with A. D: A piano recording. E: The same piano recording after convolution with C. Process Figure 4.1 shows a complete process of acquiring an impulse response and applying it to a sound. The process as I followed it is: 1. Acquire a sweep file and play it through a speaker into the intended room. Record the 31 I used 20Hz to 20KHz. Higher frequencies would have been represented badly at the 48KHz sampling rate being used. Above the Nyquist frequency (24KHz), frequencies wouldn't be represented at all and lower, alias frequencies would start to be created. 20Hz on the other hand is already getting too low for my small speakers to play.

31 31 sound. 2. Deconvolve the recorded sound with the original sweep file. This will produce an impulse response (this is what you would get from step one if using the impulse method). 3. Convolve that impulse with any other sound. The final step will add the sound of the recorded location to your sound file. Considering how simple the process actually is, as long as the original recording is good, it tends to work extremely well. The real key is finding good sounding locations to record. Equipment I took the following items to each location recorded (see figure 4.2): Dell Studio XPS 13 (1340) a laptop, to feed the speaker from the on-board sound card. I would have preferred to use my Digidesign Mbox 2 for better quality sound output but it is not supported by the laptop's operating system. Genelec 8030 small active32 studio monitor speaker with relatively flat frequency response. Good and flat frequency response is desired to reproduce the sweep file as accurately as possible. Zoom H2 Massey's four-channel portable recorder. Cables C1433 power cable and XLR to 3.5mm TRS audio cable for the speaker. The sweep file I used was taken from one used in the university paper Applied Multimedia Signal Processing. The only issue with this file was that it only had 44,100 samples/second, so the 20KHz waves were already represented badly, but this would likely make no audible difference Amplifier built-in 33 Jug plug. 34 It is often taught that the Nyquist frequency half the sample rate is where frequencies can no longer be represented and aliasing starts to occur, and hence one should endeavour to never go above this frequency. This is true, but seems to have led to a common misconception that everything up to Nyquist is magically represented with perfect clarity. Not so. A 20KHz sine wave with a 44.1KHz sample rate is only going to have 2.2 sample points per wave, producing a jagged and non-symmetrical sine wave quite unlike its true form.

32 32 Figure 4.2 Equipment used for recording sweeps. 4.2 Convy a convolution processor in MATLAB Convy is a small, simple application for extracting impulse responses via deconvolution, and applying impulses to files via convolution. It simplifies the processing of multiple files and is designed to be easy for a novice to use while also being passively educational, and robust in terms of avoiding errors. Figure 4.3 shows the user interface. Figure 4.3 Convy after deconvolving a sweep recording and applying the impulse response to another audio clip.

33 33 Convy source code files contain comments which explain the code. Typing help and the name of any of the source files will also bring up information. For example, entering help convol: >> help convol CONV Convolutes one (mono) file with another. Z = conv(x, Y) I suggest using wavread to read in wave files, e.g. x = wavread('x.wav'); Convolutes x with y, and outputs result z. Usually, x should be a sound file and y, an inpulse response. Convolution in the time domain is multiplication in the frequency domain, so the process performed here is as follows: - get the FFT of each input - multiply the two inputs - perform an inverse FFT on the result Finally, the output is normalised to 1 if necessary to avoid clipping. The main program may be run via convgui.m. Convy is essentially a front-end for simple convolution and deconvolution, and uses Fast Fourier Transforms to do so as quickly as possible. Silence at the either end of the output file is automatically cut. The program will prompt the user if a required file is missing. Files may be deconvolved to obtain an impulse response, or an impulse response may be loaded directly. The impulse response as well as the final output may be saved to the.wav file format, and the final output may be played back with or without saving. Convy was fairly simple to develop. One time consuming aspect was catching all errors when a user hasn't loaded all the necessary files for processing, or the choice of deconvolving files or not was changed by the user and files had to be cleared or reexamined. The interface also changes significantly when the Deconvolve? button is pressed, and plots must be cleared and reloaded correctly. Actual convolution and deconvolution is fairly simple thanks to MATLAB's built-in Fourier transform commands. Remaining issues with Convy Convy only accepts files with a.wav extension. This is a limitation of MATLAB's wavread command. Only one file can be processed at a time. Batch processing for e.g. four channel recordings would be a nice additional feature. Convy cannot update the GUI or the MATLAB console while files are processing or audio is playing. At other times, it can only update the console. If a file takes a long time to process, it may confuse the user. Meanwhile, commands will queue up to be executed once

34 34 processing is complete, so a flurry of commands can then occur at once. Audio cannot be stopped once Listen is pressed. This is a limitation of MATLAB's wavplay command. Convy still crashes if a non-mono file is loaded. This could be fairly easily fixed by checking the input array dimensions of each file when loaded, and/or adding support for stereo files. 4.3 Use in The Paintbrush Locations recorded I recorded several locations first to test whether my own recordings would work well in convolution. In the past I had only used pre-recorded sweeps. The earlier figure 4.2 shows recording in The Pyramid at Massey Wellington. This gave a huge reverb which was good for testing realism since it was so obvious when added to anything. It came out very well, although the effect was far too great for the actual purposes of the film. The Paintbrush called for small to medium sized indoor rooms in a home. I used my own home for this, recording several rooms and later selecting which sounded the best. These room recordings should closely match the appearance of the rooms in the film. To create the effect of the old piano being in another room, I recorded exactly that the sound of the speaker from another room. I experimented with being one or two rooms away, always with one door closed to muffle the sound. Use of convolution It transpired that background noises had a large effect on the resultant impulse response obtained from a sweep recording, and therefore several recordings were done of each location, where I would subsequently isolate the cleanest specimen. For example, in my house the meter board clicks occasionally in a fairly loud but brief manner. If one of these clicks occurred during the recording of a sweep, upon deconvolution into an impulse response it would become a quiet fweee sound like a toy ray-gun being fired (I assume the miniature frequency sweep of the click is caused by the original click transposing itself over multiple frequencies of the sine wave sweep). Upon convolution with another sound, ray-guns could be heard on every transient. Although traffic, aircraft and bird noise were impossible to filter out entirely, recording the same location several times and keeping the volume of the primary source (speaker) loud produced fairly good results. There aren't any noticeable artefacts in the final impulse responses. Occasionally there would be background noise in the recording that would not become apparent until playback. The human mind has adapted to focus on one sound and automatically filter out background noise as much as possible, so it remains important to consciously listen for noises that may enter into the recording, especially when recording the sound of a room for convolution.

35 35 The room recordings came out fairly similar, so I chose that which sounded the best but also had the cleanest recording (with little contaminating background noise). The effect was good, but I feel that pure convolution reverbs don't usually sound as pleasing as a good traditional (fake) reverb effect, and this was no exception. Convolution reverb tends to be quite dense and sometimes muddying to the sound, and professional convolution reverb plug-ins tend to have a lot of parameters which can be modified as necessary maybe not something you'd expect to be necessary on a so-called perfect recreation of any location. It may not have helped that the main sound I used the effect on the main piano recordings was not a completely dry35 recording to begin with. The convolution reverb made the piano sound less dry and boring, but the convolution track is mixed fairly quietly behind the main sound in the final soundtrack. The recordings in another room worked well. I was unsure whether the muted sound of being in another room (mainly a cut of treble frequencies) would be applied with convolution along with the echo of the room. The effect turned out well, frequency effects were obviously applied but they weren't particularly strong. The version used in the film is the cleanest recording from the furthest away, with an added 6dB per octave low pass filter in Pro Tools to roll off some high frequencies further. The version used on the old piano in the soundtrack36 is the convolved audio only (whereas the main piano is the original and the convolved audio). This may be compared against the sound of the old piano at the start of the film; this is how the later old piano section originally sounded before convolution. This recording also gave the old piano a somewhat harsh, mechanical sound, but this was actually a positive point for the film. The piano is intended to sound bad and annoying at that particular point; a harsh reality in contrast with the clear, tuneful music the girl imagines. 4.4 Potential in film Convolution is a powerful means of placing any sound into any location, in a more realistic way than has been possible in the past. Convolution can be processed in real time on today's hardware, and long audio files can often be convolved with impulse responses in less time than it would take to play back the file. Take for instance recording on location in a large hall, but bad background noise or dialogue mistakes during recording mean re-recording needs to be done later in a studio37. It can be hard to match a studio recorded voice or sound with the sounds recorded on location. Often the dubbed recording sounds dry and fake. With convolution, an impulse response of the recording location could be taken during a period with low background noise (e.g. at night). This will require a little extra time, but a simple impulse from something like a gunshot is easy to obtain. 35 Dry meaning recorded in a way that picks up no reverb from the room, only the direct sound of the instrument. 36 From 1:24 to 1:49 37 Called ADR Automated Dialogue Replacement.

36 36 This impulse response could then potentially be used on any recording done in the studio to add in the sound of the location perfectly fitting any studio recorded sound into the location. Of course, a good convolution recording with little or no background noise is needed in the first place: Re-recording is often done because the original location is too noisy. If the impulse response contains the same noise it would clearly be undesirable to add it back in. Libraries of impulse responses are already available, although these are often tied to proprietary file formats which can only be run through a certain audio plug-in. Libraries are often sold by the creators of the associated software, recorded by themselves and kept closed format to boost sales. If all you need is a direct convolution adding the sound of a location to another sound something as simple as Convy would suffice. Plug-ins may, however, provide a lot more control over the sound output, and the mix of original versus convolved audio. Libraries of impulse responses can be sold in a similar way to how one may purchase a set of Foley sounds. Dry sounds can then be recorded in the studio and added to any location without having to record on-site. Since the process of capturing and recreating any location is fairly simple, convolution can be a simple, realistic way to add ambience to studio sound effects without the effort of rerecording on location.

37 37 5 Discussion As of the writing of this report The Paintbrush is not yet complete, but is well on the way to completion. The original aims of the audio for the film have essentially been achieved. Use of quadraphonic recording for the sound effects is a successful improvement on standard mono recordings, though minor editing of volume levels will sometimes be necessary. There is extra work involved in this type of recording so time will be a factor in deciding whether it is worthwhile. The combination of sound effects with each other, and the soundtrack which plays at some level throughout, reduces the immersive effect that the recordings have when played on their own. This is unavoidable but can be reduced with careful placement of sounds, equalisation, and an overall sparse soundscape. In terms of using the Zoom H2 recorder, the device tended to pick up a lot of sound with its rear microphones, sometimes making sounds coming from the front appear to be coming from behind if the listener was nearer the rear speakers. Much of the sound effects have the speakers in the direction of the sound source turned slightly up and those opposite turned slightly down, to minimise this effect. It should be noted when a sound is coming from a certain location, there is still much more sound in the opposite speakers than standard film sound effects would usually have often these effects would come only through the speakers in the direction of the sound source. The music soundtrack sounds essentially as originally envisioned, though the benefit of instruments recorded non-traditionally (in quadraphonic) is minor. The 5.1 mix has a significant audible improvement over the stereo mix. The use of Bidule for signal routing demonstrates a versatile emerging new technology. In terms of using the Røde microphone array, spatial positioning was as good as using the Zoom H2, with clarity and accuracy of sound reproduction being slightly better than the H2. The use of convolution and the development of Convy showcases some possible applications for convolution in film. More control than Convy provides over convolution processing is needed to allow convolution to sound pleasing in all situations (for instance, equaliser and dry/wet control). Time management was an issue. Creating the audio as the film developed worked fairly well, but the musical score would have been easier to develop with a traditional schedule where work on the soundtrack could be started after the film was largely complete.

38 38 There were no significant delays in audio development caused by the audio work itself; the only minor delay coming from finding appropriate software to run string instrument samples. Final mixing and mastering of the soundtrack continued for a long time as the music was checked and re-checked on different systems to achieve the best possible sound, but sound effects could be developed concurrently. Delays in video completion caused delays in audio completion. These were caused mainly by limited rendering resources for the 3D scenes causing excessively long render times (sometimes days), and a greater than expected workload on the video aspect of the film. Audio is currently up-to-date in terms of completed work, so is on schedule to hopefully be finished in unison with the video content. The project also brought to light failings with surround sound in film in general. Lack of standardisation between formats, and even within formats is an issue. As Blackburn (2000) points out, a Dolby sticker does not guarantee correct Dolby decoding, at least in the realm of bass management. Different recommended angles for speaker placement, and disparity between speaker angles and microphone angles on recorders create audio inconsistencies and confusion as to how a mix should be done. The LFE channel is widely misunderstood as being the subwoofer channel; I shared a similar misconception at the outset of this project. Correct usage of the LFE channel is in fact fairly simple and logical, although the Pro Tools 5.1 panner has some idiosyncrasies which can confuse usage even further. Namely, using a 5.1 output on a mono track will still allow one to send audio to any single speaker, except in the case of the LFE. One may not add to the LFE channel and also cut from the remaining speakers. This could potentially cause issues if bass management later sends bass from the main channels to the subwoofer, where bass will then be doubled up. Alternatively, if the studio has small speakers whereas the consumer has large speakers, more bass will emerge from their main speakers as well as that being sent to their subwoofer through the LFE channel. To send to the LFE channel only, the output of a channel must be set only to LFE. The LFE channel is also overused, especially considering it is discarded in some formats. Blackburn (2000) himself is highly praising of films with high use of the LFE channel, the complaint being that other films are lacking in impressive bass. This should not be necessary with correct bass management, use of the LFE channel has nothing to do with the overall level of low frequencies. Elen's analysis (2002) provides a clear explanation. Massey University's system also has some unresolved issues. The main speakers need bass management, either through additional software or external hardware.38 Currently there are four identical subwoofers and identical mid-size speakers. The best possible configuration for 5.1 music mixing at this time is four full-range 38 See footnote 23.

39 39 channels with a speaker and subwoofer, a centre channel without subwoofer, and the LFE channel discarded completely. The best configuration for film may involve taking the two rear-channel subwoofers and using them for the LFE channel, leaving only two out of five channels with fullrange sound as desired. The Logitech system needs signal conversion to allow connection to the Digidesign 003. A conversion between the differing S/PDIF formats may not be possible. A conversion between +4dBu and -10dBV is easily possible but would require conversion of all six channels independently. The Logitech system may bypass the 003 entirely and connect to a sound card on the main computer, although the on-board sound only allows stereo output, so an additional card (internal or external) would be required. However, this would not be a feasible solution as only the 003 may be used for audio output directly from Pro Tools. The current system of using the direct inputs and simply turning the volume down considerably may actually be the best long-term solution. The rattle of the light fixtures should cease to be an issue once the surround room is moved to its planned new location.

40 40 6 Conclusions and Recommendations The recording and mixing style of The Paintbrush has been largely successful and I would recommend further experimentation with surround recordings in film. Seating in cinemas is a worry for those close to the side. Placing some front audio into the rear speakers, and vice versa, may help to alleviate issues with the proximity of different sources, but this would also reduce the realism of the sound somewhat. Testing of audio recorded in surround in a real cinema is needed. I believe convolution can be a useful addition to film sound, and the project has explored some useful applications. Ideally, Massey needs a system for bass management either in software, or a hardware box that fits in after the output of the mixing console and sends audio signals to the subwoofer(s) appropriately.39 The small Logitech system has this built in, so it manages bass appropriately straight out of Pro Tools. However, it has its own connection issue described earlier. Surround sound standards as proposed by format creators such as Dolby need to be read and adopted correctly, by both equipment manufacturers and audio engineers. Standards largely already exist, but these need to be adopted correctly to provide a guaranteed listening experience to all end users. Manufacturers need to make sure their receivers conform to specifications, while mixing engineers need to be aware of how these receivers will process their audio, and how to mix appropriately. Utilisation of surround sound systems can be improved in both music and film. 5.1 music mixes are beginning to fully take advantage of all channels, but film retains its front-centric approach. Although testing in a cinema environment is needed, The Paintbrush shows that better use of surround channels is possible. Convolution may have many practical uses in film, particularly in areas such as dialogue replacement. Altering the perceived location of a sound, such as having a sound appear to come from another room, also worked well when tested in The Paintbrush. However, convolution is useful as assistance for room sound and effects alongside traditional equalisation and reverb. It is not a be all and end all solution. Whenever convolution was used in The Paintbrush, some modification or addition to the sound was needed. I believe the original timeline for audio work on the film was reasonable, had the video work also been completed on schedule. The video work appears to have a greater total workload than the audio work, so with one person working on each in a project, the audio creator needs to assist the video creator as much as possible. Assuming the video work was on schedule, it would have been a tight but achievable deadline to complete all audio for the film to a high standard by the time of the final presentation. The Paintbrush has several scenes with few or no sound effects required, which makes the work on sound effects much faster, and the mix simpler. As of the final presentation on 14 October 2009, the film is on schedule for completion by mid-november. 39 See footnote 23.

41 41 7 References Bass management in Pro Tools. (2007, November 13). Retrieved September 20, 2009 from Blackburn, D. (2000). How important is the LFE subwoofer to home theater? Retrieved September 21, 2009 from Dolby (2005a). 5.1-Channel music production guidelines. Retrieved September 17, 2009 from Dolby (n.d.). Dolby digital guidelines. Retrieved September 17, 2009 from Dolby (2005b). Dolby surround mixing manual. Retrieved September 17, 2009 from Dolby (1999). Surround sound: Past, present and future. San Francisco, CA: Dolby Laboratories. Elen, R. (2001). Ambisonics: the surround alternative. Retrieved September 19, 2009 from Elen, R. (2002). All bass is covered: A dictatorial approach to bass management in surround music mixing. Retrieved September 19, 2009 from Holman, T. (2000). 5.1 surround sound: Up and running. Oxford: Focal Press. Intellexual.net (2004). BOSE Acoustimass Better Profits Through Marketing. Retrieved September from Kemmler, A. (2006). Acting on impulse. Retrieved September 23, 2009 from Meyer, E., & Moran, D. (2007). Audibility of a CD-standard A/DA/A loop inserted into highresolution audio playback. JAES, 55(9), Oohashi, T., Nishina, E., Honda, M., Yonekura, Y. Fuwamoto, Y., Kawai, N., Maekawa, T., Nakamura, S., Fukuyama, H., & Shibasaki H. (2000). Inaudible high-frequency sounds affect brain activity: Hypersonic effect. The Journal of Neurophysiology, 83(6), Pohlmann, K. (2009). Canton GLE Series home theater speaker system. Retrieved September from Taylor, J., Crawford, C., Zink, M., & Armbrust, C. (2008). Blu-ray disc demystified. New York: McGraw-Hill Professional.

42 42 8 Appendices 8.1 Appendix A: Included resources Two DVD-R discs are intended to be included with this report. DVD A contains a copy of the film The Paintbrush, playable in any standard DVD player. Audio is encoded in 5.1. Note that the visual content on the disc is incomplete. DVD B is a data disc for use on a computer with DVD drive, and has the following content: soundtrack5-1.flac - the complete soundtrack (music only) in 5.1 channels, in lossless FLAC compressed format. soundtrack2-0.mp3 the complete soundtrack (music only) in stereo (2.0), in VBR -V 2 (formerly known as 'alt-preset standard') lossy mp3 compressed format, encoded using LAME. Convy this folder contains the MATLAB source code to run the convolution processor, Convy. See section 4.2, Convy a convolution processor in MATLAB. Music PT the folder contains a Pro Tools session with the 5.1 mix of all music for The Paintbrush. All Audio PT the folder contains a Pro Tools session with a bounced down 5.1 mix of music for the film, and a mix of all other audio for The Paintbrush.

43 Appendix B: High resolution images Screenshots Waveforms Figure B1.0 Final soundtrack waveform before volume compression (L, C, R, Lfe, Ls, Rs). Figure B1.1. Final 5.1 soundtrack waveform after volume compression/limiting/boost of overall volume (L, C, R, Lfe, Ls, Rs).

44 44 Figure B1.2. Final 2.0 soundtrack downmix (L, R). Figure B 1.3 An earlier version of the soundtrack. The actual volume of everything in the mix is very similar, but note how the Lfe channel is used more. There is less bass in the main channels than in the final mix. Figure B1.4 A large portion of the final 5.1 mix of all audio with sound effects and music. L, R, C, Ls, Rs Lfe channel not shown.

45 45 Software Figure B2.1 Convy. Figure B2.2 Small Convy code sample showing deconvolution.

46 46 Figure B2.3 Pro Tools 7.4 edit window during early stereo mixing. Note orange tracks with MIDI information for string instruments.

47 47 Figure B2.4 Pro Tools 8.0 edit window showing most tracks in final soundtrack (music, no SFX).