Audio-to-MIDI Converter Real-Time Note Detection Software for Electric Guitars

Final Report Audio-to-MIDI Converter Real-Time Note Detection Software for Electric Guitars ECE4007 Senior Design Project Section L04, Team Obama Jon Kump Bryan Tyson Sean Early Andy Burkic Submitted May 6, 2010

Table of Contents Executive Summary............ 3 1. Introduction........... 4 1.1 Objective.. 4 1.2 Motivation.... 5 1.3 Background.. 6 1.3.1 The MIDI Protocol.... 6 1.3.2 MIDI Guitar Pickups.... 6 1.3.3 Existing Software Products........ 7 2. Project Description and Goals......... 7 3. Technical Specifications...... 8 3.1 Functional Specifications.... 8 3.2 Interface Specifications...... 8 4. Design Approach and Details....... 9 4.1 Design Approach........., 9 4.2 Codes and Standards....... 12 5. Schedule, Tasks, and Milestones.. 12 6. Results and Acceptance Testing.... 13 7. Budget and Cost Analysis..... 16 7.1 Marketing Analysis......... 16 7.2 Cost Analysis..... 16 8. Conclusion and Future Work 17 9. References.... 19 Appendix A..... 20 2

Executive Summary Musicians and producers are always looking for cheaper and more reliable products to create music. Virtual Studio Technology (VST) is an interface for integrating software plug-ins in host applications that support the VST format, eliminating the need for a pickup to convert the analog audio from an electric guitar to MIDI. The analog-to-midi converter is a VST software plug-in that converts audio to MIDI in real-time to drive a synthesizer or any other MIDI device. The fundamental goal of the audio-to-midi converter is to give the most accurate and realistic representation of the notes played by a guitar. Inside the VST plug-in, an AMDF algorithm analyzes the monophonic input waveform by its pitch to determine which notes have been played. Once a note has been created, it is converted to MIDI instructions and sent to a synthesizer where it can be modified in real-time to the user s preference. This means that the plug-in will only be able to detect one pitch at a time. The latency from the acceptance testing has been determined to be 23.8 ms on average which makes the delay between input and output unnoticeable. Additionally, the note duration was reduced to 190 ms to allow a guitarist to play up to 300 beats per minute. A completed project was provided for $31,357.70, with a total equipment cost of $397.70 and an estimated labor cost of $30,960. To make the product more marketable, the following improvements are suggested: smaller latency, optimized pitch bend, compatibility with other instruments as well as support for non-standard tunings. 3

1. Introduction As technology has progressed, the latency to convert an audio signal to MIDI has improved to the point that even guitarists can take advantage of it. Pickups attach to electric guitars and convert the notes being played to MIDI instructions. The converted file can be used to control a synthesizer to produce a sound or be modified as desired. Team Obama has designed a real-time, analog-to-midi converter that changes the audio signal produced by an electric guitar to a MIDI signal without using any hardware. The music industry, especially producers and artists, can now avoid the expense of purchasing pickups for their instruments. The software can also benefit musicians who are looking to add new effects to guitar recordings and live playing. The analog-to-midi converter is a software plug-in for digital audio workstations (DAW) such as Logic, which is a host application that communicates with the sound card of a computer. Labor was the main cost for developing the product since most necessary equipment and software was already available to begin the design. Team Obama is looking to make the product available for purchase for approximately $50. 1.1 Objective Virtual Studio Technology (VST) is an interface for integrating software plug-ins in host applications that support the VST format. It was used to eliminate the need for hardware to convert the analog sound from an electric guitar to MIDI. The audio-to-midi converter is a VST software plug-in that converts the analog signal from an electric guitar to MIDI to drive a synthesizer or any other MIDI device in real-time. 4

When a guitar string is struck and the analog audio reaches the sound card, it is converted to waveform audio (.wav). The.wav file is sent to the host application where it can be accessed by the VST software plug-in. Inside the plug-in, an algorithm was implemented to analyze the input waveform by its pitch to determine which notes have been played. Once these notes are created, they are converted to MIDI instructions and sent to a synthesizer where they can be modified in real-time to the user s preference. The texture of the audio is monophonic which means that the plug-in is only able to detect one pitch or note at a time. Only one input and output channel can be used because the guitar produces a mono signal. The latency is an average of 23.8 ms which makes the delay between input and output virtually unnoticeable to the human ear. The VST plug-in can detect the pitch of the entire range of the guitar, which is four octaves. 1.2 Motivation Currently, guitars are recorded through a sound card or an amp. The recording does not give the artist any way to manipulate or edit the sound. MIDI guitar pickups were created, but have become expensive and time consuming. Team Obama s real-time analog-to-midi converter can be used as a recording tool for producers to incorporate a human element to MIDI sequences. With its multiple features, which can be easily accessed through a simple virtual user interface, the musician has more options on modifying or changing the sound output. The plug-in can create an increased dramatic effect for guitarist in live productions by allowing them to edit the sound produced in real-time or later. By eliminating the need for a guitar pickup, customers avoid the cost of hardware but are still able to add human-like effects, such as pitch bending, to the MIDI recordings. 5

1.3 Background 1.3.1 The MIDI Protocol Musical Instrument Digital Interface, or MIDI, is a protocol that enables electronic musical instruments to exchange system data. Devices such as synthesizers, computers, electric drums, and sound cards have the ability to communicate, synchronize, and control one another [5]. MIDI should not be mistaken for an audio signal; rather, it is a text-based language for computers to control any sound-producing device and vice versa [1]. A MIDI cable sends instructions to a device, such as a keyboard, which in turn generates a particular sound. An advantage to the music industry is the ability to send a keyboard s data to the computer, record it as MIDI events, and then manipulate the composition with software such as Logic. It is easy for keyboards to communicate with computers via MIDI, because each key is a switch for a particular note. Communication becomes much more complex with stringed instruments. Each note must be analyzed, detected, and converted to MIDI, which requires more processing time than with a keyboard [2]. 1.3.2 MIDI Guitar Pickups The way MIDI operates has not changed since its introduction in the early 1980s. The basic measurements are note on, note off, and velocity. A guitar pickup uses individual sensors for each string to analyze and convert the note being played to MIDI. An algorithm takes the input signal of the guitar and analyzes the spectral components of the note using digital signal processing [3]. Once the note is recognized, it is converted to MIDI and stored on a computer so that it can be modified later by the artist. Guitarists utilize pickups to record their own music and edit it to their liking. Brands such as Axon, Roland, and Fender sell MIDI pickups in the ranges of $100 - $300, but they also offer electric MIDI guitars for several thousands of dollars [4]. 6

1.3.3 Existing Software Products Team Obama has eliminated the need for hardware and created a note detection process similar to the MIDI pickup using only software and the soundcard from a computer. The key building blocks are the VST architecture, a pitch recognition algorithm, and a MIDI output in real-time. Celemony s Melodyne studio software is currently available and performs similar functions to a real-time audio-to-midi converter. It sells for $699 compared to another product, Widisoft, which sells for $200 [9]. Widisoft has the ability to convert audio to MIDI in real-time as well; however, it is not widely used [10]. Team Obama s VST plug-in should be able to reduce the price of such software down to $50. 2. Project Description and Goals The fundamental goal of the audio-to-midi converter is to give the most accurate and realistic representation of the notes played by a guitar. It allows a user to play a guitar and edit the notes using a MIDI sequencer, giving the musician the ability to both visualize and manipulate the notes being played in real-time. The product was created through a VST software development kit called SynthMaker, which is available online for purchase [6]. The following features are incorporated in the project: Real-time VST software plug-in Pitch recognition by algorithm Output of detected notes as MIDI instructions Real-time control of a synthesizer Low latency for real-time performance Pitch bend Targeted audiences: music producers and artists Target price: $50 7

3. Technical Specifications 3.1 Functional Specifications Table 1 lists the functions available to the VST plug-in user. The texture of the audio is monophonic, meaning that the plug-in is only able to detect one pitch at a time. The input sampling rate is 44.1 khz, the standard sampling rate for audio CDs and most audio applications. The plug-in is able to detect the pitch of the entire range of the guitar, which is four octaves. The minimum note duration is 190 ms, allowing the guitarist to play at speeds close to 300 beats per minute. Table 1 Functional Specifications Desired Actual Input Format Analog Analog Output Format MIDI MIDI Sampling Rate 44.1 khz 44.1 khz Texture Monophonic Monophonic Channels 1 1 Latency < 20 ms 23.8 ms Note Range 4 Octaves 4 Octaves Min Note < 200 ms 190 ms Duration Texture Monophonic Monophonic 3.2 Interface Specifications The analog-to-midi converter was written as a VST plug-in to allow the software to be used with any host program that supports the VST format and MIDI events. Table 2 shows the specifications for the VST plug-in interface. 8

Table 2 Interface Specifications Format Interface VST Plug-in 3 Buttons 2 Knobs 2 Drop-down Lists 1 Level Meter 1 Reset Button 4. Design Approach and Details 4.1 Design Details An electric guitar is connected to the sound card of a PC and the audio signal is recorded and converted to a.wav file by preexisting software such as Logic. Once the.wav file is created, the audio-to-midi converter uses its pitch detection algorithm to analyze the waveform by its pitch to determine which notes have been played. As the notes are detected, they are converted to MIDI instructions and sent to a synthesizer where they can be modified in real-time, depending on the user s preference. Following below is a description of each control on the interface. Figure 1 shows a screen shot of the GUI created. Pitchbend Button: Enables and disables pitchbend feature Passthrough Button: Enables and disables guitar audio passthrough feature. Allows the raw guitar to be played over the speakers as well as the MIDI controlled synthesizer audio Articulation Button: Changes the articulation mode between legato and staccato Guitar Drop-down List: Allows user to select between guitar and bass guitar Octave Drop-down List: Selects desired octave shift level of the MIDI output Sensitivity Knob: Controls note detection sensitivity Preamp Knob: Adjusts volume level of input signal Reset Button: Sends MIDI silence to reset stuck synthesizer notes 9

Figure 1. Guitar2MIDI GUI interfaces. The plug-in was written in VST format. Any driver level interaction with the sound card or MIDI controller is handled by the host program. A top level block diagram of the plug-in is shown in Figure 2. The plug-in goes through four basic steps, outlined below. Figure 2. Design block diagram. After staging the input to the appropriate volume, the first step is to identify the pitch of the note being played. An Average Magnitude Difference Function (AMDF) pitch detection 10

algorithm is used to perform that function. AMDF works by delaying the input signal 48 times; one time for each note on the guitar. The delay for each note is equal to one half the period of that note. The original input is then added to each delayed input. The summed signal with the lowest average magnitude will be the matching pitch. After the pitch is detected, note starts and note stops need to be detected. It was performed in two separate ways, depending on the setting of the articulation button. In the first method, changes in pitch were detected to create note start and note stop events, producing a smooth legato sound. The second method uses envelope detection to find when the volume of the note went above and below a sound threshold. This method created a crisp staccato sound. The third phase is to create a MIDI output signal which can be used by the host. Once the start of a note and its pitch are detected, the note start signal will be sent to the MIDI interface, signaling the start of the note at the appropriate pitch. The software will have to detect when the note stops playing and send the note stop signal at the correct time. It is important for the MIDI signal to function in real-time to ensure accurate audio-to-midi conversion. Finally, if the pitchbend feature is enabled, pitchbend information needs to be sent to the MIDI output. Pitchbend is calculated using a method called zero-crossing. Zero-crossing is when a periodic signal crosses the zero line. This method calculates the average time difference between every zero crossing of our input guitar signal. If the average time in decreasing the pitch will start bending up, and if the average time is increasing the pitch will start bending down. This pitchbend information is then sent out along with our MIDI event. 11

4.2 Codes and Standards The software is a Virtual Studio Technology (VST) plug-in written using the SynthMaker IDE. By using the VST format, the plug-in can be used with any number of host programs that support the technology, as well as eliminating the need to configure the hardware interfaces specifically for the plug-in [6]. Using the SynthMaker IDE enabled rapid development due to automated completion of common tasks, as well as simplified debugging due to the nature of its graphical programming environment [6] The plug-in will output to a MIDI interface, which is the industry standard for use with electronic instruments. 5. Schedule, Tasks, and Milestones The project is divided into four subgroups for each building block of the project. Each building block is indicated in bold font and serves as the stepping stones towards the final product. The tasks below each milestone are in italics and will be delegated to team members. Table 3 shows the layout and timeline for how the project was designed and implemented. Table 3 Project Schedule, Tasks, and Milestones Task Name Duration Start Date End Date Difficulty Degree Proposal 2/4/2010 2/10/2010 Construct framework 4 days 2/4/2010 2/9/2010 Low Tyson Review with Professor 1 day 2/9/2010 2/9/2010 Low Burkic Turn in Proposal 1 day 2/10/2010 2/10/2010 Milestone Group VST Component 2/10/2010 4/7/2010 Research and upload Software Development Kit 6 days 2/10/2010 2/17/2010 Low Kump Develop algorithm for Pitch detection 36 days 2/17/2010 4/7/2010 High Earley Successfully create and use Pitch Detection 0 days 4/7/2010 4/7/2010 Milestone Group Complete AMDF Algorithm 15 days 2/24/2010 3/16/2010 High Earley Create GUI Interface 31 days 2/24/2010 4/7/2010 Medium Burkic PDR Presentation 3/10/2010 3/16/2010 Construct Framework 4 days 3/10/2010 3/15/2010 Low Tyson Rehearse presentation 2 days 3/15/2010 3/16/2010 Low Burkic Demonstrate presentation 0 days 3/16/2010 3/16/2010 Milestone Group Midi Generation 3/29/2010 4/21/2010 Research various sources for converting Pitch to USB Midi 8 days 3/29/2010 4/7/2010 Medium Burkic Develop process to output Midi note 18 days 3/29/2010 4/21/2010 High Earley Successfully output MIDI note 0 days 4/21/2010 4/21/2010 Milestone Group Add Pitch Bend feature 25 days 3/29/2010 4/23/2010 High Kump Responsible Person Final Presentation 0 days 4/27/2010 4/27/2010 Milestone Group Demonstration 0 days 4/27/2010 4/27/2010 Milestone Group 12

At the conclusion of the project, the VST plug-in and AMDF algorithm work in conjunction to create the correct pitch recognition through the analog-to-midi conversion. The AMDF algorithm was created using the program SynthMaker. Table 3 provides a breakdown of the project and each member responsibilities, as well as a degree level indicating the level of difficulty for each task. The design began on February 4 and concluded on April 27 for a demonstration. A Gantt Chart is provided in Appendix A. 6. Results and Acceptance Testing In order to verify the audio-to-midi converter would perform as planned, two criterions were used for acceptance testing. The first criterion describes the amount of time between the note being played by a musician and the note heard through the synthesizer, or latency. This parameter was targeted for 20 ms because at any time larger than 20ms, the human ear would be able to detect a difference when the guitar is played and when it outputs from the synthesizer. In order to measure the latency, the host program was used to measure the audio waveform from the guitar and the synthesizer waveform. The musician would play a note on the guitar, while the host program tracks the wavelength over time. Once the note is passed through the VST to the synthesizer, another waveform would be plotted on the same graph as the original waveform. Both waveforms can be seen on the host program where the difference between the start times for each wave was measured. Figure 3 shows the audio waveform on top and the synthesizer waveform at the bottom. The red lines represent the start time for each waveform and the small boxes at the bottom each measure 20 ms. After taking multiple measurements, the average latency proved to be 23.8 ms. Therefore our percent error was 16%. This clearly did not meet the intended goal of 20 ms, 13

Figure 3. Latency measurement using host program however, the delay between the audio wave and synthesizer wave were hardly noticeable. Latency was more than likely to have been halted by the sound card used. By using two sound cards on the computer as opposed to one, the sound would have the ability to travel faster. The second criterion is pitch detection accuracy. Since a musician is attempting to mimic the exact note being played into a MIDI note, it is crucial that the notes being played match up with the correct pitch. The VST should be able to accurately recognize a pitch and then determine the exact note being played. The only measure of acceptance would be if the VST could recognize all notes on a single octave. Therefore as each note is played by a guitarist, the exact match would be emulated on the host program. To test this, Team Obama physically went through each note to verify the host program would recognize it. Figure 4 shows the synthesizer used with the 14

keyboard displayed. On the far left, the impressed key represents the note being played by the guitarist. Figure 4. Host program indicating note being played During the demonstration, the user can witness each note emulated on the keyboard of the host program. As the musician plays a note on the guitar, the equivalent would indicate an impression on the same note of the keyboard. While testing, Team Obama found no room for error since all the notes played in two different octaves were accurate. The plug-in was able to generate MIDI notes as the musician played on the guitar. The notes were displayed through the host program and were seen as actual MIDI events. To prove these notes were MIDI events and not simply an audio signal, the user is able to edit and manipulate the note in any way possible. Through the host program, the user can go to the MIDI notes generated and alter the length, change pitch, and arrange the notes in a different manner than the way they were originally played. 15

7. Budget and Cost Analysis 7.1 Marketing Analysis Although musicians can mount a MIDI pickup on an electric guitar to convert analog audio to MIDI, it is also possible to perform the same process using only software. By simply installing a VST plug-in to a host application, the user can obtain similar results as with a pickup. Melodyne studio software is a plug-in with the ability to convert monophonic analog audio to MIDI for the real-time control of synthesizers. The software sells for $699, which is too high for rising musicians or producers with small budgets. One VST plug-in on the market is the WIDI Recognition System 4.0 by Widisoft, selling for $199. It also performs a real-time conversion of monophonic audio to MIDI, but the lack of user reviews on the internet questions the credibility of the product. Team Obama s VST plug-in will also be able to convert monophonic analog audio to MIDI in real-time. Its advantage over the competition lies in a realistic, natural, and accurate representation of the notes played by an electric guitar. As another benefit, the product will be available for $50 on the market. 7.2 Cost Analysis The total cost to design the project will be $41,052.69. Necessary resources and equipment were purchased for $342.69 with a labor cost of $30,960. Table 4 lists the prices for the items that were obtained using a provided budget of $400. Table 4 Cost Analysis Product Description Quantity Unit Price SynthMaker Standard License [6] 1 Price $255.00 $199.99 C++ Algorithms for Digital Signal Processing Textbook [7] 1 $62.72 $62.72 SoundTech LightSnake Guitar/Keyboard USB 10 ft Cable [8] 2 $39.99 $79.98 Product Cost Total $397.70 16

Labor costs have been estimated to approximately $30 per hour for an entry-level electrical engineering position. Table 5 shows the required labor hours as well as cost for important project components. The labor hours have been estimated to include reports, meetings, and engineering development and implementation time. PROJECT COMPONENT Table 5 Project Cost LABOR HOURS LABOR COST EQUIPMENT COST TOTAL COST Create VST Guitar Tuner 210 $6,300.00 $279.97 $6,579.97 Design Plug-in Interface 64 $1,920.00 $0.00 $1,920.00 AMDF Algorithm 260 $7,800.00 $62.72 $7,862.72 Pitchbend and Features 176 $5,280.00 $0.00 $5,280.00 Analog-to-MIDI Conversion Output 112 $3,360.00 $0.00 $3,360.00 Class Lecture/Meetings 210 $6,300.00 $0.00 $6,300.00 TOTAL LABOR 1032 $30,960.00 TOTAL PARTS COST $397.70 PROJECT TOTAL $31,357.70 8. Conclusion and Future Work Team Obama was able to successfully reach all of the initial design goals for the project. AVST software plug-in has been created to convert analog audio from a guitar to MIDI instructions representing the notes that were played. The plug-in is compatible with any digital audio workstation that supports the VST format. A latency of only 23.8 ms ensures real-time audio processing which is necessary to create special effects to the sound without noticing a delay. The design also comes with several features that allow users to modify the way the sound is recorded and heard. 17

Although the project was a success, there are still some improvements that could be made to make the product more marketable. Below is a list of recommended changes for the future with brief explanations. Improved Latency: Bring latency below 20 ms for better performance Optimized Pitch Bend: More complex and efficient code necessary Increase Compatibility: Enable program to also function as a stand-alone application without relying on host application Non-standard Tunings: Create support for guitars that are tuned in non-standard conventions Other Instruments: Create support for other stringed instruments 18

9. References [1] D. M. Huber, The MIDI Manual: A Practical Guide to MIDI in the Project Studio, 3rd ed. Burlington, NC : Focal Press, 2007. [2] M. Schonbrun, What is MIDI? godinguitars.com, Nov. 16, 2008. [Online]. Available: http://www.godinguitars.com/whatismidi.htm. [Accessed: Jan. 25, 2010]. [3] P. White, Basic MIDI, Wiltshire, UK: Antony Rowe Limited, 2006. [4] J. P. Perry, "MIDI and Music," IEEE Transactions on Musical Instrument Communications, vol. 6, pp. 134-139, April 2005. [5] S. Motoyama, Real Time Communication of Musical Tone Information, U.S. Patent no. 7072362, Jul. 2006. [6] SynthMaker, Inc. SynthMaker, [Company Website], [cited 2010 May 5]. Available: http://synthmaker.co.uk/buy.html [7] Amazon, Inc. Kindle Edition, [Company Website], [cited 2010 Feb 9]. Available: http://www.amazon.com/gp/product/b001k6l7sc/ref=pd_lpo_k2_dp_sr_2?pf_rd_p=48653985 1&pf_rd_s=lpo-top-stripe-1&pf_rd_t=201&pf_rd_i= 0023606827&pf_rd_m= ATVPDKIKX0D ER&pf_rd_r=11V83X78N1Y4WF49VDRF. [8] Musician s Friend, Inc. 332600, [Company Website], [cited 2010 Feb 8]. Available: http://accessories.musiciansfriend.com/product/soundtech-usb-intelligent-instrument- Cable?sku=332600&src=3WFRWXX&ZYXSEM=0&CAWELAID=26022915 [9] Celemony, Inc. Melodyne Studio, [Company Website], [cited 2010 Feb 10]. Available: http://www.celemony.com/cms/index.php?id=product_comparison [10] Widisoft, Inc. WIDI Recognition System, [Company Website], [cited 2010 Feb 10]. Available: http://www.widisoft.com/english/mp3-midi-products.html 19

APPENDIX A PROJECT GANTT CHART See next page for project Gantt chart 20

21