Interactive Music Instruction with Java Objects Dr. Paul E. Dworak College of Music, University of North Texas, P. O. Box 311367, Denton, TX 76203 Tel: 940-565-4906, Fax: 940-565-2002, E-mail:dworak@music.unt.edu Abstract: The Java Music Toolkit is a set of Java classes that can display and play musical scores. The classes are used in music courseware in which students hear problem sets and notate their solutions to the examples that they hear. Students are able to compare their solutions with correct answers to enhance their understanding of the musical thought processes involved. Courseware using applets based on these classes will run on multiple platforms and can be served over the internet. This provides students with a larger variety of interactive instructional programs and facilitates the offering of courses delivered by distance learning. 1. Introduction When microcomputers became available for computer based instruction in the early 1980s, courseware authors were able to develop instructional materials for students in their college or university. With the current interest in distance learning, these same instructors need tools that enable them to provide the same interactive instruction remotely. This paper describes a set of Java classes that support applets for music computer based instruction. These classes enable students at any location to complete drills served from remote hosts, with the same interactivity that is possible if they were at a microcomputer at the host institution. Serving instructional materials developed in Java over the internet has several indisputable advantages: 1. It enables students with different platforms to use the same materials, since the student's web browser will interpret the Java code and control presentation of the materials. 2. It enables students in many locations to take advantage of pedagogical methods developed and used at other colleges and universities. 2. Java for Music Instruction 2.1 Advantages of the Java Music Toolkit The Java Music Toolkit is a set of classes that reads compressed score files and converts them both to a graphical image and to a MIDI file that can be played. These files are very small a set of five harmonic dictation exercises, each four measures long, fits in less than 4 KB. Consequently, an entire drill with all its component exercises can be served in one step. The client will decode each score in the drill as the student needs it. If score images were served as gif files, they each could be 50KB to 500KB in size. MIDI files used to play scores typically range in size from 20KB to 150KB. The toolkit thus provides significant size and speed advantages over traditional means of transporting musical images and sound files. The toolkit also enables the student to create scores that are solutions to dictation exercises. The classes that display scores are the same ones that enable the student to create a new score. The student could not interact in the same way with drills using images prepared with a music notation editor. 2.2 Classes in the Java Music Toolkit The Java Music Toolkit is divided into five groups of classes: 1. Score definitions: constants that represent types of staves, ties, notes, etc. 2. Score types: data structures for notes, chords, voices, etc. 3. Score drawing: the classes that draw images, staves, clefs, notes, etc. 4. Score canvas: the graphical object onto which items are drawn
5. Score data file: the file that stores the binary representation of scores in a drill The total size of all these classes, uncompressed, is less than 200 KB. Ideally, users of the Java Music Toolkit would download and install the classes before accessing any applet that used them. 2.3 Structure of the Score Data File Scores are arrays of data structures, organized hierarchically. Each score is an array of systems. Each system is an array of staves. The array of data associated with each staff its clef, key, meter, etc describe the score s background elements, which are those that change little or not at all. Also associated with each system is an array of voices. The voice data notes, rests, chord symbols, etc. define the score s foreground elements. These are mapped to staves of the score, and they define the melodies and harmonies that characterize the score. Each item in this data structure has a class associated with it. The class knows how to read the data it needs from the score file and to convert the information into drawing and MIDI commands. 2.4 Displaying a Score Each element of the score s foreground and background has a timepoint in the score. This timepoint is associated with its physical location, or placepoint, on the screen. The placepoint of any timepoint will vary, depending on the contents of the score. A note preceded by an accidental requires more space on the screen than does a note without an accidental. When chords are drawn, notes forming the interval of a second must be offset from one another, and accidentals also must be offset if they occur within the interval of the sixth. Consequently, the ScoreCanvas class must calculate the placepoint of each element of a score with respect to the context in which it occurs. To prepare for drawing the score, ScoreCanvas creates a list of ScoreEvents, which associates the timepoints and placepoints of each element of a score. Each score event also has a type, which ScoreCanvas uses to draw the element. To draw background elements, ScoreCanvas sends a draw message to a Staff object, identifying the type of element (clef, key signature, or meter) that this object should draw on the staff. To draw foreground elements, ScoreCanvas sends a draw message to a Voice object, requesting that it draw beats on the staff at the appropriate locations. Drawing the score background entails iterating the list of systems and staves that constitute the background and determining the physical location of each on the screen. DrawStaffData then invokes the appropriate drawing routine. ScoreCanvas has the ability to transpose the key signature of a score, so that one model score can be used to create several different examples, each in a different key. To draw the score foreground, ScoreCanvas initializes voice data based on the timepoint of the current score event. After calculating note offsets and accidental offsets for chords, and aligning notes vertically, it then invokes DrawVoiceData. This method draws the score foreground by iterating the list of systems and voices that constitute the foreground. If a voice contains a beat at the timepoint specified by the score event, ScoreCanvas asks the voice to draw its beat. If not, the voice determines if its last sounding note needs a beam or tie, and if so, it draws the appropriate shape. By segmenting the score into a series of score events, ScoreCanvas is able to draw the score one beat at a time. The following example shows a harmonic dictation example drawn by the Java Music Toolkit classes. Figure 1: A harmonic dictation example.
Interactive Music Instruction with Java Objects Paul E. Dworak College of Music University of North Texas Advantages of Java Applications can run on a variety of platforms--windows 95, Macintosh, and UNIX Students can access class materials from their homes or from a remote computer lab Students can access pedagogical materials used at Colleges and Universities other than their own 1
Other Notation Applications Finale can produce TIF files that can be converted to GIF files fine graphic resolution requires large files score notation is not inherently interactive Sibelius can produce notation that can be posted to a web page the image is scrollable the notation is not inherently interactive Status of Java Sound Support MIDI sound is not supported in JDK1.1 JDK1.2 beta and Java Media Framework can read MIDI files play them through software synthesis rather than through local MIDI hardware Forthcoming JavaSound API will handle MIDI events Staccato writes to the local MIDI hardware it is not yet available 2
Advantages of Java Music Toolkit An entire drill with multiple scores can be served in one file that is less than 5 KB The local client will decode this file into graphical images and MIDI sounds The student can create scores interactively that represent the solutions for dictation drills Java Music Toolkit Classes Score Definitions--constants Score Types--fundamental objects Score Drawing--classes that draw images Score Canvas--graphical object on which drawing occurs Score File--binary score representation 3
Structure of Scores Array of Scores Array of Systems Array of Staves Array of Voices Array of Staff Data Array of Voice Data Clefs, Keys, etc. Notes, Rests, etc. BeatData public class BeatData extends PackedBeatData { public byte Stem; // The type of stem // 0 down, 1 up, 2 default, 3 none public boolean BeatGiven; // TRUE is beat is given public boolean NoteGiven; // TRUE if Note or Chord is given, // irrelevant if a rest public byte Tie; // The type of tie // 0 = none, 1 = tie, 3 = flipped tie public byte Kind; // Type of beat // 0 = note, 1 = chord analysis, // 2 = visible rest, 3 = invisible rest public byte SystemNumber; // System on which the beat is to be // displayed. public byte StaffNumber; // Staff on which the beat is to be // displayed. } 4
NoteData public class NoteData extends ScoreData { public byte NoteHead; // The type of notehead for durations // 0 = beat's value, 2 = black, 3 = white public byte NameClass; // Number of the note name class. // F C G D A E B // 0 1 2 3 4 5 6 public byte AccdClass; // Note's accidental class: // bb b n # x // 0 1 2 3 4 public byte Octave; // Note's octave -- middle C is octave 5. public byte NHT; // The nonharmonic tone: // 0 = None, 1 = UPT, 2 = UN, 3 = LN // 4 = APT, 6 = Susp, 7 = Ret, // 8 = Ant, 9 = ET, 10 = CT } Drawing a Score Each Timepoint in a score is associated with a Placepoint in a ScoreEvent The amount of space allocated to notes, accidentals, rests, etc. are calculated automatically Displaying a score is accomplished by iterating the list of ScoreEvents 5
Drawing a Score public void DrawCurrentEvents() throws IOException { ScoreEvent ascoreevent; } for (Enumeration seiter = ScoreEventList.elements(); seiter.hasmoreelements(); ) { ascoreevent = (ScoreEvent) seiter.nextelement(); if (ascoreevent.geteventtype() < Opcode.CLEF) DrawSystems(aScoreEvent); else if (ascoreevent.geteventtype() < Opcode.BEAT) DrawScoreBackground(aScoreEvent, Color.black); else DrawScoreForeground(aScoreEvent, Color.black); } Drawing the Score s Background Display staff key signature meter These are elements that do not change often 6
Drawing a Score s Foreground Display notes rests roman numerals These score elements change every beat and give the score its unique sound Drawing Score Foreground public void DrawScoreForeground(ScoreEvent ascoreevent, Color color) throws IOException { SetVoiceData(aScoreEvent); SetNoteOffsets(aScoreEvent); SetAccidentalOffsets(aScoreEvent); DrawVoiceData(aScoreEvent, color); } 7
8
9
10
11