TECHNOLOGY-ENHANCED LEARNING FOR MUSIC WITH I-MAESTRO FRAMEWORK AND TOOLS

Transcription

1 TECHNOLOGY-ENHANCED LEARNING FOR MUSIC WITH I-MAESTRO FRAMEWORK AND TOOLS ICSRiM - University of Leeds Shool of Computing & Shool of Musi Leeds LS2 9JT, UK kia@i-maestro.org Abstrat This paper presents a projet alled i-maestro ( whih develops interative multimedia environments for tehnology enhaned musi eduation. Guided by an analysis of pedagogial needs, the projet develops enabling tehnologies to support musi performane and theory training, inluding tools based on augmented instruments, gesture analysis, audio analysis and proessing, sore following, symboli musi representation, ooperative support and exerise generation for tuition, selflearning, and ollaborative work senarios. This paper briefly desribes the ontext and bakground of the projet, together with an overview of the framework and a number of different tools to support tehnology-enhaned musi learning and teahing. INTRODUCTION i-maestro [1, 18, 12] explores novel solutions for musi training in both theory and performane with a partiular fous on bowed string instruments. Musi performane is not simply to play the right note at the right time. Among the many hallenging aspets of musi eduation, we are partiularly interested in linking musi pratie and theory training, looking at interativity, expressivity and aessibility. New pedagogial approahes are being studied with interative ooperative and self-learning environments, and omputer-assisted tuition in lassrooms inluding gesture interfaes and augmented instruments. The projet is partiularly interested in linking musi pratie and theory training. i-maestro Guided by an analysis of pedagogial needs, the projet develops enabling tehnologies to support musi performane and theory training, inluding tools based on augmented instruments, gesture analysis, audio analysis and proessing, sore following, symboli musi representation, ooperative support and exerise generation for tuition, self-learning, and ollaborative work senarios. Key objetives of the projet inlude: Basi researh and development to support and enhane musi learning and teahing Exploration of new pedagogial approahes in musi eduation to improve aess to musial knowledge Enhanement of the onnetion between pratie and theory training 177

2 Creation of an interative multimedia environment with tools and servies for tehnology-enhaned musi eduation i-maestro offers a flexible, interative multimedia framework and supporting tools whih builds on reent innovations resulting from the development of omputer and information tehnologies. It offers pedagogi solutions and tools to maximise effiieny, motivation, and interests in the learning proesses and improve aessibility to musial knowledge. i-maestro omponents inlude: Prodution and authoring tools: for designing personalised musi exerises, lessons and ourses Client tools and appliations: with gesture analysis, augmented instruments, audio proessing, and sore following for interative performane and theory training Cooperative work support: for setting up ollaborative exerises on musi based appliations inluding ooperative work on MPEG Symboli Musi Representation Shool server: for sharing learning material at home and in the lassroom. The outomes are being validated by several European institutions inluding Aademia Nazionale di Santa Ceilia (Rome), the Fundaión Albéniz (Madrid) and IRCAM (Paris). FRAMEWORK AND TOOLS An overview of the framework is given in the diagram as presented in Figure 1. In addition, guidelines for aessibility in tehnology-enhaned musi training have been developed [2, 5]. This setion highlights several i-maestro tools that support different aspets of musi learning and teahing. Symboli Musi Representation Musi notation is one of the fundamentals in musi eduation. i-maestro is promoting MPEG Symboli Musi Representation (SMR), an ISO standard for the representation of musi notation with enhaned multimedia features [3, 4, 8, 9, 15]. Cooperative work is another key area of musi eduation. It allows different omponents of the i-maestro framework to be used aross a network. An MPEG SMR player/deoder, within the IM1 MPEG-4 referene software has been implemented (see Figure 2). 178

3 Sensors interfae I-MAESTRO DB aess Posture and Gesture Visual Rendering and interation Audio Rendering and aquisition Cooperative Support for Musi Training Aessible Interfae I-MAESTRO Musi Training Exerise Proessor Symboli Training proessing Tools Pratie Training Proessing Tools O and tools Musi Editing + Sore- Following I-MAESTRO Client Tool Assessment Models and Tools Audio Proessor: beat, pith, et. i-maestro Shool Server I-MAESTRO Prodution Tools Teaher, Assessment experts To other I- MAESTRO Client Tools Group of students Eduational Content, profiles, history and results individual students I-MAESTRO P2P and Cooperative Work Support Group of students Figure 1. An overview of the i-maestro arhiteture. Figure 2. An MPEG SMR player/deoder. Musi Training Supports The i-maestro Sound and Gesture Lab inludes advaned audio analysis, gestureand sore- following algorithms that provide feedbak and aompaniment allowing new kinds of musial interation (see Figure 3a). The Gesture Follower an trak a performed gesture in real time and ompare it with pre-reorded gestures for a variety of 179

4 pedagogial appliations (see Figure 3b). The Augmented Violin allows bowing gestures to be traked and studied (see Figure 3). (a) (b) () Figure 3. The Sore Follower listens to the player and provides automated page turning and aompaniment. The i-maestro 3D Augmented Mirror (AMIR) [11, 13, 14, 17, 18] aptures and visualises the performane in 3D. It offers a number of different analyses to support the teahing and learning of bowing tehnique and body posture. The tool provides interative multimodal feedbak, online and offline with visualisation and sonifiation [19]. Figure 5 shows the bowing traking with automated bowing annotation on SMR. Figure 4. AMIR for 3D visualisation and sonifiation of a bowing exerise. Using data from the extrated features and analyses we developed an approah to visualise the similarity of bowing features in order to failitate their study in relation to bowing tehnique and musial expression desriptors. We use the k-mean lustering method to ompute the mean of a partiular dataset in order to study the differenes in aptured data for a range of these qualities. For example by reording data for the same musial material played with different bowing styles (e.g. tenuto, staato, spiato et) and for eah tehnique P 1, P 2, P N, a training dataset of feature vetors may be olleted: 1 2 n i i i { P P,..., P }, where P = ( v, v,...),,1, 2 v: feature vetor 180

5 Figure 5. AMIR with automated bowing annotation. Figure 6. Real-time playing with AMIR. For eah dataset/luster we an ompute p The mean of the luster as the entroid A measure of the spread of the luster by measuring the distane d of the luster points to their respetive entroid: n 1 i µ =! d( p, p ) n 1 i = The standard deviation of the distanes: # = 1 n! n i = 1 2 ( d( p, p ) " µ ) i For an input dataset j p, for example apture of a student pratiing, we find the epsilon! for eah luster and find the luster with the minimum!. This indiates that the data from the apture is similar to that in a speifi luster (and hene orresponds to the bowing style or expression represented by that luster): 181

6 j d( p, p )# µ " =! To visualise the extrated features and the relationships between analyses of different bow strokes we have reated the mav.graph.3d objet. This objet urrently takes up to three different feature inputs (note that the lustering analysis is not limited to three features), from arbitrary mav.stats objets, and plots lusters in three-dimensional spae. Figure 7 illustrates an example of how the features an be displayed to show the grouping and variation of bow strokes based on analysis of the bow-veloity, bowsetion, and bow-length features from a reorded performane by linking eah feature to one of the dimensions in the graph. Figure 7. Clustering visualisation. Bow strokes played with a similar expression appear in the same area of the graph as lusters, and as a new stroke is added the mathing luster is highlighted to show that the stroke was similar. The more similar bow strokes are; the more ompat the lusters will appear in the graph. We also indiate the distane from the entroid of the luster. We implement this tehnique in suh a way to provide an easy to understand, high-level interfae to non-tehnial users like musiians and teahers to study the similarity of different bow strokes. As well as using this system to ompare strokes to previous strokes in the performane, it is also possible to load in analysis data sets from other reordings to failitate the omparison between two performanes. The SDIF format is being used for the storage of 3D motion and other sensor data allowing data exhange between the i-maestro gesture analysis tools [20]. 182

7 Figure 8. SDIF data rendering using S&G Lab with an example of the bow-stroke diretion and angle data from the AMIR and reorded sound. Cooperative Environment Cooperative work is another key area of musi eduation. The i-maestro Cooperative Environment allows different omponents of the i-maestro framework to be used aross a network. Figure 9. Cooperative interfae for ear training Aessibility To improve aessibility to musial knowledge and offer wider partiipation, guidelines for aessibility in tehnology-enhaned musi training have been developed and aessibility support has been added to the SMR tools. Figure 10. A view of the i-maestro Braille editor for musi. 183

8 Exerise Generator The i-maestro Exerise Generator supports (semi-)automated reation of exerises and variations of musi material using an extensible set of algorithms and templates. Figure 11. A sreen shot of the i-maestro Exerise Generator. Shool Server The i-maestro Shool Server, offers online aess to stored lesson material. Figure 12. i-maestro Shool Server Integration The ombination of tools leads to new funtionality, e.g. the automati annotation of a sore with bowing symbols in real time while a musiian is playing, whih is reahed by ombining, sore follower, motion apture and SMR support. An appliation (alled the i-maestro Start) has been reated to offer students and teahers a unique tool to start all the tools offered by i-maestro. With the tools now available, initial validation is started with teahers in musi shools and onservatories. CONCLUSION The projet ontinued its work on pedagogial aspets, enabling tehnologies, software omponents, integrations and validation ativities. An overarhing pedagogial approah and model [16] for tehnology-enhaned teahing and learning has been developed. On this basis, a set of detailed pedagogial senarios related to the use of the i-maestro tools has been reated. 184

9 This paper presented a brief overview of the i-maestro projet. With the introdution, the paper presented the overall framework design and introdued several tools to support musi learning and teahing inluding MPEG SMR for theory training, gesture analysis for performane training, with a partiular fous on the 3D gesture and posture support using the 3D Augmented Mirror. The final results onsist of a framework for tehnology-enhaned musi training, that ombines proven and novel pedagogial models with tehnologial tools suh as ollaborative work support, symboli musi proessing, audio proessing, and gesture interfaes. Offering aessible tools for musi performane and theory training will ensure wide partiipation. Prototype tools are now available and are expeted to be inorporated in various new produts and servies, whih will be made available to both the general publi and eduational establishments. These are being validated and refined and the projet is inviting musi teahers and students to take part in the validation phase. We are partiularly interested in testing the system in real pedagogial situations to see how teahers and students interat with the tehnology. At the ICSRiM - University of Leeds (UK), open lab sessions are being organised for people to ome and try out the i- Maestro 3D augmented mirror system with a 12-amera motion apture system. ACKNOWLEDGMENTS The researh is supported in part by the European Commission under Contrat IST I-MAESTRO. The authors would like to aknowledge the EC IST FP6 for the partial funding of the I-MAESTRO projet ( and to express gratitude to all I-MAESTRO projet partners and partiipants, for their interests, ontributions and ollaborations. Referenes [1] i-maestro projet website: [2] i-maestro projet Deliverable DE4.5.1 on Aessibility aspets in Musi Tuition, available online via [3] BELLINI, P Nesi, and Zoia, G: MPEG Symboli Musi Representation: A Solution for Multimedia Musi Appliations. Published on Interative Multimedia Musi Tehnologies, Copyright 2008 by IGI Global. ISBN (hardover) - ISBN (ebook). [4] BELLINI, P: XML Musi Notation Modelling for Multimedia: MPEG-SMR. Published on Interative Multimedia Musi Tehnologies, Copyright 2008 by IGI Global. ISBN (hardover) - ISBN (ebook). [5] MCKENZIE, N and Crombie, D: Creating Aessible Interfaes for i-maestro Learning Objets, in Proeedings of the Seond International Conferene on Automated Prodution of Cross Media Content for Multi-hannel Distribution (AXMEDIS 2006), Volume for Workshops, Tutorials, Appliations and Industrial, pp , 13th 15th Deember 2006, Leeds, UK, Firenze University Press (FUP), ISBN: , [6] CONT, A., Shwarz, D. (2006): Sore Following at IRCAM, MIREX 06 (Musi Information Retrival Evaluation exhange), The Seond Annual Musi 185

10 Information Retrieval Evaluation exhange Abstrat Colletion, Edited by The International Musi Information Retrieval Systems Evaluation Laboratory (IMIRSEL), Graduate Shool of Library and Information Siene, University of Illinois at Urbana-Champaign p. 94, Otober 2006, Vitoria, Canada ( [7] BEVILACQUA, F, N. Rasamimanana, E. Fléty, S. Lemouton, F. Bashet (2006) The augmented violin projet: researh, omposition and performane report, 6th International Conferene on New Interfaes for Musial Expression (NIME 06), Paris, 2006 [8] BELLINI, P, Paolo N, Maurizio Campanai, Giorgio Zoia, FCD version of the symboli musi representation standard. FCD version of the symboli musi representation standard, MPEG2006/N8632, Otober 2006, Hangzhou, China [9] BELLINI, P, Frosini, F, Liguori, G, Mitolo, N and Nesi: P MPEG-Symboli Musi Representation Editor and Viewer for Max/MSP, in Proeedings of the Seond International Conferene on Automated Prodution of Cross Media Content for Multi-hannel Distribution (AXMEDIS 2006), Volume for Workshops, Tutorials, Appliations and Industrial, pp , 13th 15th Deember 2006, Leeds, UK, Firenze University Press (FUP), ISBN: , [10] SCHNELL, N, Bevilaqua, F, Shwarz, D, Rasamimanana, N and Guedy, F: Tehnology and Paradigms to Support the Learning of Musi Performane, in Proeedings of the Seond International Conferene on Automated Prodution of Cross Media Content for Multi-hannel Distribution (AXMEDIS 2006), Volume for Workshops, Tutorials, Appliations and Industrial, pp , 13th 15th Deember 2006, Leeds, UK, Firenze University Press (FUP), ISBN: , [11] ONG, B., Khan, A., Ng, K., Nesi, P., Mitolo, N: Gesture-based Support for Tehnology-Enhaned String Instrument Playing and Learning. International Computer Musi Conferene (ICMC) 6-11 November 2006, New Orleans, Louisiana, USA, ISBN: [12] ONG, B, Ng, K, Mitolo, N and Nesi, P: i-maestro: Interative Multimedia Environments for Musi Eduation, in Proeedings of the Seond International Conferene on Automated Prodution of Cross Media Content for Multihannel Distribution (AXMEDIS 2006), Volume for Workshops, Tutorials, Appliations and Industrial, pp , 13th 15th Deember 2006, Leeds, UK, Firenze University Press (FUP), ISBN: , [13] NG, K, Larkin, O, Koerselman, T and Ong, B: i-maestro Gesture and Posture Support: 3D Motion Data Visualisation for Musi Learning And Playing, in Proeedings of EVA 2007 London International Conferene, Eds: Jonathan P. Bowen, Suzanne Keene, Lindsay MaDonald, London College of Communiation, University of the Arts London, UK, July 2007, pp [14] NG, K, Larkin, O, Koerselman, T, Ong, B, Shwarz, D, Bevilaqua, F: The 3D Augmented Mirror: Motion Analysis for String Pratie Training, p , in 186

11 Proeedings of the International Computer Musi Conferene, ICMC 2007 Immersed Musi, Volume II, pp , August 2007, Copenhagen, Denmark, ISBN: [15] NG, K, and Nesi, P (eds): Interative Multimedia Musi Tehnologies, ISBN: (hardover) (ebook), 394 pages, IGI Global, Information Siene Referene, Library of Congress , [16] WEYDE, T, Ng, K, Neubarth, K, Larkin, O, Koerselman, T, and Ong, B: A Systemi Approah to Musi Performane Learning with Multimodal Tehnology Support, in Theo Bastiaens and Saul Carliner (eds.), Proeedings of E-Learn 2007, World Conferene on E-Learning in Corporate, Government, Healthare, & Higher Eduation, Québe City, Québe, Canada, Assoiation for the Advanement of Computing in Eduation (AACE), Otober 15-19, [17] KOERSELMAN, T, Larkin, O, and Ng, K: The MAV Framework: Working with 3D Motion Data in Max MSP / Jitter, in Proeedings of the 3rd International Conferene on Automated Prodution of Cross Media Content for Multi-hannel Distribution (AXMEDIS 2007). Volume for Workshops, Tutorials, Appliations and Industrial, i-maestro 3rd Workshop, Barelona, Spain, ISBN: , November [18] NG, K, Weyde, T, Larkin, O, Neubarth, K, Koerselman, T, and Ong, B: 3D Augmented Mirror: A Multimodal Interfae for String Instrument Learning and Teahing with Gesture Support, in Proeedings of the 9th international onferene on Multimodal interfaes, Nagoya, Japan, pp , ISBN: , ACM, SIGCHI, DOI: [19] LARKIN, O, Koerselman, T, ONG, B and Ng, K: Analysis and Sonifiation of Bowing Features for String Instrument Training, in Ng (ed), Proeedings of the 4 th i-maestro Workshop on Tehnology-Enhaned Musi Eduation, o-loated with the 8th International Conferene on New Interfaes for Musial Expression (NIME 2008), Genova, Italy, ISBN: , pp. 5-10, 4 June 2008 [20] KOERSELMAN, T, Larkin, O, Ong, B, Leroy, N, Lambert, J-P, Shwarz, D, Guedy, F, Shnell, N, Bevilaqua, F, and Ng, K: SDIF Integration in i-maestro Gesture Tools, in Ng (ed), Proeedings of the 4 th i-maestro Workshop on Tehnology-Enhaned Musi Eduation, o-loated with the 8th International Conferene on New Interfaes for Musial Expression (NIME 2008), Genova, Italy, ISBN: , pp , 4 June