enterface 09 Project Proposal Video Navigation Tool: Application to browsing a database of dancers performances.

Transcription

1 enterface 09 Project Proposal Video Navigation Tool: Application to browsing a database of dancers performances. Principal Investigator: Xavier Siebert (FPMs, Mons) 1 Keywords Multimedia database; Content-based video browsing; Dance performances 2 Project objectives This project aims at creating an online navigation tool for video databases browsing. To allow for efficient browsing, the notion of "distance" between the items of the database must be defined, based on appropriate content descriptors. As part of the Numediart program ( in particular the {Audio, Media}Cycle projects) we are developing content descriptors for sound (e.g., rhythm, harmony, timbre,... ) and images (color, shape, textures,... ). The combination of sound and images paves the way for treating video databases, which nevertheless require additional specific features (e.g., quantity of motion). This enterface09 project will focus on a particular type of videos, namely short (2 min) dancers performances, that have been filmed in a precise setup. As such, it is associated with the artistic project "DANCERS!" launched by Bud Blumenthal s collective ( which aims at providing the first high quality audiovisual database of professional dancers gestures. 1

2 A major challenge of this project is to find descriptors of dancers movements that will provide relevant criteria to navigate the database. Some such descriptors are already incorporated into the EyesWeb [CRT99] XMI platform (e.g., quantity of motion, contraction index,... ), and others will need to be developed. Additional tags (name, age, city, body height,... ) will be associated with the videos, to provide a mixture between tag- and contentbased browsing. A website will be created to provide online access to the dancers information, and to interact with users voting by mobile phone. 3 Background information The evergrowing amount of publicly available multimedia libraries calls for novel tools and techniques for accessing their content efficiently. Database search initially relied on metadata (e.g., captions, keywords) associated to each item (sound, image, video,... ) which may be difficult or tedious to generate, especially when the database reaches a critical size. Contentbased descriptors have been developed as an alternative or a complement to metadata, as in the "Query by Image and Video Content" (QBIC) system [FSN + 95]. However, finding appropriate content descriptors is not a trivial task, particularly in the case of videos [PJ02, LSDJ06]. Several algorithms for compression and recognition of dance gestures have been developed recently [efc04] and could serve as a basis for providing descriptors to organize the database. Other criteria for gesture recognition and analysis will be developed a Numediart project, "Dancers Gestural Analysis by Dual Remote/Worn Sensing". The associated artistic project "DANCERS!" consists of a performance installation, that will be moved from cities to towns, featuring a 6m-wide projection screen besides a stage on which professional dancers will perform. A high quality camera will record their performance and display it on the screen, supported by a surround sound system. Dancers will perform one at a time on stage during two minutes, and the recorded performance will be stored in a database along with their biometric data (name, age, city, body height,... ). The audience could be allowed to vote to rate the performance or to select the next dancer to perform, using a tabletop booth or even cellular phones. 2

3 4 Technical description Below are listed specific points of technical relevance that will be investigated during the enterface workshop. given the recordings of two video cameras (one above and one in front of the stage), dancers movements have to be tracked in 3D and meaningful analytical descriptors of their gestures have to be extracted. To allow for comparison of whole sequences, global features have to be defined first : mean and variance of space occupation, statistics on the dynamics of the motion (e.g., quantity of motion),... The relevance of local features has to be investigated as well. organize and navigate the database containing audiovisual records, user-submitted tags and dancers biometric information. Several algorithms based on graph theory can be incorporated to efficiently retrieve similar dancer s gestures. develop social network / web 2.0 functionalities for the website improve the usability of the tabletop booth 5 Resources needed The resources listed below are necessary to process video sequences that will be provided by Bud Blumenthal. Computer resources: Development will be achieved on participant?s laptops, and the softwares are all publicly available. Some parts of the project will rely on the EyesWeb XMI or on the tools developed as part of the Numediart project. An online server will be setup to store the database of audiovisual rushes of dancers performances and to host the website with the navigation tool. Staff: 1 project leader + 5 researchers. Additional resources: a tabletop booth for public votes, to allow rating of the performances. 3

4 6 Project management The project leader will be responsible for the management, accompanied by a researcher from his team. 6.1 Work plan and implementation schedule The recording of dancers performances will take place before the enter- FACE workshop. The first video tests were done on January 28 and 29, 2009, with the participation of the principal investigator of this project. Other video recordings will take place between February and June, so that a small database of videos will be at out disposal at the beginning of the enter- FACE workshop, which we are planning to organize as follows: 1. First week Kickoff meeting Work assignment use of EyesWeb XMI to track the dancer s motions and extract representative features (space occupation, quantity of motion, contraction index,... ) (to start) design and develop the website (to start) 2. Second week continue the development of descriptors to classify the dance videos. Eventually use EyesWeb s Software Development Kit (SDK) to contribute new blocks specific to the analysis of dancers motion. design and develop the website (to continue) 3. Third week design and develop the website (to continue) incorporate the descriptors of dancers motion into the website prototype of user interface for the voting tabletop 4. Fourth week 4

5 Demonstrate the prototype of the online navigation tool and the voting tabletop Paperwork Review meeting After the enterface workshop, a public presentation of the related performance with public voting is scheduled for November Deliverables Navigation software for video browsing Rich Internet Application of the web 2.0 application for browsing the database Prototype of user interface for the voting tabletop 6.3 Benefits of the research The final product of this research will be a website for navigating a database of dancers. The mixing of tag- and content-based browsing opens the door to other studies, and will be an interesting tool for choreographers. 6.4 Profile of team 6.5 Leader Xavier Siebert graduated as a physics engineer from ULB, Belgium and holds a Ph.D. in biophysics from the Johns Hopkins University (Baltimore, USA). His thesis work focused on the determination of the three-dimensional structure of proteins and on 3D modeling. He then moved to France to work as a post-doctoral researcher at the CNRS on software development for interfacing X-Ray crystallography and Electron Microscopy. His interest is in applied mathematics and digital image processing. He now works as a Senior researcher at the Polytechnic Institute of Mons (aka FPMs), as part of the Numediart project. He is the co-director (along with Stéphane Dupont from FPMs) of the MediaCycle project, which investigates software tools for browsing sounds, images and their future extensions 5

6 to video databases. The first prototype of the software for audio browsing has been presented recently at Multitel, Belgium, and is currently being adapted to other types of media. 6.6 Staff Staff proposed by the leader: one researcher from the Numediart team will join the project. Possible candidates include Stéphane Dupont (senior researcher, co-director of the MediaCycle project) and Sullivan Hidot (post-doctoral researcher who worked on the statistical analysis of dancers movements). The availability of the researcher from the Numediart team will depend on the acceptance of other enterface projects, and cannot be further specified at this stage. Other researchers needed: 2 researchers working on video analysis 2 researchers working on website development 1 researcher working on user interface for the voting tabletop A possible industrial partner for the design of the website is Speculoos, based in Belgium, member of the Open Source Publishing agency 6.7 Artistic collaborations Bud Blumenthal s collective (along with Charleroi-danse) will provide the recordings of dancers performances. References [CRT99] [efc04] Antonio Camurri, Matteo Ricchetti, and Riccardo Trocca. Eyesweb - toward gesture and affect recognition in dance/music interactive systems. Multimedia Computing and Systems, International Conference on, 1:9643, S. Boukir et F. Chenevière. Compression and recognition of dance gestures using a deformable model. Pattern Analysis and Applications, Springer-Verlag, 7(3): , Décembre

7 [FSN + 95] Myron Flickner, Harpreet Sawhney, Wayne Niblack, Jonathan Ash ley, Qian Huang, Byron Dom, Monika Gorkani, Jim Hafner, Denis Lee, Dragutin Petkovic, David Steele, and Peter Yanker. Query by image and video content: The QBIC system. Computer, 28(9):23 32, [LSDJ06] Michael S. Lew, Nicu Sebe, Chabane Djeraba, and Ramesh Jain. Content-based multimedia information retrieval: State of the art and challenges. ACM Trans. Multimedia Comput. Commun. Appl., 2(1):1 19, February [PJ02] Milan Petkovic and Willem Jonker. Cobra: A Content-Based Video Retrieval System