Magic Mirror : A new VR platform design and its applications

Magic Mirror : A new VR platform design and its applications Ig-Jae Kim Korea Institute of Science and Technology 39-1 Hawolgokdong, Seongbukgu, Seoul, Korea +82-2-958-5766 drjay@kist.re.kr Hyun Jin Lee School of Design & Art, Hong-ik University 300 Shinan-ri, Jochiwon-eup, Youngigun, Chungnam, Korea hjlee@hongik.ac.kr Hyoung-Gon Kim Korea Institute of Science and Technology 39-1 Hawolgokdong, Seoungbukgu, Seoul, Korea +82-2-958-5771 hgk@kist.re.kr ABSTRACT This paper describes a case study of VR platform Magic Mirror and its applications that are economic in development process and cost, flexible by contents and installation conditions, and that has business potential for consumer market. Magic Mirror uses video based virtual world and tangible interaction by motion tracking. Magic Mirror platform enables a user to monitor their action and to collaborate with other users of remote place within attractive interaction feedback. Magic Mirror system gives serious distance learning experience with tutoring and group collaboration. They are presented in public exhibitions and tested by exhibition visitors. They showed application potential of Magic Mirror platform in interactive game, distance learning, and entertainment field. Categories and Subject Descriptors C.C.3 [Special-Purpose and Application-based Systems]: a case study of VR platform Magic Mirror and its applications. General Terms Design, Human Factors First, the facility cost of the platform should be low in comparison to conventional VR platforms (CAVE, VR Theater etc.) to have market value in consumer business. And the second, the technology barrier and development cost for contents implementation should be low so that more designers have accessibility to VR contents. At the third, the platform should bring intuitive, natural user interaction and also the user experience should be valuable in terms of functional and emotional satisfaction. Finally, It should afford variety of contents or experiences within basic technology environment for wide application domain and contents versioning. While we were processing this project, we planned to show our project in public media art/ design exhibitions to get real audience feedback then to evolve the platform through design iteration. We have participated in VR contents design projects for CAVE and VR theater platform in Imaging Media Research Center, KIST (Korea Institute of Science and Technology). The projects are about VR exploration of cultural heritage named Heritage Alive!. This project required heavy technology development and professional facilities. Also we spent huge amount of manpower to develop three dimensional virtual world modeling (figure 1). In addition, the main platform (VR theater) is accessible in just a few research labs as we know. Keywords VR, Tangible Interface, Interaction, Vision Tracking, IR, Distant Learning,. 1. INTRODUCTION This project is about designing a new form of VR (Virtual Reality) platform and its application contents. We want to design a VR platform that satisfies following requirements. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ACE 04, June 3-5, 2004, Singapore Copyright 2004 ACM 1-58113-882-2/04/0006 $5.00. Figure 1. Three Dimensional World Model of Heritage Alive! Although Heritage Alive project is a very valuable move in VR research, its influence on real world is fairly minimal due to high

Active IR camera IR based Tracking Part Flash Animation Part IR Image Blob Labeling Position Extraction Interface Compositing Part Effect Rendering Color Image Rendering VR Environment Creation Magic Mirror System Figure 2. Software Architecture cost and low contents variety. So we started to study small sized, market oriented, and easy and quick VR experience design as another valuable VR research track. 2. SYSTEM OVERVIEW 2.1 Project Process We started the project with a primary idea of using projection and vision tracking for platform technology. And we explored user experiences that are interactive, emotionally entertaining, and have business value. Various user interactions and experiences are generated and evaluated. So we focused on a specific experience: playing and practicing a glass xylophone in front of magical mirror. We developed several prototype versions and tested them to optimize the design of experience and platform. Finally we designed graphics, GUI, and sound interactions. For public exhibition of our project, we optimized platform installation to the specific exhibition settings. The feedback of audiences from the exhibitions brought design insight and findings for the next version of our work. 2.2 User Experience What kind of experience would be valuable to users as a VR experience? VR enables user to feel reality in virtually generated situation. But if the VR world just mimics the real world, VR would be less attractive. Generally, people want to experience fantasy or magic in VR, like in a dream, in a fiction or movie. We decided our experience concept as magical user experience, with sufficient reality of presence. The target experience should be like a very live and vivid dream that is not possible in real world. The second design concept of our project is natural interaction between users and contents. When a user feels natural in VR environment, as in real world, the VR world brings a feeling of presence and immersive interactions. For these, we studied tangible interactions, which are partially conversed with the real physical world. Tangibles enhance a feeling of touch that is comparatively hard part of VR implementation in our technology boundary. In addition to having tangibility in interaction, we thought if the system reacts on various user actions in different ways, the experience would be rich and unique. That means giving more freedom to user actions so that each user becomes contents author by creating VR experience with user s own character. So we want to afford user s creativity in our project. Based on these discussions and idea generation process, we designed the main experience concept as a user does interaction with tangible prop while watching a big magic mirror in front of the user, which reflects magical interaction feedback and visual effects to the user. We named this platform Magic Mirror. A user feels strong presence of the VR experience because he/she watches himself/herself in VR wonderland. 2.3 Platform Structure We showed the layout of Magic Mirror platform in figure 3. Magic Mirror is composed of five systems, such as rear-projection system, active IR camera system, tangible objects (for example, glasses), IR reflector for users and server computer. In server computer, there are four software parts for operating Magic Mirror system, which are input part, tracking part, Flash animation part and composition part. We could get two streams in input part, one is for tracking and the other is for composition. In tracking part, we could extract the position of IR reflectors on xylophone sticks by using connected component labeling process. We have taken conventional authorware toolkit, macromedia s Flash MX, for creating VR environment easily. Flash animation part is a type of standalone application like a general Flash application, therefore Flash animators can make VR environment application freely without any technical constraints. In this case, however, there are two critical problems. One is interface between standalone flash application and tracking system. The other is composition problem. We ve solved the first problem by filesharing. Flash application can read the data from shared file that is written by tracking system with flag and position information. We ve, also, solved the composition problem by developing the transparent window process that is explained more in the following key technologies part.

(a) (b) Figure 5. The connected component labeling process example: left figure(a) is input gray level image, right figure(b) shows the three blobs after labeling process Figure 3. Magic Mirror System Platform 3. KEY TECHNOLOGY 3.1 IR Based Tracking After labeling process, we could get the position of IR reflectors from extracted the blob position of IR image. Before we apply the information of position to the Flash interface, we had to calibrate between the position of real glasses and that of virtual glasses from the Flash animation. Figure 4. Active IR Camera Computer Vision based object tracking has been widely studied. However, it also inherits the shortcoming of the most computer vision algorithms: sensitiveness to lighting condition. Thus we adopted active IR based tracking method for robust tracking. We used active IR (Infra-Red) camera for tracking xylophone sticks. The IR camera can be made with a normal video camera and an IR filter. In figure 4, the upper part composes the IR camera. The cold mirror is an IR filter that absorbs IR rays while reflecting visible rays. We used a cold mirror that absorbs the rays of above 800nm in wave length. The IR reflectors were made with retroreflective material so that they can be viewed best from the camera with IR light source. We could get two kinds of video stream from active IR camera, one is a color video stream for composition with Flash animation and the other is a IR video stream for detecting the position of IR reflectors [2]. 3.2 Labeling Process To extract IR reflector area in IR image, we applied the connected component labeling process to the IR image. The connected components labeling scans an image and groups its pixels into components based on pixel connectivity, i.e. all pixels in a connected component share similar pixel intensity values and are connected with each other in some way. Once all groups have been determined, each pixels is labeled with a gray level or a color labeling according to the component it was assigned [3][4]. Figure 6. Snapshot of developed IR tracking program interface Figure 7. Glasses and IR reflectors on Magic Mirror System 3.3 Layered Process To compose the virtual environment made by Flash MX with the real video stream from the camera onto the desktop with little CPU consumption, we used one of the new features of Windows

that is the ability to use what's known as compositing to draw Windows onto the desktop. Put simply, this makes it possible to make a window transparent for specific color range so that the background can be seen through the window. In Windows 2000 and XP window, transparency is used during some kinds of drag and drop operations to render icons as they are dragged across the screen. We showed the example of layered composition in figure 8. We made virtual environment with blue background using Flash MX (left image in figure 8). As windows transparency process makes a window transparent for specific color, we made the background of virtual environment blue. After transparency process, real video stream rendered by DirectX replaced blue area as a background of virtual environment [5]. Figure 9. Exhibition setting and users of Glass Xylophone 2003. Figure 8. Layered composition example 4. IMPLEMENTATION To explain the detailed solution, we separated our project results to two application cases by exhibitions of our work: Glass Xylophone 2003 and VR class. Each application case has the unique user experiences and key technologies while it shares Magic Mirror platform 4.1 Glass Xylophone 2003 Glass Xylophone 2003 is a targeted application for Korea HCI design exhibition, February 2003. In Glass Xylophone 2003, users play a glass xylophone with interactive sounds and visual effects (figure 9). The 15 glasses display 15 musical scales and each glass in which has no water (figure 10). When a stick touch a certain glass, a color bar pops up from the glass opening by the height of its musical scale on the screen. When a user continues the play, color bars present dynamic animation according to the music he/she plays. The music and visual effects varies by user s action. In additions to this free style play, Glass Xylophone 2003 provides practice mode of certain music. In practice mode, a user does not need to memorize the music to play. Leading cursor shown on the glasses indicates the next hit as on figure 14. The cursor waits until the user follows the pace. We provided the practice option of Do Re Mi Song in our exhibition. When a user successfully finishes the practice, Glass Xylophone 2003 sends the congratulation message and music on the screen. We encouraged users to experience artistic self-expression with glass xylophone performance and Magic Mirror effects. Figure 10. Tangible props (grasses and sticks) During the three days of exhibition, over hundreds people experienced Glass Xylophone 2003. The majority of users were HCI researchers and their family members. They had no difficulty on understanding and enjoying the contents. No significant interface problem was found in interactions. Since many users were researchers, they have interest in implementation mechanism. They often experimented on the effects and interactions of Glass Xylophone 2003. Businessmen discussed the application possibilities of this platform in arcade game. Among the users, kids are the most excited user group at Glass Xylophone 2003. They stayed for a long time to create their own performance with Glass Xylophone 2003. We observed that kids feel more immersiveness and presence in contents. The practice mode became more popular experience since there was a rewarding when a user finished assigned practice task. We found slight time lag on our interaction. However it was not a big problem because most of the users played the glass xylophone slowly. But when a user becomes to be a good player or the music to play is quite fast, the system needs to be upgraded.

4.2 VR Class Based on Glass Xylophone 2003 experience, we had more focus on having evaluation (or rewarding) function of the performance. There was a potential of serious education through Magic Mirror platform. Since a user can do monitoring and self-correction of his/her action, if well designed feedback of the user action is provided, we can use the Magic Mirror system to a learner centered education platform. In addition to self-monitoring and evaluation feedback of Glass Xylophone 2003, we explored collaborative learning possibilities through Magic Mirror. So having a live tutor and peer learners in contents and live interaction between the players became the next experience concept. 4.2.1 Upgrade Technology In VR Class system, we modified our previous system, Glass Xylophone 2003 which was stand alone system, to distributed system for interacting with remote-side user. To upgrade our platform to a distributed system, we had to install a video streaming server-client module for each side. After video streaming system installed, we had connected to control information, namely positional information of IR reflectors, via TCP/IP protocol through the Internet for each participant s side. We also added a video avatar module as a guide or a tutor for VR Class. To make a video avatar, we ve recorded the actor s performance using camcorder in advance for various situations according to VR Class scenario and segmented captured video sequence to make several short video clips. IR Image IR based Tracking Part Position Extraction Avatar Interface Effect Rendering Remote A Active IR camera Color Image ing Server/Client Render Server VR Environment Creation Flash Animation Network Figure 11. System Diagram of VR Class Color Image ing Server/Client Render Server VR Environment Creation Remote B Active IR camera IR Image IR based Tracking Part Position Extraction Avatar Interface Effect Rendering Flash Animation Then we composed those video clips with predefined background, rendered from Flash MX, using chroma-keying technology [6] and saved in database that would be loaded according to the user s input. In figure 11, we showed the upgraded Magic Mirror system for VR Class. For each remote group, we installed the streaming server/client module (shown as green color area in figure 11) based on the MPEG4IP streaming scheme which provides an end-to-end system to explore streaming multimedia [7]. By using streaming module, we could exchange the image stream of the users for each remote group and rendered together with local video from live camera and remote users video from network streaming server. As mentioned above, we ve included video avatar module (shown orange color area in figure 11) that rendered a virtual tutor in VR Class who interacts with users. The video avatar module is connected with the position extraction system, which enables to get the user s feedback, and can load the adequate video avatar clip in database according to the user s input. Figure 12. Learners in two distributed places and a tutor in VR Class 4.2.2 Exhibition and Installation The second exhibition venue of our project was special exhibition under the exhibition theme of Ten years after in one of the noted art gallery of Korea. The curator who visited Glass Xylophone 2003 proposed to participate in Ten years after exhibition. In this exhibition, we showed interactive distance learning experience through Magic Mirror platform which is named VR Class. In VR Class, two user groups in physically distributed location share one virtual classroom and an instructor in Magic Mirror as in figure 11. The tutor in VR Class monitors users action and facilitates the collaborative practice. She (an avatar in VR Class) introduces each user then leads simple practice of glass xylophone by turns. When two users get used to play music by glass xylophone, she suggests the game which is playing a given song by her lead as in figure 13. The score of each user is counted according to the player s performance. After the game, she evaluates the game results and gives some rewarding or encouraging messages. Then she says good-bye to the learners and then disappears. She comes into the classroom again when other learner calls for glass xylophone tutoring. As mentioned above, the tutor in VR Class is a video avatar who could speak, move, and interact with users. She acts one of the 30 different pre-recorded situations according to the user s action. It simulates live and interactive tutor-learners experience successfully. Due to space constraints, we simulated the distributed places with partition wall and one shared screen in our. But in real world situation, the places can be located in any distance and screen should be installed separately in each place.

more business value, we have to study business model of contents updates and platform production. In terms of technology research, we want to try three dimensional motion detecting and presentation. We did not catch z-depth of the motion in this project. It will provide more sophisticated feedback and solid sense of visual effects for a given user action, if we adopt 3 dimensional motion tracking. Also system instability and interaction time lag should be solved for commercialization of the Magic Mirror platform. 6. REFERENCES Figure 13. Two users are participating in a glass xylophone game. 5. CONCLUSIONS Magic Mirror platform is economic and handy. We brought the platform to exhibitions and installed the system with no sophisticated technological support. The installation is flexible so that we could customize the setting according to the place s conditions and contents. We showed the design case of two different applications with Magic Mirror platform. It has a big design potential for other new experiences. Especially we confirmed the possibility in education or in entertainment market. With observation of the user experiences through our two exhibitions, we found Magic Mirror platform provides sufficient reality for the desired experience. Having tangible interaction and interactive video display is the key interaction strategies to simulate magic in real world. For our future study, we will design more contents applications which involve self-monitoring and learning experiences such as sports, dance, or DIY lesson. The learner motivation would be higher because Magic Mirror can provide accurate action analysis, repetitive tutoring and entertaining special effects in addition to conventional learning experiences of the real world. For getting [1] Lee, H.J., Kim, J., Ahn, H., Ahn, S.C., Kim, I.J., Kim, H., and Ko, H., VR Experience Design in Tangible Space: Heritage Alive!, Proceedings of IEA(International Ergonomics Association) 2003 conference (Seoul, August 2003), IEA. [2] S.C. Ahn, T.S. Lee, I.J. Kim, Y.M. Kwon and H.G. Kim, Computer Vision based Interactive Presentation System, Proceedings of Asian Conference for Computer Vision 2004, January, 2004. [3] R.C. Gonzalez and R. E. Woods, Digital Image Processing, Addison-Wesley, 1992. [4] I.J. Kim, S.H. Lee, S.C. Ahn, and H.G. Kim, 3D tracking of multi-objects using color and stereo for HCI, IEEE International Conference for Image Processing, 2001. [5] M. Linetsky, Programming Microsoft Directshow, Wordware,2001. [6] A. Smith, J. Blinn, Blue Screen Matting, In ACM SIGGRAPH 1996, pp. 59-268, August, 1996. [7] D. Wu, Y.T. Hou, W. Zho, et al, On End-to-End Architecture for Transporting MPEG-4 over the internet, IEEE Transactions on Circuits and Systems for Technology, Vol. 10, No. 6, September, 2000.