Providing Natural Means for Teacher in Smart Classroom for Tele-education *

Transcription

1 Providing Natural Means for Teacher in Smart Classroom for Tele-education * Yuanchun Shi, Weikai Xie, Guangyou Xu, Enyi Chen, Yanhua Mao, Fang Liu Department of Computer Science and Technology The State Key Laboratory on Intelligence System and Technology Tsinghua University, Beijing shiyc@tsinghua.edu.cn bstract Smart Classroom is an ongoing project dedicated to develop a Smart Space augmented classroom in which Tele-education can be seamlessly integrated with face-to-face education. In this paper, we describe the features of Smart Classroom, especially those may help to get a better user experience. Pen-based UI on the projected display of the teaching slides provides just the same experience as using a real blackboard or whiteboard for writing. Laserpointer To Cursor together with the Voice Command helps the teacher to communicate with students and control the content of presentations. Virtual ssistant can enhance the speech interaction between the teacher and the system. nd Smart Cameraman module can automatically capture the proper view of the classroom according to the context of the class as the video feed for the remote students. Keywords: Smart Spaces, Smart Classroom, Natural User Interface 1 Introduction promising application domain of UbiComp is the education. The Classroom 2000 project [1] at GIT has been devoted into the automated classroom experience capturing. Besides, the Smart Classroom project at our institute is focused on Teleeducation. Most currently deployed real-time Tele-education systems are desktopbased, in which the teacher s experience is totally different from teaching in a real classroom. For example, he/she should remain stationary in front of a desktop computer instead of moving freely in the classroom. The goal of Smart Classroom project is to eliminate the discrepancy of a teacher s experience in Tele-education and that in the traditional classroom education, as well as enable the integration of these two currently separated education practices. Our approach was to move the legacy real-time Tele-education system off the desktop, extending its user interface into the 3D space of an augmented classroom (called Smart Classroom) so that in this * The related research is supported by NSFC, 863 High-tech Plan, The Ministry of Education, China and IBM China Research Lab.

2 classroom the teacher could interact with the remote students with multiple natural modalities just like interacting with the local students. Smart Classroom Camera Camera Display Camera Server Fig. 1. Smart Classroom System Fig.1 shows the general idea of Smart Classroom. real classroom is instrumented and turned into a natural user-interface of the Web-based tele-education software for the teacher side, where teachers can move freely and use the conventional teaching metaphors that they are familiar with to give classes to remote students, instead of the cumbersome desktop-based metaphors used in most of current real-time teaching systems. Since the teacher is in a real classroom, we can accommodate local students presenting in the classroom at the same time. This way we can eliminate the difference of on-campus learning and tele-education activities, because teachers can give classes to local students and remote students simultaneously, which can reduce the required workforce of teachers as compared to separate operation of on-campus education and tele-education. Furthermore, the process of the lecture will be recorded as a hypermedia courseware for playback after class. The rest of the paper will be organized as following: First, we give a scenario of Smart Classroom as a teacher s experience. Then, we describe the setting of the classroom, user interface technologies and the smart cameraman facility respectively. fter a short statement of the software infrastructure which integrate the functional modules, we finally give a conclusion. 2. Scenario When Prof. Shi walks into the Smart Classroom, her biometrics features are detected automatically and the room recognizes her. So the classroom says Good morning, Prof. Shi, you can give your lecture now. She goes to the whiteboard. Then she finds her lecture, the lecture for Digital Logic course, displaying on the whiteboard just as where she stops in the last class and finds the representing image of

3 remote students, who attend her lecture by the Internet, are also displayed on a wall. Thus Prof. Shi begins her lecture. When she gives her presentation, just like in a normal classroom, she can make annotations on the blackboard with electronic chalk. Besides, she can also control the display of the lecture conveniently by her speech or a laser pen on her hand. When the remote students meet some difficulties on understanding her lecture, they can consult Prof. Shi, just like the local students. For example, the classroom says, Tom has a question now. Prof. Shi replies Tom, go ahead. So Tom s audio and video will be presented in the classroom. He can also sketch his question on same board where Prof. Shi presents her slides and adds annotations. The remote students can get the live video and audio of the classroom with different focus according to what the teacher is illustrating. For example, when Prof. SHI makes annotations on the blackboard, the video focus on blackboard will be fed to them; when she explains a realia in her hand, the video focus on the realia will be fed to them; when she interprets a concept on the dais, an overview scene of the classroom will be fed to them. ctually this scenario can be accessed as a video demo on our Website[2]. 3 Room Setting ccording to the scenario, besides distance learning technologies, teachers can have the similar experience in Smart Classroom as in a traditional face-to-face classroom which is most familiar and efficient. Smart Classroom is inspired by the research filed of Smart Space. Smart Space is a richly instrumented physical environment where people can get transparent access to information and assistance from computers while performing their ordinary tasks in this space.[3] There are many research institutes carrying out research on Smart Spaces with different emphases. Interactive Workspaces in Stanford [4] is the multi-device, multiuser environments for people to work together in technology-rich spaces with computing and interaction devices on many different scales. Intelligent Rooms in MIT [5] have cameras, microphone array, speech command to try to interpret what people are doing in order to help them. Easy Living in Microsoft Research [6] is an intelligent environment for people to live easily. Dream Space in IBM Research [7] allow humans to interact with the real world in natural ways, using the common skills of speaking, gesturing, glancing, moving around, reaching out. ware Home in Georgia Tech [8] builds a home, which provides services to its residents to enhance their quality of life or help them to maintain independence as they age. Despite of having different purpose, all of them must take consideration of interaction with real world. Smart Classroom is just such an effort to turn an ordinarily classroom into a Smart Space for Tele-education.[9] We augment an ordinary classroom with wall-sized displays, sensors, cameras and the associated software modules such as gesturetracking module and speech-recognition module so as to allow the teacher presenting in the classroom to access the tele-education software transparently, rather than

4 appeal to a desktop computer. Through Smart Classroom, we actually extend the user interface of the teacher s interface from a desktop computer into the 3-D space of the classroom. The space of a classroom usually can be divided into two parts - the teaching area that mostly used and occupied by the teacher, and the audience area where local students reside. The most significant physical instrumentations in a Smart Classroom, compared to an ordinary one, take place in the teaching area, as illustrated in Fig. 2. They are two large projector screens. The one on the front wall is a touch-sensitive screen, replacing the usual blackboard or whiteboard found in a classroom. It s called Mediaboard to display teaching materials. Remote students can view the content of this board on their client program (which is only desktop-based). nother screen called Student Board is on the sidewall. It is the window to remote students, on which the image of remote students participating a session will be displayed and the video and audio of the remote student who takes the floor will be played here too. The teacher and local students become aware of the presence of remote students through this facility Remote Students Studentboard Mediaboard Virtual ssistant Fig. 2. snap shot of the Smart Classroom Besides these two obvious facilities, there are about a half-dozen cameras, each with different usage, installed at different places in the classroom. Some are used to recognize the action of the teacher and the result is interpreted as an interaction command to the underlying tele-education software. Others are used to capture the live video of the classroom for broadcasting to the remote students. In addition, the teacher wears a wireless microphone for capturing his speech. 4. Natural Teaching Experience In Smart Classroom, in order to give classes to remote students, the teacher no longer needs to remain stationary in front of a desktop computer and, for most of common tasks involved in a class, the teacher no longer needs to use keyboard and mouse

5 either. To this end, the following technologies are developed and integrated in the Smart Classroom Pen-based UI s mentioned above, the Mediaboard is displayed on a touch-sensitive screen, which is actually a commercial product called SmartBoard TM. Most functions of a mouse found in a desktop setting can be accomplished by operating directly on this board. For example teachers can control the display of the slides. Moreover, using the provided pens and erasers, teacher can write comments and scribbles on the slides or wipe the strokes, as illustrated in Fig.2, which provides just the same experience as using a real blackboard or whiteboard. nd the remote students will get to see the presentations and writings on the board, as shown in Fig.3. List of participating students Video from Teacher side Mediaboard and the slides displayed on it Fig. 3. Remote Students Client t the beginning of the project, we ever decided to develop a mechanism that can have the same function as a mouse but do not ask the teacher approach to a desktop computer.[10] Therefore we developed the Virtual Mouse module, which is based on a hand-tracking algorithm. The teacher s hand movement on the vertical plane was tracked when standing steps away from the Mediaboard and turned into the movement of the cursor on the board or strokes on the slides. However, the user experience turned out to be frustrating for two reasons: 1) The skin-color modal and appearance modal of hand used to locate the teacher s hand is inevitable unreliable in a setting like Smart Classroom where front-projection projector is used, because the environment illumination changes as the projected content changes. The case is even worse when the light of the projector shot on the teacher s hand directly. 2) While the metaphor here, using teacher s finger as an extended pointer device, is natural for driving the cursor, but not for draw scribbles on the slides, for one is difficult to keep his/her hand movement smooth without a firm support point of his/her arm. That s why we gave up the idea of Virtual Mouse but developed other two mechanisms which together can achieve the same goal but more reliable. First, to allow the teacher stably draw scribbles on the slides, we adopted SmartBoard. Second mechanism is the Laser Pointer To Cursor module we developed.

6 4.2. Laser Pointer To Cursor Laser pointers are widely used as a tool for indicating the focus nowadays. Besides this common utility, in Smart Classroom the teacher can even use a laser pointer as an interactive tool. The function is provided by a computer vision module that can track the movement of the spot of the laser pointer and recognize its certain movement patterns. This recognition result is interpreted differently according to whether the spot is on the Mediaboard or the Studentboard. If on the Mediaboard, the movement of the spot is interpreted as the mouse movement event and circling the spot around a point is interpreted as the mouse-click event, so that the teacher can select and open a link on the HTML page by laser pointer. lthough the same task can be completed by manipulating on the Mediaboard directly, this is nevertheless useful, for the teacher do not need to approach the Mediaboard every time he/she need to open a link. s the laser pointer can be tracked, its position can be always recognized as the cursor s position which can also help to catch the remote students attention just like it works in the classroom for the local students. If the spot is on the Studentboard, the recognition result is interpreted as the current selection of a remote student, and as an indication, this remote student representing image will be highlighted. Together with voice command, it provides a convenient method for the teacher to control the floor between remote students. For example, the teacher can point the laser pointer on a remote student s representing image and say Go ahead (as shown in Fig.4), which will result the system to give the floor to this remote student. Go ahead The representing image of a remote student is highlighted as the teacher pointing the laser pointer on it Fig. 4. Give the floor to a remote student by a laser pointer plus voice command 4.3. vatar as Virtual ssistant The Smart Classroom incorporates a speech-recognition module as well as a text-tospeech module. Therefore the teacher can complete several common tasks involved in a class by voice command such as Give the floor to Tom or Jump to the previous page. The system can also use the synthesized voice to notify the teacher of certain

7 events. For example when a remote student named Peter asks for the floor, the system will alert the teacher Peter is asking for the floor. However interact with a dummy room with voice will seem funny in some sense, so we introduced a Virtual ssistant figure into the Smart Classroom to impersonate the classroom against the teacher. The Virtual ssistant shows a face of a virtual person, which are displayed on the StudentBoard as illustrated in Fig. 2 as well as in Fig. 5, and its face expression and lips movement is synchronized with the synthesize voice of the system. Thus the teacher can see a vivid virtual assistant that can understand her/his voice command and give notifications or feedbacks through speech, just as if there is a real assistant Login to The Classroom Based on Biometric Characteristic Since a Smart Classroom is a public space, each teacher who gives a class in it needs to identify him/herself to the system in order to be authorized to use the facilities in the classroom. The common way of authentication on a desktop computer is inputting one s ID and Password with keyboard. In Smart Classroom, we use a combination of face-recognition and speaker-verification technology to automatically identify the teacher, providing the teachers an undeterred experience while login to the classroom. s the teacher enters the Smart Classroom, he first should show up in front of a mirror (behind which a camera for capturing the teacher s face is installed) and speak out his name, if both the face-recognition and speaker-verification processes are passed, the Virtual ssistant will greet him, indicating the Smart Classroom is now ready to serve him. This procedure is showed in Fig. 5. Moreover, the system will use the teacher s identification information to load the right voice model for the teacher into the speech-recognition module if the teacher have trained such a model in advance, which is beneficial for the accuracy rate of the recognition result.

8 Virtual ssistant Your Name, Please. camera for Face- Recognition is installed behind the mirror I m Weikai Fig. 5. teacher login to the Smart Classroom 5 Smart Cameraman When taking a class in a real classroom, the students will change the focus of their sights as the context of the class changes. For example, when the teacher are writing a formula on the blackboard, the students will focus their sight on the formula, while when the teacher is showing a model in his hand, the students will focus their sight on the model. However, in most of current real-time teaching systems, the students can only get the teacher-side video with a fixed scene no mater how the context of the class changes, which significantly decreases the efficiency of understanding of the teacher s instruction. (a) The teacher was writing on the Mediaboard (b) The teacher was showing a model (c) The teacher was discussing with local students Fig.6. Different Scenes Remote Students Get ccording to Context of The Class To overcome with this problem, a facility called Smart Cameraman is introduced into Smart Classroom, which can distinguish among several kinds of context in a class by observing some clues in the classroom, and then select a camera with proper view according the context from an array of available ones as the source of the live video to remote students. Currently, this module can successfully distinguish the following three kinds of context:

9 1) Teacher Writing on Mediaboard, where the teacher is writing or scribbling on the Mediaboard. In this case, the camera which focus on the Mediaboard will be selected as Fig.6 (a) shows. 2) Teacher Showing Model, where the teacher is holding a model in his hand. In this case, the camera which follows the teacher s hand will be selected as Fig.6 (b) shows. 3) Others, for all other situations. In this case, the camera with a overview of the whole classroom will be selected as Fig.6 (c) shows. The cues used by the Smart Cameraman to estimate the current context includes the output of a Person-Tracking module which track the position of the teacher, a Gesture-Recognition module that can decide whether the teacher is holding something in his hand and the touch event reported by the SmartBoard. 6 The Software Infrastructure s we have seen, besides the tele-education software, there are many other modules running in the Smart Classroom that provides periphery services, including speech-recognition module, text-to-speech module, tracking modules and so on. Totally we have about a dozen of modules running on eight distributed computers and we developed a multi-agent system (as shown in Fig.7) called Smart Platform [11] to interconnect and coordinate these modules. Here each module is running in a separate process and communicate with each other through predefined a XML format messages. For example, the module in charge of the laser pointer tracking will periodically send a message to the module in charge of the Mediaboard to update the location of the cursor while detecting a laser pointer spot. Context Unification gent Teacher ction Recognition gent MediaBoard gent Speaker Recognition gent vatar & TTS gent User Profile gent Virtual Mouse Laser Pointer gent Tracking gent StudentBoard gent Face Recognition gent Speech Recognition gent Fig.7. Software Platform for Smart Classroom Since the Smart Classroom is a fairly complicated system, its reliability is an important issue that needs careful consideration. Besides each composing modules are

10 carefully designed and developed, the interconnection of modules takes a loosecoupling policy. First, we use message passing instead of RPC so that one module will not block for the completion of a communication request, which reduce the possibility of a halting module to block other module which is communicating with it. Second, we use a mediated communication scheme in that all the modules only need to maintain a connection with a centralized message dispatcher module and all the inter-modules communication are send to and forwarded by this dispatcher. This structure reduces the recovery work need to do when a module is restarted after failure. 7 Conclusions Currently we have made concrete achievements on each part of the project and the prototype system runs quite well in initial informal evaluations. We are planning a formal user study of this system in the near future. We are also cooperating with the Distance Learning School of Tsinghua University for the large-scope user experiments 2003 fall. With the development of computing and network technology, the portable computing device will be proliferated. The teacher may take a portable computer or PD into the Smart Classroom, it is interesting how to configure interface depending on the set of devices available. How to seamlessly add computing device into the Smart Classroom is another research topic we are now paying more attention on. Reference 1. Gregory D. bowd. Classroom 2000: n Experiment with the Instrumentation of a Living Educational Environment. IBM Systems Journal, Special issue on Pervasive Computing, Vol38, No4: Smart Classroom Project NIST Interactive Workspaces in Stanford, 5. Intelligent Rooms in MIT, 6. Easyliving in Microsoft Research, 7. Dream Spaces in IBM Research, 8. ware Home in Georgia Tech, 9. Xie, W.K., Shi, Y.C. & Xu G.Y. Smart Classroom - an Intelligent Environment for Teleeducation. In Proceedings of The Second Pacific-Rim Conference on Multimedia (PCM 2001), Beijing, China. Springer LNCS2195, Jiang Changhao, Shi Yuanchun, Xu Guangyou, Xie Weikai. Classroom in the Era of Ubiquitous Computing - Smart Classroom. ICWLHN2001, Dec 5-7, 2001, Singapore 11. Yanhua Mao, Weikai Xie, Yuanchun Shi, Guangyou Xu. Building the Software Infrastructure From O to Smart Platform PCM2002, Dec, Taiwan LNCS2532