Innovate Engineering Outreach: A Special Application of the Xbox 360 Kinect Sensor

Transcription

1 Innovate Engineering Outreach: A Special Application of the Xbox 360 Kinect Sensor Tanner Bryce Blair School of Electrical and Computer Engineering The University of Oklahoma Norman, United States tannerblair@ou.edu Chad Eric Davis School of Electrical and Computer Engineering The University of Oklahoma Norman, OK weeb@ou.edu Abstract In November of 2010, Microsoft released the Kinect sensor for the Xbox 360 video game console. This device similar to a webcam allows an individual to interact with an Xbox 360 or a computer in threedimensional space using an infrared depth-finding camera and a standard RGB camera. As of January of 2012, over 24 million units have been sold. Using a combination of custom and open-source software, we were able to develop a means for students to visualize and interact with the data allowing us to introduce the concepts and skills used in the field of Electrical and Computer Engineering. The unique technological application, visual appeal of the output, and the widespread ubiquity of the device make this an ideal platform for raising interest in the field of Electrical and Computer Engineering among high school students. In order to understand the appeal of the Kinect, a working knowledge of the technical details of the device is useful. The novelty and appeal of the Kinect sensor lies in its infrared camera, which is comprised of two distinct devices. An infrared projector sends out a 640x480 grid of infrared beams, and an infrared detector is used to measure how long the reflection of each beam takes to return to the sensor. This data set is known as a point cloud. This point cloud is a three-dimensional vector comprised of data points between 40 and 2000, which correspond to distance from the device of each beam. The data in this array can then be parsed to construct a 3d image. The Kinect s infrared camera operates at 30Hz, or 30 samples per second, so the device is able to deliver a frame rate that is sufficient to create the illusion of motion. This allows for the development of applications that give the user a sense of interacting in real time with the image on the screen. The unique visual appeal, novelty of interaction, and relatively easy-to-understand theory of operation make the Kinect an attractive platform for recruitment and outreach. Using the Kinect, a recruiter is able to quickly and effectively demonstrate a range of concepts involving hardware, software, and the design process on a platform that students are familiar with and find appealing. In a short window of time they are able to show examples and explain the fundamental principles of the system while providing tangible, meaningful, and enjoyable interactivity with the device itself. This level of approachability and familiarity is rare among highly-technical fields, and provides an excellent catalyst to develop interest in Electrical and Computer Engineering education. Keywords Kinect; engineering; outreach; education; Processing; natural interaction; embedded systems; Xbox 360 I. INTRODUCTION As referenced in a prior paper from our department [1], the University of Oklahoma s Electrical and Computer Engineering department (OU-ECE) suffered a sharp decline in undergraduate enrollment from 2004 to 2008, during which enrollment numbers dropped from 387 in 2004 to 246 in In order to rectify this, action was taken to form a studentdriven recruiting effort that emphasized hands-on, demonstrative outreach communicated in a peer dynamic [2]. This recruiting and outreach program was immensely successful, as evidenced by the undergraduate enrollment rising to 399 students in During this time a number of unique innovations in OU-ECE outreach were produced. While many successful hands-on demonstrations were born out of this program, a key omission was identified in the explanation and demonstration of embedded real-time systems and their application. In order to rectify this, several demonstrations were developed around the Microsoft Xbox 360 Kinect sensor. This paper aims to provide an overview of the Kinect platform, discuss the technical details of the Kinect and our application, analyze the perceived benefits of our approach, discuss the results seen thus far, and introduce some possible ways to expand the platform to increase its effectiveness. II. DEVICE OVERVIEW In November of 2010, Microsoft released the Kinect sensor for the Xbox 360 video game console [3]. This device similar to a webcam allows an individual to interact with an Xbox 360 or a computer in 3-dimensional space using an infrared depth-finding camera and a standard RGB camera /13/$ IEEE

2 This device has become immensely popular, and as of January of 2012, over 24 million units have been sold [4]. There are currently over 150 games available for the Xbox that utilize the Kinect sensor, and Microsoft is currently focusing on integrating it in to the Windows environment. While Microsoft has popularized the device, the actual technology was developed by an Israel-based company named PrimeSense. PrimeSense offers a range of 3d imaging solutions; all characterized by their system-on-a-chip (SoC) design and the open-source software solutions that accompany the device. III. HARDWARE OVERVIEW From the perspective of an end-user the Kinect is similar to a standard webcam, but the technology behind the device is distinct from other commercially available cameras. The device is, in fact, two distinct devices an infrared camera and a standard RGB camera that are processed into a single dataset by PrimeSense s SoC. The layout of the device is shown in Fig. 1. The Kinect s 3-dimensional imaging is accomplished by the infrared camera, which itself is comprised of two distinct parts that are managed by the SoC. The Kinect contains an infrared projector that emits a 640x480 grid of infrared beams. This grid leaves the Kinect in a conical projection and is reflected off of the environment. The Kinect then uses an infrared detector to calculate the length of time it takes each beam to return to the device. The SoC then uses that measurement to extrapolate the distance from the device at each point, and returns a three-dimensional vector to the host machine that is comprised of data points between 40 and These data points correlate directly to the number of millimeters away the person or object is from the device. This concept is illustrated In Fig. 2. In addition to creating this vector, the SoC also collects and correlates data from the RGB camera, allowing the device to determine with reasonable accuracy what color is present at each point in the cloud. Infrared Detector and Emitter Figure 1 - Physical Layout of the Kinect RGB Camera While all beams leave the Kinect simultaneously, the beam closest to the person or object of interest returns first. The Kinect s SOC uses the slight difference in time to create the mesh data that the software can then parse into a 3-dimensional image. Figure 2 - Kinect Time Delay Another key aspect of the Kinect s functionality is that it operates at 30 Hz, or 30 samples per second. This allows the device to deliver a frame rate that is sufficient to create the illusion of motion, and allows for the development of applications that give the user a sense of interacting in real time with the image on the screen. While no single component of the Kinect is necessarily revolutionary, the combination of the components in to a complete system allows for an embedded system capable of real-time data acquisition, and enables powerful applications that are relatively simple to develop. IV. SOFTWARE OVERVIEW By supplying the software to access the data from the Kinect in an open source fashion, PrimeSense has successfully fostered a large developer community around the Kinect. The PrimeSense Software Development Kit consists of two essential software solutions that facilitate development with the Kinect. The first is Open Natural Interaction (OpenNI), which provides an open-source API framework for development. Specifically, the OpenNI APIs allow access to voice command recognition, hand gestures, and body motion tracking [5]. The second software solution is Natural Interface Technology for End-User (NiTE). This software acts as a simple way to use the OpenNI framework with minimal CPU load [6]. NiTE provides the tools to quickly and easily turn the data from the Kinect device in to usable information about the depth, color, IR and audio data received from the hardware [6]. When choosing the development environment there were several key features that were important to us, both as developers and recruiters. In order for it to be an effective tool for distilling the core concepts, a simple interface and a robust development environment were vital. To this end, the demonstrations were developed in the open-source development environment Processing [7]. This environment vastly facilitates simple, rapid development, and contains many features that are a natural fit for the Kinect. By design, Processing is natively capable of generating robust, visually appealing displays using intuitive commands, and has a clean, simple user interface that lends itself to being incorporated in

3 to the demonstrations. Another key component of our choice of development environment was the SimpleOpenNI wrapper for Processing that allows for even simpler access to the core features of OpenNI. An overview of the development hierarchy is provided in Figure 3. sweep the shapes away from them, throw the shapes around, and otherwise use their physical presence to manipulate the virtual environment. Fig. 4 features a screen shot of this demonstration. User Interface Visual Output Processing Development Environment NiTE Algorithms SimpleOpenNI OpenNI PrimeSense System-on-a-chip Depth Physical Acquisition Figure 3-Software Hierarchy Overview V. USE IN OUTREACH Now that an understanding of the underpinnings of the Kinect hardware and software has been established, the device s application in our outreach efforts is discussed. We aim to accomplish this by providing an overview of the goals of this project, the product of our development, how it is employed in our outreach efforts, and what perceived benefits this project has yielded thus far. In our experience, demonstrations that allow for some sort of physical interactivity serve as the best starting point for a discussion with prospective students. While demonstrations that embody this concept are abundant for many fields, embedded systems and sensing demonstrations have either been too complicated, for the length of time and level of understanding, or too simplistic to fully illustrate the importance of this field of study. With these goals in mind, we set out to develop a series of applications and demonstrations that use the Kinect to illustrate some of the fundamental concepts of ECE, and the result was a collection of engaging, interactive demonstrations that illustrate a wide range of topics. To begin, we use a series of physicss demonstrations developed by Amnon Owed [8] that involve the students being able to interact with geometric shapes that fall from the top of the screen. The Kinect is capable of detecting the user, outlining them, and then using that outline to allow them to manipulate the shapes on the screen using their movements. Using edge detection, the software is able to provide the user with the ability to hold their arms out and gather the shapes, Figure 4 - Physics with the Kinect Another powerful use of the Kinect is employing it as a controller for other software. To demonstrate this, we developed an application that allows the user to control Ableton Live, a music creation and synthesis application. Using the Kinect and the midi protocol, the Processing application is able to send commands to Ableton Live that allow the user to select different virtual instruments, control the volume, pitch, and various sound effects using their physical movements. This is accomplished by using skeletal detection to outline the user, and then assigning values to the relative positions of their limbs. This translates to the user being able to move their right hand towards or away from the Kinect to control the intensity of the note and up or down to control the pitch, while using their left hand to control the intensity of the reverberation or distortion present in the signal. A screen shot of the skeletal data that this application uses is shown in Fig. 5. Figure 5 Skeletal data from the Kinect

4 Both of these applications provide the prospective students with a novel and unique interaction, and often lead to a discussion of how these demonstrations are accomplished. Typically the student is interested predominately in either the hardware and signal chain necessary to implement the demonstration or the software employed to accomplish the interactivity, and both demonstrations lend themselves to a discussion on either topic. When the student is predominately interested in the hardware, a quick synopsis of how the Kinect device functions is easily delivered by explaining the infrared camera as a mesh of individual data points. Using Processing we are able to show a view of the raw data and a simplistic rendering of a three dimensional image that allows the students to visualize the infrared mesh that the Kinect generates. This demonstration in itself is typically impressive, and often the students are surprised at the amount of information the Kinect is able to deliver. An example of the Kinects ability to render three dimensional images is shown in the screenshot in Fig. 6. The camera is sitting in front of the user, but the software is able to render the image from above. Figure 6 - Kinect data view During the past year, we have used these demonstrations at a variety of recruiting events, both on and off of our campus and with various age levels. At a variety of traditional-format events where our recruiters are given a table and students are free to choose whom to talk to, the demonstrations were effective at catching the interest of prospective students. These demonstrations were employed at several recruiting events targeting high school students interested in attending OU, and were very well received. Through our conversations and interactions with students, we have come to believe that much of this success is owed to the familiarity of the device. As previously noted, there are over 24 million Kinect sensors in the homes of consumers. This is a device that these students associate with having fun and playing video games, and often this connection is enough to help convey the impact that electrical and computer engineers have on a wide range of activities in their daily lives. Additionally, the Kinect demonstrations provide the opportunity to interact with students that are interested in video games that would traditionally only view Computer Science as a viable major. The associations that the Kinect brings often allow us a chance to speak with these students about the hardware and lower-level software that goes in to game development, and potentially convince them that ECE could be a better fit for them. While there are a variety of interesting demonstrations that we employ in our outreach, the Kinect creates a unique starting point for conversations with high school students about the degree and scope of ECE and helps attract students that would traditionally be interested in other areas of study. The Kinect has proven to be an effective tool for outreach as a booth demonstration, but another important aspect of our work with the Kinect has been outreach targeted specifically to high school groups that are interested in the STEM fields. In the past year, we have used the Kinect demonstrations with various school groups, including a high school mathematics club, a high school robotics club, a high school physics club, and a pre-engineering club. When presenting to the mathematics club, we were able to relate the Kinect demonstrations to the mathematical algorithms that the Kinect uses to determine the depth of an object, the trigonometry it uses to align the color data with the point cloud, and various other applications of mathematics in technology. With the robotics club we shifted our focus towards machine vision, object detection, and the creation of responsive programming. With the physics club we focused on creating computer models and simulations, and using those simulations to predict and observe how the physical world would behave under a specific set of conditions. With each subject, the Kinect has served as a powerful tool for linking a prospective student s current interests with a related application in technology and engineering. Using the demonstrations outlined above to broach the topic of studying ECE has proven to be effective. As we used these demonstrations over the past year, the response has been immensely positive due to the diverse applications of the Kinect, the novelty of being able to render objects in three dimensions, and the prospective student s familiarity with the device. While no specific, quantitative data has been collected for analysis at this point, our organization intends to develop the methodology measure the impact of this device and will begin collecting data in the coming months. Using the Kinect has allowed our recruiting team to reach a wider audience of students than other demonstrations and recruiting methods allowed. VI. FUTURE DEVELOPMENT Moving forward, there are several key features that we would like to implement in to our Kinect demonstrations. To heighten the effect of the demonstrations and to increase the appeal for students interested in physics, we would like to add the ability to adjust simulation settings in real time. This

5 would give the user the ability to select the effect of various conditions such as gravity or the mass of an object. Another possible direction that we are exploring is using the LabVIEW development environment to create functional blocks that the students could use to write their own programs within a relatively short amount of time. Using this, a recruiter could allow the student to not only experience the demonstration, but actively engage in its creation. The LabVIEW development environment also lends itself to the creation of interfaces like the ones previously described that would be used to control simulation variables. Given these advantages, the possibility of simultaneous development could be beneficial. We would also like to work on developing the Kinect in to a series of robotics demonstrations where the actions of the user would influence the behavior of a physical object. For example, the user s skeletal data could be used to determine the position of a servo motor, remote commands could be given through motion, or a robotic arm could be programmed to wave back when a user waves. The possibilities for effective, simple demonstrations are numerous, and we look forward to developing an increasing number of applications as our outreach efforts continue to grow. VII. CONCLUSION While OU-ECE s recruiting and outreach program has seen great success over the last five years, the Kinect has enabled our recruitment team to reach a greater number of students interested in a wide array of disciplines. Using a device which most students are familiar with and have had a positive association with has allowed us to introduce the fundamentals of embedded systems to a new generation of prospective students and enabled meaningful conversation pertaining to the importance of the fields of ECE. The student excitement generated by this device at recruiting events has surpassed all other demonstrations OU-ECE has deployed in the past and it has always been the most engaging demonstration of any event it has been shown. We believe others will find the Kinect an extremely useful tool for ECE outreach. ACKNOWLEDGMENT As noted, many of these demonstrations were either the direct product or derivative of other members of the open source community. To that end, we wish to thank the independent developer Amnon Owed whose work was used in our physics demonstrations, Professor Max Rheiner of the Interactive Design program at Zurich University of the Arts whom contributed countless examples that aided in our development, Professor Peter Froslie of the School of Art and Art History at the University of Oklahoma for his assistance with Processing and communication over the midi protocol, and independent developer Jordan Kuehn who assisted with various technical challenges as they arose. All of these people were an invaluable resource, occasionally without their knowledge or against their will. REFERENCES [1] C. Davis, M. Yeary, and J. Sluss, "Reversing the trend of engineering enrollment declines with innovative outreach, recruiting, and retention programs," IEEE Transactions on Education, vol 55, issue 2, pp , May [2] C. Davis, D. Vreeland, C. Griffin, and M. Yeary, Creating a Culture of Student-Driven ECE Recruiting and Retention, ASEE Annual Conference and Exposition, Paper accepted and to appear in conference proceedings. June [3] Microsoft Xbox 360 Kinect Launches November Gizmodo. Accessed: 20 February [4] MICROSOFT SAYS XBOX 360 SALES HAVE SURPASSED 76 MILLION UNITS, KINECT SALES TOP 24 MILLION BGR. Accessed: 20 February 2013 [5] ABOUT OPENNI OpenNI. Accessed: 24 February [6] NITE OpenNI. Accessed: 24 February [7] Processing Processing.org. Accessed: 8 April [8] Amnon P5 - Experiments with Processing Accessed: 8 April [9] Borenstein, Greg Making Things See: 3D vision with Kinect, Processing, Arduino, and MakerBot. Sebastopol, CA: Maker Media.