Kinect Exercise HCI Course Fall 2015 Maria Husmann husmann@inf.ethz.ch CNB E 104.1
Agenda Introducing Kinect What it does How it works Kinect SDK for Windows Skeletal Tracking Kinect Tutorial (Example Code) Introducing the exercise Demo
Introducing Kinect Kinect is a camera-based, motion sensing input device Developed by Microsoft Announced first in June 2009 at the E3 as «Project Natal» Launched in November 2010 exclusively for the Xbox 360 video game console In February 2012, Microsoft released the commercial Kinect SDK for Windows (current version 1.8)
What it does Provides full-body 3D motion capture, face tracking, and speech recognition Skeletal tracking of up to 6 persons (Kinect v2) Has a built-in microphone array to record (and locate) voices Promises to deliver a «Natural User Interface» experience, your body is the controller The Kinect Effect (Advertisement)
3D DEPTH SENSORS RGB CAMERA MULTI-ARRAY MIC MOTORIZED TILT Old Kinect Sensor (v1)
Source: buildinsider.net
Source: ifixit.com
New is better: Kinect v2 Improved sensors: 3x depth fidelity, wider field of view, 1080p color camera (30fps) Tracks more people and more skeletal joints: Hand tips, thumbs and shoulder center Lighting independent IR (can see in the dark) Advanced face tracking Current Devkit: SDK 2.0
Kinect (v2) Data Sources
Body (skeletal) Tracking Recognizes up to 6 people in the field of view Maximum of two (six with v2) players tracked (body) at once Center position for other 4 players (v1) Two range modes (Kinect v1) Default: Between 0.8m and 4m Pratical range: 1.2 to 3.5 m Near range: Between 0.4m and 3m Pratical range: 0.8 to 2.5 m One Mode (Kinect v2) Between 0.5m and 4.5m
Joints Coordinates expressed in meters, center of the Kinect is reference plane Each player has set of 20 joints (25 in v2) Each joint has attributes Associated state: Tracked, Not tracked, or Inferred Joint normal (v2): Describes rotation Inferred Occluded, clipped, or low confidence joints
New Joints!
Advanced tracking with Kinect v2 Kinect v2 tries to recognize different states for each hand: Open, Closed, Lasso, Unknown and NotTracked Hand state is tracked for the two bodies closest to the sensor Lean Tracking Determines how much a body is leaning from vertical Values range between -1 and 1 in both directions, 1 roughly corresponds to 45 degrees of lean
Kinect Tutorial
Kinect Initalisation KinectSensor sensor; BodyFrameReader bodyreader; void MyKinectApplication_Loaded(object sender, RoutedEventArgs e) { // Get the default Kinect Sensor (there can only be one anyway) sensor = KinectSensor.GetDefault(); // Request a new reader for the body frames (skeletal tracking) bodyreader = sensor.bodyframesource.openreader(); // Allocate buffer for bodies (skeletons) bodies = new Body[6]; // Start the Kinect sensor sensor.open(); } // Register event handler to process body frames bodyreader.framearrived += bodyreader_framearrived;
Processing Body Frames void bodyreader_framearrived(object sender, BodyFrameArrivedEventArgs e) { // Retrieve the current body frame and dispose it after the block using (BodyFrame bframe = e.framereference.acquireframe()) { // Check whether the frame has not expired yet (i.e. it is available) if (bframe!= null) { bframe.getandrefreshbodydata(bodies); // Copy the body data into our buffer trackedbody = (from s in bodies where s.istracked select s).firstordefault(); // Get the first tracked body } if (trackedbody!= null) { JointOrientation headorientation = trackedbody.jointorientations[jointtype.head]; CameraSpacePoint headposition = trackedbody.joints[jointtype.head].position; //... do stuff }
The Kinect Exercise
Introducing the exercise Task: Control a PowerPoint presentation with your body Due to the nature of the project, you are required to program in C# You can work in teams of 4-5 Teams will be allocated at least 2 slots à 2-3 hours each (more slots can be allocated to your team if desired) Fun exercise with moderate effort
How the exercise is being run All programming is done with Visual Studio in our lab on the real Kinect for Windows hardware 2 labs are available, each equipped with a workstation and a Kinect for Windows sensor Visual Studio 2012 pre-installed You are responsible for managing your source code. You may be assigned to a different machine each time.
Implement «recognizegestures» /// <summary> /// TODO This is where you are supposed to implement your pose/gesture recognition /// </summary> /// <param name="trackedbody">tracked body data</param> private void RecognizeGestures(Body trackedbody) { // TODO Add your pose/gesture recognition code here! } Called each time a new body frame has been processed (up to 30 frames per second) You may change other parts of the code as well if you want to add more/extended functionality
Goal of this Exercise Design gestures for at least 4 different PowerPoint operations: Next Slide, Previous Slide First Slide, Last Slide Get to know the Kinect SDK for Windows and implement some poses and/or gestures You will be given a small code framework (you do not have to start from scratch) available by the end of this week Multi-user operations possible but tricky! (You will need to change the supplied framework)
Technical challenges Constant stream of data, no «touch down» or «gesture start» events Coordinates are in 3D space and differ depending on where the users are standing Our proposal: Pseudo-Gestures Try to recognize different poses (e.g. hands above head) A pseudo-gesture is a sequence of different poses Implement some kind of State-Machine
Time Schedule 23.09.2015 01.10.2015 (1 Week) Explore the Kinect SDK, design suitable gestures Register your group with Maria (husmann@inf.ethz.ch) until Friday 25. 9. Send an e-mail with the subject "Kinect Exercise" that contains a group name and the names of all members of your group Please also specify preferred dates/times: Mo-Fr, 9-12/12-15/15-18 29.09.2015-13.10.2015 (~2 Weeks) Implementation Preparation of a small presentation (ca. 5 minutes per team) You need to send your Visual Studio Projects and accompanying presentations to Maria no later than 13.10.2015 14.10.2015 Presentation and live demonstration of your system in the exercise session
Showtime! Demo
References Learning Resources and Documentation http://www.microsoft.com/en-us/kinectforwindows/develop/learn.aspx Official MSDN Documentation https://msdn.microsoft.com/en-us/library/dn799271.aspx Programming Guid https://msdn.microsoft.com/en-us/library/dn782037.aspx Video Tutorials (Lecture 2 is the most useful for this exercise), try to open link in Chrome http://www.microsoftvirtualacademy.com/training-courses/programmingkinect-for-windows-v2-jump-start