Introduction to the Perceptual Computing

23.09.2013 1 Introduction to the Perceptual Computing Designed by employees of The Center of Innovative Education

Introduction to the Perceptual Computing 23.09.2013 2 This course is aimed to study and introduce the innovative technologies of human-computer interaction to the educational process in institutions of higher and secondary education. The goals of the course: To study the fundamentals of human-computer interaction; To get acquainted with the hardware and software features of the Intel Perceptual Computing SDK; To study how to design the interactive applications, which involve functions of natural intuitive interaction.

Introduction to the Perceptual Computing The course is designed in the collaboration with Intel Corporation The set of lectures and practical works can be taught as elective course or as distance course The course can be taught on Russian and English language 23.09.2013 3

Authors The leader of the group Olga Yufryakova, Director of the Center of Innovative Education, Institute of mathematics, information and space technologies NaRFU, senior lecturer at the Department of programming and high performance computing. Julia Berezovkaya, senior lecturer at the Department of programming and high performance computing. Vera Nekrasova, an engineer at the Center of Innovative Education, Institute of mathematics, information and space technologies NaRFU, M.Sc. In Media Technologies. Konstantin Nosov, a software developer at the Center of Innovative Education, Institute of mathematics, information and space technologies NaRFU, M.Sc. In Media Technologies. 23.09.2013 4

23.09.2013 5 Actuality Natural intuitive interaction is the evolution of the interaction with the computer through natural for human ways and interfaces (beyond keyboard and mouse); All that you are trying to solve with the help of a computer, will be done through the recognition of your face, voice, gestures. Natural and intuitive interaction involves the use of various technologies, transforming the interaction with the computer in a rich and interesting process.

23.09.2013 6 Course structure Lectures 1 New features in human-machine interface. Basic terms, the benefits of utilization. Practical work 2 Fundamentals, functionality and features of Intel Perceptual Computing SDK. 3 Gesture recognition with Intel Perceptual Computing SDK. 4 Voice recognition with Intel Perceptual Computing SDK. 5 Face detection and recognition (face landmarks and characteristics) with Intel Perceptual Computing SDK. Installation and configuration of Intel Perceptual Computing SDK. Designing of applications for gesture recognition using SDK. Designing of applications for voice recognition using SDK. Designing of applications for face detection and recognition using SDK. 6 Tracking of 2D/3D objects, augmented reality. Designing of applications for tracking of 2D/3D objects using SDK. 7 Game engines and integration with Intel Perceptual Computing SDK. OpenFrameworks: concepts, features, functionality. Configuring of SDK for use with openframeworks applications. Installation of openframeworks. An example of application.

23.09.2013 7 Hardware (1) CREATIVE Interactive Gesture Camera Sensor resolution: 720p (1280 x 720) Resolution of depth sensor: QVGA (320 x 240) The angle of view: 73 30 FPS Working distance: from 15 cm to 1 meter Power supply: USB 2.0 (< 2.5 W)

23.09.2013 8 Hardware (2) Center of Innovative Education, Institute of mathematics, information and space technologies NaRFU is equipped by: 15 working places; Three CREATIVE Interactive Gesture Camera; Installed software; Designed course.

23.09.2013 9 Software Intel Corporation provides an innovative product Intel Perceptual Computing SDK. SDK is focused on several aspects of natural intuitive humancomputer interaction: Face Detection; Voice recognition; Gesture recognition. Also SDK includes tutorials, code samples, algorithms, and application examples. This will allow developers to implement functions of natural intuitive interaction.

Modules of Intel Perceptual Computing SDK 23.09.2013 10 Gesture recognition module; Face detection module; Voice recognition module; Module for tracking of 2D/3D objects;

Gesture recognition module (1) Detection of fingers position (7 points): The tips of the fingers; Center of the palm; The most visible point, which is located close to the elbow. 23.09.2013 11

Gesture recognition module (2) Recognition of standard gestures: Thumb up; Thumb down; Victory. Open palm. Recognition of dynamic gestures: Swipe, circle, waving. 23.09.2013 12

Face detection module (1) The module allows to get a lot of information about the recognized face (7 points on the face): Corners of the eyes; Corners of the mouth; The tip of the nose. 23.09.2013 13

23.09.2013 14 Face detection module (2) The module allows to get some analytical information about the recognized face: Age group of the person who presents in the video stream (a child, an elderly person); Gender; Wink; Smile.

Voice recognition module The module offers the following functionality: Realization of voice menus; Dictation; Speech synthesizes. At the moment only English dictionary is available. Other languages will be added during co-operation with the developers of voice engine. Now for the implementation of voice control voice engine Nuance Dragon Assistant is used. Russian language will be also available soon. 23.09.2013 15

23.09.2013 16 Module for tracking of 2D/3D objects Functionality of this module allows to track 2D/3D objects as well as faces. As the result will be obtained: Position of the object; The angle of inclination, etc. The module supports tracking of three-dimensional objects that are specified in the form of a model (.obj file).

Working with game engines SDK implements several levels of interfaces that allow the immediate use of game engines. At the moment the following game engines are supported: Unity; Processing; openframeworks. 23.09.2013 17

Technology application (1) Video games: Interaction with the threedimensional world and other players; Utilization of voice or gestures to manipulate the game. 23.09.2013 18

Technology application (2) Modeling of grabbing, moving system of virtual 3D object; Pointing the object. 23.09.2013 19

23.09.2013 20 Technology application (3) Interaction with media content: For example, switching tracks in the playlist or flipping through book pages.

Technology application (4) Security systems: Simple person identification (face-login). The combination of face, voice and gesture parameters for more precise identification. 23.09.2013 21

23.09.2013 22 Technology application (5) Image segmentation: For example, to add more colors to the virtual communication. The background can be changed by any image, such as the lunar landscape or ocean beach.

Links Northern (Arctic) Federal University named after M. Lomonosov http://www.narfu.ru Intel Perceptual Computing SDK http://www.intel.com/software/perceptual Demo application http://software.intel.com/en-us/vcsource/tools/perceptual-computingsdk/demos 23.09.2013 23