DESIGN OF A TOUCHLESS USER INTERFACE. Author: Javier Onielfa Belenguer Director: Francisco José Abad Cerdá

Transcription

1 DESIGN OF A TOUCHLESS USER INTERFACE Author: Javier Onielfa Belenguer Director: Francisco José Abad Cerdá 1

2 OUTLINE 1. Problem to solve 2. Goal of the project 3. Connecting devices and the system 3.1 Microsoft Kinect 3.2 Arduino 4. Design of the rule system 4.1 Rules 4.2 Actions 4.3 Example of a data base 4.4 Managing the data base 5. Storing the data 6. Conclusions and future work 7. Demo 2

3 1. PROBLEM TO SOLVE Nowadays, the devices using a touch sensor as input have been widely extended. These devices have included gestures as an input feature. This project will explore the possibility of using touchless gestures as input. 3

4 2. GOAL OF THE PROJECT Design a touchless user interface. It should support a variety of devices. The system will be based on rules and actions. It should allow the user to access the most common windows actions. 4

5 3. CONNECTING DEVICES AND THE SYSTEM The system should be able to support differents input devices. In order to connect the devices and the system a client/server architecture will be used. The client will be placed on the device side. The server will be placed on the computer side. The server should be able to use various devices at the same time 5

6 3. CONNECTING DEVICES AND THE SYSTEM The client/server architecture will be based on TCP. The client/server architecture can be avoided if using only one device. The devices Microsoft Kinect and Arduino will be studied. Arduino Client Kinect Client Internet (TCP/IP) Rule Server 6

7 3.1 MICROSOFT KINECT Microsoft Kinect is a device designed to be used with Xbox 360. The devices packages a videocamera, an infrared camera, an infrared projector and an array of 4 micros. It allows the gamers to use his body as a controller 7

8 3.1 MICROSOFT KINECT The are three major APIs for Microsoft Kinect: Kinect SDK, OpenKinect and OpenNI. The Microsoft Kinect SDK is the official API for Microsoft Kinect. It supports audio, depth maps, skeleton tracking but gestures. OpenKinect is an open source community. It allows a low level access to the device. OpenNI is an general API for natural interfaces. It can be used with Microsoft Kinect. It allows gestures. 8

9 3.1 MICROSOFT KINECT For the project, OpenNI will be used to create the Kinect client. The language will be C++. The choice is due to the fact that OpenNI supports gestures recognition. OpenNI uses a callback system for all the interaction with the user. When a gesture is recognized a callback function is called. The client will recognize the gesture and send it to the server. 9

10 3.2 ARDUINO Arduino is an open-source electronics protoyping plataform. It based on an easy-to-use hardware and software. It allows the user to connect sensors to it. Arduino is programmed using the Arduino programming language (very similar to c++). 10

11 3.2 ARDUINO Arduino provides a flash memory where the program will be loaded. The program will be execute as Arduino is plugged in. The language allows the user to read and write the pins. As it is similar to C++ it also provides all the possibilities of an imperative language ( functions, loops, operators,etc). It can communicate from/to a computer using a virtual serial port on top of a USB port. 11

12 3.2 ARDUINO Danger Shield is an add-on for Arduino. It is designed to be connected to Arduino It is a fully self-contained shield. 12

13 3.2 ARDUINO It provides: 3 linear sliders with integrated LEDs 3 pushbuttons 2 LEDs 1 piezo buzzer 1 temperature sensor 1 light sensor 1 knock sensor 1 7-segment LED 13

14 3.2 ARDUINO In order to create an Arduino client, we will follow the next steps: On Arduino side: The program will recognize the gesture using the input sensors. The gesture name will be sent by the serial port to the computer. On computer side: A program will read from the serial port and send the data to the server. 14

15 4. DESIGN OF THE SYSTEM The server side, from now called the system, will be design based on rules and actions. A rule represents a specific situation of the operating system and the device. A rule can have actions linked to it. When a rule is validated, the list of actions linked to it will be executed. Every actions represents an operation of the user to the operating system. 15

16 4. DESIGN OF THE SYSTEM When the server recieves a gesture, it will execute it. The server will look for the first rule that is validated. 16

17 4.1 RULES A rule represents a situation in the operating system. It will have 3 parameters: Gesture. Name of the gesture linked to the rule. Application. Rute of the application linked to the rule. Foreground. Boolean value that indicates if Application should have the focus. A rule is validated when the conditions represented in the rule are valid. 17

18 4.2 ACTIONS An action is an object that represents an operation on the operating system. It has to be linked to a specific rule. The actions can be: Open, close, maximize, minimize an application. Showing the desktop Controlling the sound volume of the OS. Sending key strokes and hotkey strokes. 18

19 4.3 EXAMPLE OF A DATA BASE Example of a database of rules and actions: Rule ID:1 Gesture: Wave App: Internet Explorer Foreground: True Send key stroke F5 Rule ID:2 Gesture: Click App: Windows Media Player Foreground: True Send HotKey CTRL +P Rule ID:3 Gesture: Wave App: None Foreground: False Show desktop Mute sound volume 19

20 4.4 MANAGING THE DATA An application for managing the data have been develop. The user can visually design the rule system. 20

21 5. STORING THE DATA In order to store the data the XML format will be used. The XML format has been chosen because it is easy to represent the data and there are different free parsers in the market. XML is a format for encoding the data using tags. Each of the object of our system, rules and actions, will be represented as a tag. For the parser, the solution chosen is Pugixml. 21

22 6. CONCLUSIONS AND FUTURE WORK As a result, we have a system that can use a different set of devices. The user can determine easily the different situations. Most of common action in an OS are represented. As future work: Make the system able to be compile for other OS. Possibility to include serveral rule systems in the same file. 22

23 DEMO 23