Moving the Mouse Pointer Using Eye Gazing Computer Science Honours Documentation Ibraheem Frieslaar 2734760 Supervisors: Mr Mehrdad Ghaziasgar and Mr James Connan Department of Computer Science University of the Western Cape
ABSTRACT: With a growing number of computer devices around us, and the increasing time we spend for interacting with such devices, we are strongly interested in finding new interaction methods which ease the use of computers or increase interaction efficiency. Eye tracking seems to be a promising technology to achieve this goal. This project intends to use eye gazing techniques to allow a user to move the mouse pointer using his eyes.
TABLE OF CONTENTS ABSTRACT:... 2 CHAPTER 1... 5 INTRODUCTION... 5 1.1 Human computer interaction (HCI):... 5 1.2 Current Research:... 5 CHAPTER 2... 8 USER REQUIREMENTS DOCUMENT... 8 2.1 Users view of the problem... 8 2.2 Description of the problem... 8 2.3 Expectations from the software solution... 9 2.4 Not expected from the software solution... 9 CHAPTER 3... 10 REQUIREMENTS ANALYSIS DOCUMENT... 10 3.1 Designer's interpretation... 10 3.2 Complete Analysis of the problem... 10 3.3 Current Solution:... 11 3.5 Suggested Solution:... 11 CHAPTER 4... 12 USER INTERFACE SPECIFICATION (UIS)... 12 CHAPTER 5... 13 HIGH LEVEL DESIGN... 13 5.1 High Level Data Dictionary... 13 5.2.1 Relationship between objects... 15 5.2.2 High Level Design of Eye Gazing and Mouse Movement Diagram... 16 CHAPTER 6... 17 LOW LEVEL DESIGN... 17 6.1 Low Level Data Dictionary... 17
6.2 Low Level Design of Eye Gazing and Mouse Movement Diagram... 19 6.3 Detailed Methodology... 20 6.3.1 Haar-like feature... 20 6.3.2 Greyscale... 21 6.3.3 SVM... 21 LIST OF FIGURES... 22 REFERANCE... 23
CHAPTER 1 INTRODUCTION 1.1 Human computer interaction (HCI): HCI is the study, planning and design of the interaction between users and computers. It is often regarded as the intersection of computer science, behavioural sciences, design and several other fields of study [1]. Interaction between users and computers occurs at the user interface. There are new modalities for computer interaction like speech interaction, input by gestures or by tangible objects with sensors. A further input modality is eye gaze which nowadays finds its application in accessibility systems. Such systems typically use eye gaze as the sole input, but outside the field of accessibility eye gaze can be combined with any other input modality. Therefore, eye gaze could serve as an interaction method beyond the field of accessibility. The aim of this work is to find new forms of interactions utilizing eye gaze and suitable for standard users. 1.2 Current Research: An eye-gaze interface seems to be a promising candidate for a new interface technique, which may be more convenient than the ones we use. Traditionally, disabled people who cannot move anything except their eyes use eye gaze interaction. These systems are designed to direct the computer solely by the eyes. Such systems work well and are a great help for people who need them, but for others they are cumbersome and less efficient than keyboard and mouse. An eye-gaze interface might offer several potential benefits: I. A benefit of eye tracking could be reduced stress for hand and arm muscles by transferring the computer input from the hand to the eyes. This need not necessarily put
extra load on the eye muscles because for most interactions the eyes move anyway. For example, when clicking a button on the screen in most cases mouse and eyes move to the target. [2] II. Video-based eye tracking works contact free which means that no maintenance is necessary. There is no need to clean the device, which is a typical problem for keyboards and mouse devices. Placing the camera behind strong transparent material results in a vandalism-proofed interface, which is nearly impossible to realize for keyboards and mouse devices. III. The eyes tell a lot about what somebody is doing. Tracking the eyes provides useful information for context-aware systems. In the simplest form an eye tracker tells where the attention is, which already has a big potential for the implementation of contextawareness. Simple analysis of the eye tracker data can detect activities like reading. Analysis that is more sophisticated could reveal the physical or emotional condition of a user, her or his age, and degree of literacy. [2] There are also possible problems: I. The eyes perform unconscious movements and this might disturb their use as computer input. It is not clear to which degree people are able to control the movement of their eyes. The ability to control the eyes consists of both suppressing unintended movements and performing intended eye movements. It seems that we are at least able to control where we look because this is required by our social protocols. However, it is not clear whether we can train the motor skills of the eye muscles to the same extent as we can train the fingers for playing the guitar [2].
II. Misinterpretation by the gaze interface can trigger unwanted actions wherever we look. The situation is similar when triggering actions by eye movements, i.e. gestures. The eyegaze interface has to separate natural eye movements from intentional gaze gestures. III. From other input devices we know that extensive use of particular muscles or muscle groups can cause physical problems called RSI (repetitive strain injury) [7]. There are fears that this might happen to the eye muscles too. In this project we intend to capture the eye gaze location immediately once the user has correctly placed his face in front of the web camera.
CHAPTER 2 USER REQUIREMENTS DOCUMENT This chapter focuses on viewing the problem from the user s perspective. The solution is based on the vital information acquired from the user. 2.1 Users view of the problem The user requires a system that should determine where the user is looking with a high degree of accuracy. High accuracy is the ability for the user to place the mouse within 32 pixels of the actual location, since this is a standard icon size. The proposed hardware and software requirements are: i. A web camera ii. Windows 7 Operating System iii. Visual Studio 2010 iv. Open Source Computer Vision Library (Opencv libraries) v. Library for Support Vector Machines (LIBSVM) 2.2 Description of the problem Disabled people who cannot move anything except their eyes would need a system where they will be able to use their eyes to move the mouse to the position where the eyes is gazing. However people who have no disabilities would also want such a system.
2.3 Expectations from the software solution The user requires a system that should determine where the user is looking with a high degree of accuracy. The system should run in the background and track the eyeball without user intervention and move the mouse to the appropriate position according to their eye gaze. 1) Image Acquisition A digital image is produced by a web camera. 2) Image processing and analysis Acquire the users eyes within the image 3) Feature extraction Regions of interest (ROI) to determine user s left eye 4) Detection Eye detection is done and its location is stored and sent to the SVM 5) Object tracking Using the Support Vector Machine (SVM) we can determine the movement of the eyes. 2.4 Not expected from the software solution It is not expected to be able to process the image with more than one human head in the image at a time. People wearing spectacles will not be supported.
CHAPTER 3 REQUIREMENTS ANALYSIS DOCUMENT 3.1 Designer's interpretation The eyes perform unconscious movements and this might disturb their use as computer input. It is not clear to which degree people are able to control the movement of their eyes. The difficulty lies within identifying the eye correctly. The SVM needs accurate data to make the correct decision. 3.2 Complete Analysis of the problem I. A web camera is required to capture the users face in real time. II. From the image, Haar-like features are used to determine the eyes location. III. Having the eyes location we then set a ROI over the left eye. IV. We then detect the true location of the eye within the ROI using Hough Circles. V. That location is stored and drawn onto a white image as a black circle. VI. The pixels of that image is stored and sent to the SVM VII. The SVM determines which direction the user is looking VIII. Based on the SVM s prediction we can move the mouse.
3.3 Current Solution: I. Lenovo s eye-controlled notebook. Developed in conjunction with Swedish-based Tobii, this computer is the first fully functional prototype of Tobi si eye-tracking technology that allows users to zoom, automatically centre on what they re focusing on, auto-dim and brightens the display, scroll through documents with their eyes. [3] II. Sweden's eye tracking and control innovator Tobii has release a stand-alone eye control device called PCEye. The device tracks eye movement and translates it into mouse cursor action onscreen. Positioned in front of a PC monitor and connected via USB. [4] 3.5 Suggested Solution: The suggested solution aims to be a cost effective. All you would need is a basic web camera and the Opencv libraries installed
CHAPTER 4 USER INTERFACE SPECIFICATION (UIS) The UIS is a daemon which runs in the background, rather than under the direct control of a user, it is initiated as background processes.
CHAPTER 5 HIGH LEVEL DESIGN In this chapter we will look at it from a high level of abstraction, while the low level view will follow in the next chapter. Since the system was programmed in C/C++ we do not have an Object Oriented Analysis so we will not include a class diagram. 5.1 High Level Data Dictionary An object is given with its description to help define the diagrams that follow. Object Description OpenCV Opencv is the open source library used for this purpose. Its main focus is real-time image processing. Haar-like features A Haar-like feature considers adjacent rectangular regions at a specific location in a detection window, sums up the pixel intensities in these regions and calculates the difference between them. This difference is then used to categorize subsections of an image. [5] Region of Interest (ROI) Region of Interest is a rectangular area in an image, to segment object for further processing. Greyscale It is used to convert colour images to its
greyscale equivalent. The conversion, to a shade of grey from a colour image, is established by calculating the effective brightness or luminance of the colour. This value is then used to create the shade of grey that corresponds to the desired brightness. Locate Iris Finds the left Iris in image. Support Vector Machine (SVM) A set of related supervised learning methods that analyse data and recognize patterns, used for classification and regression analysis. The standard SVM takes a set of input data and predicts, for each given input, which possible classes the input is a member of. [6] Mouse Movement The mouse pointer will be moved using x and y coordinates
5.2.1 Relationship between objects In the figure below the relationship between the objects are identified. Original Frame 1 : 1 Haar Like Features 1 : 1 ROI 1 : 1 Mouse Movement 1 : 1 SVM 1 : 1 Locate Iris Position Figure 1: Relationships between objects
5.2.2 High Level Design of Eye Gazing and Mouse Movement Diagram In this figure the key component to the system is slightly elaborated. It shows a high-level view of how the eyes are tracked in the system using computer vision methods. Input Webcam Image Processing Move Mouse Pointer Figure 2 High Level Design of System
CHAPTER 6 LOW LEVEL DESIGN 6.1 Low Level Data Dictionary Class Attributes Haar-like features Using HaarDetectObjects we are able to detect the eyes within the frame. ROI We then use SetImageROI to set a region of interest over the left eye and its stored in eyeroi Greyscale eyeroi is then converted to greyscale and stored in the new image grey. HoughCircles HoughCircles are used to locate circles within the grey image Pixel Calculation The circles that have been found are looped through and all the pixels are counted. The circle with the darkest pixel value will be the eye. DarkPixels = DarkPixels + (255-s.val[0])
Eye Re-plotting The Correct circle is redrawn as a black circle onto a white image, which is saved in image trueeye. SVM The Pixels of image trueeye is sent to the SVM where it determines what position the eye is. Mouse Movement SetCursorPos(x,y) will Moves the mouse based on the SVM predication
6.2 Low Level Design of Eye Gazing and Mouse Movement Diagram Input Webcam Image Processing Move Mouse Pointer Haar-like features ROI Greyscale Pixel Calculation Eye Replotting Hough Circles SVM Figure 3 Low Level Design of System
6.3 Detailed Methodology We will now discuss the methodology used to create this system by elaborating on the following key computer vision techniques used: I. Haar-like features II. Greyscale III. SVM 6.3.1 Haar-like feature Viola and Jones [5] adapted the idea of using Haar wavelets and developed the so called Haarlike features. A Haar-like feature considers adjacent rectangular regions at a specific location in a detection window, sums up the pixel intensities in these regions and calculates the difference between them. This difference is then used to categorize subsections of an image. The key advantage of a Haar-like feature over most other features is its calculation speed; it minimizes computation time while achieving high detection accuracy. [5]
6.3.2 Greyscale When we have the captured frame we use this method to convert colour images to its greyscale equivalent on the fly. The greyscale image shows the effective brightness or luminance of the colour image. Red, green and blue values are multiplied by 0.3, 0.59 and 0.11 respectively. The sum of the product is calculated to give the single pixel value, which is greyscale. [8] Since our system needs to run in real-time, we need to use techniques that process faster than it would have taken to process the original frame. Greyscale is also necessary because since its only one value rather than three, it is easier to manipulate the frames. [8] 6.3.3 SVM SVM s are a useful technique for data classification. It s a set of related supervised learning methods that analyse data and recognize patterns, used for classification and regression analysis. The standard SVM takes a set of input data and predicts, for each given input, which possible classes the input is a member of. [6]. The SVM will allow us to accurately predict which region the users are gazing at.
LIST OF FIGURES Figure 1: Relationships between objects... 15 Figure 2 High Level Design of System... 16 Figure 3 Low Level Design of System... 19
REFERANCE [1] Baecker, Card, Carey, Gasen, Mantei, Perlman, Strong and Verplank Hewett, "ACM SIGCHI Curricula for Human-Computer Interaction," ACM SIGCHI, ISBN 0-89791-474-0, 2009. [2] Heiko Drewes, "Eye Gaze Tracking for Human Computer Interaction," Ludwig-Maximilians- Universität, Munich, 2010. [3] Chris Ziegler. (2011, April) Engadget. [Online]. http://www.engadget.com/2011/03/01/tobii-andlenovo-show-off-prototype-eye-controlled-laptop-we-go/ [4] Tobii Technology. (2011, April) Tobii Technology. [Online]. http://www.tobii.com/pceye [5] Paul Viola and Michael Jones, "Rapid Object Detection using a Boosted Cascade of Simple Features," in ACCEPTED CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, 2001. [6] John Shawe-Taylor & Nello Cristianini, Support Vector Machines and other kernel-based learning methods. [7] James E. McGreevey, CUMULATIVE TRAUMA DISORDERS IN OFFICE WORKERS, 2003, Public Employees Occupational Safety and Health Program. [8] Codeback. Codeback. [Online]. http://codeback.net/color-graphics-to-grayscale-algorithm