Real Time Eye Tracking and Mouse Control for Physically Disabled

Transcription

1 Real Time Eye Tracking and Mouse Control for Physically Disabled Sourabh Kanwar VIT University Keywords: Glint, Mouse control, ROI, Tracking Abstract: In the cases of paralysis a person s ability to move may be limited to the muscles around the eyes, which may be the only way for him to communicate. We developed a computationally efficient and cost effective application for controlling the mouse cursor using the eye ball by real time tracking using a USB camera. Eye ball movement can be followed by obtaining the image of the eye. The glint in the eye acts as the Region of Interest and can be located in the real time images of the eye. We begin with the extraction of human face and its features from a video stream. These extracted features, are tracked and monitored to detect any movements which form the basis of various actions. The LabVIEW VI gives the coordinates of the glint which are input the cursor to change its position in response to the movement of the eye. 1. Introduction For people with severe paralysis, communication abilities are extremely limited often to yes and no responses using eye movements or blinks, since the eye muscles are the only muscles they can control. The goal of this paper is to help stop an active mind from being trapped in a body that has difficulty communicating. As progress toward that goal, we created a camerabased computer interface that makes communication possible by detecting the eye-gaze direction and simulating computer mouse input. Existing methods for eye detection and tracking, as well as gaze analysis that do not rely on infrared lighting typically depend on high-resolution images of eyes, often obtained by pan/tilt/zoom cameras. First, note that our goal is to measure visual line of gaze, that is, the absolute position in space at which the user s eyes are pointed, rather than, for example, the position of the eye in space or the relative motion of the eye within the head; not all eye tracking techniques do this. Since both eyes generally point together, it is customary to track only one eye. The system proposed here was designed around an inexpensive visible-light webcam, and it works with images that have a lower resolution than the images used in previous approaches (e.g., [1] and [2]). These approaches require high-resolution images of the eyes for appearance- or feature based methods that locate, for example, eye corners, eyelids, or contours of the iris. The use of webcams also allows the system to be made available to people at significantly less expense than those requiring specialized hardware.this application aims to replace the most common hardware used today, the mouse. The features of the eye affect the task of tracking eye movements, determining the position of the eye and inferring the point or object regarded. In building a system for this task, it is necessary to simplify and use workable models of the eye. It is important to judge these simplifications and the performance of a system for a given application, and regard every measured characteristic separately. ICL 2009 Proceedings - Page 939 1(5)

2 2. Proposed Solution 2.1. Configuration We use an PC with LabVIEW installed in it. The user sits in front of a computer display, mouse, and keyboard. The camera/illuminator is attached to the user in the inverted position with a cap. In addition, the room lights are not to bright to keep the user s pupil from becoming too small. The eye tracker transmits the x and y coordinates for the user s visual line of gaze every 1/30 second, on a USB port. The computer performs all of the further processing. One further addition is that the system when to be set up requires two people, one is the subject whose eye is to be tracked and an experimenter who monitors and adjusts the equipment. Operation by a single user playing both roles simultaneously initially is somewhat awkward because, as soon as you look at the eye tracker control panel to make an adjustment, your eye is no longer pointed where it should be for tracking. After setting up the entire system the subject can work independently. 2.2 Pre processing of eye image In order to track a facial feature it must be first identified in an image stream. The initialization process starts with the location of face in the first image. Skin pixels can have very different RGB values for different people depending on their skin color, illumination etc, however their chromatic red and chromatic blue values defined as chr = R/(R+G+B) and chb = B/(R+G+B) are clustered in a limited region of (chr, chb) space and are largely independent of skin color or lighting conditions. Based on their chromatic red and chromatic blue values, the pixels are classified into skin or non skin pixels. After face segmentation the image is subject to blob analysis to locate the exact position of the eyes. The technique makes use of the fact that eyes when open have color different from skin and therefore appear as holes in the face, in the skin segmented image. These holes give the approximate location of the eyes. These locations are then used to track the eyes in the remaining video stream. The purpose of pre processing is to minimize the number of on-pixels in the binary image. An ideal resulting image will only contain a circle representing the outline of the glint, however it is difficult to obtain and attempt is made to remove unwanted pixels without removing the eyeball pixels. The pre processing steps are: Color filtering: Three channel RGB image is converted to a single channel image for edge detection. It has been found experimentally [3] that green channel image provides best results for edge detection even though difference with other channels is not huge. This is done using IMAQ ExtractSingleColorPlane.vi and setting the color priority as green Histogram equalization: The single channel image is equalized to compensate for different lighting conditions. Since eye image contains two regions of comparable areas and distinct intensities, one representing the eyeball and the other white area around it, equalization tends to highlight the edges of the eyeballs Edge detection: The green channel image is then subjected to edge detection to get an outline of the eyeball. Since the eyeball pixels have different intensity than the white pixels around it an edge is readily obtained however it is occluded. A 3 by 3 vertical Prewitt operator is used for this purpose. Horizontal Prewitt operator was also tried but since the outline of the eyes form horizontal edges, extra edges appear in the edge image. Vertical operator returns the vertical ICL 2009 Proceedings - Page 940 2(5)

3 edges as all the pixels are not required to obtain the circle outlining the glint. The IMAQ EdgeDetection.vi helps to implement the Prewitt Filter. 4. Particle Removal: Particle analysis is done to find statistical information such as the presence of particles, their number and size, and location and to perform vision inspection such as detecting flaws. Here we use it to locate the glint in the eye and to eliminate particles that are of no interest. All this is obtained by thresholding using IMAQ Threshold.vi. At first, the image is binarized by a proper threshold value. It is important to find the proper threshold value in order to separate the eyes, nose and eye brows from the face. The threshold Td is used to guarantee a minimal spatial difference between the minimum and maximum values when motion is detected. This helps prevent motion that is not an intentional eye movement from being detected. There are many methods to find the threshold value. We employed a manual threshold method. The threshold interval is defined by two parameters i.e. the Lower Threshold (>=) and Upper Threshold (<=). Then we include the IMAQ RemoveParticle.vi to eliminate or keep particles resistant to a specified number of 3 3 erosions. The particles that are kept are exactly of the same shape as those found in the original source image. The connectivity 4/8 specifies the algorithm for particle detection. The connectivity mode directly determines whether an adjacent pixel belongs to the same particle or a different particle. We choose connectivity-4 so that pixels adjacent in the horizontal and vertical directions are considered neighbors to increase the efficiency. The pixel frame is treated as hexagonal during the transformation. Finally the numbers of erosions are specified as 3 3 erosions to apply to the image. The default is 2 but we set it to 1. It reduces the size of an object along its boundary and eliminates isolated points in the image. The erosions eliminate pixels isolated in the background and erode the contour of particles according to the template defined by the structuring element. A high pass filter emphasizes significant variations of the light intensity usually found at the boundary of objects. High pass frequency filter helps to isolate abruptly varying patterns that correspond to sharp edges, details, and noise in the acquired image of the eye. In order to determine the location of the glint in the processed image a range for the radius of the desired circle is specified. Each threshold pixel in the binary image is examined to determine whether it fits the desired circle. The best circle gives the outline of the glint. The technique is very powerful since it can fit circles even when a partial outline of a circle is available which is common in an eye image due to the occluded eyeball. Figure 3 shows the extraction of glint from eye image using the pre-processing algorithm. The image has been equalized and vertical Prewitt operator has been used to get the approximate outline of the eyeball. The final image is obtained using the IMAQ CastImage.vi which converts the current image type to Grayscale (SGL) having 32 bits resolution. 5. Simulated Results The final image is obtained using the IMAQ CastImage.vi which converts the current image type to Grayscale (SGL) having 32 bits resolution. Fig 4. Pre processing of eye image for eyeball detection ICL 2009 Proceedings - Page 941 3(5)

4 3. Real Time Implementation The algorithm can be used to locate glint in each eye image thus determining its location. This technique however is computationally The advantage of this algorithm on the other hand is that we do not have to worry about the loss of tracking. If the If it fails to find a circle in one image, this will not affect the transform computation in the subsequent images. The ROI could be lost due to a blink or a rapid movement of the eyeball or eye itself. We address this problem by constantly monitoring the tracked ROI to determine if the tracking has been lost, in which case The algorithm is used to re initialize the glint location. The best feature points for the glint are the center of the eyeball and its two outer edges. The relative distance among these three points should remain constant as the eyeball moves within the eye. After obtaining the processed image of the eye we select the Region of Interest (ROI) glint to help with further processing. The ROI can be selected using any of the tools present like a rectangle, circle or an annulus. This application uses an annulus to select the ROI as it is independent of the shape chosen. The ROI when selected is converted into a template which has to be searched during the matching phase. The template image contains the data that defines the color template pattern for the matching stage. This data is appended to the input color template image. During the color pattern matching step, the color template descriptor is extracted from the color template image and used to search for the template in the color match image. The real time images of the eye, being input at high speed, or the tracking video is processed using the same algorithm mentioned above. After processing the input video, also gives the feature point or the glint and helps in continuously monitoring it. The IMAQ Find Pattern 2.vi helps to implement the pattern matching technique in which we compare the input template image having glint as the ROI and the real time video of the eye. The rotation invariant technique is to look for the template pattern in the image. The minimum match score, the smallest score so that a match must have to be considered valid can be varied depending on how accurate the user wants the system to be. We get the tracked glint specified with a red mark in the output image along with the position where it is found. The position is in terms of the coordinates with respect to a reference plane. These coordinates can be viewed in a graph. The graph shows the dynamic movement of the glint as the eye moves. In our application we give these coordinates as input to the mouse cursor coordinates. 4. Mouse Control The coordinates obtained by tracking the eye movement are act as input to the CodeInterfaceNode where we can load a program in any of the programming languages like C++ Java (AWT) or.net (mouse control package). Here we use a program in C++ which takes these coordinates when invoked. On execution the application makes the cursor move with the help of the eye independent of the desktop. References: [1] M. Betke and J. Kawai, Gaze detection via self-organizing gray-scale units, in Proc. Int. Workshop Recog., Anal., Tracking Faces Gestures Real-Time Syst., Kerkyra, Greece, Sep. 1999, pp [2] Z.-H. Zhou and X. Geng, Projection functions for eye detection, Pattern Recognition., vol. 37, no. 5, pp , May [3] Yassir Nawaz and Shiladitya Sircar Real Time Eye Tracking and Blink Detection Memorial University, St. John s, NF, CANADA A1B 3X5 unpublished. [4] Arne John Glenstrup and Theo Engell- Nielsen. Eye controlled media: Present and futurestate. [Last Accessed Aug. 12, 2005]. [5] Andrew T. Duchowski. Eye Tracking Methodology. Theory and Practice.Springer-Verlag, [6] Canny, 1986 a computational approaches to edge detection. IEEE Trans. PAMI, 8: ICL 2009 Proceedings - Page 942 4(5)

5 Author(s): Sourabh, Kanwar, Mr Vellore Institute of Technology University, Electronics and Instrumentation J-437,John F Kennedy Block VIT University Men s Hostel VIT University Vellore,Tamil Nadu,India sourabh.kanwar@gmail.com ICL 2009 Proceedings - Page 943 5(5)