Eye Gaze Tracking Under Natural Head Movements

Transcription

1 Eye Gaze Tracking Under Natural Head Movements Zhiwei Zhu and Qiang Ji Department o ECE, Rensselaer Polytechnic Institute, Troy, NY,12180 {zhuz,jiq}@rpi.edu Abstract Most available remote eye gaze trackers based on Pupil Center Corneal Relection (PCCR) technique have two characteristics that prevent them rom being widely used as an important computer input device or human computer interaction. First, they must oten be calibrated repeatedly or each individual; econd, they have low tolerance or head movements and require the user to hold the head uncomortably still. In this paper, we propose a novel solution or the classical PCCR technique that will simpliy the calibration procedure and allow ree head movements. The core o our method is to analytically obtain a head mapping unction to compensate head movement. peciically, the head mapping unction allows to automatically map the eye movement measurement under an arbitrary head position to a reerence head position so that the gaze can be estimated rom the mapped eye measurement with respect to the reerence head position. Furthermore, our method minimizes the calibration procedure to only one time or each individual. Our proposed method will signiicantly improve the usability o the eye gaze tracking technology, which is a major step or eye tracker to be accepted as a natural computer input device. 1 Introduction Eye Gaze is deined as the line o sight o a person. It represents a person s ocus o attention. Eye gaze tracking has been an active research topic or many decades because o its potential usages in various applications such as Human Computer Interaction, Eye Disease Diagnosis, Human Behavior tudy, etc. Earlier eye gaze trackers were airly intrusive in that they require physical contacts with the user, such as placing a relective white dot directly onto the eye [1] or attaching a number o electrodes around the eye [2]. Except the intrusive properties, most o these technologies also require the viewer s head to be motionless during eye tracking. With the rapid technological advancements in both video cameras and microcomputers, eye gaze tracking technology based on the digital video analysis o eye movements has been widely explored. ince it does not require anything attached to the user, the video technology opens the most promising direction to build a non-intrusive eye gaze tracker. Based on the eye images captured by the video cameras, various techniques [3], [4], [5], [6], [7], [8], [9] have been proposed to do the eye gaze estimation. Yet most o them suer rom two problems, the need o calibration per user session and the large restriction on head motion [10]. Among them, the PCCR technique is the most commonly used approach to perorm the video-based eye gaze tracking. The angle o the visual axis (or the location o the ixation point on the display surace) is calculated by tracking the relative position o the pupil center and a speck o light relected rom the cornea, technically known as glint as shown in Figure 1. The accuracy o the system can be urther enhanced by illuminating the eyes with low-level Inrared light, which produces the bright pupil eect as shown Figure 1. It makes the video image easier to process. glint Fig. 1. Eye image with corneal relection (or glint). everal systems [11], [12], [3], [4], [10], [8] have been built based on the PCCR technique. Most o these systems show that i the user has the ability to keep his head ixed or via the help o a chin rest or bite bar to restrict the head motion, very high accuracy o the eye gaze tracking results can be achieved. peciically, the average error can be less than 1 visual angle, which corresponds to less than 10 mm in the computer screen when the subject is sitting around 550 mm rom the computer screen. But as the head moves away rom the original position where the user perormed the eye gaze calibration, the accuracy o these eye gaze tracking systems drops dramatically. ome detailed data [10] are reported about how the calibration mapping unction decays as the head moves away rom its original position. The current solution to overcome this problem is perorming another new calibration whenever the head moves. This, however, is practically impossible since a person s head constantly moves. Thereore, the limited head movement is one o the worst issues in the current remote eye tracking systems. Jacob reports a similar act in [12], and tried to solve the problem by giving the user the possibility o making local manual re-calibration, which brings numerous troubles or the user. It is obvious that most o the existing gaze tracking systems based on PCCR technique share two common drawbacks: irst, the user must perorm certain experiments in calibrating the user-dependent parameters beore using the gaze tracking system; second, the user must keep his head uncomortably still, no signiicant head movements allowed. In this paper, a solution is proposed to handle these two issues. First, beore using the system, a calibration procedure

2 is perormed or each user to obtain a gaze mapping unction. Ater the calibration, the user does not need to calibrate the system again. econd, completely dierent rom previous eye gaze trackers, the user can move his head reely in ront o the camera, which will make the communications between the computer and user more naturally. Thereore, by using our gaze tracking technique, a more robust, accurate, comortable and useul system can be built. 2 Classical PCCR Technique The PCCR based technique consists o two major components: pupil-glint vector extraction and gaze mapping unction acquisition. 1. Pupil-glint Vector Extraction Gaze estimation starts with the pupil-glint vector extraction. Ater grabbing the eye image rom the camera, computer vision techniques [9], [13] are proposed to extract the pupil center and the glint center robustly and accurately. The pupil center and the glint center is connected to orm a 2D pupilglint vector v as shown in Figure peciic Gaze Mapping Function Acquisition Ater obtaining the pupil-glint vectors, a calibration procedure is proposed to acquire a speciic gaze mapping unction that will map the extracted pupil-glint vector to the user s ixation point in the screen or current head position. The extracted pupil-glint vector v is represented as (v x,v y ) and the screen gaze point s is represented by (x gaze,y gaze ) in the screen coordinate system. The speciic gaze mapping unction s = (v) can be modelled by the ollowing nonlinear equations [8]: x gaze = a 0 + a 1 v x + a 2 v y + a 3 v x v y 2 (1) y gaze = b 0 + b 1 v x + b 2 v y + b 3 v y The coeicients a 0,a 1,a 2,a 3 and b 0,b 1,b 2,b 3 are estimated rom a set o pairs o pupil-glint vectors and the corresponding screen gaze points. These pairs are collected in a calibration procedure. During the calibration, the user is required to visually ollow a shining dot as it displays at several predeined locations on the computer screen. In addition, he must keep his head as still as possible. I the user does not move his head signiicantly ater gaze calibration, the speciic gaze mapping unction can be used to estimate the user s gaze point in the screen accurately based on the extracted pupil-glint vector. But when the user moves his head away rom the position where the speciic gaze calibration is perormed, the speciic gaze mapping unction will ail to estimate the gaze point accurately because o the pupilglint vector changes caused by the head movement. In the ollowing, the head movement eects on the pupil-glint vector will be illustrated. 2.1 Head Motion Eects on Pupil-glint Vector Figure 2 shows the ray diagram o the pupil-glint vector generation in the image when an eye is located at two dierent } 3D positions and in ront o the camera due to head movement. For simplicity, the eye is represented by a cornea, the cornea is modelled as a convex mirror and the IR light source used to generate the glint is located at O, which are applicable to all the subsequent igures. Assume that the origin o the camera is located at O, p 1 and p 2 are the pupil centers and g 1 and g 2 are the glint centers generated in the image. Further, at both positions, the user is looking at the same point o the computer screen. According to the light ray diagram shown in Figure 2, the generated pupil-glint vectors 1 p 1 and g 2 p 2 will be signiicantly dierent in the images as shown in Figure 3. Two actors are responsible or this pupilglint vector dierence: irst, the eyes are at dierent positions in ront o the camera; second, in order to look at the same screen point, eyes at dierent positions rotate themselves dierently. Camera Z Axis Cornea G1 G2 O g 1 p 1 g 2 p 2 Image Plane creen Plane Fig. 2. Pupil and glint image ormations when eyes are located at dierent positions while gazing at the same screen point (side view). p g 1 1 p g 2 2 (a) eye image at location (b) eye image at location Fig. 3. The pupil-glint vectors generated in the eye images when the eye is located at and in Figure 2. The eye will move as the head moves. Thereore, when the user is gazing at a ixed point on the screen while moving his head in ront o the camera, a set o pupil-glint vectors in the image will be generated. These pupil-glint vectors are signiicantly dierent rom each other. I uncorrected, inaccurate gaze points will be estimated ater inputting them into the speciic gaze mapping unction obtained at the reerence eye position. Thereore, the head movement eects on these pupil-glint vectors must be eliminated in order to utilize the speciic gaze mapping unction to estimate the screen gaze points accurately. In the ollowing, a technique is proposed to eliminate the head movement eects on these pupil-glint vectors beore inputting them into the speciic gaze mapping unction. With this technique, accurate gaze screen points can be estimated accurately whether the users has moved his head or not.

3 3 Dynamic Head Compensation Model 3.1 Approach Overview The irst step o our technique is to ind a speciic gaze mapping unction O1 between the pupil-glint vector v 1 and the screen coordinate at a reerence 3D eye position.thisis usually achieved via a gaze calibration procedure using equations 1. The unction O1 can be expressed as ollows: = O1 (v 1 ) (2) pupil-glint vectors g 1 p 1 and g 2 p 2 are generated in the image. Further, as shown in Figure 4, a parallel plane A o the image plane that goes through the point will intersect the line O at G 1 1. Another parallel plane B o the image plane that goes through the point will intersect the line O at G 2 2. Thereore, g 1 p 1 is the projection o the vector G 1 and 2 p 2 is the projection o the vector G 2 in the image plane. Because plane A, plane B and the image plane are parallel, the vectors g 1 p 1, g 2 p 2, G 1 and G 2 can be represented as 2D vectors in the X Y plane o the camera coordinate system. Assume that when the eye moves to a new position as the head moves, a pupil-glint vector v 2 will be created in the image while the user is looking at the same screen point. When is signiicantly dierent rom, v 2 can not be used as the input o the gaze mapping unction O1 to estimate the screen gaze point due to the changes o the pupilglint vector caused by the head movement. I the changes o the pupil-glint vector v 2 caused by the head movement can be eliminated, then a corrected pupil-glint vector v 2 will be obtained. Ideally, this corrected pupil-glint vector v 2 is the generated pupil-glint vector v 1 o the eye at the reerence position when gazing at the same screen point. Thereore, this is equivalent to inding a head mapping unction g between two dierent pupil-glint vectors at two dierent head positions when still gazing at the same screen point. This mapping unction g can be written as ollows: creen Plane X p Image Plane 2 Y Plane A G 1 G 1 Plane B G 2 O s o p 1 c g 2 g 1 O G 2 creen Gaze Point v 2 = g(v 2,, ) (3) where v 2 is the equivalent measurement o v 1 with respect to the initial reerence head position. Thereore, the screen gaze point can be estimated accurately rom the pupil-glint vector v 2 via the speciic gaze mapping unction O1 as ollows: = O1 (g(v 2,, )) = F (v 2, ) (4) where the unction F can be called as a generalized gaze mapping unction or a computational dynamic head compensation model. It provides the gaze mapping unction or a new eye position dynamically. With the use o the proposed technique, whenever the head moves, a gaze mapping unction at each new 3D eye position will be estimated automatically, thereore, the issue o the head movement can be solved through g. 3.2 Head Movement Compensation In this section, we show how to ind the head mapping unction g. Figure 4 shows the process o the pupil-glint vector ormation in the image or an eye in ront o the camera. When the eye is located at two dierent positions and while still gazing at the same screen point, two dierent Z Axis o Camera Fig. 4. Pupil and glint image ormation when the eye is located at dierent positions in ront o the camera Image Projection o Pupil-glint Vector Assume that in the camera coordinate system, the 3D pupil centers and are represented as (x 1,y 1,z 1 ) and (x 2,y 2,z 2 ), the glint centers g 1 and g 2 are represented as (x g1,y g1, ) and (x g2,y g2, ), where is ocus length o the camera, and the screen gaze point is represented by (x s,y s,z s ). Via the pinhole camera model, the image projection o the pupil-glint vectors can be expressed as ollows: 1 p 1 = G 1 (5) z 1 2 p 2 = G 2 (6) z 2 Assume that the pupil-glint vectors g 1 p 1 and g 2 p 2 are represented as (v x1,v y1 ) and (v x2,v y2 ) respectively, and the vectors G 1 and G 2 are represented as (V x1,v y1 ) and (V x2,v y2 ) respectively. Thereore, the ollowing equation can be derived by combining the equations 5 and 6: 1 G 1 is not the actual virtual image o the IR light source 2 G 2 is not the actual virtual image o the IR light source

4 v x1 = V x1 z 2 v x2 (7) V x2 z 1 v y1 = V y1 z 2 v y2 (8) V y2 z 1 The above two equations describe how the pupil-glint vector changes as the head moves in ront o the camera. Also, based on the above equations, it is obvious that each component o the pupil-glint vector can be transormed individually. Thereore, equation 7 or the X component o the pupil-glint vector will be derived irst as ollows First Case: the cornea center and the pupil center lie on the camera s X Z plane: Figure 5 shows the ray diagram o the pupil-glint vector ormation when the cornea center and pupil center o an eye happen to lie on the X Z plane o the camera coordinate system. Thereore, either the generated pupil-glint vectors p 1 g 1 and p 2 g 2 or the vectors P 1 G 1 and P 2 G 2 can be represented as one dimensional vectors, speciically, p 1 g 1 = v x1, p 2 g 2 = v x2, P 1 G 1 = V x1 and P 2 G 2 = V x2. creen Plane X p o Image Plane 2 g 2 c p 1 g1 G P 1 1 G 1 Z Axis o Camera O s O G 2 creen Gaze Point Fig. 5. Pupil and glint image ormation when the eye is located at dierent positions in ront o the camera (Top-down view). According to Figure 5, the vectors G 1 and G 2 can be represented as ollows: G 1 = G 1 O 1 + O 1 (9) G 2 = G 2 O 2 + O 2 (10) For simplicity, r 1 is used to represent the length o O 1, r 2 is used to represent the length o, G 1 is represented as α1, G 2 is represented as α2, G 1 is represented as β1 and G 2 is represented as β2. According to the geometries shown in Figure 5, the vectors G 1 and G 2 can be urther achieved as ollows: 1 2 G 1 = r 1 sin(α1) r 1 cos(α1) tan(β1) (11) G 2 = r 2 sin(α2) r 2 cos(α2) tan(β2) (12) AsshowninFigure5,lineG 1 and line G 2 are parallel to the X axis o the camera. Thereore, tan(β1) and tan(β2) can be obtained rom the rectangles g 1 o c O and g 2 o c O individually as ollows: tan(β1) = tan(β2) = o c g 1 (13) o c g 2 (14) In the above equation, g 1 and g 2 are the glints in the image, and o c is the principal point o the camera. For simplicity, we choose x g1 to represent o c g 1 and x g2 to represent o c g 2. Thereore, ater detecting the glints in the image, tan(β1) and tan(β2) can be obtained accurately. Further, sin(α1), cos(α1), sin(α2) and cos(α2) can be obtained rom the geometries o the rectangles 1 and 2 directly. Thereore, equations 11 and 12 can be derived as ollows: V x1 = r 1 (z s z 1 ) x g1 V x2 = r 2 (z s z 2 ) x g2 + r 1 (x s x 1 ) + r 2 (x s x 2 ) (15) (16) econd Case: the cornea center and the pupil center do not lie on the camera s X Z plane: However, the cornea center and the pupil center do not always lie on the camera s X Z plane, we can obtain the ray diagram shown in Figure 5 by projecting the ray diagram in Figure 4 into X Z plane along the Y axis o the camera s coordinate system. Thereore, as shown in Figure 6, point is the projection o the pupil center P c1, point is the projection o the cornea center O c1, and point is also the projection o the screen gaze point in the X Z plane. tarting rom O c1, a parallel line O c1 o line intersects with line P c1 at. Also starting rom P c1, a parallel line P c1 o line intersects with line at. Because O c1 P c1 represents the distance r between the pupil center to the cornea center, which will not change as the eyeball rotates, can be derived as ollows: r 1 = = r 2 +(y 1 y s ) 2 (17)

5 X O s Assume that a speciic gaze mapping unction P1 is known via the calibration procedure described in ection 2. Thereore, ater integrating the head mapping unction g into the speciic gaze mapping unction P1 via equation 4, the generalized gaze mapping unction F can be rewritten as ollows: Z O c1 P c1 Y Fig. 6. Projection into camera s X Z plane. Thereore, when the eye moves to a new location as showninfigure5, can be represented as ollows: r 2 = = r 2 +(y 2 y s ) 2 (18) Ater substituting the ormulations o r 1 and r 2 into equations 15 and 16, we can obtain Vx1 V x2 as ollows: V x1 = d [(z s z 1 ) x g1 +(x s x 1 ) ] V x2 [(z s z 2 ) x g2 +(x s x 2 ) ] where d is set as ollows: d = (z 2 z s ) 2 +(x 2 x s ) 2 +(y 2 y s ) 2 (z 1 z s ) 2 +(x 1 x s ) 2 +(y 1 y s ) 2 (19) As a result, equations 7 and 8 can be inally obtained as ollows: v x1 = d [(z s z 1 ) x g1 +(x s x 1 ) ] [(z s z 2 ) x g2 +(x s x 2 ) ] z 2 z 1 v x2 (20) v y1 = d [(z s z 1 ) y g1 +(y s y 1 ) ] [(z s z 2 ) y g2 +(y s y 2 ) ] z 2 z 1 v y2 (21) The above equations constitute the head mapping unction g between the pupil-glint vectors o the eyes at dierent positions in ront o the camera, while gazing at the same screen point. 3.3 Iterative Algorithm or Gaze Estimation Actually, the derived equations 20 and 21 o the head mapping unction can not be established unless the gaze point =(x s,y s,z s ) o the screen is known. However, the gaze point is the one that needs to be estimated. As a result, the gaze point is also a variable o the head mapping unction g, which can be urther expressed as ollows: v 2 = g(v 2,,,) (22) = F (v 2,,) (23) Given the extracted pupil-glint vector v 2 rom the eye image and the new location that the eye has moved to, equation 23 becomes a recursive unction. An iterative solution is proposed to solve it. First, the screen center 0 is chosen as an initial gaze point, then a corrected pupil-glint vector v 2 can be obtained rom the detected pupil-glint vector v 2 via the head mapping unction g. By inputting the corrected pupil-glint vector v 2 into the speciic gaze mapping unction P1, a new screen gaze point can be estimated. is urther used to compute a new corrected pupil-glint vector v 2. The loop continues until the estimated screen gaze point does not change any more. Usually, the whole iteration process will converge in less than 5 loops, which is very ast. By the proposed method, the gaze point can be estimated accurately. 4 Experiment Results 4.1 ystem etup Two cameras are mounted under the monitor screen. An IR light illuminator is mounted at the center o the lens o one camera, which will produce the corneal glint in the eye image. The pupil-glint vectors extracted rom the eye images captured by this camera will be used or gaze estimation. ince our algorithm needs the 3D eye position to work, another camera is mounted close to the one with IR illuminator. These two cameras orm a stereo vision system so that the 3D eye position can be obtained accurately. One requirement or our eye tracker system is to know the 3D representation o the monitor screen plane in the coordinate system o the camera with the IR illuminator. This is accomplished via the calibration method proposed in [14]. Ater the calibration, the coniguration between the camera and the monitor is ixed during the eye tracking. 4.2 Head Compensation Model Validation The equations 20 and 21 o the head mapping unction g are validated by the ollowing experiment. A screen point c = (132.75, , ) is chosen as the gaze point. The user was gazing at this point rom twenty dierent locations in ront o the camera; at each location, the pupil-glint vector and the 3D pupil center are collected. The 3D pupil centers and the pupil-glint vectors o the irst two samples, are shown in Table I, where is served as the reerence position. The second column indicates the original pupil-glint vectors, while the third column indicates the transormed pupil-glint vectors by the head

6 mapping unction. The dierence between the transormed pupil-glint vector o and the reerence pupil-glint vector at is deined as the transormation error. Figure 7 illustrates the transormation errors or all these twenty samples. It s observed that the average transormation error is only around 1 pixel. This shows the validity o the proposed head compensation model. TABLE I PUPIL-GLINT VECTOR COMPARION AT DIFFERENT EYE LOCATION 3D pupil 2D Pupil-glint Transormed Pupil-glint position (mm.) Vector (pixel) Vector (pixel) (5.25, 15.56, ) (9.65, ) (9.65, ) (-8.13, 32.29, ) (7.17, ) (8.75, ) Pixels Transormation Error (X Component) Locations Pixels Transormation Error (Y Component) Locations Fig. 7. The pupil-glint vector transormation errors. 4.3 Gaze Estimation Accuracy When the user moves away rom the camera, the eye in the image becomes smaller. Due to the increased pixel measurement error caused by the lower image resolution, the gaze accuracy o the eye gaze tracker will decrease as the user moves away rom the camera. In this experiment, the eect o the distance to the camera on the gaze accuracy o our system is analyzed. A user was asked to perorm the gaze calibration when he was sitting around 330 mm to the camera. Ater the calibration, the user was positioned at our dierent locations, which have dierent distances to the camera as listed in Table II. At each location, the user was asked to ollow the moving objects that will display 12 predeined positions across the screen. Table II lists the computed gaze estimation accuracy at these our dierent locations. It shows that as the user moves away rom the camera, the gaze resolution will decrease. Yet within this space allowed or the head movement, approximately mm (width height depth) at 450 mm rom the camera, the average horizontal angular accuracy is around 1.3 and the average vertical angular accuracy is around 1.7, which is acceptable or most o the Human Computer Interaction applications. In addition, this space volume allowed or the head movement is large enough or a user to sit comortably in ront o the camera and communicate with the computer naturally. 5 Conclusion In this paper, a solution is proposed to improve the classical PCCR eye gaze tracking technique. While enjoying the advantages o high accuracy and stability o classical PCCR TABLE II GAZE ETIMATION ACCURACY Distance to Horizontal Accuracy Vertical Accuracy the camera (mm) (degrees) (degrees) technique, we make it work under natural head movements. Thereore, the user does not need to keep his head uncomortably still as most o eye trackers require. Instead, the user can move his head reely while using our eye tracking system. At the same time, the number o calibration procedures is minimized to only once or a new user. Also, the calibration result is long-lasting: i the user repositions his head or he wants to come back and use the system next time, he does not need to do the calibration again. Both achievements are made possible because o the proposed head mapping unction, which can automatically accommodate the head movement changes. Reerences [1]. Milekic, The more you look the more you get: intention-based interace using gaze-tracking, in Bearman, D., Trant, J.(des.) Museums and the Web 2002: elected papers rom an international onerence, Archives and Mseum Inormatics, [2] K. Hyoki, M. higeta, N. Tsuno, Y. Kawamuro, and T. Kinoshita, Quantitative electro-oculography and electroencephalography as indices o alertness, in Electroencephalography and Clinical Neurophysiology, 1998, pp [3] Y.Ebisawa, M. Ohtani, and A. ugioka, Proposal o a zoom and ocus control method using an ultrasonic distance-meter or video-based eye-gaze detection under ree-hand condition, in Proceedings o the 18th Annual International conerence o the IEEE Eng. in Medicine and Biology ociety, [4] C.H. Morimoto, D. Koons, A. Amir, and M. Flickner, Frame-rate pupil detector and gaze tracker, in IEEE ICCV 99 Frame-rate Workshop, [5] A. T. Duchowski, Eye tracking methodology: Theory and practice, in pring Verlag, [6] R.J.K. Jacob and K.. Karn, Eye tracking in human-computer interaction and usability research: Ready to deliver the promises, 2003, Oxord, Elsevier cience. [7] D. H. Yoo, Non-contact eye gaze estimation system using robsut eature, CVIU, special Issue on eye detection and tracking, [8] LC Technologies Inc, The eyegaze development sytem, [9] Z. Zhu and Q. Ji, Eye and gaze tracking or interactive graphic display, Machine Vision and Applications, vol. 15, no. 3, pp , [10] C.H. Morimoto and M. R.M. Mimica, Eye gaze tracking techniques or intractive applications, CVIU, special issue on eye detection and tracking, [11] T.E. Hutchinson, K.P. White Jr., K.C. Reichert, and L.A. Frey, Human-computer interaction using eye-gaze input, in IEEE Transactions on ystems, Man, and Cybernetics, 1989, pp [12] R.J.K. Jacob, Eye-movement-based human-computer interaction techniques: Towards non-command interaces, 1993, pp , Ablex Publishing corporation, Norwood, NJ. [13] Z. Zhu and Q. Ji, Robust real-time eye detection and tracking under variable lighting conditions and various ace orientations, CVIU, special issue on eye detection and tracking, [14]. W. hih and J. Liu, A novel approach to 3-d gaze tracking using stereo cameras, in IEEE Trans. yst. Man and Cybern., part B, 2004, number 1, pp