Framework for colocated synchronous dual eye tracking Craig Hennessey Department of Electrical and Computer Engineering University of British Columbia Mirametrix Research craigah@ece.ubc.ca Abstract Dual user eye tracking offers insight into the collaborative behaviours of participants, as well as the potential for improving communication effectiveness when direct eye contact is unavailable. In this paper we outline a system framework for performing colocated synchronous dual eye tracking. Issues around system setup and eye tracker calibration are discussed. An evaluation of accuracy found that lower accuracy was achieved for the viewer who was off axis to the primary display. The impact of lower accuracy depends on the intended application. For the test application presented here in which on-screen gaze markers indicate the general area of the viewer s interest, the accuracy achieved proved sufficient. Author Keywords DUET; eye gaze tracking; colocated; synchronous; ACM Classification Keywords H.5.m [Information interfaces and presentation (e.g., HCI)]: Miscellaneous. Copyright is held by the author/owner(s). CSCW 12, February 11 15, 2012, Seattle, Washington, USA. ACM 978-1-4503-0556-3/12/02. General Terms Design, Experimentation, Measurement
Introduction Computer supported collaborative technologies offer the potential to improve worker productivity [2]. There has been significant focus on remote collaboration, as the workforce is increasingly geographically distributed. Dual eye tracking has been investigated for remote collaborative problem-solving [6] and remote collaborative software development [1]. Certain tasks however are still frequently performed in a colocated or face to face environment [4]. The colocated environment under consideration here is one in which participants share a single workstation for joint tasks such as pair programming or collaborative document editing. An earlier work by Pietinen et al proposed a colocated collaborative system for investigating pair programming [7]. The system was noted for its somewhat complex hardware and software configuration, leading to difficulty in data synchronization and analysis. It is our goal to develop a simple framework for investigating and enhancing colocated collaborative work with dual eye tracking, and to outline common challenges that may arise in using such a system. Hardware Design The hardware block diagram for the colocated collaborative dual eye tracking system is shown in Figure 1. One eye tracker is connected directly to the primary computer while a second eye tracker is connected to the secondary computer. The primary computer is operated on by the primary participant who controls the keyboard and mouse while the secondary computer is used solely for running the second eye tracker. The secondary participant sits to the side of the primary participant with a side view of the primary computer display. The primary eye tracker may be built into the display, however the secondary eye tracker must be free to orient towards the second participant while still allowing full view of the primary screen. The primary participant is eye tracked as normal with the tracker located beneath the display. The secondary eye tracker must be placed off-axis, at an angle with the primary screen which allows the secondary participant view the display with enough room for both participants. The orientation of the secondary eye tracker must be such that infrared illumination from the eye tracker does not bleed into the primary eye tracker field of view, as this may lead to invalid corneal reflections as shown in Figure 2. This issue should not commonly occur as the secondary participant should be located to the side of the primary participant. PC HDMI USB TCP/IP Display Laptop (secondary) USB (secondary) Figure 1: Hardware block diagram for the colocated collaborative dual eye tracking system. The system consists of two computers, and two eye trackers, with the primary computer used for collaborative work.
(a) Correctly identified left and right eye pupils and glints Primary Secondary Tracker Server Tracker Server Client Gaze marker Gaze marker (secondary) Data recording Figure 3: Software block diagram for the colocated collaborative dual eye tracking system. Each computer runs one of the eye tracker servers. The client application for interaction and data recording runs on the primary computer. Calibration (b) Incorrectly identified glint in the right eye Figure 2: The pupil and glint for the left and right eyes are shown identified correctly (using colored crosses) for the primary participant in Figure 2(a). In Figure 2(b), the secondary eye tracker illumination is shown bleeding into the primary field of view resulting in extra corneal reflections. An incorrectly identified glint is selected in the right eye. Software Design The software system is outlined in Figure 3. The vendor specific software for the primary and secondary eye trackers are run on the primary (PC) and secondary (laptop) computers respectively. The eye tracker servers are responsible for calibration and for providing gaze estimates over TCP/IP to the client application running on the primary computer. Eye tracker calibration procedures require the viewer to observe points shown across the display. The primary eye tracker calibration is straightforward as it is connected to the primary display. To calibrate the secondary eye tracker the primary display may be temporarily connected to the secondary computer during the calibration process. An alternative is to show the calibration grid on the primary display, which the secondary participant observes during calibration, rather than the secondary display. Audible or visual cues seen from the peripheral vision of the secondary participant can be used to indicate completion of individual calibration points. Eye trackers are designed for a display in perpendicular to the viewer, as is the case for the primary participant. The secondary participant perceives significant parallax when viewing the display from the side. The rotated eye tracker results in lower accuracy for the secondary participant, as shown later in this paper.
Synchronization Gaze data is generated by the eye tracker software on both the primary and secondary computers. To synchronize the two data streams with the displayed content, a client application on the primary computer is necessary. The client application connects to both the primary and secondary eye tracker servers (through a vendor specific API or SDK) and collects the point of gaze estimates as well as a time stamp of when the estimate was computed. The timestamp of the first point of gaze estimate from each server is taken to be time T=0. All subsequent estimates are recorded as an offset from the initial time, synchronizing both gaze data streams, up to the transmission delay from the server to client application. The transmission delay between the secondary and primary computers may be estimated and used as a fixed offset, or as a rough estimate, over a local network a delay of 25 ms is common. Using a single client to connect to both servers at the same time reduces the complexity of trying to synchronizing both participants gaze estimates in separate applications, or by analyzing recorded data logs. The client application can use the collected gaze estimates in real-time to show the point of gaze on the display, as well as recorded for off-line processing such as for performing area of interest analysis. Implementation The framework outlined above was used to implement a system for colocated dual eye tracking. The test system is shown in Figure 4(a) in which the primary remote eye tracker is located in front of the primary display and the secondary remote eye tracker is oriented towards the right. The display was a 22 screen with resolution of 1920x1080 pixels. The eye trackers were remote systems from Mirametrix. In Figure 4(b), two participants are shown collaborating on the primary computer. The primary participant on the left controls the keyboard and mouse with the secondary participant on the right. (a) Display screen with two eye trackers (b) Two participants collaborating Figure 4: Figure 4(a) shows the system implemented, while Figure 4(b) shows the system in use by two participants.
An evaluation of accuracy of both the primary and secondary eye trackers was performed to compare the effect of the rotated secondary eye tracker. Four participants calibrated the primary eye tracker on the primary display and recorded their gaze positions on a 4x4 grid across the display. Each individual then moved to the secondary position and calibrated the secondary eye tracker on the primary display. The accuracy measurement was then repeated on the 4x4 grid. The accuracy measurements are shown in Figure 5. As can be seen, the accuracy of the secondary eye tracker is lower than the primary eye tracker. Average error over the four subjects with the primary eye tracker was 36 pixels (< 1 of visual angle) while the average error for the secondary eye tracker was 79 pixels (< 2 of visual angle). While the accuracy is lower at the secondary position, for some applications such as indicating the area of interest of the participant on the display, the performance achieved may be sufficient. Real-time application An example application common to duel eye tracking studies was developed to test system operation [1]. In this application the point of gaze for the participants were drawn in real-time on the display to provide a visual indication of each participant s focus. The gaze markers were drawn as small circular, semitransparent, overlays as shown in Figure 6. The gaze markers can promote effective communication by maintaining a common ground when communicating without direct eye contact [3, 5]. Participants often left the field of view of the eye tracker as they adjusted their seated position, at which point the gaze marker faded away. The disappearance of the gaze cursor provided indirect feedback to the participant on if they were in the correct eye tracker position. As a side effect, participants could selectively show or hide the gaze marker by consciously moving into or out of the field of view. Figure 5: Accuracy measurements on the primary and secondary eye trackers for four subjects. Figure 6: Two participants are shown collaborating on a spreadsheet. The primary participants gaze marker is shown in blue, while the secondary participants gaze marker is shown in red.
Discussion In this paper we have outlined a system for performing colocated synchronous dual eye tracking. The design uses a primary workstation for collaboration and eye tracking of the primary participant. A secondary computer is used for eye tracking the secondary participant. Accuracy of the secondary eye tracker was found to be lower than the primary eye tracker due to the angle of view of the primary display for the secondary participant. The lower accuracy was still sufficient for indicating general areas of interest with the point of gaze. A client application running on the primary PC and connected to the eye tracker servers simplified synchronization of both participants gaze data. An example application was developed to test gaze marker overlays for sharing the location of each participant s gaze and consequently their area of attention. The system described here will be used as a test bench for future research into colocated synchronous collaboration augmented with gaze information. The framework presented here will hopefully aid in the development of similar systems by other researchers. References [1] R. Bednarik, A. Shipilov, and S. Pietinen. Bidirectional gaze in remote computer mediated collaboration: setup and initial results from pair-programming. In Proceedings of the ACM 2011 conference on Computer supported cooperative work, pages 597 600, New York, NY, USA, 2011. [2] P. Dourish and V. Bellotti. Awareness and coordination in shared workspaces. In Proceedings of the 1992 ACM conference on Computer-supported cooperative work, pages 107 114, New York, NY, USA, 1992. [3] E. Horvitz, C. Kadie, T. Paek, and D. Hovel. Models of attention in computing and communication: from principles to applications. Commun. ACM, 46:52 59, March 2003. [4] R. Johansen. GroupWare: Computer Support for Business Teams. The Free Press, New York, NY, USA, 1988. [5] K. Jokinen, M. Nishida, and S. Yamamoto. Eye-gaze experiments for conversation monitoring. In Proceedings of the 3rd International Universal Communication Symposium, pages 303 308, New York, NY, USA, 2009. [6] N. Kuriyama, A. Terai, M. Yasuhara, T. Tokunaga, K. Yamagishi, and T. Kusumi. Gaze matching of referring expressions in collaborative problem solving. Proceedings of International Workshop on Dual Eye Tracking in CSCW, 2011. [7] S. Pietinen, R. Bednarik, T. Glotova, V. Tenhunen, and M. Tukiainen. A method to study visual attention aspects of collaboration: eye-tracking pair programmers simultaneously. In Proceedings of the 2008 symposium on Eye tracking research and applications, pages 39 42, New York, NY, USA, 2008.