PAPER CrossOverlayDesktop: Dynamic Overlay of Desktop Graphics Between Co-located Computers for Multi-User Interaction

Transcription

1 IEICE TRANS. INF. & SYST., VOL.Exx D, NO.xx XXXX 200x 1 PAPER CrossOverlayDesktop: Dynamic Overlay of Desktop Graphics Between Co-located Computers for Multi-User Interaction Daisuke IWAI, Nonmember and Kosuke SATO, Member SUMMARY This paper presents an intuitive interaction technique for data exchange between multiple co-located devices. In the proposed system, CrossOverlayDesktop, desktop graphics of the devices are graphically overlaid with each other (i.e., alphablended). Users can exchange file data by the usual drag-anddrop manipulation through an overlaid area. The overlaid area is determined by the physical six degrees of freedom (6-DOF) correlation of the devices and thus changes according to users direct movements of the devices. Because familiar operations such as drag-and-drop can be applied to file exchange between multiple devices, seamless, consistent, and thus intuitive multiuser collaboration is realized. Furthermore, dynamic overlay of desktop graphics allows users to intuitively establish communication, identify connected devices, and perform access control. For access control of the data, users can protect their own data by simply dragging them out of the overlaid area, because only the overlaid area becomes a public space. Several proof-of-concept experiments and evaluations were conducted. Results show the effectiveness of the proposed interaction technique. key words: Multiple displays, Handheld computer, Graphically overlaid desktop, Co-located user collaboration 1. Introduction and Motivation With the recent widespread use of handheld or portable devices, such as the Ultra-Mobile PC, tablet PC, PDA, netbook, and smartphone, data exchange between colocated users, such as friends or colleagues, has become common. However, most novice users still have difficulty connecting their own devices with others via wireless connection. This is because that they are not familiar with the cumbersome and complicated authentication procedures that require them to adopt a new mental model of the process. This paper proposes an intuitive interaction technique for data exchange between co-located multiple devices, which is based on a previous short report [1]. In the proposed system, CrossOverlayDesktop, desktop graphics of the devices are graphically overlaid with each other (i.e., alpha-blended). Users can exchange file data by the usual drag-and-drop manipulation through the overlaid area (Fig. 1). The overlaid area is determined by physical six degrees of freedom (6-DOF) correlation of the devices and thus changes according to users direct movements of the devices. Because fa- The authors are with the Graduate School of Engineering Science, Osaka University, Toyonaka-shi, Japan. (a) usual environment (b) CrossOverlayDesktop environment Fig. 1 Concept of CrossOverlayDesktop: (a) any desktop graphics are not overlaid, and (b) desktop graphics are overlaid (or alpha-blended) each other and the users exchange their data through the overlay. miliar operations such as drag-and-drop can be applied to file exchange between multiple devices, the proposed interaction technique offers seamless, consistent, and thus intuitive multi-user collaboration. Dynamic overlay of desktop graphics allows users to intuitively establish communication, identify connected devices, and perform access control. Access control or ownership of the exchanged data is an important issue for multi-user interactions where private or personalized information must be appropriately protected. In the proposed system, users can protect their own data by simply dragging them out of the overlaid area, because only the overlaid area becomes a public space. Furthermore, they are allowed to control the visibility of the overlaid desktop graphics and file icons. This paper focuses on discussing usability of the proposed interface rather than its usefulness through several user studies. The remainder of the paper is organized as follows. The subsequent section briefly describes related studies. Sections 3 and 4 present the detailed principles of dynamic graphical overlay and interaction techniques, respectively. Section 5 describes the detailed implementation of the proposed interaction technique and shows how it works. Section 6 describes several user studies to evaluate the proposed method. In section 7, we briefly discuss advantages and limitations of the proposed method. Section 8 concludes the paper with directions for future work.

2 2 IEICE TRANS. INF. & SYST., VOL.Exx D, NO.xx XXXX 200x 2. Related Works Intuitive interaction techniques to support direct data exchange between co-located portable devices have been explored in previous studies. Rekimoto proposed an interaction technique called Pick-and-Drop, which allows a user to pick a file on one computer and drop it on another with a touch pen [2]. Although pen ID recognition is required in this technique, Stitching [3] utilizes a pen-based gesture to bind multiple displays. In contrast to these previous studies, the well-known drag-and-drop operation can still be used in file exchange between co-located devices in the interaction technique proposed in this paper. A user can copy a file from another device by simply dragging a file icon through the overlaid desktop area and dropping it on his (or her) own desktop. In this interaction, a touch pen is not always needed; mice or other input interfaces can be used as well. Several studies have proposed intuitive methods to connect nearby computers. ConnecTables [4] are wheeled tables with mounted LCD displays that can be moved together so that the top edges of the two LCDs meet. The devices then recognize one another through a radio-frequency identification (RFID) tag and a reader mounted on each device. The displays can be joined only at the top edges (i.e., when they face each another). When joined, two ConnecTables create a shared workspace in which users can pass objects back and forth. Rekimoto et al. proposed detecting nearby devices with RFID tags or infrared beaming to establish wireless connections [5]. In these techniques, a user can connect a device by bringing it close to another without having any knowledge about the wireless communication itself. However, as indicated by Hinckley [6], although wireless networking and location awareness techniques allow devices to communicate and provide some information about the proximity of other devices, a user would have difficulty specifying which devices to connect to when several devices are nearby. This paper explores capabilities of an interaction technique that exploits 6-DOF correlation of co-located devices rather than their proximity. This richer information allows a user not only to establish communication intuitively but also to specify desired devices from multiple nearby devices easily. Recently, Cao et al. explored co-located multi-user interaction using multiple handheld projectors [7]. Because overlapping of projected working spaces is a natural phenomenon, they realized file exchange through the overlapped area. Although they proposed access control of projected data, all users could see all of the data. They allow users to make their own data invisible to other users; however, the users themselves cannot see this hidden data either. On the other hand, this paper proposes an interaction technique in which users can exchange data only in the overlaid area (i.e., public space). They easily protect private data by simply moving them to the non-overlaid area. In addition, visibility control of file icons and desktop graphics are implemented in the proposed technique. Several techniques that enable 6-DOF tracking of devices have been developed, such as a vision-based marker tracking method [8] or a magnetic tracking system. However, they require an external apparatus such as a camera and obtrusive visual markers or a transmitter of a magnetic field. On the other hand, many studies have been conducted on ubiquitous high-quality spatial tracking [9]. However, these projects are still in the research phase, and thus are not commercially available. Because we focus on the interaction technique itself, development of 6-DOF tracking technology is outside the focus of this paper. Therefore, an off-the-shelf magnetic tracker is employed in this paper because it offers real-time and precise 6-DOF tracking, and thus allows us to prototype future interaction scenarios. 3. Dynamic Overlay of Desktop Graphics Based on 6-DOF Correlation of Devices This section describes how to determine 2D movements, such as translation and rotation, of desktop graphics of co-located devices based on their 6-DOF correlations. This is accomplished by coordinate transformation in a 3D physical space (Fig. 2) and a 2D pixel space (Fig. 3). As a result of the transformation, desktop graphics of the devices, including file icons and mouse pointers, are overlaid in a shared 2D space. Then alpha-blending is applied to the overlaid area. Figure 2(a) shows an example of co-located devices in a 3D physical space. Two displays, I and, each have their own Cartesian coordinate systems C I (origin: O I, axes: X I, Y I, Z I ) and CA (origin: O A, axes: XA, Y A, Z A ), respectively. First, 3D distances of all combinations of a vertex of one display and an edge of the other are measured, and a combination whose distance is the minimum is chosen. The chosen vertex and edge are represented as p A and ei, respectively (Fig. 2(b)). A point s I is the intersection of e I and the perpendicular drawn from p A to ei. Second, is translated in the 3D physical space in a manner such that p B is on the XI Y I plane and the distance from s I to p B is the same as that from si to p A (Fig. 2(c)). The new coordinate system of is represented as CB. p B is translated from p A. Third, is rotated in 3D space around an axis that contains p B and is parallel to e I in a manner such that q C, which is the diagonal of p C, is on the XI Y I plane (Fig. 2(d)). The new coordinate system of is represented as CC. q C is rotated from q B. p B is invariant in this process ( p B = p C ). Finally, is rotated in the 3D physical space around a diagonal line from p C to q C in a manner such

3 IWAI and SATO: DYNAMIC OVERLAY OF DESKTOP GRAPHICS FOR CO-LOCATED MULTI-USER INTERACTION 3 (a) co-located devices (b) vertex-edge combination whose distance is minimum C ) (c) translation (CA B C ) (d) rotation (CB C C ) (e) rotation (CC D (f) final result of coordinate transformation physical space is now converted into a 2D pixel space (Fig. 3). The desktop graphics of are translated and rotated as shown in Fig. 3(a). M is a user-defined arbitrary parameter that defines how the desktop graphics of are inserted inside the desktop of I from ei in the 2D pixel space. α represents a conversion factor from millimeters to pixels. This process can be represented as a simple 2D affine transformation without scaling. Thus, the overlaid area is derived as shown in Fig. 3(b). Hereby, an overlaid desktop area changes according to the 3D movements of co-located devices. Furthermore, the overlaid area implicitly indicates the direction of a connected device, and thus a user can easily specify it. However, because the motion of overlaid desktops is performed in 2D although the devices themselves move in 3D, this visualization might be indirect and thus result in a non-intuitive interaction. This issue will be evaluated and discussed later in This section describes how to realize file exchange through an overlaid area, stabilization of the overlaid area, and access control of file data. 4.1 (g) final result of coordinate transformation (2D view) Fig. 2 Coordinate transformation in physical space (a) 2D transformation of desktop graphics Fig. 3 (b) calculated overlaid area Overlaid area calculation in pixel space that all the vertices of, such as p~idi, ~qd, and ~rd, are on I I the X Y plane (Fig. 2(e)). The new coordinate sys tem of is now represented as CD. ~rd is rotated from ~rc. p~ici and ~qc are invariant in this process (~ pici = p~idi, ~qc = ~qd ). Now all the vertices of the two displays are on the same 2D physical plane, as shown in Fig. 2(f). Therefore, they can be drawn in a 2D representation, as shown in Fig. 2(g). In this figure, d is the 2D distance between p~idi and ~si and t is that between ~si and an end-point of ei. Angle θ is formed by the two x-axes X I and XD. d, t and θ are calculated in the above processes. The result of coordinate transformation in the 3D Interaction Technique File Exchange through Overlaid Area This paper proposes an intuitive file exchange interaction technique where a user can directly access a file that another user owns in an overlaid desktop area and copy it with the familiar drag-and-drop manipulation. The overlaid area becomes a public space where file data can be exchanged, and thus the remaining area becomes a personal space for file data, which only the owner can access. The proposed technique is explained with an example case of two co-located devices, I and, whose desktop graphics, including file icons and mouse pointers, are alpha-blended in an overlaid area. Figure 4 shows the overlaid desktop graphics of each device. The left and right images correspond to I and, respectively. Figure 4(a) shows an example of alpha-blending of the desktop graphics. When the user of I clicks a file icon in the public space, which the user of owns, a rectangular frame indicating the clicking operation appears around the icon in both the desktop graphics (Fig. 4(b)). After being dragged over to the public space (Fig. 4(c)), the file icon starts to rotate in a manner such that it is oriented correctly in the desktop of I when it is across the border of the public space (Fig. 4(d)(e)). When the icon is dragged, its transparency is 50%, indicating that it is not copied yet. Once it is dropped in the private space of I s desktop (i.e., the copy operation is completed), the transparency becomes 0% (Fig. 4(f)). As shown in Fig. 4, all visual effects occurring in the public space are drawn in both

4 4 IEICE TRANS. INF. & SYST., VOL.Exx D, NO.xx XXXX 200x graphics are generated. It is not necessary to rotate or adjust the own device for file copying. 4.2 Stabilization of Overlaid Graphics (a) alpha blending of the desktop graphics (b) clicking of an icon on the overlaid area Unintended device motion has to be considered when the device is held by a user. In addition, the 6-DOF sensor may have noise in its output. Because these factors cause unintended movements of the overlaid desktop graphics and thus make file exchange interaction difficult, stabilization of the motion is required. A simple average filter is applied to sensor output in order to compensate for device instability and sensor noise. Although this filter is simple and easy to implement, there is a time delay in the filter output. We evaluate how the usability of the proposed file exchange interaction technique is improved with such a simple filter in Access Control (c) dragging of the icon (d) dragging across the border of the overlaid area Fig. 4 (e) dragging to own desktop (f) dropping of the icon File exchange through overlaid desktops the desktops so that both the users can be aware of the copy operation. Note that one of the desktops is rotated in the file copying process, as shown in Fig. 4 and some other figures in this paper. A user can copy file data irrespective of the orientations of the communicating devices if they are close enough to each other so that overlaid desktop Three access control methods are available in the proposed interaction technique. First, it is assumed that nobody wants to receive a file from another user without realizing he (or she) has received it. Therefore, the following rule is applied: in the public space, every user can copy file data from another device, but nobody can copy a file to another device. Second, users can protect their data by a natural movement. As explained in 4.1, only files in the public space can be accessed by all users. Therefore, if users would not like to share private data, they just need to move the data or their own devices in a manner such that the icons are out of the public space. The user can control accessibility of the data by direct manipulation. Finally, a user can assign three access levels to data. All file data as well as the desktop itself have their own access levels that determine how users can interact with them. The first is Public where the file/desktop is visible to all users (i.e., displayed on both displays) and all users can operate on it. The second is Semi-Public where the file/desktop is visible to all users but only operable by its owner. The last is Private where the file/desktop is visible and operable only by its owner. Owners can assign an access level to their own data from a context menu that appears with a right click on the icon or the desktop. 5. Implementation This section describes an implementation of the proposed file exchange interaction technique and shows how it works. 5.1 System Configuration The proposed system is configured as shown in Fig. 5.

5 IWAI and SATO: DYNAMIC OVERLAY OF DESKTOP GRAPHICS FOR CO-LOCATED MULTI-USER INTERACTION 5 Fig. 5 (a) (b) (c) (d) System configuration Fig. 7 Overlay of desktop graphics according to 6-DOF correlations of devices: (a) distance between two devices is not short enough to generate overlaid graphics, (b) it is shorter and overlaid graphics are automatically appear, (c) it is much shorter and the overlaid area becomes larger, (d) desktop graphics are overlaid irrespective of the orientations of the communicating devices if they are close enough to each other. Fig. 6 Prototype of the proposed system It is a server-client system and consists of two client devices, a server, and two 6-DOF trackers attached to the devices. Tracker data are processed in the server to determine the 2D movement of the desktop graphics. Clients always send their file data (file name and position) and mouse data (click information and position) to the server, and receive the same information about the clients from the server. When a file is copied from a client, all data of the file is transferred. Figure 6 shows the experimental system that consists of a server computer (IBM ThinkPad T43p; CPU: 2.26 [GHz], RAM: 1 [GB]) and two handheld client computers (KOHJINSHA SA1F00D and SA5SX12A; CPU: 800 [MHz], RAM: 512 [MB]; a touch pen can be used only with the SA5SX12A). These three devices are connected via wireless LAN. Both client devices are tracked by a Polhemus Fastrak magnetic tracking system that provides 6-DOF information with 0.04 millimeter precision. All visual effects, including file icons and desktop graphics, are drawn with OpenGL. The system runs at about 30 [Hz], which does not disturb users natural interactions. In addition to the handheld devices, we prepared a desktop PC (CPU: 3.2 [GHz], RAM: 1.0 [GB]) with a large public display (Sony Qualia KDX-46Q005) as a client. The handheld clients have pixels in 7-inch displays ( [mm]). The public client has pixels in a 46-inch display ( [mm]). Thus, conversion factors from millimeters to pixels, α, introduced in Sec. 3 are 6.7 for the handheld devices and 1.3 for the public device. In addition, 512 (hand- (a) file dragged in public space (b) icon rotated on border (c) icon rotation completed (d) file copied Fig. 8 File exchange experiment held) and 640 (public) pixels are assigned to the parameter M, also introduced in Sec System Behavior This part shows how the proposed method actually works with the working prototype. First, we show how the proposed algorithm for dynamic overlay of desktop graphics works. Two clients are placed on a table as shown in Fig. 7(a) where desktop graphics are not overlaid yet. When they are brought close to each other, the desktop graphics are overlaid according to the distance (Fig. 7(b)(c)). The desktop graphics are overlaid irrespective of the orien-

6 IEICE TRANS. INF. & SYST., VOL.Exx D, NO.xx XXXX 200x 6 (a) setup A (a) (b) Fig. 9 Usage scenarios of file exchange interaction: (a) between co-located handheld devices, (b) between a co-located handheld device and public display tations of the communicating devices if they are close enough to each other (Fig. 7(d)). It is confirmed that the overlaid desktop graphics change dynamically according to the correlation of the devices position and orientation. Second, we show how the proposed file exchange interaction works. When a file in the public space (or overlaid area) is dragged, the file icon follows the mouse pointer and is overlaid on both the desktop graphics with 50% transparency (Fig. 8(a)). When the mouse pointer crosses the border, the file icon starts to rotate (Fig. 8(b)). Fig. 8(c) shows the completed rotation. The file is finally copied by a drop operation, and then the transparency of the copied file icon becomes 0% (Fig. 8(d)). Third, Fig. 9(a) shows two users exchanging their files with the proposed interaction technique. They used the prototype for ten minutes and it worked without any technical problems. In addition, the system worked with a combination of the handheld client and the public client, as shown in Fig. 9(b). The user could also exchange file data with this large display. Thus, it is confirmed that the proposed interaction technique works in typical usage scenarios. 6. Evaluation Three types of experiments are conducted to evaluate the usability or intuitiveness of the proposed interaction technique with the working prototype introduced in Intuitiveness Graphics of Dynamic Overlaid Desktop Movement of a device in a 3D physical space results in a 2D movement of the overlaid desktop graphics, as explained in Sec. 3. When the displays are placed on a same planar surface, the transformation is 2D to 2D linear mapping. Therefore, the generated overlaid graphics are natural for users. However, it is assumed that the users hold their mobile devices in a 3D physical space. In this case, the displays are not always placed on a same plane, then the transformation becomes 3D to 2D non-linear mapping. Therefore, the generated overlaid graphics might not be intuitive for (b) setup B (c) setup C (d) setup D (e) setup E Fig. 10 Five variations of position and pose correlation of two co-located devices the users. The objective of the experiment is to check if the overlaid desktop graphics generated by the proposed method are intuitive for the users even when the displays are not on a same plane. Participants behaviors in cases where two devices are placed on a same planar surface are compared with ones in cases where they are not placed on a same plane. Two devices are positioned in five ways, as shown in Fig. 10, on the assumption that two users sit next to each other (setup A), a user faces another across a table (setups B and D), and two users sit down at a corner of a table (setups C and E). In setups A, B, and C, two displays are placed on a same plane. In the other setups, they are not on a same planar surface. In each setup, two devices are placed so that the generated overlaid area shares only one icon. One displays the overlaid desktop graphics (thus, this device is referred to as DISPLAY). In contrast, the other does not display anything, but is touched by a participant (thus, this device is referred to as TOUCH). In the experiment, the devices are not held by participants but fixed on tripods. Magnetic trackers are attached to both devices. 6-DOF information from the trackers is used to calculate the overlaid desktop area. The procedure of the experiment is as follows (see also Fig. 11). First, a file icon is randomly placed on the TOUCH desktop (Fig. 11(a)). Overlaid desktop graphics are automatically appear according to the 6-DOF correlation of the devices (Fig. 11(b)). Then desktop graphics on both the devices are made invisible and a participant is allowed to see the DISPLAY desktop (Fig. 11(c)). Second, the DISPLAY device displays the desktop graphics (Fig. 11(d)). At the same time, the participant starts to estimate where the overlaid icon

7 IWAI and SATO: DYNAMIC OVERLAY OF DESKTOP GRAPHICS FOR CO-LOCATED MULTI-USER INTERACTION 7 (a) (b) (c) (d) (e) Fig. 11 Experimental procedure of the first evaluation: (a) a file icon is randomly placed on TOUCH desktop, (b) desktop graphics are overlaid, (c) they are made invisible and a participant is let see DISPLAY desktop, (d) the desktop graphics are revealed on DISPLAY desktop, and (e) the participant estimates the position of the icon on TOUCH desktop and touches that location. (a) distance from correct position Fig. 12 (b) task completion time Results in position estimation of overlaid icon Ten participants were recruited from a local university. All of them major in computer science and used PCs for more than six hours everyday. All the system features were explained to each participant, but no demonstrations were shown. A total of 50 experiments (5 setups 10 participants) were conducted. The file icon was randomly located in each experiment. The results are shown in Fig. 12. Note that bars in all the graphs shown in this paper represent standard deviations. Figure 12(a) and (b) shows the averages of the measured Euclidean distances and ones of the task completion time. A one-way analysis of variance (ANOVA) with repeated measures showed that there were no statistically significant differences in both the Euclidean distance (F(4,36) = 0.71, p 0.59) and the task completion time (F(4,36) = 1.84, p 0.14) between the five setups. The results confirmed that the participants behaviors were not significantly changed between the cases where two devices are placed on the same planar surface and the cases where they are not placed on one. Therefore, the overlaid desktop graphics generated by the proposed method do not degrade the intuitiveness for the users. In addition, another evaluation study is also conducted to check whether a user can identify a device whose desktop graphics overlay the user s own desktop when more than two displays exist. Four devices are placed on a table as shown in Fig. 13(a). In the desktop of the bottom-center device, the other three desktop graphics are overlaid as shown in Fig. 13(b). In this experiment, each device has one file icon that is in the overlaid area. A participant is allowed to identify which file icon belongs to which device. Ten participants were recruited and 100% of them could perfectly identify the relationships. Thus, it is confirmed that a user can comprehend the correspondences between overlaid desktop graphics and nearby devices even when more than two desktops are overlaid. 6.2 (a) experimental setup (b) displayed desktop graphics Fig. 13 Device identification from overlaid desktop graphics in the case of four co-located devices is located in the TOUCH desktop. Then he (or she) touches the estimated position on the TOUCH display (Fig. 11(e)). The Euclidean distance from the touched position to the correct position of the icon is measured. Task completion time is measured from when the overlaid desktop graphics of the DISPLAY desktop are revealed to when the participant touches the TOUCH display. Effectiveness of Stabilization The effectiveness of the proposed stabilization technique explained in 4.2 is evaluated. In this evaluation, one device is held by a participant and another is set in a manner such that the top sides of the displays face each other, as shown in Fig. 14. The participant is asked to click (specifically, touch with a touch pen) a file icon appearing in the overlaid area on the held device. When the participant clicks the icon, it disappears and another one appears at a different position in the overlaid area. The positions of the icons are randomly selected. The participants have to click ten file icons one by one. Task completion time is measured and the number of clicks is counted as well. The experiment is conducted under the following three conditions. In the first condition, no stabilization

8 8 IEICE TRANS. INF. & SYST., VOL.Exx D, NO.xx XXXX 200x Fig. 14 Touching an icon appearing in an overlaid area while holding device Fig. 16 Multi-user interaction for user feedback experiments (a) task completion time (b) number of clicks Fig. 15 Results for icon touch task Table 1 Qualitative measures used in the study Measure Descriptive phrase Understandability Easy to understand? Difficulty Easy to use? Familiarity Familiar and appropriate form? Complexity Complex form? Satisfactory Fun to use? Appeal Want to use again? technique is applied (thus, this condition is referred to as W/O STAB). In the second condition, the proposed stabilization technique (i.e., a simple averaging filter) is applied (thus, this condition is referred to as W/ STAB). In the last condition, overlaid desktop graphics do not move (thus, this condition is referred to as STATIC). The last condition offers the best condition in this experiment, and thus the data gained under it are considered as reference data. Ten participants were recruited from a local university. All the system features were explained to each participant. Then each participant spent about one minute practicing with the system in each of the abovementioned cases. Eight frames were applied as the window size of the average filter. Each participant performed the task 10 times; a total of 100 (10 participants 10 times) trials were conducted. In each experiment, task completion time was measured and the number of clicks was counted. The results are shown in Fig. 15. A one-way ANOVA with repeated measures showed statistically significant differences in both the task completion time (F (2,18) = 62.40, p < 0.01) and number of clicks (F (2,18) = 60.40, p < 0.01) between the three conditions. Post hoc analysis was then performed using a Student-Newman-Keuls test for pairwise comparison. It showed statistically significant differences between each pair of conditions in both the task completion time and number of clicks. Thus, both the task completion time and number of clicks are improved in W/ STAB compared to W/O STAB. Therefore, it is confirmed that the proposed simple stabilization method is effective, despite the time delay in the output of the stabilization. However, the participants performed better in the STATIC condition than in W/ STAB. Thus, Fig. 17 Results of user feedback user interactions may be improved with an additional estimation algorithm such as a Kalman filter. 6.3 User Feedback Ten participants, working in pairs, were asked to try the system. All the system features were demonstrated to each pair of participants in advance. Then they freely tried out all the functions of the system (Fig. 16). Each pair spent about five minutes trying all the functions: dynamic overlay of desktop graphics, file exchange, and data protection. Their behaviors were observed, and post-study interviews were conducted in which they were asked to rate six questions according to a 7-point Likert scale from 1 = strongly negative to 7 = strongly positive. The questions addressed the usability criteria in Table 1. In the experiment, all participants grasped the system concepts quickly and did not show any difficulty in learning the interaction techniques. All of them could not only copy files through the overlaid area but also protect their data from their partners by dragging a file icon out of the shared space or moving their devices. The results of the interviews are shown in Fig. 17.

9 IWAI and SATO: DYNAMIC OVERLAY OF DESKTOP GRAPHICS FOR CO-LOCATED MULTI-USER INTERACTION 9 The participants gave positive responses to all the questions. These results show that the proposed interaction technique offers users an intuitive file exchange interaction. 7. Discussion Pick-and-Drop (PnD) is one of the most famous data exchange interaction techniques for co-located devices [2]. Here we would like to compare the proposed system (CrossOverlayDesktop: COD) with PnD in terms of their design philosophies. PnD is a very powerful tool for copying of data between computers. It is obvious that users can transfer data from a computer to another with PnD more easily than with COD. However, sharing of data among the users is not explicitly dealt with in PnD. Therefore, PnD is not designed to provide access control of data with the users, which is one of the most important functions for data sharing. On the other hand, data sharing is explicitly considered in COD where the users are allowed to intuitively control access levels of their data. Therefore, in this paper, we showed one possibility to realize intuitive interactions for both copying and sharing of data. As described above, PnD and COD assume completely different interaction scenarios. In the proposed method, an access level for each data file can be assigned from a context menu that appears with a right click on the file icon. However, various possibilities exist for access control implementation. For example, the access level of a data file can be automatically assigned according to the category of coworkers. If a coworker is affiliated with a company, he (or she) may freely access the file (Public). On the other hand, if a coworker belongs to a different organization, the access level may become Semi-Public. Thus, one future work could be to explore other possible access control implementations suitable for the proposed file exchange interaction technique. 8. Conclusion This paper proposed an intuitive interaction technique for information exchange between multiple co-located devices. In the system, desktop graphics of the devices are graphically overlaid with each other according to the 6-DOF correlation of the devices. The proposed file exchange interaction technique allows users to exchange their files through the overlaid area with the familiar drag-and-drop manipulation. A simple average filter is applied to stabilize overlaid desktop graphics against inadvertent motion. Three access control methods were also proposed. In particular, the second method allows users direct and natural manipulations where they control accessibility of their private data just by moving data file icons. The users can also protect their private data by moving their own devices in a manner such that data file icons are outside the public space. We implemented the proposed methods and confirmed that they worked well in a prototype system. Three pilot studies were also conducted. In the first study, we confirmed that the proposed dynamic overlay of desktop graphics is intuitive even though movement of a device in a 3D physical space results in a 2D movement of the overlaid desktop graphics. In the second study, it was also confirmed that the applied simple stabilization method is effective despite a time delay in the output. In the final study, participants freely tried out all the functions of the system. They could grasp the system concepts quickly. Furthermore, they provided positive responses to all the qualitative measures we examined, such as understandability and difficulty of the proposed interaction technique. These results show that the proposed interaction technique allows seamless, consistent, and thus intuitive multi-user collaboration. As a future work, we plan to extend the proposed technique to remote user collaboration. References [1] D. Iwai and K. Sato, CrossOverlayDesktop: Dynamic Overlay of Desktop Graphics between Mobile Computers for Multi-User Interaction, In Proceedings of Human Interface Symposium 06, pp , (in Japanese). [2] J. Rekimoto, Pick-and-Drop: A Direct Manipulation Technique for Multiple Computer Environments, In Proceedings of ACM UIST 97, pp.31 39, [3] K. Hinckley, G. Ramos, F. Guimbretiere, P. Baudisch, and M. Smith, Stitching: Pen Gestures that Span Multiple Displays, In Proceedings of ACM AVI 04, pp.23 31, [4] P. Tandler, T. Prante, C.M. ller Tomfelde, N. Streitz, and R. Steinmetz, ConnecTables: Dynamic Coupling of Displays for the Flexible Creation of Shared Workspaces, In Proceedings of ACM UIST 01, pp.11 20, [5] J. Rekimoto, Y. Ayatsuka, M. Kohno, and H. Oba, Proximal Interactions: A Direct Manipulation Technique for Wireless Networking, In Proceedings of IFIP INTERACT 03, pp , [6] K. Hinckley, Distributed and Local Sensing Techniques for Face-to-face Collaboration, In Proceedings of ACM ICMI 03, pp.81 84, [7] X. Cao, C. Forlines, and R. Balakrishnan, Multi-user interaction using handheld projectors, In Proceedings of ACM UIST 07, pp.43 52, [8] H. Kato and M. Billinghurst, Marker Tracking and HMD Calibration for a Video-Based Augmented Reality Conferencing System, In Proceedings of IEEE/ACM IWAR 99, pp.85 94, [9] G. Kortuem, C. Kray, and H. Gellersen, Sensing and Visualizing Spatial Relations of Mobile Devices, In Proceedings of ACM UIST 05, pp , 2005.

10 10 IEICE TRANS. INF. & SYST., VOL.Exx D, NO.xx XXXX 200x Daisuke Iwai received the BS degree from Osaka University, Japan, in 2003 and the MS and PhD degrees in engineering science from Osaka University, Japan, in 2005 and 2007, respectively. He was a visiting scientist at the Faculty of Media, Bauhaus-University Weimar, Germany, from 2007 to He is currently an assistant professor at the Graduate School of Engineering Science, Osaka University, Japan. His research interests include human-computer interaction and projection-based mixed reality. He is a member of VRSJ, ACM and IEEE. Kosuke Sato received the BS degree from Osaka University, Japan, in 1983 and the MS and PhD degrees in engineering science from Osaka University, Japan, in 1985 and 1988, respectively. He was a visiting scientist at the Robotics Institute, Carnegie Mellon University, from 1988 to He is currently a professor at the Graduate School of Engineering Science, Osaka University, Japan. His research interests include image sensing, 3D image processing, and virtual reality. He is a member of IEICE, IPSJ, VRSJ, ACM and IEEE.