Interactive Phone Call: Synchronous Remote Collaboration and Projected Interactive Surfaces

Transcription

1 Intertive Phone Cll: Synhronous Remote Collortion nd Projeted Intertive Surfes Christin Winkler, Christin Reinrtz, Din Nowk, Enrio Rukzio MHCI Group, pluno The Ruhr Institute for Softwre Tehnology, University of Duisurg-Essen {hristin.winkler, hristin.reinrtz, din.nowk, ABSTRACT Smrtphones provide lrge mounts of personl dt, funtionlities, nd pps nd mke sustntil prt of our dily ommunition. But during phone lls the phone nnot e used muh eyond voie ommunition nd does not offer support for synhronous ollortion. This is owed to the ft tht first, despite the vilility of lterntives, the phone is typilly held t one s er; nd seond tht the smll moile sreen is less suited to e used with existing ollortion softwre. This pper presents novel in-ll ollortion system tht leverges projetor phones s they provide lrge disply tht n e used while holding the phone to the er to projet n intertive interfe nytime nd nywhere. The system uses desktop metphor user interfe nd provides privte nd shred spe, live mirroring of the shred spe nd user defined ess rights to shred ontent. We evluted the system in omprtive user study. The results of the user study highlight the generl enefits of synhronous in-ll ollortion nd in prtiulr the dvntges of the projeted disply nd our developed onepts. Our findings inform future designers of synhronous remote ollortion softwre for intertive surfes. ACM Clssifition: H.5.2 [Informtion Interfes nd Presenttion]: User Interfes Input devies nd strtegies; Prototyping. Generl terms: Design, Experimenttion, Humn Ftors. Keywords: Personl projetion, projetor phone, phone ll, ollortion, moile phone. INTRODUCTION Moile phones re nowdys used s pervsive intertion devies supporting lrge vriety of ommunition mens, servies nd pplitions. Surprisingly, the originl funtion of moile phones, voie ommunition, didn t enefit from the servies nd fetures dded to those devies in the lst dede. We mke frequent phone lls ut while doing so it s diffiult to use other pplitions ville on moile phones or to ollorte with the other prty. As Gunrtne nd Brush pointed out, there re mny situtions, when synhronous ollortion is desired during the synhronous voie onverstion, suh s shring pitures or diretions nd sheduling ppointments [7]. Currently, the only ville mens for remote ollortion re the usge of synhronous file exhnge protools (e.g., sending of pitures vi MMS or e-mil) or through the usge of entrl servers (e.g., using Feook for shring pitures). Moreover, it hs een shown tht within the intimy nd limited time spn of phone ll, people feel more omfortle shring privte informtion thn when shring to the puli, e.g. on Feook [7]. Furthermore, operting the phone whilst in ll requires dditionl onfigurtion or hrdwre, e.g., enling the loudspeker mode, whih is not pproprite in every ontext, or usge of hedset or erphone. In this pper we re going to explore the design spe of synhronous remote ollortion etween two prties during phone ll. Different from previous reserh on tht topi [7], we don t require the user to use dditionl hrdwre like omputer or hedset. We present the Intertive Phone Cll (IPC), whih is moile pplition designed to support two or more lling prties in synhronous d-ho ollortion nd dt exhnge during phone ll. Supported ollortion senrios inlude, ut re not limited to: shring pitures, lotions, diretions, wesites, presenttions, nd sheduling ppointments. To provide for usul ll experiene during ollortion, IPC prtiulrly supports projetor phones. Those moile phones with uilt-in projetors llow the projetion of reltively lrge intertive disply in front of the user while holding the devie t one s er s usully (Figure 1). IPC uses desktop metphor split up into two djent spes, whih n e Permission to mke digitl or hrd opies of ll or prt of this work for personl or lssroom use is grnted without fee provided tht opies re not mde or distriuted for profit or ommeril dvntge nd tht opies er this notie nd the full ittion on the first pge. To opy otherwise, or repulish, to post on servers or to redistriute to lists, requires prior speifi permission nd/or fee. ITS 2011, Novemer 13-16, Koe, Jpn. Copyright 2011 ACM /11/11...$ Figure 1: IPC modes. Left: IPC Projetion, Right:

2 resized in fvor of one or the other, in order to leverge the lrger spe to provide more plesnt ollortion experiene. The left side of the projetion is used for privte view displying personl dt nd pplitions ville on the moile phone. The right side is used for displying shred spe, whih is synhronized in rel-time etween oth users. This llows instnt shring nd intertive disussion of files nd pplitions y moving files nd pplitions from the privte to the shred spe. As ll intertion hppens synhronously, ny nnoying metonverstions regrding the stte of shring eome unneessry. The IPC n e used with projetor phones (IPC Projetion mode, Figure 1 left), ut s well on the sreen of onventionl smrtphones ( mode, Figure 1 right). Due to the smller ville disply size in IPC Sreen mode the user sees only the privte or the shred spe t time nd moves etween spes with horizontl swipe gesture. The underlying pproh of the privte nd shred spe of IPC is extended, mong others, through onepts like olor sed ownership oding, showing whih ojet elongs to whih user, nd opy permission ontrol defining whether the other person n only see or n lso opy nd mnipulte dt from shred spe. After disussing relted work, we present the IPC nd its underlying onepts. Lter on, we ontrst the two different IPC modes (IPC Projetion, ) with nother moile in-ll ollortion tool (Sreen Shring) tht supports moile sreen shring etween lling prties with ntive phone softwre ut without support for projetion. Through omprtive study etween IPC Projetion,, nd Sreen Shring, we ompre the onfigurtions with/without projetion nd with/without IPC onept. The results of this study highlight the dvntges of the projetion nd the IPC pproh in terms of lrger sreen spe, etter privy ontrol for shring, nd tive loudspeker mode not eing required. Moreover, the study revels the generl enefits of remote synhronous ollortion support tht is missing in ontemporry systems. RELATED WORK The work presented in this pper uilds on personl projetors nd projetion phones, diret intertion with personl projetions, synhronous remote ollortion, ollortion vi moile devies, nd privy while shring. Personl projetors nd projetor phones In the lst 3-5 yers we oserved the emergene of pioprojetor tehnology tht llows the integrtion of personl projetors into moile nd werle devies [13]. More reently the first projetor phones, moile phones with uilt-in projetors, ppered in the mrket. Prominent exmples re the Smsung Bem, LG Expo nd doomo PRO series SH-06C. This development hs minly een driven y the possiility to inlude pio-projetors with n eptle power onsumption nd rightness. The envisioned usge senrios fous minly t the possiility to show pitures, videos, nd presenttions nywhere on ny surfe. The potentil nd hllenges in terms of intertion design, privy, nd ollortion re not yet een ddressed y urrently ville devies. Diret intertion with personl projetion Rukzio et l. identified four oneptully different pprohes to intert with personl projetion: input on the projetor, movement of projetor, diret intertion with projetion nd mnipultion of projetion surfe [13]. When onsidering the speifi ontext of nswering phone ll, then only the diret intertion with the projetion n e onsidered s the projetor phone hs to e held lose to the user s er. The first to explore this pproh were Kritsuk nd Sto who used shoulder-worn projetormer unit tht projeted on notepd held y the user [9]. The mer trked the position of the user s finger on the notepd, whih enled touh-sed intertion with projeted informtion. The Briny Hnd system extended this pproh y using the user s hnd s the projetion surfe nd hedset, whih inludes projetor nd mer [17]. Wer Ur World (etter known s SixthSense) is system worn s pendnt or hed-mounted devie whih supports projetion t ny surfe nd diret pointing sed intertions with the projetion [11]. Other diret intertions were investigted were telesopi stik, lser pointer or the user s oots [10] were used to ontrol the intertion or where the user s forerm s the projetion surfe ws used s the projetion surfe [8]. The intertion onept of the Intertive Phone Cll is different from previously investigted settings s touh-sed input on ny projetion surfe (e.g. tle or wll) is onsidered, s the user is holding the moile phone with one hnd lose to the er, nd s only the remining hnd n e used for diret input. Synhronous remote ollortion Synhronous remote ollortion tht goes eyond phone lls hs lredy een investigted in the 1970s y Chpnis et l. nd it hs een shown tht visul ollortion improves tsk ompletion mesurly [5]. Sine then we sw very lrge ody of reserh nd ommeril produts in the re tht often involves the usge of n udio / video link nd live shring of pplitions, douments nd the desktop. Nowdys multitude of pplitions suh s Mirosoft Lyn, Windows Live Messenger, Adoe Connet, or Skype (sreen shring) re used. Collortion vi moile devies The most reserh onerning ollortion with moile devies fouses on o-loted ollortion. Here severl users e.g., intert diretly with eh other vi short-rnge network onneting their moile devies to exhnge files [2], use puli disply s meditor for the ollortion [12], or shre their lotion informtion with others [4]. There exists reltively little reserh on synhronous remote ollortion etween two users lling eh other with their moile phones. The PlyByPly system supports ollortive rowsing etween two users wherey one is using moile devie (e.g. lling nd sking for diretions) nd the other one is using desktop PC [20], whih oth re synhronized. The Newport system lso supports ollortion etween lling prties of whih t lest one person is

3 lose to omputer [7]. The users re le to send eh other mps, photos, or notes tht n e nnotted ut no live sreen shring is supported unless oth persons sit in front of omputer. The ommeril system Thrutu [21] enles in-ll ollortion etween smrtphones ut is designed for eing used with loudspeker mode on the moile disply. In ontrst, oth Newport nd Intertive Phone Cll (IPC) support ollortion during phone ll on lrge shred disply, ut IPC provides distint dvntges s no desktop omputer is required. Privy while Shring Moile phones re onsidered s very privte devies s they often ontin informtion out personl ommunition (e.g. phone lls, SMS or emil) nd store privte medi. This hs e.g. een ddressed y the work of Grriss et l., whih showed the importne of user privy during moile intertions with puli kiosks [6]. MoShre is photo shring system for moile devies, whih onsiders privy spets refully s it llows users to define expliitly with whom whih pitures should e shred [14]. Ahern et l. onfirm tht people re in prtiulr onerned out the pitures stored on their moile phones when onsidering personl seurity nd soil dislosure [3]. Intertive Phone Cll ddresses this spet vi the privte nd puli spe. If the user wnts to shre file then the user hs to move it expliitly from the privte into the puli spe. INTERACTIVE PHONE CALL (IPC) The IPC onept enles users to rowse, shre, nd opy personl dt nd ollorte in rel-time during phone lls. We dded synhronous ollortion hnnel to the voie ommunition to resemle o-loted ollortion s losely s possile. We think tht in this ontext the two most importnt qulities of o-loted ollortion re tht ollortion hppens synhronously, i.e. tions of one person hve n immedite effet on the pereption of other nery persons, nd idiretionlly, i.e. ll persons n mnipulte the sme ojets t (lmost) the sme time. To the est of our knowledge exists no solely moile idiretionl ollortion system nd therefore we looked t existing systems tht support remote ollortion nd dt shring with the help of desktop omputers, suh s Newport [7], Skype, Windows Meeting Spe, Windows Communitor, Adoe Connet, nd Ciso or VNC produts. We found tht these systems uild on quite similr user interfe onepts for shring. The most prominent of these is tht users hve to hoose etween shring their desktop/pplition nd wthing nother user s desktop/pplition, sometimes with the option to tke input ontrol of the remote desktop. Another ommonly found onept is shring files synhronously, peer to peer y mens of Drg & Drop of ioni file representtions. We were surprised how few existing systems support synhronous nd idiretionl ommunition t the sme time. Yet we re onfident tht the synhronous nture of lls demnds for fitting synhronous shring experiene. Otherwise the phone onverstion would likely e luttered with phrses like Hve you lredy sent the file? I didn t reeive it., Hve you lredy opened my file nd see, nd I hve the file here, ome nd wth my sreen. While this my e tolerle in trditionl remote ollortion, moile phone lls re often likely to hppen on the go nd lst reltively short time. Therefore, ommunition nd ollortion hs to hppen s effiiently nd effetively s possile. Aprt from how shring is supported, in-ll ollortion on moile devies entils some more hllenges speifi to moile phones. Among those re the onsiderly smller sreen spe (even of projetion); the stored very personl dt tht is likely to rise privy onerns during ollortion; nd tht moile phones do not hve desktop interfe (multiple prllel pplition windows) like PCs. The IPC should s well support user s moility during phone lls, like when eing t home, on the go, eing le to projet or eing le to tivte the loudspeker, respetively. Finlly we wnted to expliitly support projetor phones for the igger disply spe they n provide whih we ssumed to e very enefiil for the tsk of o-loted ollortion. In the following we will present the IPC nd its onepts, eh inluding our design onsidertions nd how they got refleted in the implementtion. The lst setion will then present some pprent use ses supported y our implementtion of the IPC. IPC Conepts Surfe vs. Phone Metphor Computer nd phone user interfes developed quiet differently in the pst ording to their diverse usge requirements nd the ville sreen spe. While the WIMP (Windows, Ions, Menu, Pointer) metphor, inluding Drg & Drop, eme very widespred on omputer operting systems, modern moile OS t lest ndoned the Windows, Pointer, nd sometimes s well the Drg & Drop onepts. This holds true for projeted interfes of urrently ville projetor phones, whih just mirror the moile interfe. As the moile phone offers more nd more desktop PC funtionlities, onepts for dt, file, nd ojet mnipultion regin importne. An exmple is the Wetop frmework on the Motorol ATRIX phone tht resemles stndrd WIMP desktop interfe when onneted to igger HDMI disply [20]. Sine users re more fmilir with onepts for dt shring stemming from trditionl omputer operting systems, we deided to uild on these y using desktop-like interfe. The moile desktop shows sttus r, pplition ions, nd title-less windows for every opened ojet or pplition (Figure 2). Every window n e moved, sled, or rotted with Drg & Drop, Pinh to Zoom or two-finger rotte gestures. Content inside windows is mostly mnipulted with single finger touhes. Thus our interfe uilds on modified WIMP onepts s they re lso used in urrent tletop pplitions tht, e.g., trget multi-touh tles. Different from existing tletop pplitions, we lso llow the user to intert with the surfe y just hovering over it.

4 Cllee (one of possily severl) Ioni representtion of llee s gender nd IPC mode (IPC Projetion) Ken shred the piture, Lis my opy it Sttus r showing ll OS notifitions Cn e moved to resize spes Clendr view with Detils ontrol for merged view in Shred Spe Piture stemming from the devie After opying, oth own the piture Finger hover over ojets revels lose utton App ions similr to Android OS Piture opied from shred spe Lis shred the piture, Ken must not opy it Hide/unhide overlyed help inform. Show Desktop: Hide/unhide ll window ojets You (urrently in mode) Toggle defult opy permission Toggles freehnd pinting on shred spe (for use with mp for exmple) Figure 2. IPC GUI: Ken nd Lis shre pitures, mps, nd ppointments synhronously during the Intertive Phone Cll. This is for exmple used to disply lose utton on windows or hints on ertin elements only when the user s finger is lose to it (Figure 2 right middle). ritil in shring privte dt. Different window order olors re used to show the origin nd ownership of windows/ojets. The own olor is lwys the sme, i.e. drk lue, wheres lling prties hve different olors. Eh window order n thus rry numer of different olors up to the totl of ll prtiipnts. When windows re drgged from privte to shred spe they reeive order in the owner s olor (Figure 3, Piture ). When windows re opied from shred to privte spe, the ownership of the person opying the window is dded to the shred ojet, giving feedk to the former ojet owner(s) tht the item hs een opied (Figure 3 right). Shre nd Copy etween Privte nd Shred Spe The desktop spe is further divided into privte nd shred spe, whih n e resized in fvor of one or the other spe with the divider in the middle (Figure 3) to ount for hnging spe requirements. The shred spe is seen y ll other ll prtiipnts nd syned in rel-time, inluding window movement nd ontent mnipultion. Eh prtiipnt n shre windows y mens of Drg & Drop from the privte to the shred spe, nd opy windows in the opposite diretion, if permitted y the window s owner (Figure 3). The originl window returns to its former ple fter it ws shred or opied nd n identil opy is reted t the ple where it ws drgged. Supporting Copy Rights nd Privy Moile phones store lot of very personl dt, e.g., emils, SMS, pitures, ontts, ppointments. Mking this dt ville for shring is likely to rise privy onerns. Thus giving users ontrol over opy permissions nd the grnulrity of shring is importnt. The shred spe on the right, mirrored etween ll ll prtiipnts, mkes it ovious, whih windows re urrently shred nd how, even t whih size-level, ll prtiipnts urrently view them. If permitted y the owner, ontent n e mnipulted ollortively in shred spe nd opied to the privte spe t ny time t everyody s disretion. For IPC we differentite minly etween three different types of shring: not shred, view-only nd mnipulte & opy. The onept of privte nd shred spe lredy serves the not shred type, s nothing is shred tht is not expliitly drgged into the shred spe y the user. The user n further toggle etween view-only nd mnipulte & opy y opening or losing lok utton t the window s orner. The lok utton is dded to ny window tht my present privte informtion when the window is drgged from privte to shred spe ross the divider, inheriting the stte of the defult lok utton tht sits on the divider s initil stte (Figure 3d). As the nmes suggest, the window n only e moved nd hnged in size nd rottion s long s the lok is losed nd freely mnipulted, inluding ontent, nd opied when the lok is open. The rel-time synhroniztion further enles people to dd visul ommunition like gestures to their spoken words s they would do with physil ojets when o-loted. If someone tlks out ertin piture for exmple, they might point to it, gr nd move it, or even point to ertin positions in the imge. Consequently, the onept of privte nd mirrored shred spe voids ll synhronous intertion during the synhronous phone ll. Ownership olor oding Giving the user feedk out whih of the shown informtion elongs to them, the other lling prty, or oth is Privte Shred Privte Shred Privte Privte Shre Copy Shred ) Resizle privte & mirrored shred Spe ) Shre & opy vi Drg & Drop Privte Shred ) Ownership olor oding Figure 3. Intertive Phone Cll onepts Privte Shred Privte Shred Shred d) Defult & individul opy permission

5 Unfortuntely, the spes nd the lok onepts nnot serve every sitution dequtely. A lendr for exmple hs to show mny individul ppointments in one window t the sme time, whih mens they n only e shred ll together or not t ll. We think tht these ses n e est solved individully from one ontent type to nother. A lendr window, for instne, n hve n dditionl detils utton tht n e toggled in privte spe to influene shred spe. When the utton is off, only free/usy is shown in shred spe without ny further detils out the ppointments (Figure 2 left). In ft, detils utton is le to solve lot of privy issues, ut proly not ll of them. Supporting Freehnd Annottions (The Penil) Despite support for rel-time mirroring of the shred spe, some informtion n e muh etter visulized with the help of freehnd-skethed nnottions. Therefore, ll ll prtiipnts n tivte penil to drw on top of the whole shred spe in their respetive olor. This wy ontent suh s mps nd pitures n e nnotted or freehnd drwings n e pinted on the shred spe. Phone nd Cll Sttus The IPC softwre supports fetures similr to the stndrd ll pplition. It shows the phone s sttus r (ttery, signl strength, et.) t the top of the privte spe nd ll ll prtiipnts y gender-wre ions, nmes nd ontt pitures. Strting lls from the ontt list within the pplition is supported s well s ending lls y liking on the red reeiver ion on top of the other lling prty, whih is reveled s soon s the finger hovers over it. Stteful spe Moile Phone lls re inherently frgile. The onnetion n drop when one prty moves, loses network overge, or phone runs out of ttery. Soil irumstnes n disllow ontinuing the onverstion. During onverstion supported y IPC, lots of dt nd nnottions n e reted in the shred spe tht must not e lost y ident. Therefore, if dropped session is reopened, the prtiipnts re sked whether they wnt to ontinue their old shring session or strt new one. Without nother onnetion however, the shred spe nnot e essed to not undermine the privy rules of other ll prtiipnts, who intended showing their ontent only for the time of the ll. Therefore, ontent the user wnts to store persistently hs to e opied to the privte spe efore the end of the ll. Swithing IPC Modes nd Sttes An Intertive Phone Cll n e in one of severl sttes (shring or not shring), modes (IPC Projetion, IPC Sreen, or Non-IPC) nd onfigurtions, whih result from multiple users ollorting in different modes. 1. ll 2. ept ll 3. strt shred spe 5. ept shred spe 6. shred spe estlished 4. get notified 4. optionlly: strt projetion Figure 4: IPC sttes. Hndshke efore shring strts. IPC Sttes Aprt from ll support, the IPC softwre n lso e used without n ongoing ll to use the lrger projeted touh surfe to intert with personl ontent. When ll is oming in (Figure 4, 1), it is visulized on the projetion nd n e nswered from there (2). One nswered, the ioni representtion of the other lling prty is pled on the shred spe of oth users. However, the shred spe is greyed out sine shring ontent my not e the intention of ny ll prtiipnt. To enle ontent shring, one lling prty must lik the lrge Strt ommon spe utton tht overs the shred spe (3). When liked, the other prty reeives unique notifition (virtion or udio) (4) to indite tht she must tke tion to either ept or deny the shring request tht is presented on the shred spe; this might require swithing to IPC Projetion or IPC Sreen mode (4). This initil hndshke out shring prevents unknown llers from presenting indequte ontent without prior onsent. One the request is epted (5) the shred spe is estlished (6) nd prtiipnts n shre ontent until the end of the ll. IPC modes Tody, most people still telephone y holding the phone lose to their er. Despite lterntive options s loudspeker mode or the usge of hnds-free essories, the dvntge of the originl telephone ehvior is tht it does not require dditionl hrdwre like hedset, whih my not e ville t the moment or hs run out of ttery. Further it is muh more unotrusive thn loudspeker mode nd n e used in reltively noisy environments. We think projetion n serve mny irumstnes where loudspeker mode is no onsiderle lterntive. However, there my e situtions where this is vie vers. Therefore, IPC supports oth Projetion nd Sreen mode nd semless swithing in etween. Sine the urrent mode ffets the ville spe for shring, these stte hnges hve to e ounted for during lls s well (see next setion). In totl IPC knows three different modes it n operte in s depited in Figure 5. The user n yle through modes y pressing the projetor hrd utton on the side of the phone, whih urrent projetor phones suh s the Smsung Bem offer. In Non-IPC mode, the IPC softwre runs in kground, only, nd listens for utton presses nd shring requests. The IPC Projetion mode shows the IPC user interfe desried erlier on the projeted disply. The user holds the phone to her er nd interts with the projetion with her free hnd. For the version, where the phone Non-IPC projetion off, loudspeker off single press projetion on, loudspeker off doule press projetion off, loudspeker on single press Figure 5: IPC modes, hnged y different presses of the phone s projetion utton.

6 is held in front of the user, we onsidered whether we wnted to use the very sme desktop UI on the muh smller phone sreen or develop n lterntive UI tht resemles more the UI of stndrd phone softwre. Sine we did not wnt to hve ompletely different UIs for the sme pplition to not overstrin users, we only slightly dpted the projeted version to the moile sreen. Due to the smller spe on the moile sreen, we deided tht privte nd shred spes re exlusively visile, only. The user n move etween spes mnully y performing horizontl swipe gesture on the sreen or utomtilly y drgging ojets etween spe orders. Sine views re seprte, the moile version does not llow resizing spes. IPC size onfigurtions As mentioned erlier, our gol is to support semless swithing etween IPC Projetion nd mode. When user swithes from Projetor to Sreen mode, the sizes of his privte nd shred spes shrink onsiderly. If the other lling prty is in projeted mode, the shred spe sizes tht re syned 1:1 re not equl ny more. To ount for tht, the spe ville on the moile sreen is highlighted on the shred spe of the user in Projetor mode nd remining spe is greyed out. Group shring nd ollortion During the design phse of IPC we took re to limit our onepts not to two-person setup, ut find solutions tht would sle to onferene lls with multiple users. The presented onepts, in prtiulr the shred spe with ioni representtions of ll prtiipting users, olor oding, lok, nd penil work eqully well for onferene ll with multiple prtiipnts. IPC Use Cses Clendr The lendr pplition llows drgging the own lendr to the shred spe nd stking lendrs of different persons on top of eh other to merge them nd see ll ppointments together with orresponding olor oding for eh dy nd ppointment. If new ppointments re dded in shred spe, they re dded utomtilly s shred events for ll ll prtiipnts tht own the merged lendr, i.e. ll persons tht drgged their privte lendr on top of the shred instne efore. Pitures nd Videos Similr to the ntive phone gllery softwre, medi files n e opened from list of ville gllery lums nd their windows n e resized, rotted, or shred. Presenttions Shring presenttion n e used to disuss revisions efore the tul presenttion is onduted. Furthermore, if the intended udiene is not o-loted, the presenttion n e moved to nd onduted in the shred spe during onferene ll. The IPC softwre supports shring presenttions, moving etween slides in the shred spe nd nnottions y mens of the penil. Live Cmer Imge A live strem from the mer pturing the re in front of the user n e shred if the moile phone s mer is loted on the sme side s the projetor. Anything in the view of the user tht is relevnt to the onverstion, suh s hnd gestures, sight of eutiful lndspe, items of interest while shopping or pitures on trditionl pper, n esily e rought into the ollortion session. PROTOTYPE In the following we present our system setup, whih we designed to fit the trgets of the susequent user study. To riefly summrize these trgets, we wnted prtiipnts to engge in rel phone ll while performing some ollortion tsks on the projeted nd the sreen interfe. We hd to develop projetorphone prototype tht projets retngulr surfe in front of the user whilst held t the er; system tht trks the user s fingers nd mps it to the projeted surfe, independent from the surfe s position; the IPC softwre with its underlying onepts nd support for some ontent types (pitures, lendrs, mps, presenttions); nd finlly integrte forementioned with the projetorphone to hieve the IPC Projetion nd IPC Sreen modes. Moreover, we hd to doule the system nd onnet oth systems to hieve relisti ll senrio etween two persons. The resulting overll system setup for the Intertive Phone Cll is depited in Figure 7. For resons desried lter, the IPC softwre runs on omputers tht re vi LAN onneted to eh other, the system tht mnges finger trking, nd Projetor phone A IPC Projetion (VGA) (VNC) ShowWX+ IPC-Computer Cll ontrol (TCP) Nexus S Figure 6: Merge lendrs y stking them. Mps Mp windows inlude Google Mps rowser window, serh field nd zoom nd nvigtion ontrols. They n e used to ring ertin lotion into view nd then shre it, or to ollortively explore the mp in shred spe. Annottions n e used to mrk users urrent lotions, s well s spots to meet, or prk the r. IPC syn hnnel (TCP) Projetor phone & finger position (UDP) Projetor phone B Optitrk-System Sme s Projetor phone A Figure 7: System setup. Two synhronized PCs running the IPC softwre reeive input events from the trking system nd deliver their output vi VGA to phone projetors nd vi VNC to phone sreens. Cellulr line

7 Figure 8. The IPC hrdwre prototype. A SHOWWX+ Pio Projetor is equipped with retro refletive mrkers nd y two luminum ngles nd velro tpe orthogonlly nd with height offset (for etter hndling nd lrger projetion) tthed to Nexus S phone. dditionlly to respetive projetorphones vi VNC. The udio onnetion etween lling prties is over stndrd ellulr line from one phone to the other. Projetor Phone Prototype Beuse no suitle projetor phone ws on sle, we uilt projetor phone prototype tht onsists of Smsung Nexus S Android phone tthed to Mirovision SHOWWX+ lser pio-projetor Figure 8. In order tht our projetor phone n e held s usul, i.e. to the er nd prllel to the fe, we hd to tth the devies orthogonlly, sine we wnted the projetor to projet lndspe imge in front of the user. Additionlly we tthed retro refletive mrkers to the projetor, in order to trk it with the 6DOF infrredsed trking system OptiTrk from NturlPoint. The usge of lser projetor further llows the user to hnge height nd ngle of the projetion without the imge loosing fous. At typil distne of out 60m etween er nd enter of the projetion on the tle, the SHOWWX+, thnks to its wide projetion ngle, projets right 45x20m-sized imge with resolution of 848x480 pixels. Beuse of the reltively short projetion distne even smll text (> 11pt) ould e esily red. Another dvntge of the projetor-t-er setup is tht ny jitter of the projetion is lmost unpereivle to the user sine hed nd projetor move simultneously. In theory, jitter might eome prolem while the user tries to touh the surfe, ut hrdly ws n issue in our tests nd studies. Oviously, phone nd projetor re only tthed ut not diretly onneted to eh other. Nevertheless, euse the phone is idiretionlly onneted to PC nd the PC delivers its grphil output to the projetor, we hieved the illusion tht phone keys, in prtiulr the on/off utton on the side of the Nexus S, yled through different IPC modes s desired. We lso implemented nti-distortion of the projetion nd posture-independent finger trking in order tht users n independntly move, rotte, or hnge the height of the projetor during ll to support their omfortleness. However, users re ound to the lirted frme of the trking system whih does not let them stnd up or move wy from the tle in front of them. Anti-Distortion of Projetion We expet users of IPC to e more omfortle with lrgely undistorted projetion sine they re likely more used to retngulr desktop interfes. However, s [18] pointed out, it hs still to e studied in whih senrios ntidistortion enrihes the user experiene notly. Projetions follow the ontrry perspetive lws of the humn eye: the frer the distne etween projetor nd projetion surfe, the igger the projeted imge gets. We n provide for n undistorted imge on the projetion surfe y projeting ounter-distorted imge. A ounter-distorted imge is the sme imge 3D progrm would rete to imitte the perspetive view of the projeted imge from the position of the projetor. We refer to [4,18] for the required omputtionl steps to ompute the projetion mtrix. The user s fingers re trked in surfe oordintes nd mpped k to pplition oordintes y multiplying their position with the inverse projetion mtrix. IPC Softwre The IPC softwre (Figure 2) ws uilt with C# nd Mirosoft s Windows Presenttion (WPF) nd Windows Touh frmeworks, euse urrent moile phone UI frmeworks re neither designed nor suited for tle-top intertion nd the muh lrger disply spe. The projeted version is therefore served from PC vi VGA to our projetor phone. The version runs on Android OS nd uilds upon Android VNC Viewer to disply nd ontrol the sme IPC softwre vi VNC, showing one spe t time s desried erlier. The VNC onnetion introdues smll lteny, ut whih is lmost unpereivle from user stndpoint. Furthermore, the Android pplition runs servie in the kground tht ommunites phone stte (lling, ringing, off the hook) to the PC nd reeives ontrol ommnds like strt ll to x or end ll with x from the PC. The PC uses RelVNC for trnsmitting the Intertive Phone Cll output to the moile phone. Our hosen setup hs the dvntge tht it presents the user with the sme user interfe on the projetion nd the sreen, while still retining most ffordnes of the phone like hrd uttons nd the touh sreen. Rel World Deployment Considertions A rel deployment of the system poses two mjor hllenges. First, the rottion of the projetor ginst the surfe must e known to e le to projet ounter distorted imge. Seond, finger touhes must e reognized through some kind of optil trking system tht is onord the projetor phone. If the system further ws to support unplnr projetion surfes s well, these must e deteted through optil surfe estimtion s well. Although there is no stndrd, redy-to-use method ville, reent reserh nd produts showed how suh system ould e relized. The orienttion of the devie n e sensed with the help of inertil sensors like elerometers nd gyrosopes lredy present on smrtphones. This however would require n initil lirtion step whenever the user hnges the projetion surfe. If the phone fetured mer looking in the sme diretion of the projetion, the mer ould pture smll visul mrkers in the orners of the projetion to estimte sptil reltion nd distne etween projetor nd surfe without lirtion. Sugimoto et l. showed how this ould even e extended to support un-

8 plnr surfes [16]. One the sptil reltion is known, finger trking ould e relized through shdow trking like presented y Wilson [19]. Aprt from tht, very muh like 3D mers nd 3D displys lredy found their wy into moile phones, we expet depth mer systems like Mirosoft s Kinet to e introdued to projetor phones in the future. Suh system delivering depth informtion would lrgely simplify the omputtion of the projetion surfe s well s the finger trking even on un-plnr surfes nd without disturing the user visully or with lirtion proedures. In regrd to the rel-time synhroniztion, reent dvne in moile dt networks, inluding tody s support for video shring, mkes us onfident tht the reliztion of IPC s shred spe is fesile tody or t lest in the ner future. PRELIMINARY EVALUATION We onduted preliminry evlution to reeive user feedk on the IPC system. Further we wnted to nlyze nd distinguish () the effet of projeting the interfe during the ll insted of using the moile sreen nd () the extend to whih the IPC onepts enhne in-ll ollortion ompred to stndrd phone softwre. For the ske of () we treted the IPC Projetion (C1) nd (C2) modes s two seprte study onfigurtions, whih only differ in using the projetion or sreen. For the ske of () we introdued nother moile in-ll ollortion system, the Sreen Shring system, s third study onfigurtion (C3). Sreen Shring Prototype Configurtion 3 (Sreen Shring) dds support for sreen shring nd remote pointing to stndrd smrtphones. The softwre onsists of smll widget of four uttons (Figure 9) tht resides on top of ll other pplitions on the phone sreen t ny time. With the lower uttons (3 nd 4) the user n ontrol sreen shring. At the sme time they serve s inditor for the urrent stte of shring. () shows the initil stte where the user hs not yet shred his own spe (left ion) nd neither hs the other ll prtiipnt (right ion). With long press on (3) the user n llow (or prohiit) shring her own sreen. () shows the stte fter oth prtiipnts llowed sreen shring. The user n invite the other prtiipnt to strt wthing her sreen with norml press on utton (3). This my result in stte (), whih indites to the user tht the other prtiipnt is urrently wthing her sreen. With nother press on (3) in stte () the user n off-lod the other prtiipnt from wthing her sreen, whih is more polite thn disling shring ompletely with long press. By pressing (4) while the other prtiipnt llows shring, the user strts wthing the other prtiipnt s sreen (d). Pressing (4) in (d) stops wthing gin. Moreover, (4) n e pressed in () or (3) in (d) to diretly swith etween wthing jointly the own or the other s sreen. With (1) the whole widget n e moved to the next sreen orner to give free sight on the re elow. With the pointer hnd (2) oth users n idiretionlly visulize their sreen touhes with seprte olors to eh other while viewing the sme sreen. We dded support for sreen shring nd pointing to uild system tht only uses stndrd phone softwre nd t the sme time llows solving the tsks of the user study. Content n e shred view-only through sreen shring nd nnotted y mens of the pointing hnd. Content n further e opied y e-mil or MMS. Study Prtiipnts nd Setup For our user study we reruited 14 students (6 femle, 8 mle, ge 20-25) nd pid eh of them 10 Euros for their prtiiption. All of them were experiened smrtphone users nd 4 of them hd moile dt flt rte. Prtiipnts were explined ll onfigurtions t the eginning nd prtiipnts ould explore the three onfigurtions on their own then fterwrds. During the study one experimenter styed with the prtiipnt to give instrutions on the tsks to perform nd to ssist if prtiipnt would get stuk in solving tsk. Another experimenter in nother room plyed lose friend of the prtiipnt nd engged with prtiipnts in rel onverstion over the phone. We hose this setup to mimi rel senrio while t the sme time ssuring tht ll onverstions nd tions took lmost the sme ourse. We employed within-sujets design, i.e. during eh ll prtiipnts used one of three forementioned onfigurtions: IPC Projetion,, nd Sreen Shring. The order in tht prtiipnts were exposed to the onfigurtions ws ounter-lned; the order of the tsks ws lwys the sme. The experimenter in the other room used the sme hrdwre nd onfigurtion s the prtiipnt. Moreover, oth experimenters followed detiled sript to synhronize their intertion nd to ensure tht ll intertion with prtiipnts ws very similr ross ll study sessions. After eh onfigurtion we sked prtiipnts 12 questions, whih ould e nswered from strong disgree to strong gree (5-point Likert sle). We lso sked out pereived dvntges nd disdvntges of the system. After finishing the third onfigurtion nd questionnire we sked prtiipnts to ompre nd rte the onfigurtions in terms of performne nd personl liking nd tell us when, where, nd why they would use suh systems. One study session lsted pproximtely 80 minutes. Figure 9: The Sreen Shring widget.

9 Study Proedure Prtiipnts hd to fulfill series of tsks with eh onfigurtion. The first ws to ll the experimenter nd to estlish the ollortion with the mens provided y the present onfigurtion. Similrly, the lst tsk ws to end the ll. In etween, users hd to perform the following four tsks: (1) Open the gllery pplition, selet nd shre own pitures nd opy pitures from the other prty. One own piture ws onsidered privte nd therefore the prtiipnt hd to ensure tht the other lling prty ws not llowed to opy it. (2) Reommend ple to meet for offee to the other lling prty, whih pretended to not know the ple. Here the user hd to open the Mps pplition, look up the ddress y entering serh phrse, optionlly pn nd zoom, nd then shre the mp entered on the destintion. Further she nnotted it (with the penil) sed on questions sked y the experimenter on the phone. (3) Prtiipnts hd to open the lendr pplition, merge their lendr in shred spe with the lendr shred from the other prty, nd find free slot for n ppointment in the merged lendr. The other prty dded the ppointment to the merged lendr nd instruted the prtiipnt to hek the ppointment in her lendr in privte spe. (4) Finlly, the prtiipnt ws requested to open nd shre leture presenttion she hd on her phone in order tht the other prty ould look up speifi slide. During the Sreen Shring onfigurtion, pitures, mps, nd presenttions were presented through sreen shring. Pitures nd presenttions were exhnged vi e-mil. Clendr entries were viewed seprtely with turn tking sreen wthing nd the ppointment ws dded y oth lling prties seprtely. RESULTS Overll, prtiipnts gve us very positive feedk out ll three systems. In regrd to the question if nd where they would use suh systems they reported they would use them in lmost every lotion nd sitution. In the se of IPC Projetion we reeived some nswers like I would use it only in privte res, like t home or t work, ut more feedk like everywhere I m where I hve lrge projetion surfe nd I would use it in puli trnsport if spe llows. One prtiipnt highlighted the ft tht not projetion or loudspeker must e used, ut tht projetion nd loudspeker n e used together for n esy to set up teleonferening system etween remote groups. Input nd Output In the finl omprison of the onfigurtions, 12 of 14 prtiipnts mentioned the igger spe s dvntge of the projeted interfe nd 5 tht the phone n still e held t the er. Unfortuntely, the IPC onfigurtions ould not provide the very sme input experiene s the ntive Android pplitions in the Sreen Shring onfigurtion (s nswers to Q3 in Figure 10 show). In IPC Projetion this ws due to the trking system sometimes not reognizing Sreen Shring Sreen Shring Sreen Shring Sreen Shring Sreen Shring Sreen Shring Sreen Shring 0 2 strong disgree Q1 I ould solve the ssigned tsks quikly. Q2 The system supported me lot in ommuniting non-verlly. I did not experiene ny Q3 prolems giving input to the system (e.g., touhing the surfe or sreen). The ville grphil spe Q4 ws more thn enough for solving the tsks. Q5 Privte informtion I ould hide from the other prty. At ny time I surely knew whih Q6 items elonged to me nd whih were stored on my phone. At ny time I surely knew whih Q7 ontent items the other prtiipnt ould see Rting Count disgree neutrl gree strong gree Figure 10: Seleted nswers from the questionnire prtiipnts filled in fter eh onfigurtion. user s finger urtely for liking or the lik ws reognized with slight offset when the projetor ws tilted eyond ertin ngle. No user omplined out the projetion jittering. In there ws smll notiele lg in interting with ojets due to onstrints of the VNC onnetion. Responses to Q4 further show the desktop metphor ws the right deision for the projeted interfe ut it does not perform so well when used on moile sreen. Collortion nd Privy Answers to Q1 nd Q2 indite tht the three onfigurtions filitted ollortion nd tht ll tsks ould e solved in resonle time. With more roust input in IPC onfigurtions, we expet IPC pprohes to perform t lest s fst s Sreen Shring. One prolem tht prtiipnts hd with the onfigurtion ws tht euse they ould only see one spe (privte or shred) t time, they sometimes missed n tion the other prtiipnt performed in shred spe. The iggest differene however, we found in the pereived support of users privy. Answers to Q5, Q6, Q7 indite tht IPC onfigurtions performed etter in supporting the user s privy. In the Sreen Shring onfigurtion, users ould only shre their entire sreen or nothing t ll wheres the IPC onfigurtions llowed muh more finegrined ontrol. Beuse the given tsks required severl swithes etween the own nd the other lling prty s sreen, prtiipnts told us they were not lwys too sure whose ontent they were urrently looking during Sreen Shring. Further they felt unesy with the ft, tht they ould not e sure when extly the other person strted to wth their sreen (gin). Although the permission ould e revoked t ny time, users did not feel the sme ontrol s with the onept of privte nd shred spe. Moreover, users did not know whih ontent elonged to whom nd whih they hd lredy shred (Q6). Finlly we sked users to rnk the three tested onfigurtions ording to with whih onfigurtion they felt the

10 fstest in solving the tsks nd whih onfigurtion they preferred overll. The nswers we got re interesting sine lthough most users (6) felt the fstest with the Sreen Shring onfigurtion (ompred to 4 nd 4), 11 prtiipnts in ontrst fvored one of the IPC onfigurtions (6 IPC Sreen nd 5 IPC Projetion), whih we ssume is for the forementioned dvntges of the IPC onept. The positive user feedk we reeived for ll three onfigurtions shows tht we ompred three eqully sophistited systems with similr funtionlities ut opposite qulities. The results further indite tht the IPC improves in-ll remote ollortion in vriety of use ses, even etter thn remote ollortion ould e implemented on top of stndrd phone softwre. The evlution lso reveled tht IPC onepts like the desktop interfe nd the privte nd shred spe revel their full potentil only with the projeted interfe, whih lso hs the dvntges tht it does not require tivting the loudspeker mode nd my help integrting nery persons in the ll y presenting ontent on the projetion. CONCLUSION We presented the Intertive Phone Cll nd its onepts tht filitte synhronous ollortion during phone ll. Previous reserh proposed the use of dditionl hrdwre like PCs [7] while we explored the possiilities nd requirements for system tht relies solely on moile phones. Our presented system supports projetorphones s well s onventionl phones through the IPC Projetion nd IPC Sreen modes, whih n e semlessly swithed to serve different moile situtions. We presented preliminry evlution of the IPC, lso ginst nother likewise novel system lled Sreen Shring. The results of our evlution indite tht the IPC onepts, e.g., privte nd shred spes, olor oding, opy permissions, nd the projeted interfe highly improve the user experiene in synhronous remote ollortion in terms of visul ommunition nd user ontrol over shring nd privy. Aording to study prtiipnts positive feedk nd their initil reports on their moile phone usge nd shring hits, we re onfident tht the IPC ould eome n indispensile tool of moile phone users. Our results inform future designers of synhronous remote ollortion systems from udio-visully onneted touh-tles to moile projeted surfes. In our future work we pln to investigte pprohes tht llow the usge of the IPC system outside of the lortory nd to ondut omprehensive user study. ACKNOWLEDGEMENTS This reserh hs een onduted within the projet Moile Intertion with Pervsive User Interfes funded y the DFG. We thnk Ken Pfeuffer for ting in the video. REFERENCES 1. Ah Kun, L.M. nd Mrsden, G. Co-present photo shring on moile devies. In Pro. of Moile HCI'07, ACM (2007), Ahern, S., Ekles, D. et l. Over-exposed?: privy ptterns nd onsidertions in online nd moile photo shring. In pro. of CHI'07, ACM (2007), Brkhuus, L., Brown, B., Bell, M., Sherwood, S., Hll, M., nd Chlmers, M. From wreness to reprtee: shring lotion within soil groups. In pro. of CHI'08, ACM (2008), Berdsley, P., vn Br, J., Rskr, R., nd Forlines, C. Intertion using hndheld projetor. Computer Grphis nd Applitions, IEEE 25, 1 (2005), Chpnis, A., Ohsmn, R., Prrish, R., nd Weeks, G. Studies in Intertive Communition: I. The Effets of Four Communition Modes on the Behvior of Tems During Coopertive Prolem-Solving. Humn Ftors: The Journl of the Humn Ftors nd Ergonomis Soiety 14, 6 (1972), Grriss, S., Cáeres, R., Berger, S., Siler, R., vn Doorn, L., nd Zhng, X. Trustworthy nd personlized omputing on puli kiosks. In pro. of Moisys'08, ACM (2008), Gunrtne, J.A. nd Bernheim Brush, A.J. Newport: enling shring during moile lls. In pro. of CHI'10, ACM (2010), Hrrison, C., Tn, D., nd Morris, D. Skinput: ppropriting the ody s n input surfe. In pro. of CHI'10, ACM (2010), Kritsuk, T. nd Sto, K. A werle mixed relity with n on-ord projetor. In pro. ISMAR'03, MFrlne, D.C. nd Wilder, S.M. Intertive dirt. In pro. of Uiomp 09, (2009), Mistry, P., Mes, P., nd Chng, L. WUW - wer Ur world. Ext. Astrts CHI'09, ACM Press, Myers, B.A. Using hndhelds for wireless remote ontrol of PCs nd pplines. Interting with Computers 17, 3 (2005), Rskr, R. nd Berdsley, P. A self-orreting projetor. 2, (2001), II-504- II-508 vol Rukzio, E., Holleis, P., nd Gellersen, H. Personl Projetors for Pervsive Computing. IEEE Pervsive Computing, Srvs, R., Oulsvirt, A., nd Jui, G. Building soil disourse round moile photos: systemi perspetive. In pro. of Moile HCI'05, ACM (2005), Sugimoto, M. et l. An Adptive Projetion Tehnique for Monohromti Non-flt Surfes using Hndheld Projetor Cmer System. In MP 2 Workshop CHI' Tmki, E., Miyki, T., nd Rekimoto, J. Briny hnd. Ext. Astrts CHI 09, (2009), Willis, K.D.D., Poupyrev, I., nd Shirtori, T. Motionem: metphor for hrter intertion with hndheld projetors. In pro. of CHI'11, ACM (2011), Wilson, A.D. PlyAnywhere: ompt intertive tletop projetion-vision system. In pro. of UIST'05, (2005), Wiltse, H. nd Nihols, J. PlyByPly: ollortive we rowsing for desktop nd moile devies. In pro. of CHI'09, ACM (2009), Thrutu Chnging the Converstion.