Gestures for pointing devices in screen-based environments. Florian Weil,

Transcription

1 Gesures for ponng devces n screen-based envronmens Floran Wel, December 4, 2010

2 Absrac Ths paper analyses gesure desgn for ponng devces n screen-based envronmens. By explorng desgn paerns he analyss nvesgaed he gesure desgn of ve deren end-user producs: Deskop operang sysems, moble operang sysems, 3 rd Pary sofware, small sofware producs, and common hardware producs. The begnnng of he paper denes wha a gesure s, and he varous knds of gesures. Aferwards he analyss merges he gesure desgn resuls wh he basc commands for ponng devces. Ths approach pons ou whch gesures are ofen used, and n whch conex hey are used. The resuls gve neracon desgners and sofware engneers a gude for mplemenng gesures n her own producs. Furhermore, he paper proposes soluons for gesure documenaon, and a concepual framework for complcaed gesures. The las secon akes an ndusral desgn perspecve on ponng devces as an npu channel. I dscusses he evoluon of nerface desgn from a hardware drven o a sofware drven approach.

3 Conens 1 Inroducon 1 2 Gesures for ponng devces Denon of gesure n Human-Compuer-Ineracon Types of gesures Commands for gesures Analyss of gesure based Ineracon for end-user producs Concep and analyss model Resuls from a Ineracon Desgn vew Level 1 gesures Level 2 gesures Level 3 gesures Level 4 and level 5 gesures Resuls from a desgn vew Resuls from a echncal vew Deskop Operang Sysems Moble Operang Sysems rd Pary sofware Addonal Sofware producs Hardware producs General resuls An Indusral Desgn vew on he evoluon of Ponng Devces 29 5 Concluson 32 2

4 Ls of Fgures 2.1 Canoncal Vocabulary by Alan Cooper Movemen caegores of each gesure level Overvew of all nvesgaed producs Bar char of all level 1 gesures Bar char of all level 2 gesures Bar char of all level 3 gesures Command overvew of he nvesgaed gesures Evoluon of Lapop Ponng Devces Evoluon of moble devces

5 Ls of Tables 3.1 Gesures of Deskop Operang Sysems Gesures of Moble Operang Sysems Gesures of 3 rd Pary sofware Gesures of addonal sofware producs Gesures of Hardware devces

6 Chaper 1 Inroducon Ineracon desgners and sofware engneers have endeavoured o negrae gesurebased neracon n sofware producs for years. Wh he adven of mul-ouch screens and he launch of he Phone gesure-based neracon has esablshed new mehods of neracon for Human-Compuer-Ineracon. Unforunaely, gesural-based neracon for Human-Compuer-Ineracon s que deren relave o radonal neracon va a mouse devce. Ineracon desgner and sofware engneers are requred o learn how gesures funcon, whch conceps base gesures on, when makes sense o use gesures, and n whch cases gesures can harm he Human-Compuer-Ineracon. The focus of hs paper s on gesures for ponng devces n screen-based envronmens. Ths area of ponng devces mos commonly relaes o mulouch screens, mouses, rackpads, ables, and pen sysems as npu devces. Ths paper s dvded n 5 chapers. In chaper 2 he word gesure and he conex n whch hs paper uses he word gesure s dened. The las secon oulnes a funconal command se for ponng devces amed a neracon desgners and sofware engneers o creae new usable gesures. Ths paper does no nvesgae free-form gesures va gloves or camera-based neracon. Furhermore, hs paper only explores ponng devces, whch are used n screen-based envronmens. Tha ncludes small (e.g. smar phones), medum (e.g. able devces), and Deskop Screens. Bg cy screens or scalable mul-ouch screen sysems lke MulTouch Cell 1 are no par of hs nvesgaon. Chaper 3 conans gesure analyss resuls of curren sofware and hardware producs. The analyss nvesgaed gesures used for Deskop Operaon Sysems, Moble Operaon Sysems, 3 rd Pary sofware, small sofware ools, and addonal npu devces. The analyss reveals wha ype of neracon gesure paerns exs for hese deren producs. The lack of gesure documenaon 1 MulTouch Cell from Mulouch webse, hp://mulouch./producs/cell/, accessed March

7 and gudance sysems for more complcaed gesures s he focus of secon 3.4. Anoher mporan pon of he paper s he evoluon of npu devces from an ndusral desgn perspecve. Chaper 4 pus forward he hypohess ha Our npu devces wll become more absrac n he fuure, and sofware (he vrual par) wll dene he rules of neracon. Sofware s gong o play a more mporan role han he hardware desgn. Smarphones wh a Touch- Screen and have less han four buons. These generaon of smarphones are an evden for hs evoluon. Chaper 4 concludes by dscussng he emergen ssues beween organc nerfaces 2 and gesure-based neracon. The hess wll be falsed or conrmed, dependng on he research resuls from some research and developmen labs. The concluson of resuls ams o conrbue and furher research n he eld of gesural neracon. For nsance, how varous meda and screen envronmens nuence gesure-based neracon. 2 Organc User Inerface are user nerfaces wh non-planar dsplays ha may acvely or passvely change shape va analog physcal npus, see also hp:// 2

8 Chaper 2 Gesures for ponng devces The word gesure s a broadly dened erm. I s ofen used colloqually and varous scenc dscplnes use he word gesure n deren conexs. Therefore s mporan o acheve a suable and well dened concep for he word gesure, whch s usable for Human-Compuer-Ineracon. Ths s especally mporan n he conex of ponng devces n screen-based envronmens. 2.1 Denon of gesure n Human-Compuer- Ineracon The denon of gesure arculaed by Kurenbach and Huleen s useful for denng a gesure-based neracon n Human-Compuer Communcaon sysems: A gesure s a moon of he body ha he conans nformaon. Wavng goodbye s a gesure. Pressng a key on a keyboard s no a gesure because he moon of a nger on 's way o hng a key s neher observed nor sgncan. All ha maers s whch key was pressed.[10] Kurenbach and Huleen sae ha he movemen for pressng a key on a keyboard s no a gesure, because he movemen s no capured. Ponng devces have he ably o capure physcal movemens. Also he denon of a gesure from Dan Saer suppors ha a gesure s a movemen. A gesure, for he purposes of hs book, s any physcal movemen ha a dgal sysem can sense and respond o whou he ad of a radonal ponng devce such as a mouse or sylus. A wave, a head nod, a ouch, a oe ap, and even a rased eyebrow can be a gesure.[17, page 2] 3

9 There s no denon of a gesure avalable whch excludes ponng devces ha are exensons of he body. Furhermore, ponng devces mee he requremens of a gesure recognon sysem dened by Dan Saer: Every devce or envronmen ha employs gesures o conrol has a leas hree general pars: a sensor, a comparaor, and an acuaor. These hree pars can be a sngle physcal componen, or, more ypcally, mulple componens of any gesural sysem, such as a moon deecor (a sensor), a compuer (he comparaor), and a moor (he acuaor).[17, page 13+14] In he end we know every gesure s a combnaon of movemen and a ls of capured daa (evens). For hs reason gesures n hs paper conss of a leas 2 evens daa. For nsance, a mouse-buon-press s an even. A mouse-buonpress even n combnaon wh a quck followng mouse-buon-release even s already a gesure, whch s called mouse-clck gesure. A furher 3-even-daa gesure could be mouse-buon-press + mouse-move + mouse-buon-release. Ths gesure belongs o he radonal Drag & Drop gesure. How a sysem reacs o deren evens (gesures), depends on neracon desgn. The neracon and nerface desgner s responsble for eecve and nuve gesure neracon. 2.2 Types of gesures Bll Buxon[2] summarzed he mos mporan caegores of gesures for Human- Compuer Ineracons. The rs caegory s founded on Cadoz (1994). He grouped gesures no hree ypes [2, page 2] : semoc communcae meanngful nformaon ergoc manpulae he physcal world and creae arefacs epsemc learn from he envronmen hrough acle or hapc exploraon Humans lke o nerac drecly wh objecs [17, page 17]. For hs reason ergoc and epsemc gesures can be a very good nspraon source for gesure desgn wh ponng devces. We can adap or ransform hese hapc gesures no he screen-based envronmen. For nsance, pushng up and down he volume slder, works perfecly wh a physcal npu and n a screen-based envronmen va drag neracon, as well. In addon o hese gesures, Rme and Scharaura (1991) also speced a useful gesure axonomy[2, page 2+3]: symbolc gesures These gesures have a same meanng n each culure. For nsance, he humbs up hand sgn for everyhng s OK. 4

10 decc gesures The gesures of ponng, or focusng he aenon o specc evens or objecs n he envronmen. These gesures have a srong command characer and herefore hey are already well esablshed n Human- Compuer-Ineracon. conc gesures These gesures communcae nformaon abou he sze, shape or orenaon of he objecs. These gesures have a descrpve and an addonal explanaon characer. panommc gesures These gesures show he use of movemen or how a objec s used by an human. The human normally do a panommc gesure for emphaszng he done acvy. For our approach of ulzng ponng devces for gesural npu he decc and conc gesures are he bes gesures forms. Decc gesures are well esablshed and exensvely explored n radonal command-based communcaon beween human and compuer sysems. Therefore neracon desgners mus focus more on conc gesures. Iconc gesures relae o he Drec-Objec-Manpulaon concep, whch s nowadays used sofware. Drec-Objec-Manpulaon enables he user o change he sze, form or oher properes of a dgal objec drecly. Curren sofware producs mos commonly use he Drag & Drop meaphor for Drec-Objec-Manpulaon. Wh he adven of mul-ouch screens he nerface desgns have an enre new array of possbles for he Drec-Manpulaon concep n screen-based envronmens. For nsance, ransform operaons (scalng, roaon, ec.) a he same me. Iconc gesures explore such new possbles n he bes way, due o her relaed descrpve and explanaory characer. Moreover, Dan Saer denes gesures as sac gesures and dynamc gesures[17, page 179]: Sac gesures belong o he axonomy of symbolc gesures. Meanng sac gesures are performed and held lke posures. Dynamc gesures are a movemen over me, lke a Drag & Drop gesure or usng he scroll wheel of he mouse. In addon o he menoned caegores, hs paper nroduces a new caegory. Ths new caegory combnes all hese oher caegores, and dsngushes beween nuve, pre-raned and raned gesures. Transonng udly beween nuve, pre-raned and raned gesures. Inuve gesures are obvous for he user. The user does no need o learn hem. For nsance, he selecon process of an nerface em. The user has o move he cursor o an objec and hen clck on he objec. Ths selecon acon funcons he same way n he real world. Go o he objec and grab/ouch. The user does nuvely. Preraned gesures are gesures from predecessor neracve sysems. For nsance, he double clck s no an nuve and obvous gesure, bu he 5

11 mos people have already learn hs gesure from neracon wh he mouse devce. Traned gesures are he mos complcaed gesures. Traned gesures are absoluely no obvous for he user. The user mus learn hem hrough gudance from a documenaon or an nsrucor. In he mos cases only exper users of a sofware use raned gesures, whch could be compared wh keyboard shorcu commands. Traned gesures can cause problems and confuson for he end-user, whch wll be dscussed n deal n secon 3.4. In summary we have explored a varey of denons for gesures and how hey can be caegorzed. These deren caegores smplfy he search for new usable gesures: denng he scope of gesural neracon and how we mgh develop new gesures. No every gesure s useful, hus hs secon ends wh an enlghenng mnemoechnc verse from Dan Saer [17, page 38]: The more complcaed he gesure, he fewer he people who wll be able o perform. 2.3 Commands for gesures So far we have dscussed manly wha a gesure s and he varous caegores of gesures ha exs. Ths knowledge s mporan bu almos useless whou undersandng n he conex of screen-based user nerfaces. Every gesure s conneced wh a specc funcon or command. I s herefore mporan o undersand he herarches and prncple srucures of curren compuer command communcaon. Gesure based neracon wll no change hs concep of communcaon wh a screen. Gesure based neracon s able o perform asks faser and more nuvley for he user. Alan Cooper speced a very good srucure model of compuer command communcaon. Ths model s called Canoncal Vocabulary and s vald for all screen-based user nerfaces[4, page 281]. He dvded he commands n hree deren layers. 6

12 I d oms I npu del e e,cr ea e, dr aw Ou pu scr ol l ng,sor ng, d al ogs Appl c a ons pec f c c ommandsandf eedbac k Compounds doubl ecl c k, bu oncl ck,sel ec on ed f el ds,checkboxes h ghl gh ng Gener ci npuandou pu ac onsands y mbol s po n,cl c k, dr ag,keypr ess Pr m v es I nd v s bl eac onsand f eedbac kmec han s ms cur sor, ex Fgure 2.1: Canoncal Vocabulary by Alan Cooper Ths pyramd conans some neresng commands and gesures. From he command perspecve almos every sofware suppors delee, creae, sorng and scrollng funcons. So we can noce ha hese knds of commands are he absolue basc commands n curren screen-based sofware producs. From he gesure perspecve he bascs are buon-clck, double-clck, selec- on, drag and pon. All hese gesures are already mplemened n he operang sysems. Every developer and Ineracon Desgner can work wh hs gesures whou any concerns. These gesures are mosly obvous (e.g. Drag & Drop) or hey belong o he preraned gesure caegory (e.g. double-clck ). Addonal o he srucure model of basc commands and gesures for screenbased user nerfaces, Alan Cooper de ned seven caegores for Drec-Manpulaon wh ponng devces. These caegores o er a useful overvew of he command se for manpulang dgal objecs[4, page 377]. 1. Ponng 2. Selecon 3. Drag & Drop 4. Conrol manpulaon 5. Palee ools 6. Objec Manpulaon (such as posonng, shapng, and reszng) 7. Objec connecon 7

13 The rs four caegores are already well explored and sophscaed. Only some mnmal mprovemens are sll possble. For nsance, he command groupng objec has already a funcon, bu could be beer and more nuve mplemen wh he help of gesures. For he caegory Palee ools and Objec Manpulaon exs only an nconvenen se of funcons. Especally, mulouch sysems delver a more nuve neracon form for changng he poson, he shape and he sze of dgal objecs. The las caegory Objec connecon exs already n projec managemen or mnd map sofware ools. The new possbles of usng more han one ponng npu enables new neracon forms beween deren dgal objecs a he same me. Therefore hs caegory Objec connecon s very neresng for gesure based neracon. Havng dened gesures and command communcaon n screen-based envronmens for ponng devces n a heorecal manner, he nex chaper s gong o explore how gesures are mplemened and used for end-user producs. 8

14 Chaper 3 Analyss of gesure based Ineracon for end-user producs The followng analyss seeks o nd paerns n gesural neracon n curren end-user producs. Afer generang a ls of used gesures, hs chaper wll explore regularly used gesural paerns. These paerns wll oer an nsgh no whch gesures are well esablshed and whch are no. From a user experence perspecve s mporan o reveal and ulse such paerns. Users wll ry o use gesures from one produc n anoher produc. If hs ransformaon fals he user experence s broken. From a users perspecve hs assumpon usually resuls n frusraon wh he neracon desgn. A broken user experence ofen rggers an averson n usng gesure based neracon. Furhermore, analyss gves neracon desgners and sofware engneers nsghs no how o furher develop gesure based neracon n screen-based envronmens (especally n mul-ouch screen envronmens). The secon 3.1 explans he analyss model n a more deal. How apples he Bll Buxon dea of Degree of Freedom [3] for npu devces. The subcaegores are generaed on hs base of he Degree of Freedom heory. The secons 3.2 and 3.3 dscuss he resuls on each level n deal. Each subcaegory from a desgn and echncal vew has her own specal characersc. Analyss wll explan hese specal characerscs and compare hem o each oher. 3.1 Concep and analyss model The analyss model s comprsed of wo pars. One par summarzes he caegores of gesures and he oher summarzes he caegores of sofware and hardware producs. The caegores of gesures are relaed wh he heory Degree of Freedom 9

15 from Bll Buxon[3]. He explans hs heory: The rchness of neracon s hghly relaed o he rchness/numbers of degrees of freedom (DOF), and n parcular, connuous degrees of freedom, suppored by he echnology. The convenonal GUI s largely based on movng around a sngle 2D cursor, usng a mouse, for example. Ths resuls n 2DOF. If I am sensng he locaon of wo ngers, I have 4DOF, and so on. Our analyss smples Bll Buxon's concep by basng on he number of poners. In Buxon's denon he mouse has wo Degrees of Freedom, because moves nsde a wo-dmensonal world. In our concep he mouse cursor s a level 1 devce, because has only one pon. A wo nger neracon on a screen denoes 4 degrees of freedom n Bll Buxon's denon. In our case a wo nger neracon s dened as a level 2 gesure, as has wo poners. We smpled our concep because all he nvesgaed npu devces use only wo dmensons (x and y coordnaes) for neracon. In he case of Free-Form gesures and neracon n a hree dmenson envronmen hs approach becomes nvald. Every gesure level has subcaegores wh a deren amoun of movemen. The analyss model dsngushes beween one-way movemen, wo-way movemen and n-way movemen. In he gure3.1 you can see examples of each subcaegory. The crcle of hese cons shows he sarng pon of he movemen. ONEWAYMOVEMENT TWO WAYMOVEMENT N-WAYMOVEMENT Fgure 3.1: Movemen caegores of each gesure level The sofware and hardware par ncludes producs for he end-user and mass marke. The sofware par has four deren caegores Deskop Operang Sysems, Moble Operang Sysems, 3 rd Pary sofware, and addonal sofware. The hardware caegory s dened by self. A ls of all nvesgaed producs can be vewed n gure 3.1. The 3 rd Pary Tools presen he mos avalable Web Browser Plug-Ins. Many Inerne Applcaon (RIAs) are usng hese Plug-Ins. Therefore wll become mporan o consder, how hese ools deal wh gesure based neracon. Mddle-ware Tools and lbrares lke reactivson 1,NUI Group Lbrary 2, 1 reactivson. From sourceforge webse, hp://reacvson.sourceforge.ne/, accessed February Touchlb. From NUI Group webse, hp://nugroup.com/ouchlb/, accessed February

16 Desk opos Mob l eos 3rdPar ytool s -W ndows7 -MacOSXSnowLeopar d -L nux -W ndows6. 5 -Andr o d PhoneOS -Bl ackber r y( S or m 2) -Symb anos -AdobeFl ash( Pl ayer10. 1) -M r cosof S l ver l gh -JavaFX( Java) Add onal Sof war e -Oper abr owser -Moz l l af r ef ox -Au odeskmaya -BumpTopVer s on2 -MouseGes ur es ar -MacJ ouch Add onal Har dwar e -Synap ctr ackpad -MouseandAppl emag cmouse -Wacom BambooTouch&Fun Fgure 3.2: Overvew of all nvesgaed producs 11

17 SPARS-UI 3, and MT4j 4 are no ncluded n hs analyss. These ools are normally used n nsallaon sysems or n specal research envronmens. The lbrares are mporan for prooypng and realzng new mul ouch sysems, bu hey do no play a bg role for he mass marke. The las sofware based caegory addonal Sofware conans smaller sofware producs whch use gesure based neracon. The Inerne browsers Mozlla Frefox and Opera as well as Auodesk Maya, BumpTop Verson 2, MouseGesure Sar, and Mac Jouch belong o hs caegory. BumpTop, MouseGesure Sar and Mac Jouch are characersed as exensons for he operang sysem raher han beng an unque sofware produc. Because of he her exenson characersc and non-complan gesure desgn hey aord he poenal for nnovaon. A furher movaon was o nvesgae he derences n gesure based neracon for bg and small sofware producs. The hardware-based caegory explores he doman of common npu devces for screen-based compuer envronmens. Some of hese devces exend he common hardware funconaly of exsng compuer or noebook sysems. Such devces are readly avalable n compuer shops. Havng declared our he analyss model, he nex secon dscusses he resuls of hs qualave research. 3.2 Resuls from a Ineracon Desgn vew Level 1 gesures The analyss revealed ha level 1 gesures (only one poner) are he mos used gesures n curren end-user producs. The man reason for hs fac s ha he mouse s he radonal ponng devce for screen-based sofware producs, and belongs compleely n he level 1 caegory. Clck / Tap, double Clck / double Tap, and one-way movemens are he mos suppored gesures n hs caegory. These hree neracon forms are almos used wce as ofen as anoher neracons forms. Durng he analyss process was no possble o dene a common paern for usng wo-way Movemen gesures. Seven sofware producs suppor hs knd of gesure, bu every produc mplemens he gesure neracon n her own way. There exs no obvous neracon desgn convenon for wo-way gesures. The mos sysems (e.g. Google Androd, Wndows Moble 6.5 ec.), whch suppors wo-way and n-way movemen neracon, delver an addonal Gesure Manager Uly Manager Sofware for her Programmng SDKs. These ools smplfy he creaon process of cusom gesures. Ineracon desgners and sofware engneers can bene from such uly ools. They can dene he form of he gesure, and hen he gesure recognon sofware mplemens he deecon algorhm and he gesure hreshold auomacally. Furhermore, no oher 3 Sparsh UI. From Google Code webse, hp://code.google.com/p/sparsh-u/, accessed February MT4j. From MT4j Wk webse, hp:// accessed February

18 Tap / Clck Double Tap and Clck 20 SWIPE 7 wo-way Movemen Down - Lef Move Down - Rgh Move Down - Up Move Up - Rgh Move Rgh - Up Move wo-way Movemen gesures Up - Lef Move 5 2 n-way Movemen Crcle-based Pah Fgure 3.3: Bar char of all level 1 gesures sofware han Auodesk Maya s usng he Markng Menu gudance concep for wo and n-way movemen gesures. Especally n-way gesures are very dcul o memorze, because of her complexy. Ineracon Desgner should be more wllng for smplfyng hese complex gesures by usng a gudance conceps 5. The secon 3.4 abou desgn problems wll explan why s hghly recommend o use Markng Menus for n-way gesures. The las form of gesures n he level 1 caegory explores crcle-based gesures. Only wo producs use he crcle-based gesures. These wo producs have wo compleely deren purposes for hs knd of gesure. BumpTop uses hs knd of gesures for execung group selecons for ems on he screen. The Synapc rackpad uses her crcle-based neracon for scrollng funcons. They even creaed her own erm for hs gesure, callng ChralScrollng TM. In general a crcle-based gesure s easer o keep n mnd for he user n comparson o wo and n-way gesures. One dsadvanage of crcle-based gesure s, ha hey are no easy o perform, especally wh a mouse devce. For hs reason neracon desgners and sofware engneers should be cauous when usng crcle-based gesures. The hreshold for hs gesure mus be mplemened very well. I could be very dsapponng for he user, when hey ry o perform he gesure whou any success Level 2 gesures The level 2 gesure caegory can be speced n wo pars: Drec-Manpulaon operaons and raned gesures. Drec-Manpulaon gesures are he mos 5 A gudance concep s a gude, whch appears durng he user performs a gesure. Tha could be predened gesural gudes or dynamc generaed gesure suggesons for he user. 13

19 Two Fnger Tap / Clck suppored gesures n he level 2 caegory Press AND Tap Press AND SWIPE Two Fnger SWIPE ZOOM Gesure Roaon Gesure Fgure 3.4: Bar char of all level 2 gesures They emerged for he mass-marke wh he adven of he Phone and oher mul-ouch sysems. The gesures Zoom and Roaon became famous hrough he mnmal phoo edng and browsng vdeo demos on TV or YouTube. For hs reason we can already specfy hese very young gesures (Two Fnger Swpe, Zoom, and Roaon gesure) as nuve gesures for mul-ouch sofware envronmens. Ineracon desgners and sofware engneers can use whou gesures whou any doub. Only n small screen envronmens, for nsance moble phones, hey should consder he small sze of he screen and he ems. If he graphcal represenaon of a dgal em s oo small for Drec-Manpulaon, hey should no use a Drec-Manpulaon gesure. Conrol nerfaces lke slders and edng elds are more approprae n hs case. Alhough rarely used, he Selec (Press) and Swpe Gesure s even a drec manpulaon gesure. The user presses wo ngers on he dgal objec, and hen performs wh one nger a one-way movemen, lke a swpe. Ths gesure can be used for addonal croppng and reszng operaons (see also a BumpTop). The nex wo gesures are raned gesures and no very obvous for he user. The Two Fnger Tap s suppored by ve producs. Especally, he rgh clck meaphor or he command for he conex menu s on Wndows 7 derenly mplemened han on Mac OS. On Mac OS he Two Fnger Tap performs he rgh clck meaphor. On Wndows 7 sysems he user mus perform a Press one Fnger and ap wh he second nger gesure for he rgh-clck meaphor. These derences of gesure desgn for he same command, can confuse he sandard compuer user. I does no suppor an opmal ndependen plaform user experence. 14

20 3.2.3 Level 3 gesures Three Fnger Tap / Clck Three Fnger Down Three Fnger SWIPE Index Press AND Two Fnger SWIPE Clck (Tap) AND Two Fnger SWIPE Two Fnger Down AND SWIPE Two Fnger Down AND Index Double Tap Fgure 3.5: Bar char of all level 3 gesures Gesures n he level 3 caegory are no very ofen used. Only hree sofware producs are usng hese knds of gesures. The wo more or less nuve gesures Three Fnger Tap and Three Fnger Swpe are used n bgger sysems (Mac OS and Synapc Trackpad). The Swpe gesure s used agan for scrollng and skppng navgaon commands n a hgher herarchy. For nsance, he Mac Operang Sysem and he Synapc rackpad use hs gesure for skppng beween deren pages and phoos. The oher gesures are used only by he Mac Jouch gesure exenson sofware. The amoun of gesures n hs collecon s que mpressve. Unforunaely he gesures are very dcul o remember, because of he nhered coordnaon complexy wh 3 ngers Level 4 and level 5 gesures Only a few gesures exs n he level 4 and level 5 caegores. In he level 4 caegory Mac OS Leopard uses a 4 Fnger Swpe for swchng beween deren acve applcaons. The Mac Jouch Gesure exenson uses a 4 Fnger Follow Tap for mnmzng and maxmzng a wndow on he screen. Boh gesures are no obvous for he user. The user has o learn hem. Therefore hese wo gesures belong o he raned gesure caegory. The 4 Fnger Follow ap s especally dcul o memorze and perform. BumpTop Verson 2 s he only sofware produc, whch suppors a level 5 gesure. The gesure s called Scrunch. The user mus use he whole hand for hs gesure. He or she spreads he nger and hen move he ngers ogeher. All objec on he screen, whch are underneah he hand wll be grouped ogeher. Ths gesure does no seem obvous a he rs vew, alhough belongs o nuve gesure caegory. I s a nce example how gesures can mprove group- 15

21 command neracons. Ths gesure s very smlar wh our acvy when we group some ems on a desk, for nsance. The only excepon are moble screens. These screens are usually smaller han an ou sreched hand. For hs reason, hs level 5 gesure won' work wh small screens Resuls from a desgn vew From a neracon desgn vew he analyss showed ha gesural neracons are already esablshed n curren end-user producs. Drec-Manpulaon gesures are he mos suppored and common gesures. Furhermore, he Mac OS and he Synapc rackpad use he characersc of levels for herarchc-based navgaon n screen-based envronmens: 1. One Fnger SWIPE for selecon and draggng, deleng, move operaons 2. Two Fnger SWIPE for scrollng 3. Three Fnger SWIPE for skp a slde n a presenaon, a phoo or a page 4. Four Fnger SWIPE for swchng beween deren applcaons Ineracon Desgner and Sofware Engneers should adap and consder hs concep of herarchc-based commands. Applyng hs approach for oher gesures, can maybe srongly mprove gesure based neracon. Anoher pon s, whch knd of commands are sll nsucenly suded. The gure pons ou whch gesures have are relaed o he basc commands of ponng devces n screen-based envronmens, and whch commands are rarely used. 16

22 Oupu scrollng, sorng, dalogs IDIOMS Applcaon specfc commands and feedback Inpu delee, creae, draw ed felds, checkboxes, hghlghng COMPOUNDS Generc npu and oupu acons and symbols double-clck buonclck, selecon cursor, ex PRIMITIVES Invsble acons and feedback mechansms pon. clck drag, keypress 17 Tap / Clck Double Tap / Clck Swpe Two-way moevmen N-way novemen Crcle-based Pah Two Fnger Tap / Clck Press and Tap Press and Swpe Two Fnger Swpe ZOOM Gesure Roaon Gesure Three Fnger Tap / Clck Three Fnger Down Three Fnger Swpe Press and Two Fnger Swpe Tap and Two Fnger Swpe Two Fnger Down and Swpe Two Fnger Down and Double Tap Four Fnger Follow Tap Four Fnger SWIPE Fve Fnger Scrunch LEVEL 1 LEVEL 2 LEVEL 3 LEVEL 4 LEVEL 5 Ponng Selecon Drag and drop Conrol manpulaon Palee ools Objec manpulaon Objec connecon Fgure 3.6: Command overvew of he nvesgaed gesures

23 The mos basc compuer commands for graphcal user nerfaces are already mplemened wh gesures. Drec-Objec-Manpulaon s heavly used because of her nuve and obvous characer for he users. Bu some operaons are sll unused n gesure based neracon. For example, he commands relaed o Objec connecons and conrol manpulaon delvers some nnovaon space for new gesure neracons. The Tex Inpu Sofware Swype 6 shows objec connecons n form of characer connecon gesures n an exemplary way. Palee based commands are heavly conneced wh he appearance of he cursor. The deren cursor syles vsualse he acve sae of he curren command for he user. In he doman of Palee ools does no exs much space for nnovaon of gesure based neracon. Touchscreen envronmens do no use a mouse poner any more, and he neracon desgn s ofen saeless. In conras he dscplne of angble nerfaces and her new hapc-based ools could mprove palee ool neracons. In general, he gure also shows ha gesure based neracon wh ponng devces s well-esablshed. The mos basc operaons are mapped wh gesures and n he fuure neracon desgners and sofware engneers wll develop oher new gesures. 3.3 Resuls from a echncal vew Deskop Operang Sysems The Deskop Operang Sysems suppor very smlar gesure paerns n her caegory. The level 1 suppored gesures n all Operang Sysems are he same. For level 2 gesures ders a lle b. All Operaons Sysems suppor Roae and Zoom gesures, even her Programmng SDK suppors gesure recognon for hese gesures. In addon, every Operang Sysem API delvers access o he raw ouch daa. Ths raw ouch daa allows neracon desgner and sofware engneers o develop her own cusom gesures. In general he Operang Sysems suppors more Drec-Manpulaon gesures. These gesures are more nuve and obvous for he user. For hs reason he Operang Sysems ac more conservave n he opc of gesure-based ner- 6 Tex Inpu for Screens from Swype Inc. webse, hp://swypenc.com/produc.hml, accessed March Cocoa Applcaon Framework, from Mac OS X Reference Lbrary webse, hp://developer.apple.com/mac/lbrary/releasenoes/cocoa/appk.hml, accessed February Wndows Touch Gesure Overvew, from MSDN Wndows Developer Cener webse, hp://msdn.mcrosof.com/en-us/lbrary/dd940543%28vs.85%29.aspx, accessed February Wndows Touch Team, Touchng Wndows 7, March 25, 2009, pos on blog Engneerng Wndows 7, MSDN Blogs, hp://blogs.msdn.com/e7/archve/2009/03/25/ouchngwndows-7.aspx, accessed February Yochay Kray, MulTouch Capables n Wndows 7, from MSDN Magazne webse, hp://msdn.mcrosof.com/en-us/magazne/ee aspx, accessed February Apple Inc. Chaper 2: Lfe wh your MacBook Pro. Manual MacBook Pro. pp (hp://manuals.nfo.apple.com/en_us/macbook_pro_13nch_md2009.pdf, accessed February 2010) 18

24 Wndows 7 Mac OS Leopard Lnux Clck and Tap x x x Double Clck and Tap x x x SWIPE x x x Two Fnger Tap x x Press AND Tap x Two Fnger SWIPE x x Zoom x x x Roaon x x x Three Fnger SWIPE Four Fnger SWIPE x x Table 3.1: Gesures of Deskop Operang Sysems acon compared o oher smaller producs, lke BumbTop and Auodesk Maya. Unforunaely, a he me of he analyss only a prooype mplemenaon 12 of he new Lnux kernel suppored nave gesure suppor. In whch case please read he nformaon abou Lnux n hs paper crcally Moble Operang Sysems The curren Moble Operang sysems suppors only one poner, excep he Apple Phone Therefore he mos suppor gesures n hs caegory are found n he level 1 gesure caegory. The Moble Operang Sysems API of he Google Androd and Wndows Moble delver a gesure programmng framework or a leas a gesure creaon ool. How and wha knd of gesures are suppored depends heavly on he used moble hardware and less on he Moble Operang Sysem. The moble BlackBerry devce Sorm 2 18 suppors more 12 Lnux nave mulouch suppor. From ENAC Ineracve Compung Laboraory webse, hp:// accessed February Handlng Mul-Touch Evens. From Phone OS Reference Lbrary webse, hp://developer.apple.com/phone/lbrary/documenaon/phone/concepual/phoneosprogrammnggude/evenhandlng/e CH9-SW11, accessed February Apple Inc. Chaper 3: Desgnng an Phone Applcaon: From Produc Denon o Brandng. Phone Human Inerface Gudelnes. pp. 42 (hp://developer.apple.com/phone/lbrary/documenaon/userexperence/concepual/moblehig/moblehig.pdf, accessed February 2010) 15 Apple Inc. Chaper 2: Bascs. Phone User's Gude. pp (hp://manuals.nfo.apple.com/en/phone_user_gude.pdf, accessed February 2010) 16 Gesures. From Androd Developer webse, hp://developer.androd.com/resources/arcles/gesures.hml, accessed February Usng Gesures n Wndows Moble 6.5, From MSDN Wndows Moble Developer Cener webse, hp://msdn.mcrosof.com/en-us/lbrary/ee aspx, accessed February BlackBerry. BlackBerry Sorm 2: Gesures & Shorcus, (hp://ercrc.com/wpconen/uploads/2009/10/bb_sorm2_gesures.pdf, accessed February 2010) 19

25 Wndows Moble 6.5 Androd 1.6 BlackBerry Sorm 2 Symban OS Phone Clck / Tap x x x x x Double Clck x x x x and Tap SWIPE x x x x x Two-way x x x movemen n-way movemen x x Zoom x Table 3.2: Gesures of Moble Operang Sysems gesural neracon han he oher BlackBerry devces. Symban OS suppors only he basc gesures 19. In he fuure he gesure neracon for moble phones wll rapdly ncrease, because he Androd and Wndows Phone 7 are gong o suppor mulouch gesures for her nex generaon devces rd Pary sofware The 3 rd Pary sofware Flash Player 21, Slverlgh 2223 and JavaFX 24 suppor gesures. The suppored gesures are based on he used Deskop Operang Sysems. Ths relaon can be explaned by he echnology dependency wh he Operang Sysems. Each of he 3 rd pary ools uses he suppored gesures of he Operang Sysems, and pass hem hrough o her own API. Tha s he reason why hese ools suppor some gesures only for a specal plaforms. How already menoned mddleware ools lke reactivson,nui Group Lbrary, SPARS-UI, and MT4j are no ncluded n he analyss. Even Sparsh UI and MT4j have already mplemened her own gesure recognon sysem. These 19 Vdeo Player Package/feaure playback vew gesure suppor, From Symban Developer Wk webse, hp://developer.symban.org/wk/ndex.php/vdeo_player_package/feaure_playback_vew_gesure_suppor, accessed February Androd 2.1 SDK. From Androd Developer webse, hp://developer.androd.com/sdk/androd-2.1.hml, accessed February Chrsan Canrell; Mul-ouch and gesure suppor on he Flash Plaform, From Adobe Developer Connecon webse, November 17, 2009, hhp:// accessed February Tm Heuer, Slverlgh 3 Mul-ouch: The Bascs, July 30, 2009, pos on blog Mehod of faled, Tm Heuer Blog, hp://mheuer.com/blog/archve/2009/07/30/slverlgh-3-mulouch-nroducon-fundamenals-bascs.aspx, accessed February Jesse Bshop, Mul-ouch Gesure Recognon n Slverlgh 3, November 5, 2009, pos on blog jebshop.blog, hp:// accessed February Rer, Smon & Cacedo, Angela. Buld Your Own Mul-Touch Inerface wh Java and JavaFX Technology. PowerPon presenaon o JavaOneTM Conference,

26 Flash Player 10.1 Slverlgh 3 Java FX Clck / Tap x x x Double Clck / Tap x x x SWIPE x x x Two Fnger Tap x x Press AND Tap x (only Wn) x (only Wn) Two Fnger Swpe x (only Wn) x (only Wn) Zoom x x x Roaon x x x Table 3.3: Gesures of 3 rd Pary sofware ools are no used n end-user producs. They are more used for mplemenng prooypes n scenc envronmens. For hs reason hey are no nvolved n he analyss Addonal Sofware producs A few of he nvesgaed addonal sofware producs use her own gesure logc. If you compare hs gesure able wh he oher gesure ables, hen becomes obvous he addonal sofware caegory uses he bgges se of gesures. Bu quany s no equal o qualy n mos cases. Mos gesures seem o be very expermenal. Whch s a very good envronmen o explore new nnovave gesures. Some gesures wll be useless, bu oher gesures can evolve o useful gesures. Therefore, he gesures of hs caegory are nvesgaed n a more deal. The Inerne Browser Sofware Opera 25 and Mozlla Frefox 2627 are usng mouse gesures for navgaon commands. These gesures manly mee he requremens of level 1 wo-way gesures. The user has o press he rgh mouse buon and move he mouse along a ceran pah. Afer he movemen he user releases he rgh mouse buon for performng he command. A ceran amoun of one and wo-way gesures are easy o perform for he user. Three or n-way Movemen gesures are very complcaed o remember and are usually dcul o perform. Therefore Auodesk Maya uses he Markng Menu gudance concep for her movemen gesures. The user ges a very good gudance for performng he approprae mouse gesure. The sze and he movemen of he 25 Mouse Gesures n Opera. From Opera Sofware webse, hp:// accessed February Mouse Gesures. From Mozlla Developer webse, hp://opmoz.mozdev.org/gesures/, accessed February Suppored Gesures. From Mozlla Developer webse, hp://opmoz.mozdev.org/gesures/defaulmappngs.hml, accessed February Markng Menu Edor. From Auodesk Maya 2010 manual webse, hp://download.auodesk.com/us/maya/2010help/ndex.hml?url=ws1a f5-3a29af00119afd28e hm,opcnumber=d0e104204, accessed February

27 gesure could easly changed by he user. The Mouse gesure Sarer 29 [8], BumpTop 30 and Mac Jouch 31 sofware producs exend he gesure neracon of he used Operang Sysems. For hs reason hey do no suppor he basc gesures lke a Zoom and Roaon gesures. Especally, he Mac Jouch uses a bunch of new gesures (especally n he level 3 caegory). The Mac Jouch gesures are absoluely no nuve and obvous for he user. All hese gesures of he Jouch sofware belongs o he caegory of raned gesures. The user has o nves me for learnng hem. The sofware mus also delver a movaon for he user o learn hem. Even he Mac Jouch gesure collecon does no mee hese requremens, neracon desgners and sofware engneers can gan nspraon from hs gesure desgn. In he end s good o see ha hese specal sofware producs expermen wh gesural neracon. They are pushng he developmen of gesural neracon furher forward. No every mplemenaon s good, bu s an mporan vsonary eor for he bgger sofware producs lke he Deskop Operang Sysems and 3 rd Pary Tools. 29 mglaunch - Mouse Gesure Applcaon Launcher, From Mouse Gesure. webse, hp:// accessed February BumpTop Mul-Touch Gesures. From BumpTop webse, hp://bumpop.com/doc/mul-ouch-gesures/, accessed February Trackpad Gesures. From jouch 2 webse, hp:// accessed February

28 Browser Opera Browser Frefox Auodesk Maya BumpTop Verson 2 Mouse Gesure Sarer Clck / Tap x x x x Double Clck x x x x / Tap SWIPE x x x x x Two-way x x x x movemen n-way x x x movemen Crcle-based pah x x Mac Jouch Press AND Tap Press AND SWIPE Two Fnger SWIPE Zoom Roaon Three Fnger Tap Index Press AND wo nger SWIPE Tap AND wo nger SWIPE Two nger down AND SWIPE Two Fnger Down AND Index Double Tap Four Fnger Follow Tap Fve Fnger Scrunch x x x x x x x x x x x x x x Table 3.5: Gesures of addonal sofware producs 23

29 3.3.5 Hardware producs Exernal npu devces always relae o Operang Sysems. In he mos cases hey exend he npu mehods of he compuer, and aord new neracon forms. Ceran npu devces are assgned for specal sofware producs, ohers are no. For nsance, he Apple Magc Mouse 32 and he ordnary mouse devces exend he npu hardware of he used compuer. These devces send he npu daa drecly o he Operang Sysem and he Operaon Sysem decdes how o deal wh hs daa. The Synapc rackpad 3334 and he Wacom Bamboo Touch 35 & Fun 36 work deren. These devces requre a specal drver sofware. Ths drver sofware decdes how o deal wh he npu daa. For nsance, he Wacom Bamboo gesures work only n combnaon wh specal sofware producs and don' work wh sofware lke he Flash Player, Slverlgh and JavaFX. Maybe hs approach wll change n he fuure. The drver sofware wll use he Operang Sysem gesures API and pass hem drecly o he Operang Sysem. Aferwards every exernal mul-ouch devce should work wh every mul-ouchable sofware. From he neracon desgn vew he exernal npu devces suppor pre-raned gesures, lke clck, double clck, one-way movemen gesure, and he nuve Drec-Manpulaons gesures. Synapc Trackpad Wacom Bamboo Apple Magc Mouse Ordnary Mouse Clck / Tap x x x x Double Clck / Tap x x x x SWIPE x x x x Crcle-based Pah x Two Fnger Tap x Two Fnger SWIPE x x x Zoom x x Roaon x x Three Fnger Down Three Fnger SWIPE x x Table 3.6: Gesures of Hardware devces 32 Magc Mouse. From Apple Inc. webse, hp:// accessed February Synapcs Gesure Sue for TouchPads, From SynapcsTM webse, hp:// accessed February Gesures and Mul-Touch, from SynapcsTM webse, hp:// accessed February Bamboo Touch. From Wacom webse, hp:// accessed February Bamboo Fun. From Wacom webse, hp:// accessed February

30 3.4 General resuls Near he end of he analyss we proved ha gesure based neracon s already well esablshed n screen based envronmens. The mos suppored gesures are Drec-Manpulaon gesures. These gesures are more nuve and obvous for he user n screen-based envronmens han oher commands. The relaon beween basc compuer commands (see Coopers ls of basc commands for ponng devces2.3) and gesures poned ou, ha here sll exs some nnovaon possbles for developng new gesures. Especally, objec connecon gesures whch are no well nvesgaed ye. Projec Managemen Tools and Mnd-mappng Tools are usng Objec Connecon commands heavly. I would be very neresng o assgn approprae gesures for connecng hese deren objecs ogeher and manpulang hem. The Tex Inpu sofware from Swype Inc. 37 s a very nce example for Objec Connecon gesures. Swype Inc. connecs deren characer ogeher an generaes a word ou of. I works smlar o he T9 Tex Inpu gudance on moble phones. Also he herarchc-gesures concep, whch Apple s usng for scrollng (wo Fnger Swpe), skppng (hree Fnger Swpe) and swchng beween acve applcaons (four Fnger Swpe), s an neresng approach. The herarchc swpe gesures command s very well wh navgaon neracon. There may exs oher human-compuer-neracon forms, where makes sense o use hs knd of herarchcal gesure commands. Ineracon Desgners and engneers need o nd more such nuve and susanable gesures approaches. Drec-Manpulaon gesures ( Zoom, Roaon, ec.) and herarchc gesures mee he requremens of he user-menalmodel[4, page 27-32]. The user s able o merge hese gesure conceps wh real world behavor. Especally he hgher level (mul-pon / mul-ouch) neracon are no very well nvesgaed ye. A deeper research may pay ou on hs opc. The nex problem deals wh he caegory of raned gesures. In general, Ineracon Desgner should avod hs knd of gesures for novce user. Traned gesures are comparable wh shorcus on keyboards. The user has o learn hem and hey are dcul o remember. Therefore a gudance concep for hese knd of gesure would mprove he learnng curve dramacally. Forunaely, for ponng devces exs a very well nvesgaed gudance concep called Markng Menus. The Markng Menu approach was nally nvened for one pon devces (sylus and mouse). A Markng Menu exends he funconaly of a conex menu wh he aspec of radal menus. I s a able o perform a command va movng he cursor or he sylus n a ceran drecon[11, 12, 19, 1]. Markng Menus show every me he user wh a radal or ower menu wha funcons are avalable o he user. Ths approach s very useful for 2-way or 37 Tex Inpu for Screens from Swype Inc. webse, hp://swypenc.com/produc.hml, accessed March

31 n-way gesures. If he user can no remember he gesure he Markng Menu suggess he nex possble movemens. So he user can learn he gesure beer. Afer a whle he or she can perform he gesure very fas whou any gudance. The sofware produc Auodesk Maya uses hs approach successfully. Therefore s hghly recommend o use he Markng Menu approach for level 1 wo-way and n-way gesures. The same crcumsances are vald for hgher level gesures. In general hree nger gesures are already que dcul o remember. The nuve gesures Three Fnger Swpe, Three Fnger Tap and Three Fnger down are an excepon n hs caegory. All oher gesures are supposed o provde a gudance concep. Especally he sofware Mac Jouch uses many complcaed level 3 gesures. These gesures are no poorly desgned, bu hey are more dcul o remember han o perform. The Markng Menu approach can also help n hs case. Every gesure whch uses more pons (ngers) and a SWIPE gesure can use hs concep. I can sugges whch SWIPE drecon execues whch command. The Markng Menu research n mul-ouch envronmens s no well researched ye. Also provdng a gudance concep for more complcaed mul-ouch gesures s n he early days. The lack of gudance conceps n gesure based neracon s no he only man problem. Durng he analyss an addonal mporan ssue emerged. Some gesure documenaons were very dcul o access or poorly documened. The Sofware Jouch, BumpTop and he Browser Opera have exemplary documenaon of her suppored gesures. Unforunaely, a he momen here exs no real convenons how o documen gesures and how o use hem. Ryan Lee 38, Ron George 39, and Ideum 40 publshed free graphcs for conssen gesure documenaon. Unforunaely, he graphcs of Ryan Lee and Ideum (GesureWorks) have only a xed collecon of gesures. For hs reason hs paper uses he approach from Ron George. Hs documenaon graphcs suppor creang new cusom gesures. Therefore seems he bes way for documenng gesures n sac graphcs and prn meda. Dan Saer[17] and R. Clayon Mller 41 ake an mporan sep furher. They agree wh a conss gesure documenaon, bu hey also sared a dalogue for a conss gesure usage. The goal of her dalogues s o nd gesure paerns. Gesure paerns are lke user nerface and desgn paerns, whch funcon he same across deren sofware producs. The am of hs goal s ha he user wll bene from a conss user experence beween deren sofware producs and plaforms. Ths paper suppors hese deren dalogues, and hopes o smplfy he daly usage of gesures n he fuure. From he echncal perspecve smaller sofware producs are more movaed 38 GesureCons. From Ryan Lee webse, hp://gesurecons.com/, accessed March GesurCons. From Ron George Blog webse, hp://blog.rongeorge.com/desgn/gesurcons/, accessed March Open Source Mulouch Gesure Lbrary and Illusraons. From GesureWorks webse, hp://gesureworks.com/abou/open-source-gesure-lbrary/, accessed March /GUI. From R. Clayon Mller webse, hp://10gu.com, accessed March

32 o use new gesure conceps han he larger sofware producs. Furhermore, some sofware producs need o deal wh echncal ssues. For nsance, Moble Operang Sysems could suppor more gesures bu he moble hardware de- nes wha s possble. The same s vald for 3 rd Tool sofware. The Operang Sysems dene whch gesures are suppored by he 3 rd Pary Tools and whch are no. Almos no 3 rd Pary Tool manufacurer mplemened her own gesure recognon sysem. In addon, ouch screens also have dsadvanages compared o radonal ponng devces[17, page 26-27]. The rgh-clck menu s no really avalable on ouch screens envronmens. Also he mouse-hover eec dsappeared n ouch screens envronmens. For hs reason, he neracon desgner should consder ha mos ouch screen envronmens and also gesure-based neracons are saeless or modeless. Tha s no a bg problem for he user, bu he neracon desgner should ake care of hese derences compared o radonal ponng devces (such as he mouse devce). In some cases he vsual feedback works deren han for radonal ponng devces. Also he mplemenaon of gesure recognon hreshold depends on he screen ype. The poner of he nger s much bgger han he poner of a mouse devce. Gesure recognon sysems mus be able o deal wh hs naccuracy n ouch screen envronmens. Hrvoje Benko, Andrew D. Wlson and Parck Baudsch speced some gudance n her paper Precse Selecon Technques for Mul-Touch Screens [6]. Usng a ngerp nsead of a mouse poner or a sylus pen wll be more naccurae. The nex mporan pon s he characersc of movemen n he hreshold mplemenaon. Dynamc gesures are more dcul o deec, han he sac gesures. Therefore he hreshold mplemenaon s crucal facor n gesure recognon sysems. In general radonal ponng devces conan a ceran naccuracy n performng a movemen. Consequenly neracon desgners and sofware engneers need o mplemen a hreshold o her paern-mach gesure sysem. I s almos mpossble for he user o perform a gesure wh 100% accuracy. Especally, crcle-pah based gesures are more dcul o perform wh a mouse devce han n a ouch screen envronmen. The sofware BumpTop and he Synapcs rackpad mplemened her crcle-pah gesure prey well. The las echncal pon, sze and resoluon of he screen nuence he desgn process of gesures srongly. Some gesures, for nsance, he Fve Fnger Scrunch gesure from BumpTop, are no able o adap for moble devces. The screen s oo small for usng whole hand and he ngers. One possbly mgh be o reducng he amoun of ngers for performng a Scrunch gesure. Insead of usng 5 ngers use 3 ngers. The dsadvanage of hs soluon s he broken user experence of he gesure. Anoher pon s he deren usage conex of small screen compared o bgger screens. Small screen devces are also very ofen used wh one hand. An example scenaro could be n a bus. The user hold hmself wh one hand, and he devce n he oher hand. In hese knds of scenaros 27

33 here s only he humb lef for gesure neracon. Ths crcumsance needs a compleely deren gesure desgn approach. Also n bg screens envronmens he Fve Fnger Scrunch gesure wll no work. For nsance, he rng wall from Sensory Mnds 42 uses bg graphcal ems n her user nerface. Ther graphcal ems are oo bg o group hem ogeher wh one hand. Ponng and Selecon va arm gesures could be a soluon for hs problem[18]. In he end he screen sze s no a problem for gesure based neracon. The neracon desgner mus consder he screen capables, and hen choose he approprae gesure desgn approach. Ths chaper explored and hghlghed he common gesures. I menoned he problems wh gesure neracon n general, and proposed some soluons. The analyss also showed, ha he mos used gesures (Double Clck, SWIPE, Zoom and Roaon gesures) should used n he same manner, as oher end-user producs are dong. In he end wll provde a beer user experence. 42 SENSORY MINDS - NATURAL USER INTERFACE DESIGN. From Sensory Mnds webse, hp://10gu.com, accessed March

34 Chaper 4 An Indusral Desgn vew on he evoluon of Ponng Devces Afer researchng he hsory of ponng devces seems, ha neracon desgners are consanly changng her hardware approach o a sofware approach. Ths evoluon s neresng from an ndusral desgner vew. Indusral desgners wll have less n uence when desgnng a devce, or ndusral desgners mus also learn he prncples of Inerface Desgn. Fgure 4 pons ou hs basc dea of evoluon for Deskop PC and Noebooks. Tr ad onaldev ces ( ol d) Cur r eni npudev ces ( new) MouseDev ce TwoBu onsandamousewheelar edef n ng he n er ac on des gn.the n er ac onposs b l esar es r ongl ybasedon he har dwar edes gn. Tr ackpad w hscr ol l ng ThecommonTr ackpadonno ebookshasgo wobu ons( l ef andr ghcl ck),anar eaf orpos on ng hemouse( po n er ),and as pec alar eaf orver calandhor zon alscr ol l ng.the n er ac on des gn sspec f edbysof war eandhar dwar e. Tr ack padonl y TheAppl eandsynap cstr ackpadhavegoonl yonepl anear ea. Th spl anear eacande ec ouch ng nf or ma on.thesof war edef neshow odealw h hese ouc hda a.thesof war e mpl emen s cl cks,pos on ng,scr ol l ngandges ur er ecogn oncommands. LowV r ual I n er ac on ( Har dwar econ r ol l ed) H gherv r ual I n er ac on ( Sof war econ r ol l ed) Fgure 4.1: Evoluon of Lapop Ponng Devces A very smlar evoluon emerged for cellphone devces. The rs cellphone devces are srongly based on keyboard neracon. Then he cellphone manufacurer added a joysck o her devces for a beer vercal and horzonal 29

35 navgaon. Aferward he pen (sylus) neracon became more popular. Unl now he keyboard s almos compleely gone (see gure 4). The Phone and he Google Phone s heavly usng he ouchscreen for drec neracon wh he devce. Tr ad onal Dev c es ( ol d) Key boar dbas ed dev c es Thec ommonmob l edev c esl k enok a5110,ands mar phones l k egs88/penel ope,us ed hek ey boar df or n er ac onw h he mob l edev c e.thes c r eenwasonl y hev s ual f eedbac kc hannel. Hy br d dev c es Mob l edev c esbec amemor ec ompl ex.thek ey boar dasan npu c hannel wasnoenough.themob l emanuf ac ur erdev el opeda po n ngdev c e npu nf or m ofapen.us er sc oul d n er ac w h hepend r ec l yon hes c r een. Touchs c r eens Thenewgener a onofs mar phones,l k e he Phoneand he Googl ephone,ar eal mos us ngonl y he ouc hs c r eenasan npu c hannel.al s onoadd onal dev c e,l k e hepen, sneededany mor e.thes of war eand heus er n er f ac edef nes he n er ac on. Cur r eni npudev c es ( new) LowV r ual I n er ac on ( Har dwar econ r ol l ed) H gherv r ual I n er ac on ( Sof war econ r ol l ed) Fgure 4.2: Evoluon of moble devces Are hese wo developmens evdence for he hypohess, ha he npu channel for ponng devces become more vrual? Wll be he physcal npus as buons, joyscks, and (mouse) wheels be compleely removed n he fuure? Wll be he physcal pars be replaced by vrual npus va ouchscreens? The drec neracon wh he screen emphaszes he dea of Drec-Manpulaon. Drec-Manpulaon s easer o learn and more nuve for humans. I ad- dresses how a human works drecly wh physcal hngs n he real world. The physcal npus can no really realze hs Drec-Manpulaon approach n screen-based envronmens. Unforunaely n hs paper was no possble o nd an evdence, ha Drec-Manpulaon va ouch or ponng devce performs beer, han a physcal npu sysem. In relaon o vrual npus and her evoluon, s obvous ha gesure based neracon s gong o play a very mporan role. The success of he Phone and oher mul-ouch sysems con rm hs developmen n general. The fuure of gesure neracon doesn' rely only on ouch gesures. Also free-form gesures are playng an mporan role. Especally n he game ndusry. The 1 s an evden for hs evoluon. Also he camera 2 from Mcrosof and he BD Screen3 success of he Nnendo W based neracon sysem Projec Naal from he MIT con rm he evoluon of gesure based neracon. 1 Nnendo W. From W.com. webse, hp:// accessed March Projec Naal. From Mcrosof Xbox.com. webse, hp:// accessed March BDScreen, From MIT Meda Lab. webse, hp://web.meda.m.edu/~mhrsch/bd/, accessed March

36 For hs reason s approprae o ask f he hardware based npu wll compleely dsappear. Wll humans nerac wh compuer whou any devce n he fuure? Ths free form of neracon has one dsadvanage. The user needs a feedback f he gesure or neracon s successfully performed. If no, hen he user needs o be nformed abou he wrong performance. Tangble (physcal) devces can delver such gudance. The eld of organc and knec nerfaces[14, 5, 7, 13, 16, 20] can solve hs ask. They deal wh he angble characersc of neracon and feedback. These devces decrease he user's freedom of movemen. I s mporan o undersand ha hs research s no agans free-form (gesure) neracon. In some cases knec based neracon and free form neracon can bene from each oher. Knec devces can gve he user a gudance or a leas a vsual feedback for he correc gesure performance. I suppors he users movemen. Moreover he projec D20 from Olver Bau 4 and Poupyrev[15] shows he combnaon of mul-ouch gesures and a angble devce n an exemplary form. Therefore hese wo deren Human Compuer Ineracon approaches wll merge n he fuure. The Wmoe devce and s vbraon feedback channel combned wh he gesure based neracon s only he begnnng of hs evoluon. The fuure les n he combnaon of Indusral and Inerface Desgn. Indusral desgners have he knowledge abou he human knec and movemen. nerface desgners have he knowledge abou he vsual and nformaon percepon. Boh dscplnes are val for developng nex generaon devces. 4 D20: Ineracon wh Mulfaceed Dsplay Devces, From Olver Bau Insue webse, hp://nsu.lr.fr/~bau/d20.hml, accessed March

37 Chaper 5 Concluson Ths paper showed ha gesure based neracon s a very powerful ool n Human Compuer Ineracon. I has already arrved n large sofware producs, such as Deskop Operang Sysems. I s also obvous ha gesure based neracon wh ponng devces can mprove and speed up he neracon beween humans and compuers[9]. Gesures do no creae a new communcaon form or even a new neracon language. Unl he merge beween organc nerfaces and free form gesures s complee, gesure-based neracon wh ponng devces wll ad n he ranson beween hese deren Human Compuer Ineracon approaches. A resumpon of hs paper could be an nvesgaon n gesures dependng on he screen sze. In he fuure s mporan o nd he answer for he quesons: ˆ Whch gesures are approprae for whch screen sze? ˆ How do he gesure-based neracons dsngush each oher from deren screen-based envronmens? The nex pon could be n researchng he meda conen of games. Games suppor hghly neracve conen and user nerfaces. I s also neresng o know whch gesures are used n games. Whch gesure are useful and how hey dsngush from each oher n deren game genres. Sraegy games mgh use oher gesures han Ego-Shooer games. Whch genre-based gesures exs for games? I would be also very neresng o explore he free-form gesure and ponng devce gesure paerns n games. Are free-form gesures more used han ponng devce gesures n games? Maybe he resuls wll nspre he fuure gesures n common sofware producs (lke n Mnd-mappng Tools). There sll exs many deren domans o nvesgae gesural neracon. Ths paper summarzed only a small par of hs doman. The resuls gave an overvew wha s acually sae of he ar n gesure neracon for ponng 32

38 devces. Now we should accuraely observe he new developmens n free-form and mul-ouch gesures. 33

39 Bblography [1] Lecolne E Bally, G. and L. Ngay. Flower menus: A new ype of markng menu wh large menu breadh, whn groups and ecen exper mode memorzaon. In AVT2008, pages 1522, Napol, Ialy, May [2] Mark Bllnghurs and Bll Buxon. Human npu o compuer sysems: Theores, echnques and echnology. Chaper 14: Gesure Based Ineracon, [3] Bll Buxon. Mul-ouch sysems ha have known and loved. Bll Buxon Research Webse, January [4] Cronn Cooper, Remann. In Abou Face 3 - The Essenal of Ineracon Desgn. Wley Publshng, [5] Davd Holman and Roel Veregaal. Organc user nerfaces: desgnng compuers n any way, shape, or form. Commun. ACM, 51(6):4855, [6] Andrew D. Wlson Hrvoje Benko and Parck Baudsch. Precse selecon echnques for mul-ouch screens. In CHI 2006 Proceedngs - Ineracng wh Large Surfaces, pages , Monréal, Québec, Aprl CHI [7] Hrosh Ish. The angble user nerface and s evoluon. Commun. ACM, 51(6):3236, [8] Yang L Jacob O. Wobbrock, Andrew D. Wlson. Gesures whou lbrares, oolks or ranng: A one dollar recognzer for user nerface prooypes. In UIST 2007, Newpor, Rhode Island, USA, UIST. [9] Kenrck Kn, Maneesh Agrawala, and Tony DeRose. Deermnng he bene- s of drec-ouch, bmanual, and mulnger npu on a mulouch worksaon. In GI '09: Proceedngs of Graphcs Inerface 2009, pages , Torono, On., Canada, Canada, Canadan Informaon Processng Socey. [10] E. Kurenbach, G. & Huleen. The Ar of Human Compuer Inerface Desgn, chaper Gesures n Human-Compuer Communcaons, pages Addson-Wesley,

40 [11] W. Kurenbach, G. & Buxon. The lms of exper performance usng herarchc markng menus. In Proceedngs of InerCHI, pages , [12] W. Kurenbach, G. & Buxon. User learnng and performance wh markng menus. In Proceedngs of CHI, pages , [13] Kas Ooserhus and Nmsh Blora. Ineracons wh proacve archecural spaces: he muscle projecs. Commun. ACM, 51(6):7078, [14] Amanda Parkes, Ivan Poupyrev, and Hrosh Ish. Desgnng knec neracons for organc user nerfaces. Commun. ACM, 51(6):5865, [15] Ivan Poupyrev, Henry Newon-Dunn, and Olver Bau. D20: neracon wh mulfaceed dsplay devces. In CHI '06: CHI '06 exended absracs on Human facors n compung sysems, pages , New York, NY, USA, ACM. [16] Jun Rekmoo. Organc neracon echnologes: from sone o skn. Commun. ACM, 51(6):3844, [17] Dan Saer. Desgnng Gesural Inerfaces. O'Relly, [18] Alexander Schck, Floran van de Camp, Jors Ijsselmuden, and Raner Sefelhagen. Exendng ouch: owards neracon wh large-scale surfaces. In ITS '09: Proceedngs of he ACM Inernaonal Conference on Ineracve Tableops and Surfaces, pages , New York, NY, USA, ACM. [19] Ravn Balakrshnan Shengdong Zhao. Smple vs. compound mark herarchcal markng menus. In UIST, pages 3342, Sana Fe, New Mexco, USA Ocober [20] Roel Veregaal and Ivan Poupyrev. Inroducon. Commun. ACM, 51(6):2630,

41 Appendx In hs appendx all gesures are lsed wh a descrpon, he suppored command, and he suppored produc. The gesure graphcs are based on he gesure documenaon of GesurCons from Ron George 1. 1 GesurCons. From Ron George Blog webse, hp://blog.rongeorge.com/desgn/gesurcons/, accessed March

42 Tap or Clck The user presses a mouse buon and releases he same mouse buon fasly. For ouch devce he user ouches he screen a a ceran poson, and removed he fnger drecly afer he ouch conac. Level 1 Gesure graphc Gesure Name Command Producs Tap and Clck Selec Wndows 7 Tap and Clck Selec Mac OS Tap and Clck Selec Lnux Tap Selec Wndows Moble 6.5 Tap Selec Google Androd 1.6 Tap Selec BlackBerry Sorm 2 Tap Selec Symban OS Tap Selec Phone Tap and Clck Selec Adobe Flash Tap and Clck Selec Slverlgh Tap and Clck Selec JavaFX Tap Selec Wacom Bamboo Tap Selec Synapc Trackpad Tap Selec Mac Trackpad Clck Selec Mce Devce Tap and Clck Selec Apple Magc Mouse Tap and Clck Selec Opera Browser Tap and Clck Selec Frefox Browser Tap and Clck Selec Auodesk Maya Tap and Clck Selec BumpTop Verson 2 Clck Selec Mouse Gesure Sar Tap Selec Mac Jouch

43 Level 2 Gesure graphc Gesure Name Command Producs Two Fnger Tap cusom Wndows 7 Two Fnger Tap Rgh Clck Menu (Conex Menu) Mac OS Two Fnger Tap cusom Adobe Flash (only on Wndows) Two Fnger Tap cusom Slverlgh Two Fnger Tap Rgh Clck Menu (Conex Menu) Wacom Bamboo Two Fnger Tap Rgh Clck Menu (Conex Menu) Mac Trackpad Level 3 Gesure graphc Gesure Name Command Producs Three Fnger Tap Open new lnk n a ab Mac Jouch

44

45 Double Tap or Double Clck The user fasly clck he mouse buon wo mes. For ouch devce he user ouched he screen a a ceran poson, and removed he fnger drecly afer he ouch conac. Ths Tap acon he user performs wo mes. Level 1 Gesure graphc Gesure Name Command Producs Double Tap and Double Clck Open, Acvae Wndows 7 Double Tap and Double Clck Open, Acvae Mac OS Double Tap and Double Clck Open, Acvae Lnux Double Tap Open, Acvae Wndows Moble 6.5 Double Tap Open, Acvae Google Androd 1.6 Double Tap Open, Acvae BlackBerry Sorm 2 Double Tap Zoom Symban OS Double Tap and Double Clck cusom Adobe Flash Double Tap and Double Clck cusom Slverlgh Double Tap and Double Clck Open, Acvae JavaFX Double Tap Open, Acvae Wacom Bamboo Double Tap Open, Acvae Synapc Trackpad Double Tap Open, Acvae Mac Trackpad Double Clck Open, Acvae Mce Devce Double Tap and Double Clck Open, Acvae Apple Magc Mouse Double Tap and Double Clck Open, Acvae Opera Browser Double Tap and Double Clck Open, Acvae Frefox Browser Double Tap and Double Clck Open, Acvae Auodesk Maya Double Tap and Double Clck Open, Acvae BumpTop Verson 2 Double Clck Open, Acvae Mouse Gesure Sar Double Tap Open, Acvae Mac Jouch

46

47 Swpe The user presses a mouse buon, hen movng he mouse cursor o a ceran poson, or fas anywhere on he screen. Afer he movng acon he user releases he mouse buon. For ouch devce he user ouches he screen a a ceran poson (graphc objec or a specal area), and moves he fnger along he screen. The end of he movemen can be a ceran poson on he screen, or somewhere on he screen. Level 1 Gesure graphc Gesure Name Command Producs Swpe, Flck Drag, Drag and Drop, Scrollng Wndows 7 Swpe Drag, Drag and Drop Mac OS Swpe Drag, Drag and Drop Lnux Swpe Drag, Drag and Drop, Scrollng Wndows Moble 6.5 Swpe Drag, Drag and Drop, Scrollng Google Androd 1.6 Swpe Drag, Drag and Drop, Scrollng BlackBerry Sorm 2 Swpe Drag Symban OS Swpe Drag, Scrollng, Skppng Phone Pan cusom Adobe Flash Swpe cusom Slverlgh Swpe Drag JavaFX Swpe Drag, Scrollng Wacom Bamboo Momenum Drag, Drag and Drop Synapc Trackpad Swpe Drag, Drag and Drop Mac Trackpad Swpe Drag, Drag and Drop Mce Devce Swpe Drag, Drag and Drop, Scrollng Apple Magc Mouse Swpe Lef mouse for draggng and drag Rgh Clck for gesure performng Swpe Lef mouse for draggng and drag Rgh Clck for gesure performng Opera Browser Frefox Browser

48 Swpe Lef mouse for draggng and drag Rgh Clck for gesure performng Auodesk Maya Flck Drag, Drag and Drop BumpTop Verson 2 Swpe Lef mouse for draggng and drag Rgh Clck for gesure performng Mouse Gesure Sar Gesure graphc Gesure Name Command Producs Two way gesure cusom Wndows Moble 6.5 Two way gesure cusom Google Androd 1.6 Two way gesure cusom Symban OS Two way gesure Maxmze wndow, close ab, open ab, ec. Opera Browser Two way gesure Maxmze wndow, close ab, open ab, ec. Frefox Browser Two way gesure cusom Auodesk Maya Two way gesure cusom Mouse Gesure Sar Gesure graphc Gesure Name Command Producs n-way gesure cusom Wndows Moble 6.5 n-way gesure cusom Google Androd 1.6 n-way gesure cusom Symban OS n-way gesure cusom Frefox Browser n-way gesure cusom Auodesk Maya n-way gesure cusom Mouse Gesure Sar

49 Level 2 Gesure graphc Gesure Name Command Producs Two fnger swpe Scrollng Wndows 7 Two fnger swpe Scrollng Mac OS Two fnger Pan cusom Adobe Flash Two fnger Pan cusom Slverlgh Two fnger swpe Scrollng, skppng Wacom Bamboo Two fnger swpe Scrollng Synapc Trackpad Two fnger swpe Scrollng Mac Trackpad Two fnger swpe Skppng Apple Magc Mouse Fan Ou, Pan, Focus on Wall Ungrop, drag, selec BumpTop Verson 2 Level 3 Gesure graphc Gesure Name Command Producs Three fnger swpe Skppng Mac OS Three fnger swpe Skppng Synapc Trackpad Three fnger swpe Skppng Mac Trackpad

50 Level 4 Gesure graphc Gesure Name Command Producs Three fnger swpe Swchng beween Applcaons Mac OS Three fnger swpe Swchng beween Applcaons Mac Trackpad

51

52 Press and Tap For ouch devce he user ouches he screen wh one fnger (e.g. Index fnger) a a ceran poson, and wh he second fnger (e.g. Mddle fnger) he user performs a shor ap Level 2 Gesure graphc Gesure Name Command Producs Press and Tap Rgh clck menu (conex menu) Wndows 7 Press and Tap cusom Adobe Flash (only on Wndows) Press and Tap cusom Slverlgh

53

54 Zoom For ouch devce he user ouches he screen wh wo fngers. Aferwards move he wo fngers ogeher or away from each oher. Level 2 Gesure graphc Gesure Name Command Producs Zoom Scalng Wndows 7 Pnch Scalng Mac OS Zoom Scalng Lnux Pnch Scalng Phone Zoom Scalng, cusom Adobe Flash Zoom Scalng, cusom Slverlgh Zoom Scalng JavaFX Zoom Scalng Wacom Bamboo Zoom Scalng Synapc Trackpad Pnch Scalng Mac Trackpad Zoom, Grow and shrnk Scalng BumpTop Verson 2

55

56 Roae For he level 1 gesure. Press wh one fnger he mouse buon or on he screen. Then perform wh he mouse cursor or your fnger a crcle pah movemen. For ouch devces he user ouches he screen wh wo fngers (2 level). Aferwards roae he wo fngers ogeher clockwse or couner-clockwse. Level 1 Gesure graphc Gesure Name Command Producs ChralScrool TM scrollng Synapc Trackpad Lasso, Lasso'n'Cross Groupng and ungroupng BumpTop Verson Level 2 Gesure graphc Gesure Name Command Producs Roae No defnon Wndows 7 Roae roae Mac OS Roae roae Lnux Roae cusom Adobe Flash Roae cusom Slverlgh Roae roae JavaFX Roae roae Wacom Bamboo Roae roae Synapc Trackpad Roae roae Mac Trackpad Roae Roae 3D Vew BumpTop Verson 2

57

58 Fnger(s) down and Swpe The user press one or more fngers down and hen performs a swpe gesure wh one or more fngers. Level 2 Gesure graphc Gesure Name Command Producs Fnger (ndex fnger) down and Swpe Fnger (ndex fnger) down and Swpe Croppng a phoo BumpTop Verson 2 drag and scalng Mac Jouch Level 3 Gesure graphc Gesure Name Command Producs Fnger (ndex fnger) down and wo-fnger swpe Open and close ab Mac Jouch

59 Gesure graphc Gesure Name Command Producs Two fngers down and one fnger swpe Skppng (Swchng o he chosen neghbor space) Mac Jouch

60

61 Three fngers down The user presses wh hree fngers on he ouch sensve area. Level 3 Gesure graphc Gesure Name Command Producs Three fngers down Openng applcaons Mac Jouch

62

63 Tap and Swpe The user performs a ap gesure and drecly afer ha a swpe gesure s performed. Level 3 Gesure graphc Gesure Name Command Producs Tap and wo-fnger swpe Qu an applcaon Mac Jouch

64

65 Double ap and fnger down The user presses wo fngers (e.g. Mddle and rng fnger) down and hen performs wh one fnger (e.g. Index fnger) a double ap. Level 3 Gesure graphc Gesure Name Command Producs Double Tap and wo-fnger down Refresh, updae Mac Jouch

66

67 Followng fngers down The user sars pressng wh one fnger (e.g. Index fnger) on he ouch-sensve area and hen does he same wh he nex hree fngers. Level 4 Gesure graphc Gesure Name Command Producs Four fnger follow press Maxmze and Mnmze he wndow Mac Jouch