FOUR EASY PIECES FOR ASSESSING THE USABILITY OF MULTIMODAL INTERACTION: THE CARE PROPERTIES

1 FOUR EASY PIECES FOR ASSESSING THE USABILITY OF MULTIMODAL INTERACTION: THE CARE PROPERTIES Daniel Salber BP 53, 38041 Grenoble Cedex 9, France {joelle.coutaz, laurence.nigay, daniel.alber} @imag.fr Ann Blandford, Jon May, Richard M. Young MRC-APU, 15 Chaucer Road, Cambridge, U CB2 2EF {ann.blandford, jon.may, richard.young} @mrc-apu.cam.ac.uk EYWORDS: Multimodal interaction, complementarity, aignment, redundancy, equivalence, uability. ABSTRACT: We propoe the CARE propertie a a imple way of characteriing and aeing apect of multimodal interaction: the Complementarity, Aignment, Redundancy, and Equivalence that may occur between the interaction technique available in a multimodal uer interface. We provide a formal definition of thee propertie and ue the notion of compatibility to how how the ytem CARE propertie interact with uer CARE-like propertie in the deign of a ytem. The dicuion i illutrated with MATIS, a Multimodal Air Travel Information Sytem. 1. INTRODUCTION In recent year, uability teting ha been the ubject of growing interet, reulting in an exploion of method and technique. All are fundamentally baed on the identification of propertie that a ytem hould atify, election of the method and tool for upporting the uability goal, and evaluation of the deirable propertie. The deigner problem i to decide upon an appropriate et of propertie to ue. General propertie uch a conitency, obervability, and pre-emptivene are ueful criteria for characteriing interactive ytem, but they do not cover the pecific of new interaction technology uch a multimodal uer interface. Multimodal uer interface upport interaction technique which may be ued equentially or concurrently, and independently or combined ynergitically (Nigay & Coutaz, 1993). Novel apect of interaction mut be conidered, uch a the fuion and fiion of information, and the nature of temporal contraint on the interaction. In thi paper, we propoe the CARE propertie a a imple way of characteriing and aeing apect of multimodal interaction: the Complementarity, Aignment, Redundancy, and Equivalence that may occur between the interaction technique available in a multimodal uer interface. Thee term have been ued rather looely in the literature (Martin, 1994). The next two ection provide formal definition. The dicuion will be illutrated with MATIS, a multimodal interactive ytem developed in collaboration with colleague at Carnegie Mellon Univerity. In the final ection, we ue the notion of compatibility between uer preference and ytem propertie to how how the CARE propertie interact with uer modelling to predict uability during the deign of a ytem. 2. CONCEPTS: THE CARE PROPERTIES The formal expreion of the CARE propertie relie on the notion of tate, goal, modality, and temporal relationhip. A tate i a vector of obervable, that i, a et of propertie that can be meaured at a particular time to characterie a ituation. A goal i a tate that an agent intend to reach. An agent, e.g., a uer, or the ytem, or a component of the ytem, i an entity capable of initiating the performance of action. A modality i an interaction method that an agent can ue to reach a goal. To model the expreive power of a modality m, that i, it capacity to allow an agent to reach tate from tate in one tep, we ue the function Reach(, m, ). A equence of ucceive tep (or tate) i called an interaction trajectory. Thi generic definition of a modality can be interpreted at different level of refinement. For To Appear in the Proceeding of INTERACT'95, Lillehammer, June 1995

example, a modality could be pecified in general term a uing peech, or more pecifically a uing a microphone. Both of thee interpretation are valid. A temporal relationhip characterie the ue over time of a et of modalitie. The ue of thee modalitie may occur imultaneouly or in equence within a temporal window, that i, a time interval. Alternatively, only one modality from a et may be ued. Let Pick(, m, ) be a predicate that expree the ue of m among a et of modalitie to reach from. Modalitie of a et M are ued imultaneouly (or in parallel) if, within a temporal window, they happen to be active at the ame time. Let Active (m, t) be a predicate to expre that modality m i being ued at ome intant t. The imultaneou ue of modalitie of a et M over a finite temporal window tw can be formally defined a: Parallel (M, tw) Card (M) > 1) (Duration(tw) t tw m M Active (m, t)) where Card (M) i the number of modalitie in et M, and Duration(tw) i the duration of the time interval tw. Sequential event may have to occur within a temporal window to be interpreted a temporally related. If they occur outide thi window, then they may be interpreted differently. Modalitie M are ued equentially within a temporal window tw if there i at mot one modality active at a time, and if all of the modalitie in the et are ued within tw: Sequential (M, tw) Card (M) >1) (Duration (tw) t tw ( m, m' M Active(m, t) m M t tw Active(m, t)) Temporal window for parallelim and equentiality need not have identical duration. The important point i that they both expre a contraint on the pace of the interaction. For example, uing multiple modalitie imultaneouly may be appropriate for the uer but may require extra proceing reource from the ytem ide or imply a pecific oftware architecture (Nigay & Coutaz, 1993). The abence of temporal contraint i treated by conidering the duration of the temporal window a infinite. T m1 m2... mn g TR TW Figure 1: A notation for expreing CARE propertie. Figure 1 how the relationhip between the concept ued in CARE: an agent having reached tate, may have (or ha) the goal g to reach tate by mean of the non-empty et of modalitie M={m1, m2... mn}. Thee modalitie are linked by temporal relationhip TR (TR::= for parallelim, equentiality, and election of one interaction technique repectively), and contrained by a temporal window TW (TW::=<interval>. The ymbol T at the upper left denote the CARE property (T::=C A R E). The CARE propertie, which characterie four type of relationhip between tate and modalitie, can be defined and illutrated with MATIS (Nigay, 1994). Our dicuion will concentrate on input (i.e., from the uer to the ytem) although the definition hold for output a well. MATIS (Multimodal Airline Travel Information Sytem) allow a uer to retrieve information about flight chedule uing peech, direct manipulation, keyboard and moue, or a combination of thee method, upporting individual and ynergitic ue of multiple input modalitie. For example, uing a ingle modality, the uer can ay how me the USAir flight from Boton to Pittburgh, can type entence in peudo-natural language in a dedicated text window, or can fill in a form uing the keyboard and the moue. When exploiting ynergy, the uer may alo combine peech and geture a in how me the USAir flight from Boton to thi city along with the election of "Pittburgh" with the moue on the creen. 3. CARE PROPERTIES: FORMAL DEFINITION Equivalence: Modalitie of et M are equivalent for reaching from, if it i neceary and ufficient to ue any one of the modalitie. M i aumed to contain at leat two modalitie. More formally: Equivalence (, M, ) Card(M) >1) ( m M Reach (, m, )) Equivalence expree the availability of choice between multiple modalitie but doe not impoe any form of temporal contraint on them. Figure 2 how an example of equivalence between everal modalitie for pecifying Pittburgh a the detination of a trip. Uer have a choice of peaking or typing the entence Flight to Pittburgh, or keying Pittburgh in the detination lot of the requet form. Alternatively, they may interact with the Tool window and pick up Pittburgh a a detination from the menu of known citie. Aignment: Modality m i aigned in tate to reach, if no other modality i ued to reach from. In contrat to equivalence, aignment expree the abence of choice: either there i no choice at all to get from one tate to another, or there i a choice 2

but the agent alway opt for the ame modality to get between thee two tate. Thu we can define two type of aignment: StrictAignment (, m, ) Reach (, m, ) m' M. Reach(, m',) m'=m) AgentAignment (, m, M, ) (Card(M) >1) m' M. (Reach (, m', ) (Pick (, m', )) m'=m) ) In the cae of an agent aignment, it i intereting to analye the interaction trajectorie to explain and jutify it occurrence. In MATIS, window management i performed by direct manipulation only. In particular, peech cannot be ued a an alternative. Therefore, the ytem impoe a trict aignment upon the uer for window baed tak. Converely, a uer who alway ue peech to pecify trip detination would turn the Equivalence offered by the ytem into an Agent Aignment. Thi iue will be developed further in our ection on uer modelling. Equivalence and aignment both meaure the choice available at ome point in the interaction trajectory. Redundancy and complementarity go one tep further by conidering the combined ue of multiple modalitie under temporal contraint. E "Flight to Pittburgh" "Flight to Pittburgh" in NL Window "Pittburgh" in Detination Slot "Pittburgh" in Tool Window pecify detination Figure 2: Example of equivalence in MATIS. The lip ymbol denote peech, repreent the ue of the keyboard, and the moue ymbol, a moue election. Redundancy: Modalitie of a et M are ued redundantly to reach tate from tate, if they have the ame expreive power (they are equivalent) and if all of them are ued within the ame temporal window, tw. In other word, the agent how repetitive behaviour without increaing it expreive power: Redundancy (, M,, tw) Equivalence (, M, ) Sequential (M, tw) Parallel (M, tw)) Redundancy can comprie two ditinct temporal relationhip equentiality and parallelim which may have different implication for uability and oftware implementation. In particular, parallelim put retriction on the type of modalitie that can be ued imultaneouly: modalitie that compete for the ame ytem or human reource cannot be activated in parallel. The agent can then only act 1 equentially if it can comply with the temporal contraint (i.e., it mut act quickly for the multiple input to be treated a if they were parallel). For example, the MATIS ytem i able to upport parallel-redundancy between peech act and any one of the other equivalent modalitie preented in Figure 2. A hown in Figure 3, redundant typing in the NL window and in the detination lot compete for the ame ytem and human reource. Therefore thee two method mut be ued equentially (unle two keyboard and two uer were available) and within the temporal window tw, which in turn mut be kept within tw'. When the uer doe not atify the temporal contraint, MATIS create a new requet initiated with the extra redundant information. Thi ytem deciion may not be in accordance with the uer expectation or intention. R "Flight to Pittburgh" R "Pittburgh" in Tool Window "Flight to Pittburgh" in NL Window "Pittburgh" in Detination Slot pecify detination Figure 3: Example of redundancy in MATIS. ; tw tw' According to our formal definition, redundancy require equivalence but equivalence only tipulate the exitence, not the activation, of multiple method. Thi aymmetry ha implication for the ytem robutne. Suppoe for example that a uer i ending the ame content concurrently via the equivalent modalitie m and m'. If the ytem doe not upport redundancy for m and m', either it i unable to ene m while ening m', or it ene both of them and may get confued. Therefore, oftware deigner mut have a clear architectural model a well a the appropriate oftware mechanim to handle thee iue appropriately. Complementarity: Modalitie of a et M mut be ued in a complementary way to reach tate from tate within a temporal window, if all of them mut be ued to reach from, i.e., none of them taken individually can cover the target tate. To expre thi adequately, we need to extend the notion of reachability to encompa et of modalitie: REACH(,M,) mean that tate can be reached from tate uing the modalitie in et M. Complementarity (, M,, tw) (Card(M) >1) ( M' PM (M' M (Duration(tw) REACH (, M, ) Sequential (M, tw) Parallel (M, tw)) 3

Deictic expreion, characteried by cro-modality reference, are example of complementarity. A hown in Figure 4, a MATIS uer can type or peak the entence flight to thi city (or imply flight to ) and elect a city name on the creen. Here, the entence pecifie the focu of interet (i.e., the detination of the trip) while the moue election denote a location. Thee two modalitie complement each other and mut be combined to reach the intended goal. A with redundancy, complementarity may occur in parallel or equentially within a temporal window. C E "Flight to" 1 "Flight to thi city" "Flight to" "Flight to thi city" "Pittburgh" on creen ; tw Figure 4: Example of complementarity in MATIS. Complementarity can be parallel or equential. In contrat to redundancy, which doe not favour any modality, complementarity may be driven by a dominant modality, which require the ue of other. Typically, in MATIS, deictic reference in peech require the ue of the moue to point to a creen object. Cro modality reference may draw upon both complementarity and redundancy. For example, a MATIS uer may ay flight to thi city while typing Pittburgh in the detination lot of the requet form. In thi cae, from the ytem perpective, the peech act denote the topic of interet while the typing action pecifie both the topic and it value. The peech act, which i covered by the typing act in the detination lot, hould be ignored by the ytem (i.e., the ytem hould not wait for the reolution of the deictic reference). Again, the CARE propertie have implication for ytem implementation. Another ource of complexity for oftware deigner i that ditinct action produced within the ame temporal window through different modalitie are not necearily complementary. In thi cae, fuion mut not be performed. For example, a MATIS uer may ay Flight to Pittburgh while electing an irrelevant object on the creen. Elewhere (Nigay & Coutaz 1993; Nigay 1994) we decribe how thee iue can be upported conitently through a reuable oftware mechanim. Our formal definition of the CARE propertie provide conceptual foundation for reaoning about multimodal interaction. To put them to work, we need to dicu iue uch a coverage and refinement. Coverage ha to do with the et of tate for which a particular property hold. For example, in MATIS, the ituation decribed in Figure 2 to 4 hold for any goal related to requet pecification. Coverage of a property over tate can be ued a a metric for aeing conitency. The CARE propertie, which include the notion of modality and goal, can alo be intantiated at multiple level of refinement. Modality refinement in term of device and interaction language i dicued in (Nigay, 1994). Typically, goal are recurively decompoed into ubgoal. In our modelling technique, thi decompoition i expreed a a refinement of interaction trajectorie. Depending on the level of refinement, interaction trajectorie can be viewed either a a one tep encapulation or a a equence of tep. For example, in Figure 5 the intended goal i to reach a tate where both the departure and the detination of the trip are pecified. Thi goal may be een a a ingle chunk or a an encapulation of two ubgoal: pecify departure and pecify detination. (a) (b) R E "Flight from Boton to Pittburgh" "Flight from Boton to Pittburgh" pecify departure pecify departure&detination Method = a in Fig.2-4 1 pecify detination Method = a in Fig.2-4 Figure 5: CARE and goal refinement. The uer may ue one modality, aying the entence Flight from Boton to Pittburgh, or typing it into the peech recognition window, or doing both redundantly (cae a). Alternatively, they could pecify the departure uing any method decribed in Figure 2 to 4 and then pecify the detination (cae b). In cae (a), the uer action mut be performed within ome interval and the ytem provide feedback once the temporal window ha elaped. In (b), the ytem provide feedback for each ubgoal. Conidering the decription in Figure 5 at a high level, method (a) and (b) are functionally equivalent. At a finer grain of analyi however, (a) and (b) differ in the interaction trajectory. A uch, they may not be perceived a equivalent. In ummary, the CARE propertie can be ued at multiple level of goal refinement. Deigner can exploit the recurive nature of CARE to reaon about multimodality at the appropriate level. For example, in the early tage of the life cycle, one may reaon at a coare grain to fleh out the mot alient requirement about multimodality. Once the ytem i deigned, one may need to go into more detail to make ound predictive aement. Once the ytem i implemented, effective ue of the ytem can be oberved and interaction trace may be 4

analyzed at fairly low level of detail. In all of thee circumtance, the CARE propertie apply. 4. CARE-LIE PROPERTIES OF THE USER The CARE propertie of the computer ytem have a counterpart in correponding propertie of the uer: the CARE-like propertie. A with the ytem CARE propertie, the uer propertie are concerned with the choice between different modalitie for communicating with the computer. A above, our dicuion will be confined to the choice of modality for communication in the input direction, that i, from the uer to the ytem. Becaue of the uer circumtance including her tak, her background, her training, her knowledge, and the phyical and interactive behaviour of the computer interface the uer may well have preference a to how he communicate with the computer. A familiar example i that if the uer i engaged in a tak which occupie her hand, he may prefer to ue peech. A little more ubtly, uppoe that the uer wihe to book a flight from omewhere in Europe to La Vega. She may not know what i the nearet international airport, o he would prefer to indicate her detination by pointing on a map or at the very leat, by chooing from an appropriately filtered lit of airport. We refer to uch preference by the uer, affecting her choice of input modalitie, a U-preference. Certain pattern of U-preference are worth identifying. (a) If only one modality i acceptable to the uer, or if he ha a trong preference for one particular modality, then we have a cae of U-aignment. (b) If there exit a ubet of the poible modalitie which he prefer to all other, but between which he i indifferent, then we have a cae of U-equivalence. (c) If the uer prefer to employ two or more mean of communication to convey the ame information, then we have a cae of U-redundancy. (d) If the uer preference i to ue one modality for one apect of the tak and another modality for another apect, then we have a cae of U-complementarity. The crucial requirement on the deign of the ytem i that it propertie mut be compatible with the uer U-preference. In the example above, in order to pecify a detination of La Vega, a ytem Aignment to either clicking on a map or providing a filtered lit of airport i compatible with the U-preference, a i a ytem Equivalence between them. But if the pecification of the detination i Aigned to either typing or talking, then the uer i in trouble and the ytem deign i incompatible with her need. The iue of compatibility can be addreed more ytematically by conidering the requirement for each of the CARE propertie to be compatible with each of the U-CARE propertie. We regard a ytem deign a being compatible with the uer need provided there exit at leat one modality which i acceptable to both ytem and uer: (a) U-aignment, where the uer require one particular modality Ua. For ytem Aignment to modality Sa the condition for compatibility i that Sa=Ua, i.e., that the modality aigned by the ytem i the ame a the one the uer need. For ytem Equivalence or Redundancy over a et Se, the condition i that Ua Se, i.e., that the modality the uer need i among thoe treated a acceptable by the ytem. (b) U-equivalence, where the uer i prepared to ue any one of a et of modalitie Ue. For ytem Aignment, the compatibility condition i imilarly that the et include the modality the ytem i Aigned to, Sa Ue. For ytem Equivalence or Redundancy, the condition i that the et Ue hare at leat one member with the ytem Equivalent or Redundant et, Ue Se Ø. In thi latter cae, the uer need to know which modalitie the ytem will find acceptable, i.e., he need to know the et Se, in order to chooe a modality in the interection. Thi quetion of what the uer need to know i dicued below. (c) U-Redundancy, where all modalitie in a et Ur are ued. The compatibility condition are imilar to thoe for U-equivalence, provided that the ytem i prepared to ignore extra input. If the ytem were aigned to a modality Sa, it would preumably be able to ignore input via other modalitie, to which it i not currently paying attention. If thi i the cae, then if Sa Ur, the propertie would be compatible. Thi i often the cae our worktation do not mind if we talk to them along with our typing or mouing. Sytem Equivalence or Redundancy over the et Se will be compatible with Uer redundancy provided that at leat one of the modalitie the uer chooe i in the et, i.e., Ur Se Ø, again provided that the ytem can ignore input in modalitie not in Se, and that in the cae of Equivalence it can ignore input beyond one modality in Se. (d) Complementarity and U-Complementarity (where the uer provide part of the information in one modality, and the remainder in one or more other modalitie) are bet analyed in term of the propertie of the component part of the meage. Thee compatibility condition ummarie the main way in which CARE and uer CARE-like propertie interact in the deign of the computer ytem. They provide a framework which put the computer and the uer on an equal footing, and within which the concrete circumtance of a particular deign can be flehed out to guide the aement of uability. An important concluion i that neither the propertie of 5

the ytem alone, nor thoe of the uer alone, determine uability. Sytem modelling can determine the propertie of the ytem, but to undertand thoe of the uer, and hence uability, we need to turn to uer modelling. One approach i to conider what a uer need to know about a ytem in order to develop any of the U-preference decribed above. Firt of all, they need to know what the poible modalitie are. The viible preence of a keyboard may ugget to a uer that typing i allowed; but will they alo know that they can peak to the ytem? If they remain unaware of the availability of a modality, then any U-preference that include it will not develop. Secondly, once they know that a modality i available, they will need to know what contraint the ytem impoe on it ue. In MATIS, for example, uer can pecify all or part of their requet by peaking. But they cannot ue peech to open, cloe or move a window containing that requet. Finally, if there i a choice between the ue of different modalitie, or combination of modalitie, uer need to know how they hould make that choice. A technique uch a Programmable Uer Modelling (Blandford & Young, 1993) can be ued to ae the likelihood that a uer will be able to acquire the particular knowledge needed to develop the appropriate U-preference. The formal apect of contructing a PUM involve the decription of the ytem in term of thi required knowledge, followed by the production of a runnable cognitive model that imulate the problem olving of the uer. The behaviour of thi model indicate whether or not a uer i likely to obtain and ue the knowledge appropriately. At an earlier tage of deign, even the le formal proce of identifying the required knowledge can help the deigner undertand the uability iue of alternative deign olution. Another approach i to conider the cognitive reource a uer will require in order to ue or chooe between the modalitie included in the variou U-preference et. A general cognitive architecture uch a ICS (Barnard & May, 1993) model the flow of information through different mental repreentation from enation and perception, through comprehenion, to action. It identifie cognitive apect uch a the influence of experience, memory requirement, and the potential for learning. The architecture alo contrain the way that different enory repreentation (i.e., the uer input modalitie) and effector repreentation (i.e., their output modalitie) can be combined. For example, for much of the time the uer might have U-equivalence over poken language, form-filling, or pointing in order to pecify Flight From. But at other time, circumtance might drive her to U-aignment to a pecific modality (e.g., becaue of difficulty in locating the information to point at, or by not being confident how to pell Pittburgh, or how to pronounce Loughborough, and o on). Experience with a modality in ome ituation may lead to U-aignment in other ituation where U-complementarity or U-equivalence would be more appropriate, thu undermining the flexibility of the ytem CARE propertie. 5. CONCLUSION The current definition of the CARE propertie provide a formal framework for reaoning about the deign of multimodal ytem. We have previouly hown how the CARE propertie can be ued a contraint for the implementer (Nigay, 1994). In thi paper, we have introduced the relationhip between the ytem propertie and the uer CARE-like propertie. In empirical tudie, thee propertie could be ued to tructure the experimental proce. Protocol can be devied according to each property and oberved at different level of granularity (e.g., goal and modality refinement). Quetion raied by uer modelling can erve a a ueful lit of concern during deign. ACNOWLEDGEMENT Thi work ha been upported by project ESPRIT BR 7040 Amodeu II. It wa influenced by timulating dicuion with Phil Barnard (MRC- APU), Jame L. Crowley (LIFIA-IMAG), David Duke (York Univerity), our Amodeu partner, and member of PRC-CHM. REFERENCES Barnard, P. J., & May, J. (1993). Cognitive Modelling for Uer Requirement. In P. F. Byerley, P. J. Barnard, & J. May (Ed.), Computer, Communication and Uability: Elevier Blandford, A. and Young, R. M. (1993) Developing Runnable Uer Model: eparating the problem olving technique from the domain knowledge in J. Alty, D. Diaper and S. Guet (ed.) People and Computer VIII, Cambridge Univerity Pre. Interface to Real&Virtual World, EC2 Publ., 97-106 Nigay, L. & Coutaz, J. (1993) A deign pace for multimodal interface: concurrent proceing and data fuion. InterCHI 93 ACM: New York, 172-178 6

application aux interface multimodale, PhD diertation, Grenoble Univerity. 7