A semiotic engineering approach to user interface design

Knowledge-Based Systems 14 2001) 461±465 www.elsevier.com/locate/knosys A semiotic engineering approach to user interface design Clarisse Sieckenius de Souza*, Simone Diniz Junqueira Barbosa, Raquel Oliveira Prates Departamento de InformaÂtica, Semiotic Engineering Research Group, PUC-Rio, Brazil Abstract Designing software involves good perception, good reasoning, and a talent to express oneself effectively through programming and interactive languages. Semiotic theories can help HCI designers increase their power to perceive, reason and communicate. By presenting some of the results we have reached with semiotic engineering over the last few years, we suggest that the main contributions of semiotic theory in supporting HCI design are: to provide designers with new perceptions on the process and product of HCI design; to bind together all the stages of software development and use, giving them a unique homogeneous treatment; and to pose innovative questions that extend the frontiers of HCI investigations. q 2001 Elsevier Science B.V. All rights reserved. Keywords: Semiotic engineering; Interdisciplinary studies; HCI design 1. Introduction Design tasks have been challenging the value of theories in all domains where physics, logic and mathematics cannot account for the basic concepts one needs to know in order to synthesize high-quality products. Although impressive theoretical scaffolding has been used to analyze a great variety of human factor dimensions in artifacts, one question still calls for a de nitive answer. How can this or that theory help me make the right choices in design? When it comes to software design, especially in the case of interactive systems, the twofold nature of the product Ð a computer program that by necessity incorporates both highly precise mathematical principles and highly subjective representational and communicative choices Ð expectations and claims about the role of theories tend to get confusing. Some try to evade confusion by arbitrating that the good theories are the ones that allow you to generate true theorems, for instance. When applied to systems design, good theories should allow you to state truths of the kind ªa computer program is correct if p and qº. This group expects non-mathematical theories to do something analogous, and generate truths of the following kind: ªa computer program is usable or plainly good, or successful) if p 1 and q 2 º. This should be the rst step to building tools to monitor or verify that qualities expressed by p 1 and q 2 are incorporated in a product, as a result of a good design process. * Corresponding author. E-mail addresses: clarisse@inf.puc-rio.br C.S. de Souza), sim@inf.- puc-rio.br S.D.J. Barbosa), raquel@inf.puc-rio.br R.O. Prates). Others take a different approach and say that good theories are the ones that allow you to understand processes and products better. They should make you aware of pitfalls and false qualitative assessments. This group expects theories to bring up new insights, and tell that ªa computer program is this; it will present features p if you have p 1 because of p 2 ; if you wish it to exhibit qualities such as q, you should know that q 1 and q 2 º. This approach places less value on veri cation tools and more on decision-support information. It assumes that the quality of products is directly proportional to how much knowledge the designer has. In a recent article [2], Philip Armour has insightfully characterized some aspects of the two extreme stereotypes we have just illustrated. He calls our attention to the fact that software is not a product, but a media, where our knowledge is encoded. If so, we can expect information-oriented theories to play a more productive role than veri cationoriented ones. So far, mainstream theoretical approaches to HCI design have typically drawn on cognitive perspectives. They have had the power to explain what users do when they interact with computers, and to predict a lot of what they will or will not do depending on the quality of the system's interface. The user-centered approach to system design [14] has shed light on the fact that although user-centeredness might at rst glance look like a de ning quality of any system which is built because of and for its users), the cognitive activity that dominates HCI requires support that algorithms and data structures alone fall short of providing. As a consequence, much of what has been done in HCI lately aims at discovering, understanding, and supporting users in their 0950-7051/01/$ - see front matter q 2001 Elsevier Science B.V. All rights reserved. PII: S0950-7051 01)00136-8

462 C.S. de Souza et al. / Knowledge-Based Systems 14 2001) 461±465 attempt to make sense and make use of software applications. A multitude of target features have been found, and handy compilations of them are offered in the form of guidelines and heuristics for HCI design. In a recent publication, Hollan and his co-authors [10] have pointed out that new theoretical foundations for HCI are required in view of the new challenges for communication and interaction posed by current networked environments. In their view, Distributed Cognition is a good theoretical alternative because it ªseeks to understand the organization of cognitive systems but, unlike traditional theories, it extends the reach of what is considered cognitive beyond the individual to encompass interactions between people and with resources and materials in the environment.º This is what ethnomethodology, anthropology and sociology have brought to HCI, helping us understand the contexts where users work, the role of social structures, the impact of cultural values, and the patterns of collective and individual behavior affecting organizational systems and technological change policies. Theoretical contributions from these disciplines have provided software designers with powerful insights and valuable knowledge that helped increase the value of corporate systems and of computer-supported collaborative work environments. Nevertheless, here again, what designers got from colleagues in these other elds of knowledge were more insights and new targets. They served to analyze the quality of existing systems, and/or to tell designers what they should aim at, but little if any) design recipes were produced. But, are there any design recipes? What do we really gain as we apply design patterns, for example? Software development is a costly process, and the price to pay for bad design is too high. There are talented designers who seem to be able to get it right without knowing anything about any theory. These are artists who excel in their craft, and inspire us to watch them and learn. But they are very few, and very expensive. So, we should be able to study these artists in the light of some theory, extract from their performance an organized body of knowledge, and present it in a consistent and coherent theoretical discourse. This should eventually be transformed into a set of techniques that can be taught to people, leading them to be better professionals. It is our belief that semiotic theories are particularly well suited to this endeavor, in that they can produce an account of the process of HCI designs, using models of human interpretation and expression. Moreover, they can also be used to analyze the product of HCI design, as it is used by people in a variety of situated contexts. This homogeneous semiotic framework used for talking about process and product allows us to contrast design intentions that emerge in the process) with perceived meanings that are derived from the product), at the very end of the line Ð the user interface. Therefore, like most of our colleagues have done in this volume, we single out semiotics as a promising new player on the stage where cognitive and social theories contribute to provide insights about HCI, and encourage scientists and technologists to devise methods and tools to help designers achieve better quality in software design. In order to illustrate our particular way of applying semiotics to HCI design, we will brie y sketch our own semiotic engineering approach [6]. The word engineering, just as in Norman's Cognitive Engineering [15], should draw attention to the synthesizing nature of the task we are trying to support an important deviation from the mostly analytical approaches found in traditional semiotic studies). Even though the kinds of theories we draw upon to achieve design HCI are quite different from those we use to design bridges and power plants, they all have a most important thing in common. They are generated, interpreted, validated and applied by people. This means that, to a greater or lesser extent, they are all subject to biases instilled by the effect of expectation on observation [16] and the interplay of subjective judgment on determining objective goals [11]. Unlike bridges and power plants, however, software is, in spite of all mathematics and logic that warrant its computational status, a most subjective and arbitrary artifact. The strict rules that it must abide to are only those relative to formal symbol processing and computability. But the relations by which symbols gain meaning to programmers and users alike are in the minds of the beholders, outside the reach of algorithms and data structures. Designing software is thus close to writing about what one perceives as being the case. It is a matter of good perception, good reasoning, and a talent to express oneself effectively. Semiotic theories can then help HCI designers increase their power to perceive, reason and communicate, and semiotic engineering helps us draw a different map of this territory. 2. A semiotic engineering approach The de ning trait of our semiotic engineering approach is that software artifacts are meta-communication artifacts. They are one-shot messages sent from designers to users about the range of messages users can exchange with systems in order to achieve certain effects. This view is radically different from the typical user-centered design model of HCI, inspired by Norman's metaphor of the two gulfs a user must bridge to interact with systems: the execution and the evaluation gulf. It explicitly characterizes HCI as a two-tiered communicative process involving designerto-user communication and user-system interaction. In our view, effective HCI can only be achieved if both levels of communication are successful in their particular way. In fact, since the designer-to-user message tells the user what range of messages she or he can send and receive during interaction, and what goals and effects can be associated to them, the success of interaction is actually dependent on the

C.S. de Souza et al. / Knowledge-Based Systems 14 2001) 461±465 463 Fig. 1. The meta-communication process of semiotic engineering. success of the one-shot message. This view gives us a handle to deal with many issues related to apparently elusive qualities of interfaces such as intuitiveness, usability, and consistency, to name a few. If the user gets the designer's top-level message about the kind of messages the system is prepared to interpret how the user should phrase them, how the user can expect the system to react, and so on), then the interface may feel intuitive, usable and consistent to the user. In other words, these qualities in our view result from a deliberate and successful communication of arbitrary choices. They are not inherent to one or another choice of interface signs widgets, images, words, layout, or dialog structure). The role of a semiotic theory in this perspective is then to tell designers how they should a) make the communicative choices, so that the interactive system is good, and b) convey their choice to users so that this communication does not get in the way of interaction, but serves as a secure scaffold for productive user activity. Compared to most of mainstream HCI design approaches, we see that a) is what they are after; although in our view b) is what really increases the chances of achieving success in HCI. In Fig. 1 we can see how the elements borrowed from Roman Jakobson's characterization of a communicative process [5] contribute to raise an HCI designer's awareness of the processes involved in semiotic engineering. The sender, or designer, is communicating her message about system/user communication, its rules, purposes and possibilities) to the user using the computer as medium [1] and a whole range of visual, tactile, and aural codes. The second distinguishing trait of our approach is that the designer is present at the interface, through what we call the designer's deputy [7,18]. This abstract gure is in fact the portion of software, which carries explanations; help information, con rmation dialogs, error messages and warnings about why and how some speci c user command or expression needs further consideration. It should convey ef ciently direct indications of design choices and principles underlying the application, and thus achieve the metalevel communication of contents that are implicit in the user±system interaction level. The role of a semiotic theory is now to indicate if meanings implicit in the user±system interaction level i.e. in the way users command systems to perform actions and systems react to users expectations) are consistent with and reinforced or explained by meanings that are explicitly conveyed by the designer's deputy discourse i.e. by readonly messages present in help modules, error messages, con rmation dialogs, and the like). For this, a theory must also structure the space of analysis, providing us with useful classi cations of signs and messages, so that correct predictions can be made about the communicative effects of choosing one or another means and mode of expression at one or another level of communication via the interface [12]. The double articulation of HCI in semiotic engineering gives us some leverage to deal with different types of interactive systems. For example, multiuser applications such as groupware present to HCI designers the challenge of designing three different systems of communication: a) the user± system interaction; b) the user±user interaction; and c) the designer-to-user communication [8,17,19]. In other words, rst and foremost, the designers of such systems must have a conception of the domain of application, of the group structure s) they want to support, of the mechanisms of coordination and communication needed for productive collaboration, and of the direct patterns of system activation and system response that best t their design. This overall view should be conveyed directly and indirectly at all levels of communication and by all instances of interfaces, a user is likely to encounter. Achieving consistency and nding the adequate means of reinforcing correct interpretations and discouraging incorrect ones is a major design task a semiotic theory should be able to support. Another example of how semiotic engineering can instruct the design of special interactive systems is that of extensible applications. The need to accommodate creative activity with nite symbol processing mechanisms has motivated the development of software that users can customize to their current contexts by programming new functions. Macro recording and scripting are the most popular means to allow users to program applications, although parameter con guration, depending on the granularity of parameters, can also achieve this effect [13]. Semiotic engineering provides a characterization of end user programming as an essentially linguistic design activity, where users become designers of extensions to existing applications [7,9]. The role of designers is to communicate not only the messages we have already described for standard interactive applications, but also the messages relative to the syntax and semantics of another language, a metalanguage, with which users can specify new conversations and achievements for the original application. A special means by which designers can achieve this effect is by designing systems that can interpret metaphors and metonymies with which users refer to new features they want to add to applications [3,4]. They tell the system how new things are metaphorically or metonymically related to old things, and thus the system learns how to react to new input. The role of a semiotic theory, in this case, is to illustrate how

464 C.S. de Souza et al. / Knowledge-Based Systems 14 2001) 461±465 language can work as its own meta-language, supporting expression of novel contents in terms of existing signs that are combined in speci c rhetorical constructs the symbolic processor is prepared to interpret. Our semiotic engineering approach can also support the software development cycle itself, with an evaluation method that can be applied at different stages of development. It carries different messages to designers, depending on whether the subjects performing the steps of the method are designers, users, or experts in semiotics. The communicability evaluation method [18] fully embraces our view that interactive systems are meta-communication artifacts. It serves to tell designers, in a number of ways, how well their message is getting across. A small set of prototypical utterances one might expect users to exclaim as they encounter breakdowns and dif culties with an application's interface is tagged to certain patterns of interaction one observes in logged sessions. Given the tasks, users were trying to perform and the tagging of utterances to breakdown situations, we can delineate a communicative pro le of the interactive system. As designers examine this pro le, they can ponder about needs and opportunities to improve on their message to users, or on the patterns of conversation users can have with the system. Semiotic theory, in the latter case, can provide designers with a wealth of indications about how they are being interpreted by users and why. Also, drawing on communication techniques well known to theorists of human communication, semiotics can present designers with alternative means and modes of expression, in view of the effects they want to achieve. Experimental data may even be used to build case bases for supporting other design tasks and helping identify patterns that can eventually be part of computer-aided user interface design environments. 3. Conclusions The conclusions we have reached in our own pursuit of how semiotic approaches can contribute to user interface design can be simply stated. First, we believe that the role of semiotic theory in supporting HCI design is not one of discriminating true from false or correct from incorrect, predominantly, but rather that of providing designers with new perceptions on the process and product of HCI designs. What can be gained by such perceptions is exempli ed by our semiotic engineering approach and the implications of a two-fold communicative structuring of our HCI model. Causes and consequences of design choices, compared to mainstream user-centered models, are stated in a substantially different perspective, which engenders different problem formulations and, by the same token, different solution spaces. Second, we do not believe that semiotic theory can, by itself alone, account for all the results and insights brought forth to HCI by cognitive and social theories. It cannot. However, because it proposes to account for human interpretive and expressive processes underlying all activities that involve representation and communication, semiotics can bind together all the stages of software development and use. There is a uniform theoretical framework that can be applied from requirements elicitation to nal product evaluation. This supports the traceability of design and implementation decisions in terms of a coherent body of knowledge, a feature other theoretical approaches have much more dif culty to support. Nevertheless, turning potential contributions into actual value requires critical mass and specialization. Critical mass can only be achieved through coordinated research and education programs, geared speci cally towards the investigation and dissemination of new brands of Applied Semiotics, such as Computer Semiotics [1]. Specialization, in its turn, requires that HCI researchers and professional practitioners embrace semiotic approaches and begin to produce knowledge, techniques and tools that consistently incorporate semiotic principles and, when put to use, observably lend distinct qualities to software products so achieved. References [1] P.B. Andersen, A Theory of Computer Semiotics, Cambridge University Press, Cambridge, 1990. [2] P.G. Armour, The Business of Software: the case for a new business model, Commun. ACM 43 8) 2000) 19±22. [3] S.D.J. Barbosa, C.S. de Souza, Extending software through metaphors and metonymies, Knowledge-based Systems 14 1±2) 2001) pp. 15± 27. [4] S.D.J. Barbosa, ProgramacËaÄo via Interface, Doctoral Thesis, Departamento de InformaÂtica, PontifõÂcia Universidade CatoÂlica do Rio de Janeiro, RJ, Brasil, May 1999. [5] Jacobson, R. 1960) Linguistics and Poetics. in T.A. Sebeok ed.) Style in Language. New York. John Wiley and Sons. pp. 350±377. [6] C.S. de Souza, The semiotic engineering of user interface languages, Int. J. Man±Machine Studies 39 1993) 753±773. [7] C.S. de Souza, Supporting end-user programming with explanatory discourse, Proceedings of ISAS'97 Ð Intelligent Systems and Semiotics'97 Ð A Learning Perspective, 22±25 September, NIST, Gaithersburg, USA, 1997 pp. 461±466. [8] C.K.V. Cunha, C.S. de Souza, V.S.T.D.B. Quental, D. Schwabe, A model for extensible web-based information-intensive task-oriented systems, in: M. de Sharon, Y. Waern, G. Cockton Eds.), People and Computers XIV Ð Usability or Else!, HCI2000 Ð 14th Annual Conference of the British HCI Group, The British Computing Society, University of Sunderland UK), 5±8 SeptemberSpringer, London, 2000. [9] C.S. de Souza, S.D.J. Barbosa, S.R.P. da Silva, Semiotic engineering principles for evaluating end-user programming environments, Interacting with Computers 13 4) 2001) pp. 467±495. [10] J. Hollan, E. Hutchins, D. Kirsh, Distributed cognition: toward a new foundations for human-computer interaction research, ACM Trans. Computer±Human Interact. 7 2) 2000) 174±196. [11] I. Kant, Critique of Pure Reason, St. Martin Press, New York, 1965. [12] J.C. Leite. Modelos e Formalismos para a Engenharia SemioÂtica de Interfaces de UsuaÂrio, Doctoral Thesis, Departamento de InformaÂtica, PontifõÂcia Universidade CatoÂlica do Rio de Janeiro, RJ, Brasil, October 1998.

C.S. de Souza et al. / Knowledge-Based Systems 14 2001) 461±465 465 [13] B. Nardi, A Small Matter of Programming, Mass. MIT Press, Cambridge, 1993. [14] D.A. Norman, S.W. Draper, User Centered System Design, Lawrence Erlbaum Associates, Hillsdale, NJ, 1986. [15] D.A. Norman, Cognitive Engineering, in: D.A. Norman, S.W. Draper Eds.), User Centered System Design, Lawrence Erlbaum Associates, Hillsdale, NJ, 1986, pp. 31±61. [16] K. Popper, Objective Knowledge, Clarendon Press, London, 1972. [17] R.O. Prates, C.S. de Souza, Towards a semiotic environment for supporting the development of multi-user interfaces, Proceedings of CRIWG'98 Ð IV International Workshop On Groupware, ArmacËaÄo de BuÂzios, Brazil, 1998 pp. 53±67. [18] R.O. Prates, C.S. de Souza, S.D.J.B. Barbosa, A method for evaluating the communicability of user interfaces, ACM Interact. Jan-Feb 2000) 31±38. [19] R.O. Prates. A Engenharia SemioÂtica de Linguagens de Interfaces Multi-UsuaÂrio, Doctoral Thesis, Departamento de InformaÂtica, PontifõÂcia Universidade CatoÂlica do Rio de Janeiro, RJ, Brasil, October 1998.