Intelligent agents: some ethical issues and dilemmas

Transcription

1 Intelligent agents: some ethical issues and dilemmas Carolyn Dowling Australian Catholic University 115 Victoria Parade Fitzroy, Victoria 3065 Not so long ago, intelligent software agents belonged in the realms of speculation and esoteric research. Today, like it or not, they are an important part of our everyday lives online. Characteristics commonly acknowledged as desirable include a degree of anthropomorphism or personification, the ability to learn from experience and consequently to respond flexibly to new situations, the capacity to interact both with users and with other agents, and a significant measure of autonomy in the pursuit of goals. The widespread implementation of agent based systems raises a number of issues that have clear ethical, even moral dimensions. While the image of an agent may suggest that the technology is under the control of the user, the metaphor is less straightforward than it appears, and the confidence and security it engenders may sometimes be misplaced. Implicit in our understanding of the concept of agency is the notion of delegation. Important ethical aspects of this concept include the balance between autonomy and control, and questions related to trust, responsibility and privacy. While the usefulness of agents in achieving certain outcomes is undeniable, this utility perhaps comes at a certain cost that we may or may not consider acceptable. Through consideration of our experiences of agents both in the physical world and online, this paper explores some of the ethical issues arising from the widespread implementation of intelligent agents within computer based environments 1 Introduction It is not so long ago that intelligent software agents belonged in the realms of speculation and esoteric research. Today, whether or not we are aware of it, they are an important part of our everyday lives online. From the invisible armies of knowbots and related entities scurrying around the Net in the service of increasingly sophisticated search engines to the cheery little characters who pop up on our screens offering assistance with anything from formatting a date to constructing a complex multimedia presentation, or the personalities with whom we interact in chat rooms in happy ignorance of their purely digital nature, intelligent agents are alive and multiplying. 1 Copyright ' 2001, Australian Computer Society, Inc. This paper appeared at the 2nd Australian Institute of Computer Ethics Conference, Canberra, Australia, December 2000, J. Weckert, Ed. Conferences in Research and Practice in Information Technology, Vol 1. Reproduction for academic, not-for profit purposes permitted provided this text is included. An early but still useful conception of an agent is, A character, enacted by the computer, who acts on behalf of the user in a virtual environment, useful in mediating a relationship between the labyrinthine precision of computers and the fuzzy complexity of man (Laurel, p. 355). Typical of recent and more functional descriptions is: An agent can be viewed as an object which has a goal and autonomously solves problems through interaction, such as collaboration, competition, negotiation and so on (Kinoshita & Sugawara 1995). This has obvious similarities with Maes definition of an agent as: A computational system which: is long lived; has goals, sensors and effectors; decides autonomously which actions to take in the current situation to maximize progress towards its (time-varying) goals Maes (1997, slide 5). Other qualities frequently proposed, but not supported by all researchers or indeed by all users, include the ability to learn from experience and consequently to respond in flexible and possibly unforeseen ways to particular situations, and the possession of a believable character, or personality. The widespread implementation of agent based systems raises a number of issues that have clear ethical, even moral dimensions, including the balance between autonomy and control, and questions related to trust, responsibility and privacy. Some of these have been recognised by researchers for a number of years. Friedman and Kahn (1992), for instance, discuss ethical difficulties related to personification and to the granting of excessive autonomy to agents. Despite this, computing professionals as a group still lack a well articulated and broadly accepted understanding of the issues and of their implications. As a result, many developers appear to pay scant attention to these aspects of their systems. 2 Why agents? Why have we been so quick to accept software agents as an essentially unproblematic element of our computing experience? Their usefulness, indeed perhaps their necessity, becomes clear when we consider the degree to which our day to day activities now take place within computing environments characterised by rapid change, large quantities of extraordinarily complex information, and a lack of common organisational structures through which information may be accessed and managed. As Laurel predicted, there are now many situations in which,

2 in the interests of efficiency, some form of intelligent mediation is required between computer systems and the needs of users. Agents are a means of masking the complexity that lies behind our use of computers, thereby facilitating the achievement of our various aims and goals. The notion of agency appears to many to be a natural metaphor. It strikes a comforting note of familiarity with many computer users. For these people, the term most commonly conjures up an image of a docile and amenable servant undertaking particular tasks at the behest and in the interests of the user. As a way of conceptualising our relationship with the technology it has much in common with an earlier metaphor which proved extremely popular, that of the computer as a tool. Both of these images encompass the reasssuring suggestion that the technology is under the control of the user. In each case, however, the metaphor is less straightforward than it appears, and the confidence and security engendered in users may well be misplaced. In many instances the notion of an agent masks not just a functional complexity, but also a range of activities which are not of the users choosing, and not necessarily even clearly in their interest. Both in the real world and online, the concept of an agent is understood and instantiated in a number of different ways. In scientific contexts an agent may be defined as a causative factor (as in a chemical agent ). In other circumstances an agent is one who represents us and our interests when we ourselves are not present. This type of agent, when modelled in some detail upon our personal qualities and preferences, may have a good deal in common with the idea of an avatar a representation of ourselves through which we may interact with others in electronically mediated environments where we are not able to be present in person. Alternatively, we may envisage an agent more in the nature of a trustworthy personal servant, obedient to our orders and committed to our goals. Some of the most interesting types of human agents to consider in relation to the ethical and moral complexities of the roles which software agents might undertake are those which actually represent the interests of more than one party, and must effect a balance between competing claims. Examples of these include estate agents, theatrical or literary agents, and employment agents. Such agents are not the property of individual clients, but function independently as brokers of deals involving various types of resources. This relates to yet another aspect of our conceptions of an agent a so-called free agent being one who possesses autonomy. In certain philosophical discourses agency in fact refers to the degree to which humans are understood as being able to initiate actions rather than simply respond the extent, in other words, to which they possess free will. It is this aspect of our understanding of agency that resonates most obviously with the development of highly autonomous software agents as exemplified by the work of Pattie Maes and her team at MIT. Another type of agent with which we are all familiar but are probably less comfortable, is the secret agent, generally a gatherer of supposedly confidential information operating under an assumed identity. There are clear links to be made here with the activities of both existing and potential software agents! Indeed, in considering the online experiences of many computer users this model is particularly relevant, and certainly raises ethical issues of significance in relation to our awareness of and control over some of the activities of agents. It is ironic that many of the agents with whom users most commonly interact through the Internet are in fact far removed from the popular image of the personal servant, and interaction with them is certainly not initiated through a contractual arrangement entered into voluntarily by the user, but is an invisible component of an apparently straightforward task such as a search for information. Common to all of the understandings of the roles and functions of human agents outlined above is the assumption that their usefulness derives from the possession of specialised skills, which qualify them to mediate between an individual and a particular environment for the more effective achievement of various ends. A further entailment of the agent metaphor is a commonly held understanding that agents can be relied upon to act in accordance with an understood set of principles that ensure that the client or user is not disadvantaged. Assumptions such as these, related to expertise and trust, play a significant part in encouraging the acceptance of programs characterised as agents. 3 The personification issue There are many different forms that the mediation between human beings and their computing environments could have taken. The agent metaphor lends itself particularly well to reinforcement through a personified interface. The strong propensity for most users to readily accept assistance in the form of a more or less personified computer based entity undoubtedly derives at least in part from the anthropomorphic elements implicit in most computer interfaces from the earliest days of computing. It can be strongly argued that a degree of personification has always been automatically and inevitably conferred as much by a program s use of language as a component of the interface as by our everyday understandings of the intelligence, albeit artificial, of computers. Intelligence and language use are, after all, key defining attributes of human beings. As Shirk puts it: Although there is some dispute among software critics concerning the advisability of having personalities in computer programs, their presence seems unavoidable. Any time there is communication between a computer and a human, the information presented by the computer has a certain style, diction, and tone of voice which impact upon the human s attitude and response toward the software (Shirk 1988, p. 320). Not only are we accustomed to interacting with computers as though they share with us a degree of humanity, but in a number of areas of activity we have been persuaded to value social interaction particularly

3 highly. An example is the field of education, where our current understandings of learning depend very much on an acceptance of the role of the social construction of knowledge. It is in this regard that online interactions of a social nature with a significantly personified entity can be perceived as adding value to the activity in which we are engaged. This propensity to value the social also lends acceptability to the interlocking activities of multiple agents when they are metaphorically characterised as a society (Franklin & Graesser 1996; Costa & Perkusich 1997). However, while we might acknowledge the inadvertent attribution of human qualities to computers and their programs, the deliberate cultivation of personified or anthropomorphic interfaces is more problematic, as evidenced by a range of discarded attempts such as the many incarnations of Apple Computer s Phil, or Microsoft s Bob. The issue of the degree to which software agents can and should be personified is extremely important when considering the ethical aspect of agent systems. It is this aspect of agency which propels questions of right and wrong to a level beyond that pertaining to the human use of more traditional types of programs. To the extent that personification is successfully implemented, the expectations of users in relation to the behaviour of the program move into the arena governing the understandings upon which we base our interpersonal interactions, rather than merely those upon which we base our use of technology. For instance, from an ethical point of view one must consider the implications of an agent who successfully passes for a human being, to the extent that users are unaware of its true nature. Already this is a not uncommon occurrence within some interactive online environments where the agent is implemented through dialogue within a limited discourse domain, generally requiring only a short-term relationship with the user and involving little need for a comprehensive memory of previous exchanges. Where this is the case, even if there is no obvious intent to harm, there can be no pretence that the user is in control of the technology. Such a situation is inherently and deliberately misleading, and is a poor basis for the development of the type of trust in agency that arguably should be an important part of a fruitful ongoing relationship between human beings and software agents. Other arguments supporting the efficacy of less robust forms of personification include the effects of a circumscribed level of performance on the user s perception of the program. Apart from the issue of deceit, it is possible that users might be more tolerant of the limitations of an agent if they are conscious of the fact that it is only a computer program. In addition to language use, a significant aspect of the representation of character or personality is visual appearance. Both research and experience suggest that a mismatch between realism in appearance and the apparent knowledge level of the agent as revealed through its use of language and other capabilities can have a deleterious effect on credibility and on acceptance. As Masterton, writes, A common problem with AI programs that interact with humans is that they must present themselves in a way that reflects their ability. Where there is a conflict between the ability of the system and the users perception of that ability a breakdown occurs and users may either fail to exploit its full potential or become frustrated with its shortcomings (Masterton 1998, p. 215). The more visually realistic the representation, the higher the expectations of the user in relation to the appropriateness and intelligence of utterances and actions. Agents that look smart and act or talk dumb are poorly received by many users, who express a higher tolerance for the limitations of a character more sketchily represented, for instance through cartoon-like graphics. Masterton goes on to describe the use of a degree of anthropomorphism rather than full personification, intended to convey qualities such as friendliness and usefulness, without the implication of possession of full human capabilities (Masterton 1998, p. 211). A further instance of this principle is the characterisation of agents as animals, as suggested early on by theorists such as Laurel (1990). Our expectations in regard to the cognitive skills of animals may well be more appropriate to the capabilities of software agents than are our experiences of human-to-human interactions. Well known examples in the field of education include the learning companions of Chan (1996, 1998) pedagogical agents in the guise of animal characters who fulfil various roles within electronically mediated learning environments for young students. From a further perspective, the issue of the optimal degree of personification and anthropomorphism needs to be considered in terms of the particular function being fulfilled by the agent and its status relative to the user. While in many instances we may feel more comfortable giving instructions to a character that is not fully human, we may not feel the same way about receiving advice or even correction from such an entity. Users of Microsoft Office will be all to familiar with the indignity of being subjected to the whims of an animated paperclip. 4 Autonomy, trust and responsibility The delegation of any task to a software agent raises questions in relation to its autonomy of action and decision, the degree of trust which can be vested in the outcomes it achieves, and the location of responsibility, both moral and legal, for those outcomes. Autonomy in this context is generally understood as meaning that the agent has the ability to make certain decisions in the course of undertaking an activity, without the need for constant monitoring and intervention on the part of the user. Implicit in the recognition of the need for a capacity to exercise initiative is an acknowledgment that some outcomes of agent activity may not be easily predictable by the user. In some cases they may even be contrary to what the user might perceive as his or her interests and wishes. Loeffler (1996), for instance, notes that the unpredictability resulting from significant autonomy might well result in agents who are less helpful to us than we might hope or indeed expect. As Minsky puts it, There s the old paradox of having a very smart slave. If you keep the

4 slave from learning too much, you are limiting its usefulness. But, if you help it to become smarter than you are, then you may not be able to trust it not to make better plans for itself than it does for you (Minsky, 1994, p. 25). It is easy to slip from such considerations into the need for a contemporary version of Asimov s laws of robotics as conceived in fictional terms more than 30 years ago. A further issue emanating from consideration of agent autonomy is that of responsibility for outcomes and actions resulting from decisions which are out of the user s control, and which may indeed relate to capacities of the program of which the user had no knowledge at all. It is inevitable that such instances will occur, precipitating the need for renewed examination both of community and legal understandings of liability. Despite these areas of potential difficulty the need for some degree of autonomy in the interests of usefulness is accepted by most researchers. As Maes (1994), for instance, writes in relation to the implementation of agent based systems, The currently dominant interaction metaphor of direct manipulation [as promulgated most notably by Schneiderman (1983)] requires the user to initiate all tasks explicitly and to monitor all events. This metaphor will have to change if untrained users are to make effective use of the computers and networks of tomorrow (Maes, 1994). As previously noted, the issue of trust is closely associated with our ambivalence towards agent autonomy. This is particularly important in view of the mission critical nature of many of our computing activities. The reverse side of the coin where trust is concerned is risk. Without an element of risk, there is no need for trust. How then might we come to trust an agent? Words which come to mind are those which might well be applied to our relationships with human agents. A reputation for credibility in most fields is contingent on a verifiable history demonstrating qualities such as accuracy, reliability, efficiency and so on. Where such assurances are available in relation to the software agents to whom we delegate responsibility, perhaps we can be justified in taking the risk of trusting them. As with some forms of delegation to other human beings, however, we may have to accept that there are no guarantees. Militating against our entrusting our affairs to agents according to such criteria is the fact that, implicit in the whole notion of agency, is the masking of complexity referred to earlier in this paper. While to an extent we can judge the competency, even the good intentions of an agent through the outcomes of its actions, if we have had no hand in its programming we have little insight into its capabilities and limitations, even into what we might describe as its motivations. For many purposes, a more accurate agent model than that suggested by a personified interface is desirable. This suggests that a range of types of interfaces of differing degrees of transparency might be appreciated, particularly by users who are entrusting the agent with undertakings of major significance. Allied to this is the capacity for users to monitor the activities of agents at a lower level within the system. Even if we are prepared to trust the agents with whom we deal directly, a further question remains as to how agents might reasonably decide how to trust one another. In a number of fields, agents already have the capacity to interact with other agents emanating from different systems. As this develops further, a number of other ethical issues, including the important one of personal privacy, come into play, creating substantial challenges for software developers. 5 Conclusion It is clear that the development of agent based systems has the potential to raise a range of ethical and moral issues pertaining to the delegation of tasks to intelligent computer programs, in particular those vested with a significant degree of autonomy. This paper has concentrated on those related to our understanding of the agent metaphor as a way of characterising such programs, exploring both the benefits and some of the potential problems with its implementation. While the usefulness of agents in achieving certain outcomes is undeniable, this utility perhaps comes at a certain cost that we may or may not consider acceptable. More experience of this type of software and a great deal more discussion between and among users and developers is needed in order to reach a common understanding and strike the most efficacious compromise, particularly in relation to the ethical issues involved. 6 References CHAN, T.-W. (1996): Learning companion systems, social learning systems, and the global social learning club. Journal of AI in Education 7 (2). CHAN, T.-W. (1998): The past, present, and future of e d u c a t i o n a l a g e n t s. COSTA, E.deB. and PERKUSICH, A. (1997): Designing a Multi-Agent Interactive Learning Environment. Proceedings of ICCE 97, International Conference on Computers in Education, Kuching, Sarawak, Malaysia, 2 6 December 1997, Charlottesville VA: AACE. FRANKLIN, S. and GRAESSER, A. (1996): Is it an Agent, or just a Program?: A Taxonomy for Autonomous Agents. Proceedings of the Third International Workshop on Agent Theories, Architectures and Languages, Berlin: Springer-Verlag. FRIEDMAN, B. and KAHN, P. (1992): Human Agency and responsible computer: Implications for computer system design. Journal of Systems Software, 17. KINOSHITA, T. and SUGAWARA, K. (1995): Agent oriented computing. Tokyo: Soft Research Center. LAUREL, B. (1990): Interface agents: metaphors with character. In The Art of Human-Computer Interface Design, LAUREL, B. (ed.), Reading, Massachusetts, Addison-Wesley. LOEFFLER, C.E. (1996): Artificial Life of Agents. Advanced IT Tools. Proceedings of the IFIP World

5 Conference on IT Tools, Canberra, Australia, 2 6 September 1996, Chapman & Hall. MAES, P. (1994): Agents that Reduce Work and Information Overload. MAES, P. (1997): Software Agents Tutorial. MASTERTON, S. (1998): Computer support for learners using intelligent educational agents: the way forward. Proceedings of ICCE 98, the Sixth International Conference on Computers in Education, Beijing, China, October 1998, Beijing, China Higher Education Press and Springer-Verlag. MINSKY, M. (1994): A conversation with Marvin Minsky about Agents. Communications of the ACM, July 1994, vol. 37, no. 7. SCHNEIDERMAN, B. (1983): Direct Manipulation: A Step Beyond Programming Languages. IEEE Computer, vol 16, no. 8. SHIRK, H.N. (1988): Technical writers as computer scientists: the challenges of online documentation. In Text, Context and Hypertext: writing with and for the computer, BARRETT, E. (ed.), Cambridge, Massachusetts, MIT Press.