Conversational Agents in Virtual Worlds: Immersion and the

Conversational Agents in Virtual Worlds: Immersion and the Conversational Record. Bob Heller and Mike Procter Abstract Conversational agents (CAs) are computer programs designed to chat with users in natural language using the rules of conversation and turn-taking. In our research, we have been investigating historical figure applications (i.e. Freudbot), their role in virtual worlds and in particular, whether conversational logs can reveal evidence of social presence that occurs as a consequence of immersion in a virtual world. In this paper we adopted a linguistic approach to analyze the conversational logs from participants who chatted with Freudbot in a virtual world (i.e. Second Life) as opposed to participants who chatted with Freudbot in a text-only condition. We hypothesized that the virtual office space and a Freudbot avatar would give rise to greater sense of social presence in comparison to a non-virtual context. The results supported the hypothesis as conversation logs from Second Life had significantly higher proportions of lexical categories arguably associated with social presence (Personal Pronouns, Biological, & Perceptual) and lower proportions of lexical categories less likely to be associated with social presence (Cognition). These findings build on previous work conducted on the conversational logs and provide directions for future work. Conversational Agents & Virtual Worlds Conversational agents (CAs), sometimes known as chatbots, are computer programs designed to chat with users using the rules of conversation and turn-taking (see http://www.chatbots.org/ for examples). As noted by Kerly, Ellis, and Bull (2009), the significant growth of CAs in education is a result of maturing conversational technologies and the wide variety of educational applications available (e.g. tutoring, question-answering, second language learning, learning companions, reflective dialogues.) The value of CAs is often linked to the 1

Persona effect, a hypothesis that an animated and anthropomorphized intelligent agent would lead to better learning outcomes than non-animated and non-anthropomorphized agent delivery. Although evidence for a persona effect may be equivocal (see Clark & Choi, 2005; Dehn & van Mulken, 2000; Gulz, 2004; Gulz & Haake, 2006 for reviews of animated pedagogical agents, a closely related technology) the rationale for the use of CAs is sound. Conversational agents provide an engaging and intuitive interface to a body of knowledge that can be accessed in a personalized and adaptable format (Cassell, Sullivan, Prevost, & Churchill, 2000). Virtual worlds can be defined as persistent 3-dimensional computer generated spaces that support users as avatars with navigation and communication abilities. The value of VWs has been linked to a sense of presence or immersion (Witmer and Singer, 1998) in which greater immersion leads to better outcomes. Together, virtual worlds and CAs provide an extraordinary opportunity to create engaging simulations and game based learning opportunities (Heller & Procter, 2009). Rickel (2001) argues that virtual worlds should be enriched with intelligent CAs that support face-toface interactions in a variety of roles. Similarly, one of the leading developers of Artificial Intelligence (AI) in virtual worlds notes that non-player characters (characters controlled by a computer) were the norm in the progenitors of virtual worlds (i.e., Multi-User Dungeons (MUDs) and Massively Multiplayer Online Role-Playing Games (MMORPGs)) and that their deployment in virtual worlds like Second Life offers a unique opportunity to evaluate embodied AI (Burden, 2009). In our studies with CAs we have focused on historical figure applications (Heller, Procter, Mah, Jewell, & Cheung, 2005; Payr, 2003; Veletsianos & Miller, 2008). Our first historical agent was Freudbot (Heller et al., 2005) based on Sigmund Freud and created using 2

Artificial Intelligence Markup Language (AIML), an XML-based programming language developed by Richard Wallace (see http://www.alicebot.org) and one of the most widely used programming platforms for chatbot development. The website Pandorabots (http://www.pandorabots.com), is a hosting service for AIML based chatbots with over 1.3 billion responses served. Typically, users converse with AIML chatbots by typing in their comment or question in a text window and the agent responds immediately after the user submits their input. At its core, AIML is a pattern matching program and thus similar to Eliza, the grandmother of all chatbots (Weizebaum, 1966). However, unlike Eliza, AIML enables a recursion property that reduces user input into different symbolic categories thus enabling a chatbot response appropriate to the conversational context. Freudbot s knowledge is stored in the form of first person narratives. The importance of narratives as a vehicle for knowledge transmission has been recognized in intelligent tutoring systems (McQuiggan, Robinson, & Lester, 2010) and is well suited as a medium for CAs as historical figures. Freud s narratives are presented in chunks provided that the user types explicit ( Tell me more ) or implicit ( That s interesting ) conversational prompts to advance the story. In some cases Freudbot can answer questions that are anticipated from the narrative output. In cases where the user input was not recognized Freudbot would default to a conversationally appropriate request in order to direct the user to a relevant conversational narrative. In total, Freudbot, contains almost 90 stories related to his theories and his biography. In a proof of concept study (Heller et al., 2005), Freudbot was evaluated by 53 students after a ten-minute conversation mediated by a standard web page interface. Questionnaire data on Freudbot performance indicated that the potential of the application was widely recognized, but there was some criticism on the actual performance of the agent. When asked if they would 3

chat again with Freudbot, over two-thirds of the participants indicated yes and, not surprisingly, the ratings of Freudbot provided by this subsample was significantly higher than those who would not chat again. In the words of one participant: It was pretty cool the way it felt like I was actually interacting with Freud... he's deceased though, yeah, but the picture, the fast answers... made me pay attention to the answers a lot more than if I had been simply reading a text written by someone else. Plus it was cool to feel like I could voice my own opinion with the most well-known psychoanalyst of all time. Further research on Freudbot lead us to investigate the visual aspects of the task including animated CAs (REF) and virtual worlds like Second Life. As others had noted (Gulz and Hake), the visual look and feel of CAs, and by extension, the virtual environment, can be critically influential on agent performance and effectiveness. Freudbot was first deployed in Second Life in late 2007 in a recreation of a period office on Athabasca University Island. Freudbot sat on a chair by a couch and rose to greet people when they walked into his office. He would then invite them to sit down on his couch and chat with him if interested. During the first four years of his residency on Athabasca University Island, Freudbot chatted with 351 users or approximately 87 users per year. In March, 2009, Freudbot was also added to an educational build in Second Life known as The Theorists Project developed by Kelton and Renfro-Michel. The project was dedicated to a handful of well-known counselling theorists, including Freud, and Renfro-Michel used the build with her counselling students who created content in Second Life in the offices of each theorist. Like the Freudbot on Athabasca University Island, Freudbot on The Theorists Project would rise to greet visitors and invite them to chat with him by sitting on his couch. However, in contrast to the Athabasca University site, Freudbot on the Theorists Project chatted with approximately 634 4

users over the first 32 months or approximately 238 per year with an average duration of 5.7 minutes and an average of 14 exchanges. The higher traffic is likely a reflection of the course design as well as the presence of The Iceberg, a separate build dedicated to the theories of Sigmund Freud. More importantly, the higher traffic provided a pool of conversational records that could be compared to conversational records collected under different visual conditions in order to isolate the unique effects of virtual world immersion. Immersion and the Conversational Record The research question of interest is whether immersion in a virtual world and chatting with a CA has measurable effects on the conversational record as opposed to chatting with a CA in the absence of a virtual world. We hypothesized that the immersive experience of being an avatar in the virtual office space conversing with a Freudbot avatar would give rise to a greater sense of social presence and engagement and reflected in the conversational record in comparison to a non-immersive conversation with Freudbot. Social presence has been defined as the degree of salience (i.e. quality or state of being there) between two communicators over some communicative medium, the quality of which varies such that some media enable more social presence (e.g., video) than others (e.g., texting) (Short, Williams, & Christie, 1976). The role of social presence in virtual worlds began with investigations of computer-mediated communication in the 90s (Lowental, 2010). According to Lowenthal (2010), social presence is now a central concept in online learning, a key component in a number of theoretical frameworks, and generally associated with greater student satisfaction. However, social presence comes from a family of presence constructs and measurement issues are predominantly questionnaire-based. 5

In our investigations, we have been examining the conversational record for evidence of immersion and greater social presence. In previous work with conversational records (Heller & Procter, in press), we established 2 conditions involving conversations with Freudbot, one in Second Life and one with text-only. We initially focused on the quality and quantity of user compliance with Freudbot as proxies for social presence and user engagement. We hypothesized that the immersive world of Second Life would be more engaging and, as a result, engender greater conversational compliance to Freudbot s request in comparison to the text-only condition. Compliance with Freudbot can also be thought of as persistence in the role-play exercise. As long as students are following Freudbot s questions and directions, they are more fully participating in the role-play exercise as engaged learners. We found that there was a non-significant trend to greater compliance in Second Life, compared to the text-only condition, except when Freudbot suggested a topic for discussion. We also found that initial compliance to Freudbot s greeting was focused more on Freud s theory in the text-only condition compared to the Second Life condition. Finally, a 3- point rating developed to evaluate the degree of social presence reflected in each user s exchange revealed no differences between conditions although levels were high in both (text-only: 85%; Second Life: 90%). The Current Study In the present study, we adopted more of a linguistic approach to investigate the differences in social presence between conditions following on the work of Kramer, Oh, & Fusell (2006). Kramer et al. (2006) analyzed CMC based on linguistic features and found that certain task independent linguistic features were reliably correlated with self-rated sense of 6

presence. Following Kramer et al., we used the Linguistic Inquiry and Word Count program (Pennebaker, Booth, and Francis, 2001) to further analyze the conversational logs in Heller and Procter (in press) in order to examine selected lexical categories that might be affected differently under the two conditions; text-only Freudbot vs. Second Life Freudbot. In the textonly condition, there were 30 participants who chatted with Freudbot for exactly 10 minutes using a text window and in the absence of any visual information. In the Second Life condition, there were 39 participants who chatted with Freudbot for approximately 10 minutes on The Theorists Project in Second Life. Further details on the selection of logs can be found in Heller and Procter (in press) but with the selection criteria in place, the average number of exchanges per protocol in the Second Life sample was 31.1 compared to 29.7 from the text only sample, a difference that was not statistically different, t(62) =.38, p >.05. In order to prepare the logs for LIWC, the text chat was edited by the primary author to correct spelling mistakes, add punctuation, and convert nonlinguistic characters as recommended. Based on Kramer et al. (2006), we selected the following linguistic features as proxies for social presence: personal pronouns, positive and negative emotions, cognitive terms, social terms, and tense. Kramer et al (2006) found relations with social presence for all features with the exception of emotions. In addition to these categories, we also examined biology, perceptual, and relativity as lexical features that could be affected by higher levels of social presence. Table 1 provides a list of lexical categories we examined, the mean for each condition, and the results of a t test to compare conditions. There were no significant differences in the total word count between the Second Life condition (M= 91.2) and the text-only conditions (M=114.5), t(66) = -1.43, p >.05. 7

Table 1. Summary of LIWC differences for Freudbot conversations under two conditions. LIWC Category Second Life Text-only t-value Df p-value Personal Pronoun 13.7% 11.2%.98 65.05* Positive Affect 7.4% 7.5% -.02 66.98 Negative Affect 2.0% 1.4%.87 66.39 Social 13.8% 12.5%.84 66.41 Cognition 13.4% 16.4% -2.5 66.01** Perception 2.1%.8% 1.86 66.07* Biology 3.07% 1.7% 2.06 55.04** Relativity 7.2% 7.1% 0.02 66.99 Past 2.1% 3.4% 0.02 66.05** Present 13.6% 11.2% 0.02 66.06* ** Significant difference.01>p<05; * Significant difference.05>p<.10 8

There was a significant difference in the use of personal pronouns as participants in second Life used more personal pronouns than did those in the text only condition. Although the pattern of personal pronoun usage differed somewhat from Kramer, the greater use of personal pronouns in general is consistent with a social presence effect. There were no differences between conditions in either positive affect or negative affect, a finding consistent with Kramer et al. (2006) and is somewhat surprising given the role of affect in defining social presence in CMC (Garrison, Anderson, & Archer, 2000). We will return to this observation in the discussion. There was no significant difference in the social category although there was a higher percentage in the Second Life condition as would be expected. Kramer et al. found an unexplained significant negative correlation between social category and presence. There was a significant difference in the cognitive category as participants in Second Life used less cognitive terms than did those in the text only condition, a finding that Kramer et al. (2006) also reported and one predicted by social presence. The three new categories examined were expected to show greater usage in Second Life because of the connection to social presence. There was a significant difference in the biological and perceptual categories as participants in Second Life used more of these terms than did those in the text-only condition. There was no differences between conditions in use of relative categories. Finally, the use of past tense was significantly greater and present tense significantly less in the text-only condition compared to the Second Life condition. Although Kramer s et al s (2006) did not examine tense, they did report differences in remote and local deixis (i.e. how objects are referenced) which seems related to present tense. 9

Discussion The findings from the current study provide support for the hypothesis that an immersive virtual environment leads to more social presence than a text-only environment when chatting with a CA based on a historical figure. This conclusion is based on the pattern of pronoun usage, and the distribution of word categories that one would anticipate to occur more or less frequently under conditions of high and low social presence. Those in an immersive space had conversations that expressed a higher use of personal pronouns, perceptual, and biological terms accompanied with a lower use of cognitive terms. Moreover, those in an immersive space used more of a present tense voice and less of a past tense voice compared to the text-only condition. The overall pattern across all categories is remarkably consistent with a social presence explanation. However, we were surprised that there were no differences in affective terms (and to a lesser extent, relativity terms), a finding that was consistent with Kramer et al. (2006). One explanation is that the use of affective terms reflects a persona effect that operates equally in both conversations, an effect which may be primarily responsible for activating affective conversations. Distinguishing between persona effects and immersion effects is beyond the scope of the present design but it is an important empirical question. The present findings build on past work using conversational records to investigate social presence effects. Previous study identified broad differences in the topics of discussion and a pattern of compliance that was suggestive of social presence. In the current study, the linguistic analysis added greater precision over our previous measure of social presence which had identified high levels of social presence but was too blunt to reveal differences between conditions. As noted by Annand (2011) there are very few direct or objective measures of social 10

presence, a general weakness of the social presence construct. The use of linguistic indices may provide a complementary and reliable approach to the assessment of social presence (see Kramer et al. 2006) and maybe equally usueful to evaluate CAs in virtual worlds. Our next steps are to examine some correlates of the LIWC categories in relation to demographics and self- rated experiences. There are some limitations to the present study. The foundational feature of social presence, the concept of immediacy (Mehrabian, 1969), is poorly implemented in Second Life since immediacy is usually operationalized by paralinguistic communication channels (i.e., eye gaze, gestures, facial expressions). However, as virtual spaces increase, there is a growing family of presence constructs that require further methodological and theoretical consensus. The participants under each condition participated under different conditions with different expectations. They came from different schools in different countries and likely differed in their year of study. The data were also collected at different points in time. These confounded variables make it difficult to attribute the differences in the conversational logs solely to the different conditions of conversation. Finally, there was no control of the camera view that participants had when they chatted with Freudbot in Second Life. Given how important visual data could be to the establishment of social presence it would be helpful to confirm the actual views experienced. However, we are confident that the majority experienced the conversation in the default view that occurs once seated on the couch. Acknowledgements 11

The authors acknowledge the contribution of Edino Renfro-Michel and AJ Kelton of Montclair State University and all the participants in The Theorists Project. The authors also acknowledge a grant from the Mission Critical Research Fund of Athabasca University. Dedication The first author would like to dedicate this paper to his father, Joseph. 12

References Annand, D. (2011). Social presence within the community of inquiry framework. International Review of Research in Open and Distance Education, 12, 40 56. Burden, D. (2009). Deploying embodied AI into virtual worlds. Knowledge-Based Systems, 22, 540 544. Cassell, J., Bickmore, T., Campbell, L., Vilhjálmsson, H., & Yan, H. (2000). Human conversation as a system framework: Designing embodied conversational agents. In J. Cassell, J. Sullivan, S. Prevost, & E. Churchhill (Eds.), Embodied conversational agents (pp. 29 63). Cambridge, MA: MIT Press. Cassell, J., Sullivan, J., Prevost, S., & Churchhill, E. (2000). Embodied conversational agents. Cambridge, MA: MIT Press. Clark, R. E., & Choi, S. (2005). Five design principles for experiments on the effects of animated pedagogical agents. Journal of Educational Computing Research, 32(3), 209 225. Danforth, D., Procter, M., Heller, R., Chen, R., & Johnson, M. (2009). Development of virtual patient simulations for medical education. Journal of Virtual Worlds Research, 2(3), 4 11. Dehn, D. M., & van Mulken, S. (2000). The impact of animated interface agents: A review of the empirical literature. International Journal of Human-Computer Studies, 52, 1 22. Garrison, D. R., Anderson, T., & Archer, W. (2000). Critical inquiry in a text-based environment: Computer conferencing in higher education. The Internet and Higher Education, 2(2 3), 87 105. 13

Gulz, A., (2004). Benefits of virtual characters in computer based learning environments: Claims and evidence. International Journal of Artificial Intelligence in Education, 14, 313 334. Gulz, A., & Haake, M. (2006). Design of animated pedagogical agents: A look at their look. International Journal of Human-Computer Studies, 64(4), 322 339. Heller, R. B. & Procter, M. (2009). Animated pedagogical agents: The effect of visual information on a historical figure application. Journal of Web-based Learning and Teaching Technologies, 4(1), 54 65. Heller, R. B., Procter, M., Mah, D., Jewell, L., & Cheung, B. (2005). Freudbot: An investigation of chatbot technology in distance education. In P. Kommers & G. Richards (Eds.), Proceedings of the World Conference on Multimedia, Hypermedia and Telecommunications, 2005 (pp. 3913 3918). Chesapeake, VA: AACE. Kerly, A., Ellis, R., & Bull, S. (2009). Conversational agents in e-learning. In T. Allen, R. Ellis, & M. Petridis (Eds.) Applications and Innovations in Intelligent Systems XVI: Proceedings of AI-2008, the Twenty-eighth SGAI International Conference on Innovative Techniques and Applications of Artificial Intelligence (Part 5, pp. 169 182). London, England: Springer. Lester, J. C., Converse, S. A., Kahler, S. E., Barlow, S. T., Stone, B. A., & Bhogal, R. S. (1997). The persona effect: Affective impact of animated pedagogical agents. In S. Pemberton (Ed.), Human Factors in Computing Systems: CHI'97 Conference Proceedings (pp. 359 366). New York, NY: ACM Press. 14

Lowenthal, P. R. (2010). The evolution and influence of social presence theory on online learning. In S. Dasgupta (Ed.), Social computing: Concepts, methodologies, tools, and applications (pp. 113 128). Hershey, PA: IGI Global. Mehrabian, A. (1969). Some referents and measures of non-verbal behaviour. Behaviour Research Methods and Instrumentation, 1(6), 205 207. McQuiggan, S. W., Robinson, J. L., & Lester, J. C. (2010). Affective transitions in narrativecentered learning environments. Journal of Educational Technology & Society, 13(1), 40 53. Mori, M. (1970). Bukimi no tami [The Uncanny Valley]. Energy, 7(4), 33 35. Payr, S. (2003). The virtual university s faculty: An overview of educational agents. Applied Artificial Intelligence, 17, 1 19. Rickel, J. (2001). Intelligent virtual agents for education and training: Opportunities and challenges. In A. de Antonio, R. Aylett, & D. Ballin (Eds.): Intelligent Virtual Agents: Third International Workshop, IVA, Madrid, Spain, September 10 11, 2001, Proceedings, (LNAI 2190) (pp. 15 22). Berlin, Germany: Springer. Short, J., Williams, E., & Christie, B. (1976). The social psychology of telecommunications. Toronto, ON: Wiley. Veletsianos, G., & Miller, C. (2008). Conversing with pedagogical agents: A phenomenological exploration of interacting with digital entities. British Journal of Educational Technology, 39(6), 969 986. Weizenbaum, J. (1966). ELIZA A computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36 45. 15