Usability and Effectiveness Evaluation of a Course-Advising Chat Bot Hyekyung Kim Informatics Program, University at Buffalo, State University of New York hk67@buffalo.edu Miguel E Ruiz Department of Library and Information Studies, University at Buffalo, State University of New York, 534 Baldy Hall, Buffalo, New York 14260 meruiz@buffalo.edu Lorna Peterson Department of Library and Information Studies, University at Buffalo, State University of New York, 537 Baldy Hall, Buffalo, New York 14260 lpeterso@buffalo.edu This research compares the usability and efficiency of a course-advising chat bot with menu driven FAQs (frequently asked questions). Based on a survey and user interviews, a text-based FAQ system was created and compared with a chat bot that was developed to address library and information science (LIS) graduate student course and program related questions. The students conducted tasks with both the chat bot and FAQ systems. The usability and effectiveness of the functionality and user-interface of both systems is assessed. Introduction A chat bot is an interactive computer program that enables users to converse with a computer like a real person through natural language. Natural language processing (NLP), one of several areas of artificial intelligence, is used to construct computer interfaces programmed to understand natural human languages (Artificial Intelligence, 2003). A natural language interface allows the user to type in his or her own words to form his or her inquiry. Natural language interaction is defined as the operation of computers that people use as a familiar
natural language to give instructions and receive responses. The main advantage of using this type of interfaces is that users do not need to learn any artificial language because the user can use the natural language which they already know (Shneiderman, 1998). Among chat bots, A.L.I.C.E. (Artificial Linguistic Internet Computer Entity) is one of the strongest chatterbots. A.L.I.C.E. encourages users to converse by applying some heuristic pattern matching rules to human s input. A.L.I.C.E. uses a programming language AIML (Artificial Intelligence Markup Language) specified in its program (Wallace, n.d.). AIML is an XML-compliant language for programming chat bot ( AIML ). For example, Stella which was built based on novomindiq software is the library chat bot for Hamburg State and University Library, Germany. (fn: http://www.novomind.com/index_ht_en.html?press/2006/rel_124_en.html ) She helps students use the digital library and find information in a position of all web pages. Stella also eases the workload of librarians by providing research advice, informing people of open hours, and assisting borrowers in extending their lending periods (novomind, 2006). Pandorabots, which has been used for this research, is a software robot hosting service based on AIML. Pandorabots was selected because it provides the services to create and publish one s own chat bot via the web. Chat bots created by Pandorabots have controlled answering and voice response capabilities with an avatar. To launch a well-developed chat bot on the web, we also used the SitePal software, providing commercial services to create animated characters and artificial intelligence interface. Chat bots have remarkable features regarding information search methods and feedback as well as interactive user interface with the function stimulating human conversation. First, users can search information not only by keywords but by using natural language with complete sentences. Second, through typing questions on a form interface, users can actively retrieve information that they want. Since web users of all levels are familiar and comfortable with form, form interfaces have the advantage of being intuitive and easy to use (Meng, 1999). In addition, chat bots can increase understanding of information by providing text and voice responses with an avatar. Since chat bots offer convenience in the method and process of information search, this research will explore two aspects of a course-advising chat bot so that it could be used as an academic tool to address course-related questions. In terms of functionality, is a course-advising chat bot more effective than a text-based FAQ to address course-related questions? Does the interactive user interface of the course-advising chat bot positively affect the process of searching and understanding information? Usability Test
This research conducts a comparative usability test to explore that a course-advising chat bot effectively contributes to address course-related questions in terms of its functionality and interface. Materials For the test, two systems have been created: One is a text-based FAQ interface and the other is a LIS course-advising chat bot. The text-based FAQs interface was built based on a preferred FAQs interface selected from a short web-survey and user interviews. 40 students participated in the FAQ interface web-survey. 46% of participants preferred hierarchically organized menu among four types of FAQ websites. They also preferred the FAQs interface providing answers close to questions and less clicking. We conducted interviews with the LIS students and analyzed the current contents of our FAQ to gather the content required to create both systems. LISAbot, which is the LIS course-advising chat bot, was created by using 'Pandorabots'. The LISAbot was technically controlled for supporting LIS course-related questions by adding potential questions and answers in the AIML database. By using the SitePal software, we built an avatar and an artificial intelligent interface and published the LISAbot on the web. Participants All participants for the FAQs interface web-survey, the FAQs interface interview and usability test have been recruited among the LIS graduate students enrolled at the University at Buffalo, State University of New York in the fall semester 2006. Forty LIS graduate students participated in the web-survey. Six LIS students participated in the interview. Sixteen participants for the usability test were voluntarily recruited. Procedure A set of eight questions were designed to assess performance, the information finding process, and interaction behaviors of users on two systems. The presentation order of the systems and questions are systematically controlled through a randomized block experimental design. The test consists of two sets of task sessions with a ten minute break in between. During each task session (20 min.), participants use a system in turn to answer a set of four assigned questions. Before beginning the first task test session, participants filled out a short questionnaire to
collect information regarding Internet usage and familiarity with chat programs and avatars, information seeking attitudes, and a preference for animated characters. After each task test session, they answered a system usage questionnaire for collecting data for usefulness, effectiveness, satisfaction, and the perception of the time to complete tasks in each system. After completing two task test sessions, they answered a post-test questionnaire about the preference of the user interface and comparative functionality between both systems. We collected data for the observation of the information finding process, performed tasks, logs files, and user feedback. Results and Analysis Results are analyzed for all data collected in task test. The t-test method is used for data analysis obtained from three kinds of questionnaires to compare two systems regarding usefulness, effectiveness, learnability, satisfaction, and the perception of the time to complete tasks. Conclusion Although this is a research in progress our initial results from the user interviews indicate that the LISAbot could be a potentially successful interface in terms of easy accessibility of FAQs and information search method. However, users have numerous open questions and various expressions for questions on LISAbot. Our future work includes a formal evaluation of the effectiveness of this interface in helping course-related questions. We plan to complete user tests by the time we present the poster and include the data gathered at that point. Reference AIML (n.d). Retrieve June 30, 2006, from http://www.alicebot.org/aiml.html Artificial Intelligence (2003). Retrieved July 29, 2006, from the AccessScience database Meng, F. (1999). A natural language interface for information retrieval from forms on the World Wide Web. Paper presented at Proceeding of the 20th international conference on Information Systems. Retrieved August 3. from http://portal.acm.org novomind (2006). Hamburg University library wins government-backed IT award. novidmind Press, November 21st, 2006. Retrieved March 15, 2007, from http://www.novomind.com/index_ht_en.html?press/2006/rel_124_en.html Shneiderman, B. (1998). Designing the User Interface (3rd ed.). Wesley Longman, Inc. Harlow, England: Addison
Wallace, R. S. (n.d.). The Anatomy of A.L.I.C.E. A.L.I.C.E. AI Foundation Retrieve July 29, 2006, from http://www.alicebot.org/anatomy.html