Quality Evaluation of Bochum Restaurant Information System: A Spoken Dialogue System

Transcription

1 ISCA Archive Quality Evaluation of Bochum Restaurant Information System: A Spoken Dialogue System Christine S. Pellegrini Institute of Communication Acoustics Ruhr-University Bochum, Germany Tina_Pellegrini@yahoo.de Abstract BoRIS, the Bochumer Restaurant-Information-System, is a spoken dialogue system that returns information about restaurants via telephone. This paper describes an auditory test in which different components of the dialogue system can be changed. Configurations are arranged with different features of the system s components. All configurations are assessed by subjects. Further, technical parameters are collected, e.g. the dialogue s length, or expert-based parameters, such as transaction success. Correlations can be found between the features of the system s components, the technical data, and the assessment by the subjects. The results of this test will be discussed. It is conceivable that these results can be used to improve similar interactive dialogue systems.. Introduction A preliminary restaurant information system was implemented at the IDIAP (Dalle Molle Institute for Perceptual Artificial Intelligence) in Martigny. A German version was realized by Sebastian Möller at the Institute for Communication Acoustics, Ruhr-University Bochum. In the context of a diploma thesis [] the German version was modified for an experiment as described in this paper. The purpose of this test was to analyze the influence of different components of a spoken dialogue system on the quality of the system. The dialogue system consisted of a speech recognizer, a dialogue manager, a database, and a speech generator. The speech recognition accuracy could be changed and the system s utterances were made either by a human voice or by synthesized speech. Additionally, different confirmation strategies were assessed.. BoRIS - Das Bochumer Restaurant- Informations-System.. Implementation BoRIS is an information system for restaurants in Bochum. It is implemented with the speech toolkit from the Center for Spoken Language Understanding (CSLU []). The toolkit is based on a state-machine. The links between the states are shown in figure. In these states the system s possibilities are implemented. This may be the command to open a window, or to play a file with an utterance, or to decide by keywords how to react that means to step to a certain state... Application At the beginning of the program the system welcomes the user with a human voice. After a introduction to the user it expects an utterance. The computer analyses the information from the speech recogniser and decides how to proceed: BoRIS looks for restaurants, which are classified by the food, the opening hours (day and time), the price of a meal and the location. After each utterance BoRIS gives the specification to the database. If there are more than three fitting restaurants it asks for further specification. If there are up to three restaurants, BoRIS announces their names and addresses. If it doesn t find any restaurant it asks the user to modify the specification and again enumerates the five criteria to specify a restaurant. The system is able to answer help requests. After all restaurants are announced the system says goodbye and can be restarted... Components Several components are responsible for a successful dialogue. In this paragraph the setup of the dialogue system BoRIS is presented. It s different components will be explained.... Speech recogniser and speech understanding unit To understand an utterance the system needs a speech recogniser. Errors of the speech recogniser have a big influence on the progress of the conversation. The vocabulary of the recogniser consists of the week days, including expressions like today or tomorrow, time data (morning evening, now, eight o clock etc.), districts of Bochum and places like the main station or the university, pricing information like cheap, expensive, not too much or 0 Mark and different types of food (Greek, Italian, vegetarian etc.). There are also expressions with a certain meaning, which control the system s reaction like yes (e.g. yes, why not, it s okay), no (e.g. no, don t), unimportant (e.g. unimportant, no preference, don t know), help (e.g. which, other).

2 These expressions are collected in a keyword list of approximately 50 words and expressions, which provides their meaning to the system. To test different speech recognition accuracies a person in another room types the utterances without the subject s awareness. It is easy to change the accuracy: To reach a rate of 75%, every four words are typed wrongly, so the system can not find them in the keyword list, but it is possible to reconstruct the dialogue situation afterwards by reading the log file. Four different accuracies are tested: 67%, 75%, 8% and 00%. information: ( Do you want to go to a cheap Chinese restaurant Friday evening at the main station?) Collective confirmation II: It s the same confirmation like collective confirmation I, but it is made when BoRIS doesn t find a restaurant an asked the user for a change. This confirmation makes sense because often the user is unsure what information the system still has in mind since the beginning of the dialogue. When a subject s utterance is typed into the correspondent window, the string has to end with the return-key. Then the system goes on with the dialogue. If such an utterance window is open, it is possible for the typing person to control the next Figure : The main window of the toolkit shows the state-machine of the system.... Dialogue manager and dialogue strategies One topic of this experiment is to test different dialogue strategies. The subjects shall confirm given information. BoRIS can ask for confirmation in four different dialogue situations: Explicit confirmation: After each utterance the system asks if it understood the information properly. ( You want to go to a restaurant on Friday? ) Implicit confirmation: The given information is part of the next question. ( Where do you want to go to a restaurant on Friday? ) Collective confirmation I: After collecting all information to the five criteria and before starting a request to the database, BoRIS asks to confirm all part of the dialogue. This is done when the confirmation has to be made. In another window parts of sentences are clicked on and speech files are played. However, the information in the subject s utterances has to be typed in the utterance window, or at least the keywords. At the end of the confirmation part the system gets the necessary information an takes over the control of the dialogue, when the typing person presses the return-key. Some information missing in the log file, such as the number of user turns etc., are written down and added to the log file after the experiment.... Database The database consists of the names of approximately 70 restaurants in Bochum, their address, additional location information, the food type, the price, and the opening hours. The database doesn t change during all experiments. 5

3 ... Speech generator All system utterances existes as two sets of wave files. One set has been spoken by a woman; another set has been generated by the speech synthesis of the Institute of Communication Acoustics []. Human or synthesized speech can be chosen separately for the system s utterances and the announcement of the restaurants... Log file The system automatically generates a log file that consists of the system s utterances, the strings typed into the utterance windows, the recognised words in the keyword list and the criteria for the request to the database.. The Experiment subjects participated in the experiment. All of them received the same tasks with the same set of six scenarios. Every call was made with a different configuration of the system... Scenarios The six scenarios were arranged in a way that at the beginning BoRIS functionality was not transparent. No vocabulary was offered which the system could find in the keyword list. In the next four scenarios the tasks were ly specified, including information for possible modification. For the last scenario the subjects had to invent their own request and write it down before calling... Configurations different configurations were tested by subjects (table ). They were assigned in a way that every subject had six different configurations, and they were equally distributed concerning the different scenarios. Table : System configurations Speech recognition accuracy Confirmation System s utterances Announcement of restaurants A 00% Without Human speech Human speech B 00% Without Synth. speech Human speech C 00% Explicit Human speech Human speech D 00% Implicit Human speech Human speech E 00% Collective I Human speech Human speech F 8% Without Human speech Human speech G 75% Without Human speech Human speech H 67% Without Human speech Human speech I 00% Without Human speech Synth. speech J 00% Without Synth. speech Synth. speech K 00% Collective II Human speech Human speech.. Questionnaire At the beginning of the test the subjects were asked to fill out a form relating to their expectation concerning dialogue systems. After every dialogue they had to rate on continuous rating scales [] how well they felt understood by the system, how well the subjects knew what the system wanted them to do, how well they could understand the system acoustically. They also had to fill out a semantic differential, based on Möller [5] and Hempel [6], with antonyms like un,, -,, -,, -. Finally, the overall impression should be shown on the basis of the same criteria, and the subjects were asked for suggestions for improvement... Log file With the log file and the handwritten completions the dialogue s length, the number of help requests, number of system and user turns and correction turns, the contextual appropriateness, and the transaction success were determined. The definition of the transaction success as described by Simpson and Fraser [7] was modified to fit in this context [].. Results.. Speech recognition accuracy All results are shown in figure. The 00% accuracy is rated best except for the antonyms. The recognition accuracy doesn t have an influence on this aspect, as the rating is almost the same for all recognition accuracies. The rating for the question if people felt understood by the system is very eye-catching. The 67% accuracy is rated much worse than 75% or 8%, and of course than 00%. Unexpectedly the 75% accuracy is rated better than the 8% accuracy. It is supposable that both accuracies are similarly problematic for the subjects. The questions if the by the system and if the dialogue was rather or show the largest differences in the results. The subjects expressed that they understood the system quite well independent from the accuracy. 6

4 5.5.5 un 00% 8% 75% 67% Figure : Assessment of the speech-recognition accuracy.. Confirmation The worst rating is given for the collective confirmation I, as shown in figure. It is astonishing that the strategy without any confirmation is often rated best; the implicit confirmation seems to be an alternative. The explicit confirmation makes the system ; the subjects didn t know what the system wanted them to do. The collective confirmation II is rated very much better than the collective confirmation I, sometimes also better than any other strategy... Speech These results are shown in figure. The dialogues with the human voice are rated best. The system using only human speech is best to understand. In the second-best configuration the synthesis makes only the announcements of the restaurants. The rating of the inverse configuration with the synthesis speaking the dialogue and the woman announcing the restaurants is rated very poor for the question how well the system can be understood acoustically un utterances: human; restaurants: human utterances: synth.; restaurants: human utterances: human; restaurants: synth. utterances: synth.; restaurants: synth. Figure : Assessment of natural versus synthesized speech un without explicit implicit collective I collective II Figure : Assessment of the different confirmation strategies 5. Discussion Two aspects must be taken into consideration while interpreting the results: The test situation has an influence on the rating of the subjects. They do not react as spontaneously and emotionally as they would during a usual call. They rather want to do their task successfully. They do not want to go to a restaurant and are not very motivated to understand the announcements. The typing of the utterances is an inevitable source of error. While operating the confirmation window the dialogue s process has to be the same as if the system reacted itself, e.g. if there are help requests. In some situations decisions hat to be made quickly and were not always correct. Dialogues with operating errors are consequently excluded from the evaluation, but the progress of those dialogues probably might have an influence on the assessment of the following. 5.. Speech recognition accuracy As expected, the best recognition accuracy is rated best. Already the 8% accuracy is rated worse, but the overall quality 75% accuracy seems to be comparable. 7

5 5.. Confirmation There is no best confirmation strategy; but with the implicit confirmation the subjects felt very. Dialogues with the collective confirmation seem to be very. The reason might be that all five criteria are asked and have to be confirmed before the system can control the dialogue again to search the restaurants. Often there are no restaurants found, because the inputs are too specific. Those dialogues continue with modification and were indeed much er than others. Ratings for the collective confirmation II may lead to wrong conclusions, because many dialogues have been over before the confirmation was made. This configuration is surely rated that well for other reasons. [6] Hempel, T., Untersuchungen zur Korrelation auditiver und instrumenteller Messergebnisse für die Bewertung von Fahrzeuginnenraumgeräuschen als Grundlage eines Beitrags zur Klassifikation von Hörereignissen, Herbert Utz Verlag, München, 00. [7] Simpson, A. und Fraser, N.A., Black Box and Glass Box Evaluation of the SUNDIAL System, in Proceedings of the Third European Conference on Speech Communication and Technology, EUROSPEECH 99, Berlin, pages - 6, Speech The human voice can be understood best, a synthetic voice is rated bad. The results show that the ratings for the combination with the woman speaking the system s utterances and the synthetic speech announcing the restaurants much better than the inverse configuration. A reason for this might be, that the subjects have been asked to write down only the name of the restaurant. They shall not get unsure because they are not able to write down that much information and then rate the successful dialogues worse. They probably would rate the configuration with the synthetic voice announcing the restaurants much worse, if they have to write down all information including the address of the restaurants. If possible, all utterances should be made by a human speaker. The only disadvantage for restaurant information systems is that for any changes in the database new recordings have to be made with the same speaker. 6. Outlook In future work some aspects should be analysed more exactly. The different recognizing accuracies and confirmation strategies could be implemented in the toolkit to avoid mistakes. Experiments should be done with more subjects to get significant results. It would be desirable to describe the experiments in a model to estimate the dependence of the components on the quality assessment of the subjects. 7. References [] Dudda, C. S., Evaluierung eines natürlichsprachlichen Dialogsystems für Restaurantauskünfte, diploma thesis, Institute for Communication Acoustics, Ruhr-University Bochum, 00. [] Center for Spoken Language Understanding, Oregon Graduate Institute of Science and Technology. [] Arnd Böhm, Maschinelle Sprachausgabe deutschen & engl. Textes, Verlag Shaker Aachen, 99. [] Markus Bodden and Ute Jekosch. Entwicklung und Durchführung von Tests mit Versuchspersonen zur Verifizierung von Modellen zur Berechnung der Sprachübertragungsqualität, Final Report, Institute for Communication Acoustics, Ruhr-University Bochum, 996. [5] Möller, S., Assessment and Prediction of Speech Quality in Telecommunications, Kluwer Academic Publishers, Boston,