TOOLS FOR RESEARCH AND EDUCATION IN SPEECH SCIENCE

Transcription

1 TOOLS FOR RESEARCH AND EDUCATION IN SPEECH SCIENCE Ronald A. Cole Center for Spoken Language Understanding, Univ. of Colorado, Boulder ABSTRACT The Center for Spoken Language Understanding (CSLU) provides free language resources to researchers and educators in all areas of speech and hearing science. These resources are of great potential value to speech scientists for analyzing speech, for diagnosing and treating speech and language problems, for researching and evaluating language technologies, and for training students in the theory and practice of speech science. This article describes language resources from CSLU, and some of the ways in which these resources can be used. 1. ACCESSIBLE LANGUAGE RESOURCES In 1991, the Center for Spoken Language Understanding received a grant from the National Science Foundation to develop the OGI speech tools free software for analyzing, displaying and transcribing speech [1]. Since that time, CSLU has been developing and distributing free language resources to interested users in educational and other not-for-profit institutions. The original OGI speech tools have been transformed into the CSLU Toolkit, a comprehensive set of tools and technologies supporting research, development and education in speech and language technologies. In addition to software tools, CSLU has developed a large number of speech corpora to support basic research and development of language technologies and systems. These language resources software tools and corpora have been distributed to over 2000 sites in 65 countries, and have supported research reported in over 300 publications. In this article I describe these resources and how they might benefit educators, practitioners and researchers in speech and language science. 2. SPEECH CORPORA In areas of speech and language technology, including speech recognition, speaker recognition, language identification and message understanding, progress has been measured for the past fifteen years in terms of performance on shared speech corpora. In national and international competitions in each of these areas sponsored by DARPA and other agencies, corpora have been used to train recognition systems on designated training data, and then to evaluate and compare systems on new (and unseen) test data using standardized scoring techniques. By increasing task difficulty (e.g., from transcription of stories read from the Wall Street Journal using a 5000-word vocabulary to transcription of unknown news broadcasts with unlimited vocabulary), the field has been able to demonstrate progress in a scientific manner. While the needs and goals of speech scientists may differ from those of speech technologists, publicly available language resources are just as important to progress in speech science. Speech corpora enable researchers to study the acoustic and linguistic properties of speech, perform various analyses, and propose and test theories. When other researchers are able to share the same data and software, results can be replicated, extended or challenged. Moreover, because just about everyone today has access to powerful yet affordable computers (under $3000) that can store and process large amount of data, speech from thousands of speakers can now be collected, annotated, analyzed and shared. Speech scientists can now study statistical patterns and trends as well as the behaviors of individual speakers. CSLU provides 17 annotated speech corpora with over 1000 hours of speech from children and adults in 22 languages [2]. These corpora have been designed to support research on acoustic phonetic, lexical, syntactic and semantic manifestations of speech and to reveal how these representations change over time within and across speakers in different acoustic environments. The range of acoustic environments includes recording studios, classrooms, laboratories, cars and public places using high quality microphones, telephones and cellular phones. Callers have produced a wide range of utterances, from isolated words and phrases to one-minute intervals of extemporaneous speech. They have called a single time or multiple times over a period of years. The CSLU Web site describes all available corpora, including recording conditions, speaker population and the data collection protocol [2]. Recent corpora include Kids Speech, with utterances from 1100 children grades K-10; Speaker Recognition, with 12 calls each from 90 speakers over a two year period; and National Cellular, with 1500 calls on cellular phones from across the U.S. 3. THE CSLU TOOLKIT Since 1993, the Center for Spoken Language Understanding has focused on developing, acquiring and incorporating spokenlanguage technology into a portable, comprehensive and easy-touse software environment. The result of these efforts is the CSLU Toolkit, an affordable, accessible, portable, comprehensive and east-to-use set of tools and technologies for learning about, researching and developing interactive language systems and their underlying technologies [3, 4, 5].

2 Development of the CSLU Toolkit is motivated by our belief that the best way to accelerate progress in language technology is by involving as many people as possible by making interactive language systems and technologies accessible, affordable and easy to use. This is precisely the role that the CSLU Toolkit fulfills. We have developed learning materials, core technologies, infrastructure, and software tools that are truly universal in that they benefit a range of users, from novice to expert, and are applicable to a wide range of tasks. The toolkit provides a modular, open architecture supporting distributed, cross-platform, client/server-based networking. It includes interfaces for standard telephony and audio devices, and software interfaces for speech recognition, natural language understanding, text-to-speech synthesis, speech reading (video) and animation components. This flexible environment makes it possible to easily integrate new components and to develop scalable, portable speech-related applications. In the remainder of this section we describe the main component of the CSLU Toolkit. RAD seamlessly integrates the core technologies of facial animation, speech recognition and understanding and speech synthesis with other useful features such as word-spotting, barge-in, dialogue repair, telephone and microphone interfaces, and open-microphone capability. RAD s graphical authoring environment enables users to design interactive dialogues by specifying prompts, recognition vocabularies and actions. Prompts can be either recorded or typed in as text, in which case they are produced as speech using the toolkit s text-to-speech system. Both recorded and synthesized prompts are produced automatically by Baldi, the animated talking face [6]. The words or phrases to be recognized at any dialogue state are simply typed in by the system builder. RAD automatically creates a pronunciation model for each word. (When running an application, the recognizer spots these words and phrases in the speaker s utterance and assigns them confidence scores. These scores are used to reject extraneous speech or poorly recognized words and engage the user in conversational repair.) Arbitrary actions can be associated with recognized utterances, such as producing a new prompt, displaying an image or retrieving and displaying information from a Web site. RAD contains many useful objects for retrieving, organizing and presenting information. In addition, users can develop new objects using the Tcl/Tk programming language. By connecting RAD objects, dialogues of arbitrary complexity can be designed. Based on teachers and students requests, a large number of objects have been incorporated into RAD to make design and evaluation of learning and language training activities easier and more powerful. For example, it is easy to incorporate images into applications, highlight parts of images, associate audio files with parts of images (e.g., the sound of a waterfall or monkeys playing), have Baldi ask questions about these objects, and have the user respond by naming or clicking on the object. Figure 1. RAD Application with Baldi Rapid Application Developer (RAD). RAD is the toolkit s graphical authoring environment. This software makes it possible for people with little or no knowledge of speech technology to learn to develop speech interfaces and applications. RAD s drag and drop interface is easy to use and learn to use naïve users can build simple dialogues in minutes for conversing with an animated talking face. Experienced users can develop and deploy sophisticated interactive media systems for a variety of useful tasks in a few hours. Facial animation. Baldi is an animated three dimensional talking head developed at the in the Perceptual Science Laboratory at UC Santa Cruz. Baldi s synthesis program controls a wireframe model, with a control strategy for coarticulation, controls for paralinguistic information and affect in the face, text-to-speech synthesis, and synchronization of auditory and visual speech [6]. Most of the current parameters move vertices (and the polygons formed from these vertices) on the face by geometric functions such as rotation (e.g. jaw rotation) or translation of the vertices in one or more dimensions (e.g., lower and upper lip height, mouth widening). Other parameters work by interpolating between two different face subareas; e.g., interpolation is used to determine parameters for cheek, neck, and forehead shape. Some affect parameters such as smiling also use interpolation. Basic emotions of surprise, happiness, anger, sadness, disgust, and fear can be communicated through facial expressions. Baldi produces accurate visual speech that can be understood by skilled speech readers. Recently, a more complex and accurate tongue (consistent with electropalatography and

3 imaging data) a hard palate, and three-dimensional teeth have been added. The face can be made transparent during speech revealing the movements of the teeth and tongue, and the orientation of the face can also be changed while speaking so it can be viewed from different perspectives. These features offer unique capabilities for language instruction features that cannot be easily controlled in real faces. Speech Recognition. The toolkit includes working speech recognition systems, and tools for researching, developing and testing new speech recognition systems. It comes complete with general purpose speaker- and vocabulary-independent speech recognition engines for both word spotting and continuous speech recognition applications [7, 8]. In addition, several vocabulary-specific recognizers (e.g., alpha-digits) are included. Mexican Spanish recognizers are also included for vocabularyindependent word spotting applications and continuous digit recognition [9]. The toolkit supports research and education for several approaches to computer speech recognition including artificial neural network (ANN) classifiers, hidden Markov models (HMM) and segmental systems. The toolkit also includes stepby-step tutorials for training and testing new ANN and HMM recognizers. Natural Language Understanding. PROFER (Predictive RObust Finite-state parser,) is a semantic parser modeled after Carnegie Mellon University s Phoenix system [10]. As a robust semantic parser, PROFER can be used to extract semantic patterns from the output of the toolkit s recognizers while tolerating many of the features of spontaneous speech, such as false starts, filled pauses and ungrammatical constructions. It enables users to design conversational systems combining continuous speech recognition and natural language understanding. A step-by-step tutorial has been developed for PROFER in which students learn to develop a conversational system for retrieving movie times and locations from a Web site. Festival Speech Synthesis System. The toolkit integrates the Festival text-to-speech synthesis system, developed at the University of Edinburgh [11]. Festival provides a complete environment for learning, researching and developing synthetic speech, including modules for normalizing text (e.g., dealing with abbreviations), transforming text into a sequence of phonetic segments with appropriate durations, assigning prosodic contours (e.g., pitch, amplitude) to utterances, and generating speech using either diphone or unit-selection concatenative synthesis. CSLU has developed a waveform-synthesis plug-in component and seven voices, including male and female versions of American English and Mexican Spanish, and a German male voice. In addition, we have developed a graphical user interface that enables users to mark up a text string to control many features of the resulting synthesized speech (e.g., pitch, amplitude) and to insert pauses, filled pauses, coughs, sneezes, etc. SpeechView is the toolkit s interactive analysis and display tool. It allows users to create new waveform and label files, display data that are associated with a waveform (such as spectrograms or pitch contours), and modify existing waveforms and label files. It is used at CSLU for research, corpus development activities, and forms the basis for an interactive spectrogram reading class [12]. Several independent waveform windows, each with zero or more spectrogram and label windows may be displayed simultaneously within SpeechView for comparison and manipulation. Several spectrogram formats with user-defined signal processing and display options are available. Waveform sections corresponding to a phoneme or word label can be played back in isolation from adjacent phonemes or words. Zooming in and out is accomplished by highlighting the segment of interest. BaldiSync enables users to synchronize any speech waveform with Baldi's facial movements by supplying the waveform and the text of the utterance as a sequence of typed words. This module integrates Baldi, the Festival TTS system, the Toolkit's automatic phonetic alignment package, and SpeechView. BaldiSync assigns a phonetic sequence to each word using a pronunciation dictionary or TTS letter-to-sound rules and uses a Viterbi search to force an alignment of the phonetic segments to the speech waveform. The user can then play the entire utterance or any desired interval of labeled speech and watch Baldi produce it. Perceptual Science Laboratory (PSL) provides tools to support research in perception and cognition [13]. PSL provides a userfriendly research environment for designing and conducting multimodal experiments in speech perception, psycholinguistics, and memory. It enables users to manipulate auditory and visual stimuli; design interactive protocols for multi-media data presentation and multi-modal data capture, transcribe and analyze subjects' responses; perform statistical analyses, and summarize and display results. It supports a variety of experimental designs including factorial designs and expanded factorial designs (see [6]) enabling researchers to investigate the manner in which perceivers combine information from different knowledge sources (e.g., speech sounds and facial movements). Since PSL tools can be used to teach students to conduct research using the scientific method, it offers them new ways to conceptualize problems and investigate the world. Speech Performance Assessment and Measurement (SPAM) is a database program designed to capture and analyze all behaviors produced by the user and system during an interactive dialogue. Since all user and system behaviors are indexed in a database, data can be retrieved and analyzed to explore and test various hypotheses. In conjunction with PSL, SPAM provides an invaluable tool for designing and evaluating user interfaces for

4 conversational interactions. SPAM benefited greatly from the experiences of it s developer Daniel Solcher, who has been profoundly deaf since birth, and who overcame many obstacles to become a successful engineer in a hearing world. Programming environment: The toolkit comes with complete programming environments for both C and Tcl, which incorporate a collection of software libraries and a set of API's [14]. These libraries serve as basic building blocks for toolkit programming. They are portable across platforms and provide the speech, language, networking, input, output, and data transport capabilities of the toolkit. Natural language processing modules, developed in Prolog, interface with the toolkit through sockets. 4. OPPORTUNITIES The language resources described here offer new opportunities for research, education and practice in all areas of speech science. In education, the toolkit can be used to help students learn about speech and speech technologies through hands-on experience with state of the art software. The toolkit comes with several course modules, such as those described in [12] in a computer-based spectrogram reading course. Students in this course learn about the acoustic phonetic structure of English by recording, displaying and transcribing utterances, and playing utterances produced by Baldi with visible articulators. Toolkit courses have also been taught on building spoken dialogue systems [15,16] and text-to-speech synthesis. Course modules in speech perception and production, recognition, natural language understanding, text-to-speech synthesis and multimodal dialogue systems are all available in the toolkit. The toolkit also supports learning through interactive media systems that act like intelligent tutors. In the next three articles, my colleagues describe interactive language systems used to teach classroom subjects and speech, hearing and language skills. I believe that near-term advances in language and interface technologies will result in interactive media systems becoming the most effective means to acquire and create new knowledge. The CSLU Toolkit provides unlimited opportunities for new research. For example, phoneticians can use the toolkit to design protocols to record speech in different languages, transcribe the speech at different linguistic levels, and then compare labeling performance of different transcribers automatically [17]. Using RAD, useful systems can be developed and deployed quickly to investigate how people interact with machines using speech and to evaluate new language technologies in real-world applications. Educational researchers can use the toolkit to investigate learning using interactive language systems. In clinics and laboratories, the potential of the toolkit has just begun to be explored. Today, most evaluation of speech production is done with paper and pencil instruments. A tool such as SpeechView could be of enormous benefit to speech therapists by recording, transcribing and analyzing speech behaviors, and evaluating the effects of treatments over time. ACKNOWLEDGMENTS This work was supported in part by NSF grant ECS , NSF Challenge Grant CDA , a joint grant from the Office of Naval Research and DARPA, and Intel Corporation. REFERENCES 1. Fanty, M., Pochmara, J., and Cole, R. An interactive environment for speech recognition research. Proceedings of the International Conference on Spoken Language Processing, Banff, Canada, October Corpus information available at 3. Sutton, S., Novick, D., Cole, R.A., and Fanty, M. Building 10,000 spoken-dialogue systems. Proceedings of the ICSLP, Philadelphia, PA, October Cole, R., Sutton, S., Yan, Y., Vermeulen, P., and Fanty, M. Accessible technology for interactive systems: A new approach to spoken language research, Proceedings of the International Conference on Acoustics, Speech and Signal Processing, Seattle, WA, May Sutton, S., Cole, R.A., de Villiers, J., Schalkwyk, J., Vermeulen, P., Macon, M., Yan, Y., Kaiser, E., Rundle, B., Shobaki, K., Hosom, J.P., Kain, A., Wouters, J., Massaro, M. and Cohen, M. Universal Speech Tools: the CSLU Toolkit. Proceedings of the International Conference on Spoken Language Processing page , Sydney, Australia, November Massaro, D., Perceiving Talking Faces: From Speech Perception to a Behavioral Principle, MIT Press, Cambridge, Cosi, P., Hosom, J.P., Schalkwyk, J., Sutton, S., and Cole, R.A., Connected digit recognition experiments with the OGI toolkit s neural network and hmm-based recognizers. Proceedings of 4th IEEE Workshop on Interactive Voice Technology for Telecommunications Applications, Turin, Italy, September Hosom, J.P., Cole, R.A., and Cosi, P. Evaluation and integration of neural-network training techniques for continuous digit recognition. In Proceedings of the International Conference on Spoken Language Processing, Sydney, Australia, November Serridge, B., Cole, R. A., Barbosa, A., Munive, N., and Vargas, A. Creating a Mexican Spanish version of the CSLU toolkit. In Proceedings of the International Conference on Spoken Language Processing, Sydney, Australia, November Kaiser, E.C., Johnston, M., and Heeman, P. A. Profer: Predictive, Robust Finite-State Parsing for Spoken Language. In Proceedings of ICASSP, Phoenix, Arizona, March Black, A., and Taylor, P., "Festival Speech Synthesis System: System documentation (1.1.1)," Human Communication Research Centre Technical Report HCRC/TR-83, Edinburgh, Carmell, T., Hosom, J.P., and Cole, R. A Computer-based course in spectrogram reading. Proceedings of ESCA/SOCRATES Workshop on Method and Tool Innovations for Speech Science

5 Education, London, UK, April PSL tools and tutorials: Schalkwyk, J., de Villiers, J., van Vuuren, S., and Vermeulen, P. CSLUsh: an extendible research environment. In Proc. EUROSPEECH 97, pages , Rhodes, Greece, Colton, D., Cole, R.A., Novick, D., and Sutton, S. A laboratory course for designing and testing spoken dialogue systems. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Atlanta, Georgia, May Sutton, S., Kaiser, E., Cronk, A., and Cole, R. Bringing Spoken Language Systems to the Classroom. In Proceedings of EUROSPEECH, Rhodes, Greece, Cole, R.A., Oshika, B.T., Noel, M., Lander, T., and Fanty, M. Labeler agreement in phonetic labeling of continuous speech. In Proceedings of the International Conference on Spoken Language Processing, pages , Yokohama, Japan, September 1994.