Building a Regionally Inclusive Dictionary for Speech Recognition

Transcription

1 SPRING 2004 Computer Science & Linguistics Building a Regionally Inclusive Dictionary for Speech Recognition Speech Recognition (SR) is the automated conversion of speech into written text. Applications range from simple phone-based information services to commercial-grade automated customer service systems, such as those for airline phone reservations. While this process is complex in and of itself, it is further complicated by the fact that speakers (the users) from different parts of the country have varying accents and pronounce the same words differently. Our aim is to create a more speaker-independent SR system while maintaining speed and accuracy of transcription. This requires the construction of an SR dictionary that takes into account the existence of multiple pronunciations for the same word. However, the existence of too many alternate pronunciations overloads the system and is detrimental to accuracy and speed. By finding the optimal number of pronunciations per word, the percentage of words correctly identified by the SR system increased from 78% using the traditional technique to 85.7% using the improved method outlined in this article. This brings the technology 35% closer to the goal of complete recognition and the use of speech as the primary method of human-computer interaction. Justin Burdick Speech Recognition (SR) is a process that transcribes speech into text using a computer. Many repetitive phone tasks can be automated with speech recognition technology, saving businesses significant amounts of money. However, the transcription of everyday human speech is significantly more difficult than simply the recognition of a small set of words, as is the case in a SR-based phone information-retrieval system (e.g. 411 services). A complete Speech Recognition system is fairly complex and consists of various subsystems that interact together to convert speech into written text, as shown in Figure 1. A typical SR system utilizes a common tool for pattern-matching called Hidden Markov Models (HMM) to model parts of speech known as phonemes (similar to syllables) [1]. The models are first trained using a set of training data, which consists of a large number of wave sound files with speech and the corresponding text transcriptions. Each speech file is converted into a series of observations known as feature vectors." These are sets of numbers which describe the sound mathematically by extracting specific information from the waveform at 25-millisecond intervals. The computer then matches this numerical representation to the corresponding text transcription included in the training data. Applied over the entire training set, this process creates a series of models which later allow the SR system to translate speech into text. The training process results in the creation of two data files: one consisting of a model of properties for each phoneme and another of a list of pronunciations for each word. Phonemes, the most fundamental units of speech, are single sounds; each phoneme is modeled by its average sound, the variation of that sound across various speakers, and the transition probabilities between it and other phonemes. The list of pronunciations, termed the pronouncing dictionary, contains each of the words used during training along with the sequence of phonemes that compose that word. Once the models are trained, the system can be used to perform recognition. In order to produce meaningful sentences, the SR system relies on a dictionary and a grammar model as well as the trained phoneme models from the training phase. Transcription is accomplished using the Viterbi algorithm. This algorithm divides the incoming speech into individual observations and processes one observation at a time by comparing it to every possible phoneme. When the algorithm moves to the next observation, it eliminates paths that have a low matching probability. By this elimination at each observation, only the most probable paths survive until the end of the observation sequence; the recognized word sequence can then be generated. The dictionary is important in this process. As the system processes each observation, it must check the dictionary to see if it has made a word yet. Once the word is recognized, the system uses the grammar file to influence the next word chosen, which is very valuable for making the sentence meaningful, similar to grammar check in a wordprocessing program. Implementing an effective SR system requires the synthesis of concepts 1

2 SURJ. Building the Dictionary Figure 1. A simplified model of an HMM-based speech recognition system from several fields, including: Signal Processing (for extracting important features from the sound waves) Probability and Statistics (for defining and training the models and recognition) Linguistics (for building the dictionary and the grammar model) Computer Science (for creating efficient search algorithms) Improvements in any of the subsystems above could result in better overall performance. The following performance goals are among the main objectives of any SR system [2]: 1. Increased Accuracy Accuracy is measured as a percentage of words (or sentences) that are correctly detected. To emphasize the importance of this value, consider a certain SR system that has an accuracy of 99%. This means that during the task of dictation, one word in every 100 words is identified incorrectly. For the average dictation, such an error rate would result in approximately six errors per single-spaced page. Searching for and correcting these typos is a tedious and time-consuming task. This shows that even an accuracy of 99% may not be sufficiently high for the task of dictation. 2. Increased Speed Many speech recognition applications must run on small hand-held devices with limited CPU power. It is important for the SR system to be as computationally inexpensive as possible. This allows the system to transcribe speech in realtime, even on a low-performance system like a PDA or cell phone. 3. Speaker Independence Most SR systems can operate with very high accuracy if they are trained to a specific speaker. However, problems can arise when the system attempts to transcribe the speech of a different person. The reasons for this include, but are not limited to: a. Different speakers may pronounce the same phonemes (sub-words) differently, e.g. a speaker from Brooklyn may pronounce certain vowels differently than a speaker from California. b. Different speakers may pronounce the same words differently, with a different sequence of phonemes, e.g. the word maybe could be pronounced as m ey b iy or as m ey v iy. In this project, our goal is to improve recognition of speaker independent SR systems by taking into account that different speakers may pronounce similar words differently. This requires building a dictionary that has multiple pronunciations for any given word. The details of building such a dictionary are explained below. In order to test the effects of this approach, a set of carefully designed experiments were conducted, the results of which are in the Results and Discussion section. The TIMIT (Texas Instruments- Massachusetts Institute of Technology) database was used for both training and testing of the SR system. This database contains audio files of sample sentences as well as word-level and phonemelevel transcriptions. However, we needed to create our own "pronouncing dictionary" before we could do training and recognition. Each word in a pronouncing dictionary is associated with a series of phonemes that approximate the sound of that word. However, some words need multiple pronunciations in order to be best represented (see Table 1 for examples). When this sequence of phonemes is pronounced, it accurately approximates the sound of the word being spoken. All of the TIMIT data was used in the experiments. About 70% was used to train the models, and the rest was used for performing the testing. Table 2 summarizes key characteristics of the TIMIT data. Method limitations In order to provide a versatile training set, data was collected from speakers spanning eight different geographical areas to capture regional dialects [4] (shown in Figure 2). Unfortunately, these regional dialects introduce problems for the construction of a dictionary. Since many of the speakers pronounce words differently especially in the case of short, common words multiple transcriptions for the same word were common. In fact, the Word As Coauthors Coffee Phoneme-level Transcription ae z ax z kcl k ow ao th axr z k aa iy f k ao f iy Table 1. Sample dictionary entries. Multiple pronunciations can exist for the same word. For example, the first pronunciation of coffee is that of a typical New Yorker, while the second is that of someone from the old Northwest. 2

3 Number of... Speakers 630 (70% male, 30% female) Utterances 6300 (10 per speaker) Distinct texts 2342 Words 6099 Utterances in the training set 4620 Utterances in the test set 1680 Male speakers 326 (training set) (test set) Female speakers 136 (training set) + 56 (test set) Table 2. Key characteristics of the TIMIT data unedited dictionary contained an average of two pronunciations per word [1]. However, these pronunciations tended to cluster around certain words, with most words having only one pronunciation but with others having six or more. A certain level of multiple pronunciations can be helpful at capturing common divergences in pronunciation (e.g. potato/pot_to), as even the best hand-made dictionaries contain 1.1 to 1.2 pronunciations per word [5]. Too many pronunciations per word, however, could be misleading when the computer performs recognition, because a poorly recognized string of phonemes could cause a word mismatch that makes a sentence meaningless. Methodology In order to reduce the number of redundant definitions in the dictionary, two methods were investigated. The first, called skimming, simply edits out and deletes low-frequency transcriptions. This method defined a threshold based on the most commonly occurring pronunciations. Anything below this threshold was removed from the dictionary. The second method, called percentaging, encodes the frequency at which each pronunciation was encountered into the definitions themselves, which a speech recognizer can use to modify its recognition network. As mentioned earlier, the dictionary was created by examining all the speech transcriptions during training mode. The original system just added all of the different combinations of word and phonemes into the system. The new method does the same thing, except that it keeps count of how many times each combination occurs. These counts can be used to perform either skimming (removal of low-occurrence pairs), or percentaging (placing percentage occurrence information with the data). This paper examines the effect of skimming alone, percentaging alone, and finally skimming followed by percentaging on SR system accuracy. Results and Discussion The two systems of dictionary making (skimming and percentaging) were tested independently and then with the 2-step method (skimming followed by percentaging). As shown in Figure 4, skimming examples of dictionaries after processing provided a much larger accuracy gain than did percentaging. When the two were used simultaneously, percentaging added a Figure 2: Speaker Breakdown by Region (total number of speakers= 630). SPRING 2004 very slim increase in accuracy to the method of skimming. An unexpected bonus of using the skimming method is a reduction in required recognition time. Since there are fewer possible recognition sequences, the computer can process each speech fragment more rapidly. At optimal skimming (30%), the test took only two hours, rather than the full three, which is a 33% reduction in time. Furthermore, at this level of skimming, there was an average of 1.6 pronunciations per word, as opposed to 2.0 pronunciations without skimming. Thus, not only does skimming provide more accurate speech recognition, it also allows for faster recognition. Conclusion Since people speak in many differ-ent ways, it is expected that having a large dictionary, inclusive of all pronunciation and dialectic differences, would be a valuable aid to recognition. However, this paper demonstrates that the most effective SR system dictionary may be one with a small number of select alternate pronunciations per word. The dictionary must be built so that a balance is found between being inclusive of pronunciation variants and of conforming to the limitations of a computerized recognition system. This study observed that while a typical SR system contains on average 2.0 pronunciations per word, maximum 3

4 SURJ. SR system accuracy is achieved at 1.6 pronunciations per word. Tailoring a dictionary to contain fewer pronunciations per word is best achieved manually; however, given the size of a typical SR system dictionary, (13,000 entries) a computerized method of deleting erroneous and infrequently used pronunciations is more practical. This study revealed that the method of skimming both effectively increases recognition accuracy and results in a 33% reduction in the time required for speech recognition tasks. An example of an unaltered dictionary Clearly, about ax b aw t is the most commonly given pronunciation for the word about, and so it should be given the highest weight, whereas definitions bah and baw should just be erased. Examples of dictionaries after processing After Skimming 4 about ax b aw d 17 about ax b aw t Here we have done skimming, deleting the least frequently occurring pronunciations. After Percentaging 3.85% about ax b ae t 7.69% about ax b ah 15.38% about ax b aw d 65.38% about ax b aw t 3.85% about b ah 3.85% about b aw Original 1 about ax b ae t 2 about ax b ah 4 about ax b aw d 17 about ax b aw t 1 about b ah 1 about b aw Here we have done percentaging, inserting the frequency of each pronunciation as a percentage. Figure 3: The process of modifying the dictionary Table 3. Simulation Test Results Figure 4. Test Results 4

5 SPRING 2004 References 1. Mohajer, K., Zhong-Min Hu, Pratt, V. Time Boundary Assisted speech recognition. International Journal of Information Fusion (Special Issue on Multi-Sensor Information Fusion For Speech Processing Applications). 2. Rabiner, Juang. Fundamentals of Speech Recognition, Chapter 6, Prentice Hall Signal Processing Series, New Jersey, Young, Evermann, Kershaw, Moore, Odell, Ollason, Valtchev, Woodland. The HTK Book, Ch5, Cambridge University Engineering Department, Garofolo, Lamel, Fisher, Fiscus, Pallett, Dahlgren. TIMIT Printed Documentation, Ch5, U.S. Depart. Of Commerce, Hain, T., Woodland, P.C., Evermann, G. et al. New features in the CU-HTK system for transcription of conversational telephone speech. Acoustics, Speech, and Signal Processing, Proceedings. (ICASSP 01) IEEE International Conference on, Vol.1, Iss., Pages:57-60 vol Culicover, Peter. Lecture: An Introduction to Language in the Humanities. Spring variation.pdf. Justy Burdick Justy Burdick is a sophomore majoring in Electrical Engineering and is considering a minor in Computer Science. He would like to thank Professor Vaughan Pratt and the URP office for sponsoring this Research Experience for Undergraduates project. Furthermore, he would like to thank Keyvan Mohajer for the guidance and help he received at every step of the research process. Finally, he would like to thank the 2002 and 2003 speech group REU team members: Simon Hu, Melissa Mansur, Ryan Bickerstaff, Kenneth Lee, Gu Pan Grace. 5