Quarterly Progress and Status Report. Linguistic methods in communication aids

Transcription

1 Dept. for Speech, Music and Hearing Quarterly Progress and Status Report Linguistic methods in communication aids Hunnicutt, S. journal: STL-QPSR volume: 29 number: 1 year: 1988 pages:

2

3 STL-QPSR 1/ SPEECH ANALYSIS AND SPEECH PERCEPTION A. LINGUISTIC METHODS IN COMMUNICATION AIDS* S. Hunnicutt Linguistic components of text-to-speech systems are the subject of the earliest contributions to the thesis. These are sets of graphemeto-phoneme rules for several such systems, and a morphological classification of a large lexicon for one system. These systems are used as speech prostheses for non-vocal persons and as computer display readers for blind computer users. The next section deals with the linguistic components of a symbolto-speech system, an expanded version of one of the text-to-speech systems described in the first section. To convert symbols to phoneme strings, both lexicons and parsers were developed for four languages. In two languages, symbol-to-text systems were also developed. These systems are used as communication aids by persons experiencing severe speech and motor disabilities. Speech perception and recognition are the subject matter of the third section. The relationship of context redundancy and key-word intelligibility were examined in sentence frames. An intelligibility advantage was found for words in lower-redundancy contexts for text-type sentences, but no such advantage was found for words in either context in the adage and spoken-sentence pairs. It was found in pairwise comparisons, that differences in intensity, duration and FO occurred in the expected direction in a majority of word pairs, i.e., less redundant words were prosodically more prominent. Effects due to redundancy and recency of word use contribute to the segmental quality and prosody in natural speech. Knowledge of these effects helps us to improve the naturalness of synthetic speech in communication aids. In a further experiment, it was found that in communication with noise and with seemingly random requests for repeated reading, speakers increased their overall speech intensity, and raised their mean f undamental frequency and intonation range. One of two speakers increased sentence length as well. There was little evidence of segmental enhancement. A psycholinguistic study examined a set of perceptually similar words, and led to several organizin3 principles for a lexicon for automatic speech recognition. It was suggested that stressed vowels, initial segments and final sonorants in multisyllabic words be given high priority. Further examination of vocabularies for speech recognition revealed varying discrimination supplied by several coarse phonetic classifications for vocabularies in five European languages. Calculations were made of the number of equivalence classes defined by these classifications and the number of one-member classes. Maximum, mean and expected sizes of these classes were also calculated. The final section describes several algorithms which are designed to predict words from partial information. Experience in each of the areas described in the first three sections is brought to bear in the specification of these algorithms. One algorithm predicts words from * Summary of doctoral dissertation.

4 STL-QPSR 1/1988 initial letters. This program is being used as a speaking or writing aid. A second algorithm predicts words from prosodic and grammatical, as well as orthographic, information. This program is being tested as an aid in aphasia rehabilitation. The third algorithm accepts a phonetic description and is used in conjunction with a speech recognition system. Recognition systems such as this will be used in the future by speaking persons in need of written output capability. Introduction The linguistic methods and their application in communication aids discussed in this summary are grouped into four areas: text-to-speech! symbol-to-speech, speech perception and recognition, and lexical prediction. Their ordering reflects the time course of research initiated in these areas. Linguistic forms and entities described in this work which have been employed in communication aids include lexiconsl parsers, graphemeto-phoneme rules and symbols-to-sentence rules. Research methods have involved the study of large corpora to compile rule sets and lexicons, and percept ion experiments to determine saliency of phonetic, prosodic and lexical features in speech. A knowledge of these features is essential to high-quality speech recognition and speech synthesis. These linguistic studies and compilations have found application in several communication aids. One application is the English language speech synthesis used in voice prostheses for non-vocal persons and computer screen reading facilities for blind persons. Other applications are a Blissymbol-to-speech system in four languages! and two lexical prediction programs, also in several languages. Tex t-to-speech Grapheme-to-phoneme rules The rules presented in Hunnicutt (1976a) are part of a text-tospeech system called MITalk developed at the Massachusetts Institute of Technology in the 1970s (Allen, Hunnicutt, & Klatt, 1987). The system was further developed and marketed by Telesensory Systems, Inc. (the developers of the Optacon) and later by Speech Plus. The phonemic rules were developed by a process of extensive statistical analysis of English words; the lexical stress rules are an adaptation of rules proposed by Halle and Keyser. Contained in the MITalk system was a large morph lexicon which, together with a word analysis algorithm was capable of correctly analyzing more than 100,000 words of English. The grapheme-to-phoneme rules were used as an adjunct just for those words which the algorithm could not analyze into constituent morphs present in the lexicon. The format chosen for the rules was that used in generative grammar, and a rule compiler designed to accept such rules was developed by F.X. Carroll. The rules operated by first strip ping aff ixes from a wordr converting the consonants in the rootr and then, using this dependable consonant framework to provide the context! converting vowels and affixes. This conversion to phonemes was accomplished by one pass through a set of ordered rules. Lexical stress was placed by a further set of ordered rules including a core of cyclic rules which applied to an increasing string of morphs within a word. These stress rules were not written as rules, but were rather written in a conventional programming language. Studies of allowable suffix combinations and the effect of suffixation on stress and vowel quality contributed to the originality of the work.

5

6

7 STL-QPSR 1/1988 tion between any one of these factors and word intelligibility was quite low. However I it was observed in a pairwise comparisonr that differences in the prosodic factors occurred in the expected direction in a majority of word pairs. This was true for all sentence pairs, even for the adage/spoken-sentence pairs, in which intelligibility was not correlated with redundancy. This suggests that a talker often produces subtle, but measurable, prosodic emphasis in his production of words in non-redundant contexts, even though he may not always be successful in improving word intelligibility for his listener. The experiment described in Clark, Lubker, & Hunnicutt (1988) was inspired, in part, by the questions posed and the results obtained in Hunnicutt (1985). It was designed to further investigate speech production due to perceived difficulty in communication. In particular, it was planned to manipulate the difficulty of the communication environment and to study the response of the speaker in phonetic terms. This was accomplished by having two speakers read short sentences from cards, first in quiet, and then in noise with planned but seemingly random requests for repeated reading by a listener. In communication with noise, both speakers increased their overall speech intensity, and one speaker showed a striking increase in spectral energy between 1 khz and 3 khz. Both speakers raised their mean fundamental frequency substantially, one speaker shifting his pitch almost an octave upwards. Pitch range was also increased, and a seemingly semantically motivated increase of pitch was produced on the word perceived to be most important. One speaker additionally increased sentence duration, partially by the insertion of pauses after words perceived to be critical. There was little evidence of segmental enhancement; however, somewhat more peripheral vowel qualities were observed in the noisy conditions. Thus, the most marked differences were found in prosody and voice quality. Analysis of the data, and manuscript preparation for Clark & al. (1988) were undertaken by J. Clark at Macquarie University. These three papers (Hunnicutt, 1985; 1987a; Clark & al., 19888) deal with comparatively small, but important effects in speech which need to be considered to improve the quality of speech synthesis and phonemic speech recognit ion. Effects due to redundancy (Hunnicutt, 1985; 1987a) and recency of word use (new/old information) contribute to the variation in segmental quality and prosody in natural speech. Contributions due to these ef fects in synthetic speech would improve its naturalness. They are, of course I extremely difficult to calculate -- semantics is a field of the future in synthetic speech research. The effects observed in Clark & al. (1988) contribute to understanding of the ways speakers are likely to interact with automatic speech recognizers which fail to accept an initial spoken input. This study also has implications for the understanding of speech production used by normal speakers in communication with hearing-impaired listeners. A Psycholinguistic Study This paper is a token of a long and continued conviction that human lexical organization and the matching or retrieval of word representations in the mental lexicon referred to as "lexical access" are of vital importance. Studies in this field are essential both for automatic machine speech recognition where modelling human speech perception is one possible method, and in the specification of a communication aid where human speech production can be the model. In Jakimik & Hunnicutt (1951), the perceptual similarity of words is considered, and an attempt is made to characterize what words might be confused by a human listener. First, a space of potentially confus-

8 STL-QPSR 1/ able words was constructed based on what is known of listeners' perceptual confusions. Holding number of segments and stress pattern constant, and considering the most likely confusable phoneme sets, a space of word confusions was derived. Only 11% of the space of possibilities was found to contain actual Ehglish words. Data generated from a larger confusion base yielded a space over an order of magnitude larger with a population density of 9% actual English words. Secondly, an experiment was performed in which listeners ranked the similarity of sound of words. The hypothesis that words sharing a stressed vowel would be judged as more similar in sound was confirmed. Two other "anchor points" or dimensions of similarity were also found: words beginning with the same segment were judged as very similar, and for multisyllabic words, a final syllabic sonorant served to relate words. It was suggested that in organizing or indexing a lexicon for automatic speech recognition that stressed vowels, initial segments and final unstressed sonorant syllables of multisyllabic words be given high priority. Lexical Stat istics An interest in the use of word properties and the distribution of words in "phonetic space" in automatic speech recognition led us to an investigation of large vocabularies (approximately the 10,000 most frequent words) of five European languages: Swedish, English, German, French and Italian. These analyses are presented and discussed in Carlson, Elenius, Granstrom, & Hunnicutt (1985). In order to approximate the benefit that would be derived in a speech recognition system by a coarse classification of the phonemes of a word, four such classifications were stipulated. Two of the classifications varied from the other two by inclusion of stress information. One classification simply mapped phonemes onto C (consonant) or V (vowel); the other major classification mapped vowels onto V and consonants into five separate classes. Calculations were then made of the number of equivalence classes (cohorts) defined by these classifications and the number of unique (one-member) cohorts. The maximum, mean and expected cohort sizes were also calculated. For all four classif ications, the German vocabulary was best discriminated, having the largest number of cohorts and unique cohorts, and with one exception, the smallest maximum cohort size. English and Swedish vocabularies take the middle ground, being less well discriminated by the classifications than German, but better than French and Italian. The French vocabulary was by far the least well discriminated by the classifications. The more discriminatory 5-consonant classification gave a large increase in number of cohorts for all languages. For the Germanic languages, over 80% of the increase in number of cohorts (frorn 1 to 5 consonant classes) was in unique cohorts. For all languages, however, the number of unique cohorts was a linear function of the total number of cohorts. The 5-consonant classification also produced a highly significant decrease in expected cohort size of at least an order of magnitude. Several studies were also under taken to investigate how lexical search is facilitated by partial knowledge of phonemic word structure or by knowledge of a word's grammatical class. Words in the five vocabularies were classified according to the number of phonemes it takes (from the beginning of the word) to uniquely specify them, and according to these "points of identification" relative to the word's primary stressed vowel. The so-called "lexical redundancy" defined by this identif ication point was also calculatedl both from the word's beginning and from

9

10 STL-QPSR 1/ experiencing disabling effects of aphasia. Trials of the program with these two user groups are currently being planned. Options included in the two prediction programs PREDICT and ACCESS are output via synthetic speech and input via a pattern-matching speech recognition system. Any set of distinct utterances! one corresponding to each letter! can be used to train the recognizer. In both prediction programs, two additional files are generated during the program's use. One file saves the final text and can be accessed at a later time for printing or further editing with a text processing program. The other file keeps information about what has transpired during a session in terms of all keys typed and elapsed time. This information can be used to check a user's progress and to test program efficiency and use ulness. COHORT is the program developed to be used together with a phonetic word recognition system. It is also designed to access a word given partial information. That information can be in the form of phonemes, stress! or phonetic features. It is also possible to specify an ordered set of options for a phoneme. This program is a component of a speech recognition scheme called NEBULA under development at our laboratory, part of which includes a phonemically-based speech recognizer. COdORT has the same structure of lexicons as the other two prediction programs, but additionally includes a phonetic transcription for each word. Phonetic speech recognizers, such as the one from which COHORT can take input, are able to support much larger vocabularies than systems strictly based on pattern-matching. In the future, these systems will be useable for generating written output by persons whose only (or best) ability to communicate is vocal. In the more near future, a person will be able to say one word at a time and have that word recognized and transcribed! or have a list of alternatives given which can then be accessed by voice. Final Remarks The thesis has presented research in the areas of lexical structure, devising of phonological rules for English and parsing rules for several languages. It has also presented three perception experiments! and phonetic! primarily prosodic! analysis of the results of two of these experiments. The knowledge gained from the experiments and lexical studies, and the lexicons and rule sets derived have contributed to our efforts to describe speech and language. It is such understanding which allows us to see the relationships needed to design an aid to communication. Further research in semantics, lexical theory, syntax! phonetics and psycholinguistics will help us to make more useful communication aids. There is much still to be learned, and we will doubtless continue to be inspired by the opportunity to apply what we learn in helping to remove communication barriers. A paper describing the use of speech synthesis and recognition in comunication aids has been included as an appendix (publ. in STL-QPSR 4/1986, Blomberg & al.). In it will be found short descriptions and pictures of some of the systems to which the research and development described in this thesis have contributed. References Allen, J., Hunnicutt! S., & Klatt,D. (1987): From Text to Speech: the MITalk System, Cambridge University Press, Cambridge.

11 STL-QPSR 1/ Blomberg, M., CarlsonI Rat Elenius, K., Galyas, K.1 Granstrom, B., Hunnicutt, S., & Neovius, L. (1986): "Speech synthesis and recognition in technical aids", STL-QPSR 4/1986, pp Carlson, R.I Granstrom, B., & Hunnicutt, S. (1982): "Bliss Communication with Speech or Text Output," pp in Proc. ICASSP Paris, Vol. 3. Carlson, R., Elenius, K.1 Granstrom, B., & Hunnicutt, S. (1985): "Phonetic and Orthographic Properties of the Basic Vocabulary of Five European Lanquages," STL-QPSR 1/ 1985, pp Clark, J.1 Lubker, J.1 & Hunnicutt, S. (1938): "Some Preliminary Evidence for Phonetic Ad justmen t Strategies in Communication Dif f icultv", * - in (R. Steele & T. ~hriad~old, eds.) Language Topics: Essays in Honour of Michael Halliday, John Benjamins, Amsterdam. Hunnicutt, S. (1976a): "Phonological Rules for a Text-to-Speech Am. J. of Computational Linguistics, Microfiche 57. System," Hunnicutt, S. (197613) : "A New Morph Lexicon for English," in Proceedings COLINGt Ottawa. Hunnicut t, S. ( 1980) : "Grapheme-to-Phoneme Rules: A Review," STL-QPSR 2-3/1980t pp Hunnicutt, S. ( 1985) : "Intelligibility versus Redundancy - Conditions of Dependency," Language & Speech 28:11 pp ~unnicutt, S. (1986a): "Bliss Symbol-to-Speech Conversion: '~lisstalk,'" J. of the American Voice I/O Society, 31 pp Hunnicutt, S. (1986b): "Lexical Prediction for a Text-to-Speech System" pp in (E. Hjelmquist & L-G. Nilsson, eds.) Communication and Handicap: Aspects of Psychological Compensation and Technical Aids, Elsev ier Science Publishers, Amsterdam. Hunnicutt, S. (1987a) : "Acoustic Correlates of Redundancy and Intelliq ibility," STL-QPSR 2-3/1987, pp Hunnicutt, S. (198713) : "Input and Output A1 ternat ives in Word Prediction t " STL-QPSR 2-3/1987 t pp Jakimik, J. & Hunnicutt, S. (1981): "Organizing the lexicon for recognition," written version of paper presented at the meeting of the Acoustical Society of America, Ottawa. -