ANALYSIS AND SYNTHESIS OF F 0 CONTOURS OF DECLARATIVE, INTERROGATIVE, AND IMPERATIVE UTTERANCES OF BANGLA. Professor Emeritus, The University of Tokyo

Transcription

1 ANALYSIS AND SYNTHESIS OF F 0 CONTOURS OF DECLARATIVE, INTERROGATIVE, AND IMPERATIVE UTTERANCES OF BANGLA Anal Haque Warsi 1, Tulika Basu 1, Keikichi Hirose 2, Hiroya Fujisaki 3 1 Centre for Development of Advanced Computing(C-DAC), Kolkata 2 Department of Information and Communication Engineering, The University of Tokyo 3 Professor Emeritus, The University of Tokyo anal.warsi@cdackolkata.in, tulika.basu@cdackolkata.in, hirose@gavo.t.u-tokyo.ac.jp, fujisaki@alum.mit.edu ABSTRACT This study first examines the differences in the gross features of the fundamental frequency contour (the F 0 contour) responsible for discriminating utterances of three sentence types, namely declarative, imperative and interrogative, in Bangla. In order to realize these differences in speech synthesis, these differences are then interpreted in terms of differences in the parameters of the commandresponse model for F 0 contour generation. Finally, the results of model-based analysis were used to generate synthetic speech stimuli for a perceptual experiment in order to verify the results of analysis. The result of the experiment indicates that the synthesized F 0 contours are quite satisfactory for the perception of utterances of three sentence types in Bangla, and thus can be successfully used in a concatenative Text-to-Speech System of Standard Colloquial Bangla (SCB) developed by C-DAC, Kolkata. Index Terms: Sentence type, F 0 contour, analysis, synthesis, Bangla, 1. INTRODUCTION Speech has been used as the most efficient means of communication amongst human beings. One of the features of speech as an effective communication medium is to convey simultaneously not only linguistic information but also to transmit paralinguistic and nonlinguistic information. Segmental features alone do not serve all these purposes. Communication becomes meaningful only when the suprasegmental features are imposed and is useful in speech production as well as perception. The rising and falling tones of language, segmental duration and amplitude separate ideas, distinguish questions and statements and show special emphasis. Hence the technique of imposing knowledge of prosody of a language is extremely important in machine-generated speech in the arena of spoken language technologies. But these prosodic cues which are inherent in speech cannot be quantified in an absolute manner and are highly relative to individual speaking style, gender, dialect and other phonological factors [1]. The difficulty in characterizing supra-segmental features has resulted in various schemes for labeling or modeling the intonation of various languages across the world. The ToBI labeling system and the command-response model are most widely known among them. ToBI [2] is basically meant to capture the phonological aspect of speech. It is designed for transcribing the prosodic events of utterances in terms of a set of labels. Since the prosodic organization differs from language to language, there are many different ToBI systems, each one specific to each language [3]. Since, however, it does not provide quantitative description, ToBI labels cannot be used for high-quality speech synthesis. On the other hand, the command-response model of fundamental frequency contour (henceforth F 0 contour) generation by Fujisaki and his coworkers [4] is based on quantitative formulation of the physiological and physical mechanisms of speech production, and thus can provide highly accurate reproduction/prediction of tonal features of speech regardless of difference of language [5]. Production/detection of prosodic features is vital in all applications of spoken language technology including textto-speech (TTS) synthesis [6], speech recognition [7], speech understanding [8], and speech-to-speech translation [9]. The preliminary research work described in this paper is motivated by the desire to enhance the intonation model within a text-to-speech synthesis framework of Bangla. C-DAC, Kolkata, jointly with the University of Tokyo, had analyzed the F 0 contours of Bangla declarative sentences using the command-response model. The obtained results were used to identify the prosodic words and the prosodic phrases from the text and to assign appropriate parameter values to their respective commands for use in the TTS system [10]. In the present study, an attempt is made to identify the differences in intonation that exist among utterances of the three types of Bangla sentences, i.e., declarative, imperative and interrogative, so that the findings can be used for improving the naturalness of the synthesized output of

2 Bangla Text-to-Speech system developed by C-DAC, Kolkata [11]. The rest of the paper is organized as follows. Section 2 gives a brief description of the command-response model. Section 3 describes the design of speech corpus. Section 4 presents experimental procedures along with their results. Finally, discussion and conclusion are presented in section THE COMMAND-RESPONSE MODEL The command-response model describes an F 0 contour in the logarithmic scale as the superposition of a baseline value, phrase components, and accent components as given by Equation (1) [4]. I F 0(t) = ln(f b) + ( 0 ) + { ( 1 ) 2 ApiGp t T i Aaj Ga t T j G a(t T j)} i= 1 j= 1 α 2 t exp( α t), t 0 G p( t) = 0, t < 0 min[1 (1 )exp( ), ], ( ) + β t β t γ Ga t = 0, J t 0 t < 0 Equation (2) represents the response of a second-order critically-damped linear filter (the phrase control mechanism) to an impulse called phrase command, and Equation (3) represents the response of another secondorder critically-damped linear filter (the accent control mechanism) to a step function called accent command. The parameter α is the natural angular frequency of the phrase control mechanism which essentially characterizes the rate of declination of the phrase component, β is the natural angular frequency of the accent control mechanism which characterizes the rate of local rise/fall of the F 0 contour, and γ is the ceiling parameter. In Equation (1), F b is the baseline frequency. Its value is dependent on the size, mass, and stiffness of the vocal folds, and therefore varies from individual to individual. Among utterances of a single speaker it does not vary greatly as long as the speaking style remains the same, but can vary appreciably if the speaking style changes. In the context of the present analysis, F b was allowed to vary from utterance to utterance since it was felt to be desirable to produce the closest approximation to measured F 0 contour of each utterance. I and J are the numbers of phrase and accent commands, respectively. A pi is the magnitude of the i th phrase command and A aj is the amplitude of the j th accent command. T 0i is the time of occurrence of the i th phrase command, while T 1j and T 2j are the onset and offset times of the j th accent command. 3. DESIGN OF BANGLA SPEECH CORPUS The Bangla speech corpus used in this study was designed and created by the authors at CDAC-Kolkata. (1) (2) (3) 3.1. Speech Material Fifty sentences each of declarative, imperative and interrogative ( yes-no question) types were designed for this study in such a way that except for the punctuation mark there was no difference in the textual content. Hence except for prosody there was no linguistic difference in the utterances of each of the three types of sentences. Table 1. Example of two sentences recorded in three utterance types, i.e., declarative, imperative and interrogative. Bangla IPA English Translation Bangla IPA English Translation / / /ebar gǥm lagaben/ Declarative: (You will) Sow wheat seeds this season. Imperative: (You must) Sow wheat seeds this season. Interrogative: Will you sow wheat seeds this season? / / /tumi baȴar ȴao/ Declarative: (Generally) You go to the market. Imperative: Go to the market. Interrogative: Do you go to the market? 3.2. Informant Selection and Recording The speech signal was initially recorded by 6 female speakers. All of them are native speakers of Standard Colloquial Bangla (SCB), the official dialect of West Bengal, and are in the age group between 20 to 40 years. The metadata of the speakers is given in Table 2. Table 2. Metadata of informants. Serial Age Group Sex Number 1. Between Years F 2. Between Years F 3. Between Years F 4. Between Years F 5. Between Years F 6. Between Years F

3 The recordings of speech data for all six speakers were done in a studio environment and digitized at a sampling rate of 22,050 Hz with an accuracy of 16 bits/sample. The data was recorded using the legendary Shure SM58 vocal microphone with the help of Cool Edit Pro Software. During the recording a constant distance from the microphone element and the speaker s mouth was maintained. Identical environment was maintained for every speaker. For each of the six speakers recording was done in two sessions. In the first session the speaker was asked to give three repetitions such that each declarative utterance is followed by interrogative and then by imperative utterance. In the second session the speakers were asked to give another set of three repetitions. But this time they were asked first to speak all the declarative utterances, then all the interrogative utterances and finally all the imperative utterances. In designing the speech material, the following points were kept in mind: (1) the sentences should be natural in their meaning. (2) Unvoiced stops and fricatives were avoided in order to minimize the segmental effect on F 0 contours. All the sentences used in this study were 5 to 8 syllables long, and were uttered without intra-sentential pause. After recording, a listening test was conducted in order to select the data for the analysis. For this purpose the utterances were randomly mixed and presented to 10 native listeners for giving their judgments on the utterance type. Finally, for analysis, only those utterances were selected which all the listeners perceived as the respective type of utterances. 4. EXPERIMENTAL PROCEDURE The experiment was conducted in the following three stages: (i) analysis of gross features of F 0 contours, (ii) model-based analysis of F 0 contours and extraction of model parameters, and (iii) synthesis of speech based on the analysis results and perceptual verification of validity of synthesis Analysis of Gross Features of F 0 Contours Results F 0 analysis of the recorded data shows that Declaratives generally have falling intonation and a terminal fall. Interrogatives utterances have a gradually rising intonation and a terminal rise. Compared with Declaratives and Interrogatives, Imperatives tend to have a rather flat intonation, i.e., neither gradually falling (typical for Declaratives) nor gradually rising (typical for the Interrogatives). It is evident from the data in Table 3 that maximum F 0 (Max. F 0 ) value plays a vital role in distinguishing Interrogatives from the other two types of utterances in Bangla. Namely, Interrogatives have much higher values of Max. F 0 compared to their Declarative and Imperative counterparts. However, there is no significant difference in the minimum F 0 (Min. F 0 ) values between the three types of utterances. Table 3. Comparison of maximum and minimum F 0 values. Max. F 0 µ: σ: Min. F 0 µ: σ: 6.72 µ: σ: µ: σ: 7.55 µ: σ: µ: σ: Analysis of prosodic words in the utterances of all the six speakers reveals that compared to Declaratives and Imperatives, Interrogatives tend to have a much larger dynamic range of F 0 in the last prosodic word of the utterance. Figure 1 represents the dynamic range of F 0 expressed as the ratio of Max. F 0 to Min. F 0 of the final prosodic word of each utterance. In case of Interrogatives, F 0 starts at Hertz and ends at Hertz. On the other hand, in Declaratives and Imperatives F 0 starts at Hertz and ends at a value which is Hertz lower. In the first place, analysis of gross features of the F 0 contours of declarative, imperative and interrogative utterances (henceforth denoted by Declaratives, Imperatives and Interrogatives, respectively) for all the six speakers was done Method of analysis In order to capture the gross features of the F 0 contours, the F 0 values were extracted at every 20ms. After obtaining frame-wise F 0 values for the entire utterance, five values were measured as gross features representing the entire profile of the utterance. They were: maximum, minimum, mean, standard deviation, and ratio of maximum to minimum within the entire utterance. The computed features were then analyzed to identify the differences in terms of F 0, if any, that exist among the three types of the utterances. Figure 1: Dynamic range of F 0 in the final prosodic word of Declaratives, Imperatives and Interrogatives of six speakers.

4 4.2. Analysis of F 0 Contours Using the Command- Response Model Since the differences in the gross features cannot be directly utilized for speech synthesis, model-based analysis of the F 0 contours is necessary. For this purpose, automatic extraction of the command-response model parameters followed by fine manual tuning was carried out for accurate extraction of model parameters of F 0 contours. The extracted command parameters were then analyzed to find out how the observed differences in the gross features of F 0 contours of the three types of utterances can be explained in terms of the command response model Method of analysis For analyzing the command parameters of the utterance, the extracted F 0 contour (in the logarithmic scale) was decomposed into its constituents, i.e., the baseline frequency, the phrase components and the accent components. The magnitude and timing of the underlying phrase and accent commands were estimated using the Analysis-by-Synthesis method. The extracted parameters were then compared to identify the differences that exist among them. Figure 2 shows how the command parameters were estimated Results From Tables 4 and 5 it is clear that Interrogatives have a much lower baseline frequency (F b ) and have a higher value of utterance-initial A p. Imperatives, on the other hand, have a higher F b and a lower utterance-initial A p value. In addition to the initial positive phrase command, Declaratives and Imperatives have an additional negative phrase command towards the end of the utterance. The lead times of the phrase commands are presented in Table 6. From the data presented in Table 6 it is clear that in case of Imperatives the onset of the utterance initial phrase command is delayed by milliseconds as compared with Declaratives and Interrogatives. In case of Declaratives the negative phrase commands start approximately 80 milliseconds before the segmental onset of the first syllable of the final prosodic word. In case of Imperatives they start approximately 33 milliseconds before the segmental onset of the last syllable of the final prosodic word of the utterance. Thus the duration of the falling F 0 slope is greater in case of Declaratives than Imperatives. This negative slope causes the terminal fall characterizing Declaratives. Table 4. Base frequency in hertz (F b ) µ: µ: µ: σ: σ: 4.54 σ: 8.15 Table 5. Phrase command magnitude (A p ) Initial Final µ: µ: µ: σ: σ: σ: µ: µ: σ: σ: Table 6. Phrase command lead-time [in s] Figure 2: Examples of modeled F 0 contour for three utterance types. The upper panel shows the speech waveform, the original (pink-crossed) and matched contour (blue-solid). The lower panel consists of two parts: the phrase command onset and magnitude, and accent command onset, offset and amplitude. Initial Final µ: µ: µ: σ: σ: σ: µ: µ: σ: σ: From the study of accent commands it is observed that intonation of most of the prosodic words can be modeled using a single negative accent command except that of the final prosodic word of Interrogatives, which requires an additional positive accent command (µ: and σ:

5 0.078). It is also observed that in Declaratives the negative accent command amplitude (A a ) increases from the phraseinitial position to phrase-medial positions. On the other hand, the amplitude of the negative accent command in Interrogatives decreases from the phrase-initial position to phrase-initial positions. In Imperatives, the negative accent command amplitude remains more or less the same within a prosodic phrase (Figure 3). As for the onset and offset times of the accent commands, it was observed that there is no significant difference across the three types of utterances in Bangla and the value of onset and offset times are almost identical to those already reported for declaratives in [10]. Figure 3: Negative accent command amplitude (A a ) of Declaratives, Imperatives and Interrogatives within a prosodic phrase Speech Synthesis and Verification of Results The synthesized F 0 contour was evaluated using two methods: a) the objective method, and b) the subjective method. A set of 25 Bangla sentences of each type (not included in the analysis data) is used for the evaluation Method of analysis In the objective evaluation, the original F 0 contour is compared with the synthesized F 0 contour and the root mean squared value of the difference is calculated in log F 0. For the subjective evaluation, 5 subjects - 2 males (L1, L2) and 3 females (L3, L4, and L5) - were selected. All subjects are native speakers of Standard Colloquial Bangla and are not speech experts. In order to verify the results of the analysis, 25 synthetic stimuli of each type were generated using the ESNOLA-based Bangla Text to Speech system developed by C-DAC, Kolkata. The stimuli were ordered randomly and were presented to the subjects for categorizing the sentence type as Declaratives, Imperatives or Interrogatives. The subjects were also asked to provide an indication of the degree of confidence of judgment on a 5-point scale (1: least confident, 5: most confident). The steps which are required for the generation of F 0 contour is described below: Step 1: Determine the sentence type, i.e., whether it is Declarative, Imperative or Interrogative, on the basis of parts of speech tag and punctuation mark. Step 2: Select the value for base-frequency based on the sentence type (Table 3). Step 3: Predict the position of the phrase command. Step 4: Select the value for the magnitude (Table 5) and lead time (Table 6) of the phrase command on the basis of sentence type and position of the phrase command. Step 5: Predict the position and type of the accent command by identifying the prosodic words [10]. Step 6: Select an appropriate value for the amplitude of the accent command based on the sentence type and its position within a phrase. Step 7: Selection of t 1 and t 2 values based on the position and length of the prosodic word [10]. Step 8: Using the command-response model, synthesize the F 0 contour based on the values predicted by the above steps Results The objective evaluation of the synthesized F 0 contour reveals that the root mean squared difference per sample of the synthesized and original contour for the above 75 sentences is quite small (0.051). The result of subjective evaluation is presented in Table 7. It is evident from the table that majority of the identified sentence type matches their intended type. Declarative stimuli are identified as intended 92.6 % of the times, while 7.4% of the time they are identified as either Interrogatives or Imperatives. The mean certainty rating for declarative stimuli is 4.32 and it is highly significant for Declaratives (p<0.0001, F=18.12). Imperatives have the identification rate of 84.5%. They are mostly confused with Declaratives (12.3%) and sometimes with Interrogatives (3.2%). The mean certainty rating for imperative stimuli is 3.70 and it is significant for Imperatives (p<0.0001, F=34.8). Interrogatives stimuli are identified as intended 90.2% of the time, while 9.8% of the time they are categorized as either Declaratives or Imperatives. The mean certainty rating for interrogative stimuli is 4.52 and it is also highly significant for Interrogatives (p<0.0001, F=24.37). Table 7. Identification result in percentage. Declaratives Imperatives Interrogatives

6 5. DISCUSSION AND CONCLUSION The present study first showed prosodic differences that exist among the three major utterance types of Bangla in terms of gross features of F 0 contours. The differences were then more precisely analyzed and quantified using the command-response model, and expressed in terms of differences in its model parameters. The results were further used for generating F 0 contours of synthetic speech and were perceptually verified, proving the usefulness of the command-response model in analysis as well as in highquality speech synthesis of Bangla. This encourages us to explore further along this direction, and analyze F 0 contours of other types of Bangla utterances for the purpose of highquality text-to-speech synthesis. 6. ACKNOWLEDGEMENTS The authors of this paper are greatly thankful to all the informants and evaluators for their unconditional participation in the experiments. 7. REFERENCES [1] Rangarajan, Vivek., Narayanan, S., Bangalore, S., Acoustic-Syntactic Maximum Entropy Model For Automatic Prosody Labeling, Proc. of the IEEE/ACL Workshop on Spoken Language Technology, pp , [2] Silverman, K., Beckman, M., Pitrelli, J., Ostendorf, M., Wightman, W. C., Price, P., Pierrehumbert, J., Hirschberg, J., ToBI: A Standard Scheme for Labeling Prosody, Proc. of the Second International Conference on Spoken Language Processing, Banff, Canada, , [3] [4] Fujisaki, H., Hirose, K., "Analysis of Voice Fundamental Frequency Contours for Declarative Sentences of Japanese," J. Acoust. Soc. Japan (E), vol.5, no.4, pp , [5] Fujisaki, H., Information, Prosody, and Modeling, Proc. of Speech Prosody 2004, Nara, Japan, 1-10, [6] Bulyko, I., and Ostendorf, M., Joint Prosody Prediction and Unit Selection for Concatenative Speech Synthesis, Proc. Of ICASSP, [7] Hasegawa-Johnson, M., Chen, K., Cole, J., Borys, S., Kim, S.-S., Cohen, A., Zhang, T. Choi, Y., Kim, H., Yoon, T.-J., Chavara, S., Simultaneous Recognition of Words and Prosody in the Boston University Radio Speech Corpus, Speech Commun., vol. 46, pp , [8] Nöth, E., Batliner, A., KieBling, A., Kompe, R., and Niemann, H., VERBMOBIL: The Use of Prosody in the Linguistic Components of a Speech Understanding System, IEEE Trans. Speech Audio Process. vol. 8, no. 5, pp , Sep [9] Aguero, P.D., Adell, J., Bonafonte, A., Prosody Generation for Speech-to-Speech Translation, Proc. of ICASSP, [10] Das Mandal, S. K., Warsi, A. H., Basu, T., Hirose K., Fujisaki, H., Analysis and Synthesis of F 0 Contours for Bangla Readout Speech, Proc. of Oriental COCOSDA 2010, Kathmandu, Nepal, [11] Das Mandal, S. K. and Datta, A. K., Epoch Synchronous Non-OverLapping Add (ESNOLA) Method Based Concatenative Synthesis System for Bangla, Proc. of 6th ISCA Workshop on Speech Synthesis, University of Bonn, Germany, , 2007.