How To Create An Emoton Recognzer

Transcription

1 Recognzng Low/Hgh Anger n Speech for Call Centers FU-MING LEE*, LI-HUA LI, RU-YI HUANG Department of Informaton Management Chaoyang Unversty of Technology 168 Jfong E. Rd., Wufong Townshp Tachung County, 41349, Tawan (R.O.C.) Abstract: - Automatc mult-level anger recognton n speech s an mportant factor to enhance user satsfacton for call centers. In ths research, three emotonal states, namely, neutral, low anger, and hgh anger of acted corpora wth telephone qualty are specfed for emoton recognton. The corpora are collected from amateur actors and, thereafter, verfed by the actors themselves. The emoton recognzer s mplemented by usng Back-propagaton Network (BPN) wth acoustc features of speech. Due to the varaton of expresson methods by dfferent person, the feature values of the tranng used are too complex to make the BPN model convergent. To overcome the problem, a codfed method s developed to smplfy the feature values. Wth the codfed nputs, the BPN model and a comparatve Decson Tree C5.0 gve qute satsfactory test performances for anger recognton. Therefore, they can be used as a part of a decson support system for proper applcatons n call centers. Key-Words: - Mult-level anger recognton, call center, acoustc features, Back-propagaton Network 1 Introducton Recently, there has been an ncreasng nterest n automatc emoton recognton n speech [9][13][15] [17][18], especally the negatve emoton recognton. One motvaton comes from the desre to develop human machne nterfaces that are more adaptve and responsve to a user s behavor [5][9][10]. The other motvaton comes from the need of a varety of applcatons for busness [11][13][18]. In partcular, n a call center envronment, anger s dentfed as the negatve and the most mportant emoton. Three types of spoken materal, namely, acted, nduced, and real-lfe corpora, are generally collected for emoton analyss and recognton [2][7]. To test the performance of the emoton recognzer, we must know the actual emotonal state of each test example of the corpora. Although, nduced and real-lfe corpora are more natural than acted one, the state of each example of those corpora s unknown unless the speaker tells us the true answer rght after the corpora beng collected. In servce practces, t s almost mpossble to get the answer. Even f the state s labeled by several ndependent labelers, the nter-labeler agreement s very low [5]. Furthermore, even f all labelers acheve agreement, the answer may be wrong. On the contrary, the state of each example of acted corpora can be easly verfed by the speaker. Therefore, n ths research acted corpora are collected for buldng and evaluatng the recognton model. Features of the speech sgnal need to be extracted for automatc emoton recognton. There are large numbers of lngustc (lexcal and dalogc) features and paralngustc (acoustc, fluent, etc.) features that can be attrbuted to the emotonal state of the sgnal [7]. Among them acoustc features are the most wdely employed [9][13][19][20]. In ths research lngustc features are excluded snce the sentences spoken n acted corpora are controlled and expressed wth the whole spectrum of consdered emotons. Furthermore, because there s no agreement on the best set of relevant acoustc features for emoton recognton, our strategy s to use as many features as possble. Many types of emoton recognzer of spoken materal have been used, e.g. Artfcal Neural Networks (ANN) [3][11][13][14][15][20], K-nearest Neghbors [6][9][10][11][13][15][16][20], Decson Trees [7][16], Gaussan models [5][12][15], and Support Vector Machnes [7][11][15][16][18][20]. However, the most effcent model s stll not well establshed and from publshed results appears to be data-dependent [18]. Because the major capablty of ANN s ther flexblty n approxmatng operaton between nputs and outputs and one of the most popular models n ANN s Back-propagaton * Correspondng author (E-mal: fmlee@cyut.edu.tw; TEL: ext. 4286) ISSN:

2 Network (BPN), the recognton model s mplemented by usng BPN n ths research. Furthermore, the performance of BPN s compared wth the performance of Decson Tree C5.0, whch s one of the most wdely used recognzer. The goal of ths research s to create an emoton recognzer that can process telephone qualty voce messages n real tme and can be used as a part of a decson support system for proper applcatons n call centers. Three emotonal states, namely, neutral, low anger, and hgh anger are consdered for emoton recognton. Ths dstncton s motvated by the wsh to recognze low/hgh anger, especally low anger, whch wll be handled by the conclaton strateges. In our opnon mult-level anger recognton s an mportant factor to enhance caller satsfacton. However, recognton results are sparse n the lterature. In the work of Burkhardt et al. [5], prosodc features of real-lfe corpora are extracted; an algorthm based on Gaussan denstes are used to recognze emotons; the accuracy of the recognzer for no anger, low anger, and hgh anger emotons are 89%, 49% and 16%, respectvely, whch shows that the organzer does not dstngush between these emotons very well. In ths research, the acted corpora are collected from 48 amateur actors and, thereafter, verfed by the actors themselves. From the corpora, 1401 are created and are parttoned nto tranng set and test set,.e and 294, respectvely. Each data set has even for three emotonal states. From each example, 47 acoustc features are extracted by usng the Praat program [4] and, then, are normalzed for developng and evaluatng the emoton recognzers. A standard F1 measure s used to assess the performance of the emoton recognzers. Due to the varaton of expresson methods by dfferent people, the BPN model can not converge n tranng stage. To overcome the convergent problem, a codfed method s proposed to smplfy the complexty of the normalzed feature values. Thereafter, the BPN model wth codfed nputs s convergent; F1 measures of the model for test wth neutral, low anger, hgh anger, and anger (.e. low anger or hgh anger) emotonal states are 71.5%, 47.0%, 64.4%, and 86.0%, respectvely. F1 measures of Decson Tree C5.0 wth codfed nputs for test wth neutral, low anger, hgh anger, and anger emotonal states are 66.0%, 41.2%, 64.6%, and 82.4%, respectvely. The results show that the performance of anger emoton detecton s satsfactory. However, the performance of low anger emoton detecton s poor snce low anger s often confused wth neutral or hgh anger. 2 Corpora Collecton In ths research, three emotonal states, namely, neutral, low anger, and hgh anger of acted corpora are specfed for emoton analyss and recognton. The followng s the detals about the corpora and the collectng method. Recordng equpments: Each utterance of the corpora s recorded by usng a close-talk mcrophone wth a recorder equpped n Mcrosoft Wndows. To smulate the voce n telephone, the format of the recorded utterance s set at 8000 Hz, 8 bt, and mono. Amateur actors: There are 25 male actors and 23 female actors partcpatng n ths actvty. They are undergraduate or graduate students wth age between 20 and 30. Gven sentences: Thrty short sentences are collected from the FAQs of webstes of two telecommuncaton servce provders n Tawan. Each actor s asked to select ten of them whch he (or she) supposes that t would be easer for hm (or her) to speak at all three emotonal states consdered n ths research. Recordng envronment: The recordng actvty takes place at a normal offce. In a recordng case, an actor speaks successvely a sentence three tmes wth neutral, low anger, and hgh anger emotonal states, respectvely. Qualty verfcaton: Rght after a recordng case the actor s asked to verfy the qualty of the utterance by playng t back. Unless the actor feels that the states of the utterance he portrays on demand are ndeed consstent wth the states he usually acts n the real lfe, the case wll be recorded agan. Emotonal : Three emotonal of an utterance are cut by a research member of us. After examnng all the utterances, we fnd and dscard 13 utterances wth hgh background noses. Thus, the corpora of 1401 are created wth 467 per emotonal state. 3 Features Extracton In ths research, we ntend to extract the acoustc features, namely, ptch, ntensty, formant, pulse, shmmer, jtter, harmoncs to nose rato (HNR), and duraton, by usng the Praat program [4]. Before features extractng, every emotonal example s fltered by a band-pass flter of the Praat program wth band frequency set between 200 Hz and 3200 Hz whch s close to the band frequency of voce n telephone. In features extractng, wndow length s ISSN:

3 set at 0.02 seconds. The formant frequences are estmated by usng the lnear predcted codng (LPC) method. The predcton order s set as 16. Ptch floor and ptch celng are set accordng to the gender of the actor. For male s emotonal example, the 2 parameters are set at 75 Hz and 300 Hz, respectvely. For female s emotonal example, the 2 parameters are set at 100 Hz and 500 Hz, respectvely. The rest parameters n the Praat program are set as default values. The followng s the detals about the 47 features extracted from each emotonal example. Ptch features: A ptch object represents perodcty canddates as a functon of tme,.e. a ptch contour. Related ptch features are statstcal propertes of the ptch contour. Sx ptch features are used n ths research: maxmum, mnmum, mean, standard devaton, mean absolute slope, mean absolute slope wthout octave jumps. Intensty features: An ntensty object represents an ntensty contour at certan lnearly spaced tme ponts. Related ntensty features are statstcal propertes of the ntensty contour. Four ntensty features are used n ths research: maxmum, mnmum, mean, standard devaton. Formant features: A formant object represents spectral structure as a functon of tme,.e. a formant contour. Related formant features are statstcal propertes of the formant contour. Twenty formant features are used n ths research. The followng s the detals about the features. Frst formant (F1): maxmum, mnmum, mean, standard devaton. Second formant (F2): maxmum, mnmum, mean, standard devaton. Thrd formant (F3): maxmum, mnmum, mean, standard devaton. Fourth formant (F4): maxmum, mnmum, mean, standard devaton. Bandwdths: B1, B2, B3, B4. Pulse features: It s generated at every pont n the pont process. Two related pulse features are used n ths research: mean perod, standard devaton perod. Jtter features: Fve jtter features are used n ths research: jtter (local), jtter (local, absolute), jtter (rap), jtter (ppq5), jtter (ddp). The defntons of these features please referred to the Praat program [4]. Shmmer feature: Fve shmmer features are used n ths research: shmmer (local), shmmer (apq3), shmmer (apq5), shmmer (apq11), shmmer (ddp). The defntons of these features please referred to the Praat program [4]. HNR features: It represents the degree of acoustc perodcty. Four related HNR features are used n ths research: maxmum, mnmum, mean, standard devaton. Duraton feature: It represents the rato of voced frame s number and unvoced frame s number. 4 Features Preprocessng To tran the emoton recognzer and test ts performance, the 1401 emotonal are parttoned nto tranng set and test set. The tranng set conssts of 1107 emotonal created by randomly chosen 20 male actors and 18 female actors; the test set conssts of 294 emotonal created by the remanng 5 male actors and 5 female actors. Because the unts of the features are dfferent and the ranges of the features values of the emotonal are dverse, all of the 47 features are normalzed as follows. For all e tranng set test set, let f e f ( e) f,mn ( ) =, f,max f,mn 1, f f ( e) > 1, f ( e) = f ( e), f 0 f ( e) 1, 0, otherwse, (1) where f (e) denotes the th feature value of emotonal example e, f( e ) denotes the normalzed value of f (e), f,max denotes the 2nd maxmum value of {f (e): e tranng set}, f,mn denotes the 2nd mnmum value of {f (e): e tranng set}. To avod to get unusual extreme feature values, both f,max and f,mn are determned as the 2nd extreme values mentoned above. Furthermore, to conform to practces, we suppose that the nformaton about the test set s unknown n advance. Therefore, both f,max and f,mn are determned only based on emotonal of tranng set. 5 Performance Measure In ths research, F1 measure s used to assess the performance of recognzers. It has been used very commonly as a standard performance measure n the recognton problem, and s defned as follows. 2PR nc nc F1 =, R =, P = P + R m m (2) c c ISSN:

4 where n c denotes the number of test whose emotonal state s correctly state c, m c denotes the number of test wth emotonal state c, and m' c denotes the number of test whose emotonal state s emoton state c. 6 BPN wth Normalzed Inputs The BPN model proposed n ths research s depcted on Fg. 1. It conssts of three knds of layers,.e. nput layer, hdden layer, and output layer. The nputs conssts of 47 normalzed features. The number of hdden nodes s set as 25. The number of output nodes s set as the number of emotonal states. In tranng stage, the target values of the tranng example for the outputs are set accordng to the emotonal state of the example descrbed n Table 1. In test stage, the recognzed emotonal state of the test example s determned accordng to the output wth hghest value. For example, f y l (e) has the hghest value, the emotonal state of the test example s low anger. f1( e) f2( e) f ( e) 47 Fg. 1 The BPN model In the experments, all the 1107 tranng are used to tran the BPN model by usng the Matlab s Neural Network Toolbox. However, the BPN model can not converge. To fnd out the reason, the tranng set and the test set are regrouped by dscardng wth certan emotonal state; the 47 features are preprocessed as (1) wth regrouped tranng set and test set; the BPN model s reformed by dscardng the output node correspondng to the certan emotonal state; the target values of the tranng example for the remanng 2 outputs are set as shown n Table 1, and the reformed BPN model s traned by usng the regrouped tranng. The reformed BPN model can not converge ether n the case where hgh anger emotonal are dscarded or n the case where neutral emotonal are dscarded. Nevertheless, n the case where low anger emotonal are dscarded, the reformed BPN model s convergent and ts performance s shown n Table 2. The F1 measures for test wth neutral or hgh anger emotonal states are above 85%. Table 1 Target value of tranng example for outputs output y emotonal state n (e) y l (e) y h (e) neutral low anger hgh anger Table 2 Performance of BPN wthout node y l (e) hgh measures (%) neutral anger P R F1 neutral hgh anger In collectng the acted corpora, we observe that the ways to speak a gven sentence wth neutral, low anger, and hgh anger emoton are qute dfferent for dfferent person. Thus, for each feature, the feature values of the three emotonal and the patterns of the values dstrbuton are qute dfferent for varous persons. Furthermore, for the three emotonal states, some actors expresson methods are hard to dscrmnate, especally the methods between neutral and low anger or between low anger and hgh anger. Ths s the reason why the BPN model and the reformed BPN models wth node y l (e) can not converge, and the reformed BPN model wthout node y l (e) are convergent and has excellent test performance. Because the qualty of the s verfed by the actor of the rght after a recordng case, the low anger are stll vald and should be consdered. To overcome the convergent problem of the BPN model, the complexty of the normalzed feature values of the emotonal s smplfed by codfyng method descrbed n the next secton. 7 Features Codfcaton To codfy th normalzed feature value of any emotonal example, th normalzed feature values of the 1107 tranng are collected. They are ordered by ther values and are parttoned nto fve segments depcted on Fg. 2, n whch four partton ponts are determned as (3). Fg. 2 Partton ponts for the th normalzed feature f, s, lp + f, s+ 1, fp p, s =, for = 1, 2,..., 47, s = 1, 2, 3, 4, and (3) 2 for the th normalzed feature p,s denotes the sth partton pont, f, s, lp denotes the value of the last pont at segment s, f, s+ 1, fp denotes the value of the frst pont at segment s+1. ISSN:

5 Every normalzed feature value of an emotonal example s codfed nto four dgts defned n Table 3. For nstance, when th normalzed feature value of an emotonal example belongs n certan segment depcted on Fg. 2, the value s codfed nto four bnary dgts correspondng to the segment. In Table 3, the bnary codes are desgned such that the hammng dstance of any two codes represents the segment dstance of the correspondng two segments. For example, the hammng dstance of two codes correspondng to segment 1 and segment 5 s 4. The detals are descrbed as n Table 4. Table 3 The bnary code for each segment segment seg. 1 seg. 2 seg. 3 seg. 4 seg. 5 bnary code Table 4 The hammng dstance of any two codes segment seg. 1 seg. 2 seg. 3 seg. 4 seg. 5 seg seg seg seg seg BPN wth Codfed Inputs To accommodate the codfed nputs, the BPN model s redesgned wth 188 bnary nput nodes and 95 hdden nodes. The number of outputs and the target value of the tranng example are desgned by the same way for the BPN model mentoned n Secton 6. The BPN model wth codfed nputs s traned by the 1107 tranng. Fnally, the BPN model wth codfed nputs s converged and the performance of the model s shown n Table 5. The F1 measures for test wth neutral, low anger, and hgh anger emotonal states are 71.5%, 47.0%, and 64.4%, respectvely. Table 5 Performance of BPN wth codfed nputs neutral low anger hgh measures (%) anger P R F1 neutral low anger hgh anger Table 6 Performance of BPN wth codfed nputs for neutral and anger emotons measures (%) neutral anger P R F1 neutral anger As mentoned n Secton 6, low anger emotonal are often confused wth neutral or hgh anger emotonal. Thus, although the convergent problem of the BPN model can be overcome by codfyng the complex normalzed features, the performance of the BPN model wth codfed nputs are stll not good for the three emotonal states, especally low anger. However, wth respect to applcatons n call center, the performance for anger,.e. low anger or hgh anger, emoton recognton s the man concern. Therefore, after combnng the results of both low anger and hgh anger, the performance of the BPN model wth codfed nputs s recalculated as Table 6. The F1 measure for test wth anger emotonal state s 86.0%. To compare the performance of the BPN model wth codfed nputs, Decson Tree C5.0 wth codfed nputs s developed and evaluated wth the same tranng set and the test set by usng SPSS Clementne. The performance of Decson Tree C5.0 s shown n Table 7 and Table 8. The F1 measures for test wth neutral, low anger, hgh anger, and anger emotonal states are 66.0%, 41.2%, 64.6%, and 82.4%, respectvely. The results of the two models show that the performance of anger emoton detecton s satsfactory. However, the performance of low anger emoton detecton s poor snce low anger s often confused wth neutral or hgh anger. From the experments, we notce that the BPN model s superor to the Decson Tree C5.0. Table 7 Performance of Decson Tree C5.0 wth codfed nputs neutral low anger hgh measures (%) anger P R F1 neutral low anger hgh anger Table 8 Performance of Decson Tree C5.0 wth codfed nputs for neutral and anger emotons measures (%) neutral anger P R F1 neutral anger Conclusons Automatc mult-level anger recognton n speech s an mportant factor to enhance user satsfacton for call centers. To recognze the level of anger, one dffculty comes from the lack of clear defnton or descrpton to dstnct the levels of anger and the other dffculty comes from the need of actual emotonal state nformaton of the telephone speech. In our opnon, the caller hmself s the only person who exactly knows the answer and the emotonal state of the caller s the man concern of call centers. In ths research, neutral, low anger, and hgh anger emotons are consdered. To have the exact emotonal state nformaton of speech, the acted corpora are verfed by the actor hmself. Due to the varaton of expresson methods by dfferent people, the feature values of the tranng ISSN:

6 used are too complex to make the BPN model convergent. To overcome the problem, a codfed method s developed to smplfy the feature values. Wth the codfed nputs, the results of the BPN model and a comparatve Decson Tree C5.0 show that the performance of anger (.e. low anger or hgh anger) emoton detecton s satsfactory. Therefore, they can be used as a part of a decson support system for proper applcatons n call centers. References: [1] J. Adell, A. Bonafonte, and D. Escudero, Analyss of Prosodc Features: Towards Modelng of Emotonal and Pragmatc Attrbutes of Speech, XXI Cogreso de la Socedad Española para el Procesamento del Lenguaje Natural, [2] A. Batlner, K. Fscher, R. Huber, J. Splker, and E. Nöth, Desperately Seekng Emotons or: Actors, Wzards, and Human Bengs, Proc. of The Inter. Speech Communcaton Assocaton Workshop on Speech and Emoton, 2000, pp [3] M. W. Bhatt, Y. Wang, and L. Guan, A Neural Network Approach for Human Emoton Recognton n Speech, Proc. of IEEE Inter. Symposum on Crcuts and Systems, Vol. 2, 2004, pp [4] P. Boersma and D. Weennk, Praat: Dong Phonetcs by Computer (Verson ), [5] F. Burkhardt, M. V. Ballegooy, R. Englert, and R. Huber, An Emoton-Aware Voce Portal, Proc. of The 16th Conf. on Electronc Speech Sgnal Processng, 2005, pp [6] F. Dellaert, T. Polzn, and A. Wabel, Recognzng Emoton n Speech, Proc. of The Inter. Conf. on Spoken Language Processng, 1996, pp [7] L. Devllers, L. Vdrascu, and L. Lamel, Challenges n Real-Lfe Emoton Annotaton and Machne Learnng Based Detecton, Neural Networks, Vol. 18, No. 4, 2005, pp [8] O.-W. Kwon, K. Chan, J. Hao, and T.-W. Lee, Emoton Recognton by Speech Sgnals, Proc. of The 8th European Conf. on Speech Communcaton and Technology, 2003, pp [9] C. M. Lee, S. S. Narayanan, and R. Peraccn, Recognton of Negatve Emotons from the Speech Sgnal, Proc. of Automatc Speech Recognton and Understandng, 2001, pp [10] C. M. Lee and S. S. Narayanan, Toward Detectng Emotons n Spoken Dalogs, IEEE Trans. on Speech and Audo Processng, Vol. 13, No. 2, 2005, pp [11] D. Morrson, R. Wang, and L. C. De Slva, Ensemble Methods for Spoken Emoton Recognton n Call-Centers, Speech Communcaton, Vol. 49, 2007, pp [12] D. Neberg, K. Elenus, and K. Laskowsk, Emoton Recognton n Spontaneous Speech Usng GMMs, Proc. of The 19th Inter. Conf. on Spoken Language Processng, 2006, pp [13] V. A. Petrushn, Emoton Recognton n Speech Sgnal: Expermental Study, Development, and Applcaton, Proc. of The 6th Inter. Conf. on Spoken Language Processng, 2000, pp [14] H. Sato, Y. Mtsukura, M. Fukum, and N. Akamatsu, Emotonal Speech Classfcaton wth Prosodc Parameters by Usng Neural Networks, Proc. of The 7th Australan and New Zealand Intellgent Informaton Systems Conf., 2001, pp [15] B. Schuller, G. Rgoll, and M. Lang, Speech Emoton Recognton Combnng Acoustc Features and Lngustc Informaton n a Hybrd Support Vector Machne - Belef Network Archtecture, Proc. of Internat Conf. on Acoustcs, Speech, and Sgnal Processng, Vol. 1, 2004, pp [16] M. Sham and W. Verhelst, An Evaluaton of the Robustness of Exstng Supervsed Machne Learnng Approaches to The Classfcaton of Emotons n Speech, Speech Communcaton, Vol. 49, 2007, pp [17] D. Ververds and C. Kotropoulos, Emotonal Speech Recognton: Resources, Features, and Methods, Speech Communcaton, Vol. 48, 2006, pp [18] L. Vdrascu and L. Devllers, Detecton of Real-Lfe Emotons n Call Centers, Proc. of The 18th Inter. Conf. on Spoken Language Processng, 2005, pp [19] S. Yldrm, M. Bulut, C. M. Lee, A. Kazemzadeh, C. Busso, Z. Deng, S. Lee, and S. Narayanan, An Acoustc Study of Emotons Expressed n Speech, Proc. of Internat Conf. on Spoken Language Processng, Vol. 1, 2004, pp [20] F. Yu, E. Chang, Y.-Q. Xu, and H.-Y. Shum, Emoton Detecton from Speech to Enrch Multmeda Content, Proc. of The 2nd IEEE Pacfc Rm Conf. on Multmeda, 2001, pp ISSN: