VoIP Text. A study into the effectiveness of speech-to-text as an assistive tool in VoIP communications. Phase 1 Report

Transcription

1 VoIP Text A study into the effectiveness of speech-to-text as an assistive tool in VoIP communications Phase 1 Report 1

2 Introduction...3 Software Licensing Issues...3 Libjingle Licence...3 GNU GPL (1/2/3) Licence...4 Software Evaluation...4 Analysis...5 Summary of Results...6 Directions Example...6 Conversation Example...6 Conclusions...7 APPENDIX A...8 Example 1: Travel Directions Results...8 APPENDIX B Example 2: Conversation Results

3 Introduction The spread of Voice over Internet Protocol (VoIP) services, equipment and clients is transforming telephony worldwide. In addition to providing very inexpensive, or even free, international telephone calls there is potentially additional benefit in using computer networks to facilitate telephony. Currently hearing impaired and deaf users are excluded from VoIP services; unless the message is in text form to begin with the hearing impaired user cannot access these services effectively. A current draft ETSI standard (ETSI ES V0.0.6r3 ( )) outlines suggested acceptable performance measures for such a system but as yet these have not been verified. This project looks at the feasibility of utilizing speech to text software in order to generate text from natural speech over VoIP and so improve accessibility. The project will assess the accuracy of current state of the art Automatic Speech Recognition (ASR) software and also investigate the rationale behind the acceptable performance measure suggested by the draft ETSI document and assess if this is a reasonable level of performance. During phase 1 of this work, documented here, University of Salford has developed a software VoIP, voice chat, client based on open source libraries intended to assess the potential for accessible access to VoIP. The client has been implemented utilizing GoogleTalk open source libraries and has been extensively debugged so providing a robust test bed for formal assessment of whether the current state of the art of Automatic Speech Recognition (ASR) is sufficient to provide benefits for hearing impaired and deaf users. The scenario considered most useful, and that which is least catered for currently, is that of informal contact with friends and family which may be considered too personal to utilize services involving a 3 rd party transcriber and too trivial to make the expense of such services appropriate. As part of the development an informal assessment has been carried out that has yielded results indicating that the software may indeed be useful in this context. Formal assessment is planned for phase 2 of the project. Software Licensing Issues The software that has been developed during phase 1 has made use of open source libraries that impact on the resulting programs licensing and distribution. All of these have been chosen so that the software can be freely distributed with or without source code. For details see below. Libjingle Licence libjingle Copyright , Google Inc. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 3

4 Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. This is essentially a BSD license. The typical BSD license contains 3 major clauses, allowing unlimited redistribution for any purpose as long as its copyright notices and the license's disclaimers of warranty are maintained. The license also contains a clause restricting use of the names of contributors for endorsement of a derived work without specific permission. GNU GPL (1/2/3) Licence Please see: These libraries are fully open source, can be distributed with or without source code, modifications can be made without releasing the updates, but the software cannot be sold without releasing the updated GPL source code. There's nothing in there that stops the software from being: Given away, for free, without the source code (Freeware) Given away, for free, with the source code (Opensource) It would, however, be contrary to the licence to sell the software without the source code. Software Evaluation The software has been tested on Windows and, more recently, in Windows 7. After considerable debugging and several versions the software functions well without crashes or errors on all five of the PCs on which testing was carried out. An informal preliminary evaluation of the software has been carried out as part of phase 1 prior to full evaluation in phase 2. The aim of this testing was not a full assessment of the speech engines (scheduled to take place in phase 2) but instead to assess the potential of speech to text in the context of VoIP software. Two genres of conversation were assessed for this evaluation: firstly a series of detailed directions, and, in the second example, typical phrases that might form a 4

5 more informal conversation. The second of these is considered to be more typical of likely applications of the software, the first is considered to be more challenging for the speech recognition engine. Example 1 To get to the train station, take your first right, then walk down this road for about 5 minutes. You'll see a large red building on your left, and a police station to your right, head right past the police station, and then continue up this road for about 6 minutes. Stop when you come to a coffee shop on your left hand side. You are close to the station now, turn right, then take your second right and continue straight on, you'll be able to see the train station from here, just keep walking straight ahead. Example 2 Hello Auntie how are you today? Sebastian is now five years old He starts school this year I hope dinner was nice We had chicken and rice, it was very nice Although the chicken was a little chewy it was still a very nice meal I think you can get a cheap flight on August 15 th It will only cost you 300 pounds which I think is very cheap Hopefully the weather won't be too bad if you decide to come it's been raining all winter Last week it rained every day although, saying that, it's actually quite nice outside now Still, it's not as hot as where you live! Maybe we should come over there to visit instead It would be nice to see Gary and Steve again It was nice to talk to you again Hopefully we will see you soon Give my love to Hannah Goodbye! Analysis In this analysis the number of incorrect words was based on the number of words originally spoken and misinterpreted by the speech recognition software. Incorrect words were counted, extra words added were not counted as extra words incorrectly identified. For example the word, Sebastian, when interpreted as, so bastion is considered as a single incorrect word. This decision was taken as otherwise it would in theory be possible to have more words incorrectly identified than were originally spoken. It is important to point out that these informal tests were carried out using a single voice and that the results cannot be extrapolated further. Tests planned for phase 2 will utilise multiple voices, both male and female, and are expected to provide more useful statistical data. 5

6 In all cases the full original text and interpreted text is included in Appendices A and B. Summary of Results Directions Example ASR Engine % Incorrect ASR (untrained) 55.7 ASR (trained) 27.8 Naturally Speaking ASR (untrained) 49.5 Naturally Speaking ASR (trained) 8.2 ASR (untrained) 26.8 Conversation Example ASR Engine % Incorrect ASR (trained) 6.0 Naturally Speaking ASR (trained) 4.0 ASR (untrained)

7 Conclusions The software has been demonstrated to be effective in providing speech to text capability for VoIP, in this case a voice chat client application based on GoogleTalk. The software has been assessed on 2 different operating systems, Windows and (RC1). It is clear from the results documented here that in these cases training the ASR engines has provided dramatic improvements in performance. In both cases ( and Naturally Speaking) the training that was carried out was the minimal training permissible, taking around 5 minutes for each ASR engine. Longer training programmes are available and may well provide further benefit. This will be investigated further as part of phase 2 of the project. The results indicate that the built in speech recognition engine, although claimed to be an improvement on that bundled with Windows XP, has poorer performance in this test than the current leading commercial application, Naturally Speaking. In both tests Naturally Speaking using a trained profile performed well with over 90% accurate word recognition. The ASR performed very well in the more typical conversational exchange but failed to provide good results in the more challenging task. One interesting feature of this informal assessment is the strong performance of the built in ASR client using an untrained profile. The limited testing in this phase means that it is not possible to compare this with other untrained results however it indicates that may possibly provide good trained performance at a zero cost option, once the cost of the operating system itself is met. This can be explored further in phase 2 of the project. 7

8 APPENDIX A Example 1: Travel Directions Results In this example the conditions compared were as follows: ASR, both trained and untrained, Naturally Speaking 10 ASR, both trained and untrained, untrained using RC1 version. Both the and Naturally speaking results come from untrained and trained profiles, the results are from an untrained profile using the RC1 version. To get to the train station, take your first right, (untrained) (trained) (untrained) (trained) (untrained) To get the train station at your own right To get to the train station pain of birthright forget the train station in the right to get to the train station take your first right to get to the translation take your first ride then walk down this road for about 5 minutes. (untrained) (trained) (untrained) (trained) (untrained) and what down the road for about a minute and what down this road for about 5 minutes and what Dallas wrote about back then and walk down this road for about five minutes in what comes rolled from of 5 minutes You'll see a large red building on your left, (untrained) (trained) (untrained) (trained) (untrained) you're here like that building on your lap your fear large made building on your left you'll feel I read building until you'll see a large red building on your left you'll see a larger building on your left and a police station to your right, (untrained) (trained) (untrained) (trained) (untrained) and believe they can do right and a police station to your right a police station to your right and a police station to your right and the police station to your right head right past the police station, (untrained) (trained) (untrained) (trained) (untrained) had right at the polling station had tried to pass the police station head right at the police station head right at the police station to have a drive past the police station and then continue up this road for about 6 minutes. 8

9 (untrained) (trained) (untrained) (trained) (untrained) and then continue of the world about a minute and a continual of this world for about 6 minute and they continue at this road are about six minute and then continue up this road for about six minute and then continual visual for about 6 minutes Stop when you come to a coffee shop on your left hand side. (untrained) (trained) (untrained) (trained) (untrained) goal when you come to a proper job on your left hand light start when you come to a coffee shop on your left hand side stop when you come to a copy shop on your left and the start when you come to a copy shop on your left hand side stopped when you come to a coffee shop on your left hand side You are close to the station now, turn right, (untrained) (trained) (untrained) (trained) (untrained) you are close to the nation now her like you are close to the station now turn right good motivation now turn right you are close to the station now turn right and you are close to the station LU turn right then take your second right and continue straight on, (untrained) (trained) (untrained) (trained) (untrained) a at all that can write and continued account and take your back and right and continue straight on into your second right and continue straight on and take your second right and continue straight on and take your second rate and continue straight on you'll be able to see the train station from here, (untrained) (trained) (untrained) (trained) (untrained) you'll be able to claim they can come and get what they had your be to see the train station from here the elderly the train station from here you be able to see the train station from here your be able to see the train station from here just keep walking straight ahead. (untrained) (trained) (untrained) (trained) (untrained) heard that go well with the a M. Bacher the had a go at to keep walking straight ahead [no text] to sleepwalking straight ahead just keep walking straight ahead 9

10 APPENDIX B Example 2: Conversation Results Example "Real world" Conversation Both the and Naturally speaking results come from trained profiles, the results are from an untrained profile using the RC1 version. Hello Auntie Hello auntie hello auntie Hello it How are you today? How are you today how are you today how are you today Sebastian is now five years old So bastion is now five years old Sebastien is now five years old as a bastion is now five years old He starts school this year If he starts school this year list at school this year east at school this year I hope dinner was nice I hope dinner was nice I hope dinner was nice I hope dinner was nice We had chicken and rice, it was very nice we had chicken and rice it was very nice We had chicken and rice it was very nice we had chicken and rice it was very nice Although the chicken was a little chewy Although the chicken was a little too weak Although the chicken was a little chewy other than taking was a little chilly It was still a very nice meal it was still a very nice meal it was still a very nice meal it was still a very nice meal 10

11 I think you can get a cheap flight on August 15th I think you can get a cheap flight on August 15 I think you can get a cheap flight on August 15 I think you can get a cheap flight on August 15 It will only cost you 300 pounds It will only cost you 300 pounds it will only cost you 300 it will only cost you 300 pounds Which I think is very cheap Which I think is very cheap which I think is very cheap which I think he's buried G Hopefully the weather won't be too bad if you decide to come Hopefully the weather won't be too bad if you decide to call hopefully the weather will be too bad if you decide to call hopefully the weather will be too bad if you decide to come It's been raining all winter It's been raining or winter it's been raining all winter it's been raining or winter Last week it rained every day Last week in rained every don't last week it rained every day last week in rained every day Although, saying that, it's actually quite nice outside now Although in saying that it's actually quite nice outside now although saying that it's actually quite nice outside now although saying that it's actually quite nice outside now Still, it's not as hot as where you live! Still it's not as hot as where you live still is not as hot as where you live still is not as part is where you live Maybe we should come over there to visit instead Maybe we should come over there to visit instead mate we should come over there to visit instead maybe we should come over there to visit instead It would be nice to see Gary and Steve again it would be nice to see Gary and Steve again 11

12 it would be nice to see Gary and Steve again he'll be nice to see Gary and Steve again It was nice to talk to you again It was nice to talk to you again it was nice to talk to you again it was nice to talk to you again Hopefully we will see you soon Hopefully we will see you soon hopefully we will see you soon hopefully we will see you soon Give my love to Hannah Give my love to hammer give my love to Hannah give my love to hander Goodbye! Goodbye goodbye goodbye 12