Developing speech recognition software for command-and-control applications

Transcription

1 Developing speech recognition software for command-and-control applications Author: Contact: Ivan A. Uemlianin Contents Introduction Workflow Set up the project infrastructure Describe the application Specify requirements Requirements for recogniser Acoustic Model requirements Language Model requirements Develop the application A note on testing and integration Language Model Acoustic Model Prepare training data Build Acoustic Model with SphinxTrain Initial configuration and setup Editing sphinx_train.cfg Lines 18f Lines 69f Line 120 Lines 90 & 101 Files & directories needed cyw.transcription cyw.fileids cyw.dic For a word model cyw.filler cyw.phone cyw/wav/ directory Running SphinxTrain Using the recogniser Package the application for distribution References Appendices Python script to derive project files from transcription Runall_wordmodel.pl 1

2 2 Introduction This document provides guidelines for developing the necessary speech recognition software for a commandand-control (C&C) application, for example a speech-driven calculator. As well as promoting best practice, the discussion will point out common errors and hazards. Some programs to automate parts of the process are given in appendices. This document assumes the use of the Sphinx speech recognition toolkit [sphx]. Sphinx is mature, open-source and freely available, with reasonable documentation and support. There are other toolkits avaiable, notably HTK [htk]. Some details will vary depending on which toolkit you use, but much of the design and planning will be the same. Workflow The five main stages of the project should run consecutively, as each stage depends on the previous stage having been more or less completed: Set up the project infrastructure Describe the application Specify requirements Develop the application Package the application for distribution The development stage (stage 4) is much more complex than the other stages, and some tasks can run concurrently. Development workflow diagram

3 3 In the workflow diagram above, boxes represent major tasks and arrows connecting boxes represent dependency (so, for example, "Specify Langauge Model (LM)" must be completed before "Build LM" can begin). One can imagine a timeline running down the diagram from top to bottom. Tasks which are next to each other are independent of each other, and can be dealt with at the same time. For example, development of the GUI and building the LM are completely unrelated and can be tackled in any order, or concurrently. This is an important point because recording the data is by far the longest and most labour-intensive task. As it involves finding speakers and scheduling recording sessions, the time taken may be partly outside of the control of the project team. On no account should work on the interfaces or the LM be postponed until the recording has been completed. Even parts of the Acoustic Model development can begin before all of the data has been collected. If you wait until finishing the recording before starting other parts of the development, the project may run out of time or budget, or unforeseen problems may arise, before the project can produce anything useful. Set up the project infrastructure The most important thing first of all is to set up an infrastructure to support the project's activities. This infrastructure should ensure that work is saved centrally and kept secure; it should allow co-workers to share work; it should keep track of previous or different versions of parts of the project (e.g., versions of development code); it should also include a task management system so that progress can be monitored either by a project manager or by the team as a whole. We used subversion [svn] for version control and trac [trac] for task management. These systems are mature and widely-used open-source systems. Subversion is versatile and easy to use, and can be set up on an external server to increase security (e.g., our external subversion server is backed up offsite daily). Trac provides a web browser interface to the subversion repository, a wiki for team sharing information, and a simple task management system, which can again be accessed through a web browser. Describe the application Before writing any code, before recording any data, and before making any detailed plans in those areas, it is very important to establish firmly a concrete idea of the application to be developed. One of the first items produced by the project should be a document describing the application which the project intends to develop. Depending on the complexity of the application, this document could be just a single page of A4 --- a couple of hundred words --- but it should be produced, and it should be understood and agreed on by the whole team. The document can focus on issues like why the team is developing the application, and what needs the application is to meet, but it can and should incorporate other levels of description. In particular, the document should include descriptions of how the application is to be used. In larger development projects, this document might end up as a set of dcuments addressing different aspects (e.g., the "business case", "usage scenarios", "operating context", and so on), but even the smallest project must begin with a definite concrete description of the target. Specify requirements Once we have a concrete idea of the target, we can set about deciding in detail what the software should do: we can specify the software requirements. The description document gives a foundation to decisions on requirements, and stops discussion becoming too open-ended. In a command-and-control application there are three main components:

4 4 there is the component which does the processing required (e.g., in a speech-driven calculator, the mathematical processing); there is the device as seen by the user (e.g., a graphical user interface on a computer desktop); there is the speech recogniser, which takes audio data (i.e., speech) as input as returns text as output (or other material, suitable for input to the processing component). In each application these components will be slightly different. The ways the components need to interact with each other will differ also: for example, should the applcation always be listening for user speech, or should it be a "push-to-talk" system, in which the user clicks a button to enable input? How should the application behave when the recogniser component fails to recognise the given input? Questions like this should be addressed now. A full discussion of software requirements specification is beyond the scope of this document (and there are many textbooks and websites available). Here I shall cover requirements specification for the recogniser component. Requirements for recogniser Using current technology, a speech recogniser comprises two components, which acn be developed separately. Each component models mathematically an aspect of the language under consideration: the Acoustic Model (AM) models the basic sound units of the language (e.g., phonemes) and how they combine into larger units (e.g., words and phrases); the Language Model (LM) models the basic meaning units of the language (e.g., words) and how they combine into larger units (e.g., words and phrases). Acoustic Model requirements The AM requirements should address the issue of what kind of speech the application must recognise. For example, questions like: will the application work with many unknown speakers, or will it be attuned to only one speaker? what kind of environment will the application work in --- a quiet office, a construction site, a spacesuit? These questions have a direct impact on the recording. The speech data you record should be similar to the speech which the application is intended to recognise. If you want the application to work with unknown speakers, you should record data from as many and various speakers as possible. Similarly, data should be recorded in a variety of environments: silent room, quiet (and not so quiet) office, outside, by the roadside,... You should err on the side of variety. If you record all or a large art of your data in "ideal conditions" you may find that the recogniser is too sensitive to background noise, mishears what is said, and hears background noise as speech. Language Model requirements For a large vocabulary speech recogniser, the LM approximates a grammar of the lanuage being recognised. For a C&C application, the LM begins from a concise description of the "task language" in other words, the subset of language which is being used for the application. In the simplest applications, where the recogniser will recognise a set of single spoken commands, the best description can be a list of the commands required. For more complicted applications a small grammar can provide a more concise, but still easily comprehensible description. For example, an application designed to recognise phone numbers. It would be possible to describe the LM required as simply a list of all possible phone numbers. However, it will be more useful to write a simple set of rules like the following:

5 5 phone_number --> std_code + local_number std_code --> 0 + digit + digit + digit + digit local_number --> digit + digit + digit + digit + digit + digit (+ digit)? digit --> one of {1, 2, 3, 4, 5, 6, 7, 8, 9, 0} 1 --> "un" 2 --> "dau" 3 --> "tri" 4 --> "pedwar" 5 --> "pump" 6 --> "chwech" 7 --> "saith" 8 --> "wyth" 9 --> "naw" 0 --> "dim" Collecting LM requirements tells you what to record: i.e., what words or phrases. For a small vocabulary C&C application you should record all of the commands required (e.g., for a simple calculator you'll need the digits, the decimal point, function keys, and whatever other commands you want to support). For something with a larger vocabulary, like a phone number recogniser, you might not need to record every possible input, but you should ensure that all possible combinations are covered (e.g., for a phone number recogniser, you should record each digit in each position). If you skip this step before you record, you may find that you have failed to record the data you need for you application. If you fail to record the data you need, it will at best be very difficult, and probably impossible, to develop the desired application. Develop the application A note on testing and integration Testing code, integrating components, and testing integration should not be thought of as separate activities which follow development. Testing and integration are a part of development and should be carried out as part of development. In particular, development of the internal interfaces (e.g., between the recogniser and the other components) should not wait until after recogniser has been developed. An interface can be designed and developed, and the recogniser can be developed to provide output in the format required by the interface. The interface can be tested by using a "dummy" recogniser which produces output in the same way as the real recogniser, but does no recognition. A full discussion of software component interfaces is beyond the scope of this document (and there are many textbooks and websites available, for example on threading or event-driven programming). Here I shall cover developing the recogniser component. Language Model The grammar given above in Language Model requirements is a finite state grammar [fsg]. Such a grammar can be represented by a list of rules. Sphinx requires finite state grammars to be written in a particular format. Below are two examples: 1. Word command grammar for a calculator List of rules: command --> one of {1, 2, 3, 4, 5, 6, 7, 8, 9, 0, +, -, *, /, =, C, OFF} 1 --> "un" 2 --> "dau" 3 --> "tri" 4 --> "pedwar" 5 --> "pump" 6 --> "chwech" 7 --> "saith" 8 --> "wyth"

6 6 9 --> "naw" 0 --> "dim" + --> "adio" - --> "tynnu" * --> "lluosi" / --> "rhannu" = --> "yn gwneud" C --> "clirio" OFF --> "diffodd" Sphinx format: FSG_BEGIN cyfrifiannell_19 NUM_STATES 19 START_STATE 0 FINAL_STATE 18 TRANSITION adio TRANSITION chwech TRANSITION clirio TRANSITION dau TRANSITION diffodd TRANSITION dim TRANSITION lluosi TRANSITION naw TRANSITION pedwar TRANSITION pump TRANSITION rhannu TRANSITION saith TRANSITION tri TRANSITION tynnu TRANSITION un TRANSITION wyth TRANSITION yn_gwneud TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION TRANSITION FSG_END 2. Number formats for a phone number recogniser List of rules: phone_number --> std_code + local_number std_code --> 0 + digit + digit + digit + digit local_number --> digit + digit + digit + digit + digit + digit (+ digit)? digit --> one of {1, 2, 3, 4, 5, 6, 7, 8, 9, 0} 1 --> "un" 2 --> "dau" 3 --> "tri" 4 --> "pedwar" 5 --> "pump" 6 --> "chwech" 7 --> "saith" 8 --> "wyth" 9 --> "naw" 0 --> "dim"

7 7 Sphinx format: The sphinx format for fsgs quickly becomes very verbose. For more involved grammars we recommend using the JSpeech Grammar Format [jsgf] which is also supported by Sphinx [sphx-jsgf]. Acoustic Model Prepare training data From the LM specification you can generate a list of utterances required. A prompt sheet should be prepared for the speaker. This can be a sheet of paper, or it can be onscreen, depending on your recording set up. The crucial thing is that it should indicate explicitly and exactly what you want the speaker to say. For example, for the number '4', you should indicate whether you want the speaker to say "four" or "pedwar"; for the letter 'C' you should indicate whether you want the speaker to say "see", "eck" or "kuh". If you want all, you should collect all, and indicate this explicitly on the prompt sheet. Bad Good 4 Four Pedwar C C_en ("see") C_cy1 ("eck") C_cy2 ("kuh") If you fail to do this and, for example collect alphabets from 100 speakers without indicating whether you require the English or Welsh pronunciation, you will end up with more or less random data, far too variable to train a recogniser, and therefore useless wasted effort. Each utterance should be recorded into a separate audio file. The speaker should leave at least half a second of silence before speaking, and the recorder should leave at least half a second of silence before stopping recording. It might be tempting to record more than one utterance at a time, and plan to split up each file later into separate utterances (e.g., record "un... dau... tri... pedwar... pump" and later split the audio file into five separate files). This is false efficiency, even if you think you have a proven procedure which can automatically split the audio files. The software you have in mind might not work with these speakers, speaking in this way, in this variety of environments. After specifying the requirements, collecting the data is the most important part of the project, and it should be as uncomplicated as possible. Speech recognition trainers like SphinxTrain or HTK use audio sampled in mono, with a sampling rate of 16 khz, and with a sample size of 16 bits. Ideally, you should record at these settings. If necessary, use audio procesing software (e.g., sox [sox]) to process the audio after you have recorded it. Once recorded, data should be stored in the version control system as soon as possible, and before any post-processing. Build Acoustic Model with SphinxTrain Note that, although the AM itself can only be built once all the data has been recorded, almost all of the configuration and set up (see Initial configuration and setup and Files & directories needed) can be done while still recording.

8 8 SphinxTrain is the AM training component of Sphinx. It can be downloaded from the project website [sphx]. Sphinx and SphinxTrain (and HTK) were designed to work in a Unix environment (e.g., Linux and Mac OS X) but they will compile and work on Microsoft Windows. Much of SphinxTrain's configuration and set up uses scripts written in the programming languge Perl. Perl is generally already available on Unix platforms, but on MS Windows you might have to download and install it yourself before proceeding. Perl can be downloaded from the Perl website [perl]. The following sections describe in detail how to configure, set up and use Sphinx Train. I shall use the project name "cyw" (short for "cywaith"). Initial configuration and setup Create a new directory with the project name and cd into that directory: $ mkdir cyw $ cd cyw Use the SphinxTrain command setup_sphinxtrain.pl to set up the directory structure, scripts and config files: $ perl /path/to/sphinxtrain/scripts_pl/setup_sphinxtrain.pl -task cyw Editing sphinx_train.cfg The main config file is put into cyw/etc/sphinx_train.cfg. Certain lines should be editted to reflect the needs of your project. Lines 18f Edit these lines to match the type of your audio files. If your audio files are not one of the supported formats, you must reformat them (e.g. with sox). 18 # Audio waveform and feature file information 19 $CFG_WAVFILES_DIR = "$CFG_BASE_DIR/wav"; 20 $CFG_WAVFILE_EXTENSION = 'wav'; 21 $CFG_WAVFILE_TYPE = 'mswav'; # one of nist, mswav, raw Lines 69f $CFG_LISTOFFILES and $CFG_TRANSCRIPTFILE (lines 73 & 74) should be changed so that those two filenames follow the general model (and fit in with tinydoc.txt & most other scraps of SphinxTrain documentation). This is a bug in SphinxTrain. 69 #*******variables used in main training of models******* 70 $CFG_DICTIONARY = "$CFG_LIST_DIR/$CFG_DB_NAME.dic"; 71 $CFG_RAWPHONEFILE = "$CFG_LIST_DIR/$CFG_DB_NAME.phone"; 72 $CFG_FILLERDICT = "$CFG_LIST_DIR/$CFG_DB_NAME.filler"; 73 $CFG_LISTOFFILES = "$CFG_LIST_DIR/${CFG_DB_NAME}_train.fileids"; 74 $CFG_TRANSCRIPTFILE = "$CFG_LIST_DIR/${CFG_DB_NAME}_train.transcription"; 75 $CFG_FEATPARAMS = "$CFG_LIST_DIR/feat.params"; Change lines 73 & 74 to: 73 $CFG_LISTOFFILES = "$CFG_LIST_DIR/$CFG_DB_NAME.fileids"; 74 $CFG_TRANSCRIPTFILE = "$CFG_LIST_DIR/$CFG_DB_NAME.transcription"; Line # Number of tied states (senones) to create in decision-tree clustering 120 $CFG_N_TIED_STATES = 1000;

9 9 This should be edited depending on the amount of training data. The following table is from the SphinxTrain manual [sman]. Amount of training data (hours) No. of senones Greater than are enough Lines 90 & 101 $CFG_STATESPERHMM = 5; SphinxTrain builds a mathematical model called a hidden Markov model for each unit to be recognised. This number, states per HMM, relates to the complexity of the HMMs. If the units are very small and homogeneous --- i.e. basic sound units like phones --- the HMMs are relatively simple and can have fewer states (e.g., around 5-7). If the units are longer and more heterogeneous --- like words or phrases --- the HMMs often need to be much more complex, with around states. If you are developing a recogniser with a large vocabulary --- a hundred words or more --- you should develop a phone recogniser, and leave this line as it is. If you are developing a recogniser with a small vocabulary, such as a basic C&C applcation, a word recogniser will be easier and quicker to develop: set this line to depending on the length of your units. $CFG_STATESPERHMM = 15; Files & directories needed SphinxTrain uses a number of text files as input along with the audio data. All of these text files should be saved into the cyw/etc/ directory. cyw.transcription is the most important file, and all other files should be dervied from that (see Appendix: Python script to derive project files from transcription for a script which will do this automatically). All of the text files must be in the same character encoding (e.g. utf-8). cyw.transcription Orthographical transcription of each audio file, one per line. Note that the transcription should not include punctuation. Essentally it's just a space separated list of words. To stress this, many projects make all words upper case. Line format: <s> IS THIS A DAGGER I SEE BEFORE ME </s> (cyw0001)... <s> </s> Starting silence

10 10 Closing silence (fstem) The name of the audio file, without the filetype extension cyw.fileids A list of filestems (i.e., filename without the filetype extension) for each audio file. n.b.: The filestems should be in the same order as in cyw.transcription. Best is to finish with cyw.transcription and derive cyw.fileids from that. File format: cyw0001 cyw0002 cyw cyw.dic Pronunciation dictionary for the words in cyw.transcription. SphinxTrain will crash if there is not an exact match between the words in cyw.transcription and cyw.dic. Line format: CYLLELL k ax ll eh ll The headword must appear exactly as it does in cyw.transcription (e.g. upper or lower case). Each headword must be unique. If your data contains orthographically identical words with different pronunciation, you must differentiate them in some way in the orthographic transcription. e.g.: C en s ii C cy1 eh k C cy2 k ax MINUTE N m ih n ih t MINUTE ADJ m ai n j u t The phonemic transcription must not contain stress or length markings. For a word model For a word model, just give the word itself as its own pronunciation, e.g. UP UP DOWN DOWN LEFT LEFT RIGHT RIGHT cyw.filler A 'pronunciation dictionary' for filler words, non-speech sounds, and other noises. File format as for cyw.transcription. Must contain at least opening and closing silence and internal silence. For a C&C application this should be sufficient. <s> SIL </s> SIL <sil> SIL

11 11 cyw.phone A list of all and only the phones used in cyw.dic and the sound elements used in cyw.filler (which are regarded as phones by SphinxTrain). No particular order necessary. File format: SIL ai ax eh ih ii j k m n s t u... cyw/wav/ directory Put all audio files mentioned in cyw.transcription into a directory called wav. Despite the name, they do not have to be '.wav' files. They must all be the same format, but their format can be specified in the config file (see above). Running SphinxTrain The first step is to pre-process the audio data: $ perl scripts_pl/make_feats.pl -ctl etc/cyw.fileids If you are building a phone model you can now run: $ perl scripts_pl/runall.pl If you are building a word model, much of the processing in RunAll.pl is superfluous. See Appendix for RunAll_wordmodel.pl, a script to build a word model. Using the recogniser The code appropriate for using the recogniser varies widely depending on the application and the programming language used. Sphinx itself comes with a python script which gives an example of usage with a simple scripting language (in the directory sphinx3/python), and several small C programs are provided as more serious demos (in the directory sphinx3/src/programs). The calculator (cyfrifiannell) and launcher (lansiwr) show how the recogniser can be used in a MS Windows.Net environment. Package the application for distribution After all development has been completed, the application can be packaged in a format appropriate to the distribution requirements.

12 12 References [fsg] [htk] [jsgf] [perl] [sman] [sox] [sphx] (1, 2) [sphx-jsgf] [svn] [trac] Appendices Python script to derive project files from transcription See make_st_files.py Runall_wordmodel.pl See Runall_wordmodel.pl