SPRACH - WP 6 & 8: Software engineering work at ICSI

Transcription

1 SPRACH - WP 6 & 8: Software engineering work at ICSI March 1998 Dan Ellis International Computer Science Institute, Berkeley CA <[email protected]> Hardware: MultiSPERT Software: speech & visualization tools Packaging: SPRACHworks SPRACH WP6 & 8 - Dan Ellis 1998mar24-1

2 1 Hardware: MultiSPERT Multiple SPERT boards on single host MultiSPERT New software, firmware, hardware - for nets: pattern parallel and network parallel Fastest performance: 500 MCUPS (5 boards, 10x previous single-board) Current neural-net trainings: 8000 hidden units, 3.2M params, 18M frames ~10 15 ops per training iteration - this size not previously possible SPRACH WP6 & 8 - Dan Ellis 1998mar24-2

3 2 Software: Speech tools New components: feacalc: front-end feature calculation - uses core RASTA-PLP routines from rasta - mnemonic options ( -L -rasta log ) - comprehensive file format support featools: scaffolding for novel features - handles finding input files & assembling output - user provides (chain of) online-feature applets pfile_utils: feature archive manipulation - merging, splitting, rearranging, summary stats - pfile_gaussian, pfile_klt... SPRACH WP6 & 8 - Dan Ellis 1998mar24-3

4 Visualization & User Interface tools ICSI is developing a toolkit of Tcl/Tk modules for visualization & demos Current pieces - visualization of signals, features, probabilities, label alignments - interactive & file-based audio input/output - web-interface tools Some current applications - sgramimg.cgi on-demand spectrogram gifs - recogviz comparing different ASR configs - berpdemo98 speech application plus graphics SPRACH WP6 & 8 - Dan Ellis 1998mar24-4

5 sgramimg.cgi CGI script: spectrogram GIF on-demand - point to in <img src..> tag of other pages Automatically includes xlabel annotation Try it, e.g. links from: SPRACH WP6 & 8 - Dan Ellis 1998mar24-5

6 recogviz Motivation: compare recognition techniques - at each stage in process (signal, features, probs) Easily to incorporate novel features etc. SPRACH WP6 & 8 - Dan Ellis 1998mar24-6

7 berpdemo98 Existing application plus recogviz modules Demo available... SPRACH WP6 & 8 - Dan Ellis 1998mar24-7

8 3 Packaging: SPRACHworks Original SPRACHworks goal: Source-level distribution of all partner tools - integrated: compatibility between tools - portable: easily compiled - ready-to-run: demos to play with HUGE amount of work! Reduced scope: - several separate but conformal packages (ICSI, Cambridge, FPMs/Strut) - incremental incorporation of packages Status: - first collection: April 97 (Cambridge) - latest: single make for complete speech demo ( SPRACHcore = 19 packages) - ICSI-developed, installed at FPMs & IDIAP SPRACH WP6 & 8 - Dan Ellis 1998mar24-8

9 Future work Tools - language modelling, two-pass decoder, etc... - formal modular structure for visualization drop-in demos Packaging - include more packages: ASR training - tutorial documentation - more platforms? SPRACH WP6 & 8 - Dan Ellis 1998mar24-9

10 Combined speech/nonspeech analysis: How to combine ASR with Computational Auditory Scene Analysis (CASA) for nonspeech transients? Preprocessor (Weintraub 85, Nakatani/Okuno 97) mixture CASA (f 0 -based) single element ASR words Use extra CASA info in ASR (Cooke,Morris & Green 97) Simultaneous search for speech & nonspeech elements mixture mixture CASA Hypothesis management Compare prediction errors mask element Missing-data ASR scene hypothesis Speech elements Nonspeech elements Predict words words casa-appr 1998mar Difference between observations & predictions drives analysis ThisL/ICSI - Dan Ellis 1998mar25-1

11 Speech/nonspeech: preliminary results Original speech +clap Bootstrap recognition db Reconstructed speech element Recovered nonspeech (a) Speech plus clap (223cl-env.pf) (b) Recognizer output h# w n ay n tclt uw f ay ah s ay ow h# v s eh v ah n h# h# n ay n tcl t uw f ay v ow h# s eh v ax n <SIL> nine two five oh <SIL> seven (c) Predicted speech element (223cl-renv.pf) (d) Click5 from nonspeech analysis (justclick.pf) s Issues: - iterative refinement of each element use nonspeech to relax speech recognition - reconstructing speech features from ASR output - use f 0 -based separation to bootstrap ASR; recognizer trained on periodic + noise ThisL/ICSI - Dan Ellis 1998mar25-2

12 ICSI GUI tools in ThisL Interactive audio_frontend for commands Web-embeddable recognition display tools ThisL/ICSI - Dan Ellis 1998mar25-3

13 ThisL Overview Archive Query Database Segmentation Control IR NLP ASR Receiver ASR Text Audio http Video Domains: BBC news (3hr/day) NIST TREC French? Issues: Size of episodes decoder Size of database speed Segmentation/side info Information Retrieval Dan Ellis 1998apr02