Speech Applications - Accessing information by voice. Li Haizhou, PhD Vice President, InfoTalk haizhou.li@infotalkcorp.com

Transcription

1 Speech Applications - Accessing information by voice Li Haizhou, PhD Vice President, InfoTalk haizhou.li@infotalkcorp.com

2 nabling technology for voice portal Voice portal Speech recognition & spoken dialogue Text-to-speech Voice print technology Voice XML speech enabling platform

3 nfotalk R.S.V.P Recognizer (Automatic speech recognition) Speaker (Text-to-speech) InfoTalk Verifier (voice print) InfoTalk Platform (Voice XML)

4 Voice portal

5 A call centre PSTN

6 A voice portal Automatic speech recognition PSTN Business Logic Text-to to-speech

7 arket trend

8 hy Now? 50 Speech Algorithm Need 40 MIPS Processor Speed Processor speed becomes faster every year Speech technology becomes more efficient every year Commercialization becomes viable

9 enefits of Speech Technology Difficulty of inputting Asian languages on a keyboard or telephone keypad Prevalence of using mixed languages

10 enefits of Speech Technology Existing Number (Millions) Phones PC's connected to the Internet 6 Worldwide China Higher penetration of telephones relative to PCs in the region

11 Benefits of Speech Technology 35,000 30,000 25,000 20,000 15,000 10,000 5,000 0 Live agent cost/call $1.06 Live agent/year Speech IVR/year (US$) Speech IVR cost/call $0.10 Source: speech recognition magazine Saving of US$1.2m annually for a speech system of 100,000 calls/month Source: TMA Associates

12 alue Proposition Full, round-theclock service Easier, faster navigation Consistent service, brand No holding or dial through menus Customer satisfaction and retention speech understanding solution Operational efficiency Operators not needed for routine tasks Reduces call backlogs Automates routine tasks Larger call-volume capacity High ROI Fewer resources necessary Lower call times = lower telecom costs

13 o automate a call centre PSTN Trunk Group (T1, E1, ISDN, Analog) Existing IVR / CRM System Database TCP/IP Local Area Network

14 o automate a call centre PSTN Trunk Group (T1, E1, ISDN, Analog) Existing IVR / CRM System Database ASR servers TCP/IP Local Area Network TTS servers Speech-enabling your existing platform by simply adding ASR and TTS modules

15 oice portal business model Enterprise portal (eg. CAAS) Corporate Websites Self-service VRU B2B, B2C and B2E Consumer portal Value-add services as pay-per-use Time-critical, location-insensitive telcom products

16 oice Portal Utility Acquire information Engage in transactions interactively Navigate information Voice remote control

17 Speech Recognition & Spoken Dialogue

18 SR: How it works?

19 SR: In a simple way Acoustic models Grammars PSTN

20 SR: Telephony speech recognition Speech recognition & understanding Syntactic Semantic Lexical Phonetic

21 SR: Syntactic & semantic rules ( Grammar )

22 SR: Lexical & phonetic rules

23 SR: Lexical & phonetic rules

24 poken dialogue: Moving through states (CAAS) Greetings Arrival/Departure Flight number City of origin Flight number City of Dest. no Yes/no no Yes/no no Yes/no no Yes/no yes yes yes yes

25 poken dialogue: Mode of spoken dialogue (I) Full duplex/half duplex dialogue Turn-taking Barge-in (CAAS Telefiqs) Mixed-initiative dialogue Computer-directed Form filling conversation

26 poken dialogue: Mode of spoken dialogue (II) Mixed-modality dialogue DTMF Speech Mixed-lingual dialogue Speak in your customer s language Multilingual

27 poken dialogue: Mode of spoken dialogue (III) Natural language understanding Partial parsing Keyword barge-in Dialogue memory Audio media indexing (Demo) Rejection of OOV and disfluency

28 poken dialogue: Natural language understanding the conversion rate from HK dollar to Singapore dollar. Slot: from Slot: to Value: HKD Value: SGD Speech: Hong Kong dollar Speech: Singapore dollar Confidence: 72.4% Confidence: 80.1%

29 poken dialogue: Common practices in spoken dialogue design Understand characteristics of speech Be aware of limitation of speech recognition Identify the computer Prompting Accommodation/flexibility Responsiveness & confirmation Error recoverability Select appropriate vocabulary

30 poken dialogue: Prompting Use Avoid Say your credit card number? Say the departure date of travel. For example, say October 1 st, 2002 Tell me your credit card number. May I have your departure date?

31 poken dialogue: Common practices in spoken dialogue esign Understand characteristics of speech Be aware of limitation of speech recognition Identify the computer Prompting Accommodation/flexibility Responsiveness & confirmation Error recoverability Select appropriate vocabulary

32 Human Agents Speech Understanding Touch Tone Price poken dialogue: Accommodation/Flexibility Complexity

33 poken dialogue: Common practices in spoken dialogue design Understand characteristics of speech Be aware of limitation of speech recognition Identify the computer Prompting Accommodation/flexibility Responsiveness & confirmation Error recoverability Select appropriate vocabulary

34 poken dialogue: Responsiveness & confirmation Supply alternative guesses Do you mean Paris or Bali? Acknowledge the user s speech Paris, there are ten flights to Paris Pass talking-turn to user by using auditory icon Confirm if unsure

35 poken dialogue: Common practices in spoken dialogue design Understand characteristics of speech Be aware of limitation of speech recognition Identify the computer Prompting Accommodation/flexibility Responsiveness & confirmation Error recoverability Select appropriate vocabulary

36 poken dialogue: Error recoverability Use specific error message System: Say the departure date User: Tomorrow System: Say the departure date User: I want to travel tomorrow System: Say the departure date User: Ohh it does not understand! Provide info about error Do not repeat same error

37 poken dialogue: Error recoverability Avoid User: I need to sell my Keppel Land stock System: Do you mean Post Bank. If that s correct, please say the number of shares to sell. Otherwise, repeat the stock name User: 50 nnoo it is Keppel Land hold on a minute System: Confirming fifteen shares of Keppel Land User: No, no, no, it s all wrong.

38 poken dialogue: Error recoverability Use User: I need to sell my Keppel Land stock System: How many shares? User: 50 shares System: At what price? User: thirty-eight and a half System: Please repeat the stock name User: Keppel Land System: Confirming fifty shares of Keppel Land at

39 poken dialogue: Common practices in spoken dialogue design Knowing unique characteristics of speech Knowing limitation of speech recognition Identify the computer Prompting Accommodation/flexibility Responsiveness & confirmation Error recoverability Select appropriate vocabulary

40 poken dialogue: Select appropriate vocabulary Avoid using confusable words, such as correct vs. incorrect Apply back-end information to n-best choices

41 maturing technology From Radio Rex (1911) to speech recognition From recognition to understanding

42 Text-to-Speech

43 he Linguistic Processing Module Text Phonetics Köln linguistic processing k9ln prosody generation EPT synthesis k[90(1,98)]9[150(5,95)(50,99) (120,90)]l[100]n[95(95,80)]

44 he Linguistic Processing Module Convert a text into a phonetic transcription. Text: link with actual pronunciation may be vague or absent Phonetic transcription: unambiguous representation of sounds Examples Dr. Jones, 243 Medicine Dr. Doctor Jones, 243 Medicine Drive. 晚 间 气 温 -10

45 he Linguistic Processing Module rapheme-to-phoneme Conversion pronunciation dictionaries pronunciation rules morpheme dictionaries and morphological analysis fast & accurate, include additional information (PoS), but: closed set always effective, compact, but slower and not always accurate fast if limited analysis, accurate, more economic & dynamic than word form lists

46 he Linguistic Processing Module entence Analysis Example Input: The president will address the Congress in a few moments. After word by word processing???????????????????????????????????????????????????????????????????????? 1?# 'D$ 'pre-zi-d$nt 'wil $-'dres 'D$ 'kan-gres 'In $-'fju 'mo-m$nts. #? 2?# the president~~~ will address the congress~ in~ a few~ moment~~s~. #? 3?# d~~ N~~~~~~~~~~~ M\N~ H8~~~~~ d~~ N~~~~~~~~ p\a d2\p~~ N~~~~~~~ ~. #? 4?# ~~~ ~~~~~~~~~~~~ 444~ 'æ-dres ~~~ ~ ~~~ ~ ~~~~~~~~ ~ #????????????????????????????????????????????????????????????????????????? After sentence processing???????????????????????????????????????????????????????????????????????? 1?# 'D$ 'pre-zi-d$nt 'wil $-'dres 'D$ 'kan-gres 'In $-'fju 'mo-m$nts. #? 2?# the president~~~ will address the congress~ in~ a few~ moment~~s~. #? 3?#~d~~~N~~~~~~~~~~~~M~~~~v~~~~~~~d~~~N~~~~~~~~~p~~~d~~~~~~N~~~~~~~~~.~#? 4?~ N~~~~~~~~~~~~~~~ V~~~~~~~~~~~ N~~~~~~~~~~~~ P~~~~~~~~~~~~~~~~~~~~~~~?????????????????????????????????????????????????????????????????????????

47 he Prosody Module Text Phonetics Köln linguistic processing k9ln prosody generation EPT synthesis k[90(1,98)]9[150(5,95)(50,99) (120,90)]l[100]n[95(95,80)]

48 he Prosody Module Supra-segmentals structure the speech flow: expression level: encoding of linguistic intentions perception level: stress, intonation, tone, rhythm, speech rate, loudness acoustic level: fundamental frequency, intensity, duration Importance of prosody in TTS: Intelligibility Naturalness Demonstration:

49 ext-to-speech pronunciation rules InfoTalk Speaker User Rule File Language: putonghua Codeset: gb [ ](<NUM:11>)[ ]{<NAD>} Applying pronunciation rule Default interpretation

50 he Synthesis Module Text Phonetics Köln linguistic processing k9ln prosody generation EPT synthesis k[90(1,98)]9[150(5,95)(50,99) (120,90)]l[100]n[95(95,80)]

51 TS Unit Selection

52 ost Functions Candidate to Target matching Node Cost Sum of contributions from various symbolic and acoustic features Typically ~ 20 features per diphone Candidate to Candidate matching Transition Cost Numerical match between properties of diphones at boundaries Typically ~ 5 features at diphone boundary Balanced match

53 Voice print technology

54 oice print technology

55 oice print for speaker verification & speaker dentification

56 peaker verification System: Good Morning! Welcome to Jockey Club automated call centre. Please say enquiry for racing report or betting for placing your bet. Speaker: betting System: May I have your name? Speaker: Peter Tan System: Please spell your name. Speaker: P E T E R T A N System: Please say the horse name you wish to bet. Speaker: Golden Lion System: How many bets? Speaker: two System: Your bet is accepted. Payment will be deducted from your account.

57 peaker verification: User Enrollment User Enrolment User s voice InfoTalk-Verifier (Voiceprint Trainer) User s Voiceprint

58 peaker verification: Authentication Authentication Process User s Voiceprint Test of use s voice InfoTalk-Verifier Result Accept / Reject

59 Voice XML Platform

60 oicexml Standards Open, non-proprietary standard defined by VoiceXML Forum Voice XML is designed for creating audio dialogs that feature synthesized speech, digitized audio, recognition of spoken and DTMF key input, recording of spoken input, telephony, and mixed-initiative conversations. Voice XML s major goal is to bring the advantages of web-based development and content delivery to interactive voice response applications. Voice XML 2.0 is at its Last Call Working Draft status as of May 2002.

61 Traditional voice portal architecture Telephony Server (IVR) Database Server Telephony Interface Application Integrated ASR&Telephony API Database access over LAN or Internet (e.g. SQL, ODBC) Database ASR Engine TTS Engine

62 Voice XML based voice portal architecture Telephony Interface InfoTalk VoiceXML Interpreter (Voice Browser) InfoTalk API VoiceXML via HTTP over LAN or Internet VoiceXML Application Documents VoiceXML Application Documents VoiceXML Application Documents Database access over LAN or Internet (e.g. SQL, ODBC) Database InfoTalk CSU Engine InfoTalk TTS Engine Legend: API=Application Programming Interface CSU=Conversational Speech Understanding TTS=Text-to-Speech

63 oicexml Scripts -- ================== Stock Quote ===================== --> orm id="stock_quote"> <field name="stock_field"> <prompt count="1"> <audio src="message/english/ask_stock_name.vox"> Whick stock would you like? </audio> </prompt> <prompt count="2"> <audio src="message/english/ask_stock_name2.vox"> Please say the stock name you want to inquire, such as Hutchison Whampoa. </audio> </prompt> <grammar src="grammar/stock_name.gdo" type="application/gdo"/> <filled> <assign name= stock_name" expr= stock_field"/> <submit next="stockdata.asp" namelist="stock_name" fetchtimeout="5"/> </filled> </field> form>

64 oicexml and HTML paradigm VoiceXML HTML PSTN Internet Voice Browser Web Server

65 uilding a voice portal Human-Machine Interface Design ASR Engine Pilot Data Analysis & Tuning Production Speech Database Grammars & Models

66 he users of voice portal Infotainment Voice portal Call center Telecom Call center Travel Airline ticket booking Banking & insurance Check balance Transfer funds Brokerage Stock quote Trading Telephony voice portal Distribution Order tracking Logistics Utilities Bill payment Manufacturing/IT CRM Healthcare Appointment registration