MULTIMODAL VIRTUAL ASSISTANTS FOR CONSUMER AND ENTERPRISE

Transcription

1 MULTIMODAL VIRTUAL ASSISTANTS FOR CONSUMER AND ENTERPRISE Michael Johnston Lead Inventive Scientist 1 mjohnston@interactions.net

2 VIRTUAL ASSISTANTS: CONSUMER Calling Local Search Messaging Web Search Calendar Reminders Alarms 2

3 VIRTUAL ASSISTANT: ENTERPRISE Billing Sales can I upgrade to iphone 6? when is my payment due? Communications Provider Smart Home set the thermostat in the living room to 65 TV Control comedy movies on HBO tonight? Conn Tur Conn Set Hote Gas 14 Account I m locked out of my account can I reset my password Technical Support my wifi is not working Appointments I need to reschedule my service appointment W Co Sw

4 MULTIMODAL VIRTUAL ASSISTANT (INTERACT) Restaurants Map / Geo sushi restaurants near madison wisconsin what about pizza I d like to reserve a table show portland oregon zoom to the west village Business search Events hotels [circle area] gas stations [draw route] country music shows near dallas texas next weekend what about punk rock TV / Media Weather / General Comedy shows on HBO tonight what about showtime reviews for this show weather forecast for las vegas for tomorrow web search alternate side parking 4

5 INTERACTIONS WATSON: MULTIMODAL VIRTUAL ASSISTANT Intelligent interactive applications empowered by natural modalities to make it effortless for users to access information and execute tasks MOBILE DEVICES CONNECTED CAR DIGITAL HOME CONSUMER ENTERPRISE NATURAL INPUT AND OUTPUT Speech recognition and synthesis Motivation: Vast range of commands through single point of entry Natural language Key for mobile, hands free MULTIMODAL Gesture / Visual displays Multimodal integration Motivation: Input/output by most effective means Maps and other visual displays Adapt to environment: physical/social CONTEXTUAL Interpret user input with respect to flow of conversation Motivation: Natural, concise interaction CONVERSATIONAL Multi-step cooperative dialog Motivation: Enable fulfillment of complex intents through multi-step interaction ADAPTIVE Machine learning technology applied throughout Motivation: Learns from experience Performance improves with use PERSONALIZED Knows who you are Speaker verification/id Proactive adaptation to user Motivation: Effortless, Retain user 5

6 MULTIMODAL VA: INTERACT Interactions Watson Multimodal VA Platform Incremental interaction Natural language Contextual Cooperative conversational dialog Multimodality Personalized and adaptive 6 6

7 INTERACTIONS WATSON: MULTIMODAL VA TECHNOLOGIES Coordinated Graphics / Gesture GUI Generation graphics Speech Synthesis! Natural! Language Generation feedback words meaning ADAPTIVE UNDERSTANDING Data/ Learning/ NLP! Multimodal Dialog Manager! meaning words! SAL Automatic Speech Recognition GESTURE Natural Language Understanding Gesture Recognition Semantic Abstraction Layer task reasoning, learning 7 ianalyst Desktop ianalyst Desktop ianalyst Desktop API API API

8 MULTIMODALITY ACCESS TO INFORMATION, CONTENT, SERVICES, AND CARE IS RAPIDLY EVOLVING TO MULTIMODAL Telephony SPEECH ENABLES SINGLE-POINT-OF-ENTRY AND CONVERSATIONAL INTERACTION but not the best for everything Certain tasks and functions cry out for particular modalities (Rudnicky and Hauptmann 1992) Ability to switch modes Mobile Devices Connected Car MULTIMODAL IN VIRTUAL ASSISTANT Parallel presentation of complementary info Automation of more complex intents Fusion of multiple modalities Wearables Smart Home

9 MULTIMODAL EXAMPLE: SALES / DATA USAGE Dynamic presentation combining synthetic speech with graphical displays Deeply personalized Seamless connecting capabilities 9 9

10 MULTIMODAL EXAMPLE: APPOINTMENT SCHEDULE Spoken dialog interspersed with graphical interaction Enables automated fulfillment of complex intents 10 10

11 CROSS-PLATFORM VIRTUAL ASSISTANTS FOR ENTERPRISE Synergy within particular hardware or software ecosystem Unified customer experience needed for enterprise VA 11

12 VIRTUAL AMBASSADORS TO THE INTERNET OF THINGS DIM We just want to relax tonight SEARCH ORDER 12

13 VIRTUAL AMBASSADORS TO THE INTERNET OF THINGS Users will expect a consistent interface of things across devices What is the temperature in the bedroom? Movies on Showtime tonight? Make it 65 degrees What about HBO? 13

14 THE INTERFACE OF THINGS We make speech and multi-modal technologies universally accessible to all devices and industries through a wide ecosystem of partners Data & Analytics Voice Biometrics Machine Learning Data & Analytics Speech Recognition Dialog Manager Large Enterprises Text to Speech Devices Language Processing Video 3 rd Party Developers AT&T Solution Providers Consumer Applications Select Licensing (Large OEMs) Partners Systems Integrators Healthcare Connected Car Industries Industrial Automation 1 Home Automation Customer Service

15 CONCLUSION Enterprise virtual assistants provide customers with a single-point-ofentry to a broad array of information and services Cross-platform solutions are critical for enterprise applications, even more with emergence of IoT Interactions Watson Virtual Assistant Platform: Multimodal interaction Adaptive understanding (human-in-the-loop) Natural context sensitive language understanding Cooperative conversational dialog mjohnston@interactions.net 15