Intelligent Personal Assistants: A view from the intersection of Data Mining and Mobile Services

Transcription

1 Intelligent Personal Assistants: A view from the intersection of Data Mining and Mobile Services Y. Akinaga S. Mukherjee [speaker] H.-S. Shin P. Subasic R. Sujithan Y. Sun H. Yin DOCOMO Innovations, Inc., Palo Alto, CA H. Fujimoto T. Hayashi S. Iizuka J. Jacintha T. Yoshimura NTT DOCOMO, Inc. Japan

2 The age of intelligent virtual assistants Name Clippy Siri 2012 Google Now By (in): MS (Off97) Apple (ios5) Google (And4.1JB) MS (WP8.1) For: Desktop (local app) Mobile (OS) [Cloud] Mobile (OS) [Cloud] Mobile (OS) [Cloud] Access: Typing Voice Voice Voice Limited, definiton- based: is my wife. Rule- based, e.g., show traffic map for appointment Machine learning algorithms on collected data Learning Very limited, method machine (if any) learning 5/15/14 Cortana 2

3 DOCOMO s intelligent assistant: Shabette-Concier DOCOMO serves about 60M subscribers in Japan DOCOMO didn t carry iphone until 2013 only Android! 2011: iphone big in Japan, but Siri didn t speak Japanese! [Japanese support introduced in Mar. 2012] Feb. 2012: DOCOMO introduced (on Android) an intelligent virtual assistant (exclusively Japanese): Named Shabette-Concier [ talking concierge ] Used more accurate local information sources (e.g., restaurants) than Siri s Japanese service Google introduced Google Now in Jul. 2012, but Japanese support only came in 2013 Shabette-Concier was DOCOMO s weapon for selling Android handsets in Japan 5/15/14 3

4 The key to global expansion: language-independent processing! Voice Voice recognition, text transcription Task Recog., Query Extraction Display the result or Initiate Func. Content organizer Content Provider Search Engine From Japanese team Well-defined interface 5/15/ v=ytbf8lcofpm&feature=youtube_gdata

5 Example: processing a text query Find a good Italian restaurant in Palo Alto Input sentence Slots in Restaurant CP API Location query Restaurant Search task Query extracton is done using rule- based text processing and matching to a large dictonary. The dictonary is created by combining the input from many corpora and searching knowledge databases like Wikipedia and Freebase. Example of rendering of results of restaurant search query More examples: YVrkwKutFlg! I found restaurants below, 1. Osteria detail 2. Vero detail 247 Hamilton Ave Palo Alto, CA (650) Bryant St Palo Alto, CA (650) Café Pro Bono 2437 Birch St detail Palo Alto, CA (650) Terrone 448 S California Ave detail see more Palo Alto, CA (650) Find a good Italian restaurant in Palo Alto 5/15/14 5

6 Supported Search tasks (US, English) Category Contents Provider Query extraction Location extraction Transportation Google map No need Location for search Date(time) extraction Map search Google map No need Center location No need Weather info World Weather Online No need Location for weather Date for weather Music search Yahoo BOSS (search) Name of music Movie Yahoo BOSS (search) Name of movie Location for reservation Date for reservation Photo/Image search Yahoo BOSS (search) Named Entity No need No need News search Yahoo BOSS (search) No need No need No need Game search Yahoo Game Search Appl. name or type No need No need Apps search Google play Appl. name or type No need No need Restaurant search Yelp (restaurant) No need Location for search / reservation Date for reservation Recipe search Yahoo BOSS (search) Name of dish No need No need Book search Yahoo BOSS (search) Name of book No need No need Shop search Yelp (shop) No need Location for search Travel search Yahoo BOSS No need Location for search Date for reservation Wikipedia search Wikipedia Search word No need No need Hospital search Yelp (Health & Medical) No need Location for search FAQ search Wolfram Alpha No need No need No need 5/15/14 6

7 Supported Initiation and Chatting tasks Category Query extraction Date(time) extraction IdenTfying/evaluaTng temporal expressions Initiate call Called person s name No need Initiate Destination No need Initiate camera No need No need Check schedule Schedule contents Date and time for schedule Book schedule Schedule contents Date and time for schedule Make memo Memo contents No need Launch timer No need Time for timer Set alarm No need Alarm date and time Music player Search queries for music No need Greeting No need No need Praise No need No need Chatting tasks Dispraise No need No need Boredom No need No need Check schedule: What do I have on the calendar for Thursday at 2pm? check calendar for march 25 at 8am Weather: Show me the 5 day weather forecast What are the current weather conditions in Cleveland, Ohio? What is Friday's weather forecast? Book Schedule: schedule dentist appointment next Tuesday at 9am Timer: Set a timer for 6 minutes Love No need No need Alarm: set alarm for 10am tomorrow 5/15/14 7

8 High-level architecture Web Client (Dev.) Mobile Client (Android) Web Service + Pipeline (Supervised) Machine Learning Feature Engineering AWS Deployment + Core Libraries Task classifier C/C++ code (linear SVM) Python Bindings (low-level) Python APIs (high-level) 5/15/14 8

9 Crowdsourcing training data (text) for ML (in each language!) English -- Amazon Mechanical Turk A marketplace for work that requires human intelligence Workers ( turkers ) submit text, get paid after tasks complete Other languages too! We did it for Chinese using a different marketplace (in China) Caveat: careful task definition and scrutiny of submissions to maintain quality Mech. Turk: a fake chess-playing automaton from late 18 th century moves are made by a human hiding under the table! 5/15/14 File:Tuerkischer_schachspieler_windisch4.jpg! 9

10 An example of a set of tasks on AMT 5/15/14 10

11 Now in the car! DOCOMO DriveNet Demo: ITS World Congr., Tokyo, Oct. 13 Restaurant in Palo Alto GPOI/UPOI tagger Query extraction for Point of Interest (POI) Unique (UPOI), e.g. Palo Alto General (GPOI), e.g. Restaurant task classifier <GPOI> in <UPOI> 2 Incorrect task classification Task ID 004:POI search 3 Failure to extract query keywords Failure could occur at any of these 3 points: Task classifier has fairly low probability of failure Logic needs to handle failure to tag/extract query 5/15/14 Slot 1: Restaurant Slot 2: Palo Alto 1 Failure to tag/ incorrect tagging of GPOI/UPOI Tagged Text <task id, Extracted query keywords> Text Query extractor 11

12 The next frontier: DOCOMO s (silent) predictive assistant -- Tap de Concier Applications supported: TOP (slide-in widget) o Call : address (called tel. number) o Mail : address ( addr.) o Train route search : origin / destination o SMS : address (text tel. number) Dynamic: One-shot tasks Call More one-shot suggestions for train route search Enter your own train route search Train route Search Pressing the static button for train route gets you this screen Static buttons Mail 5/15/14 12

13 The next frontier: DOCOMO s (silent) predictive assistant -- Tap de Concier (contd.) CA team: design and implement predictor for: One-Shot candidates on top screen Query candidates (content) on each app screen One-Shot candidates on top screen List of apps and content with scores Score and compare Mail - Alice Alarm 08:00 Mail - Bob Call Alice Query candidates on app screen Score and compare List of mail content with scores Alice Bob Charlie Many subscribers = big data, but Personalization = working with small data (individual subscriber s usage) 5/15/14 13

14 Predictor Framework Usage log data Feature ExtracTon Training data Log data is converted to an RFM feature vector RFM = Recency, Frequency, Monetary (utility value) record PredicTon For top screen: One-shot candidates For app screens: Query candidates Modeling Multivariate linear regression Input: matrix of RFM entries Output: weighting vector RFM feature vector format is app-dependent 1 vector per: contact [for call, sms, and apps] origin or destination station [for transit app] Training data: RFM vectors and clickthrough rates measured from usage log 5/15/14 14

15 What comes next? Contextual awareness Some idea of what state you are in May require detailed probing of your likes/dislikes/personal data Example: Cortana s notebook Specialization (with expert knowledge) Assistants may be embedded into the apps you use E.g., you open a flight reservation app and the assistant talks you through your choices and creates an itinerary Would help if all apps on your mobile device had access to the underlying processing pipeline of the virtual assistant Example: Siri not available to third-party apps 5/15/14 15

16 Duchess:... When I was younger, boys of your age used to be nice and innocent. Reginald: Now we are only nice. One must specialise in these days. [From Reginald at the Theatre, by Saki (H.H. Munro), London, 1911.] Thank you! smukherjee@docomoinnovations.com 5/15/14 16