Luis A. Pineda Héctor Avilés Paty Pérez Pavón Ivan Meza Wendy Aguilar Montserrat Alvarado External collaborators Students

Transcription

1 Dialogue model specification and interpretation for interacting with service robots Luis Pineda and The Golem group IIMAS, UNAM, México September, 2009 The Golem project team (current) Luis A. Pineda Héctor Avilés Paty Pérez Pavón Ivan Meza Wendy Aguilar Montserrat Alvarado External collaborators Students Introduction Content A multimodal architecture for service robots Dialogue models Implementation Applications: Tell me about the magazine! Guess the card! A guided tour with pointing acts Conclusion and future work Introduction Content A multimodal architecture for service robots Dialogue models Implementation Applications: Tell me about the magazine! Guess the card! A guided tour with pointing acts Conclusion and future work Main assumptions The interaction cycle must be tight! The agent needs to keep in touch with the world! Only reactive behavior? Human-level communication (e.g. Coordinated language and vision) needs to rely on symbolic representations The objects of representation are interpretations (not the world directly!) Perception is guided by intentions and memory depends on the context The Context Every interpretation act is performed in context: A spatial and time situation (indexical) A set of agents (I, you also indices) A discourse or interaction history (anaphoric) A conceptual domain A set of potential intention and action types that can be expressed by the agents during the interaction The context is a big thing! Can it be modelled explicitly in a simple way? 1

2 Agent s expectations Linguistic expectations (communicative) What are the speech acts that can be expressed by interlocutorn specific conversational situations? Environmental expectations (without intent) What events are expected to appear in the situation that are perceived by the different senses? How to deal with the unexpected? Introduction Content A multimodal architecture for service robots Dialogue models Implementation Applications: Tell me about the magazine! Guess the card! A guided tour with pointing acts Future work A three layers architecture Contextual knowledge (modality independent) + Inference Perception Modality specific or combination of specific modalities 2

3 Modality specific images driven interpretation The context An interpretation act Contextual expectations An interpretation act Contextual expectations Uninterpreted image! Message/stimulus indexed by expectations! Contextual expectations (Indices) retrieval Uninterpreted image! A qualitative match 3

4 The elements of interpretation Selected intention/expectation relative to the context Interpreted message/expectation! 1) The stimulus 2) The expectations 3) The objects of representation are interpretations! s Examples: Linguistic & visual s of the world (aspects of it) Linguistic Spoken expression Expected intentions Spoken recognition Expected intentions Recognized text The text is an uninterpreted image! 4

5 The memory objects The qualitative match Expected Intentions (Indices) Text Expected. Int i --> Reg. Exp i Expected Intentionndex Regular Expressions Text Selected intentions are those whose associated Reg. Exp. is satisfied by the text (i.e. grep) The output of the interpretation Selected intention (grounded) The intended speech act! In relation to the context Interpreted spoken input! may not be required! Visual name-of-visitor(x) name-of-visitor(peter) Semantic parser Visual expectation Hum, my name I m Peter Concrete visual image 5

6 Spoken recognition Visual expectation SIFT features The memory objects Visual expectations (Indices) SIFT features The qualitative match index images SIFT vector The qualitative match Selected visual expectation Selected expectations are those whose associated vector is the closest to the vector of the external Dr. Luis image Pineda, IIMAS, UNAM, 2009 of the visual act In relation to the context Interpreted image (symbolic rep.) The full architecture: and Action 6

7 The full architecture Output actions Specification Specification Modality specific Realization Realization Reactive behavior Schematic behavior Specification Specification Realization Realization Our current working setting Content Specification Realization Introduction A multimodal architecture for service robots Dialogue models Implementation Applications: Tell me about the magazine! Guess the card! A guided tour with pointing acts Conclusion and future work 7

8 Dialogue Models Dialogue Models Specification Realization The formalism: Functional Recursive Transition Networks (F-RTN) Interactive situation The conversational situation Information state Modality oriented Speech acts and expectations Rhetorical Acts: actions performed by the agent (spoken, motor, etc.) Relative to a context! Conversational Situation Conversational Situation Input SpAct SpAct:RhtAct SpAct:RhtAct SpAct:RhtAct SpAct:RhtAct Only a specific set of intentions meaningful 8

9 Conversational Situation SpAct:RhtAct SpAct:RhtAct Listening Visual Telling Recursive Error Final Intention types RhtAct Rhetorical acts (Actions) Structure: Lists of basic acts Basic Acts: Modality specific actions Function: Direct external behavior Can direct internal behavior (embedded in the interpretation state): Razoning, planning, theorem-proving, problemsolving The Model Conversational model: Set of dialogue models Dialogue model: Set of situations Set of rhetorical acts Declarative specification from a task analysis A Dialogue Model A Recursive Situation is ai:ra 4(ai) rs 1 ε :ra 5(ai) is ai:ra 4(ai) rs 1 ε :ra 5(ai) ε:ra 1(tour) ls 2 pr:ra 4 (pr) rs 2 ε :ra 5(pr) ls 3 ε:ra 1(tour) ls 2 pr:ra 4 (pr) rs 2 ε :ra 5(pr) ls 3 ok:ra2([ai, pr, pl]) pl:ra 4(pl) ε:ra5(pl) ok:ra2([ai, pr, pl]) pl:ra 4(pl) ε:ra5(pl) ls 1 no:ra 3(tour) rs 3 no:ra 3(tour) no:ra3(tour) ok:w ls 1 no:ra 3(tour) rs 3 no:ra 3(tour) no:ra3(tour) ok:w fs ls 2 fs ls 2 9

10 is A subordinated DM ok:ra 1([per, area, proy]) ε:ma(ai) ms 1 no:ra 2 (error) ls 1 per:ra 2 (per) area:ra 2 (area) rs help ts 1 ts 2 proy:ra 2 (proy) ε:ra 3 (proy) no:ra 4(ai) fs ε ts 3 no: ra 4(ai) is ε:ra 3 (per) ε:ra 3 (area) ls 1 ls 2 ok:w Modality independent specification ls 1 ok:ra 1([per, area, proy]) ε:ma(ai) is ms 1 no:ra 2 (error) per:ra 2 (per) area:ra 2 (area) ts 1 ts 2 ε:ra 3 (per) ε:ra 3 (area ) ls 2 proy:ra 2 (proy) ε:ra 3 (proy) no:ra 4(ai) ts 3 ok:w fs no: ra 4(ai) ls 1 ε rs help is Declarative specification of situations Basic expressive power: Recursive Transition Networks [id ==> ls_2, type ==> listening, in_pairs ==> [ls_1 => ok:ra_2([ai,pr,pl]), ls_3 => ok:ra_2([ai,pr]), ls_3 => ok:ra_2([ai,pl]), ls_3 => ok:ra_2([pr,pl])], out_pairs ==> [ai:ra_4(ai) => rs_1, pr:ra_4(pr) => rs_2, pl:ra_4(pl) => rs_3, no:ra_3(tour) => fs], expected_intentions ==> [ai,pr, pl, no] ], Dialogue Manager = Interpreter program Dialogue Models Generation (NL & motor action) Basic Rhetorical Acts (Modality Specific) Speech Act Interpreter Recursive Transition Network Rhetorical Act Rhetorical Act Interpreter Spec. of Mod. Spec. Action 10

11 Specification of actions Specification Realization al objects (Speech acts, Rhetorical acts and Situations) Constants Grounded predicates Variables Predicates with free variables Functions Constants Grounded Predicates a:b +1 p(a,b):q(c,d) +1 Definite intentions and actions Definite intentions and actions Variables Predicates with free-variables (The indexical context) x:p(x) +1 p(a,x):q(x) +1 Abstract intentions and actions Abstract intentions and actions 11

12 Collecting arguments from perception p(a,x) p(a,b) Passing values from input to output p(a,x) p(a,b) q(b) Specification Realization Situations can have parameters The interaction history (anaphoric context) p(a,x):q(x) s j (x) p(a,b):q(c,d) +1 The full path is recorded once an arc has been traveled (i.e. grounded), for all arcs traveled: Abstract intentions, actions and situations dm 1 : =>p(a,b):q(c,d) => s j The interaction history is a list of these terms Functions λx.f 1 (X):λY. f 2 (Y) λz. f 3 (Z) Functional definition of intentions, actions and next situations: X, Y and Z depend on the interaction history, the current DM and the current situation. Functional Actions Suppose a scenario in which the robot is giving a tour to a visitor and there is a situation where it has to make an offer, but taking into account the offert has made before 12

13 Functional Expression of Actions Evaluation _:λy. f 2 (Y) +1 _:λy. f 2 (Y) = p(a,b,c) +1 Where p = do you want me to do Robot: Do you want me to do a or b or c? User: Please, do b But if we come again to this situation _:λy. f 2 (Y) = p(a,c) Where p = do you want me to do Robot: Do you want me to do a or c? User: Please, do a +1 Functional Definition of Suppose the robot is expecting to see one among a number of posters at a specific situation during the interaction (at the same position) Functional Definition of Evaluation λx.f 1 (X):_ +1 λx.f 1 (X) = p(a, b, c) :_ +1 Robot expects to see a poster a or b or c at the current situation Robot recognizes poster a 13

14 But if we come to this situation again λx.f 1 (X) = p(b, c) :_ Robot focusen recognizing b or c +1 The unexpected Every situation can be followed by a recursive situation embedding a recovery dialogue model The information gathered is passed as a parameter to the recovery model The recovery model may use the interaction history You already saw this poster do you want so see it again? Functional Definition of Next Situations Suppose the robot responds up to four questions from the user for each poster then he or she might get bored! Functional Definition of Next Situations _:_ λz. f 3 (Z) Functional definition of next situations Providing information But after the fourth time _: :_ λz. f 3 (Z) = λz. f 3 (Z) = s j Where = +1 Where the system offers to explain a topic of the current poster (not explained before!) Where s j is a continuation dialogue : Do you want to see another poster? 14

15 Declarative specification [id ==> ls_2, type ==> listening, in_pairs ==> [ls_1 => λx.f(x): λx.g(y), ls_3 => ok:λx.h(x), out_pairs ==> [λx.f(a,x):λx.h(y) => λz.rs(z)), no:ra_3(tour) => fs], ], Possible value s types of functions Constants Grounded predicates Variables Predicates with free variables Functions Content The full expressive power: Functional Recursive Transition Networks (F-RTN) Introduction A multimodal architecture for service robots Dialogue models Implementation Applications: Tell me about the magazine! Guess the card! A guided tour with pointing acts Conclusion and future work Implementation Agents environment: Open Agent Architecture (linux) Dialogue models spec. & interpretation: Prolog Speech recognition: Sphinx Acoustic models: The Corpus DIMEx100 Speech synthesis: Mbrola & Festival Implementation Robot navigation: Player/Stage Vision: OpenCV SIFT Robot: Magellan Pro (RWI) Fixed Applications with PCs 15

16 Introduction Content A multimodal architecture for service robots Dialogue models Implementation Applications: Tell me about the magazine! Guess the card! A guided tour with pointing acts Conclusion and Future work Introduction Content A multimodal architecture for service robots Dialogue models Implementation Applications: Tell me about the magazine! Guess the card! A guided tour with pointing acts Conclusion and future work A Bayesian Architecture (with symbolic knowledge) p(e) p(e/o) Future work Speech and language processing: More roboust speech recognition (Language Models) More general NLP capabilities (e.g. Semantic parsing) Richer linguistic explanations p(o/e) Future work Dialogue Models A new DM interpreter (more general) Introduce multimodal situations: listening and seeing Extend the range of speech acts that can be understood (grounding strategies, turn taken strategies) More general recovery strategien the DM Introduce a stochastic component in the F-RTN Introduce internal actions (i.e. thinking ) Future work Vision and navigation More intelligent navigation (e.g. Metric maps, qualitative plans and interaction cycles of move, see, ask) More varied recognition and action devices More control input devices handled from DM directly (contact, laser, ultrasonic, infrared, etc.) 16

17 Future work Architecture Integration of reactive behavior Intergration of schematic behavior Use a multimodal data-base to implement the memory component Construct diverse applicatios with the same platform! Many thanks! 17