Raciocínio e Percepção Espacial: Uma abordagem lógica

Transcription

1 Raciocínio e Percepção Espacial: Uma abordagem lógica Paulo Santos FEI - São Paulo September 10, 2010 Paulo Santos ( FEI - São Paulo ) September 10, / 136

2 Outline I 1 Preface Where/what is FEI? 2 PART I: The Big Picture Introduction and motivation Automated Reasoning 101 Qualitative Spatial Reasoning at a glance 3 Part II: Qualitative Spatial Reasoning Region Connection Calculus Lines of Sight Calculus Region Occlusion Calculus Cardinal Direction Calculus Double Cross Calculus Other calculi Tractability and computability 4 Coffee Break 20 min? Paulo Santos ( FEI - São Paulo ) September 10, / 136

3 Outline II 5 Part III: Cognitive Vision Foundations Cognitive Vision at a glance Early systems Modern systems (from 2000) 6 Part IV: QSR in CogVis Depth Profile Calculus Reasoning about Shadows in Robotics The future: Probabilistic Logic Encoding of Spatial Domains 7 Conclusion Paulo Santos ( FEI - São Paulo ) September 10, / 136

6 Where is FEI? S. Paulo, SP FEI Campus FEI is the largest engineering school in Brazil, with over 8,000 students it is already a regional centre of scientific development for the automotive industry and started investing intensively to become also a regional centre for intelligent robotics. Paulo Santos ( FEI - São Paulo ) September 10, / 136

12 Introduction and motivation reasoning about space is ubiquitous: it is necessary in situations from tying a shoe-lace to urban traffic navigation; automation of spatial reasoning has led to the development of a number of application domains: geographical information systems (GIS) robotics commonsense reasoning natural language processing virtual world modelling and animation medical analysis and diagnosis systems computer vision Paulo Santos ( FEI - São Paulo ) September 10, / 136

13 Introduction and motivation reasoning about space is ubiquitous: it is necessary in situations from tying a shoe-lace to urban traffic navigation; automation of spatial reasoning has led to the development of a number of application domains: geographical information systems (GIS) robotics commonsense reasoning natural language processing virtual world modelling and animation medical analysis and diagnosis systems computer vision Paulo Santos ( FEI - São Paulo ) September 10, / 136

14 Introduction and motivation We acquire knowledge about spatial relationships mainly in two ways: sensory processing: intensively studied in Computer Vision and Robotics; being told, or reading, about spatial arrangements (high-level reasoning). This is the kind of information processing we re concerned about here. In other words, we ll be talking about qualitative reasoning, in contrast to numerical processing. We will also present computer vision systems whose aim is to bridge the gap between sensory processing and high-level reasoning: Cognitive Vision systems. Paulo Santos ( FEI - São Paulo ) September 10, / 136

17 Aim and inspiration for this tutorial We follow the believe expressed in Takeo Kanade Keynote Lecture (given at IJCAI 2003): it is now the time to combine Computer Vision with Relational models and Reasoning. This tutorial presents: tools and methodology of QSR; an overview of major QSR calculi; an overview of Cognitive Vision Systems; examples of QSR systems for Cognitive Vision. Aim: make a brief overview of these areas, presenting the context and foundations to kick start new projects. Paulo Santos ( FEI - São Paulo ) September 10, / 136

22 Why a logic-based approach? has a semantics large variety of distinct logics for different kinds of reasoning variety of inference mechanisms provides a tool kit from which it is possible to characterise the calculi developing relational calculi is well understood Paulo Santos ( FEI - São Paulo ) September 10, / 136

28 Logical Reasoning Every man is mortal. Socrates is a man. Ergo: Socrates is mortal. Paulo Santos ( FEI - São Paulo ) September 10, / 136

32 Logical Reasoning x m(x) mo(x). m(socrates). Ergo: mo(socrates). MODUS PONENS: A B, A B Paulo Santos ( FEI - São Paulo ) September 10, / 136

33 Logical Reasoning x m(x) mo(x). m(socrates). Ergo: mo(socrates). MODUS PONENS: A B, A B Paulo Santos ( FEI - São Paulo ) September 10, / 136

34 Logic programming m(x) : mo(x). m(s).?- m(x)?- X = s Computational processes: resolution, model checking,... Paulo Santos ( FEI - São Paulo ) September 10, / 136

38 Logical Reasoning Deduction: Inferring logical truths All the beans from this bag are white. These beans are from this bag. Ergo, these beans are white. (result) Prolog, Otter, Spass, and much logic programming systems. Abduction: jumping to conclusions I took one bean from this bag and it is white Ergo, all the beans from this bag are white. ACLIP, CIFF, ProLogICA,... Induction: generalising from examples one bean from this bag is white; another bean from this bag is white; another bean from this bag is white;... Ergo, all the beans from this bag are white. Progol, HR, Claudien, FOIL,... Paulo Santos ( FEI - São Paulo ) September 10, / 136

50 Knowledge Representation and Reasoning the logic formalisation of reasoning processes, capable of inferring knowledge from representations of the world; the construction of a medium for efficient computation, in which the formal representation provides the means to organise domain knowledge allowing for efficient (and consistent) queries, updates and revisions of the knowledge base; the rigorous treatment of ontological commitments, which provide the base rules that guide reasoning about the world. For instance, what should or should not be considered as the effects of actions nature of knowledge about temporal entities belief change vagueness spatial entities Davis et al. what is Knowledge Representation?, AI Magazine vol 14, 1993 Paulo Santos ( FEI - São Paulo ) September 10, / 136

61 What is QSR? The formal representation of (qualitative) spatial knowledge in terms of some basic entities and primitive relations in order to allow meaningful and, sometimes, efficient inference methods about space. Paulo Santos ( FEI - São Paulo ) September 10, / 136

62 Spatial Reasoning The basic stories we know best are small stories of events in space: The wind blows clouds through the sky, a child throws a rock, a mother pours milk into a glass, a whale swims through the water. These stories constitute our world. (M.Turner, The Literary Mind) Paulo Santos ( FEI - São Paulo ) September 10, / 136

63 Philosophical origins of QSR the fundamental principles of Geometry were first investigated in ancient Greece by Thales circa 600 B.C.) the laws of valid argument in terms of logical modes of inference were studied separately by early Greek philosophers: analytic geometry (Descartes 1637) 19th century revolution on spatial reasoning: Non-euclidean geometries (e.g. Lobachevsky s hyperbolic geometry (1829) Cantor s point-set topology ( ): Poincaré s algebraic topology Paulo Santos ( FEI - São Paulo ) September 10, / 136

72 Philosophical origins of QSR: 20th century Bertrand Russell and Alfred Whitehead: principles of logic to phenomenological theories, describing the world as it is perceived through sense data; Whitehead (1920): a theory of the perceived world should have as basic entities the very phenomena : integral objects or events geometry becomes concerned with relationships between regions occupied by bodies and dynamical laws with qualitative rules about world events Paulo Santos ( FEI - São Paulo ) September 10, / 136

75 Origins of QSR in Artificial Intelligence phenomenological theories: closer to human reasoning would it be possible to automate them? at least for some particular domains? the construction of formal theories about the qualitative relationships between basic spatial ( phenomenological ) theories is the main goal of qualitative spatial reasoning in AI. but perhaps we should talk about time before talking about space... Paulo Santos ( FEI - São Paulo ) September 10, / 136

80 Allen s interval calculus Figure: adapted from M. Ragni, Reasoning in Dynamic Environments, KI relations any two intervals expressed by one and only one relation (JEPD) temporal reasoning: derive facts about temporal intervals Paulo Santos ( FEI - São Paulo ) September 10, / 136

81 Allen s interval calculus: applications represent activities and temporal knowledge planning and scheduling temporal databases natural language understanding Paulo Santos ( FEI - São Paulo ) September 10, / 136

82 Tools of temporal reasoning Conceptual Neighbourhood Diagrams: graphs representing in their vertices relations on some specific objects; whereas their edges represent continuous transitions between these relations. Continuous transitions: in between adjacent vertices of the graph there is no other relations that the entities in the relation s arguments can assume. Composition (or transitivity) tables: given two relations on any objects a, b, ad c (e.g., R 1 (a, b) and R 2 (b, c)), the composition table entry for R 1 (a, b) and R 2 (b, c) gives the minimal set of disjunctions R 3 (b, c) of the possible relations between a and c. Paulo Santos ( FEI - São Paulo ) September 10, / 136

85 Conceptual Neighbourhood Diagram: illustration Figure: adapted from M. Ragni, Reasoning in Dynamic Environments, KI2006 Paulo Santos ( FEI - São Paulo ) September 10, / 136

86 Transitivity table < > d di... before (<) < no info <; o; m; d; <... s after (>) no info > >; oi; mi; >... d; f during (d) < > no info. d... contains <; o; m; di; >; oi; di; o; oi; dur; di... (di) fi mi; si con; = overlaps < >; oi; di; o; d; s <; o; m... (o) mi; si overlapped <; o; m; di; > oi; d; f >; oi; mi;... by (oi) fi di;si meets (m) < >; oi; mi; o; d; s <... di; si met-by (mi) <; o; m; di; fi > d <; o; m; di; fi... Paulo Santos ( FEI - São Paulo ) September 10, / 136

87 Basic elements in a spatial theory space: absolute or relative; global or local; basic entities: points, regions, directions, bodies, shapes, things, sense-data,... primitive relations: meet, between, connect, part-of,... the set of relations is usually Jointly Exhaustive and Pairwise Disjoints (JEPD) formal tools: axiomatic (deriving axioms and proving theorems), algebraic (encode knowledge with operators and equations), purely logical (design of a spatial logic) Paulo Santos ( FEI - São Paulo ) September 10, / 136

88 Tools of QSR Conceptual Neighbourhood Diagrams Composition (or transitivity) tables Paulo Santos ( FEI - São Paulo ) September 10, / 136

91 Region Connection Calculus (RCC) many sorted, first order logic axiomatisation of spatial regions based on a primitive binary relation about the connection between two regions (C/2). C(x, y) ( x is connected to y ), i.e., the topological closures of x and y share at least one point x C(x, x); xy C(x, y) C(y, x); xyz (C(z, x) C(z, y)) x = y. Paulo Santos ( FEI - São Paulo ) September 10, / 136

92 Region Connection Calculus P(x, y), x is part of y ; O(x, y), x overlaps y ; PP(x, y), x is proper part of y ; Pi/2 and PPi/2 are inverse relations of P/2 and PP/2, resp; DC(x, y), x is disconnected from y ; EQ(x, y), x is equal to y ; PO(x, y), x partially overlaps y ; EC(x, y), x is externally connected to y ; TPP(x, y), x is tangencial proper part of y ; NTPP(x, y), x is non-tangential proper part of y ; TPPi/2 NTPPi/2 are inverses of TPP/2 and NTPP/2 {DC(x, y), EQ(x, y), PO(x, y), EC(x, y), TPP(x, y), NTPP(x, y), TPPi/2, NTPPi/2 } is a JEPD set known as RCC8. Paulo Santos ( FEI - São Paulo ) September 10, / 136

93 RCC: conceptual neighbourhood diagram y x TPP x y x x y x y y DC EC PO x y TPPi x y EQ y NTPP NTPPi x Paulo Santos ( FEI - São Paulo ) September 10, / 136

94 Region Connection Calculus ((DC(x, y) C(x, y))). (1) ((P(x, y) ( z(c(z, x) C(z, y))))). (2) ((PP(x, y) (P(x, y) P(y, x)))). (3) ((EQ(x, y) (P(x, y) P(y, x)))). (4) ((O(x, y) ( z(p(z, x) P(z, y))))). (5) ((PO(x, y) (O(x, y) P(x, y) P(y, x)))). (6) ((DR(x, y) O(x, y))). (7) ((EC(x, y) (C(x, y) O(x, y)))). (8) ((TPP(x, y) (PP(x, y) ( z(ec(z, x) EC(z, y)))))). (9) ((NTPP(x, y) (PP(x, y) ( z(ec(z, x) EC(z, y)))))). (10) ((Pi(x, y) P(y, x))). (11) ((PPi(x, y) PP(y, x))). (12) ((TPPi(x, y) TPP(y, x))). (13) Paulo Santos ( FEI - São Paulo ) September 10, / 136

95 RCC: Transitivity table DC EC PO TPP... DC no info. DR,PO, PP DR,PO, PP DR,PO, PP... EC DR,PO, DR,PO, DR,PO, EC,PO,... PPi TPP PP PP PO DR,PO, DR,PO, no info. PO,PP... PPi PPi TPP DC DR DR,PO, PP... PP NTPP DC DC DR,PO, NTPP... PP TPPi DR,PO, EC,PO, PO,PPi PO,TPP... PPi PPi NTPPi DR,PO, PO,PPi PO,PPi PO,PPi... PPi EQ DC EC PO TPP... Paulo Santos ( FEI - São Paulo ) September 10, / 136

98 Lines of Sight Calculus Represent the relative positions between pairs of (non-overlapping) convex bodies Figure: Randell et a. From Images to Bodies: Modelling and Exploiting Spatial Occlusion and Motion Parallax. IJCAI 2001 Paulo Santos ( FEI - São Paulo ) September 10, / 136

99 Lines of Sight Calculus B A C B A JC JH PH JF JHI PHI JFI B A B A A B A B B A B A B A H A B EH A F B A HI B B A EHI B FI A Paulo Santos ( FEI - São Paulo ) September 10, / 136

100 Lines of Sight Calculus 14 relations: C(x, y), is clear from; JC(x, y), is just clear from; PH(x, y), partially hides; PHI(x, y), is partially hidden by; JH(x, y), just hides; JHI(x, y), is just hidden; H(x, y), hides; HI(x, y), is hidden by; EH(x, y), exactly hides; EHI(x, y), is exactly hidden; F(x, y), is in front of; FI(x, y), has y in front of it; JF (x, y), is just in front of; JFI(x, y), has y just in front of it; Paulo Santos ( FEI - São Paulo ) September 10, / 136

103 Region Occlusion Calculus (ROC) Region Occlusion Calculus (ROC) is an extension of RCC to represent the various possibilities of interposition (occlusion) between arbitrary shaped bodies. Two functions: region and image. region: maps a physical body to is occupancy region. image maps a physical body to its bi-dimensional projection from a particular viewpoint. Paulo Santos ( FEI - São Paulo ) September 10, / 136

104 Region Occlusion Calculus (ROC) ROC primitive relations: C/2 and TotallyOccludes(x, y, ν) ( body x totally ocludes body y from the viewpoint ν ). x ν TotallyOccludes(x, x, ν) x y z νtotallyoccludes(x, y, ν) TotallyOccludes(y, z, ν)) TotallyOccludes(x, z, ν) Paulo Santos ( FEI - São Paulo ) September 10, / 136

105 Region Occlusion Calculus The following axioms introduce RCC8 in ROC:. ((TotallyOccludes(x, y, ν) P(region(z), region(y))) TotallyOccludes(x, z, ν)) (TotallyOccludes(x, y, ν) z(p(region(z), region(y))) TotallyOccludes(z, x, ν)) (TotallyOccludes(x, y, ν) z u(p(region(z), region(x)) P(region(u), region(y))) TotallyOccludes(u, z, ν)) y z(p(region(y), region(x)) P(region(z), region(x)) TotallyOccludes(y, z, ν)) (TotallyOccludes(x, y, ν) P(image(y, ν), image(x, ν))) Paulo Santos ( FEI - São Paulo ) September 10, / 136

106 Region Occlusion Calculus We can introduce a weaker notion of occlusion Occludes/3: Occludes(x, y, ν) z u(p(region(z), region(x)) P(region(u), region(y)) TotallyOccludes(z, u, ν)) Paulo Santos ( FEI - São Paulo ) September 10, / 136

107 Region Occlusion Calculus Non occlusion, partial occlusion, mutual occlusion: PartiallyOccludes(x, y, ν) Occludes(x, y, ν) TotallyOccludes(x, y, ν) Occludes(y, x, ν) MutuallyOccludes(x, y, ν) Occludes(x, y, ν) Occludes(y, x, ν) NonOccludes(x, y, ν) Occludes(x, y, ν) Occludes(y, x, ν) NonOccludes(x, y, ν) DR(image(x, ν), image(y, ν)) PartiallyOccludes(x, y, ν) (PO(image(x, ν), image(y, ν)) PP(image(x, ν), image(y, ν))) MutuallyOccludes(x, y, ν) (PO(image(x, ν), image(y, ν)) P(image(x, ν), image(y, ν)) PI(image(x, ν), image(y, ν))) Paulo Santos ( FEI - São Paulo ) September 10, / 136

108 Region Occlusion Calculus NonOccludesDC NonOccludesEC PartiallyOccludesPO MutuallyOccludesPO PartiallyOccludesTPP PartiallyOccludesPO!1 TotallyOccludesTPPI TotallyOccludesNTPPI TotallyOccludesTPPI!1 MutuallyOccludesNTPP MutuallyOccludesTPP MutuallyOccludesEQ PartiallyOccludesNTPP TotallyOccludesEQ MutuallyOccludesTPP!1 PartiallyOccludesTPP!1!1 PartiallyOccludesNTPP MutuallyOccludesNTPP!1 TotallyOccludesNTPPI!1 TotallyOccludesEQ!1 Paulo Santos ( FEI - São Paulo ) September 10, / 136

111 Cardinal Direction Calculus Cardinal Direction Calculus (CDC) is a formalism for reasoning about the directions between spatial objects 9 base relations: north, south, east, west, northeast, northwest, southeast, southwest e EQ (EQ(x, y) means x is at the same direction as y ). Main goal of CDC is to infer facts about the relative direction of two objects A and B from the known directions between A and C (A C and B C). E.g., from north(a, B) and northeast(b, C), the task is to calculate the possible directions between A e C. Paulo Santos ( FEI - São Paulo ) September 10, / 136

112 Cardinal Direction Calculus Cardinal Direction Calculus (CDC) is a formalism for reasoning about the directions between spatial objects 9 base relations: north, south, east, west, northeast, northwest, southeast, southwest e EQ (EQ(x, y) means x is at the same direction as y ). Main goal of CDC is to infer facts about the relative direction of two objects A and B from the known directions between A and C (A C and B C). E.g., from north(a, B) and northeast(b, C), the task is to calculate the possible directions between A e C. Paulo Santos ( FEI - São Paulo ) September 10, / 136

113 Cardinal Direction Calculus Paulo Santos ( FEI - São Figure: Paulo ) Adapted from SparQ User Manual v0.7 September 10, / 136

116 Double Cross Calculus is a calculus that defines the direction of a point with respect to a directed line segment 15 ternary relations on points represents every distinct relation between the directions left-right and front-back (e.g.left-front, left-back, left-line, left -perpendicular, straight-front,...) motivation: qualitative description of paths Paulo Santos ( FEI - São Paulo ) September 10, / 136

120 Double Cross Calculus Figure: Adapted from SparQ User Manual v0.7 Paulo Santos ( FEI - São Paulo ) September 10, / 136

121 Double Cross Calculus Paulo Santos ( FEI - São Paulo ) September 10, / 136

124 Other Calculi size and distance: are defined either in absolute scale (whereas the relations < or >) are introduced in the usual way), or in relative terms, where the relative connection between three objects is used to define the relations of proximity (near than, far than) and equidistance. Size and distance calculi are usually coupled with other calculi to extend their expressivity, shapes: one of the least understood areas of QSR. In general, shape is defined by means of a number of primitives such as interior or boundary. Paulo Santos ( FEI - São Paulo ) September 10, / 136

125 Other Calculi default: not much has been done wrt default theories about space; Shanahan formalises a pre-condition about spatial occupancy of objects assuming that space is empty by default. spatial change: base to the development of spatial change is the work of Galton, where both time instants and intervals are included. Two predicates are used to this end: Holds T represents a spatial state that is true at a time instant, whereas Holds I represents true stated during a time interval. From this, 8 distinct kinds of transitions between pairs of states are defined in order to represent the relation of two states in time. Qualitative Trajectory Calculi, Line Segments, Dipole Calculi, and many others! Paulo Santos ( FEI - São Paulo ) September 10, / 136

126 SparQ toolbox for QSR in applications reference implementations of QSR typical QSR procedures uniform interface Paulo Santos ( FEI - São Paulo ) September 10, / 136

127 Applications of QSR qualitative simulation of physical systems syntax and semantics of visual programming languages databases integration GIS real time event recognition robotics Paulo Santos ( FEI - São Paulo ) September 10, / 136

135 Tractability and computability efficiency is inversely proportional to expressivity first order formulation of mereotopology is not decidable find a decidable subset (e.g. RCC8) look for tractable subsets Paulo Santos ( FEI - São Paulo ) September 10, / 136

140 Tractability and computability Tractable subsets of QSR formalism is the subject of a number of papers. In particular (Renz e Cohn 2008) presents the following ingredients for finding tractable subsets of such formalism: a method to prove that a given subset if tractable a method to suggest possible tractable subsets in order to prove that a given set of relations is tractable, it is sufficient to prove that the inclusion of any new relation makes the set intractable J. Renz, Qualitative Spatial and Temporal Reasoning: Efficient Algorithms for Everyone, in: Proc (IJCAI-07) Paulo Santos ( FEI - São Paulo ) September 10, / 136

141 Tractability and computability Tractable subsets of QSR formalism is the subject of a number of papers. In particular (Renz e Cohn 2008) presents the following ingredients for finding tractable subsets of such formalism: a method to prove that a given subset if tractable a method to suggest possible tractable subsets in order to prove that a given set of relations is tractable, it is sufficient to prove that the inclusion of any new relation makes the set intractable J. Renz, Qualitative Spatial and Temporal Reasoning: Efficient Algorithms for Everyone, in: Proc (IJCAI-07) Paulo Santos ( FEI - São Paulo ) September 10, / 136

146 Our knowledge of the external world (...) what is actually given in sense is much less than most people would naturally suppose, and (...) much of what at first sight seems to be given is really inferred. This applies especially in regard to our space-perceptions. For instance, we unconsciously infer the real size and shape of a visible object from its apparent size and shape, according to its distance and our point of view. (...). Thus, the first step in the analyses of data, namely, the discovery of what is really given in sense, is full of difficulty. the External World, pp.75-76] [B. Russell (1914), Our Knowledge of Although these ideas are at the foundations of QSR, their application in scene analysis is still work in progress. Paulo Santos ( FEI - São Paulo ) September 10, / 136

147 Our knowledge of the external world (...) what is actually given in sense is much less than most people would naturally suppose, and (...) much of what at first sight seems to be given is really inferred. This applies especially in regard to our space-perceptions. For instance, we unconsciously infer the real size and shape of a visible object from its apparent size and shape, according to its distance and our point of view. (...). Thus, the first step in the analyses of data, namely, the discovery of what is really given in sense, is full of difficulty. the External World, pp.75-76] [B. Russell (1914), Our Knowledge of Although these ideas are at the foundations of QSR, their application in scene analysis is still work in progress. Paulo Santos ( FEI - São Paulo ) September 10, / 136

150 What is CogVis? from ECvision network A Cognitive Vision System can achieve the four levels of generic visual functionality: Detection, Localisation, Recognition, Understanding (role, context, purpose) and exhibits purposive goal-directed behaviour, is adaptive to unforeseen changes, and can anticipate the occurrence of objects and events. This is achieved through: Learning semantic knowledge (form, function and behaviours) Retention of knowledge (about the cognitive system, its environment, and the relationship with the environment) Deliberation about objects and events, including the cognitive system itself. Paulo Santos ( FEI - São Paulo ) September 10, / 136

151 Some approaches for CogVis VITRA system- Visual Translator: Integration of computer vision and natural language processing ALVEN system - textual description of all heart dynamics via X-Ray image sequences. Brand s visual understanding through causal analysis Siskind s systems - Event classification from camera input using force dynamics Leeds traffic interaction - Modelling traffic interaction using learnt qualitative spatio-temporal relations and variable length Markov models Leeds Protocol learning - Combining continuous and symbolic models to learn games from observation FEI contributions - Spatial reasoning image analysis Paulo Santos ( FEI - São Paulo ) September 10, / 136

154 Nagel s Image Sequence Evaluation Requirements and purpose: exhaustive internal representation for all tasks and experimental conditions it is expected to handle situation: suitable intermediate representation during the evaluation of image sequences derive scene-specific conceptual descriptions from image sequences, based on general assumptions about motion as the cause of observable changes Image sequence evaluation is based on the idea that the flow of image sequences reflects (to some extent) the coherency of the conceptual world. Paulo Santos ( FEI - São Paulo ) September 10, / 136

158 Nagel s Image Sequence Evaluation Intermediate levels of description: change: any deviation of sensor signal which significantly differs from noise event: any change which has been defined a priori as a primitive for the construction of more complex descriptions verb: describe activities history: extended sequence of activities Paulo Santos ( FEI - São Paulo ) September 10, / 136

159 Some early systems Morio (Nagel 1983), Naos (Neumann 1986), Epex (Nagel 1987), CityTour (Herzog 1986) usually describing traffic scenes/situations from an static viewpoint (using motion verbs such as overtake, approach, move, beside, recede but also to describe a walking person (Hogg 1983), heart motion (Tsotos 1980) and so on Paulo Santos ( FEI - São Paulo ) September 10, / 136

162 Project Vitra: Visual Translator long term project: 80 s and 90 s design and construction of integrated knowledge-based systems capable of translating visual information into natural language descriptions general goals: extend the scope of scene analysis beyond the level of object recognition explicit description of spatial configurations by means of spatial relations interpretation of object movement automatic recognition of goals and plans of observed agents specific goals: answering queries about traffic situations generating reports of football games communicating with a mobile robot Paulo Santos ( FEI - São Paulo ) September 10, / 136

171 Project Vitra: Visual Translator Overview of the system: from raw data, image analysis generates a geometrical representation of the scene, with object s locations through time the geometric description is interpreted by the cognitive level (or high-level scene analysis) this high-level analysis extracts: spatial relations interesting motion events presumed intentions, plans and plan interactions between agents the final, linguistic, level transform the conceptual descriptions into utterances Paulo Santos ( FEI - São Paulo ) September 10, / 136

176 Alven (1984) natural language description of ventricular wall motion data extracted from X-ray image sequences knowledge organisation: ontology Analyse pre-operative and post-operative marker films to evaluate the efficacy of surgery analysis using both quantitative and qualitative representations major issues: understanding visual motion from image tokens over time reasoning about spatio-temporal relationships Paulo Santos ( FEI - São Paulo ) September 10, / 136

182 Alven Paulo Santos ( FEI - São Paulo ) September 10, / 136

183 Alven s knowledge base Set of classes organised using relations about IS_A: generalisation/specialisations (taxonomy) PART _OF: part/whole (partonomy) temporal precedence Should satisfy: 1 motion classes should be sufficient to express the domain 2 image tokens are connected to general knowledge in the leaves of a PART _OF hierarchy of motion concepts Paulo Santos ( FEI - São Paulo ) September 10, / 136

184 Alven s knowledge base Set of classes organised using relations about IS_A: generalisation/specialisations (taxonomy) PART _OF: part/whole (partonomy) temporal precedence Should satisfy: 1 motion classes should be sufficient to express the domain 2 image tokens are connected to general knowledge in the leaves of a PART _OF hierarchy of motion concepts Paulo Santos ( FEI - São Paulo ) September 10, / 136

185 Alven s control structure extract tokens from the input signal, instantiating then in the leaves of the PART _OF hierarchy follows this hierarchy to activated hypotheses that are aggregates of the input tokens this set of hypotheses is specialised by going down one level in the IS_A hierarchy each hypothesis is matched with other data instances: matching leads to further specialisations, failure leads to the selection of other hypotheses the best hypotheses generate a set of predictions predictions are mapped back to the image level and are used as a guidance to the token extraction procedure Paulo Santos ( FEI - São Paulo ) September 10, / 136

191 Brand s visual understanding through causal analysis describe the causal structure from images knowledge about physical causality is used in the interpretation scenes hypothesis: a small core set of qualitative rules accounts for most of what humans ordinarily see it is possible to make useful inferences with qualitative knowledge about connectivity and free space qualitative rules of connectivity, friction, attachment and penetration interpretation of simple mechanical machines purpose Analyse diagnosis prediction inspection Paulo Santos ( FEI - São Paulo ) September 10, / 136

198 Brand s SPROCKET (1997) Paulo Santos ( FEI - São Paulo ) September 10, / 136

201 Siskind s Event classification from camera input using force dynamics similar to Brand s but on image sequences force dynamics: SUPPORTED, ATTACHED, PICKUP, PUTDOWN, CONTACT more recently: supervised learning of visual event definitions from video Paulo Santos ( FEI - São Paulo ) September 10, / 136

204 Siskind s Leonard System Paulo Santos ( FEI - São Paulo ) September 10, / 136

205 Leeds Traffic Interaction apply QSR to dynamic scene analysis learning events from the observation of traffic scenes recognising these events and generate predictions a scene is interpreted from the comparison of its temporal development with a (previously learned) transition diagram representing change in the domain. Paulo Santos ( FEI - São Paulo ) September 10, / 136

206 CND of relative positions Paulo Santos ( FEI - São Paulo ) September 10, / 136

207 Leeds Traffic Interaction Paulo Santos ( FEI - São Paulo ) September 10, / 136

208 Protocol learning A system for learning protocol behaviour from computer vision data using ILP Unsupervised continuous learning of perceptual categories and unsupervised symbolic learning of protocols (as sets of Horn clauses); Paulo Santos ( FEI - São Paulo ) September 10, / 136

209 Protocol Learning Off-the-shelf vision system Attention: identifies key frames with no motion preceded by a number of frames with notion; Statistical classifier: assigns different classes to clusters of features Off-the-shelf ILP system: Progol Generalises a set of positive only examples according to user defined mode declarations Mode declarations restrict the possible form of the proposed generalisations Paulo Santos ( FEI - São Paulo ) September 10, / 136

210 Protocol Learning Paulo Santos ( FEI - São Paulo ) September 10, / 136

211 Protocol Learning Inducing axioms of ordering and equivalence, without knowing about numbers Building equivalence classes to cope with over clustering sound and completeness of the agent actually playing the game [AIJ 05] Paulo Santos ( FEI - São Paulo ) September 10, / 136

214 Example of QSR from Vision: Depth Profile Calculus How much knowledge about a robot s environment can be from vision alone? How can we construct the knowledge about objects in the world from sensor data of the robot? Construction of a qualitative spatial reasoning (QSR) system based on sensor data Use abduction for sensor data assimilation and deduction for predictions Paulo Santos ( FEI - São Paulo ) September 10, / 136

218 Simplified environment Paulo Santos ( FEI - São Paulo ) September 10, / 136

219 Spatial Reasoning about Robot Sensor Data Attributes: Distance, disparity, size; Changes in the sensor data; Representation: Depth profiles and time points; Displacement between regions; Mapping function between images and objects Paulo Santos ( FEI - São Paulo ) September 10, / 136

220 Assimilating changes Axioms of the system: < Dynamic spatial rel > < desc. sensor transition > < Dynamicspatialrel. > < obj obs relation > Paulo Santos ( FEI - São Paulo ) September 10, / 136

221 Depth Profiles Paulo Santos ( FEI - São Paulo ) September 10, / 136

222 The model Extract one horizontal depth profile of each scene from the visual data; Objects in the scenes are represented as peaks; Axiomatise relations on the depth and size of these profiles as well as displacements; Paulo Santos ( FEI - São Paulo ) September 10, / 136

223 Depth Profile Calculus A theory about displacement, size and depth; 27 base relations; Large and complex conceptual neighbourhood diagrams and composition tables; Paulo Santos ( FEI - São Paulo ) September 10, / 136

224 Depth Profile Calculus Paulo Santos ( FEI - São Paulo ) September 10, / 136

225 DPC example Paulo Santos ( FEI - São Paulo ) September 10, / 136

228 Reasoning about Shadows in Robotics making explicit the knowledge contained in cast shadows use it to reason about the robot environment Computer vision however has largely been filtering out cast shadows as noise Paulo Santos ( FEI - São Paulo ) September 10, / 136

231 Illusory Motion from Shadows Paulo Santos ( FEI - São Paulo ) September 10, / 136

232 Perception of shadows no luminous body ever sees the shadows that it generates [da Vinci, Notebooks of Leonardo Da Vinci. Project Gutenberg (1888)] from the light source viewpoint shadows are occluded by their casters We model observer-caster-shadow within qualitative spatial reasoning: ROC + an axiom about shadow : Shadow(s, o, Scr, L) PO(r(s), r(scr)) TotallyOccludes(o, s, L) o TotallyOccludes(o, o, L). a shadow is totally occluded by its caster from the lightsource viewpoint Paulo Santos ( FEI - São Paulo ) September 10, / 136

236 Qualitative regions for self-localisation * L o 2 1 S 1 Paulo Santos ( FEI - São Paulo ) September 10, / 136

237 In practice Qualitative robot self-localisation relative depth from the observation of shadows threshold finding from qualitative regions Paulo Santos ( FEI - São Paulo ) September 10, / 136

238 In practice 3 * L o 2 1 S 1 located(region 1, ν, o, s) Is_a_Shadow(s, o) NonOccludesDC(o, s, v) v o; located(region 2, ν, o, s) Is_a_Shadow(s, o) NonOccludesEC(o, s, v) v o; located(region 3, ν, o, s) Is_a_Shadow(s, o) PartiallyOccludesPO(o, s, v) v o; located(region 4, ν, o, s) Is_a_Shadow(s, o) TotallyOccludesTPPI(o, s, v) v o; Paulo Santos ( FEI - São Paulo ) September 10, / 136

241 Probabilistic Logic Encoding of Spatial Domains incorporate incomplete sensor data and domain knowledge in a probabilistic logic setting explore inferences about space Paulo Santos ( FEI - São Paulo ) September 10, / 136

242 Traffic Scenario Paulo Santos ( FEI - São Paulo ) September 10, / 136