Towards robust symbolic reasoning about information from the web

Transcription

1 Towards robust symbolic reasoning about information from the web Steven Schockaert School of Computer Science & Informatics, Cardiff University, 5 The Parade, CF24 3AA, Cardiff, United Kingdom s.schockaert@cs.cardiff.ac.uk Abstract. The need for vast amounts of machine-readable commonsense knowledge is one of the key obstacles hampering progress towards artificial general intelligence (AGI). While machine-readable knowledge is increasingly becoming available on the web, two important challenges remain. First, available information is mainly fact based, whereas more complex domain knowledge (e.g. in the form of rules) is needed to support common-sense reasoning. Second, because information on the web is inherently noisy, methods that take advantage of it need to be sufficiently robust. State-of-the art solutions to these challenges are mostly based on some form of statistical relational learning to induce probabilistic rule bases from the set of available facts. In this position paper, we argue that to support AGI a greater emphasis on qualitative methods is needed (i) to further increase the robustness of inference results, (ii) to implement strategies for dealing with vagueness, inconsistency and missing or incomplete knowledge, (iii) to ensure that knowledge bases are easy to understand and edit by users, and (iv) to allow inference results to be explained in an intuitive way. 1 Introduction More and more data is becoming available on the web in a structured, machinereadable form. Semantic wikis such as Google s Freebase 1, for instance, are collaborative efforts to compile formal knowledge bases from assertions made by their users (covering more than 23 million entities in the case of Freebase). Structured data can also be obtained by analysing statements in natural language (e.g. [3, 4, 24]), and from existing datasets published as linked data on the web (e.g. open government data 2 ). Although a number of formal ontologies and rule bases have been made available as well (e.g. the open-domain ontology Cyc 3 and various domain-specific ontologies) most knowledge bases on the web are essentially repositories of facts. For reasoning on the other hand, we need rules that can be used to infer new

2 facts from known ones and to verify whether sets of facts are logically consistent. Recent progress in statistical relational learning makes it possible to learn such rules automatically. One popular approach is to learn Markov logic networks from available facts [22], in which case learnt knowledge is represented as weighted first-order formulas, with the weights corresponding to those of an associated Markov network. Along similar lines, [11] proposes two strategies for learning probabilistic rules from noisy facts that have been extracted from the web. The resulting probabilistic theories have proven useful for learning new facts. For example, [22] reports an experiment in which 31,000 probabilistic inference rules have been learned from 250,000 facts. By using these inference rules the total number of facts known to the system increased to over 1,300,000 although the precision (i.e. the percentage of correct facts) at the same time dropped from about 90% to about 50%. Similarly, probabilistic rules are shown in [11] to lead to substantially better performance in a system that is aimed at learning instances of relations and concepts from the web (e.g. person X plays for team Y). These approaches to learning facts and rules from the web represent a crucial step towards semantic search engines, which can infer answers to questions from users from structured knowledge, and semantic search engines can in turn be seen as a crucial step towards AGI. However, relying exclusively on probabilistic inference over learned rules will not be sufficient, for a number of reasons: The probabilistic weights associated with rules have to be learned from noisy input data, and often from positive examples alone. As a result these weights should be interpreted as heuristic confidence scores rather than accurate probability estimates. Moreover, probabilities are strongly contextdependent, and it is doubtful that the corpora which are used to learn the probabilities always reflect context in which they are applied to a sufficient extent. Moreover, information extraction methods tend to introduce structural errors, often leading to overly optimistic probability estimates [22]. To be useful in practice, a semantic search engine would need the ability to explain how it has inferred a given conclusion to users without any training in probability theory or logic, e.g. using some form of controlled natural language [6]. Without an adequate explanation, users would not be able to judge whether they can trust the answer. However, there are often a large number of formulas which contribute to the probability with which a given conclusion can be derived, and the weights associated with each of these formulas is not always easily interpretable (e.g. in Markov logic networks). When explicit knowledge to answer a given question is missing, humans tend to resort to similarity-based or analogical reasoning, which is difficult to formalise in a probabilistic framework. More precisely, while probabilistic models may be extended with similarity-based reasoning capabilities (see e.g. [26]), it is not clear how the parameters of such rich models can be estimated sufficiently accurately from noisy web data. To address these issues, inference should as much as possible be based on commonsense reasoning over symbolic knowledge. Specifically, a probabilistic theory

3 which has been learned from the web could be approximated as a set of default rules, where a default rule if α then generally β can be interpreted as P (α β) P (α β), a view which can be formalized using the notion of big-stepped probabilities [2]. These default rules could then be published as part of a semantic wiki, allowing users to edit, remove or expand available knowledge. This user interaction is crucial, as fully automated methods are not likely to be sufficiently accurate. It is also in line with previous crowd-sourced approaches to knowledge acquisition, such as Wikipedia, Freebase, and the Open Mind Common Sense project. As the default rules only rely on orders-of-magnitude of probabilities, they are less sensitive to changes in context. To implement this symbolic approach to reasoning about information from the web, a number of fundamental challenges still need to be addressed. Most importantly, a form of commonsense reasoning needs to be developed which is tolerant to inconsistency and to the vagueness of natural language, and which can still draw plausible conclusions when explicit knowledge is missing. Such forms of commonsense reasoning would need to crucially rely on similarity information. While similarity relations can be induced from the web in a data-driven manner [7, 17, 25], such numerical similarity degrees have issues which are similar to those of probability degrees that are estimated from the web. In particular, similarity degrees are highly context-dependent and may be subjective, and the actual degrees that are obtained may not be reliable if insufficient high-quality data is available. Thus there is a need to develop a qualitative counterpart of similarity degrees, analogous to the way default reasoning offers a qualitative counterpart to probability theory. The next section briefly discusses a possible solution, using qualitative spatial relations between geometric cognitive representations to express similarity information in a qualitative way. Finally, Section 3 outlines how these spatial relations can be used to develop a theory of commonsense reasoning that is sufficiently robust to the noisiness of information from the web. 2 Conceptual relations as qualitative similarity Conceptual spaces [8] are geometric spaces in which the meaning of natural language properties and concepts can be represented. Conceptual spaces are often assumed to be Euclidean spaces, even though this is a simplification. The dimensions of a conceptual space represent cognitively primitive features. An individual object has precise values for these features and thus corresponds to a point, while properties and concepts correspond to regions. A central assumption in the theory of conceptual spaces is that the regions corresponding to natural properties or concepts are convex. This assumption finds its roots in prototype theory [14], which posits that the membership of an object to a category is based on its distance to the prototypes of that category, relative to the distances to prototypes of other categories. A standard example of conceptual spaces is in the domain of colours. Colours can be represented in a three-dimensional conceptual space, with dimensions corresponding to hue, saturation and intensity.

4 Every point in this space corresponds to a specific colour, and the colours that correspond to natural language terms such as red have been shown to indeed correspond to convex regions [9], although the exact boundaries of such regions may be vague. In most domains, conceptual space representations of properties and concepts are not accessible to us. Concepts such as restaurant, for example, can only be represented in a high-dimensional space, and it is not even clear what exactly would be the relevant quality dimensions. Even in the colour domain, the exact extension of regions corresponding to specific colour terms may not be clear. However, for commonsense reasoning, all we need is information about particular qualitative spatial relations that hold between the regions in a conceptual space. For example, an important problem in commonsense reasoning concerns the modelling of vague natural language terms. When merging information from different sources, logical inconsistencies may arise due to slightly different interpretations of vague terms. In [19] a solution was presented based on knowledge of which concepts correspond to adjacent regions in a conceptual space. The rationale is that the exact boundary between two regions is only partially determined for vague concepts, hence some objects may be considered to be instances of some concept by one source and instances of another concept by a second source. To resolve inconsistencies caused by vagueness, we thus need to be aware of which concepts share a (vague) boundary. As another example, [18] and [20] studied approaches to interpolative and analogical reasoning, aimed at completing rule bases with missing rules. For instance, assume that we know that undergraduate students do not pay tax and PhD students do not pay tax, but have no knowledge about whether master s students pay tax. Given that master s students are conceptually between undergraduate students and PhD students, we may draw the plausible conclusion that master s students do not pay tax. This conceptual betweenness can be identified with geometric betweenness in an underlying (unknown) conceptual space. Note that betweenness relates to comparative similarity. In particular, if the point b is between the points a and c in a Euclidean space (with a, b and c distinct points), then for every point x it holds that d(b, x) < max(d(a, x), d(c, x)). The notion of comparative similarity has also been used in a more direct way in [23]. Finally, note that analogical proportions of the form a is to b as c is to d correspond to parallelograms in an underlying conceptual space. In other words, a is to b as c is to d holds if the way a and b are different is similar to the way c and d are different [12]. Obtaining sufficiently accurate information from the web about qualitative spatial relationships in conceptual spaces should be considerably simpler than attempting to acquire the actual conceptual space representations. For example, [27] and [1] propose methods to discover analogies from analyzing natural language text. Another possible strategy, used in [25] and [17], is to use dimensionality reduction techniques such as multi-dimensional scaling and singular value decomposition to induce geometric models from data. These geometric models may be viewed as very coarse approximations of conceptual spaces, which may however be sufficiently accurate for discovering relevant qualitative relationships.

5 One important challenge that remains is reasoning about such qualitative spatial relations, to maintain consistency when integrating information from different sources, and to make explicit the consequences of known relations. For example, knowing that taiga is between tundra and temperateforest, and that taiga subsumes borealforest, we should be able to conclude that borealforest is between tundra and temperateforest. Spatial relations such as adjacency and parthood can be expressed in the Region Connection Calculus (RCC [13]), for which sound and complete decision methods are available. In [15], it was moreover shown that the requirement in conceptual spaces that regions be convex does not affect these decision procedures. In [16], it was shown how sound and complete reasoning can be done when RCC5, a fragment of the RCC, is extended with the notion of betweenness. However, no results are available for sound and complete reasoning about parallelism or comparative similarity, in combination with (fragments of) the RCC, and in this sense a general theory for reasoning about qualitative similarity information is still missing. 3 Reasoning about information from the web As explained above, we can derive two types of qualitative knowledge from the web. On the one hand, we can derive default rules which encode information about the order-of-magnitude of probabilities, and on the other hand we can derive conceptual relations which encode how different natural language categories are semantically related. In [21] it was shown how default reasoning in the sense of Kraus, Lehmann and Magidor [10] can be interpreted in a spatial way, thus reducing commonsense reasoning to the problem of reasoning over the qualitative spatial structure of conceptual spaces. In particular, to interpret defaults, we need order-of-magnitude information about how frequently an agent has encountered exemplars that belong to different areas of a conceptual space. To interpret a default rule if α then β it is useful to note that α and β correspond to regions in some conceptual space. This default rule then encodes the constraint that α β includes the part of α where the density of exemplars is highest (where we identify α and β with their corresponding region for the ease of presentation). Being inherently data-driven, the resulting approach would be more robust than approaches based on classical logic, while it is still symbolic and can thus naturally be used to interact with users (to explain results or to take into account user feedback). Still, by reducing numerical similarity degrees and probabilities to qualitative structures, important information is lost, and any purely qualitative approach is likely to be too cautious. While such numerical values should be seen as heuristic information, they can be very useful in practice. For example, we may use the qualitative approach outlined in this paper to generate arguments supporting or refuting particular conclusions, and then rely on available numerical information to rank these arguments from the most to the least plausible. In this way, we would rely on a combination of reliable symbolic knowledge (which is generic, and intuitive to understand and edit by humans) with heuris-

6 tic, data-driven methods (which are application-dependent and mostly hidden from users). Such an approach would be in line with the dual process account of human reasoning [5], according to which human reasoning relies on a combination of unconscious, heuristic processing and explicit, rule-based processing. 4 Conclusions This paper considered the problem of common-sense reasoning about information from the web, as a stepping stone to semantic search, and ultimately AGI. Any method which is to be successful for this task should reflect the quality of the input data. In particular, as accurate probabilities cannot be estimated from noisy positive examples alone, we argued that qualitative structures may be more appropriate. We also emphasized the importance of user interaction, as feedback is needed to correct systematic and other errors from the system, and intuitive explanations need to be provided to allow users to assess the plausibility of a conclusion. Finally, we stressed the fact that, in addition to uncertainty, similarity also has a crucial role to play in the formalization of commonsense reasoning. References 1. M. Bayoudh, H. Prade, and G. Richard. Evaluation of analogical proportions through kolmogorov complexity. Knowledge-Based Systems, 29(0):20 30, S. Benferhat, D. Dubois, and H. Prade. Possibilistic and standard probabilistic semantics of conditional knowledge bases. Journal of Logic and Computation, 9(6): , A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. R. Hruschka Jr, and T. M. Mitchell. Toward an architecture for never-ending language learning. In Proceedings of the Twenty-Fourth Conference on Artificial Intelligence, pages , O. Etzioni, A. Fader, J. Christensen, S. Soderland, and M. Mausam. Open information extraction: The second generation. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence, pages AAAI, J. S. Evans. In two minds: dual-process accounts of reasoning. Trends in Cognitive Sciences, 7(10): , N. E. Fuchs, K. Kaljurand, and T. Kuhn. Attempto Controlled English for knowledge representation. In Reasoning Web, volume 5224 of Lecture Notes in Computer Science, pages Springer, E. Gabrilovich and S. Markovitch. Computing semantic relatedness using wikipedia-based explicit semantic analysis. In Proceedings of the 20th International Joint Conference on Artificial Intelligence, volume 6, pages , P. Gärdenfors. Conceptual Spaces: The Geometry of Thought. MIT Press, G. Jäger. Natural color categories are convex sets. In M. Aloni, H. Bastiaanse, T. Jager, and K. Schulz, editors, Logic, Language and Meaning, volume 6042 of Lecture Notes in Computer Science, pages Springer Berlin Heidelberg, S. Kraus, D. Lehmann, and M. Magidor. Nonmonotonic reasoning, preferential models and cumulative logics. Artificial Intelligence, 44(1-2): , 1990.

7 11. N. Lao, T. Mitchell, and W. W. Cohen. Random walk inference and learning in a large scale knowledge base. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages , H. Prade and G. Richard. Reasoning with logical proportions. In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, pages , D. Randell, Z. Cui, and A. Cohn. A spatial logic based on regions and connection. In Proceedings of the 3rd International Conference on Knowledge Representation and Reasoning, pages , E. H. Rosch. Natural categories. Cognitive Psychology, 4(3): , S. Schockaert and S. Li. Convex solutions of RCC8 networks. In Proceedings of the 20th European Conference on Artificial Intelligence, pages , S. Schockaert and S. Li. Combining rcc5 relations with betweenness information. In Proceedings of the International Joint Conference on artificial Intelligence, S. Schockaert and H. Prade. Interpolation and extrapolation in conceptual spaces: A case study in the music domain. In Proceedings of the 5th International Conference on Web Reasoning and Rule Systems, pages , S. Schockaert and H. Prade. Qualitative reasoning about incomplete categorization rules based on interpolation and extrapolation in conceptual spaces. In Proceedings of the Fifth International Conference on Scalable Uncertainty Management, pages , S. Schockaert and H. Prade. Solving conflicts in information merging by a flexible interpretation of atomic propositions. Artificial Intelligence, 175(11): , S. Schockaert and H. Prade. Cautious analogical-proportion based reasoning using qualitative conceptual relations. In Proceedings of the ECAI 2012 Workshop on Similarity and Analogy-based Methods in AI, pages 41 48, S. Schockaert and H. Prade. Interpolative reasoning with default rules. In Proceedings of the International Joint Conference on Artificial Intelligence, S. Schoenmackers, O. Etzioni, D. S. Weld, and J. Davis. Learning first-order horn clauses from web text. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages , M. Sheremet, D. Tishkovsky, F. Wolter, and M. Zakharyaschev. A logic for concepts and similarity. 17(3): , R. Speer and C. Havasi. ConceptNet 5: A large semantic network for relational knowledge. In I. Gurevych and J. Kim, editors, The People s Web Meets NLP: Collaboratively Constructed Language Resources, Theory and Applications of Natural Language Processing, pages Springer-Verlag, R. Speer, C. Havasi, and H. Lieberman. Analogyspace: reducing the dimensionality of common sense knowledge. In Proceedings of the 23rd AAAI Conference on Artificial intelligence, pages , J. B. Tenenbaum and T. L. Griffiths. Generalization, similarity, and bayesian inference. Behavioral and Brain Sciences, 24: , P. D. Turney. Measuring semantic similarity by latent relational analysis. In Proceedings of the 19th International Joint Conference on Artificial Intelligence, pages , 2005.