Natural Language Processing Semantics Compositional

Transcription

1 Natural Language Processing Semantics Compositional Mats Dahllöf Institutionen för lingvistik och filologi December 2013

2 Natural language processing and semantics Two directions : Analysis: NL meaning representation NL free, in principle. Classification, retrieval. Synthesis (generation): meaning representation NL NL designed. Easier. Meaning representation logic.

3 Logic and semantics Aristotelian logic important ever since. Inference. (Patterns in AL: syllogisms.) E.g.: Premise: No reptiles have fur. Premise: All snakes are reptiles. Conclusion: No snakes have fur. Modern logic develops, late 19th Century more general and systematic. Formal semantics in linguistics and philosophy based on logic (20th Century). Applications in language technology and computer science.

4 Formal and computational semantics Computational semantics is the study of how to automate the process of constructing and reasoning with meaning representations of natural language expressions. (Wikipedia). Early systems rule-based, most famous example: Montague grammar (1970). Sophisticated mechanisms for translation of English into a very rich logic. (NLTK: a version of this.) Language technology: Recent interest in data-driven and machine learning-based methods. More robust less sophistcated/fine-grained.

5 Semantics in NLP NLP semantics is typically more limited in scope than NL semantics as analysed in linguistics and philosophy. NLP applications often handle semantic aspects without having explicitly semantic components, e.g. in machine translation. Other aspects of language morphology, syntax, etc. can be seen as support systems for semantics: The purpose of language lies in the use of expressions as carriers of semantic meaning. And that is what many NLP systems have to respect, e.g. MT, retrieval, classification, etc.

6 Semantics and truth Semantics, meanings and states of affairs: What a sentence means: a structure involving (lexical) concepts and relations among them. Can be articulated as a semantic representation. E.g. I ate a turkey sandwich. in predicte logic: x(ate(i, x) Sandwich(x) ContainsTurkey(x)) A sentence and the semantic representation of a sentence is also the representation of a possible state of affairs.

7 Semantics and truth Correspondence theory of truth: If the content of a sentence corresponds to an actual state of affairs if it is true; otherwise, it is false. Ignoring philosophical complications, in many cases we can extract knowledge from texts. E.g. Warmer climate entails increased release of carbon dioxide by inland lakes. (From uu.se press release.) Related issue: Which texts should we trust? Many sentences are difficult to formalize in logic. (Modality, conditionality, vague quantification, tense, etc.)

8 Semantics vs pragmatics/discourse What does a word, a phrase, a text segment mean as an NL expression? ( Linguistic meaning semantics.) Conventional, static, systemic aspect of meaning. What does the author intend to convey by means of a word, a phrase, a text segment? ( Speaker meaning pragmatics/discourse.) Contextual, dynamic aspect of meaning. The two aspects depend on each other, of course.

9 Semantics vs pragmatics/discourse After a stellar two-year run, the bond market is stumbling and a number of investors are betting that stocks will post better returns in the coming months. (Wall Street Journal.) semantics pragmatics/discourse communicative intentions e.g. a wish to advise, caution, etc. relation to context (coherence) open reference specific reference intended (reference resolution)

10 Semantics vs pragmatics/discourse After a stellar two-year run, the bond market is stumbling and a number of investors are betting that stocks will post better returns in the coming months. (Wall Street Journal.) semantics pragmatics/discourse vagueness more well-defined concepts intended run, bond, market (NLP: WSD) ambiguity literal content specific interpretation intended metaphoric interpretation stellar, stumbling

11 Semantics-oriented NLP applications Machine translation: The translation of a text segment should mean the same as the original (to emphasize linguistic meaning) or should convey the same content (to emphasize speaker meaning). Information extraction is to extract components of the information conveyed by a text. Question answering is extraction combined with inference of an answer to a given question. Text classification, in typical cases, relates to the meanings of the texts being classified.

12 Semantics and generation Generation: semantic representation NL. Less challenging than analysis the structure of the input is under control. Needed in e.g. dialogue systems. Interlingua semantic representation in machine translation: Analysis: source language interlingua. Generation: interlingua target language. Would be economic if many languages are involved. The idea has not proved very successful so far.

13 Reference Reference is very important what statements are about. Referring expressions are very common. Reference is a discourse phenomenon. Resolving reference is a crucial step in e.g. extraction, e.g.in sentiment analysis translation, e.g. to get agreement right English it vs French il/elle vs Swedish den/det.

14 Reference, an example The portrait, comments, and updates provide constant reminders of the existence of friends. The content is not all that important, but the effect is that we perceive our Facebook friends as closer than other acquaintances who are not on Facebook. (From a uu.se press release.) Referring expressions are interpreted under coherence assumptions: The content of a Facebook page the effect of Facebook interaction with people we who are (potential) users of Facebook

15 Kinds of referring expressions Indefinite noun phrases. E.g. a book. Introduce new entities. Pronouns. E.g. he. Typically coreferent with a previous referring expression (antecedent). Names. E.g. Mats Dahllöf/Mats. Demonstrative. E.g. this room. Other definite noun phrases. E.g. the first chapter. Reference to somehow known entity, often previously mentioned.

16 Named entity recognition (NER) To identify expressions being used as names. (What characterizes a name?) Also to identify what kind of name it is: E.g. of a person, or a place, or a stretch of time, or a chemical compound, or a gene, etc. State-of-the-art NER systems for English produce near-human performance. For example, the best system entering MUC-7 scored 93.39% of F-measure while human annotators scored 97.60% and 96.95% (Wikipedia, Named-entity recognition).

17 Anaphora and deixis resolution Pronouns (they), pronominal adverbs (there, then), and definite NP s refer to entities by means of contextually given information. E.g. by referring to previously mentioned referents anaphora. E.g. by reference based on the participants, time, and place of the discourse deixis (e.g. I, you, here, yesterday). Anaphora and deixis resolution is much more challenging task than NER. The reference of name-like graph words is much more predictible. Compare Barack Obama and he.

18 Sentiment analysis an extraction task What views do people express in blogs and reviews? That s interesting for politicans and marketing people. Opinions are often expressed in a personal and informal way. E.g. Peter bought me a Baileys marzipan chocolate thing which I washed down with Gluehwein and that, in combination with the bright lights and cheery faces really made me feel warm inside! (From a blog post.) Sentiment analysis: to extract the referent of a sentiment and the polarity positive negative associated with it. E.g. Baileys marzipan chocolate positive.

19 Sentiment analysis, two dimensions Subjectivity: neutral vs polar. Polarity: positive vs negative. Demo (NLTK):

20 Sentiment analysis, two dimensions Demo (NLTK): This movie is tremendously enjoyable, and considering how exotic it is, The Desolation of Smaug is weirdly unassuming. It rattles along, never drags, and is always terrifically likable: open, genial and good-natured. (From a movie review, Subjectivity neutral: 0.1 polar: 0.9 Polarity pos: 0.8 neg: 0.2

21 Opinion definition An opinion is a quintuple (e,a,s,h,t), where e is a target entity. a is an aspect/feature of the entity e. s is the sentiment value of the opinion from the opinion holder h on aspect a of entity e at time t. s is +ve, -ve, or neu, or a more granular rating. h is an opinion holder. t is the time when the opinion is expressed. (Bing Liu)

22 Lexical concepts Words are often grammatically simple, but carry a structured conceptual content. Definitions unpack the content of concepts: friend a person whom one knows well, is loyal to, etc. turkey a kind of animal, a bird, etc. sandwich a kind of food item, contains bread, etc. eat a relation (holding in/of an event) between an organism and a food item, the food is chewed and ingested, etc.

23 Lexical concepts decomposition Componential analysis hen +chicken +female +adult rooster +chicken -female +adult chick +chicken -adult sow +pig +female +adult boar +pig -female +adult piglet +pig -adult

24 Lexical concepts relations Hypernym (superordinate concept): novel book hen chicken bird animal But all the publicity surrounding Franzen s newest novel, Freedom, edged me on to read the first chapter of the book How To Truss Your Chicken [... ] Catch the wings with the string, pulling them close to the sides of the bird. from blog posts.

25 Lexical concepts relations Part meronym book chapter bird wing; bird side; But all the publicity surrounding Franzen s newest novel, Freedom, edged me on to read the first chapter of the book How To Truss Your Chicken [... ] Catch the wings with the string, pulling them close to the sides of the bird. from blog posts.

26 Synonymy Synonymy holds between two words (word tokens) which express the same or similar concepts. Unsupervised detection of synonymy can be based on The Distributional Hypothesis: words with similar distributions have similar meanings. Random Indexing is a method here. Synonymy knowledge useful in e.g. translation, text classification, and information extraction. Also query expansion in retrieval.

27 Formalizing meaning (see J&M Ch. 17) Linguistic content has at least to a certain degree a logical structure that can be formalized by means of logical calculi meaning representations. The representation languages should be simple and unambiguous in contrast to complex and ambiguous NL. Logical calculi come with accounts of logical inference. They are useful for reasoning-based applications. Meaning formalization faces far-reaching conceptual and computational difficulties.

28 Compositionality (J&M Ch. 18) Linguistic content is compositional: Simple expressions have a given (lexical) meaning; the meaning of complex expressions is determined by the meanings of their constituents. People produce and understand new phrases and sentences all the time. (NLP must also deal with these.) Compositionality is studied in detail in compositional syntax-driven semantics. Work in this field is typically about hand-coded rule systems for small fragments of NL.

29 Compositional aspects I ate a turkey sandwich. a: some item was thus involved. Logic: existential quantification. This item was a turkey sandwich. turkey sandwich: a kind of sandwich somehow being of a turkey kind.

30 Predicates and arguments Predicates when associated (through the argument roles) with arguments give us propositions, which are true or false, or at least say something about how things are. Many tasks in information extraction are about finding propositions of a certain kind. Predicate words: verbs, adjectives, nouns, adverbs. Predicate argument analysis in NLP is typically course-grained compared to what formal logical semantics would require.

31 Compositional aspects argument structure I ate a turkey sandwich. Predicate argument structure analysis or semantic role labeling would find: I: The speaker was in relation to ate the eater or first argument. (Pronominal reference resolution is a matter of discourse-related processing.) a turkey sandwich: Was in relation to ate the eaten or second argument. Time could be seen as a third argument expressed as the verb s tense.

32 Syntactic function vs semantic role There are systematic connections between syntactic functions and semantic roles, but these relations also vary: One of the primary difficulties in labeling semantic roles is that one predicate may be used with different argument structures: for example, in the sentences He opened the door and The door opened, the verb open assigns different semantic roles to its syntactic subject. (Gildea and Jurafsky, 2002)

33 Discourse-related aspects I ate a turkey sandwich. I: pronoun resolution (deixis). ate: tense preterite this event took place in the past. Pragmatics: When? How far away in the past would be relevant? Does the context provide a setting for this eating? turkey sandwich: sandwich somehow related to turkey Comtext-sensitive interpretation: sandwich made with turkey meat as an ingredient.

34 Compositional semantics in NLP I ate a turkey sandwich. Just predicate argument structure (more basic): predicate: ate eater: I eaten: a turkey sandwich NLP: semantic role labelling. Predicate logic (something like): x(ate(i, x) Sandwich(x) ContainsTurkey(x)) (Referent of I: pronoun resolution.)

35 Predicate argument data If we want to do supervised predicate argument analysis, we need training data. Would unsupervised methods be possible when it comes to predicate argument analysis? Most well-known corpus of this kind: PropBank (Proposition Bank) English (mostly Wall Street Journal) annotated with the predicate-argument structure of the verbs. (Also other languages.) (Martha Palmer, and co-workers, 2002.) There is also a NomBank with noun annotation.

36 PropBank semantic roles neutral semantic roles, as opposed to verb-specific ones E.g. give: Arg0: giver Arg1: thing given Arg2: entity given to Rather than giver, gift, recipient. (PropBank does not give us propositions where the predicate comes from e.g. a noun or an adjective.)

37 System: Semantic role labeler, Lund joint syntactic-semantic analysis bottom-up projective parser using pseudo-projective transformations the semantic model uses global inference mechanisms on top of a pipeline of classifiers Richard Johansson and Pierre Nugues, 2008, Dependency-based Syntactic-Semantic Analysis with PropBank and NomBank, CoNLL 2008: Proceedings of the 12th Conference on Computational Natural Language Learning,

38 System: Semantic role labeler, Lund Semantic model in three steps: A SRL classifier pipeline that generates a list of candidate predicate-argument structures. A constraint system that filters the candidate list to enforce linguistic restrictions on the global configuration of arguments. A global classifier that rescores [... ] the filtered candidate list.

39 System: Semantic role labeler, Lund Linguistically motivated global constraints CORE ARGUMENT CONSISTENCY. Core argument labels must not appear more than once. DISCONTINUITY CONSISTENCY. If there is a label C-X, it must be preceded by a label X. REFERENCE CONSISTENCY. If there is a label R-X and the label is inside a relative clause, it must be preceded by a label X.

40 System: Semantic role labeler, Lund Crucial to our success was the high performance of the syntactic parser, which achieved a high accuracy. integration of syntactic and semantic analysis is beneficial for both sub-tasks. F1-scores around 0.8.

41 System: Enju Enju is a syntactic parser for English. wide-coverage probabilistic HPSG grammar efficient parsing algorithm provides phrase structures and predicate-argument structures useful for high-level NLP applications, including information extraction, automatic summarization, and question answering

42 Enju output example The company that he runs is small output: ROOT ROOT ROOT ROOT -1 ROOT ROOT is be is be VBZ VB 5 verb_arg12 ARG1 company company is be VBZ VB 5 verb_arg12 ARG2 small small small small JJ JJ 6 adj_arg1 ARG1 company company The the DT DT 0 det_arg1 ARG1 company company that that IN IN 2 relative_arg1 ARG1 company company runs run VBZ VB 4 verb_arg12 ARG1 he he runs run VBZ VB 4 verb_arg12 ARG2 company company

43 LinGO English Resource Grammar broad-coverage, linguistically precise HPSG-based grammar of English semantically grounded in Minimal Recursion Semantics (MRS), which is a form of flat semantic representation developed around the turn of the century.

44 First-order predicate logic flexible, well-understood, and computationally tractable approach to the representation of knowledge [and] meaning (J&M p. 589) expressive verifiability against a knowledge base (related to database languages) inference model-theoretic semantics

45 First-order predicate logic Boolean operators: negation and connectives Existential/universal quantification Individual constants Predicates (taking a number of arguments)

46 When to assume compositionality? Many phrases are lexicalized in a way that gives a non-compositional lexicalized sense and a less plausible compositional reading: hard disc digital memory device part of speech word category green card permanent residence permit (U.S.) This kind of ambiguity is a problem in e.g. machine translation.

47 Major points Logic-based semantics is a theoretical foundation for NLP semantics, but implemented systems are typically more coarse-grained and of a more limited scope. Meaning depends both on literal content and contextual information. This is a challenge for most NLP tasks. Most NLP applications have to be highly sensitive to semantics.

48 Major points Finding and interpreting names and other referential expressions is a central issue for NLP semantics. Disambiguation of polysemous lexical tokens is also a central issue for NLP semantics. Accessing the content of lexical tokens is also useful. Meaning representation involves predicate-argument structure, which captures a basic aspect of NL compositionality.