A Dynamic Oracle for Arc-Eager Dependency Parsing
|
|
- Maximillian Whitehead
- 8 years ago
- Views:
Transcription
1 A Dynamic Oracle for Arc-Eager Dependency Parsing Yoav Gold ber g 1 Joakim N ivre 1,2 (1) Google Inc. (2) Uppsala University yogo@google.com, joakim.nivre@lingfil.uu.se ABSTRACT The standard training regime for transition-based dependency parsers makes use of an oracle, which predicts an optimal transition sequence for a sentence and its gold tree. We present an improved oracle for the arc-eager transition system, which provides a set of optimal transitions for every valid parser configuration, including configurations from which the gold tree is not reachable. In such cases, the oracle provides transitions that will lead to the best reachable tree from the given configuration. The oracle is efficient to implement and provably correct. We use the oracle to train a deterministic left-to-right dependency parser that is less sensitive to error propagation, using an online training procedure that also explores parser configurations resulting from non-optimal sequences of transitions. This new parser outperforms greedy parsers trained using conventional oracles on a range of data sets, with an average improvement of over 1.2 LAS points and up to almost 3 LAS points on some data sets. KEYWORDS: dependency parsing, transition system, oracle. Proceedings of COLING 2012: Technical Papers, pages , COLING 2012, Mumbai, December
2 1 Introduction The basic idea in transition-based dependency parsing is to define a nondeterministic transition system for mapping sentences to dependency trees and to perform parsing as search for the optimal transition sequence for a given sentence (Nivre, 2008). A key component in training transition-based parsers is an oracle, which is used to derive optimal transition sequences from gold parse trees. These sequences can then be used as training data for a classifier that approximates the oracle at parsing time in deterministic classifier-based parsing (Yamada and Matsumoto, 2003; Nivre et al., 2004), or it can be used to determine when to perform updates in online training of a beam search parser (Zhang and Clark, 2008). Currently, such oracles work by translating a given tree to a static sequence of parser transitions which, if run in sequence, will produce the gold tree. Most transition systems, including the arc-eager and arc-standard systems described in Nivre (2003, 2004), exhibit spurious ambiguity and map several sequences to the same gold tree. In such cases, the oracles implicitly define a canonical derivation order. We call such oracles static, because they produce a single static sequence of transitions that is supposed to be followed in its entirety. Static oracles are usually specified as rules over individual parser configurations if the configuration has properties X and the gold tree is Y, then the correct transition is Z giving the impression that they define a function from configurations to transitions. However, these rules are only correct for configurations that are part of the canonical transition sequence defined by the oracle. Thus, static parsing oracles are only correct as functions from gold-trees to transition sequences, and not as functions from configurations to transitions. There are at least two limitations to training parsers using static oracles. First, because of spurious ambiguity, it is not clear that the canonical transition sequence proposed by the oracle is indeed the easiest to learn. It could well be the case that a different sequence which also leads to a gold tree is preferable. Second, it happens often in greedy deterministic parsing that the parser deviates from the gold sequence, reaching configurations from which the correct tree is not derivable. The static oracle does not provide any means of dealing with such deviations. The parser s classifier is then faced with configurations which were not observed in training, and this often leads to a sequence of errors. It would be preferable for the parser to explore non-gold configurations at training time, thus mitigating the effect of error-propagation. In this paper, we introduce the concept of a dynamic parsing oracle. Rather than defining a single static canonical transition sequence for producing a given gold tree, the dynamic oracle answers queries of the form: Is transition Z valid in configuration X for producing the best possible tree Y? A key difference compared to a static oracle is that the dynamic oracle no longer forces a unique transition sequence in situations where multiple sequences derive the gold tree. In this case, the dynamic oracle permits all valid transition sequences by answering yes on more than one transition Z in a given configuration. 1 The second crucial difference to a static oracle is that the dynamic oracle defined in this work is well-defined and correct for all configurations, including configurations which are not a part of a gold derivation. For configurations which are not part of a gold derivation (and from which the gold tree is not reachable), the dynamic oracle permits all transitions that can lead to a tree with minimum loss compared to the gold tree. In this paper, we provide a provably correct dynamic oracle for the arc-eager transition system of Nivre (2003, 2008). One important use for a dynamic oracle is in training a parser that (a) is not restricted to a 1 This is similar to the parsing oracle used in the EasyFirst parser of Goldberg and Elhadad (2010). 960
3 PRD SBJ IOBJ ROOT 0 He 1 wrote 2 her 3 P DOBJ a 4 DET letter 5. 6 Figure 1: Projective dependency tree particular canonical order of transitions and (b) can handle configurations that are not part of any gold sequence, thus mitigating the effect of error propagation. To this end, we provide an online training procedure based on the dynamic oracle that deals with spurious ambiguity by treating all sequences leading to a gold tree as correct, and with error-propagation by exploring transition sequences that are not optimal in the sense that they do not derive the gold tree, while training the parser to perform the optimal transitions on these non-optimal configurations. We show that both of these properties improve the accuracy of a deterministic left-to-right arc-eager parser by over 1.2 LAS points on average and up to almost 3 LAS points on some data sets, as compared to the conventional training procedure. 2 Background 2.1 Arc-Eager Transition-Based Dependency Parsing Given a set L of dependency labels, we define a dependency graph for a sentence x = w 1,..., w n as a labeled directed graph G = (V x, A), consisting of a set of nodes V x = {0, 1,..., n}, where each node i corresponds to the linear position of a word w i in the sentence, plus an extra artificial root node 0, and a set of labeled arcs A V x L V x, where each arc (i, l, j) represents a dependency with head w i, dependent w j, and label l. In order for a dependency graph to be well-formed, we usually require it to be a dependency tree, which is a directed spanning tree rooted at the node 0. In this paper, we further restrict our attention to dependency trees that are projective, that is, where the presence of an arc (i, l, j) entails that there is directed path from i to every node k such that min(i, j)<k<max(i, j). Figure 1 shows a projective dependency tree for a simple English sentence. In the arc-eager transition system of Nivre (2003), a parser configuration is a triple c = (Σ, B, A) such that Σ (referred to as the stack) and B (the buffer) are disjoint sublists of the nodes V x of some sentence x, and A is a set of dependency arcs over V x (and some label set L); we take the initial configuration for a sentence x = w 1,..., w n to be c s (x) = ([0], [1,..., n], { }); and we take a terminal configuration to be any configuration of the form c = (Σ, [ ], A) (for any stack Σ and arc set A). 2 There are four types of transitions, defined formally in Figure 2, for going from one configuration to the next: 1. A LEFT-ARC l transition (for any dependency label l) adds the arc (b, l, s) to A, where s is the node on top of the stack and b is the first node in the buffer, and pops the stack. It has as a precondition that the token s is not the artificial root node 0 and does not already have a head. 2 As is customary, we use the variablesσandβ for arbitrary sublists of the stack and the buffer, respectively. For reasons of perspicuity, we will write Σ with its head (top) to the right and B with its head to the left. Thus, c = (σ s, b β, A) is a configuration with the node s on top of the stack Σ and the node b as the first node in the buffer B. 961
4 Transition Precondition LEFT-ARC l (σ i, j β, A) (σ, j β, A {( j, l, i)}) [i = 0] k l [(k, l, i) A] RIGHT-ARC l (σ i, j β, A) (σ i j,β, A {(i, l, j)}) REDUCE (σ i,β, A) (σ,β, A) k l[(k, l, i) A] SHIFT (σ, i β, A) (σ i,β, A) Figure 2: Transitions for the arc-eager transition system 2. A RIGHT-ARC l transition (for any dependency label l) adds the arc (s, l, b) to A, where s is the node on top of the stack and b is the first node in the buffer, and pushes the node b onto the stack. 3. The REDUCE transition pops the stack and is subject to the preconditions that the top token has a head. 4. The SHIFT transition removes the first node in the buffer and pushes it onto the stack. A transition sequence for a sentence x is a sequence C 0,m = (c 0, c 1,..., c m ) of configurations, such that c 0 is the initial configuration c s (x), c m is a terminal configuration, and there is a legal transition t such that c i = t(c i 1 ) for every i, 1 i m. The dependency graph derived by C 0,m is G cm = (V x, A cm ), where A cm is the set of arcs in c m. The arc-eager system is sound and complete for the class of projective dependency forests, meaning that every legal transition sequence derives a projective dependency forest (soundness) and that every projective dependency forest is derived by at least one transition sequence (completeness) (Nivre, 2008). A projective dependency forest is a dependency graph where every connected component is a projective tree, with the special case of a projective dependency tree in case there is only one connected component. Hence, the arc-eager transition system is also complete (but not sound) for the class of projective dependency trees. 2.2 Static Oracles for Transition-Based Parsing In the transition-based framework, parsing is implemented as search for an optimal transition sequence, that is, a transition sequence that derives the correct parse tree for a given sentence. The simplest version of this, and also the most efficient, is to train a classifier to predict the single best transition in each configuration and use a greedy deterministic procedure to derive a single dependency graph, which results in linear-time parsing provided that each transition can be predicted and executed in constant time (Nivre, 2003, 2008). The classifier is trained on a set of configuration-transition pairs, which can be derived from a dependency treebank by finding an optimal transition sequence for each sentence x with gold tree G gold = (V x, A gold ). Algorithm 1 defines the standard oracle function used to find the next transition t for a configuration c and gold tree G gold = (V x, A gold ). This algorithm is provably correct in the sense that, for every sentence x and projective dependency tree G gold = (V x, A gold ), if we initialize c to c s (x) and repeatedly execute the oracle transition, then we derive exactly G gold = (V x, A gold ) (Nivre, 2006). Nevertheless, it has two important limitations. 962
5 Algorithm 1 Standard oracle for arc-eager dependency parsing 1: if c = (σ i, j β, A) and ( j, l, i) A gold then 2: t LEFT-ARC l 3: else if c = (σ i, j β, A) and (i, l, j) A gold then 4: t RIGHT-ARC l 5: else if c = (σ i, j β, A) and k[k<i l[(k, l, j) A gold ( j, l, k) A gold ]] then 6: t REDUCE 7: else 8: t SHIFT 9: return t The first is that it ignores spurious ambiguity in the transition system, that is, cases where a given dependency tree can be derived in more than one way. The dependency tree in Figure 1 is derived by two distinct transition sequences: 3 (1) SH, LA SBJ, RA PRD, RA IOBJ, SH, LA DET, RE, RA DOBJ, RE RA P (2) SH, LA SBJ, RA PRD, RA IOBJ, RE, SH, LA DET, RA DOBJ, RE RA P Algorithm 1 will predict (1) but not (2). More generally, whenever there is a SH-RE ambiguity, which is the only ambiguity that exists in the arc-eager system, the oracle prediction will always be SH. In this way, the oracle implicitly defines a canonical transition sequence for every tree. The second limitation is that we have no guarantee for what happens if we apply the oracle to a configuration that does not belong to the canonical transition sequence. In fact, it is easy to show that the oracle prediction in such cases can be suboptimal. For example, suppose that we erroneously choose the SH transition instead of RA IOBJ after the first three transitions in sequence (1). This results in the following parser configuration: ([0, 2, 3], [4, 5, 6], {(0, PRD, 2), (2, SBJ, 1)}) Starting from this configuration, the oracle defined by Algorithm 1 will predict SH, LA DET, SH, SH, which derives the dependency graph in the left-hand side of Figure 3. Using labeled attachment score to measure loss, this graph has a loss of 3 compared to the correct tree in Figure 1, since it fails to include the arcs (2, IOBJ, 3), (2, DOBJ, 5), (2, P, 6). 4 However, if we instead apply the transitions SH, LA DET, LA DET, RA DOBJ, RE, RA P, we end up with the tree in the right-hand side of Figure 3, which only has a loss of 1. We say that Algorithm 1 defines a static oracle, because it produces a single static sequence of transitions that is supposed to be followed in its entirety. The main contribution of this paper is the notion of a dynamic oracle, which does not presuppose a single canonical transition sequence for each dependency tree and which can dynamically adapt to arbitrary configurations that arise during parsing and still make optimal predictions. 3 To save space, we sometimes use the following abbreviations: LA l = LEFT-ARC l, RA l = RIGHT-ARC l, RE = REDUCE, SH = SHIFT. 4 In most practical parser implementations, this graph is converted into a tree by adding arcs from the root node to all words that lack a head. However, the loss will be exactly the same. 963
6 PRD SBJ ROOT 0 He 1 wrote 2 PRD DET SBJ her 3 a 4 letter 5. 6 ROOT 0 He 1 wrote 2 P DOBJ DET DET her 3 a 4 letter 5. 6 Figure 3: Dependency graphs with loss 3 (left) and loss 1 (right) 3 Dynamic Parsing Oracles We want to define an oracle that (a) allows more than one transition sequence for a given tree and (b) makes optimal predictions in all configurations (not only configurations that are part of a globally optimal transition sequence). It follows from the first requirement that the oracle should define a relation from configurations to transitions, rather than a function, and we will represent it as a boolean function o(t; c, G gold ), which returns true just in case t is optimal in c relative to G gold. But what does it mean for a transition to be optimal? Intuitively, a transition t is optimal if it does not commit us to a parsing error, which we take to mean that the best dependency tree reachable from c is also reachable from t(c). Consider the set of all dependency trees that are reachable from c. From this set, we pick the set of trees that minimize some loss function relative to G gold and say that t is optimal if and only if at least one tree in this set is still reachable from t(c). More precisely, we define the cost of t to be the difference in loss between the best tree reachable in c and the best tree reachable in t(c) and say that t is optimal if it has zero cost. In the special case where the gold tree G gold is reachable from c, the set of trees that minimize the loss function is the singleton set containing G gold, which entails that t is optimal if and only if G gold is still reachable from t(c). Let us now try to make this precise. 3.1 Defining the Oracle First, we define the loss (G, G gold ) of a dependency graph G = (V x, A) with respect to the gold tree G gold = (V x, A gold ) to be the number of arcs that are in G gold but not in G: (G, G gold ) = A gold \ A We then say that a dependency graph G = (V x, A) for a sentence x is reachable from a nonterminal configuration c for x, written c G if and only if there is a non-empty sequence of transitions t 1,..., t m such that [t m t 1 ](c) = (Σ, [], A) and G = (V x, A). Next, we define the cost (t; c, G gold ) of the transition t in the configuration c relative to the gold tree G gold as the loss difference between the minimum loss tree reachable before and after the transition: (t; c, G gold ) = min (G, G gold ) G:t(c) G min (G, G gold) G:c G Note that, by definition, there must be at least one zero cost transition for every configuration c and gold tree G gold. To see why, let G be some dependency graph with minimum loss reachable from c. Since G is reachable from c, there must be at least one transition t such that G is reachable from t(c). And since (G, G gold ) (G, G gold ) = 0, it follows that (t; c, G gold ) =
7 Finally, we define the oracle o(t; c, G gold ) to return true just in case t has zero cost relative to c and G gold : true if (t; c, Ggold ) = 0 o(t; c, G gold ) = false otherwise 3.2 A Dynamic Oracle for Arc-Eager Parsing In order to implement the dynamic oracle in practice, we need an efficient method for computing the cost of each transition in a given configuration. A key property of the arc-eager system (stated here without proof) is that a dependency graph G = (V x, A) is reachable from a configuration c if G is a projective forest and if each individual arc in A is reachable in c. In the Arc-Eager system, an arc (i, l, j) is reachable in c = (Σ, B, A ) if either (i, l, j) is already in A (since arcs can never be removed) or if min(i, j) is in Σ or B, max(i, j) is in B, and there is no arc in A that already assigns a head to j (since it is always possible to reach a later configuration where min(i, j) is at the top of the stack and max(i, j) is at the head of the buffer, in which case the arc can be added in a LEFT-ARC l or RIGHT-ARC l transition). Given that our loss function (and hence our cost function) also decomposes into the loss of individual arcs, we can compute the cost of each transition by simply counting the gold arcs that are no longer reachable after that transition. We do this on a case by case basis. In all the cases, we assume a configuration c of the form c = (σ s, b β, A). 5 (LEFT-ARC l ; c, G gold ): Adding the arc (b, l, s) and popping s from the stack means that s will not be able to acquire any head or dependents inβ. The cost is therefore the number of arcs in A gold of the form (k, l, s) or (s, l, k) such that k β. Note that the cost is 0 for the trivial case where (b, l, s) A gold, but also for the case where b is not the gold head of s but the real head is not inβ (due to an erroneous previous transition) and there are no gold dependents of s inβ. 6 (RIGHT-ARC l ; c, G gold ): Adding the arc (s, l, b) and pushing b onto the stack means that b will not be able to acquire any head inσorβ, nor any dependents inσ. The cost is therefore the number of arcs in A gold of the form (k, l, b), such that k σ or k β, or of the form (b, l, k) such that k σ. Note again that the cost is 0 for the trivial case where (s, l, b) A gold, but also for the case where s is not the gold head of b but the real head is not inσorβ (due to an erroneous previous transition) and there are no gold dependents of b inσ. (REDUCE; c, G gold ): Popping s from the stack means that s will not be able to acquire any dependents in b β. The cost is therefore the number of arcs in A gold of the form (s, l, k) such that k b β. While it may seem that a gold arc of the form (k, l, s) should be accounted for as well, note that a gold arc of that form, if it exists, is already accounted for by a previous (erroneous) RIGHT-ARC l transition when s acquired its head. (SHIFT; c, G gold ): Pushing b onto the stack means that b will not be able to acquire any head or dependents in s σ. The cost is therefore the number of arcs in A gold of the form (k, l, b) or (b, l, k) such that k s σ. 5 This is a slight abuse of notation, since for the SHIFT transition s may not exist, and for the REDUCE transition b may not exist. 6 One may want to associate a lower cost with cases in which the arc endpoints are correct and only the label is wrong. This extension is trivial. 965
8 (LA l ; c = (σ s, b β, A), G g ) (RA l ; c = (σ s, b β, A), G g ) (RE; c = (σ s,β, A), G g ) (SH; c = (σ, b β, A), G g ) = {(k, l, s) A g k β} {(s, l, k) A g k β} = {(k, l, b) A g k σ k β} {(b, l, k) A g k σ} = {(s, l, k) A g k β} = {(k, l, b) A g k σ} {(b, l, k) A g k σ} Figure 4: Transition costs for the arc-eager transition system with gold tree G g = (V x, A g ). The computation of transition costs is summarized in Figure 4. We can now return to the example in Section 2.2 and analyze the behavior of the static oracle in the presence of erroneous transitions. In the example there, the last two SHIFT transitions predicted by the static oracle each has a cost of 1, because they each lose an arc going into the first word of the buffer. By contrast, the transition LEFT-ARC DET in place of the first SHIFT has a cost of 0, despite the fact that the arc (5, DET, 3) is not in the gold tree, because it does not eliminate any gold arc that is still reachable the cost of the incorrect attachment is already accounted for in the cost of the erroneous SHIFT action. After defining the concept of a dynamic oracle which is correct over the entire configuration space of a transition system and providing a concrete instantiation of it for the arc-eager transition system, we now go on to present one useful application of such an oracle. 4 Training Parsers with the Dynamic Oracle Greedy transition-based parsers trained with static oracles are known to suffer from error propagation (McDonald and Nivre, 2007). We may hope to mitigate the error propagation problem by letting the parser explore larger portions of the configuration space during training and learn how to behave optimally also after committing previous errors. While this is not possible with the usual static oracles, the dynamic oracle defined above allows us to do just that, as it returns a set of optimal transitions for each possible configuration. Algorithm 2 is a standard online training algorithm for transition-based dependency parsers using a static oracle. Given a training sentence x with gold tree G gold, it starts in the initial configuration c s (x) and repeatedly predicts a transition t p given its current feature weights w and compares this to the transition t o predicted by the static oracle. If the model prediction is different from the oracle prediction, the feature weights are updated, but the new configuration is always derived using the oracle transition (line 10), which means that only configurations in the canonical oracle-induced transition sequence are explored. 7 Algorithm 3 is a modification of the standard algorithm which makes use of the dynamic oracle to explore a larger part of the configuration space. The first difference is in line 7, where the new algorithm, instead of getting the single prediction of the static oracle, gets the set of 7 Some readers may be more familiar with a two-stage process in which first the oracle is used to create oracle transition sequences from the entire training set, which are then transformed to individual training examples and passed on to an external classifier. This process is equivalent to the one in Algorithm 2 in case the external classifier is a multiclass perceptron (or any other online classifier), with the only difference being that the training examples are generated on-the-fly whenever they are needed. The online formulation is used to facilitate a smooth transition to Algorithm
9 Algorithm 2 Online training with a static oracle 1: w 0 2: for I = 1 ITERATIONS do 3: for sentence x with gold tree G gold in corpus do 4: c c s (x) 5: while c is not terminal do 6: t p arg max t w φ(c, t) 7: t o o(c, G gold ) 8: if t p t o then 9: w w +φ(c, t o ) φ(c, t p ) 10: c t o (c) 11: return w transitions that have zero cost according to the dynamic oracle. The weight update is then performed only if the model prediction does not have zero cost (lines 9 10), which means that updates no longer need to reflect a single canonical transition sequence. Finally, the transition used to update the parser configuration is no longer the single transition predicted by the static oracle, but a transition that is chosen by the function CHOOSE_NEXT, which may be a transition that does not have zero-cost (lines 11-12). In our current implementation, CHOOSE_NEXT is conditioned on the predicted transition t p, the set of zero cost transitions, and the iteration number. However, more elaborate conditioning schemes are also possible. We propose two versions of the CHOOSE_NEXT function. In the first version, CHOOSE_NEXT AMB, the training algorithm only follows optimal (zero cost) transitions but permits spurious ambiguity by following the model prediction t p if this has zero cost and a random zero cost transition otherwise. In the second version, CHOOSE_NEXT EXP, the training algorithm also explores nonoptimal transitions. More precisely, after the first k training iterations, it follows the model prediction t p regardless of its cost in 100(1-p)% of the cases and falls back on CHOOSE_NEXT AMB in the remaining cases. It is worth noting that Algorithm 3 subsumes Algorithm 2 as a special case if we define ZERO_COST to contain only the prediction t o of the static oracle, and define CHOOSE_NEXT to always return t o. The novel training algorithm presented here is based on perceptron learning. 8 Since the dynamic oracle provides a cost for every transition-configuration pair, it could be used also for cost-sensitive learning. Our preliminary attempts with cost-sensitive learning through the max-loss and prediction-based passive-aggressive algorithms of Crammer et al. (2006) show that the cost-sensitive variants of the algorithms indeed improve upon the non-cost-sensitive variants. However, the best passive-aggressive results were still significantly lower than those obtained using the averaged perceptron. We do not elaborate on cost-sensitive training in this work, and leave this direction for future investigation. 5 Experiments We present experiments comparing greedy arc-eager transition-based parsers trained (a) using the static oracle (Algorithm 2), (b) using the dynamic oracle with spurious ambiguity (Algorithm 3 with CHOOSE_NEXT AMB ), and (c) using the dynamic oracle with spurious ambiguity 8 In practice, we use an averaged perceptron, although this is not reflected in the algorithm descriptions above. 967
10 Algorithm 3 Online training with a dynamic oracle 1: w 0 2: for I = 1 ITERATIONS do 3: for sentence x with gold tree G gold in corpus do 4: c c s (x) 5: while c is not terminal do 6: t p arg max t w φ(c, t) 7: ZERO_COST {t o(t; c, G gold ) = true} 8: t o arg max t ZERO_COST w φ(c, t) 9: if t p ZERO_COST then 10: w w +φ(c, t o ) φ(c, t p ) 11: t n CHOOSE_NEXT(I,t p,zero_cost) 12: c t n (c) 13: return w 1: function CHOOSE_NEXT AMB (I,t,ZERO_COST) 2: if t ZERO_COST then 3: return t 4: else 5: return RANDOM_ELEMENT(ZERO_COST) 1: function CHOOSE_NEXT EXP (I,t,ZERO_COST) 2: if I> k and RAND()> p then 3: return t 4: else 5: return CHOOSE_NEXT AMB (I,t,ZERO_COST) and non-optimal transitions (Algorithm 3 with CHOOSE_NEXT EXP ). We evaluate the models on a wide range of English data sets, as well as the data sets from the CoNLL 2007 shared task on multilingual dependency parsing (Nivre et al., 2007). The parser is a greedy transition-based parser using the arc-eager transition system of Nivre (2003, 2008) with the feature representations defined by Zhang and Nivre (2011). As our primary goal is to compare the training methods, and not to achieve the highest possible score for each data set, we use the exact same feature representations and training parameters across all experiments. Specifically, we train an averaged perceptron model for 15 iterations. When using CHOOSE_NEXT EXP, we set k = 2 and p = 0.1, meaning that the algorithm allows non-optimal transitions in 90% of the cases, starting from the third training iteration. Note that many of these transitions will nevertheless be correct, as the first training iterations already put the model in a good region of the parameter space. The English model is trained on Sections 2 21 of the Penn-WSJ Treebank (Marcus et al., 1993), converted to Stanford basic dependencies (de Marneffe et al., 2006), with part-of-speech tags assigned by a structured-perceptron tagger trained on the same corpus with 4-fold jackknifing. We use Section 22 to tune parameters, and we evaluate on the following data sets, which are also converted to the same dependency scheme, and pos-tagged using the same 968
11 WSJ22 WSJ23 BNC BRN FTBL QTB ANS EML GRPS REVS BLGS Unlabeled Attachment Scores Static Dynamic-ambiguity Dynamic-explore Labeled Attachment Scores Static Dynamic-ambiguity Dynamic-explore Table 1: Results on the English data sets structured-perceptron tagger, trained on the entire training set. WSJ22: Section 22 of the Penn-WSJ Treebank (development set). WSJ23: Section 23 of the Penn-WSJ Treebank (test set). BNC: 1,000 manually annotated sentences from the British National Corpus (Foster and van Genabith, 2008). BRN: The entire Brown Corpus (Kucera and Francis, 1967). FTBL: The entire Football Corpus (Foster et al., 2011). QB: The entire QuestionBank (Judge et al., 2006). ANS, EML, GRPS, REVS, BLGS: the question-answers, s, newsgroups, reviews and weblogs portions of the English Web Treebank (Bies et al., 2012; Petrov and McDonald, 2012). The CoNLL models are trained on the dedicated training set for each of the ten languages and evaluated on the corresponding test set, with gold standard part-of-speech tags in both cases. Since the arc-eager parser can only handle projective dependency trees, all trees in the training set are projectivized before training, using the baseline pseudo-projective transformation in Nivre and Nilsson (2005). However, non-projective trees are kept intact in the test sets for evaluation. We include all ten languages from the CoNLL 2007 shared task: ARA: Arabic (Hajič et al., 2004) BAS: Basque (Aduriz et al., 2003) CAT: Catalan (Martí et al., 2007) CHI: Chinese (Chen et al., 2003) CZE: Czech (Hajič et al., 2001; Böhmová et al., 2003) ENG: English (Marcus et al., 1993) GRE: Greek (Prokopidis et al., 2005) HUN: Hungarian (Czendes et al., 2005) 969
12 ARA BAS CAT CHI CZE ENG GRE HUN ITA TUR Unlabeled Attachment Scores Static Dynamic-ambiguity Dynamic-explore Labeled Attachment Scores Static Dynamic-ambiguity Dynamic-explore Table 2: Results on the CoNLL 2007 data sets ITA: Italian (Montemagni et al., 2003) TUR: Turkish (Oflazer et al., 2003) Table 1 gives the results for the English model, while Table 2 presents the multilingual evaluation. In both cases, we present unlabeled and labeled attachment scores excluding punctuation. For the English data sets, we see that adding spurious ambiguity (Dynamic-ambiguity) generally improves both labeled and unlabeled attachment scores by up to 0.5 percent absolute. The only exception is GRPS, where there is a small decrease in unlabeled attachment score. When the training procedure in addition explores non-optimal transitions (Dynamic-explore), the improvement is even greater, in some cases up to about 1.5 percentage points, which is quite substantial given that our baseline parser already performs at the state-of-the-art level for greedy deterministic transition-based parsers on English Stanford-dependencies. 9 For the CoNLL data sets, results for the Dynamic-ambiguity condition are mixed causing a drop in accuracy for some data sets, but the Dynamic-explore condition makes up for it and brings substantial improvement in accuracy for all except two languages. We see very substantial improvements in both UAS and LAS for Arabic, Basque, Czech, English and Greek, as well as improvements for Catalan, Chinese and Italian. The average LAS improvement across all the CoNLL datasets is 1.5 LAS points. The only two exceptions are Hungarian and Turkish, where unlabeled attachment scores drop slightly as a result of not using the static oracle, and labeled attachment scores are practically unaffected. More analysis is needed to find out what is going on for these languages, but it is likely that results could be improved with language-specific tuning. 10 Overall, the experimental results show a considerable improvement in the accuracy of deterministic linear-time classifier-based dependency parsing through training procedures that explore a larger part of the search space than traditional methods based on static oracles. 9 Note that the web data sets (ANS, EML, GRPS, REVS, BLGS) are annotated according to the Ontonotes corpus guidelines, which are somewhat different than the original Penn Treebank guidelines used in the training corpora. In particular, base-nps in the web data sets are more nested. Our scores on these data sets are thus artificially lower than they could be. We could get better scores for these data sets for all training conditions by training on the Ontonotes corpora instead, but as our main concern is not in achieving the best scores, we opted for the simpler experimental setup. 10 Language-specific tuning is likely to improve results for the other languages as well we did not perform any language-specific tuning, and it is well established that individual languages parsing accuracies can greatly benefit from tuning of the feature set, the transition system being used and the learning parameters (Hall et al., 2007). 970
13 6 Related Work Deterministic classifier-based dependency parsing is an instance of independent sequential classification-based structured prediction. In sequential classification models, such as Maximum- Entropy Markov Models (McCallum et al., 2000), a structured output is produced by repeated application of a locally trained classifier, where each classification decision is conditioned on the structure created by previous decisions. Several methods have been developed to cope with error propagation in sequential classification, including stacked sequential learning (Cohen and Carvalho, 2005), LaSO (Daumé III and Marcu, 2005), Searn (Daumé III et al., 2009) and its followers SMILe (Ross and Bagnell, 2010) and DAgger (Ross et al., 2011). While stacked learning is well suited for sequence prediction tasks such as tagging and chunking, it is not clear how to apply it to parsing. 11 Searn and the closely related DAgger algorithm are more promising for dealing with the complexity of dependency parsing, but it appears that previous attempts to apply Searn-based learning to dependency parsing have been unsuccessful. A key component in the specification of a Searn learning algorithm is an optimal policy mapping states in the search space (such as parser configurations) to optimal outcomes (such as transitions). Attempts to approximate the optimal policy for parsing using static oracles are unlikely to work very well, since a static oracle is only correct for a small subset of the search space. The dynamic oracle introduced in this paper, which is correct for arbitrary parser configurations, can be used to define an optimal policy for Searn-based learning. Both Searn and DAgger require several complete training rounds over the entire data set and take a relatively long time to train. We instead use a simpler online algorithm which can be viewed as a stochastic approximation of the DAgger algorithm, which is itself heavily inspired by the Searn algorithm. Recent work on beam search and structured prediction for transition-based dependency parsing has shown that parsing accuracy can be improved considerably if models are trained to perform beam search instead of greedy one-best search, and if training is done using a global structured learning objective instead of local learning of individual decisions (Zhang and Clark, 2008; Zhang and Nivre, 2011; Bohnet and Kuhn, 2012; Huang et al., 2012). Like our method, beam search with global structured learning mitigates the effects of error propagation by exploring non-canonical configurations at training time, but the use of a beam reduces parsing speed by a factor that is roughly proportional to the size of the beam, making parsing less efficient. Our method in contrast still trains classifiers to perform local decisions, and thus incurs no efficiency penalty at parsing time, but each local decision is trained to take into account the consequences of previous, possibly erroneous, decisions. Although we may not be able to reach the accuracy level of a beam-search parser, we show that a substantial improvement in accuracy is possible also for a purely deterministic classifier-based parser, making our method suitable for training accurate parsers in situations where maximum efficiency is needed, e.g., when there is a need to process very large corpora. Integrating our dynamic oracle in the training procedure for a transition-based parser with beam search is an interesting question for future work. The work that probably comes closest to ours is Choi and Palmer (2011), who improve the accuracy of a greedy transition-based dependency parser through an iterative training procedure that they call bootstrapping. They start by training a classifier using a standard static oracle for a hybrid transition system combining elements of the arc-eager system and the algorithm of 11 Stacked learning has been explored to some extent in the context of parsing for integrating approximate higher order features as well as for combining the predictions of different parsers (Nivre and McDonald, 2008; Martins et al., 2008). 971
14 Covington (2001). In a second step, they then use this classifier to parse the training corpus, creating one new training instance for every configuration visited during parsing, using an adapted version of the static oracle to predict the optimal transition for each configuration. They iterate this procedure as long as parsing accuracy improves on a held-out development set and report improvements in parsing accuracy of about 0.5 percent absolute for English and almost 2 percent absolute for Czech. The main difference compared to our approach, except for the fact that they use a different transition system, is that their method for finding the optimal transition after the first training round is heuristic and does not guarantee that the best parse is still reachable. Finally, Cohen et al. (2012) tackle the problem of spurious ambiguity for static oracles by eliminating ambiguity from the underlying transition system instead of modifying the oracle. They show how this can be achieved for the arc-standard system of Nivre (2004) as well as the non-projective extension by Attardi (2006). It is still an open question whether their technique can also be applied to the arc-eager system targeted in this paper. 7 Conclusion We have highlighted the shortcoming of traditional static oracles used to train transition-based dependency parsers, and instead proposed the notion of a dynamic oracle, which allows more than one correct transition sequence in the case of spurious ambiguity, and which can predict an optimal transition also for non-optimal configurations. We have defined a concrete dynamic oracle for the arc-eager transition system and showed how it can be used in online training of greedy deterministic parsers. Greedy deterministic transition-based dependency parsers are among the most efficient systems available for syntactic parsing of natural language. In terms of parsing accuracy, they perform near the state-of-the-art level for many languages but tend to suffer from prediction errors and subsequent error propagation. This problem can be mitigated by using our proposed training method. Experimental results for English show consistent improvements in parsing accuracy of up to almost 1.5 percent absolute on a wide range of data sets. Experimental results on the ten languages from the CoNLL 2007 shared task on dependency parsing show significant improvements of up to almost 3 LAS points for some languages, but there are also few cases where we see little or no improvement in parsing accuracy, a phenomenon that requires further investigation. Other topics for future research is the effective use of cost-sensitive learning instead of the perceptron loss used in this paper, the derivation of dynamic oracles for other transition systems, and utilizing the dynamic oracles in non-greedy settings, such as beam-search parsers. References Aduriz, I., Aranzabe, M. J., Arriola, J. M., Atutxa, A., Díaz de Ilarraza, A., Garmendia, A., and Oronoz, M. (2003). Construction of a Basque dependency treebank. In Proceedings of the 2nd Workshop on Treebanks and Linguistic Theories (TLT), pages Attardi, G. (2006). Experiments with a multilanguage non-projective dependency parser. In Proceedings of the 10th Conference on Computational Natural Language Learning (CoNLL), pages Bies, A., Mott, J., Warner, C., and Kulick, S. (2012). English web treebank. Linguistic Data Consortium, LDC2012T
15 Böhmová, A., Hajič, J., Hajičová, E., and Hladká, B. (2003). The Prague Dependency Treebank: A three-level annotation scenario. In Abeillé, A., editor, Treebanks: Building and Using Parsed Corpora, pages Kluwer. Bohnet, B. and Kuhn, J. (2012). The best of bothworlds a graph-based completion model for transition-based parsers. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages Chen, K., Luo, C., Chang, M., Chen, F., Chen, C., Huang, C., and Gao, Z. (2003). Sinica treebank: Design criteria, representational issues and implementation. In Abeillé, A., editor, Treebanks, pages Kluwer. Choi, J. D. and Palmer, M. (2011). Getting the most out of transition-based dependency parsing. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages Cohen, S. B., Gómez-Rodríguez, C., and Satta, G. (2012). Elimination of spurious ambiguity in transition-based dependency parsing. Technical report. Cohen, W. W. and Carvalho, V. R. (2005). Stacked sequential learning. In Proceedings of the International Joint Conference on Artificial Intelligence, pages Covington, M. A. (2001). A fundamental algorithm for dependency parsing. In Proceedings of the 39th Annual ACM Southeast Conference, pages Crammer, K., Dekel, O., Keshet, J., Shalev-Shwartz, S., and Singer, Y. (2006). Online passiveaggressive algorithms. Journal of Machine Learning Research, 7: Czendes, D., Csirik, J., Guimóthy, and Kocsor, A. (2005). The Szeged Treebank. Springer. Daumé III, H., Langford, J., and Marcu, D. (2009). Machine Learning, 75: Search-based structured prediction. Daumé III, H. and Marcu, D. (2005). Learning as search optimization: Approximate large margin methods for structured prediction. In Proceedings of the 22nd International Conference on Machine Learning, pages de Marneffe, M.-C., MacCartney, B., and Manning, C. D. (2006). Generating typed dependency parses from phrase structure parses. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC). Foster, J., ÇetinoÄŸlu, O., Wagner, J., and van Genabith, J. (2011). Comparing the use of edited and unedited text in parser self-training. In Proceedings of the 12th International Conference on Parsing Technologies, pages Foster, J. and van Genabith, J. (2008). Parser evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC). Goldberg, Y. and Elhadad, M. (2010). An efficient algorithm for easy-first non-directional dependency parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL HLT), pages
16 Hajič, J., Smrž, O., Zemánek, P., Šnaidauf, J., and Beška, E. (2004). Prague Arabic Dependency Treebank: Development in data and tools. In Proceedings of the NEMLAR International Conference on Arabic Language Resources and Tools. Hajič, J., Vidova Hladka, B., Panevová, J., Hajičová, E., Sgall, P., and Pajas, P. (2001). Prague Dependency Treebank 1.0. LDC, 2001T10. Hall, J., Nilsson, J., Nivre, J., Eryiğit, G., Megyesi, B., Nilsson, M., and Saers, M. (2007). Single malt or blended? A study in multilingual parser optimization. In Proceedings of the CoNLL Shared Task of EMNLP-CoNLL 2007, pages Huang, L., Fayong, S., and Guo, Y. (2012). Structured perceptron with inexact search. In Proceedings of Human Language Technologies: The 2012 Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL HLT). Judge, J., Cahill, A., and van Genabith, J. (2006). Questionbank: Creating a corpus of parseannotated questions. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics, pages Association for Computational Linguistics. Kucera, H. and Francis, W. N. (1967). Computational Analysis of Present-Day American English. Brown University Press. Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. (1993). Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19: Martí, M. A., Taule, M., Màrquez, L., and Bertran, M. (2007). CESS-ECE: A multilingual and multilevel annotated corpus. Available for download from: mbertran/cess-ece/. Martins, A. F., Das, D., Smith, N. A., and Xing, E. P. (2008). Stacking dependency parsers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages McCallum, A., Freitag, D., and Pereira, F. (2000). Maximum entropy markov models for information extraction and segmentation. In Proceedings of the 17th International Conference on Machine Learning, pages McDonald, R. and Nivre, J. (2007). Characterizing the errors of data-driven dependency parsing models. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages Montemagni, S., Barsotti, F., Battista, M., Calzolari, N., Corazzari, O., Lenci, A., Zampolli, A., Fanciulli, F., Massetani, M., Raffaelli, R., Basili, R., Pazienza, M. T., Saracino, D., Zanzotto, F., Nana, N., Pianesi, F., and Delmonte, R. (2003). Building the Italian Syntactic-Semantic Treebank. In Abeillé, A., editor, Treebanks: Building and Using Parsed Corpora, pages Kluwer. Nivre, J. (2003). An efficient algorithm for projective dependency parsing. In Proceedings of the 8th International Workshop on Parsing Technologies (IWPT), pages
17 Nivre, J. (2004). Incrementality in deterministic dependency parsing. In Proceedings of the Workshop on Incremental Parsing: Bringing Engineering and Cognition Together (ACL), pages Nivre, J. (2006). Inductive Dependency Parsing. Springer. Nivre, J. (2008). Algorithms for deterministic incremental dependency parsing. Computational Linguistics, 34: Nivre, J., Hall, J., Kübler, S., McDonald, R., Nilsson, J., Riedel, S., and Yuret, D. (2007). The CoNLL 2007 shared task on dependency parsing. In Proceedings of the CoNLL Shared Task of EMNLP-CoNLL 2007, pages Nivre, J., Hall, J., and Nilsson, J. (2004). Memory-based dependency parsing. In Proceedings of the 8th Conference on Computational Natural Language Learning (CoNLL), pages Nivre, J. and McDonald, R. (2008). Integrating graph-based and transition-based dependency parsers. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL), pages Nivre, J. and Nilsson, J. (2005). Pseudo-projective dependency parsing. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL), pages Oflazer, K., Say, B., Hakkani-Tür, D. Z., and Tür, G. (2003). Building a Turkish treebank. In Abeillé, A., editor, Treebanks: Building and Using Parsed Corpora, pages Kluwer. Petrov, S. and McDonald, R. (2012). Overview of the 2012 shared task on parsing the web. In Notes on the First Workshop on Syntatctic Analysis for Non-Canonical Language. Prokopidis, P., Desypri, E., Koutsombogera, M., Papageorgiou, H., and Piperidis, S. (2005). Theoretical and practical issues in the construction of a Greek dependency treebank. In Proceedings of the 3rd Workshop on Treebanks and Linguistic Theories (TLT), pages Ross, S. and Bagnell, J. A. (2010). Efficient reductions for imitation learning. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics. Ross, S., Gordon, G. J., and Bagnell, J. A. (2011). A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics. Yamada, H. and Matsumoto, Y. (2003). Statistical dependency analysis with support vector machines. In Proceedings of the 8th International Workshop on Parsing Technologies (IWPT), pages Zhang, Y. and Clark, S. (2008). A tale of two parsers: Investigating and combining graph-based and transition-based dependency parsing. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages Zhang, Y. and Nivre, J. (2011). Transition-based dependency parsing with rich non-local features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages
18
Online learning for Deterministic Dependency Parsing
Online learning for Deterministic Dependency Parsing Prashanth Reddy Mannem Language Technologies Research Center IIIT-Hyderabad, India prashanth@research.iiit.ac.in Abstract Deterministic parsing has
Transition-Based Dependency Parsing with Long Distance Collocations
Transition-Based Dependency Parsing with Long Distance Collocations Chenxi Zhu, Xipeng Qiu (B), and Xuanjing Huang Shanghai Key Laboratory of Intelligent Information Processing, School of Computer Science,
An Efficient Dynamic Oracle for Unrestricted Non-Projective Parsing
An Efficient Dynamic Oracle for Unrestricted Non-Projective Parsing Carlos Gómez-Rodríguez Departamento de Computación Universidade da Coruña Campus de Elviña, s/n 15071 A Coruña, Spain carlos.gomez@udc.es
DEPENDENCY PARSING JOAKIM NIVRE
DEPENDENCY PARSING JOAKIM NIVRE Contents 1. Dependency Trees 1 2. Arc-Factored Models 3 3. Online Learning 3 4. Eisner s Algorithm 4 5. Spanning Tree Parsing 6 References 7 A dependency parser analyzes
The Evalita 2011 Parsing Task: the Dependency Track
The Evalita 2011 Parsing Task: the Dependency Track Cristina Bosco and Alessandro Mazzei Dipartimento di Informatica, Università di Torino Corso Svizzera 185, 101049 Torino, Italy {bosco,mazzei}@di.unito.it
Automatic Detection and Correction of Errors in Dependency Treebanks
Automatic Detection and Correction of Errors in Dependency Treebanks Alexander Volokh DFKI Stuhlsatzenhausweg 3 66123 Saarbrücken, Germany alexander.volokh@dfki.de Günter Neumann DFKI Stuhlsatzenhausweg
Evalita 09 Parsing Task: constituency parsers and the Penn format for Italian
Evalita 09 Parsing Task: constituency parsers and the Penn format for Italian Cristina Bosco, Alessandro Mazzei, and Vincenzo Lombardo Dipartimento di Informatica, Università di Torino, Corso Svizzera
Tekniker för storskalig parsning
Tekniker för storskalig parsning Diskriminativa modeller Joakim Nivre Uppsala Universitet Institutionen för lingvistik och filologi joakim.nivre@lingfil.uu.se Tekniker för storskalig parsning 1(19) Generative
Online Large-Margin Training of Dependency Parsers
Online Large-Margin Training of Dependency Parsers Ryan McDonald Koby Crammer Fernando Pereira Department of Computer and Information Science University of Pennsylvania Philadelphia, PA {ryantm,crammer,pereira}@cis.upenn.edu
Introduction to Data-Driven Dependency Parsing
Introduction to Data-Driven Dependency Parsing Introductory Course, ESSLLI 2007 Ryan McDonald 1 Joakim Nivre 2 1 Google Inc., New York, USA E-mail: ryanmcd@google.com 2 Uppsala University and Växjö University,
Online Learning of Approximate Dependency Parsing Algorithms
Online Learning of Approximate Dependency Parsing Algorithms Ryan McDonald Fernando Pereira Department of Computer and Information Science University of Pennsylvania Philadelphia, PA 19104 {ryantm,pereira}@cis.upenn.edu
Easy-First, Chinese, POS Tagging and Dependency Parsing 1
Easy-First, Chinese, POS Tagging and Dependency Parsing 1 ABSTRACT Ji Ma, Tong Xiao, Jing Bo Zhu, Fei Liang Ren Natural Language Processing Laboratory, Northeastern University, Shenyang, China majineu@outlook.com,
Domain Adaptation for Dependency Parsing via Self-training
Domain Adaptation for Dependency Parsing via Self-training Juntao Yu 1, Mohab Elkaref 1, Bernd Bohnet 2 1 University of Birmingham, Birmingham, UK 2 Google, London, UK 1 {j.yu.1, m.e.a.r.elkaref}@cs.bham.ac.uk,
Semi-Supervised Learning for Blog Classification
Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence (2008) Semi-Supervised Learning for Blog Classification Daisuke Ikeda Department of Computational Intelligence and Systems Science,
EvalIta 2011: the Frame Labeling over Italian Texts Task
EvalIta 2011: the Frame Labeling over Italian Texts Task Roberto Basili, Diego De Cao, Alessandro Lenci, Alessandro Moschitti, and Giulia Venturi University of Roma, Tor Vergata, Italy {basili,decao}@info.uniroma2.it
Peking: Profiling Syntactic Tree Parsing Techniques for Semantic Graph Parsing
Peking: Profiling Syntactic Tree Parsing Techniques for Semantic Graph Parsing Yantao Du, Fan Zhang, Weiwei Sun and Xiaojun Wan Institute of Computer Science and Technology, Peking University The MOE Key
Open Domain Information Extraction. Günter Neumann, DFKI, 2012
Open Domain Information Extraction Günter Neumann, DFKI, 2012 Improving TextRunner Wu and Weld (2010) Open Information Extraction using Wikipedia, ACL 2010 Fader et al. (2011) Identifying Relations for
Learning and Inference over Constrained Output
IJCAI 05 Learning and Inference over Constrained Output Vasin Punyakanok Dan Roth Wen-tau Yih Dav Zimak Department of Computer Science University of Illinois at Urbana-Champaign {punyakan, danr, yih, davzimak}@uiuc.edu
A Systematic Cross-Comparison of Sequence Classifiers
A Systematic Cross-Comparison of Sequence Classifiers Binyamin Rozenfeld, Ronen Feldman, Moshe Fresko Bar-Ilan University, Computer Science Department, Israel grurgrur@gmail.com, feldman@cs.biu.ac.il,
Chapter 8. Final Results on Dutch Senseval-2 Test Data
Chapter 8 Final Results on Dutch Senseval-2 Test Data The general idea of testing is to assess how well a given model works and that can only be done properly on data that has not been seen before. Supervised
Social Media Mining. Data Mining Essentials
Introduction Data production rate has been increased dramatically (Big Data) and we are able store much more data than before E.g., purchase data, social media data, mobile phone data Businesses and customers
Chapter 6. The stacking ensemble approach
82 This chapter proposes the stacking ensemble approach for combining different data mining classifiers to get better performance. Other combination techniques like voting, bagging etc are also described
Data Mining Practical Machine Learning Tools and Techniques
Ensemble learning Data Mining Practical Machine Learning Tools and Techniques Slides for Chapter 8 of Data Mining by I. H. Witten, E. Frank and M. A. Hall Combining multiple models Bagging The basic idea
Identifying Focus, Techniques and Domain of Scientific Papers
Identifying Focus, Techniques and Domain of Scientific Papers Sonal Gupta Department of Computer Science Stanford University Stanford, CA 94305 sonal@cs.stanford.edu Christopher D. Manning Department of
EVALITA 07 parsing task
EVALITA 07 parsing task Cristina BOSCO Alessandro MAZZEI Vincenzo LOMBARDO (Dipartimento di Informatica Università di Torino) 1 overview 1. task 2. development data 3. evaluation 4. conclusions 2 task
Effective Self-Training for Parsing
Effective Self-Training for Parsing David McClosky dmcc@cs.brown.edu Brown Laboratory for Linguistic Information Processing (BLLIP) Joint work with Eugene Charniak and Mark Johnson David McClosky - dmcc@cs.brown.edu
Very High Accuracy and Fast Dependency Parsing is not a Contradiction
Very High Accuracy and Fast Dependency Parsing is not a Contradiction Bernd Bohnet University of Stuttgart Institut für Maschinelle Sprachverarbeitung bernd.bohnet@ims.uni-stuttgart.de Abstract In addition
Learning Morphological Disambiguation Rules for Turkish
Learning Morphological Disambiguation Rules for Turkish Deniz Yuret Dept. of Computer Engineering Koç University İstanbul, Turkey dyuret@ku.edu.tr Ferhan Türe Dept. of Computer Engineering Koç University
Ngram Search Engine with Patterns Combining Token, POS, Chunk and NE Information
Ngram Search Engine with Patterns Combining Token, POS, Chunk and NE Information Satoshi Sekine Computer Science Department New York University sekine@cs.nyu.edu Kapil Dalwani Computer Science Department
Offline sorting buffers on Line
Offline sorting buffers on Line Rohit Khandekar 1 and Vinayaka Pandit 2 1 University of Waterloo, ON, Canada. email: rkhandekar@gmail.com 2 IBM India Research Lab, New Delhi. email: pvinayak@in.ibm.com
Mining the Software Change Repository of a Legacy Telephony System
Mining the Software Change Repository of a Legacy Telephony System Jelber Sayyad Shirabad, Timothy C. Lethbridge, Stan Matwin School of Information Technology and Engineering University of Ottawa, Ottawa,
Overview. Evaluation Connectionist and Statistical Language Processing. Test and Validation Set. Training and Test Set
Overview Evaluation Connectionist and Statistical Language Processing Frank Keller keller@coli.uni-sb.de Computerlinguistik Universität des Saarlandes training set, validation set, test set holdout, stratification
Workshop. Neil Barrett PhD, Jens Weber PhD, Vincent Thai MD. Engineering & Health Informa2on Science
Engineering & Health Informa2on Science Engineering NLP Solu/ons for Structured Informa/on from Clinical Text: Extrac'ng Sen'nel Events from Pallia've Care Consult Le8ers Canada-China Clean Energy Initiative
Adaptive Online Gradient Descent
Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650
Making Sense of the Mayhem: Machine Learning and March Madness
Making Sense of the Mayhem: Machine Learning and March Madness Alex Tran and Adam Ginzberg Stanford University atran3@stanford.edu ginzberg@stanford.edu I. Introduction III. Model The goal of our research
Parsing Software Requirements with an Ontology-based Semantic Role Labeler
Parsing Software Requirements with an Ontology-based Semantic Role Labeler Michael Roth University of Edinburgh mroth@inf.ed.ac.uk Ewan Klein University of Edinburgh ewan@inf.ed.ac.uk Abstract Software
Semantic parsing with Structured SVM Ensemble Classification Models
Semantic parsing with Structured SVM Ensemble Classification Models Le-Minh Nguyen, Akira Shimazu, and Xuan-Hieu Phan Japan Advanced Institute of Science and Technology (JAIST) Asahidai 1-1, Nomi, Ishikawa,
Web Document Clustering
Web Document Clustering Lab Project based on the MDL clustering suite http://www.cs.ccsu.edu/~markov/mdlclustering/ Zdravko Markov Computer Science Department Central Connecticut State University New Britain,
NATURAL LANGUAGE QUERY PROCESSING USING PROBABILISTIC CONTEXT FREE GRAMMAR
NATURAL LANGUAGE QUERY PROCESSING USING PROBABILISTIC CONTEXT FREE GRAMMAR Arati K. Deshpande 1 and Prakash. R. Devale 2 1 Student and 2 Professor & Head, Department of Information Technology, Bharati
Online Ensembles for Financial Trading
Online Ensembles for Financial Trading Jorge Barbosa 1 and Luis Torgo 2 1 MADSAD/FEP, University of Porto, R. Dr. Roberto Frias, 4200-464 Porto, Portugal jorgebarbosa@iol.pt 2 LIACC-FEP, University of
Bootstrapping Big Data
Bootstrapping Big Data Ariel Kleiner Ameet Talwalkar Purnamrita Sarkar Michael I. Jordan Computer Science Division University of California, Berkeley {akleiner, ameet, psarkar, jordan}@eecs.berkeley.edu
Cost Model: Work, Span and Parallelism. 1 The RAM model for sequential computation:
CSE341T 08/31/2015 Lecture 3 Cost Model: Work, Span and Parallelism In this lecture, we will look at how one analyze a parallel program written using Cilk Plus. When we analyze the cost of an algorithm
Brill s rule-based PoS tagger
Beáta Megyesi Department of Linguistics University of Stockholm Extract from D-level thesis (section 3) Brill s rule-based PoS tagger Beáta Megyesi Eric Brill introduced a PoS tagger in 1992 that was based
The Goldberg Rao Algorithm for the Maximum Flow Problem
The Goldberg Rao Algorithm for the Maximum Flow Problem COS 528 class notes October 18, 2006 Scribe: Dávid Papp Main idea: use of the blocking flow paradigm to achieve essentially O(min{m 2/3, n 1/2 }
PoS-tagging Italian texts with CORISTagger
PoS-tagging Italian texts with CORISTagger Fabio Tamburini DSLO, University of Bologna, Italy fabio.tamburini@unibo.it Abstract. This paper presents an evolution of CORISTagger [1], an high-performance
Experiments in Web Page Classification for Semantic Web
Experiments in Web Page Classification for Semantic Web Asad Satti, Nick Cercone, Vlado Kešelj Faculty of Computer Science, Dalhousie University E-mail: {rashid,nick,vlado}@cs.dal.ca Abstract We address
Building a Question Classifier for a TREC-Style Question Answering System
Building a Question Classifier for a TREC-Style Question Answering System Richard May & Ari Steinberg Topic: Question Classification We define Question Classification (QC) here to be the task that, given
Generating SQL Queries Using Natural Language Syntactic Dependencies and Metadata
Generating SQL Queries Using Natural Language Syntactic Dependencies and Metadata Alessandra Giordani and Alessandro Moschitti Department of Computer Science and Engineering University of Trento Via Sommarive
Towards Domain Adaptation for Parsing Web Data
Towards Domain Adaptation for Parsing Web Data Mohammad Khan Indiana University Bloomington, IN USA khanms@indiana.edu Markus Dickinson Indiana University Bloomington, IN USA md7@indiana.edu Sandra Kübler
THUTR: A Translation Retrieval System
THUTR: A Translation Retrieval System Chunyang Liu, Qi Liu, Yang Liu, and Maosong Sun Department of Computer Science and Technology State Key Lab on Intelligent Technology and Systems National Lab for
Applying Co-Training Methods to Statistical Parsing. Anoop Sarkar http://www.cis.upenn.edu/ anoop/ anoop@linc.cis.upenn.edu
Applying Co-Training Methods to Statistical Parsing Anoop Sarkar http://www.cis.upenn.edu/ anoop/ anoop@linc.cis.upenn.edu 1 Statistical Parsing: the company s clinical trials of both its animal and human-based
Testing Data-Driven Learning Algorithms for PoS Tagging of Icelandic
Testing Data-Driven Learning Algorithms for PoS Tagging of Icelandic by Sigrún Helgadóttir Abstract This paper gives the results of an experiment concerned with training three different taggers on tagged
Accelerating and Evaluation of Syntactic Parsing in Natural Language Question Answering Systems
Accelerating and Evaluation of Syntactic Parsing in Natural Language Question Answering Systems cation systems. For example, NLP could be used in Question Answering (QA) systems to understand users natural
A Shortest-path Method for Arc-factored Semantic Role Labeling
A Shortest-path Method for Arc-factored Semantic Role Labeling Xavier Lluís TALP Research Center Universitat Politècnica de Catalunya xlluis@cs.upc.edu Xavier Carreras Xerox Research Centre Europe xavier.carreras@xrce.xerox.com
Structured Models for Fine-to-Coarse Sentiment Analysis
Structured Models for Fine-to-Coarse Sentiment Analysis Ryan McDonald Kerry Hannan Tyler Neylon Mike Wells Jeff Reynar Google, Inc. 76 Ninth Avenue New York, NY 10011 Contact email: ryanmcd@google.com
SVM Based Learning System For Information Extraction
SVM Based Learning System For Information Extraction Yaoyong Li, Kalina Bontcheva, and Hamish Cunningham Department of Computer Science, The University of Sheffield, Sheffield, S1 4DP, UK {yaoyong,kalina,hamish}@dcs.shef.ac.uk
Self-Training for Parsing Learner Text
elf-training for Parsing Learner Text Aoife Cahill, Binod Gyawali and James V. Bruno Educational Testing ervice, 660 Rosedale Road, Princeton, NJ 0854, UA {acahill, bgyawali, jbruno}@ets.org Abstract We
B-bleaching: Agile Overtraining Avoidance in the WiSARD Weightless Neural Classifier
B-bleaching: Agile Overtraining Avoidance in the WiSARD Weightless Neural Classifier Danilo S. Carvalho 1,HugoC.C.Carneiro 1,FelipeM.G.França 1, Priscila M. V. Lima 2 1- Universidade Federal do Rio de
The English-Swedish-Turkish Parallel Treebank
The English-Swedish-Turkish Parallel Treebank Beáta Megyesi, Bengt Dahlqvist, Éva Á. Csató and Joakim Nivre Department of Linguistics and Philology, Uppsala University first.last@lingfil.uu.se Abstract
npsolver A SAT Based Solver for Optimization Problems
npsolver A SAT Based Solver for Optimization Problems Norbert Manthey and Peter Steinke Knowledge Representation and Reasoning Group Technische Universität Dresden, 01062 Dresden, Germany peter@janeway.inf.tu-dresden.de
Shallow Parsing with Apache UIMA
Shallow Parsing with Apache UIMA Graham Wilcock University of Helsinki Finland graham.wilcock@helsinki.fi Abstract Apache UIMA (Unstructured Information Management Architecture) is a framework for linguistic
Blog Post Extraction Using Title Finding
Blog Post Extraction Using Title Finding Linhai Song 1, 2, Xueqi Cheng 1, Yan Guo 1, Bo Wu 1, 2, Yu Wang 1, 2 1 Institute of Computing Technology, Chinese Academy of Sciences, Beijing 2 Graduate School
Predicting the Stock Market with News Articles
Predicting the Stock Market with News Articles Kari Lee and Ryan Timmons CS224N Final Project Introduction Stock market prediction is an area of extreme importance to an entire industry. Stock price is
Statistical Machine Translation
Statistical Machine Translation Some of the content of this lecture is taken from previous lectures and presentations given by Philipp Koehn and Andy Way. Dr. Jennifer Foster National Centre for Language
Customizing an English-Korean Machine Translation System for Patent Translation *
Customizing an English-Korean Machine Translation System for Patent Translation * Sung-Kwon Choi, Young-Gil Kim Natural Language Processing Team, Electronics and Telecommunications Research Institute,
Micro blogs Oriented Word Segmentation System
Micro blogs Oriented Word Segmentation System Yijia Liu, Meishan Zhang, Wanxiang Che, Ting Liu, Yihe Deng Research Center for Social Computing and Information Retrieval Harbin Institute of Technology,
6.2.8 Neural networks for data mining
6.2.8 Neural networks for data mining Walter Kosters 1 In many application areas neural networks are known to be valuable tools. This also holds for data mining. In this chapter we discuss the use of neural
Advanced Ensemble Strategies for Polynomial Models
Advanced Ensemble Strategies for Polynomial Models Pavel Kordík 1, Jan Černý 2 1 Dept. of Computer Science, Faculty of Information Technology, Czech Technical University in Prague, 2 Dept. of Computer
SYSM 6304: Risk and Decision Analysis Lecture 5: Methods of Risk Analysis
SYSM 6304: Risk and Decision Analysis Lecture 5: Methods of Risk Analysis M. Vidyasagar Cecil & Ida Green Chair The University of Texas at Dallas Email: M.Vidyasagar@utdallas.edu October 17, 2015 Outline
Multiple Kernel Learning on the Limit Order Book
JMLR: Workshop and Conference Proceedings 11 (2010) 167 174 Workshop on Applications of Pattern Analysis Multiple Kernel Learning on the Limit Order Book Tristan Fletcher Zakria Hussain John Shawe-Taylor
Stock Market Forecasting Using Machine Learning Algorithms
Stock Market Forecasting Using Machine Learning Algorithms Shunrong Shen, Haomiao Jiang Department of Electrical Engineering Stanford University {conank,hjiang36}@stanford.edu Tongda Zhang Department of
CI6227: Data Mining. Lesson 11b: Ensemble Learning. Data Analytics Department, Institute for Infocomm Research, A*STAR, Singapore.
CI6227: Data Mining Lesson 11b: Ensemble Learning Sinno Jialin PAN Data Analytics Department, Institute for Infocomm Research, A*STAR, Singapore Acknowledgements: slides are adapted from the lecture notes
Email Spam Detection Using Customized SimHash Function
International Journal of Research Studies in Computer Science and Engineering (IJRSCSE) Volume 1, Issue 8, December 2014, PP 35-40 ISSN 2349-4840 (Print) & ISSN 2349-4859 (Online) www.arcjournals.org Email
English Grammar Checker
International l Journal of Computer Sciences and Engineering Open Access Review Paper Volume-4, Issue-3 E-ISSN: 2347-2693 English Grammar Checker Pratik Ghosalkar 1*, Sarvesh Malagi 2, Vatsal Nagda 3,
Ensemble Methods. Knowledge Discovery and Data Mining 2 (VU) (707.004) Roman Kern. KTI, TU Graz 2015-03-05
Ensemble Methods Knowledge Discovery and Data Mining 2 (VU) (707004) Roman Kern KTI, TU Graz 2015-03-05 Roman Kern (KTI, TU Graz) Ensemble Methods 2015-03-05 1 / 38 Outline 1 Introduction 2 Classification
Attribution. Modified from Stuart Russell s slides (Berkeley) Parts of the slides are inspired by Dan Klein s lecture material for CS 188 (Berkeley)
Machine Learning 1 Attribution Modified from Stuart Russell s slides (Berkeley) Parts of the slides are inspired by Dan Klein s lecture material for CS 188 (Berkeley) 2 Outline Inductive learning Decision
Chapter 6: The Information Function 129. CHAPTER 7 Test Calibration
Chapter 6: The Information Function 129 CHAPTER 7 Test Calibration 130 Chapter 7: Test Calibration CHAPTER 7 Test Calibration For didactic purposes, all of the preceding chapters have assumed that the
The XMU Phrase-Based Statistical Machine Translation System for IWSLT 2006
The XMU Phrase-Based Statistical Machine Translation System for IWSLT 2006 Yidong Chen, Xiaodong Shi Institute of Artificial Intelligence Xiamen University P. R. China November 28, 2006 - Kyoto 13:46 1
A Brief Introduction to Property Testing
A Brief Introduction to Property Testing Oded Goldreich Abstract. This short article provides a brief description of the main issues that underly the study of property testing. It is meant to serve as
Beating the MLB Moneyline
Beating the MLB Moneyline Leland Chen llxchen@stanford.edu Andrew He andu@stanford.edu 1 Abstract Sports forecasting is a challenging task that has similarities to stock market prediction, requiring time-series
Tibetan-Chinese Bilingual Sentences Alignment Method based on Multiple Features
, pp.273-280 http://dx.doi.org/10.14257/ijdta.2015.8.4.27 Tibetan-Chinese Bilingual Sentences Alignment Method based on Multiple Features Lirong Qiu School of Information Engineering, MinzuUniversity of
Language Interface for an XML. Constructing a Generic Natural. Database. Rohit Paravastu
Constructing a Generic Natural Language Interface for an XML Database Rohit Paravastu Motivation Ability to communicate with a database in natural language regarded as the ultimate goal for DB query interfaces
The Role of Sentence Structure in Recognizing Textual Entailment
Blake,C. (In Press) The Role of Sentence Structure in Recognizing Textual Entailment. ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, Prague, Czech Republic. The Role of Sentence Structure
An NLP Curator (or: How I Learned to Stop Worrying and Love NLP Pipelines)
An NLP Curator (or: How I Learned to Stop Worrying and Love NLP Pipelines) James Clarke, Vivek Srikumar, Mark Sammons, Dan Roth Department of Computer Science, University of Illinois, Urbana-Champaign.
Impact of Boolean factorization as preprocessing methods for classification of Boolean data
Impact of Boolean factorization as preprocessing methods for classification of Boolean data Radim Belohlavek, Jan Outrata, Martin Trnecka Data Analysis and Modeling Lab (DAMOL) Dept. Computer Science,
Benefits of HPC for NLP besides big data
Benefits of HPC for NLP besides big data Barbara Plank Center for Sprogteknologie (CST) University of Copenhagen, Denmark http://cst.dk/bplank Web-Scale Natural Language Processing in Northern Europe Oslo,
Training and Testing Low-degree Polynomial Data Mappings via Linear SVM
Journal of Machine Learning Research 11 (2010) 1471-1490 Submitted 8/09; Revised 1/10; Published 4/10 Training and Testing Low-degree Polynomial Data Mappings via Linear SVM Yin-Wen Chang Cho-Jui Hsieh
SYSTRAN Chinese-English and English-Chinese Hybrid Machine Translation Systems for CWMT2011 SYSTRAN 混 合 策 略 汉 英 和 英 汉 机 器 翻 译 系 CWMT2011 技 术 报 告
SYSTRAN Chinese-English and English-Chinese Hybrid Machine Translation Systems for CWMT2011 Jin Yang and Satoshi Enoue SYSTRAN Software, Inc. 4444 Eastgate Mall, Suite 310 San Diego, CA 92121, USA E-mail:
FUZZY CLUSTERING ANALYSIS OF DATA MINING: APPLICATION TO AN ACCIDENT MINING SYSTEM
International Journal of Innovative Computing, Information and Control ICIC International c 0 ISSN 34-48 Volume 8, Number 8, August 0 pp. 4 FUZZY CLUSTERING ANALYSIS OF DATA MINING: APPLICATION TO AN ACCIDENT
Online Latent Structure Training for Language Acquisition
IJCAI 11 Online Latent Structure Training for Language Acquisition Michael Connor University of Illinois connor2@illinois.edu Cynthia Fisher University of Illinois cfisher@cyrus.psych.uiuc.edu Dan Roth
Language and Computation
Language and Computation week 13, Thursday, April 24 Tamás Biró Yale University tamas.biro@yale.edu http://www.birot.hu/courses/2014-lc/ Tamás Biró, Yale U., Language and Computation p. 1 Practical matters
Introduction to Support Vector Machines. Colin Campbell, Bristol University
Introduction to Support Vector Machines Colin Campbell, Bristol University 1 Outline of talk. Part 1. An Introduction to SVMs 1.1. SVMs for binary classification. 1.2. Soft margins and multi-class classification.
SYSTRAN 混 合 策 略 汉 英 和 英 汉 机 器 翻 译 系 统
SYSTRAN Chinese-English and English-Chinese Hybrid Machine Translation Systems Jin Yang, Satoshi Enoue Jean Senellart, Tristan Croiset SYSTRAN Software, Inc. SYSTRAN SA 9333 Genesee Ave. Suite PL1 La Grande
CIRGIRDISCO at RepLab2014 Reputation Dimension Task: Using Wikipedia Graph Structure for Classifying the Reputation Dimension of a Tweet
CIRGIRDISCO at RepLab2014 Reputation Dimension Task: Using Wikipedia Graph Structure for Classifying the Reputation Dimension of a Tweet Muhammad Atif Qureshi 1,2, Arjumand Younus 1,2, Colm O Riordan 1,
Distributed forests for MapReduce-based machine learning
Distributed forests for MapReduce-based machine learning Ryoji Wakayama, Ryuei Murata, Akisato Kimura, Takayoshi Yamashita, Yuji Yamauchi, Hironobu Fujiyoshi Chubu University, Japan. NTT Communication
Simple and efficient online algorithms for real world applications
Simple and efficient online algorithms for real world applications Università degli Studi di Milano Milano, Italy Talk @ Centro de Visión por Computador Something about me PhD in Robotics at LIRA-Lab,
DUOL: A Double Updating Approach for Online Learning
: A Double Updating Approach for Online Learning Peilin Zhao School of Comp. Eng. Nanyang Tech. University Singapore 69798 zhao6@ntu.edu.sg Steven C.H. Hoi School of Comp. Eng. Nanyang Tech. University
Active Learning SVM for Blogs recommendation
Active Learning SVM for Blogs recommendation Xin Guan Computer Science, George Mason University Ⅰ.Introduction In the DH Now website, they try to review a big amount of blogs and articles and find the
A Mixed Trigrams Approach for Context Sensitive Spell Checking
A Mixed Trigrams Approach for Context Sensitive Spell Checking Davide Fossati and Barbara Di Eugenio Department of Computer Science University of Illinois at Chicago Chicago, IL, USA dfossa1@uic.edu, bdieugen@cs.uic.edu
Domain Classification of Technical Terms Using the Web
Systems and Computers in Japan, Vol. 38, No. 14, 2007 Translated from Denshi Joho Tsushin Gakkai Ronbunshi, Vol. J89-D, No. 11, November 2006, pp. 2470 2482 Domain Classification of Technical Terms Using