# An End-to-End Discriminative Approach to Machine Translation

Save this PDF as:

Size: px
Start display at page:

## Transcription

3 Figure 1: Given the current prediction (a), there are two possible updates, local (b) and bold (c). Although the bold update (c) reaches the reference translation, a bad correspondence is used. The local update (b) does not reach the reference, but is more reasonable than (c). Discriminative training with hidden variables has been handled in this probabilistic framework (Quattoni et al., 2004; Koo and Collins, 2005), but we choose Equation 3 for efficiency. It turns out that using the Viterbi approximation (which we call bold updating) is not always the best strategy. To appreciate the difficulty, consider the example in Figure 1. Suppose we make the prediction (a) with the current set of parameters. There are often several acceptable output translations y, for example, (b) and (c). Since (c) s output matches the reference translation, should we update towards (c)? In this case, the answer is negative. The problem with (c) is that the correspondence h contains an incorrect alignment (, a). However, since h is unobserved, the training procedure has no way of knowing this. While the output in (b) is farther from the reference, its correspondence h is much more reasonable. In short, it does not suffice for y t to be good; both y t and h t need to be good. A major challenge in using the perceptron algorithm for machine translation is determining the target (y t,h t ) in Equation 3. Section 5 discusses possible targets to update towards. 3 Dataset Our experiments were done on the French-English portion of the Europarl corpus (Koehn, 2002), Dataset TRAIN DEV TEST Years # sentences 67K first 1K first 1K # words (unk.) 715K 10.4K (35) 10.8K (48) Table 1: The Europarl dataset split we used and various statistics on length 5 15 sentences. The number of French word tokens is given, along with the number that were not seen among the 414K total sentences in TRAIN (which includes all lengths). which consists of European parliamentary proceedings from 1996 to We split the data into three sets according to Table 1. TRAIN served two purposes: it was used to construct the features, and the 5 15 length sentences were used for tuning the parameters of those features. DEV, which consisted of the first 1K length 5 15 sentences in 2002, was used to evaluate the performance of the system as we developed it. Note that the DEV set was not used to tune any parameters; tuning was done exclusively on TRAIN. At the end we ran our models once on TEST to get final numbers. 2 4 Models Our experiments used phrase-based models (Koehn et al., 2003), which require a translation table and language model for decoding and feature computation. To facilitate comparison with previous work, we created the translation tables using the same techniques as Koehn et al. (2003). 3 The language model was a Kneser-Ney interpolated trigram model generated using the SRILM toolkit (Stolcke, 2002). We built our own phrase-based beam decoder that can handle arbitrary features. 4 The contributions of features are incrementally added into the score as decoding 2 We also experimented with several combinations of jackknifing to prevent overfitting, in which we selected features on TRAIN-OLD ( Europarl corpus) and tuned the parameters on TRAIN, or vice-versa. However, it turned out that using TRAIN-OLD was suboptimal since that data is less relevant to DEV. Another alternative is to combine TRAIN- OLD and TRAIN into one dual-purpose dataset. The differences between this and our current approach were inconclusive. 3 In other words, we used GIZA++ to construct a word alignment in each direction and a growth heuristic to combine them. We extracted all the substrings that are closed under this high-quality word alignment and computed surface statistics from cooccurrences counts. 4 In our experiments, we used a beam size of 10, which we found to be only slightly worse than using a beam of 100.

4 proceeds. We experimented with two levels of distortion: monotonic, where the phrasal alignment is monotonic (but word reordering is still possible within a phrase) and limited distortion, where only adjacent phrases are allowed to exchange positions (Zens and Ney, 2004). In the future, we plan to explore our discriminative framework on a full distortion model (Koehn et al., 2003) or even a hierarchical model (Chiang, 2005). Throughout the following experiments, we trained the perceptron algorithm for 10 iterations. The weights were initialized to 1 on the translation table, 1 on the language model (the blanket features in Section 6), and 0 elsewhere. The next two sections give experiments on the two key components of a discriminative machine translation system: choosing the proper update strategy (Section 5) and including powerful features (Section 6). 5 Update strategies This section describes the importance of choosing a good update strategy the difference in BLEU score can be as large as 1.2 between different strategies. An update strategy specifies the target (y t,h t ) that we update towards (Equation 3) given the current set of parameters and a provided reference translation (x i,y i ). As mentioned in Section 2.2, faithful output (i.e. y t = y i ) does not imply that updating towards (y t,h t ) is desirable. In fact, such a constrained target might not even be reachable by the decoder, for example, if the reference is very non-literal. We explored the following three ways to choose the target (y t,h t ): Bold updating: Update towards the highest scoring option (y,h), where y is constrained to be the reference y i but h is unconstrained. Examples not reachable by the decoder are skipped. Local updating: Generate an n-best list using the current parameters. Update towards the option with the highest BLEU score. 5 5 Since BLEU score (k-bleu with k = 4) involves computing a geometric mean over i-grams, i = 1,..., k, it is zero if the translation does not have at least one k-gram in common with the reference translation. Since a BLEU score of zero is both unhelpful for choosing from the n-best and common when computed on just a single example, we instead used a smoothed version for choosing the target: P 4 We still report NIST s usual 4-gram BLEU. i=1 i-bleu(x,y) 2 4 i+1. # \$ % & '! " # \$ % & ' ( Figure 2: The three update strategies under two scenarios. Hybrid updating: Do a bold update if the reference is reachable. Otherwise, do a local update. Figure 2 shows the space of translations schematically. On each training example, our decoder produces an n-best list. The reference translation may or may not be reachable. Bold updating most resembles the traditional perceptron update rule (Equation 2). We are ensured that the target output y will be correct, although the correspondence h might be bad. Another weakness of bold updating is that we might not make full use of the training data. Local updating uses every example, but its steps are more cautious. It can be viewed as dynamic reranking, where parameters are updated using the best option on the n-best list, similar to standard static reranking. The key difference is that, unlike static reranking, the parameter updates propagate back to the baseline classifier, so that the n-best list improves over time. In this regard, dynamic reranking remedies one of the main weaknesses of static reranking, which is that the performance of the system is directly limited by the quality of the baseline classifier. Hybrid updating combines the two strategies: it makes full use of the training data as in local updating, but still tries to make swift progress towards the reference translation as in bold updating. We conducted experiments to see which of the updating strategies worked best. We trained on

5 Decoder Bold Local Hybrid Monotonic Limited distortion Table 2: Comparison of BLEU scores between different updating strategies for the monotonic and limited distortion decoders on DEV of the 67K available examples, using the BLANKET+LEX+POS feature set (Section 6). Table 2 shows that local updating is the most effective, especially when using the limited distortion decoder. In bold updating, only a small fraction of the 5000 examples (1296 for the monotonic decoder and 1601 for the limited distortion decoder) had reachable reference translations, and, therefore, contributed to parameter updates. One might therefore hypothesize that local updating performs better simply because it is able to leverage more data. This is not the full story, however, since the hybrid approach (which makes the same number of updates) performs significantly worse than local updating when using the limited distortion decoder. To see the problem with bold updating, recall the example in Figure 1. Bold updating tries to reach the reference at all costs, even if it means abusing the hidden correspondence in the process. In the example, the alignment (, a) is unreasonable, but the algorithm has no way to recognize this. Local updating is much more stable since it only updates towards sentences in the n-best list. When using the limited distortion decoder, bold updating is even more problematic because the added flexibility of phrase swaps allows more preposterous alignments to be produced. Limited distortion decoding actually performs worse than monotonic decoding with bold updating, but better with local updating. Another difference between bold updating and local updating is that the BLEU score on the training data is dramatically higher for bold updating than for local (or hybrid) updating: 80 for the former versus 40 for the latter. This is not surprising given that bold updating aggressively tries to obtain the references. However, what is surprising is that although bold updating appears to be overfitting severely, its BLEU score on the DEV does not suffer much in the monotonic case. Model DEV BLEU TEST BLEU Monotonic BLANKET (untuned) BLANKET BLANKET+LEX BLANKET+LEX+POS Pharaoh (MERT) Full-distortion Pharaoh (MERT) Table 3: Main results on our system with different feature sets compared to minimum error-rate trained Pharaoh. 6 Features This section shows that by adding an array of expressive features and discriminatively learning their weights, we can obtain a 2.3 increase in BLEU score on DEV. We add these features incrementally, first tuning blanket features (Section 6.1), then adding lexical features (Section 6.2), and finally adding part-of-speech (POS) features (Section 6.3). Table 3 summarizes the performance gains. For the experiments in this section, we used the local updating strategy and the monotonic decoder for efficiency. We train on all 67K of the length 5 15 sentences in TRAIN Blanket features The blanket features (BLANKET) consist of the translation log-probability and the language model log-probability, which are two of the components of the Pharaoh model (Section 2.1). After discriminative training, the relative weight of these two features is roughly 2:1, resulting in a BLEU score increase from 33.0 (setting both weights to 1) to The following simple example gives a flavor of the discriminative approach. The untuned system translated the French phrase trente-cinq langues into five languages in a DEV example. Although the probability P(five trente-cinq) = is rightly much smaller than P(thirty-five trente-cinq) = 0.279, the language model favors five languages over thirty-five languages. The trained system downweights the language model and recovers the correct translation. 6 We used sentences of length 5 15 to facilitate comparisons with Koehn et al. (2003) and to enable rapid experimentation with various feature sets. Experiments on sentences of length 5 50 showed similar gains in performance.

7 secure refuge abri sûr zero growth rate croissance zéro, that same (a) (b) (c), ce même Figure 3: Three constellation features with example phrase pairs. Constellations (a) and (b) have large positive weights and (c) has a large negative weight. phrase is constructed after a (NN, NN) phrase, they are reasonable candidates for swapping because of regular word-order differences between French and English. While the bulk of our results are presented for the monotonic case, the limited distortion results of Table 2 use these lexical swap features; without parameterized swap features, accuracy was below the untuned monotonic baseline. An interesting statistic is the number of nonzero feature weights that were learned using each feature set. BLANKET has only 4 features, while BLANKET+LEX has 1.55 million features. 8 Remarkably, BLANKET+LEX+POS has fewer features only 1.24 million. This is an effect of generalization ability POS information somewhat reduces the need for specific lexical features. 6.4 Alignment constellation features Koehn et al. (2003) demonstrated that choosing the appropriate heuristic for extracting phrases is very important. They showed that the difference in BLEU score between various heuristics was as large as 2.0. The process of phrase extraction is difficult to optimize in a non-discriminative setting: many heuristics have been proposed (Koehn et al., 2003), but it is not obvious which one should be chosen for a given language pair. We propose a natural way to handle this part of the translation pipeline. The idea is to push the learning process all the way down to the phrase extraction by parameterizing the phrase extraction heuristic itself. The heuristics in Koehn et al. (2003) decide whether to extract a given phrase pair based on the underlying word alignments (see Figure 3 for three examples), which we call constellations. Since we do not know which constellations correspond to 8 Both the language model and translation table components have two features, one for known words and one for unknown words. Features -CONST +CONST BLANKET BLANKET+LEX BLANKET+LEX+POS Table 4: DEV BLEU score increase resulting from adding constellation features. good phrase pairs, we introduce an alignment constellation feature to indicate the presence of a particular alignment constellation. 9 Table 4 details the effect of adding constellation features on top of our previous feature sets. 10 We get a minor increase in BLEU score from each feature set, although there is no gain by adding POS features in addition to constellation features, probably because POS and constellation features provide redundant information for French-English translations. It is interesting to look at the constellations with highest and lowest weights, which are perhaps surprising at first glance. At the top of the list are word inversions (Figure 3 (a) and (b)), while long monotonic constellations fall at the bottom of the list (c). Although monotonic translations are much more frequent than word inversions in our dataset, when translations are monotonic, shorter segmentations are preferred. This phenomenon is another manifestation of the complex interaction of phrase segmentations. 7 Final results The last column of Table 3 shows the performance of our methods on the final TEST set. Our best test BLEU score is 29.6 using BLANKET+LEX+POS, an increase of 1.3 BLEU over our untuned feature set BLANKET. The discrepancy between DEV performance and TEST performance is due to temporal distance from TRAIN and high variance in BLEU score. 11 We also compared our model with Pharaoh (Koehn et al., 2003). We tuned Pharaoh s four parameters using minimum error rate training (Och, 2003) on DEV. 12 We obtained an increase of As in the POS features, we map each phrase pair to its majority constellation. 10 Due to time constraints, we ran these experiments on 5000 training examples using bold updating. 11 For example, the DEV BLEU score for BLANKET+LEX ranges from 28.6 to 33.2, depending on which block of 1000 sentences we chose. 12 We used the training scripts from the 2006 MT Shared Task. We still tuned our model parameters on TRAIN and

8 BLEU over the Pharaoh, run with the monotone flag. 13 Even though we are using a monotonic decoder, our best results are still slightly better than the version of Pharaoh that permits arbitrary distortion. 8 Related work In machine translation, most discriminative approaches currently fall into two general categories. The first approach is to reuse the components of a generative model, but tune their relative weights in a discriminative fashion (Och and Ney, 2002; Och, 2003; Chiang, 2005). This approach only works in practice with a small handful of parameters. The second approach is to use reranking, in which a baseline classifier generates an n-best list of candidate translations, and a separate discriminative classifier chooses amongst them (Shen et al., 2004; Och et al., 2004). The major limitation of a reranking system is its dependence on the underlying baseline system, which bounds the potential improvement from discriminative training. In machine translation, this limitation is a real concern; it is common for all translations on moderately-sized n-best lists to be of poor quality. For instance, Och et al. (2004) reported that a 1000-best list was required to achieve performance gains from reranking. In contrast, the decoder in our system can use the feature weights learned in the previous iteration. Tillmann and Zhang (2005) present a discriminative approach based on local models. Their formulation explicitly decomposed the score of a translation into a sequence of local decisions, while our formulation allows global estimation. 9 Conclusion We have presented a novel end-to-end discriminative system for machine translation. We studied update strategies, an important issue in online discriminative training for MT, and conclude that making many smaller (conservative) updates is better than making few large (aggressive) updates. We also investigated the effect of adding many expressive features, which yielded a 0.8 increase in BLEU score over monotonic Pharaoh. Acknowledgments We would like to thank our reviewers for their comments. This work was suponly used DEV to optimize the number of training iterations. 13 This result is significant with p-value based on approximate randomization (Riezler and Maxwell, 2005). ported by a FQRNT fellowship to second author and a Microsoft Research New Faculty Fellowship to the third author. References Peter F. Brown, Vincent J. Della Pietra, Peter V. desouza, Jennifer C. Lai, and Robert L. Mercer Class-Based n-gram Models of Natural Language. Computational Linguistics, 18(4): Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19: David Chiang A Hierarchical Phrase-Based Model for Statistical Machine Translation. In ACL Michael Collins and Brian Roark Incremental Parsing with the Perceptron Algorithm. In ACL Michael Collins Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithms. In EMNLP Philipp Koehn, Franz Josef Och, and Daniel Marcu Statistical Phrase-Based Translation. In HLT-NAACL Philipp Koehn Europarl: A Multilingual Corpus for Evaluation of Machine Translation. Terry Koo and Michael Collins Hidden-Variable Models for Discriminative Reranking. In EMNLP Franz Josef Och and Hermann Ney Discriminative Training and Maximum Entropy Models for Statistical Machine Translation. In ACL Franz Josef Och, Daniel Gildea, Sanjeev Khudanpur, and Anoop Sarkar A Smorgasbord of Features for Statistical Machine Translation. In HLT-NAACL Franz Josef Och Minimum Error Rate Training in Statistical Machine Translation. In ACL Ariadna Quattoni, Michael Collins, and Trevor Darrell Conditional Random Fields for Object Recognition. In NIPS Stefan Riezler and John T. Maxwell On Some Pitfalls in Automatic Evaluation and Significance Testing for MT. In Workshop on Intrinsic and Extrinsic Evaluation Methods for MT and Summarization (MTSE). Brian Roark, Murat Saraclar, Michael Collins, and Mark Johnson Discriminative Language Modeling with Conditional Random Fields and the Perceptron Algorithm. In ACL Helmut Schmid Probabilistic Part-of-Speech Tagging Using Decision Trees. In International Conference on New Methods in Language Processing. Libin Shen, Anoop Sarkar, and Franz Josef Och Discriminative Reranking for Machine Translation. In HLT- NAACL Andreas Stolcke SRILM An Extensible Language Modeling Toolkit. In ICSLP Christoph Tillmann and Tong Zhang A Localized Prediction Model for Statistical Machine Translation. In ACL Richard Zens and Hermann Ney Improvements in Statistical Phrase-Based Translation. In HLT-NAACL 2004.

### The XMU Phrase-Based Statistical Machine Translation System for IWSLT 2006

The XMU Phrase-Based Statistical Machine Translation System for IWSLT 2006 Yidong Chen, Xiaodong Shi Institute of Artificial Intelligence Xiamen University P. R. China November 28, 2006 - Kyoto 13:46 1

### An Iteratively-Trained Segmentation-Free Phrase Translation Model for Statistical Machine Translation

An Iteratively-Trained Segmentation-Free Phrase Translation Model for Statistical Machine Translation Robert C. Moore Chris Quirk Microsoft Research Redmond, WA 98052, USA {bobmoore,chrisq}@microsoft.com

### N-best. N-best Reranking Using Optimal Phrase Alignment for Statistical Machine Translation

Vol. 51 No. 8 1443 1451 (Aug. 2010) N-best 1 2 3 4 1 N-best Reranking Using Optimal Phrase Alignment for Statistical Machine Translation Mitsuru Koshikawa, 1 Masao Utiyama, 2 Shunji Umetani, 3 Tomomi Matsui

### Tekniker för storskalig parsning

Tekniker för storskalig parsning Diskriminativa modeller Joakim Nivre Uppsala Universitet Institutionen för lingvistik och filologi joakim.nivre@lingfil.uu.se Tekniker för storskalig parsning 1(19) Generative

### THUTR: A Translation Retrieval System

THUTR: A Translation Retrieval System Chunyang Liu, Qi Liu, Yang Liu, and Maosong Sun Department of Computer Science and Technology State Key Lab on Intelligent Technology and Systems National Lab for

### Statistical Machine Translation: IBM Models 1 and 2

Statistical Machine Translation: IBM Models 1 and 2 Michael Collins 1 Introduction The next few lectures of the course will be focused on machine translation, and in particular on statistical machine translation

### LIUM s Statistical Machine Translation System for IWSLT 2010

LIUM s Statistical Machine Translation System for IWSLT 2010 Anthony Rousseau, Loïc Barrault, Paul Deléglise, Yannick Estève Laboratoire Informatique de l Université du Maine (LIUM) University of Le Mans,

### The University of Maryland Statistical Machine Translation System for the Fifth Workshop on Machine Translation

The University of Maryland Statistical Machine Translation System for the Fifth Workshop on Machine Translation Vladimir Eidelman, Chris Dyer, and Philip Resnik UMIACS Laboratory for Computational Linguistics

### Adaptive Development Data Selection for Log-linear Model in Statistical Machine Translation

Adaptive Development Data Selection for Log-linear Model in Statistical Machine Translation Mu Li Microsoft Research Asia muli@microsoft.com Dongdong Zhang Microsoft Research Asia dozhang@microsoft.com

### The KIT Translation system for IWSLT 2010

The KIT Translation system for IWSLT 2010 Jan Niehues 1, Mohammed Mediani 1, Teresa Herrmann 1, Michael Heck 2, Christian Herff 2, Alex Waibel 1 Institute of Anthropomatics KIT - Karlsruhe Institute of

### A Quirk Review of Translation Models

A Quirk Review of Translation Models Jianfeng Gao, Microsoft Research Prepared in connection with the 2010 ACL/SIGIR summer school July 22, 2011 The goal of machine translation (MT) is to use a computer

### Effective Self-Training for Parsing

Effective Self-Training for Parsing David McClosky dmcc@cs.brown.edu Brown Laboratory for Linguistic Information Processing (BLLIP) Joint work with Eugene Charniak and Mark Johnson David McClosky - dmcc@cs.brown.edu

### A Joint Sequence Translation Model with Integrated Reordering

A Joint Sequence Translation Model with Integrated Reordering Nadir Durrani, Helmut Schmid and Alexander Fraser Institute for Natural Language Processing University of Stuttgart Introduction Generation

### Incremental Training and Intentional Over-fitting of Word Alignment

Incremental Training and Intentional Over-fitting of Word Alignment Qin Gao Will Lewis, Chris Quirk and Mei-Yuh Hwang Language Technologies Institute, Microsoft Research Carnegie Mellon University One

### The Prague Bulletin of Mathematical Linguistics NUMBER 93 JANUARY 2010 37 46. Training Phrase-Based Machine Translation Models on the Cloud

The Prague Bulletin of Mathematical Linguistics NUMBER 93 JANUARY 2010 37 46 Training Phrase-Based Machine Translation Models on the Cloud Open Source Machine Translation Toolkit Chaski Qin Gao, Stephan

### Statistical Significance Tests for Machine Translation Evaluation

Statistical Significance Tests for Machine Translation Evaluation Philipp Koehn Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology The Stata Center, 32 Vassar

### Statistical Machine Translation

Statistical Machine Translation Some of the content of this lecture is taken from previous lectures and presentations given by Philipp Koehn and Andy Way. Dr. Jennifer Foster National Centre for Language

### The Prague Bulletin of Mathematical Linguistics NUMBER 96 OCTOBER 2011 49 58. Ncode: an Open Source Bilingual N-gram SMT Toolkit

The Prague Bulletin of Mathematical Linguistics NUMBER 96 OCTOBER 2011 49 58 Ncode: an Open Source Bilingual N-gram SMT Toolkit Josep M. Crego a, François Yvon ab, José B. Mariño c c a LIMSI-CNRS, BP 133,

### SYSTRAN Chinese-English and English-Chinese Hybrid Machine Translation Systems for CWMT2011 SYSTRAN 混 合 策 略 汉 英 和 英 汉 机 器 翻 译 系 CWMT2011 技 术 报 告

SYSTRAN Chinese-English and English-Chinese Hybrid Machine Translation Systems for CWMT2011 Jin Yang and Satoshi Enoue SYSTRAN Software, Inc. 4444 Eastgate Mall, Suite 310 San Diego, CA 92121, USA E-mail:

### Chapter 5. Phrase-based models. Statistical Machine Translation

Chapter 5 Phrase-based models Statistical Machine Translation Motivation Word-Based Models translate words as atomic units Phrase-Based Models translate phrases as atomic units Advantages: many-to-many

### Chapter 6. Decoding. Statistical Machine Translation

Chapter 6 Decoding Statistical Machine Translation Decoding We have a mathematical model for translation p(e f) Task of decoding: find the translation e best with highest probability Two types of error

### Phrase-Based Translation Models

Phrase-Based Translation Models Michael Collins April 10, 2013 1 Introduction In previous lectures we ve seen IBM translation models 1 and 2. In this note we will describe phrasebased translation models.

### Systematic Comparison of Professional and Crowdsourced Reference Translations for Machine Translation

Systematic Comparison of Professional and Crowdsourced Reference Translations for Machine Translation Rabih Zbib, Gretchen Markiewicz, Spyros Matsoukas, Richard Schwartz, John Makhoul Raytheon BBN Technologies

### The Impact of Morphological Errors in Phrase-based Statistical Machine Translation from English and German into Swedish

The Impact of Morphological Errors in Phrase-based Statistical Machine Translation from English and German into Swedish Oscar Täckström Swedish Institute of Computer Science SE-16429, Kista, Sweden oscar@sics.se

### SYSTRAN 混 合 策 略 汉 英 和 英 汉 机 器 翻 译 系 统

SYSTRAN Chinese-English and English-Chinese Hybrid Machine Translation Systems Jin Yang, Satoshi Enoue Jean Senellart, Tristan Croiset SYSTRAN Software, Inc. SYSTRAN SA 9333 Genesee Ave. Suite PL1 La Grande

### Analyzing the Dynamics of Research by Extracting Key Aspects of Scientific Papers

Analyzing the Dynamics of Research by Extracting Key Aspects of Scientific Papers Sonal Gupta Christopher Manning Natural Language Processing Group Department of Computer Science Stanford University Columbia

### Statistical Machine Translation Lecture 4. Beyond IBM Model 1 to Phrase-Based Models

p. Statistical Machine Translation Lecture 4 Beyond IBM Model 1 to Phrase-Based Models Stephen Clark based on slides by Philipp Koehn p. Model 2 p Introduces more realistic assumption for the alignment

### The TCH Machine Translation System for IWSLT 2008

The TCH Machine Translation System for IWSLT 2008 Haifeng Wang, Hua Wu, Xiaoguang Hu, Zhanyi Liu, Jianfeng Li, Dengjun Ren, Zhengyu Niu Toshiba (China) Research and Development Center 5/F., Tower W2, Oriental

### NLP Programming Tutorial 5 - Part of Speech Tagging with Hidden Markov Models

NLP Programming Tutorial 5 - Part of Speech Tagging with Hidden Markov Models Graham Neubig Nara Institute of Science and Technology (NAIST) 1 Part of Speech (POS) Tagging Given a sentence X, predict its

### PBML. logo. The Prague Bulletin of Mathematical Linguistics NUMBER??? JANUARY 2009 1 15. Grammar based statistical MT on Hadoop

PBML logo The Prague Bulletin of Mathematical Linguistics NUMBER??? JANUARY 2009 1 15 Grammar based statistical MT on Hadoop An end-to-end toolkit for large scale PSCFG based MT Ashish Venugopal, Andreas

### Translating the Penn Treebank with an Interactive-Predictive MT System

IJCLA VOL. 2, NO. 1 2, JAN-DEC 2011, PP. 225 237 RECEIVED 31/10/10 ACCEPTED 26/11/10 FINAL 11/02/11 Translating the Penn Treebank with an Interactive-Predictive MT System MARTHA ALICIA ROCHA 1 AND JOAN

### Factored Translation Models

Factored Translation s Philipp Koehn and Hieu Hoang pkoehn@inf.ed.ac.uk, H.Hoang@sms.ed.ac.uk School of Informatics University of Edinburgh 2 Buccleuch Place, Edinburgh EH8 9LW Scotland, United Kingdom

### Automatic Speech Recognition and Hybrid Machine Translation for High-Quality Closed-Captioning and Subtitling for Video Broadcast

Automatic Speech Recognition and Hybrid Machine Translation for High-Quality Closed-Captioning and Subtitling for Video Broadcast Hassan Sawaf Science Applications International Corporation (SAIC) 7990

### Improved Discriminative Bilingual Word Alignment

Improved Discriminative Bilingual Word Alignment Robert C. Moore Wen-tau Yih Andreas Bode Microsoft Research Redmond, WA 98052, USA {bobmoore,scottyhi,abode}@microsoft.com Abstract For many years, statistical

### Context-based Arabic Morphological Analysis for Machine Translation

Context-based Morphological Analysis for Machine Translation ThuyLinh Nguyen Language Technologies Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213, USA thuylinh@cs.cmu.edu

### Hybrid Machine Translation Guided by a Rule Based System

Hybrid Machine Translation Guided by a Rule Based System Cristina España-Bonet, Gorka Labaka, Arantza Díaz de Ilarraza, Lluís Màrquez Kepa Sarasola Universitat Politècnica de Catalunya University of the

### Statistical Phrase-Based Translation

Statistical Phrase-Based Translation Philipp Koehn, Franz Josef Och, Daniel Marcu Information Sciences Institute Department of Computer Science University of Southern California koehn@isi.edu, och@isi.edu,

### Machine Translation. Agenda

Agenda Introduction to Machine Translation Data-driven statistical machine translation Translation models Parallel corpora Document-, sentence-, word-alignment Phrase-based translation MT decoding algorithm

### Towards a General and Extensible Phrase-Extraction Algorithm

Towards a General and Extensible Phrase-Extraction Algorithm Wang Ling, Tiago Luís, João Graça, Luísa Coheur and Isabel Trancoso L 2 F Spoken Systems Lab INESC-ID Lisboa {wang.ling,tiago.luis,joao.graca,luisa.coheur,imt}@l2f.inesc-id.pt

### Sentence Segmentation Using IBM Word Alignment Model 1

Sentence Segmentation Using IBM Word Alignment Model 1 Jia Xu and Richard Zens and Hermann Ney Chair of Computer Science VI, Computer Science Department RWTH Aachen University D-52056 Aachen, Germany {xujia,zens,ney}@cs.rwth-aachen.de

### T U R K A L A T O R 1

T U R K A L A T O R 1 A Suite of Tools for Augmenting English-to-Turkish Statistical Machine Translation by Gorkem Ozbek [gorkem@stanford.edu] Siddharth Jonathan [jonsid@stanford.edu] CS224N: Natural Language

### Factored Markov Translation with Robust Modeling

Factored Markov Translation with Robust Modeling Yang Feng Trevor Cohn Xinkai Du Information Sciences Institue Computing and Information Systems Computer Science Department The University of Melbourne

### Building a Question Classifier for a TREC-Style Question Answering System

Building a Question Classifier for a TREC-Style Question Answering System Richard May & Ari Steinberg Topic: Question Classification We define Question Classification (QC) here to be the task that, given

### Chapter 8. Final Results on Dutch Senseval-2 Test Data

Chapter 8 Final Results on Dutch Senseval-2 Test Data The general idea of testing is to assess how well a given model works and that can only be done properly on data that has not been seen before. Supervised

### Guest Editors Introduction: Machine Learning in Speech and Language Technologies

Guest Editors Introduction: Machine Learning in Speech and Language Technologies Pascale Fung (pascale@ee.ust.hk) Department of Electrical and Electronic Engineering Hong Kong University of Science and

### Phrase-Based MT. Machine Translation Lecture 7. Instructor: Chris Callison-Burch TAs: Mitchell Stern, Justin Chiu. Website: mt-class.

Phrase-Based MT Machine Translation Lecture 7 Instructor: Chris Callison-Burch TAs: Mitchell Stern, Justin Chiu Website: mt-class.org/penn Translational Equivalence Er hat die Prüfung bestanden, jedoch

### Online Large-Margin Training of Syntactic and Structural Translation Features

Online Large-Margin Training of Syntactic and Structural Translation Features David Chiang Information Sciences Institute University of Southern California 4676 Admiralty Way, Suite 1001 Marina del Rey,

### The Prague Bulletin of Mathematical Linguistics NUMBER 95 APRIL 2011 5 18. A Guide to Jane, an Open Source Hierarchical Translation Toolkit

The Prague Bulletin of Mathematical Linguistics NUMBER 95 APRIL 2011 5 18 A Guide to Jane, an Open Source Hierarchical Translation Toolkit Daniel Stein, David Vilar, Stephan Peitz, Markus Freitag, Matthias

### The KIT Translation Systems for IWSLT 2013

The KIT Translation Systems for IWSLT 2013 Thanh-Le Ha, Teresa Herrmann, Jan Niehues, Mohammed Mediani, Eunah Cho, Yuqi Zhang, Isabel Slawik and Alex Waibel Institute for Anthropomatics KIT - Karlsruhe

### Nonparametric Word Segmentation for Machine Translation

Nonparametric Word Segmentation for Machine Translation ThuyLinh Nguyen Stephan Vogel Noah A. Smith Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213, USA {thuylinh,vogel,nasmith}@cs.cmu.edu

### Course: Model, Learning, and Inference: Lecture 5

Course: Model, Learning, and Inference: Lecture 5 Alan Yuille Department of Statistics, UCLA Los Angeles, CA 90095 yuille@stat.ucla.edu Abstract Probability distributions on structured representation.

### Tagging with Hidden Markov Models

Tagging with Hidden Markov Models Michael Collins 1 Tagging Problems In many NLP problems, we would like to model pairs of sequences. Part-of-speech (POS) tagging is perhaps the earliest, and most famous,

### Convergence of Translation Memory and Statistical Machine Translation

Convergence of Translation Memory and Statistical Machine Translation Philipp Koehn and Jean Senellart 4 November 2010 Progress in Translation Automation 1 Translation Memory (TM) translators store past

### A Block Bigram Prediction Model for Statistical Machine Translation

A Block Bigram Prediction Model for Statistical Machine Translation CHRISTOPH TILLMANN IBM T.J. Watson Research Center and TONG ZHANG Yahoo Research In this paper, we present a novel training method for

### A Flexible Online Server for Machine Translation Evaluation

A Flexible Online Server for Machine Translation Evaluation Matthias Eck, Stephan Vogel, and Alex Waibel InterACT Research Carnegie Mellon University Pittsburgh, PA, 15213, USA {matteck, vogel, waibel}@cs.cmu.edu

### Structured Models for Fine-to-Coarse Sentiment Analysis

Structured Models for Fine-to-Coarse Sentiment Analysis Ryan McDonald Kerry Hannan Tyler Neylon Mike Wells Jeff Reynar Google, Inc. 76 Ninth Avenue New York, NY 10011 Contact email: ryanmcd@google.com

### Visualizing Data Structures in Parsing-based Machine Translation. Jonathan Weese, Chris Callison-Burch

The Prague Bulletin of Mathematical Linguistics NUMBER 93 JANUARY 2010 127 136 Visualizing Data Structures in Parsing-based Machine Translation Jonathan Weese, Chris Callison-Burch Abstract As machine

### Making Sense of the Mayhem: Machine Learning and March Madness

Making Sense of the Mayhem: Machine Learning and March Madness Alex Tran and Adam Ginzberg Stanford University atran3@stanford.edu ginzberg@stanford.edu I. Introduction III. Model The goal of our research

### Scaling Shrinkage-Based Language Models

Scaling Shrinkage-Based Language Models Stanley F. Chen, Lidia Mangu, Bhuvana Ramabhadran, Ruhi Sarikaya, Abhinav Sethy IBM T.J. Watson Research Center P.O. Box 218, Yorktown Heights, NY 10598 USA {stanchen,mangu,bhuvana,sarikaya,asethy}@us.ibm.com

### PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION Introduction In the previous chapter, we explored a class of regression models having particularly simple analytical

### Genetic Algorithm-based Multi-Word Automatic Language Translation

Recent Advances in Intelligent Information Systems ISBN 978-83-60434-59-8, pages 751 760 Genetic Algorithm-based Multi-Word Automatic Language Translation Ali Zogheib IT-Universitetet i Goteborg - Department

### Phrases. Topics for Today. Phrases. POS Tagging. ! Text transformation. ! Text processing issues

Topics for Today! Text transformation Word occurrence statistics Tokenizing Stopping and stemming Phrases Document structure Link analysis Information extraction Internationalization Phrases! Many queries

### Natural Language Processing. Today. Logistic Regression Models. Lecture 13 10/6/2015. Jim Martin. Multinomial Logistic Regression

Natural Language Processing Lecture 13 10/6/2015 Jim Martin Today Multinomial Logistic Regression Aka log-linear models or maximum entropy (maxent) Components of the model Learning the parameters 10/1/15

### HIERARCHICAL HYBRID TRANSLATION BETWEEN ENGLISH AND GERMAN

HIERARCHICAL HYBRID TRANSLATION BETWEEN ENGLISH AND GERMAN Yu Chen, Andreas Eisele DFKI GmbH, Saarbrücken, Germany May 28, 2010 OUTLINE INTRODUCTION ARCHITECTURE EXPERIMENTS CONCLUSION SMT VS. RBMT [K.

### English Grammar Checker

International l Journal of Computer Sciences and Engineering Open Access Review Paper Volume-4, Issue-3 E-ISSN: 2347-2693 English Grammar Checker Pratik Ghosalkar 1*, Sarvesh Malagi 2, Vatsal Nagda 3,

### Efficient Discriminative Training of Long-span Language Models

Efficient Discriminative Training of Long-span Language Models Ariya Rastrow, Mark Dredze, Sanjeev Khudanpur Human Language Technology Center of Excellence Center for Language and Speech Processing, Johns

### Word Completion and Prediction in Hebrew

Experiments with Language Models for בס"ד Word Completion and Prediction in Hebrew 1 Yaakov HaCohen-Kerner, Asaf Applebaum, Jacob Bitterman Department of Computer Science Jerusalem College of Technology

### 7-2 Speech-to-Speech Translation System Field Experiments in All Over Japan

7-2 Speech-to-Speech Translation System Field Experiments in All Over Japan We explain field experiments conducted during the 2009 fiscal year in five areas of Japan. We also show the experiments of evaluation

### Customizing an English-Korean Machine Translation System for Patent Translation *

Customizing an English-Korean Machine Translation System for Patent Translation * Sung-Kwon Choi, Young-Gil Kim Natural Language Processing Team, Electronics and Telecommunications Research Institute,

### Building English-Chinese and Chinese-English MT engines for the computer software domain

Proceedings of the 16th EAMT Conference, 28-30 May 2012, Trento, Italy Building English-Chinese and Chinese-English MT engines for the computer software domain Maxim Khalilov TAUS Labs Oudeschans 85-3

### DEVELOPMENT AND ANALYSIS OF HINDI-URDU PARALLEL CORPUS

DEVELOPMENT AND ANALYSIS OF HINDI-URDU PARALLEL CORPUS Mandeep Kaur GNE, Ludhiana Ludhiana, India Navdeep Kaur SLIET, Longowal Sangrur, India Abstract This paper deals with Development and Analysis of

### The Artificial Prediction Market

The Artificial Prediction Market Adrian Barbu Department of Statistics Florida State University Joint work with Nathan Lay, Siemens Corporate Research 1 Overview Main Contributions A mathematical theory

### of Statistical Machine Translation as

The Open A.I. Kit TM : General Machine Learning Modules from Statistical Machine Translation Daniel J. Walker The University of Tennessee Knoxville, TN USA Abstract The Open A.I. Kit implements the major

### Factored bilingual n-gram language models for statistical machine translation

Mach Translat DOI 10.1007/s10590-010-9082-5 Factored bilingual n-gram language models for statistical machine translation Josep M. Crego François Yvon Received: 2 November 2009 / Accepted: 12 June 2010

### Online Large-Margin Training of Dependency Parsers

Online Large-Margin Training of Dependency Parsers Ryan McDonald Koby Crammer Fernando Pereira Department of Computer and Information Science University of Pennsylvania Philadelphia, PA {ryantm,crammer,pereira}@cis.upenn.edu

### Improved Statistical Machine Translation Using Paraphrases

Improved Statistical Machine Translation Using Paraphrases Chris Callison-Burch Philipp Koehn Miles Osborne School of Informatics University of Edinburgh 2 Buccleuch Place Edinburgh, EH8 9LW callison-burch@ed.ac.uk

### Chapter 4. Word-based models. Statistical Machine Translation

Chapter 4 Word-based models Statistical Machine Translation Lexical Translation How to translate a word look up in dictionary Haus house, building, home, household, shell. Multiple translations some more

### Machine Learning Final Project Spam Email Filtering

Machine Learning Final Project Spam Email Filtering March 2013 Shahar Yifrah Guy Lev Table of Content 1. OVERVIEW... 3 2. DATASET... 3 2.1 SOURCE... 3 2.2 CREATION OF TRAINING AND TEST SETS... 4 2.3 FEATURE

### Question of the Day. Machine Translation. Statistical Word Alignment. Centauri/Arcturan (Knight, 1997) Centauri/Arcturan (Knight, 1997)

Question of the Day Is it possible to learn to translate from plain example translations? Machine Translation Statistical Word Alignment Based on slides by Philipp Koehn and Kevin Knight Word Alignment

### Dublin City University at CLEF 2004: Experiments with the ImageCLEF St Andrew s Collection

Dublin City University at CLEF 2004: Experiments with the ImageCLEF St Andrew s Collection Gareth J. F. Jones, Declan Groves, Anna Khasin, Adenike Lam-Adesina, Bart Mellebeek. Andy Way School of Computing,

### Applying Co-Training Methods to Statistical Parsing. Anoop Sarkar http://www.cis.upenn.edu/ anoop/ anoop@linc.cis.upenn.edu

Applying Co-Training Methods to Statistical Parsing Anoop Sarkar http://www.cis.upenn.edu/ anoop/ anoop@linc.cis.upenn.edu 1 Statistical Parsing: the company s clinical trials of both its animal and human-based

### Recurrent (Neural) Networks: Part 1. Tomas Mikolov, Facebook AI Research Deep Learning NYU, 2015

Recurrent (Neural) Networks: Part 1 Tomas Mikolov, Facebook AI Research Deep Learning course @ NYU, 2015 Structure of the talk Introduction Architecture of simple recurrent nets Training: Backpropagation

### Strategies for Training Large Scale Neural Network Language Models

Strategies for Training Large Scale Neural Network Language Models Tomáš Mikolov #1, Anoop Deoras 2, Daniel Povey 3, Lukáš Burget #4, Jan Honza Černocký #5 # Brno University of Technology, Speech@FIT,

### Turker-Assisted Paraphrasing for English-Arabic Machine Translation

Turker-Assisted Paraphrasing for English-Arabic Machine Translation Michael Denkowski and Hassan Al-Haj and Alon Lavie Language Technologies Institute School of Computer Science Carnegie Mellon University

### Direct Loss Minimization for Structured Prediction

Direct Loss Minimization for Structured Prediction David McAllester TTI-Chicago mcallester@ttic.edu Tamir Hazan TTI-Chicago tamir@ttic.edu Joseph Keshet TTI-Chicago jkeshet@ttic.edu Abstract In discriminative

### Accurately and Efficiently Measuring Individual Account Credit Risk On Existing Portfolios

Accurately and Efficiently Measuring Individual Account Credit Risk On Existing Portfolios By: Michael Banasiak & By: Daniel Tantum, Ph.D. What Are Statistical Based Behavior Scoring Models And How Are

### Principles of Dat Da a t Mining Pham Tho Hoan hoanpt@hnue.edu.v hoanpt@hnue.edu. n

Principles of Data Mining Pham Tho Hoan hoanpt@hnue.edu.vn References [1] David Hand, Heikki Mannila and Padhraic Smyth, Principles of Data Mining, MIT press, 2002 [2] Jiawei Han and Micheline Kamber,

### Linear Threshold Units

Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear

### Toward Statistical Machine Translation without Parallel Corpora

Toward Statistical Machine Translation without Parallel Corpora Alex Klementiev, Ann Irvine, Chris Callison-Burch, and David Yarowsky Johns Hopkins University EACL 2012 Motivation State-of-the-art statistical

### DEPENDENCY PARSING JOAKIM NIVRE

DEPENDENCY PARSING JOAKIM NIVRE Contents 1. Dependency Trees 1 2. Arc-Factored Models 3 3. Online Learning 3 4. Eisner s Algorithm 4 5. Spanning Tree Parsing 6 References 7 A dependency parser analyzes

### Chinese-Japanese Machine Translation Exploiting Chinese Characters

Chinese-Japanese Machine Translation Exploiting Chinese Characters CHENHUI CHU, TOSHIAKI NAKAZAWA, DAISUKE KAWAHARA, and SADAO KUROHASHI, Kyoto University The Chinese and Japanese languages share Chinese

### Polish - English Statistical Machine Translation of Medical Texts.

Polish - English Statistical Machine Translation of Medical Texts. Krzysztof Wołk, Krzysztof Marasek Department of Multimedia Polish Japanese Institute of Information Technology kwolk@pjwstk.edu.pl Abstract.

### Interlocking Phrases in Phrase-based Statistical Machine Translation

Interlocking Phrases in Phrase-based Statistical Machine Translation Ye Kyaw Thu, Andrew Finch and Eiichiro Sumita Multilingual Translation Lab., Advanced Speech Translation Research and Development Promotion

### Learning. Artificial Intelligence. Learning. Types of Learning. Inductive Learning Method. Inductive Learning. Learning.

Learning Learning is essential for unknown environments, i.e., when designer lacks omniscience Artificial Intelligence Learning Chapter 8 Learning is useful as a system construction method, i.e., expose

### Jane 2: Open Source Phrase-based and Hierarchical Statistical Machine Translation

Jane 2: Open Source Phrase-based and Hierarchical Statistical Machine Translation Joern Wuebker M at thias Huck Stephan Peitz M al te Nuhn M arkus F reitag Jan-Thorsten Peter Saab M ansour Hermann N e

### Automatic slide assignation for language model adaptation

Automatic slide assignation for language model adaptation Applications of Computational Linguistics Adrià Agustí Martínez Villaronga May 23, 2013 1 Introduction Online multimedia repositories are rapidly

### Offline Recognition of Unconstrained Handwritten Texts Using HMMs and Statistical Language Models. Alessandro Vinciarelli, Samy Bengio and Horst Bunke

1 Offline Recognition of Unconstrained Handwritten Texts Using HMMs and Statistical Language Models Alessandro Vinciarelli, Samy Bengio and Horst Bunke Abstract This paper presents a system for the offline

### Search Aware Tuning for Machine Translation

Search Aware Tuning for Machine Translation 0 1 2 3 4 Lemao Liu Liang Huang City University of New York EMNLP 2014. Presented by Taro Watanabe. Search Aware Tuning for Machine Translation Lemao Liu Liang