Empirical Machine Translation and its Evaluation

Transcription

1 Empirical Machine Translation and its Evaluation EAMT Best Thesis Award 2008 Jesús Giménez (Advisor, Lluís Màrquez) Universitat Politècnica de Catalunya May 28, 2010

2 Empirical Machine Translation

3 Empirical Machine Translation

4 Empirical Machine Translation

5 Empirical Machine Translation

6 Empirical Machine Translation

7 Discriminative Phrase Translation A brilliant play written by William Locke Una obra brillante escrita por William Locke vs. A brilliant play by Lionel Messi that produced a wonderful goal Una brillante jugada de Lionel Messi que resultó en un bello gol

8 Discriminative Phrase Translation A brilliant play written by William Locke Una obra brillante escrita por William Locke vs. A brilliant play by Lionel Messi that produced a wonderful goal Una brillante jugada de Lionel Messi que resultó en un bello gol

9 Discriminative Phrase Translation A brilliant play written by William Locke Una obra brillante escrita por William Locke vs. A brilliant play by Lionel Messi that produced a wonderful goal Una brillante jugada de Lionel Messi que resultó en un bello gol

10 Discriminative Phrase Translation A brilliant play written by William Locke Una obra brillante escrita por William Locke vs. A brilliant play by Lionel Messi that produced a wonderful goal Una brillante jugada de Lionel Messi que resultó en un bello gol

11 Discriminative Phrase Translation A brilliant play written by William Locke Una obra brillante escrita por William Locke vs. A brilliant play by Lionel Messi that produced a wonderful goal Una brillante jugada de Lionel Messi que resultó en un bello gol

12 Discriminative Phrase Translation A brilliant play written by William Locke Una obra brillante escrita por William Locke vs. A brilliant play by Lionel Messi that produced a wonderful goal Una brillante jugada de Lionel Messi que resultó en un bello gol

13 Discriminative Phrase Translation Idea Use discriminative Machine Learning techniques in SMT to estimate P(e i f j ), actually, P(e i f j, context f j ) Translation modeling is addressed as a classification problem

14 Discriminative Phrase Translation Idea Use discriminative Machine Learning techniques in SMT to estimate P(e i f j ), actually, P(e i f j, context f j ) Translation modeling is addressed as a classification problem

15 Contributions 1 Discriminative Phrase Selection for SMT Shallow-syntactic features word, lemma, parts of speech, chunking local / global context Improved translation quality 2 Domain Dependence in SMT Parliament proceedings Dictionary definitions Adaptation based on: EuroWordNet out-of-domain data a small amount of in-domain data Improved translation quality

16 Contributions 1 Discriminative Phrase Selection for SMT Shallow-syntactic features word, lemma, parts of speech, chunking local / global context Improved translation quality 2 Domain Dependence in SMT Parliament proceedings Dictionary definitions Adaptation based on: EuroWordNet out-of-domain data a small amount of in-domain data Improved translation quality

17 Contributions 1 Discriminative Phrase Selection for SMT Shallow-syntactic features word, lemma, parts of speech, chunking local / global context Improved translation quality 2 Domain Dependence in SMT Parliament proceedings Dictionary definitions Adaptation based on: EuroWordNet out-of-domain data a small amount of in-domain data Improved translation quality

18 Contributions 1 Discriminative Phrase Selection for SMT Shallow-syntactic features word, lemma, parts of speech, chunking local / global context Improved translation quality 2 Domain Dependence in SMT Parliament proceedings Dictionary definitions Adaptation based on: EuroWordNet out-of-domain data a small amount of in-domain data Improved translation quality

19 ... and its Evaluation

20 ... and its Evaluation

21 ... and its Evaluation

22 Limits of Lexical Similarity Candidate On Tuesday several missiles and mortar Translation shells fell in southern Israel, but there were no casualties. Reference Several Qassam rockets and mortar shells Translation fell today, Tuesday, in southern Israel without causing any casualties. Only one 4-gram in common!

23 Limits of Lexical Similarity Candidate On Tuesday several missiles and mortar Translation shells fell in southern Israel, but there were no casualties. Reference Several Qassam rockets and mortar shells Translation fell today, Tuesday, in southern Israel without causing any casualties. Only one 4-gram in common!

24 Limits of Lexical Similarity Candidate On Tuesday several missiles and mortar Translation shells fell in southern Israel, but there were no casualties. Reference Several Qassam rockets and mortar shells Translation fell today, Tuesday, in southern Israel without causing any casualties. Only one 4-gram in common!

25 Linguistic Features for Automatic MT Evaluation Idea Define similarity measures based on deeper linguistic information Compare linguistic structures and their lexical realizations Linguistic levels Syntax Parts-of-speech Base phrase chunks Phrase constituents Dependency relationships Semantics Named entities Semantic roles Discourse representations

26 Linguistic Features for Automatic MT Evaluation S PP TMP S. On NP NP A1 VP, but S Tuesday several missiles and mortar shells fell PP LOC NP VP in NP there were NP southern Israel no casualties

27 Linguistic Features for Automatic MT Evaluation S PP TMP S. On NP NP A1 VP, but S Tuesday several missiles and mortar shells fell PP LOC NP VP in NP there were NP southern Israel no casualties

28 Linguistic Features for Automatic MT Evaluation S NP A1 A0 VP. NP and NP fell NP PP LOC PP ADV Several Qassam rockets mortar shells NP TMP, NP, in NP without S today Tuesday southern VP Israel causing NP A1 any casualties

29 Linguistic Features for Automatic MT Evaluation S NP A1 A0 VP. NP and NP fell NP PP LOC PP ADV Several Qassam rockets mortar shells NP TMP, NP, in NP without S today Tuesday southern VP Israel causing NP A1 any casualties

30 Contributions 1 Linguistic measures provide more reliable rankings when the systems under evaluation are based on different paradigms (statistical vs. rule-based, fully-automatic vs. human-aided) 2 Linguistic measures have proven effective in shared tasks (WMT ) both at the system and sentence levels 3 Some linguistic measures suffer a substantial quality decrease at the sentence level (due to parsing errors!) 4 Lexical and Linguistic measures are complementary suitable for being combined!

31 Contributions 1 Linguistic measures provide more reliable rankings when the systems under evaluation are based on different paradigms (statistical vs. rule-based, fully-automatic vs. human-aided) 2 Linguistic measures have proven effective in shared tasks (WMT ) both at the system and sentence levels 3 Some linguistic measures suffer a substantial quality decrease at the sentence level (due to parsing errors!) 4 Lexical and Linguistic measures are complementary suitable for being combined!

32 Contributions 1 Linguistic measures provide more reliable rankings when the systems under evaluation are based on different paradigms (statistical vs. rule-based, fully-automatic vs. human-aided) 2 Linguistic measures have proven effective in shared tasks (WMT ) both at the system and sentence levels 3 Some linguistic measures suffer a substantial quality decrease at the sentence level (due to parsing errors!) 4 Lexical and Linguistic measures are complementary suitable for being combined!

33 Contributions 1 Linguistic measures provide more reliable rankings when the systems under evaluation are based on different paradigms (statistical vs. rule-based, fully-automatic vs. human-aided) 2 Linguistic measures have proven effective in shared tasks (WMT ) both at the system and sentence levels 3 Some linguistic measures suffer a substantial quality decrease at the sentence level (due to parsing errors!) 4 Lexical and Linguistic measures are complementary suitable for being combined!

34 Contributions 1 Linguistic measures provide more reliable rankings when the systems under evaluation are based on different paradigms (statistical vs. rule-based, fully-automatic vs. human-aided) 2 Linguistic measures have proven effective in shared tasks (WMT ) both at the system and sentence levels 3 Some linguistic measures suffer a substantial quality decrease at the sentence level (due to parsing errors!) 4 Lexical and Linguistic measures are complementary suitable for being combined!

35 Acknowledgements 1 Spanish Government Ministry of Science and Technology Ministry of Education 2 Organizers and participants of the NIST, WMT and IWSLT Evaluation Campaigns 3 A number of NLP researchers worldwide sharing their software 4 Enrique Amigó and German Rigau

36 Acknowledgements 1 Spanish Government Ministry of Science and Technology Ministry of Education 2 Organizers and participants of the NIST, WMT and IWSLT Evaluation Campaigns 3 A number of NLP researchers worldwide sharing their software 4 Enrique Amigó and German Rigau

37 Acknowledgements 1 Spanish Government Ministry of Science and Technology Ministry of Education 2 Organizers and participants of the NIST, WMT and IWSLT Evaluation Campaigns 3 A number of NLP researchers worldwide sharing their software 4 Enrique Amigó and German Rigau

38 Acknowledgements 1 Spanish Government Ministry of Science and Technology Ministry of Education 2 Organizers and participants of the NIST, WMT and IWSLT Evaluation Campaigns 3 A number of NLP researchers worldwide sharing their software 4 Enrique Amigó and German Rigau

39 Empirical Machine Translation and its Evaluation EAMT Best Thesis Award 2008 Thanks!