How to Read Clearly Without Having a Brainstorm

Transcription

1 Evaluating translation quality Machine Translation Lecture 9 Instructor: Chris Callison-Burch TAs: Mitchell Stern, Justin Chiu Website: mt-class.org/penn

2 Goals for this lecture Understanding advantages of human versus automatic evaluation Details of BLEU How to validate automatic evaluation metrics What makes a good {manual / automatic} evaluation?

3 Evaluating MT Quality Why do we want to do it? Want to rank systems Want to evaluate incremental changes What to make scientific claims How not to do it Back translation The vodka is not good

4 Human Evaluation of MT v. Automatic Evaluation Human evaluation is Ultimately what we're interested in, but Very time consuming Not re-usable Automatic evaluation is Cheap and reusable, but Not necessarily reliable

5 Rank Sentences Manual Evaluation You have judged 25 sentences for WMT09 Spanish-English News Corpus, 427 sentences total taking 64.9 seconds per sentence. Source: Estos tejidos están analizados, transformados y congelados antes de ser almacenados en Hema- Québec, que gestiona también el único banco público de sangre del cordón umbilical en Quebec. Reference: These tissues are analyzed, processed and frozen before being stored at Héma-Québec, which manages also the only bank of placental blood in Quebec. Translation These weavings are analyzed, transformed and frozen before being stored in Hema-Quebec, that negotiates also the public only bank of blood of the umbilical cord in Quebec. These tissues analysed, processed and before frozen of stored in Hema- Québec, which also operates the only public bank umbilical cord blood in Quebec. These tissues are analyzed, processed and frozen before being stored in Hema-Québec, which also manages the only public bank umbilical cord blood in Quebec. These tissues are analyzed, processed and frozen before being stored in Hema-Quebec, which also operates the only public bank of umbilical cord blood in Quebec. These fabrics are analyzed, are transformed and are frozen before being stored in Hema-Québec, who manages also the only public bank of blood of the umbilical cord in Quebec. Annotator: ccb Task: WMT09 Spanish-English News Corpus Rank 1 Best 1 Best 1 Best 1 Best 1 Best Worst Worst Worst Worst Worst

6 Goals for Automatic Evaluation No cost evaluation for incremental changes Ability to rank systems Ability to identify which sentences we're doing poorly on, and categorize errors Correlation with human judgments Interpretability of the score

7 Methodology Comparison against reference translations Intuition: closer we get to human translations, the better we're doing Could use WER like in speech recognition

8 Word Error Rate Levenshtein Distance (also "edit distance") Minimum number of insertions, substitutions, and deletions needed to transform one string into another Useful measure in speech recognition This shows how easy it is to recognize speech This shows how easy it is to wreck a nice beach

9 Problems with using WER for translation? (discuss with your neighbor)

10 Problems with WER Unlike speech recognition we don't have the assumption of exact match against the reference In machine translation there can be many possible (and equally valid) ways of translating a sentence This shows how easy it is to recognize speech It illustrates how simple it is to transcribe the spoken word

11 Problems with WER Unlike speech recognition we don't have the assumption of linearity Clauses can move around, since we're not doing transcription This shows how easy it is to recognize speech It is easy to recognize speech, as this shows This shows that recognizing speech is easy

12 Solutions? (Talk to your neighbor)

13 Solutions Compare against lots of test sentences Use multiple reference translations for each test sentence Look for phrase / n-gram matches, allow movement

14 BLEU BiLingual Evaluation Understudy Uses multiple reference translations Look for n-grams that occur anywhere in the sentence

15 Multiple references Ref 1 Ref 2 Ref 3 Ref 4 Orejuela appeared calm as he was led to the American plane which will take him to Miami, Florida. Orejuela appeared calm while being escorted to the plane that would take him to Miami, Florida. Orejuela appeared calm as he was being led to the American plane that was to carry him to Miami in Florida. Orejuela seemed quite calm as he was being led to the American plane that would take him to Miami in Florida.

16 (2002) define Bleu in terms of n-gram precision. T core, p n, for each n-gram length by summing over n-gram precision entence S in the complete corpus C as: p n = Â S2C Â ngram2s Count matched (ngram) Â S2C Â ngram2s Count(ngram) cision is modified slightly to eliminate repetitions t BLEU modifies this precision to eliminate repetitions that occur across sentences. mple, even though the bigram to Miami is repeate ns in Table 6.1, it is counted only once in a hypoth as clipped n-gram precision.

17 Modified precision Ref 1 Ref 2 Ref 3 Ref 4 Orejuela appeared calm as he was led to the American plane which will take him to Miami, Florida. Orejuela appeared calm while being escorted to the plane that would take him to Miami, Florida. Orejuela appeared calm as he was being led to the American plane that was to carry him to Miami in Florida. Orejuela seemed quite calm as he was being led to the American plane that would take him to Miami in Florida. to Miami can only be counted as correct once

18 Ref 1 Ref 2 Ref 3 Ref 4 Orejuela appeared calm as he was led to the American plane which will take him to Miami, Florida. Orejuela appeared calm while being escorted to the plane that would take him to Miami, Florida. Orejuela appeared calm as he was being led to the American plane that was to carry him to Miami in Florida. Orejuela seemed quite calm as he was being led to the American plane that would take him to Miami in Florida. Hyp appeared calm when he was taken to the American plane, which will to Miami, Florida.

19 American, Florida, Miami, Orejuela, appeared, as, being, calm, carry, escorted, he, him, in, led, plane, quite, seemed, take, that, the, to, to, to, was, was, which, while, will, would,,,. 1-gram precision = 15/18 Hyp appeared calm when he was taken to the American plane, which will to Miami, Florida.

20 American plane, Florida., Miami,, Miami in, Orejuela appeared, Orejuela seemed, appeared calm, as he, being escorted, being led, calm as, calm while, carry him, escorted to, he was, him to, in Florida, led to, plane that, plane which, quite calm, seemed quite, take him, that was, that would, the American, the plane, to Miami, to carry, to the, was being, was led, was to, which will, while being, will take, would take,, Florida 2-gram precision = 10/17 Hyp appeared calm when he was taken to the American plane, which will to Miami, Florida.

21 n-gram precision Hyp appeared calm when he was taken to the American plane, which will to Miami, Florida. 1-gram precision = 15/18 =.83 2-gram precision = 10/17 =.59 3-gram precision = 5/16 =.31 4-gram precision = 3/15 =.20 Geometric average (0.83 * 0.59 * 0.31 * 0.2)^(1/4) = or equivalently exp(ln.83 + ln.59 + ln.31 + ln.2/4) = 0.417

22 Ref 1 Ref 2 Ref 3 Ref 4 Orejuela appeared calm as he was led to the American plane which will take him to Miami, Florida. Orejuela appeared calm while being escorted to the plane that would take him to Miami, Florida. Orejuela appeared calm as he was being led to the American plane that was to carry him to Miami in Florida. Orejuela seemed quite calm as he was being led to the American plane that would take him to Miami in Florida. Hyp to the American plane

23 Better? Hyp to the American plane 1-gram precision = 4/4 = gram precision = 3/3 = gram precision = 2/2 = gram precision = 1/1 = 1.0 exp(ln 1 + ln 1 + ln 1 + ln 1) = 1

24 than defining n-gram recall Bleu instead introduces a brevity penalty Brevity Penalty the possibility of proposing high-precision hypothesis translations rt. The brevity penalty is calculated as: ( 1 if c > r BP = e 1 r/c if c apple r the length c of is the the corpus length of of hypothesis the corpus translations, of hypothesis and r is the e orpus length. translations The effective reference corpus length is calculated as t le reference r is translation the effective fromreference each set corpus which is length closest to the hyp The effective reference corpus length is the sum of the single reference translation from vity penalty is combined with the weighted sum of n-gram precision each set that is closest to the hypothesis u score. Bleu translation. is thus calculated as

25 Brevity Penalty BP MT is Longer MT is Shorter Difference with effective reference length (%)

26 Ref 1 Hyp Orejuela appeared calm as he was led to the American plane which will take him to Miami, Florida. r = 20 appeared calm when he was taken to the American plane, which will to Miami, Florida. c = 18 BP = exp(1-(20/18)) = 0.89 Ref 1 Orejuela appeared calm as he was led to the American plane which will take him to Miami, Florida. r = 20 Hyp to the American plane c = 4 BP = exp(1-(20/4)) = 0.02

27 Chapter 6. Evaluating Translation BLEU Bleu = BP exp( N Â n=1 w n logp n ) ore can range from 0 to 1, where higher scores indicate closer ma translations, and where a score of 1 is assigned to a hypothesis tra Geometric average of the n-gram precisions y matches Optionally one of the reference weight them translations. with w A score of 1 is also a is translation Multiplied which has by matches the brevity forpenalty all its n-grams (up to the max Bleu) in the clipped reference n-grams, and which has no brevity n idea of how Bleu is calculated we will walk through what t

28 BLEU Hyp appeared calm when he was taken to the American plane, which will to Miami, Florida. exp(1-(20/18)) * exp((ln.83 + ln.59 + ln.31 + ln.2)/4) = Hyp to the American plane exp(1-(20/4)) * exp((ln 1 + ln 1 + ln 1 + ln 1)/4) = 0.018

29 Problems with BLEU (Discuss with your neighbor)

30 Problems with BLEU Synonyms and paraphrases are only handled if they are in the set of multiple reference translations The scores for words are equally weighted so missing out on content-bearing material brings no additional penalty. The brevity penalty is a stop-gap measure to compensate for the fairly serious problem of not being able to calculate recall.

31 More Metrics WER - word error rate PI-WER - position independent WER METEOR - Metric for Evaluation of Translation with Explicit ORdering TERp - Translation Edit Rate plus

32 Even More Metrics Metric IDs Participant AMBER, AMBER-NL, AMBER-IT National Research Council Canad F15, F15G3 Koç University (Bicici and Yuret, METEOR-1.3-ADQ, METEOR-1.3-RANK Carnegie Mellon University (Denk MTERATER, MTERATER-PLUS Columbia / ETS (Parton et al., 201 MP4IBM1, MPF, WMPF DFKI (Popović, 2011; Popović et PARSECONF DFKI (Avramidis et al., 2011) ROSE, ROSE-POS The University of Sheffield (Song TESLA-B, TESLA-F, TESLA-M National University of Singapore ( TINE University of Wolverhampton (Rio BLEU provided baseline (Papineni et al., TER provided baseline (Snover et al., 2 Table 11: Participants in the evaluation shared task. For comparison purposes, we includ

33 How do we know which metric is best? Measure correlation with human judgments How do people evaluation MT quality

34 Rank Sentences Manual Evaluation You have judged 25 sentences for WMT09 Spanish-English News Corpus, 427 sentences total taking 64.9 seconds per sentence. Source: Estos tejidos están analizados, transformados y congelados antes de ser almacenados en Hema- Québec, que gestiona también el único banco público de sangre del cordón umbilical en Quebec. Reference: These tissues are analyzed, processed and frozen before being stored at Héma-Québec, which manages also the only bank of placental blood in Quebec. Translation These weavings are analyzed, transformed and frozen before being stored in Hema-Quebec, that negotiates also the public only bank of blood of the umbilical cord in Quebec. These tissues analysed, processed and before frozen of stored in Hema- Québec, which also operates the only public bank umbilical cord blood in Quebec. These tissues are analyzed, processed and frozen before being stored in Hema-Québec, which also manages the only public bank umbilical cord blood in Quebec. These tissues are analyzed, processed and frozen before being stored in Hema-Quebec, which also operates the only public bank of umbilical cord blood in Quebec. These fabrics are analyzed, are transformed and are frozen before being stored in Hema-Québec, who manages also the only public bank of blood of the umbilical cord in Quebec. Annotator: ccb Task: WMT09 Spanish-English News Corpus Rank 1 Best 1 Best 1 Best 1 Best 1 Best Worst Worst Worst Worst Worst

35 5-point scales Fluency How do you judge the fluency of this translation? 5 = Flawless English 4 = Good English 3 = Non-native English 2 = Disfluent English 1 = Incomprehensible Adequacy How much of the meaning expressed in the reference translation is also expressed in the hypothesis translation? 5 = All 4 = Most 3 = Much 2 = Little 1 = None Iran has already s ments to the confe danian King Abdu accused Iran of int n-gram matches: 15 trigrams, and te human scores: Ad Iran already annou attend the confere ments made by th dullah II who has in Iraqi affairs. n-gram matches: 15 trigrams, and 12 human scores: Ad Reference: Iran Kharazi would bo

36 Heather Locklear Arrested for driving under the influence of drugs The actress Heather Locklear, Amanda of the popular series Melrose Place, was arrested this weekend in Santa Barbara (California) after driving under the influence of drugs. A witness viewed her performing inappropriate maneuvers while trying to take her car out from a parking in Montecito, as revealed to People magazine by a spokesman for the Californian Highway Police. The witness stated that around 4.30pm Ms. Locklear "hit the accelerator very violently, making excessive noise while trying to take her car out from the parking with abrupt back and forth maneuvers. While reversing, she passed several times in front of his sunglasses." Shortly after, the witness, who, in a first time, apparently had not recognized the actress, saw Ms. Why was Heather Locklear arrested? She was arrested on suspicion of driving under the influence of drugs. Why did the bystander call emergency services? He was concerned for Ms. Locklear s life. Where did the witness see her acting abnormally? Pulling out of parking in Montecito

37 Heather Was arrested Locklear actress Arrested Heather for Locklear because of the driving under driving under the influence of drugs the effect of an unknown medicine The actress Heather Locklear, that Amanda is known of to the popular Amanda series through Melrose the role from Place, the was series arrested "Melrose this weekend Place" was in arrested Santa Barbara at this (California) weekend in after Santa driving Barbara under the influence (Californium) of drugs. because A witness of the viewed driving under her performing the effect of an inappropriate unknown medicine. maneuvers A female while trying witness to observed take her car she out attempted from a parking quite strange in Montecito, way how as revealed to go to from People their parking magazine space by a in spokesman Montecito, speaker for the Californian of the traffic Highway police of californium Police. The told witness the stated warehouse that around `People'. 4.30pm The female Ms. Locklear witness told "hit in the detail, accelerator that Locklear very violently, 'pressed `after making 16:30 excessive clock noise while accelerator trying and to take a lot her of car noise out did from when the she parking attempted with to abrupt move back their and car towards forth maneuvers. behind or While forward reversing, from the parking she passed space, several and when times it went in backwards, front of his she sunglasses." pulled Shortly itself together after, the unites witness, Male who, at their in a sunglasses'. first time, apparently A little later had the not recognized female witness the actress, that did probably saw Ms. Why was Heather Locklear arrested? Driving while medicated Why did the bystander call emergency services? There was a lot of noise Where did the witness see her acting abnormally? In a parking lot

38 Heather Was Actress. Medikamentes arrested Heather Locklear actress unknown Locklear Arrested Heather was have for due the to Locklear driving effect of under a because fahrens the influence of under the driving actress of an under driving under the influence of drugs unknown heather effect locklear drug of an unknown arrested medicine The Actress In Santa. actress Heather One Heather is, Locklear, melrose Locklear, place by the that Amanda is role the known series of Amanda of to of the popular Amanda from role the of the through series the "Melrose 'remember role from Place, Place" the the locklear was is series known, arrested actress "Melrose was this weekend Place" this the heather weekend was in arrested Santa this in Santa weekend, Barbara at Barbara this weekend (California) because of in after because the Santa fahrens driving Barbara of driving Barbara under the influence (Californium) under (California) the of influence drugs. because effect A of unknown witness an of the viewed driving unknown medikamentes under her drug performing the arrested. effect of People A an witness inappropriate unknown had 'magazine. observed medicine. The maneuvers how traffic it A quite female police while strange trying witness way California, tried to observed take to spokesman park her their car she out extended attempted for from the a parking gap auszufahren quite Montecito, strange in montecito way reported how as reported revealed to go in to from spokesman its People way their from parking magazine for tried the parklücke space traffic by a in police type spokesman Montecito, from strange California right, speaker you to the the have Californian of magazine the seen traffic as a Highway police `People witness. of '.. californium Police. In The some witness The Zeitung, told witness told the in as and stated warehouse detail when that they Locklear tried around `People'. to a 4.30pm `after great The 16:30 female deal Ms. of Locklear witness clock 30 p.m., durchdrückte told witness "hit in the detail, accelerator the pedal detail that and Locklear of very a lot violently, 'pressed of history noise locklear made `after making by 16:30 after excessive trying 16: clock to that her noise car while accelerator durchdrückte the trying rear or and to front peddle take a lot of her of noise the car noise park out and did gap its from when to progress move, the she parking and was attempted when made with she parklücke to abrupt move went back their for and car backwards, the towards car forth maneuvers. moving took behind it backwards, a or few While forward times they in reversing, from their had they Sunglass'. parking times she passed of Somewhat space, their several and later when times it the sonnenbrille went witness in backwards, front saw ' of. The his the she sunglasses." first beginning pulled was of Shortly itself the probably actress together after, recognised probably the unites witness, Male had that not locklear who, at their in sunglasses'. recognized a first nearby time, road apparently that A and little Locklear anhielt, later had the on had not a recognized female nearby not, with witness road the the and witness actress, that stopped did the probably saw the car Ms. car off Why was Heather Locklear arrested? Why did the bystander call emergency services? Where did the witness see her acting abnormally?

39 Was Actress. Heather Medikamentes arrested Heather Locklear actress unknown Locklear Arrested Heather was have for due the to Locklear driving effect of under a because fahrens the influence of under the driving actress of an under driving under the influence of drugs unknown heather effect locklear drug of an unknown arrested medicine Actress In The Santa. actress Heather One Heather is, Locklear, melrose Locklear, place by the that is role the Amanda known series of Amanda of to of the the popular Amanda from role the of series the through the "Melrose 'remember role from Place, Place" the the locklear was is series known, arrested actress "Melrose was this Place" this the weekend heather weekend was in arrested Santa this in Santa weekend, Barbara at Barbara this weekend because (California) of in because after the Santa fahrens driving Barbara of driving Barbara under the (Californium) under (California) influence the of influence drugs. because effect A of unknown witness an of the driving unknown medikamentes viewed under her drug performing the arrested. effect of People A an witness unknown had 'magazine. inappropriate observed medicine. The maneuvers how traffic it A quite female police while strange witness way California, trying tried to observed take to spokesman park her their car she out extended attempted for from the a gap auszufahren parking quite strange in Montecito, montecito way reported how as reported revealed to go in from spokesman its to People way their from parking magazine for tried the parklücke space traffic by a in police type Montecito, from strange spokesman California right, speaker you to the the have Californian of magazine the seen traffic as a police `People witness. Highway of '.. californium Police. In The some witness The Zeitung, told witness told the in as and warehouse detail when stated they that Locklear tried around `People'. to a 4.30pm `after great The 16:30 female deal Ms. of witness clock 30 Locklear p.m., durchdrückte told witness "hit in the detail, accelerator the pedal detail that and Locklear of very a lot 'pressed of history violently, noise locklear made `after making by 16:30 after excessive trying 16: clock to that her noise car accelerator durchdrückte while the trying rear or and to front peddle take a lot of her of noise the car noise park out and did gap its when to progress from move, the she parking and was attempted when made with she parklücke to abrupt move went back their for car backwards, the and towards car forth maneuvers. moving took behind it backwards, a or few While forward times they in from their had reversing, they Sunglass'. parking times she passed of Somewhat space, their several and later when it the sonnenbrille times went witness in backwards, front saw ' of. The his the she sunglasses." first beginning pulled was of itself the probably Shortly actress together after, recognised probably the unites witness, Male had that not locklear who, at their in sunglasses'. recognized a nearby first time, road apparently that A and little Locklear anhielt, later had the on had not a female nearby not, recognized with witness road the the and witness actress, that stopped did the probably saw the car Ms. car off Why was Heather Locklear arrested? She was arrested on suspicion of driving under the influence of drugs. Driving under the influence Driving while medicated DUI Driving while using drugs Medikamentes

40 Was Actress. Heather Medikamentes arrested Heather Locklear actress unknown Locklear Arrested Heather was have for due the to Locklear driving effect of under a because fahrens the influence of under the driving actress of an under driving under the influence of drugs unknown heather effect locklear drug of an unknown arrested medicine System Actress In The Santa. actress Heather One Heather is, Locklear, melrose Locklear, place by the that is role the Amanda known series of Amanda of to of the the popular Amanda from role the of series the through the "Melrose 'remember role from Place, Place" the the locklear was is series known, arrested actress "Melrose was this Place" this the weekend heather weekend was in arrested Santa this in Santa weekend, Barbara at Barbara this weekend because (California) of in because after the Santa fahrens driving Barbara of driving Barbara under the (Californium) under (California) influence the of influence drugs. because effect A of unknown witness an of the driving unknown medikamentes viewed under her drug performing the arrested. effect of People A an witness unknown had 'magazine. inappropriate observed medicine. The maneuvers how traffic it A quite female police while strange witness way California, trying tried to observed take to spokesman park her their car she out extended attempted for from the a gap auszufahren parking quite strange in Montecito, montecito way reported how as reported revealed to go in from spokesman its to People way their from parking magazine for tried the parklücke space traffic by a in police type Montecito, from strange spokesman California right, speaker you to the the have Californian of magazine the seen traffic as a police `People witness. Highway of '.. californium Police. In The some witness The Zeitung, told witness told the in as and warehouse detail when stated they that Locklear tried around `People'. to a 4.30pm `after great The 16:30 female deal Ms. of witness clock 30 Locklear p.m., durchdrückte told witness "hit in the detail, accelerator the pedal detail that and Locklear of very a lot 'pressed of history violently, noise locklear made `after making by 16:30 after excessive trying 16: clock to that her noise car accelerator durchdrückte while the trying rear or and to front peddle take a lot of her of noise the car noise park out and did gap its when to progress from move, the she parking and was attempted when made with she parklücke to abrupt move went back their for car backwards, the and towards car forth maneuvers. moving took behind it backwards, a or few While forward times they in from their had reversing, they Sunglass'. parking times she passed of Somewhat space, their several and later when it the sonnenbrille times went witness in backwards, front saw ' of. The his the she sunglasses." first beginning pulled was of itself the probably Shortly actress together after, recognised probably the unites witness, Male had that not locklear who, at their in sunglasses'. recognized a nearby first time, road apparently that A and little Locklear anhielt, later had the on had not a female nearby not, recognized with witness road the the and witness actress, that stopped did the probably saw the car Ms. car off Why was Heather Locklear arrested? She was arrested on suspicion of driving under the influence of drugs. Correct Answers Reference 94% Google 80% RBMT5 77% Geneva 63% Driving under the influence Driving while medicated DUI JHU - Tromble 50% Driving while using drugs Medikamentes

41 HTER - costs to edit Reference translation The man was on assignment from the Ministry of Defense when he left two highly classified documents on a train to Waterloo. Machine translation The man was seconded by the Ministry of Defense when he was two extremely confidential documents in a train to Waterloo lost. Edited machine translation The man was working seconded for by the Ministry of Defense when he was lost two extremely confidential documents in a train to Waterloo. lost.

42 Reading Comprehension of Machine Translation Jones et al (2005) - Measured translation quality by testing English speakers on a Defense Language Proficiency Test for Arabic Read the MT output, and assess how many questions were answered correctly Nice, intuitive gauge of how good MT quality actually is

43 Reading Read chapter 8 from the textbook

44 Announcements HW2 due on Thursday Feb 19th