Language Modeling. Introduc*on to N- grams
|
|
- Jonas Bond
- 7 years ago
- Views:
Transcription
1 Language Modeling Introduc*on to N- grams
2 Probabilis1c Language Models Today s goal: assign a probability to a sentence Machine Transla*on: Why? P(high winds tonite) > P(large winds tonite) Spell Correc*on The office is about fiieen minuets from my house P(about fiieen minutes from) > P(about fiieen minuets from) Speech Recogni*on P(I saw a van) >> P(eyes awe of an) + Summariza*on, ques*on- answering, etc., etc.!!
3 Probabilis1c Language Modeling Goal: compute the probability of a sentence or sequence of words: P(W) = P(w 1,w 2,w 3,w 4,w 5 w n ) Related task: probability of an upcoming word: P(w 5 w 1,w 2,w 3,w 4 ) A model that computes either of these: P(W) or P(w n w 1,w 2 w n- 1 ) is called a language model. Be_er: the grammar But language model or LM is standard
4 How to compute P(W) How to compute this joint probability: P(its, water, is, so, transparent, that) Intui*on: let s rely on the Chain Rule of Probability
5 Reminder: The Chain Rule Recall the defini*on of condi*onal probabili*es Rewri*ng: More variables: P(A,B,C,D) = P(A)P(B A)P(C A,B)P(D A,B,C) The Chain Rule in General P(x 1,x 2,x 3,,x n ) = P(x 1 )P(x 2 x 1 )P(x 3 x 1,x 2 ) P(x n x 1,,x n- 1 )
6 The Chain Rule applied to compute joint probability of words in sentence # i P(w 1 w 2 w n ) = P(w i w 1 w 2 w i"1 )! P( its water is so transparent ) = P(its) P(water its) P(is its water) P(so its water is) P(transparent its water is so)
7 How to es1mate these probabili1es Could we just count and divide? P(the its water is so transparent that) = Count(its water is so transparent that the) Count(its water is so transparent that) No! Too many possible sentences! We ll never see enough data for es*ma*ng these
8 Markov Assump1on Simplifying assump*on: Andrei Markov P(the its water is so transparent that) " P(the that) Or maybe P(the its water is so transparent that) " P(the transparent that)
9 Markov Assump1on $ i P(w 1 w 2 w n ) " P(w i w i#k w i#1 ) In other words, we approximate each component in the product P(w i w 1 w 2 w i"1 ) # P(w i w i"k w i"1 )
10 Simplest case: Unigram model # i P(w 1 w 2 w n ) " P(w i ) Some automa*cally generated sentences from a unigram model! fifth, an, of, futures, the, an, incorporated, a, a, the, inflation, most, dollars, quarter, in, is, mass!! thrift, did, eighty, said, hard, 'm, july, bullish!! that, or, limited, the!
11 Bigram model " Condi*on on the previous word: P(w i w 1 w 2 w i"1 ) # P(w i w i"1 ) texaco, rose, one, in, this, issue, is, pursuing, growth, in, a, boiler, house, said, mr., gurria, mexico, 's, motion, control, proposal, without, permission, from, five, hundred, fifty, five, yen!! outside, new, car, parking, lot, of, the, agreement, reached!! this, would, be, a, record, november!
12 N- gram models We can extend to trigrams, 4- grams, 5- grams In general this is an insufficient model of language because language has long- distance dependencies: The computer which I had just put into the machine room on the fiih floor crashed. But we can oien get away with N- gram models
13 Language Modeling Introduc*on to N- grams
14 Language Modeling Es*ma*ng N- gram Probabili*es
15 Es1ma1ng bigram probabili1es The Maximum Likelihood Es*mate P(w i w i"1 ) = count(w i"1,w i ) count(w i"1 )! P(w i w i"1 ) = c(w i"1,w i ) c(w i"1 )
16 An example P(w i w i"1 ) = c(w i"1,w i ) c(w i"1 ) <s> I am Sam </s> <s> Sam I am </s> <s> I do not like green eggs and ham </s>
17 More examples: Berkeley Restaurant Project sentences can you tell me about any good cantonese restaurants close by mid priced thai food is what i m looking for tell me about chez panisse can you give me a lis*ng of the kinds of food that are available i m looking for a good place to eat breakfast when is caffe venezia open during the day
18 Raw bigram counts Out of 9222 sentences
19 Raw bigram probabili1es Normalize by unigrams: Result:
20 Bigram es1mates of sentence probabili1es P(<s> I want english food </s>) = P(I <s>) P(want I) P(english want) P(food english) P(</s> food) =
21 What kinds of knowledge? P(english want) =.0011 P(chinese want) =.0065 P(to want) =.66 P(eat to) =.28 P(food to) = 0 P(want spend) = 0 P (i <s>) =.25
22 Prac1cal Issues We do everything in log space Avoid underflow (also adding is faster than mul*plying) log(p 1! p 2! p 3! p 4 ) = log p 1 + log p 2 + log p 3 + log p 4
23 SRILM Language Modeling Toolkits h_p://
24 Google N- Gram Release, August 2006
25 Google N- Gram Release serve as the incoming 92! serve as the incubator 99! serve as the independent 794! serve as the index 223! serve as the indication 72! serve as the indicator 120! serve as the indicators 45! serve as the indispensable 111! serve as the indispensible 40! serve as the individual 234!
26 Google Book N- grams h_p://ngrams.googlelabs.com/
27 Language Modeling Es*ma*ng N- gram Probabili*es
28 Language Modeling Evalua*on and Perplexity
29 Evalua1on: How good is our model? Does our language model prefer good sentences to bad ones? Assign higher probability to real or frequently observed sentences Than ungramma*cal or rarely observed sentences? We train parameters of our model on a training set. We test the model s performance on data we haven t seen. A test set is an unseen dataset that is different from our training set, totally unused. An evalua1on metric tells us how well our model does on the test set.
30 Extrinsic evalua1on of N- gram models Best evalua*on for comparing models A and B Put each model in a task spelling corrector, speech recognizer, MT system Run the task, get an accuracy for A and for B How many misspelled words corrected properly How many words translated correctly Compare accuracy for A and B
31 Difficulty of extrinsic (in- vivo) evalua1on of N- gram models Extrinsic evalua*on Time- consuming; can take days or weeks So Some*mes use intrinsic evalua*on: perplexity Bad approxima*on unless the test data looks just like the training data So generally only useful in pilot experiments But is helpful to think about.
32 Intui1on of Perplexity The Shannon Game: How well can we predict the next word? I always order pizza with cheese and The 33 rd President of the US was I saw a Unigrams are terrible at this game. (Why?) A be_er model of a text mushrooms 0.1 pepperoni 0.1 anchovies 0.01 is one which assigns a higher probability to the word that actually occurs. fried rice and 1e-100
33 Perplexity The best language model is one that best predicts an unseen test set Gives the highest P(sentence) Perplexity is the inverse probability of the test set, normalized by the number of words: PP(W ) = P(w 1 w 2...w N )! 1 N = N 1 P(w 1 w 2...w N ) Chain rule: For bigrams: Minimizing perplexity is the same as maximizing probability
34 The Shannon Game intui1on for perplexity From Josh Goodman How hard is the task of recognizing digits 0,1,2,3,4,5,6,7,8,9 Perplexity 10 How hard is recognizing (30,000) names at MicrosoI. Perplexity = 30,000 If a system has to recognize Operator (1 in 4) Sales (1 in 4) Technical Support (1 in 4) 30,000 names (1 in 120,000 each) Perplexity is 53 Perplexity is weighted equivalent branching factor
35 Perplexity as branching factor Let s suppose a sentence consis*ng of random digits What is the perplexity of this sentence according to a model that assign P=1/10 to each digit?
36 Lower perplexity = bexer model Training 38 million words, test 1.5 million words, WSJ N- gram Order Unigram Bigram Trigram Perplexity
37 Language Modeling Evalua*on and Perplexity
38 Language Modeling Generaliza*on and zeros
39 The Shannon Visualiza1on Method Choose a random bigram (<s>, w) according to its probability Now choose a random bigram (w, x) according to its probability And so on un*l we choose </s> Then string the words together <s> I! I want! want to! to eat! eat Chinese! Chinese food! food </s>! I want to eat Chinese food!
40 Approxima1ng Shakespeare
41 Shakespeare as corpus N=884,647 tokens, V=29,066 Shakespeare produced 300,000 bigram types out of V 2 = 844 million possible bigrams. So 99.96% of the possible bigrams were never seen (have zero entries in the table) Quadrigrams worse: What's coming out looks like Shakespeare because it is Shakespeare
42 The wall street journal is not shakespeare (no offense)
43 The perils of overfi]ng N- grams only work well for word predic*on if the test corpus looks like the training corpus In real life, it oien doesn t We need to train robust models that generalize! One kind of generaliza*on: Zeros! Things that don t ever occur in the training set But occur in the test set
44 Zeros Training set: denied the allega*ons denied the reports denied the claims denied the request Test set denied the offer denied the loan P( offer denied the) = 0
45 Zero probability bigrams Bigrams with zero probability mean that we will assign 0 probability to the test set! And hence we cannot compute perplexity (can t divide by 0)!
46 Language Modeling Generaliza*on and zeros
47 Language Modeling Smoothing: Add- one (Laplace) smoothing
48 The intuition of smoothing (from Dan Klein) When we have sparse sta*s*cs: P(w denied the) 3 allega*ons 2 reports 1 claims 1 request 7 total allegations reports claims request attack man outcome Steal probability mass to generalize be_er P(w denied the) 2.5 allega*ons 1.5 reports 0.5 claims 0.5 request 2 other 7 total allegations reports claims request attack man outcome
49 Add- one es1ma1on Also called Laplace smoothing Pretend we saw each word one more *me than we did Just add one to all the counts! MLE es*mate: P MLE (w i w i!1 ) = c(w i!1, w i ) c(w i!1 ) Add- 1 es*mate: P Add!1 (w i w i!1 ) = c(w i!1, w i )+1 c(w i!1 )+V
50 Maximum Likelihood Es1mates The maximum likelihood es*mate of some parameter of a model M from a training set T maximizes the likelihood of the training set T given the model M Suppose the word bagel occurs 400 *mes in a corpus of a million words What is the probability that a random word from some other text will be bagel? MLE es*mate is 400/1,000,000 =.0004 This may be a bad es*mate for some other corpus But it is the es1mate that makes it most likely that bagel will occur 400 *mes in a million word corpus.
51 Berkeley Restaurant Corpus: Laplace smoothed bigram counts
52 Laplace-smoothed bigrams
53 Reconstituted counts
54 Compare with raw bigram counts
55 Add- 1 es1ma1on is a blunt instrument So add- 1 isn t used for N- grams: We ll see be_er methods But add- 1 is used to smooth other NLP models For text classifica*on In domains where the number of zeros isn t so huge.
56 Language Modeling Smoothing: Add- one (Laplace) smoothing
57 Language Modeling Interpola*on, Backoff, and Web- Scale LMs
58 Backoff and Interpolation Some*mes it helps to use less context Condi*on on less context for contexts you haven t learned much about Backoff: use trigram if you have good evidence, otherwise bigram, otherwise unigram Interpola1on: mix unigram, bigram, trigram Interpola*on works be_er
59 Linear Interpola1on Simple interpola*on Lambdas condi*onal on context:
60 How to set the lambdas? Use a held- out corpus Training Data Held- Out Data Test Data Choose λs to maximize the probability of held- out data: Fix the N- gram probabili*es (on the training data) Then search for λs that give largest probability to held- out set: " i log P(w 1...w n M(! 1...! k )) = log P M (!1...! k )(w i w i!1 )
61 Unknown words: Open versus closed vocabulary tasks If we know all the words in advanced Vocabulary V is fixed Closed vocabulary task OIen we don t know this Out Of Vocabulary = OOV words Open vocabulary task Instead: create an unknown word token <UNK> Training of <UNK> probabili*es Create a fixed lexicon L of size V At text normaliza*on phase, any training word not in L changed to <UNK> Now we train its probabili*es like a normal word At decoding *me If text input: Use UNK probabili*es for any word not in training
62 Huge web- scale n- grams How to deal with, e.g., Google N- gram corpus Pruning Only store N- grams with count > threshold. Remove singletons of higher- order n- grams Entropy- based pruning Efficiency Efficient data structures like tries Bloom filters: approximate language models Store words as indexes, not strings Use Huffman coding to fit large numbers of words into two bytes Quan*ze probabili*es (4-8 bits instead of 8- byte float)
63 Smoothing for Web- scale N- grams Stupid backoff (Brants et al. 2007) No discoun*ng, just use rela*ve frequencies i!1 S(w i w i!k+1 ) = " $ # $ % $ i count(w i!k+1 i!1 count(w i!k+1 ) ) i!1 0.4S(w i w i!k+2 i if count(w i!k+1 ) > 0 ) otherwise 63 S(w i ) = count(w i ) N
64 N- gram Smoothing Summary Add- 1 smoothing: OK for text categoriza*on, not for language modeling The most commonly used method: Extended Interpolated Kneser- Ney For very large N- grams like the Web: Stupid backoff 64
65 Advanced Language Modeling Discrimina*ve models: choose n- gram weights to improve a task, not to fit the training set Parsing- based models Caching Models Recently used words are more likely to appear P CACHE (w history) =!P(w i w i!2 w i!1 )+ (1!!) c(w " history) history These perform very poorly for speech recogni*on (why?)
66 Language Modeling Interpola*on, Backoff, and Web- Scale LMs
67 Language Modeling Advanced: Good Turing Smoothing
68 Reminder: Add-1 (Laplace) Smoothing P Add!1 (w i w i!1 ) = c(w i!1, w i )+1 c(w i!1 )+V
69 More general formulations: Add-k P Add!k (w i w i!1 ) = c(w i!1, w i )+ k c(w i!1 )+ kv P Add!k (w i w i!1 ) = c(w i!1, w i )+ m( 1 V ) c(w i!1 )+ m
70 Unigram prior smoothing P Add!k (w i w i!1 ) = c(w i!1, w i )+ m( 1 V ) c(w i!1 )+ m P UnigramPrior (w i w i!1 ) = c(w i!1, w i )+ mp(w i ) c(w i!1 )+ m
71 Advanced smoothing algorithms Intui*on used by many smoothing algorithms Good- Turing Kneser- Ney Wi_en- Bell Use the count of things we ve seen once to help es*mate the count of things we ve never seen
72 Nota1on: N c = Frequency of frequency c N c = the count of things we ve seen c *mes Sam I am I am Sam I do not eat I 3! sam 2! am 2! do 1! not 1! eat 1! 72 N 1 = 3 N 2 = 2 N 3 = 1
73 Good-Turing smoothing intuition You are fishing (a scenario from Josh Goodman), and caught: 10 carp, 3 perch, 2 whitefish, 1 trout, 1 salmon, 1 eel = 18 fish How likely is it that next species is trout? 1/18 How likely is it that next species is new (i.e. caish or bass) Let s use our es*mate of things- we- saw- once to es*mate the new things. 3/18 (because N 1 =3) Assuming so, how likely is it that next species is trout? Must be less than 1/18 How to es*mate?
74 Good Turing calculations P GT * (things with zero frequency) = N 1 N c* = (c +1)N c+1 N c Unseen (bass or catfish) c = 0: MLE p = 0/18 = 0 Seen once (trout) c = 1 MLE p = 1/18 P * GT (unseen) = N 1 /N = 3/18 C*(trout) = 2 * N2/N1 = 2 * 1/3 = 2/3 P * GT (trout) = 2/3 / 18 = 1/27
75 Ney et al. s Good Turing Intuition H. Ney, U. Essen, and R. Kneser, On the es*ma*on of 'small' probabili*es by leaving- one- out. IEEE Trans. PAMI. 17:12, Held-out words: 75
76 Ney et al. Good Turing Intuition (slide from Dan Klein) Training Held out Intui*on from leave- one- out valida*on Take each of the c training words out in turn c training sets of size c 1, held- out of size 1 What frac*on of held- out words are unseen in training? N 1 /c What frac*on of held- out words are seen k *mes in training? (k+1)n k+1 /c So in the future we expect (k+1)n k+1 /c of the words to be those with training count k There are N k words with training count k Each should occur with probability: (k+1)n k+1 /c/n k or expected count: k* = (k +1)N k+1 N k N 1 N 2 N N 3511 N 4417 N 0 N 1 N N 3510 N 4416
77 Good-Turing complications (slide from Dan Klein) Problem: what about the? (say c=4417) For small k, N k > N k+1 For large k, too jumpy, zeros wreck es*mates N 1 N 2 N3 Simple Good- Turing [Gale and Sampson]: replace empirical N k with a best- fit power law once counts get unreliable N 1 N 2
78 Resulting Good-Turing numbers Numbers from Church and Gale (1991) 22 million words of AP Newswire c* = (c +1)N c+1 N c Count c Good Turing c*
79 Language Modeling Advanced: Good Turing Smoothing
80 Language Modeling Advanced: Kneser-Ney Smoothing
81 Resulting Good-Turing numbers Numbers from Church and Gale (1991) 22 million words of AP Newswire c* = (c +1)N c+1 N c It sure looks like c* = (c -.75) Count c Good Turing c*
82 Absolute Discounting Interpolation Save ourselves some *me and just subtract 0.75 (or some d)! P AbsoluteDiscounting (w i w i!1 ) = c(w i!1, w i )! d c(w i!1 ) (Maybe keeping a couple extra values of d for counts 1 and 2) But should we really just use the regular unigram P(w)? 82 discounted bigram Interpolation weight +!(w i!1 )P(w) unigram
83 Kneser-Ney Smoothing I Be_er es*mate for probabili*es of lower- order unigrams! Shannon game: I can t see without my reading? Francisco glasses Francisco is more common than glasses but Francisco always follows San The unigram is useful exactly when we haven t seen this bigram! Instead of P(w): How likely is w P con*nua*on (w): How likely is w to appear as a novel con*nua*on? For each word, count the number of bigram types it completes Every bigram type was a novel con*nua*on the first *me it was seen P CONTINUATION (w)! {w i"1 : c(w i"1, w) > 0}
84 Kneser-Ney Smoothing II How many *mes does w appear as a novel con*nua*on: P CONTINUATION (w)! {w i"1 : c(w i"1, w) > 0} Normalized by the total number of word bigram types {(w j!1, w j ) : c(w j!1, w j ) > 0} P CONTINUATION (w) = {w i!1 : c(w i!1, w) > 0} {(w j!1, w j ) : c(w j!1, w j ) > 0}
85 Kneser-Ney Smoothing III Alterna*ve metaphor: The number of # of word types seen to precede w {w i!1 : c(w i!1, w) > 0} normalized by the # of words preceding all words: P CONTINUATION (w) = " w' {w i!1 : c(w i!1, w) > 0} {w' i!1 : c(w' i!1, w') > 0} A frequent word (Francisco) occurring in only one context (San) will have a low con*nua*on probability
86 Kneser-Ney Smoothing IV P KN (w i w i!1 ) = max(c(w i!1, w i )! d, 0) c(w i!1 ) +!(w i!1 )P CONTINUATION (w i ) λ is a normalizing constant; the probability mass we ve discounted!(w i!1 ) = d c(w i!1 ) {w : c(w i!1, w) > 0} 86 the normalized discount The number of word types that can follow w i-1 = # of word types we discounted = # of times we applied normalized discount
87 Kneser-Ney Smoothing: Recursive formulation i!1 P KN (w i w i!n+1 ) = max(c i KN (w i!n+1 i!1 c KN (w i!n+1 )! d, 0) ) i!1 i!1 +!(w i!n+1 )P KN (w i w i!n+2 ) c KN ( ) =!# " $# count( ) for the highest order continuationcount( ) for lower order Continuation count = Number of unique single word contexts for 87
88 Language Modeling Advanced: Kneser-Ney Smoothing
CS 533: Natural Language. Word Prediction
CS 533: Natural Language Processing Lecture 03 N-Gram Models and Algorithms CS 533: Natural Language Processing Lecture 01 1 Word Prediction Suppose you read the following sequence of words: Sue swallowed
More informationSpeech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright c 2014. All rights reserved. Draft of September 1, 2014.
Speech and Language Processing. Daniel Jurafsky & James H. Martin. Copyright c 204. All rights reserved. Draft of September, 204. CHAPTER 4 N-Grams You are uniformly charming! cried he, with a smile of
More informationChapter 7. Language models. Statistical Machine Translation
Chapter 7 Language models Statistical Machine Translation Language models Language models answer the question: How likely is a string of English words good English? Help with reordering p lm (the house
More informationLanguage Modeling. Chapter 1. 1.1 Introduction
Chapter 1 Language Modeling (Course notes for NLP by Michael Collins, Columbia University) 1.1 Introduction In this chapter we will consider the the problem of constructing a language model from a set
More informationDRAFT N-GRAMS. ... all of a sudden I notice three guys standing on the sidewalk...
Speech and Language Processing: An introduction to speech recognition, computational linguistics and natural language processing. Daniel Jurafsky & James H. Martin. Copyright c 2006, All rights reserved.
More informationManaged N-gram Language Model Based on Hadoop Framework and a Hbase Tables
Managed N-gram Language Model Based on Hadoop Framework and a Hbase Tables Tahani Mahmoud Allam Assistance lecture in Computer and Automatic Control Dept - Faculty of Engineering-Tanta University, Tanta,
More informationEstimating Probability Distributions
Estimating Probability Distributions Readings: Manning and Schutze, Section 6.2 Jurafsky & Martin, Section 6.3 One of the central problems we face in using probability models for NLP is obtaining the actual
More informationTagging with Hidden Markov Models
Tagging with Hidden Markov Models Michael Collins 1 Tagging Problems In many NLP problems, we would like to model pairs of sequences. Part-of-speech (POS) tagging is perhaps the earliest, and most famous,
More informationBasic probability theory and n-gram models
theory and n-gram models INF4820 H2010 Institutt for Informatikk Universitetet i Oslo 28. september Outline 1 2 3 Outline 1 2 3 Gambling the beginning of probabilities Example Throwing a fair dice Example
More informationPerplexity Method on the N-gram Language Model Based on Hadoop Framework
94 International Arab Journal of e-technology, Vol. 4, No. 2, June 2015 Perplexity Method on the N-gram Language Model Based on Hadoop Framework Tahani Mahmoud Allam 1, Hatem Abdelkader 2 and Elsayed Sallam
More informationMicro blogs Oriented Word Segmentation System
Micro blogs Oriented Word Segmentation System Yijia Liu, Meishan Zhang, Wanxiang Che, Ting Liu, Yihe Deng Research Center for Social Computing and Information Retrieval Harbin Institute of Technology,
More informationLinear Programming Notes VII Sensitivity Analysis
Linear Programming Notes VII Sensitivity Analysis 1 Introduction When you use a mathematical model to describe reality you must make approximations. The world is more complicated than the kinds of optimization
More informationPart III: Machine Learning. CS 188: Artificial Intelligence. Machine Learning This Set of Slides. Parameter Estimation. Estimation: Smoothing
CS 188: Artificial Intelligence Lecture 20: Dynamic Bayes Nets, Naïve Bayes Pieter Abbeel UC Berkeley Slides adapted from Dan Klein. Part III: Machine Learning Up until now: how to reason in a model and
More informationOPTIMIZATION OF NEURAL NETWORK LANGUAGE MODELS FOR KEYWORD SEARCH. Ankur Gandhe, Florian Metze, Alex Waibel, Ian Lane
OPTIMIZATION OF NEURAL NETWORK LANGUAGE MODELS FOR KEYWORD SEARCH Ankur Gandhe, Florian Metze, Alex Waibel, Ian Lane Carnegie Mellon University Language Technology Institute {ankurgan,fmetze,ahw,lane}@cs.cmu.edu
More informationA Mixed Trigrams Approach for Context Sensitive Spell Checking
A Mixed Trigrams Approach for Context Sensitive Spell Checking Davide Fossati and Barbara Di Eugenio Department of Computer Science University of Illinois at Chicago Chicago, IL, USA dfossa1@uic.edu, bdieugen@cs.uic.edu
More informationPa8ern Recogni6on. and Machine Learning. Chapter 4: Linear Models for Classifica6on
Pa8ern Recogni6on and Machine Learning Chapter 4: Linear Models for Classifica6on Represen'ng the target values for classifica'on If there are only two classes, we typically use a single real valued output
More informationUnit 1 Number Sense. In this unit, students will study repeating decimals, percents, fractions, decimals, and proportions.
Unit 1 Number Sense In this unit, students will study repeating decimals, percents, fractions, decimals, and proportions. BLM Three Types of Percent Problems (p L-34) is a summary BLM for the material
More informationSimple Language Models for Spam Detection
Simple Language Models for Spam Detection Egidio Terra Faculty of Informatics PUC/RS - Brazil Abstract For this year s Spam track we used classifiers based on language models. These models are used to
More information1 Maximum likelihood estimation
COS 424: Interacting with Data Lecturer: David Blei Lecture #4 Scribes: Wei Ho, Michael Ye February 14, 2008 1 Maximum likelihood estimation 1.1 MLE of a Bernoulli random variable (coin flips) Given N
More informationCAs and Turing Machines. The Basis for Universal Computation
CAs and Turing Machines The Basis for Universal Computation What We Mean By Universal When we claim universal computation we mean that the CA is capable of calculating anything that could possibly be calculated*.
More informationText Classification and Naïve Bayes. The Task of Text Classifica1on
Text Classification and Naïve Bayes The Task of Text Classifica1on Is this spam? Who wrote which Federalist papers? 1787-8: anonymous essays try to convince New York to ra1fy U.S Cons1tu1on: Jay, Madison,
More informationWord Completion and Prediction in Hebrew
Experiments with Language Models for בס"ד Word Completion and Prediction in Hebrew 1 Yaakov HaCohen-Kerner, Asaf Applebaum, Jacob Bitterman Department of Computer Science Jerusalem College of Technology
More informationLog-Linear Models. Michael Collins
Log-Linear Models Michael Collins 1 Introduction This note describes log-linear models, which are very widely used in natural language processing. A key advantage of log-linear models is their flexibility:
More informationScaling Shrinkage-Based Language Models
Scaling Shrinkage-Based Language Models Stanley F. Chen, Lidia Mangu, Bhuvana Ramabhadran, Ruhi Sarikaya, Abhinav Sethy IBM T.J. Watson Research Center P.O. Box 218, Yorktown Heights, NY 10598 USA {stanchen,mangu,bhuvana,sarikaya,asethy}@us.ibm.com
More informationAttribution. Modified from Stuart Russell s slides (Berkeley) Parts of the slides are inspired by Dan Klein s lecture material for CS 188 (Berkeley)
Machine Learning 1 Attribution Modified from Stuart Russell s slides (Berkeley) Parts of the slides are inspired by Dan Klein s lecture material for CS 188 (Berkeley) 2 Outline Inductive learning Decision
More informationPre-Algebra Lecture 6
Pre-Algebra Lecture 6 Today we will discuss Decimals and Percentages. Outline: 1. Decimals 2. Ordering Decimals 3. Rounding Decimals 4. Adding and subtracting Decimals 5. Multiplying and Dividing Decimals
More informationSignificant Figures, Propagation of Error, Graphs and Graphing
Chapter Two Significant Figures, Propagation of Error, Graphs and Graphing Every measurement has an error associated with it. If you were to put an object on a balance and weight it several times you will
More informationGrammars and introduction to machine learning. Computers Playing Jeopardy! Course Stony Brook University
Grammars and introduction to machine learning Computers Playing Jeopardy! Course Stony Brook University Last class: grammars and parsing in Prolog Noun -> roller Verb thrills VP Verb NP S NP VP NP S VP
More informationMachine Learning. CS 188: Artificial Intelligence Naïve Bayes. Example: Digit Recognition. Other Classification Tasks
CS 188: Artificial Intelligence Naïve Bayes Machine Learning Up until now: how use a model to make optimal decisions Machine learning: how to acquire a model from data / experience Learning parameters
More informationOffline Recognition of Unconstrained Handwritten Texts Using HMMs and Statistical Language Models. Alessandro Vinciarelli, Samy Bengio and Horst Bunke
1 Offline Recognition of Unconstrained Handwritten Texts Using HMMs and Statistical Language Models Alessandro Vinciarelli, Samy Bengio and Horst Bunke Abstract This paper presents a system for the offline
More informationUnderstanding Options: Calls and Puts
2 Understanding Options: Calls and Puts Important: in their simplest forms, options trades sound like, and are, very high risk investments. If reading about options makes you think they are too risky for
More informationECBDL 14: Evolu/onary Computa/on for Big Data and Big Learning Workshop July 13 th, 2014 Big Data Compe//on
ECBDL 14: Evolu/onary Computa/on for Big Data and Big Learning Workshop July 13 th, 2014 Big Data Compe//on Jaume Bacardit jaume.bacardit@ncl.ac.uk The Interdisciplinary Compu/ng and Complex BioSystems
More informationANALYTICAL TECHNIQUES FOR DATA VISUALIZATION
ANALYTICAL TECHNIQUES FOR DATA VISUALIZATION CSE 537 Ar@ficial Intelligence Professor Anita Wasilewska GROUP 2 TEAM MEMBERS: SAEED BOOR BOOR - 110564337 SHIH- YU TSAI - 110385129 HAN LI 110168054 SOURCES
More informationCSE 473: Artificial Intelligence Autumn 2010
CSE 473: Artificial Intelligence Autumn 2010 Machine Learning: Naive Bayes and Perceptron Luke Zettlemoyer Many slides over the course adapted from Dan Klein. 1 Outline Learning: Naive Bayes and Perceptron
More informationEnsemble Methods. Adapted from slides by Todd Holloway h8p://abeau<fulwww.com/2007/11/23/ ensemble- machine- learning- tutorial/
Ensemble Methods Adapted from slides by Todd Holloway h8p://abeau
More informationIdentifying Focus, Techniques and Domain of Scientific Papers
Identifying Focus, Techniques and Domain of Scientific Papers Sonal Gupta Department of Computer Science Stanford University Stanford, CA 94305 sonal@cs.stanford.edu Christopher D. Manning Department of
More informationLog-Linear Models a.k.a. Logistic Regression, Maximum Entropy Models
Log-Linear Models a.k.a. Logistic Regression, Maximum Entropy Models Natural Language Processing CS 6120 Spring 2014 Northeastern University David Smith (some slides from Jason Eisner and Dan Klein) summary
More informationIntroduction to Bayesian Classification (A Practical Discussion) Todd Holloway Lecture for B551 Nov. 27, 2007
Introduction to Bayesian Classification (A Practical Discussion) Todd Holloway Lecture for B551 Nov. 27, 2007 Naïve Bayes Components ML vs. MAP Benefits Feature Preparation Filtering Decay Extended Examples
More information+ = has become. has become. Maths in School. Fraction Calculations in School. by Kate Robinson
+ has become 0 Maths in School has become 0 Fraction Calculations in School by Kate Robinson Fractions Calculations in School Contents Introduction p. Simplifying fractions (cancelling down) p. Adding
More informationBBC LEARNING ENGLISH 6 Minute Grammar Conditionals review
BBC LEARNING ENGLISH 6 Minute Grammar Conditionals review This is not a word-for-word transcript Hello and welcome to 6 Minute Grammar with me, And me, So, what is our topic today? I don t know. Haven
More information3. Mathematical Induction
3. MATHEMATICAL INDUCTION 83 3. Mathematical Induction 3.1. First Principle of Mathematical Induction. Let P (n) be a predicate with domain of discourse (over) the natural numbers N = {0, 1,,...}. If (1)
More information10 Proofreading Tips for Error-Free Writing
October 2013 10 Proofreading Tips for Error-Free Writing 2013 Administrative Assistant Resource, a division of Lorman Business Center. All Rights Reserved. 10 Proofreading Tips for Error-Free Writing,
More informationLarge Language Models in Machine Translation
Large Language Models in Machine Translation Thorsten Brants Ashok C. Popat Peng Xu Franz J. Och Jeffrey Dean Google, Inc. 1600 Amphitheatre Parkway Mountain View, CA 94303, USA {brants,popat,xp,och,jeff}@google.com
More informationOA4-13 Rounding on a Number Line Pages 80 81
OA4-13 Rounding on a Number Line Pages 80 81 STANDARDS 3.NBT.A.1, 4.NBT.A.3 Goals Students will round to the closest ten, except when the number is exactly halfway between a multiple of ten. PRIOR KNOWLEDGE
More information6 3 4 9 = 6 10 + 3 10 + 4 10 + 9 10
Lesson The Binary Number System. Why Binary? The number system that you are familiar with, that you use every day, is the decimal number system, also commonly referred to as the base- system. When you
More informationApplying Machine Learning to Network Security Monitoring. Alex Pinto Chief Data Scien2st MLSec Project @alexcpsec @MLSecProject!
Applying Machine Learning to Network Security Monitoring Alex Pinto Chief Data Scien2st MLSec Project @alexcpsec @MLSecProject! whoami Almost 15 years in Informa2on Security, done a licle bit of everything.
More informationLecture 10: Regression Trees
Lecture 10: Regression Trees 36-350: Data Mining October 11, 2006 Reading: Textbook, sections 5.2 and 10.5. The next three lectures are going to be about a particular kind of nonlinear predictive model,
More informationECON 459 Game Theory. Lecture Notes Auctions. Luca Anderlini Spring 2015
ECON 459 Game Theory Lecture Notes Auctions Luca Anderlini Spring 2015 These notes have been used before. If you can still spot any errors or have any suggestions for improvement, please let me know. 1
More informationChapter 4 Online Appendix: The Mathematics of Utility Functions
Chapter 4 Online Appendix: The Mathematics of Utility Functions We saw in the text that utility functions and indifference curves are different ways to represent a consumer s preferences. Calculus can
More informationThe equivalence of logistic regression and maximum entropy models
The equivalence of logistic regression and maximum entropy models John Mount September 23, 20 Abstract As our colleague so aptly demonstrated ( http://www.win-vector.com/blog/20/09/the-simplerderivation-of-logistic-regression/
More informationDecimal Notations for Fractions Number and Operations Fractions /4.NF
Decimal Notations for Fractions Number and Operations Fractions /4.NF Domain: Cluster: Standard: 4.NF Number and Operations Fractions Understand decimal notation for fractions, and compare decimal fractions.
More informationReading 13 : Finite State Automata and Regular Expressions
CS/Math 24: Introduction to Discrete Mathematics Fall 25 Reading 3 : Finite State Automata and Regular Expressions Instructors: Beck Hasti, Gautam Prakriya In this reading we study a mathematical model
More informationWe can express this in decimal notation (in contrast to the underline notation we have been using) as follows: 9081 + 900b + 90c = 9001 + 100c + 10b
In this session, we ll learn how to solve problems related to place value. This is one of the fundamental concepts in arithmetic, something every elementary and middle school mathematics teacher should
More informationLanguage Model Adaptation for Video Lecture Transcription
UNIVERSITAT POLITÈCNICA DE VALÈNCIA DEPARTAMENT DE SISTEMES INFORMÀTICS I COMPUTACIÓ Language Model Adaptation for Video Lecture Transcription Master Thesis - MIARFID Adrià A. Martínez Villaronga Directors:
More informationIntroduction to Learning & Decision Trees
Artificial Intelligence: Representation and Problem Solving 5-38 April 0, 2007 Introduction to Learning & Decision Trees Learning and Decision Trees to learning What is learning? - more than just memorizing
More informationNLP Lab Session Week 3 Bigram Frequencies and Mutual Information Scores in NLTK September 16, 2015
NLP Lab Session Week 3 Bigram Frequencies and Mutual Information Scores in NLTK September 16, 2015 Starting a Python and an NLTK Session Open a Python 2.7 IDLE (Python GUI) window or a Python interpreter
More informationAll the examples in this worksheet and all the answers to questions are available as answer sheets or videos.
BIRKBECK MATHS SUPPORT www.mathsupport.wordpress.com Numbers 3 In this section we will look at - improper fractions and mixed fractions - multiplying and dividing fractions - what decimals mean and exponents
More informationStrategies for Training Large Scale Neural Network Language Models
Strategies for Training Large Scale Neural Network Language Models Tomáš Mikolov #1, Anoop Deoras 2, Daniel Povey 3, Lukáš Burget #4, Jan Honza Černocký #5 # Brno University of Technology, Speech@FIT,
More informationBayesian Networks. Read R&N Ch. 14.1-14.2. Next lecture: Read R&N 18.1-18.4
Bayesian Networks Read R&N Ch. 14.1-14.2 Next lecture: Read R&N 18.1-18.4 You will be expected to know Basic concepts and vocabulary of Bayesian networks. Nodes represent random variables. Directed arcs
More informationLinear Threshold Units
Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear
More informationCHAPTER 5 Round-off errors
CHAPTER 5 Round-off errors In the two previous chapters we have seen how numbers can be represented in the binary numeral system and how this is the basis for representing numbers in computers. Since any
More informationTopic Extrac,on from Online Reviews for Classifica,on and Recommenda,on (2013) R. Dong, M. Schaal, M. P. O Mahony, B. Smyth
Topic Extrac,on from Online Reviews for Classifica,on and Recommenda,on (2013) R. Dong, M. Schaal, M. P. O Mahony, B. Smyth Lecture Algorithms to Analyze Big Data Speaker Hüseyin Dagaydin Heidelberg, 27
More informationChristmas Theme: The Greatest Gift
Christmas Theme: The Greatest Gift OVERVIEW Key Point: Jesus is the greatest gift of all. Bible Story: The wise men brought gifts Bible Reference: Matthew 2:1-2 Challenge Verse: And we have seen and testify
More informationOn Attacking Statistical Spam Filters
On Attacking Statistical Spam Filters Gregory L. Wittel and S. Felix Wu Department of Computer Science University of California, Davis One Shields Avenue, Davis, CA 95616 USA Paper review by Deepak Chinavle
More informationThe State of Split Testing Survey Analysis
The State of Split Testing Survey Analysis Analysed by Parry Malm CEO, Phrasee parry@phrasee.co.uk www.phrasee.co.uk 1 Welcome to the State of Split Testing Email subject line split testing is nothing
More informationAutomatic slide assignation for language model adaptation
Automatic slide assignation for language model adaptation Applications of Computational Linguistics Adrià Agustí Martínez Villaronga May 23, 2013 1 Introduction Online multimedia repositories are rapidly
More informationReliable and Cost-Effective PoS-Tagging
Reliable and Cost-Effective PoS-Tagging Yu-Fang Tsai Keh-Jiann Chen Institute of Information Science, Academia Sinica Nanang, Taipei, Taiwan 5 eddie,chen@iis.sinica.edu.tw Abstract In order to achieve
More informationHealth Care Vocabulary Lesson
Hello. This is AJ Hoge again. Welcome to the vocabulary lesson for Health Care. Let s start. * * * * * At the beginning of the conversation Joe and Kristin talk about a friend, Joe s friend, whose name
More informationNodes, Ties and Influence
Nodes, Ties and Influence Chapter 2 Chapter 2, Community Detec:on and Mining in Social Media. Lei Tang and Huan Liu, Morgan & Claypool, September, 2010. 1 IMPORTANCE OF NODES 2 Importance of Nodes Not
More informationDouble Deck Blackjack
Double Deck Blackjack Double Deck Blackjack is far more volatile than Multi Deck for the following reasons; The Running & True Card Counts can swing drastically from one Round to the next So few cards
More informationDecision Making under Uncertainty
6.825 Techniques in Artificial Intelligence Decision Making under Uncertainty How to make one decision in the face of uncertainty Lecture 19 1 In the next two lectures, we ll look at the question of how
More informationThis Unit: Floating Point Arithmetic. CIS 371 Computer Organization and Design. Readings. Floating Point (FP) Numbers
This Unit: Floating Point Arithmetic CIS 371 Computer Organization and Design Unit 7: Floating Point App App App System software Mem CPU I/O Formats Precision and range IEEE 754 standard Operations Addition
More informationA Sales Strategy to Increase Function Bookings
A Sales Strategy to Increase Function Bookings It s Time to Start Selling Again! It s time to take on a sales oriented focus for the bowling business. Why? Most bowling centres have lost the art and the
More informationEstimating Language Models Using Hadoop and Hbase
Estimating Language Models Using Hadoop and Hbase Xiaoyang Yu E H U N I V E R S I T Y T O H F G R E D I N B U Master of Science Artificial Intelligence School of Informatics University of Edinburgh 2008
More informationQA or the Highway 2016 Presentation Notes
QA or the Highway 2016 Presentation Notes Making QA Strategic Let s Get Real (Panel Discussion) Does testing belong at the strategic table? What is that strategic value that testing provides? Conquering
More informationA STATE-SPACE METHOD FOR LANGUAGE MODELING. Vesa Siivola and Antti Honkela. Helsinki University of Technology Neural Networks Research Centre
A STATE-SPACE METHOD FOR LANGUAGE MODELING Vesa Siivola and Antti Honkela Helsinki University of Technology Neural Networks Research Centre Vesa.Siivola@hut.fi, Antti.Honkela@hut.fi ABSTRACT In this paper,
More informationSolving Linear Equations in One Variable. Worked Examples
Solving Linear Equations in One Variable Worked Examples Solve the equation 30 x 1 22x Solve the equation 30 x 1 22x Our goal is to isolate the x on one side. We ll do that by adding (or subtracting) quantities
More informationPredicting Flight Delays
Predicting Flight Delays Dieterich Lawson jdlawson@stanford.edu William Castillo will.castillo@stanford.edu Introduction Every year approximately 20% of airline flights are delayed or cancelled, costing
More informationHomework 2. Page 154: Exercise 8.10. Page 145: Exercise 8.3 Page 150: Exercise 8.9
Homework 2 Page 110: Exercise 6.10; Exercise 6.12 Page 116: Exercise 6.15; Exercise 6.17 Page 121: Exercise 6.19 Page 122: Exercise 6.20; Exercise 6.23; Exercise 6.24 Page 131: Exercise 7.3; Exercise 7.5;
More informationCOMP3420: Advanced Databases and Data Mining. Classification and prediction: Introduction and Decision Tree Induction
COMP3420: Advanced Databases and Data Mining Classification and prediction: Introduction and Decision Tree Induction Lecture outline Classification versus prediction Classification A two step process Supervised
More informationFrequently Asked Questions about New Leaf s National Accounts Program
Frequently Asked Questions about New Leaf s National Accounts Program What if I am already selling to some of these national accounts? Simple. Just let us know that you do not need us to present to these
More informationEasy Casino Profits. Congratulations!!
Easy Casino Profits The Easy Way To Beat The Online Casinos Everytime! www.easycasinoprofits.com Disclaimer The authors of this ebook do not promote illegal, underage gambling or gambling to those living
More informationMath 4310 Handout - Quotient Vector Spaces
Math 4310 Handout - Quotient Vector Spaces Dan Collins The textbook defines a subspace of a vector space in Chapter 4, but it avoids ever discussing the notion of a quotient space. This is understandable
More informationUnit 7 The Number System: Multiplying and Dividing Integers
Unit 7 The Number System: Multiplying and Dividing Integers Introduction In this unit, students will multiply and divide integers, and multiply positive and negative fractions by integers. Students will
More informationLarge-Scale Test Mining
Large-Scale Test Mining SIAM Conference on Data Mining Text Mining 2010 Alan Ratner Northrop Grumman Information Systems NORTHROP GRUMMAN PRIVATE / PROPRIETARY LEVEL I Aim Identify topic and language/script/coding
More informationNumerical Matrix Analysis
Numerical Matrix Analysis Lecture Notes #10 Conditioning and / Peter Blomgren, blomgren.peter@gmail.com Department of Mathematics and Statistics Dynamical Systems Group Computational Sciences Research
More informationNgram Search Engine with Patterns Combining Token, POS, Chunk and NE Information
Ngram Search Engine with Patterns Combining Token, POS, Chunk and NE Information Satoshi Sekine Computer Science Department New York University sekine@cs.nyu.edu Kapil Dalwani Computer Science Department
More informationSpeech Recognition on Cell Broadband Engine UCRL-PRES-223890
Speech Recognition on Cell Broadband Engine UCRL-PRES-223890 Yang Liu, Holger Jones, John Johnson, Sheila Vaidya (Lawrence Livermore National Laboratory) Michael Perrone, Borivoj Tydlitat, Ashwini Nanda
More informationPart 1 Expressions, Equations, and Inequalities: Simplifying and Solving
Section 7 Algebraic Manipulations and Solving Part 1 Expressions, Equations, and Inequalities: Simplifying and Solving Before launching into the mathematics, let s take a moment to talk about the words
More informationPUSD High Frequency Word List
PUSD High Frequency Word List For Reading and Spelling Grades K-5 High Frequency or instant words are important because: 1. You can t read a sentence or a paragraph without knowing at least the most common.
More informationAnalysis of Compression Algorithms for Program Data
Analysis of Compression Algorithms for Program Data Matthew Simpson, Clemson University with Dr. Rajeev Barua and Surupa Biswas, University of Maryland 12 August 3 Abstract Insufficient available memory
More informationLevel 2 6.4 Lesson Plan Session 1
Session 1 Materials Materials provided: image of 3R symbol; 4 environment images; Word Map; homework puzzle. Suggested additional materials: examples of compostable and non-compostable waste, i.e., apple
More informationIBM Research Report. Scaling Shrinkage-Based Language Models
RC24970 (W1004-019) April 6, 2010 Computer Science IBM Research Report Scaling Shrinkage-Based Language Models Stanley F. Chen, Lidia Mangu, Bhuvana Ramabhadran, Ruhi Sarikaya, Abhinav Sethy IBM Research
More informationLecture 7: Continuous Random Variables
Lecture 7: Continuous Random Variables 21 September 2005 1 Our First Continuous Random Variable The back of the lecture hall is roughly 10 meters across. Suppose it were exactly 10 meters, and consider
More informationThe Purchase Price in M&A Deals
The Purchase Price in M&A Deals Question that came in the other day In an M&A deal, does the buyer pay the Equity Value or the Enterprise Value to acquire the seller? What does it mean in press releases
More informationEncoding Text with a Small Alphabet
Chapter 2 Encoding Text with a Small Alphabet Given the nature of the Internet, we can break the process of understanding how information is transmitted into two components. First, we have to figure out
More informationTelemarketing Selling Script for Mobile Websites
Telemarketing Selling Script for Mobile Websites INTRODUCTION - - - - - - - To person who answers phone - - - - - - - Record name of company, phone Good Morning (or Good Afternoon) I would like to speak
More informationUnit 6 Number and Operations in Base Ten: Decimals
Unit 6 Number and Operations in Base Ten: Decimals Introduction Students will extend the place value system to decimals. They will apply their understanding of models for decimals and decimal notation,
More informationSession 7 Fractions and Decimals
Key Terms in This Session Session 7 Fractions and Decimals Previously Introduced prime number rational numbers New in This Session period repeating decimal terminating decimal Introduction In this session,
More informationThe 5 P s in Problem Solving *prob lem: a source of perplexity, distress, or vexation. *solve: to find a solution, explanation, or answer for
The 5 P s in Problem Solving 1 How do other people solve problems? The 5 P s in Problem Solving *prob lem: a source of perplexity, distress, or vexation *solve: to find a solution, explanation, or answer
More information