Overview. Naive Bayes Classifiers. A Sample Data Set. Frequencies and Probabilities. Connectionist and Statistical Language Processing

Transcription

1 Overvew Nave Bayes Classfers Connectonst and Statstcal Language Processng Frank Keller Computerlngustk Unverstät des Saarlandes Sample data set wth frequences and probabltes Classfcaton based on Bayes rule Maxmum a posteror and maxmum lkelhood Propertes of Bayes classfers Nave Bayes classfers Parameter estmaton, propertes, example Dealng wth sparse data Applcaton: emal classfcaton Lterature: Wtten and Frank (2000: ch. 4), Mtchell (1997: ch. 6). Nave Bayes Classfers p.1/22 Nave Bayes Classfers p.2/22 A Sample Data Set Frequences and Probabltes Fctonal data set that descrbes the weather condtons for playng some unspecfed game. outlook temp. humdty wndy play sunny hot hgh false no sunny hot hgh true no overcast hot hgh false yes rany mld hgh false yes rany cool normal false yes rany cool normal true no overcast cool normal true yes outlook temp. humdty wndy play sunny mld hgh false no sunny cool normal false yes rany mld normal false yes sunny mld normal true yes overcast mld hgh true yes overcast hot normal false yes rany mld hgh true no Frequences and probabltes for the weather data: outlook temperature humdty wndy play yes no yes no yes no yes no yes no sunny 2 3 hot 2 2 hgh 3 4 false overcast 4 0 mld 4 2 normal 6 1 true 3 3 rany 3 2 cool 3 1 yes no yes no yes no yes no yes no sunny 2/9 3/5 hot 2/9 2/5 hgh 3/9 4/5 false 6/9 2/5 9/14 5/14 overcast 4/9 0/5 mld 4/9 2/5 normal 6/9 1/5 true 3/9 3/5 rany 3/9 2/5 cool 3/9 1/5 Nave Bayes Classfers p.3/22 Nave Bayes Classfers p.4/22

2 Classfyng an Unseen Example Classfyng an Unseen Example Now assume that we have to classfy the followng new nstance: outlook temp. humdty wndy play sunny cool hgh true? Key dea: compute a probablty for each class based on the probablty dstrbuton n the tranng data. Frst take nto account the the probablty of each attrbute. Treat all attrbutes equally mportant,.e., multply the probabltes: P(yes)=2/9 3/9 3/9 3/9 = P(no)=3/5 1/5 4/5 3/5 = Now take nto account the overall probablty of a gven class. Multply t wth the probabltes of the attrbutes: P(yes)= /14 = P(no)= /14 = Now choose the class so that t maxmzes ths probablty. Ths means that the new nstance wll be classfed as no. Nave Bayes Classfers p.5/22 Nave Bayes Classfers p.6/22 Bayes Rule Maxmum A Posteror Ths procedure s based on Bayes Rule, whch says: f you have a hypothess h and data D whch bears on the hypothess, then: (1) P(h D)= P(D h)p(h) P(D) P(h): ndependent probablty of h: pror probablty P(D): ndependent probablty of D P(D h): condtonal probablty of D gven h: lkelhood P(h D): cond. probablty of h gven D: posteror probablty Based on Bayes Rule, we can compute the maxmum a posteror hypothess for the data: (2) H: set of all hypotheses h MAP h H P(h D) h H P(D h)p(h) P(D) h H P(D h)p(h) Note that we can drop P(D) as the probablty of the data s constant (and ndependent of the hypothess). Nave Bayes Classfers p.7/22 Nave Bayes Classfers p.8/22

3 Maxmum Lkelhood Propertes of Bayes Classfers Now assume that all hypotheses are equally probable a pror,.e, P(h )=P(h j ) for all h,h j H. Ths s called assumng a unform pror. It smplfes computng the posteror: (3) h ML h H P(D h) Ths hypothess s called the maxmum lkelhood hypothess. Incrementalty: wth each tranng example, the pror and the lkelhood can be updated dynamcally: flexble and robust to errors. Combnes pror knowledge and observed data: pror probablty of a hypothess multpled wth probablty of the hypothess gven the tranng data. Probablstc hypotheses: outputs not only a classfcaton, but a probablty dstrbuton over all classes. Meta-classfcaton: the outputs of several classfers can be combned, e.g., by multplyng the probabltes that all classfers predct for a gven class. Nave Bayes Classfers p.9/22 Nave Bayes Classfers p.10/22 Nave Bayes Classfer Assumpton: tranng set conssts of nstances descrbed as conjunctons of attrbutes values, target classfcaton based on fnte set of classes V. The task of the learner s to predct the correct class for a new nstance a 1,a 2,...,a n. Key dea: assgn most probable class v MAP usng Bayes Rule. (4) v MAP v j V P(v j a 1,a 2,...,a n ) v j V P(a 1,a 2,...,a n v j )P(v j ) P(a 1,a 2,...,a n ) v j V P(a 1,a 2,...,a n v j )P(v j ) Nave Bayes: Parameter Estmaton Estmatng P(v j ) s smple: compute the relatve frequency of each target class n the tranng set. Estmatng P(a 1,a 2,...,a n v j ) s dffcult: typcally not enough nstances for each attrbute combnaton n the tranng set: sparse data problem. Independence assumpton: attrbute values are condtonally ndependent gven the target value: nave Bayes. (5) P(a 1,a 2,...,a n v j )= P(a v j ) Hence we get the followng classfer: (6) v NB P(v v j j) P(a v j ) V Nave Bayes Classfers p.11/22 Nave Bayes Classfers p.12/22

4 Nave Bayes: Propertes Estmatng P(a v j ) nstead of P(a 1,a 2,...,a n v j ) greatly reduces the number of parameters (and data sparseness). The learnng step n Nave Bayes conssts of estmatng P(a v j ) and P(v j ) based on the frequences n the tranng data. There s no explct search durng tranng (as opposed to decson trees). An unseen nstance s classfed by computng the class that maxmzes the posteror. When condtonal ndependence s satsfed, Nave Bayes corresponds to MAP classfcaton. Nave Bayes: Example Apply Nave Bayes to the weather tranng data. The hypothess space s V = {yes,no}. Classfy the followng new nstance: outlook temp. humdty wndy play sunny cool hgh true? v NB = arg max v j {yes,no} P(v j ) P(a v j ) = arg max P(v j)p(outlook = sunny v j )P(temp = cool v j ) v j {yes,no} P(humdty = hgh v j )P(wndy = true v j ) Compute prors: P(play = yes)=9/14 P(play = no)=5/14 Nave Bayes Classfers p.13/22 Nave Bayes Classfers p.14/22 Nave Bayes: Example Compute condtonals (examples): P(wndy = true play = yes)= 3/9 P(wndy = true play = no)=3/5 Then compute the best class: P(yes)P(sunny yes)p(cool yes)p(hgh yes)p(true yes) = 9/14 2/9 3/9 3/9 3/9 = P(no)P(sunny no)p(cool no)p(hgh no)p(true no) = 5/14 3/5 1/5 4/5 3/5 = Now classfy the unseen nstance: v NB = arg max j)p(sunny v j )P(cool v j )P(hgh v j )P(true v j ) v j {yes,no} = no Nave Bayes Classfers p.15/22 Nave Bayes: Sparse Data Condtonal probabltes can be estmated drectly as relatve frequences: P(a v j )= n c n where n s the total number of tranng nstances wth class v j, and n c s the number of nstances wth attrbute a and class v. Problem: ths provdes a poor estmate f n c s very small. Extreme case: f n c = 0, then the whole posteror wll be zero. Nave Bayes Classfers p.16/22

5 Nave Bayes: Sparse Data Soluton: use the m-estmate of probabltes: P(a v j )= n c + mp n + m p: pror estmate of the probablty m: equvalent sample sze (constant) In the absence of other nformaton, assume a unform pror: p = 1 k where k s the number of values that the attrbute a can take. Applcaton: Emal Classfcaton Tranng data: a corpus of emal messages, each message annotated as spam or no spam. Task: classfy new emal messages as spam/no spam. To use a nave Bayes classfer for ths task, we have to frst fnd an attrbute representaton of the data. Treat each text poston as an attrbute, wth as ts value the word at ths poston. Example: emal starts: get rch. The nave Bayes classfer s then: v NB = arg max v j j) P(a v j ) {spam,nospam} = arg max j)p(a 1 = get v j )P(a 2 = rch v j ) v j {spam,nospam} Nave Bayes Classfers p.17/22 Nave Bayes Classfers p.18/22 Applcaton: Emal Classfcaton Applcaton: Emal Classfcaton Usng nave Bayes means we assume that words are ndependent of each other. Clearly ncorrect, but doesn t hurt a lot for our task. The classfer uses P(a = w k v j ),.e., the probablty that the -th word n the emal s the k-word n our vocabulary, gven the emal has been classfed as v j. Smplfy by assumng that poston s rrelevant: estmate P(w k v j ),.e., the probablty that word w k occurs n the emal, gven class v j. Create a vocabulary: make a lst of all words n the tranng corpus, dscard words wth very hgh or very low frequency. Tranng: estmate prors: P(v j )= n N Estmate lkelhoods usng the m-estmate: P(w k v j )= n k +1 n+ Vocabulary N: total number of words n all emals n: number of words n emals wth class v j n k : number of tmes word w k occurs n emals wth class v j Vocabulary : szeofthevocabulary Testng: to classfy a new emal, assgn t the class wth the hghest posteror probablty. Ignore unknown words. Nave Bayes Classfers p.19/22 Nave Bayes Classfers p.20/22

6 Summary References Bayes classfer combnes pror knowledge wth observed data: assgns a posteror probablty to a class based on ts pror probablty and ts lkelhood gven the tranng data. Mtchell, Tom. M Machne Learnng. New York: McGraw-Hll. Wtten, Ian H., and Ebe Frank Data Mnng: Practcal Machne Learng Tools and Technques wth Java Implementatons. San Dego, CA: Morgan Kaufmann. Computes the maxmum a posteror (MAP) hypothess or the maxmum lkelhood (ML) hypothess. Nave Bayes classfer assumes condtonal ndependence between attrbutes and assgns the MAP class to new nstances. Lkelhoods can be estmated based on frequences. Problem: sparse data. Soluton: usng the m-estmate (addng a constant). Nave Bayes Classfers p.21/22 Nave Bayes Classfers p.22/22