Naïve Bayes classifier & Evaluation framework

Transcription

1 Lecture aïve Bayes classfer & Evaluaton framework Mlos Hauskrecht 539 Sennott Square Generatve approach to classfcaton Idea:. Represent and learn the dstrbuton p x, y. Use t to defne probablstc dscrmnant functons E.g. g o x = p y = x g x = p y = x Typcal model p x, y = p x y p y p x y = Class-condtonal dstrbutons denstes bnary classfcaton: two class- condtonal dstrbutons p x y = p x y = p y = Prors on classes - probablty of class y bnary classfcaton: Bernoull dstrbuton p y = + p y = = y x

2 aïve Bayes classfer a generatve classfer model wth an addtonal smplfyng assumpton: All nput attrbutes are condtonally ndependent of each other gven the class. So we have: C p x, y = p x y p y X X X n p x y = = p x y Learnng of parameters of the model Much smpler densty estmaton problems We need to learn: p x y = and p x y = and p y Because of the assumpton of the condtonal ndependence we need to learn: for every varable : p x y = and p x y = If the number of nput attrbutes s large ths much easer Also, the model gves us a flexblty to represent nput attrbutes dfferent of dfferent forms!!! E.g. one attrbute can be modeled usng the Bernoull, the other as Gaussan densty, or as a Posson dstrbuton

3 Makng a class decson for the aïve Bayes Dscrmnant functons. Lkelhood of data choose the class that explans the nput data x better lkelhood of the data p x Θ, > p x Θ then y=, = = else y= g x g x Posteror of a class choose the class wth better posteror probablty p y = x > p y = x then y= else y= p y = x = = p x Θ, p y = = p x Θ, p y = + p x Θ =, p y = Expermental evaluaton Dataset: a set of samples Splt the dataset to: Tranng and testng data Learn on the Tranng data Test on the Testng data Test errors gve an honest assesment of the error for future cases recall the overft ssue

4 Prevent the tran/test splt bas If we use only one tran/test splt we can be lucky or unlucky A much better less based opton s to use multple tran/test splts and average the test errors obtaned on these splts How to do the splts? Random subsamplng: choose the test and tran set randomly k tmes Cross-fold valdaton: a more systematc approach Splt data to k equal parttons Create a tran data usng k- parttons, test data on the remanng partton Gves us k dfferent tran test splts Evaluaton For any data set we used to test the model we can buld a confuson matrx: Counts of examples wth: class label ω that are classfed wth a label target 4 7 model 54 α

5 Evaluaton For any data set we used to test the model we can buld a confuson matrx: model 4 target 7 54 agreement Error:? Evaluaton for the bnary classfcaton For any data set we used to test the model we can buld a confuson matrx: model TP F target FP T TP: True postve ht FP: False postve false alarm T: True negatve correct reecton F: False negatve a mss

6 Addtonal statstcs Senstvty SES TP = TP + F Specfcty SPEC T = T + FP Postve predctve value TP PPT = TP + FP egatve predctve value T PV = T + F Bnary classfcaton. Addtonal quanttes. Confuson matrx model target 4 8 PPV = 4/5 PV = 8/ SES= 4/6 SPEC= 8/9 Row and column quanttes: Senstvty SES Specfcty SPEC Postve predctve value PPV egatve predctve value PV

7 Recever operatng characterstc ROC shows the dscrmnablty between the two classes under dfferent decson bases types of errors we make matter ROC curve s created by plottng: the true postve rate aganst false postve rates or senstvty aganst -specfcty Bnary decsons: accuracy ω ω.4. x * Probabltes: True postve ht False postve false alarm True negatve correct reecton False negatve a mss p p p p x > x* x ω x > x* x ω x < x* x ω x < x* x ω

8 Decson threshold..8.6 ω ω.4. x * Movement of x* changes the probabltes: True postve ht p x > x* x ω False postve false alarm p x > x* x ω True negatve correct reecton p x < x* x ω False negatve a mss p x < x* x ω Recever Operatng Characterstc ROC ROC curve plots : p x > x* x ω vs p x > x* x ω for dfferent x* ω x * ω p x > x* x ω p x > x* x ω

9 ROC curve Case Case Case p x > x* x ω p x > x* x ω Bayesan decson theory Assume we want to ncorporate our bas about the learnng nto the learnng process Assume a multway classfcaton problem and more general confuson matrx Counts of examples wth: class label ω that are classfed wth a label α agreement

10 Zero-one loss functon Msclassfcaton error Based on the zero- one loss functon Any msclassfed example counts as Correctly classfed example counts as agreement What s the zero- one loss for the confuson matrx? General loss functon Error functon based on a more general loss functon Dfferent msclassfcatons have dfferent weght loss α our choce ω true label λ α ω loss for classfcaton Example: λ α ω 3 3 5

11 Bayesan decson theory More general loss functon Dfferent msclassfcatons have dfferent weght loss λ α ω Expected loss for choce acton R α x = λ α ω P ω x Also called condtonal rsk Decson rule: α x Chooses label acton accordng to the nput Overall expected loss for the decson rule α R α = R α x, x P x d x Bayesan decson theory The optmal decson rule α * x = arg max λ α ω P ω x α How to modfy classfers to handle dfferent loss? Dscrmnatve models: Drectly optmze the parameters accordng to the new loss functon Generatve models: Learn probabltes as before Decsons about classes are based to mnmze the emprcal loss as seen above