Receiver Operating Characteristic Analysis (ROC)

Transcription

1 Receiver Operting Chrcteristic Anlysis (ROC) ROC nlysis is n pproch cn be used to compre clssifiers. It is clerest when there re two clsses. On the y-xis, you plot the true positives (TP). On the x-xis, you plot the flse positives (FP).

2 ROC Curve True Positive Rte Flse Positive Rte

3 ROC

4 How re points on n ROC curve creted? For neurl network you cn vry the threshold t which n output unit is considered ctive. For Nïve Byes use threshold on the probbility needed to predict the clss of interest. For clssifiers tht do not produce continuous outputs, some other pproch must be found. The TP/FP rtio will be estimted on the vlidtion or test set.

5 Perhps, we should try k-fold cross vlidtion nd tke n verge t ech point to smooth out vlidtion set vrince. Alterntively, Provost, et l. suggest different wy of verging performnce on multiple folds.

6 When is Clssifier better? When it's ROC curve domintes tht of other lgorithms. This mens every point on the ROC curve hs greter TP percentge for fixed FP percentge thn competitors. The more usul cse is tht there is crossover. Hence, n lgorithm my be better in some regions nd worse in others.

7 Question A point on n ROC curve cn come from: ) The best trined clssifier b) A threshold pplied to the probbility of the clss of interest. c) the verge vlue of cross vlidtion d) none of these

8 ROC.

9 Convex Hull The convex hull is the set of points on n ROC curve with the highest TP s for ech FP vlue. If we cn chieve the convex hull, this is optiml with the compred clssifiers.

10 Convex Hull

11 When is Clssifier better? We cn compre curves tht crossover one nother by clculting the re under the curve (AUC). The lerning lgorithm tht hs greter AUC my be considered generlly better.

12 Stimge ROC

13 Threshold t ech probbility nd plot TP/FP (from Tble 5.6 Witten) Thresholded TP/FP points TP Thresholded TP/FP points FP

14 Threshold t ech probbility nd plot TP/FP (from Tble 5.6 Witten) Thresholded TP/FP points TP Thresholded TP/FP points FP

15 Mesures in informtion retrievl Percentge of retrieved documents tht re relevnt: precision=tp/(tp+fp) Percentge of relevnt documents tht re returned: recll =TP/(TP+FN) Precision/recll curves hve hyperbolic shpe Summry mesures: verge precision t 20%, 50% nd 80% recll (three-point verge recll) F-mesure=(2 recll precision)/(recll +precision)

16 Summry of mesures Domin Plot Explntion Lift chrt Mrketing TP Subset size ROC curve Recllprecision curve Communictions Informtion retrievl TP rte FP rte Recll Precision TP (TP+FP)/(TP+FP+TN+FN) TP/(TP+FN) FP/(FP+TN) TP/(TP+FN) TP/(TP+FP)

17 Question The F-mesure compres to the AUC ) it is snpshot t one point on n ROC curve b) you could verge it over points on n ROC curve c) we expect the AUC my be more stble d) they re relly the sme.

18 Evluting numeric prediction Sme strtegies: independent test set, crossvlidtion, significnce tests, etc. Difference: error mesures Actul trget vlues: 1 2 n Predicted trget vlues: p 1 p 2 p n Most populr mesure: men-squred error n) ( p 1) ( pn n Esy to mnipulte mthemticlly

19 Other mesures The root men-squred error : n) ( p 1) ( pn n The men bsolute error is less sensitive to outliers thn the men-squred error: p pn n n Sometimes reltive error vlues re more pproprite (e.g. 10% for n error of 50 when predicting 500)

20 Improvement on the men How much does the scheme improve on simply predicting the verge? The reltive squred error is ( ): The reltive bsolute error is: ) (... ) ( ) (... ) ( n n n p p is the verge n n n p p

21 Correltion coefficient Mesures the sttisticl correltion between the predicted vlues nd the ctul vlues Scle independent, between 1 nd +1 Good performnce leds to lrge vlues! A P PA S S S 1 ) )( ( = n p p S i i i PA 1 ) ( 2 = n p p S i i P 1 ) ( 2 = n S i i A

22 Which mesure? Best to look t ll of them Often it doesn t mtter Exmple: A B C D Root men-squred error Men bsolute error Root rel squred error 42.2% 57.2% 39.4% 35.8% Reltive bsolute error 43.1% 40.1% 34.8% 30.4% Correltion coefficient v D best v C second-best v A, B rguble

23 Question For numeric dt in mesuring error ) only squred error is good b) the bsolute error nd squred error re good c) ll mesures re the sme in terms of mesuring performnce d) we my find differences in performnce by using different error mesures (i.e. different best pproch).

24 The MDL principle MDL stnds for minimum description length The description length is defined s: spce required to describe theory + spce required to describe the theory s mistkes In our cse the theory is the clssifier nd the mistkes re the errors on the trining dt Aim: we seek clssifier with miniml DL MDL principle is model selection criterion

25 Model selection criteri Model selection criteri ttempt to find good compromise between: The complexity of model Its prediction ccurcy on the trining dt Resoning: good model is simple model tht chieves high ccurcy on the given dt Also known s Occm s Rzor : the best theory is the smllest one tht describes ll the fcts

26 Elegnce vs. errors l l l l Theory 1: very simple, elegnt theory tht explins the dt lmost perfectly Theory 2: significntly more complex theory tht reproduces the dt without mistkes Theory 1 is probbly preferble Clssicl exmple: Kepler s three lws on plnetry motion Less ccurte thn Copernicus s ltest refinement of the Ptolemic theory of epicycles 26

27 MDL nd compression l MDL principle reltes to dt compression: l The best theory is the one tht compresses the dt the most l I.e. to compress dtset we generte model nd then store the model nd its mistkes l We need to compute () size of the model, nd (b) spce needed to encode the errors l (b) esy: use the informtionl loss function l () need method to encode the model 27

28 MDL nd Byes s theorem l l l l l L[T]= length of the theory L[E T]=trining set encoded wrt the theory Description length= L[T] + L[E T] Byes s theorem gives posteriori probbility of theory given the dt: Equivlent to: Pr[T E] = Pr[T E]= Pr[E T]Pr[T ] Pr[E ] log Pr[T E]= log Pr[E T] log Pr[T ] log Pr[E ] 28 Dt Mining: Prcticl Mchine Lerning Tools nd Techniques Pr[E T ]Pr[T ] Pr[E] log(pr[t e]) = log(pr[e T ) log(pr[t ]) log(pr[e]) constnt

29 MDL nd MAP MAP stnds for mximum posteriori probbility Finding the MAP theory corresponds to finding the MDL theory Difficult bit in pplying the MAP principle: determining the prior probbility Pr[T] of the theory Corresponds to difficult prt in pplying the MDL principle: coding scheme for the theory I.e. if we know priori tht prticulr theory is more likely we need less bits to encode it

30 Discussion of MDL principle Advntge: mkes full use of the trining dt when selecting model Disdvntge 1: pproprite coding scheme/prior probbilities for theories re crucil Disdvntge 2: no gurntee tht the MDL theory is the one which minimizes the expected error Note: Occm s Rzor is n xiom! Epicurus s principle of multiple explntions: keep ll theories tht re consistent with the dt

31 Question MDL is of interest becuse: ) it uses trdeoff of model size nd ccurcy to decide the best clssifier model b) n encoding my improve the lerning lgorithm c) it equtes to both informtion theory nd Byes theory d) requires dditionl work for ech clssifier type nd error mesure (for encoding)