# The Relationship Between Precision-Recall and ROC Curves

Save this PDF as:

Size: px
Start display at page:

## Transcription

2 Algorithm Algorithm Algorithm Algorithm (a) Comparison in ROC space (b) Comparison in PR space Figure. The difference between comparing algorithms in ROC vs PR space tween these two spaces, and whether some of the interesting properties of ROC space also hold for PR space. We show that for any dataset, and hence a fixed number of positive and negative examples, the ROC curve and PR curve for a given algorithm contain the same points. Therefore the PR curves for Algorithm I and Algorithm II in Figure (b) are, in a sense that we formally define, equivalent to the ROC curves for Algorithm I and Algorithm II, respectively in Figure (a). Based on this equivalence for ROC and PR curves, we show that a curve dominates in ROC space if and only if it dominates in PR space. Second, we introduce the PR space analog to the convex hull in ROC space, which we call the achievable PR curve. We show that due to the equivalence of these two spaces we can efficiently compute the achievable PR curve. Third we demonstrate that in PR space it is insufficient to linearly interpolate between points. Finally, we show that an algorithm that optimizes the area under the ROC curve is not guaranteed to optimize the area under the PR curve. 2. Review of ROC and - In a binary decision problem, a classifier labels examples as either positive or negative. The decision made by the classifier can be represented in a structure known as a confusion matrix or contingency table. The confusion matrix has four categories: True positives (TP) are examples correctly labeled as positives. False positives (FP) refer to negative examples incorrectly labeled as positive. True negatives (TN) correspond to negatives correctly labeled as negative. Finally, false negatives (FN) refer to positive examples incorrectly labeled as negative. A confusion matrix is shown in Figure 2(a). The confusion matrix can be used to construct a point in either ROC space or PR space. Given the confusion matrix, we are able to define the metrics used in each space as in Figure 2(b). In ROC space, one plots the False Positive Rate (FPR) on the x-axis and the True Positive Rate (TPR) on the y-axis. The FPR measures the fraction of negative examples that are misclassified as positive. The TPR measures the fraction of positive examples that are correctly labeled. In PR space, one plots on the x-axis and on the y-axis. is the same as TPR, whereas measures that fraction of examples classified as positive that are truly positive. Figure 2(b) gives the definitions for each metric. We will treat the metrics as functions that act on the underlying confusion matrix which defines a point in either ROC space or PR space. Thus, given a confusion matrix A, RECALL(A) returns the associated with A. 3. Relationship between ROC Space and PR Space ROC and PR curves are typically generated to evaluate the performance of a machine learning algorithm on a given dataset. Each dataset contains a fixed number of positive and negative examples. We show here that there exists a deep relationship between ROC and PR spaces. Theorem 3.. For a given dataset of positive and negative examples, there exists a one-to-one correspondence between a curve in ROC space and a curve in PR space, such that the curves contain exactly the same confusion matrices, if.

3 actual actual positive negative predicted positive F P predicted negative F N T N (a) Confusion Matrix = +F N = +F P = +F N = F P F P +T N (b) Definitions of metrics Figure 2. Common machine learning evaluation metrics Proof. Note that a point in ROC space defines a unique confusion matrix when the dataset is fixed. Since in PR space we ignore T N, one might worry that each point may correspond to multiple confusion matrices. However, with a fixed number of positive and negative examples, given the other three entries in a matrix, T N is uniquely determined. If =, we are unable to recover F P, and thus cannot find a unique confusion matrix. Consequently, we have a one-to-one mapping between confusion matrices and points in PR space. This implies that we also have a one-to-one mapping between points (each defined by a confusion matrix) in ROC space and PR space; hence, we can translate a curve in ROC space to PR space and vice-versa. One important definition we need for our next theorem is the notion that one curve dominates another curve, meaning that all other...curves are beneath it or equal to it (Provost et al., 998). Theorem 3.2. For a fixed number of positive and negative examples, one curve dominates a second curve in ROC space if and only if the first dominates the second in - space. Proof. Claim ( ): If a curve dominates in ROC space then it dominates in PR space. Proof by contradiction. Suppose we have curve I and curve II (as shown in Figure 3) such that curve I dominates in ROC space, yet, once we translate these curves in PR space, curve I no longer dominates. Since curve I does not dominate in PR space, there exists some point A on curve II such that the point B on curve I with identical has lower. In other words, P RECISION(A) > P RECISION(B) yet RECALL(A) = RECALL(B). Since RECALL(A) = RECALL(B) and is identical to R, we have that R(A) = R(B). Since curve I dominates curve II in ROC space F P R(A) F P R(B). Remember that total positives and total negatives are fixed and since R(A) = R(B): A R(A) = Total Positives B R(B) = Total Positives we now have A = B and thus denote both as. Remember that F P R(A) F P R(B) and F P A F P R(A) = Total Negatives F P B F P R(B) = Total Negatives This implies that F P A F P B because P RECISION(A) = F P A + P RECISION(B) = F P B + we now have that P RECISION(A) P RECISION(B). But this contradicts our original assumption that P RECISION(A) > P RECISION(B). Claim 2 ( ): If a curve dominates in PR space then it dominates in ROC space. Proof by contradiction. Suppose we have curve I and curve II (as shown in Figure 4) such that curve I dominates curve II in PR space, but once translated in ROC space curve I no longer dominates. Since curve I does not dominate in ROC space, there exists some point A on curve II such that point B on curve I with identical R yet F P R(A) < R(B). Since RECALL and R are the same, we get that RECALL(A) = RECALL(B). Because curve I dominates in PR space we know that P RECISION(A) P RECISION(B). Remember

4 B A B = A (a) Case : F P R(A) > F P R(B) (b) Case 2: F P R(A) = F P R(B) Figure 3. Two cases for Claim of Theorem B.6.4 B = A A (a) Case : P RECISION(A) < P RECISION(B) (b) Case 2: P RECISION(A) = P RECISION(B) Figure 4. Two cases of Claim 2 of Theorem 3.2 that RECALL(A) = RECALL(B) and A RECALL(A) = Total Positives B RECALL(B) = Total Positives We know that A = B, so we will now denote them simply as. Because P RECISION(A) P RECISION(B) and P RECISION(A) = + F P A P RECISION(B) = + F P B we find that F P A F P B. Now we have F P A F P R(A) = Total Negatives F P B F P R(B) = Total Negatives This implies that F P R(A) F P R(B) and this contradicts our original assumption that F P R(A) < F P R(B). In ROC space the convex hull is a crucial idea. Given a set of points in ROC space, the convex hull must meet the following three criteria:. Linear interpolation is used between adjacent points. 2. No point lies above the final curve.

5 (a) Convex hull in ROC space (b) Curves in ROC space (c) Equivalent curves in PR space Figure 5. Convex hull and its PR analog dominate the naïve method for curve construction in each space. Note that this achievable PR curve is not a true convex hull due to non-linear interpolation. Linear interpolation in PR space is typically not achievable. 3. For any pair of points used to construct the curve, the line segment connecting them is equal to or below the curve. Figure 5(a) shows an example of a convex hull in ROC space. For a detailed algorithm of how to efficiently construct the convex hull, see Cormen et al. (99). In PR space, there exists an analogous curve to the convex hull in ROC space, which we call the achievable PR curve, although it cannot be achieved by linear interpolation. The issue of dominance in ROC space is directly related to this convex hull analog. Corollary 3.. Given a set of points in PR space, there exists an achievable PR curve that dominates the other valid curves that could be constructed with these points. Proof. First, convert the points into ROC space (Theorem 3.), and construct the convex hull of these points in ROC space. By definition, the convex hull dominates all other curves that could be constructed with those points when using linear interpolation between the points. Thus converting the points of the ROC convex hull back into PR space will yield a curve that dominates in PR space as shown in Figures 5(b) and 5(c). This follows from Theorem 3.2. The achievable PR curve will exclude exactly those points beneath the convex hull in ROC space. The convex hull in ROC space is the best legal curve that can be constructed from a set of given ROC points. Many researchers, ourselves included, argue that PR curves are preferable when presented with highly-skewed datasets. Therefore it is surprising that we can find the achievable PR curve (the best legal PR curve) by first computing the convex hull in ROC space and the converting that curve into PR space. Thus the best curve in one space gives you the best curve in the other space. An important methodological issue must be addressed when building a convex hull in ROC space or an achievable curve in PR space. When constructing a ROC curve (or PR curve) from an algorithm that outputs a probability, the following approach is usually taken: first find the probability that each test set example is positive, next sort this list and then traverse the sorted list in ascending order. To simplify the discussion, let class(i) refer to the true classification of the example at position i in the array and prob(i) refer to the probability that the example at position i is positive. For each i such that class(i) class(i + ) and prob(i) < prob(i + ), create a classifier by calling every example j such that j i + positive and all other examples negative. Thus each point in ROC space or PR space represents a specific classifier, with a threshold for calling an example positive. Building the convex hull can be seen as constructing a new classifier, as one picks the best points. Therefore it would be methodologically incorrect to construct a convex hull or achievable PR curve by looking at performance on the test data and then constructing a convex hull. To combat this problem, the convex hull must be constructed using a tuning set as follows: First, use the method described above to find a candidate set of thresholds on the tuning data. Then, build a convex hull over the tuning data. Finally use the thresholds selected on the tuning data, when

6 building an ROC or PR curve for the test data. While this test-data curve is not guaranteed to be a convex hull, it preserves the split between training data and testing data. 4. Interpolation and AUC A key practical issue to address is how to interpolate between points in each space. It is straightforward to interpolate between points in ROC space by simply drawing a straight line connecting the two points. One can achieve any level of performance on this line by flipping a weighted coin to decide between the classifiers that the two end points represent. However, in - space, interpolation is more complicated. As the level of varies, the does not necessarily change linearly due to the fact that F P replaces F N in the denominator of the metric. In these cases, linear interpolation is a mistake that yields an overly-optimistic estimate of performance. Corollary 3. shows how to find the achievable PR curve by simply converting the analogous ROC convex hull; this yields the correct interpolation in PR space. However, a curve consists of infinitely many points, and thus we need a practical, approximate method for translation. We expand here on the method proposed by Goadrich et al. (24) to approximate the interpolation between two points in PR space. Remember that any point A in a - space is generated from the underlying true positive ( A ) and false positive (F P A ) counts. Suppose we have two points, A and B which are far apart in - space. To find some intermediate values, we must interpolate between their counts A and B, and F P A and F P B. We find out how many negative examples it takes to equal one positive, or the F PB F PA local skew, defined by B A. Now we can create new points A +x for all integer values of x such that x B A, i.e. A +, A +2,..., B, and calculate corresponding FP by linearly increasing the false positives for each new point by the local skew. Our resulting intermediate - points will be ( A + x Total Pos, A + x A + x + F P A + F PB F PA B A x For example, suppose we have a dataset with 2 positive examples and 2 negative examples. Let A = 5, F P A = 5, B =, and F P B = 3. Table shows the proper interpolation of the intermediate points between A and B, with the local skew of 5 negatives for ). Table. Correct interpolation between two points in PR space for a dataset with 2 positive and 2 negative examples TP FP REC PREC A B Correct Incorrect Figure 6. The effect of incorrect interpolation in PR space every positive. Notice how the resulting interpolation is not linear between.5 and 5. Often, the area under the curve is used as a simple metric to define how an algorithm performs over the whole space (Bradley, 997; Davis et al., 25; Goadrich et al., 24; Kok & Domingos, 25; Macskassy & Provost, 25; Singla & Domingos, 25). The area under the ROC curve (AUC-ROC) can be calculated by using the trapezoidal areas created between each ROC point, and is equivalent to the Wilcoxon-Mann- Whitney statistic (Cortes & Mohri, 23). By including our intermediate PR points, we can now use the composite trapezoidal method to approximate the area under the PR curve (AUC-PR). The effect of incorrect interpolation on the AUC-PR is especially pronounced when two points are far away in and and the local skew is high. Consider a curve (Figure 6) constructed from a single point of (.2, ), and extended to the endpoints of (, ) and (,.8) as described above (for this example, our dataset contains 433 positives and 56,64 negatives). Interpolating as we have described would have an AUC-PR of.3; a linear connection would severely overestimate with an AUC-PR of.5. Now that we have developed interpolation for PR space, we can give the complete algorithm for find-

7 (a) Comparing AUC-ROC for two algorithms (b) Comparing AUC-PR for two algorithms Figure 7. Difference in optimizing area under the curve in each space ing the achievable PR curve. First, we find the convex hull in ROC space (Corollary 3.). Next, for each point selected by the algorithm to be included in the hull, we use the confusion matrix that defines that point to construct the corresponding point in PR space (Theorem 3.). Finally, we perform the correct interpolation between the newly created PR points. 5. Optimizing Area Under the Curve. Several researchers have investigated using AUC-ROC to inform the search heuristics of their algorithms. Ferri et al. (22) alter decision trees to use the AUC-ROC as their splitting criterion, Cortes and Mohri (23) show that the boosting algorithm Rank- Boost (Freund et al., 998) is also well-suited to optimize the AUC-ROC, Joachims (25) presents a generalization of Support Vector Machines which can optimize AUC-ROC among other ranking metrics, Prati and Flach (25) use a rule selection algorithm to directly create the convex hull in ROC space, and both Yan et al. (23) and Herschtal and Raskutti (24) explore ways to optimize the AUC-ROC within neural networks. Also, ILP algorithms such as Aleph (Srinivasan, 23) can be changed to use heuristics related to ROC or PR space, at least in relation to an individual rule. Knowing that a convex hull in ROC space can be translated into the achievable curve in - space leads to another open question: do algorithms which optimize the AUC-ROC also optimize the AUC- PR? Unfortunately, the answer generally is no, and we prove this by the following counter-example. Figure 7(a) shows two overlapping curves in ROC space for a domain with 2 positive examples and 2 negative examples, where each curve individually is a convex hull. The AUC-ROC for curve I is.83 and the AUC-ROC for curve II is.875, so an algorithm optimizing the AUC-ROC and choosing between these two rankings would choose curve II. However, Figure 7(b) shows the same curves translated into PR space, and the difference here is drastic. The AUC-PR for curve I is now.54 due to the high ranking of over half of the positive examples, while the AUC-PR for curve II is far less at.38, so the direct opposite choice of curve I should be made to optimize the AUC-PR. This is because in PR space the main contribution comes from achieving a lower range with higher. Nevertheless, based on Theorem 3.2 ROC curves are useful in an algorithm that optimizes AUC-PR. An algorithm can find the convex hull in ROC space, convert that curve to PR space for an achievable PR curve, and score the classifier by the area under this achievable PR curve. 6. Conclusions This work makes four important contributions. First, for any dataset, the ROC curve and PR curve for a given algorithm contain the same points. This equivalence, leads to the surprising theorem that a curve dominates in ROC space if and only if it dominates in PR space. Second, as a corollary to the theorem we show the existence of the PR space analog to the convex hull in ROC space, which we call an achievable PR curve. Remarkably, when constructing the achievable PR curve one discards exactly the same points omitted by the convex hull in ROC space. Consequently, we can efficiently compute the achievable PR curve. Third, we show that simple linear interpolation is insufficient between points in PR space. Finally, we show

8 that an algorithm that optimizes the area under the ROC curve is not guaranteed to optimize the area under the PR curve. Acknowledgements A Java program for calculating all of the discussed metrics can be found at We gratefully acknowledge the funding from USA NLM Grant 5T5LM and USA Air Force Grant F , Vítor Santos Costa, Louis Oliphant, our advisors David Page and Jude Shavlik and our anonymous reviewers for their helpful comments and suggestions. References Bockhorst, J., & Craven, M. (25). Markov networks for detecting overlapping elements in sequence data. Neural Information Processing Systems 7 (NIPS). MIress. Bradley, A. (997). The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognition, 3, Bunescu, R., Ge, R., Kate, R., Marcotte, E., Mooney, R., Ramani, A., & Wong, Y. (24). Comparative Experiments on Learning Information Extractors for Proteins and their Interactions. Journal of Artificial Intelligence in Medicine, Cormen, T. H., Leiserson, Charles, E., & Rivest, R. L. (99). Introduction to algorithms. MIress. Cortes, C., & Mohri, M. (23). AUC optimization vs. error rate minimization. Neural Information Processing Systems 5 (NIPS). MIress. Davis, J., Burnside, E., Dutra, I., Page, D., Ramakrishnan, R., Costa, V. S., & Shavlik, J. (25). View learning for statistical relational learning: With an application to mammography. Proceeding of the 9th International Joint Conference on Artificial Intelligence. Edinburgh, Scotland. Drummond, C., & Holte, R. (2). Explicitly representing expected cost: an alternative to ROC representation. Proceeding of Knowledge Discovery and Datamining (pp ). Drummond, C., & Holte, R. C. (24). What ROC curves can t do (and cost curves can). ROCAI (pp. 9 26). Ferri, C., Flach, P., & Henrandez-Orallo, J. (22). Learning decision trees using area under the ROC curve. Proceedings of the 9th International Conference on Machine Learning (pp ). Morgan Kaufmann. Freund, Y., Iyer, R., Schapire, R., & Singer, Y. (998). An efficient boosting algorithm for combining preferences. Proceedings of the 5th International Conference on Machine Learning (pp. 7 78). Madison, US: Morgan Kaufmann Publishers, San Francisco, US. Goadrich, M., Oliphant, L., & Shavlik, J. (24). Learning ensembles of first-order clauses for recall-precision curves: A case study in biomedical information extraction. Proceedings of the 4th International Conference on Inductive Logic Programming (ILP). Porto, Portugal. Herschtal, A., & Raskutti, B. (24). Optimising area under the ROC curve using gradient descent. Proceedings of the 2st International Conference on Machine Learning (p. 49). New York, NY, USA: ACM Press. Joachims, T. (25). A support vector method for multivariate performance measures. Proceedings of the 22nd International Conference on Machine Learning. ACM Press. Kok, S., & Domingos, P. (25). Learning the structure of Markov Logic Networks. Proceedings of 22nd International Conference on Machine Learning (pp ). ACM Press. Macskassy, S., & Provost, F. (25). Suspicion scoring based on guilt-by-association, collective inference, and focused data access. International Conference on Intelligence Analysis. Manning, C., & Schutze, H. (999). Foundations of statistical natural language processing. MIress. Prati, R., & Flach, P. (25). ROCCER: an algorithm for rule learning based on ROC analysis. Proceeding of the 9th International Joint Conference on Artificial Intelligence. Edinburgh, Scotland. Provost, F., Fawcett, T., & Kohavi, R. (998). The case against accuracy estimation for comparing induction algorithms. Proceeding of the 5th International Conference on Machine Learning (pp ). Morgan Kaufmann, San Francisco, CA. Raghavan, V., Bollmann, P., & Jung, G. S. (989). A critical investigation of recall and precision as measures of retrieval system performance. ACM Trans. Inf. Syst., 7, Singla, P., & Domingos, P. (25). Discriminative training of Markov Logic Networks. Proceedings of the 2th National Conference on Artificial Intelligene (AAAI) (pp ). AAAI Press. Srinivasan, A. (23). The Aleph Manual Version 4. oucl/ research/ areas/ machlearn/ Aleph/. Yan, L., Dodier, R., Mozer, M., & Wolniewicz, R. (23). Optimizing classifier performance via the Wilcoxon- Mann-Whitney statistics. Proceedings of the 2th International Conference on Machine Learning.

### CSEP 546: Decision Trees and Experimental Methodology. Jesse Davis

CSEP 546: Decision Trees and Experimental Methodology Jesse Davis Outline Decision Trees Representation Learning Algorithm Potential pitfalls Experimental Methodology Decision Trees Popular hypothesis

### ROC Graphs: Notes and Practical Considerations for Data Mining Researchers

ROC Graphs: Notes and Practical Considerations for Data Mining Researchers Tom Fawcett 1 1 Intelligent Enterprise Technologies Laboratory, HP Laboratories Palo Alto Alexandre Savio (GIC) ROC Graphs GIC

### Mining the Software Change Repository of a Legacy Telephony System

Mining the Software Change Repository of a Legacy Telephony System Jelber Sayyad Shirabad, Timothy C. Lethbridge, Stan Matwin School of Information Technology and Engineering University of Ottawa, Ottawa,

### Chapter 6. The stacking ensemble approach

82 This chapter proposes the stacking ensemble approach for combining different data mining classifiers to get better performance. Other combination techniques like voting, bagging etc are also described

### COMP9417: Machine Learning and Data Mining

COMP9417: Machine Learning and Data Mining A note on the two-class confusion matrix, lift charts and ROC curves Session 1, 2003 Acknowledgement The material in this supplementary note is based in part

### The many faces of ROC analysis in machine learning. Peter A. Flach University of Bristol, UK

The many faces of ROC analysis in machine learning Peter A. Flach University of Bristol, UK www.cs.bris.ac.uk/~flach/ Objectives After this tutorial, you will be able to [model evaluation] produce ROC

### Uplift Modeling with ROC: An SRL Case Study

Uplift Modeling with ROC: An SRL Case Study Houssam Nassif, Finn Kuusisto, Elizabeth S. Burnside, and Jude Shavlik University of Wisconsin, Madison, USA Abstract. Uplift modeling is a classification method

### Evaluating Protein-protein Interaction Predictors with a Novel 3-Dimensional Metric

Evaluating Protein-protein Interaction Predictors with a Novel 3-Dimensional Metric Haohan Wang 1, Madhavi K. Ganapathiraju, PhD 2 1 Language Technologies Institute, School of Computer Science, Carnegie

### Knowledge Discovery and Data Mining

Knowledge Discovery and Data Mining Lecture 15 - ROC, AUC & Lift Tom Kelsey School of Computer Science University of St Andrews http://tom.home.cs.st-andrews.ac.uk twk@st-andrews.ac.uk Tom Kelsey ID5059-17-AUC

### Evaluation in Machine Learning (2) Aims. Introduction

Acknowledgements Evaluation in Machine Learning (2) 15s1: COMP9417 Machine Learning and Data Mining School of Computer Science and Engineering, University of New South Wales Material derived from slides

### Using Random Forest to Learn Imbalanced Data

Using Random Forest to Learn Imbalanced Data Chao Chen, chenchao@stat.berkeley.edu Department of Statistics,UC Berkeley Andy Liaw, andy liaw@merck.com Biometrics Research,Merck Research Labs Leo Breiman,

### The Optimality of Naive Bayes

The Optimality of Naive Bayes Harry Zhang Faculty of Computer Science University of New Brunswick Fredericton, New Brunswick, Canada email: hzhang@unbca E3B 5A3 Abstract Naive Bayes is one of the most

### Experiments in Web Page Classification for Semantic Web

Experiments in Web Page Classification for Semantic Web Asad Satti, Nick Cercone, Vlado Kešelj Faculty of Computer Science, Dalhousie University E-mail: {rashid,nick,vlado}@cs.dal.ca Abstract We address

### Analysis on Weighted AUC for Imbalanced Data Learning Through Isometrics

Journal of Computational Information Systems 8: (22) 37 378 Available at http://www.jofcis.com Analysis on Weighted AUC for Imbalanced Data Learning Through Isometrics Yuanfang DONG, 2, Xiongfei LI,, Jun

### Machine Learning (CS 567)

Machine Learning (CS 567) Time: T-Th 5:00pm - 6:20pm Location: GFS 118 Instructor: Sofus A. Macskassy (macskass@usc.edu) Office: SAL 216 Office hours: by appointment Teaching assistant: Cheol Han (cheolhan@usc.edu)

### Machine Learning model evaluation. Luigi Cerulo Department of Science and Technology University of Sannio

Machine Learning model evaluation Luigi Cerulo Department of Science and Technology University of Sannio Accuracy To measure classification performance the most intuitive measure of accuracy divides the

### Evaluation and Cost-Sensitive Learning. Cost-Sensitive Learning

1 J. Fürnkranz Evaluation and Cost-Sensitive Learning Evaluation Hold-out Estimates Cross-validation Significance Testing Sign test ROC Analysis Cost-Sensitive Evaluation ROC space ROC convex hull Rankers

### CLASSIFICATION JELENA JOVANOVIĆ. Web:

CLASSIFICATION JELENA JOVANOVIĆ Email: jeljov@gmail.com Web: http://jelenajovanovic.net OUTLINE What is classification? Binary and multiclass classification Classification algorithms Performance measures

### PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION Introduction In the previous chapter, we explored a class of regression models having particularly simple analytical

### Performance Measures for Machine Learning

Performance Measures for Machine Learning 1 Performance Measures Accuracy Weighted (Cost-Sensitive) Accuracy Lift Precision/Recall F Break Even Point ROC ROC Area 2 Accuracy Target: 0/1, -1/+1, True/False,

### Performance Measures in Data Mining

Performance Measures in Data Mining Common Performance Measures used in Data Mining and Machine Learning Approaches L. Richter J.M. Cejuela Department of Computer Science Technische Universität München

### A Support Vector Method for Multivariate Performance Measures

Thorsten Joachims Cornell University, Dept. of Computer Science, 4153 Upson Hall, Ithaca, NY 14853 USA tj@cs.cornell.edu Abstract This paper presents a Support Vector Method for optimizing multivariate

### COMP 551 Applied Machine Learning Lecture 6: Performance evaluation. Model assessment and selection.

COMP 551 Applied Machine Learning Lecture 6: Performance evaluation. Model assessment and selection. Instructor: (jpineau@cs.mcgill.ca) Class web page: www.cs.mcgill.ca/~jpineau/comp551 Unless otherwise

### Evaluation & Validation: Credibility: Evaluating what has been learned

Evaluation & Validation: Credibility: Evaluating what has been learned How predictive is a learned model? How can we evaluate a model Test the model Statistical tests Considerations in evaluating a Model

### ROC Analysis. Peter Flach, University of Bristol

ROC Analysis Peter Flach, University of Bristol An entry to appear in the forthcoming Encyclopedia of Machine Learning (Springer) Synonyms Receiver Operating Characteristic Analysis Definition ROC analysis

### Evaluation in Machine Learning (1) Evaluation in machine learning. Aims. 14s1: COMP9417 Machine Learning and Data Mining.

Acknowledgements Evaluation in Machine Learning (1) 14s1: COMP9417 Machine Learning and Data Mining School of Computer Science and Engineering, University of New South Wales March 19, 2014 Material derived

### ROC Graphs: Notes and Practical Considerations for Data Mining Researchers

ROC Graphs: Notes and Practical Considerations for Data Mining Researchers Tom Fawcett Intelligent Enterprise Technologies Laboratory HP Laboratories Palo Alto HPL-23-4 January 7 th, 23* E-mail: tom_fawcett@hp.com

### Supervised Learning (Big Data Analytics)

Supervised Learning (Big Data Analytics) Vibhav Gogate Department of Computer Science The University of Texas at Dallas Practical advice Goal of Big Data Analytics Uncover patterns in Data. Can be used

### Overview. Evaluation Connectionist and Statistical Language Processing. Test and Validation Set. Training and Test Set

Overview Evaluation Connectionist and Statistical Language Processing Frank Keller keller@coli.uni-sb.de Computerlinguistik Universität des Saarlandes training set, validation set, test set holdout, stratification

### Data Mining - Evaluation of Classifiers

Data Mining - Evaluation of Classifiers Lecturer: JERZY STEFANOWSKI Institute of Computing Sciences Poznan University of Technology Poznan, Poland Lecture 4 SE Master Course 2008/2009 revised for 2010

### Data Mining Practical Machine Learning Tools and Techniques

Counting the cost Data Mining Practical Machine Learning Tools and Techniques Slides for Section 5.7 In practice, different types of classification errors often incur different costs Examples: Loan decisions

### Hani Tamim, PhD Associate Professor Department of Internal Medicine American University of Beirut

Hani Tamim, PhD Associate Professor Department of Internal Medicine American University of Beirut A graphical plot which illustrates the performance of a binary classifier system as its discrimination

### ROC Curve, Lift Chart and Calibration Plot

Metodološki zvezki, Vol. 3, No. 1, 26, 89-18 ROC Curve, Lift Chart and Calibration Plot Miha Vuk 1, Tomaž Curk 2 Abstract This paper presents ROC curve, lift chart and calibration plot, three well known

### Data Mining Practical Machine Learning Tools and Techniques

Credibility: Evaluating what s been learned Data Mining Practical Machine Learning Tools and Techniques Slides for Chapter 5 of Data Mining by I. H. Witten, E. Frank and M. A. Hall Issues: training, testing,

### Performance Metrics. number of mistakes total number of observations. err = p.1/1

p.1/1 Performance Metrics The simplest performance metric is the model error defined as the number of mistakes the model makes on a data set divided by the number of observations in the data set, err =

### Data Mining. Nonlinear Classification

Data Mining Unit # 6 Sajjad Haider Fall 2014 1 Nonlinear Classification Classes may not be separable by a linear boundary Suppose we randomly generate a data set as follows: X has range between 0 to 15

### T-61.3050 : Email Classification as Spam or Ham using Naive Bayes Classifier. Santosh Tirunagari : 245577

T-61.3050 : Email Classification as Spam or Ham using Naive Bayes Classifier Santosh Tirunagari : 245577 January 20, 2011 Abstract This term project gives a solution how to classify an email as spam or

### A Comparison of Different ROC Measures for Ordinal Regression

Willem Waegeman Willem.Waegeman@UGent.be Department of Electrical Energy, Systems and Automation, Ghent University, Technologiepark 913, B-905 Ghent, Belgium Bernard De Baets Bernard.DeBaets@UGent.be Department

### Roulette Sampling for Cost-Sensitive Learning

Roulette Sampling for Cost-Sensitive Learning Victor S. Sheng and Charles X. Ling Department of Computer Science, University of Western Ontario, London, Ontario, Canada N6A 5B7 {ssheng,cling}@csd.uwo.ca

### An Approach to Detect Spam Emails by Using Majority Voting

An Approach to Detect Spam Emails by Using Majority Voting Roohi Hussain Department of Computer Engineering, National University of Science and Technology, H-12 Islamabad, Pakistan Usman Qamar Faculty,

### Learning with Skewed Class Distributions

CADERNOS DE COMPUTAÇÃO XX (2003) Learning with Skewed Class Distributions Maria Carolina Monard and Gustavo E.A.P.A. Batista Laboratory of Computational Intelligence LABIC Department of Computer Science

### Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1 Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 2. Tid Refund Marital Status

Data Mining Classification: Basic Concepts, Decision Trees, and Evaluation Lecture tes for Chapter 4 Introduction to Data Mining by Tan, Steinbach, Kumar Classification: Definition Given a collection of

### A Hybrid Approach to Learn with Imbalanced Classes using Evolutionary Algorithms

Proceedings of the International Conference on Computational and Mathematical Methods in Science and Engineering, CMMSE 2009 30 June, 1 3 July 2009. A Hybrid Approach to Learn with Imbalanced Classes using

### A Two-Pass Statistical Approach for Automatic Personalized Spam Filtering

A Two-Pass Statistical Approach for Automatic Personalized Spam Filtering Khurum Nazir Junejo, Mirza Muhammad Yousaf, and Asim Karim Dept. of Computer Science, Lahore University of Management Sciences

### On the effect of data set size on bias and variance in classification learning

On the effect of data set size on bias and variance in classification learning Abstract Damien Brain Geoffrey I Webb School of Computing and Mathematics Deakin University Geelong Vic 3217 With the advent

### Constrained Classification of Large Imbalanced Data by Logistic Regression and Genetic Algorithm

Constrained Classification of Large Imbalanced Data by Logistic Regression and Genetic Algorithm Martin Hlosta, Rostislav Stríž, Jan Kupčík, Jaroslav Zendulka, and Tomáš Hruška A. Imbalanced Data Classification

### Efficient AUC Optimization for Classification

Efficient AUC Optimization for Classification Toon Calders 1 and Szymon Jaroszewicz 1 Eindhoven University of Technology, the Netherlands National Institute of Telecommunications, Warsaw, Poland Abstract.

### Ensemble Methods. Knowledge Discovery and Data Mining 2 (VU) (707.004) Roman Kern. KTI, TU Graz 2015-03-05

Ensemble Methods Knowledge Discovery and Data Mining 2 (VU) (707004) Roman Kern KTI, TU Graz 2015-03-05 Roman Kern (KTI, TU Graz) Ensemble Methods 2015-03-05 1 / 38 Outline 1 Introduction 2 Classification

### 1. Classification problems

Neural and Evolutionary Computing. Lab 1: Classification problems Machine Learning test data repository Weka data mining platform Introduction Scilab 1. Classification problems The main aim of a classification

### Machine Learning. Chapter 18, 21. Some material adopted from notes by Chuck Dyer

Machine Learning Chapter 18, 21 Some material adopted from notes by Chuck Dyer What is learning? Learning denotes changes in a system that... enable a system to do the same task more efficiently the next

### Low Cost Correction of OCR Errors Using Learning in a Multi-Engine Environment

2009 10th International Conference on Document Analysis and Recognition Low Cost Correction of OCR Errors Using Learning in a Multi-Engine Environment Ahmad Abdulkader Matthew R. Casey Google Inc. ahmad@abdulkader.org

### Classification of Bad Accounts in Credit Card Industry

Classification of Bad Accounts in Credit Card Industry Chengwei Yuan December 12, 2014 Introduction Risk management is critical for a credit card company to survive in such competing industry. In addition

### arxiv:1112.0829v1 [math.pr] 5 Dec 2011

How Not to Win a Million Dollars: A Counterexample to a Conjecture of L. Breiman Thomas P. Hayes arxiv:1112.0829v1 [math.pr] 5 Dec 2011 Abstract Consider a gambling game in which we are allowed to repeatedly

### AUC: a Statistically Consistent and more Discriminating Measure than Accuracy

AUC: a Statistically Consistent and more Discriminating Measure than Accuracy Charles X Ling Jin Huang Harry Zhang Department of Computer Science Faculty of Computer Science The University of Western Ontario

### Performance Metrics for Graph Mining Tasks

Performance Metrics for Graph Mining Tasks 1 Outline Introduction to Performance Metrics Supervised Learning Performance Metrics Unsupervised Learning Performance Metrics Optimizing Metrics Statistical

Adaptive Online Gradient Descent Peter L Bartlett Division of Computer Science Department of Statistics UC Berkeley Berkeley, CA 94709 bartlett@csberkeleyedu Elad Hazan IBM Almaden Research Center 650

### Towards better accuracy for Spam predictions

Towards better accuracy for Spam predictions Chengyan Zhao Department of Computer Science University of Toronto Toronto, Ontario, Canada M5S 2E4 czhao@cs.toronto.edu Abstract Spam identification is crucial

### Neural Networks. Introduction to Artificial Intelligence CSE 150 May 29, 2007

Neural Networks Introduction to Artificial Intelligence CSE 150 May 29, 2007 Administration Last programming assignment has been posted! Final Exam: Tuesday, June 12, 11:30-2:30 Last Lecture Naïve Bayes

### Addressing the Class Imbalance Problem in Medical Datasets

Addressing the Class Imbalance Problem in Medical Datasets M. Mostafizur Rahman and D. N. Davis the size of the training set is significantly increased [5]. If the time taken to resample is not considered,

### WE DEFINE spam as an e-mail message that is unwanted basically

1048 IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 10, NO. 5, SEPTEMBER 1999 Support Vector Machines for Spam Categorization Harris Drucker, Senior Member, IEEE, Donghui Wu, Student Member, IEEE, and Vladimir

### Notes on Factoring. MA 206 Kurt Bryan

The General Approach Notes on Factoring MA 26 Kurt Bryan Suppose I hand you n, a 2 digit integer and tell you that n is composite, with smallest prime factor around 5 digits. Finding a nontrivial factor

### Abstract. The DNA promoter sequences domain theory and database have become popular for

Journal of Artiæcial Intelligence Research 2 è1995è 361í367 Submitted 8è94; published 3è95 Research Note On the Informativeness of the DNA Promoter Sequences Domain Theory Julio Ortega Computer Science

### Simple Language Models for Spam Detection

Simple Language Models for Spam Detection Egidio Terra Faculty of Informatics PUC/RS - Brazil Abstract For this year s Spam track we used classifiers based on language models. These models are used to

### International Journal of Electronics and Computer Science Engineering 1449

International Journal of Electronics and Computer Science Engineering 1449 Available Online at www.ijecse.org ISSN- 2277-1956 Neural Networks in Data Mining Priyanka Gaur Department of Information and

### Getting Even More Out of Ensemble Selection

Getting Even More Out of Ensemble Selection Quan Sun Department of Computer Science The University of Waikato Hamilton, New Zealand qs12@cs.waikato.ac.nz ABSTRACT Ensemble Selection uses forward stepwise

### Globally Optimal Crowdsourcing Quality Management

Globally Optimal Crowdsourcing Quality Management Akash Das Sarma Stanford University akashds@stanford.edu Aditya G. Parameswaran University of Illinois (UIUC) adityagp@illinois.edu Jennifer Widom Stanford

### Feature Selection with Data Balancing for. Prediction of Bank Telemarketing

Applied Mathematical Sciences, Vol. 8, 2014, no. 114, 5667-5672 HIKARI Ltd, www.m-hikari.com http://dx.doi.org/10.12988/ams.2014.47222 Feature Selection with Data Balancing for Prediction of Bank Telemarketing

### Performance Analysis of Naive Bayes and J48 Classification Algorithm for Data Classification

Performance Analysis of Naive Bayes and J48 Classification Algorithm for Data Classification Tina R. Patil, Mrs. S. S. Sherekar Sant Gadgebaba Amravati University, Amravati tnpatil2@gmail.com, ss_sherekar@rediffmail.com

### Example: Credit card default, we may be more interested in predicting the probabilty of a default than classifying individuals as default or not.

Statistical Learning: Chapter 4 Classification 4.1 Introduction Supervised learning with a categorical (Qualitative) response Notation: - Feature vector X, - qualitative response Y, taking values in C

### III. DATA SETS. Training the Matching Model

A Machine-Learning Approach to Discovering Company Home Pages Wojciech Gryc Oxford Internet Institute University of Oxford Oxford, UK OX1 3JS Email: wojciech.gryc@oii.ox.ac.uk Prem Melville IBM T.J. Watson

### Mining Life Insurance Data for Customer Attrition Analysis

Mining Life Insurance Data for Customer Attrition Analysis T. L. Oshini Goonetilleke Informatics Institute of Technology/Department of Computing, Colombo, Sri Lanka Email: oshini.g@iit.ac.lk H. A. Caldera

### Artificial Neural Networks and Support Vector Machines. CS 486/686: Introduction to Artificial Intelligence

Artificial Neural Networks and Support Vector Machines CS 486/686: Introduction to Artificial Intelligence 1 Outline What is a Neural Network? - Perceptron learners - Multi-layer networks What is a Support

### DATA MINING FOR IMBALANCED DATASETS: AN OVERVIEW

Chapter 40 DATA MINING FOR IMBALANCED DATASETS: AN OVERVIEW Nitesh V. Chawla Department of Computer Science and Engineering University of Notre Dame IN 46530, USA Abstract Keywords: A dataset is imbalanced

### Predict Influencers in the Social Network

Predict Influencers in the Social Network Ruishan Liu, Yang Zhao and Liuyu Zhou Email: rliu2, yzhao2, lyzhou@stanford.edu Department of Electrical Engineering, Stanford University Abstract Given two persons

### Reducing multiclass to binary by coupling probability estimates

Reducing multiclass to inary y coupling proaility estimates Bianca Zadrozny Department of Computer Science and Engineering University of California, San Diego La Jolla, CA 92093-0114 zadrozny@cs.ucsd.edu

### Cross-Validation. Synonyms Rotation estimation

Comp. by: BVijayalakshmiGalleys0000875816 Date:6/11/08 Time:19:52:53 Stage:First Proof C PAYAM REFAEILZADEH, LEI TANG, HUAN LIU Arizona State University Synonyms Rotation estimation Definition is a statistical

### Feature vs. Classifier Fusion for Predictive Data Mining a Case Study in Pesticide Classification

Feature vs. Classifier Fusion for Predictive Data Mining a Case Study in Pesticide Classification Henrik Boström School of Humanities and Informatics University of Skövde P.O. Box 408, SE-541 28 Skövde

### Spyware Prevention by Classifying End User License Agreements

Spyware Prevention by Classifying End User License Agreements Niklas Lavesson, Paul Davidsson, Martin Boldt, and Andreas Jacobsson Blekinge Institute of Technology, Dept. of Software and Systems Engineering,

### Ensemble Data Mining Methods

Ensemble Data Mining Methods Nikunj C. Oza, Ph.D., NASA Ames Research Center, USA INTRODUCTION Ensemble Data Mining Methods, also known as Committee Methods or Model Combiners, are machine learning methods

### Evaluating Machine Learning Algorithms. Machine DIT

Evaluating Machine Learning Algorithms Machine Learning @ DIT 2 Evaluating Classification Accuracy During development, and in testing before deploying a classifier in the wild, we need to be able to quantify

### W6.B.1. FAQs CS535 BIG DATA W6.B.3. 4. If the distance of the point is additionally less than the tight distance T 2, remove it from the original set

http://wwwcscolostateedu/~cs535 W6B W6B2 CS535 BIG DAA FAQs Please prepare for the last minute rush Store your output files safely Partial score will be given for the output from less than 50GB input Computer

### Measuring Lift Quality in Database Marketing

Measuring Lift Quality in Database Marketing Gregory Piatetsky-Shapiro Xchange Inc. One Lincoln Plaza, 89 South Street Boston, MA 2111 gps@xchange.com Sam Steingold Xchange Inc. One Lincoln Plaza, 89 South

### The Role of Size Normalization on the Recognition Rate of Handwritten Numerals

The Role of Size Normalization on the Recognition Rate of Handwritten Numerals Chun Lei He, Ping Zhang, Jianxiong Dong, Ching Y. Suen, Tien D. Bui Centre for Pattern Recognition and Machine Intelligence,

### Incorporating Evidence in Bayesian networks with the Select Operator

Incorporating Evidence in Bayesian networks with the Select Operator C.J. Butz and F. Fang Department of Computer Science, University of Regina Regina, Saskatchewan, Canada SAS 0A2 {butz, fang11fa}@cs.uregina.ca

### Statistical Validation and Data Analytics in ediscovery. Jesse Kornblum

Statistical Validation and Data Analytics in ediscovery Jesse Kornblum Administrivia Silence your mobile Interactive talk Please ask questions 2 Outline Introduction Big Questions What Makes Things Similar?

### U.C. Berkeley CS276: Cryptography Handout 0.1 Luca Trevisan January, 2009. Notes on Algebra

U.C. Berkeley CS276: Cryptography Handout 0.1 Luca Trevisan January, 2009 Notes on Algebra These notes contain as little theory as possible, and most results are stated without proof. Any introductory

### A General Framework for Mining Concept-Drifting Data Streams with Skewed Distributions

A General Framework for Mining Concept-Drifting Data Streams with Skewed Distributions Jing Gao Wei Fan Jiawei Han Philip S. Yu University of Illinois at Urbana-Champaign IBM T. J. Watson Research Center

### Comparing the Results of Support Vector Machines with Traditional Data Mining Algorithms

Comparing the Results of Support Vector Machines with Traditional Data Mining Algorithms Scott Pion and Lutz Hamel Abstract This paper presents the results of a series of analyses performed on direct mail

### PREDICTING SUCCESS IN THE COMPUTER SCIENCE DEGREE USING ROC ANALYSIS

PREDICTING SUCCESS IN THE COMPUTER SCIENCE DEGREE USING ROC ANALYSIS Arturo Fornés arforser@fiv.upv.es, José A. Conejero aconejero@mat.upv.es 1, Antonio Molina amolina@dsic.upv.es, Antonio Pérez aperez@upvnet.upv.es,

### FRAUD DETECTION IN ELECTRIC POWER DISTRIBUTION NETWORKS USING AN ANN-BASED KNOWLEDGE-DISCOVERY PROCESS

FRAUD DETECTION IN ELECTRIC POWER DISTRIBUTION NETWORKS USING AN ANN-BASED KNOWLEDGE-DISCOVERY PROCESS Breno C. Costa, Bruno. L. A. Alberto, André M. Portela, W. Maduro, Esdras O. Eler PDITec, Belo Horizonte,

### Evaluation and Credibility. How much should we believe in what was learned?

Evaluation and Credibility How much should we believe in what was learned? Outline Introduction Classification with Train, Test, and Validation sets Handling Unbalanced Data; Parameter Tuning Cross-validation

### Azure Machine Learning, SQL Data Mining and R

Azure Machine Learning, SQL Data Mining and R Day-by-day Agenda Prerequisites No formal prerequisites. Basic knowledge of SQL Server Data Tools, Excel and any analytical experience helps. Best of all:

### Anne Kraus (Autor) Recent Methods from Statistics and Machine Learning for Credit Scoring

Anne Kraus (Autor) Recent Methods from Statistics and Machine Learning for Credit Scoring https://cuvillier.de/de/shop/publications/6703 Copyright: Cuvillier Verlag, Inhaberin Annette Jentzsch-Cuvillier,

### ROC analysis of classifiers in machine learning: A survey

Intelligent Data Analysis 17 (2013) 531 558 531 DOI 10.3233/IDA-130592 IOS Press ROC analysis of classifiers in machine learning: A survey Matjaž Majnik and Zoran Bosnić Faculty of Computer and Information

### NOTES ON LINEAR TRANSFORMATIONS

NOTES ON LINEAR TRANSFORMATIONS Definition 1. Let V and W be vector spaces. A function T : V W is a linear transformation from V to W if the following two properties hold. i T v + v = T v + T v for all

### Knowledge Discovery from patents using KMX Text Analytics

Knowledge Discovery from patents using KMX Text Analytics Dr. Anton Heijs anton.heijs@treparel.com Treparel Abstract In this white paper we discuss how the KMX technology of Treparel can help searchers

### Consolidated Tree Classifier Learning in a Car Insurance Fraud Detection Domain with Class Imbalance

Consolidated Tree Classifier Learning in a Car Insurance Fraud Detection Domain with Class Imbalance Jesús M. Pérez, Javier Muguerza, Olatz Arbelaitz, Ibai Gurrutxaga, and José I. Martín Dept. of Computer

### Statistical Machine Learning

Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes