Performance Metrics for Graph Mining Tasks

Transcription

1 Performance Metrics for Graph Mining Tasks 1

2 Outline Introduction to Performance Metrics Supervised Learning Performance Metrics Unsupervised Learning Performance Metrics Optimizing Metrics Statistical Significance Techniques Model Comparison 2

3 Outline Introduction to Performance Metrics Supervised Learning Performance Metrics Unsupervised Learning Performance Metrics Optimizing Metrics Statistical Significance Techniques Model Comparison 3

4 Introduction to Performance Metrics Performance metric measures how well your data mining algorithm is performing on a given dataset. For example, if we apply a classification algorithm on a dataset, we first check to see how many of the data points were classified correctly. This is a performance metric and the formal name for it is accuracy. Performance metrics also help us decide is one algorithm is better or worse than another. For example, one classification algorithm A classifies 80% of data points correctly and another classification algorithm B classifies 90% of data points correctly. We immediately realize that algorithm B is doing better. There are some intricacies that we will discuss in this chapter. 4

5 Outline Introduction to Performance Metrics Supervised Learning Performance Metrics Unsupervised Learning Performance Metrics Optimizing Metrics Statistical Significance Techniques Model Comparison 5

6 Supervised Learning Performance Metrics Metrics that are applied when the ground truth is known (E.g., ification tasks) Outline: 2 X 2 Confusion Matrix Multi-level Confusion Matrix Visual Metrics Cross-validation 6

7 2X2 Confusion Matrix An 2X2 matrix, is used to tabulate the results of 2-class supervised learning problem and entry (i,j) represents the number of elements with class label i, but predicted to have class label j. True Positive Actual Predicted + - False Negative + f ++ f +- C = f ++ + f +- - f -+ f -- D = f -+ + f -- A = f ++ + f -+ B = f +- + f -- T = f ++ + f -+ + f +- + f -- False Positive + and are two class labels True Negative 7

8 2X2 Confusion Metrics Example Results from a ification Algorithms Corresponding 2x2 matrix for the given table Vertex ID Actual Predicted Actual Predicted C = D = 3 A = 6 B = 2 T = 8 True positive = 4 False positive = 1 True Negative = 1 False Negative =2 8

9 2X2 Confusion Metrics Performance Metrics Walk-through different metrics using the following example 1. Accuracy is proportion of correct predictions 2. Error rate is proportion of incorrect predictions 3. Recall is the proportion of + data points predicted as + 4. Precision is the proportion of data points predicted as + that are truly + 9

10 Multi-level Confusion Matrix An nxn matrix, where n is the number of classes and entry (i,j) represents the number of elements with class label i, but predicted to have class label j 10

11 Multi-level Confusion Matrix Example Actual Marginal Sum of Predictions Predicted Marginal Sum of Actual Values T = 14 11

12 Multi-level Confusion Matrix Conversion to 2X2 f ++ Predicted f f +- f-- Actual Actual 2X2 Matrix Specific to Predicted 1 (+) Not 1 (-) 1 (+) 2 2 C = 4 Not 1 (-) 2 8 D = 10 A = 4 B = 10 T = 14 We can now apply all the 2X2 metrics Accuracy = 2/14 Error = 8/14 Recall = 2/4 Precision = 2/4

13 Multi-level Confusion Matrix Performance Metrics Actual Predicted Critical Success Index or Threat Score is the ratio of correct predictions for class L to the sum of vertices that belong to L and those predicted as L 2. Bias - For each class L, it is the ratio of the total points with class label L to the number of points predicted as L. Bias helps understand if a model is over or under-predicting a class 13

14 Confusion Metrics R-code library(performancemetrics) data(m) M [,1] [,2] [1,] 4 1 [2,] 2 1 twocrossconfusionmatrixmetrics(m) data(multilevelm) MultiLevelM [,1] [,2] [,3] [1,] [2,] [3,] multilevelconfusionmatrixmetrics(multilevelm) 14

15 Visual Metrics Metrics that are plotted on a graph to obtain the visual picture of the performance of two class classifiers (0,1) - Ideal ROC plot 1 True Positive Rate AUC = 0.5 (1,1) Predicts the +ve class all the time (0,0) Predicts the ve class all the time 0 0 False Positive Rate 1 Plot the performance of multiple models to decide which one performs best 15

16 Understanding Model Performance based on ROC Plot Models that lie in this upper left have good performance Note: This is where you aim to get the model 1. Models that lie in lower left are conservative. 2. Will not predict + unless strong evidence 3. Low False positives but high False Negatives 1 True Positive Rate 0 0 AUC = 0.5 False Positive Rate 1 1. Models that lie in upper right are liberal. 2. Will predict + with little evidence 3. High False positives Models that lie in this area perform worse than random Note: Models here can be negated to move them to the upper right corner 16

17 ROC Plot Example 1 True Positive Rate M 1 (0.1,0.8) M 3 (0.3,0.5) M 2 (0.5,0.5) 0 0 False Positive Rate M 1 s performance occurs furthest in the upper-right direction and hence is considered the best model. 1 17

18 Cross-validation Cross-validation also called rotation estimation, is a way to analyze how a predictive data mining model will perform on an unknown dataset, i.e., how well the model generalizes Strategy: 1. Divide up the dataset into two non-overlapping subsets 2. One subset is called the test and the other the training 3. Build the model using the training dataset 4. Obtain predictions of the test set 5. Utilize the test set predictions to calculate all the performance metrics Typically cross-validation is performed for multiple iterations, selecting a different non-overlapping test and training set each time 18

19 Types of Cross-validation hold-out: Random 1/3 rd of the data is used as test and remaining 2/3 rd as training k-fold: Divide the data into k partitions, use one partition as test and remaining k-1 partitions for training Leave-one-out: Special case of k-fold, where k=1 Note: Selection of data points is typically done in stratified manner, i.e., the class distribution in the test set is similar to the training set 19

20 Outline Introduction to Performance Metrics Supervised Learning Performance Metrics Unsupervised Learning Performance Metrics Optimizing Metrics Statistical Significance Techniques Model Comparison 20

21 Unsupervised Learning Performance Metrics Metrics that are applied when the ground truth is not always available (E.g., Clustering tasks) Outline: Evaluation Using Prior Knowledge Evaluation Using Cluster Properties 21

22 Evaluation Using Prior Knowledge To test the effectiveness of unsupervised learning methods is by considering a dataset D with known class labels, stripping the labels and providing the set as input to any unsupervised leaning algorithm, U. The resulting clusters are then compared with the knowledge priors to judge the performance of U To evaluate performance 1. Contingency Table 2. Ideal and Observed Matrices 22

23 Contingency Table Cluster Same Cluster Different Cluster Same u 11 u 10 Different u 01 u 00 (A) To fill the table, initialize u 11, u 01, u 10, u 00 to 0 (B) Then, for each pair of points of form (v,w): 1. if v and w belong to the same class and cluster then increment u if v and w belong to the same class but different cluster then increment u if v and w belong to the different class but same cluster then increment u if v and w belong to the different class and cluster then increment u 00 23

24 Contingency Table Performance Metrics Example Matrix Same Cluster Cluster Different Cluster Same 9 4 Different 3 12 Rand Statistic also called simple matching coefficient is a measure where both placing a pair of points with the same class label in the same cluster and placing a pair of points with different class labels in different clusters are given equal importance, i.e., it accounts for both specificity and sensitivity of the clustering Jaccard Coefficient can be utilized when placing a pair of points with the same class label in the same cluster is primarily important 24

25 Ideal and Observed Matrices Given that the number of points is T, the ideal-matrix is a TxT matrix, where each cell (i,j) has a 1 if the points i and j belong to the same class and a 0 if they belong to different clusters. The observed-matrix is a TxT matrix, where a cell (i,j) has a 1 if the points i and j belong to the same cluster and a 0 if they belong to different cluster Mantel Test is a statistical test of the correlation between two matrices of the same rank. The two matrices, in this case, are symmetric and, hence, it is sufficient to analyze lower or upper diagonals of each matrix 25

26 Evaluation Using Prior Knowledge R-code library(performancemetrics) data(contingencytable) ContingencyTable [,1] [,2] [1,] 9 4 [2,] 3 12 contingencytablemetrics(contingencytable) 26

27 Evaluation Using Cluster Properties In the absence of prior knowledge we have to rely on the information from the clusters themselves to evaluate performance. 1. Cohesion measures how closely objects in the same cluster are related 2. Separation measures how distinct or separated a cluster is from all the other clusters Here, g i refers to cluster i, W is total number of clusters, x and y are data points, proximity can be any similarity measure (e.g., cosine similarity) We want the cohesion to be close to 1 and separation to be close to 0 27

28 Outline Introduction to Performance Metrics Supervised Learning Performance Metrics Unsupervised Learning Performance Metrics Optimizing Metrics Statistical Significance Techniques Model Comparison 28

29 Optimizing Metrics Performance metrics that act as optimization functions for a data mining algorithm Outline: Sum of Squared Errors Preserved Variability 29

30 Sum of Squared Errors Squared sum error (SSE) is typically used in clustering algorithms to measure the quality of the clusters obtained. This parameter takes into consideration the distance between each point in a cluster to its cluster center (centroid or some other chosen representative). For d j, a point in cluster g i, where m i is the cluster center of g i, and W, the total number of clusters, SSE is defined as follows: This value is small when points are close to their cluster center, indicating a good clustering. Similarly, a large SSE indicates a poor clustering. Thus, clustering algorithms aim to minimize SSE. 30

31 Preserved Variability Preserved variability is typically used in eigenvector-based dimension reduction techniques to quantify the variance preserved by the chosen dimensions. The objective of the dimension reduction technique is to maximize this parameter. Given that the point is represented in r dimensions (k << r), the eigenvalues are λ 1 >=λ 2 >=.. λ r-1 >=λ r. The preserved variability (PV) is calculated as follows: The value of this parameter depends on the number of dimensions chosen: the more included, the higher the value. Choosing all the dimensions will result in the perfect score of 1. 31

32 Outline Introduction to Performance Metrics Supervised Learning Performance Metrics Unsupervised Learning Performance Metrics Optimizing Metrics Statistical Significance Techniques Model Comparison 32

33 Statistical Significance Techniques Methods used to asses a p-value for the different performance metrics Scenario: We obtain say cohesion =0.99 for clustering algorithm A. From the first look it feels like 0.99 is a very good score. However, it is possible that the underlying data is structured in such a way that you would get 0.99 no matter how you cluster the data. Thus, 0.99 is not very significant. One way to decide that is by using statistical significance estimation. We will discuss the Monte Carlo Procedure in next slide! 33

34 Monte Carlo Procedure Empirical p-value Estimation Monte Carlo procedure uses random sampling to assess the significance of a particular performance metric we obtain could have been attained at random. For example, if we obtain a cohesion score of a cluster of size 5 is 0.99, we would be inclined to think that it is a very cohesive score. However, this value could have resulted due to the nature of the data and not due to the algorithm. To test the significance of this 0.99 value we 1. Sample N (usually 1000) random sets of size 5 from the dataset 2. Recalculate the cohesion for each of the 1000 sets 3. Count R: number of random sets with value >= 0.99 (original score of cluster) 4. Empirical p-value for the cluster of size 5 with 0.99 score is given by R/N 5. We apply a cutoff say 0.05 to decide if 0.99 is significant Steps 1-4 is the Monte Carlo method for p-value estimation. 34

35 Outline Introduction to Performance Metrics Supervised Learning Performance Metrics Unsupervised Learning Performance Metrics Optimizing Metrics Statistical Significance Techniques Model Comparison 35

36 Model Comparison Metrics that compare the performance of different algorithms Scenario: 1) Model 1 provides an accuracy of 70% and Model 2 provides an accuracy of 75% 2) From the first look, Model 2 seems better, however it could be that Model 1 is predicting 1 better than 2 3) However, 1 is indeed more important than 2 for our problem 4) We can use model comparison methods to take this notion of importance into consideration when we pick one model over another Cost-based Analysis is an important model comparison method discussed in the next few slides. 36

37 Cost-based Analysis In real-world applications, certain aspects of model performance are considered more important than others. For example: if a person with cancer was diagnosed as cancer-free or vice-versa then the prediction model should be especially penalized. This penalty can be introduced in the form of a cost-matrix. Cost Matrix Actual Predicted c 11 c 10 - c 01 c 00 Associated with f 10 or u 10 Associated with f 11 or u 11 Associated with f 01 or u 01 Associated with f 00 or u 00 37

38 Cost-based Analysis Cost of a Model The cost and confusion matrices for Model M are given below Confusion Matrix Predicted + - Cost Matrix Predicted + - Actual + f 11 f 10 - f 01 f 00 Actual + c 11 c 10 - c 01 c 00 Cost of Model M is given as: 38

39 Cost-based Analysis Comparing Two Models This analysis is typically used to select one model when we have more than one choice through using different algorithms or different parameters to the learning algorithms. Cost Matrix Actual Predicted Confusion Matrix of M y Actual Predicted Cost of M y : 200 Cost of M x : 100 Confusion Matrix of M x Actual Predicted C Mx < C My Purely, based on cost model, M x is a better model 39

40 Cost-based Analysis R-code library(performancemetrics) data(mx) data(my) data(costmatrix) Mx [,1] [,2] [1,] 4 1 [2,] 2 1 My [,1] [,2] [1,] 3 2 [2,] 2 1 costanalysis(mx,costmatrix) costanalysis(my,costmatrix) 40