Information Processing and Management

Transcription

1 Information Processing and Management 45 (2009) Contents ists avaiabe at ScienceDirect Information Processing and Management journa homepage: A systematic anaysis of performance measures for cassification tasks Marina Sokoova a, *, Guy Lapame b a Eectronic Heath Information Lab, Chidren s Hospita of Eastern Ontario, Ottawa, Canada b Département d informatique et de recherche opérationnee Université de Montréa, Montréa, Canada artice info abstract Artice history: Received 14 February 2008 Received in revised form 21 November 2008 Accepted 6 March 2009 Avaiabe onine 8 May 2009 Keywords: Performance evauation Machine Learning Text cassification This paper presents a systematic anaysis of twenty four performance measures used in the compete spectrum of Machine Learning cassification tasks, i.e., binary, muti-cass, muti-abeed, and hierarchica. For each cassification task, the study reates a set of changes in a confusion matrix to specific characteristics of data. Then the anaysis concentrates on the type of changes to a confusion matrix that do not change a measure, therefore, preserve a cassifier s evauation (measure invariance). The resut is the measure invariance taxonomy with respect to a reevant abe distribution changes in a cassification probem. This forma anaysis is supported by exampes of appications where invariance properties of measures ead to a more reiabe evauation of cassifiers. Text cassification suppements the discussion with severa case studies. Ó 2009 Esevier Ltd. A rights reserved. 1. Motivation Machine Learning (ML) divides cassification onto binary, muti-cass, muti-abeed, and hierarchica tasks. In this work we present a systematic anaysis of twenty four performance measures used in these cassification subfieds. We focus on how we casses are identified without reference to computation cost or time. We consider a set of changes in a confusion matrix that correspond to specific characteristics of data. We then anayze the type of changes that do not change a measure s vaue and therefore preserve a cassifier s evauation. This is what we ca measure invariance. As a resut, we buid the measure invariance taxonomy with respect to a reevant abe distribution changes in a cassification probem. We suppement the forma anaysis by exampes of appications where invariance properties of measures ead to a more reiabe evauation of cassifiers; exampes are taken from text cassification. Note, that we focus on recent ML deveopments; more detais on ML measures can be found, for exampe, in Sokoova, Japkowicz, and Szpakowicz (2006) which ooks into reations between the measures and assessment of medica trias. To the best of our knowedge, our current study is the first reviews of ML measures which comprehensivey evauates the invariant properties of measures. Preiminary resuts on binary cassification appear in (Sokoova & Lapame, 2007). This study expands the resuts two-fod, with discussion of new invariant properties, in some cases, adding monotonicity properties, and consideration of muti-cass, muti-abeed, and hierarchica measures. Empirica evauation remains the most used approach for the agorithm assessment, athough ML agorithms can be evauated through empirica assessment or theory or both, e.g., derived generaized bounds and empirica resuts (Marchand & Shawe-Tayor, 2002). Evauation techniques based on mutipe experiments are considered in Dietterich (1998), one of the most cited work on empirica evauation of ML agorithms. An extensive critique of ML evauation practice can be found in Sazberg (1999). The author anayzes the currenty used methods and their statistica vaidity. The paper distinguishes two goas of evauation: a comparison of agorithms, and the feasibiity of agorithms on a specific domain. Demsar (2006) surveys how cassifiers are compared over mutipe data sets. Empirica comparison is most often done by appying agorithms on various data sets and then evauating the performance of the cassifiers that the agorithms have produced; accuracy * Corresponding author. E-mai addresses: msokoova@eheathinformation.ca (M. Sokoova), apame@iro.umontrea.ca (G. Lapame) /$ - see front matter Ó 2009 Esevier Ltd. A rights reserved. doi: /j.ipm

2 428 M. Sokoova, G. Lapame / Information Processing and Management 45 (2009) being the most often used measure. In a these assessment approaches, the agorithm and the output cassifiers take the centra stage. We take an aternative route ooking how characteristics affect the objectivity of measures. Our forma discussion of ML performance measures compements popuar statistica and empirica comparisons such as the ones presented in Goutte and Gaussier (2005). We show that, in some earning settings, the correct identification of positive exampes may be important whereas in others, the correct identification of negative exampes or disagreement between data and cassifier abes may be more significant. Thus, standard performance measures shoud be re-evauated with respect to those scenarios. Previousy, ML studies of performance measures have primariy focused on binary cassification. For a compete review, we add muticass, muti-topic and hierarchica cassification measures. The current study can be usefu for measure design. So far, the ML community did not consider measures invariance when new ones were introduced (Bengio, Mariéthoz, & Keer, 2005; Huang & Ling, 2007) or suggested for adoption from other discipines (Sokoova et a., 2006). 2. Overview of cassification tasks Supervised ML aows access to the data abes during the agorithm s training and testing stages. Consider categorica abes when data entries x 1 ;...; x n have to be assigned into predefined casses C 1 ;...; C. Then cassification fas into one of the foowing tasks: Binary: the input is to be cassified into one, and ony one, of two non-overapping casses ðc 1 ; C 2 Þ; Binary cassification is the most popuar cassification task. Assigned categories can be objective, independent of manua evauation (e.g, repubican or democrat in the votes data of the UCI repository (Asuncion & Newman, 2007)) or subjective, dependent on manua evauation (e.g., positive or negative reviews in Amazon.com (Bitzer et a., 2007)). Casses can be we-defined (e.g., the votes abes), ambiguous (e.g., the review opinion abes), or both (e.g., medica vs. other texts in the Newsgroups coection 1 ). Muti-cass: the input is to be cassified into one, and ony one, of non-overapping casses. Muticass probems incude the identification of the iris type in a three-cass data set popuar in pattern recognition (Duda & Hart, 1973), in the earning the origina 135 categories in the benchmark Reuters coection, 2 or in tagging utterances as objective, subjective, or neutra (Wison, Wiebe, & Hwa, 2006). As for the binary case, muti-cass categorization can be objective or subjective, we-defined or ambiguous. Muti-abeed: the input is to be cassified into severa of non-overapping C j. Exampes incude cassification of functions of yeast genes (Mewes, Abermann, Heumann, Lieb, & Pfeiffer, 1997), identifying scenes from image data (Li, Zhang, & Zhu, 2006) or text-database aignment and word aignment in machine transation (Snyder & Barziay, 2007). In text mining of medica information, muti-abe cassification methods are often evauated on OHSUMED, a coection of medica references (Hersh, Buckey, Leone, & Hickam, 1997). When the earning task is document topic cassification, muti-abeing is often referred as muti-topic cassification such as for cinica texts that are assigned mutipe disease codes from ICD-9-CM (Sasaki, Rea, & Ananiadou, 2007). Binary, muti-cass, and muti-abeed probems form fat cassification (Yang, 1999), in which categories are isoated and their reations are not considered important. The next, hierarchica, probem addresses reations among categories and incudes their structure into earning targets. Hierarchica: the input is to be cassified into one, and ony one, C j which are be divided into subcasses or grouped into supercasses. The hierarchy is defined and cannot be changed during cassification. Text cassification and bioinformatics suppy many exampes, e.g., protein function prediction (Eisner, Pouin, Szafron, Lu, & Greiner, 2005). Hierarchica cassification can be transformed into fat cassification. For exampe, the Reuters coection cassification can be muticass (Bobicev & Sokoova, 2008), muti-abeed (Tikk & Biró, 2003), and hierarchica (Sun, Lim, & Ng, 2003). A frequent appearance of anguage and text probems among the isted above exampes can be expained by a specia roe Natura Language Processing (NLP) hods in ML appications. The richness of anguage characteristics and the fast-increasing voume of readiy avaiabe digita texts make texts not ony a neary inexhaustibe research area, but aso one of the most important data formats for ML appications (Shawe-Tayor & Christianini, 2004). Text Cassification has achieved a prominent pace among ML appications to NLP probems. It is dedicated to finding texts according to a given criteria (Sebastiani, 2002) and it incudes the cassification of documents (research papers, technica reports, magazine artices, etc.). For topic cassification (e.g., identification of documents about a given city or documents about bands and artists, etc.) documents are simpy cassified as being reevant to the topic or not; hence, casses are buit as positive vs everything ese. Retrieva of reevant documents being the more important task, the focus in this case is on true positive cassification. First comprehensive books on Machine Learning were pubished in ate 1990 s (Langey, 1996; Mitche, 1997). As a discipine, ML borrowed measures from assortment of discipines traditionay reied on empirica evidence, e.g., medica trias (Issebacher & Braunwad, 1994), behavioura research (Cohen, 1988), information retrieva (IR) (van Rijsbergen, 1979; Saton & McGi, 1983). In some ways, text cassification borrows from Information Extraction (IE) which preuded the use of Machine

3 M. Sokoova, G. Lapame / Information Processing and Management 45 (2009) Tabe 1 Confusion matrix for binary cassification and the corresponding array representation used in this paper. Learning in automated text processing and understanding, e.g., the automated anaysis and generation of synonymous texts (Boyer & Lapame, 1985). IE and IR metrics in the evauation of ML agorithms are an exampe of such borrowing. The evauation metrics commony used in Text Cassification (Precision, Reca, Fscore) have their origin in IE. The formuas for these measures negect the correct cassification of negative exampes, they instead refect the importance of retrieva of positive exampes in text/document cassification: Precision: the number of correcty cassified positive exampes divided by the number of exampes abeed by the system as positive Reca: the number of correcty cassified positive exampes divided by the number of positive exampes in the data Fscore: a combination of the above. In recent years, the NLP and ML communities have turned their attention to the study of opinions, subjective statements, and sentiments. The corresponding empirica probems are represented by the cassification of poitica debates, web postings or phone cas in which the main task is non-topic cassification, e.g. vote cassification, gender cassification, sentiment cassification, etc. Data for these studies are gathered from chart-boards, bogs, product and movie reviews, emai, records of phone conversations and poitica debates, eectronic negotiation transcripts, etc. Chart-boards, bogs and movie reviews are often used in sentiment anaysis to find whether texts refect a positive or negative opinion of the author on certain products or events. In this case, texts are cassified according to opinion/sentiment abes (Pang, Lee, & Vaithyanathan, 2002). Emai discussions, records of phone conversation and eectronic negotiation transcripts are used in studies of individua behavior. The aim of such studies is to find what factors infuence the behavior of a person in a specific situation. Cassification of texts depends on the probem statement. Transcripts of the US Congress debates are used in the socia network anaysis, a new area of Artificia Inteigence research. Here a common task is to define important infuence factors and predict the future behavior of members of a socia group. In this case, records are cassified according to the actions of speakers (Thomas, Pang, & Lee, 2006). These sources represent records of human communication that convey meanings sent by a speaker and received by a hearer. These meanings can be compex and subty expressed and constituted from both what is said and what is impied. So far, there is no common consensus on the choice of measures used to evauate the performance of cassifiers in opinion, subjectivity and sentiment anaysis. Additiona performance measures other than the above are Accuracy used in Pang et a. (2002) and Thomas et a. (2006), or the correspondence between Precision and Reca in Gamon, Aue, Corston-Oiver, and Ringger (2005). When going from document cassification to the cassification of human communication, it is important to know how different performance measures reate to each other in order to hep resove disagreements among performance evauations. This phenomenon happens quite often in experimenta studies. 3. Performance measures for cassification The correctness of a cassification can be evauated by computing the number of correcty recognized cass exampes (true positives), the number of correcty recognized exampes that do not beong to the cass (true negatives), and exampes that either were incorrecty assigned to the cass (fase positives) or that were not recognized as cass exampes (fase negatives). These four counts constitute a confusion matrix shown in Tabe 1 for the case of the binary cassification. Tabe 2 presents the most often used measures for binary cassification based on the vaues of the confusion matrix. AUC (Area Under the Curve), 3 captures a singe point on the Reception Operating Characteristic curve; it can aso be viewed as a inear transformation of Youden Index (Youden, 1950). We omit measures such as BreakEvenPoint, the point at which Precision = Reca (Goutte & Gaussier, 2005), and the combined AUC (Huang & Ling, 2007) because their properties can be derived from the basic Accuracy measures. However, we present Fscore s properties because of its extensive use in text cassification. Tabe 3 presents the measures for muti-cass cassification. For an individua cass C i, the assessment is defined by tp i ; fn i ; tn i ; fp i :Accuracy i ; Precision i ; Reca i are cacuated from the counts for C i. Quaity of the overa cassification is usuay 3 AUC, sometimes referred to as Baanced Accuracy.

4 430 M. Sokoova, G. Lapame / Information Processing and Management 45 (2009) Tabe 2 Measures for binary cassification using the notation of Tabe 1. Measure Formua Evauation focus tpþtn Accuracy tpþfnþfpþtn tp Precision tpþfp tp Reca (Sensitivity) tpþfn ðb Fscore 2 þ1þtp ðb 2 þ1þtpþb 2 fnþfp tn Specificity fpþtn 1 tp AUC 2 tpþfn þ tn tnþfp Overa effectiveness of a cassifier Cass agreement of the data abes with the positive abes given by the cassifier Effectiveness of a cassifier to identify positive abes Reations between data s positive abes and those given by a cassifier How effectivey a cassifier identifies negative abes Cassifier s abiity to avoid fase cassification Tabe 3 Measures for muti-cass cassification based on a generaization of the measures of Tabe 1 for many casses C i: tp i are true positive for C i, and fp i fase positive, fn i fase negative, and tn i true negative counts respectivey. and M indices represent micro- and macro-averaging. Measure Formua Evauation focus Average Accuracy Error Rate Precision Reca Fscore Precision M Reca M Fscore M P tp i þtn i i¼1 tp i þfn i þfp i þtn i P fp i þfn i i¼1 tp i þfn i þfp i þtn i P tp i¼1 i P ðtp i¼1 iþfp i Þ P tp i¼1 i P ðtp i¼1 iþfn i Þ ðb 2 þ1þprecisionreca b 2 PrecisionþReca P tp i i¼1 tp i þfp i P tp i i¼1 tp i þfn i ðb 2 þ1þprecisionmrecam b 2 PrecisionMþRecaM The average per-cass effectiveness of a cassifier The average per-cass cassification error Agreement of the data cass abes with those of a cassifiers if cacuated from sums of per-text decisions Effectiveness of a cassifier to identify cass abes if cacuated from sums of per-text decisions Reations between data s positive abes and those given by a cassifier based on sums of per-text decisions An average per-cass agreement of the data cass abes with those of a cassifiers An average per-cass effectiveness of a cassifier to identify cass abes Reations between data s positive abes and those given by a cassifier based on a per-cass average assessed in two ways: a measure is the average of the same measures cacuated for C 1 ;...; C (macro-averaging shown with an M index), or the sum of counts to obtain cumuative tp; fn; tn; fp and then cacuating a performance measure (micro-averaging shown with indices). Macro-averaging treats a casses equay whie micro-averaging favors bigger casses. As there is yet no we-deveoped muti-cass Reception Operating Characteristic anaysis (Lachiche & Fach, 2003), we do not incude AUC in the ist of muti-cassification measures. The quaity of muti-topic cassification (Tabe 4) is assessed through either partia or compete cass abe matching (Kazawa, Izumitani, Taira, & Maeda, 2005); the atter is often referred to as exact matching. We consider a casses and their abes as being equivaent. These measures thus count correct or incorrect abe identification independenty of their order or rank. We do not incude such measures as One-error which counts how many times the top-ranked abe was not a member of the predicted abe set (Li et a., 2006). Some authors refer to it Exact Match Ratio as Accuracy (Zhu, Ji, Xu, & Gong, 2005). In Section 4, we show that these two measures are not interchangeabe with respect to confusion matrix transformations; thus, they may not be equay appicabe to simiar settings. For hierarchica cassification measures (Tabe 5), we consider measures that incorporate the probem s hierarchy. These measures either evauate descendant or ancestor performance (Kiritchenko, Matwin, Nock, & Famii, 2006). We omit Tabe 4 Measures for muti-topic cassification; I is the indicator function; L i ¼ L i ½1Š;...; L i ½Š denotes a set of cass abes for x i ; L i ½jŠ ¼1ifC j is present among the abes and 0, otherwise; L c i are abes given by a cassifier, L d i are the data abes. Measure Formua Evauation focus Exact Match Ratio Labeing Fscore Retrieva Fscore Hamming Loss P n i¼1 IðLd i ¼Ld i Þ n P P 2 n j¼1 Lc i ½jŠLd i P ½jŠ i¼1 j¼1 ðlc i ½jŠþLd i ½jŠÞ n P P n 2 P i¼1 Lc i ½jŠLd i ½jŠ j¼1 n i¼1 ðlc i ½jŠþLd i ½jŠÞ P n P i¼1 j¼1 IðLc i ½jŠ Ld i ½jŠÞ n The average per-text exact cassification The average per-text cassification with partia matches The average per-cass cassification with partia matches The average per-exampe per-cass tota error

5 M. Sokoova, G. Lapame / Information Processing and Management 45 (2009) Tabe 5 Measures for hierarchica cassification: C # means subcasses of cass C; C c # denotes subcasses assigned by a cassifier; Cd # data cass abes; simiar notations appy to supercasses, which are denoted by C ". Measure Formua Evauation focus Precision # jc c # \Cd # j jc c # j Positive agreement on subcass abes w.r.t. the subcass abes given by a cassifier Reca # jc c # \Cd # j jc d # j Positive agreement on subcass abes w.r.t. the subcass abes given by data Fscore # Precision " Reca " Fscore " ðb 2 þ1þprecision#reca# b 2 Precision#þReca# jc c " \Cd " j jc c " j jc c " \Cd " j jc d " j ðb 2 þ1þprecision"precision" b 2 Precision"þPrecision" Reations between data s positive subcass abes and those given by a cassifier Positive agreement on supercass abes w.r.t. the supercass abes given by a cassifier Positive agreement on supercass abes w.r.t. the supercass abes given by data Reations between data s positive supercass abes and those given by a cassifier distance- and semantics-based measures suggested for hierarchica cassification (Bockee, Bruynooghe, Dzeroski, Ramon, & Struyf, 2002; Sun et a., 2003). These measures extend fat, non-hierarchica, measures by estimating differences and simiarities between casses. However, in these measures, acceptabe differences and simiarities are often specified by users (Costa, Lorena, Carvaho, & Freitas, 2007). Thus, the obtained resuts may be subjective and user-specific. A simiar restriction appies to depth-dependent measures, which reate casses by imposing vertica distances (Bockee et a., 2002). Data Mining has successfuy expoited the invariant properties of interestingness measures for comparison of association and cassification rues (Tan, Kumar, & Srivastava, 2004). Some invariant properties of binary cassification measures have been discussed within broader studies of the cassification of communication records (Sokoova & Lapame, 2007). In the current study, we consider new invariant properties and expand discussed measures by incuding muti-cass, muti-topic and hierarchica cassification measures. Athough the atter three types of cassification are quite popuar, their measures have not been studied to the same extent as for binary cassification measures. 4. Invariance properties of measures We focus on the abiity of a measure to preserve its vaue under a change in the confusion matrix. A measure is invariant if its vaue does not change when a confusion matrix changes, i.e. invariance indicates that the measure does not detect the change in the confusion matrix. This inabiity can be beneficia or adverse, depending on the goas. Let s consider a case when invariance to the change of tn is beneficia. Text cassification extensivey uses Precision and Reca (Sensitivity) which do not detect changes in tn when a other matrix entries remain the same. In document cassification, a arge number of unreated documents constitute a negative cass without having a singe unifying characteristic. The criterion for the performance of a cassifier is its performance on reevant documents, a we-defined unimoda positive cass, independenty of performance on the irreevant documents. Precision and Reca do not depend on tn, but ony on the correct abeing of positive exampes ðtpþ and the incorrect abeing of exampes (fp and fn). These measures provide the best perspective on a cassifier s performance for document cassification. On contrast, the same invariance for the tn change can be an adversary. Consider the cassification of human communication where negative casses are aso important. In those probems, casses often have distinct features (mae or femae) for which both positive and negative casses are we-defined. The retrieva of a positive cass, the discrimination between casses or the baance between retrieva from both casses are probem-dependent tasks. Thus, an appropriate evauation measure shoud take into account the cassification of negative exampes and refect the changes in tn when the other matrix eements stay the same. We now examine eight invariance properties ði 16k68 Þ with respect to changes of eements in a confusion matrix. A the eight changes are resuts of eementary operations on matrices: addition, scaar mutipication, interchange of rows or coumns. This set covers a reevant abe distribution changes in a cassification probem: interchange of positive and negative abes provided by data, interchange of those abes output by a cassifier, change of a singe segment (e.g., true positives), a uniform increase in the number of exampes. Henceforth, I k denotes the non-invariance of a transformation. We in detai discuss binary cassification because other evauation measures are derived from the binary confusion matrix and its performance measures. In severa parts of the discussion, we refer to data quaity. By this we understand how we exampes represent the underying notion (especiay, ease of understanding and interpretabiity), how accurate is the data, incuding its abes, and the amount of noise (based on Wang & Strong (1996)). Thereinafter, f ð½tp; fp; tn; fnšþ denotes a measure s vaue. Our caim is that the foowing invariance properties affect the appicabiity and trustworthiness of a measure ði 1 Þ Exchange of positives and negatives A measure f([tp; fp; tn; fn]) is invariant under exchange of positives and negatives if f([tp; fp; tn; fn]) = f([tn; fp; tp; fn]). tp fn tn fp! fp tn fn tp

6 432 M. Sokoova, G. Lapame / Information Processing and Management 45 (2009) This shows measure invariance with respect to the distribution of cassification resuts because it does not distinguish tp from tn and fn from fp and may not recognize the asymmetry of cassification resuts. Thus it may not be trustworthy when cassifiers are compared on data sets with different and/or unbaanced cass distributions. For exampe, invariant measures may be more appropriate for assessing the cassification of consumer reviews than for document cassification ði 2 Þ Change of true negative counts A measure f([tp; fp; tn; fn]) is invariant under the change of tn if a other matrix counts remain the same f([tp; fp; tn; fn]) = f([tp; fp; tn 0 ; fn]). tp fn tp fn! fp tn fp tn 0 This measure does not recognize the specifying abiity of cassifiers. Such evauation may be more appicabe to domains with a muti-moda negative cass taken as everything not positive. In the case of text cassification, these invariant measures are suitabe for the evauation of document cassification. If the measure is non-invariant, then it acknowedges the abiity of cassifiers to correcty identify negative exampes. In this case, it may be reiabe for comparison in domains with a we-defined, unimoda, negative cass. Non-invariant measures are preferabe for evauating communications in which there are criteria for both positive and negative resuts ði 3 Þ Change of true positive counts A measure f([tp; fp; tn; fn]) is invariant under the change of tp if a other matrix counts remain the same f([tp; fp; tn; fn]) = f([tp 0 ; fp; tn; fn]). tp fn! tp0 fn fp tn fp tn This measure does not recognize a cassifier s sensitivity. Such evauation can be compementary to other measures, but can hardy stay on its own. It may be reiabe for comparison in domains with a we-defined, unimoda, negative cass. As opposed to I 2, these invariant measures are not suitabe for the evauation of document cassification. Non-invariant measures can be used as stand aone for evauating cassification with a strong positive cass ði 4 Þ Change of fase negative counts A measure f([tp; fp; tn; fn]) is invariant under the change of fn if a other matrix counts remain the same f([tp; fp; tn; fn]) = f([tp; fp; tn; fn 0 ]) " # tp fn tp fn0! fp tn fp tn Invariance indicates measure stabiity under disagreement between the data and the negative abes assigned by a cassifier. This is especiay important for probems invoving manua abeing. If a negative cass has unreiabe abes (Nigam & Hurst (2004) argue that humans can agree on ony 74% of abes for negative opinion), an invariant measure may give miseading resuts. For non-invariant measures, its vaue s monotonicity is important. If the cassifier evauation improves when fn increases, the measure may favor a cassifier prone to fase negatives. The use of invariant and non-invariant measures shoud be decided based on probem and data characteristics ði 5 Þ Change of fase positive counts A measure f([tp; fp; tn; fn]) is invariant under the change of fp if a other matrix counts remain the same f([tp; fp; tn; fn]) = f([tp; fp 0 ; tn; fn]). tp fn tp fn! fp tn fp 0 tn An invariant measure may provide reiabe resuts when some of positive data abes are counter-intuitive. This can happen when the positive exampes have outiers that cannot be expained by the mainstream data. We ca such outiers counterexampes. A non-invariant measure may not be suitabe for data with many counterexampes. If the cassifier evauation improves when fp increases, the measure may favor a cassifier prone to fase positives. This is especiay important for probems invoving subjective abeing. Some data entries may not have consistent abes because of the difficuty of imposing

7 M. Sokoova, G. Lapame / Information Processing and Management 45 (2009) rigorous abeing rues. This can occur in the cassification of records of ong-term communications in which the data may contain a substantia number of counterexampes ði 6 Þ Uniform change of positives and negatives A measure f([tp; fp; tn; fn]) is invariant under a uniform change of positives and negatives if f ð½tp; fp; tn; fnšþ ¼ f ð½k 1 tp; k 1 fp; k 1 tn; k 1 fnšþ; k 1 1. tp fn! k 1tp k 1 fn fp tn k 1 fp k 1 tn An invariant measure is stabe with respect to the uniform increase of data size, i.e., scaar mutipication of the confusion matrix. If we expect that for different data sizes the same proportion of exampes wi exhibit positive and negative characteristics, then the invariant measure may be a better choice for the evauation of cassifiers. When a measure is non-invariant, then its appicabiity may depend on data sizes. The non-invariant measures may be more reiabe if we do not know how representative the data sampe is in terms of the proportion of positive/negative exampes ði 7 Þ Change of positive and negative coumns A measure f([tp; fp; tn; fn]) is invariant under coumns change if f ð½tp; fp; tn; fnšþ ¼ f ð½k 1 tp; k 1 fp; k 2 tn; k 2 fnšþ; k 1 k 2. tp fn! k 1tp k 2 fn fp tn k 1 fp k 2 tn Suppose that different data sizes have the same proportion of positive and negative exampes. This change in the confusion matrix is caused by changes in the proportion of positive and negative abes issued by an agorithm, i.e., the coumns are mutipied by different scaars. This may happen when the quaity of additiona data substantiay differs from the initia data sampe (e.g., the information infow can add more noise). However, an invariant measure does not show the performance change. Thus, it requires support of other measures to assess a cassifier s performance on different casses. A non-invariant measure refects on the performance of a cassifier on different casses. It is more appropriate if we can expect a change in the agorithm s performance across casses ði 8 Þ Change of positive and negative rows A measure f([tp; fp; tn; fn]) is invariant under rows change if f ð½tp; fp; tn; fnšþ ¼ f ð½k 1 tp; k 2 fp; k 2 tn; k 1 fnšþ; k 1 k 2. tp fn! k 1tp k 1 fn fp tn k 2 fp k 2 tn We again expect that different data sizes have the same proportion of positive and negative exampes. Then the change in the confusion matrix corresponds to changes of an agorithm s performance within a positive (negative) cass, i.e., the rows are mutipied by different scaars. For exampe, this may happen when a positive (negative) cass is better represented in the new data. If we expect that different data sizes exhibit same quaity of positive (negative) characteristics, then the invariant measure may be a better choice for the evauation of cassifiers. When a measure is non-invariant, its appicabiity may depend on the quaity of data casses. The non-invariant measures may be more reiabe if we do not know how representative the data sampe is in terms of the quaity of positive and negative casses, which might be the case in web-posted consumer reviews. For muti-cass cassification, we consider transformations of the confusion matrix for each cass C j. As expected, the measures retain their invariance properties regardess of micro- ormacro-averaging. For muti-topic cassification, Exact Match Ratio and Accuracy have different invariant properties. Thus, referring to Exact Match Ratio as Accuracy may be miseading. Measures used in hierarchica cassification have a somewhat imited reiabiity because they evauate the performance of a cassifier either on subcasses or on supercasses, but not on both. Thus, invariance properties shoud be assessed with respect to the cassification of subcasses for Precision # and Reca #, and supercasses for Precision " and Precision ". Tabe 6 dispays the invariance properties of the measures described in Tabes 2 5. By assessing the invariant properties of commony used measures, we show that Precision; Precision ; Precision M ; Precision # ; Precision " exhibit same invariance characteristics. Thus, we group them as Precision G for genera. Simiary, we group Reca; Reca ; Reca M ; Reca #, and Reca " as Reca G ; Fscore; Fscore ; Fscore M ; Fscore #, and Fscore " as Fscore G, and, finay, Accuracy, Average Accuracy, and Error Rate, essentiay 1-Average Accuracy, asaccuracy G. As a resut, we further consider ony those performance measures that vary in their invariance properties. Tabe 7 ists the measures and their properties. Our next step is to associate the invariant properties with particuar settings.

8 434 M. Sokoova, G. Lapame / Information Processing and Management 45 (2009) Tabe 6 Invariance properties of performance measures ði k Þ for different types of cassification tasks. + denotes invariance and non-invariance of the measure. I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 Binary cassification Tabe 2 Accuracy + + Precision Reca (Sensitivity) Fscore + + Specificity AUC + + Muti-cass cassification Tabe 3 Average Accuracy + + Error Rate + + Precision Reca Fscore + + Precision M Reca M Fscore M + + Muti-topic cassification Tabe 4 Exact Match Ratio + + Labeing Fscore + + Retrieva Fscore + Hamming Loss Hierarchica cassification Tabe 5 Precision # Reca # Fscore # + + Precision " Precision " Fscore " + + Tabe 7 Performance measures that exhibit different invariance properties. + denotes invariance and non-invariance of the measure. I 1 I 2 I 3 I 4 I 5 I 6 I 7 I 8 Accuracy G + + Precision G Reca G ðsensitivityþ Fscore G + + Specificity AUC + + Exact Match Ratio + + Labeing Fscore + + Retrieva Fscore + Hamming Loss Anaysis of invariant properties To identify simiarities among the measures, we compare them according to their invariance and non-invariance properties shown in Tabe 7. First, we present measure outiers whose properties remarkaby differ them from others. Two measures hod unique invariant properties: Precision G is the ony measure invariant under vertica scaing ði 7 Þ and Exact Match Ratio is the ony measure non-invariant under uniform scaing ði 6 Þ. Another exception is Retrieva Fscore which is sensitive to a the changes in the confusion matrix except for uniform scaing. Next we generaize on the properties: The invariance I 1 The invariance I 2 has been much discussed in the Machine Learning community, abeit from a negative point of view (Japkowicz, 2006). But we want to emphasize that this invariance makes Accuracy G and Hamming Loss robust measures for an agorithm s overa performance and insensitive to performance on a specific cass. The corresponding non-invariance I 1 means that the measures are sensitive to asymmetry of cassification. This is a we-known characteristic for Precision, Reca, Fscore and Specificity, but not for AUC, which has been introduced ony recenty in text cassification. is a we-known property of Precision, Reca, and Fscore and ess known for Labeing Fscore and Hamming Loss. Invariance under the change of tn has made them a too of choice for the evauation of document cassification. The non-invariance I 2 signifies that the use of non-invariant measures is more appropriate

9 M. Sokoova, G. Lapame / Information Processing and Management 45 (2009) The invariance I 3 The invariance I 4 The invariance I 5 The invariance I 6 The invariance I 7 The invariance I 8 on data with a unimoda negative cass than with a muti-moda one. This impication is more important for AUC than for Specificity. The atter is usuay used in combination with other measures, whereas the former might be appied separatey. so far eudes thorough studies. Measures are expected to be non-invariant under the change of tp. The non-invariant measures are used for evauating cassification with a strong positive cass, such as for the evauation of document cassification. Ony Specificity and Hamming Loss do not measure the tp change. Specificity was purposefuy designed to avoid tp. The non-invariance of Specificity and Hamming Loss suggests they may be used in a combination with other measures. These two measures may be reiabe for comparison in domains with a we-defined, unimoda, negative cass. under change in fn indicates that Precision, Specificity, and Exact Match Ratio may be more reiabe when manua abeing foows rigorous rues for a negative cass. In the absence of such rues, disagreement between the data abes and the negative abes assigned by a cassifier can depend on subjective factors and fuctuate. Under such conditions, an invariant measure may give miseading resuts. A the I 4 measures discussed above are monotone decreasing when fn increase hence, wi not favor a cassifier prone to fase negatives. under fp change indicates that Reca and Exact Match Ratio may provide reasonaby conservative estimate when a positive cass has counterexampes, i.e., outiers not expained by the mainstream positive exampes. The other eight measures are non-invariant. However, they are monotone decreasing when fp increase, hence, they wi not favor a cassifier prone to fase positives. under uniform scaing hods for a the measures except Exact Match Ratio. The nine invariant measures adapt to different sizes of data. The non-invariance of Exact Match Ratio indicates that its resuts may not be comparabe when obtained on data of widey different sizes. under the scaar coumn change hods ony for Precision. This supports a common practice of combining Precision with other measures when assessing cassifier performance. The combination assures that the evauation is ess dependent on the data quaity. A the other measures are non-invariant under the scaar coumn change. Thus, they are more reiabe if an agorithm s performance is expected to change across casses with new data. under the scaar row change indicates that Reca, Specificity, and AUC may be a better choice for the evauation of cassifiers if different data sizes exhibit same quaity of positive (negative) characteristics. Exampes are simuated or generated data under the same distribution. The other measures are non-invariant. They may be more reiabe if the representative power of positive and negative casses is uncertain. Invariance with respect to the matrix transformations is especiay important because it connects evauation measures to particuar earning settings. We now summarize the appicabiity of these measures to two subfieds of text cassification: document cassification and cassification of human communications. One might be tempted to appy Fscore measures on any text cassification evauation. However, various cassification probems exhibit different characteristics which may require different evauation measures. Based on our anaysis, we propose the foowing. Since document cassification data is often highy imbaanced, reevant documents constitute a sma we-defined positive cass, but the rest is a heterogeneous negative cass buit from non-reevant documents as everything non-positive. Presence of a negative cass that compements the positive cass favors the use of the Fscore measures. In many such probems, exampes of the positive cass remain the same and the cass keeps its modaity, whereas exampes of the negative cass change. Since the Fscore measures invariance under the change of correcty cassified negative exampes ði 2 Þ prevents drastic changes, they wi be ess sensitive to changes in the negative cass. Cassification of human communications is most often represented by sentiment cassification appied to coections of free form texts containing product evauations. The number and ratio of positive and negative exampes depends on the popuarity of a particuar product. If reviewers have strong ikes and disikes, then both casses have we-defined characteristics. In this case, Area Under the Curve (AUC) may provide a more reiabe cassifier evauation than Precision and Reca. Since AUC is non-invariant under the change of correcty cassified negative exampes ði 2 Þ, it wi detect possibe changes in the negative cass better than Fscore measures. For other types of cassification of communications in socia activities, other measure combinations might aso be suitabe. Poitica debates and eectronic negotiations are exampes of such communications. Their data can exhibit a unimoda negative cass and a arge number of counterexampes. In poitica debates, counterexampes are records that praise the discussed matter, but vote against it at the end, either because of a hidden motive or randomness of behavior (Sokoova & Lapame, 2007). In such cases, which are difficut even for human cassification, Accuracy, with its invariance under the exchange of positives and negatives cassification ði 1 Þ, and Precision, with its invariance under the change of fase negative exampes ði 5 Þ, may be used for a reiabe evauation of cassifiers. 6. Concusion and future work In this study, we have anayzed twenty four performance measures used in the compete spectrum of Machine Learning cassification tasks: binary, muti-cass, muti-abeed, and hierarchica. Effects of changes in the confusion matrix on severa

10 436 M. Sokoova, G. Lapame / Information Processing and Management 45 (2009) we-known measures have been studied. In a the cases, we have shown that the evauation of cassification resuts can depend on the invariance properties of the measures. A few cases required that we additionay considered monotonicity of the measure. These properties have aowed us to make fine distinctions in the reations between the measures. One way to insure a reiabe evauation is to empoy a measure corresponding to the expected quaity of the data, e.g., representativeness of cass distribution, reiabiity of cass abes, uni- and muti-modaity of casses. To match measures with the data characteristics, we constructed the measure invariance taxonomy with respect to a reevant abe distribution changes in a cassification probem. We suppemented the forma discussion by anayzing the appicabiity of performance measures on different subfieds of text cassification. We have shown that the cassification of human communications differs from document cassification, and thus that these two types of text cassification may require different performance measures. Our study has deat with measures used in text cassification but it coud be extended to other anguage appications of Machine Learning. The next step woud be to study measures used in Machine Transation. This wi consideraby expand the measure ist. Appicabiity of the measures to traditiona Natura Language Processing tasks, e.g., word sense disambiguation, parsing, is another topic of considerabe interest. It woud aso be usefu to anayze in more detais a measure s monotonicity, especiay its behavior with respect to extreme cassification resuts, such as when the abes provided by the data and a cassifier are independent. Person authentication probems, in which the appropriate measures are a fase acceptance rate and a fase rejection rate (Bengio et a., 2005), is another exampe of possibe appications. Acknowedgments This work has been funded by the Natura Sciences and Engineering Research Counci of Canada and the Ontario Centres of Exceence. We thank Eiott Mackovitch for fruitfu suggestions on an eary draft. We thank anonymous reviewers for hepfu comments. References Asuncion, A., & Newman, D. (2007). UCI Machine Learning Repository. Irvine, CA: University of Caifornia, Schoo of Information and Computer Science. < Bengio, S., Mariéthoz, J., & Keer, M. (2005). The expected performance curve. In Proceedings of the ICML 05 workshop on ROC anaysis in machine earning (pp ). Bitzer, J., Dredze, M., & Pereira, F. (2007). Biographies, boywood, boom-boxes and benders: Domain adaptation for sentiment cassification. In Proceedings of the 45th annua meeting of the association of computationa inguistics (pp ). Association for Computationa Linguistics. Bockee, H., Bruynooghe, M., Dzeroski, S., Ramon, J., & Struyf, J. (2002). Hierarchica muti-cassification. In KDD-2002: Workshop on muti-reationa data mining (pp ). Bobicev, V., & Sokoova, M. (2008). An effective and robust method for short text cassification. In Proceedings of the association for the advancement of artificia inteigence (AAAI-2008) (pp ). AAAI Press. Cohen, J. (1988). Statistica power anaysis for the behaviora sciences. Hisdae, NJ: Lawrence Erbaum. Costa, E., Lorena, A., Carvaho, A., & Freitas, A. (2007). A review of performance evauation measures for hierarchica cassifiers. In Proceedings of the AAAI 2007 workshop Evauation methods for machine earning (pp. 1 6). Demsar, J. (2006). Statistica comparisons of cassifiers over mutipe data sets. Journa of Machine Learning Research, 7, Dietterich, T. (1998). Approximate statistica tests for comparing supervised cassification earning agorithms. Neura Computation, 10, Duda, R. O., & Hart, P. E. (1973). Pattern cassification and scene anaysis. John Wiey & Sons. Eisner, R., Pouin, B., Szafron, D., Lu, P., & Greiner, R. (2005). Improving protein function prediction using the hierarchica structure of the gene ontoogy. In Proceedings of IEEE symposium on computationa inteigence in bioinformatics and computationa bioogy (pp. 1 10). Gamon, M., Aue, A., Corston-Oiver, S., & Ringger, E. (2005). Puse: Mining customer opinions from free text. In Proceedings of the 6th internationa symposium on inteigent data anaysis (IDA 2005) (pp ). Goutte, C., & Gaussier, E. (2005). A probabiistic interpretation of precision, reca and f-score, with impication for evauation. In Proceedings of 27th European conference on IR research (ECIR 2005) (pp ). Hersh, W., Buckey, C., Leone, T., & Hickam, D. (1997). OHSUMED: An interactive retrieva evauation and new arge test coection for research. In Proceedings of the 17th annua internationa ACM SIGIR conference on research and deveopment in information retrieva (SIGIR-97) (pp ). Huang, J., & Ling, C. (2007). Constructing new and better evauation measures for machine earning. In Proceedings of the 20th internationa joint conference on artificia inteigence (IJCAI 2007) (pp ). Issebacher, K., & Braunwad, E. (1994). Harrison s principes of interna medicine. McGraw-Hi. Japkowicz, N. (2006). Why question machine earning evauation methods? In Proceedings of the AAAI 06 workshop on evauation methods for machine earning (pp. 6 11). Kazawa, H., Izumitani, T., Taira, H., & Maeda, E. (2005). Maxima margin abeing for muti-topic text categorization. In Advances in neura information processing systems (NIPS 04), (Vo. 17, pp ). Kiritchenko, S., Matwin, S., Nock, R., & Famii, A. F. (2006). Learning and evauation in the presence of cass hierarchies: Appication to text categorization. In Proceedings of the 19th Canadian conference on AI (AI 2006) (pp ). Lachiche, N., & Fach, P. A. (2003). Improving accuracy and cost of two-cass and muti-cass probabiistic cassifiers using ROC curves. In Proceedings of ICML 2003 (pp ). Langey, P. (1996). Eements of machine earning. San Francisco, Caif: Morgan Kaufmann. Li, T., Zhang, C., & Zhu, S. (2006). Empirica studies on muti-abe cassification. In Proceedings of the 18th IEEE internationa conference on toos with artificia inteigence (pp ). Marchand, M., & Shawe-Tayor, J. (2002). The set covering machine. Journa of Machine Learning Research, 3, Mewes, H.-W., Abermann, K., Heumann, K., Lieb, S., & Pfeiffer, F. (1997). MIPS: A database for protein sequences, homoogy data and yeast genome information. Nuceic Acids Research, 25(1), Mitche, T. (1997). Machine earning. McGraw-Hi. Nigam, K., & Hurst, M. (2004). Towards a robust metric of opinion. In Proceedings of the AAAI spring symposium on exporing attitude and affect in text (pp ). AAAI Press. Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up? Sentiment cassification using machine earning techniques. In Proceedings of empirica methods of natura anguage processing (EMNLP 02) (pp ).

11 M. Sokoova, G. Lapame / Information Processing and Management 45 (2009) Rijsbergen, C. (1979). Information retrieva (2nd ed.). London: Butterworths. Saton, G., & McGi, M. (1993). Introduction to modern information retrieva. New York: McGraw-Hi. Sazberg, S. L. (1999). On comparing cassifiers: A critique of current research and methods. Data Mining and Knowedge Discovery, 1, Sasaki, Y., Rea, B., & Ananiadou, S. (2007). Muti-topic aspects in cinica text cassification. In Proceedings of the 2007 IEEE internationa conference on bioinformatics and biomedicine (pp ). IEEE Computer Society. Sebastiani, F. (2002). Machine earning in automated text categorization. ACM Computing Surveys, 34(1), Shawe-Tayor, J., & Christianini, N. (2004). Kerne methods for pattern anaysis. Cambridge University Press. Snyder, B., & Barziay, R. (2007). Database-text aignment via structured mutiabe cassification. In Proceedings of the internationa joint conference on artificia inteigence (IJCAI-2007) (pp ). Sokoova, M., Japkowicz, N., & Szpakowicz, S. (2006). Beyond accuracy, F-score and ROC: A famiy of discriminant measures for performance evauation. In Proceedings of the ACS Austraian joint conference on artificia inteigence (pp ). Sokoova, M., & Lapame, G. (2007). Performance measures in cassification of human communication. In Proceedings of the 20th Canadian conference on artificia inteigence ( AI 2007) (pp ). Sun, A., Lim, E.-P., & Ng, W.-K. (2003). Performance measurement framework for hierarchica text cassification. Journa of the America Society for Information Science and Technoogy, 54(11), Tan, P., Kumar, V., & Srivastava, J. (2004). Seecting the right objective measure for association anaysis. Information Systems, 29(4), Tikk, D., & Biró, G. (2003). Experiments with muti-abe text cassifier on the Reuters coection. In Proceedings of the internationa conference on computationa cybernetics (ICCC 03) (pp ). Thomas, M., Pang, B., & Lee, L. (2006). Get out the vote: Determining support or opposition from congressiona foor-debate transcripts. In: Proceedings of the 2006 conference on empirica methods in natura anguage processing (pp ). Wang, R., & Strong, D. (1996). Beyond accuracy: What data quaity means to data consumers. Journa of Management Information Systems, 12(4), Wison, T., Wiebe, J., & Hwa, R. (2006). Recognizing strong and weak opinion causes. Computationa Inteigence, 22(2), Yang, Y. (1999). An evauation of statistica approaches to text categorization. Information Retrieva, 1, Youden, W. (1950). Index for rating diagnostic tests. Cancer, 3, Zhu, S., Ji, X., Xu, W., & Gong, Y. (2005). Muti-abeed cassification using maximum entropy method. In Proceedings of the 28th annua internationa ACM SIGIR conference on Research and deveopment in information retrieva (pp ).