Cluster-based Sampling Approaches to Imbalanced Data Distributions for PAKDD 2007 Competition

Transcription

1 Cluster-based Samplng Approaches to Imbalanced Data Dstrbutons for PAKDD 2007 Competton Yue-Sh Lee and Show-Jane Yen Department of Computer Scence and Informaton Engneerng, Mng Chuan Unversty 5 The-Mng Rd., Gwe Shan Dstrct, Taoyuan County 333, Tawan {sjyen,leeys}@mcu.edu.tw Abstract. For classfcaton problem, the tranng data wll sgnfcantly nfluence the classfcaton accuracy. When the data set s hghly unbalanced, classfcaton algorthms tend to degenerate by assgnng all cases to the most common outcome. Hence, t s mportant to select the sutable tranng data for classfcaton n the mbalanced class dstrbuton problem. In ths paper, we propose cluster-based under-samplng approaches for selectng the representatve data as tranng data to mprove the classfcaton accuracy n the mbalanced class dstrbuton envronment,.e., PAKDD competton data set. The CART (Classfcaton and Regresson Tree) classfcaton algorthm s consdered. The expermental results show that our cluster-based under-samplng approaches can perform the tradtonal approaches. 1 Introducton The classfcaton technques usually assume that the tranng samples are unformly-dstrbuted between dfferent classes. A classfer performs well when the classfcaton technque s appled to a dataset evenly dstrbuted among dfferent classes. However, many datasets n real applcatons nvolve mbalanced class dstrbuton problem [5, 7]. The mbalanced class dstrbuton problem occurs whle there are much more samples n one class than the other class n a tranng dataset. In an mbalanced dataset, the majorty class has a large percent of all the samples, whle the samples n mnorty class just occupy a small part of all the samples. In ths case, a classfer usually tends to predct that samples have the majorty class and completely gnore the mnorty class. One smple method of under-samplng s to select a subset of MA randomly and then combne them wth MI as a tranng set, whch s called random under-samplng approach. Several advanced researches are proposed to make the selectve samples more representatve. The under-samplng approach based on dstance [7] uses dstnct modes: the nearest, the farthest, the average nearest, and the average farthest dstances between MI and MA, as four standards to select the representatve samples from MA. For every mnorty class sample n the dataset, the frst method nearest calculates the dstances between all majorty class samples and the mnorty class

2 samples, and selects k majorty class samples whch have the smallest dstances to the mnorty class sample. If there are n mnorty class samples n the dataset, the nearest approach would fnally select k n majorty class samples (k 1). However, some samples wthn the selected majorty class samples mght duplcate. Smlar to the nearest approach, the farthest approach selects the majorty class samples whch have the farthest dstances to each mnorty class samples. For every majorty class samples n the dataset, the thrd method average nearest calculates the average dstance between one majorty class sample and all mnorty class samples. Ths approach selects the majorty class samples whch have the smallest average dstances. The last method average farthest s smlar to the average nearest approach; t selects the majorty class samples whch have the farthest average dstances wth all the mnorty class samples. The above under-samplng approaches based on dstance n [7] spend a lot of tme selectng the majorty class samples n the large dataset, and they are not effcent n real applcatons. In 2003, J. Zhang and I. Man [6] presented the compared results wthn four nformed under-samplng approaches and random under-samplng approach. The frst method NearMss-1 selects the majorty class samples whch are close to some mnorty class samples. In ths method, majorty class samples are selected whle ther average dstances to three closest mnorty class samples are the smallest. The second method NearMss-2 selects the majorty class samples whle ther average dstances to three farthest mnorty class samples are the smallest. The thrd method NearMss- 3 take out a gven number of the closest majorty class samples for each mnorty class sample. Fnally, the fourth method Most dstant selects the majorty class samples whose average dstances to the three closest mnorty class samples are the largest. The fnal expermental results n [6] showed that the NearMss-2 approach and random under-samplng approach perform the best. In ths paper, we study the effects of under-samplng [1, 3, 6] on the CART (Classfcaton and Regresson Tree) classfcaton algorthm and propose some new under-samplng approaches based on clusterng, such that the nfluence of mbalanced class dstrbuton can be decreased and the accuracy of predctng the mnorty class can be ncreased. 2 Our Approaches In ths secton, we present our approach SBC (under-samplng Based on Clusterng) whch focuses on the under-samplng approach and uses clusterng technques to solve the mbalanced class dstrbuton problem. Our approach frst clusters all the tranng samples nto some clusters. The man dea s that there are dfferent clusters n a dataset, and each cluster seems to have dstnct characterstcs. If a cluster has more majorty class samples and less mnorty class samples, t wll behave lke the majorty class samples. On the opposte, f a cluster has more mnorty class samples and less majorty class samples, t doesn t hold the characterstcs of the majorty class samples and behaves more lke the mnorty class samples. Therefore, our approach SBC selects a sutable number of majorty class samples from each cluster by

3 consderng the rato of the number of majorty class samples to the number of mnorty class samples n the cluster. 2.1 Under-samplng based on clusterng Assume that the number of samples n the class-mbalanced dataset s N, whch ncludes majorty class samples (MA) and mnorty class samples (MI). The sze of the dataset s the number of the samples n ths dataset. The sze of MA s represented as Sze MA, and Sze MI s the number of samples n MI. In the class-mbalanced dataset, Sze MA s far larger than Sze MI. For our under-samplng method SBC, we frst cluster all samples n the dataset nto K clusters. The number of majorty class samples and the number of mnorty class samples n the th cluster (1 K) are Sze MA and Sze MI, respectvely. Therefore, the rato of the number of majorty class samples to the number of mnorty class samples n the th cluster s Sze MA / SzeMI. If the rato of Sze MA to Sze MI n the tranng dataset s set to be m:1, the number of selected majorty class samples n the th cluster s shown n expresson (1): SzeMA SzeMI SSze MA = ( m SzeMI) (1) K Sze MA Sze = 1 MI In expresson (1), m SzeMI s the total number of selected majorty class samples that we suppose to have n the fnal tranng dataset. K Sze MA s the total = 1 SzeMI rato of the number of majorty class samples to the number of mnorty class samples n all clusters. Expresson (1) determnes that more majorty class samples would be selected n the cluster whch behaves more lke the majorty class samples. In other words, SSze MA s larger whle the th cluster has more majorty class samples and less mnorty class samples. After determnng the number of majorty class samples whch are selected n the th cluster, 1 K, by usng expresson (1), we randomly choose majorty class samples n the th cluster. The total number of selected majorty class samples s m Sze MI after mergng all the selected majorty class samples n each cluster. At last, we combne the whole mnorty class samples wth the selected majorty class samples to construct a new tranng dataset. Table 1 shows the steps for our under-samplng approach. For example, assume that an mbalanced class dstrbuton dataset has totally 1100 samples. The sze of MA s 1000 and the sze of MI s 100. In ths example, we cluster ths dataset nto three clusters. Table 2 shows the number of majorty class

4 samples Sze MA, the number of mnorty class samples Sze MI, and the rato of Sze MA to Sze MI for the th cluster. Table 1. The structure of the under-samplng based on clusterng approach SBC Step1. Determne the rato of Sze MA to Sze MI n the tranng dataset. Step2. Cluster all the samples n the dataset nto some clusters. Step3. Determne the number of selected majorty class samples n each cluster by usng expresson (1), and then randomly select the majorty class samples n each cluster. Step4. Combne the selected majorty class samples and all the mnorty class samples to obtan the tranng dataset. Table 2. Cluster descrptons Cluster ID Number of majorty Number of mnorty Sze MA / SzeMI class samples class samples /10= /50= /40=5 Assume that the rato of Sze MA to Sze MI n the tranng data s set to be 1:1, n other words, there are 100 selected majorty class samples and the whole 100 mnorty class samples n ths tranng dataset. The number of selected majorty class samples n each cluster can be calculated by expresson (1). Table 3 shows the number of selected majorty class samples n each cluster. We fnally select the majorty samples randomly from each cluster and combne them wth the mnorty samples to form the new dataset. Table 3. The number of selected majorty class samples n each cluster Cluster ID The number of selected majorty class samples / (50+6+5) = / (50+6+5) = / (50+6+5)= Under-samplng based on clusterng and dstances In SBC method, all the samples are clustered nto several clusters and the number of selected majorty class samples s determned by expresson (1). Fnally, the majorty class samples are randomly selected from each cluster. In ths secton, we propose other two under-samplng methods, whch are based on SBC approach. The dfference between the two proposed under-samplng methods and SBC method s the way to select the majorty class samples from each cluster. For the two proposed methods, the majorty class samples are selected accordng to the dstances between

5 MaxA MnA the majorty class samples and the mnorty class samples n each cluster. Hence, the dstances between samples wll be computed. For a contnuous attrbute, the values of all samples for ths attrbute need to be normalzed n order to avod the effect of dfferent scales for dfferent attrbutes. For example, suppose A s a contnuous attrbute. In order to normalze the values of attrbute A for all the samples, we frst fnd the maxmum value Max A and the mnmum value Mn A of A for all samples. To le an attrbute value a n between 0 to 1, a a MnA s normalzed to. For a categorcal or dscrete attrbute, the dstance between two attrbute values x 1 and x 2 s 1 (.e. x 1 -x 2 =1) whle x 1 s not equal to x 2, and the dstance s 0 (.e. x 1 -x 2 =0) whle they are the same. X Assume that there are N attrbutes n a dataset and represents the value of attrbute A n sample X, for 1 N. The Eucldean dstance between two samples X and Y s shown n expresson (2). V dstance ( X, Y ) = N X Y 2 ( V V ) = 1 (2) The two approaches we proposed n ths secton frst cluster all samples nto K (K 1) clusters as well, and determne the number of selected majorty class samples for each cluster by expresson (1). For each cluster, the representatve majorty class samples are selected n dfferent ways. The frst method SBCMD (Samplng Based on Clusterng wth Most Dstant) selects the majorty class samples whose average dstances to M closest mnorty class samples n the th cluster are the farthest. The second method, whch s called SBCMF (Samplng Based on Clusterng wth Most Far), selects the majorty class samples whose average dstances to all mnorty class samples n the cluster are the farthest. 3 Data Preprocessng Because PAKDD competton data set contans many noses such as null or mssng value and wrong value. Therefore, we need to perform data preprocessng. Frst of all, we compute the null value dstrbuton for all attrbutes. After ths step, we remove 6 attrbutes (AMEX_CARD, DINERS_CARD, VISA_CARD, MASTERCARD, RETAIL_CARDS, DISP_INCOME_CODE) from the tranng data set due to ther hgh percentage of null value. Then, we encode the attrbute values for several attrbutes to get a reasonable value. For example, we encode the attrbute value 98 to 0 and 99 to 12 n B_ENQ_L1M attrbute. After encode each attrbute, we calculate the attrbute sgnfcance for each attrbute and fnally only select 14 attrbutes for tranng our classfcaton model. They are lsted below.

6 B_ENQ_L6M_GR3 B_ENQ_L12M_GR3 CURR_RES_MTHS AGE_AT_APPLICATION B_ENQ_L3M B_ENQ_L1M B_ENQ_L6M CURR_EMPL_MTHS B_ENQ_L12M_GR2 B_ENQ_L6M_GR2 PREV_RES_MTHS NBR_OF_DEPENDANTS B_ENQ_L6M_GR2C Number of Bureau Enqures n the Last 6 Months for Mortgages Number of Bureau Enqures n the Last 12 Month for Mortgages Total Number of Months at Current Resdence Customer's Age (n Years) upon Applcaton Number of Bureau Enqures n the Last 3 months Number of Bureau Enqures n the Last 1 months Number of Bureau Enqures n the Last 6 months Total Number of Months at Current Employment Number of Bureau Enqures n the Last 12 Month for Loans Number of Bureau Enqures n the Last 6 Months for Loans Total Number of Months at Prevous Address Number of Dependents Number of Bureau Enqures n the Last 6 Months for Loans ANNUAL_INCOME_RANGE Number of Bureau Enqures n the Last 6 months 4 Expermental Results We do several experments and the best one s lsted below. Fg. 1 Expermental Results

7 5 Concluson In a classfcaton task, the effect of mbalanced class dstrbuton problem s often gnored. Many studes [2, 4] focused on mprovng the classfcaton accuracy but dd not consder the mbalanced class dstrbuton problem. Hence, the classfers whch are constructed by these studes lose the ablty to correctly predct the correct decson class for the mnorty class samples n the datasets whch the number of majorty class samples are much greater than the number of mnorty class samples. Many real applcatons, lke rarely-seen dsease nvestgaton, credt card fraud detecton, and nternet ntruson detecton always nvolve the mbalanced class dstrbuton problem. It s hard to make rght predctons on the customers or patents who that we are nterested n. In ths study, we propose cluster-based under-samplng approaches to solve the mbalanced class dstrbuton problem by usng CART (Classfcaton and Regresson Tree). The expermental results show that our cluster-based under-samplng approaches can perform the tradtonal approaches. References 1. Chawla, N. V.: C4.5 and Imbalanced Datasets: Investgatng the Effect of Samplng Method, Probablstc Estmate, and Decson Tree Structure. Proceedngs of the ICML 03 Workshop on Class Imbalances (2003). 2. Caragea, D., Cook, D., Honavar, V.: Ganng Insghts nto Support Vector Machne Pattern Classfers Usng Projecton-Based Tour Methods. Proceedngs of the KDD Conference, San Francsco, CA (2001) Drummond, C., Holte, R. C.: C4.5, Class Imbalance, and Cost Senstvty: Why Under- Samplng Beats Over-Samplng. Proceedngs of the ICML 03 Workshop on Learnng from Imbalanced Datasets (2003). 4. del-hoyo, R., Buldan, D., Marco, A.: Supervsed Classfcaton wth Assocatve SOM. Lecture Notes n Computer Scence, Vol.2686 (2003) Japkowcz, N.: Concept-learnng n the Presence of Between-class and Wthn-class Imbalances. Proceedngs of the Fourteenth Conference of the Canadan Socety for Computatonal Studes of Intellgence (2001) Zhang, J., Man, I.: knn Approach to Unbalanced Data Dstrbutons: A Case Study Involvng Informaton Extracton. Proceedngs of the ICML 2003 Workshop on Learnng from Imbalanced Datasets (2003). 7. Chy, Y.M.: Classfcaton Analyss Technques for Skewed Class Dstrbuton Problems, Master Thess, Department of Informaton Management, Natonal Sun Yat-Sen Unversty (2003).