Hgh Dmensonal Data Analysis proposeations

Transcription

1 Yuntao Qan, Xaoxu Du, and Q Wang Sem-supervsed Herarhal Clusterng Analyss for Hgh Dmensonal Data Sem-supervsed Herarhal Clusterng Analyss for Hgh Dmensonal Data Yuntao Qan, Xaoxu Du, and Q Wang College of Computer Sene, Zheang Unversty Hangzhou 30027, Chna ytqan@zu.edu.n, Duxaoxu68@hotmal.om, angq@hz.n Abstrat In many data mnng tasks, there s a large supply of unlabeled data but lmted labeled data sne t s expensve generated. herefore, a number of sem-supervsed lusterng algorthms have been proposed, but fe of them are speally desgned for hgh dmensonal data. Hgh dmensonalty s a dffult hallenge for lusterng analyss due to the nherent sparse dstrbuton, and most of popular lusterng algorthms nludng sem-supervsed ones ll be nvald n hgh dmensonal spae. In ths paper, a sem-supervsed herarhal lusterng algorthm for hgh dmensonal data s proposed, hh s based on the ombnaton of semsupervsed lusterng and dmensonalty reduton. In order to aheve hgh harmony beteen dmensonalty reduton and nherent luster struture deteton, the number of dmensons s redued sequentally as the lusters are gradually formed n the herarhal lusterng proedure. he expermental results sho the effetveness of our method. Keyords: Clusterng analyss, dmensonalty reduton, sem-supervsed learnng, hgh dmensonal spae I. Introduton As the data mnng problem often nvolves a large amount of unlabeled data and a relatve small amount of labeled data, onsequently, learnng th both labeled and unlabeled data that s knon as sem-supervsed learnng, has beome a top of sgnfant reent nterest [,2]. In ths paper, e fous on sem-supervsed lusterng n hh labeled data s employed to mprove the lusterng performane. Some sem-supervsed lusterng algorthms have been presented [3,4], but fe of them are speally desgned for hgh dmensonal data. In most of lusterng applatons suh as mage proessng, pattern reognton, omputatonal bology, and eb nformaton retreval, the data need to be proessed are alays n hghdmensonal spae. Hgh dmensonalty not only makes omputatonal ost very expensve, but also makes many popular lusterng algorthms nvald due to sparse 54

2 Internatonal ournal of Informaton ehnology, Vol.2, No.3, 2006 densty dstrbuton. herefore, the urse of dmensonalty must be gven a sgnfant amount of researh attenton n sem-supervsed lusterng. Dmensonalty reduton s thought as an effetve ay to solve hgh dmensonal problem. In most ases, dmensonalty reduton s arred out as a preproessng step, for example, lnear/nonlnear dsrmnant analyss (LDA/NDA and prnpal omponent analyss (PCA are popular used n lassfaton and lusterng problems respetvely [5]. Hoever, PCA does not alays enhane the qualty of lusterng sne PCA based dmensonalty reduton may destroy the nherent luster struture n most ases. Reently spetral embeddng and manfold learnng suh as spetral lusterng [6], Neghbourhood omponent analyss [7], Isomap [8], and loally lnear embeddng [9], are used nstead of PCA, hh an estmate ntrns dmensons that ell reflet lusterng strutures. he foundaton of these methods s dependent on the estmaton of loal densty dstrbuton, but ths estmaton s very dffult n hghdmensonal spae. herefore, most of studes of hgh-dmensonal data lusterng alays use more omplated shemes to norporate dmensonalty reduton nto lusterng proedure nstead of usng dmensonalty reduton as preproessng step,.e., to problems of parttonng a data set and fndng a redued dmensonaltes are solved at the same tme. Amng at to prnpal lusterng methods of parttonng lusterng and herarhal lusterng, the exstng dmensonalty reduton based lusterng methods an be also dvded nto these to lasses. Fredman and Meulman propose a ne parttonng rteron based on subsets of attrbutes, n hh the dstane beteen a par of ponts s measured on a subset of attrbutes rather than on all of them smultaneously [0]. Roth and Lange use a lassal EM algorthm to ombne feature seleton and lusterng, n hh the E-step s estmated by the fuzzy/probablst lusterng based on Gaussan mxture model, and the M-step s formulated as a LDA problem here the fuzzy/probablst membershp s used as the nformaton of labelng []. Dng et al use luster membershp as the brdge to onnet the lusters defned n the redued dmensonal spae and those defned n the full dmensonal spae [2]. Aggaral and Yu propose a generalzed proeted lusterng method for hgh dmensonal data [3], hh fnds the best proeton for eah luster n suh a ay that the greatest amount of smlarty among the ponts n that luster an be deteted. Herarhal lusterng tehnque s used n ths method as a man frame, but the dea of parttonng lusterng s also used to alays assoate a urrent luster th eah of the ponts. In general, parttonng tehnque based methods are very senstve to data dstrbuton. If a dstrbuton does not onform to the presumed prototype, t ll beome less effetve. And herarhal tehnque based methods are more versatle sne they an dentfy some rregularly shaped or non-globular lusters, but ther omputatonal ost s very hgh for large data set. Compared th the above methods, our sem-supervsed herarhal lusterng algorthm for hgh dmensonal data has to dstnt features: Agglomeratve herarhal lusterng tehnque s used to deal th the luster strutures th non-spher shapes. he labeled examples are used as the ntal seeds of luster. In lusterng proedure, the unlabeled examples that are losest 55

3 Yuntao Qan, Xaoxu Du, and Q Wang Sem-supervsed Herarhal Clusterng Analyss for Hgh Dmensonal Data to the exsted lusters, are assgned to the orrespondng lusters and gven a spef luster label, hh s dfferent from the unsupervsed herarhal lusterng algorthm that ntalzes eah unlabeled examples as a luster, then suessvely agglomerates these lusters. Sne e only need to alulate the dstane beteen the resdual unlabeled examples and the exsted lusters n eah teratve phase, the omputatonal ost an be saved. In lusterng proedure, as the unlabeled examples are gradually transformed to the labeled examples by agglomeratve operatons, the number of dmensons s also gradually redued by dmensonalty reduton algorthm. Clusterng and dmensonalty reduton are run as an teratve sequental proedure. It s based on the dea that lusterng and dmensonalty reduton are dependent on eah other, and they are an ntegrated system. Sne both labeled and unlabeled examples exst n lusterng proedure, a sem-supervsed dmensonalty reduton algorthm s proposed, by hh the nformaton ontaned n both of the labeled and unlabeled examples an be utlzed to determne a redued subspae. hs paper s organzed as follos. In the next seton, our method s ntrodued n detal. he empral results are dsussed n seton III. Fnally, e present the onlusons and summary n seton IV. II. Algorthms he key ssue of hgh dmensonal lusterng s formulatng a sutable mehansm to keep funtonal harmony beteen dmensonalty reduton and lusterng. For lusterng th unlabeled examples, although some unsupervsed dmensonalty reduton or feature extraton methods suh as PCA, ndependent omponent analyss, fator analyss an preserve the man nformaton of a data set aordng to ther respetve fouses, but ther rtera are alays not onsst th the rteron of lusterng,.e., nhert luster strutures are destroyed severely n many ases. We thnk that some nformaton onernng lass labelng may be requred as a brdge to reah funtonal harmony for lusterng problem. Obvously, the luster membershp an be utlzed nstead of lass labelng, but there s a dlemma hether to do lusterng frst or do dmensonalty reduton frst. In order to solve ths problem, e propose a sem-supervsed herarhal lusterng algorthm, n hh the dmensonalty s gradually redued th a sem-supervsed dmensonalty reduton algorthm as lusters are gradually formed. he valdty of our method s based on the fat that the mportane of the labelng nformaton beomes more and more mportant as number of dmensons s gradually redued. A. Sem-Supervsed Dmensonalty Reduton d Gven a data set D = {( x, y, L, ( x n, y n } R {lass/luster} and n s the total number of data, d s the number of orgnal dmensons, x s the data vetor, and y s the lass label. D an be dvded nto to sets D= { L U}, here L s the set of labeled data th knon lass labels and U s the set of unlabeled data. he semsupervsed dmensonalty reduton algorthm s an optmzaton based on these to 56

4 Internatonal ournal of Informaton ehnology, Vol.2, No.3, 2006 types of data, and the rteron of optmzaton s a ombnaton of LDA and PCA that are popular dmensonalty reduton tehnques used for labeled and unlabeled data respetvely. For the set of labeled examples L, LDA s gven by a lnear transformaton matrx d l W R maxmzng the so-alled Fsher rteron W SbW F ( W = ( W S W S S b = p ( m m0 ( m m0 = = pe{( x m ( x m = 0 = p m = x C } m (4 Eah example x n L belongs to a spef lass C, and s the number of lasses; m and p are the mean vetor and a pror probablty of lass C respetvely; S b and S are the beteen-lass and thn-lass satter matres respetvely. he purpose of LDA s to maxmze the beteen-lass satter hle smultaneously mnmzng the thn-lass satter n the redued l-dmensonal spae. For the set of unlabeled examples U, PCA derves a relatve small number of deorrelated lnear ombnatons (prnpal omponents of a set of random varables hle retanng as muh of the nformaton from the orgnal varables as possble. If e let the mean of the data set be zero, PCA s gven by a lnear transform d l W R maxmzng the ovarane matrx based rteron ( W = W R W (5 P xx R xx = E{ xk xk xk U} (6 PCA an be onverted to an egenvalue problem of R xx, and the l prnpal omponents n W are the egenvetors orrespondng to the l largest egenvalues of R. xx If there exst a hybrd data set th labeled and unlabled examples, a good dmensonalty reduton method should make full use of them together. herefore, the follong rteron an be naturally proposed for the hybrd data set D H W ( W = W SbW + αw S W R xx W (2 (3 (7 57

5 Yuntao Qan, Xaoxu Du, and Q Wang Sem-supervsed Herarhal Clusterng Analyss for Hgh Dmensonal Data W an be obtaned by maxmzng the rteron H (W. In rght sde of the above equaton, the frst term s the rteron of LDA and the seond term s the rteron of PCA, and ther mportane s determned by the eghted varable α. In order to redue ts omputatonal ost, a smpler rteron s requred. L et al. proposed a ne feature extraton rteron, the maxmum margn rteron (MMC, for labeled data set, hh has been proved effent, aurate, and robust [4]. In ths paper, e propose a MCC based sem-supervsed dmensonalty reduton algorthm. MMC based dmensonalty reduton for the labeled set L, makes a example be lose to those n the same lass but far from those n dfferent lasses after the dmensonalty reduton. he margn rteron s defned as M = p p d( C, C 2 = = (8 d( C,C = d( m, m v( C v( C (9 d ( m, m = ( m m ( m m (0 v( C = tr( E{( x m ( x m x C} ( he margn rteron (8 an be easly smplfed to M = tr( Sb S (2 he redued dmensonal vetor W s obtaned by maxmzng M ( W = tr( W ( Sb S W (3 For unlabeled data set U, the margn rteron an be rtten as P ( W = tr( W RxxW (4 he MMC rteron of sem-supervsed dmensonalty reduton for the hybrd set D s defned as H ( W = tr( W ( Sb S +α Rxx W (5 Let W = [, L, Ll ] be an orthogonal matrx hose olumns satsfy = δ ( W W = Il (6 hen e solve the follong onstraned optmzaton max subet to l k = W k ( S b S W = I l + α R xx k (7 58

6 Internatonal ournal of Informaton ehnology, Vol.2, No.3, 2006 H (W s maxmzed hen W s omposed of the frst l largest egenvetors of S b S + α R xx [5]. Compared th maxmzng the rteron (7, the rteron (7 s easer to be optmzed. B. Clusterng Algorthm As the labeled data set L an be used as the seeds of the luster to buld an ntal lusterng result n the herarhal lusterng proedure, the latter agglomeratve operatons are smplfed to a proedure of the unlabeled examples beng assgned to the exstng lusters. herefore, n every teratve lusterng phase, the examples are lassfed nto to sets: a urrent labeled set that nludes the examples havng been exsted n the lusters and a urrent unlabeled set that nludes the examples havng not been lustered. At the same tme, the sem-supervsed dmensonalty reduton proedure s ompleted th these to types of data sets n eah teratve phase, and the next agglomeratve proedure an be done n the ne dmensonal spae. After the teratve phases, fnally all unlabeled examples are assgned to the lusters, and the number of dmensons s dereased to the pror defned value. he detaled algorthm for sem-supervsed herarhal lusterng s desrbed as follos. Algorthm: sem-supervsed herarhal lusterng (Number of lusters: Number of data ponts: n, hh ontans n labeled ponts and n 2 Number of orgnal dmensons: d Number of redued dmensons: l unlabeled ponts, and n + n2 {l k the number of urrent dmensons; Ck = { Ck, LCk} urrent lusters; n 2 k the number of urrent resdual unlabeled ponts}. Pk all labeled ponts as the seeds of lusters, and eah ntal luster n C0 = { C0, L C0 } nludes the examples th the same lass label. 2. Whle n 2k > 0 2. begn Clusterng proess for =: m fnd an unlabeled pont x that s losest to one of the urrent lusters C k merge x nto C k end C = ne{ C, L C } k k k n = n m 2k 2k end 2.2 begn dmensonalty reduton proess redue the number of dmensonalty from l k to lk = lk d th the sem-supervsed dmenson reduton algorthm. he 59

7 Yuntao Qan, Xaoxu Du, and Q Wang Sem-supervsed Herarhal Clusterng Analyss for Hgh Dmensonal Data end 3. end ponts n Ck = { Ck, L Ck} are used as labeled data, and the resdual unlabeled n 2k ponts are used as unlabeled data. {In eah teratve phase, the number of dmensons and unlabeled ponts are redued by d and m respetvely} C. Parameter Settng In our algorthm, some parameters should be set n advane. he eghted varable α that determnes the mportane of the PCA rteron relatve to the LDA rteron. Generally the value of α an be gven as. If α >>, sem-supervsed dmensonalty reduton degenerates to PCA, hle f 0 < α <<, t degenerates to LDA. he number of dmensons and the number of unlabeled ponts, hh should be redued n eah teratve phase. If e suppose that the number of dmensons s redued by d n eah teratve phase, the number of teratons an be determned by k = ( d l / d, and the number of unlabeled labeled examples that are assgned to the urrent lusters s m = n2 / k n eah teratve phase. In our experments, d s valued by. If e ant to save omputatonal tme,.e., dereasng the number of teratons, d an be valued by a large postve nteger. III. Expermental Evaluaton We llustrate the sem-supervsed dmensonalty reduton based herarhal lusterng on several data sets of UCI mahne learnng repostory. he performane measure s based on the mathng beteen the lusterng result and the lassfaton benhmark Corret Rate max Y has not been mathed th CLC = = = Y = Y, L, Y } s the lassfaton benhmark. { C { C I Y } For the sem-supervsed lusterng, e randomly selet a part of examples n the dataset to buld a labeled set L, and the rest examples are used as the unlabeled set U. he proporton of labeled examples n the hole data sze s defned as 5% n our experments. In the follong experments, e ompare three lusterng methods: sem-supervsed herarhal lusterng on the orgnal dmensonal spae thout dmensonalty (8 60

8 Internatonal ournal of Informaton ehnology, Vol.2, No.3, 2006 reduton (M, sem-supervsed herarhal lusterng on the redued dmensonal spae that s derved by unsupervsed dmensonalty reduton method - PCA (M2, and our sem-supervsed herarhal lusterng (M3. he mean and mnmum dstane measures are hosen as the dstane measure beteen an unlabeled pont and a luster. Pma Indans Dabetes Dagnoses data set (table : hs data set nludes 2 lasses, 765 nstanes, and 8 attrbutes. ohns Hopkns Unversty Ionosphere data set (table 2: hs data set nludes 2 lasses, 35 nstanes, and 34 attrbutes. Optal Reognton of Handrtten Dgts data set (table 3: hs data set nludes 0 lasses, 3822 nstanes, and 64 attrbutes. From the above three experments, t s shon the performane of our method s alays better than the other to methods. he value of α affets the lusterng performane, so ths parameter needs to be arefully hosen. In general, 0. < α < s sutable, that s the labelng nformaton should be gven a large eght relatve to unlabelng nformaton. IV. Conlusons In ths paper, the problem of sem-supervsed lusterng for hgh dmensonal data set s dsussed. he proposed algorthm s based on sem-supervsed herarhal lusterng frame n hh the lusters are formed gradually from a small amount of labeled examples as seeds by assgnng unlabeled examples to the exsted lusters aordng to ther dstanes. In the herarhal lusterng proedure, dmensonalty reduton s norporated, and the number of dmensons s redued gradually as the fnal lusters are formed. he rteron of dmensonalty reduton s dependent on both the labeled data n the urrent lusters and the unlabeled data that have not been assgned to the urrent lusters. hrough the teratve lusterng dmensonalty reduton lusterng proedure, the harmony beteen lusterng and dmensonalty reduton s reahed, and these to tasks are ntegrated nto a harmonous system. he expermental results also demonstrate the effetveness of our method. Hoever, ho to automatally determne sutable values for the parameters n our methods, and ho to mprove the omputatonal effetveness for large sale data sets, are need to be further studed n the future. able. Comparatve expermental results on Pma Indans Dabetes Dagnoses data set Method: lusterng methods Proporton: the proporton of the labeled examples n a data set Dmenson: the fnal redued number of dmensons α : eghted varable Measure: the type of dstane measure Auray: lusterng performane measure (orret rate Method Proporton Dmenson Measure α Auray M 5% 8 mean 63.28% 6

9 Yuntao Qan, Xaoxu Du, and Q Wang Sem-supervsed Herarhal Clusterng Analyss for Hgh Dmensonal Data M2 5% 2 mean 63.5% M3 5% 2 mean % M3 5% 2 mean 67.34% M3 5% 2 mean % M 5% 8 mn 70.70% M2 5% 2 mn 69.40% M3 5% 2 mn % M3 5% 2 mn 72.53% M3 5% 2 mn % able 2. Comparatve expermental results on ohns Hopkns Unversty Ionosphere data set Method Proporton Dmenson Measure α Auray M 5% 34 mean 7.79% M2 5% 5 mean 72.65% M3 5% 5 mean % M3 5% 5 mean 82.93% M3 5% 5 mean % M 5% 34 mn 68.38% M2 5% 5 mn 69.23% M3 5% 5 mn % M3 5% 5 mn 83.48% M3 5% 5 mn % able 3. Comparatve expermental results on Optal Reognton of Handrtten Dgts data set Method Proporton Dmenson Measure α Auray M 5% 64 mean 90.4% M2 5% 8 mean 89.22% M3 5% 8 mean % M3 5% 8 mean 9.3% M3 5% 8 mean % M 5% 64 mn 90.37% M2 5% 8 mn 88.63% M3 5% 8 mn % M3 5% 8 mn 9.45% M3 5% 8 mn 0 9.3% Referenes 62

10 Internatonal ournal of Informaton ehnology, Vol.2, No.3, 2006 [] Blum, A., Mthell,. Combnng labeled and unlabeled data th o-tranng. In Proeedngs of the th Annual Conferene on Computatonal Learnng heory (Madson, WI, 998, [2] Szummer, M., aakkola,. Partally labeled lassfaton th Markov random alks. In Advanes n Neural Informaton Proessng Systems 5 (NIPS 2002 (Vanouver, Brtsh Columba, Canada, Deember 9-4, MI Press, [3] Basu, S., Bleoko, M., Mooney R.. Comparng and unfyng searh-based and smlarty-based approahes to sem-supervsed lusterng, In Proeedngs of Int'l Conf. Mahne Learnng (ICML 2003(Washngton DC, August [4] Xng, E.P., Ng, A. Y., ordan, M. I., Russell S. Dstane metr learnng, th applaton to lusterng th sde-nformaton, In Advanes n Neural Informaton Proessng Systems 6 (NIPS 2003 (Vanouver and Whstler, Brtsh Columba, Canada, Deember 8-3, MI Press, [5] Fukunaga, K. Introduton to Statstal Pattern Reognton. Aadem Press, San Dego, Calforna, 990. [6] Ng, A.Y., ordan, M.I., Wess, Y. On spetral lusterng: Analyss and an algorthm. In Advanes n Neural Informaton Proessng Systems 5 (NIPS 2002 (Vanouver, Brtsh Columba, Canada, Deember 9-4, MI Press, 2003, [7] Goldberger,., Roes, S., Hnton, G., Salakhutdnov, R. Neghbourhood omponents analyss. NIPS 2004, Vanouver, Brtsh Columba, Canada, Deember 3-6, [8] enenbaum,.b., Slva, V., Langford,.C. A global geometr frameork for nonlnear dmensonalty reduton. Sene, 290(Deember 2000, [9] Roes, S.., Saul, L.K. Nonlnear dmensonalty reduton by loal lnear embeddng. Sene, 290(Deember 2000, [0] Fredman,. H., Meulman,.. Clusterng obets on subsets of attrbutes [] Roth, V., Lange,. Feature seleton n lusterng problems. In Advanes n Neural Informaton Proessng Systems 6 (NIPS 2003 (Vanouver and Whstler, Brtsh Columba, Canada, Deember 8-3, MI Press, [2] Dng, D., He, X.F., Zha, H.Y., Smon, H. Adaptve dmenson reduton for lusterng hgh dmensonal data. In Proeedngs of 2nd IEEE Int'l Conf. Data Mnng (Maebash, apan, De [3] Aggaral, C.C., Yu, P.S. Fndng generalzed proeted lusters n hgh dmensonal spaes. SIGMOD 2000, Dallas, exas, USA, May 6-8, 2000, [4] L, H.F., ang,., Zhang, K.S. Effent and robust feature extraton by maxmum margn rteron. In Advanes n Neural Informaton Proessng Systems 6 (NIPS 2003 (Vanouver and Whstler, Brtsh Columba, Canada, Deember 8-3, MI Press, [5] Guo, Y.F., L S.., Yang,.Y., Shu,.., Wu, L.D. A generalzed Foley-Sammon transform based on generalzed fsher dsrmnant rteron and ts applaton to fae reognton, Pattern Reognton Letters, 24(anuary

11 Yuntao Qan, Xaoxu Du, and Q Wang Sem-supervsed Herarhal Clusterng Analyss for Hgh Dmensonal Data Yuntao Qan reeved the B.E. and M.E. degrees n automat ontrol from X an aotong Unversty n 989 and 992 respetvely, and the Ph.D. degree n sgnal proessng from Xdan Unversty n 996. From 996 to 998, he as a postdotoral fello n Northestern Polytehnal Unversty. In Aprl 998, he oned the College of Computer Sene of Zheang Unversty here he beame a Professor n From 999 to 200, he as a vstng sholar at the Centre of Pattern Reognton and Mahne Intellgene, Conorda Unversty, Canada, and at Department of Computer Sene, Hong Kong Baptst Unversty. He has publshed more than 50 tehnal papers n aadem ournals and onferene proeedngs. Hs present researh nterests nlude data lusterng analyss, pattern reognton, mage proessng, avelet theory, and neural netorks. Xaoxu Du s a MS student n College of Comupter Sene of Zheang Unversty sne He obtaned hs bahelor degree n Management Informaton System from Huazhong Unnversty of Sene and ehnology. Hs maor researh nterests nlude fae reognton, lusterng analyss. 64