Combining Statistics and Semantics via Ensemble Model for Document Clustering

Transcription

1 ombining tatitic and emantic via Enemble Model or Document lutering amah Jamal Fodeh Michigan tate Univerity Eat Laning, MI, William F Punch Michigan tate Univerity Eat Laning, MI, punch@mu.edu Pang-Ning an Michigan tate Univerity Eat Laning, MI, ptan@mu.edu ABRA Incorporating background knowledge into data mining algorithm i an important but challenging problem. urrent approache in emi-upervied learning require explicit knowledge provided by domain expert, knowledge peciic to the particular data et. In thi tudy, we propoe an enemble model that couple two ource o inormation: tatitic inormation that i derived rom the data et, and ene inormation retrieved rom WordNet that i ued to build a emantic binary model. We evaluated the eicacy o uing our combined enemble model on the Reuter and 20newgroup data et. Keyword WordNet, enemble learning, text clutering, diambiguation. 1. INRODUION he rapidly growing availability o large tract o textual data uch a online new eed, blog poting, , and dicuion board meage, ha made the need or improved text clutering an important current reearch area. However, depite the extenive reearch, clutering untructured, textual inormation remain a challenging problem. For example, the nature o the untructured textual inormation make it hard or current clutering algorithm to capture the intrinic tructure that we deire [3]. Individual data et alo have unique characteritic which add more complexity to mapping or deciding upon the clutering methodology that work bet or a particular data et. Moreover, the lack o labeled example in unupervied clutering make the partitioning tak an ill-poed problem ince there i no adopted methodology wellknown to produce the ideal clutering [3]. o overcome thee challenge, reearcher have begun to invetigate alternative clutering approache that incorporate background knowledge to guide each partitioning tak and thu alleviate the diiculty o inding a ingle, bet approach [2][6]. One way to add background knowledge i through emiupervied clutering [1], where the domain inormation i Permiion to make digital or hard copie o all or part o thi work or peronal or claroom ue i granted without ee provided that copie are not made or ditributed or proit or commercial advantage and that copie bear thi notice and the ull citation on the irt page. o copy otherwie, or republih, to pot on erver or to reditribute to lit, require prior peciic permiion and/or a ee. A 09, March 8-12, 2009, Honolulu, Hawaii, U..A. opyright 2009 AM /09/03 $5.00. provided in the orm o labeled example or mut-link (ML) and cannot-link (L) contraint. hi explicit inormation ugget the availability o an expert in the domain who would annotate the label o the document or ummarize the important aociation between document in the data et. In practice, thi human intervention can be expenive and could produce inaccurate reult depending on the reliability o the inormation provided. o addre thee problem, there ha been ome recent work that attempt to incorporate background knowledge into the partitioning tak without any uer or expert dynamic interaction [8]. Unlike emi-upervied clutering, thi kind o background knowledge provide general inormation about the relationhip between the eature and i applicable to any data et with imilar type o eature. For document clutering, recent interet ha ocued on incorporating contextual knowledge in the orm o linguitic ontologie into clutering algorithm uch a WordNet [2][6][8] WordNet i an ontology that include not only the ene o the word, but alo their relationhip with each other. WordNet addree the ynonymy and polyemy problem o text document by replacing the word by their mot appropriate ene a ued in the context o the document. In thi paper, we invetigate the eectivene o combining term tatitic with emantic knowledge acquired rom WordNet to improve document clutering. Our analyi how that a traightorward replacement o the word by their correponding ene rom WordNet may not alway improve the clutering reult. hi i becaue the clutering algorithm mut deal with iue uch a the increaing dimenionality o the data (when the noun are replaced by their correponding ene) and noie (when incorrect ene are elected a eature). We propoe to addre thee iue uing a compound enemble clutering algorithm that combine the tatitic inormation rom the data with the ene inormation rom WordNet. Our approach or combining the model take into account the conitency o each clutering olution and i applicable to any clutering algorithm (including k-mean). We evaluated the eectivene our method uing two benchmark dataet: Reuter and 20newgroup. Our experimental reult ugget that the propoed method help to improve the clutering reult igniicantly when applied to data et where ene inormation i valuable to diambiguate word that are ued in multiple context. 2. emantic imilarity uing WordNet WordNet i a hierarchical linguitic ontology that group word into ynet. Each ynet deine a certain concept. ynet are

2 linked by emantic relation uch a hypernym and hyponym, indicating cla-ubcla relationhip between the ynet. everal emantic ditance meaure can be ued to compute the emantic imilarity between two ynet in the Wordnet hierarchy. In particular, we ued the Wu-Palmer meaure, which utilize inormation uch a ene depth. Wu-Palmer compute the imilarity between two ene by inding the leat common ubumer (L) node that connect their ene. he L o two ene, p and q, i the lowet interecting node between the path o p and q rom the root o the WordNet cla-ubcla hierarchy. Once the L ha been identiied, the Wu-Palmer ditance i given by the ollowing equation: 2d Wu-Palmer( p, q ) (1) L L 2d p where d i the depth o the L rom the root, L p i the path length between p and the L, and L q i the path length between q and the L. A key tep in our propoed approach or incorporating emantic knowledge rom WordNet i to identiy the mot appropriate ene aociated with each noun in a given document. Our approach i baed on the aumption that the ene o a noun i determined by the context in which it i being ued in the document. For example, conider the word cat, which ha eight meaning a a noun in WordNet. I it i ued in a document that contain other word uch a kitten, Perian, and pet, we expect it ene reer to the eline mammal ene o cat, and not one o the other even (uch a a arm machine or a particular type o X-ray). However, i the word cat appear in a document that contain other noun uch a contruction and builder, it ene mot likely reer to a aterpillar, which i a large tracked vehicle ued or moving earth in contruction. Let i = { i1, i2, ik } denote the et o all ene aociated with the noun t i according to the WordNet ontology. Given a document d, we determine the mot appropriate ene o a noun t i i by computing the um o it imilarity to other noun ene in d, i.e.: i argmax max ( il, jm) il i jm j t j d where ( p, q ) i the WordNet imilarity between two ene, p and q. Furthermore, ince the ene o a given noun in the WordNet hierarchy are arranged in decending order according to their popularity, we retrict our conideration to the top 3 ene o each given noun. Once the appropriate ene have been choen or the noun, we tranorm each document vector o term into a binary vector o their correponding ene. 3. ENEMBLE LUERING he propoed enemble clutering ramework combine the clutering olution obtained rom the emantic imilarity o the document (emantic binary model) with thoe obtained baed on requency imilaritie (noun requency model). Our rationale or uing enemble clutering i that, although individual clutering olution (uing either requency or emantic imilarity) may make poor deciion regarding the cluter aignment or ome document, one may be able to improve clutering reult by q (2) conidering their collective deciion 1. he propoed ramework i alo highly lexible becaue it may accommodate any baeline clutering algorithm a well a methodology or creating dierent intance o the enemble. he enemble clutering ramework ue two type o data input: (1) a document-noun requency matrix A, and (2) a documentene binary matrix A. A et o requency-baed clutering olution, M Fenemble, i generated rom A uing the methodology preented in ection 3.1. Analogouly, a ene o ene-baed clutering olution, M enemble, i obtained by applying the methodology given in ection 3.2. he clutering olution rom M Fenemble and M enemble are then aggregated to obtain the inal clutering uing the approach decribed in ection Noun Frequency Model he noun requency model i generated rom applying enemble clutering to the document-noun requency matrix A. Firt, we randomly ample a ubet o the noun, V V. o alleviate bia in the ample ize V, the number o noun to be ampled i an integer choen randomly between m/2 and m-1, where m i the total number o noun in the original dataet. A truncated document-noun requency matrix, A i then created by chooing only the noun in each document that belong to the ubet V. Once the truncated matrix i obtained, the weight or each document vector are urther normalized uing the FIDF method. Finally we apply the tandard k-mean algorithm to obtain an N k requency-baed cluter memberhip matrix,, whoe (i,j) th element i equal to 1 i the document d i belong to cluter j and 0 otherwie. 3.2 emantic Binary Model Our approach or creating the emantic binary model i quite imilar to the approach decribed in the previou ection. However, intead o uing the document-noun requency matrix, we ue the document-ene matrix a input to the enemble clutering algorithm. he ample ize i a random integer choen between m/2 and m-1. A truncated document-ene binary matrix A i then created by removing all the ene not elected by the ample. Ater normalization uing the FIDF method, we apply the k-mean clutering algorithm to obtain the ene-baed cluter memberhip matrix,. 3.3 ombined Enemble lutering hi ection decribe our propoed approach or combining the noun requency model with the emantic binary model. Firt, an N N weighted co-aociation matrix i computed rom the et o requency-baed cluter memberhip matrice in M Fenemble. he co-aociation matrix repreent the number o time a pair o document i aigned to the ame cluter in the enemble, weighted by the quality o the individual clutering olution. Formally it i iteratively computed rom the requency-baed cluter memberhip matrice a ollow: ( t1) ( t ) wt where the matrix product t) (3) ( i a binary 0/1 matrix that indicate whether a pair o document belong to the ame cluter 1 Auming each clutering olution i independent and i doing better than random cluter aignment.

3 during the t th iteration o the enemble and the weighting actor w t meaure the quality o the clutering. he matrix product i alo known a an incidence matrix in clutering literature. he weighting actor w t i then computed by correlating the coine imilarity matrix between each pair o document with the incidence matrix. he higher the correlation i, the greater the level o agreement between the clutering reult and the document imilarity matrix. Equation (3) i iteratively updated uing all the clutering olution rom the noun requency model. he weighted co-aociation matrix eectively encode the likelihood that a pair o document i in the ame cluter baed on it term requency inormation. It i certainly poible to apply a clutering algorithm uch a k-mean on to produce a inal clutering or the noun requency model. imilarly, we may repeat thi procedure to obtain a ene-baed weighted co-aociation matrix : ( t1) where the incidence matrix w t (4) depend on the clutering olution o the emantic binary model and the weighting actor w t i the correlation coeicient between the coine imilarity o document (computed rom the document-ene matrix A ) and the incidence matrix. he overall emantic binary model can be obtained by applying the kmean algorithm to the weighted coaociation matrix. However, ince our intention i to combine the noun requency model with emantic binary model, we may aggregate their weighted co-aociation matrice a ollow: ( 1 ) (5) ombined where α i a parameter that govern the tradeo between uing both model. We will apply a clutering algorithm uch a kmean on the combined weighted co-aociation matrix to obtain the inal clutering reult. 4. EXPERIMEN electing a data et to tet our approach upon ha to be done careully. We wanted to work with a data et that reveal the power o ubtituting the noun by it ene. o teting with a dataet that i rich with vocabulary ued to expre the ame meaning, or vocabulary that ha multiple meaning would give our algorithm a better chance to demontrate the eect o incorporating the emantic knowledge rom WordNet. wo benchmark data et were elected or our experiment the Reuter and 20newgroup data et. For Reuter dataet we elected only document that belong to the top 20 larget categorie or our experiment. We then ampled at mot 200 document rom each category. he inal ize o our text corpu wa 2655 document. For 20newgroup dataet: we aggregated the original training et into 6 ditinct categorie. For example, we aggregate categorie uch a rec.auto, rec.motorcycle, rec.port.baeball, and rec.port.hockey into one cla. he original data wa divided into 60% training et and 40% tet et. For our experiment, we ampled 6000 article rom the training et and applied our algorithm to detect the 6 ditinct categorie. All the experiment conducted in thi tudy employ kmean a the baeline algorithm. he number o cluter i equal to the number o categorie in the original data. ince kmean i enitive to the choice o initial centroid, we repeat each experiment 50 time and report their average entropy or purity. We compared our method againt Latent emantic Indexing (LI) which i a tatitical method or identiying the latent tructure in a et o document by analyzing the relationhip between the document and their correponding term. able 1: Intra-Document lutering or Reuter Data et. Document call rate, loan creen luter degree, monetary_value department, militia corn, grain, barley, wheat, oat liberation,agribuine average, iodine national, ubtitute,orghum hundredweight,two, three, our,ix tallow tallion, anadian authority, agreement purchae, bargain department, point, Hondura tranhipment, agribuine corn, wheat port, creen, hip meaure, metric_ton board etimate, intervention, beginning metric_ton, let, French eaon barley, cereal, wheat, corn prognoi, manner_o_peaking 4.1 Intra Document ene Diambiguation hi experiment aim to demontrate the eectivene o our method in electing the mot appropriate ene to ubtitute or a noun in the context o a particular document. One way to do thi i to cluter the ene and examine the reulting cluter. I we oberve ditinctive topic in the cluter then thi indicate that the elected ene were able to reveal ome o the emantic content o the document. In order to apply clutering within the document, we built a imilarity matrix or the ene uing Wu- Palmer imilarity meaure [10]. Intra-document clutering i then perormed uing the complete-link agglomerative hierarchical clutering algorithm. able 1 how the reult o clutering three document that belong to the grain category in the Reuter dataet. Document 2 how well-related emantic cluter uch a (purchae, bargain), (corn, wheat), and (meaure, metric_ton). hee cluter repreent the dierent emantic concept that are important in the document. Neverthele, we do oberve ome impure cluter uing the ene that were choen. For example, in document 1, although we ound a cloely-related emantic cluter (corn, grain, barley, wheat, oat), we alo mied a ene (orghum) that hould be added to thi cluter. Intead, orghum wa added to a cluter that doe not have a high emantic imilarity with gain.

4 hee reult ugget that our technique or ixing the ene o a word uing WordNet wa quite ucceul to orm a group o emantically related ene. We thereore argue that thi method wa able -to a good extent- to eparate the dierent topic in the document which hould improve document clutering reult. 4.2 Document lutering uing ene Ater ixing the ene o each word uing the approach decribed in ection 2, we tranorm each document vector into a binary vector o their ene. For the Reuter dataet, we tarted with 5922 noun a eature. Ater applying WordNet, the data i tranormed into a document-ene matrix with 6559 ene. Likewie, in the 20newgroup dataet, our method convert the noun into ene. able 2 how the number o eature ued or the dierent approache (including LI). Dataet able 2. Number o eature ued Noun-requency emantic model binary model LI 20newgroup Reuter he number o eature ha increaed ater word ene diambiguation ince the algorithm may reolve the ame noun into multiple ene depending on the context o the document. Next, we apply k-mean clutering on both the document-noun requency matrix A and document-ene binary matrix A. able 3 how the reult o the clutering, which are baed on the average entropy or 50 iteration o applying kmean with dierent initial centroid. hee reult however do not eem to jutiy the need to replace the noun with their correponding ene. In act, the ue o ene inormation eem to degrade the entropy igniicantly or the Reuter data et. able 3. omparion o average entropy or K-mean Dataet Noun-requency emantic model binary model 20newgroup Reuter One poible explanation or the poor reult i due to increaing number o eature when we replace the noun by their ene. o invetigate the eect o the number o dimenion, we have perormed ingular Value Decompoition (VD) on both the document-noun requency matrix and document-ene binary matrix prior to applying the kmean algorithm. he ormer i almot equivalent to a Latent emantic Indexing (LI) except we had removed the adjective, verb, and noun that are not regitered in WordNet. Figure 1 how the reult o applying k- mean clutering on the document-noun requency matrix, document-ene binary matrix, and LI. For the 20newgroup data, the entropy value improve igniicantly ater incorporating ene inormation. Neverthele, thee value are till wore than the reult without applying VD (ee able 3). For the Reuter data, the entropy value or clutering uing ene inormation are generally wore than thoe obtained by applying VD on the noun and the LI method. he reult provided in thi ection ugget that tranorming the noun into ene alone i inuicient to improve the clutering reult. In act, the reult may be wore due to the increaed dimenionality o the data or when the wrong ene i choen to replace ome o the noun. Figure 1. omparion o Avg entropy or K-mean with VD Neverthele, we till expect ome document to be correctly placed in the right cluter becaue o the word ene diambiguation. What i needed i a clutering algorithm that: (1) better utilize the ene inormation in combination with the term tatitic inormation. (2) deal with the increaing number o dimenion when replacing the noun by their ene. (3) tolerant to noie when incorrect ene are ued. Becaue o it lexibility and robutne, we conjecture that enemble clutering i an appropriate approach or combining term tatitic with emantic knowledge or document clutering. 4.3 Enemble lutering In thi work, we combine the clutering reult rom both the emantic binary model (denoted a M enemble ) and the noun requency model (denoted a M Fenemble ) into one inal clutering (denoted a M enemble ). Our motivation or uing enemble clutering i becaue o it lexibility to accommodate any input data matrix (either the document-noun requency matrix, the document-ene binary matrix, or both), it ability to deal with high dimenionality via eature ubampling, and it reilience to noie and other variability in the data. Figure 2. omparion o entropy or enemble clutering Figure 2 how the reult o applying the three enemble clutering method to the Reuter and 20newgroup data. he number o iteration or the enemble in thi experiment i varied rom 10 to 30 run. For each number o iteration, we combine hal o the clutering reult rom M enemble with another hal rom M Fenemble to generate the inal clutering M enemble. A mentioned in ection 3.3, each clutering olution in the enemble will be

5 weighted according to the quality o their cluter (which i meaured in term o the correlation between the coine imilarity o the document and the reulting incidence matrix o the cluter). We oberved that the weighting actor w t aociated with the olution in M Fenemble were generally greater than the weighting actor or M enemble. For example, or the Reuter dataet the average o the weight o the M Fenemble wa approximately.5 compared to.3 in the M enemble. hi obervation i conitent with our reult in ection 4.2 where the noun requency model appear to produce better cluter than uing the emantic binary model. he weighted cluter in each enemble were aggregated in a co-aociation matrix (with α = 0.8) that relect the conenu o the individual run on allocating the document acro the cluter. Figure 2 how a comparion o the entropy value between the three enemble or both data et. For the 20newgroup dataet, the compound enemble M enemble achieved the lowet entropy core. For example, ater 20 iteration, the entropy core or the compound enemble i coniderably lower (.559) than the core or the noun requency model (.636) and emantic binary model (.787). hi reult ugget that our compound enemble method i capable o enhancing the clutering reult by taking advantage o the variability in the clutering olution obtained rom the term tatitic and emantic knowledge. Even though the emantic binary model ha higher entropy than noun requency model, it till provide ueul olution that can be exploited by our compound enemble. Furthermore, all the enemble clutering reult are igniicantly better than the reult or individual run (ection 4.2) even though it ue only a ample o the original eature. In the Reuter data et, no igniicant improvement wa oberved or the compound enemble M enemble over the requency enemble M Fenemble. hi reult ugget that the nature o the data et alo play a igniicant role in determining the eectivene o incorporating emantic knowledge rom WordNet. Figure 3. omparion o purity or enemble clutering Figure 3 how the purity value or the dierent enemble clutering method. Once again, the compound enemble achieved the highet purity core (.802) or the 20newgroup data et wherea the noun requency enemble model ha the highet purity or the Reuter data et. Finally, it i worth noting that our propoed method uing the compound enemble clutering igniicantly improved the cluter quality compared to LI or both dataet. he lowet entropy obtained or the Reuter in LI wa 1.01 uing 100 component compared to.947 ater 10 iteration uing our compound enemble method. For the 20- newgroup dataet, LI achieved an entropy o 1.13 with 100 component compared to.559 when applying the compound enemble at 20 iteration. 5. ONLUION hi paper preent a methodology or combining term tatitic with emantic knowledge rom WordNet or document clutering. Our analyi how that a traightorward replacement o the word with their ene may not necearily improve the clutering reult, which i conitent with ome o the previou reult reported in [6] and [8]. he clutering algorithm need to be lexible and robut enough to deal with the higher dimenionality and noie due to improper election o ene. o overcome thee challenge, we propoe an enemble clutering method that ytematically combine clutering olution rom a noun requency model with thoe rom a emantic binary model baed on the conitency o their clutering olution. Our experimental reult how that the enemble method i eective on ome but not all data. We are preently doing urther reearch to better determine what characteritic o the data are mot uitable or the approach. 6. AKNOWLEDGMEN We acknowledge the upport o the National Archive and Record Adminitration and the MARIX group at Michigan tate Univerity. 7. REFERENE [1] Bradley P., Bennett K., and Demiriz A., ontrained k- mean clutering. Microot Reearch echnical Report, MR-R , [2] Hotho A., taab., tumme G, WordNet improve text document clutering. In Proc. o the IGIR 2003 emantic Web Workhop, 2003, [3] Goe J., an P. N., and heng H., emi-upervied lutering with Partial Background Inormation. In Proc. o IAM Int l on on Data Mining, Betheda, MD [4] Mann H. B., Whitney D. R. On a tet whether one o two random variable i tochatically larger than the other. Annal o Mathmatical tatitic, 18, 1947, [5] Miller J., WordNet: a lexical databae or Englih, ommunication o the AM [6] edding J., Kazakov D., WordNet-baed text document clutering. In Proc. o the 3 rd Workhop on Robut Method in Analyi o Natural Language Proceing Data. 2004, [7] teinbach M. and Karypi G. and Kumar V., A comparion o document clutering technique. In proc. o KDD Workhop on ext Mining, [8] ermier A., Rouet M, ebag M, ombining tatitic and emantic or word and document clutering, In Proc. o IJAI, 2001, [9] opchy A., Jain A.K., Punch W., A mixture model or clutering enemble, In Proc. o IAM onerence on Data Mining, 2004, [10] Wu Z. and Palmer M. Verb emantic and Lexical election. In Proc. o the 32nd Annual Meeting o the Aoc. or omputational Linguitic, 1994,