High Performance Latent Dirichlet Allocation for Text Mining

Transcription

1 Hgh Performance Latent Drchlet Allocaton for Text Mnng A thess submtte for Degree of Doctor of Phlosophy By Department of Electronc an Computer Engneerng School of Engneerng an Desgn Brunel Unversty September 2013

2 Abstract Latent Drchlet Allocaton (LDA), a total probablty generatve moel, s a three-ter Bayesan moel. LDA computes the latent topc structure of the ata an obtans the sgnfcant nformaton of ocuments. However, tratonal LDA has several lmtatons n practcal applcatons. LDA cannot be rectly use n classfcaton because t s a non-supervse learnng moel. It nees to be embee nto approprate classfcaton algorthms. LDA s a generatve moel as t normally generates the latent topcs n the categores where the target ocuments o not belong to, proucng the evaton n computaton an reucng the classfcaton accuracy. The number of topcs n LDA nfluences the learnng process of moel parameters greatly. Nose samples n the tranng ata also affect the fnal text classfcaton result. An, the qualty of LDA base classfers epens on the qualty of the tranng samples to a great extent. Although parallel LDA algorthms are propose to eal wth huge amounts of ata, balancng computng loas n a computer cluster poses another challenge. Ths thess presents a text classfcaton metho whch combnes the LDA moel an Support Vector Machne (SVM) classfcaton algorthm for an mprove accuracy n classfcaton when reucng the menson of atasets. Base on Densty-Base Spatal Clusterng of Applcatons wth Nose (DBSCAN), the algorthm automatcally optmzes the number of topcs to be selecte whch reuces the number of teratons n computaton. Furthermore, ths thess presents a nose ata reucton scheme to process nose ata. When the nose rato s large n the tranng ata set, the nose reucton scheme can always prouce a hgh level of accuracy n classfcaton. Fnally, the thess parallelzes LDA usng the MapReuce moel whch s the e facto computng stanar n supportng ata ntensve applcatons. A genetc algorthm base loa balancng algorthm s esgne to balance the workloas among computers n a heterogeneous MapReuce cluster where the computers have a varety of computng resources n terms of CPU spee, memory space an har sk space.

3 Acknowlegement I woul thank many people for ther help. Frst of all, I woul lke to thank my PhD supervsor, Prof. Maozhen L. In the process of the whole research, he always gave me the most help an guance from en to en. In aton, I have been encourage greatly wth hs avce an support so that I coul face all the ffcultes. Not only I learn more about my research, but also I learne how to analyze an solve the problem. I also thank Xaoyu Chen, Yang Lu, Yu Zhao. They gave me lots of sgnfcant opnons. Especally ther support an care n the usual lfe. Moreover, I stll woul thank the School of Engneerng an Desgn, an Brunel Unversty. Durng my PhD research years, I acheve all the aspects of the help from the School an the Unversty. Fnally, I thank my parents, grlfren an housemates. They gave me the greatest support an courage when I was n the most ffcult tme. Here, I woul lke to express my heartfelt thanks to all the frens who have helpe me. I shall remember the ays I spent at Brunel forever.

4 Author s Declaraton The work escrbe n ths thess has not been prevously submtte for a egree n ths or any other unversty an unless otherwse reference t s the author s own work.

5 Statement of Copyrght The copyrght of ths thess rests wth the author. No quotaton from t shoul be publshe wthout hs pror wrtten consent an nformaton erve from t shoul be acknowlege. v

6 Lst of Abbrevatons AD-LDA API BP CTM DAG DBSCAN DLB DP EM FIFO GA GFS GS HDFS HD-LDA HDP ICE Intel TBB JDK KLD Approxmate Dstrbute LDA Applcaton Program Interface Belef Propagaton Correlate Topc Moel Drecte Acyclc Graph Densty-Base Spatal Clusterng of Applcatons wth Nose Dynamc Loa Balancng Drchlet Process Expectaton-Maxmzaton Frst n Frst out Genetc Algorthm Google Fle System Gbbs Samplng Haoop Dstrbute Fle System Herarchcal Dstrbute LDA Herarchcal Drchlet Process Internet Communcatons Engne Intel Threang Bulng Blocks Java Development Kt Kullback-Lebler Dvergence v

7 K-NN LDA LSI MCMC MPI PAM PLDA PLSI SGA SSH SVD SVM TF-IDF VB VI VSM K-Nearest Neghbor Latent Drchlet Allocaton Latent Semantc Inexng Markov Chan Monte Carlo Message Passng Interface Pachnko Allocaton Moel Parallel Latent Drchlet Allocaton Probablstc Latent Semantc Inexng Smple Genetc Algorthm Secure Shell Sngular Value Decomposton Support Vector Machnes Term Frequency-Invert Document Frequency Varatonal Bayes Varatonal Inference Vector Space Moule v

8 Table of Contents Abstract... Acknowlegement... Author s Declaraton... Statement of Copyrght... v Lst of Abbrevatons... v Table of Contents... v Lst of Fgures... x Lst of Tables... xv Chapter 1 Introucton Backgroun Text Mnng Technques Hgh Performance Computng for Text Mnng Motvaton of Work Major Contrbutons Structure of the Thess Chapter 2 Lterature Revew Introucton Probablty Topc Moels TF-IDF Moel Mxture of Ungrams LSI Moel Basc Concepts The Moelng Process The Avantages of LSI The Dsavantages of LSI PLSI Moel Basc Concepts v

9 The Moelng Process The Avantages of PLSI The Dsavantages of PLSI LDA Moel Basc Concepts The Moelng Process The Avantages of LDA An Overvew of the Man Inference Algorthms of LDA Moel Parameters Varatonal Inference (VI) Belef Propagaton (BP) Gbbs Samplng (GS) Analyss an Dscusson An Overvew of Genetc Algorthm The Basc Iea of Genetc Algorthm The Man Steps of Genetc Algorthm Cong Mechansm Ftness Functon Selecton Crossover Mutaton The Parameter Settngs of Genetc Algorthms An Overvew of Haoop HDFS MapReuce Programmng Moel n Haoop Haoop Scheulng Algorthms Dvsble Loa Theory Summary Chapter 3 Text Classfcaton wth Latent Drchlet Allocaton Introucton Overvew of Text Classfcaton v

10 3.2.1 The Content of Text Classfcaton Text Preprocessng Text Representaton Text Feature Extracton an Dmenson Reucton Text Classfcaton Algorthms Classfcaton Performance Evaluaton System Text Classfcaton base on LDA Gbbs Samplng Approxmate Inference Parameters of LDA Moel The Specfc Steps of Text Classfcaton Experment an Analyss Expermental Envronment Tranng Envronment for SVM The Data Set Evaluaton Methos Expermental Results Summary Chapter 4 Accuracy Enhancement wth Optmze Topcs an Nose Processng Introucton The Metho of Selectng the Optmal Number of Topcs Current Man Selecton Methos of the Optmal Number of Topcs Base on LDA Moel The Metho of Selectng the Optmal Number of Topcs Base on HDP The Stanar Metho of Bayesan Statstcs A Densty-base Clusterng Metho of Selectng the Optmal Number of Topcs n LDA DBSCAN Algorthm The Relatonshp between the Optmal Moel an the Topc Smlarty A Metho of Selectng the Optmal Number of Topcs Base on x

11 DBSCAN Experment an Result Analyss Nosy Data Reucton The Nose Problem n Text Classfcaton The Current Man LDA-base Methos of Nose Processng The Data Smoothng Base on the Generaton Process of Topc Moel The Category Entropy Base on LDA Moel A Nose Data Reucton Scheme The Experment an Result Analyss Summary Chapter 5 Genetc Algorthm base Statc Loa Balancng for Parallel Latent Drchlet Allocaton Introucton The Current Man Parallelzaton Methos of LDA Mahout s Parallelzaton of LDA Yahoo s Parallel Topc Moel The Algorthm of Parallel LDA Base on Belef Propagaton Google s PLDA Analyss an Dscusson Parallelng LDA wth MapReuce/Haoop The Workng Process of MapReuce on Haoop The Algorthm an Implementaton of PLDA A Statc Loa Balancng Strategy Base on Genetc Algorthm for PLDA n Haoop The Algorthm Desgn The Desgn an Implementaton of Genetc Algorthm Encong Scheme The Intalzaton of Populaton Ftness Functon x

12 Crossover Mutaton The optmal retenton strategy Experment an Analyss Evaluatng PLDA n Haoop The Expermental Envronment Expermental Data The Experment n the Homogeneous Envronment The Experment n the Heterogeneous Envronment The Experment of Loa Balancng n a Smulate Envronment Summary Chapter 6 Concluson an Future Works Concluson Future Works The Supervse LDA The Improve PLDA PLDA Dynamc Loa Balancng Problem The Applcaton of Clou Computng Platform The Applcaton of Interscplnary Fel References x

13 Lst of Fgures Fgure 2.1: The generatve process of the topc moel Fgure 2.2: (Left) The ungrams (Rght) The mxture of ungrams Fgure 2.3: The agram of sngular value ecomposton (SVD) Fgure 2.4: The graphcal moel representaton of PLSI Fgure 2.5: The network topology of LDA latent topcs Fgure 2.6: (Left) The structure agram of LDA latent topcs (Rght) The graphcal moel representaton of LDA Fgure 2.7: The LDA probablstc graphcal moel wth the varatonal parameters. 39 Fgure 2.8: Belef propagaton n the LDA moel base on the factor graph Fgure 2.9: The typcal MapReuce framework n Haoop Fgure 3.1: The typcal process of automatc text classfcaton Fgure 3.2: The separatng hyperplane of SVM algorthm Fgure 3.3: Comparson of the performance of three methos on each class Fgure 4.1: HDP moel Fgure 4.2: The relatonshp between logp(w T) an T Fgure 4.3: The ata smoothng base on LDA weakens the nfluence of nose samples Fgure 4.4: The flow agram of a nose ata reucton scheme Fgure 4.5: The relatonshp between the number of teratons an the effect of nose processng wth fferent nose ratos Fgure 4.6: Classfcaton results of fferent methos wth varous nose ratos n frst group of ata Fgure 4.7: Classfcaton results of fferent methos wth varous nose ratos n secon group of ata Fgure 4.8: Classfcaton results of fferent methos wth varous nose ratos n thr group of ata x

14 Fgure 5.1: The framework of one Gbbs samplng teraton n MapReuce-LDA Fgure 5.2: The computng tme wth fferent number of ata noes n a homogeneous cluster Fgure 5.3: A comparson of computng tme of ealng wth fferent szes of ata wth eght ata noes Fgure 5.4: The performance comparson of the loa balancng strategy wth the fferent heterogenety n a smulate envronment Fgure 5.5: The performance comparson of PLDA wth fferent szes of ata n a smulate envronment Fgure 5.6: The convergence of genetc algorthm n the loa balancng strategy x

15 Lst of Tables Table 3.1: The strbuton of fve classes of text n the 20newsgroup corpus Table 3.2: Comparson of three methos macro-average an mcro-average Table 3.3: Comparson of three methos mensonalty reucton egree to corpus Table 4.1: Results of the propose algorthm to fn the optmal value of topc Table 4.2: Expermental ata sets Table 5.1: The features comparson of mplementng PLDA wth MPI an MapReuce Table 5.2: The confguraton of the expermental envronment Table 5.3: The confguraton of noes n a Haoop cluster Table 5.4: The expermental result of sngle machne an one ata noe n the cluster processng the ata Table 5.5: The confguraton of the smulate Haoop envronment xv

16 Chapter 1 Chapter 1 Introucton 1.1 Backgroun So far, the Internet has accumulate a huge number of gtal nformaton nclung news, blogs, web pages, e-books, mages, auo, veo, socal networkng an other forms of ata, an the number of them has been growng at the spee of the exploson contnually [1]. Thus, how people can organze an manage large-scale ata effectvely an obtan the requre useful nformaton quckly has become a huge challenge. For example, the ata s too large to use the tratonal ata analyss tools an technques to eal wth them. Sometmes, even f the ata set s relatvely small, because of untratonal characterstcs of the ata, the tratonal methos also cannot be use [2] [3]. In aton, the expert system technology can put knowlege nto the knowlege base manually by specal users or oman experts. Unfortunately, ths process often has some evaton an mstake, an t s tme-consumng an hgh-cost [4] [5]. Therefore, t s necessary to evelop new technologes an automatc tools whch can convert massve ata nto useful nformaton an knowlege ntellgently. Data mnng s a technque, an t s able to combne tratonal ata analyss methos wth complex algorthms that can eal wth large amounts of ata. Data mnng s a complex process where the unknown an valuable moes or rules are extracte from mass ata. Furthermore, t s an nterscplne, whch s closely relate to atabase system, statstcs, machne learnng, nformaton scence an other scplnes [1] [2] [4]. So, ata mnng can be seen as the result of the natural evoluton of nformaton technology. Accorng to the processng object, t can be ve nto object ata mnng, spatal ata mnng, multmea ata mnng, Web mnng an text mnng [3] [6]. Hgh Performance Latent Drchlet Allocaton for Text Mnng 1

17 Chapter 1 Text s the most mportant representaton of the nformaton. The statstcs research showe that 80 percent of nformaton n an organzaton was store n the form of text, whch nclue books, research papers, news artcles, Web pages, e-mal an so on [7]. Text s able to express vast an abunant nformaton meanwhle t contans lots of unetecte potental knowlege. The whole text set s not structure ata an t lacks machne-unerstanable semantcs so that t s qute ffcult to eal wth a huge number of ocuments. So n the fel of ata mnng, a new technology whch can process the above text ata effectvely was propose, whch was calle text mnng [5] [6] [8] Text Mnng Technques Text mnng was frst propose by Ronen Felman et al n 1995, whch was escrbe lke The Process of extractng nterestng Patterns from very large text collectons for the purpose of scoverng knowlege [6]. Text mnng was also known as text ata mnng or knowlege scovery n texts, whch was a process where the unknown, potental, unerstanable an useful knowlege can be foun from mass text ata [2] [6] [9] [10]. Meanwhle, t was also a process of analyzng text ata, extractng text nformaton an fnng out text knowlege [9] [11] [12]. The man technques of text mnng contan text classfcaton, text clusterng, text summarzaton, correlaton analyss, nformaton extracton, strbuton analyss, tren precton an so on [13] [14] [15]. Here, text classfcaton an text clusterng are the mnng methos whose object s the text set. But, the processng object of text summarzaton an nformaton extracton s a sngle ocument. There are many classfcaton methos n text classfcaton, an the frequently-use methos nclue Natve Bayes (NB), K-Nearest Neghbor (K-NN), Support Vector Machnes (SVM), Vector Space Moule (VSM) an Lnear Least Square Ft (LLSF) [6][7][8][9][12][13][14][15]. Hgh Performance Latent Drchlet Allocaton for Text Mnng 2

18 Chapter 1 In the fel of text mnng, machne learnng experts researche an put forwar probablstc topc moel, an fast unsupervse machne learnng algorthms were use to fn out the text hen nformaton automatcally [16] [17] [18] [19]. At present, man mnng latent semantc knowlege moels are Latent Semantc Inexng (LSI) [20] [21] [22] [23] [24], Probablstc Latent Semantc Inexng (PLSI) [25] [26] [27] [28] [29] an Latent Drchlet Allocaton (LDA) [30] [31] [32]. Ther applcaton almost covers all areas of text mnng an nformaton processng, such as text summarzaton, nformaton retreval an text classfcaton, etc [33] [34] [35]. Especally, Ble et al put forwar the LDA moel n 2003, whch was wely use to solve text classfcaton, text annotatons an other ssues. Beses, t create a seres of text processng methos whch were base on probablstc topc moelng [17] [36], an t was expane to mages, auo, veo an other multmea ata processng fels [37] [38]. LDA s a probablstc topc moelng whch moels screte ata sets such as text set [30]. It treats ocuments as the probablty strbuton of topcs an smplfes the generatve process of the text, whch helps to hanle large-scale text sets effcently. In aton, LDA s a three-ter Bayesan moel, whch nclues wors, topcs an ocuments three-ter structure. It makes each ocument express as a topc mxture where each topc s a probablty strbuton of the fxe wor lst. In bref, the basc moelng process of LDA s to establsh a ocument-wor co-occurrence matrx frstly. An then, a text tranng set s moele. Next, nference methos are use to obtan moel parameters, such as ocument-topc matrx an topc-wor matrx. Fnally, the learne moel wll be use to prect the topc probablty strbuton of new ocuments so as to express text nformaton [17] [30] [32]. Compare wth other topc moels, LDA has some unque avantages: Frstly, LDA Hgh Performance Latent Drchlet Allocaton for Text Mnng 3

19 Chapter 1 topc moel s a completely probablstc generatve moel, whch can use mature probablty algorthms to tran moels rectly. An, t s easy to use the moel [39]. Seconly, the sze of the parameter space of LDA s fxe an has nothng to o wth the sze of a text set so that t s more sutable for large-scale text sets [40]. Thrly, LDA s a herarchcal moel, whch s more stable than non-herarchcal moels [39] Hgh Performance Computng for Text Mnng In the aspects of processng spee, storage space, fault tolerance an access spee, the tratonal techncal archtecture an sngle computer wth seral-base approach are more an more unsutable to hanle mass ata [41] [42] [43] [44]. Parallel computng s an effectve metho of mprovng the computaton spee an processng capacty of the computer system. Its basc ea s to ecompose a problem nto several parts, an each part s compute by an nepenent processor n parallel [45] [46] [47]. A Parallel computng system can be a supercomputer wth multple processors, an t also can be a cluster whch s mae up of several nepenent computers that are nterconnecte n some way [44] [48]. However, for parallel programmng moels n the tratonal hgh-performance computng, such as PThrea, MPI an OpenMP [49] [50], evelopers have to unerstan bottome confguratons an parallel mplementaton etals. In 2004, Jeffrey Dean an Sanjay Ghemawat who were two engneers of Google propose the programmng ea of MapReuce [51], an apple t to the parallel strbute computng of large-scale ata sets. Accorng to functonal programmng eas, the MapReuce framework can ve a computatonal process nto two phases Map an Reuce [52] [53] [54]. In the Map phase, the nput ata s transmtte to map functons n the form of key-value pars. After operatons, an ntermeate set of key-value pars s generate. In the Reuce phase, all the ntermeate values sets wth the same keys wll be merge. The fnal result wll be output n the form of key-value Hgh Performance Latent Drchlet Allocaton for Text Mnng 4

20 Chapter 1 pars. For users, they only nee to mplement two functons map an reuce, reucng the complexty of esgnng parallel programmng sgnfcantly [51] [52]. Then, Google CEO Erc Schmt frst propose the concept of clou computng n 2006 [43]. Each bg nternatonal Internet nformaton technology company launche a seres of proucts n successon, showng ther research results [55] [56]. For example, Google launche Google App Engne whch was base on the clou computng envronment evelopment platform. Amazon launche Amazon EC2 (Amazon Elastc Compute Clou) whch was a powerful clou computng platform an Amazon S3 (Amazon Smple Storage Servce) whch can prove clou storage servce. Mcrosoft launche Wnows Azure whch was a clou computng platform. Clou computng s the result of comprehensve evelopment of parallel computng, strbute computng an gr computng. In other wors, clou computng s the commercal mplementaton of the above computng scence concepts [43] [57]. Clou computng s a new computng moe, s also a new composte moe of computer resources, an represents an nnovatve busness moe [55]. The basc prncple of clou computng s to strbute the requre calculaton, transmsson an storage n local computers or remote servers nto a large number of strbute computers, whch means that each computer shares huge tasks of calculaton, transmsson an storage together. Users can gan corresponng servces through the network, mprovng the resource utlzaton rate effectvely an realzng resource acquston on eman [58] [59]. The above clou computng platforms are commercal proucts, whch cannot be freely avalable to the major researchers. Thus, the emergence of open source clou computng platforms s able to gve the major researchers opportuntes. The researchers can use these open source projects to consttute a cluster system whch are mae up of several machnes n a laboratory envronment, an smulate the clou computng envronment n busness systems [57] [58]. Haoop s one of the most Hgh Performance Latent Drchlet Allocaton for Text Mnng 5

21 Chapter 1 famous open source strbute computng frameworks, whch acheves man technologes of Google clou computng [60] [61] [62]. Bascally, ts core content contans Haoop Dstrbute Fle System (HDFS) [53] [60] [61] [63] an MapReuce programmng moel [64] [65] [66]. Haoop orgnate n the projects of Lucene an Nutch, an then t evelope nto a separate project of Apache founaton. It s manly use for processng mass ata [53] [60] [63]. Beses, t proves a MapReuce framework base on Java wth strong portablty, whch can make the strbute computng apply to large low-cost clusters. Meanwhle, for nvuals an companes, t has a hgh performance so as to lay the founaton for the research an applcaton of ther strbute computng [67] [68]. Yahoo was the frst company whch use, researche an evelope Haoop eeply. In Yahoo, there were more than ten thousan CPUs n a sngle cluster usng Haoop, an there were hunres of researchers n the use of Haoop [52] [54]. Nowaays, t has become the manstream technology of clou computng. For example, unverstes, research nsttutons an the Internet companes research an use t [67]. Along wth the ncreasng popularty of the clou computng concept, Haoop wll get more attenton an faster evelopment [65]. Accorng to the gtze nformaton report whch was publshe by Internatonal Data Corporaton (IDC) n 2011, the amount of global nformaton woul be ouble an reouble every two years [1]. So, there s no oubt that t wll gve the ata storage an computng power of servers much pressure, whch cannot use tratonal topc moel algorthms to process large-scale text sets n hgh-performance servers [44] [69]. In aton, people cannot only hope that the mprovement of computer harware technology s able to enhance the processng effcency, but they nee to use effcent parallel computng technologes to apply to mult-core computers an cluster systems wth hgh-performance. Through a combnaton of harware an software, the accelerate processng can be accomplshe n the process of complex calculatons, solvng the bottleneck of computaton tme an memory capacty. At the Hgh Performance Latent Drchlet Allocaton for Text Mnng 6

22 Chapter 1 same tme, parallel computng can save cost to a certan extent, an complete large-scale computng tasks wth a lower nvestment [51] [68] [70]. When ealng wth mass ata, the savantage of LDA s that the computaton complexty wll be hgher an processng tme wll be longer [17] [71] [72]. Thus, how LDA can moel large-scale text effectvely, meet the requrements of the computaton tme an memory capacty, an fnally fn out latent topc rules s another bg ssue. In orer to mprove the LDA moel n processng mass ata, the parallel LDA algorthm has become one of the research hotspots. To research MapReuce parallel algorthm an apply t to topc moels has much practcal sgnfcance [51] [52] [66]. Frstly, parallel algorthms can solve the problem of calculaton an storage, an they can process large-scale ata more effcently. Seconly, parallel topc moels are able to aress many practcal applcaton problems, such as text classfcaton, nformaton retreval, text summarzaton an so on [17] [73]. Fnally, parallel topc moels can analyze the massve Internet user behavor ata to obtan useful nformaton, whch s able to be use n socal networkng webstes, ntellgent search an other Internet proucts [18] [72]. 1.2 Motvaton of Work Automatc text classfcaton nvolves ata mnng, computatonal lngustcs, nformatcs, artfcal ntellgence an other scplnes, whch s the bass an core of text mnng. It s an mportant applcaton fel of natural language processng, an also a sgnfcant applcaton technology of processng large-scale nformaton [2] [4] [74]. In short, text classfcaton technology s to make a large number of ocuments ve nto one or a group of categores where each category represents fferent concept topcs. Automatc text classfcaton systems can help users organze an obtan nformaton well, playng a sgnfcant role n mprovng the spee an Hgh Performance Latent Drchlet Allocaton for Text Mnng 7

23 Chapter 1 precson of nformaton retreval [5] [8]. Therefore, t has a very mportant research value. In text classfcaton, the selecton an mplementaton of classfcaton methos are the core parts of classfcaton systems. How to choose an approprate classfcaton moel s an mportant ssue [75] [76]. In aton, text feature selecton s also another key technology n text classfcaton [77] [78] [79]. The man role of makng probablstc topc moel apply to text classfcaton s to acheve the mensonalty reucton of the text ata representaton space by fnng out latent topc nformaton from ocuments. In the early stage, the most typcal representatve was the LSI moel. The mensonalty reucton effect of ts feature space s sgnfcant, but ts classfcaton performance often ecreases [80] [81]. Furthermore, ts parameter space grows lnearly wth the tranng ata because of the hgh computaton complexty of SVD operaton n LSI. These are lmtatons of LSI when t s apple to practcal problems. PLSI can be seen as a probablstc mprove verson of LSI, whch can escrbe the probablstc structure of the latent semantc space [25] [26]. However, PLSI stll has the problem of the parameter space growng lnearly wth the tranng set [29] [40]. Compare wth LSI an PLSI, the LDA moel s a completely probablstc generatve moel so that t has a complete nternal structure an t s able to use useful probablstc algorthms to tran an make use of the moel. In aton, the sze of the parameter space of the LDA moel has nothng to o wth the number of ocuments. Thus, LDA s more sutable to construct text representaton moel n a large-scale corpus [30] [32]. An, the LDA moel has got successful applcatons n machne learnng, nformaton retreval an other many fels. In the research of text classfcaton, the LDA moel s effectve but ts performance s not remarkable. The reason s that LDA s a non-supervse learnng moel so that t cannot be rectly use n classfcaton. It nees to be embee nto approprate classfcaton algorthms [82]. Hgh Performance Latent Drchlet Allocaton for Text Mnng 8

24 Chapter 1 The LDA moel tself s a generatve moel, an t s usually ntegrate wth generatve classfcaton algorthms. In ths generatve ntegrate moe, the tranng of classfers oes not use the entre tranng corpus to gan a sngle LDA moel, but the sub-lda moels of corresponng categores are obtane by each category of the tranng corpus. In fact, each sub-lda moel escrbes latent topcs of corresponng categores of ocuments [32] [83]. Because of separate tranng, each category shares a group of topcs, an topcs between categores are solate. But, one of man problems of ths classfcaton metho s that the target ocument wll take place the force strbuton of latent topcs n categores whch o not contan the target ocument, resultng n that the calculaton of the generatve probablty prouces the evaton n these categores so as to reuce the classfcaton performance [84]. Therefore, how to choose a sutable classfcaton algorthm combne wth the LDA moel to construct an effectve classfer that has become a challenge. In the LDA moel, topcs obey Drchet strbuton whch assumes that the emergence of a topc has nothng to o wth the emergence of other topcs. But n the real ata, many topcs have assocatons between them. For example, when PC appears, the probablty of Computer appearng wll be qute hgh but t s unlkely for Hosptal to appear. Obvously, ths nepenent assumpton s nconsstent wth the real ata. So, when usng the LDA moel to moel the whole text set, the number of topcs n the LDA moel wll nfluence the performance of moelng greatly [82]. Therefore, how to etermne the optmal number of topcs n the LDA moel s another research hotspot of LDA. At present, to etermne the optmal number of topcs n the LDA moel has two man methos: the selecton metho base on Herarchcal Drchlet Process (HDP) [86] [87] an the stanar metho n Bayesan statstcs [82]. The former uses the nonparametrc feature of Drchlet Process (DP) to solve the selecton problem of the optmal number of topcs n LDA. But, t nees to establsh a HDP moel an a LDA moel respectvely for the same one ata set. Obvously, when processng practcal Hgh Performance Latent Drchlet Allocaton for Text Mnng 9

25 Chapter 1 text classfcaton problems, t wll spen a lot of tme n computng [86] [88]. The latter nees to specfy a seres of values of T manually, where T stans for the optmal number of topcs n LDA. After carryng out Gbbs samplng algorthm an the relate calculaton, the value of T n the LDA moel can be etermne fnally [82]. Thus, an algorthm nees to be esgne to fn out the optmal number of topcs automatcally wth low consumng tme an operaton. In aton, n text classfcaton, the qualty of classfers wll affect the fnal classfcaton result greatly. An, the qualty of classfers epens on the qualty of the tranng text set to a great extent. In general, f classes of the tranng text set are more accurate an ts content s more comprehensve, the qualty of the obtane classfer wll be hgher [89] [90]. However, n practcal applcatons, t s qute ffcult to obtan ths hgh qualty of tranng text sets, especally large-scale text sets. Usually, the tranng ata contans nose unavoably, an the nose samples wll have a sgnfcant mpact on the fnal classfcaton result [91]. Usually, a manstream approach of nose processng s to entfy an remove nose samples from ata sets [90] [91]. At present, there are two man nose processng methos base on LDA, the ata smoothng metho base on the LDA moel [92] an the nose entfcaton metho base on the category entropy [93] [94]. They can remove a majorty of nose samples from text sets effectvely to some extent, but they cannot elete all the nose completely [95]. Furthermore, some normal samples wll be wrongly remove as nose unavoably [96]. Therefore, these ssues also become a challenge n text classfcaton. In orer to meet the requrements of storage capacty an computaton spee, the sngle-core seral programmng s turnng to the mult-core parallel programmng technology graually [41] [42]. Parallel programmng applyng to clusters or servers can solve some lmtatons when topc moel learnng algorthms process mass ata [69]. So, parallel topc moels are able to eal wth large-scale ata effectvely. But, Hgh Performance Latent Drchlet Allocaton for Text Mnng 10