Naive Rule Induction for Text Classification based on Key-phrases

Transcription

1 Nave Rule Inducton for Text Classfcaton based on Key-phrases Nktas N. Karankolas & Chrstos Skourlas Department of Informatcs, Technologcal Educatonal Insttute of Athens, Greece. Abstract In ths paper, we focus on the nducton of nave rules for classfyng text documents. An algorthm s brefly descrbed for the creaton of key-phrases from a gven set of documents and these key-phrases are organzed and used as features for the automatc classfcaton of documents. An Authorty lst of key-phrases s specfed by the algorthm contanng key-phrases that are frequent wthn the documents of only one or few classes n the tranng set. In ths framework, ths last property permtted us the creaton of nave rules that measure the smlarty of documents wth the exstng classes. Keywords: text data mnng, text classfcaton, nstance based learnng, rule nducton. Introducton Key-phrases or search terms could be defned as sequences of adjacent words wthn a text wndow (e.g. fve successve words of the text / a sentence) formng a meanngful, descrptve phrase related to the content of the text document. Such terms can be used as features for classfyng (text) documents. Snce, not every key-phrase s approprate for dscrmnatng between documents, we have to examne and apply methods for selectng the approprate ones. Hence, a prerequste for such a classfcaton method s the use and mantenance of a lst of key-phrases, the so-called Authorty Lst Karankolas et al [4]. An nterestng problem s related to the reducton of the search space that s needed for the extracton of canddate key-phrases. In Classfcaton learnng, a learnng scheme takes a set of classfed examples from whch t s expected to learn a way of classfyng unseen

2 examples. In Instance-based learnng methods we start wth a partcular nstance (example) and see how t can be generalzed to cover other nearby nstances (examples) n the same class Wtten et al [0]. Therefore, Instance based learnng methods seems to be a novel way for classfyng documents. In such a method a smlarty measure adapted from the doman of Text Retreval (Kowalsk [7], Deal []) can be used to evaluate how close to the exstng classfed documents s a, under classfcaton, document.. Defntons - Notatons Every classfed document could be represented (or accompaned) by a vector of (m+) tems. The frst m tems are assgned to Boolean values, representng the exstence or not of a correspondng key-phrase n the document. It means that the elements of the Authorty lst of key-phrases are m or n other words that the number of key-phrases selected for classfcaton s m. In the last (m+)-tem of the vector s located the class, called the label, of the document. Havng a tranng set of labeled vectors we eventually form a table wth m+ attrbutes (columns). Thus, data mnng algorthms (or knowledge dscovery) can be easly appled to such a representaton of documents, n order to extract rules or trees for classfyng documents Karankolas et al [4]. 2 Materals and Methods In the proposed classfcaton scheme we use an authorty lst of key-phrases and a set of classfed documents and try to apply a smlarty measure whch offers a way of classfyng (unclassfed) documents. Hence, n our work, Text Classfcaton s the task of applyng (sem) automatc methods n order to select the approprate class for a gven document, between a set of predefned ones (classes). More precsely, our nstance based learnng approach for the classfcaton of documents s based on the use of the key-phrases and the related concept of the smlarty between exstng documents (the tranng set) and the (under classfcaton) documents Karankolas et al [5]. The method assumes that smlar documents must be classfed n the same category,.e. must share the same classfcaton code. It remans to select the features of documents that must be used for measurng ther smlarty. In our approach these features are key-phrases that exst n the documents of the tranng set and are extracted from the Authorty Lst. The representaton of our documents s a vector wth (m+2) attrbutes. The frst attrbute contans the document. Each of the next m attrbutes s assgned to a Boolean value that represents the exstence or not of a correspondng key-phrase n the document. The last tem of the vector s the classfcaton code (the label) of document. The followng fgure (fg. ) depcts the vector and ts relatonshp wth the lst of key-phrases and the lst of avalable classes. Ths approach explots a well-known approach n the doman of Informaton Retreval n order to defne a measure of smlarty between the document

3 and the documents of the tranng set. Our proposal for a smlarty measure s an adaptaton of the cosne measure (see for example Lucarella [8]). Authorty Lst of Key-Phrases K K 2 K 3 K 4 K m Doc. ID T)rue F)alse T F F F T CC s CC CC 2 CC 3 CC 4 CC s CC n Classfcaton Codes (CC), T= True, F= False Fgure : The representaton of our documents vectors and ts relatonshp wth the lst of key-phrases and the lst of avalable classes The followng equaton calculates the smlarty: m q k m j j j= j= S( D, D ) = = () LD LD k m m 2 q j j= j= 2 j where m s the number of key-phrases used n the collecton, k j s equal to f the key-phrase j exsts n document D (of the tranng set), otherwse s equal to 0 and q j s the weght of key-phrase j n the document. The followng equaton can be used to measure the term q j : ClassCount q j = log 2 ClassFreq j where ClassCount s the number of classes of the tranng set, and ClassFreq j s the number of classes that nclude the key-phrase j. Selectng key-phrases (Georgantopoulos et al [3], Turney [9]) that wll be used as attrbutes for text classfcaton s a very mportant step n our approach. The choce of the key-phrases has not to be based on frequent (wthn the whole text collecton) canddate phrases. We must also avod choosng key-phrases that exst n a few texts of the collecton (as a whole) but are qute frequent wthn these texts Karankolas et al [6]. The creaton of an Authorty Lst and ts use for document classfcaton must be based on the selecton of key-phrases that are frequent wthn the documents of only one or few classes n the tranng set. q j k j

4 2. Phrase Extracton We formalze the problem of phrase extracton for classfcaton n the followng way. Gven a collecton of documents subdvded nto classes, a wndow wdth and a frequency threshold, fnd all key-phrases that occur frequently enough n only one or few classes. The followng algorthm can solve the problem of extractng the approprate key-phrases and has two alternatng phases: buldng canddate key-phrases, and evaluatng how often these ones occur n a class of the collecton. Algorthm For every class (CL ) of the tranng set do 2 For every document of the class (DCL ) do 3 Stemmng 4 stop word removal 5 End {For every document of the class} 6 Choose the most frequent stems of the class (P0 - parameter) 7 Form the canddate double word phrases (C 2 ) from the frequent stems (L ) 8 Choose the most frequent double word phrases (L 2 ) (W and P - parameters) 9 For x=3,4 do 0 Form the canddate x - wdth word phrases (C x ) from the frequent (x-) wdth word phrases (L x- ) Choose the most frequent x wdth word phrases (L x ) (P2 and W2, P3 and W3 parameters) 2 End {For x=3,4 do} 3 Compose an ntegrated lst by jonng L x (for x=2,3,4). Ths jon, forms the frequent word phrases of class (FCL ) 4 End {For every class of the tranng set} 5 Integrate / Jon the lsts of frequent word phrases of all classes of the tranng set 6 Reject the frequent word phrases that exst n many classes (Pt - parameter). The rest of the frequent word phrases form the set of key-phrases or «Authorty lst» 7 Form the dctonary of «Terms». It s the lst of stems that are components of the key-phrases of the «Authorty lst». Where Parameter P0 s the percentage of texts of the class that must contan a stem, W s the wdth of wndow that covers 2-word phrases, P s the percentage of texts of the class that must contan a 2-words phrase, etc. Pt s the percentage of classes that can contan a key-phrase. The way that step0 forms the canddate x-wdth word phrases (C x ) from the frequent (x-) wdth word phrases (L x- ) s an nterestng part of our algorthm. The effcency of the algorthm s based on the followng estmaton: Potentally, a very large number of canddate key-phrases has to be checked. Hence, we can reduce the search space by buldng larger key-phrases from smaller ones. In

5 other words, t s only necessary to test the occurrences of key-phrases whose all sub-(key-phrases) are frequent. 3 Nave Rules nducton CL : { }, 2 In ths secton we focus on some key ponts for the nducton of nave rules that measure the smlarty of a, unclassfed, document wth the pre-defned classes of documents. There s an nterestng property ncorporated nto the above Key-phrase Extracton algorthm: The Authorty lst created by the algorthm contans keyphrases that are frequent wthn the documents of only one or few classes. Consequently, for each class we can have a lst of representatve key-phrases. A framework for extractng nave rules can be the followng one: The smlarty of a document wth the elements of some class CL s estmated by the number of key-phrases from the lst of the representatve ones for the class CL. More precsely, the calculaton of smlarty s equal to the number of tems of CL class representatve key-phrases lst that actually exst n the document, dvded (normalzed) by the populaton of the lst. The followng notaton denotes the lst of representatve key-phrases of class RCL = kpcl kpcl,..., kpcl, where kpcl j s the j-(keyphrase) of the representatve key-phrases of class CL. The measure RCL defnes the number of the representatve key-phrases for the specfed class CL. A smple framework for extractng nave rules that measure the smlarty of a, unclassfed, document wth the class CL s defned by the equaton: ' S ( D, CL ) count( exst( kpcl, D RCL j r = (2a) where the operaton (functon) exsts denotes f a representatve key-phrase kpcl j exsts n the document and counts means the calculaton of the exstng key-phrases. If we want to proceed nductng more complcated rules we can use weghts based on the number of words (stems) that consttute a key-phrase. ' S ( D, CL ) = RCL j= count( conttuent( exst( kpcl, D RCL j= )) count( consttuent( kpcl )) (2b) Thus, gven a document we can measure ts smlarty wth every class, usng formula (2a) or (2b), and suggest the most plausble classes,.e. the classes that have the greater smlarty wth the document. In other words, we have to apply the smlarty measures S ' ( D, CL ) for any avalable class and choose the class that gets the hghest value. On the contrary, an nstance based learnng approach needs the applcaton of the formula for the calculaton j j ))) CL

6 of the smlarty S D, D ) as many tmes as the sze of the tranng set. ( Moreover, n order to select the most promsng class, we have to apply a second step that fnds the average smlarty of the document wth the documents of each class of the tranng set. Ths can be obtaned wth the followng formula: '' S ( D, CL ) = D j DCL S( D, D j DCL (3) where DCL s the subset of the tranng s set documents that are pre-classfed as members of class CL. Example Let say that we have a tranng set that contans a thousand (000) documents pre-classfed n eght (8) classes. In order to select the most relevant class for a document, wth the usage of the nstance based learnng approach, we have to: apply formula (), between every document of the tranng set and the unclassfed document,.e. a thousand of tmes apply formula (3), to get the average smlarty between each class and the unclassfed document,.e. eght tmes On the contrary, the applcaton of nave rule (2a), eght (8) tmes s the only step needed to select the most relevant class for the document. Thus, the nave rule nducton based approach s smple and could be easly appled n varous applcatons. On the other hand, the nstance based learnng approach can be appled only when the tranng set s not a very extended one. 4 Conclusons We have presented methods for (sem) automatc classfcaton of documents based on nstance based learnng and nave rule nducton. Our approach s based on the use of key-phrases from a controlled lst (Authorty Lst). The creaton and mantenance of the key-phrase s (authorty) lst s an mportant subject n the doman of document classfcaton. Our algorthm for key-phrase selecton permts the nducton of nave rules for classfcaton. There s an nterestng property n our algorthm, for selectng the approprate keyphrases for classfcaton that permtted us to nvestgate a nave rule nducton for classfcaton, as an alternatve to the classcal data mnng algorthms. The Authorty lst s created by the proposed algorthm and contans key-phrases that are frequent wthn the documents of only one or few classes n the tranng set. Consequently, for each class we can have a lst of representatve key-phrases. A framework for nave rules nducton s based on a measure of the smlarty between a document and the representatve key-phrases of some class CL. Hence, f a document s gven we can measure ts smlarty wth every class and suggest the most plausble classes,.e. the classes that have the )

7 greater smlarty wth the document. It s also possble to extract more complcated rules usng weghts e.g. weghts based on the number of words (stems) that consttute a key-phrase. Acknowledgements Ths work was co-funded by 75% from E.E. and 25% from the Greek Government under the framework of the Educaton and Intal Vocatonal Tranng Program Archmedes. References [] Deal D.C., Technques of Document Management: A revew of Text Retreval and related technologes. Journal of Documentaton, 57, pp , 200. [2] Frank Ebe, et al. Doman-Specfc Key-phrase Extracton. Internatonal Jont Conference of Artfcal Intellgence, 999. [3] Georgantopoulos B. & Pperds S. Automatc Term Extracton Based on Pattern Grammars. LogoNavgaton, Issue 5, May 999. [4] Karankolas N.N. & Skourlas C. Automatc Dagnoss Classfcaton of patent ds charge letters. MIE 2002: XVIIth Internatonal Congress of the European Federaton for Medcal Informatcs, Budapest, August, IOS Press, ISBN: , ISSN: [5] Karankolas N.N., Skourlas C., Chrstopoulou A. & Alevzos T. Medcal Text Classfcaton based on Text Retreval technques. MEDINF st Internatonal Conference on Medcal Informatcs & Engneerng, October 9 -, 2003, Craova, Romana. [6] Karankolas N.N. & Skourlas C. Key-Phrase Extracton for Classfcaton. MEDICON and HEALTH TELEMATICS X Medterranean Conference on Medcal and Bologcal Engneerng, July 3 - August 5, 2004, Ischa, Italy. [7] Kowalsk G. Informaton Retreval Systems. Theory and Implementaton, Kluwer Academc Publshers, st edn, 997, ISBN [8] Lucarella, D., A document retreval system based on nearest neghbour searchng, Journal of Informaton Scence, 4, pp , 988. [9] Turney P. Extracton of Key-phrases from Text: Evaluaton of Four Algorthms. Natonal Research Councl of Canada, Techncal Report ERB- 05, October 23, 997. [0] Wtten Ian & Frank Ebe, Data Mnng: Practcal Machne Learnng tools and Technques wth Java mplementaton, Morgan Kaufmann, 999, ISBN: