A Comparative Study for Classification

Transcription

1 A Coparatve Study for Eal Classfcato Seogwook You ad Des McLeod Uversty of Souther Calfora, Los Ageles, CA USA Abstract - Eal has becoe oe of the fastest ad ost ecoocal fors of coucato. However, the crease of eal users have resulted the draatc crease of spa eals durg the past few years. I ths paper, eal data was classfed usg four dfferet classfers (Neural Network, classfer, Naïve Bayesa Classfer, ad classfer). The experet was perfored based o dfferet data sze ad dfferet feature sze. The fal classfcato result should be f t s fally spa, otherwse, t should be 0. Ths paper shows that sple classfer whch ake a bary tree, could be effcet for the dataset whch could be classfed as bary tree. I. INTRODUCTION Eal has bee a effcet ad popular coucato echas as the uber of Iteret users crease. Therefore, eal aageet s a portat ad growg proble for dvduals ad orgazatos because t s proe to suse. The bld postg of usolcted eal essages, kow as spa, s a exaple of suse. Spa s cooly defed as the sedg of usolcted bulk eal - that s, eal that was ot asked for by ultple recpets. A further coo defto of a spa restrcts t to usolcted coercal eal, a defto that does ot cosder o-coercal solctatos such as poltcal or relgous ptches, eve f usolcted, as spa. Eal was by far the ost coo for of spag o the teret. Text classfcato cludg eal classfcato presets challeges because of large ad varous uber of features the dataset ad large uber of docuets. Applcablty these datasets wth exstg classfcato techques was lted because the large uber of features ake ost docuets udstgushable. I ay docuet datasets, oly a sall percetage of the total features ay be useful classfyg docuets, ad usg all the features ay adversely affect perforace. The qualty of trag dataset decdes the perforace of both the text classfcato algorths ad feature selecto algorths. A deal trag docuet dataset for each partcular category wll clude all the portat ters ad ther possble dstrbuto the category. The classfcato algorths such as Neural Network (), Support Vector Mache (), ad Naïve Bayesa () are curretly used varous datasets ad showg a good classfcato result. The proble of spa flterg s ot a ew oe ad there are already a doze dfferet approaches to the proble that have bee pleeted. The proble was ore specfc to areas lke Artfcal tellgece ad Mache Learg. Several pleetatos had varous trade-offs, dfferece perforace etrcs, ad dfferet classfcato effceces. The techques such as decso tree (), Nave Bayesa classfers, Neural Networks, Support Vector Mache, etc had varous classfcato effceces. The reader of the paper s orgazed as follows: Secto 2 descrbes exstg related works; Secto 3 troduces four spa classfcato ethods used the experet; Secto 4 dscusses the experetal results; Secto 5 cocludes the paper wth possble drectos for future work. II. RELATED WORKS [7] copared a cross-experet betwee 4 classfcato ethods, cludg decso tree, Naïve Bayesa, Neural Network, lear squares ft, Roccho. K s oe of top perforers, ad t perfors well scalg up to very large ad osy classfcato probles. [4] showed that brgg other kds of features, whch are spa-specfc features ther work, could prove the classfcato results. [] showed a good perforace reducg the classfcato error by dscoverg teporal relatos a eal sequece the for of teporal sequece patters ad ebeddg the dscovered forato to cotet-based learg ethods. [3] showed that the work o spa flterg usg feature selecto based o heurstcs. Aproaches to flterg juk eal are cosdered [2, 5, 4]. [6] ad [7] showed approaches to flterg eals volve the deployet of data g techques. [3] proposed a odel based o the Neural Network to classfy persoal eals ad the use of Prcpal Copoet Aalyss (PCA) as a preprocessor of to reduce the data ters of both desoalty as well as sze. [] copared the perforace of the Naïve Bayesa flter to a alteratve eory based learg approach o spa flterg. [5] ad [8] developed a algorth to reduce the feature space wthout sacrfcg rearkable classfcato

2 accuracy, but the effectveess was based o the qualty of the trag dataset. I the classfcato experet for spa al flterg, showed better result tha,, or classfer. x2 arg ( γ ) x III. SPAM CLASSIFICATION METHODS Geerally, the a tool for eal aageet s text classfcato. A classfer s a syste that classfes texts to the dscrete sets of predefed categores. For the eal classfcato, cog essages wll be classfed as spa or legtate usg classfcato ethods. A. Neural Network () Classfcato ethod usg a was used for eal flterg log te ago. Geerally, the classfcato procedure usg the cossts of three steps, data preprocessg, data trag, ad testg. The data preprocessg refers to the feature selecto. Feature selecto s the way of selectg a set of features whch s ore foratve the task whle reovg rrelevat or redudat features. For the text doa, feature selecto process wll be forulated to the proble of detfyg the ost relevat word features wth a set of text docuets for a gve text learg task. For the data trag, the selected features fro the data preprocessg step were fed to the, ad a eal classfer was geerated through the. For the testg, the eal classfer was used to verfy the effcecy of. I the experet, a error BP (Back Propagato) algorth was used. B. Support Vector Maches () Classfer s are a relatvely ew learg process flueced hghly by advaces statstcal learg theory. s have led to a growg uber of applcatos age classfcato ad hadwrtg recogto. Before the dscovery of s, aches were ot very successful learg ad geeralzato tasks, wth ay probles beg possble to solve. s are very effectve a wde rage of boforatc probles. s lear by exaple. Each exaple cossts of a uber of data pots(x, x) followed by a label, whch the two class classfcato we wll cosder later, wll be + or -. - represetg oe state ad represetg aother. The two classes are the separated by a optu hyperplae, llustrated fgure, zg the dstace betwee the closest + ad - pots, whch are kow as support vectors. The rght had sde of the separatg hyperplae represets the + class ad the left had sde represets the - class. Ths classfcato dvdes two separate classes, whch are geerated fro trag exaples. The overall a s to geeralze well to test data. Ths s obtaed by troducg a separatg hyperplae, whch ust axze the arg () betwee the two classes, ths s kow as the optu separatg hyperplae wx + = w x+ b= 0 w x + b= 2 b Let s cosder the above classfcato task wth data pots x, =...,, wth correspodg labels y = ±, wth the followg decso fucto: f ( x) = sg( w x+ b) By cosderg the support vectors x ad x2, defg a caocal hyperplae, axzg the arg, addg Lagrage ultplers, whch are axzed wth respect to α: W( α) = α αα y y ( x x ) = j j j =, j= ( α y = 0, α 0) C. Naïve Bayesa () Classfer Naïve Bayesa classfer s based o Bayes theore ad the theore of total probablty. The probablty that a docuet d wth vector x =< x,..., x > belogs to category c s PC ( = c) PX ( = x C= c) PC ( = c X= x) = PC ( = k) PX ( = x C= k) k { spa, legt} However, the possble values of X are too ay ad there are also data sparseess probles. Hece, Naïve Bayesa classfer assues that X,... X are codtoally depedet gve the category C. Therefore, practce, the probablty that a docuet d wth vector x =< x,..., x > belogs to category c s PC ( = c) PX ( = x C= c) PC ( = c X= x) = = PC ( = k) PX ( = x C= k) k { spa, legt} = PX ( C ) ad P(C) are easy to obta fro the frequeces of the trag dataset. So far, a lot of Support vectors Separatg hyperplae

3 researches showed that the Naïve Bayesa classfer s surprsgly effectve. D. Classfer classfer s a sple C4.5 decso tree for classfcato. It creates a bary tree. IV. RESULTS for both ad was over 95%. Dataset sze was ot a portat factor easurg precso ad recall. The results show that the perforace of classfcato was ot stable. For four dfferet classfcato ethods, precso of spa al was show Fg. 2, lkewse, precso of legtate al was show Fg. 3. I ths secto, four classfcato ethods (Neural Network, Support Vector Mache classfer, Naïve Bayesa classfer, ad classfer) were evaluated the effects based o dfferet datasets ad dfferet features. Fally, the best classfcato ethod was obtaed fro the trag dataset eals were used as a trag dataset. 38.% of dataset were spa ad 6.9% were legtate eal. To evaluate the classfers o trag dataset, we defed a accuracy easure as follows. Precso 5 5 Legtate Precso Correctly _ Classfed _ Eals Accuracy(%) = *00 Total _ Eals Fg. 3. Legtate precso based o data sze Spa Recall Also, Precso ad Recall were used as the etrcs for evaluatg the perforace of each eal classfcato approach. A. Effect of dataset o perforace A experet easurg the perforace agast the sze of dataset was coducted usg dataset of dfferet szes lsted Fg.. The experet was perfored wth 55 features fro TF/IDF. For exaple, case of 000 dataset, Accuracy was 95.80% usg classfer. Naïve Bayesa % 92.70% 97.20% 95.80% % 95.00% 98.5% 98.25% % 92.40% 97.83% 97.27% % 9.93% 97.75% 97.63% % 97% 96.47% 97.56% Wth 55 features Fg.. Classfcato result based o data sze Spa Precso Recal Fg. 4. Spa recall based o data sze Legtate Recall (spa) (spa) (spa) (spa) Precso 5 5 (spa) (spa) (spa) (spa) Fg. 2. Spa precso based o data sze A few observatos ca be ade fro ths experet. As show Fg., the average of correct classfcato rate Fg. 5. Legtate recall based o data sze As show Fg. 2, 3, 4, ad 5, the precso ad recall curves of ad classfcato were better tha the oes of ad. Also, the average precso ad recall for both ad was over 95%. I Fg. 5, legtate recall values were sharply decreased at the data sze The crease of spa al the trag dataset betwee 000 ad 2000 result a sharp decrease of legtate recall values for all classfers

4 B. Effect of feature sze o perforace The other experet easurg the perforace agast the sze of dataset was coducted usg dfferet features lsted Fg eal dataset was used for the experet. For exaple, case of 0 features, Accuracy was 94.84% usg classfer. The ost frequet words spa al were selected as features. Geerally, the result of classfcato was creased for all classfcato ethods accordg the feature sze creased. classfer provded the precso over 95% for every feature sze rrespectve of spa or legtate. Also, classfer supported over 97% of classfcato accuracy for ore tha 30 feature sze. For the recall, ad showed better result tha ad for both spa ad legtate al, but was a lttle bt better tha. 5 Spa Recall Feature Sze Naïve Bayesa % 8.9% 92.42% 94.84% % 85.73% 95.60% 96.9% % 88.87% 95.64% 97.56% % 89.93% 97.49% 97.3% % 90.27% 96.84% 97.67% % 94% 97.64% 97.56% Fg. 6. Classfcato result based o feature sze Feature Sze Fg. 9. Spa recall based o feature sze Legtate Recall (spa) (spa) (spa) (spa) Spa Precso 5 Precso. 5 5 (spa) (spa) (spa) (spa) Feature Sze Feature Sze Fg. 0. Legtate recall based o feature sze Precso 5 5 Fg. 7. Spa precso based o feature sze Legtate Precso Feature Sze Fg. 8. Legtate precso based o feature sze As show Fg. 7, 8, 9, ad 0, good classfcato result order the experet was,,, ad for all cases (spa precso, legtate precso, spa recall, ad legtate recall). The overall precso ad recall for eal classfcato crease ad becoe stable accordg to the crease of the uber of feature. Gradually, the accuracy crease ad fally saturated wth the creased feature sze. As show Fg. 7 ad 8, V. CONCLUSTION AND FUTURE WORK I ths paper, four classfers cludg Neural Network,, Naïve Bayesa, ad were tested to flter spas fro the dataset of eals. All the eals were classfed as spa () or ot (0). That was the characterstc of the dataset of eal for spa flterg. s very sple classfer to ake a decso tree, but t gave the effcet result the experet. Naïve Bayesa classfer also showed good result, but Neural Network ad dd t show good result copared wth or Naïve Bayesa classfer. Neural Network ad were ot approprate for the dataset to ake a bary decso. Fro ths experet, we ca fd t that a sple classfer ca provde better classfcato result for spa al flterg. I the ear future, we pla to corporate other techques lke dfferet ways of feature selecto, classfcato usg otology. Also, classfed result could be used Seatc Web by creatg a odularzed otology based o classfed result. There are ay dfferet g ad classfcato algorths, ad paraeter settgs each algorth. Experetal results ths paper are based o the default settgs. Extesve experets wth dfferet

5 settgs are applcable WEKA. Moreover, dfferet algorths whch are ot cluded WEKA ca be tested. Also, experets wth varous feature selecto techques should be copared. Furtherore, we pla to create a adaptve otology as a spa flter based o classfcato result. The, ths otology wll be evolved ad custozed based o user s report whe a user requests spa report. By creatg a spa flter the for of otology, a flter wll be user custozed, scalable, ad odularzed, so t ca be ebedded to ay other systes. Ths otology also ay be used to block por web ste or flter out spa eals o the Seatc Web. [6] I. Stuart, S. Cha, ad C. Tappert, A Neural Network Classfer for Juk E-Mal, Docuet Aalyss Systes, 2004, pp [7] Y. Yag, A Evaluato of Statstcal Approaches to Text Categorzato, Joural of Iforato Retreval, Vol, No. /2, 999, pp [8] Y. Yag ad J. Pederse, A Coparatve Study o Feature Selecto Text Categorzato, I ICML, 997, pp [9] S. You ad D. McLeod, Otology Developet Tools for Otology-Based Kowledge Maageet, I Ecyclopeda of E- Coerce, E-Goveret ad Moble Coerce. Idea Group Ic, ACKNOWLEDGEMENT Ths research has bee fuded part by the Itegrated Meda Systes Ceter, a Natoal Scece Foudato Egeerg Research Ceter, Cooperatve Agreeet No. EEC REFERENCES [] I. Adroutsopoulos, G. Palouras, V. Karkaletss, G. Sakks, C. Spyropoulos, ad P. Staatopoulos, Learg to Flter Spa E- Mal: A Coparso of a Nave Bayesa ad a Meory-Based Approach, CoRR cs.cl/ , [2] W. Cohe, Learg rules that classfy e-al, I Proc. of the AAAI Sprg Syposu o Mache Learg Iforato Access, 996. [3] B. Cu, A. Modal, J. She, G. Cog, ad K. Ta, O Effectve E- al Classfcato va Neural Networks, I Proc. of DEXA, 2005, pp [4] E. Crawford, I. Koprska, ad J. Patrck, Phrases ad Feature Selecto E-Mal Classfcato, I syposu of ADCS, 2004, pp [5] Y. Dao, H. Lu, ad D. Wu, A coparatve study of classfcato based persoal e-al flterg, I Proc. of fourth PAKDD, [6] T. Fawcett, vvo spa flterg: A challege proble for data g, I Proc. of th KDD Exploratos vol.5 o.2, [7] K. Gee, Usg latet seatc dexg to flter spa, I Proc. of eghteeth ACM Syposu o Appled Coputg, Data Mg Track, [8] Z. Gyögy, H. Garca-Mola, ad J. Pederse, Cobatg Web Spa wth TrustRak, I VLDB, 2004, pp [9] T. Joachs, A Probablstc Aalyss of the Roccho Algorth wth TFIDF for Text Categorzato, I ICML, 997, pp [0] T. Joachs, Structured Output Predcto wth Support Vector Maches, SSPR/SPR, 2006, pp. -7 [] S. Krtcheko, S. Matw, ad S. Abu-Haka, Eal Classfcato wth Teporal Features, Itellget Iforato Systes 2004, pp [2] S. Mart, B. Nelso, A. Sewa, K. Che, ad A. Joseph, Aalyzg Behavoral Features for Eal Classfcato, CEAS, [3] T. Meyer, ad B. Whateley, SpaBayes: Effectve ope-source, Bayesa based, eal classfcato syste, I Proc. of frst Coferece of Eal ad At-Spa, [4] M. Saha, S. Duas, D. Heckera, ad E. Horvtz, A Bayesa Approach to Flterg Juk E-Mal, I Proc. of the AAAI Workshop o Learg for Text Categorzato, 998. [5] S. Shakar ad G. Karyps, Weght adjustet schees for a cetrod based classfer, Coputer Scece Techcal Report TR00-035, 2000.