Feature Selection Method with Proportionate Fitness Based Binary Particle Swarm Optimization

Transcription

1 Feature Selecton Method wth Proportonate Ftness Based Bnary Partcle Swarm Optmzaton Zhe Zhou, Xng Lu, Png L, and Ln Shang State Key Laboratory of Novel Software Technology, Department of Computer Scence and Technology, Nanjng Unversty, Nanjng , Chna, Abstract. Partcle swarm optmzaton(pso) has been appled on feature selecton wth many mproved results. Tradtonal PSO methods have some drawbacks when dealng wth bnary space, whch may have negatve effects on the selecton result. In ths paper, an algorthm based on ftness proportonate selecton bnary partcle swarm optmzaton(fpsbpso) wll be dscussed n detal amng to overcome the problems of tradtonal PSO methods. FPSBPSO wll be utlzed n the feature subset selecton doman. The performance of feature selecton wll be compared n a benchmark dataset, and expermental results prove that the FPSBPSO-based feature selecton methods can avo premature convergence and mprove the classfcaton accuracy. 1 Introducton Feature Selecton refers to the process of selectng a subset of relevant features n model constructon, whch s often seen n classfcaton problems. A well selected feature subset can boost the classfer s performance and reduce the tme cost at the same tme. Current feature selecton methods nclude frequency-based methods, nformaton-gan-based methods, lexcon-based methods and so on. Beses, some evolutonary algorthms can be utlzed nto feature selecton as well, such as generc algorthms[1] and partcle swarm optmzaton. Partcle swarm optmzaton(pso) s an optmzaton technque frstly ntroduced by Kennedy and Eberhart n 1995 [6], whch was nspred by the socal behavor of brds flockng or fsh schoolng. Snce then, PSO has attracted sgnfcant attenton. Researchers report that compared wth other evolutonary algorthms lke genetc algorthms(ga), PSO s computatonally more effcent [4]. Despte many successful applcatons of PSO, varous varatons of the classc PSO model were proposed amng to acheve better performance. However, they stll share the same basc ea: a swarm s made up by many partcles, each of whch represents a candate soluton. In each teraton, partcles update ther

2 2 poston n the search space accordng to ther own hstorcal nformaton and ther neghbours nformaton. After the teratons reach the end, an optmal soluton s supposed to be found. PSO was orgnally desgned to solve contnuous optmzaton problems. In order to deal wth dscrete problems, Kennedy and Eberhart proposed bnary partcle swarm optmzaton(bpso) [7] n BPSO has attracted much attenton snce nvented, but most researchers have been focusng on the applcaton of BPSO rather than ts theoretcal analyss. Although there are several mproved versons of BPSO, none of them has become a real replacement of BPSO. The orgnal BPSO model s used n most publcatons reportng the applcaton of PSO n dealng wth dscrete problems. The basc ea of PSO s that a partcle can explot ts own hstorcal poston and performance nformaton, as well as the hstorcal poston and performance nformaton of the whole swarm. A partcle tends to move closer to ts own hstorcal best poston and the best poston found by the whole swarm durng teratons. Ths ea enables PSO to fnd an optmal soluton, and t s also the ea shared by most varatons of PSO. However, as we analysed the detals of BPSO, we found that BPSO doesn t follow ths ea, or at least not n the way other PSO varatons do. In fact, a partcle n BPSO wll probably move away from the best poston found by tself or the best poston found by the whole swarm. Ths makes BPSO dffcult to nterpret. Our goal s to fnd a feature subset for a bnary classfcaton problem wth PSO-based methods. In order to make the selected feature subset as powerful as possble, we am to mprove PSO s performance n dscrete space. In order to acheve ths goal, we wll dscuss two major problems of PSO when the search space s dscrete and try to prove that the tradtonal way of calculatng veloctes s the man cause of the two problems. A new way to calculate veloctes based on ftness values s then proposed and based on that we wll propose a new bnary verson of PSO called FPSBPSO. We wll prove the effcency of our method by utlzng the method to perform the feature selecton process n classfcaton problems. If the selected feature subset wth FPSBPSO can return better classfcaton results than that wth tradtonal PSO, our proposed scheme can be regarded as an effcent one. The remander of ths paper s organzed as follows. In secton 2, we revew the development of PSO and BPSO. In secton 3, we show two problems PSO wll have to face n dscrete space, and ntroduce a modfed BPSO method called FPSBPSO. Then we wll utlze FPSBPSO to deal wth the task of feature subset selecton. In secton 4, we dscuss dfferent expermental results whch can prove FPSBPSO s effcency both n optmzaton problems and n feature subset selecton. Conclusons and future work are ncluded n secton 5.

3 3 2 Background 2.1 Contnuous Partcle Swarm Optmzaton PSO was proposed by Kennedy and Eberhart n 1995 [6]. It was nspred by the socal behavor of brds flockng or fsh schoolng. A swarm has some partcles, each partcle has a poston component representng a specfc soluton, and a velocty component representng the drecton of a partcle s movement n the soluton space. PSO s an teratve optmzaton algorthm, and lke other teratve algorthms, t also has manly three steps. The frst step s to ntalze the populaton by generatng each partcle s velocty component and poston component randomly. The second step s to evaluate solutons represented by partcles postons. The fnal step s to update partcles veloctes and then update partcles postons usng the followng formulas. v t+1 = w v t + c 1 rand ( pbest x t ) + c 2 rand ( gbest x t ) (1) x t+1 = x t + v t (2) Where v t+1 and x t+1 represents the velocty component and the poston component of partcle p at the (t+1) th teraton respectvely, c 1 and c 2 are two fxed confence coeffcent, rnd s a unformally dstrbuted random varable rangng from 0 to 1, w s the nerta weght. pbest means the poston of partcle p s personal best whle gbest means the poston of partcle p s global best. The rght se of equaton (1) can be dved nto three parts. w v t represents the prevous drecton, c 1 rand (pbest x t ) represents the tendency of movng towards a partcle s personal best, c 2 rand (gbest x t ) represents the tendency of movng towards the swarm s global best. The three parts cooperate wth each other to gue a partcle s movement. The followng fgure shows the effect of ther cooperaton. Fg. 1. The effect of pbest and gbest. Snce the basc PSO was proposed, researchers have proposed many varatons amng to mprove ts performance. In fact, there was no nerta weght n the

4 4 basc PSO, Sh and Eberhart ntroduced nerta weght nto PSO n 1998 [12], and ths parameter has been adopted n almost every PSO varaton. Other mportant modfcatons to the basc PSO model nclude Bare Bones PSO proposed by Kennedy [5], and the ea to clamp veloctes [3] proposed by Eberhart et al.. Ioan gave a good analyss of PSO s convergence and parameter selecton problem n [14]. 2.2 Bnary Partcle Swarm Optmzaton In order to use PSO to solve dscrete problems, Kennedy proposed BPSO [7], whch became the most wely used PSO model dealng wth combnatoral optmzaton problems. Kennedy manly made two changes to the contnuous PSO, and the frst s the representaton of a partcle s poston. Unlke n the contnuous PSO where a partcle s poston s a set of real values, n BPSO, a partcle s poston becomes a bnary strng lke The second change s that the velocty of a partcle no longer represents the drecton of a partcle s movement, rather, t means the possblty of choong 1 at a specfc bt of a partcle s poston component. And the update fomula of the velocty component becomes the followng one. x = { 1, rand < S(v ) 0, otherwse. (3) where 1 S(v ) = 1 + e v There are many applcatons of BPSO, ncludng usng BPSO to solve feature subset selecton problems [10] [15], the travellng salesman problem [11], the lot szng problem [13], flowshop schedulng problems [9], and many other combnatoral optmzaton problems. (4) 3 Feature Selecton wth FPSBPSO In ths secton, we wll propose a modfed BPSO algorthm called FPSBPSO, whch suts the nature of PSO n dscrete space better. Then we wll perform a feature subset selecton wth the concept of FPSBPSO and test whether our proposed method can be benefcal to the classfcaton process. 3.1 Problems of Tradtonal BPSO In the case of bnary space, the tradtonal velocty component becomes useless because partcles now jump rather than move. In BPSO, veloctes become the probablty of choosng 1 n the next teraton, whch s a reasonable nterpretaton. However, n BPSO, the update formula of veloctes keep unchanged, whch s not reasonable. In classc PSO models, veloctes are calculated usng dstance

5 5 nformaton, for example, the dstance between p and ts global best partcle p gb. It makes sense to use dstance measures to calculate veloctes n contnuous space because partcles are movng from one poston to another poston. However, n bnary space, t does not seem to make sense to follow the same ea. There are actually only three possble dstance values n bnary space, 0, 1, and 1, and t s nature to thnk why these values can determne the possblty of jumpng from 0 to 1 or vce versa, why not use 2, 0, and 2? Or other real numbers? we thnk t s not reasonable to use dstance nformaton to calculate veloctes n dscrete space. There s another problem PSO wll have to face n dscrete space. As we all know, partcles tend to follow a leader of the whole swam or a small group. In contnuous space, the result of ths strategy s that a partcle wll be closer to ts personal best and global best n every dmenson n the next teraton. However, n bnary space, the concept of closer can not work well. We can stll say two partcles are closer when they share the same values on many dmensons, but we can only say two partcles p and p j are closer n the d th dmenson when x d = xd j. Now closer means equalty, whch s a problem. The leader partcle s choosen by comparng every partcle s overall performance, whch means that the leader partcle does not have to make the best choce on every dmenson to be selected as the leader. However, the leader partcle wll effect other partcles choce on every dmenson, causng them to move toward t. In contnuous space, the strategy works okay because partcles wll move to new postons closer to the leader partcle, but not the same poston. Whle n bnary space, snce movng closer to the leader partcle wll decrease the dversty of the swarm because partcles tend to select the same value wth the leader partcle, whch wll lead to premature convergence. 3.2 FPSBPSO: Ftness Proportonate Selecton BPSO The two problems dscussed above greatly affect PSO s performance n dscrete space and we propose solutons to them n ths paper. For the frst problem, we beleve t s better to use performance nformaton rather than dstance nformaton to dece whch state wll be selected. Better performance means hgher possblty to be selected, whch s a reasonable strategy commonly known as ftness proportonate selecton. The soluton to the second problem s based on the frst soluton. In bnary space, x d can only be 0 or 1. For a partcle p, ts global best partcle p gb has the best overall performance, however, ts selecton on the d th bt may not be good. Assume p s global best select 1 on the d th bt. If p s personal best partcle or p tself make the same selecton wth the global best partcle on the d th bt, then the average performance of 1 wll be worse than s sngle performance. So, we use average performance of 0 and 1 to calculate the possblty of selectng 0 and 1, whch can depress the tendency of movng towards the leader partcle to a lower level and thus mantan the dversty of the swarm. Now we descrbe the newly desgned update formulas for poston component and velocty component whch are the essental part of the new model. We adopt p gb

6 6 ftness proportonate selecton to update a partcle s poston component, and that s why we call the new model FPSBPSO. In the followng content, when we refer to nvolved partcles n the follong content, we actually mean the three partcles p, p s personal best partcle p pb newly desgned update formulas are as follows. mr, f n 0 == 0 v t+1 = 1 mr, f n 1 == 0 f 1 f 1 +f 0, otherwse. and p s global best partcle p gb. The (5) { x t+1 0, rand < v t+1 = 1, otherwse. (6), Where n 0 s the number of nvolved partcles wth x = 0 and n 1 s the number of nvolved partcles wth x = 1. f 1 and f 0 are computed as follows: Frst, nvolved partcles are dved nto two sets S 1 and S 0 based on whether they select 1 or 0 at the d th bt. Then, f 1 and f 0 are calculated by averagng the ftness values of partcles n S 1 and S 0 respectvely. v t+1 s the probablty of settng x t+1 to 0. The hgher f 1 s, the hgher probablty x t+1 wll be 0. In ths new method, the update formula for veloctes s dfferent from that of tradtonal BPSO. Frst, n the proposed method, the velocty component s calculated usng the current state nformaton, whle n tradtonal PSO models, prevous veloctes have an mpact on current veloctes. However, abandonng prevous veloctes does not mean that the prevous drecton nformaton s lost n FPSBPSO. We use prevous state to play the role of prevous veloctes n classc PSO. Beses, the new veloctes are deced by partcles ftness values, whle veloctes n tradtonal PSO models are calculated manly based on partcles dstance nformaton, for example, the dstance between p and ts global best partcle p gb. As we have argued n prevous subsecton, t s not reasonable to use dstance nformaton to update veloctes. So we desgned a vote-lke s- trategy n the new model. f 1 and f 0 represents the average performance of 1 and 0 respectvely, and better average performance means that the correspondng bt value wll have hgher chance to be selected. In ths way, we can also solve the second problem we proposed n the prevous subsecton. Our proposed FPSBPSO method has another advantage, whch s that there s only one free parameter n FPSBPSO mr. Fewer parameters means that the model wll be easer to be tuned because the dffculty of fndng a good combnaton of parameter settngs wll ncrease exponentally as the number of parameter ncreases. The parameter mr s ntroduced to prevent the problem of premature convergence. In the proposed model, f x pb = xgb = xt, then xt+1 wll be fxed. For example, f x pb = xgb = xt = 1, then vt+1 wll be 0, whch means x t+1 wll be set to 1 wth probablty 1. Ths wll lead to premature convergence. In ths case, the parameter of mr wll force the bt to mutate at tmes, whch s often seen n generc algorthms. How to select a proper value of mr s an mportant problem n the new model. By comparng the results of FPSBPSO usng dfferent settngs of mr, we found

7 7 two basc regulartes. Frst, f mr s set to 0 or a too small value, premature convergence wll happen. Second, f mr s set to a large value, then t wll be very dffcult for the swarm to convergence. Expermental results show that 0.01 s a good choce for many dfferent problems. However, we suggest that t s better to use early-exploraton-late-explotaton strategy, whch means settng mr to a much larger value n the early stage so that partcles can explore more space and gradually decreasng mr to ensure that the partcles can pay more attenton to explotng the neghbourhood nformaton n the later stage. 3.3 FPSBPSO Based Feature Subset Selecton In most classfcaton tasks, the process of feature selecton must be executed, durng whch a smaller set of all features s ganed n order to reach better classfcaton performance. There are many feature selecton methods, such as nformaton gan based method, lexcon based method, generc algorthm based method and so on. PSO can be used to perform feature selecton as well. Smlar to the generc algorthm based method, every feature selecton result s transformed nto a bt strng n PSO based method. The length of strng s equal to the sze of all dfferent features and the value of each bt represents whether the correspondng feature s selected or not. The detaled algorthm s shown n Table 1. As the man am of feature selecton s to rase the classfcaton accuracy, the ftness value of a partcle s represented by the classfer s accuracy usng the correspondng feature subset. Our am s to use FPSBPSO to perform the feature selecton process to obtan a feature subset whch s supposed to get better classfcaton results than that brought by a tradtonal-bpso-based one. 4 Expermental Results In ths secton, we am to prove FPSBPSO s effectveness n feature selecton. Ths s done through two steps. We frst compare the performance of tradtonal BPSO and FPSBPSO on the MKP problem to show that our modfcaton on tradtonal BPSO s reasonable. Then we utlze FPSBPSO nto the feature selecton n a bnary classfcaton problem to test whether the feature subset can mprove the accuracy of classfcaton. 4.1 Experments for the Problem of MKP Frst we want to prove that FPSBPSO works better than tradtonal BPSO n bnary space by comparng the two algorthms on the problem of MKP. MKP s a an ntensvely studed dscrete optmzaton problem because many real world problems can be formulated as a MKP, for example, cargo loadng, cuttng stock problems. MKP s a constraned optmzaton problem, whch means that there exsts llegal soluton space. In our tests, we dn t try to gnore or avo llegal soluton space, nstead, we add a penalty term nto the ftness functon of

8 8 Table 1. FPSBPSO Based Feature Selecton FPSBPSO Based Feature Selecton 1: Intalze parameters of BPSO 2: Randomly ntalze swarm 3: WHILE stoppng crteron not met DO 4: calculate each partcle s ftness functon 6: For = 1 to swarmsze DO 7: update the lbest of P 8: update the pbest of P 9: END 10: FOR = 1 to swarmsze DO 11: FOR j = 1 to dmenson DO 12: update the velocty of P accordng 13: to equaton (5) 14: update the poston of P accordng 15: to equaton (6) 16: END 17: END 18: END 19: Return the best feature subset found by the swarm FPSBPSO. The fnal ftness functon we used was proposed by Khur et al. n [8]. n f(x) = p x s maxp (7) where 1 s = j m w j x > c j (8) 1 Where p s the value of tem, w j s the cost of tem for knapsack j, and c j s the sze of knapsack j. n s the number of tems and m s the number of constrants. The goal s to maxmze n n p x wth the constrans j, w j x <= 1 c j. We used 7 benchmarks named mknap1 n OR-Lbrary [2] to test FPSBPSO s performance on MKP. Optmums of all the 7 benchmarks are all already known. Detaled nformaton of the 7 benchmark problems are gven n Table 2, where n s the number of tems and m s the number of constrants and the column ttled Best shows the known best of the correspondng problem. For standard BPSO, we use the followng parameter settngs: w = , c = , and V max = 4. For FPSBPSO, mr decreases from 0.1 to as teraton process goes on usng formula gven follows. mr = (1 ter/max ter) (9) 1

9 9 Where ter s the current number of teratons and Max ter s the maxmum number of teratons. Max teraton tmes and populaton sze are 1000 and 40 respectvely. In order to reduce the tme cost of tranng, the teraton process wll mmedately stop f the value of global best has kept unchanged for over 120 tmes. For each problem, we run BPSO and FPSBPSO for 20 tmes. We record the average and best value found durng the 20 tmes. Expermental results are gven n Table 3. As can be seen, BPSO succeeded n fndng the best soluton n 4 out of 7 problems whle FPSBPSO succeeded n all 7 problems. Beses, FPSBPSO has the best average performance n most of the problems, and ts advantage over BPSO wll expand as the dmenson of problems ncreases. In a word, FPSBPSO has better performance than BPSO, especally n hgh-dmensonal problems. Table 2. MKP Benchmark n m Best Known n m Best Known Table 3. MKP Results BPSO FPSBPSO Best Avg. Best Avg Feature Selecton wth FPSBPSO Our next goal s to evaluate FPSBPSO s performance n feature selecton doman. To acheve ths, we use the benchmark dataset of Madelon to perform the bnary classfcaton. We use basc BPSO method and FPSBPSO method to get two dfferent feature subsets, then the two subsets are separately used on the dataset to test the classfcaton accuracy. In both cases, the classfcaton algorthm s the CART decson tree. In ths experment, each algorthm s ran for 5 tmes wth dfferent ntalzaton settngs. The expermental results are shown n Table 4 and Table 5. From Table 4, we can know that the average accuracy of results by BPSO s From Table 5, we can see the average accuracy of results by FPSBPSO s , whch s over that by BPSO by 3%. The accuracy mproves much. However, It may take

10 10 more teraton tme for FPSBPSO to return the feature selecton result, whch can be explaned as that FPSBPSO s more precous than tradtonal BPSO and can avo the premature convergence. Table 4. Expermental results by BPSO tme accuracy teraton tmes average Table 5. Expermental results by FPSBPSO tme accuracy teraton tmes average Conclusons and Future Work In ths paper, we dscussed the feasblty of utlzng PSO based methods to select feature subset for classfcaton problems. Frst, we nvestgated the detal of tradtonal BPSO model and some of ts varatons, from whch we argue that BPSO s not a good bnary verson of PSO as t doesn t follow the essence of PSO n some cases. By analysng two major problems PSO wll face n solvng dscrete problems, we proposed a new bnary verson of PSO called FPSBPSO. We redefned the velocty component n FPSBPSO to make t sut the character of PSO better and more computatonally effcent. Expermental results showed that the feature selecton result from our proposed FPSBPSO based method s superor to the one from tradtonal BPSO, as the former one can reach better classfcaton results than the latter one. Although prelmnary results show that the new model can prove sol performance, there are stll many problems to be dealt wth. Frstly, n the proposed

11 11 scheme we used only three partcles nformaton to calculate the average performance of 1 and 0 at a specfc bt, t remans to be seen f more partcles nformaton could mprove the model s performance. Secondly, we have only e- valuated the model s performance on only one benchmark dataset, whch s not suffcent enough. More experments need to be taken to prove the valty and stablty of the FPSBPSO based feature selecton method. In the future, we wll manly focus on the two problems. References 1. A. Abbas, H. Chen, and A. Salem. Sentment analyss n multple languages: Feature selecton for opnon classfcaton n web forums. ACM Transactons on Informaton Systems (TOIS), 26(3):12, J. E. Beasley. Or-lbrary: dstrbutng test problems by electronc mal. Journal of the Operatonal Research Socety, 41(11): , R. Eberhart, P. Smpson, and R. Dobbns. Computatonal ntellgence PC tools. Academc Press Professonal, Inc., San Dego, CA, USA, R. Hassan, B. Cohanm, and O. de Weck. A comparson of partcle swarm optmzaton and the genetc algorthm. In Proceedngs of the 1st AIAA multscplnary desgn optmzaton specalst conference, Aprl J. Kennedy. Bare bones partcle swarms. In Proceedngs of IEEE Swarm Intellgence Symposum, pages 80 87, Aprl J. Kennedy and R. Eberhart. Partcle swarm optmzaton. In Proceedngs of IEEE Internatonal Conference on Neural Networks, volume 4, pages , J. Kennedy and R. Eberhart. A dscrete bnary verson of the partcle swarm optmzaton. In Proceedngs of IEEE Internatonal Conference on Systems, Man, and Cybernetcs, Computatonal Cybernetcs and Smulaton, volume 5, pages , October S. Khur, T. Back, and J. Hetkotter. The zero/one multple knapsack problem and genetc algorthms. In Proceedngs of the 1994 ACM Symposum on Appled Computng, pages , Aprl C.-J. Lao, C.-T. Tseng, and P. Luarn. A dscrete verson of partcle swarm optmzaton for flowshop schedulng problems. Computers & Operatons Research, 34(10): , X. Lu and L. Shang. A fast wrapper feature subset selecton method based on bnary partcle swarm optmzaton. In Proceedngs of IEEE Congress on Evolutonary Computaton, pages , June X. Sh, Y. Lang, H. Lee, C. Lu, and Q. Wang. Partcle swarm optmzaton-based algorthms for tsp and generalzed tsp. Informaton Processng Letters, 103(5): , March Y. Sh and R. Eberhart. A modfed partcle swarm optmzer. In Proceedngs of IEEE World Congress on Computatonal Intellgence, pages 69 73, May M. F. Tasgetren and Y.-C. Lang. A bnary partcle swarm optmzaton algorthm for lot szng problem. Journal of Economc and Socal Research, 5(2):1 20, I. C. Trelea. The partcle swarm optmzaton algorthm: convergence analyss and parameter selecton. Inf. Process. Lett., 85(6): , B. Xue, M. Zhang, and W. N. Browne. Novel ntalsaton and updatng mechansms n pso for feature selecton n classfcaton. In EvoApplcatons, pages , 2013.