A Tunable Radial Basis Function Model for Nonlinear System Identification Using Particle Swarm Optimisation

Transcription

1 A Tunable Radal Bass Functon Model for Nonlnear System Identfcaton Usng Partcle Swarm Optmsaton S. Chen, X. Hong, B.L. Lu and C.J. Harrs Abstract A tunable radal bass functon (RBF) networ model s proposed for nonlnear system dentfcaton usng partcle swarm optmsaton (PSO). At each stage of orthogonal forward regresson (OFR) model constructon, PSO optmses one RBF unt s centre vector and dagonal covarance matrx by mnmsng the leave-one-out (LOO) mean square error (MSE). Ths PSO aded OFR automatcally determnes how many tunable RBF nodes are suffcent for modellng. Compared wth the-state-of-the-art local regularsaton asssted orthogonal least squares algorthm based on the LOO MSE crteron for constructng fxed-node RBF networ models, the PSO tuned RBF model constructon produces more parsmonous RBF models wth better generalsaton performance and s computatonally more effcent. I. INTRODUCTION The radal bass functon (RBF) networ s a popular neural networ archtecture for nonlnear system dentfcaton [1]. In ths applcaton, the parameters of a RBF networ, ncludng nodes centre vectors and varances or covarance matrces as well as connectng weghts, can be estmated based on nonlnear optmsaton usng gradent-descent algorthms [2], [3], the expectaton-maxmsaton algorthm [4], [5] or the populaton-based evolutonary algorthms [6], [7]. However, these nonlnear estmaton methods are computatonally expensve and, moreover, the RBF model structure or the number of RBF nodes has to be determned va other means, typcally based on costly cross valdaton. Clusterng algorthms can alternatvely be appled to fnd the RBF centre vectors as well as the assocated bass functon varances [8] [10]. The remanng RBF weghts can then be determned usng the smple lnear least squares (LS) estmate. However, the number of the clusters agan has to be determned va other means, such as cross valdaton. A most popular approach for dentfyng RBF models s to formulate the problem as a lnear regresson by consderng the tranng nput data ponts as canddate RBF centres and to employ a common varance for every RBF node. A parsmonous RBF networ can then be dentfed effcently usng the orthogonal least squares (OLS) algorthm [11] [14]. Smlarly, the support vector machne (SVM) and other sparse ernel modellng methods [15] [17] also place the ernel centres to the tranng nput data ponts and adopt a common ernel varance for every ernel. A sparse ernel model s then sought. Snce the common varance n ths S. Chen and C.J. Harrs are wth School of Electroncs and Computer Scence, Unversty of Southampton, Southampton SO17 1BJ, UK X. Hong s wth School of Systems Engneerng, Unversty of Readng, Readng RG6 6AY, UK B.L. Lu s wth Department of Manufacturng Engneerng and Engneerng Management, Cty Unversty of Hong Kong, Hong Kong, Chna fxed-node RBF model s not provded by the learnng algorthms, t must be determned va cross valdaton. For the ernel methods, such as the SVM, two more hyperparameters must also be specfed va cross valdaton. Our prevous results [14] show that the local regularsaton asssted OLS (LROLS) algorthm based on the leave-one-out (LOO) cross valdaton compares favourably wth other ernel methods, n terms of model sparsty and generalsaton performance as well as effcency of model constructon. Ths LROLS-LOO algorthm [14] offers a state-of-the-art constructon method for fxed-node RBF models. Ths paper consders the tunable RBF model, where each RBF node has a tunable centre vector and an adjustable dagonal covarance matrx. We do not attempt to optmse all the RBF parameters together, as t s a too large and complex nonlnear optmsaton tas. Instead, we adopt an orthogonal forward regresson (OFR) to optmse RBF unts one by one based on the LOO mean square error (MSE). Specfcally, we use partcle swarm optmsaton (PSO) [18], [19] to optmse one RBF node s centre vector and covarance matrx at each stage of the OFR. PSO s a populaton based stochastc optmsaton technque [18], [19] nspred by socal behavour of brd flocng or fsh schools. It s becomng popular due to ts smplcty n mplementaton, ablty to qucly converge to a reasonably good soluton and ts robustness aganst local mnma. The method has been appled to many optmsaton problems successfully [7], [19] [25]. We demonstrate that the proposed PSO aded OFR algorthm for tunable-node RBF models not only produces sparser models and better generalsaton performance but also offers computatonal advantages n model constructon, compared wth the state-of-the-art LROLS-LOO constructon algorthm for fxed-node RBF models [14]. II. IDENTIFICATION USING TUNABLE RBF MODELS Consder the class of dscrete-tme nonlnear systems that can be represented by the followng NARX structure y = f s (y 1,,y my,u 1,,u mu ) + e = f s (x ) + e (1) where u and y are the system nput and output varables, respectvely, m u and m y are the nown lags for u and y, respectvely, e s a zero-mean uncorrelated observaton nose, f s ( ) denotes the unnown system mappng, and x = [x 1, x 2, x m, ] T = [y 1 y my u 1 u mu ] T (2)

2 denotes the system nput vector wth the nown dmenson m = m y + m u. The NARX system (1) s a specal case of the followng generc NARMAX system [26] y = f s (y 1,,y my,u 1,,u mu, e 1,,e me ) + e. (3) The technques developed for the NARX structure can be extended to the general NARMAX system [11], [26], [27]. Gven the tranng data set D K = {x,y } K =1, the tas s to dentfy the system (1) usng the RBF networ model ŷ (M) ˆf (M) = ˆf (M) RBF (x ) = M θ p (x ) = p T M()θ M (4) where RBF ( ) denotes the mappng of the M-term RBF model, M s the number of RBF unts, θ M = [θ 1 θ 2 θ M ] T s the RBF weght vector, and p T M() = [p 1 (x ) p 2 (x ) p M (x )] (5) s the response vector of the M RBF nodes to the nput x. We consder the general RBF regressor of the form ( ) p (x) = ϕ (x µ ) T Σ 1 (x µ ), (6) where µ and Σ = dag{σ 2,1,σ2,2,,σ,m} are the centre vector and dagonal covarance matrx of the th RBF node, respectvely, ϕ( ) s the chosen RBF bass functon. In ths study, the Gaussan bass functon s employed. Let us defne the modellng error at the th tranng data pont (x,y ) as ε (M) = y ŷ (M). (7) Then the regresson model (4) over the tranng set D K can be wrtten n the matrx form y = P M θ M + ε (M), (8) where y = [y 1 y 2 y K ] T s the desred output vector, ε (M) = [ε (M) 1 ε (M) 2 ε (M) K ]T s the modellng error vector of the M-term model, and the regresson matrx P M = [p 1 p 2 p M ] wth the th regressor gven by p = [p (x 1 ) p (x 2 ) p (x K )] T, (9) where 1 M. Note that p s the th column of P M whle p T M () denotes the th row of P M. A. Orthogonal Decomposton Let an orthogonal decomposton of P M be P M = W M A M, where W M = [w 1 w 2 w M ] wth the orthogonal columns that satsfy w Tw l = 0 for l and 1 α 1,2 α 1,M. A M = αm 1,M. (10) The regresson model (8) can alternatvely be presented as y = W M g M + ε (M), (11) where g M = [g 1 g 2 g M ] T satsfes the trangular system A M θ M = g M. Snce the space spanned by the orgnal model bases p ( ), 1 M, s dentcal to the space spanned by the orthogonal model bases, the RBF model output can equvalently be expressed as ŷ (M) = w T M()g M, (12) where w T M () = [w 1() w 2 () w M ()] s the th row of W M. Orthogonal decomposton can be carred out usng the Gram-Schmdt procedure [11]. Usng the model (11) nstead of the orgnal one (8) facltates an effcent OFR model constructon. In partcular, calculaton of the LOO MSE becomes very fast, mang t possble to construct the model by drectly optmsng the model generalsaton capablty rather than mnmsng the tranng MSE [14]. B. OFR Based on LOO Cross Valdaton The evaluaton of model generalsaton capablty s drectly based on the concept of cross valdaton [28], and a commonly used cross valdaton s the LOO cross valdaton wth ts assocated LOO test MSE [29]. Consder the OFR modellng process that has produced the n-node RBF model. Denote the constructed n columns of regressors as W n = [w 1 w 2 w n ], the th model output of ths n-node RBF model dentfed usng the entre tranng data set D K as ŷ (n) = n g w (), (13) and the correspondng th modellng error as ε (n) = y ŷ (n). If we remove the th data pont from the tranng set D K and use the remanng K 1 data ponts D K \(x,y ) to dentfy the n-node RBF model nstead, the test error of the resultng model can be calculated on the data pont (x,y ) not used n tranng. Ths LOO modellng error, denoted as ε (n, ), s gven by [29] ε (n, ) = ε (n) /η(n), (14) where η (n) s referred to as the LOO error weghtng [29]. The LOO MSE for the n-node RBF model s defned as J n = 1 K K ( =1 ε (n, ) ) 2. (15) The LOO MSE J n s a measure of the model generalsaton capablty [28], [29]. For the model (11), J n can be computed very effcently because ε (n) and η (n) can be calculated recursvely usng [14] n = y g w () = ε (n 1) g n w n () (16) and η (n) ε (n) = 1 n w 2 () w T w + λ = η(n 1) w2 n() w T nw n + λ, (17)

3 respectvely, where λ 0 s a small regularsaton parameter [14]. The regularsaton parameter can smply be set to λ = 0 (no regularsaton) or a very small value (10 6 ). We can use an OFR procedure based on ths LOO MSE to construct the RBF nodes one by one. At the nth stage of the constructon, the nth RBF node s determned by mnmsng J n wth respect to µ n and Σ n mn J n (µ n,σ n ). (18) µ n,σ n In the next secton, we wll detal how to use PSO to perform ths optmsaton. The LOO MSE J n s locally convex wth respect to the model sze n [14]. Thus, there exsts an optmal number of RBF nodes M such that: for n M J n decreases as the model sze n ncreases whle J M J M+1. (19) Therefore, ths OFR constructon process s automatcally termnated when the condton (19) s met, yeldng a very small model set contanng only M RBF nodes. After constructng the M-node RBF model usng ths OFR procedure, we may apply the LROLS-LOO algorthm of [14] to automatcally update ndvdual regularsaton parameter for each RBF weght whch may further reduce the model sze. Ths refnement wth the LROLS-LOO algorthm requres a very small amount of computaton snce the regresson matrx P M s completely specfed wth only a few columns. III. PARTICLE SWARM OPTIMISATION AIDED OFR The tas at the nth stage of the OFR for constructng a tunable RBF networ s to solve the optmsaton problem (18). Ths optmsaton problem s non-convex wth respect to µ n and Σ n, and we adopt PSO [18], [19] to determne µ n and Σ n. Our study demonstrates that PSO s partcularly suted for the optmsaton tas (18). A. Partcle Swarm Optmsaton In a PSO algorthm [19], a group of S partcles that represent potental solutons are ntalsed over the search space randomly. Each partcle has a ftness value assocated wth t, based on the related cost functon of the optmsaton problem, and ths ftness value s evaluated at each teraton. The best poston, pb, vsted by each partcle provdes the partcle the so-called cogntve nformaton, whle the best poston vsted so far among the entre group, gb, offers the socal nformaton. The pbs and gb are updated at each teraton. Each partcle has ts own velocty to drect ts flyng, whch reles on ts prevous speed as well as ts cogntve and socal nformaton. In each teraton, the velocty and the poston of the partcle are updated based on the followng equatons = ξ v [l] + rand() c 1 (pb [l] u [l] ) +rand() c 2 (gb [l] u [l] ), (20) = u [l] +, (21) l teraton ndex, 1 l L, L s the maxmum number of teratons partcle ndex, 1 S, S s the partcle sze v [l] velocty of th partcle at lth teraton. The jth elements of v [l] are n range [ V jmax, V jmax ] ξ nerta weght c j the acceleraton coeffcents, j = 1,2 rand() unform random number between 0 and 1 u [l] pb [l] gb [l] poston of th partcle at lth teraton. The jth elements of u [l] are n range [U jmn, U jmax ] best poston that the th partcle has vsted upto lth teraton best poston that all the partcles have vsted upto lth teraton It s reported n [20] that usng a tme varyng acceleraton coeffcent (TVAC) enhances the performance of PSO. The reason s that at the ntal stages, a large cogntve component and a small socal component help partcles to wander around the search space and to avod local mnma. In the later stages, a small cogntve component and a large socal component help partcles to converge qucly to a global mnmum. We adopt ths TVAC mechansm n whch c 1 for the cogntve component s reduced from 2.5 to 0.5 and c 2 for the socal component vares from 0.5 to 2.5 durng the teratve procedure accordng to c 1 = l 1.0 L and c 2 = l 1.0 L, (22) respectvely. In our experment, we have found out that usng a random nerta weght, ξ = rand(), acheves better performance than usng ξ = 0 or constant ξ. If the velocty n (20) approaches zero, t s rentalsed randomly to proportonal to the maxmum velocty j = ±rand() γ V jmax, (23) where j denotes the jth element of and γ = 0.1 s a constant. B. PSO Aded OFR for Tunable RBF Model The procedure of usng the PSO aded OFR to determne the nth RBF node s now summarsed. Let u be the vector that contans µ n and Σ n. The dmenson of u s thus 2m. The search space s specfed by { = mn{x Ujmn j,,1 K}, 1 j m, (24) U jmax = max{x j,,1 K}, U jmn = σ 2 mn, U jmax = σ 2 max, 1 + m j 2m. (25) The velocty bounds are defned by V jmax = 0.5 (U jmax U jmn ), 1 j 2m. (26) Gve the followng ntal condtons } ε (0) = y and η (0) = 1, 1 K, J 0 = 1 N yt y = 1 K N =1 y2. (27) Specfy the number of teratons L and the partcle sze S.

4 Intalsaton. Randomly generate the partcles u [0], 1 S, n the search space defned by (24) and (25). Set the ntal veloctes v [0] = 0 2m, 1 S, where 0 2m denotes the zero vector of dmenson 2m. Intalse J n (gb [0] ) and J n (pb [0] ), 1 S, to a value larger than J 0. Iteraton loop. For (l = 0; l L; l + +) { Orthogonalsaton and cost functon evaluaton. 1) For 1 S, generate p ) n from u [l], the canddates for the nth model column, accordng to (9), and orthogonalse them accordng to [11] α ) j,n = wt j p ) n/w T j w j, 1 j < n, (28) ( gn ) = n 1 wn ) = p ) n α ) j,n w j, (29) w ) n j=1 ) ( T ( ) T y/ wn ) w ) n + λ ). (30) 2) For 1 S, calculate the LOO cost for each u [l] ε (n) η (n) () = ε(n 1) wn ) ()gn ), 1 K, (31) ( ) 2 wn ) () () = η(n 1) ( ) T, 1 K, ) wn + λ J ) n = 1 K w ) n ( K =1 ε (n) η (n) ) (32) 2 (), (33) () where wn ) () s the th element of wn ). Update cogntve and socal nformaton. 1) For ( = 1; S; ++) If (Jn ) < J n (pb [l] )) J n (pb [l] ) = J) n ; = u [l] pb [l] End f; 2) Fnd ; = arg mn 1 S J n(pb [l] ). If (J n (pb [l] ) < J n(gb [l] )) J n (gb [l] ) = J n (pb [l] ); gb [l] = pb [l] ; End f; Update veloctes and postons of partcles 1) For ( = 1; S; ++) = rand() v [l] + rand() c 1 (pb [l] u [l] +rand() c 2 (gb [l] u [l] ); For (j = 1; j 2m; j++) If ( j == 0) If (rand() < 0.5) j = rand() γ V jmax ; Else j = rand() γ V jmax ; End f; ) End f; If ( j > V jmax ) j = V jmax ; Else f ( j < V jmax ) v l+1 End f; j = V jmax ; 2) For ( = 1; S; ++) = u [l] + ; For (j = 1; j 2m; j++) If ( j > U jmax ) j = U jmax ; Else f ( j < U jmn ) j = U jmn ; End f } End of teraton loop Ths yelds the soluton u = gb [L],.e. µ n and Σ n of the nth RBF node, the nth model column p n, the orthogonalsaton coeffcents α j,n, 1 j < n, the correspondng orthogonal model column w n, and the weght g n, as well as the n- term modellng errors ε (n) and the assocated LOO error weghtngs η (n) for 1 K. C. Computatonal Complexty Comparson We compare the computatonal complexty of the proposed PSO aded OFR algorthm for tunable RBF models wth that of the LROLS-LOO algorthm for fxed-node RBF models [14]. The LROLS-LOO algorthm nvolves a few teratons. The frst teraton wors on the K K full regresson matrx and selects a subset of M RBF nodes, where M K. The computatonal complexty of the algorthm s domnated by ths frst teraton, and the complexty of the rest teratons s neglgble. For the LROLS-LOO algorthm, t can be verfy that the computatonal complexty of one model column orthogonalsaton and the assocated LOO cost functon evaluaton s O(K). Thus, we can characterse the complexty of the algorthm by the requred number of the LOO cost functon evaluatons and assocated model column orthogonalsatons, whch s gven by C LROLS LOO M +1 (K ( 1)) ( M + 1 ) K, (34) where the second approxmaton arrves because M K. Snce for the PSO aded OFR algorthm, the computatonal requrement of one model column orthogonalsaton and the assocated LOO cost functon evaluaton s also O(K), we can also characterse the computatonal requrements of the algorthm by the number of the LOO cost functon evaluatons and assocated model column orthogonalsatons.

5 TABLE I COMPARISON OF THE TWO GAUSSIAN RBF NETWORK MODELS OBTAINED BY THE LROLS-LOO AND PSO-OFR ALGORITHMS. Algorthm RBF type Model sze Tranng MSE Test MSE complexty LROLS-LOO fxed O(500) PSO-OFR tunable O(500) Ths number s gven as C PSO OFR (M + 1) S L, (35) where M s the constructed model sze, S the partcle sze and L the number of teratons. Snce the model sze M s usually much smaller than the model sze M obtaned by the LROLS-LOO algorthm, we always have C PSO OFR < C LROLS LOO whenever K S L. Thus, the PSO aded OFR algorthm for constructng tunable-node RBF models has clearly computatonal advantages over the LROLS-LOO algorthm for selectng fxed-node RBF models when the sze of the tranng data set s large. Note that the PSO algorthm s very effcent. Our expermental results have shown that typcally L = 20 and S = 10 to 20 are often suffcent. Furthermore, the complexty of (35) s the true complexty of the PSO aded OFR algorthm, whle the complexty of (34) s the complexty of the LROLS-LOO algorthm gven a RBF varance. Snce the RBF varance s not provded by the LROLS-LOO algorthm, t must be determned based cross valdaton. Tang ths fact nto account, computatonal advantages of the proposed PSO aded OFR algorthm becomes even more sgnfcant. (b) Fg. 1. Lqud level data: (a) system nput u, and (b) system output y. (a) IV. LIQUID LEVEL DATA SET MODELLING The data set was collected from a nonlnear lqud level system, whch conssted of a DC water pump feedng a concal flas whch n turn fed a square tan. The system nput u was the voltage to the pump motor and the system output y was the water level n the concal flas [30]. Fg. 1 shows the 1000 data ponts of the data set used n ths experment. From the data set, 1000 data ponts {x,y } were constructed wth x gven by x = [y 1 y 2 y 3 u 1 u 2 u 3 u 4 ] T. (36) The frst 500 pars of the data were used for tranng and the remanng 500 pars for testng the constructed model. For the fxed-node RBF model wth every tranng nput data used as a candadate RBF centre vector, an approprate RBF varance was found to be σ 2 = 2.0 va a grd search based cross valdaton usng the LROLS-LOO algorthm [14]. Wth σ 2 = 2.0, the LROLS-LOO algorthm automatcally selected a model set of M = 30 nodes from the canddate set of K = 500 potental nodes. The results obtaned by the LROLS-LOO algorthm are gven n Table I, where the complexty was computed as C LROLS LOO = = for the gven σ 2 = 2.0. For constructng the tunable-node RBF model, we set the partcle sze to S = 10 and the number of teratons to L = 20. The PSO aded OFR algorthm automatcally constructed a model set of M = 20 nodes. The results produced by the the PSO aded OFR are also lsted n Table I, where the complexty was gven by C PSO OFR = = Fg. 2 shows the model predcton ŷ and the predcton error ε = y ŷ produced by the 20-node RBF model constructed usng the PSO aded OFR algorthm. For ths example, the PSO aded OFR algorthm has clear advantages over the benchmar LROLS-LOO algorthm, n terms of model sze and generalsaton capablty as well as complexty of model constructon. V. CONCLUSIONS In ths contrbuton we have proposed a novel PSO aded OFR algorthm to construct tunable-node RBF networ models for nonlnear system dentfcaton. Unle the standard fxed-node RBF model where the RBF centre vectors are placed at the tranng nput data ponts and a common RBF varance s used for every RBF node, the proposed algorthm optmses one RBF node s centre vector and dagonal covarance matrx by mnmsng the LOO MSE at each stage of the OFR. The model constructon procedure automatcally determnes how many tunable nodes are suffcent, and PSO ensures that ths model constructon procedure s computatonally very effcent. Usng the best exstng algorthm for fxed-node RBF models, the LROLS-LOO algorthm, as

6 (a) (b) Fg. 2. Modellng of the lqud level data by the 20-node RBF networ constructed usng the PSO aded OFR: (a) model predcton ŷ supermposed on system output y, and (b) model predcton error ε = y ŷ. the benchmar, t has been shown that the proposed PSO aded OFR algorthm for constructng tunable-node RBF models offers clear advantages over the benchmar LROLS- LOO algorthm for constructng fxed-node RBF models, n terms of more parsmonous model and better generalsaton performance as well as more effcent model constructon. REFERENCES [1] S. Chen, S.A. Bllngs, C.F.N. Cowan and P.M. Grant, Non-lnear systems dentfcaton usng radal bass functons, Int. J. Systems Sc., Vol.21, pp , [2] S. Chen, C.F.N. Cowan, S.A. Bllngs and P.M. Grant, Parallel recursve predcton error algorthm for tranng layered neural networs, Int. J. Control, Vol.51, pp , [3] S. McLoone, M.D. Brown, G. Irwn and A. Lghtbody, A hybrd lnear/nonlnear tranng algorthm for feedforward neural networs, IEEE Trans. Neural Networs, Vol.9, No.4, pp , [4] Z.R. Yang and S. Chen, Robust maxmum lelhood tranng of heteroscedastc probablstc neural networs, Neural Networs, Vol.11, pp , [5] M.-W. Ma and S.-Y. Kung, Estmaton of ellptcal bass functon parameters by the EM algorthm wth applcaton to speaer verfcaton, IEEE Trans. Neural Networs, Vol.11, No.4, pp , [6] J. Gonzalez, I. Rojas, J. Ortega, H. Pomares, F.J. Fernandez and A.F. Daz, Multobjectve evolutonary optmzaton of the sze, shape, and poston parameters of radal bass functon networs for functon approxmaton, IEEE Trans. Neural Networs, Vol.14, No.6, pp , [7] H.-M. Feng, Self-generaton RBFNs usng evolutonal PSO learnng, Neurocomputng, Vol.70, No.1-3, pp , [8] J. Moody and C.J. Daren, Fast learnng n networs of locally-tuned processng unts, Neural Computaton, Vol.1, pp , [9] S. Chen, S.A. Bllngs and P.M. Grant, Recursve hybrd algorthm for non-lnear system dentfcaton usng radal bass functon networs, Int. J. Control, Vol.55, pp , [10] S. Chen, Nonlnear tme seres modellng and predcton usng Gaussan RBF networs wth enhanced clusterng and RLS learnng, Electroncs Letters, Vol.31, No.2, pp , [11] S. Chen, S.A. Bllngs and W. Luo, Orthogonal least squares methods and ther applcaton to non-lnear system dentfcaton, Int. J. Control, Vol.50, No.5, pp , [12] S. Chen, C.F.N. Cowan and P.M. Grant, Orthogonal least squares learnng algorthm for radal bass functon networs, IEEE Trans. Neural Networs, Vol.2, No.2, pp , [13] S. Chen, X. Hong and C.J. Harrs, Sparse ernel regresson modellng usng combned locally regularzed orthogonal least squares and D- optmalty expermental desgn, IEEE Trans. Automatc Control, Vol.48, No.6, pp , [14] S. Chen, X. Hong, C.J. Harrs and P.M. Sharey, Sparse modellng usng orthogonal forward regresson wth PRESS statstc and regularzaton, IEEE Trans. Systems, Man and Cybernetcs, Part B, Vol.34, No.2, pp , [15] V. Vapn, The Nature of Statstcal Learnng Theory. New Yor: Sprnger-Verlag, [16] M.E. Tppng, Sparse Bayesan learnng and the relevance vector machne, J. Machne Learnng Research, Vol.1, pp , [17] B. Schölopf and A.J. Smola, Learnng wth Kernels: Support Vector Machnes, Regularzaton, Optmzaton, and Beyond. Cambrdge, MA: MIT Press, [18] J. Kennedy and R. Eberhart, Partcle swarm optmzaton, n Proc IEEE Int. Conf. Neural Networs (Perth, Australa), Nov.27- Dec.1, 1995, Vol.4, pp [19] J. Kennedy and R. Eberhart, Swarm Intellgence. Morgan Kaufmann, [20] A. Ratnaweera, S.K. Halgamuge and H.C. Watson, Self-organzng herarchcal partcle swarm optmzer wth tme-varyng acceleraton coeffcents, IEEE Trans. Evolutonary Computaton, Vol.8, No.3, pp , [21] S.M. Guru, S.K. Halgamuge and S. Fernando, Partcle swarm optmsers for cluster formaton n wreless sensor networs, n Proc Int. Conf. Intellgent Sensors, Sensor Networs and Informaton Processng (Melbourne, Australa), Dec.5-8, 2005, pp [22] K.K. Soo, Y.M. Su, W.S. Chan, L. Yang and R.S. Chen, Partcleswarm-optmzaton-based multuser detector for CDMA communcatons, IEEE Trans. Vehcular Technology, Vol.56, No.5, pp , [23] T.-Y. Sun, C.-C. Lu, T.-Y. Tsa and S.-T. Hseh, Adequate determnaton of a band of wavelet threshold for nose cancellaton usng partcle swarm optmzaton, n Proc. CEC 2008 (Hong Kong, Chna), June 1-6, 2008, pp [24] F.A. Guerra and L.S. Coelho, Mult-step ahead nonlnear dentfcaton of Lorenz s chaotc system usng radal bass functon neural networ wth learnng by clusterng and partcle swarm optmszaton, Chaos, Soltons and Fractals, Vol.35, No.5, pp , [25] W.-F. Leong and G.G. Yen, PSO-based multobjectve optmzaton wth dynamc populaton sze and adaptve local archves, IEEE Trans. Systems, Man and Cybernetcs, Part B, Vol.38, No.5, pp , [26] S. Chen and S.A. Bllngs, Representaton of non-lnear systems: The NARMAX model, Int. J. Control, Vol.49, No.3, pp , [27] S. Chen, S.A. Bllngs, C.F.N. Cowan and P.M. Grant, Practcal dentfcaton of NARMAX models usng radal bass functons, Int. J. Control, Vol.52, pp , [28] M. Stone, Cross valdaton choce and assessment of statstcal predctons, J. Royal Statstcs Socety Seres B, Vol.36, pp , [29] R.H. Myers, Classcal and Modern Regresson wth Applcatons. 2nd Edton, Boston: PWS-KENT, [30] S.A. Bllngs and W.S.F. Voon, A predcton-error and stepwseregresson estmaton algorthm for non-lnear systems, Int. J. Control, Vol.44, pp , 1986.