Splicing System Based Genetic Algorithms for Developing RBF Networks

Transcription

1 Chin. J. Chem. Eng., 5() (007) Spicing System Based Genetic Agorithms for Deveoping RBF Networks Modes * TAO Jii( 陶吉利 ) and WANG Ning( 王宁 )** Nationa Laboratory of Industria Contro Technoogy, Institute of Advanced Process Contro, Zhejiang University, Hangzhou 3007, China Abstract A spicing system based genetic agorithm is proposed to optimize dynamica radia basis function (RBF) neura network, which is used to extract vauabe process information from input output data. The nove RBF network training technique incudes the network structure into the set of function centers by compromising between the conficting requirements of reducing prediction error and simutaneousy decreasing mode compexity. The effectiveness of the proposed method is iustrated through the deveopment of dynamic modes as a benchmark discrete exampe and a continuous stirred tank reactor by comparing with severa different RBF network training methods. Keywords RBF network, structure optimization, genetic agorithm, spicing system INTRODUCTION Radia basis function (RBF) networks attracted considerabe interest in the past because of its severa advantages compared with other types of artificia neura networks (ANNs), such as better approximation capabiities, simper network structures, and faster earning agorithms[]. However, the seection of appropriate number of basis functions is a critica issue for RBF networks[]. The number of basis functions contros the compexity of the structure, i.e., the generaization capabiity of RBF networks. A RBF network, containing very few basis functions, yieds poor predictions on new data, i.e., poor generaization, as the mode has imited fexibiity. The RBF network, containing severa basis functions, aso yieds poor generaization, as it is too fexibe and fits the noise in the training data. The best generaization performance is obtained via the compromise between the conficting requirements of simutaneousy reducing the prediction error and decreasing the compexity of the mode. This trade-off highights the importance of optimizing the compexity of RBF network to achieve the best generaization. More specificay, most of the standard RBF training methods require the designer to fix the network structure. These training procedures usuay proceed via two steps[3]: First, the centers of basis function are determined using custering method. Second, the cacuation of the fina-ayer weights is reduced to sove a simpe inear system using east squares method. Therefore, the first stage is an unsupervised method, and separated from the actua objective to minimize the output prediction error. In this study, the RBF networks are constructed using the input data supervised by the output data. The incusion of the structure seection in the formuation of the network optimization probem is desirabe, but it resuts in a rather difficut probem, which cannot be easiy soved using the standard optimization methods. An interesting aternative for soving this compicated probem is offered by the use of the recenty deveoped evoutionary computation methods. Perhaps the most popuar and successfu strategies are the so-caed genetic agorithms (GAs), which are stochastic methods based on the principes of natura seection and evoution[4]. GAs have proved to be successfu in the structure seection of severa types of neura networks, such as BP neura networks[5,6] and recurrent neura networks[7,8]. As to the optimization of RBF networks, Vesin and Gruter used GA to sove the compete optimization probem, but the centers of the potentia nodes were restricted among the set of training data[9]. Esposito et a. empoyed a GA based technique to determine the widths of Gaussian functions in RBF networks[0], whereas Sarimveis et a. used GA approach to optimize the parameters of RBF networks in terms of the error minimization criterion[]. In this study, the structure seection is incuded, and the fitness of each chromosome is cacuated on the basis of the prediction error and the structure compexity criterion. To simpify the optimization of RBF network, the radia basis function is chosen as thin-pate-spine function[], where the determination of widths is not required. Therefore, the GA in this study is used to determine the centers of basis functions and the network structure. The fina-ayer weights are derived using recursive east squares (RLS) method with the same initia weight vector. The proposed agorithm starts with a random popuation of RBF networks, which are coded as chromosomes. As a the function centers generated by stochastic chromosomes are not feasibe, two nove operators, i.e., eongation and deetion, enightened by DNA spicing system[3,4], are introduced in the GA approach. SPLICING SYSTEM BASED GA FOR RBF NETWORKS Generay, the determination of the RBF centers Received , accepted * Supported by the Nationa Natura Science Foundation of China (No.60400), and the Nationa High Technoogy Research and Deveopment Program of China (863 Program, 006AA040308). ** To whom correspondence shoud be addressed. E-mai: nwang@iipc.zju.edu.cn

2 Spicing System Based Genetic Agorithms for Deveoping RBF Networks Modes 4 is based on a sef-organizing custering approach, such as k-means custering[5], the nearest neighbor custering method[6]. The appication of the above agorithms requires the transcendenta knowedge of an appropriate custering degree which is difficut to determine, and it considers ony the input data. The proposed approach in this study does not require the transcendenta knowedge of the pant; moreover, the structure and RBF centers can be synchronousy optimized by utiizing the input output data. GA is an optimization agorithm on the basis of Darwinism, which is very fexibe. Depending on the features of the probem's soution space, there is a wide range of choices of fitness functions, the coding method, and the genetic operations, and a these factors affect the efficiency of genetic agorithm. This study is focused on the optimization of RBF network using the spicing system based GA.. Coding method There are totay n r n rea number parameters to be optimized in the RBF network, which means one chromosome shoud be abe to give n r n rea number vaues, where n r is the number of hidden nodes, n is the number of input nodes. Hence, binary coding chromosome wi become very compex, and decima coding chromosome is used. The structure of the th chromosome is shown beow c, c, c, n c, c, c, n C = c n r, c n r, c () n r, n where =,,, L, L is the size of the popuation, n r is randomy produced between and D, D is the maximum number of hidden nodes, the rows beow n r are set to zeros and do not correspond to the center. The eements of C are computed using the foowing equation: c = x + r ( x x ) j j,min j,max j,min nr, j n () where r is the random number between 0 and, x j,min and x j,max is the minimum and the maximum vaues of input variabes given in the probem.. Fitness function As mentioned in the above sections, the drawbacks of the genera training methods of RBF network mainy ie in the absence of goba optimization of both the approximation capabiity and the generaization performance. To overcome these drawbacks, the choice of appropriate fitness function is crucia. In this study, the training procedures using spicing system based GA are aso preceded in two steps: First, the network structure and the basis function parameters are determined using the chromosomes of one popuation. Second, the fina-ayer weights are cacuated using east squares method. As the direct east squares method cannot obtain the soutions for a bad-conditioned matrix, the output weights of the th RBF neura network are cacuated using the foowing RLS method[7]: T w( k) = w( k ) + K( k)[ y( k) Xr ( k) w( k )] T K( k) = P( k ) Xr( k)[ Xr ( k) P( k ) Xr( k) + ] T T P( k) = P( k ) K( k) K ( k)[ Xr ( k) P( k ) Xr( k) + ] (3) where k N, N is the maximum iterative time, X r (k) is the n r dimension output vector of the hidden ayer, y(k) is the output of the actua system, K(k) is the n r dimension assistant vector, P(k) is the n r -by- n r assistant matrix. From Eq.(3), the computationa compexity of RLS soution for one iterative time is ob- ( ) tained as ( ) O n r. Hence, the compexity of RLS soution for the th network weight vector is ( ( ) ) O N n. r In every generation of GA, the cacuation of the output weights competes the formuation of L RBF networks, which can be represented by the pairs (C, w ), (C, w ), and (C L, w L ). To obtain good generaization capabiity of RBF networks, the training data are divided into two groups, one group of data (X, Y ) are used to cacuate the fina-ayer weights, herein, N=N (N is the number of the first group data), and the other group of data (X, Y ) are utiized to evauate the produced RBF networks in each generation. This scheme incorporates a testing procedure into the training agorithm, and guarantees good generaization performance of the RBF networks. However, to obtain good approximation capabiity of RBF networks, the network structure sti becomes much compex. The structure compexity of RBF network conficts with the generaization performance of neura networks. Therefore, the objective function considering both approximation capabiity and generaization performance is shown as foows. N ˆ J( C, w ) = Nn Y( t) Y( t) + ηnr n( N) (4) t = Equation (4) expresses a compromise between the cost of modeing errors and the compexity of network structure[3], where Y ˆ () t is the output of RBF network, N is the number of the second group data, n r is the number of hidden nodes in the th chromosome, η is the weight coefficient, and 0 η, the greater the η is, the stronger constraint of structure compexity woud be considered. In this study, η is set as. Chin. J. Ch. E. 5() 40 (007)

3 4 Chin. J. Ch. E. (Vo. 5, No.).3 Operators in spicing system based GA Li et a.[3] summarized a possibe operations of the DNA spicing systems, such as eongation operation, deetion operation, absent operation, insertion operation, transocation operation, transformation operation permutation operation, etc. Above operations actuay incude three basic operations: seection, crossover, and mutation. Other operations adopted by standard genetic agorithm (SGA) for specia probems may improve the performance of SGA..3. Seection operator A set of individuas from the previous popuation must be seected for reproduction. This seection depends on their fitness vaues. Individuas with good fitness vaues wi most probaby survive. There exist different types of seection operators, and in this study rouette whee method is appied. The probabiity of the seected individua, P(C ), is given by: L P( C ) = f( C ) f( C ) (5) = where f(c ) is the fitness function of the individua C, which is obtained by /J(C, w ). The rouette whee is paced with L equay spaced pointers. A singe spin of the rouette whee wi simutaneousy pick a the members of the next popuation.as the computationa compexity in Eq.(4) is O(N ), the compexity of seection operator in one generation is O(LN )..3. Crossover operator The crossover operator is appied after seection with a probabiity (p c ), which produces nove individuas, i.e., the nove structure and centers of RBF networks. It is executed between the currenty seected individua C and its subsequent individua C +, and yieds the offspring chromosomes C, C. As the number of input nodes (n) is fixed during the whoe GA optimization process, the crossover point is chosen between and n. The procedure is demonstrated in Fig., which represents the singe-point crossover. As the computationa compexity of the pus operator for two D-by-n matrices is O(nD), in the worst case (a chromosome pairs execute crossover operator), the crossover operator in one generation requires O(LnD/) computations..3.3 Mutation operator To effectivey expore the search space, mutation is carried out. When the eement of an individua is ' ' + mutated with a probabiity (p m ), it is repaced by a nove generated eement in terms of Eq.(). Simiar compexity can be obtained as in the case of crossover operator, which is O(LnD)..3.4 Spicing operators As shown in the description of the crossover operation, different structures of RBF networks can be produced using this genetic operator. However, the operator may yied unreasonabe RBF structure as shown in Fig., where most of the centers in row 5 and row 6 of C ' are zeros. Moreover, the crossover operator does not aways modify the structure of the parent chromosomes. Therefore, enightened by the DNA spicing system[3], two more genetic operators, i.e., eongation operator and deetion operator are introduced. The eongation operator is used to add a nove node, and a random nonzero vector is created as described in Eq.(), whereas the deetion operator is utiized to repace the existing unreasonabe node c (,,, i ci ci c in ) with a zero vector. Suppose r is a random number between 0 and, when p e >r, the eongation operator is executed. In the worst case (the number of hidden nodes in each chromosome add up L to D), the compexity is O( ( D nr ) n = ). The deetion operator is executed in the pace of existing unreasonabe node, which is determined by the number of zeros in the node centers. If the number of zeros goes beyond, the node is considered as an unreasonabe one and wi be repaced by deetion operator. In the worst case (a nodes are deeted), the computationa compexity is O( nr n) L. =.4 The procedure of spicing system based GA The whoe processes of the optimization of RBF networks are described in the foowing steps. Step : Generate the code for L chromosomes randomy in the search space. Step : Cacuate the corresponding L weight vectors using RLS method and compute the performance index f for each individua. Step 3: Seect the chromosomes for the generation of new chromosomes of the next generation according to the seection operator. Figure Schematic diagram of the crossover operation Apri, 007

4 Spicing System Based Genetic Agorithms for Deveoping RBF Networks Modes 43 Step 4: Choose a point randomy in the range [, n ], and exchange the codes of the pairs of chromosomes reproduced in Step 3. Repeat this for a the p c L/ pairs of parents. Step 5: Impement mutation, repace the eement of the current chromosome with the nove eement generated by Eq.(). Step 6: Execute the spicing operators, when the conditions are met. Step 7: Repeat Steps to 6 unti a termination criterion is met. This can be the set of maximum number of evoutions, or the set of minimum improvement of the best performance in successive generations. Moreover, Eitism, the incusion of the best current set in the next popuation, is used throughout. Considering the compexity of one generation in the whoe procedure, the basic operations of one generation being performed and the worst case compexities associated with it, are as foows: () L RLS soutions of weight vector is L O( N( nr) ) ; = () Seection operator is O(LN ); (3) Crossover operator is O(LnD/), (4) Mutation operator is O(LnD); L (5) Eongation operator is ( ( O D n r ) n = ), and L (6) Deetion operator is O( n rn = ). As can be seen, the overa compexity of the ( ) L above agorithm is O N( nr) = number of nodes ( ) r. Once the n in the hidden ayer increases, the computation significanty increases. Moreover, as the eitism strategy is adopted throughout the whoe evoutionary procedure, the proposed GA can be competey converged[8]. 3 SIMULATION RESULTS The performance of the proposed methodoogy is evauated by appying it on two different systems: a noninear benchmark probem described by a discrete input output mode and a noninear continuous stirred tank reactor. 3. Simuation tests on a discrete input output mode The discrete input output mode is described as foows[9]. yk ( ) = yk ( ) yk ( ) yk ( 3) uk ( ) [ yk ( 3) ] + u( k ) + yk ( ) + yk ( 3) (6) where πk sin( ), k uk ( ) = πk πk 0.8sin( ) + 0.sin( ), k > The objective of this appication is to utiize the proposed methodoogy to obtain suitabe RBF configuration for modeing the aforementioned system. The input of RBF mode consists of two previous vaues of u and three previous vaues of y. x ( k) = [ u( k ) u( k ) y( k ) y( k ) y( k 3)] (7) The proposed methodoogy is compared with nearest neighbor custering method and k-means custering method, which are used to train the centers of RBF network. 000 data points are produced according to Eq.(6), where the first 500 data points are used to optimize the RBF networks, and the other 500 data points are used to test the performance of the given RBF networks. The operationa parameters used by the proposed agorithm can be seen in Tabe. Tabe GA parameters used in the discrete mode and CSTR exampes Agorithm parameters Discrete exampe CSTR exampe number of chromosomes L maximum of hidden nodes D number of generations G probabiity of crossover p c probabiity of mutation p m probabiity of eongation p e The simuation resuts obtained using the above two methods and the proposed GA are shown in Figs. 5. Fig. shows the fitting curve of prediction vaues and rea vaues using nearest neighbor custering method, and the pots of the estimation error are depicted in Figs.3 5. As the nearest neighbor custering method was used for cacuating the number of Gaussian functions and their corresponding function centers, σ shoud be set in advance. In Fig.3, σ is seected as 0. using tria and error method. In Fig.4, the structure of RBF network is obtained using k-means custering method with 0 custers in terms of nearest neighbor custering method, and σ is optimized using SGA. The number of hidden nodes in RBF network and the sum of absoute vaues of modeing error (S) using the above 3 methods are isted in Tabe. By comparing the resuts in Tabe and the pots in Figs.3 5, it can be observed that the proposed approach minimizes the error to a sma extent using an amost equivaent network structure. Moreover, it is needess to set the network parameter, such as σ to 0. and custers to 0. Tabe Simuation resuts using the 3 different methods Methods Number of hidden nodes S nearest neighbor custering k-means custering & SGA proposed GA Chin. J. Ch. E. 5() 40 (007)

5 44 Chin. J. Ch. E. (Vo. 5, No.) Figure RBF network predictions using nearest neighbor custering method neura predictions; rea vaues Figure 3 Estimation of error using nearest neighbor custering method Figure 4 Estimation of error using k-means custering method and SGA Figure 5 Estimation error using proposed method 3. Simuation tests on a continuous stirred tank reactor In this study, a nonisotherma CSTR process is considered, which is characterized using the foowing dynamic equations[0]: x x = x + Da( x) exp + x ϕ x x = ( + δ ) x + BDa( x) exp + δ u + x ϕ (8) where x and x represent the dimensioness reactant concentration and reactant temperature, respectivey, the physica parameters in the CSTR mode equations are: Da, φ, B and δ, which correspond to the Damökher number, the activated energy, heat of reaction, and heat transfer coefficient, respectivey. The vaues of the system parameters can be obtained as: Da= 0.07, φ=0, B=8, δ=0.3. The contro input, u, is the dimensioness temperature of the cooant. Because of the hard input constraints, u ies between -5 and 5. The objective is to buid discrete dynamic modes for predicting the state variabe x and x using the input output data. The input vector of RBF network mode is seected as foows. x ( k) = [ u( k ) u( k ) u( k 3) y( k) y( k )] (9) The proposed method is appied to optimize both the structure and the centers of RBF networks. As the probem is reativey compex, the number of hidden nodes obtained using nearest neighbor custering is 69 corresponding to σ=0.3, which is too compicated to be adopted. Hence, the resuts of the proposed agorithm are compared with another RBF identification scheme, i.e., the improved S&A GA in Ref.[]. The same radia basis function is adopted and the maximum number of hidden nodes is chosen as 50. For a simuations that foow, a set of 700 input output data points are created by randomy seecting the vaues of the input variabe within the space [-5, 5]. The first 300 data are used in the training procedure to cacuate the connection weights; the second 00 data are aso used during the training process to evauate the RBF network in each generation, whereas the remaining 00 data are used to test the efficiency of the utimate RBF network. To test the generaization performance of RBF networks, the vaues of the output variabes are modified by adding random noise chosen from a uniform distribution at the interva [-3%, +3%] of the maximum vaues of the given variabe. To evauate the approximation capabiity and generaization performance, both the training error and testing error are compared as shown in Figs.6 9. Once the best structure and the centers of RBF network are optimized using GA according to the first and second data sets, the weights between the hidden ayer and the output ayer are updated using RLS method according to the second 00 data, the training errors are thus obtained, and then, the weights are fixed and the testing errors are cacuated in terms of the third 00 data. Figs.6 and 8 show the responses of RBF network predictor optimized using S&A GA with RLS method because the LS method in Ref.[] coud not aways obtain the inverse of the matrix. By comparing with Figs.7 and 9 through RBF networks using S&A GA, training errors are obtained, which are smaer than that by the use of proposed GA, i.e., the sum of absoute vaues of the training error (S ) using S&A GA is smaer than that by the use of proposed GA, and they demand much more hidden nodes regardess of the compexity of the Apri, 007

6 Spicing System Based Genetic Agorithms for Deveoping RBF Networks Modes 45 Figure 6 Training and testing error for x using S&A GA in RBF network Figure 8 Training and testing error for x using S&A GA in RBF network Figure 7 Training and testing error for x using proposed GA in RBF network Figure 9 Training and testing error for x using proposed GA in RBF network network structure. Moreover, the sum of the absoute vaue of the testing error (S ) are quite simiar, and some points of testing error in Figs.6 and 8 are smaer than that in Figs.7 and 9, which can aso be testified using the maximum testing error (E max ). This may be caused by the over-fitting of RBF network with severa hidden nodes. Tabe 3 shows that, as the number of hidden nodes using S&A GA is much more than that by the use of proposed GA, the runtime has markedy increased, which is consistent with the anaysis of computationa compexity. A agorithms are programmed by MATLAB7.0 using the computer with Ceeron(R) CPU.4GHz and RAM 56MB. 4 CONCLUSIONS This artice presents a spicing system based GA for the optimization of both structure and centers of the RBF network mode. This agorithm is based on Chin. J. Ch. E. 5() 40 (007)

7 46 Chin. J. Ch. E. (Vo. 5, No.) Tabe 3 The simuation resuts using two improved GA x x Methods Number of nodes S S E max Runtime, s Number of nodes S S E max Runtime, s S&A GA proposed GA input-output data, and its objective function considers both approximation capabiity and generaization performance of the RBF network. In this manner, the network simutaneousy retains a reasonabe size and effectivey describes the whoe system. Different simuations of benchmark probem and typica noninear dynamic CSTR system are performed to iustrate the effectiveness of the proposed method. The resuts show that this method can produce highy accurate prediction and keep a reativey simpe network structure. NOMENCLATURE B heat of the reaction C the th n-by-d chromosome matrix c i the ith node center vector D maximum number of the hidden nodes Da Damökher number E max maximum of the testing error f the fitness function G maximum number of generations J the objective function K n r dimension assistant vector in the th network in RLS agorithm L size of the popuation N maximum iterative time n number of the input nodes n r number of hidden nodes of the th network P nr -by- n r assistant matrix in the th network in RLS agorithm p c probabiity of crossover p e probabiity of eongation p m probabiity of mutation r random number between 0 and S sum of the absoute vaue of the modeing error S sum of the absoute vaue of the training error S sum of the absoute vaue of the testing error u contro input of the actua system w weight vector of the th RBF network X r n r dimension output vector of the hidden ayer in RLS agorithm X N n-dimension input vectors X N n-dimension input vectors x input vector of the RBF network x reactant concentration x reactor temperature Y N output vaues of the system Y N output vaues of the system y output vaue of the actua system δ heat transfer coefficient σ width of the Gaussian function φ activated energy REFERENCES Park, J., Sandberg, I., Universa approximation using radia basis function networks, Neura Comput., 3(), 46 57(99). Prechet, L., Automatic eary stopping using cross vaidation: quantifying the criteria, Neura Network., (4), (998). 3 Aexandridis, A., Sarimveis, H., Bafas, G., A new agorithm for onine structure and parameter adaptation of RBF networks, Neura Network., 6(7), (003). 4 Hoand, J.H., Adaptation in Natura and Artificia Systems, MIT Press, Cambridge, MA, USA (99). 5 Wang, Y., Yao, P., Simuation and optimization for thermay couped distiation using artificia neura network and genetic agorithm, Chin. J. Chem. Eng., (3), 307 3(003). 6 Yang, T., Lin, H.C., Chen, M.L., Metamodeing approach in soving the machine parameters optimization probem using neura network and genetic agorithms: A case study, Robot Comput. Integrated Manuf., (4), 3 33(006). 7 Banco, A., Degado, M., Pegaajar, M.C., Rea-coded genetic agorithm for training recurrent neura networks, Neura Network., 4(), 93 05(00). 8 Degado, M., Pegaajar, M.C., A mutiobjective genetic agorithm for obtaining the optima size of a recurrent neura network for grammatica inference, Pattern Recognit., 38(9), (005). 9 Vesin, J.M., Gruter, R., Mode seection using a simpex reproduction genetic agorithm, Signa Process., 78(3), 3 37(999). 0 Esposito, A., Marinaro, M., Oricchio, D., Scarpetta, S., Approximation of continuous and discontinuous mappings by a growing neura RBF-based agorithm, Neura Network., 3(6), (000). Sarimveis, H., Aexandridis, A., Mazarakis, S., Bafas, G., A new agorithm for deveoping dynamic radia basis function neura network modes based on genetic agorithms, Comput. Chem. Eng., 8(), 09 7(004). Chen, S., Cowan, C.F.N., Grant, P.M., Orthogona east squares earning agorithm for radia basis function networks, IEEE Trans. Neura Netw., (), (99). 3 Li, S.C., Xu, J., Pan, L.Q., Operationa rues for digita coding of RNA sequences based on DNA computing in high dimensiona space, Bu. Sci. Tech. Soc., 8(6), (003). 4 Jonoska, N., Seeman, N.C., DNA Computing, Springer-Verag, New York (00). 5 Pham, D.T., Dimov, S.S., Nguyen, C.D., Seection of K in k-means custering, Proc. Inst. Mech. Eng. C. Mech. Eng. Sci., 9(), 03 9(005). 6 Zhang, X., Song, J., RBF neura networks based on dynamic nearest neighbor-custering agorithm and its appication in prediction of MH-Ni battery capacity, Transactions of China Eectrotechnica Society, 0(), 84 87(005). 7 Hou, Y.B., Wang M., Wang, L.Q., System Identification and Its MATLAB Simuation, Science Press, Beijing (004). 8 Li, M.Q., Kou, J.S., Basic Theories and Appications of Genetic Agorithm, Science Press, Beijing (00). 9 Liu, G.P., Kadirkamanathan, V., Biings, S.A., On-ine identification of noninear systems using Voterra poynomia basis function neura networks, Neura Netw., (9), (998). 0 Chen, C.T., Peng, S.T., Learning contro of process systems with hard input constraints, J. Process Contro, 9(), 5 60(999). Apri, 007