On the Use of Neural Network as a Universal Approximator

Transcription

1 Internatonal Journal of Scences and Technques of Automatc control & computer engneerng IJ-STA, Volume, N, Jul 8, pp On the Use of Neural Network as a Unversal Appromator Amel SIFAOUI, Afef ABDELKRIM, Mohamed BENREJEB UR LARA Automatque, Ecole Natonale d Ingéneurs de Tuns, BP 37 Tuns Belvédère AmelSfaou@entrnutn, afefabdelkrm@estrnutn, mohamedbenrejeb@entrnutn Abstract Neural network process modellng needs the use of epermental desgn and studes A new neural network constructve algorthm s proposed Moreover, the paper deals wth the nfluence of the parameters of radal bass functon neural networks and multlaer perceptrons network n process modellng Partcularl, t s shown that the neural modellng, dependng on learnng approach, cannot be alwas valdated for large classes of dnamc comple processes n comparson wth Kolgomorov theorem kewords Functon appromaton, Radal Bass Functon (RBF), MultLaer Perceptron (MLP), Chaotc behavour Introducton The general scheme of appromaton supposes the estence of a relatonshp between several nput varables and output dependent varables For unknown relatonshp, an appromator s bult between them usng nput-output couples from the learnng data One of the most mportant aspects of neural network concepton s the selecton of the archtecture The objectve s to fnd the smallest archtecture that accuratel fts the true functon descrbed b the tranng data Research n archtecture selecton has resulted n varous dfferent approaches to solve ths problem such as network constructon and prunng Network constructon algorthms start tranng wth a small network and ncrementall add hdden unts durng tranng when the network s trapped n a local mnmum [-3] Neural network prunng algorthms start wth an overszed network and remove unnecessar network parameters, ether durng tranng or after convergence to a local mnmum Multlaer Perceptrons networks (MLP networks) and Radal Bass Functon networks (RBF networks) are two of the most commonl tpes of feedforward Ths paper was recommended for publcaton n revsed form b the edtor Staff Edton: CPU of Tuns, Tunsa, ISSN:

2 On the Use of Neural Network as a Unversal Appromator A Sfaou et al 387 neural networks used as neural network appromators Multlaer perceptrons networks have a nonparametrc archtecture, wth an nput laer, one or more hdden laers, and an output laer The nput sgnal propagates through the network n a forward drecton, laer b laer RBF networks are non-lnear parametrc appromaton models based on combnatons of gaussan functons The MLP and RBF neural networks have been wdel used for functon appromaton, pattern classfcaton and recognton because to ther structural smplct and fast learnng abltes [4-8] The two tpes of network dffer The frst dfference s structure of archtecture: RBF network has a smple archtecture wth a nonlnear hdden laer and a lnear output one; MLP network ma have one or more hdden laers The second dfference s the mode of learnng: for MLP network snaptc weghts are adapted for each pattern but for RBF network the are traned smultaneousl In ths paper, frst we propose a constructve algorthm based on a method known as epermental desgn; secondl, we use Kolgomorov theorem, to prove that a neural network wth one hdden laer such as RBF neural network consttutes a good appromator of a multvarable functon A dnamc sstem wth a statc non lneart s consdered n order to show the applcablt condtons of ths theorem ANN desgn approach Basc dea A ver mportant aspect of neural network model selecton s to fnd the optmal archtecture that accuratel fts the true functon descrbed b the tranng data The man problem n the desgn s that a lot of parameters need to be determned: number of laers, number of neurons n a laer, tpe of actvaton functons, number of patterns for the tranng process, etc Therefore, t s not eas to fnd the optmal values of all these parameters, leadng to a neural network satsfng the problem to whch we are seekng a soluton A ver large archtecture ma accuratel ft the tranng data, but ma have bad generalzaton due to over fttng of the tranng data On the other hand, a too small archtecture wll save the computatonal costs but ma not have enough processng elements to accuratel appromate the true functon Thus archtecture selecton algorthms have to balance network complet wth the best ft of the functon beng appromated One possble soluton to ths problem s the proposed teratve desgn approach to be descrbed n the followng ANN constructve algorthm Man methods n neural network desgn, tr to defne the optmal archtecture In ths wa we propose a neural network constructve algorthm Startng wth one hdden laer network contanng onl one neuron, ths ntal neural network s

3 388 IJ-STA, Volume, N, Jul 8 traned usng one couple of pattern/target The desgn process conssts of addng successvel neurons n the hdden laer, and then ncreasng the number of pattern/target couples, and fnall the number of laers The algorthm permts the eploraton of one part of the possble soluton The possble varatons on the archtecture of the network obtaned can be done onl on the number of neurons of the last laer, e the number of neurons s equal to a predefned mamum for all the hdden laers ecept the last one, where ths number can be lower than ths mamum The behavour epected from the neural network s to represent as close as possble a gven dnamc sstem In order to go further n the neural network desgn process, other loops can be added n the proposed algorthm, allowng eplorng the effect of ntal weghts and the nature of actvaton functons onto the network under constructon 3 Nonlnear functon appromaton In conventonal non artfcal ntellgence-based sstems, the functon appromaton s usuall performed usng a mathematcal model of the consdered sstem However, sometmes t s not possble to have an accurate mathematcal model [9], [], or there ma not est an conventonal tpe of model at all In such cases, artfcal-ntellgence-based functon appromators can be used [5], [6], [], [] For nonlnear multvarable functons, Kolgomorov theor [] defnes a class of multlaer or RBF neural appromators Kolgomorov s theorem A contnuous nonlnear real functon = f ( ) wth n varables, = (,,, n ), can be appromated b the sums and superpostons of ( n +) contnuous functons wth sngle varables z : wth: n ( ) + = g ( z ) () = n z = h ( ) () j= j and h j are nonlnear monovarable functons Kolgomorov s theorem states that the nonlnear functon appromator can be mathematcall descrbed b:

4 On the Use of Neural Network as a Unversal Appromator A Sfaou et al 389 (,,, n+ ) = g ( h ( )) (3) n j = j= It s also possble to modf Kolgomorov s theorem as follows The h j are replaced b λ hj, where λ are constants and h j are strctl monotonc functons The nonlnear functon appromator can thus be defned as: (,,, n+ n ) = g ( λ h ( )) (4) n j j = j= n The theorem does not ndcate the forms of ( ) j j n g λ h j j= h and ( ) Furthermore there s no known constructve technque to obtan them Kolgomorov s theorem has been emploed n [], [] [5], to demonstrate the nonlnear functon apromaton capabltes of artfcal neural networks Let g be non-constant, bounded and monotonc ncreasng contnuous functon, defned b: ( ) g S Gven f (,,, n ) appromaton of, f ( ) ( =,,, M = n+, j =,,, n) = + ep ( S ) = and b usng equaton (4), the goal s to obtan an h = S = w b, where w j and b are real constants j j j It follows that, [3] : M = f (,,, ) = λ g ( w ) (6) n j j = j= If for smplct b = and g = g = = g, then M M M f(,,, ) λ g( w ) = = (7) n j j = j= The last equaton can represent the nonlnear functon apromator f ( ) b the neural network shown n fgure It should be noted that there are n nput nodes to the network and these can be consdered as nodes of the so-called nput laer Ths laer s followed b a socalled hdden laer, whch contans M hdden nodes All nput nodes are connected to all hdden nodes and an th th hdden node s connected to the,,, n, nput nodes, (5)

5 39 IJ-STA, Volume, N, Jul 8 va the w, w,, wn wegths, whch are constant All the hdden nodes are then connected to a so-called output node va the weghts λ j, j =,,, M The output, on the output node, s the lnear combnaton of the outputs of the hdden nodes The wegths can be obtaned b usng nput/output tranng data and applng a tranng method such as the backpropagaton tranng algorthm [8] n w w n w g λ λ λ M Fg Neural network structure 4 RBF and MLP neural networks lmtatons In ths secton, dfferent RBF and MLP neural networks parameters are vared to appromate a nonlnear functon and the result of each test s dscussed and nterpreted 4 Sstem descrpton In order to llustrate the nfluence of the neural network parameters, a nonlnear sstem, gven b the followng dfferental equatons s smulated wth a nonlneart f() represented b a neural network d = sn dt d 6 = f ( ) dt f ( ) = + 5 ( πt ) 8 f ( ) 3, ( ) = 4, ( ) = When the neural network parameters var, several sstem responses can be obtaned The sstem s response wth the consdered real nonlneart s presented n fgure B varng the parameters, dfferent results can be obtaned when usng a neural network structure In fact, the depend on the number of the laers, the hdden number of nodes, the actvaton functons, the ntal weghts, the bas, the number of (8)

6 On the Use of Neural Network as a Unversal Appromator A Sfaou et al 39 patterns, the learnng algorthm, the centers c and the wdths σ of the Gaussan functons 4 RBF neural network nfluence on the sstem s response The frst part of ths secton descrbes the RBF neural network used The second part deals wth the nfluence of the number of nodes and patterns, and the ntal weghts on a unque response of the studed dnamc sstem The result of each test s nterpreted The RBF neural network archtecture conssts of one nput laer wth one nput, one hdden laer wth N nodes and one output laer wth one node The nput neuron s full connected to the N hdden laer neurons ecept the bas neuron Agan, each of the hdden laer neurons and the bas neuron also full connected to the output neurons The output of a hdden laer neuron s usuall generated b a gaussan functon φ as follows: c ep φ ( ) = σ (9) where s an nput vector, c and σ are the center and the wdth of the receptve feld of the th neuron of the hdden laer respectvel and s the eucldean dstance between the nput vector and the center vector tme(s) Fg Smulaton of the dfferental sstem (8) usng the real non lneart The output laer neuron computes a lnear weghed sum of the outputs of the hdden laer neurons as follows:

7 39 IJ-STA, Volume, N, Jul 8 = N = wφ ( ) () where w s the weght between the th hdden laer neuron and the output laer neuron In ths paper, the hbrd learnng method s used for tranng the RBF neural network The learnng stage s dvded nto two successve steps In the frst step, the centers c of the hdden laer neurons are selected b usng k- means clusterng algorthm [4-6], then the wdth σ [7] of the radal bass functons are determned b P-nearest neghbour heurstcs: p k = ( c ) ck σ = () p where c k s the P-nearest neghbour of σ and p s determned heurstcall Fnall, the weghts w between the hdden and output laers are estmated b usng the Least Means Square (LMS) algorthm The followng equaton s used to update the weght: t t t t w + = w +ηe φ () wth : t w : weght vector at teraton t, : hdden laer output vector, t φ η : tranng rate, d : desred output, : network output, t e : error (3) The purpose of ths secton s to analse the nfluence of the RBF neural network s parameters on the unqueness of the second order dnamc sstem s response, descrbed above The nonlneart f () s appromated b an RBF neural network Frst, the number of hdden nodes s changed for the same ntal weghts For hdden nodes, fgure 3a, 5 hdden nodes, fgure 3b, the neural network response show the estence of a lmt ccle n contrast wth the real response sstem, fgure, whch show the estence of a jump to another lmt ccle The second vared parameter s the ntal weghts for hdden nodes, fgure 4a and fgure 4b Fnall, the number of patterns s changed In fact, for hdden d

8 On the Use of Neural Network as a Unversal Appromator A Sfaou et al 393 nodes wth the same ntal weghts the number of patterns s ncreased from 4 samples, fgure 3b, to 67 samples, fgure 5a, then to samples, fgure 5b The dfferent tests show that the neural network has lmts n appromatng the nonlneart, especall n sstems presentng chaotc phenomena [8-] Indeed, n addton to the fact that the neural network can t alwas show the estence of two lmt ccles, the obtaned shapes have great dfferences wth the real sstem response gven n fgure (a) N = neurons (b) N =5 neurons Fg3 RBF neural network: Hdden neurons numbers nfluence for the same ntal weghts 43 MLP neural network nfluence on the sstem response In ths secton, the nfluence of the actvaton functons of hdden laers and the number of nodes s dscussed The result of each test s nterpreted In order to stud the nfluence of these parameters, an MLP network made of two hdden laers wth the same sgmoïd tangent (tansg) actvaton functons, fgure 6, then wth dfferent actvaton functons, fgure 7, and one output laer wth a lnear (pureln) actvaton functon, s chosen and traned b the error backpropagaton

9 394 IJ-STA, Volume, N, Jul 8 algorthm [8] Usuall, ntal weghts are randoml selected, but n these tests, the weghts, the bas and the patterns reman the same from one test to another The number of neurons of the second hdden laer N has been fed Then man tests have been carred out where the number of the neurons of the frst hdden laer N s ncreased It s noted that the actvaton functons are the same (tansg) for the two laers The dfferent llustratons of fgure 6 show that the sstem gves dfferent responses The second set of tests seral conssts of varng the actvaton functons of the hdden laers The actvaton functons are (tansg) and (logsg) respectvel for the frst and second hdden laers, fgure 7a, then the are nterchanged, fgure 7c The results consequentl acheved are dfferent In fact, a permanent flow s obtaned when drfts towards 8 n the frst case, and when drfts towards -8 n the second case In the case where the two hdden laers have the same actvaton functon, fgure 7b, the sstem reacton s dfferent from the two prevous ones Consequentl, the llustrate the ncapact of the artfcal neural network desgn approach to represent chaotc behavour (a) Frst run tme(s) tme(s) (b) Second run Fg4 RBF neural network: Intal weghts nfluence for the same hdden number of neurons, case of neurons

10 On the Use of Neural Network as a Unversal Appromator A Sfaou et al tme(s) (a) Case of 67 patterns tme(s) (b) Case of patterns Fg5 RBF neural network: Number of patterns nfluence, for hdden neurons

11 396 IJ-STA, Volume, N, Jul tme(s) (a) N = / tansg, N = 5 / tansg Goal = tme(s) (b) N = 5 / tansg, N = 5 / tansg Goal = (c) N = / tansg, N = / tansg Goal = Fg6 Hdden neurons numbers N and N nfluence for the same ntal weghts

12 On the Use of Neural Network as a Unversal Appromator A Sfaou et al tme(s) (a) tansg and logsg neurons of frst and second hdden laer respectvel tme(s) (b) tansg and tansg neurons of frst and second hdden laer respectvel tme(s) (c) logsg and tansg neurons of frst and second hdden laer respectvel Fg7 Actvaton functons nfluence N = 5, N =, goal = for the same ntal weghts

13 398 IJ-STA, Volume, N, Jul 8 5 Concluson The selecton of neural networks parameters s of a great mportance n modellng ether the whole or a part of a process Man algorthms have been proposed n the lterature, n order to select the best neural archtecture combnng the network complet and ts capact to appromate a gven functon The frst part of ths paper proposes a network constructve algorthm The second part deals wth neural appromaton of nonlnear functon presentng chaotc behavour Two tpes of neural network are used: MLP and RBF The obtaned results show that the appromators of nonlnear functons or part of them s not a proper appromaton Besdes, the do not lead to a unque behavour of the sstem as t must have been epected b Kolgomorov theorem References Mood, J, Antsakls, PJ: The dependence dentfcaton neural network constructon algorthm IEEE Trans Neural Networks, Vol7 (996) 3 5 Fahlman, SE, Lebere, C: The cascade-correlaton learnng archtecture School Comput, Sc, Carnege Mellon Unv, Pttsburgh, PA, Tech Rep CMU-CS,(99) 9 3 Zhang, J, Morrs, A: A sequental learnng approach for sngle hdden laer neural networks Neural Networks, Vol (997) Gor, M, F S: A Multlaer Perceptron adequate for pattern recogton and verfcaton IEE Transactons on Pattern Analss and Machne Intellgence, Vol (998) 3 5 Hornk, K, Stnchcombe, M, Whte, H: Multlaer feedforward networks are unversal appromators Neural Networks, Vol (989) Funahash, K: On the appromate realzaton of contnuous mappngs b neural networks Neural Networks, Vol (989) Borne, P, Benrejeb, M, Haggege, J: Les réseau de neurones Présentaton et applcatons Ed Technp, Pars (7) 8 Borne, P: Les réseau de neurones Apprentssage REE, No 9, (6) Watson, GA: Appromaton theor and numercal methods Wle New York (989) Kolgomorov, A N: On the representaton of contnuous functons of several varables b the superposton of contnuous functons of one varable and addton Doklad Akadema Nauk SSR, 4 (957) Gros, F, Poggo T: Representaton propertes of networks Kolgomorov s theorem s rrelevant Neural Computaton, (989) Kurkova, V: Kolgomorov s theorem s relevant Neural Computaton 3 (99) 67 3 Vas, P: Artfcal-Intellgence-Based Electrcal Machnes and Drves Applcaton of Fuzz, Neural Fuzz-Neural, and Genetc-Algorthm-Based Technques - Oford Unverst Press, New York (999) 4 Hwang, Y S, Bang, S Y: An effcent method to construct a radal bass functon neural network classfer Neural Networks,Vol (997) Orr, M J: Optmsng the Wdths of Radal Bass Functons n Proc of 5 th Brazlan Smposum on Neural Networks Belo Horzonte (998)

14 On the Use of Neural Network as a Unversal Appromator A Sfaou et al Leonard, J A, Kramer M A: Radal bass functon networks for classfng process faults IEEE Control Sstems (99) Benoudjt, N, Archambeau, C, Lendasse, A, Lee, J, Verlesen, M: Wdth optmzaton of the Gaussan kernels n Radal Bass Functon Networks In Proc of th European Smposum on Artfcal Neural Networks, d-sde, Brussels () 8 Zarouan, M, Deulot, JY, Benrejeb, M: Sur l uncté de la réponse d un réseau d énerge électrque en régme de défauts Revue des Scences et Technologes de l Automatque, Revue e-sta, Vol 3 (6) 9 Robert, B, IU, HHC, Fek, M: Adaptatve tme-delaed feedback for chaos control n a PWM sngle phase nverter Journal of Crcuts, Sstems, and Computers, Vol 3, No 3 (6) -6 Fek, M, Fourat, A: Sldng mode-based snchronzaton and control of chaotc sstems wth parametrc uncertantes 4 th Internatonal Mult-Conference on Sstems, Sgnals and Devces, (7)