A Comprehensive Evaluation of Weight Growth and Weight Elimination Methods Using the Tangent Plane Algorithm

Transcription

1 (IJACSA) International Journal o Advanced Coputer Science and Applications, A Coprehensive Evaluation o Weight Groth and Weight Eliination Methods Using the Tangent Plane Algorith P May K College, Broo Street, Tonbridge, Kent, UK E Zhou Engineering, Sports and Sciences Acadeic Group, University o Bolton, UK C. W. Lee Engineering, Sports and Sciences Acadeic Group University o Bolton, UK Abstract The tangent plane algorith is a ast sequential learning ethod or ultilayered eedorard neural netors that accepts alost zero initial conditions or the connection eights ith the expectation that only the iniu nuber o eights ill be activated. Hoever, the inclusion o a tendency to ove aay ro the origin in eight space can lead to large eights that are harul to generalization. This paper evaluates to techniques used to liit the size o the eights, eight groing and eight eliination, in the tangent plane algorith. Coparative tests ere carried out using the Extree Learning Machine hich is a ast global iniiser giving good generalization. Experiental results sho that the generalization perorance o the tangent plane algorith ith eight eliination is at least as good as the ELM algorith aing it a suitable alternative or probles that involve tie varying data such as EEG and ECG signals. Keyords neural netors; bacpropagation; generalization; tangent plane; eight eliination; extree learning achine I. INTRODUCTION In Lee [] an algorith as described or supervised training in ultilayered eedorard neural netors giving aster convergence and iproved generalization relative to the gradient descent bacpropagation algorith. This tangent plane algorith starts the training ith the connection eights set to values close to zero in the expectation that the iniu eights necessary ill be activated. The results based on to real orld datasets indicated that the tangent plane algorith gives iproved generalization over a range o netor sizes and that it is robust ith respect to the choice o its internal paraeters. Despite the success o the tangent plane algorith there is, hoever, strong evidence to suggest that groing the eights to assue large values can actually hurt generalization in dierent ays. Excessively large eights eeding into output units can cause ild outputs ar beyond the range o the data i an output activation unction is not included. To put it another ay, large eights can cause excessively large variances in the output. According to Bartlett [], the size o the eights is ore iportant than the nuber o eights in deterining good generalization. This poses the olloing question: can e odiy this algorith so that it discourages the oration o eights ith large values? Further, can the algorith encourage eights ith sall values to decay rapidly to zero thus producing a netor having the optiu size or good generalization? Weight decay is a subset o regularization ethods. The principal idea o eight decay is to penalize connection eights ith sall values so that the netor reoves the superluous eights itsel. The siplest ethod is to subtract a sall proportion o a eight ater it has been updated [3]. This is equivalent to adding a penalty ter to the obective unction and peroring gradient descent on the resulting total error. Unortunately this ethod penalizes ore o the s than necessary hilst eeping the relative iportance o the eights unchanged. This can be cured by using a dierent penalty ter, / ( + ), so that the sall s decay aster than the larger ones [4]. Willias [5] proposed yet another type o penalty unction hich is proportional to the logarith o the l nor o the eights,. It as shon in [5] that using this penalty ter is ore appropriate or internal eights than eight decay. Hoyer [6] proposed a sparseness easure based on the l nor and l nor o eights. Experients ith Hoyer s ethod indicate that it perors ell in coparison ith eight decay and eight eliination. A urther reineent involves using a ixed nor penalty ter [7]. In this procedure the l nor o the eight vector is iniised subect to the constraint that the l nor equals unity. II. OBJECTIVES The principal obective o this paper is to describe an alternative strategy or iproving generalization in neural netors trained using the tangent plane algorith. In the nely developed algorith, the training is started ro arbitrary initial conditions and the inactive eights in the netor encouraged decaying to zero by using the eight eliination procedure. Unlie other ipleentations o eight eliination procedures [4, 6, 8-9], the ethod used here is built into the geoetry used in the derivation o the algorith. A secondary obective is to copare the nely developed algorith ith the extree learning achine [], hich obtains the least squares solution ith the iniu training error and iniu nor o eights..iacsa.thesai.org 49 P a g e

2 (IJACSA) International Journal o Advanced Coputer Science and Applications, III. DERIVATION OF THE ALGORITHM In Lee [] an algorith is described that accepts alost zero staring conditions or the connection eights, and hich oves aay ro the origin in a direction indicated by the training data ith the expectation that only the iniu eights ould be activated. This tangent plane algorith uses the target values o the training data to deine a (n ) surace in eight space R n. The eights are adusted by oving ro the current position to a point near to the oot o the perpendicular to the tangent plane to this surace, but displaced soehat in the direction aay ro the origin, on the expectation that the saller the distance oved ro the oot o the perpendicular the less disturbance there ill be to the previous learning. To enhanceents are ade to the tangent plane algorith to obtain the iproved tangent plane algorith reerred to as itpa. Firstly, a directional oveent vector is introduced into the training process to push the oveent in eight space toards the origin. This oveent vector siulates eight decay hich is non to have a beneicial eect on generalization in bacpropagation learning. Secondly, the directional vector is urther odiied to give a heavier eighting to eights ith sall eight values to avoid penalizing ore o the eights than necessary; one large eight costs uch ore than any saller ones. A high degree polynoial ter is used to select the proportion o eights or pruning. This ter can be adusted so that a eight decay procedure is ipleented or reined in a ay that speciic eights are reoved by causing the to decay ore rapidly to zero. d c nˆ b Fig. Moveent ro the present position a to the point d inclined at an angle β to the perpendicular ro a to the tangent plane to the constraint surace ф = - (y ) at point b in the eight space R n. The vector represents the orthogonal proection o the eight eliination vector orthogonally onto the noral n to the constraint surace at point b The ethod assues a eed-orard neural netor o units { u }, here the connection beteen u i and u is ediated by. ф and θ denote the input and output o u, so that θ = (ф ), and ф = i θ i. The single output unit u is trained to iic the target value y. Let n denote the nuber o eights y a a i o in the netor. For a given set o inputs e can consider ф to be a unction o the eights ф : R R n. The tangent plane algorith adusts the eights by oving along the line at an angle β to the perpendicular ro the current position a to the tangent plane to the surace ф = - (y ), on the side o the perpendicular aay ro the origin (see Fig. ). Let a = i be the current values o the eights, here i is a unit vector in the direction o the axis. Use the equation - (y ) = + i i θ i to ind a value,, or the bias eight ro the values o the other eights, so that the surace ф = - (y ) contains the point b = i +,i, i. No, i e use the equation - (y ) = + i i θ i and - (θ ) = + i i θ i, and note that b diers ro a only in the value o, e get b a ( ( " ( y ) i ) ( )) i Let nˆ be the unit noral to the surace at b, so n ˆ. The length o the perpendicular ro a to the tangent plane at b is ( b a ). nˆ. I c is the oot o the perpendicular ro a to the tangent plane at b, And c a ( ( y ) ( )) ( i ( y ( y ) ) ( ). nˆ ) nˆ () () c a (3) ( ) The vector that is directed toards the origin and biased along the axes o the eights that have sall eight values relative to soe sall positive constant a is = -,i ( / a ) i / ( + / a). The proection o onto the tangent plane is given by Where (. nˆ ) nˆ (4) l l, l ( a i (5), i, i ( a ) Thus, i d is the point o intersection ith the tangent plane o a line ro a inclined at angle β to the perpendicular, then d a c a ) ( d c) ( c a ) tan ( c a ) (6).iacsa.thesai.org 5 P a g e

3 (IJACSA) International Journal o Advanced Coputer Science and Applications, Let δ = (y ) (θ ) be the error in the input to inal unit. Hence using equations (), (3) and (4) in (5) yields here d a l l Thus, to adust a given eight tan l l l tan l (7) (8) l i (9) l l,, The ter is the partial derivative o the net input to the output unit. The treatent o this ter ollos ro Lee [] and, i ( ) M i, i Where M is the set o units to hich u passes its output. The ne itpa algorith requires to paraeters that need to be set anually. First paraeter is the angle paraeter tanβ, hich gives the angle beteen the oveent vector and the perpendicular ro the current position to the tangent plane. Its value is usually chosen to be sall, typically.5, so that oveent is to a point nearby the oot o the perpendicular. Second paraeter is the eight sensitivity paraeter a, hich gives the value an individual eight receives a large push toards the origin. a is preerred to be sall, typically.5, so that eights ith sall values are selected or reoval ro the netor. This ill produce the required separation o active and inactive eights in the netor. An individual ter = - ( / a ) / ( + ( / a ) ) in the directional vector varies according to ( / a ) in an antisyetric ashion. This perits the necessary sign changes in so that the oveent along a eight diension is alays directed toards the origin. When < a the directional ter or that eight is approxiately linear. On the other hand, hen > a the directional ter approaches to zero. Thus a eight ill receive a large push hen equals a. A potential diiculty arises hen both and a are less than.5. I = a, the resulting push ay be large enough to overshoot the origin. This situation can be avoided through the appropriate choice o tanβ. The approach o the ne itpa algorith is reiniscent o the Neton-Raphson ethod o irst degree [] used to ind the zero points o unctions that depend on one value. An iportant dierence is that it provides a hole R n plane o suitable points to ove toards. Any ethod that does a zero-point search o a unction cannot get trapped in a local iniu unless it hits one by accident. The ne itpa algorith uses the vector ф to do a linear extrapolation o the surace ф = (y ) in order to gain a ne eight vector that is hoped to be on, or at least close, to this surace. A siple cost saving can be ade by replacing the ter = (. nˆ ) nˆ in the algorith ith. is greater than or equal to (. nˆ ) nˆ ith equality holding hen is perpendicular to nˆ. Its use ill result in a reduction in the size o, but this ter is scaled by tanβ anyay. This reduction is greatest hen is perpendicular to the tangent plane. According to equation (9) involves adding n products o the ters l and ф / l, and then using this result to scale n partial derivatives, ф /. Thus the total coputational saving is n operations per eight update. The algorith can be urther iproved by using a high degree polynoial in the denoinator o each ter in the directional vector, so that = -( / a )/( + ( / a ) n ) here n is a positive constant. The curves o an individual directional ter or three values o paraeter n are shon in Fig.. Exaination o the curves or > a yield the olloing observation. When n is large (typically > 6), the directional ter or that eight decays rapidly to zero. Thus the proper choice o the paraeter n ill perit soe eights in the netor to assue values that are larger than ith n = a =., n = a =., n = 4 a =., n = Fig. Variation o an individual directional ter or dierent values o the paraeter n.iacsa.thesai.org 5 P a g e

4 (IJACSA) International Journal o Advanced Coputer Science and Applications, IV. ESTIMATING THE WEIGHT SENSITIVITY The rationale o pruning is to reduce the nuber o ree paraeters in the netor by reoving dispensable ones. I applied properly, this approach oten reduces overitting and iproves generalization. At the sae tie it produces a saller netor. The approach adopted in this paper is to autoatically prune superluous eights by using the ethod o eight eliination [4]. But ho do e no hether the ethod o eight eliination actually produces the required separation o active and inactive eights? One approach ight be to easure the signiicance or iportance o each eight, as the agnitude o the eights is not the best easure o their contribution to the training process []. There are several ethods suggested or calculating the iportance o connection eights. Karnin [] easures the sensitivity s o each eight by onitoring the su o all the changes to the eights during training. Thus the saliency o a eight is given as s = t (t) / (t) / ( ), here t is the nuber o epochs trained, and are the inal and initial values o eight. LeCun et al [3] easure the saliency o a eight by estiating the second derivative o the error. They also reduce the netor coplexity by constraining certain eights to be equal. Lo saliency eans lo iportance o the eights. A ore sophisticated approach avoids the drabacs o approxiating the second derivatives by coputing the exactly [4]. The last to ethods have the disadvantage o requiring training don to the error iniu. The autoprune ethod [] avoids this proble. It uses a statistic t to allocate an iportance coeicient to each eight based upon the assuption that a eight becoes zero during the training process t ( ) log t t t t ( ( )) () In the above orula, sus are over all training exaples t o the training set, and the overline eans arithetic ean over all exaples. A large value o t indicates high iportance o eight. V. SIMULATIONS AND RESULTS The convergence behaviour o the ne itpa algorith as evaluated and copared ith the gradient descent bacpropagation algorith. The dataset used as the to spiral proble [5 6]. Lie ost published or classiying the to spiral proble [7], a netor ith three hidden layers as used. trials ere perored ith the classiication error on the training and test sets, ean nuber o epochs to converge, and nuber o successul trials recorded. Netor training as terinated hen all the training patterns ere learned correctly or 5, epochs or presentations o the entire dataset. Next, the ability o the ne itpa and original tangent plane algoriths to generalise ro a given set o training data as evaluated and copared ith the Extree Learning Machine [8]. The Extree Learning Machine (ELM) is a ast learning algorith that obtains the least squares solution ith the iniu training error and iniu nor o the eights. The benchar datasets used ere the Henon ap [9] and the non-linear dynaic plant []. A standard eedorard neural netor ith to hidden layers as utilised ith the nuber o hidden units deterined by grid search. For each test, trials ere perored ith the noralised ean square error on the training and test sets recorded together ith the nuber o successul trials. Netor training as terinated ater 5, epochs or presentations o the entire training set. Finally, the evolution and developent o the eights in the ne itpa algorith as evaluated and copared ith the original tangent plane algorith. The benchar datasets used ere the Henon ap [9] and the non-linear dynaic plant []. The ethod used to estiate the sensitivity o the eights as autoprune []. Histogras o eight sensitivities ere plotted ater, 3 and 5 epochs. A. Netor initialization The algoriths used in the study require anually set paraeters. Preliinary tests shoed that the best results ere obtained ith the paraeters set as ollos. First test is the ne itpa algorith. For the to spiral proble, tanβ =.. The eight sensitivity paraeter a and n ere varied according to a grid search. The input eights ere set to rando values in the range [-, ]. For the Henon ap and non-linear dynaic plant, tanβ =., a =.5, and n = 4.. The input eights ere set to rando values in the range [-.5,.5]. Next test is the original tangent plane algorith. The angle paraeter tanβ =.. The input eights ere set to rando values in the range [-.,.]. Finally test is the standard bac-propagation algorith. For the to spiral proble, the learning rate η =., and oentu coeicient α =.3. The input eights ere set to rando values in the range [-, ]. B. Siulation probles The to spiral proble consists o to interlocing spirals, each ade up o 97 data points. The netor ust learn to discriinate the to spirals. Traditionally this is non to be a very diicult proble or the bac-propagation algorith to solve. There are to inputs and one output. The inputs are the x and y co-ordinates, and the output notiies hich spiral the point belongs to. For the points in the irst spiral the output is set to +, and or points on the other spiral the output is set to -. The nuber o training saples is 94. A test set o 9 saples as generated by rotating the to spirals by a sall angle. The Henon ap proble is a chaotic tie-series prediction proble. The tie series is coputed by x ( t) ( t) ( t) c [ x ] b x () Where x (t) is the value at taen tie t, and the paraeters b =.3, and c =.4. Initial values or the tie series are x () = x () = This point is called the ixed point o the tie series. In neural netor siulations, our successive values o the tie series are used in predicting the next value..iacsa.thesai.org 5 P a g e

5 NMSE x (IJACSA) International Journal o Advanced Coputer Science and Applications, Thus, the nuber o inputs is our and the nuber o output is one. Data values ere taen ro the range [3,3] as given in Lahnaärvi et al [9]. The nuber o training saples is, and testing saples is. The non-linear dynaic plant proble is a high order nonlinear syste introduced in Narendra and Parthasarathy []. It is odelled by the olloing discrete tie equation y ( t) ( t) ( t) ( t3) ( t) ( t3) t y y y u [ y ] u () ( t) ( t3) [ y ] [ y ] Where y (t) is the odel output at tie t. Lie Narendra and Parthasarathy [], training data as generated using a rando input signal uniorly distributed over the interval [-, ]. Five hundred data points ere generated, the irst three hundred used as training data hilst the reaining used as test data. C. Error etrics used to deterine convergence The error etrics used in the siulations ere CERR (Classiication ERRor) or classiication probles and NMSE (Noralized Mean Square Error) or regression probles. The CERR as calculated using the 4--4 criteria e.g. the actual output does not dier ro the target output by ore than.4 [5 6]. The NMSE as calculated by dividing the MSE by the variance o the target output. D. Discussion o results To spiral proble. The irst test is a diicult nonlinearly separable proble here a set o co-ordinates (x,y) is classiied as belonging to one o to interoven spirals. A netor topology as chosen as given in [6]. For the ne itpa algorith, trails gave no ailures and a ean nuber o steps to converge o 8 ith standard deviation 9. The classiication error on the test set as.3 - (e.g. % test set learned = 98.7) ith all o the points on the training set correctly classiied. Using the standard bacpropagation algorith, there as one ailure and a ean nuber o steps to converge o 736 ith standard deviation 46. The classiication error on the test set as.5 - (e.g. % test set learned = 98.5). The results copare very avourably ith those given in Linder et al [6] (Aprop: epochs = 67, % test set learned = 96.6; Rprop: epochs = 46, % test set learned = 65.6). Table deonstrates the eects o changing the eight sensitivity paraeters a and n o the ne itpa algorith. A ---- architecture as chosen to deterine the degree to hich the itpa algorith could generalize in a large netor ith any ree paraeters. It as ound that the classiication error iproved slightly hen large values ere chosen or the eight sensitivity paraeter a. Further, increasing the value o the paraeter n caused the classiication error to dip to a clearly deined iniu. When a is large (e.g. typically >.5) the directional vector ill push ore o the netor eights to sall values close to zero thus ipleenting a eight decay procedure. This suggests that eight decay is a ar ore eective strategy or iproving generalization than eight eliination in large neural netors trained using the tangent plane algorith. TABLE I. CONVERGENCE SPEED AND CLASSIFICATION ACCURACY FOR DIFFERENT PARAMETERS IN THE ITPA ALGORITHM Note: the coluns in Table reer to the classiication error on the training set (Cerr) and test set (Cerr*), the ean a n Cerr Cerr* Steps Succ nuber o steps to converge, and nuber o success trials or dierent values o the eight sensitivity paraeter a and n. Fig 3 and 4 sho soe typical test curves or both algoriths on the to spiral probles. Dierent sets o initial eights ere used in each test. The test curves o the ne itpa algorith sho soe variation (Fig 3). In any o the curves generated the test error as ound to diinish sloly at the start o the training run ith interittent rises in the test error airly typical (test ). When the ne algorith as close to a solution, the convergence as usually rapid (test and 3). The test curves o the standard bac-propagation algorith also sho ide variation in the test error. Soe curves exhibited turbulent behaviour siilar to the ne itpa algorith (test 3). Other curves got trapped in local inia o the error landscape resulting in very long runties (test ). Generally speaing the ne itpa algorith is prone to probles ith stability. Introducing a 5% staged reduction in the step size resulted in aster convergence speeds Epochs itpa test itpa test itpa test 3 Fig. 3 Typical generalization behaviour o the ne itpa algorith on the to spiral proble.iacsa.thesai.org 53 P a g e

6 NMSE x (IJACSA) International Journal o Advanced Coputer Science and Applications, 5 GD-BP test GD-BP test GD-BP test Epoch Epoch 3 Epoch Epochs Fig. 4 Typical generalization behaviour o the gradient descent bacpropagation algorith on the to spiral proble Henon ap tie series. The second test is a classical deterinistic one-step-ahead prediction proble. Preliinary tests shoed that the best results or both tangent plane algoriths ere obtained using a architecture. Netor training as terinated ater 5, epochs or presentations o the entire dataset. For the ne itpa algorith, trials gave a noralised ean square error on the training set and test set o.7 and.8 respectively. Using the original tangent plane algorith, trials gave (training set =.5, test set =.9). There as little evidence o overtraining. The perorance o both tangent plane algoriths copare avourably ith the Extree Learning Machine. For ELM a single hidden layer eedorard neural net ith 8 hidden units gave (training set =., test set =.). Fig 5 and 6 sho histogras o the iportance coeicients o the eights or both algoriths on the Henon ap proble. The iportance coeicients ere recorded ro the sae trial at epochs, 3 and 5. The coeicient sizes ere grouped in classes o idth one and histogras plotted to sho the distribution o the t values at three dierent stages o training. The ne itpa algorith gave average coeicient sizes at, 3, 5 epochs o.8,.95, and.95 respectively. The original algorith gave.7,.96, and.5. Notice the right seness o the histogras produced by the ne algorith (see Fig 5). Ater 5 epochs the histogra is doinated by a single high pea and long right tail. This suggests that any o the eights have taen on equally iportant roles in the netor. Thus are liely to be eer outlier eights ith extree values that are non to produce overtraining in neural netors. Fig. 5 Iportance coeicient histogras or the ne itpa algorith (Henon ap proble). Horizontal axis: coeicient size grouped in classes o idth. Vertical axis: absolute requency o eights ith this coeicient size Epoch Epoch 3 Epoch Fig. 6 Iportance coeicient histogras or the original tangent plane algorith (Henon ap proble). Horizontal axis: coeicient size grouped in classes o idth. Vertical axis: absolute requency o eights ith this coeicient size Non-linear dynaic plant. The third test is a high order non-linear discrete tie syste. Preliinary tests shoed that the best results or both tangent plane algoriths ere obtained using a --- architecture. Netor training as terinated ater 5, epochs or presentations o the entire dataset. For the ne itpa algorith, trials gave an average noralised ean square error on the training and test sets o.45 and.86 respectively. Using the original tangent plane algorith, trials gave (training set =.4, test set =.393)..iacsa.thesai.org 54 P a g e

7 (IJACSA) International Journal o Advanced Coputer Science and Applications, Once again the perorance o both tangent plane algoriths copare avourably ith the Extree Learning Machine. For ELM a single hidden layer neural net ith hidden units gave (training set =.7, test set =.568). The results on the training set suggest that the ne itpa algorith is an eective global iniizer capable o reaching the sallest training error. Fig 7 and 8 sho histogras o the iportance coeicients o the eights or both algoriths on the nonlinear dynaic plant proble. The iportance coeicients ere recorded ro the sae trial at epochs, 3 and 5. The coeicient sizes ere grouped in classes o idth one and histogras plotted to sho the distribution o the t values at three dierent stages o training. The ne itpa algorith gave average coeicient sizes at, 3, 5 epochs o.4,.68, and.79 respectively. The original algorith gave.56, 3.33, and 3.8. Notice the let drit o the histogras produced by the ne algorith (see Fig 7). In contrast the histogras produced by the original algorith tend to drit right, as expected (Fig 8). Further, these histogras have a long right tail hich suggests that soe eights are taing on a ar ore active role in the netor. This ight account or the orse generalization o the original algorith on this proble. Fig. 7 Iportance coeicient histogras or the ne itpa algorith (non-linear dynaic plant). Horizontal axis: coeicient size grouped in classes o idth. Vertical axis: absolute requency o eights ith this coeicient size VI Epoch Epoch 3 Epoch COMPARISON OF THE DIFFERENT ALGORITHMS In order to deterine hether the dierence in the results is statistically signiicant, e peror soe hypothesis tests. The test used as a standard t-test ith the saple o test errors ro the itpa algorith copared ith the corresponding saple ro the original tangent plane algorith or each dataset used in the study. A second test as carried out by coparing these test results ith the ELM algorith on the sae set o probles. 6 Fig. 8 Iportance coeicient histogras or the original tangent plane algorith (non-linear dynaic plant). Horizontal axis: coeicient size grouped in classes o idth. Vertical axis: absolute requency o eights ith this coeicient size The ELM algorith requires the nuber o hidden units to be set hich as ound be grid search. For the correct application o the t-test, it as necessary to tae the logarith o the test errors (since the test errors have log-noral distribution) and reove any outliers, olloing the sae procedure in []. The resulting saples ere tested or norality using the Kologorov-Sirnov test. TABLE II. Proble RESULTS OF A T-TEST COMPARING THE MEAN TEST ERRORS OF THE DIFFERENT ALGORITHMS Training saples Test saples Epoch Epoch 3 Epoch 5 Inputs (a) (b) (c) L Spiral Henon L L Plant E 7.3 Note: The entries sho dierences that are statistically signiicant on a % level and dashes ean no signiicance ound. Colun (a): itpa algorith ( L ) vs. original tangent plane algorith ( T ). Colun (b): itpa algorith vs. ELM algorith ( E ). Colun. (c): original tangent plane algorith vs. ELM algorith. The results are tabulated in Table. Dashes ean dierences that are not signiicant at the % level i.e. the probability that the dierences are purely accidental. Other entries indicate the superior algorith (e.g. itpa algorith - L, original tangent plane algorith T, ELM algorith - E), and the value o the t statistic. Colun (a) gives a coparison beteen the ne itpa algorith and the original tangent plane algorith. The results sho to ties L is better (spiral and non-linear dynaic plant) and once T is better (Henon ap)..iacsa.thesai.org 55 P a g e

8 (IJACSA) International Journal o Advanced Coputer Science and Applications, The ne itpa Algorith perored better on the datasets that ere ore diicult to learn and so convergence speeds tended to be sloer. Where convergence occurred quicly the original algorith as the better ethod. Colun (b) and (c) give coparisons beteen the ne itpa and original tangent plane algoriths, and the ELM algorith. The results sho 4 ties no statistical dierence, once L is better and tice E is better. This suggests that the generalization perorance o the tangent plane ethod is at least as good as the ELM algorith, hich is one o the best neural netor classiiers. In situations here tie varying signals are required, such as EEG and ECG signals, the sequential learning ability o the tangent plane algorith ight be the preerred ethod. VII. CONCLUSIONS A ne variant o the tangent plane algorith reerred to as itpa is proposed or eed-orard neural netors. This ne algorith includes to odiications to the existing algorith. Firstly, a directional oveent vector is introduced into the training process to push the oveent in eight space toards the origin. This directional vector is built into the geoetry o the tangent plane algorith and ipleents a eight eliination procedure. Secondly, a high degree polynoial ter is utilised to adust the proportion o eights that receive an inards push. Thus the algorith can be tuned to decay speciic eights to zero (hich can help generalization). Coparative tests ere carried out using the ne itpa and original tangent plane algoriths, the gradient descent bac-propagation algorith and the Extree Learning Machine. The results indicate that the ne itpa algorith retains the ast convergence speed o the original ethod. Hoever, the ne itpa algorith is prone to probles ith stability. Including a 5% reduction in step size oten iproves convergence behaviour ithout any diinution in learning speed. The results also sho that the ne algorith gives iproved generalization relative to the original algorith in soe probles, and has coparable generalization perorance in yet others. Further, the generalization perorance o the tangent plane ethod is at least as good as the Extree Learning Machine, hich is one o the best neural netor classiiers. VIII. FUTURE WORK This paper shos that the nely developed iproved tangent plane algorith (itpa) is at least as good as the extree learning achine, hich is one o the best neural netor classiiers. In situations here tie varying signals are required, such as EEG and ECG signals, the sequential learning ability o the iproved tangent plane algorith ight be the preerred ethod. REFERENCES [] C.W. Lee, Training eedorard neural netors: an algorith giving iproved generalization, Neural Netors, vol pp [] P. Bartlett, For valid generalization, the size o the eights is ore iportant than the size o the netor, Advances in Neural Inoration Processing Systes 9, Cabridge, MA: The MIT Press, 997, pp [3] S. J. Nolan, and G. E. Hinton, Sipliying neural netors by sot eight sharing, Neural Coputation, vol. 4, no. 4, pp , 99 [4] A.S. Weigend, D.E. Ruelhart and B.A. Huberan, Generalization by eight eliination ith application to orecasting, Advances in neural inoration processing (3), 99, pp [5] P.M. Willias, Baysian regularisation and pruning using a Laplacian prior, Technical report, (3), 994 [6] P.O. Hoyer, Non-negative atrix actorisation ith sparseness constraints, Journal o achine learning research, (5): [7] Huien Zeng, Diensionality reduction using a ixed ter penalty reduction, IEEE orshop on achine learning or signal processing, 5 [8] C.M. Ennett and M. Frize, Weight eliination neural netors applied to coronary surgery orality prediction. IEEE Trans In Technol Bioed, 3, 7():86-9. [9] R.M. Zur, Yulei Jiang, L. Pesce, and K. Druer, Noise inection or training neural netors: a coparison ith eight decay and early stopping, Med. Phys. 9, 36(): [] J. Stoer, Einührung in die nuerische Matheati,, Springer, Vol.. S, 976 [] W. Finno, F. Hergert, and H.G. Zierann, Iproving odel selection by non-convergent ethods, Neural Netors, vol.6, 993, [] E.D. Karnin, A siple procedure or pruning bac-propagation trained neural netors, IEEE trans neural netors, vol., no., 99, pp [3] Y.L, LeCun, J.S., Dener,and S.A. Solla, Optial brain daage, Advances in neural inoration processing systes, vol., 99, pp [4] B. Hassibi, and D.G. Stor, Second order derivatives or netor pruning, Advances in neural inoration processing systes, vol.5, 993, pp [5] S. Fahlan, and C. Lebiere, The cascade correlation learning architecture, Advances in neural inoration processing systes, vol.. 99, pp [6] R. Linder, S. Wirtz, and S.J. Poppl, Speeding up bacpropagation learning by the APROP algorith, Proceedings o the second international ICSC syposiu on neural coputation, ICSC Acadeic Press,, pp. -8. [7] K.L. Lang, and M.J. Witbroc, Learning to Tell To Spirals Apart, Proceedings o the 988 Connectionist Models Suer School, Morgan Kauann 988. [8] Guang-Bin Huang, Qin-Yu Zhu, and Chee-Kheong Sle, Extree learning achine: Theory and applications, Neurocoputing, 7, 489-5, 6 [9] J.J.T. Lahnaärvi, M.I. Lehtoangas, and J.P.P. Saarinen, Evaluation o constructive neural netors ith cascade architectures, Neurocoputing, vol. 48,, pp [] K.S Narandra, and K. Parthasarathy, Identiication and control o dynaical systes using neural netors, IEEE transactions on neural netors, (), [] L. Prechelt, Connection pruning ith static and adaptive pruning schedules, Neurocoputing, Volue 6, Issue, 997, pp iacsa.thesai.org 56 P a g e