AN ALGORITHM OF CHAOTIC DYNAMIC ADAPTIVE LOCAL SEARCH METHOD FOR ELMAN NEURAL NETWORK. Received September 2009; revised January 2010

Transcription

1 International Journal of Innovative Computing, Information and Control ICIC International c 2011 ISSN Volume 7, Number 2, February 2011 pp AN ALGORITHM OF CHAOTIC DYNAMIC ADAPTIVE LOCAL SEARCH METHOD FOR ELMAN NEURAL NETWORK Zhiqiang Zhang 1,2, Zheng Tang 2, Shangce Gao 2 and Gang Yang 2 1 Department of Research and Development NST Inc Tokyo, Japan dalaosha@hotmail.com 2 Faculty of Engineering Toyama University Gofuku 3190, Toyama shi , Japan Received September 2009; revised January 2010 Abstract. In this paper, we present an efficient algorithm for the prediction of sunspotrelated time series, namely the Chaotic Dynamic Adaptive Local Search (CDALS) algorithm. This algorithm is based on exploiting partially recurrent Elman Neural Network (ENN) and it can be divided into two main steps: the first one is the basic model of the Adaptive Local Search (ALS) proposed in our previous work. After that, a hybrid local search method is proposed by introducing the chaos signals into ALS. Thus, ALS and chaos are hybridized to form a powerful CDALS algorithm, which reasonably combines the searching ability of ALS and chaotic searching behavior. Simulation results show that the CDALS algorithm can eventually reach the global optimum or its good approximation with high probability, effectively enhance the searching efficiency and quality within reasonable number of iterations. Keywords: Elman neural network (ENN), Sunspot-related time series prediction, Adaptive local search (ALS), Chaotic dynamic 1. Introduction. As we know, Elman Neural Network (ENN) consists of a two-layer back propagation network with an additional feedback connection from the output of the hidden layer to its input. The advantage of this feedback path is that it allows the ENN to recognize and generate temporal patterns and spatial patterns. This means that after training, interrelations between the current input and internal states are processed to produce the output and to represent the relevant past information in the internal states. As a result, the ENN has been widely used in various fields, in which prediction is an important application direction [1-5]. Since the ENN usually uses the Back-Propagation (BP) based algorithms to deal with the various signals, it has been proved that it is difficult to overcome the weakness or inherent characteristics of the BP algorithm which suffers from slowness of convergence speed and easily gets stuck into the local minima. In contrast to the past learning algorithms employed by the ENN, we have found that all these learning algorithms, either the error BP based algorithms or the non error BP based algorithms employed by the ENN such as genetic algorithm [6,7] and simulated annealing algorithm [8] are difficult to achieve the satisfied learning solutions that is far from the optimal solution. So these problems call for the development of alternative training algorithms for ENN. As a novel optimization technique, chaos has gained much attention and some applications during the past decades. For a given energy or cost function, by following chaotic ergodic orbits, a chaotic dynamic system can eventually reach the global optimum or 647

2 648 Z. ZHANG, Z. TANG, S. GAO AND G. YANG Output units y(k) w=1.0 w=1.0 Hidden units x(k) c x(k) Context units Input units u(k) Figure 1. Structure of an Elman neural network model its good approximation with high probability. On the other hand, chaos which exhibits bounded dynamic unstable, pseudo random, non-period behavior enables the search system more capable of hill-climbing and escaping from local optima [9]. To enhance the performance of the Adaptive Local Search method (ALS) proposed by us [10,11], inspired by the basic traditional local search algorithm, a hybrid local search method is proposed by incorporating chaos into ALS. Thus, ALS and chaos are hybridized to form a chaotic dynamic adaptive local search method (CDALS), which reasonably combines the searching ability of ALS and chaotic searching behavior. In this paper, a CDALS algorithm based on the logistic equation is presented to learn ENN. Simulation results and comparisons with the other algorithms show that the CDALS can effectively enhance the searching efficiency and greatly improve the searching quality within reasonable number of iterations. The rest of the paper is organized as following: in Section 2, the structure of the ENN is described, and then the basic ALS algorithm is given. The CDALS algorithm is briefly described in Section 3. In Section 4, we apply the proposed algorithm into the prediction of sunspot-related time series to show its effectiveness. Finally we give the conclusion and expectation. 2. Adaptive Local Search Algorithm. Figure 1 shows the structure of a simple ENN. It is easy to observe that the ENN consists of four layers: input layer, hidden layer, context layer and output layer. The activations are copied from hidden layer to context layer on a one for one basis, generally, with fixed recurrent connection weight (w = 1.0). The forward connection weights are trainable between hidden units and context units as other weights. If self feedback gain a are introduced to the context unit for the ENN, it is called modified ENN [12], and the values of the self-connection weights (a) can be changed according to practical applications. When the value of a is 0, the network converts to the original ENN [13]. From Figure 1 we can see that the output of the jth context unit in the modified ENN is given by: x cj (k + 1) = ax cj (k) + x j (k) (1)

3 AN ALGORITHM OF CDALS METHOD FOR ENN 649 where x cj (k) and x j (k) are the outputs of the jth context unit and jth hidden unit, respectively. The value of the feedback gain a is the same for all self-connections and it is not modified by the training algorithm. Equation (1) can be unfolded through the dynamical recurrent trace by following: x cj (k + 1) = x j (k) + ax j (k 1) + a 2 x j (k 2) + a 3 x j (k 3) +... (2) Equation (2) means that the output of the context unit is an integration of the output of the hidden unit. Usually a is between 0 and 1, a value of a enables the ENN to trace further back into the past. So the ENN can perform prediction tasks by using its good memory for the learning data. According to our previous research work [10], we can get the learning algorithm of ALS as following rules: V k+1 = V k + k e l if MSE + < 0 and MSE + < MSE V k k e l if MSE < 0 and MSE < MSE + V k otherwise Also, in order to accelerate the convergence of the proposed algorithm, we use the parameter k in different intervals of the MSE values. (3) k = βmse 0 < β < 1 (4) where β parameter is analogous to a learning rate, and we use different β values to test the proposed algorithm. Equations (3) and (4) will perform the proposed algorithm iteratively and minimizes the error function along with the set of decent directions directly. Thus, the proposed algorithm could cause the error function M SE to decrease by all parameters change produced in a cycle. In addition, for the ALS algorithm [10], in this paper, we will add some new technical supplement. As we know, the condition that the ENN architecture can employ the BP algorithm is that the weights between hidden layer and context layer are fixed at 1. But now, we delete the mechanism of the fixed weights, and use random weights in ALS algorithm since we did not adopt BP-based algorithm. Thus, the memory capacity of the ENN will be enhanced, and it should perform better than the condition of fixed weights in generalization capacity aspects for the prediction tasks. 3. Chaotic Dynamic Adaptive Local Search (CDALS) Algorithm. Although the ALS learning algorithm described above may lead a convergence to a local minimum for the ENN, however it is not a thoroughly efficient way for the learning to reach the global minimum from the local minimum [10,14]. In order to avoid these disadvantages, we propose a hybrid local search method of solving the local minimum problem by incorporating chaos into ALS and apply it into sunspots related prediction tasks. As a kind of characteristic of nonlinear systems, chaos is a bounded unstable dynamic behavior that exhibits sensitive dependence on initial conditions and infinite unstable periodic motions. Although it appears to be stochastic, it occurs in a deterministic nonlinear system under deterministic conditions. In recently years, growing interests from physics, chemistry,biology and engineering have stimulated the studies of chaos for control [15,16], synchronization [17] and optimization [18-21]. There are several chaotic systems, such as Logistic map [22], mapping drawn from chaotic neuron [23], Tent mapping [24], Lorenz System [25], Chen System [26], and so on. The first three referenced chaotic systems can show good chaotic properties. If a

4 650 Z. ZHANG, Z. TANG, S. GAO AND G. YANG Figure 2. Dynamic of logistic map designed algorithm needs the candidate points to distribute in search space as much as possible, the three chaotic systems can meet this need. It has been demonstrated that these three chaotic systems display better randomness than other systems [27,28]. We adopt the Logistic map as a studying case. Nevertheless, it does not indicate that the adopted Logistic map outperforms the other two systems. Further works can be considered regarding to this issue. The logistic equation is defined by the following equation: cx i (k + 1) = 4cX i (k)(1 cx i (k)) 0 < β < 1, k = 1, 2... (5) where cx i denotes the ith chaotic variable and k represents the iteration number. Obviously, cx i (k) is distributed in the interval (0, 1) under the conditions that the initial cx i (0) (0, 1) and that cx i (0) {0.25, 0.5, 0.75}. Figure 6 shows its chaotic dynamics, where X i (0) = 0.1, k = 100. In this paper, the CDALS algorithm was developed to get a high quality solution for learning ENN networks by introducing the chaotic signal. The learning rule Equation (3) is modified by using Equation (5) as follows: V k+1 = V k + p 1 cv (k + 1) p 2 (6) cv (k + 1) = 4 cv (k) (1 cv (k)) where k denotes the iteration number, and the function x removes the fractional part of x and returns the resulting integer value. Furthermore, if x is negative, x returns the first negative integer less than or equal to x. The used architectures present only one input neuron, three number of hidden units chosen by performing several selection trials and one output unit. After the network design, we initialize the weights and thresholds of the connections in [ 1, 1]. Finally, the entire input sequence is presented into the network which is trained using CDALS algorithm. After training phase, we use the well trained network with steady and fixed parameters to test the prediction generalization capacity. Generally, the search procedures of the CDALS algorithm for learning ENN networks are described as Figure 3. The detailed learning algorithm proposed can be described by the following pseudo-codes: Step 1. Pre-process the training data. Step 2. Assign a three layered ENN as Figure 1.

5 AN ALGORITHM OF CDALS METHOD FOR ENN 651 Figure 3. General process of the proposed CDALS algorithm Step 3. Initialize all the parameters. Initialize the vector V (weights and thresholds) to small random values between 1 and 1 and other parameters (the maximal CPU limit time and error criterion), and set the maximum iteration times (M ax Epoch). Step 4. Present Inputs and Desired Outputs. Present a training input sequence and specify the desired outputs. Step 5. Calculate Actual Outputs. Using the employed functions in the hidden layer and the output layer to calculate the actual outputs. Step 6. Use the CDALS learning rule Equation (6) to adapt the weights and biasing parameters. Select a weight or a threshold randomly, and adjust it using Equation (6), If the new solution is better than V k or the predefined maximum iteration Max k is reached, output the new solution as the result of the CDALS and go to Step 7; otherwise, let k = k + 1 and repeat Step 4. Step 7. Use the traditional ALS learning rule Equations (3) and (4) to update the parameters. Step 8. Repeat by going to Step 4, until the window and biasing parameters stabilize. In science, chaos is used as a synonym for irregular behavior, whose long-term prediction is essentially unpredictable. Chaotic differential equations exhibit not only irregular behavior but they are also unstable with respect to small perturbations of their initial condition. Consequently it is difficult to forecast the future of time series only based on chaotic differential equations, so we design the CDALS algorithm by combining the ALS and chaos with their advantages, respectively. 4. Simulations The design of the sunspot experiment. Solar activity is regularly monitored by many world observatories and research centers that are able to provide the relative number of dark spots observed on the sun day after day. These data are recorded to give the so-called sunspot related time series. In the last years, there are many studies that have been made on the forecast of solar activity [29-32]. The sunspot series was the first time with series ever studied with autoregressive models, and thus has served as a benchmark in many literatures [33,34]. Consistent with these appraisals, we use the sunspot series time prediction to testify the validity of our proposed algorithm.

6 652 Z. ZHANG, Z. TANG, S. GAO AND G. YANG 200 Yearly sunspot number from 1700 to Number of dark spot Year Figure 4. The yearly sunspot number Also, according to the common practice,the accuracy of the prediction is evaluated in terms of the Normalized Mean Squared Error (NMSE), also called by some authors the Average Relative Variance (AVR): NMSE = 1 N (y σ 2 i ỹ i ) 2 (7) N where y i is the actual value of the ith point of the series of length N, ỹ i is the predicted values and σ 2 denotes a variance of true time series in the prediction interval N. In this paper, we use data provided by the Sunspot Index Data Center of the Federation of Astronomical and Geophysical Data Analysis Services [35]. It is common practice to use as the training data from 1700 to 1920 and to evaluate the performance of the model on two sets, from 1921 to 1955 (Test1) and from 1956 to 1979 (Test2). Test2 is non-stationary and is considered to be more difficult. We employ the Yearly Sunspot Number, showed in Figure 4, that contains the yearly number of dark spots from 1700 to Obviously, in order to make correct comparisons we have used the same set of values of other previous works. Also, we will compare our algorithm with numerous results found in literature. The algorithm proposed in this paper to predict sunspot-related time series is principally based on using ENN as a prediction framework, and it is characterized by a fine data preprocessing technique in order to prepare the most suitable training sets. In this paper, for the training data pre-processing, we adopted the pre-processing technique proposed by K. Kutza [36]. The pre-processed training data of sunspot are depicted as Figure Analysis of simulation results. For the time series prediction we adopted is the Yearly Sunspot Number. We use the data from 1700 to 1920 as training data in the training phase, and use two test sets to evaluate the performance of the trained network with steady learning parameters, one test set is from 1921 to 1955 and another test set is from 1980 to By testing several experiments, we have achieved the optimal performance using an ENN with three hidden layer neurons, β as 0.5. The prediction capacity and the comparison results with previous works are summarized in Table 1. From the simulation results of Table 1 we can see that the proposed CDALS algorithm outperformed simple AR (Auto-Regressive) models, fully recurrent neural networks DRNN (Dynamical Recurrent Neural Networks) presented in [37], the Committee Prediction method of Wan [29], the i=1

7 AN ALGORITHM OF CDALS METHOD FOR ENN Number of Sunspot processed by pre processing technique Year Figure 5. Pre-processed training data from 1700 to 1920 Table 1. Comparison of generalization capacity NMSE to the same testing sets for the single step prediction Training set Algorithms NMSE of Test1 NMSE of Test2 NMSE of Test3 AR(12) WNet DRNN N/A Training Set COMM ( ) RNN+BPTT N/A RNB+CBPTT N/A VGBP Propose CDALS recurrent networks trained with the Back Propagation Through Time algorithm (BPTT) and one of its variant, the Constructive BPTT developed in [38]. Only the Violation Guided Back Propagation technique (VGBP) [38], based on a recurrent Finite Impulse Response (FIR) network, is more accurate than our algorithm on the Test1. As we know, the prediction of Test2 and Test3 is more difficult than Test1, so we can conclude that our proposed algorithm has powerful prediction capacity and more accurate rate than those algorithms that often used as comparison object. We also gave the specific simulation curve Figure 6, in order to better understand the prediction capacity of our proposed algorithm. For the multi-steps prediction, there are two strategies often mentioned, one is to directly forecast the desired prediction horizon by training a network, another is to use the values predicted in the previous time steps as inputs for subsequent prediction. By analyzing both methods we have found that the direct approach is more suitable for our ENN. Table 2 is the mean results obtained until a prediction horizon from 2 to 6. From Table 2 we can see that our proposed CDALS algorithm has overwhelming advantage compared with existing other methods. So although the ALS is a traditional local search approach, but by introducing the chaotic dynamic signals, the CDALS exerted its robust and generalization capacity for the sunspot-related time prediction tasks. In order to make a comparison with the most recent results published in literature about a medium-term horizon prediction of the Yearly Sunspot Number, we have computed the

8 654 Z. ZHANG, Z. TANG, S. GAO AND G. YANG Table 2. Comparison of generalization capacity MSE test to the same testing sets for all algorithms Steps ahead Proposed CDALS RNN+BPTT RNN+CBPTT RNN+EBPTT Test2: Number of sunspots (processed) Test1: Test3: Year Figure 6. Single step prediction for sunspots from 1921 to 1990 NMSEs on the cumulated test set involving Test1, Test2 and Test3. In all the multi-step predictions, the trained well ENN present the best performance, as one can see also from Figure 6, which illustrates a graph showing a comparison between the actual values and the prediction values by using the proposed algorithm. Through above analysis of the simulation results from single prediction task to multisteps predictions we can see that it was suitable for the ENN to train and learn the sunspots number prediction by introducing the chaos signals into ALS algorithm, and the proposed CDALS algorithm not only achieved better training results but also obtained overwhelming advantage in prediction aspects. 5. Conclusion. In this paper, we proposed a Chaotic Dynamic Adaptive Local Search algorithm for ENN. The proposed algorithm was shown to be of high prediction capacity for the sunspot-related time series prediction than existing other algorithms. The proposed algorithm has been applied to the benchmark task including single prediction and multi-steps prediction of sun spots. So it could be concluded that the proposed algorithm was an effective algorithm for ENN to learning sequence time tasks. Besides that by using the CDALS algorithm to replace BP-based algorithm, calculations are easier since

9 AN ALGORITHM OF CDALS METHOD FOR ENN 655 no derivatives are required and local search is straightforward. However, one of the drawback of the proposed algorithm was that the consumption of the time take longer than some algorithms, we think it is our future work direction to shorten the learning time by developing other optimization techniques. Acknowledgment. The authors would like to thank many people who helped us in the experiments. The authors also gratefully acknowledge the helpful comments and suggestions of the reviewers, which have improved the presentation. REFERENCES [1] F. Fessant, S. Bengio and D. Collobert, On the prediction of solar activity using different neural network models, Annales Geophysicae, vol.14, pp.20-26, [2] T. Koskela, M. Lehtokangas, J. Saarinen and K. Kaski, Time series prediction with multilayer perceptron, FIR and Elman neural networks, Proc. of the 1996 World Congress on Neural Networks, [3] P. Stagge and B. Sendhoff, An extended Elman net for modeling time series, Int. Conf. Artificial Neural Networks, LNCS, vol.1327, pp , [4] H. He and X. Tian, An improved Elman network and its application in flatness prediction modeling, Proc. of the 2nd International Conference on Innovative Computing, Information and Control, Kumamoto, Japan, pp.552, [5] C. P. Lim and W. Y. Goh, The application of an ensemble of boosted Elman networks to time series prediction: A benchmark study, International Journal of Computational Intelligence, vol.3, no.2, [6] D. T. Pham and D. Karaboga, Training Elman and Jordan networks for system identification using genetic algorithms, Artificial Intelligence in Engineering, vol.13, pp , [7] X. Z. Gao and S. J. Ovaska, Genetic algorithm training of Elman neural network in motor fault detection, Neural Computation and Application, vol.11, pp.37-44, [8] D. T. Pham and D. Karaboga, Intelligent Optimization Techniques: Genetic Algorithm, Tabu Search, Simulated Annealing and Neural Networks, Springer, London, [9] B. Li and W. S. Jiang, Optimizing complex function by chaos search, Cybernetics and Systems, vol.64, pp , [10] Z. Zhang, Z. Tang and C. Vairappan, A novel learning method for Elman neural network using local search, Neural Information Processing Letters and Reviews, vol.11, no.8, pp , [11] Z. Zhang, G. Yang, J. Yi, Y. Zhu and Z. Tang, A new stochastic dynamic adaptive local search algorithm for Elman neural network, International Journal of Innovative Computing Information and Control, vol.4, no.11, pp , [12] D. T. Pham and X. Liu, Identification of linear and nonlinear dynamic systems using recurrent neural networks, Artificial Intelligence in Engineering, vol.8, pp.67-75, [13] J. Elman, Finding structure in time, Trends in Cognitive Sciences, vol.14, pp , [14] C. Vairappan, S. Gao, H. Tamura and Z. Tang, An improved learning of local search for fuzzy controller network, International Journal of Innovative Computing, Information and Control, vol.5, no.4, pp , [15] Z. Tang, O. Ishizuka and K. Tanno, A learning multiple-valued logic network that can explain reasoning, T. IEE Japan, vol.119-c, no.8-9, [16] E. Ott, C. Grebogi and J. A. Yorke, Controlling chaos, Phys. Rec. Lett., vol.64, pp , [17] K. Aihara, T. Takabe and M. Toyada, Chaotic neural networks, Phys. Lett. A, vol.144, pp , [18] B. Li and W. S. Jiang, Optimizing completes function by chaos search, Cybernet System, vol.29, pp , [19] Z. Lu, L. S. Shieh and G. R. Chen, On robust control of uncertain chaotic systems: A sliding-mode synthesis via chaos optimization, Chaos, Solutions and Fractals, vol.18, pp , [20] L. Zhao, L. S. Shieh, G. Chen and N. P. Coleman, Simplex sliding mode control for nonlinear uncertain systems via chaos optimization, Chaos, Solutions and Fractals, vol.23, pp , [21] R. M. May, Simple mathematical models with very complicated dynamics, Nature, vol.261, pp , 1976.

10 656 Z. ZHANG, Z. TANG, S. GAO AND G. YANG [22] L. Yang and T. Chen, Application of chaos in genetic algorithm, Communications in Theoretical Physics, vol.38, pp , [23] P. Collet and J. P. Eckmann, Iterated Maps of the Interval as Dynamical Systems, Birkhauser, Boston, [24] E. N. Lorenz, Deterministic nonperiodic flow, Journal of the Atmospheric Sciences, vol.20, pp , [25] G. Chen and T. Ueta, Yet another chaotic attractor, International Journal of Bifurcations and Chaos, vol.9, pp , [26] M. J. Ji and H. W. Tang, Application of chaos in simulated annealing, Chaos, Solitons and Fractals, vol.21, pp , [27] B. Liu, L. Wang, Y. H. Jin, F. Tang and D. X. Huang, Improved particle swarm optimization combined with chaos, Chaos, Solitons and Fractals, vol.25, pp , [28] E. A. Wan, Combining fossils and sunspots: Committee predictions, Proc. of the IEEE International Conference on Neural Networks, vol.4, pp , [29] T. J. Cholewo and J. M. Zurada, Sequential network construction for time series prediction, Proc. of the IEEE International Joint Conference on Neural Networks, Houston, TX, pp , [30] M. G. Genet and A. Petrowsky, Sunspot number prediction by a conditional distribution discrimination tree, Proc. of the IEEE International Joint Conference on Neural Networks, vol.1, pp , [31] F. Liu, G. Ng and C. Quek, Rldde: A novel reinforcement learning based dimension and delay estimator for neural networks in time series prediction, Neurocomputing, vol.70, no.7-9, pp , [32] H. Tamura, K. Tanno, H. Tanaka and Z. Zhang, Bi-directional function learning method for time series prediction, ICIC Express Letters, vol.2, no.4, pp , [33] A. S. Weigend, B. A. Huberman and D. Rumelhart, Predicting the future: A connectionist approach, Int. J. Neural System, no.1, pp , [34] R. A. M. van der Linden and the SIDC Team, Online Catalogue of the Sunspot Index, oma.be/html/sunspot.html. [35] K. Kutza, Neural Network at Your Fingertips, [36] A. Aussem, Dynamical recurrent neural networks towards prediction and modeling of dynamical systems, Neurocomputing, vol.28, pp , [37] R. Bone and M. Crucianu, An evaluation of constructive algorithms for recurrent networks on multistep ahead prediction, Proc. of the International Conference on Neural Information Processing, pp , [38] B. W. Wah and M. L. Qian, Time-series predictions using constrained formulations for neuralnetwork training and cross validation, Proc. of the International Conference on Intelligent Information Processing, the 16th IFIP World Computer Congress, pp , 2000.