A Comparison of BA, GA, PSO, BP and LM for Training Feed forward Neural Networks in e-learning Context

Transcription

1 I.J. Intelligent Systems and Applications, 2012, 7, Published Online June 2012 in MECS ( DOI: /ijisa A Comparison BA, GA, PSO, BP and LM for Training Feed forward Neural Networks in e-learning Context Kfka Khan Department Computing and Information Technology, University the West Indies, St. Augustine, Trinidad And Tobago (W.I.) [email protected] Ashok Sahai Department Mathematics and Statistics, University the West Indies, St. Augustine, Trinidad And Tobago (W.I.) [email protected] Abstract Training neural networks is a complex task great importance in the supervised learning field research. We intend to show the superiority (time performance and quality solution) the new metaheuristic bat algorithm (BA) over other more standard algorithms in neural network training. In this work we tackle this problem with five algorithms, and try to over a set results that could hopefully foster future comparisons by using a standard (Proben1: selected benchmark composed problems arising in the field Medicine) and presentation the results. We have selected two gradient descent algorithms: Back propagation and Levenberg- Marquardt, and three population based heuristic: Bat Algorithm, Genetic Algorithm, and Particle Swarm Optimization. Our conclusions clearly establish the advantages the new metaheuristic bat algorithm over the other algorithms in the context elearning. Index Terms Neural networks, supervised learning, metaheuristic, bat algorithm, backpropagation, Levenberg-Marquardt, genetic algorithm, particle swarm optimization I. Introduction Optimization has been an active area research for several decades. As many real-world optimization problems become more complex, better optimization algorithms were needed. In all optimization problems the goal is to find the minimum or maximum the objective function. Thus, unconstrained optimization problems can be formulated as minimization or maximization D dimensional function: Min (or max) f(x), x=(x 1, x 2, x 3,, x D ) (1) where D is the number parameters to be optimized. Many population based algorithms were proposed for solving unconstrained optimization problems. Genetic algorithms (GA), particle swarm optimization (PSO), and bat algorithms (BA) are most popular optimization algorithms which employ a population individuals to solve the problem on hand. The success or failure a population based algorithms depends on its ability to establish proper trade-f between exploration and exploitation. A poor balance between exploration and exploitation may result in a weak optimization method which may suffer from premature convergence, trapping in a local optima and stagnation. GA is one the most popular evolutionary algorithms in which a population individuals evolves (moves through the fitness landscape) according to a set rules such as selection, crossover and mutation [1]. PSO algorithm is another example population based algorithms [2]. PSO is a stochastic optimization technique which is well adapted to the optimization nonlinear functions in multidimensional space and it has been applied to several real-world problems [3]. BA is a relatively new population based metaheuristic approach based on hunting behavior bats [8]. In this algorithm possible solution the problem is represented by bat positions. Quality the solution is indicated by the best position a bat to its prey. BA has been tested for continuous constrained optimization problems [23]. The interest the research in Artificial Neural Networks (ANNs) resides in the appealing properties they exhibit: adaptability, learning capability, and ability to generalize. Nowadays, ANNs are receiving a lot attention from the international research community with a large number studies concerning training, structure design, and real world applications, ranging from classification to robot control or vision [1]. Feed-forward neural networks (NNs) are popular classification tools. Each feed-forward NN consists

2 24 A Comparison BA, GA, PSO, BP and LM for Training Feed forward an input layer neurons. In case the classification problem the input layer consists as many neurons as there are measurements in the patterns, that is, for each measurement there exists exactly one input neuron. Furthermore, a feed-forward NN consists an arbitrary number hidden layers neurons, and an output layer (cf. Figure 1). The outline this paper is as follows. Section II describes the different types metaheuristics used in this paper, two artificial neural network metaheuristics, 1) gradient-descent algorithm, and 2) Levenberg- Marquardt backpropaction algorithm, three population based methheuristics 1) genetic algorithm, 2) particle swarm algorithm, and 3) bat algorithm. In Section III, benchmarking testing and elearning s are described together with the performance analysis the test results. Most importantly the parameters for the algorithms are given for the individual tests. Section IV presents conclusions and final comments. II. Metaheuristics Metaheuristics have been established as one the most practical approach to simulation optimization. However, these methods are generally designed for combinatorial optimization, and their implementations do not always adequately account for the presence simulation noise. Research in simulation optimization, on the other hand, has focused on convergent algorithms, giving rise to the impression a gap between research and practice. This paper surveys the use metaheuristics for simulation optimization, focusing on work bridging the current gap between the practical use such methods and research, and points out some promising directions for research in this area. The main emphasis is on two issues: accounting for simulation noise in the implementation metaheuristics, and convergence analysis metaheuristics that is both rigorous and practical value. To illustrate the key points, metaheuristics are discussed and used, namely artificial neural networks, Levenberg-Marquardt search algorithm, genetic algorithm, particle swarm optimization, and the bat algorithm. Figure 1. ANN Neuron Hence, BP is a gradient-descent local search procedure (expected to stagnate in local optima in complex landscapes). The squared error the ANN for a set patterns is: ( ) (2) The actual value the previous expression depends on the weights the network. The basic BP algorithm calculates the gradient E (for all the patterns) and updates the weights by moving them along the gradientdescendent direction. This can be summarized with the expression w = η E, where the parameter η > 0 is the learning rate that controls the learning speed. The pseudo-code the BP algorithm is shown (cf. Figure 2). Initialize Weights; While not Stop-Criterion do For all i, j do End For End While η A. Artificial Neural Networks (ANNs) The backpropagation algorithm (BP) [2] is a classical domain-dependent technique for supervised training (cf. Figure 1). It works by measuring the output error, calculating the gradient this error, and adjusting the ANN weights (and biases) in the descending gradient direction. Figure 2. Pseudocode backpropagation algorithm B. Levenberg-Marquardt The Levenberg-Marquardt algorithm (LM) [9] is an approximation to the Newton method (cf. Figure 3) used also for training ANNs.

3 A Comparison BA, GA, PSO, BP and LM for Training Feed forward 25 C. Genetic Algorithm (GA) A GA [12] is a stochastic general search method. It proceeds in an iterative manner by generating new populations individuals from the old ones (cf. Figure 4). Every individual is the encoded (binary, real, etc.) version a tentative solution. Figure 5 shows the selection and recombination phases the genetic algorithm. Figure 3. Flow chart Levenberg-Marquardt The Newton method approximates the error the network with a second order expression, which contrasts to the Backpropagation algorithm that does it with a first order expression. LM updates the ANN weights as follows: [ ( ) ( )] ( ) (3) Where J p (w) is the Jacobian matrix the error vector e p (w) evaluated in w, and I is the identity matrix. The vector error e p (w) is the error the network for pattern p, that is, e p (w) = t p o p (w). The parameter µ is increased or decreased at each step. If the error is reduced, then µ is divided by a factor β, and it is multiplied by β in other case. Levenberg-Marquardt performs the steps detailed in Fig. 3. It calculates the network output, the error vectors, and the Jacobian matrix for each pattern. Then, it computes w using (3) and recalculates the error with w + w as network weights. If the error has decreased, µ is divided by β, the new weights are maintained, and the process starts again; otherwise, µ is multiplied by β, w is calculated with a new value, and it iterates again. Initialize Weights; While not Stop-Criterion do Calculates c p (w) for each pattern e 1 = ( ) ( ) Calculates J p (w) for each pattern Repeat Calculates e2 = ( ) ( ) If e1 e2 Then μ = μ * β end If Until e1 < e2 μ = μ/β w = w + w End While Figure 4. Pseudocode Levenberg-Marquardt algorithm Figure 5. Selection and recombination phase the genetic algorithm The canonical algorithm applies stochastic operators such as selection, crossover, and mutation on an initially random population in order to compute a new population. In generational GAs all the population is replaced with new individuals. In steady-state GAs (used in this work) only one new individual is created and it replaces the worst one in the population if it is better. The pseudo-code the GA we are using here can be seen in Fig. 4. The search features the GA contrast with those the BP and LM in that it is not trajectory-driven, but population-driven. The GA is expected to avoid local optima frequently by promoting exploration the search space, in opposition to the exploitative trend usually allocated to local search algorithms like BP or LM. The pseudocode for the genetic algorithm is shown on Figure 6. t = 0 Initialize: P(0) = {a 1 (0),, a μ (0)} I μ Evaluate: P(0): {Φ(a 1 (0)),, Φ(a μ (0))} While (P(t)) true //Reproductive loop Select: P (t) = s θz {P(t)} Recombine: P (t) = θz {P (t)} Mutate: P (t) = m θm {P (t)} Evaluate: P (t): {Φ(a 1 (t)),, Φ(a λ (t))} Replace: P(t+1) = r θr (P (t) Q) t = t + 1 End While Figure 6. Pseudocode genetic algorithm D. Particle Swarm Optimization (PSO) The PSO algorithm was first introduced by Eberhart and Kennedy [5, 6, 7, and 8]. Instead using evolutionary operators to manipulate the individuals,

4 26 A Comparison BA, GA, PSO, BP and LM for Training Feed forward like in other evolutionary computational algorithms, each individual in PSO flies in the search space with a velocity which is dynamically adjusted according to its own flying experience and its companions flying experience. Each individual is treated as a volume-less particle (a point) in the D-dimensional search space (cf. Figure 7). better than the p g, assign the current best fitness value to p. g Step6 For each particle, update particle velocity, Step7 Update particle position. Step8 Repeat Step2-7 until a stop criterion is satisfied or a predefined number iterations are completed. The particle swarm flowchart is shown on Figure 8. Figure 7. Particles movement in PSO The ith particle is represented as X i = (x il, x i2,..., x id ). The best previous position (the position giving the best fitness value) the ith particle is recorded and represented as P i = (p il, p i2,..., p id ). The index the best particle among all the particles in the population is represented by the symbol gb representing global best. The index the best position for each particle in the population is represented by the symbol ib representing the individual s best. The rate the position change (velocity) for particle i is represented as V i to the following equation: (v il, v i2,..., v id ). The particles are manipulated according to the following equations: () ( ) () ( ) (4) The algorithm can be summarized as follows: Step1 Initialize position and velocity all the particles randomly in the N dimension space. Step2 Evaluate the fitness value each particle, and update the global optimum position. Step3 According to changing the gathering degree and the steady degree particle swarm, determine whether all the particles are re-initialized or not. Step4 Determine the individual best fitness value. Compare the p every individual with its current i fitness value. If the current fitness value is better, assign the current fitness value to p. i Step5 Determine the current best fitness value in the entire population. If the current best fitness value is (5) E. Bat Algorithm (BA) Figure 8. PSO Flowchart The bat algorithm [23] uses the echolocation behaviour bats (cf. Figure 9). These bats emit a very loud sound pulse (echolocation) and listens for the echo that bounces back from the surrounding objects (cf. Figure 7). Their signal bandwidth varies depending on the species, and increases using harmonics. The ith bat flies randomly with velocity v i at position X i with a fixed frequency f min. The bat varies its wavelength λ and loudness A 0 to search for food. Figure 9. Micro-bat hunting for prey using echolocation

5 A Comparison BA, GA, PSO, BP and LM for Training Feed forward 27 The number bats is n b. (6) ( ) (7) ( ) (8) It is assumed that the loudness varies from a large (positive) A 0 to a minimum constant value A min. The new solutions the ith bat at time step t are given by X i t and v i t. (9) where [0, 1] is a random vector drawn from a uniform distribution. The domain size the problem in context determines the values f min and f max. Initially, each bat is randomly assigned a frequency which is drawn uniformly from [f min, f max ]. For local search procedure (exploitation) each bat takes a random walk creating a new solution for itself based on the best selected current solution. (10) where [ ] is a random number, A t is the average loudness all bats at this time step. The loudness decreases as a bat tends closer to its food and pulse emissions rate increases. (11) [ ] (12) where (comparative to the cooling factor schedule in simulated annealing) and are constants. III. Benchmarking Testing and e-learning s Proben1 is a collection problems for neural network learning in the realm pattern classification and function approximation plus a set rules and conventions for carrying out benchmark tests with these or similar problems. Proben1 contains 15 data sets from 12 different domains. All s represent realistic problems which could be called diagnosis tasks and all but one consist real world data. The s are all presented in the same simple format, using an attribute representation that can directly be used for neural network training. Along with the s, Proben1 defines a set rules for how to conduct and how to document neural network benchmarking. The purpose the problem and rule collection is to give researchers easy access to data for the evaluation their algorithms and networks and to make direct comparison the published results feasible [18]. A. Standard s We chose three problems from the well-known Proben1 benchmark set (cf. Table 1). The standard Proben1 benchmarking rules were applied [18]. s were used with even-numbered examples used for the training set and odd-numbered examples used for the test set. Cancer Diabetes Heart Thyroid patterns B. Elearning datset Table 1 Standard s parameters outputs Classification Pathological or normal Pathological or normal Pathological or normal Hyper hypo function A group medical students were tested using a medical health application. Students were given one hour to complete the elearning tasks and complete a set questions online. Ninety students participated in the experiment. Six students were tested thoroughly in the Usability Lab. The average time taken to complete the tasks were 48 minutes and the average percentage correct questions was 76 percent. elearni ng patterns Table 2 Elearning s parameters outputs C. Results and algorithm s settings Classificati on Bad or Good design The mean squared error function was used as the training error. For all population based algorithms the population size was set at 30 [23]. This was done to uniformly survey the performance the algorithms without too much variation. The parameters for the genetic algorithm were mutation rate=0.001, crossover rate=0.7 with the selection method being uniform (suitably fit members are chosen to breed at random) and crossover method is single-point (parents' genes are swapped over at one point along their length) [5]. The two constants for the PSO algorithm were given values 2 [5]. The bat algorithm used the following settings: i A 0 = 1.0, A min = 0.0, r 0 = 0.01, α= γ=0.6 [23]. The learning rate the steepest decent and Levenberg- Marquardt back-propagation algorithms were set to 0.3 [18]. Each run stops either when the error obtained

6 28 A Comparison BA, GA, PSO, BP and LM for Training Feed forward is less than 10-8, or when the maximal number evaluations ( ) is achieved. All tests were carried out on a 64-bit operating system with a 2.4GHz CPU and 2GB RAM. Table 1 shows the average number function evaluations obtained for each algorithm for each the benchmark s considered. Table 2 shows the average error obtained for each algorithm for each the benchmark s considered. Best performances are highlighted in both tables. Table 3 Average number function evaluations Average number function evaluations BP LM GA PSO BA Heart Thyroid Cancer Diabetes elearning Table 4 Average error Average error BP LM GA PSO BA Cancer e e e e e-5 Diabetes e e e e e-5 Heart e e e e e-6 Thyroid 1 1.1e e e e e-6 elearning 8.46e e e e e-6 The bat algorithm incorporates a combination particle swarm optimization (PSO), harmony search (HS) and simulated annealing (SA) using a certain combination parameters. The results clearly show that the combination these algorithms into the bat algorithm enhances its performance as an optimization algorithm under different conditions. This is shown by the fact that for all the test s the BA outperforms the other algorithms. However, it is noted that in some cases the improvements are only minimal. IV. Conclusions A comparison algorithms for training feed forward neural networks was done. Tests were done on two gradient descent algorithms: Backpropagation and Levenberg-Marquardt, and three population based heuristic: Bat Algorithm, Genetic Algorithm, and Particle Swarm Optimization. Experimental results show that bat algorithm (BA) outperforms all other algorithms in training feed forward neural networks. It supports the usage the BA in further experiments and in further real world applications. In this study, the Bat Algorithm (BA) Algorithm, which is a new, simple and robust optimization algorithm, has been used to train standard and elearning s for classification purposes. First, well-known classification problems obtained from Proben1 [18] repository have been used for comparison. Then, an experimental setup has been designed for an elearning based on health classification. Training procedures involves selecting the optimal values the parameters such as weights between the hidden layer and the output layer, spread parameters the hidden layer base function, center vectors the hidden layer and bias parameters the neurons the output layer. Trapping a local minimum is a disadvantage for these algorithms. The GA and ABC algorithms are population based evolutional heuristic optimization algorithms. These algorithms show better performance than derivative based methods for well known classification problems such as Iris, Glass, and Wine and also for experimental elearning based health classification. However, these algorithms have the disadvantage a slow convergence rate. If the classification performances are compared, experimental results show that the performance the BA algorithm is better than those the others. The success the classification results test problems are superior and also correlates with the results many papers in the literature. In real-time applications, number neurons may affect the time complexity the system. The results elearning classification problem are reasonable and may help to the planning algorithms in other elearning applications. References [1] Alavi, M., Yoo, Y., &Vogel. D. R. Using Information Technology to Add Value to Management Education. The Academy Management Journal, 40: , [2] Ardito, C. et al. An approach to usability evaluation elearning applications. Universal Access in the Information Society, 4(3): , [3] Boehner, K., DePaula, R., Dourish, P. & Sengers, P. How emotion is made and measured. Int. Journal Human-Computer Studies, 65: , [4] Branco, P., Firth, P., Encarnacao, M., & Bonato, P. Faces Emotion in Human-Computer Interaction. CHI 2005, [5] Chiu, C., Hsu, M., Sun, S., Lin, T., Sun, P. Usability, quality, value and elearning continuance decisions. Computers & Education, 45: 4, [6] Council Ministers Education. UNESCO World Conference on Higher Education, Report, 1998.

7 A Comparison BA, GA, PSO, BP and LM for Training Feed forward 29 [7] Hannula, M. Motivation in mathematics: Goals reflected in emotions. Educational Studies in Mathematics, 63(2), [8] ISO Stware Ergonomics Requirements for fice work with visual display terminal (VDT). Part 11: Guidance on usability, International Organisation for Standardisation, [9] Juutinen, S. & Saariluoma, P. Usability and emotional obstacles in adopting elearning - A case study. Proc. IRMA International conference, [10] Juutinen, S., Saariluoma, P., Iaitos, T. Emotional obstacles for elearning - a user psychological analysis. European Journal Open, Distance and ELearning, [11] Keevil, B. Measuring the usability index your website. Computer and Information Science, , [12] Koohang, A. Foundations Informing Science. T. Grandon Gill & Eli Cohen (eds.), (5): , [13] Kumar, A., Kumar, P. & Basu, S.C. Student perceptions virtual education: An exploratory study. Proc. Information Resources Management Association International Conference, , [14] Likert, R. A technique for the measurement attitudes. Archives psychology, 22(140), [15] Nebro, A., Durillo, J., Coello, C., Luna, F., Alba, E. A Study Convergence Speed in Multi-objective Metaheuristics. Parallel problem solving from nature, Springer, Lecture Notes in Computer Science, 5199: , [16] Paneva, D., Zhelev, Y. Models, Techniques and Applications elearning Personalization. International Journal. ITK, 1(3): , [17] Parrott, W. G. (). The nature emotion. Emotions and motivation. In M. Brewer and M. Hewstone (Eds.), [18] Prechelt, L. Proben1 A Set Neural Network Benchmark Problems and Benchmarking Rules. Technical Report. Fakultat fur Informatik, Universitat Karlsruhe, Karlsruhe, Germany. [19] Ruiz, G.J., Mintzer, J.M., Leipzig, M.R. The Impact ELearning on Medical Education. ELearning and Digital Media, 8(1): 31-35, [20] Shneiderman, B. Exploratory experiments in programmer behavior. International Journal Computer and Information Sciences, 5: , [21] Trentin, G. Using a wiki to evaluate individual contribution to a collaborative learning project. Journal computer assisted learning, 25(1): 43-55, [22] Villiers, R.D. Usability evaluation an elearning tutorial: criteria, questions and case study. Proc. IT research in developing countries, ACM, 75: [23] Xin-She Yang. A New Metaheuristic Bat-Inspired Algorithm. Nature Inspired Cooperative Strategies for Optimization (NICSO). Studies in Computational Intelligence, 284:65-74, Mr. Kfka Khan was born in San Fernando, Trinidad and Tobago in He received the B.Sc. and M.Sc. degrees from University the West Indies, in 2002 and 2008, respectively. He is working presently at The University The West Indies; St. Augustine Campus (TRINIDAD & TOBAGO) as a Tutor in Computer Science in the Department Computing and Information Technology (Faculty Science & Agriculture) since September Mr. Khan started his teaching-n-research career as a Demonstrator in Computer Science at the University The West Indies at the Department Mathematics and Computer Science. He has up-to-date, published seven research papers in journals international repute & in the proceedings international conferences. Dr. Ashok Sahai is working presently at The University The West Indies; St. Augustine Campus (TRINIDAD & TOBAGO) as a Pressor Statistics in the Department Mathematics & Computer Science(Faculty Science & Agriculture) since February Dr. Sahai started his teaching-n-research career as a Lecturer in Statistics Department at Lucknow University (INDIA) in July 1966, and continued thereat till April He has, upto-date, published more than one hundred research papers in journals international repute & in the proceedings international conferences. He worked as Reader in Statistics and as Pressor Statistics in the Department Mathematics at University Roorkee (Now IIT Roorkee) during the period: April July Pr. Sahai had also worked as an Assoc. Pressor Statistics at University Dar-Es-Salaam; TANZANIA (East Africa) during the period: July June 1984, and as a Pressor Statistics at University Swaziland (Southern Africa) during the period: July June He worked as a Guest PharmacoEconomic Research Centre; University Arizona, TUCSON during the period from July 2003 to October 2003 and as Visiting Hyderabad; INDIA during December 2003 to January 2006 in ICFAI Tech. University, Medchel Rd.; in Aurora School Management at Chikkadpally; and in St. Ann s P.G. (Management) College For Women at Mallapur.