# degrees of freedom and are able to adapt to the task they are supposed to do [Gupta].

Save this PDF as:

Size: px
Start display at page:

Download "degrees of freedom and are able to adapt to the task they are supposed to do [Gupta]."

## Transcription

1 1.3 Neural Networks 19 Neural Networks are large structured systems of equations. These systems have many degrees of freedom and are able to adapt to the task they are supposed to do [Gupta]. Two very different types of Neural Networks exist: ackpropagation Neural Networks and Probabilistic Neural Networks. ackpropagation Neural Networks use equations that are connected using weighting factors. The selection of the weighting factors makes these Neural Nets so powerful. Probabilistic Neural Nets use statistical methods to select the equations within the structure and do not weight these functions differently [Chrekassky] ackpropagation Neural Networks Structure and Algorithm of a ackpropagation Neural Network The ackpropagation Neural Networks consist of neurons organized in one input layer and one output layer and several hidden layers of neurons (Fig ). Several planes can exist that are generally built up like the first plane [Zurada]. Neurons perform some kind of calculation. They use inputs to compute an output that represents the system. The outputs are given on to a next neuron. An arc indicates to which neurons the output is given. These arcs carry weights.

2 20 In (Fig ) the structure of a backpropagation neural network is shown. The Input vector consists of variables that are used to predict the desired variable. The input vector, using an example of a heat exchanger with inputs and outputs. The inputs could consist of the mass flow rate of water, the inlet temperature of water, the velocity of the air flowing across the pipe and the temperature of the air. The output could be the effeciveness of the heat exchanger. This information of the input is given to the input layer of the neural network. In the input layer the data are normalized. A neural network does not work in a unit system. A neural network looks solely at the numbers. It is therefore necessary to normalize the input. This means if the temperature changes over the whole range from 300K-400K the range of the temperature is normalized to a range of 0-1. The same normalization happens for each individual input.

3 Input Vector 21 Neurons in first Layer Fixed weights on arcs Hidden Layers Output value can be more than one Other prediction plane The planes can be interconnected if necessary Fig Fig organization of neural network The normalized signal is given on to the next neuron. Each neuron receives signals from any number of other neurons [Zurada]. The architecture of the neural network does not need be as shown in (Fig ) but for now we assume the architecture of the neural network as shown in (Fig ). Each signal is weighted as it is given from the input layer to the first hidden layer. As the new signals reaches a neuron in the hidden layer they all the signals that are received by a neuron are summed up. This process can be looked at as a vector multiplication (Eqn ) of the weight vector w and the signal vector y previous of the previous

4 22 layer [Zurada]. In the hidden layer new signals are computed and given on to the next hidden layer. This process continues until the output layer is reached. x = w * y previous Eqn The scalar x causes the neuron to react in a certain way. There are two very widelyused functions that are used in backpropagation neural networks: the threshold function (Eqn ) and the sigmoid function (Eqn ). y new i, = threshold ( x) Eqn ynew,i = sigmoid ( x) Eqn The process going on in a neuron is shown in (Fig ). It can be seen that the sigmoid function has a continuous output whereas the output from the threshold function is either 0 or a value, usually one.

5 Neurons 23 weight weight weight Σ Summation of stimulating signals coming into neuron θ Threshold or other actication function Activating signal for next layer of neurons Threshold function Y binary signal X Sigmoid Function Y continuous signal X Fig Sigmoid function and Threshold function The following process in the neural network is basically a repetition of the process that was just described. The values of the signals and the weights will be different but the process

6 24 itself will continue similarly. The signals are weighted and then summed up, they cause a reaction of the neuron and the reaction, which is the new signal, will be given on to the next neuron. The last step is to give the signal to the output layer. The output layer can consist of one or more neurons. More neurons would mean that the plane of the neural network has multiple outputs. In the output neuron a calculation is again necessary to yield a value that is not normalized and has again a physical meaning. The weights on the connections are not chosen arbitrarily, but result from training. Several training algorithms exist. The well known backpropagation training, that actually gave the name for the backpropagation neural network, is discussed in Section Training of a ackpropagation Neural Network Initially the network predicts an output for one input vector for which the true output is known. This combination of known output and input vector is called training sample. The predicted output is compared to the known value. The weights on the arcs are adjusted depending on how the prediction of the actual result. In (Fig ) the general process in a backpropagation neural network is shown again. This figure is used for nomenclature in the process shown in (Fig ) for the training.

7 Inputs Hidden Layer Outputs 25 z 1 1 v 11 1 y 1 v J1 v j1 w 11 1 o 1 z i i v 1i v Ji v ji j y j w 1j w Kj w K1 z I I v 1I v ji v JI J y J w KJ K o K Fig ackpropagation neural network, nomenclature used for learning description In (Fig ) the first step in the training is to assume initial weights. With these initial weights the first output for one training sample, o, is calculated. The third step in the training is to calculate the cycle error made in the prediction. The cycle error is as shown in (Fig ) the sum of the squared differences between the known result the neural network should predict, d, and the result, o, the neural network does predict. The cycle error is computed for each run through the training data. The cycle error is used in each cycle of the training to compare to the maximal allowable cycle error in step eight of (Fig ). In the fourth step, the training signals are calculated. The weights between the different layers are changed according to these training signals, that are calculated in step four in (Fig ). The process is repeated until no more training samples are available. If there are no more training samples available the cycle error then is calculated and compared to the maximal allowable error. If the cycle error exceeds the maximal allowable cycle error, then the training has to start with a new

8 26 cycle. This is repeated until the cycle error is small enough. The training of ackpropagation Neural Networks is a very time consuming process [Zurada]. Note that the error signal vector must be determined in the output layer first and then it is propagated back toward the network input nodes.

9 27 Step 1 Initialize weights W,V eginning of a new training cycle eginning of a new training step Step 2 Submit pattern z and compute layer's response y=g [Vz] o=g [Wy] Step 3 Compute cycle error E=E+1/2 d-o 2 E=0 N Step 4 Compute learning signals δo, δy d o =1/2[(d k -o k )(1-o k 2 )] d y =w j t d o f' y f' y =1/2[1-y i 2 ] Step 8 Y E<E max Step 5 Adjust weights of output layer W=W+h d o y' Stop Step 6 Adjust weights of hidden layer V=V+h d y z' N Step 7 More Patterns in the Training set? Y Fig lock Diagram for ackpropagation Learning Algorithm [Zurada]

10 28 Training of ackpropagation Neural Nets depends on the order in which the training samples are used [Zurada]. The dependence on order is a very critical point. It is not known how the order influences the selection of the weights. A frequently used method is to randomly choose the order of the samples and repeat it several times to see how the weights change depending on the order of the training samples [Gupta] Pros and Cons of ackpropagation Neural Nets ackpropagation Neural Networks have the very desirable characteristic of being very flexible. They can be used for pattern recognition as well as for decision making problems. Another advantage is like for every other neural network that the process is highly parallel and therefore the use of parallel processors is possible and cuts down the necessary time for calculations [Specht 91, Gupta, Cherkassky]. ackpropagation Neural Networks have negative characteristics. The training of the network can need a substantial amount of time [Gupta]. As soon as the training is done though the network performs very fast. In some respect a disqualifying aspect, is that the size of the training data for ackpropagation Neural Nets has to be very large. Often it is almost impossible to provide enough training samples [Zurada]. Examples would be when training samples result out of very expensive experiments. Or when the data are observations of nature and these observations can only occur very rarely.

11 1.3.2 Probabilistic Neural Network 29 Probabilistic Neural Nets use a statistical approach in their prediction algorithm. The bases for the statistical approach is given in the The ayes Strategy for Pattern Classification [Specht 90] The ayes Strategy for Pattern Classification ayes strategies are strategies used to classify patterns. These strategies can be applied to problems containing any number of categories[parzen, Specht 90]. Consider a situation for which the state of nature θ is known to be either θ A and θ. It has to be decided in which of these two states θ is. The available information is a p-dimensional input vector X T = [ X 1,... X i,... Xp ]. The ayes decision rule becomes ( X) = θa if A A A( X) > ( X) ( X) = θ if ( X) < ( X) d h I f h I f d h I f h I f A A A Eqn where f A ( X ) and f ( X ) are probability density functions for categories A and. I A is the loss function associated with the decision d( X) for the decision d( X) = θ A when θ = θ. I is the loss function = θ when θ = θ A, respectively. The losses for the correct decision are equal to zero. The ratio of the loss functions in (Eqn ) can usually be set to -1 (an inverter) if there is no reason for biasing the decision. Otherwise it would mean that the decision d( X) = θ A for example cannot be trusted as much as the decision d( X) = θ. h A

12 30 is the prior probability of occurrence of patterns of the category A. h = 1 h is the prior A probability of θ = θ A [Parzen]. The two category decision surface according to (Eqn ) therefore is fa ( X) = K f ( X) with Eqn K = h I h I A A. Eqn A similar decision rule can be stated for multi-category problems [Parzen, Specht 90]. A decision surface separating two states of nature is shown in (Fig ). The decision surface can be arbitrarily complex since there are no limitations on the densities except the restrictions on the probability density functions. Namely non negative integrable integral over space equals unity.

13 31 A A A A A A A A A Decision Surface A Fig Decision surface between states of nature A and To be able to use (Eqn ) it is necessary to estimate the probability density function accurately. The only available information to estimate the density functions are the training samples. The loss functions require subjective evaluation if the decision is biased. A loss function is a function that subtracts a value of the overall value for a wrong decision [Specht 90]. According to [Parzen] a class of probability density function estimates asymptotically approaches the underlying parent density function. The only requirement is that the density function is continuous. Parzen [Parzen] showed how to construct a family of estimates of the density function

14 32 f n ( X) = 1 nλ n i= 1 X X W λ Ai Eqn which is consistent at all points X at which the probability density function is continuous. Parzen imposes on the weighting function W( y ) which are sup W( y) < y< < Eqn where sup indicates the supremum. (Eqn ) says that the weights are not unbound and cannot reach infinity. Where W ( y) dy < Eqn ( y) lim yw = 0 y and Eqn

15 W( y) dy = Eqn In (Eqn ) λ is chosen as a function of n such that n lim ( n) λ = 0 Eqn and lim nλ n ( n) = Eqn Parzen further proved in [Parzen] that the estimate of the density function is consistent in quadratic mean in the sense that the expected error goes to zero with the number of training samples going to infinity ( ) f ( ) 2 E fn X X 0 as n. Eqn This definition of consistency says that the expected error gets smaller as the number of training samples gets bigger. This consistency is very important since it means that the prediction will converge towards the underlying function with more given training samples.

16 34 Murthy [Murthy] and Cacoullos [Cacoullos] relaxed and extended Parzen's results. Using Gausian distribution for the weight function, the density function can be expressed as f ( ) A X = ( 2 ) T ( X X ) ( X X ) m 1 1 Ai p exp p π 2σ m i= 1 2σ 2 Ai. Eqn The density function here is simply the sum of Gaussian distributions centered at each training sample. Several more distributions can be used [Specht 90]. The smoothing parameter σ has a very important influence on the approximation. The influence will be shown later. ut it is stated already here that this parameter needs special attention in the further use of Gaussian Probabilistic Neural Networks. The structure of the calculations for the Probability density function has striking similarities to a feed-forward neural network. The structure of the probabilistic approach is very parallel, like a feed-forward neural network. A probabilistic neural network has certain differences compared to a backpropagation neural network the approach using. The biggest advantage is the fact that the probabilistic approach works with a one-step-only learning. The learning of ackpropagation Nerual Networks can be described as trial and error. This is no longer the case for the probabilistic neural network. The experience is learned not by trial but by experience others made for the neural network [Specht 90, Gupta].

17 Probabilistic Neural Network 35 Probabilistic Neural networks are frequently used to classify patterns based on learning from examples. Probabilistic neural nets base the algorithm on The ayes Strategy for Pattern Classification. Different rules determine pattern statistics from the training samples. ackpropagation uses methods as previously described that are not based on statistical methods. ackpropagation needs a long time and many iterations and feedbacks until it gradually approaches the underlying function [Specht 91]. It would be desirable to approach the parameters in a one-step-only approach. The ayes Strategy for Pattern Classification extracts characteristics from the training samples to come to knowledge about the underlying function.

18 36 X 1 X 2 X n-1 X n Input Units Pattern Units Summation Unit Output Unit Y(X) Fig lock diagram of a Probabilistic Neural Network The general structure of a probabilistic neural network (Fig ) is the same as discussed in Section A Probabilistic Neural Network consists of one input layer, and two hidden layers. The first hidden layer contains the pattern units. The pattern units contain the important functional form that was previously discussed. The calculations in the pattern unit are shown in (Fig ). Each pattern unit represents information on one training sample. Each pattern unit calculates the probability on how well the input vector fits into the pattern unit. In the second hidden layer there is only one summation unit. Here it is decided upon the

19 individual results of each pattern unit in which pattern the input vector finally belongs. The 37 output unit performs again a calculation to give the output a physical meaning. For a probabilistic neural network it is not always possible to have multiple outputs. A futher big difference that exists between a backpropagation neural network and a probabilistic neural network is the difference in the process inside the neurons. A probabilistic neural network uses functions that are based on knowledge resulting from The ayes Strategy for Pattern Classification. The strength of a probabilistic neural network therefore is not the selection of weights that fit the data in the best way. The strength of a probabilistic neural network lies in the function that is used inside the neuron.

20 38 Neurons in Input Layer T ( ) ( ) Di = X XAi X X Ai f 1 1 A( X) = exp[ D ] p i m ( ) p 2π 2σ 1 1 p ( 2π ) 2 p σ m T ( ) ( ) Di = X X Ai X X Ai Probability Density function fa( X 1 1 ) = exp p m ( ) p 2π 2 σ T ( X X ) ( X X ) Ai 2σ 2 Ai X Ai X Fig Process in a Pattern Unit The function used in a neuron of a pattern unit is a probability density function. The probability density function needs the distance between the sample point and the position

21 where the prediction should take place to calculate the output (Fig ). The output of each 39 pattern unit is summed up and in the summation unit and then converted to a result with physical meaning. Known Value Estimated Value Y Known Value X 1 X 2 Available Data Point Distance Point to estimate Value of function Distance Available Data Point Fig Distance between the training sample and the point of prediction The ayes Strategy for Pattern Classification is valid as well for continuous results [Parzen62]. It is therefore possible to predict outputs that are continuous Why Probabilistic Neural Networks and not ackpropagation Neural Networks? As Cherkassky, Friedman and Wechsler wrote in [Cherkassky94] there is tension between neural network researchers and statisticians because they have different backgrounds and different objectives in designing algorithms or analytical methods. Statistical methods tend

22 40 to put more emphasis on the structure of the data. For Neural network methods the structure of the data is secondary. Therefore the amount of data needed for statistical methods is a lot smaller than the amount of data needed for artificial neural network approaches. Most methods are asymptotically good [Cherkassky94] but most of them have severe drawbacks as well. ackpropagation neural networks need a very large number of training samples and need a lot of time to gradually approach good values of the weights. Adding of new information requires retraining and this is computationally very expensive for ackpropagation Neural Nets but not for Probabilistic Neural Nets. Probabilistic Neural Nets have the big advantage that the prediction algorithm works with only few training samples. The other very big advantage is that they are very flexible and new informations can be added immediately with almost no retraining.

23 1.4 Chapter Summary 41 Different approaches to model a system with data available have been shown. Each one of them has its very own qualities and therefore advantages. The parameters in curve fitting can have physical meaning if a physical model is available and can be used for extrapolation. On the other hand if no physical relation is being used the function chosen to use for the fitting needs some more attention, especially if the problem is multi-dimensional. Ordinary Splines and -Splines are very flexible tools to perform curve fitting. Problems that occur can be severe. Ordinary spline functions deviate from the expected very easily as soon as the available data is not without error and would need interpolation. - Splines do not have this kind of problem but a minimization problem has to be performed instead. Probabilistic Neural Networks do not need very much additional input but the results are not interpretable. Probabilistic Neural Nets have a very simple structure and are therefore very stable procedures. Probabilistic Neural Networks perform very well for only few available training samples, the quality increases as the number of training samples increases.

### TRAINING A LIMITED-INTERCONNECT, SYNTHETIC NEURAL IC

777 TRAINING A LIMITED-INTERCONNECT, SYNTHETIC NEURAL IC M.R. Walker. S. Haghighi. A. Afghan. and L.A. Akers Center for Solid State Electronics Research Arizona State University Tempe. AZ 85287-6206 mwalker@enuxha.eas.asu.edu

### Neural Networks. CAP5610 Machine Learning Instructor: Guo-Jun Qi

Neural Networks CAP5610 Machine Learning Instructor: Guo-Jun Qi Recap: linear classifier Logistic regression Maximizing the posterior distribution of class Y conditional on the input vector X Support vector

### An Introduction to Neural Networks

An Introduction to Vincent Cheung Kevin Cannons Signal & Data Compression Laboratory Electrical & Computer Engineering University of Manitoba Winnipeg, Manitoba, Canada Advisor: Dr. W. Kinsner May 27,

### Lecture 8 February 4

ICS273A: Machine Learning Winter 2008 Lecture 8 February 4 Scribe: Carlos Agell (Student) Lecturer: Deva Ramanan 8.1 Neural Nets 8.1.1 Logistic Regression Recall the logistic function: g(x) = 1 1 + e θt

### Chapter 4: Artificial Neural Networks

Chapter 4: Artificial Neural Networks CS 536: Machine Learning Littman (Wu, TA) Administration icml-03: instructional Conference on Machine Learning http://www.cs.rutgers.edu/~mlittman/courses/ml03/icml03/

### 6.2.8 Neural networks for data mining

6.2.8 Neural networks for data mining Walter Kosters 1 In many application areas neural networks are known to be valuable tools. This also holds for data mining. In this chapter we discuss the use of neural

### Artificial Neural Computation Systems

Artificial Neural Computation Systems Spring 2003 Technical University of Szczecin Department of Electrical Engineering Lecturer: Prof. Adam Krzyzak,PS 5. Lecture 15.03.2003 147 1. Multilayer Perceptrons............

### Follow links Class Use and other Permissions. For more information, send email to: permissions@pupress.princeton.edu

COPYRIGHT NOTICE: David A. Kendrick, P. Ruben Mercado, and Hans M. Amman: Computational Economics is published by Princeton University Press and copyrighted, 2006, by Princeton University Press. All rights

### Stock Prediction using Artificial Neural Networks

Stock Prediction using Artificial Neural Networks Abhishek Kar (Y8021), Dept. of Computer Science and Engineering, IIT Kanpur Abstract In this work we present an Artificial Neural Network approach to predict

### Introduction to Machine Learning and Data Mining. Prof. Dr. Igor Trajkovski trajkovski@nyus.edu.mk

Introduction to Machine Learning and Data Mining Prof. Dr. Igor Trakovski trakovski@nyus.edu.mk Neural Networks 2 Neural Networks Analogy to biological neural systems, the most robust learning systems

### PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION

PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION Introduction In the previous chapter, we explored a class of regression models having particularly simple analytical

### Neural Nets. General Model Building

Neural Nets To give you an idea of how new this material is, let s do a little history lesson. The origins are typically dated back to the early 1940 s and work by two physiologists, McCulloch and Pitts.

### CHAPTER 5 PREDICTIVE MODELING STUDIES TO DETERMINE THE CONVEYING VELOCITY OF PARTS ON VIBRATORY FEEDER

93 CHAPTER 5 PREDICTIVE MODELING STUDIES TO DETERMINE THE CONVEYING VELOCITY OF PARTS ON VIBRATORY FEEDER 5.1 INTRODUCTION The development of an active trap based feeder for handling brakeliners was discussed

### Speech Synthesis by Artificial Neural Networks (AI / Speech processing / Signal processing)

Speech Synthesis by Artificial Neural Networks (AI / Speech processing / Signal processing) Christos P. Yiakoumettis Department of Informatics University of Sussex, UK (Email: c.yiakoumettis@sussex.ac.uk)

### INTRODUCTION TO NEURAL NETWORKS

INTRODUCTION TO NEURAL NETWORKS Pictures are taken from http://www.cs.cmu.edu/~tom/mlbook-chapter-slides.html http://research.microsoft.com/~cmbishop/prml/index.htm By Nobel Khandaker Neural Networks An

### Self Organizing Maps: Fundamentals

Self Organizing Maps: Fundamentals Introduction to Neural Networks : Lecture 16 John A. Bullinaria, 2004 1. What is a Self Organizing Map? 2. Topographic Maps 3. Setting up a Self Organizing Map 4. Kohonen

### Neural Networks. Introduction to Artificial Intelligence CSE 150 May 29, 2007

Neural Networks Introduction to Artificial Intelligence CSE 150 May 29, 2007 Administration Last programming assignment has been posted! Final Exam: Tuesday, June 12, 11:30-2:30 Last Lecture Naïve Bayes

### Chapter 12 Discovering New Knowledge Data Mining

Chapter 12 Discovering New Knowledge Data Mining Becerra-Fernandez, et al. -- Knowledge Management 1/e -- 2004 Prentice Hall Additional material 2007 Dekai Wu Chapter Objectives Introduce the student to

### These slides follow closely the (English) course textbook Pattern Recognition and Machine Learning by Christopher Bishop

Music and Machine Learning (IFT6080 Winter 08) Prof. Douglas Eck, Université de Montréal These slides follow closely the (English) course textbook Pattern Recognition and Machine Learning by Christopher

### It can be shown that by replacing the Sigmoid activation function often

PROBABALISTIC NEURAL NETWORKS FOR CLASSIFICATION, MAPPING, OR ASSOCIATIVE MEMORY Donald F. Specht Lockheed Palo Alto Research Laboratories 3251 Hanover St. Palo Alto, California 94304 Abs tract It can

### Modular Neural Networks

16 Modular Neural Networks In the previous chapters we have discussed different models of neural networks linear, recurrent, supervised, unsupervised, self-organizing, etc. Each kind of network relies

### Learning. Artificial Intelligence. Learning. Types of Learning. Inductive Learning Method. Inductive Learning. Learning.

Learning Learning is essential for unknown environments, i.e., when designer lacks omniscience Artificial Intelligence Learning Chapter 8 Learning is useful as a system construction method, i.e., expose

### Artificial Neural Networks and Support Vector Machines. CS 486/686: Introduction to Artificial Intelligence

Artificial Neural Networks and Support Vector Machines CS 486/686: Introduction to Artificial Intelligence 1 Outline What is a Neural Network? - Perceptron learners - Multi-layer networks What is a Support

### Impact of Feature Selection on the Performance of Wireless Intrusion Detection Systems

2009 International Conference on Computer Engineering and Applications IPCSIT vol.2 (2011) (2011) IACSIT Press, Singapore Impact of Feature Selection on the Performance of ireless Intrusion Detection Systems

### INTELLIGENT ENERGY MANAGEMENT OF ELECTRICAL POWER SYSTEMS WITH DISTRIBUTED FEEDING ON THE BASIS OF FORECASTS OF DEMAND AND GENERATION Chr.

INTELLIGENT ENERGY MANAGEMENT OF ELECTRICAL POWER SYSTEMS WITH DISTRIBUTED FEEDING ON THE BASIS OF FORECASTS OF DEMAND AND GENERATION Chr. Meisenbach M. Hable G. Winkler P. Meier Technology, Laboratory

### Linear Threshold Units

Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear

### Automated Stellar Classification for Large Surveys with EKF and RBF Neural Networks

Chin. J. Astron. Astrophys. Vol. 5 (2005), No. 2, 203 210 (http:/www.chjaa.org) Chinese Journal of Astronomy and Astrophysics Automated Stellar Classification for Large Surveys with EKF and RBF Neural

### PMR5406 Redes Neurais e Lógica Fuzzy Aula 3 Multilayer Percetrons

PMR5406 Redes Neurais e Aula 3 Multilayer Percetrons Baseado em: Neural Networks, Simon Haykin, Prentice-Hall, 2 nd edition Slides do curso por Elena Marchiori, Vrie Unviersity Multilayer Perceptrons Architecture

### 1 Maximum likelihood estimation

COS 424: Interacting with Data Lecturer: David Blei Lecture #4 Scribes: Wei Ho, Michael Ye February 14, 2008 1 Maximum likelihood estimation 1.1 MLE of a Bernoulli random variable (coin flips) Given N

### Feed-Forward mapping networks KAIST 바이오및뇌공학과 정재승

Feed-Forward mapping networks KAIST 바이오및뇌공학과 정재승 How much energy do we need for brain functions? Information processing: Trade-off between energy consumption and wiring cost Trade-off between energy consumption

### SELECTING NEURAL NETWORK ARCHITECTURE FOR INVESTMENT PROFITABILITY PREDICTIONS

UDC: 004.8 Original scientific paper SELECTING NEURAL NETWORK ARCHITECTURE FOR INVESTMENT PROFITABILITY PREDICTIONS Tonimir Kišasondi, Alen Lovren i University of Zagreb, Faculty of Organization and Informatics,

### Maximum Likelihood Estimation

Math 541: Statistical Theory II Lecturer: Songfeng Zheng Maximum Likelihood Estimation 1 Maximum Likelihood Estimation Maximum likelihood is a relatively simple method of constructing an estimator for

### 3 An Illustrative Example

Objectives An Illustrative Example Objectives - Theory and Examples -2 Problem Statement -2 Perceptron - Two-Input Case -4 Pattern Recognition Example -5 Hamming Network -8 Feedforward Layer -8 Recurrent

### Introduction to Neural Networks : Revision Lectures

Introduction to Neural Networks : Revision Lectures John A. Bullinaria, 2004 1. Module Aims and Learning Outcomes 2. Biological and Artificial Neural Networks 3. Training Methods for Multi Layer Perceptrons

### (Refer Slide Time: 01:11-01:27)

Digital Signal Processing Prof. S. C. Dutta Roy Department of Electrical Engineering Indian Institute of Technology, Delhi Lecture - 6 Digital systems (contd.); inverse systems, stability, FIR and IIR,

### Lecture 6. Artificial Neural Networks

Lecture 6 Artificial Neural Networks 1 1 Artificial Neural Networks In this note we provide an overview of the key concepts that have led to the emergence of Artificial Neural Networks as a major paradigm

### CHAPTER 6 NEURAL NETWORK BASED SURFACE ROUGHNESS ESTIMATION

CHAPTER 6 NEURAL NETWORK BASED SURFACE ROUGHNESS ESTIMATION 6.1. KNOWLEDGE REPRESENTATION The function of any representation scheme is to capture the essential features of a problem domain and make that

### Neural Networks and Back Propagation Algorithm

Neural Networks and Back Propagation Algorithm Mirza Cilimkovic Institute of Technology Blanchardstown Blanchardstown Road North Dublin 15 Ireland mirzac@gmail.com Abstract Neural Networks (NN) are important

### Introduction to Neural Networks for Senior Design

Introduction to Neural Networks for Senior Design Intro-1 Neural Networks: The Big Picture Artificial Intelligence Neural Networks Expert Systems Machine Learning not ruleoriented ruleoriented Intro-2

### Research of Digital Character Recognition Technology Based on BP Algorithm

Research of Digital Character Recognition Technology Based on BP Algorithm Xianmin Wei Computer and Communication Engineering School of Weifang University Weifang, China wfxyweixm@126.com Abstract. This

### Role of Neural network in data mining

Role of Neural network in data mining Chitranjanjit kaur Associate Prof Guru Nanak College, Sukhchainana Phagwara,(GNDU) Punjab, India Pooja kapoor Associate Prof Swami Sarvanand Group Of Institutes Dinanagar(PTU)

### Programming Exercise 3: Multi-class Classification and Neural Networks

Programming Exercise 3: Multi-class Classification and Neural Networks Machine Learning November 4, 2011 Introduction In this exercise, you will implement one-vs-all logistic regression and neural networks

### Biological Neurons and Neural Networks, Artificial Neurons

Biological Neurons and Neural Networks, Artificial Neurons Neural Computation : Lecture 2 John A. Bullinaria, 2015 1. Organization of the Nervous System and Brain 2. Brains versus Computers: Some Numbers

### Temporal Difference Learning in the Tetris Game

Temporal Difference Learning in the Tetris Game Hans Pirnay, Slava Arabagi February 6, 2009 1 Introduction Learning to play the game Tetris has been a common challenge on a few past machine learning competitions.

### Machine Learning and Pattern Recognition Logistic Regression

Machine Learning and Pattern Recognition Logistic Regression Course Lecturer:Amos J Storkey Institute for Adaptive and Neural Computation School of Informatics University of Edinburgh Crichton Street,

### Lecture 3: Linear methods for classification

Lecture 3: Linear methods for classification Rafael A. Irizarry and Hector Corrada Bravo February, 2010 Today we describe four specific algorithms useful for classification problems: linear regression,

### Chapter 4: Non-Parametric Classification

Chapter 4: Non-Parametric Classification Introduction Density Estimation Parzen Windows Kn-Nearest Neighbor Density Estimation K-Nearest Neighbor (KNN) Decision Rule Gaussian Mixture Model A weighted combination

### Classification Problems

Classification Read Chapter 4 in the text by Bishop, except omit Sections 4.1.6, 4.1.7, 4.2.4, 4.3.3, 4.3.5, 4.3.6, 4.4, and 4.5. Also, review sections 1.5.1, 1.5.2, 1.5.3, and 1.5.4. Classification Problems

### Chapter 2 The Research on Fault Diagnosis of Building Electrical System Based on RBF Neural Network

Chapter 2 The Research on Fault Diagnosis of Building Electrical System Based on RBF Neural Network Qian Wu, Yahui Wang, Long Zhang and Li Shen Abstract Building electrical system fault diagnosis is the

### Prediction Model for Crude Oil Price Using Artificial Neural Networks

Applied Mathematical Sciences, Vol. 8, 2014, no. 80, 3953-3965 HIKARI Ltd, www.m-hikari.com http://dx.doi.org/10.12988/ams.2014.43193 Prediction Model for Crude Oil Price Using Artificial Neural Networks

### Recurrent Neural Networks

Recurrent Neural Networks Neural Computation : Lecture 12 John A. Bullinaria, 2015 1. Recurrent Neural Network Architectures 2. State Space Models and Dynamical Systems 3. Backpropagation Through Time

### A Simple Feature Extraction Technique of a Pattern By Hopfield Network

A Simple Feature Extraction Technique of a Pattern By Hopfield Network A.Nag!, S. Biswas *, D. Sarkar *, P.P. Sarkar *, B. Gupta **! Academy of Technology, Hoogly - 722 *USIC, University of Kalyani, Kalyani

### SPEAKER VERIFICATION BASED ON SPEAKER P OSITION IN A MULTIDIMENSIONAL SPEAKER IDENTIFICATION SPACE

SPEAKER VERIFICATION BASED ON SPEAKER P OSITION IN A MULTIDIMENSIONAL SPEAKER IDENTIFICATION SPACE CLAUDE A. NORTON, III AND STEPHEN A. ZAHORIAN Department of Electrical and Computer Engineering Old Dominion

### 1. Classification problems

Neural and Evolutionary Computing. Lab 1: Classification problems Machine Learning test data repository Weka data mining platform Introduction Scilab 1. Classification problems The main aim of a classification

### Neural Networks and Support Vector Machines

INF5390 - Kunstig intelligens Neural Networks and Support Vector Machines Roar Fjellheim INF5390-13 Neural Networks and SVM 1 Outline Neural networks Perceptrons Neural networks Support vector machines

### ANN Based Fault Classifier and Fault Locator for Double Circuit Transmission Line

International Journal of Computer Sciences and Engineering Open Access Research Paper Volume-4, Special Issue-2, April 2016 E-ISSN: 2347-2693 ANN Based Fault Classifier and Fault Locator for Double Circuit

### Neural Networks. Neural network is a network or circuit of neurons. Neurons can be. Biological neurons Artificial neurons

Neural Networks Neural network is a network or circuit of neurons Neurons can be Biological neurons Artificial neurons Biological neurons Building block of the brain Human brain contains over 10 billion

### /SOLUTIONS/ where a, b, c and d are positive constants. Study the stability of the equilibria of this system based on linearization.

echnische Universiteit Eindhoven Faculteit Elektrotechniek NIE-LINEAIRE SYSEMEN / NEURALE NEWERKEN (P6) gehouden op donderdag maart 7, van 9: tot : uur. Dit examenonderdeel bestaat uit 8 opgaven. /SOLUIONS/

### SUCCESSFUL PREDICTION OF HORSE RACING RESULTS USING A NEURAL NETWORK

SUCCESSFUL PREDICTION OF HORSE RACING RESULTS USING A NEURAL NETWORK N M Allinson and D Merritt 1 Introduction This contribution has two main sections. The first discusses some aspects of multilayer perceptrons,

### Application of Neural Network in User Authentication for Smart Home System

Application of Neural Network in User Authentication for Smart Home System A. Joseph, D.B.L. Bong, D.A.A. Mat Abstract Security has been an important issue and concern in the smart home systems. Smart

### A TUTORIAL. BY: Negin Yousefpour PhD Student Civil Engineering Department TEXAS A&M UNIVERSITY

ARTIFICIAL NEURAL NETWORKS: A TUTORIAL BY: Negin Yousefpour PhD Student Civil Engineering Department TEXAS A&M UNIVERSITY Contents Introduction Origin Of Neural Network Biological Neural Networks ANN Overview

### Power Prediction Analysis using Artificial Neural Network in MS Excel

Power Prediction Analysis using Artificial Neural Network in MS Excel NURHASHINMAH MAHAMAD, MUHAMAD KAMAL B. MOHAMMED AMIN Electronic System Engineering Department Malaysia Japan International Institute

### Impelling Heart Attack Prediction System using Data Mining and Artificial Neural Network

General Article International Journal of Current Engineering and Technology E-ISSN 2277 4106, P-ISSN 2347-5161 2014 INPRESSCO, All Rights Reserved Available at http://inpressco.com/category/ijcet Impelling

### University of Cambridge Engineering Part IIB Module 4F10: Statistical Pattern Processing Handout 8: Multi-Layer Perceptrons

University of Cambridge Engineering Part IIB Module 4F0: Statistical Pattern Processing Handout 8: Multi-Layer Perceptrons x y (x) Inputs x 2 y (x) 2 Outputs x d First layer Second Output layer layer y

### 1 SELF-ORGANIZATION MECHANISM IN THE NETWORKS

Mathematical literature reveals that the number of neural network structures, concepts, methods, and their applications have been well known in neural modeling literature for sometime. It started with

### Lecture 1: Introduction to Neural Networks Kevin Swingler / Bruce Graham

Lecture 1: Introduction to Neural Networks Kevin Swingler / Bruce Graham kms@cs.stir.ac.uk 1 What are Neural Networks? Neural Networks are networks of neurons, for example, as found in real (i.e. biological)

### Gasses at High Temperatures

Gasses at High Temperatures The Maxwell Speed Distribution for Relativistic Speeds Marcel Haas, 333 Summary In this article we consider the Maxwell Speed Distribution (MSD) for gasses at very high temperatures.

### CS 2750 Machine Learning. Lecture 1. Machine Learning. http://www.cs.pitt.edu/~milos/courses/cs2750/ CS 2750 Machine Learning.

Lecture Machine Learning Milos Hauskrecht milos@cs.pitt.edu 539 Sennott Square, x5 http://www.cs.pitt.edu/~milos/courses/cs75/ Administration Instructor: Milos Hauskrecht milos@cs.pitt.edu 539 Sennott

### x 2 would be a solution to d y

CATHOLIC JUNIOR COLLEGE H MATHEMATICS JC PRELIMINARY EXAMINATION PAPER I 0 System of Linear Equations Assessment Objectives Solution Feedback To use a system of linear c equations to model and solve y

### Neural Computation - Assignment

Neural Computation - Assignment Analysing a Neural Network trained by Backpropagation AA SSt t aa t i iss i t i icc aa l l AA nn aa l lyy l ss i iss i oo f vv aa r i ioo i uu ss l lee l aa r nn i inn gg

### Logistic Regression. Jia Li. Department of Statistics The Pennsylvania State University. Logistic Regression

Logistic Regression Department of Statistics The Pennsylvania State University Email: jiali@stat.psu.edu Logistic Regression Preserve linear classification boundaries. By the Bayes rule: Ĝ(x) = arg max

### A Content based Spam Filtering Using Optical Back Propagation Technique

A Content based Spam Filtering Using Optical Back Propagation Technique Sarab M. Hameed 1, Noor Alhuda J. Mohammed 2 Department of Computer Science, College of Science, University of Baghdad - Iraq ABSTRACT

### Models of Cortical Maps II

CN510: Principles and Methods of Cognitive and Neural Modeling Models of Cortical Maps II Lecture 19 Instructor: Anatoli Gorchetchnikov dy dt The Network of Grossberg (1976) Ay B y f (

### Principles of training multi-layer neural network using backpropagation algorithm

Page 1 Students' Project: "Neural networks and artificial inteligence in electronics" Teacher: mgr inż. Adam Gołda Authors: Mariusz Bernacki Przemysław Włodarczyk Electronics V ( 2005) Principles of training

### 3. Regression & Exponential Smoothing

3. Regression & Exponential Smoothing 3.1 Forecasting a Single Time Series Two main approaches are traditionally used to model a single time series z 1, z 2,..., z n 1. Models the observation z t as a

### Some stability results of parameter identification in a jump diffusion model

Some stability results of parameter identification in a jump diffusion model D. Düvelmeyer Technische Universität Chemnitz, Fakultät für Mathematik, 09107 Chemnitz, Germany Abstract In this paper we discuss

### CSCI567 Machine Learning (Fall 2014)

CSCI567 Machine Learning (Fall 2014) Drs. Sha & Liu {feisha,yanliu.cs}@usc.edu September 22, 2014 Drs. Sha & Liu ({feisha,yanliu.cs}@usc.edu) CSCI567 Machine Learning (Fall 2014) September 22, 2014 1 /

### A simple application of Artificial Neural Network to cloud classification

A simple application of Artificial Neural Network to cloud classification Tianle Yuan For AOSC 630 (by Prof. Kalnay) Introduction to Pattern Recognition (PR) Example1: visual separation between the character

### PREDICTION OF RHEOLOGICAL PROPERTIES OF MORTAR FOR SHOTCRETE USING NEURAL NETWORK

PREDICTION OF RHEOLOGICAL PROPERTIES OF MORTAR FOR SHOTCRETE USING NEURAL NETWORK Quoc Huu Duy PHAN* 1, Koji BABA* 2 and Taketo UOMOTO* 3 ABSTRACT: Rheological properties of shotcrete mortar largely affect

### Stock Market Prediction System with Modular Neural Networks

Stock Market Prediction System with Modular Neural Networks Takashi Kimoto and Kazuo Asakawa Computer-based Systems Laboratory FUJTSU LABORATORES LTD., KAWASAK 1015 Kamikodanaka, Nakahara-Ku, Kawasaki

### Monotonicity Hints. Abstract

Monotonicity Hints Joseph Sill Computation and Neural Systems program California Institute of Technology email: joe@cs.caltech.edu Yaser S. Abu-Mostafa EE and CS Deptartments California Institute of Technology

### Neural Networks algorithms and applications

Neural Networks algorithms and applications By Fiona Nielsen 4i 12/12-2001 Supervisor: Geert Rasmussen Niels Brock Business College 1 Introduction Neural Networks is a field of Artificial Intelligence

### 13.4 THE CROSS PRODUCT

710 Chapter Thirteen A FUNDAMENTAL TOOL: VECTORS 62. Use the following steps and the results of Problems 59 60 to show (without trigonometry) that the geometric and algebraic definitions of the dot product

### Mechanics 1: Conservation of Energy and Momentum

Mechanics : Conservation of Energy and Momentum If a certain quantity associated with a system does not change in time. We say that it is conserved, and the system possesses a conservation law. Conservation

### Introduction to Artificial Neural Networks MAE-491/591

Introduction to Artificial Neural Networks MAE-491/591 Artificial Neural Networks: Biological Inspiration The brain has been extensively studied by scientists. Vast complexity prevents all but rudimentary

### Data Mining using Artificial Neural Network Rules

Data Mining using Artificial Neural Network Rules Pushkar Shinde MCOERC, Nasik Abstract - Diabetes patients are increasing in number so it is necessary to predict, treat and diagnose the disease. Data

### CS 688 Pattern Recognition Lecture 4. Linear Models for Classification

CS 688 Pattern Recognition Lecture 4 Linear Models for Classification Probabilistic generative models Probabilistic discriminative models 1 Generative Approach ( x ) p C k p( C k ) Ck p ( ) ( x Ck ) p(

### Open Access Research on Application of Neural Network in Computer Network Security Evaluation. Shujuan Jin *

Send Orders for Reprints to reprints@benthamscience.ae 766 The Open Electrical & Electronic Engineering Journal, 2014, 8, 766-771 Open Access Research on Application of Neural Network in Computer Network

### The Role of Size Normalization on the Recognition Rate of Handwritten Numerals

The Role of Size Normalization on the Recognition Rate of Handwritten Numerals Chun Lei He, Ping Zhang, Jianxiong Dong, Ching Y. Suen, Tien D. Bui Centre for Pattern Recognition and Machine Intelligence,

### Neural network software tool development: exploring programming language options

INEB- PSI Technical Report 2006-1 Neural network software tool development: exploring programming language options Alexandra Oliveira aao@fe.up.pt Supervisor: Professor Joaquim Marques de Sá June 2006

### Introduction to Artificial Neural Networks

POLYTECHNIC UNIVERSITY Department of Computer and Information Science Introduction to Artificial Neural Networks K. Ming Leung Abstract: A computing paradigm known as artificial neural network is introduced.

### Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur

Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur Module No. #01 Lecture No. #15 Special Distributions-VI Today, I am going to introduce

### jorge s. marques image processing

image processing images images: what are they? what is shown in this image? What is this? what is an image images describe the evolution of physical variables (intensity, color, reflectance, condutivity)

### Forecasting Trade Direction and Size of Future Contracts Using Deep Belief Network

Forecasting Trade Direction and Size of Future Contracts Using Deep Belief Network Anthony Lai (aslai), MK Li (lilemon), Foon Wang Pong (ppong) Abstract Algorithmic trading, high frequency trading (HFT)

### Finite Element Analysis Prof. Dr. B. N. Rao Department of Civil Engineering Indian Institute of Technology, Madras. Lecture - 28

Finite Element Analysis Prof. Dr. B. N. Rao Department of Civil Engineering Indian Institute of Technology, Madras Lecture - 28 In the earlier lectures, we have seen formulation for 3 node linear triangular

### Neural networks. Chapter 20, Section 5 1

Neural networks Chapter 20, Section 5 Chapter 20, Section 5 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 20, Section 5 2 Brains 0 neurons of

### NEUROMATHEMATICS: DEVELOPMENT TENDENCIES. 1. Which tasks are adequate of neurocomputers?

Appl. Comput. Math. 2 (2003), no. 1, pp. 57-64 NEUROMATHEMATICS: DEVELOPMENT TENDENCIES GALUSHKIN A.I., KOROBKOVA. S.V., KAZANTSEV P.A. Abstract. This article is the summary of a set of Russian scientists

### NEURAL NETWORKS IN DATA MINING

NEURAL NETWORKS IN DATA MINING 1 DR. YASHPAL SINGH, 2 ALOK SINGH CHAUHAN 1 Reader, Bundelkhand Institute of Engineering & Technology, Jhansi, India 2 Lecturer, United Institute of Management, Allahabad,