Neural Networks (Chapter 9)

Transcription

1 Neural Networks (Chapter 9) Kai Goebel, Bill Cheetham GE Corporate Research & Development Outline Introduction Categories Hopfield Net Perceptron Single Layer Multi Layer Page

2 3 Introduction Human brain is superior to digital computer at many tasks + e.g., processing of visual information + robust and fault tolerant (nerve cells in the brain die every day) + flexible; adjusts to new environment + can deal with information that is sparse, imprecise, noisy, inconsistent + highly parallel + small, compact, dissipates very little power - slower in primarily (simple) arithmetic operations Neurons McCulloch & Pitts (943) - simple model of neuron as a binary threshold unit - uses step function to fire when threshold µ is surpassed x w 4 x x3 w Σ w3 µ Page

3 Real Neurons Real Neurons - use not even approximately threshold devices - it is assumed they use a non-linear summation method - produce a sequence of pulses (not a single output level) - do not have the same fixed delay (t-> t+) - are not updated synchronously - amount of transmitter substance varies unpredictably 5 Issues 6 What does that leave us with? What is the best architecture? (layers, connections, activation functions, updating, # units?) How can it be programmed? (can it learn, # examples needed, time to learn, amount of supervision, real-time learning) What can it do? (how many tasks, how well, how fast, how robust, level of generalization) Page 3 3

4 Neural Nets: Categorization 7 Supervised Learning Multilayer perceptrons Radial basis function networks Modular neural networks LVQ (learning vector quantization) Reinforcement Learning Temporal Difference Learning Q-Learning Unsupervised Learning Competitive learning networks Kohonen self-organizing networks ART (adaptive resonant theory) Supervised Neural Networks Requirement: known input-output relations input pattern output 8 Page 4 4

5 Hopfield Model 9 Associative Memory is considered the fruit fly of this field. It illustrates in the simplest possible manner the way that collective computation can work. Store a set of patterns in such a way that when presented with a new pattern, the network responds by producing the closest stored pattern. Conventional approach: store a list of patterns, compute the Hamming distance, find the smallest, et voila! Hopfield Network Operation Picture is pattern; stored as attractor in the configuration space. From arbitrary starting points, one attractor will be found Page 5 5

6 Hopfield Network Operation Picture is pattern; stored as attractor in the configuration space. From arbitrary starting points, one attractor will be found Hopfield Architecture - Recurrent Network - Symmetric Architecture - Evaluation until no more changes are observed i.e., network settles into local minimum config. w ij local minimum corresponds to energy function E = wij xi x j i, j Page 6 6

7 Hopfield Network Equations The operative equation, i.e., the network output at each step is where y = sgn w x i ij j i ( ) sgn x = if x - if x < 3 Learning in Hopfield Models Learning Rule: ( n+ ) ( n) ij ij w = w + x x w ii = i j 4 Page 7 7

8 5 Hopfield Example Learn x=[ - -] which gives us the weight matrix w=[ ] Now let s check the slightly corrupted pattern p=[ - ] which will restore the pattern found close y=[ - -] with an energy level of E=-6 w 4 =- w 3 =- w =- w 34 =- w ij 3 w 3 =- 4 w 4 =- 6 Hopfield Example Learn second pattern x=[- - ] which gives us the new weight matrix w=[ ] Now let s check the slightly corrupted pattern p=[- - - ] which will restore the pattern y=[- - ] with an energy level of E=- Page 8 8

9 More Complex Hopfield Examples Reconstruction of Images 7 binary images are 3x8 pixels Hopfield Book Example Character Recognition eight exemplar patterns 8 output pattern for noisy 3 input Page 9 9

10 Hopfield: Issues - Other memories can get lost - Memories are created that were not supposed to be there - crosstalk: if there are many memories, they might interfere - no emphasis on learning; rather handcrafting to get desired properties - goes towards optimization 9 Perceptrons -Rosenblatt: 95s -Input patterns represented is binary -Single layer network can be trained easily -Output o is computed by n o = f w i x i θ i= where w i is a (modifiable) weight x i is the input signal θ is some threshold (weight of constant input) f(. ) is the activation function f ( x) = sgn( x) = if x > otherwise Page

11 Single-Layer Perceptrons Network architecture x w w x x3 w w3 y = signum(σwi xi + w) Single-Layer Perceptron Example: Gender classification (according to Jang) Network Arch. Training data h v w w w y = signum(hw+vw+w) y v (voice freq.) = - if female if male h (hair length) Page

12 Perceptron Learning: select an input vector if the response is incorrect, modify all weights w = η t x i i i where t i is a target output η is the learning rate 3 If a set of weights for converged state exists, then a method for tuning towards convergence exists (Rosenblatt, 96) ADALINE single layer network (conceived by Widrow and Hoff) output is weighted linear combination of weights n i i i= o = w x w error is described as ( ) E = t o p p p (for pattern p) 4 where t p is the target output o p is the actual output Page

13 ADALINE To decrease the error, the derivative wrt the weights is taken E p w i The delta rule is: η ( ) = t o x p p i ( ) p w i = t p o p x i Intuitive appeal: 5 if t p >o p, boost o p by increasing w i x i increase w i if x i is positive decrease w i is x i is negative ADALINE and MADALINE + Simplicity of learning procedure + Distributed learning; can be performed locally at node level + on-line (pattern by pattern) learning + connect several ADALINEs to MADALINEs to deal with XOR problem + were used for noise cancellation, adaptive inverse control - only one layer; no suitable training method for multi-layer perceptron why? 6 Page 3 3

14 XOR Minsky and Papert reported a severe shortcoming of single layer perceptrons, the XOR problem x x output x O O not linearly separable x 7 XOR 8 Minsky and Papert reported a severe shortcoming of single layer perceptrons, the XOR problem x x output w + w w w w + w + w + w + w w + w > w + w > w + w w > w > w + w x O O not linearly separable w x Page 4 4

15 Enter the Dark Ages of NNs which (together with a lack or proper training techniques for multi-layer perceptrons) all but killed interest in neural nets in the 7s and early 8s. 9 3 Page 5 5

16 Multilayer Perceptrons Two-layer perceptron x x x x x5 3 Two-Layer Perceptron: XOR Node output as surface of their two inputs.5 x x + x x x 5 3 note location of o and x Page 6 6

17 Multilayer Perceptrons (MLPs) Network architecture x y x y hyperbolic tangent or logistic function Steepest descent (Backprop) Conjugate gradient method All optim. methods using first derivative Derivative-free optim. Learning rule: 33 Multi-Layer Perceptrons -Recall the output n o = f w i x i θ i= -and the squared error measure n p p p E E = t o k = which is amended to ( tk ok ) ( ) p p p -and the activation function f ( x) = + e x or f ( x) k = x e x = x = tanh or f ( x) = x + e 34 then the learning rule for each node can be derived using the chain rule... Page 7 7

18 y 9/9/3 Multi-Layer Perceptrons -Recall the output n o = f w i x i θ i= -and the squared error measure n p p p E E = t o k = which is amended to ( tk ok ) ( ) p p p -and the activation function f ( x) = + - x e x or f ( x) squashing functions k = x e x = x = tanh or f ( x) = x + e Backpropagation make incremental change in the direction de/dw to decrease the error. The learning rule for each node can be derived using the chain rule... to propagate the error back through a multilayer perceptron. E parameters 36 w ki E = η w p p ki error Page 8 8

19 Back-prop procedure 37. Initialize weights to small random values. Choose a pattern and apply it to input layer 3. Propagate the signal forward through the network 4. Compute the deltas for the output layer 5. Compute the deltas for the preceding layers by propagating the error backwards 6. Update all weights 7. Go back to step and repeat for next pattern 8. Repeat until error rate is acceptable Step. Initialize weights to small random values Example: Weight Value x x = O = O w w w w O O w w 3 O w -.4 w -.3 Threshold Value 38 w w w w t t t Page 9 9

20 Step. Choose a pattern and apply it to input layer x x Target output 39 x x Step 3 Propagate signal forward through the network m m Oi = g wij O j j until all outputs have been calculated For m= (input layer), the output is the pattern. Example: w =-.4 = O = w =-.3 = O = t =. m O = /(+exp(-*β*(*(-.4)+*(-.3)+.)) 4 Page

21 Step 4 Compute the deltas for the output layer M i δ = g' M p m ( h )[ d O ] i i i by comparing the actual output O with the target output t for the pattern p considered Example: x = O t O 3 =-. =.3 4 x = O h =.745 δ m i = f ' m p m ( h ) ( d O ) i i i Step 5 Compute the deltas for the preceding layers by propagating the errors backwards δ m i = g' n m ( hi ) j w m m ij δ j for m=m, M-, M-, until a delta has been calculated for every unit 4 Page

22 Step 6 Use w m ij m i = ηδ O m j to update all connections to w new ij = w old ij + w ij 43 Step Size, Initial Weights 44 Step size: too big too small variable: error compute error backpropagate compute error again if error bigger, reduce step size (.5) otherwise, increase a little (.) Initial weights: randomize (= ) Page

23 Momentum If error minimum in long narrow valley, then updating can happen to zig-zag down the valley 45 smoothes weight updating can speed learning up = η w + α w E w prev Overfitting Error on learning cases Error on validation cases epoch error error 46 epoch trained things that are accidental and unimportant Page 3 3

24 Local Minima There is no guarantee that the algorithm converges to a global minimum error epoch 47 - check with different initial conditions (different weights, etc.) - perturb the system (data) with noise to improve result Architectures and other Techniques 48 Normalize weights move weights same Euclidean distance each epoch Data scaling Input scaling: allows weights to have same order of magnitude Output scaling: let target go between +-.9 to avoid saturation What number of nodes per layer? How many layers? Page 4 4

25 MLP Decision Boundaries XOR Interwined General -layer: Half planes A B B A A B -layer: Convex A B B A A B 3-layer: Arbitrary A B B A A B 49 Radial Basis Function (RBF) Networks Network architecture x x Σ Σ y y Each node is described by a bell shaped function x c i oi = exp σ i 5 where c i is the center of the curve Page 5 5

26 RBF Output: weighted sum weighted average linear combination Location of Center: Use (fuzzy) k-means clustering Size of Variance: Use knn-classifier and take average distance 5 XOR, revisited x y output 5 Page 6 6

27 RBF and FIS Consider the radial basis functions:. c c c 3 x 53 and a linear combination of the output variables y = a o + b i i i then the response is equivalent to... Modular Networks Task decomposition Local Experts Fuse information local expert O Input x local expert... O weigh experts opinions by g i fuse y local expert n O n 54 Page 7 7

28 last slide 55 Page 8 8