Rate-based artificial neural networks and error backpropagation learning. Scott Murdison Machine learning journal club May 16, 2016

Transcription

1 Rate-based artificial neural networks and error backpropagation learning Scott Murdison Machine learning journal club May 16, 2016 Murdison, Leclercq, Lefèvre and Blohm J Neurophys 2015

2 Neural networks??? Why would we use those?! Some problems are just too complicated (i.e. nonlinear) to solve with simple programming Computer vision/audition applications: - digit recognition - facial recognition - reading - speech recognition - object recognition Others: - detection of medical disorders - industrial process control - stock market predictions - the list goes on Could theoretically write a program to solve each of these problems, but the rules governing such a program would be incredibly complicated! Geoff Hinton sweet intro vid 2

3 Neural networks??? Why would we use those?! Some problems are just too complicated to solve with simple programming Computer vision/audition applications: - facial recognition Paul Debevec. A Neural Network for Facial Feature Location. UC Berkeley CS283 Project Report, December

4 Neural networks??? Why would we use those?! Some problems are just too complicated to solve with simple programming Computer vision/audition applications: - facial recognition manually located left eye, nose and mh polar-transformed images around each feature Paul Debevec. A Neural Network for Facial Feature Location. UC Berkeley CS283 Project Report, December subsampled images to feed to neural network A. Avoids local minima Why? B. 4MB RAM in 1992 = $150 USD 4

5 Neural networks??? Why would we use those?! Some problems are just too complicated to solve with simple programming Computer vision/audition applications: - facial recognition After training a simple, feedforward network with backprop using yearbook photos it could successfully detect each eye, nose and mh in previously unseen photo! Paul Debevec. A Neural Network for Facial Feature Location. UC Berkeley CS283 Project Report, December

6 The choice for rate-based over spiking for machine learning spiking rate-based Pros neuron-level resolution time-resolved (spiking dynamics, variability) several levels of abstraction available (H-H, leaky integrate-and-fire, Izhikevich) Cons computationally expensive for large populations of neurons intractable for simulations of whole brain areas not obviously useful for machine learning Pros describes average firing of functional populations of neurons computationally cheap can address network structure/function mathematical simplicity Universal function approximator Cons lose benefits of spiking spatiotemporal resolution averages over timing/dynamics/variability/ etc. biological analogs not always obvious 6

7 General architecture Simplest network is the perceptron r1 in r2 in w1 w2 g r Transfer function g( ) general engineering term used to describe put of some processing unit as a function of input in if r i = x i and r = y then y = g(w 1 x 1 + w 2 x 2 ) and if g( ) = (i.e. g is purely linear) then y = w1 x1 + w2 x2 General single-layer mapping r i = g( w ij r in j ) + r = g(wr in ) i 7

8 r1 in The XOR problem w1 XOR x1 x2 y x2 r2 in w2 Boolean OR x1 x2 y x2 g r x not linearly separable: more complicated! x1 linearly separable: can be easily separated with single threshold! g(x)= G 1 if x > H 0 else 8

9 r1 in The XOR problem w1 XOR x1 x2 y x1 r2 in w2 g r becomes not linearly separable: more complicated! x2 With enough hidden units, multilayer perceptron is the universal function approximator! 9

10 The XOR problem r1 in w1 g r r2 in w2 becomes With enough hidden units, multilayer perceptron is the universal function approximator! 10

11 Multi-layer perceptron HLU1 HLU2 put HLU3 E.g. sine wave approximation using logistic transfer function With enough hidden units, multilayer perceptron is the universal function approximator! 11

12 Multi-layer perceptron Limitations - Brain-like performance doesn t equate with actual performance - Training rules are non-physiologic Strengths - Hidden layer activity might resemble brain function (with appropriate inputs/puts) - Brain = mapping network - Self-Organization, like the brain - High flexibility in possible computations Point: MLP is usually good for machine learning purposes, but is not necessarily good for neuroscience theory all the time! 12

13 Two types of NNets feedforward rate-based recurrent Feed-forward: information only flows forward, simplest connectivity - Kernel machines - Radial basis function networks - Probabilistic mapping networks - Bayesian nets - Stochastic nets - Modular nets - Committee machines - Associative neural nets - Holographic associative nets (???) - Fuzzy nets - Compositional pattern-producing nets - etc. Recurrent: information can flow forward and backward, useful for time-resolved problems 13

14 How do we train a neural network? Minimize some cost function gradient descent! r1 in w1 MSE g r 1 E = 2 r2 in w2 i (r i - y i ) 2 desired put (data, training set) w ij! w ij + Dw ij 2E Dw ij = -f( ) 2wij learning gradient of MSE rate wrt weights 14

15 How do we train a neural network? r1 in r2 in w1 w2 g r E = 2 1 i (r i - y i ) 2 w ij! w ij + Dw ij 2E Dw ij = -f( ) 2wij 2E 1 2 2w = ij 2 2wij i j in (g( w ij r j ) - y i ) hi f change in weight i,j depends on learning rate and dependence of error on weight change at i,j 2f 2w ij 2f = 2g 2g 2h (using chain rule) 2h 2w ij error s dependence on weight i,j, rewritten using MSE equation Dw ij = f(g'(h i )(y i - r i )r j in ) learning rule (derivation?) 15

16 Adding layers Start by finding the put rates 2-layer (1 hidden) perceptron r i = g( w r ) r = g( w r ) j h ij h j E = 2 1 i (r i - y i ) 2 Dw ij = f(g'(h i )(y i - r i )r j in ) h h in r = g ( w g ( w r )) 3-layer (2 hidden) perceptron h h h h in r = g ( w g -1( w -1g -2( w -2r ))) n-layer (n-1 hidden) perceptron h h h h h h h h in r = g ( w g -1( w -1g -2( w g - n+ 1( w - n+ 1g -n( w -nr ))))) Idea is to nest each layer s put rates within the next 16

17 Training through the layers Generalized delta rule (put wts) 2E 2wij with d i 2E 2w 2E 2w with d h i 1 2 = 2 ( ri y) w - i 2 ij i h = di r j = g '( h h i)( r i -yi) h ij h ij = E = ( r ) w 2 = h i -yi 2 ij i 1 2 h 2 ( g ( wij g ( w r )) yi) w h = h in jk k - 2 ij i j k h in = dir j h g '( h in i ) w k ik d k 2 i delta rule for put weights (r i - y i ) 2 Dw ij = f(g'(h i )(y i - r i )r j in ) Hidden layer weights Error propagates BACKWARDS through the layers! delta rule for hidden wts (depends on delta rule for put wts) 2 17

18 Training protocol (gradient descent) Training set Test set 1. Train until gradient of error function reaches minimum. A. Batch: use full training set with each iteration (smooth convergence, but more prone to local minima) B. Online: use different sample of training set with each iteration (more memory efficient, but messy convergence) 2. Test generalization of network using previously unseen test data set. 18

19 Caveats Local minima Can use momentum term in weight update to incorporate history of weight changes. Overfitting Use heuristics to determine appropriate number of nodes for solving particular problem. Can usually use 2*number of nodes for training set. 19

20 Recurrent networks are a whole new game! but I ll spare you Backprop. through time (BPTT) 20