[Read Ch. 4] [Recommended exercises 4.1, 4.2, 4.5, 4.9, 4.11]

Transcription

1 Articial Neural Networks [Read Ch. 4] [Recommended exercises 4.1, 4.2, 4.5, 4.9, 4.11] Threshold units Gradient descent Multilayer networks Backpropagation Hidden layer representations Example: Face Recognition Advanced topics 74 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

2 Connectionist Models Consider humans: Neuron switching time ~ :001 second Number of neurons ~ Connections per neuron ~ 10 4?5 Scene recognition time ~ :1 second 100 inference steps doesn't seem like enough! much parallel computation Properties of articial neural nets (ANN's): Many neuron-like threshold switching units Many weighted interconnections among units Highly parallel, distributed process Emphasis on tuning weights automatically 75 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

3 When to Consider Neural Networks Input is high-dimensional discrete or real-valued (e.g. raw sensor input) Output is discrete or real valued Output is a vector of values Possibly noisy data Form of target function is unknown Human readability of result is unimportant Examples: Speech phoneme recognition [Waibel] Image classication [Kanade, Baluja, Rowley] Financial prediction 76 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

4 ALVINN drives 70 mph on highways Sharp Left Straight Ahead Sharp Right 30 Output Units 4 Hidden Units 30x32 Sensor Input Retina 77 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

5 Perceptron x 1 w 1 x 0 =1 x 2 x n... w 2 w n o(x 1 ; : : : ; x n ) = w 0 8 >< >: Σ n Σ w i x i i=0 { n 1 if Σ w > 0 o = i x i=0 i -1 otherwise 1 if w 0 + w 1 x w n x n > 0?1 otherwise. Sometimes we'll use simpler vector notation: o(~x) = 8 >< >: 1 if ~w ~x > 0?1 otherwise. 78 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

6 Decision Surface of a Perceptron x 2 + x x x 1 (a) (b) Represents some useful functions What weights represent g(x 1 ; x 2 ) = AND(x 1 ; x 2 )? But some functions not representable e.g., not linearly separable Therefore, we'll want networks of these lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

7 Perceptron training rule where w i w i + w i w i = (t? o)x i Where: t = c(~x) is target value o is perceptron output is small constant (e.g.,.1) called learning rate 80 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

8 Perceptron training rule Can prove it will converge If training data is linearly separable and suciently small 81 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

9 Gradient Descent To understand, consider simpler linear unit, where o = w 0 + w 1 x w n x n Let's learn w i 's that minimize the squared error E[ ~w] 1 (t d? o d ) 2 2 d2d Where D is set of training examples X 82 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

10 Gradient Descent E[w] Gradient 0 w0-1 Training rule: i.e., 3 2 re[ ~w] 1 w ~w =?re[ ~w] -1 w i n lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

11 Gradient 1 X (t d? o d ) i 2 d = 1 (t d? o d ) 2 2 i = 1 X 2(t d? o d (t d? o d ) 2 i = X (t d? o d (t d? ~w ~x d ) i i = X d (t d? o d )(?x i;d ) 84 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

12 Gradient Descent Gradient-Descent(training examples; ) Each training example is a pair of the form h~x; ti, where ~x is the vector of input values, and t is the target output value. is the learning rate (e.g.,.05). Initialize each w i to some small random value Until the termination condition is met, Do { Initialize each w i to zero. { For each h~x; ti in training examples, Do Input the instance ~x to the unit and compute the output o For each linear unit weight w i, Do w i w i + (t? o)x i { For each linear unit weight w i, Do w i w i + w i 85 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

13 Summary Perceptron training rule guaranteed to succeed if Training examples are linearly separable Suciently small learning rate Linear unit training rule uses gradient descent Guaranteed to converge to hypothesis with minimum squared error Given suciently small learning rate Even when training data contains noise Even when training data not separable by H 86 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

14 Incremental (Stochastic) Gradient Descent Batch mode Gradient Descent: Do until satised 1. Compute the gradient re D [ ~w] 2. ~w ~w? re D [ ~w] Incremental mode Gradient Descent: Do until satised For each training example d in D 1. Compute the gradient re d [ ~w] 2. ~w ~w? re d [ ~w] E D [ ~w] 1 2 X d2d (t d? o d ) 2 E d [ ~w] 1 2 (t d? o d ) 2 Incremental Gradient Descent can approximate Batch Gradient Descent arbitrarily closely if made small enough 87 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

15 Multilayer Networks of Sigmoid Units head hid who d hood F1 F2 88 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

16 Sigmoid Unit x 1 w 1 x 0 = 1 x 2 x n... w 2 w n w 0 Σ n net = Σ w i x i=0 i o = σ(net) = ē net (x) is the sigmoid function Nice property: d(x) dx e?x = (x)(1? (x)) We can derive gradient decent rules to train One sigmoid unit Multilayer networks of sigmoid units! Backpropagation 89 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

17 Error Gradient for a Sigmoid 1 X (t d? o d ) i 2 d2d = 1 (t d? o d ) 2 2 i = 1 2 X d 2(t d? o d ) = X d (t d? o d ) =? X d (t d? o d i (t d? o d d 1 C i But we d d = o d (1? o d i ~w ~x i =? X d2d = x i;d (t d? o d )o d (1? o d )x i;d 90 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

18 Backpropagation Algorithm Initialize all weights to small random numbers. Until satised, Do For each training example, Do 1. Input the training example to the network and compute the network outputs 2. For each output unit k 3. For each hidden unit h k o k (1? o k )(t k? o k ) h o h (1? o h ) X k2outputs 4. Update each network weight w i;j w h;k k where w i;j w i;j + w i;j w i;j = j x i;j 91 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

19 More on Backpropagation Gradient descent over entire network weight vector Easily generalized to arbitrary directed graphs Will nd a local, not necessarily global error minimum { In practice, often works well (can run multiple times) Often include weight momentum w i;j (n) = j x i;j + w i;j (n? 1) Minimizes error over training examples { Will it generalize well to subsequent examples? Training can take thousands of iterations! slow! Using network after training is very fast 92 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

20 Learning Hidden Layer Representations Inputs Outputs A target function: Input Output ! ! ! ! ! ! ! ! Can this be learned?? 93 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

21 Learning Hidden Layer Representations A network: Inputs Outputs Learned hidden layer representation: Input Hidden Output Values ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

22 Training Sum of squared errors for each output unit lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

23 Training Hidden unit encoding for input lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

24 Training Weights from inputs to one hidden unit lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

25 Convergence of Backpropagation Gradient descent to some local minimum Perhaps not global minimum... Add momentum Stochastic gradient descent Train multiple nets with dierent inital weights Nature of convergence Initialize weights near zero Therefore, initial networks near-linear Increasingly non-linear functions possible as training progresses 98 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

26 Expressive Capabilities of ANNs Boolean functions: Every boolean function can be represented by network with single hidden layer but might require exponential (in number of inputs) hidden units Continuous functions: Every bounded continuous function can be approximated with arbitrarily small error, by network with one hidden layer [Cybenko 1989; Hornik et al. 1989] Any function can be approximated to arbitrary accuracy by a network with two hidden layers [Cybenko 1988]. 99 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

27 Overtting in ANNs Error Error Error versus weight updates (example 1) Training set error Validation set error Number of weight updates Error versus weight updates (example 2) Training set error Validation set error Number of weight updates 100 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

28 Neural Nets for Face Recognition left strt rght up x32 inputs Typical input images 90% accurate learning head pose, and recognizing 1-of-20 faces 101 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

29 Learned Hidden Unit Weights left strt rght up Learned Weights x32 inputs Typical input images lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

30 Alternative Error Functions Penalize large weights: E( ~w) 1 2 X d2d X k2outputs(t kd? o kd ) 2 + X i;j w 2 ji Train on target slopes as well as values: E( ~w) 1 2 X X d2d k2outputs 2 6 4(t kd? o kd ) 2 + X j2inputs j j d 1 A23 7 C 775 Tie together weights: e.g., in phoneme recognition network 103 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

31 Recurrent Networks y(t + 1) y(t + 1) b x(t) x(t) c(t) (a) Feedforward network (b) Recurrent network y(t + 1) x(t) c(t) y(t) x(t 1) c(t 1) y(t 1) (c) Recurrent network unfolded in time x(t 2) c(t 2) 104 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997