Neural Networks. CAP5610 Machine Learning Instructor: Guo-Jun Qi

Transcription

1 Neural Networks CAP5610 Machine Learning Instructor: Guo-Jun Qi

2 Recap: linear classifier Logistic regression Maximizing the posterior distribution of class Y conditional on the input vector X Support vector machines Maximizing the maximum margin, and Hard margin: subject to the constraints that no training error shall be made Soft margin: minimizing the slack variables that represent how much an associated training example violates the classification rule. Extended to nonlinear classifier with kernel trick Mapping input vectors to high dimensional space linear classifier in high dimensional space, nonlinear in original space.

3 Building nonlinear classifier With a network of logistic units? A single logistic unit is linear classifier : f: X Y 1 f X = 1 + exp( W 0 N n=1 W n X n )

4 Graph representation of a logistic unit Input layer: An input X=(X 1,, X n ) Output: logistic function of the input features

5 A logistic unit as an neuron: Input layer: An input X=(X 1,, X n ) Activation: weighted sum of input features a = W 0 + N n=1 W n X n Activation function: logistic function h applied to the weighted sum Output: z = h(a)

6 Neural Network: Multiple layers of neurons Output of a layer is the input into the upper layer

7 An example A three layer neural network w 10 z 1 a 1 w 11 a 1 y 1 w 12 a 2 z2 w 10 w 20 a1 = w11 x1 + w 12 x2 + w 10 z 1 = h(a 1 ) a 2 = w21 x1 + w 22 x2 + w 20 z 2 = h(a 2 ) w 11 w 21 w 12 w 22 x 1 x 2 a 1 = w11 z1 + w 12 x2 + w 10 y 1 = f(a 1 )

8 XOR Problem It is impossible to linearly separate these two classes x 2 (0,1) (1,1) (0,0) (1,0) x 1

9 XOR Problem Two classes become separable by putting a threshold 0.5 to the output y 1 x 2 (0,1) (1,1) 6.06 z 1 y z Input Output (0,0) (1,0) (0,1) (1,1) (0,0) (1,0) x 1 x 1 x 2

10 Application: Drive a car Input: real-time videos captured by a camera Output: signals that steer a car From the sharp left, straight to sharp right

11 Training Neural Network Given a training set of M examples {(x (i),t (i) ) i=1,,m} Training neural network is equivalent to minimizing the least square error between the network output and the true value min w L w = 1 (y (i) t i ) 2 2 i=1 Where y (i) is the output depending on the network parameters w. M

12 Recap: Gradient decent Method Gradient descent method is an iterative algorithm hill climbing method to find the peak point of a mountain At each point, compute its gradient L L L L,,..., w w w 0 1 N Gradient is a vector that points to the steepest direction climbing up the mountain. At each point, w is updated so it moves a size of step λ in the gradient direction w w L( w)

13 Stochastic Gradient Ascent Method Making the learning algorithm scalable to big data Computing the gradient of square error for only one example L w = M i=1 (y (i) t i ) 2 L L L L,,..., w w w 0 1 N L (i) w = (y (i) t i ) 2 L () i L L L,,..., w w w ( i) ( i) ( i) 0 1 N

14 Boiling down to computing the gradient y 1 z 1 w 11 a 1 w 12 z2 w 10 Square loss: L 1 (y t ) 2 k k 2 y f ( a ), a w z k k k kj j j Derivative to the activation in the second layer: L y (y t ) (y t ) f '( a ) k k k k k k k ak ak x 1 x 2 Derivative to the parameter in the second layer: L L a k ki k ki w a w k z i

15 Boiling down to computing the gradient Computing the derivatives to the parameters in the first layer w 10 z 1 w 11 a 1 y 1 w 12 z2 w 10 Relation between activations of the first and second layers By chain rule: a w h( a ) L k kj j j k j k k j h '( a ) L a a a a w j k kj j k w 11 w 21 w w The derivative to the parameter in the first layer: a w x j jn n n L L a w a w j jn j jn j x n x 1 x 2

16 Summary: Back propagation δ 1 For each training example (x,y), For each output unit k For each hidden unit j (y t ) f '( a ) k k k k δ 1 δ 2 h '( a ) w j j k kj k

17 Summary: Back propagation δ 1 For each training example (x,y), For each weight w ki : L w = δ k ki zi δ 1 δ 2 Update w ki wki δk zi For each weight w jn : Update L w jn = δ j xn w jn wjn δj xn

18 Regularized Square Error Add a zero mean Gaussian prior on the weights w ij (l) ~N(0, σ 2 ) MAP estimate of w L (i) w = 1 2 (y(i) t i ) 2 + γ 2 (w ij (l) ) 2

19 Summary: Back propagation δ 1 For each training example (x,y), For each weight w ki : L w = δ k zi + γw ki ki δ 1 δ 2 Update w ki wki δk zi γw ki For each weight w jn : Update L w = δ j xn + γw jn jn w jn wjn δj xn γw jn

20 Multiple outputs encoding multiple classes MNIST: ten classes of digits Encoding multiple classes as multiple outputs: An output variable is set to 1 if the corresponding class is positive for the example Otherwise, the output is set to 0. The posterior probability of an example belonging to class k P Class k x = y k K k =1 y k

21 Overfitting Tuning the number of update iterations on validation set

22 How expressive is NN? Boolean functions: Every Boolean function can be represented by network with single hidden layer But might require exponential number of hidden units Continuous functions: Every bounded continuous function can be approximated with arbitrarily small error by neural network with one hidden layer Any function can be approximated to arbitrary accuracy by a network with two hidden layers

23 Learning feature representation by neural networks A compact representation for high dimensional input vectors A large image with thousands of pixels High dimensional input vectors might cause curse of dimensionality Needs more examples for training (in lecture 1) Not well capture the intrinsic variations an arbitrary point in a high dimensional space probably does not represent a valid real object. A meaningful low dimensional space is preferred!

24 Autoencoder Set output to input Hidden layers as feature representation, since it contains sufficient information to reconstruct the input in the output layer

25 An example

26 Deep Learning: A Deep Feature Representation If you build multiple layers to reconstruct the input at the output layer

27 Summary Neural Networks: Multiple layers of neurons Each upper layer neuron encodes weighted sum of inputs from the other neurons at a lower layer by an activation function BP training: a stochastic gradient descent method From the output layer down to the hidden and input layers Autoencoder: feature representation