Accepted Notation. x i w ij. Net input to unit Y j : s j = b j + i

Transcription

1 Accepted Notation x i input signal of activation unit X i. y j y j = F (s j ) for an activation unit Y j. w i,j Weight on connection from unit X i to unit Y j. b j Bias on unit Y j. s j Net input to unit Y j : s j = b j + i x i w ij W weight matrix {w ij }. w j Vector of weights w j = (w 1j, w 2j,..., w nj ) T. u j Threshold for activation of neuron Y j. x Input vector for classification or response x = (x 1,..., x i,..., x n ). w ij Change in w ij. w ij = [w ij (new) w ij (old)]. Dr. E.C. Kulasekere () Neural Networks 1 / 36

2 Mathematical Models of Activation Functions Typically the same activation function is used for all neurons in any particular layer (this is not a requirement). In a multi-layer network if the neurons have linear activation functions the capabilities are no better than a single layer network with a linear activation function. Hence in most cases nonlinear activation functions are used. Linear Activation Function f (x) = x, x. Dr. E.C. Kulasekere () Neural Networks 2 / 36

3 Mathematical Models of Activation Functions... Binary Step Function with Threshold θ { 1 if x θ f (x) = 0 if x < θ f(x) 1 θ x Dr. E.C. Kulasekere () Neural Networks 3 / 36

4 Mathematical Models of Activation Functions... Binary Sigmoid Function f (x) = exp ( σx). f (x) = σf (x)[1 f (x)]. f(x) 1 This is especially useful when training the backpropagation neural network where the derivatives are used to obtain the output value. Dr. E.C. Kulasekere () Neural Networks 4 / 36 x

5 Mathematical Models of Activation Functions... Bipolar Sigmoid Function g(x) = 2f (x) 1 = 1 exp ( σx) 1 + exp ( σx). g (x) = σ [1 + g(x)][1 g(x)]. 2 The above is actually the binary sigmoid scaled to have a set of values ranging from -1 to +1. The bipolar sigmoid is also closely related to the hyperbolic tangent function. exp (x) exp ( x) h(x) = tanh (x) = exp (x) + exp ( x) 1 exp ( 2x) = 1 + exp ( 2x). Dr. E.C. Kulasekere () Neural Networks 5 / 36

6 Matrix Manipulations 1 X 1 w 1j b j X i w ij Y j X n w nj s j = w j x t = n i=1 x iw ij, if the bias is neglected If the bias is not neglected, the x includes the bias value. Here row vectors for the input and weights are assumed. Ex is a direct application of the material learned thus far. Dr. E.C. Kulasekere () Neural Networks 6 / 36

7 McCulloch-Pitts Neuron Characteristics The McCulloch-Pitt neuron is considered to be the first neural network. It is a fixed weight network that can be used to implement boolean functions. Its characteristics are: Binary activation (1 ON, 0 OFF). i.e. it either fires with an activation 1 or does not fire with an activation of 0. Neurons are connected by directed weighted paths. If w > 0, excitatory, else inhibitory. Excitatory weights are identical; inhibitory weights too. Each neuron has a fixed threshold for firing. That is if the net input to the neuron is greater than the threshold it fires. The threshold is set such that the inhibition is absolute. Any non-zero inhibitory input will prevent it from firing. It takes one time step for the signal to pass over one link. Dr. E.C. Kulasekere () Neural Networks 7 / 36

8 McCulloch-Pitts Neuron Architecture X 1 w The inhibition should be absolute. Hence the threshold satisfies θ > nw p X n X n+1 X n+m w p p Y The output neuron will fire if it receives k or more excitatory inputs with no inhibitory inputs, where kw θ > (k 1)w The activation function { is 1 if s θ f (s) = 0 if s < θ Dr. E.C. Kulasekere () Neural Networks 8 / 36

9 Neural net to perform the AND function X 1 X Y Write the truth table for the above network. Threshold = 2 Given another configuration of the AND function with Threshold 1. Dr. E.C. Kulasekere () Neural Networks 9 / 36

10 Neural net to perform the OR function X 1 X Y Write the truth table for the above network. Threshold=2 Give another configuration of the OR function with Threshold 1. Dr. E.C. Kulasekere () Neural Networks 10 / 36

11 Neural net to perform the AND NOT function X 1 X Y Write the truth table for the AND NOT function. Threshold=2 Can you find another configuration for the weights to implement the AND NOT function? Dr. E.C. Kulasekere () Neural Networks 11 / 36

12 Neural net to perform the XOR function X Z 1 2 Y X Z 2 2 Write the truth table for the XOR function. Threshold=2 Write the layer equations for the XOR function and show that it is AND NOT combined with an OR function. Dr. E.C. Kulasekere () Neural Networks 12 / 36

13 Geometric View of Neural Architectures Single output neuron Note that the threshold function can be reduced to the discrete activation function when the threshold is considered as a bias. Then the synaptic input can be written as s = wx t. For two input neurons with inputs x 1 and x 2, s = 0 is a straight line. On the x 1 x 2 -plane, s > 0 gives one half-plane and s < 0 gives another half-plane. If the weights and threshold are multiplied by a constant c, then the only difference is that the half-planes in which the earlier classification occurred will be switched. (pp.27) If you have more that two inputs you will be plotting on a x 1 x 2... x n -hyper plane. For example if n = 3 you will end up with a plane. Dr. E.C. Kulasekere () Neural Networks 13 / 36

14 Geometric View of Neural Architectures Multiple output neurons Each output neuron will have a summation block of s j = wx t. For j = 2 two input neurons the above will give two straight lines in the x 1, x 2 -plane. For j = 2 with more than two input neurons, the above will give two n-dimensional hyper-planes. For j > 2 and several inputs will give higher order planes. Dr. E.C. Kulasekere () Neural Networks 14 / 36

15 Example Note that the threshold function can be converted to the discrete activation function by converting s > 0 to s u > 0. By doing this we are converting a threshold to a bias. We have to mark the area in which the output turns out to be 1. That is by looking at the vertices, we determine it is x 1 > 0, x 1 < 1, x 2 > 0, and, x 2 < 1. In order to obtain the required output the threshold function has to satisfy s u > 0. That is w 1 x 1 + w 2 x 2 u > 0. Now compare the above two items to obtain the required w 1, w 2 and u Even for Exercise we have a region bonded by a triangle in which the same computation can be carried out. Even when you are building a two layer network the arguments used in MCP networks can be used for example the question in pp.33. All of these techniques produce fixed weight ANNs. Dr. E.C. Kulasekere () Neural Networks 15 / 36

16 Pattern Classification Using ANNs This is the simplest task a neural network can be trained to perform. Each input vector pattern either belongs or does not belong to a class. The correct input data is assumed to be known. The activation function is either binary or bipolar. The output of the trained network will be 1 when presented with a pattern from the class. If the pattern is not in the class a -1 (or 0 for a binary output) is output. Since it is either one set of outputs the activation is a hard step. Dr. E.C. Kulasekere () Neural Networks 16 / 36

17 Biases and Thresholds The bias acts exactly as a weight on a connection from a unit whose activation is always 1. Increasing the bias increases the net input to to the unit. The Activation with a bias is f (s) = { 1 if s 0; 1 if s < 0; The Activation with a threshold is where s = b + i x i w i,. f (s) = { 1 if s θ; 1 if s < θ; where s = i x i w i,. Essentially the above are equivalent for most cases. However the bias becomes essential for certain problems associated with linear separability. Dr. E.C. Kulasekere () Neural Networks 17 / 36

18 Requirement of Linear Separability In pattern classification, the network is trained to output an indicator to show the presented input is either a member of one class or another class. This will depend on the decision boundary that will either fire the neuron or not fire it. This can be determined by setting y_in = 0 for the case with a bias. The decision boundary is b + i x i w i = 0. If the system is linearly separable, the two classes should lie on either side of the decision boundary (y_in > 0 or y_in < 0). The boundary is non unique x 2 = w 1 w 2 x 1 b w 2. Dr. E.C. Kulasekere () Neural Networks 18 / 36

19 Requirement of Linear Separability... For bipolar signals the outputs for the two classes are -1 and +1. For unipolar signals it is 0 and 1. Depending on the number of inputs the decision boundary can be a line, plane or a hyperplane. Eg. For two inputs its a line and for three inputs its a plane. If all of the training input vectors for which the correct response is +1 lie on one side of the decision boundary we say that this system is linearly separable. It has been shown that a single layer network can only learn linearly separable problems. The trained weights are not unique. Dr. E.C. Kulasekere () Neural Networks 19 / 36

20 Response Regions for the AND Function The AND function for Bipolar inputs and targets is Input (x 1, x 2 ) Output (t) (1,1) +1 (1,-1) -1 (-1,1) -1 (-1,-1) -1 x 2 = x is the decision boundary where w 1 = w 2 = 1 and b = 1. This boundary is not unique. - - x x 1 Dr. E.C. Kulasekere () Neural Networks 20 / 36

21 Response Regions for the OR function The OR function for Bipolar inputs and targets is Input (x 1, x 2 ) Output (t) (1,1) +1 (1,-1) +1 One (-1,1) +1 (-1,-1) -1 possible decision boundary is given by x 2 = x 1 1 where b = 1, w 1 = 1 = w 2. Draw the above graphically. Importance of Bias If the bias weight was not included in the previous example the decision boundary would have been forced to go through the origin. This changes the problem from a solvable one to an unsolvable problem. Dr. E.C. Kulasekere () Neural Networks 21 / 36

22 Response Regions for the XOR Function The XOR function for bipolar inputs and targets is given by Input (x 1, x 2 ) Output (t) (1,1) -1 (1,-1) +1 This (-1,1) +1 (-1,-1) -1 is not linearly separable. + x 2 - x Dr. E.C. Kulasekere () Neural Networks 22 / 36

23 Binary/Bipolar Data Representation In most cases the binary input data can be modified to bipolar data. However the form of the data can change the problem from one that is solvable to a problem that cannot be solved. Binary representation is also not as good as the bipolar if we want the net to generalize. i.e. to respond to input data that is similar but not the same as the training data. Bipolar inputs and targets are also better during training where the adaptive weight change is computed. For example w = xy is the weight change in the Hebb learning algorithm. If either the training input vector or the target vector is binary (unipolar) the update becomes zero and the learning stops. Using bipolar missing data can be distinguished from the mistaken data by assigning 0 to missing data and a mistake by changing -1 to +1 or vise versa. Dr. E.C. Kulasekere () Neural Networks 23 / 36

24 The HEBB Net This is the earliest and the most simplest learning rule for the neural networks. Hebb proposed that if two interconnected neurons are both on at the same time, then the weight between them should be increased. However the original statement did not discuss neurons that are connected but do not fire together. Later this was also included in the Hebb rule so that the weight increased of not firing together and the original rule was made a strongest learning algorithm. The weight update can be represented as w i (new) = w i (old) + x i y. Note that if the signals are binary, the update rule cannot distinguish between a pair of inputs outputs for the following conditions. Input is on and the target is off. Both the input and the target units are off. Dr. E.C. Kulasekere () Neural Networks 24 / 36

25 Hebb Algorithm Step 0 Initialize all weights: w i = 0, i = 1,..., n. Step 1 For each input training vector and target output pair, s : t, do steps 2-4. Step 2 Set activations for input units: x i = s i, i = 1,..., n. Step 3 Set activation for output unit: y = t. Step 4 Adjust the weight and bias: w i (new) = w i (old) + x i y for i = 1,..., n b(new) = b(old) + y. If the bias is considered to be an input signal that is always 1, the weight change can be written as where w = xy. w(new) = w(old) + w Dr. E.C. Kulasekere () Neural Networks 25 / 36

26 Hebb Net for AND Function Binary inputs and targets Input Target (x 1, x 2, 1) The truth table for the AND function is (1,1,1) 1 (1,0,1) 0 (0,1,1) 0 (0,0,1) 0 The initial values are w(old) = (0, 0) and b(old) = 0. The first step of the algorithm is Input Target Weight Changes Weights (x 1, x 2, 1) ( w 1, w 2, b) (w 1, w 2, b) (0,0,0) (1,1,1) 1 (1,1,1) (1,1,1) The separating line after the first step is x 2 = x 1 1 Dr. E.C. Kulasekere () Neural Networks 26 / 36

27 Hebb Net for AND Function... Binary inputs and targets Now if we present the second, third and fourth training vectors the weight change is given by Input Target Weight Changes Weights (x 1, x 2, 1) ( w 1, w 2, b) (w 1, w 2, b) (1,1,1) (1,0,1) 0 (0,0,0) (1,1,1) (0,1,1) 0 (0,0,0) (1,1,1) (0,0,1) 0 (0,0,0) (1,1,1) We note that in the above problem when the target value is zero, no learning occurs and hence no weight change. With this we can determine that the bipolar inputs have resulted in a short coming in the learning method. Dr. E.C. Kulasekere () Neural Networks 27 / 36

28 Hebb Net for the AND function Binary inputs, bipolar targets The truth table in this case is Input Target (x 1, x 2, 1) (1,1,1) +1 (1,0,1) -1 (0,1,1) -1 (0,0,1) -1 Presenting the first input Input Target Weight Changes Weights (x 1, x 2, 1) ( w 1, w 2, b) (w 1, w 2, b) (0,0,0) (1,1,1) 1 (1,1,1) (1,1,1) The separating line becomes x 2 = x 1 1. This is the correct classification for the first input. Dr. E.C. Kulasekere () Neural Networks 28 / 36

29 Hebb Net for the AND function... Binary inputs, bipolar targets x x 1 Dr. E.C. Kulasekere () Neural Networks 29 / 36

30 Hebb Net for the AND function... Binary inputs, bipolar targets Presenting the other inputs we have the following Input Target Weight Changes Weights (x 1, x 2, 1) ( w 1, w 2, b) (w 1, w 2, b) (1,0,1) -1 (-1,0,-1) (0,1,0) (0,1,1) -1 (0,-1,-1) (0,0,-1) (0,0,1) -1 (0,0,-1) (0,0,-2) Again we see that the problem has not correctly classified the output. Hence alternate training data should be used. Dr. E.C. Kulasekere () Neural Networks 30 / 36

31 Hebb Net for the AND function Bipolar inputs and targets The truth table is given by Input Target (x 1, x 2, 1) (1,1,1) +1 (1,-1,1) -1 (-1,1,1) -1 (-1,-1,1) -1 Presenting the first input:target pair yields the same result as before with the separation line being x 2 = x 1 1. Now the classification is correct for the point (1,1) and also for (-1,-1). - - x x 1 Dr. E.C. Kulasekere () Neural Networks 31 / 36

32 Hebb Net for the AND function... Bipolar inputs and targets For the second input:target pair we obtain the separating line as x 2 = 0. Input Target Weight Changes Weights (x 1, x 2, 1) ( w 1, w 2, b) (w 1, w 2, b) (1,1,1) (1,-1,1) -1 (-1,1,-1) (0,2,0) With the third pair we obtain the separating line as x 2 = x Input Target Weight Changes Weights (x 1, x 2, 1) ( w 1, w 2, b) (w 1, w 2, b) (0,2,0) (-1,1,1) -1 (1,-1,-1) (1,1,-1) Presenting the fourth pair, the separating line does not change from what we obtained previously even though the weights have changed. Input Target Weight Changes Weights (x 1, x 2, 1) ( w 1, w 2, b) (w 1, w 2, b) (1,1,-1) (-1,-1,1) -1 (1,1,-1) (2,2,-2) Dr. E.C. Kulasekere () Neural Networks 32 / 36

33 Hebb Net for the AND function... Bipolar inputs and targets x x Figure: Decision boundary for bipolar AND function using Hebb rule after third/fourth training. Draw the diagram for the intermediate stages as well. Dr. E.C. Kulasekere () Neural Networks 33 / 36

34 Character Recognition Example Using Hebb Net The Problem: Distinguish between the two patterns. # # # # # # # # # # # # # # # # # # # # # The steps involved are as follows: We use the bipolar representation to convert the patterns into the input values that can be used for training. We assign # to 1 and to -1. The correct target for the first pattern is +1 and for the second pattern it is -1. This is used as the classification. Then use the Hebb rule to find the weights by repeatedly presenting the input:target pairs ti find the weights. Check the system with training patterns that are similar but not identical to see if the system will still react with the correct classification. Dr. E.C. Kulasekere () Neural Networks 34 / 36

35 Missing and Mistaken Data Binary representation of inputs and targets (0 and 1 levels) Missing Data: Cannot be represented. Mistaken Data: A mistake of 1 is represented as 0 and vise versa. Bipolar representation of inputs and targets (-1 and 1 levels) Missing Data: this is represented by a 0. Mistaken Data: A mistake in +1 is represented as -1 and vise versa. In general a net can handle more missing components than wrong components. For input data the above translates to It s better not to guess. The reason is that if the value is zero the weight change during training may not get affected. Dr. E.C. Kulasekere () Neural Networks 35 / 36

36 Additional Comments ANN Training is completely ad hoc. Sometimes the result will not converge. The weight vector ca be multiplied by a positive number without changing the actions of the neurons. This normalization can eliminate rounding errors account for input variations etc. The value of η is significant for convergence as well as the rate of convergence. The order in which the training vectors are presented is also important. Randomizing is generally preferable. In multilayer networks, it is accepted that more than two layers are rarely used. Dr. E.C. Kulasekere () Neural Networks 36 / 36