4 Perceptron Learning Rule

Transcription

1 Percetron Learning Rule Objectives Objectives - Theory and Examles - Learning Rules - Percetron Architecture -3 Single-Neuron Percetron -5 Multile-Neuron Percetron -8 Percetron Learning Rule -8 Test Problem -9 Constructing Learning Rules - Unified Learning Rule - Training Multile-Neuron Percetrons -3 Proof of Convergence -5 Notation -5 Proof -6 Limitations -8 Summary of Results - Solved Problems - Eilogue -33 Further Reading -3 Exercises -36 Objectives One of the questions we raised in Chater 3 was: ÒHow do we determine the weight matrix and bias for ercetron networks with many inuts, where it is imossible to visualize the decision boundaries?ó In this chater we will describe an algorithm for training ercetron networks, so that they can learn to solve classification roblems. We will begin by exlaining what a learning rule is and will then develo the ercetron learning rule. We will conclude by discussing the advantages and limitations of the singlelayer ercetron network. This discussion will lead us into future chaters. -

2 Percetron Learning Rule Theory and Examles Learning Rule In 93, Warren McCulloch and Walter Pitts introduced one of the first artificial neurons [McPi3]. The main feature of their neuron model is that a weighted sum of inut signals is comared to a threshold to determine the neuron outut. When the sum is greater than or equal to the threshold, the outut is. When the sum is less than the threshold, the outut is. They went on to show that networks of these neurons could, in rincile, comute any arithmetic or logical function. Unlike biological networks, the arameters of their networks had to be designed, as no training method was available. However, the erceived connection between biology and digital comuters generated a great deal of interest. In the late 95s, Frank Rosenblatt and several other researchers develoed a class of neural networks called ercetrons. The neurons in these networks were similar to those of McCulloch and Pitts. RosenblattÕs key contribution was the introduction of a learning rule for training ercetron networks to solve attern recognition roblems [Rose58]. He roved that his learning rule will always converge to the correct network weights, if weights exist that solve the roblem. Learning was simle and automatic. Examles of roer behavior were resented to the network, which learned from its mistakes. The ercetron could even learn when initialized with random values for its weights and biases. Unfortunately, the ercetron network is inherently limited. These limitations were widely ublicized in the book Percetrons [MiPa69] by Marvin Minsky and Seymour Paert. They demonstrated that the ercetron networks were incaable of imlementing certain elementary functions. It was not until the 98s that these limitations were overcome with imroved (multilayer) ercetron networks and associated learning rules. We will discuss these imrovements in Chaters and. Today the ercetron is still viewed as an imortant network. It remains a fast and reliable network for the class of roblems that it can solve. In addition, an understanding of the oerations of the ercetron rovides a good basis for understanding more comlex networks. Thus, the ercetron network, and its associated learning rule, are well worth discussion here. In the remainder of this chater we will define what we mean by a learning rule, exlain the ercetron network and learning rule, and discuss the limitations of the ercetron network. Learning Rules As we begin our discussion of the ercetron learning rule, we want to discuss learning rules in general. By learning rule we mean a rocedure for modifying the weights and biases of a network. (This rocedure may also -

3 Percetron Architecture be referred to as a training algorithm.) The urose of the learning rule is to train the network to erform some task. There are many tyes of neural network learning rules. They fall into three broad categories: suervised learning, unsuervised learning and reinforcement (or graded) learning. Suervised Learning Training Set In suervised learning, the learning rule is rovided with a set of examles (the training set) of roer network behavior: Target Reinforcement Learning Unsuervised Learning {, t },{, t },, { Q, t Q }, (.) where q is an inut to the network and t q is the corresonding correct (target) outut. As the inuts are alied to the network, the network oututs are comared to the targets. The learning rule is then used to adjust the weights and biases of the network in order to move the network oututs closer to the targets. The ercetron learning rule falls in this suervised learning category. We will also investigate suervised learning algorithms in Chaters 7Ð. Reinforcement learning is similar to suervised learning, excet that, instead of being rovided with the correct outut for each network inut, the algorithm is only given a grade. The grade (or score) is a measure of the network erformance over some sequence of inuts. This tye of learning is currently much less common than suervised learning. It aears to be most suited to control system alications (see [BaSu83], [WhSo9]). In unsuervised learning, the weights and biases are modified in resonse to network inuts only. There are no target oututs available. At first glance this might seem to be imractical. How can you train a network if you donõt know what it is suosed to do? Most of these algorithms erform some kind of clustering oeration. They learn to categorize the inut atterns into a finite number of classes. This is esecially useful in such alications as vector quantization. We will see in Chaters 3Ð6 that there are a number of unsuervised learning algorithms. Percetron Architecture Before we resent the ercetron learning rule, letõs exand our investigation of the ercetron network, which we began in Chater 3. The general ercetron network is shown in Figure.. The outut of the network is given by a hardlim( W + b). (.) (Note that in Chater 3 we used the hardlims transfer function, instead of hardlim. This does not affect the caabilities of the network. See Exercise E.6.) -3

4 Percetron Learning Rule Inut Hard Limit Layer R R x W S x R b S x n S x S a S x a hardlim (W + b) Figure. Percetron Network It will be useful in our develoment of the ercetron learning rule to be able to conveniently reference individual elements of the network outut. LetÕs see how this can be done. First, consider the network weight matrix: w, w, w, R W w, w, w, R w S, w S, w S, R. (.3) We will define a vector comosed of the elements of the ith row of W : i w w i, w i,. (.) w i, R Now we can artition the weight matrix: wt W wt S wt. (.5) This allows us to write the ith element of the network outut vector as -

5 Percetron Architecture a hardlim (n) n W + b. (.6) Recall that the hardlim transfer function (shown at left) is defined as: (.7) Therefore, if the inner roduct of the ith row of the weight matrix with the inut vector is greater than or equal to b i, the outut will be, otherwise the outut will be. Thus each neuron in the network divides the inut sace into two regions. It is useful to investigate the boundaries between these regions. We will begin with the simle case of a single-neuron ercetron with two inuts. Single-Neuron Percetron a i hardlim( n i ) hardlim( w T i + b i ) a hardlim( n) if n otherwise. LetÕs consider a two-inut ercetron with one neuron, as shown in Figure.. Inuts Two-Inut Neuron Figure. Two-Inut/Single-Outut Percetron The outut of this network is determined by w, w, Σ b n a a hardlim (W + b) a hardlim( n) hardlim( W + b) hardlim( + b) hardlim( w, + w, + b) w T (.8) Decision Boundary The decision boundary is determined by the inut vectors for which the net inut n is zero: n + b w, + w, + b. (.9) w T To make the examle more concrete, letõs assign the following values for the weights and bias: -5

6 Percetron Learning Rule w, The decision boundary is then, w,, b. (.) n + b w, + w, + b +. (.) w T This defines a line in the inut sace. On one side of the line the network outut will be ; on the line and on the other side of the line the outut will be. To draw the line, we can find the oints where it intersects the and axes. To find the intercet set : b if. (.) w, To find the intercet, set : b if. (.3) w, The resulting decision boundary is illustrated in Figure.3. To find out which side of the boundary corresonds to an outut of, we just need to test one oint. For the inut T, the network outut will be a hardlim w T ( + b) hardlim. (.) Therefore, the network outut will be for the region above and to the right of the decision boundary. This region is indicated by the shaded area in Figure.3. w T + b w a a Figure.3 Decision Boundary for Two-Inut Percetron -6

7 Percetron Architecture w We can also find the decision boundary grahically. The first ste is to note that the boundary is always orthogonal to w, as illustrated in the adjacent figures. The boundary is defined by w T + b. (.5) w For all oints on the boundary, the inner roduct of the inut vector with the weight vector is the same. This imlies that these inut vectors will all have the same rojection onto the weight vector, so they must lie on a line orthogonal to the weight vector. (These concets will be covered in more detail in Chater 5.) In addition, any vector in the shaded region of Figure.3 will have an inner roduct greater than b, and vectors in the unshaded region will have inner roducts less than b. Therefore the weight vector will always oint toward the region where the neuron outut is. w After we have selected a weight vector with the correct angular orientation, the bias value can be comuted by selecting a oint on the boundary and satisfying Eq. (.5). + LetÕs aly some of these concets to the design of a ercetron network to imlement a simle logic function: the AND gate. The inut/target airs for the AND gate are, t, t 3, t 3, t. The figure to the left illustrates the roblem grahically. It dislays the inut sace, with each inut vector labeled according to its target. The dark circles indicate that the target is, and the light circles indicate that the target is. The first ste of the design is to select a decision boundary. We want to have a line that searates the dark circles and the light circles. There are an infinite number of solutions to this roblem. It seems reasonable to choose the line that falls ÒhalfwayÓ between the two categories of inuts, as shown in the adjacent figure. Next we want to choose a weight vector that is orthogonal to the decision boundary. The weight vector can be any length, so there are infinite ossibilities. One choice is AND w w, (.6) as dislayed in the figure to the left. -7

8 Percetron Learning Rule Finally, we need to find the bias, b. We can do this by icking a oint on the decision boundary and satisfying Eq. (.5). If we use.5 T we find w T + b.5 + b 3 + b b 3. (.7) We can now test the network on one of the inut/target airs. If we aly to the network, the outut will be a hardlim w T ( + b) hardlim a hardlim( ), 3 (.8) which is equal to the target outut correctly classified.. Verify for yourself that all inuts are To exeriment with decision boundaries, use the Neural Network Design Demonstration Decision Boundaries (nnddb). Multile-Neuron Percetron Note that for ercetrons with multile neurons, as in Figure., there will be one decision boundary for each neuron. The decision boundary for neuron i will be defined by w T i + b i. (.9) A single-neuron ercetron can classify inut vectors into two categories, since its outut can be either or. A multile-neuron ercetron can classify inuts into many categories. Each category is reresented by a different outut vector. Since each element of the outut vector can be either or, there are a total of S ossible categories, where S is the number of neurons. Percetron Learning Rule t Now that we have examined the erformance of ercetron networks, we are in a osition to introduce the ercetron learning rule. This learning rule is an examle of suervised training, in which the learning rule is rovided with a set of examles of roer network behavior: {, t },{, t },, { Q, t Q }, (.) -8

9 Percetron Learning Rule q where is an inut to the network and t q is the corresonding target outut. As each inut is alied to the network, the network outut is comared to the target. The learning rule then adjusts the weights and biases of the network in order to move the network outut closer to the target. Test Problem In our resentation of the ercetron learning rule we will begin with a simle test roblem and will exeriment with ossible rules to develo some intuition about how the rule should work. The inut/target airs for our test roblem are, t, t 3, t 3. 3 The roblem is dislayed grahically in the adjacent figure, where the two inut vectors whose target is are reresented with a light circle, and the vector whose target is is reresented with a dark circle. This is a very simle roblem, and we could almost obtain a solution by insection. This simlicity will hel us gain some intuitive understanding of the basic concets of the ercetron learning rule. The network for this roblem should have two-inuts and one outut. To simlify our develoment of the learning rule, we will begin with a network without a bias. The network will then have just two arameters, w, and, as shown in Figure.. w, Inuts No-Bias Neuron 3 w, Σ w, Figure. Test Problem Network By removing the bias we are left with a network whose decision boundary must ass through the origin. We need to be sure that this network is still able to solve the test roblem. There must be an allowable decision boundary that can searate the vectors and 3 from the vector. The figure to the left illustrates that there are indeed an infinite number of such boundaries. n a hardlim(w) a -9

10 Percetron Learning Rule 3 The adjacent figure shows the weight vectors that corresond to the allowable decision boundaries. (Recall that the weight vector is orthogonal to the decision boundary.) We would like a learning rule that will find a weight vector that oints in one of these directions. Remember that the length of the weight vector does not matter; only its direction is imortant. Constructing Learning Rules Training begins by assigning some initial values for the network arameters. In this case we are training a two-inut/single-outut network without a bias, so we only have to initialize its two weights. Here we set the elements of the weight vector, w, to the following randomly generated values: w T..8. (.) We will now begin resenting the inut vectors to the network. We begin with : a hardlim w T ( ) hardlim..8 a hardlim(.6). (.) 3 w The network has not returned the correct value. The network outut is, while the target resonse,, is. t We can see what haened by looking at the adjacent diagram. The initial weight vector results in a decision boundary that incorrectly classifies the vector. We need to alter the weight vector so that it oints more toward, so that in the future it has a better chance of classifying it correctly. One aroach would be to set w equal to. This is simle and would ensure that was classified roerly in the future. Unfortunately, it is easy to construct a roblem for which this rule cannot find a solution. The diagram to the lower left shows a roblem that cannot be solved with the weight vector ointing directly at either of the two class vectors. If we aly the rule w every time one of these vectors is misclassified, the networkõs weights will simly oscillate back and forth and will never find a solution. Another ossibility would be to add to w. Adding to w would make w oint more in the direction of. Reeated resentations of would cause the direction of w to asymtotically aroach the direction of. This rule can be stated: If t and a, then w new w old +. (.3) -

11 Percetron Learning Rule Alying this rule to our test roblem results in new values for : w w new w old (.) 3 w This oeration is illustrated in the adjacent figure. We now move on to the next inut vector and will continue making changes to the weights and cycling through the inuts until they are all classified correctly. The next inut vector is. When it is resented to the network we find: a hardlim w T ( ) hardlim.. hardlim(.). (.5) The target t associated with is and the outut a is. A class vector was misclassified as a. Since we would now like to move the weight vector w away from the inut, we can simly change the addition in Eq. (.3) to subtraction: If t and a, then w new w old. (.6) If we aly this to the test roblem we find: w new w old , (.7) which is illustrated in the adjacent figure. 3 w Now we resent the third vector : 3 a hardlim w T ( 3 ) hardlim 3..8 hardlim(.8). (.8) The current w results in a decision boundary that misclassifies 3. This is a situation for which we already have a rule, so w will be udated again, according to Eq. (.6): w new w old (.9) -

12 Percetron Learning Rule 3 w The diagram to the left shows that the ercetron has finally learned to classify the three vectors roerly. If we resent any of the inut vectors to the neuron, it will outut the correct class for that inut vector. This brings us to our third and final rule: if it works, donõt fix it. If t new a, then w wold. (.3) Here are the three rules, which cover all ossible combinations of outut and target values: If t and a, then w new If t and a, then w If t new new a, then w w old +. wold. w old. (.3) Unified Learning Rule The three rules in Eq. (.3) can be rewritten as a single exression. First we will define a new variable, the ercetron error e: e t a We can now rewrite the three rules of Eq. (.3) as:. (.3) If e new, then w If e, then w If e new new, then w w old +. w old. w old. (.33) Looking carefully at the first two rules in Eq. (.33) we can see that the sign of is the same as the sign on the error, e. Furthermore, the absence of in the third rule corresonds to an e of. Thus, we can unify the three rules into a single exression: w new w old + e w old + ( t a). (.3) This rule can be extended to train the bias by noting that a bias is simly a weight whose inut is always. We can thus relace the inut in Eq. (.3) with the inut to the bias, which is. The result is the ercetron rule for a bias: b new b old + e. (.35) -

13 Percetron Learning Rule Training Multile-Neuron Percetrons The ercetron rule, as given by Eq. (.3) and Eq. (.35), udates the weight vector of a single neuron ercetron. We can generalize this rule for the multile-neuron ercetron of Figure. as follows. To udate the ith row of the weight matrix use: i w new i w old + e i. (.36) Percetron Rule + To udate the ith element of the bias vector use: + e i. (.37) The ercetron rule can be written conveniently in matrix notation: and b i new b i old W new W old + e T b new b old + e, (.38). (.39) To test the ercetron learning rule, consider again the ale/orange recognition roblem of Chater 3. The inut/outut rototye vectors will be, t, t. (.) (Note that we are using as the target outut for the orange attern,, instead of -, as was used in Chater 3. This is because we are using the hardlim transfer function, instead of hardlims.) Tyically the weights and biases are initialized to small random numbers. Suose that here we start with the initial weight matrix and bias: W.5.5, b.5. (.) The first ste is to aly the first inut vector,, to the network: a hardlim( W + b) hardlim.5.5 hardlim(.5) +.5 (.) -3

14 Percetron Learning Rule Then we calculate the error: e t a. (.3) The weight udate is W new W old + e T ( ).5.5. (.) The bias udate is b new b old + e.5 + ( ).5. (.5) This comletes the first iteration. The second iteration of the ercetron rule is: a hardlim ( W + b) hardlim ( (.5)) (.6) hardlim (.5) e t a W new W old + e T b new b old + e (.7) (.8) (.9) The third iteration begins again with the first inut vector: a hardlim ( W + b) hardlim ( ) (.5) hardlim (.5) e t a W new W old + e T ( ).5.5 (.5) (.5) -

15 Proof of Convergence b new b old + e.5 + ( ).5. (.53) If you continue with the iterations you will find that both inut vectors will now be correctly classified. The algorithm has converged to a solution. Note that the final decision boundary is not the same as the one we develoed in Chater 3, although both boundaries correctly classify the two inut vectors. To exeriment with the ercetron learning rule, use the Neural Network Design Demonstration Percetron Rule (nndr). Proof of Convergence Although the ercetron learning rule is simle, it is quite owerful. In fact, it can be shown that the rule will always converge to weights that accomlish the desired classification (assuming that such weights exist). In this section we will resent a roof of convergence for the ercetron learning rule for the single-neuron ercetron shown in Figure.5. Inuts Hard Limit Neuron w, 3 Σ n a w,r b R a hardlim ( w T + b) Figure.5 Single-Neuron Percetron The outut of this ercetron is obtained from a hardlim( w T + b). (.5) The network is rovided with the following examles of roer network behavior: where each target outut, t q, is either or. {, t },{, t },,{ Q, t Q }. (.55) Notation To conveniently resent the roof we will first introduce some new notation. We will combine the weight matrix and the bias into a single vector: -5

16 Percetron Learning Rule x w b. (.56) We will also augment the inut vectors with a, corresonding to the bias inut: z q q. (.57) Now we can exress the net inut to the neuron as follows: n + b x T z. (.58) w T The ercetron learning rule for a single-neuron ercetron (Eq. (.3) and Eq. (.35)) can now be written x new x old + ez. (.59) The error e can be either, or. If e, then no change is made to the weights. If e, then the inut vector is added to the weight vector. If e, then the negative of the inut vector is added to the weight vector. If we count only those iterations for which the weight vector is changed, the learning rule becomes x( k) x( k ) + z' ( k ), (.6) where z' ( k ) is the aroriate member of the set { z, z,, z Q, z, z,, z Q }. (.6) We will assume that a weight vector exists that can correctly categorize all Q inut vectors. This solution will be denoted x. For this weight vector we will assume that and x T z q > δ > if t q, (.6) x T z q < δ < if t q. (.63) Proof We are now ready to begin the roof of the ercetron convergence theorem. The objective of the roof is to find uer and lower bounds on the length of the weight vector at each stage of the algorithm. -6

17 Proof of Convergence Assume that the algorithm is initialized with the zero weight vector: x( ). (This does not affect the generality of our argument.) Then, after k iterations (changes to the weight vector), we find from Eq. (.6): x( k) z' ( ) + z' ( ) + + z' ( k ). (.6) If we take the inner roduct of the solution weight vector with the weight vector at iteration k we obtain x T x( k) x T z' ( ) + x T z' ( ) + + x T z' ( k ). (.65) From Eq. (.6)ÐEq. (.63) we can show that Therefore x T z' () i > δ. (.66) x T x( k) > kδ From the Cauchy-Schwartz inequality (see [Brog9]). (.67) ( x T x( k) ) x x( k), (.68) where x x T x. (.69) If we combine Eq. (.67) and Eq. (.68) we can ut a lower bound on the squared length of the weight vector at iteration k : x( k) ( x T x( k) ) ( kδ) > x x. (.7) Next we want to find an uer bound for the length of the weight vector. We begin by finding the change in the length at iteration k : x( k) x T ( k)x( k) [ x( k ) + z' ( k ) ] T [ x( k ) + z' ( k ) ] x T ( k )x( k ) + x T ( k )z' ( k ) (.7) + z' T ( k )z' ( k ) Note that -7

18 Percetron Learning Rule Limitations The ercetron learning rule is guaranteed to converge to a solution in a finite number of stes, so long as a solution exists. This brings us to an imx T ( k )z' ( k ), (.7) since the weights would not be udated unless the revious inut vector had been misclassified. Now Eq. (.7) can be simlified to x( k) x( k ) + z' ( k ). (.73) We can reeat this rocess for x( k ), x( k ), etc., to obtain If x( k) z' ( ) + + z' ( k ). (.7) Π max{ z' () i }, this uer bound can be simlified to x( k) kπ. (.75) We now have an uer bound (Eq. (.75)) and a lower bound (Eq. (.7)) on the squared length of the weight vector at iteration k. If we combine the two inequalities we find kπ x( k) > ( kδ) x Π x or k < (.76) δ Because k has an uer bound, this means that the weights will only be changed a finite number of times. Therefore, the ercetron learning rule will converge in a finite number of iterations. The maximum number of iterations (changes to the weight vector) is inversely related to the square of δ. This arameter is a measure of how close the solution decision boundary is to the inut atterns. This means that if the inut classes are difficult to searate (are close to the decision boundary) it will take many iterations for the algorithm to converge. Note that there are only three key assumtions required for the roof:. A solution to the roblem exists, so that Eq. (.66) is satisfied.. The weights are only udated when the inut vector is misclassified, therefore Eq. (.7) is satisfied. 3. An uer bound, Π, exists for the length of the inut vectors. Because of the generality of the roof, there are many variations of the ercetron learning rule that can also be shown to converge. (See Exercise E.9.) -8

19 Proof of Convergence Linear Searability ortant question. What roblems can a ercetron solve? Recall that a single-neuron ercetron is able to divide the inut sace into two regions. The boundary between the regions is defined by the equation w T + b. (.77) This is a linear boundary (hyerlane). The ercetron can be used to classify inut vectors that can be searated by a linear boundary. We call such vectors linearly searable. The logical AND gate examle on age -7 illustrates a two-dimensional examle of a linearly searable roblem. The ale/orange recognition roblem of Chater 3 was a three-dimensional examle. Unfortunately, many roblems are not linearly searable. The classic examle is the XOR gate. The inut/target airs for the XOR gate are, t, t 3, t 3, t. This roblem is illustrated grahically on the left side of Figure.6, which also shows two other linearly insearable roblems. Try drawing a straight line between the vectors with targets of and those with targets of in any of the diagrams of Figure.6. Figure.6 Linearly Insearable Problems It was the inability of the basic ercetron to solve such simle roblems that led, in art, to a reduction in interest in neural network research during the 97s. Rosenblatt had investigated more comlex networks, which he felt would overcome the limitations of the basic ercetron, but he was never able to effectively extend the ercetron rule to such networks. In Chater we will introduce multilayer ercetrons, which can solve arbitrary classification roblems, and will describe the backroagation algorithm, which can be used to train them. -9

20 Percetron Learning Rule Summary of Results Percetron Architecture Inut Hard Limit Layer R R x W S x R b S x n S x S a S x a hardlim (W + b) a hardlim( W + b) W wt wt S wt a i hardlim( n i ) hardlim( w T i + b i ) Decision Boundary w T i + b i. The decision boundary is always orthogonal to the weight vector. Single-layer ercetrons can only classify linearly searable vectors. Percetron Learning Rule W new W old + e T b new b old + e where e t a. -

21 Solved Problems Solved Problems P. Solve the three simle classification roblems shown in Figure P. by drawing a decision boundary. Find weight and bias values that result in single-neuron ercetrons with the chosen decision boundaries. (a) (b) (c) Figure P. Simle Classification Problems First we draw a line between each set of dark and light data oints. (a) (b) (c) The next ste is to find the weights and biases. The weight vectors must be orthogonal to the decision boundaries, and ointing in the direction of oints to be classified as (the dark oints). The weight vectors can have any length we like. w w (a) (b) (c) w Here is one set of choices for the weight vectors: (a) w T, (b) w T, (c) w T. -

22 Percetron Learning Rule Now we find the bias values for each ercetron by icking a oint on the decision boundary and satisfying Eq. (.5). + b b w T w T This gives us the following three biases: (a) b, (b) b, (c) b 6 We can now check our solution against the original oints. Here we test the first network on the inut vector T. a hardlim( w T + b) hardlim + hardlim( 6)» + ans We can use MATLAB to automate the testing rocess and to try new oints. Here the first network is used to classify a oint that was not in the original roblem. w[- ]; b ; a hardlim(w*[;]+b) a P. Convert the classification roblem defined below into an equivalent roblem definition consisting of inequalities constraining weight and bias values., t, t 3, t 3, t Each target t i indicates whether or not the net inut in resonse to i must be less than, or greater than or equal to. For examle, since is, we t -

23 Solved Problems know that the net inut corresonding to to. Thus we get the following inequality: must be greater than or equal Alying the same rocedure to the inut/target airs for {, t }, { 3, t 3 } and {, t } results in the following set of inequalities. W + b +, + b +. w, w w, b w w w, w, + b () i, + b ( ii) + b < ( iii), + b < ( iv) Solving a set of inequalities is more difficult than solving a set of equalities. One added comlexity is that there are often an infinite number of solutions (just as there are often an infinite number of linear decision boundaries that can solve a linearly searable classification roblem). However, because of the simlicity of this roblem, we can solve it by grahing the solution saces defined by the inequalities. Note that w, only aears in inequalities (ii) and (iv), and w, only aears in inequalities (i) and (iii). We can lot each air of inequalities with two grahs. ii w, w, iv b iii i b Any weight and bias values that fall in both dark gray regions will solve the classification roblem. Here is one such solution: W 3 b 3. -3

24 Percetron Learning Rule P.3 We have a classification roblem with four classes of inut vector. The four classes are class :,, class : 3,, class 3: 5, 6, class : 7, 8. Design a ercetron network to solve this roblem. To solve a roblem with four classes of inut vector we will need a ercetron with at least two neurons, since an S-neuron ercetron can categorize S classes. The two-neuron ercetron is shown in Figure P.. Inut Hard Limit Layer x W x b x n Figure P. Two-Neuron Percetron LetÕs begin by dislaying the inut vectors, as in Figure P.3. The light circles indicate class vectors, the light squares indicate class vectors, the dark circles indicate class 3 vectors, and the dark squares indicate class vectors. A two-neuron ercetron creates two decision boundaries. Therefore, to divide the inut sace into the four categories, we need to have one decision boundary divide the four classes into two sets of two. The remaining boundary must then isolate each class. Two such boundaries are illustrated in Figure P.. We now know that our atterns are linearly searable. x a hardlim (W + b) a x -

25 Solved Problems 3 Figure P.3 Inut Vectors for Problem P.3 3 Figure P. Tentative Decision Boundaries for Problem P.3 The weight vectors should be orthogonal to the decision boundaries and should oint toward the regions where the neuron oututs are. The next ste is to decide which side of each boundary should roduce a. One choice is illustrated in Figure P.5, where the shaded areas reresent oututs of. The darkest shading indicates that both neuron oututs are. Note that this solution corresonds to target values of class : t t,, class : t 3 t,, class 3: t 5 t, 6, class : t 7 t, 8. We can now select the weight vectors: -5

26 Percetron Learning Rule w 3 and w. Note that the lengths of the weight vectors is not imortant, only their directions. They must be orthogonal to the decision boundaries. Now we can calculate the bias by icking a oint on a boundary and satisfying Eq. (.5): b w T 3, b w T. 3 In matrix form we have Figure P.5 Decision Regions for Problem P.3 W wt 3 and b, wt which comletes our design. P. Solve the following classification roblem with the ercetron rule. Aly each inut vector in order, for as many reetitions as it takes to ensure that the roblem is solved. Draw a grah of the roblem only after you have found a solution. -6

27 Solved Problems, t, t 3, t 3, t Use the initial weights and bias: W( ) b( ). We start by calculating the ercetronõs outut a, using the initial weights and bias. for the first inut vector a hardlim( W( ) + b( ) ) hardlim + hardlim( ) The outut a does not equal the target value t, so we use the ercetron rule to find new weights and biases based on the error. e t a T W( ) W( ) + e + ( ) b( ) b( ) + e + ( ) We now aly the second inut vector bias., using the udated weights and a hardlim( W( ) + b( ) ) hardlim hardlim( ) This time the outut a is equal to the target t. Alication of the ercetron rule will not result in any changes. W( ) W( ) b( ) b( ) We now aly the third inut vector. -7

28 Percetron Learning Rule a hardlim( W( ) 3 + b( ) ) hardlim hardlim( ) The outut in resonse to inut vector 3 is equal to the target t 3, so there will be no changes. W( 3) W( ) b( 3) b( ) We now move on to the last inut vector. a hardlim( W( 3) + b( 3) ) hardlim hardlim( ) This time the outut a does not equal the aroriate target t. The ercetron rule will result in a new set of values for W and b. e t a T W( ) W( 3) + e + ( ) 3 b( ) b( 3) + e + We now must check the first vector again. This time the outut a is equal to the associated target. t a hardlim( W( ) + b( ) ) hardlim 3 + hardlim( 8) Therefore there are no changes. W( 5) W( ) b( 5) b( ) The second resentation of of weight and bias values. results in an error and therefore a new set -8

29 Solved Problems a hardlim( W( 5) + b( 5) ) hardlim 3 + hardlim( ) Here are those new values: e t a T W( 6) W( 5) + e 3 + ( ) 3 b( 6) b( 5) + e +. Cycling through each inut vector once more results in no errors. a hardlim( W( 6) 3 + b( 6) ) hardlim 3 + t 3 a hardlim( W( 6) + b( 6) ) hardlim 3 + t a hardlim( W( 6) + b( 6) ) hardlim 3 + t a hardlim( W( 6) + b( 6) ) hardlim 3 + t Therefore the algorithm has converged. The final solution is: W 3 b. Now we can grah the training data and the decision boundary of the solution. The decision boundary is given by n W + b w, + w, + b 3 +. To find the intercet of the decision boundary, set : b if. 3 3 w, To find the intercet, set : b if. w, -9

30 Percetron Learning Rule The resulting decision boundary is illustrated in Figure P.6. W Figure P.6 Decision Boundary for Problem P. Note that the decision boundary falls across one of the training vectors. This is accetable, given the roblem definition, since the hard limit function returns when given an inut of, and the target for the vector in question is indeed. P.5 Consider again the four-class decision roblem that we introduced in Problem P.3. Train a ercetron network to solve this roblem using the ercetron learning rule. If we use the same target vectors that we introduced in Problem P.3, the training set will be: t, t, 3 t, 3 t, LetÕs begin the algorithm with the following initial weights and biases: The first iteration is 5 t, 5 7 t, 7 W( ) 8 t, 8, b( ). 6 t, 6. a hardlim ( W( ) + b( ) ) hardlim ( + ), -3

31 Solved Problems e t a, T W( ) W( ) + e +, b( ) b( ) + e +. The second iteration is a hardlim ( W( ) + b( ) ) hardlim ( + ), e t a, T W( ) W( ) + e +, b( ) b( ) + e + The third iteration is. a hardlim ( W( ) 3 + b( ) ) hardlim ( + ), e t 3 a, T W( 3) W( ) + e 3 +, -3

32 Percetron Learning Rule b( 3) b( ) + e +. Iterations four through eight roduce no changes in the weights. W( 8) W( 7) W( 6) W( 5) W( ) W( 3) b( 8) b( 7) b( 6) b( 5) b( ) b( 3) The ninth iteration roduces a hardlim ( W( 8) + b( 8) ) hardlim ( + ), e t a, T W( 9) W( 8) + e +, b( 9) b( 8) + e +. At this oint the algorithm has converged, since all inut atterns will be correctly classified. The final decision boundaries are dislayed in Figure P.7. Comare this result with the network we designed in Problem P.3. 3 Figure P.7 Final Decision Boundaries for Problem P.5-3

33 Eilogue Eilogue In this chater we have introduced our first learning rule Ñ the ercetron learning rule. It is a tye of learning called suervised learning, in which the learning rule is rovided with a set of examles of roer network behavior. As each inut is alied to the network, the learning rule adjusts the network arameters so that the network outut will move closer to the target. The ercetron learning rule is very simle, but it is also quite owerful. We have shown that the rule will always converge to a correct solution, if such a solution exists. The weakness of the ercetron network lies not with the learning rule, but with the structure of the network. The standard ercetron is only able to classify vectors that are linearly searable. We will see in Chater that the ercetron architecture can be generalized to mutlilayer ercetrons, which can solve arbitrary classification roblems. The backroagation learning rule, which is introduced in Chater, can be used to train these networks. In Chaters 3 and we have used many concets from the field of linear algebra, such as inner roduct, rojection, distance (norm), etc. We will find in later chaters that a good foundation in linear algebra is essential to our understanding of all neural networks. In Chaters 5 and 6 we will review some of the key concets from linear algebra that will be most imortant in our study of neural networks. Our objective will be to obtain a fundamental understanding of how neural networks work. -33

34 Percetron Learning Rule Further Reading [BaSu83] [Brog9] [McPi3] [MiPa69] [Rose58] A. Barto, R. Sutton and C. Anderson, ÒNeuron-like adative elements can solve difficult learning control roblems,ó IEEE Transactions on Systems, Man and Cybernetics, Vol. 3, No. 5,. 83Ð86, 983. A classic aer in which a reinforcement learning algorithm is used to train a neural network to balance an inverted endulum. W. L. Brogan, Modern Control Theory, 3rd Ed., Englewood Cliffs, NJ: Prentice-Hall, 99. A well-written book on the subject of linear systems. The first half of the book is devoted to linear algebra. It also has good sections on the solution of linear differential equations and the stability of linear and nonlinear systems. It has many worked roblems. W. McCulloch and W. Pitts, ÒA logical calculus of the ideas immanent in nervous activity,ó Bulletin of Mathematical Biohysics, Vol. 5,. 5Ð33, 93. This article introduces the first mathematical model of a neuron, in which a weighted sum of inut signals is comared to a threshold to determine whether or not the neuron fires. M. Minsky and S. Paert, Percetrons, Cambridge, MA: MIT Press, 969. A landmark book that contains the first rigorous study devoted to determining what a ercetron network is caable of learning. A formal treatment of the ercetron was needed both to exlain the ercetronõs limitations and to indicate directions for overcoming them. Unfortunately, the book essimistically redicted that the limitations of ercetrons indicated that the field of neural networks was a dead end. Although this was not true, it temorarily cooled research and funding for research for several years. F. Rosenblatt, ÒThe ercetron: A robabilistic model for information storage and organization in the brain,ó Psychological Review, Vol. 65,. 386Ð8, 958. This aer resents the first ractical artificial neural network Ñ the ercetron. -3

35 Further Reading [Rose6] [WhSo9] F. Rosenblatt, Princiles of Neurodynamics, Washington DC: Sartan Press, 96. One of the first books on neurocomuting. D. White and D. Sofge (Eds.), Handbook of Intelligent Control, New York: Van Nostrand Reinhold, 99. Collection of articles describing current research and alications of neural networks and fuzzy logic to control systems. -35

36 Percetron Learning Rule Exercises E. Consider the classification roblem defined below:, t, t 3, t 3 5, t 5., t i. Draw a diagram of the single-neuron ercetron you would use to solve this roblem. How many inuts are required? ii. Draw a grah of the data oints, labeled according to their targets. Is this roblem solvable with the network you defined in art (i)? Why or why not? E. Consider the classification roblem defined below., t, t 3, t 3, t. i. Design a single-neuron ercetron to solve this roblem. Design the network grahically, by choosing weight vectors that are orthogonal to the decision boundaries.» + ans ii. Test your solution with all four inut vectors. iii. Classify the following inut vectors with your solution. You can either erform the calculations manually or with MATLAB iv. Which of the vectors in art (iii) will always be classified the same way, regardless of the solution values for W and b? Which may vary deending on the solution? Why? E.3 Solve the classification roblem in Exercise E. by solving inequalities (as in Problem P.), and reeat arts (ii) and (iii) with the new solution. (The solution is more difficult than Problem P., since you canõt isolate the weights and biases in a airwise manner.) -36

37 Exercises E. Solve the classification roblem in Exercise E. by alying the ercetron rule to the following initial arameters, and reeat arts (ii) and (iii) with the new solution. W( ) b( ) E.5 Prove mathematically (not grahically) that the following roblem is unsolvable for a two-inut/single-neuron ercetron., t, t 3, t 3, t (Hint: start by rewriting the inut/target requirements as inequalities that constrain the weight and bias values.) a hardlims (n) n W + b E.6 The symmetric hard limit function is sometimes used in ercetron networks, instead of the hard limit function. Target values are then taken from the set [-, ] instead of [, ]. i. Write a simle exression that mas numbers in the ordered set [, ] into the ordered set [-, ]. Write the exression that erforms the inverse maing. ii. Consider two single-neuron ercetrons with the same weight and bias values. The first network uses the hard limit function ([, ] values), and the second network uses the symmetric hard limit function. If the two networks are given the same inut, and udated with the ercetron learning rule, will their weights continue to have the same value? iii. If the changes to the weights of the two neurons are different, how do they differ? Why? iv. Given initial weight and bias values for a standard hard limit ercetron, create a method for initializing a symmetric hard limit ercetron so that the two neurons will always resond identically when trained on identical data.» + ans E.7 The vectors in the ordered set defined below were obtained by measuring the weight and ear lengths of toy rabbits and bears in the Fuzzy Wuzzy Animal Factory. The target values indicate whether the resective inut vector was taken from a rabbit () or a bear (). The first element of the inut vector is the weight of the toy, and the second element is the ear length., t, t 3, t 3, t

38 Percetron Learning Rule 3 5, t 5 3 6, t 6 7, t 7 8, t 8 i. Use MATLAB to initialize and train a network to solve this ÒracticalÓ roblem. ii. Use MATLAB to test the resulting weight and bias values against the inut vectors. iii. Alter the inut vectors to ensure that the decision boundary of any solution will not intersect one of the original inut vectors (i.e., to ensure only robust solutions are found). Then retrain the network. E.8 Consider again the four-category classification roblem described in Problems P.3 and P.5. Suose that we change the inut vector to 3. 3» + ans i. Is the roblem still linearly searable? Demonstrate your answer grahically. ii. Use MATLAB and to initialize and train a network to solve this roblem. Exlain your results. iii. If 3 is changed to 3.5 is the roblem linearly searable? iv. With the 3 from (iii), use MATLAB to initialize and train a network to solve this roblem. Exlain your results. E.9 One variation of the ercetron learning rule is W new W old + αe T b new b old + αe where α is called the learning rate. Prove convergence of this algorithm. Does the roof require a limit on the learning rate? Exlain. -38