Using Neural Networks for Pattern Classification Problems

Transcription

1 Using Neural Networks for Pattern Classification Problems Converting an Image Camera captures an image Image needs to be converted to a form that can be processed by the Neural Network

2 Converting an Image Image consists of pixels Values can be assigned to color of each pixel A vector can represent the pixel values in an image If we let + represent black and represent white Converting an Image p = [.. 2

3 Neural Network Pattern Classification Problem Tank image = [. ] House image = [.] Neural Network Tank or house? Types of Neural Networks Perceptron Hebbian Adeline Multilayer with Backpropagation Radial Basis Function Network 3

4 A single neuron perceptron: Output: 2-Input Single Neuron Perceptron: Architecture p w, w,2 a = hardlims Wp + b + n a b! # " ( ) = hardlims [ w, w,2 ] p ) = hardlims w, p + w,2 + b ' ( / Symmetrical hard limiter $ * & + b, % + ( ) = -, if w,p + w,2 + b < +, if w, p + w,2 + b. p w, w,2 2-Input Single Neuron Perceptron: Example + n a b ( ) a = hardlims w, p + w,2 + b # =!, w p + w p + b <,,2 2 $ % +, w, p + w,2 + b " Example: w, = - w,2 = b = - # a =!,! p + p! < or! p + p < 2 2 $ % +,! p +! " or! p + " This separates the inputs p = [p, ] T into two categories separated by the boundary: -p + = 4

5 2-Input Single Neuron Perceptron: Decision Boundary p - + n a - (-2,) - decision boundary - p + = p (2,-) # a =!,! p + p! < or! p + p < 2 2 $ % +,! p +!" or! p + " Inputs in this region have an output of - Inputs in this region have an output of + 2-Input Single Neuron Perceptron: Weight Vector p - + n a - W = [-, ] decision boundary - p + = - p The weight vector, W, is orthogonal to the decision boundary 5

6 2-Input Single Neuron Perceptron: Weight Vector (-2,) W - decision boundary - p + = p (2,-) W points towards the class with an output of + Simple Perceptron Design The design of a simple perceptron is based upon: A single neuron divides inputs into two classifications or categories The weight vector, W, is orthogonal to the decision boundary The weight vector, W, points towards the classification corresponding to the output 6

7 Orthogonal Vectors For any hyperplane of the form: a p + a 2 + a 3 p a n p n = b the vector c*[ a, a 2,, a n ] is orthogonal to the hyperplane (where c is a constant). - p + = - * p + * = W = [ -, ] AND Gate: Description A perceptron can be used to implement most logic functions Example: Logical AND Truth table: Inputs Output 7

8 AND Gate: Architecture Input/Target pairs: '! p = $ *'! ( # &, t " = + = $ * ( # &, t ) %, " 2 = + ) %, '! p 3 = $ *'! ( # &, t " 3 = + p 4 = $ * ( # &, t ) %, " 4 =+ ) %, Two input AND p w, w,2 hardlim is used here to provide outputs of and + n a b Graphically: Inputs Output AND Gate: Graphical Description p = zero output = one output Where do we place the decision boundary? 8

9 AND Gate: Decision Boundary There are an infinite number of solutions One possible decision boundary What is the corresponding value of W? AND Gate: Weight Vector W must be orthogonal to the decision boundary W must point towards the class with an output of.5 W One possible value is [2 2].5 Output: '! a = hardlim( 2 2 # ) " [ ] p $ * & + b+ %, = hardlim { 2p b} Decision boundary 9

10 AND Gate: Bias Decision Boundary: 2p b =.5 W Passes through (.5, ).5 At (.5, ): 2(.5) + 2() + b = b = -3 AND Gate: Final Design Final Design: p Test: 2 + n a 2-3 (! a = hardlim [ 2 2] $ + ) # &' 3, * " % - = (! a = hardlim) 2 2 # * " [ ] p $ + &' 3, % - (! a = hardlim [ 2 2] $ + ) # &' 3, * " % - = a = hardlim ( [ 2 2]! $ + ) # " &' 3, * % - = a = hardlim ( [ 2 2]! $ + ) # " &' 3, * % - =

11 Perceptron Learning Rule Most real problems involve input vectors, p, that have length greater than three Images are described by vectors with s of elements Graphical approach is not feasible in dimensions higher than three An iterative approach known as the Perceptron Learning Rule is used Character Recognition Problem Given: A network has two possible inputs, x and o. These two characters are described by the 25 pixel (5 x 5) patterns shown below. x o Problem: Design a neural network using the perceptron learning rule to correctly identify these input characters.

12 Character Recognition Problem: Input Description The inputs must be described as column vectors Pixel representation: = white = black The x is represented as: [ ] T The o is represented as: [ ] T Character Recognition Problem: Output Description The output will indicate that either an x or o was received Let: = o received = x received A hard limiter will be used The inputs are divided into two classes requiring a single neuron Training set: p = [ ] T, t = = [ ] T, t 2 = 2

13 Character Recognition Problem: Network Architecture The input, p, has 25 components p w,2 w,25 5 w, + n a b hardlim is used to provide an output of or a = hardlim(wp + b) Perceptron Learning Rule: Summary Step : Initialize W and b (if non zero) to small random numbers. Ste: Apply the first input vector to the network and find the output, a. Step 3: Update W and b based on: W new = W old + (t-a)p T b new = b old + (t-a) Repeat steps 2 and 3 for all input vectors repeatedly until the targets are achieved for all inputs 3

14 Character Recognition Problem: Perceptron Learning Rule Step : Initialize W and b (if non zero) to small random numbers. Assume W = [... ] (length 25) and b = Ste: Apply the first input vector to the network p = [ ] T, t = a = hardlim(w()p + b()) = hardlim() = Step 3: Update W and b based on: W new = W old + (t-a)p T = W old + (-)p T = [ ] b new = b old + (t-a) = b old + (-) = Character Recognition Problem: Perceptron Learning Rule Ste (repeated): Apply the second input vector to the network = [ ] T, t 2 = a = hardlim(w() + b()) = Step 3 (repeated): Update W and b based on W new = W old + (t-a)p T = W old + (-) T = [ ] b new = b old + (t-a) = b old + (-) = - 4

15 Character Recognition Problem: Perceptron Learning Rule W b p t a e [ ] p [ ] - [ ] - p [ ] [ ] p Character Recognition Problem: Results After three epochs, W and b converge to: W = [ ] b = One possible solution based on the initial condition selected. Other solutions are obtained when the initial values of W and b are changed. Check the solution: a = hardlim(w*p + b) both both inputs 5

16 Character Recognition Problem: Results How does this network perform in the presence of noise? x and o with three pixel errors in each For the x with noise: a = hardlim{w*[ ] + } = For the o with noise: a = hardlim{w*[ ] + } = The network recognizes both the noisy x and o. Character Recognition Problem: Simulation Use MATLAB to perform the following simulation: Apply noisy inputs to the network with pixel errors ranging from to 25 per character and find the network output Each type of error (number of pixels) was repeated times for each character with the incorrect pixels being selected at random The network output was compared to the target in each case. The number of detection errors was tabulated. 6

17 Character Recognition Problem: Performance Results No. of Pixel Errors No. of Character Errors x o Probability of Error x o An o with pixel errors Perceptrons: Limitations Perceptrons only work for inputs that are linearly separable x x x x o o o Linearly separable x x o x o o x Not Linearly separable 7

18 Other Neural Networks How do the other types of neural networks differ from the perceptron? Topology Function Learning Rule Perceptron Problem: Part Design a neural network that can identify a tank and a house. Find W and b by hand as illustrated with the x- o example. Use the Neural Network Toolbox to find W and b Tank (t = ) House (t = ) 8

19 Perceptron Problem: Part 2 Design a neural network that can find a tank among houses and trees. Repeat the previous problem but now with a tree included. Both the house and tree have targets of zero. Tank (t = ) House (t = ) Tree (t = ) Perceptron Problem: Part 3 Design a neural network that can find a tank among houses, trees and other items. Create other images on the 9 x 9 grid. Everything other than a tank will have a target of zero. How many items can you introduce before the perceptron learning rule no longer converges? Tank (t = ) House (t = ) +???? Tree (t = ) 9

20 MATLAB: Neural Network Toolbox >> nntool MATLAB: Neural Networks Toolbox Go to MATLAB Help and review the documentation on the Neural Networks Toolbox Use the GUI interface (>> nntool) to reproduce the results you obtained for the perceptron (tank vs. house, tree, etc.) Data can be imported/exported from the workspace to the NN Tool. 2