One Solution to XOR problem using Multilayer Perceptron having Minimum Configuration

Transcription

1 International Journal of Science and Engineering Volume 3, Number PP: IJSE Available at ISSN: One Solution to XOR problem using Multilayer Perceptron having Minimum Configuration Vaibhav Kant Singh Department of Computer Science and Engineering Institute of Technology, Guru Ghasidas Vishwavidyalaya, Central University, Bilaspur, (C.G.), India 1 vibhu200427@gmail.com Abstract- Artificial Neural Network (ANN) is the branch of Computer Science which deals with the Construction of programs that are having analogy with the Biological Neural Network (BNN). There are various types of ANN systems which are used to solve variety of problems. When we will look into the history of development ANN we saw the concept of linear separabilty. The problems which are supposed to be linearly separable are solved easily my making use of Single layer perceptron model proposed by Rosenblatt. XOR problem the solution to which is discussed in this paper is a nonlinearly separable problem. The problem is a complex problem and requires new type of ANN system for its solution. In this paper we will see Architectural Graph and Signal Flow Graphs representing the ANN equivalent to Minimum Configuration Multilayer perceptron (MLP). We have utilized hyperbolic tangent function as a Activation function for Hidden Layers and Threshold Function as Activation Function for Output Layer. The learning employed is Error Correction Learning and the algorithm employed is Back propagation Algorithm (BPN). In this paper one solution is proposed for the solution of XOR problem. Keywords ANN, BNN, Activation Function, Hyperbolic Tangent Function, BPN, MLP. I. INTRODUCTION TO ARTIFICIAL NEURAL NETWORK Artificial Neural Network is a parallel and distributed processor which is simulated in a digital computer and whose working is analogous to the working of Human Brain. Humans are having nervous system that performs operation in parallel after attaining inputs from the five basic sense organs. The cells which are responsible for processing the stimulus obtained by the environment are called nerve cells or neurons. ANN resembles human brain in two aspects i.e. knowledge is acquired from the environment through an interactive process of weight change and inter-neuron connection strength i.e. synaptic weights are used to store the acquired knowledge. With every iteration of the learning process the ANN becomes more knowledgeable about the environment in which it is operating. ANN are represented using three techniques namely Block diagram representation, Signal flow graph and Architectural Graph. The basic components of a neuron are set of adjustable synaptic weigths attached to the inputs and bias, a Summing Junction and a linear or non-linear activation function. The three basic elements of any ANN are neuron, network topology and learning algorithm. The learning algorithms employed in ANN is classified into three basic types in first level i.e. Supervised learning, Reinforcement learning and Unsupervised learning. Under supervised learning comes Error Correction and Stochastic learning. Error correction is further classified into LMS and BPN. Unsupervised learning on the other hand is classified into Hebbian and Competitive learning. Some of the Neural Network Systems include SOFM (Self Organizing Feature Map), Perceptron, MLP, Neoconition, ADALINE (Adaptive Linear Neural Element), MADALINE (Multiple ADALINE), LVQ (Learning Vector Quantization), AM (Associative Memory), BAM (Bidirectional Associative Memory), Boltzmann machine, BSB (Brain-State-in-a- Box), Cauchy machines, Hopfield network, ART (Adaptive Resonance Theory), RBF (Radial Basis Function), RNN (Recurrent Neural Network) etc. II. SOLUTION OF AND, OR, NAND AND NOR GATES USING MCCULLOCH AND PITTS MODEL (1) AND GATE A.B LOGIC for AND GATE Where A and B are input values

2 One Solution to XOR problem using Multilayer Perceptron having Minimum Configuration Figure 1. Block Diagram Representing a model of single neuron in ANN displaying the solution for AND, OR, NOR, NAND GATES vk=( xi.wi+bk)=uk+bk and Since uk= xi.wi So, from the Neural Network framework we are able to analyze that there are four parameters which are required for generating output yk i.e. A, B, W1 and W2.Therefore the truth-table for the AND, OR, NAND and NOR GATE is given below:- Table -1 Truth Table of AND,OR,NAND and NOR GATE A B (AND) =. (OR) = + (NAND) =. (NOR) = In this case we will be using threshold function as the activation function. The definition of Threshold function is:- 1 if vk 2 Ψ vk = =where 2 is the Threshold value 0 if vk<2 Table -2 Derivation of Solution of AND GATE using single neuron in ANN Now, we will consider the four input patterns, Taking W1=W2=1 and B k=0 a) When, A=0, B=0, W1=1, W2=1 Since vk= xi.wi=aw1+bw2=0x1+0x1=0 Since 0<2 therefore Ψ(vk)=yk=0 Therefore, yk=0 when A=0 and B=0 b) When, A=0, B=1, W1=1, W2=1 Since vk= xi.wi=aw1+bw2=0x1+1x1=1 Since 1<2 therefore Ψ(vk)=yk=0 Therefore, yk=0 when A=0 and B=1 c) When, A=1, B=0, W1=1, W2=1 Since vk= xi.wi=aw1+bw2=1x1+0x1=1 Since 1<2 therefore Ψ(vk)=yk=0 Therefore, yk=0 when A=1 and B=0 d) When, A=1, B=1, W1=1, W2=1 Since vk= xi.wi=aw1+bw2=1x1+1x1=2 Since 2=2 therefore Ψ(vk)=yk=1 Therefore, yk=1 when A=1 and B=1 From Table-2 we are able to conclude that when W1=W2=1 and threshold value set to 2 we are able to find a solution for construction of ANN equivalent to AND GATE. (2) OR GATE A+B LOGIC for AND GATE Where A and B are input values vk=( xi.wi+bk)=uk+bk and Since uk= xi.wi In this case we will be using threshold function as the activation function. The definition of Threshold function is:-

3 IJSE,Volume 3, Number 2 V K Singh 1 if vk 1 Ψ vk = =where 1 is the Threshold value 0 if vk<1 Table -3 Derivation of Solution of OR GATE using single neuron in ANN Now, we will consider the four input patterns, Taking W1=W2=1 and B k=0 a) When, A=0, B=0, W1=1, W2=1 Since vk= xi.wi=aw1+bw2=0x1+0x1=0 Since 0<2 therefore Ψ(vk)=yk=0 Therefore, yk=0 when A=0 and B=0 b) When, A=0, B=1, W1=1, W2=1 Since vk= xi.wi=aw1+bw2=0x1+1x1=1 Since 1=1 therefore Ψ(vk)=yk=1 Therefore, yk=1 when A=0 and B=1 c) When, A=1, B=0, W1=1, W2=1 Since vk= xi.wi=aw1+bw2=1x1+0x1=1 Since 1=1 therefore Ψ(vk)=yk=1 Therefore, yk=1 when A=1 and B=0 d) When, A=1, B=1, W1=1, W2=1 Since vk= xi.wi=aw1+bw2=1x1+1x1=2 Since 2>1 therefore Ψ(vk)=yk=1 Therefore, yk=1 when A=1 and B=1 From Table-3 we are able to conclude that when W1=W2=1 and threshold value set to 1 we are able to find a solution for construction of ANN equivalent to OR GATE. (3) NAND GATE. LOGIC for NAND GATE vk=( xi.wi+bk)=uk+bk and Since uk= xi.wi Where A and B are input values In this case we will be using threshold function as the activation function. The definition of Threshold function is:- 1 if vk 1 Ψ vk = =where 1 is the Threshold value 0 if vk< 1 Table -4 Derivation of Solution of NAND GATE using single neuron in ANN Now, we will consider the four input patterns, Taking W1=W2=-1 and B k=0 a) When, A=0, B=0, W1=-1, W2=-1 Since vk= xi.wi=aw1+bw2=0x-1+0x-1=0 Since 0>-1 therefore Ψ(vk)=yk=1 Therefore, yk=1 when A=0 and B=0 b) When, A=0, B=1, W1=-1, W2=-1 Since vk= xi.wi=aw1+bw2=0x-1+1x-1=-1 Since -1=-1 therefore Ψ(vk)=yk=1 Therefore, yk=1 when A=0 and B=1 c) When, A=1, B=0, W1=1, W2=1 Since vk= xi.wi=aw1+bw2=1x-1+0x-1=-1 Since -1=-1 therefore Ψ(vk)=yk=1 Therefore, yk=1 when A=1 and B=0 d) When, A=1, B=1, W1=-1, W2=-1 Since vk= xi.wi=aw1+bw2=1x-1+1x-1=-2 Since -2<-1 therefore Ψ(vk)=yk=0 Therefore, yk=0 when A=1 and B=1 From Table-4 we are able to conclude that when W1=W2=-1 and threshold value set to -1 we are able to find a solution for construction of ANN equivalent to NAND GATE. (4) NOR GATE LOGIC + for NOR GATE vk=( xi.wi+bk)=uk+bk and Since uk= xi.wi Where A and B are input values In this case we will be using threshold function as the activation function. The definition of Threshold function is:-

4 One Solution to XOR problem using Multilayer Perceptron having Minimum Configuration 1 if vk 0 Ψ vk = =where 0 is the Threshold value 0 if vk<0 Table -5 Derivation of Solution of NOR GATE using single neuron in ANN Now, we will consider the four input patterns, Taking W1=W2=-1 and B k=0 a) When, A=0, B=0, W1=-1, W2=-1 Since vk= xi.wi=aw1+bw2=0x-1+0x-1=0 Since 0=0 therefore Ψ(vk)=yk=1 Therefore, yk=1 when A=0 and B=0 b) When, A=0, B=1, W1=-1, W2=-1 Since vk= xi.wi=aw1+bw2=0x-1+1x-1=-1 Since -1<0 therefore Ψ(vk)=yk=0 Therefore, yk=0 when A=0 and B=1 c) When, A=1, B=0, W1=-1, W2=-1 Since vk= xi.wi=aw1+bw2=1x1+0x1=-1 Since -1<0 therefore Ψ(vk)=yk=0 Therefore, yk=0 when A=1 and B=0 d) When, A=1, B=1, W1=-1, W2=-1 Since vk= xi.wi=aw1+bw2=1x-1+1x-1=-2 Since -2<0 therefore Ψ(vk)=yk=0 Therefore, yk=0 when A=1 and B=1 From Table-5 we are able to conclude that when W1=W2=-1 and threshold value set to 0 we are able to find a solution for construction of ANN equivalent to NOR GATE. III. PROBLEM STATEMENT The solutions proposed for the problems in the above section are portraying a domain which exhibits a common characteristic. The common characteristic which is exhibited is called Linear Separability. The Definition of Linear Separability is Two sets of points A and B in an n-dimensional space are called linearly separable if (n+1) real numbers w1, w2, w(n+1) exist, such that every point x1,x2.xn satisfies +1 exist, and every point x1,x2.xn satisfies < +1. Rosenblatt in 1958 proposed perceptron model for solving the problems which are linearly separable using supervised learning algorithm which was named perceptron convergence algorithm. Since XOR is a non-linearly separable problem thus require special proposal for its solution. Since, the outputs that XOR produce can t make classification of the inputs using one line in two dimensions. Table -6 Representation of the Inputs in two dimension separated into two classes on the basis of the output that it produce a)graph representing the Linear separablity in AND and NAND GATE where inputs could be classified into classes. b)graph representing the Linear separablity in OR and NOR GATE where inputs could be classified into classes. IV. LITERATURE SURVEY In [1] Abu and Jaques showed the information capacity of general form of memory is formalized. Estimation is made of the number of bits of information that can be stored in the Hopfield model of Associative Memory. In [2] Amari proposed an advance theory of learning and self-organization, covering backpropagation and its generalization as well as the formation of topological maps and neural representations of information. In [3] Akaike reviewed the classical maximum likelihood estimation procedure and a new estimate minimum information theoretical criterion (AIC) estimate (MAICE) which is designed for the purpose of statistical identification is introduced. In [4] Atiya and

5 IJSE,Volume 3, Number 2 V K Singh Abu developed a method for the storage of analog vectors i.e. vectors whose components are real valued, the method is developed for the Hopfield continuous-time network. In [5] Barron established an approximation of properties of a class of ANN. It is shown that Feedforward networks with one layer of sigmoidal non linearities achieve integrated squared errors of order O(1/n), where n is the number of nodes. In [6] Bruck showed the convergence properties of the Hopfield model are dependent on the structure of the interconnection matrix w and the method by which the nodes are updated. In [7] Freeman aim is to emplify the two nodes of information, described in the paper. In [8] the authors proposed a theoretical framework for backpropagation (BP) in order to identify some of its limitation as a general learning procedure and the reasons for its success in several experiments on pattern recognition. In [9] Giles et. al. proved that one method, recurrent cascade correlation, due to its topology has fundamental limitations in representation and there in its learning capabilities. In [10] Cardoso and Laheld introduced a class of adaptive algorithms for source separation which implements an adaptive versions of equivalent estimation and is henceforth called EASI. In [11] the authors Feldkamp and Puskorius presented a coherent neural net based framework for solving various signal processing problem. V. MULTILAYER PERCEPTRON Multilayer Perceptron as the name implies concerns with multiple layers of Neurons. Generally there are three distinguishing feature of Multilayer perceptron which are:- A. Generally the neurons present in the Neural Network are non-linear i.e. the activation function used at each neuron is generally non-linear. Sigmoid function is generally used as activation function. Logistic function or hyperbolic tangent function is used as activation function. B. The neurons present in the network offers a high degree of Connectivity. Generally the neurons present in the network are fully connected. Between the networks the input nodes may directly make connection with the output node. First hidden layer may have connection with the third hidden layer and several variations of this sort may exist between the neurons of the MLP. C. In MLP the Hidden neurons are meant to achieve higher order statistics. Either you may increase the number hidden neurons in the same layer or the number of hidden layers may be increased to transform the problem into simpler form. The learning algorithm used in the case of Multilayer perceptron is called Back Propagation Network (BPN) algorithm. BPN is a type of supervised learning algorithm. It employs error correction learning. BPN comprises of two passes in its framework. Forward pass and Backward pass. In forward pass for the current set of Input actual output is generated. Then since the learning is supervised learning and that too error correction learning, Actual output is compared with the desired output. If error is acknowledged in the forward pass. The error invokes control mechanism which will propagate weight change in the network in backward direction. The procedure continues until the system i.e. ANN learns all the patterns applicable for that domain. BPN training network requires the following steps: STEP1: Select the next training pair from the training set; apply the input vector to the network input. STEP2: Calculate the output of the network. STEP3: Calculate the error between the network output (the target vector from the training pair). STEP4: Adjust the weights of the network in a way that minimizes the error. STEP5: Repeat Steps1 through 4 for each vector in the training set until the error for the entire set is acceptably low. The Correction applied to Wji(n) is defined by the delta rule = Here, η=learning rate constant and ξ(n)=cost function or instantaneous value of error energy.

6 One Solution to XOR problem using Multilayer Perceptron having Minimum Configuration Figure 2. Architectural Graph Representing MLP having two hidden layers VI. MINIMUM CONFIGURATION MULTILAYER PERCEPTRON Figure 3. Architectural Graph Representing MLP having minimum configuration i.e. in the output layer linear activation function could be used MLP beside of having multiple layers in which every element exhibits non linearity by virtue of non linear activation function, provides a variant where there could be one or more hidden layer with Non linear element whereas in the output layer there are going to be linear elements. It means in Minimum configuration MLP in the output layer there could be one or more linear neurons.

7 IJSE,Volume 3, Number 2 V K Singh VII. FIRST SOLUTION TO THE XOR PROBLEM USING MINIMUM CONFIGURATION MULTILAYER PERCEPTRON Figure 4. Architectural Graph Representing MLP having minimum configuration for one solution to XOR problem described below Table -7 Truth Table for XOR GATE x1 x2 = In the first solution for XOR problem in the hidden layer two neurons are present in the proposed solution the activation function used is hyperbolic tangent function. In the output layer the activation function used is Threshold function. The Derivation of the Solution to the XOR problem is given below for the four possible inputs. Figure 5 is having the internal configuration of the MLP. Figure 5 is used to derive the solution. Figure 5. Signal flow graph representing First solution to XOR problem using minimum configuration MLP In the first solution for XOR problem in the hidden layer two neurons are present in the proposed solution the activation function used is hyperbolic tangent function. In the output layer the activation function used is Threshold function. The Derivation of the Solution to the XOR problem is given below for the four possible inputs. Figure 5 is having the internal configuration of the MLP. Figure 5 is used to derive the solution.

8 One Solution to XOR problem using Multilayer Perceptron having Minimum Configuration CASE 1:- When x1=0 and x2=0, At node 1 value of signal will be (1) = = 0.5..(2) At node 2 Signal value will be (3) = = 0.5..(4) Here, the activation function used for hidden layer is Hyperbolic tangent function. The Def. of which is given below:- tanh = sinh 1 = 1 cosh = + = e e h = x=induced local field value, e= The natural logarithm base also known as Euler s number & Range=[-1,+1] At node 3 the signal value will be from Eq(2and 5) = 0.5 = = = At node 4 the signal value will be from Eq (4 and 5) = 0.5 = = = At node 5 the signal value will be from Eq 6 and Eq 7= = = In the output layer the function used is Threshold function. The Definition of the threshold function is given below:- 1 h h = h h h < h h From Eq(9) and Eq(10) the value of output for the first case i.e. x1=0 and x2=0 will be = =0, h h < CASE 2:- When x1=1 and x2=0, At node 1 value of signal will be h 1 2 = = 0.5 (12) At node 2 Signal value will be , h 1 2 = = 1.5..(13) At node 3 the signal value will be from Eq(12and 5) = 0.5 = = = At node 4 the signal value will be from Eq (13 and 5) = 1.5 = = = At node 5 the signal value will be from Eq 14, Eq. 15 and Eq 8 = = From Eq(16) and Eq(10) the value of output for the second case i.e. x1=1 and x2=0 will be =1, h h >

9 IJSE,Volume 3, Number 2 V K Singh CASE 3:- When x1=0 and x2=1, At node 1 value of signal will be h 1 2 = = 1.5 (18) At node 2 Signal value will be h 1 2 = = 0.5..(19) At node 3 the signal value will be from Eq(18and 5) = 1.5 = = = At node 4 the signal value will be from Eq (19 and 5) = 0.5 = = = At node 5 the signal value will be from Eq 20, Eq 21 and Eq 8 = = From Eq(22) and Eq(10) the value of output for the third case i.e. x1=0 and x2=1 will be = =1, h h > CASE 4:- When x1=0 and x2=1, At node 1 value of signal will be = h 1 2 = = 0.5 (24) At node 2 Signal value will be , h 1 2 = = 0.5..(25) At node 3 the signal value will be from Eq(24and 5) = 0.5 = = = At node 4 the signal value will be from Eq (25 and 5) = 0.5 = = = At node 5 the signal value will be from Eq 26, Eq 27 and Eq 8 = = From Eq 28 and Eq 10 the value of the output y for input x1=1 and x2=1 will be = =0, h h < VIII.CONCLUSION From Eq. (11), Eq. (17), Eq. (23) and Eq. (29) it is concluded that the solution proposed proves that it is possible to solve XOR problem using minimum configuration Multilayer Perceptron. MLP provides very nice framework for solving problems that are specifying a non linearly separable domain. Hyperbolic function could be utilized as an activation function for training in the MLP. Hyperbolic tangent function which is a non linear function is utilized as an activation function for squashing the induced local field value or activation value produced after summation to produce output. By inclusion of hidden layer it was possible to solve problem which was complex.

10 One Solution to XOR problem using Multilayer Perceptron having Minimum Configuration REFERENCE [1] Y.S. Abu-Mostafa and J.M. St. Jacques, Information capacity of the Hopfield model, IEEE Transactions on Information Theory, vol. IT- 31, pp , [2] S. Amari, Mathematical foundations of neurocomputing, Proceeding of IEEE, vol. 78, pp , [3] H. Akaike, A new look at the statistical model identification, IEEE Transactions on Automatic Control, vol AC-19, pp , [4] A.F. Atiya and Y.S. Abu-Mostafa, An analog feedback associative memory, IEEE Transactions on Neural Networks, vol. 4, pp , [5] A.R. Barron, Universal approximation bounds for superpositions of a sigmoidal function, IEEE Transactions on Information Theory, vol. 39, pp , [6] J. Bruck, On the convergence properties of the Hopfield model, Proceedings of the IEEE, vol. 78, pp , [7] W.J. Freeman, Why neural networks don t yet fly: Inquiry into the neurodynamics of biological intelligence, IEEE International Conference on Neural Networks, vol. II, pp. 1-7, San Diego, CA, [8] M. Gori and A. Tesi, On the problem of local minima in backpropagation, IEEE Transaction Pattern Analysis and Machine Intellifgence, vol. 14, pp , [9] C.L. Giles, D. Chen, G.Z. Sun, H.H. Chen, Y.C. Lee and M.W. Goudreau, Constructive learning of recurrent neural networks: Limitations of recurrent cascade correlation with a simple solution, IEEE Transactions on Neural Networks, vol. 6, pp , [10] J.F. Cardoso and B. Laheld, Equivariant adaptive source separation, IEEE Transactions on Signal Processing, vol. 44, pp , [11] L.A. Feldkamp and G.V. Puskorius, A signal processing framework based on dynamic neural network with application to problems in adaptation, filtering and classification, Proceeding of the IEEE, vol. 86, 1998.