# ON THE K-WINNERS-TAKE-ALL NETWORK

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 634 ON THE K-WINNERS-TAKE-ALL NETWORK E. Majani Jet Propulsion Laboratory California Institute of Technology R. Erlanson, Y. Abu-Mostafa Department of Electrical Engineering California Institute of Technology ABSTRACT We present and rigorously analyze a generalization of the Winner Take-All Network: the K-Winners-Take-All Network. This network identifies the K largest of a set of N real numbers. The network model used is the continuous Hopfield model. I - INTRODUCTION The Winner-Take-All Network is a network which identifies the largest of N real numbers. Winner-Take-All Networks have been developed using various neural networks models (Grossberg-73, Lippman-87, Feldman-82, Lazzaro-89). We present here a generalization of the Winner-Take-All Network: the K-Winners-Take-All (KWTA) Network. The KWTA Network identifies the K largest of N real numbers. The neural network model we use throughout the paper is the continuous Hopfield network model (Hopfield-84). If the states of the N nodes are initialized to the N real numbers, then, if the gain of the sigmoid is large enough, the network converges to the state with K positive real numbers in the positions of the nodes with the K largest initial states, and N - K negative real numbers everywhere else. Consider the following example: N = 4, K = 2. There are 6 = (~) stable states:(++-_)t, (+_+_)T, (+--+)T, ( ++)T, (_+_+)T, and (_++_)T. If the initial state of the network is (0.3, -0.4, 0.7, O.l)T, then the network will converge to (Vi,V2,V3,v4)T where Vi> 0, V2 < 0, V3 > 0, V4 < 0 ((+ _ +_)T). In Section II, we define the KWTA Network (connection weights, external inputs). In Section III, we analyze the equilibrium states and in Section IV, we identify all the stable equilibrium states of the KWTA Network. In Section V, we describe the dynamics of the KWTA Network. In Section VI, we give two important examples of the KWTA Network and comment on an alternate implementation of the KWTA Network.

2 On the K-Winners-Take-All Network 635 II - THE K-WINNERS-TAKE-ALL NETWORK The continuous Hopfield network model (Hopfield-84) (also known as the Grossberg additive model (Grossberg-88)), is characterized by a system of first order differential equations which governs the evolution of the state of the network (i = 1,..., N) : The sigmoid function g(u) is defined by: g(u) = f(g u), where G > 0 is the gain of the sigmoid, and f(u) is defined by: 1. "f/u, 0 < f'(u) < f'(o) = 1, 2. limu... +oo f( u) = 1, 3. limu... -oo f( u) = -l. The KWTA Network is characterized by mutually inhibitory interconnections Taj = -1 for i = j, a self connection Tai = a, (Ial < 1) and'an external input (identical for every node) which depends on the number K of winners desired and the size of the network N : ti = 2K - N. The differential equations for the KWTA Network are therefore: for all i, Cd~i = -Aui + (a + l)g(ui) - (tg(uj ) - t), (1) where A = N lal, -1 < a < +1, and t = 2K - N. Let us now study the equilibrium states of the dynamical system defined in (1). We already know from previous work (Hopfield-84) that the network is guaranteed to converge to a stable equilibrium state if the connection matrix (T) is symmetric (and it is here). J=l III - EQUILIBRIUM STATES OF THE NETWORK The equilibrium states u of the KWTA network are defined by I.e., for all i, for all i, dui 0 dt -, A (E. g(u~) - (2K - N)) I a+1 I a+1 g(u'!') = --u'!' + J J Let us now develop necessary conditions for a state u to be an equilibrium state of the network. Theorem 1: For a given equilibrium state u, every component ui of u can be one of at most three distinct values. (2)

3 636 Majani, Erlanson and Abu-Mostafa Proof of Theorem 1. If we look at equation (2), we see that the last term of the righthandside expression is independent of i; let us denote this term by H(u*). Therefore, the components ut of the equilibrium state u* must be solutions of the equation: g(ui) = _A_u; + H(u*). a+1 Since the sigmoid function g(u) is monotone increasing and A/(a + 1) > 0, then the sigmoid and the line a~l ut + H(u*) intersect in at least one point and at most three (see Figure 1). Note that the constant H(u*) can be different for different equilibrium states u*. The following theorem shows that the sum of the node outputs is constrained to being close to 2K - N, as desired. Theorem 2: If u* is an equilibrium state of (1), then we have: N (a+ 1)maxg(ui) < '" g(uj~) -2K +N < (a+ 1) min g(ui). (3) u~<o L..J u'!'>o j=l Proof of Theorem 2. Let us rewrite equation (2) in the following way: Aut = (a + 1)g(ui) - (Eg(uj) - (2K - N)). Since ut and g( un are of the same sign, the term (Lj g( un - (2K - N)) can neither be too large (if ut > 0) nor too low (if ui < 0). Therefore, we must have which yields (3). { (Lj g(uj) - (2K - N)) < (a + 1)g(un, (Lj g(uj) - (2K - N)) > (a + 1)g(ut), j for all ut > 0, for all ut < 0, Theorem 1 states that the components of an equilibrium state can only be one of at most three distinct values. We will distinguish between two types of equilibrium states, for the purposes of our analysis: those which have one or more components ut such that g'( un > a~l' which we categorize as type I, and those which do not (type II). We will show in the next section that for a gain G large enough, no equilibrium state of type II is stable.

4 On the K-Winners-Take-All Network 637 IV - ASYMPTOTIC STABILITY OF EQUILIBRIUM STATES We will first derive a necessary condition for an equilibrium state of (1) to be asymptotically stable. Then we will find the stable equilibrium states of the KWTA Network. IV-I. A NECESSARY CONDITION FOR ASYMPTOTIC STABILITY An important necessary condition for asymptotic stability is given in the following theorem. Theorem 3: Given any asymptotically stable equilibrium state u*, at most one of the components ut of u* may satisfy: Proof of Theorem 3. 9 '( u *) > --. A, - a+ 1 Theorem 3 is obtained by proving the following three lemmas. Lemma 1: Given any asymptotically stable equilibrium state u*, we always have for all i and j such that i # j : g'(u~) + g'(u~) Ja 2 (g'(un - g'(ujn 2 + 4g'(ung'(uj) A> a '2 J + 2. (4) Proof of Lemma l. System (1) can be linearized around any equilibrium state u* : d(u ~ u*) ~ L(u*)(u _ u*), where L(u*) = T diag (g'(ui),...,g'( un» - AI. A necessary and sufficient condition for the asymptotic stability of u* is for L(u*) to be negative definite. A necessary condition for L(u*) to be negative definite is for all 2 X 2 matrices Lij(U*) of the type * (ag'(u~)-a -g'(u~») Lij(U ) = -g,'(ut) ag'(uj):'- A ' (i # j) to be negative definite. This results from an infinitesimal perturbation of components i and j only. Any matrix Lij (u*) has two real eigenvalues. Since the largest one has to be negative, we obtain: ~ (ag'(ui) - A + ag'(uj) - A + Ja 2 (g'(ut) - g'(ujn '(Ut)g'(Uj») < 0.

5 638 Majani, Erlan80n and Abu-Mostafa Lemma 2: Equation (4) implies: min (g'(u:),g'(u1)) < 2-1. a+ (5) Proof of Lemma 2. Consider the function h of three variables:,, * _ g'(u;) + g'(u;) va 2 (g'(u;) - g'(u;))2 + 4g'(u;)g'(uj) h (a,g (ua),g (Uj)) - a If we differentiate h with respect to its third variable g'(uj), we obtain: {)h (a, g'(ut), g'(uj)) = ~ + {)g'(uj) a2g'(uj) + (2 - a2)g'(ut) 2 2va2 (g'(un-g'(uj))2 +4g'(ung'(uj) which can be shown to be positive if and only if a > -1. But since lal < 1, then if g'(u;) < g'(uj) (without loss of generality), we have: h (a,g'(ui),g'(u1)) > h(a,g'(ui),g'(ui)) = (a+ 1)g'(ut), which, with (4), yields: which yields Lemma 2. Lemma 3: If for all i # j, 9 '( Us *) < --1' A a+ min (g'(ut),g'(u1)) < 2-1, a+ then there can be at most one ui such that: Proof of Lemma 3. A g'(u~) > --. I - a+ 1 Let us assume there exists a pair (ui, uj) with i # j such that g'( ut) > 0;1 and g'(uj) > 0;1' then (5) would be violated. I

6 On the K-Winners-Take-All Network 639 IV-2. STABLE EQUILmRIUM STATES From Theorem 3, all stable equilibrium states of type I have exactly one component, (at least one and at most one) such that g' (,) ~ 0; l' Let N + be the number of components a with g'(a) < 0;1 and a > 0, and let N_ be the number of components (3 with g'(f3) < 0;1 and f3 < 0 (note that N+ + N_ + 1 = N). For a large enough gain G, g(a) and g(f3) can be made arbitrarily close to +1 and -1 respectively. Using Theorem 2, and assuming a large enough gain, we obtain: -1 < N + - K < O. N + and K being integers, there is therefore no stable equilibrium state of type I. For the equilibrium states of type II, we have for all i, ut = a(> 0) or f3( < 0) where g'(a) < 0~1 and g'(f3) < 0;1' For a large enough gain, g(a) and g(f3) can be made arbitrarily close to +1 and -1 respectively. Using theorem 2 and assuming a large enough gain, we obtain: -(a + 1) < 2(N+ - K) < (a + 1), which yields N+ = K. Let us now summarize our results in the following theorem: Theorem 4: For a large enough gain, the only possible asymptotically stable equilibrium states u of (1) must have K components equal to a > 0 and N - K components equal to f3 < 0, with { ( ) -...L + K(g(a)-g(p)-2)+N(1+g(P» g a a 0+1, g({3) -...Lf3 + K(g(a)-g(p)-2)+N(1+g(,8» Since we are guaranteed to have at least one stable equilibrium state (Hopfield-84), and since any state whose components are a permutation of the components of a stable equilibrium state, is clearly a stable equilibrium state, then we have: Theorem 5: There exist at least (~) stable equilibrium states as defined in Theorem 4. They correspond to the (~) different states obtained by the N! permutations of one stable state with K positive components and N - K positive components. (7) v - THE DYNAMICS OF THE KWTA NETWORK Now that we know the characteristics of the stable equilibrium states of the KWTA Network, we need to show that the KWTA Network will converge to the stable state which has a > 0 in the positions of the K largest initial components. This can be seen clearly by observing that for all i ;/; j : d(u' - u ) C 'dt J =.>t(ui- uj)+(a+1)(g(ui)-g(uj». If at some time T, ui(t) = uj(t), then one can show that Vt, Ui(t) = Uj(t). Therefore, for all i ;/; j, Ui(t) - Uj(t) always keeps the same sign. This leads to the following theorem.

7 640 Majani, Erlan80n and Abu-Mostafa Theorem 6: (Preservation of order) For all nodes i # j, We shall now summarize the results of the last two sections. Theorem 7: Given an initial state u-(o) and a gain G large enough, the KWTA Network will converge to a stable equilibrium state with K components equal to a positive real number (Q > 0) in the positions of the K largest initial components, and N - K components equal to a negative real number (13 < 0) in all other N - K positions. This can be derived directly from Theorems 4, 5 and 6: we know the form of all stable equilibrium states, the order of the initial node states is preserved through time, and there is guaranteed convergence to an equilibrium state. VI - DISCUSSION The well-known Winner-Take-All Network is obtained by setting K to 1. The N/2-Winners-Take-All Network, given a set gf N real numbers, identifies which numbers are above or below the mediail~ This task is slightly more complex computationally (~ O(N log(n» than that of the Winner-Take-All (~ O(N». The number of stable states is much larger, ( N) 2N N/2 ~ J21rN' i.e., asymptotically exponential in the size of the network. Although the number of connection weights is N2, there exists an alternate implementation of the KWTA Network which has O(N) connections (see Figure 2). The sum of the outputs of all nodes and the external input is computed, then negated and fed back to all the nodes. In addition, a positive self-connection (a + 1) is needed at every node. The analysis was done for a "large enough" gain G. In practice, the critical value of Gis a~i for the N/2-Winners-Take-All Network, and slightly higher for K # N/2. Also, the analysis was done for an arbitrary value of the self-connection weight a (Ial < 1). In general, if a is close to +1, this will lead to faster convergence and a smaller value of the critical gain than if a is close to -1.

8 On the K-Winners-Take-All Network 641 VII - CONCLUSION The KWTA Network lets all nodes compete until the desired number of winners (K) is obtained. The competition is ibatained by using mutual inhibition between all nodes, while the number of winners K is selected by setting all external inputs to 2K - N. This paper illustrates the capability of the continuous Hopfield Network to solve exactly an interesting decision problem, i.e., identifying the K largest of N real numbers. Acknowledgments The authors would like to thank John Hopfield and Stephen DeWeerth from the California Institute of Technology and Marvin Perlman from the Jet Propulsion Laboratory for insightful discussions about material presented in this paper. Part of the research described in this paper was performed at the Jet Propulsion Laboratory under contract with NASA. References J.A. Feldman, D.H. Ballard, "Connectionist Models and their properties," Cognitive Science, Vol. 6, pp , 1982 S. Grossberg, "Contour Enhancement, Short Term Memory, and Constancies in Reverberating Neural Networks," Studies in Applied Mathematics, Vol. LII (52), No.3, pp , September 1973 S. Grossberg, "Non-Linear Neural Networks: Principles, Mechanisms, and Architectures," Neural Networks, Vol. 1, pp , 1988 J.J. Hopfield, "Neurons with graded response have collective computational properties like those of two-state neurons," Proc. Natl. Acad. Sci. USA, Vol. 81, pp , May 1984 J. Lazzaro, S. Ryckebusch, M.A. Mahovald, C.A. Mead, "Winner-Take-All Networks of O(N) Complexity," in this volume, 1989 R.P. Lippman, B. Gold, M.L. Malpass, "A Comparison of Hamming and Hopfield Neural Nets for Pattern Classification," MIT Lincoln Lab. Tech. Rep. TR-769, 21 May 1987

9 642 Majani, Erlanson and Abu-Mostafa u,1 /, Fj gure 1; I ntersecti on of si gmoj d and line, a+1 N-2K Figure 2; An Implementation of the KWTA Network,

### Lecture 13 Linear quadratic Lyapunov theory

EE363 Winter 28-9 Lecture 13 Linear quadratic Lyapunov theory the Lyapunov equation Lyapunov stability conditions the Lyapunov operator and integral evaluating quadratic integrals analysis of ARE discrete-time

### 7 Gaussian Elimination and LU Factorization

7 Gaussian Elimination and LU Factorization In this final section on matrix factorization methods for solving Ax = b we want to take a closer look at Gaussian elimination (probably the best known method

### Classification of Cartan matrices

Chapter 7 Classification of Cartan matrices In this chapter we describe a classification of generalised Cartan matrices This classification can be compared as the rough classification of varieties in terms

### Some Problems of Second-Order Rational Difference Equations with Quadratic Terms

International Journal of Difference Equations ISSN 0973-6069, Volume 9, Number 1, pp. 11 21 (2014) http://campus.mst.edu/ijde Some Problems of Second-Order Rational Difference Equations with Quadratic

### Solving Nonlinear Equations Using Recurrent Neural Networks

Solving Nonlinear Equations Using Recurrent Neural Networks Karl Mathia and Richard Saeks, Ph.D. Accurate Automation Corporation 71 Shallowford Road Chattanooga, Tennessee 37421 Abstract A class of recurrent

### Chapter 7. Lyapunov Exponents. 7.1 Maps

Chapter 7 Lyapunov Exponents Lyapunov exponents tell us the rate of divergence of nearby trajectories a key component of chaotic dynamics. For one dimensional maps the exponent is simply the average

### Lecture 7: Finding Lyapunov Functions 1

Massachusetts Institute of Technology Department of Electrical Engineering and Computer Science 6.243j (Fall 2003): DYNAMICS OF NONLINEAR SYSTEMS by A. Megretski Lecture 7: Finding Lyapunov Functions 1

### WINNER-TAKE-ALL ABSTRACT

703 WINNER-TAKE-ALL NETWORKS OF O(N) COMPLEXITY J. Lazzaro, S. Ryckebusch, M.A. Mahowald, and C. A. Mead California Institute of Technology Pasadena, CA 91125 ABSTRACT We have designed, fabricated, and

### 2.5 Complex Eigenvalues

1 25 Complex Eigenvalues Real Canonical Form A semisimple matrix with complex conjugate eigenvalues can be diagonalized using the procedure previously described However, the eigenvectors corresponding

### Example 4.1 (nonlinear pendulum dynamics with friction) Figure 4.1: Pendulum. asin. k, a, and b. We study stability of the origin x

Lecture 4. LaSalle s Invariance Principle We begin with a motivating eample. Eample 4.1 (nonlinear pendulum dynamics with friction) Figure 4.1: Pendulum Dynamics of a pendulum with friction can be written

### Models of Cortical Maps II

CN510: Principles and Methods of Cognitive and Neural Modeling Models of Cortical Maps II Lecture 19 Instructor: Anatoli Gorchetchnikov dy dt The Network of Grossberg (1976) Ay B y f (

### About the inverse football pool problem for 9 games 1

Seventh International Workshop on Optimal Codes and Related Topics September 6-1, 013, Albena, Bulgaria pp. 15-133 About the inverse football pool problem for 9 games 1 Emil Kolev Tsonka Baicheva Institute

### Lecture 1: Schur s Unitary Triangularization Theorem

Lecture 1: Schur s Unitary Triangularization Theorem This lecture introduces the notion of unitary equivalence and presents Schur s theorem and some of its consequences It roughly corresponds to Sections

### On Neural Networks with Minimal Weights

On Neural Networks with Minimal Weights Vasken Bohossian J ehoshua Bruck California Institute of Technology Mail Code 136-93 Pasadena, CA 91125 E-mail: {vincent, bruck }«Iparadise. cal tech. edu Abstract

### A New Nature-inspired Algorithm for Load Balancing

A New Nature-inspired Algorithm for Load Balancing Xiang Feng East China University of Science and Technology Shanghai, China 200237 Email: xfeng{@ecusteducn, @cshkuhk} Francis CM Lau The University of

### THE DYING FIBONACCI TREE. 1. Introduction. Consider a tree with two types of nodes, say A and B, and the following properties:

THE DYING FIBONACCI TREE BERNHARD GITTENBERGER 1. Introduction Consider a tree with two types of nodes, say A and B, and the following properties: 1. Let the root be of type A.. Each node of type A produces

### Two classes of ternary codes and their weight distributions

Two classes of ternary codes and their weight distributions Cunsheng Ding, Torleiv Kløve, and Francesco Sica Abstract In this paper we describe two classes of ternary codes, determine their minimum weight

### Positive Semidefiniteness and Positive Definiteness of a Linear Parametric Interval Matrix

Positive Semidefiniteness and Positive Definiteness of a Linear Parametric Interval Matrix Milan Hladík Faculty of Mathematics and Physics, Charles University in Prague, Czech Republic http://kam.mff.cuni.cz/~hladik/

### Integer Factorization using the Quadratic Sieve

Integer Factorization using the Quadratic Sieve Chad Seibert* Division of Science and Mathematics University of Minnesota, Morris Morris, MN 56567 seib0060@morris.umn.edu March 16, 2011 Abstract We give

### CITY UNIVERSITY LONDON. BEng Degree in Computer Systems Engineering Part II BSc Degree in Computer Systems Engineering Part III PART 2 EXAMINATION

No: CITY UNIVERSITY LONDON BEng Degree in Computer Systems Engineering Part II BSc Degree in Computer Systems Engineering Part III PART 2 EXAMINATION ENGINEERING MATHEMATICS 2 (resit) EX2005 Date: August

### Fuzzy Differential Systems and the New Concept of Stability

Nonlinear Dynamics and Systems Theory, 1(2) (2001) 111 119 Fuzzy Differential Systems and the New Concept of Stability V. Lakshmikantham 1 and S. Leela 2 1 Department of Mathematical Sciences, Florida

### Linear Systems. Singular and Nonsingular Matrices. Find x 1, x 2, x 3 such that the following three equations hold:

Linear Systems Example: Find x, x, x such that the following three equations hold: x + x + x = 4x + x + x = x + x + x = 6 We can write this using matrix-vector notation as 4 {{ A x x x {{ x = 6 {{ b General

### The Characteristic Polynomial

Physics 116A Winter 2011 The Characteristic Polynomial 1 Coefficients of the characteristic polynomial Consider the eigenvalue problem for an n n matrix A, A v = λ v, v 0 (1) The solution to this problem

### SCORE SETS IN ORIENTED GRAPHS

Applicable Analysis and Discrete Mathematics, 2 (2008), 107 113. Available electronically at http://pefmath.etf.bg.ac.yu SCORE SETS IN ORIENTED GRAPHS S. Pirzada, T. A. Naikoo The score of a vertex v in

### Properties of BMO functions whose reciprocals are also BMO

Properties of BMO functions whose reciprocals are also BMO R. L. Johnson and C. J. Neugebauer The main result says that a non-negative BMO-function w, whose reciprocal is also in BMO, belongs to p> A p,and

### EXISTENCE AND NON-EXISTENCE RESULTS FOR A NONLINEAR HEAT EQUATION

Sixth Mississippi State Conference on Differential Equations and Computational Simulations, Electronic Journal of Differential Equations, Conference 5 (7), pp. 5 65. ISSN: 7-669. UL: http://ejde.math.txstate.edu

### Notes on Symmetric Matrices

CPSC 536N: Randomized Algorithms 2011-12 Term 2 Notes on Symmetric Matrices Prof. Nick Harvey University of British Columbia 1 Symmetric Matrices We review some basic results concerning symmetric matrices.

### The Relation between Two Present Value Formulae

James Ciecka, Gary Skoog, and Gerald Martin. 009. The Relation between Two Present Value Formulae. Journal of Legal Economics 15(): pp. 61-74. The Relation between Two Present Value Formulae James E. Ciecka,

### Lecture 12 Basic Lyapunov theory

EE363 Winter 2008-09 Lecture 12 Basic Lyapunov theory stability positive definite functions global Lyapunov stability theorems Lasalle s theorem converse Lyapunov theorems finding Lyapunov functions 12

ONDERZOEKSRAPPORT NR 8904 OPTIMAl PREMIUM CONTROl IN A NON-liFE INSURANCE BUSINESS BY M. VANDEBROEK & J. DHAENE D/1989/2376/5 1 IN A OPTIMAl PREMIUM CONTROl NON-liFE INSURANCE BUSINESS By Martina Vandebroek

### 3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field

3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field 77 3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field Overview: The antiderivative in one variable calculus is an important

### Continuity of the Perron Root

Linear and Multilinear Algebra http://dx.doi.org/10.1080/03081087.2014.934233 ArXiv: 1407.7564 (http://arxiv.org/abs/1407.7564) Continuity of the Perron Root Carl D. Meyer Department of Mathematics, North

### Fourth-Order Compact Schemes of a Heat Conduction Problem with Neumann Boundary Conditions

Fourth-Order Compact Schemes of a Heat Conduction Problem with Neumann Boundary Conditions Jennifer Zhao, 1 Weizhong Dai, Tianchan Niu 1 Department of Mathematics and Statistics, University of Michigan-Dearborn,

### 4.6 Null Space, Column Space, Row Space

NULL SPACE, COLUMN SPACE, ROW SPACE Null Space, Column Space, Row Space In applications of linear algebra, subspaces of R n typically arise in one of two situations: ) as the set of solutions of a linear

### A characterization of trace zero symmetric nonnegative 5x5 matrices

A characterization of trace zero symmetric nonnegative 5x5 matrices Oren Spector June 1, 009 Abstract The problem of determining necessary and sufficient conditions for a set of real numbers to be the

### Reaction diffusion systems and pattern formation

Chapter 5 Reaction diffusion systems and pattern formation 5.1 Reaction diffusion systems from biology In ecological problems, different species interact with each other, and in chemical reactions, different

### 1 2 3 1 1 2 x = + x 2 + x 4 1 0 1

(d) If the vector b is the sum of the four columns of A, write down the complete solution to Ax = b. 1 2 3 1 1 2 x = + x 2 + x 4 1 0 0 1 0 1 2. (11 points) This problem finds the curve y = C + D 2 t which

### SMT 2014 Algebra Test Solutions February 15, 2014

1. Alice and Bob are painting a house. If Alice and Bob do not take any breaks, they will finish painting the house in 20 hours. If, however, Bob stops painting once the house is half-finished, then the

### Recurrent Neural Networks

Recurrent Neural Networks Neural Computation : Lecture 12 John A. Bullinaria, 2015 1. Recurrent Neural Network Architectures 2. State Space Models and Dynamical Systems 3. Backpropagation Through Time

### DETERMINANTS. b 2. x 2

DETERMINANTS 1 Systems of two equations in two unknowns A system of two equations in two unknowns has the form a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 This can be written more concisely in

### Relations Graphical View

Relations Slides by Christopher M. Bourke Instructor: Berthe Y. Choueiry Introduction Recall that a relation between elements of two sets is a subset of their Cartesian product (of ordered pairs). A binary

### Note on some explicit formulae for twin prime counting function

Notes on Number Theory and Discrete Mathematics Vol. 9, 03, No., 43 48 Note on some explicit formulae for twin prime counting function Mladen Vassilev-Missana 5 V. Hugo Str., 4 Sofia, Bulgaria e-mail:

### Analysis of Algorithms I: Optimal Binary Search Trees

Analysis of Algorithms I: Optimal Binary Search Trees Xi Chen Columbia University Given a set of n keys K = {k 1,..., k n } in sorted order: k 1 < k 2 < < k n we wish to build an optimal binary search

### Iterative Techniques in Matrix Algebra. Jacobi & Gauss-Seidel Iterative Techniques II

Iterative Techniques in Matrix Algebra Jacobi & Gauss-Seidel Iterative Techniques II Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin

### LS.6 Solution Matrices

LS.6 Solution Matrices In the literature, solutions to linear systems often are expressed using square matrices rather than vectors. You need to get used to the terminology. As before, we state the definitions

### 2.1: Determinants by Cofactor Expansion. Math 214 Chapter 2 Notes and Homework. Evaluate a Determinant by Expanding by Cofactors

2.1: Determinants by Cofactor Expansion Math 214 Chapter 2 Notes and Homework Determinants The minor M ij of the entry a ij is the determinant of the submatrix obtained from deleting the i th row and the

### Applied Algorithm Design Lecture 5

Applied Algorithm Design Lecture 5 Pietro Michiardi Eurecom Pietro Michiardi (Eurecom) Applied Algorithm Design Lecture 5 1 / 86 Approximation Algorithms Pietro Michiardi (Eurecom) Applied Algorithm Design

### THREE DIMENSIONAL GEOMETRY

Chapter 8 THREE DIMENSIONAL GEOMETRY 8.1 Introduction In this chapter we present a vector algebra approach to three dimensional geometry. The aim is to present standard properties of lines and planes,

### DATA ANALYSIS II. Matrix Algorithms

DATA ANALYSIS II Matrix Algorithms Similarity Matrix Given a dataset D = {x i }, i=1,..,n consisting of n points in R d, let A denote the n n symmetric similarity matrix between the points, given as where

### Thesis Title. A. U. Thor. A.A.S., University of Southern Swampland, 1988 M.S., Art Therapy, University of New Mexico, 1991 THESIS

Thesis Title by A. U. Thor A.A.S., University of Southern Swampland, 1988 M.S., Art Therapy, University of New Mexico, 1991 THESIS Submitted in Partial Fulllment of the Requirements for the Degree of Master

### α = u v. In other words, Orthogonal Projection

Orthogonal Projection Given any nonzero vector v, it is possible to decompose an arbitrary vector u into a component that points in the direction of v and one that points in a direction orthogonal to v

### A Branch and Bound Algorithm for Solving the Binary Bi-level Linear Programming Problem

A Branch and Bound Algorithm for Solving the Binary Bi-level Linear Programming Problem John Karlof and Peter Hocking Mathematics and Statistics Department University of North Carolina Wilmington Wilmington,

### Chapter 6. Orthogonality

6.3 Orthogonal Matrices 1 Chapter 6. Orthogonality 6.3 Orthogonal Matrices Definition 6.4. An n n matrix A is orthogonal if A T A = I. Note. We will see that the columns of an orthogonal matrix must be

### 2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

### Nonlinear Systems of Ordinary Differential Equations

Differential Equations Massoud Malek Nonlinear Systems of Ordinary Differential Equations Dynamical System. A dynamical system has a state determined by a collection of real numbers, or more generally

### 1 Formulating The Low Degree Testing Problem

6.895 PCP and Hardness of Approximation MIT, Fall 2010 Lecture 5: Linearity Testing Lecturer: Dana Moshkovitz Scribe: Gregory Minton and Dana Moshkovitz In the last lecture, we proved a weak PCP Theorem,

### 1 A duality between descents and connectivity.

The Descent Set and Connectivity Set of a Permutation 1 Richard P. Stanley 2 Department of Mathematics, Massachusetts Institute of Technology Cambridge, MA 02139, USA rstan@math.mit.edu version of 16 August

### Fixed Point Theorems

Fixed Point Theorems Definition: Let X be a set and let T : X X be a function that maps X into itself. (Such a function is often called an operator, a transformation, or a transform on X, and the notation

### Brendan D. McKay Computer Science Dept., Vanderbilt University, Nashville, Tennessee 37235

139. Proc. 3rd Car. Conf. Comb. & Comp. pp. 139-143 SPANNING TREES IN RANDOM REGULAR GRAPHS Brendan D. McKay Computer Science Dept., Vanderbilt University, Nashville, Tennessee 37235 Let n~ < n2

### AN ITERATIVE METHOD FOR THE HERMITIAN-GENERALIZED HAMILTONIAN SOLUTIONS TO THE INVERSE PROBLEM AX = B WITH A SUBMATRIX CONSTRAINT

Bulletin of the Iranian Mathematical Society Vol. 39 No. 6 (2013), pp 129-1260. AN ITERATIVE METHOD FOR THE HERMITIAN-GENERALIZED HAMILTONIAN SOLUTIONS TO THE INVERSE PROBLEM AX = B WITH A SUBMATRIX CONSTRAINT

### SOLVING LINEAR SYSTEMS

SOLVING LINEAR SYSTEMS Linear systems Ax = b occur widely in applied mathematics They occur as direct formulations of real world problems; but more often, they occur as a part of the numerical analysis

### ECEN 5682 Theory and Practice of Error Control Codes

ECEN 5682 Theory and Practice of Error Control Codes Convolutional Codes University of Colorado Spring 2007 Linear (n, k) block codes take k data symbols at a time and encode them into n code symbols.

### Spectral Measure of Large Random Toeplitz Matrices

Spectral Measure of Large Random Toeplitz Matrices Yongwhan Lim June 5, 2012 Definition (Toepliz Matrix) The symmetric Toeplitz matrix is defined to be [X i j ] where 1 i, j n; that is, X 0 X 1 X 2 X n

### SHARP BOUNDS FOR THE SUM OF THE SQUARES OF THE DEGREES OF A GRAPH

31 Kragujevac J. Math. 25 (2003) 31 49. SHARP BOUNDS FOR THE SUM OF THE SQUARES OF THE DEGREES OF A GRAPH Kinkar Ch. Das Department of Mathematics, Indian Institute of Technology, Kharagpur 721302, W.B.,

### Linear Codes. In the V[n,q] setting, the terms word and vector are interchangeable.

Linear Codes Linear Codes In the V[n,q] setting, an important class of codes are the linear codes, these codes are the ones whose code words form a sub-vector space of V[n,q]. If the subspace of V[n,q]

### A MIXED TYPE IDENTIFICATION PROBLEM RELATED TO A PHASE-FIELD MODEL WITH MEMORY

Guidetti, D. and Lorenzi, A. Osaka J. Math. 44 (27), 579 613 A MIXED TYPE IDENTIFICATION PROBLEM RELATED TO A PHASE-FIELD MODEL WITH MEMORY DAVIDE GUIDETTI and ALFREDO LORENZI (Received January 23, 26,

Chapter 2 Remodulization of Congruences Proceedings NCUR VI. è1992è, Vol. II, pp. 1036í1041. Jeærey F. Gold Department of Mathematics, Department of Physics University of Utah Don H. Tucker Department

### CONTROLLABILITY. Chapter 2. 2.1 Reachable Set and Controllability. Suppose we have a linear system described by the state equation

Chapter 2 CONTROLLABILITY 2 Reachable Set and Controllability Suppose we have a linear system described by the state equation ẋ Ax + Bu (2) x() x Consider the following problem For a given vector x in

### 6.852: Distributed Algorithms Fall, 2009. Class 2

.8: Distributed Algorithms Fall, 009 Class Today s plan Leader election in a synchronous ring: Lower bound for comparison-based algorithms. Basic computation in general synchronous networks: Leader election

### 10.3 POWER METHOD FOR APPROXIMATING EIGENVALUES

58 CHAPTER NUMERICAL METHODS. POWER METHOD FOR APPROXIMATING EIGENVALUES In Chapter 7 you saw that the eigenvalues of an n n matrix A are obtained by solving its characteristic equation n c nn c nn...

### Dynamics. Figure 1: Dynamics used to generate an exemplar of the letter A. To generate

Dynamics Any physical system, such as neurons or muscles, will not respond instantaneously in time but will have a time-varying response termed the dynamics. The dynamics of neurons are an inevitable constraint

### The Inverse of a Square Matrix

These notes closely follow the presentation of the material given in David C Lay s textbook Linear Algebra and its Applications (3rd edition) These notes are intended primarily for in-class presentation

### 3. Reaction Diffusion Equations Consider the following ODE model for population growth

3. Reaction Diffusion Equations Consider the following ODE model for population growth u t a u t u t, u 0 u 0 where u t denotes the population size at time t, and a u plays the role of the population dependent

### SECTION 1.5: PIECEWISE-DEFINED FUNCTIONS; LIMITS AND CONTINUITY IN CALCULUS

(Section.5: Piecewise-Defined Functions; Limits and Continuity in Calculus).5. SECTION.5: PIECEWISE-DEFINED FUNCTIONS; LIMITS AND CONTINUITY IN CALCULUS LEARNING OBJECTIVES Know how to evaluate and graph

### The Heat Equation. Lectures INF2320 p. 1/88

The Heat Equation Lectures INF232 p. 1/88 Lectures INF232 p. 2/88 The Heat Equation We study the heat equation: u t = u xx for x (,1), t >, (1) u(,t) = u(1,t) = for t >, (2) u(x,) = f(x) for x (,1), (3)

### An example of a computable

An example of a computable absolutely normal number Verónica Becher Santiago Figueira Abstract The first example of an absolutely normal number was given by Sierpinski in 96, twenty years before the concept

### Manipulability of the Price Mechanism for Data Centers

Manipulability of the Price Mechanism for Data Centers Greg Bodwin 1, Eric Friedman 2,3,4, and Scott Shenker 3,4 1 Department of Computer Science, Tufts University, Medford, Massachusetts 02155 2 School

### Zachary Monaco Georgia College Olympic Coloring: Go For The Gold

Zachary Monaco Georgia College Olympic Coloring: Go For The Gold Coloring the vertices or edges of a graph leads to a variety of interesting applications in graph theory These applications include various

### MATH 10550, EXAM 2 SOLUTIONS. x 2 + 2xy y 2 + x = 2

MATH 10550, EXAM SOLUTIONS (1) Find an equation for the tangent line to at the point (1, ). + y y + = Solution: The equation of a line requires a point and a slope. The problem gives us the point so we

### SCHOOL DISTRICT OF THE CHATHAMS CURRICULUM PROFILE

CONTENT AREA(S): Mathematics COURSE/GRADE LEVEL(S): Honors Algebra 2 (10/11) I. Course Overview In Honors Algebra 2, the concept of mathematical function is developed and refined through the study of real

### Midterm 1. Solutions

Stony Brook University Introduction to Calculus Mathematics Department MAT 13, Fall 01 J. Viro October 17th, 01 Midterm 1. Solutions 1 (6pt). Under each picture state whether it is the graph of a function

### N E W S A N D L E T T E R S

N E W S A N D L E T T E R S 73rd Annual William Lowell Putnam Mathematical Competition Editor s Note: Additional solutions will be printed in the Monthly later in the year. PROBLEMS A1. Let d 1, d,...,

### MATH PROBLEMS, WITH SOLUTIONS

MATH PROBLEMS, WITH SOLUTIONS OVIDIU MUNTEANU These are free online notes that I wrote to assist students that wish to test their math skills with some problems that go beyond the usual curriculum. These

### Local periods and binary partial words: An algorithm

Local periods and binary partial words: An algorithm F. Blanchet-Sadri and Ajay Chriscoe Department of Mathematical Sciences University of North Carolina P.O. Box 26170 Greensboro, NC 27402 6170, USA E-mail:

### A REMARK ON ALMOST MOORE DIGRAPHS OF DEGREE THREE. 1. Introduction and Preliminaries

Acta Math. Univ. Comenianae Vol. LXVI, 2(1997), pp. 285 291 285 A REMARK ON ALMOST MOORE DIGRAPHS OF DEGREE THREE E. T. BASKORO, M. MILLER and J. ŠIRÁŇ Abstract. It is well known that Moore digraphs do

### 3.7 Non-autonomous linear systems of ODE. General theory

3.7 Non-autonomous linear systems of ODE. General theory Now I will study the ODE in the form ẋ = A(t)x + g(t), x(t) R k, A, g C(I), (3.1) where now the matrix A is time dependent and continuous on some

### 8.1 Min Degree Spanning Tree

CS880: Approximations Algorithms Scribe: Siddharth Barman Lecturer: Shuchi Chawla Topic: Min Degree Spanning Tree Date: 02/15/07 In this lecture we give a local search based algorithm for the Min Degree

### Exercises with solutions (1)

Exercises with solutions (). Investigate the relationship between independence and correlation. (a) Two random variables X and Y are said to be correlated if and only if their covariance C XY is not equal

### A Coefficient of Variation for Skewed and Heavy-Tailed Insurance Losses. Michael R. Powers[ 1 ] Temple University and Tsinghua University

A Coefficient of Variation for Skewed and Heavy-Tailed Insurance Losses Michael R. Powers[ ] Temple University and Tsinghua University Thomas Y. Powers Yale University [June 2009] Abstract We propose a

### MATH 321 EQUIVALENCE RELATIONS, WELL-DEFINEDNESS, MODULAR ARITHMETIC, AND THE RATIONAL NUMBERS

MATH 321 EQUIVALENCE RELATIONS, WELL-DEFINEDNESS, MODULAR ARITHMETIC, AND THE RATIONAL NUMBERS ALLAN YASHINSKI Abstract. We explore the notion of well-definedness when defining functions whose domain is

### DISTRIBUTIVE PROPERTIES OF ADDITION OVER MULTIPLICATION OF IDEMPOTENT MATRICES

J. Appl. Math. & Informatics Vol. 29(2011), No. 5-6, pp. 1603-1608 Website: http://www.kcam.biz DISTRIBUTIVE PROPERTIES OF ADDITION OVER MULTIPLICATION OF IDEMPOTENT MATRICES WIWAT WANICHARPICHAT Abstract.

### Taylor Series and Asymptotic Expansions

Taylor Series and Asymptotic Epansions The importance of power series as a convenient representation, as an approimation tool, as a tool for solving differential equations and so on, is pretty obvious.

### TRIPLE POSITIVE SOLUTIONS FOR BOUNDARY VALUE PROBLEM OF A NONLINEAR FRACTIONAL DIFFERENTIAL EQUATION. Communicated by Mohammad Asadzadeh

Bulletin of the Iranian Mathematical Society Vol. 33 No. 2 (27), pp -. TRIPLE POSITIVE SOLUTIONS FOR BOUNDARY VALUE PROBLEM OF A NONLINEAR FRACTIONAL DIFFERENTIAL EQUATION R. DEHGHANI AND K. GHANBARI*

### BANACH AND HILBERT SPACE REVIEW

BANACH AND HILBET SPACE EVIEW CHISTOPHE HEIL These notes will briefly review some basic concepts related to the theory of Banach and Hilbert spaces. We are not trying to give a complete development, but

### CS420/520 Algorithm Analysis Spring 2010 Lecture 04

CS420/520 Algorithm Analysis Spring 2010 Lecture 04 I. Chapter 4 Recurrences A. Introduction 1. Three steps applied at each level of a recursion a) Divide into a number of subproblems that are smaller

### A note on properties for a complementary graph and its tree graph

A note on properties for a complementary graph and its tree graph Abulimiti Yiming Department of Mathematics Institute of Mathematics & Informations Xinjiang Normal University China Masami Yasuda Department

### OPTIMAL CONTROL OF A COMMERCIAL LOAN REPAYMENT PLAN. E.V. Grigorieva. E.N. Khailov

DISCRETE AND CONTINUOUS Website: http://aimsciences.org DYNAMICAL SYSTEMS Supplement Volume 2005 pp. 345 354 OPTIMAL CONTROL OF A COMMERCIAL LOAN REPAYMENT PLAN E.V. Grigorieva Department of Mathematics