CS 295 (UVM) The LMS Algorithm Fall / 23
|
|
|
- Elfrieda Moore
- 8 years ago
- Views:
Transcription
1 The LMS Algorithm The LMS Objective Function. Global solution. Pseudoinverse of a matrix. Optimization (learning) by gradient descent. LMS or Widrow-Hoff Algorithms. Convergence of the Batch LMS Rule. Compiled by LAT E X at :50 PM on February 7, 0. CS 95 (UVM) The LMS Algorithm Fall 0 /
2 LMS Objective Function Again we consider the problem of programming a linear threshold function x w x x w w w 0 y x n w n y D sgn.w T x C w 0 / D ( ; if w T x C w 0 > 0; ; if w T x C w 0 0: so that it agrees with a given dichotomy of m feature vectors, X m D f.x ; `/; : : : ;.x m ; `m/g: where x i R n and `i f ; Cg, for i D ; ; : : : ; m. CS 95 (UVM) The LMS Algorithm Fall 0 /
3 LMS Objective Function (cont.) Thus, given a dichotomy X m D f.x ; `/; : : : ;.x m ; `m/g; we seek a solution weight vector w and bias w 0, such that, sgn w T x i C w 0 D `i; for i D ; ; : : : ; m. Equivalently, using homogeneous coordinates (or augmented feature vectors), for i D ; ; : : : ; m, where bx i D ; x T i Using normalized coordinates, for i D ; ; : : : ; m, where bx 0 i D `ibx i. sgn bw T bx i D `i; T. sgn bw T bx 0 i D ; or, bw T bx 0 i > 0; CS 95 (UVM) The LMS Algorithm Fall 0 /
4 LMS Objective Function (cont.) Alternatively, let b R m satisfy b i > 0 for i D ; ; : : : ; m. We call b a margin vector. Frequently we will assume that b i D. Our learning criterion is certainly satisfied if bw T bx 0 i D b i for i D ; ; : : : ; m. (Note that this condition is sufficient, but not necessary.) Equivalently, the above is satisfied if the expression E.bw/ D id b i bw T bx 0 i : equals zero. The above expression we call the LMS objective function, where LMS stands for least mean square. (N.B. we could normalize the above by dividing the right side by m.) CS 95 (UVM) The LMS Algorithm Fall 0 4 /
5 LMS Objective Function: Remarks Converting the original problem of classification (satisfying a system of inequalities) into one of optimization is somewhat ad hoc. And there is no guarantee that we can find a bw? R nc that satisfies E.bw? / D 0. However, this conversion can lead to practical compromises if the original inequalities possess inconsistencies. Also, there is no unique objective function. The LMS expression, E.bw/ D id b i bw T bx 0 i : can be replaced by numerous candidates, e.g. ˇ ˇbi ˇ bw T bx 0 i ˇ; sgn bw T bx 0 i ; sgn bw T bx 0 i ; etc. id id id However, minimizing E.bw? / is generally an easy task. CS 95 (UVM) The LMS Algorithm Fall 0 5 /
6 Minimizing the LMS Objective Function Inspection suggests that the LMS Objective Function E.bw/ D b i bw T bx 0 i id describes a parabolic function. It may have a unique global minimum, or an infinite number of global minima which occupy a connected linear set. (The latter can occur if m < n C.) Letting, 0 bx 0T 0 ` Ox 0 bx 0T ; Ox 0 ; Ox 0 ;n X D B : A D ` Ox 0 ; Ox 0 ; Ox 0 ;n : : : : :: C : A Rm.nC/ ; then, E.bw/ D id b i bx 0T m bx 0T i bw D `m Ox 0 m; Ox 0 m; Ox 0 m;n 0 b b b m bx 0T bw bx 0T bw C D b : A bx 0T m bw 0 bx 0T bx 0T B C bw D kb Xbwj : A bx 0T m CS 95 (UVM) The LMS Algorithm Fall 0 6 /
7 Minimizing the LMS Objective Function (cont.) As an aside, note that E.bw/ D kb Xbwk X T X D.bx 0 ; : : : ;bx 0 B D.b Xbw/ T.b Xbw/ D b T bw T X T.b Xbw/ D bw T X T X bw bw T X T b b T X bw C b T b D bw T X T X bw b T X bw C kbk 0 0 bx 0T : : bx 0T m bx 0T b T B X D.b ; : : : ; b m : bx 0T m C A D C A D id id bx 0 ibx 0T i R.nC/.nC/ ; b i bx 0T i R nc : CS 95 (UVM) The LMS Algorithm Fall 0 7 /
8 Minimizing the LMS Objective Function (cont.) Okay, so how do we minimize E.bw/ D bw T X T X bw b T X bw C kbk Using calculus (e.g., Math ), we can compute the gradient of E.bw/, and algebraically determine a value of bw? which makes each component vanish. That is, re.bw/ D C D 0: C A It is straightforward to show n re.bw/ D X T Xbw X T b: CS 95 (UVM) The LMS Algorithm Fall 0 8 /
9 Minimizing the LMS Objective Function (cont.) Thus, if re.bw/ D X T Xbw X T b D 0 bw? D.X T X/ X T b D X b; where the matrix, X D def.x T X/ X T R.nC/m is called the pseudoinverse of X. If X T X is singular, one defines Observe that if X T X is nonsingular, X D def lim.x T X C I/ X T :!0 X X D.X T X/ X T X D I: CS 95 (UVM) The LMS Algorithm Fall 0 9 /
10 Example The following example appears in R. O. Duda, P. E. Hart, and D. G. Stork, Pattern Classification, Second Edition, Wiley, NY, 00, p. 4. Given the dichotomy, X 4 D.; / T ; ;.; 0/ T ; ;.; / T ; ;.; / T ; we obtain, Whence, 0 X D bx 0T bx 0T bx 0T bx 0T 4 0 D B 0 C A : A X T X 8 8 A ; and, X D.X T X/ X T D C A : CS 95 (UVM) The LMS Algorithm Fall 0 0 /
11 Example (cont.) Letting, b D.; ; ; / T, then bw D X b D C A B A D 4 C A : Whence, x 5 w 0 D and w D T 4 ; 4 4 x CS 95 (UVM) The LMS Algorithm Fall 0 /
12 Method of Steepest Descent An alternative approach is the method of steepest descent. We begin by representing Taylor s theorem for functions of more than one variable: let x R n, and f W R n! R, so Now let ıx R n, and consider Define F W R! R, such that, f.x/ D f.x ; x ; : : : ; x n / R: f.x C ıx/ D f.x C ıx ; : : : ; x n C ıx n /: F.s/ D f.x C s ıx/: Thus, F.0/ D f.x/; and, F./ D f.x C ıx/: CS 95 (UVM) The LMS Algorithm Fall 0 /
13 Method of Steepest Descent (cont.) Taylor s theorem for a single variable (Math /), F.s/ D F.0/ C Š F 0.0/s C Š F 00.0/s C Š F 000.0/s C : Our plan is to set s D and replace F./ by f.x C ıx/, F.0/ by f.x/, etc. To evaluate F 0.0/ we will invoke the multivariate chain rule, e.g., d ds f u.s/; v/ v/ v 0.s/: Thus, F 0.s/ D df df.s/ D ds ds.x C s ıx ; : : : ; x n C s ıx n C s ıx/ d ds.x C s ıx / n.x C s ıx/ d ds.x n C s ıx n C s ıx/ ıx n.x C s ıx/ ıx n : CS 95 (UVM) The LMS Algorithm Fall 0 /
14 Method of Steepest Descent (cont.) Thus, F ıx n.x/ ıx n D rf.x/ T ıx: CS 95 (UVM) The LMS Algorithm Fall 0 4 /
15 Method of Steepest Descent (cont.) Thus, it is possible to show f.x C ıx/ D f.x/ C rf.x/ T ıx C O kıxk D f.x/ C krf.x/kkıxk cos C O kıxk ; where defines the angle between rf.x/ and ıx. If kıxk, then ıf D f.x C ıx/ f.x/ krf.x/kkıxk cos : Thus, the greatest reduction ıf occurs if cos D, that is if ıx D rf, where > 0. We thus seek a local minimum of the LMS objective function by taking a sequence of steps bw.t C / D bw.t/ re bw.t/ : CS 95 (UVM) The LMS Algorithm Fall 0 5 /
16 Training an LTU using Steepest Descent We now return to our original problem. Given a dichotomy X m D f.x ; `/; : : : ;.x m ; `m/g of m feature vectors x i R n with `i f ; g for i D ; : : : ; m, we construct the set of normalized, augmented feature vectors bx 0 m D f.`i; `ix i; ; : : : ; `ix i;n / T R nc ji D ; : : : ; mg: Given a margin vector, b R m, with b i > 0 for i D ; : : : ; m, we construct the LMS objective function, E.bw/ D.bw T bx 0 i b i / D kxbw bk ; id (the factor of is added with some foresight), and then evaluate its gradient re.bw/ D.bw T bx 0 i b i /bx 0 i D X T.Xbw b/: id CS 95 (UVM) The LMS Algorithm Fall 0 6 /
17 Training an LTU using Steepest Descent (cont.) Substitution into the steepest descent update rule, yields the batch LMS update rule, bw.t C / D bw.t/ re bw.t/ ; bw.t C / D bw.t/ C id b i bw.t/ T bx 0 i bx 0 i: Alternatively, one can abstract the sequential LMS, or Widrow-Hoff rule, from the above: bw.t C / D bw.t/ C b bw.t/ T bx 0.t/ bx 0.t/: where bx 0.t/ bx 0 m is the element of the dichotomy that is presented to the LTU at epoch t. (Here, we assume that b is fixed; otherwise, replace it by b.t/.) CS 95 (UVM) The LMS Algorithm Fall 0 7 /
18 Sequential LMS Rule The sequential LMS rule h bw.t C / D bw.t/ C b i bw.t/ T bx 0.t/ bx 0.t/; resembles the sequential perceptron rule, bw.t C / D bw.t/ C h sgn bw.t/ T bx 0.t/ i bx 0.t/: Sequential rules are well suited to real-time implementations, as only the current values for the weights, i.e. the configuration of the LTU itself, need to be stored. They also work with dichotomies of infinite sets. CS 95 (UVM) The LMS Algorithm Fall 0 8 /
19 Convergence of the batch LMS rule Recall, the LMS objective function in the form E.bw/ D Xbw b ; has as its gradient, re.bw/ D X T Xbw X T b D X T.Xbw b/: Substitution into the rule of steepest descent, bw.t C / D bw.t/ re bw.t/ ; yields, bw.t C / D bw.t/ X T Xbw.t/ b CS 95 (UVM) The LMS Algorithm Fall 0 9 /
20 Convergence of the batch LMS rule (cont.) The algorithm is said to converge to a fixed point bw,? if for every finite initial value bw.0/ <, lim bw.t/ D t! bw? : The fixed points bw? satisfy re.bw? / D 0, whence X T Xbw? D X T b: The update rule becomes, bw.t C / D bw.t/ X T Xbw.t/ b D bw.t/ X T X bw.t/ bw? Let ıbw.t/ D def bw.t/ bw?. Then, ıbw.t C / D ıbw.t/ X T Xıbw.t/ D I X T X/ıbw.t/: CS 95 (UVM) The LMS Algorithm Fall 0 0 /
21 Convergence of the batch LMS rule (cont.) Convergence bw.t/! bw? occurs if ıbw.t/ D bw.t/ bw?! 0. Thus we require that ıbw.t C / < ıbw.t/. Inspecting the update rule, ıbw.t C / D I X T X ıbw.t/; this reduces to the condition that all the eigenvalues of I X T X have magnitudes less than. We will now evaluate the eigenvalues of the above matrix. CS 95 (UVM) The LMS Algorithm Fall 0 /
22 Convergence of the batch LMS rule (cont.) Let S R.nC/.nC/ denote the similarity transform that reduces the symmetric matrix X T X to a diagonal matrix ƒ R.nC/.nC/. Thus, S T S D SS T D I, and S X T X S T D ƒ D diag. 0 ; ; : : : ; n /; The numbers, 0 ; ; : : : ; n, represent the eigenvalues of X T X. Note that 0 i for i D 0; ; : : : ; n. Thus, S ıbw.t C / D S.I D.I X T X/S T S ıbw.t/; ƒ/s ıbw.t/: Thus convergence occurs if ks ıbw.t C /k < ks ıbw.t/k; which occurs if the eigenvalues of I ƒ all have magnitudes less than one. CS 95 (UVM) The LMS Algorithm Fall 0 /
23 Convergence of the batch LMS rule (cont.) The eigenvalues of I ƒ equal i, for i D 0; ; : : : ; n. (These are in fact the eigenvalues of I X T X.) Thus, we require that < i < ; or 0 < i < ; for all i. Let max D max i denote the largest eigenvalue of X T X, then convergence 0in requires that 0 < < : max CS 95 (UVM) The LMS Algorithm Fall 0 /
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION
PATTERN RECOGNITION AND MACHINE LEARNING CHAPTER 4: LINEAR MODELS FOR CLASSIFICATION Introduction In the previous chapter, we explored a class of regression models having particularly simple analytical
Math 312 Homework 1 Solutions
Math 31 Homework 1 Solutions Last modified: July 15, 01 This homework is due on Thursday, July 1th, 01 at 1:10pm Please turn it in during class, or in my mailbox in the main math office (next to 4W1) Please
(Quasi-)Newton methods
(Quasi-)Newton methods 1 Introduction 1.1 Newton method Newton method is a method to find the zeros of a differentiable non-linear function g, x such that g(x) = 0, where g : R n R n. Given a starting
MATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1.
MATH10212 Linear Algebra Textbook: D. Poole, Linear Algebra: A Modern Introduction. Thompson, 2006. ISBN 0-534-40596-7. Systems of Linear Equations Definition. An n-dimensional vector is a row or a column
Solving Linear Systems, Continued and The Inverse of a Matrix
, Continued and The of a Matrix Calculus III Summer 2013, Session II Monday, July 15, 2013 Agenda 1. The rank of a matrix 2. The inverse of a square matrix Gaussian Gaussian solves a linear system by reducing
MATH 304 Linear Algebra Lecture 20: Inner product spaces. Orthogonal sets.
MATH 304 Linear Algebra Lecture 20: Inner product spaces. Orthogonal sets. Norm The notion of norm generalizes the notion of length of a vector in R n. Definition. Let V be a vector space. A function α
Statistical Machine Learning
Statistical Machine Learning UoC Stats 37700, Winter quarter Lecture 4: classical linear and quadratic discriminants. 1 / 25 Linear separation For two classes in R d : simple idea: separate the classes
Linear smoother. ŷ = S y. where s ij = s ij (x) e.g. s ij = diag(l i (x)) To go the other way, you need to diagonalize S
Linear smoother ŷ = S y where s ij = s ij (x) e.g. s ij = diag(l i (x)) To go the other way, you need to diagonalize S 2 Online Learning: LMS and Perceptrons Partially adapted from slides by Ryan Gabbard
A Stochastic 3MG Algorithm with Application to 2D Filter Identification
A Stochastic 3MG Algorithm with Application to 2D Filter Identification Emilie Chouzenoux 1, Jean-Christophe Pesquet 1, and Anisia Florescu 2 1 Laboratoire d Informatique Gaspard Monge - CNRS Univ. Paris-Est,
Linear algebra and the geometry of quadratic equations. Similarity transformations and orthogonal matrices
MATH 30 Differential Equations Spring 006 Linear algebra and the geometry of quadratic equations Similarity transformations and orthogonal matrices First, some things to recall from linear algebra Two
3. INNER PRODUCT SPACES
. INNER PRODUCT SPACES.. Definition So far we have studied abstract vector spaces. These are a generalisation of the geometric spaces R and R. But these have more structure than just that of a vector space.
Numerisches Rechnen. (für Informatiker) M. Grepl J. Berger & J.T. Frings. Institut für Geometrie und Praktische Mathematik RWTH Aachen
(für Informatiker) M. Grepl J. Berger & J.T. Frings Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2010/11 Problem Statement Unconstrained Optimality Conditions Constrained
MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors. Jordan canonical form (continued).
MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors Jordan canonical form (continued) Jordan canonical form A Jordan block is a square matrix of the form λ 1 0 0 0 0 λ 1 0 0 0 0 λ 0 0 J = 0
The Singular Value Decomposition in Symmetric (Löwdin) Orthogonalization and Data Compression
The Singular Value Decomposition in Symmetric (Löwdin) Orthogonalization and Data Compression The SVD is the most generally applicable of the orthogonal-diagonal-orthogonal type matrix decompositions Every
F Matrix Calculus F 1
F Matrix Calculus F 1 Appendix F: MATRIX CALCULUS TABLE OF CONTENTS Page F1 Introduction F 3 F2 The Derivatives of Vector Functions F 3 F21 Derivative of Vector with Respect to Vector F 3 F22 Derivative
Linear Threshold Units
Linear Threshold Units w x hx (... w n x n w We assume that each feature x j and each weight w j is a real number (we will relax this later) We will study three different algorithms for learning linear
Introduction to the Finite Element Method (FEM)
Introduction to the Finite Element Method (FEM) ecture First and Second Order One Dimensional Shape Functions Dr. J. Dean Discretisation Consider the temperature distribution along the one-dimensional
Notes on metric spaces
Notes on metric spaces 1 Introduction The purpose of these notes is to quickly review some of the basic concepts from Real Analysis, Metric Spaces and some related results that will be used in this course.
Using row reduction to calculate the inverse and the determinant of a square matrix
Using row reduction to calculate the inverse and the determinant of a square matrix Notes for MATH 0290 Honors by Prof. Anna Vainchtein 1 Inverse of a square matrix An n n square matrix A is called invertible
Inner Product Spaces
Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and
Solving Systems of Linear Equations
LECTURE 5 Solving Systems of Linear Equations Recall that we introduced the notion of matrices as a way of standardizing the expression of systems of linear equations In today s lecture I shall show how
Chapter 7 Nonlinear Systems
Chapter 7 Nonlinear Systems Nonlinear systems in R n : X = B x. x n X = F (t; X) F (t; x ; :::; x n ) B C A ; F (t; X) =. F n (t; x ; :::; x n ) When F (t; X) = F (X) is independent of t; it is an example
Orthogonal Diagonalization of Symmetric Matrices
MATH10212 Linear Algebra Brief lecture notes 57 Gram Schmidt Process enables us to find an orthogonal basis of a subspace. Let u 1,..., u k be a basis of a subspace V of R n. We begin the process of finding
PUTNAM TRAINING POLYNOMIALS. Exercises 1. Find a polynomial with integral coefficients whose zeros include 2 + 5.
PUTNAM TRAINING POLYNOMIALS (Last updated: November 17, 2015) Remark. This is a list of exercises on polynomials. Miguel A. Lerma Exercises 1. Find a polynomial with integral coefficients whose zeros include
Applications to Data Smoothing and Image Processing I
Applications to Data Smoothing and Image Processing I MA 348 Kurt Bryan Signals and Images Let t denote time and consider a signal a(t) on some time interval, say t. We ll assume that the signal a(t) is
t := maxγ ν subject to ν {0,1,2,...} and f(x c +γ ν d) f(x c )+cγ ν f (x c ;d).
1. Line Search Methods Let f : R n R be given and suppose that x c is our current best estimate of a solution to P min x R nf(x). A standard method for improving the estimate x c is to choose a direction
Inner Product Spaces and Orthogonality
Inner Product Spaces and Orthogonality week 3-4 Fall 2006 Dot product of R n The inner product or dot product of R n is a function, defined by u, v a b + a 2 b 2 + + a n b n for u a, a 2,, a n T, v b,
3 Orthogonal Vectors and Matrices
3 Orthogonal Vectors and Matrices The linear algebra portion of this course focuses on three matrix factorizations: QR factorization, singular valued decomposition (SVD), and LU factorization The first
CS 688 Pattern Recognition Lecture 4. Linear Models for Classification
CS 688 Pattern Recognition Lecture 4 Linear Models for Classification Probabilistic generative models Probabilistic discriminative models 1 Generative Approach ( x ) p C k p( C k ) Ck p ( ) ( x Ck ) p(
Nonlinear Algebraic Equations Example
Nonlinear Algebraic Equations Example Continuous Stirred Tank Reactor (CSTR). Look for steady state concentrations & temperature. s r (in) p,i (in) i In: N spieces with concentrations c, heat capacities
SOLUTIONS. f x = 6x 2 6xy 24x, f y = 3x 2 6y. To find the critical points, we solve
SOLUTIONS Problem. Find the critical points of the function f(x, y = 2x 3 3x 2 y 2x 2 3y 2 and determine their type i.e. local min/local max/saddle point. Are there any global min/max? Partial derivatives
DATA ANALYSIS II. Matrix Algorithms
DATA ANALYSIS II Matrix Algorithms Similarity Matrix Given a dataset D = {x i }, i=1,..,n consisting of n points in R d, let A denote the n n symmetric similarity matrix between the points, given as where
Continuity of the Perron Root
Linear and Multilinear Algebra http://dx.doi.org/10.1080/03081087.2014.934233 ArXiv: 1407.7564 (http://arxiv.org/abs/1407.7564) Continuity of the Perron Root Carl D. Meyer Department of Mathematics, North
Taylor and Maclaurin Series
Taylor and Maclaurin Series In the preceding section we were able to find power series representations for a certain restricted class of functions. Here we investigate more general problems: Which functions
Introduction to Algebraic Geometry. Bézout s Theorem and Inflection Points
Introduction to Algebraic Geometry Bézout s Theorem and Inflection Points 1. The resultant. Let K be a field. Then the polynomial ring K[x] is a unique factorisation domain (UFD). Another example of a
Lecture Notes to Accompany. Scientific Computing An Introductory Survey. by Michael T. Heath. Chapter 10
Lecture Notes to Accompany Scientific Computing An Introductory Survey Second Edition by Michael T. Heath Chapter 10 Boundary Value Problems for Ordinary Differential Equations Copyright c 2001. Reproduction
Logistic Regression. Vibhav Gogate The University of Texas at Dallas. Some Slides from Carlos Guestrin, Luke Zettlemoyer and Dan Weld.
Logistic Regression Vibhav Gogate The University of Texas at Dallas Some Slides from Carlos Guestrin, Luke Zettlemoyer and Dan Weld. Generative vs. Discriminative Classifiers Want to Learn: h:x Y X features
9 Multiplication of Vectors: The Scalar or Dot Product
Arkansas Tech University MATH 934: Calculus III Dr. Marcel B Finan 9 Multiplication of Vectors: The Scalar or Dot Product Up to this point we have defined what vectors are and discussed basic notation
Bindel, Spring 2012 Intro to Scientific Computing (CS 3220) Week 3: Wednesday, Feb 8
Spaces and bases Week 3: Wednesday, Feb 8 I have two favorite vector spaces 1 : R n and the space P d of polynomials of degree at most d. For R n, we have a canonical basis: R n = span{e 1, e 2,..., e
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a
Continued Fractions and the Euclidean Algorithm
Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction
Systems of Linear Equations
Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and
ASEN 3112 - Structures. MDOF Dynamic Systems. ASEN 3112 Lecture 1 Slide 1
19 MDOF Dynamic Systems ASEN 3112 Lecture 1 Slide 1 A Two-DOF Mass-Spring-Dashpot Dynamic System Consider the lumped-parameter, mass-spring-dashpot dynamic system shown in the Figure. It has two point
1 Introduction to Matrices
1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m
MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +
HOMEWORK 5 SOLUTIONS. n!f n (1) lim. ln x n! + xn x. 1 = G n 1 (x). (2) k + 1 n. (n 1)!
Math 7 Fall 205 HOMEWORK 5 SOLUTIONS Problem. 2008 B2 Let F 0 x = ln x. For n 0 and x > 0, let F n+ x = 0 F ntdt. Evaluate n!f n lim n ln n. By directly computing F n x for small n s, we obtain the following
1.5. Factorisation. Introduction. Prerequisites. Learning Outcomes. Learning Style
Factorisation 1.5 Introduction In Block 4 we showed the way in which brackets were removed from algebraic expressions. Factorisation, which can be considered as the reverse of this process, is dealt with
Single item inventory control under periodic review and a minimum order quantity
Single item inventory control under periodic review and a minimum order quantity G. P. Kiesmüller, A.G. de Kok, S. Dabia Faculty of Technology Management, Technische Universiteit Eindhoven, P.O. Box 513,
Math 120 Final Exam Practice Problems, Form: A
Math 120 Final Exam Practice Problems, Form: A Name: While every attempt was made to be complete in the types of problems given below, we make no guarantees about the completeness of the problems. Specifically,
Numerical Analysis Lecture Notes
Numerical Analysis Lecture Notes Peter J. Olver 5. Inner Products and Norms The norm of a vector is a measure of its size. Besides the familiar Euclidean norm based on the dot product, there are a number
December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B. KITCHENS
December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B KITCHENS The equation 1 Lines in two-dimensional space (1) 2x y = 3 describes a line in two-dimensional space The coefficients of x and y in the equation
The dynamic equation for the angular motion of the wheel is R w F t R w F w ]/ J w
![The dynamic equation for the angular motion of the wheel is R w F t R w F w ]/ J w](/thumbs/25/5134815.jpg "The dynamic equation for the angular motion of the wheel is R w F t R w F w ]/ J w")
Chapter 4 Vehicle Dynamics 4.. Introduction In order to design a controller, a good representative model of the system is needed. A vehicle mathematical model, which is appropriate for both acceleration
Chapter 4: Artificial Neural Networks
Chapter 4: Artificial Neural Networks CS 536: Machine Learning Littman (Wu, TA) Administration icml-03: instructional Conference on Machine Learning http://www.cs.rutgers.edu/~mlittman/courses/ml03/icml03/
Preliminary Version: December 1998
On the Number of Prime Numbers less than a Given Quantity. (Ueber die Anzahl der Primzahlen unter einer gegebenen Grösse.) Bernhard Riemann [Monatsberichte der Berliner Akademie, November 859.] Translated
Factorization Theorems
Chapter 7 Factorization Theorems This chapter highlights a few of the many factorization theorems for matrices While some factorization results are relatively direct, others are iterative While some factorization
Methods for Finding Bases
Methods for Finding Bases Bases for the subspaces of a matrix Row-reduction methods can be used to find bases. Let us now look at an example illustrating how to obtain bases for the row space, null space,
Row Echelon Form and Reduced Row Echelon Form
These notes closely follow the presentation of the material given in David C Lay s textbook Linear Algebra and its Applications (3rd edition) These notes are intended primarily for in-class presentation
FACTORING POLYNOMIALS IN THE RING OF FORMAL POWER SERIES OVER Z
FACTORING POLYNOMIALS IN THE RING OF FORMAL POWER SERIES OVER Z DANIEL BIRMAJER, JUAN B GIL, AND MICHAEL WEINER Abstract We consider polynomials with integer coefficients and discuss their factorization
Matrices and Linear Algebra
Chapter 2 Matrices and Linear Algebra 2. Basics Definition 2... A matrix is an m n array of scalars from a given field F. The individual values in the matrix are called entries. Examples. 2 3 A = 2 4 2
Table 1: Summary of the settings and parameters employed by the additive PA algorithm for classification, regression, and uniclass.
Online Passive-Aggressive Algorithms Koby Crammer Ofer Dekel Shai Shalev-Shwartz Yoram Singer School of Computer Science & Engineering The Hebrew University, Jerusalem 91904, Israel {kobics,oferd,shais,singer}@cs.huji.ac.il
Vector and Matrix Norms
Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty
Høgskolen i Narvik Sivilingeniørutdanningen STE6237 ELEMENTMETODER. Oppgaver
Høgskolen i Narvik Sivilingeniørutdanningen STE637 ELEMENTMETODER Oppgaver Klasse: 4.ID, 4.IT Ekstern Professor: Gregory A. Chechkin e-mail: chechkin@mech.math.msu.su Narvik 6 PART I Task. Consider two-point
MATH 304 Linear Algebra Lecture 9: Subspaces of vector spaces (continued). Span. Spanning set.
MATH 304 Linear Algebra Lecture 9: Subspaces of vector spaces (continued). Span. Spanning set. Vector space A vector space is a set V equipped with two operations, addition V V (x,y) x + y V and scalar
Mean value theorem, Taylors Theorem, Maxima and Minima.
MA 001 Preparatory Mathematics I. Complex numbers as ordered pairs. Argand s diagram. Triangle inequality. De Moivre s Theorem. Algebra: Quadratic equations and express-ions. Permutations and Combinations.
Direct Methods for Solving Linear Systems. Matrix Factorization
Direct Methods for Solving Linear Systems Matrix Factorization Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University c 2011
SF2940: Probability theory Lecture 8: Multivariate Normal Distribution
SF2940: Probability theory Lecture 8: Multivariate Normal Distribution Timo Koski 24.09.2015 Timo Koski Matematisk statistik 24.09.2015 1 / 1 Learning outcomes Random vectors, mean vector, covariance matrix,
Background 2. Lecture 2 1. The Least Mean Square (LMS) algorithm 4. The Least Mean Square (LMS) algorithm 3. br(n) = u(n)u H (n) bp(n) = u(n)d (n)
Lecture 2 1 During this lecture you will learn about The Least Mean Squares algorithm (LMS) Convergence analysis of the LMS Equalizer (Kanalutjämnare) Background 2 The method of the Steepest descent that
Lecture 6. Artificial Neural Networks
Lecture 6 Artificial Neural Networks 1 1 Artificial Neural Networks In this note we provide an overview of the key concepts that have led to the emergence of Artificial Neural Networks as a major paradigm
1 Sets and Set Notation.
LINEAR ALGEBRA MATH 27.6 SPRING 23 (COHEN) LECTURE NOTES Sets and Set Notation. Definition (Naive Definition of a Set). A set is any collection of objects, called the elements of that set. We will most
Solutions for Review Problems
olutions for Review Problems 1. Let be the triangle with vertices A (,, ), B (4,, 1) and C (,, 1). (a) Find the cosine of the angle BAC at vertex A. (b) Find the area of the triangle ABC. (c) Find a vector
15.062 Data Mining: Algorithms and Applications Matrix Math Review
.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop
Notes on Symmetric Matrices
CPSC 536N: Randomized Algorithms 2011-12 Term 2 Notes on Symmetric Matrices Prof. Nick Harvey University of British Columbia 1 Symmetric Matrices We review some basic results concerning symmetric matrices.
CONTROLLABILITY. Chapter 2. 2.1 Reachable Set and Controllability. Suppose we have a linear system described by the state equation
Chapter 2 CONTROLLABILITY 2 Reachable Set and Controllability Suppose we have a linear system described by the state equation ẋ Ax + Bu (2) x() x Consider the following problem For a given vector x in
Examination paper for TMA4205 Numerical Linear Algebra
Department of Mathematical Sciences Examination paper for TMA4205 Numerical Linear Algebra Academic contact during examination: Markus Grasmair Phone: 97580435 Examination date: December 16, 2015 Examination
Dot product and vector projections (Sect. 12.3) There are two main ways to introduce the dot product
Dot product and vector projections (Sect. 12.3) Two definitions for the dot product. Geometric definition of dot product. Orthogonal vectors. Dot product and orthogonal projections. Properties of the dot
Elasticity Theory Basics
G22.3033-002: Topics in Computer Graphics: Lecture #7 Geometric Modeling New York University Elasticity Theory Basics Lecture #7: 20 October 2003 Lecturer: Denis Zorin Scribe: Adrian Secord, Yotam Gingold
CS3220 Lecture Notes: QR factorization and orthogonal transformations
CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss
1 VECTOR SPACES AND SUBSPACES
1 VECTOR SPACES AND SUBSPACES What is a vector? Many are familiar with the concept of a vector as: Something which has magnitude and direction. an ordered pair or triple. a description for quantities such
CHAPTER 8 FACTOR EXTRACTION BY MATRIX FACTORING TECHNIQUES. From Exploratory Factor Analysis Ledyard R Tucker and Robert C.
CHAPTER 8 FACTOR EXTRACTION BY MATRIX FACTORING TECHNIQUES From Exploratory Factor Analysis Ledyard R Tucker and Robert C MacCallum 1997 180 CHAPTER 8 FACTOR EXTRACTION BY MATRIX FACTORING TECHNIQUES In
constraint. Let us penalize ourselves for making the constraint too big. We end up with a
Chapter 4 Constrained Optimization 4.1 Equality Constraints (Lagrangians) Suppose we have a problem: Maximize 5, (x 1, 2) 2, 2(x 2, 1) 2 subject to x 1 +4x 2 =3 If we ignore the constraint, we get the
LECTURE 1: DIFFERENTIAL FORMS. 1. 1-forms on R n
LECTURE 1: DIFFERENTIAL FORMS 1. 1-forms on R n In calculus, you may have seen the differential or exterior derivative df of a function f(x, y, z) defined to be df = f f f dx + dy + x y z dz. The expression
Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model
Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written
Practice Final Math 122 Spring 12 Instructor: Jeff Lang
Practice Final Math Spring Instructor: Jeff Lang. Find the limit of the sequence a n = ln (n 5) ln (3n + 8). A) ln ( ) 3 B) ln C) ln ( ) 3 D) does not exist. Find the limit of the sequence a n = (ln n)6
Recall that two vectors in are perpendicular or orthogonal provided that their dot
Orthogonal Complements and Projections Recall that two vectors in are perpendicular or orthogonal provided that their dot product vanishes That is, if and only if Example 1 The vectors in are orthogonal
Lecture 2: The SVM classifier
Lecture 2: The SVM classifier C19 Machine Learning Hilary 2015 A. Zisserman Review of linear classifiers Linear separability Perceptron Support Vector Machine (SVM) classifier Wide margin Cost function
Torgerson s Classical MDS derivation: 1: Determining Coordinates from Euclidean Distances
Torgerson s Classical MDS derivation: 1: Determining Coordinates from Euclidean Distances It is possible to construct a matrix X of Cartesian coordinates of points in Euclidean space when we know the Euclidean
ALGEBRA 2 CRA 2 REVIEW - Chapters 1-6 Answer Section
ALGEBRA 2 CRA 2 REVIEW - Chapters 1-6 Answer Section MULTIPLE CHOICE 1. ANS: C 2. ANS: A 3. ANS: A OBJ: 5-3.1 Using Vertex Form SHORT ANSWER 4. ANS: (x + 6)(x 2 6x + 36) OBJ: 6-4.2 Solving Equations by
Matrix Calculations: Applications of Eigenvalues and Eigenvectors; Inner Products
Matrix Calculations: Applications of Eigenvalues and Eigenvectors; Inner Products H. Geuvers Institute for Computing and Information Sciences Intelligent Systems Version: spring 2015 H. Geuvers Version:
Introduction to Machine Learning Using Python. Vikram Kamath
Introduction to Machine Learning Using Python Vikram Kamath Contents: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Introduction/Definition Where and Why ML is used Types of Learning Supervised Learning Linear Regression
A note on companion matrices
Linear Algebra and its Applications 372 (2003) 325 33 www.elsevier.com/locate/laa A note on companion matrices Miroslav Fiedler Academy of Sciences of the Czech Republic Institute of Computer Science Pod
A QUICK GUIDE TO THE FORMULAS OF MULTIVARIABLE CALCULUS
A QUIK GUIDE TO THE FOMULAS OF MULTIVAIABLE ALULUS ontents 1. Analytic Geometry 2 1.1. Definition of a Vector 2 1.2. Scalar Product 2 1.3. Properties of the Scalar Product 2 1.4. Length and Unit Vectors
9 MATRICES AND TRANSFORMATIONS
9 MATRICES AND TRANSFORMATIONS Chapter 9 Matrices and Transformations Objectives After studying this chapter you should be able to handle matrix (and vector) algebra with confidence, and understand the
Linear Models for Classification
Linear Models for Classification Sumeet Agarwal, EEL709 (Most figures from Bishop, PRML) Approaches to classification Discriminant function: Directly assigns each data point x to a particular class Ci
Chapter 13 Introduction to Nonlinear Regression( 非 線 性 迴 歸 )
Chapter 13 Introduction to Nonlinear Regression( 非 線 性 迴 歸 ) and Neural Networks( 類 神 經 網 路 ) 許 湘 伶 Applied Linear Regression Models (Kutner, Nachtsheim, Neter, Li) hsuhl (NUK) LR Chap 10 1 / 35 13 Examples
Chapter 5. Banach Spaces
9 Chapter 5 Banach Spaces Many linear equations may be formulated in terms of a suitable linear operator acting on a Banach space. In this chapter, we study Banach spaces and linear operators acting on
Nonlinear Algebraic Equations. Lectures INF2320 p. 1/88
Nonlinear Algebraic Equations Lectures INF2320 p. 1/88 Lectures INF2320 p. 2/88 Nonlinear algebraic equations When solving the system u (t) = g(u), u(0) = u 0, (1) with an implicit Euler scheme we have
State of Stress at Point
State of Stress at Point Einstein Notation The basic idea of Einstein notation is that a covector and a vector can form a scalar: This is typically written as an explicit sum: According to this convention,
Date: April 12, 2001. Contents
2 Lagrange Multipliers Date: April 12, 2001 Contents 2.1. Introduction to Lagrange Multipliers......... p. 2 2.2. Enhanced Fritz John Optimality Conditions...... p. 12 2.3. Informative Lagrange Multipliers...........
Computing a Nearest Correlation Matrix with Factor Structure
Computing a Nearest Correlation Matrix with Factor Structure Nick Higham School of Mathematics The University of Manchester higham@ma.man.ac.uk http://www.ma.man.ac.uk/~higham/ Joint work with Rüdiger