# Inner Product Spaces

Size: px
Start display at page:

Transcription

1 Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and which satisfies: (1) u, u 0 with equality if and only if u = 0 (2) u, v = v, u and (3) αu + v, w = α u, w + v, w Combining (2) and (3), we also have u, αv + w = α u, v + u, w. Condition (1) is called positive definite, condition (2) is called symmetric and condition (3) with the note above is called bilinear. Thus an inner product is an example of a positive definite, symmetric bilinear function or form on the vector space V. Definition Let V be an inner product space and u and v be vectors in V. definitions: We make the following (1) The length or norm of the vector u is: u = u, u (2) The distance between u and v is: u v (3) The angle between u and v is: ( ) u, v θ = cos 1 u v (4) We say that u and v are orthogonal if u, v = 0 (5) The orthogonal projection of u onto the space spanned by v is: Note that, a priori, we do not know that 1 p = ( ) u, v v v, v u,v u v 1 so θ may not be defined. Later we will show that these bounds are valid and so our definition of θ is also valid. When referring to (5), we will usually say the projection of u onto v. Now we give some preliminary results: Theorem (Pythagorean Theorem). Suppose that u and v are orthogonal vectors. Then u ± v 2 = u 2 + v 2 1

2 Proof: We ll just do the case u + v 2 as the argument for the other case is similar. Note, as u and v are orthogonal, u, v = 0. Now: u + v 2 = u + v, u + v = u, u + u, v + v, u + v, v = u, u + v, v = u 2 + v 2 Theorem Suppose that p is the orthogonal projection of u onto the space spanned by v. Then p and u p are orthogonal. Proof: Recall that the projection of u onto v is given by: For notational convenience, we set β = u,v v,v. Then Therefore they are orthogonal. p = u, v v, v v p, u p = p, u p, p = βv, u βv, βv = β v, u β 2 v, v = = = 0 u, v u, v 2 v, u v, v v, v v, v 2 u, v 2 v, v u, v 2 v, v Theorem (Cauchy-Schwarz Inequality). Suppose that v and v are vectors in an inner product space. Then u, v u v Proof: Let p be the orthogonal projection of u onto v. From the previous result, p and u p are orthogonal. We may then apply the Pythagorean Theorem to get p 2 + u p 2 = (p) + (u p) 2 = u 2 Subtracting u p 2 from both sides and noting 0 u p 2 gives: In the proof of the previous theorem, we found that p 2 = u 2 u p 2 u 2 p 2 u, v 2 = p, p = v, v 2 = u, v 2 v 2

3 So we have u, v 2 v 2 u 2 u, v 2 u 2 v 2 which finally leads to u, v u v 2. Examples of Inner Product Spaces 2.1. Example: R n. Just as R n is our template for a real vector space, it serves in the same way as the archetypical inner product space. The usual inner product on R n is called the dot product or scalar product on R n. It is defined by: x, y = x T y where the right-hand side is just matrix multiplication. In particular, if then x = x 1 x 2. x n x, y = x T y = (x 1,..., x n ) and y = y 1 y 2. y n y 1 y 2. y n = x 1 y x n y n ****PROOF OF DOT PRODUCT BEING INNER PRODUCT GOES HERE**** ****GEOMETRIC PROOF OF ORTHOGONAL PROJECTIONS GOES HERE**** ****SPECIFIC EXAMPLE GOES HERE**** p = ( x T ) y y T y y x T y = x y cos(θ) An alternate inner product can be defined on R n by: x, y = x T A y where the right-hand side is just matrix multiplication. The n n matrix A must be a type of matrix known as a symmetric, positive definite matrix in order for this to satisfy the conditions of an inner product. If we choose A to be a symmetric matrix in which all of its entries are non-negative and has only positive entries on the main diagonal, then it will be such a matrix. (More generally a symmetric, positive definite matrix is a symmetric matrix with only positive eigenvalues.) 3

4 2.2. Example: R m n. Suppose A = (a ij ) and B = (b ij ) are matrices in R m n. The usual inner product on R m n is given by: m A, B = a ij b ij j=1 Note that this is the sum of the point-wise products of the elements of A and B. ****PROOF OF THIS PRODUCT BEING INNER PRODUCT GOES HERE**** ****SPECIFIC EXAMPLE GOES HERE**** 2.3. Example: P n. Here we will describe a type of inner product on P n which we will term a discrete inner product on P n. Let {x 1,..., x n } be distinct real numbers. If p(x) is a polynomial in P n, then it has degree at most (n 1), and therefore has at most (n 1) roots. This means that p(x i ) 0 for at least one i. We know define an inner product by: p(x), q(x) = p(x i )q(x i ) ****PROOF OF THIS PRODUCT BEING INNER PRODUCT GOES HERE**** ****SPECIFIC EXAMPLE GOES HERE**** Since every polynomial is continuous at every real number, we can use the next example of an inner product as an inner product on P n. Each of these are a continuous inner product on P n Example: C[a, b]. An inner product on C[a, b] is given by: f(x), g(x) = b a f(x)g(x)w(x) dx where w(x) is some continuous, positive real-valued function on [a, b]. The standard inner product on C[a, b] is where w(x) = 1 in the above definition. ****PROOF OF THIS PRODUCT BEING INNER PRODUCT GOES HERE**** ****SPECIFIC EXAMPLE GOES HERE**** 3. Subspaces, Orthogonal Complements and Projections Definition Let U and V be subspaces of a vector space W such that U V = {0}. The direct sum of U and V is the set U V = {u + v u U and v V }. Definition Let S be a subspace of the inner product space V. The the orthogonal complement of S is the set S = {v V v, s = 0 for all s S}. Theorem (1) If U and V are subspaces of a vector space W with U V = {0}, then U V is also a subspace of W. (2) If S is a subspace of the inner product space V, then S is also a subspace of V. Proof: (1.) Note that 0+0 = 0 is in U V. Now suppose w 1, w 2 U V, then w 1 = u 1 +v 1 and w 2 = u 2 +v 2 with u i U and v i V and w 1 + w 2 = (u 1 + v 1 ) + (u 2 + v 2 ) = (u 1 + u 2 ) + (v 1 + v 2 ). Since U and V are subspaces, it follows that w 1 +w 2 U V. Suppose now that α is a scalar, then αw 1 = α(u 1 +v 1 ) = αu 1 +αv 1. As above, it then follows that αw 1 U V. Thus U V is a subspace for W. 4

5 For (2.), first note that 0 S. Now suppose that v 1 and v 2 S. Then v 1, s = v 2, s = 0 for all s S. So v 1 + v 2, s = v 1, s + v 2, s = = 0 for all s S. Thus v 1 + v 2 S. Similarly, if α is a scalar, then αv 1, s = α v 1, s = α 0 = 0 for all s S. Thus S is a subspace of V. Theorem If U and V are subspaces of W with U V = {0} and w U V, then w = u + v for unique u U and v V. Proof: Write w = u 1 + v 1 and w = u 2 + v 2. Then u 1 + v 1 = u 2 + v 2 u 1 u 2 = v 2 v 1 u 1 u 2 = 0 = v 2 v 1 u 1 = u 2 and v 2 = v 1. Recall that one of the axioms of an inner product is that x, x 0 with equality if and only if x = 0. An immediate consequence of this is that S S = 0. Definition Let S be a subspace of the inner product space V and let {x 1,..., x n } be a basis for S such that x i, x j = 0 if i j, then this basis is called an orthogonal basis. Furthermore, if x i, x i = 1 then this basis is called an orthonormal basis. Definition Let S be a finite dimensional subspace of the inner product space V and let {x 1,..., x n } be an orthogonal basis for S. If v is any vector in V then the orthogonal projection of v onto S is the vector: v, x i p = x i, x i x i Note that if {x 1,..., x n } is an orthonormal basis, then we have the simpler expression: p = v, x i x i Also in the special case where S is spanned be the single vector x 1, then p is just the usual orthogonal projection of v onto S, which is the line spanned by x 1. Now we can prove the main theorem of this section: Theorem Let S be a finite dimensional subspace of the inner product space V and v be some vector in V. Moreover let {x 1,..., x n } be an orthogonal basis for S and p be the orthogonal projection of v onto S. Then (1) v p S. (2) V = S S. (3) If y is any vector in S with y p, then v p < v y Note that part (3.) says that p is the vector in S which is closest to v. Moreover, an immediate consequence of (2.) is that the orthogonal projection p of v onto S is independent of the choice of orthogonal basis for S. Proof: (1.) We need to show that p and v p are orthogonal. So consider p, v p = p, v p, p. Note that x i, x j = 0 when i j, so that p, v = c i x i, v with c i = v, x i n x i, x i and p, p = c i x i, c i x i p, v = v, x i x i, x i x i, v and p, p = 5 v, x i 2 x i, x i 2 x i, x i

6 Thus p, v = v, x i 2 n x i, x i and p, p = v, x i 2 x i, x i and the result follows for this part. Now let v be any vector in V, then v = p + (v p). Note that p S and from (1.), v p S, and S S = {0}. Therefore we must have V = S S, proving (2.). For part (3.), let y be some vector in S with y p. Then v p = v p + p y. Since p y S and v p S by (1.), we have By the Pythagorean Theorem. So v p 2 + p y 2 = v y 2 v p 2 = v y 2 p y 2 Since y p, p y = 0. Therefore v p 2 < v y 2 and v y < v y. Note that by (3.) of the above theorem, if v is actually in S, then p = v. Definition Let S be a subspace of the inner product space V, v be a vector in V and p be the orthogonal projection of v onto S. Then p is called the least squares approximation of v (in S) and the vector r = v p is called the residual vector of v. 4. Least squares in R n In this section we consider the following situation: Suppose that A is an m n real matrix with m > n. If b is a vector in R m then the matrix equation Ax = b corresponds to an overdetermined linear system. Generally such a system does not have a solution, however we would like to find an ˆx such that Aˆx is as close to b as possible. In this case Aˆx is the least squares approximation to b and we refer to ˆx as the least squares solution to this system. Recall that if r = b Ax, then r is the residual of this system. Moreover, our goal is then to find a x which minimizes r. Before we continue, we mention a result without proof: Theorem Suppose that A is a real matrix. Then Col(A) = N(A T ) and Col(A T ) = N(A). We will use the results of the previous section to find ˆx, or more precisely Aˆx. Given b there is a unique vector p in Col(A) such that b p is minimal by theorem Moreover, by the same theorem, b p N(A T ). Thus: A T (b p) = 0 A T b A T p = 0 A T p = A T b However, p = Aˆx for some vector ˆx (note: ˆx is not necessarily unique, but Aˆx is). So A T p = A T b A T Aˆx = A T b Thus to find ˆx we simply solve for ˆx in the equation: A T Aˆx = A T b which is necessarily consistent. Note that in this case we did not need to know an orthogonal basis for Col(A). This is because we never explicitly calculate p. 6

7 Another general fact about A in this case is that the rank of A is generally n. That is, the columns of A will usually be linearly independent. We have the following theorem which gives us an additional way to solve for ˆx in this situation: Theorem If A is an m n matrix and the rank of A is n then A T A is invertible. Proof: Clearly, N(A) is a subset of N(A T A). We now wish to show that N(A T A) is a subset of N(A). This would establish that N(A) = N(A T A). Let x N(A T A), then (A T A)x = 0 A T (Ax) = 0 Ax N(A T ). Note also that Ax Col(A) so that Ax N(A T ) Col(A). Since Col(A) = N(A T ) Ax = 0, thus x N(A) and N(A T A) = N(A). By the rank-nullity theorem we see that the rank of A T A is the same as the rank of A which is assumed to be n. As A T A is an n n matrix, it must be invertible. Thus, when A has rank n, A T A is invertible, and ˆx = (A T A) 1 A T b Now we proceed with some examples: Example 1: Consider the linear system: x 1 + x 2 = 10 2x 1 + x 2 = 5 x 1 2x 2 = 20 This system is overdetermined and inconsistent. We would like to find the least squares approximation to b and the least squares solution ˆx to this system. We can rewrite this linear system as a matrix system Ax = b where: A = 2 1 and b = It is easy to check that A has rank 2, hence A T A is invertible. Therefore: ˆx = (A T A) 1 A T b = , Aˆx = 1.71 and r = b Aˆx = Example 2: Suppose some system is modeled by a quadratic function f(x), so that f(x) = ax 2 + bx + c. Experimental data is recorded in the form (x, f(x)) with the following results: (1, 1), (2, 10), (3, 9), (4, 16) We would like to find the best approximation for f(x). Using these data points, we see that: a(1) + b(1) + c = 1 a(4) + b(2) + c = 10 a(9) + b(3) + c = 9 a(16) + b(4) + c = 16 7

8 This corresponds to the matrix equation Ax = b where: A = and b = 9 16 As in the previous example, the matrix A has full rank, hence A T A is invertible. Therefore the least squares solution to this system is: 0.5 ˆx = (A T A) 1 A T b = Therefore f(x) is approximately 0.5x x 4.5 Example 3: The orbit of a comet around the sun is either elliptical, parabolic, or hyperbolic. In particular, the orbit can be expressed by the polar equation: r = β e(r cos θ) where β is some positive constant and e is the eccentricity. Note that the orbit is elliptical if 0 e < 1, parabolic if e = 1 and hyperbolic if e > 1. A certain comet is observed over time and the following set of data (r, θ) was recorded: (1.2, 0.3), (2.1, 1.2), (4.1, 2.6), (6.3, 3.8) Using this data we would like to find the approximate orbital equation of this comet. Plugging these data points in the equation above gives us the linear system: β e(1.146) = 1.2 β e(0.761) = 2.1 β e( 3.513) = 4.1 β e( 4.983) = 6.3 This system corresponds to the matrix equation Ax = b where A = and b = Once again, the matrix A has full rank so that the least squares solution is given by: ˆx = (A T A) 1 A T b = Therefore the orbital equation is approximately r = (r cos θ). This example is similar to one of the first applications of least squares. Gauss is credited with developing the method of least squares and applying it to predicting the path of the asteroid Ceres. He did this to a remarkable degree of accuracy in

9 5. Least squares in C[a, b] Recall that an inner product in C[a, b] is given by f(x), g(x) = b a f(x)g(x)w(x) dx where w(x) is some continuous, positive function on [a, b]. Consider that we have a collection of functions {f 1 (x),..., f n (x)} which are mutually orthogonal. Moreover, assume that they form an orthogonal basis for S. Then, given any function f(x) in C[a, b], we can approximate f(x) by a linear combination of the f i. The best such approximation (in terms of least squares) will be given by the orthogonal projection p(x) of f(x) onto S. The most common application of such an approximation is in Fourier Series which will be covered in the next section. There is an analytical consideration which has to be made, and that is how good can we make this approximation. In particular, can we enlarge S is a regular way so that the limit of this process is f(x). This is a question which is beyond our scope, but the answers is yes in some cases and no in others. 6. Fourier Series In this section we consider the function space C[, π] and we wish to know if given a function f(x) in C[, π] how can we approximate this function using functions of the form cos mx and sin mx. This is obviously useful for periodic functions. Our setup is as follows: and f(x), g(x) = 1 π f(x)g(x) dx S n = Span( 1 2, sin x,..., sin nx, cos x,..., cos nx) We can check that the following are true: 1, cos αx = 1 π cos αx dx = 0 1, sin βx = 1 sin βx dx = 0 π cos αx, cos βx = 1 1 if α = β cos αx cos βx dx = π 0 if α β sin αx, sin βx = 1 1 if α = β sin αx sin βx dx = π 0 if α β Therefore is an orthonormal basis for S n. cos αx, sin βx = 1 π cos αx sin βx dx = 0 { 1 2, sin x,..., sin nx, cos x,..., cos nx} 9

10 Given a function f(x) in C[, π], the least squares approximation of f(x) in S n will be where a 0 n 2 + (a k cos kx + b k sin kx) a 0 = 1 π a k = 1 π b k = 1 π f(x) dx f(x) cos kx dx f(x) sin kx dx Note that a k and b k are just the inner products of f(x) with the basis vectors of S n and a 0 n 2 + (a k cos kx + b k sin kx) is just the orthogonal projection of f(x) onto S n. The series a0 2 + n (a k cos kx+b k sin kx) is called the nth order or nth degree Fourier series approximation of f(x) and a k, b k are called the Fourier coefficients. If we consider the approximations as partial sums, then as n we get the usual Fourier series a0 2 + (a k cos kx + b k sin kx). There still remains the question of how good an approximation this is and if this series actually converges to f(x). Now lets try see what the projection of an easy function onto S n is: Example 4: Let f(x) = x. Then clearly f(x) is a vector in C[, π]. The Fourier coefficients are: a 0 = 1 x dx = 0 π a k = 1 π x sin kx x cos kx dx = π kπ 1 sin kx dx = 0 kπ b k = 1 π x cos kx x sin kx dx = π kπ 1 cos kx dx = ( 1) k+1 2 kπ k Therfore the closest vector in S n to f(x) is p n (x) = b k sin kx = ( 1) k+1 2 sin kx k It is beyond our scope, but as n, these approximations do converge to f(x) on the interval (, π). In particular, on that interval, and If we take x = π/2, then we see that x = f(x) = ( 1) k+1 2 sin kx k x 2 = ( 1) n+1 1 sin kx k π 4 = ( 1) k+1 1 k sin k π 2 =

### MATH 304 Linear Algebra Lecture 20: Inner product spaces. Orthogonal sets.

MATH 304 Linear Algebra Lecture 20: Inner product spaces. Orthogonal sets. Norm The notion of norm generalizes the notion of length of a vector in R n. Definition. Let V be a vector space. A function α

### Inner product. Definition of inner product

Math 20F Linear Algebra Lecture 25 1 Inner product Review: Definition of inner product. Slide 1 Norm and distance. Orthogonal vectors. Orthogonal complement. Orthogonal basis. Definition of inner product

### 5. Orthogonal matrices

L Vandenberghe EE133A (Spring 2016) 5 Orthogonal matrices matrices with orthonormal columns orthogonal matrices tall matrices with orthonormal columns complex matrices with orthonormal columns 5-1 Orthonormal

### Section 4.4 Inner Product Spaces

Section 4.4 Inner Product Spaces In our discussion of vector spaces the specific nature of F as a field, other than the fact that it is a field, has played virtually no role. In this section we no longer

### 3. INNER PRODUCT SPACES

. INNER PRODUCT SPACES.. Definition So far we have studied abstract vector spaces. These are a generalisation of the geometric spaces R and R. But these have more structure than just that of a vector space.

### Similarity and Diagonalization. Similar Matrices

MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that

### Linear Algebra Notes for Marsden and Tromba Vector Calculus

Linear Algebra Notes for Marsden and Tromba Vector Calculus n-dimensional Euclidean Space and Matrices Definition of n space As was learned in Math b, a point in Euclidean three space can be thought of

### Vector and Matrix Norms

Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty

### Inner Product Spaces and Orthogonality

Inner Product Spaces and Orthogonality week 3-4 Fall 2006 Dot product of R n The inner product or dot product of R n is a function, defined by u, v a b + a 2 b 2 + + a n b n for u a, a 2,, a n T, v b,

### Orthogonal Diagonalization of Symmetric Matrices

MATH10212 Linear Algebra Brief lecture notes 57 Gram Schmidt Process enables us to find an orthogonal basis of a subspace. Let u 1,..., u k be a basis of a subspace V of R n. We begin the process of finding

### MAT 242 Test 3 SOLUTIONS, FORM A

MAT Test SOLUTIONS, FORM A. Let v =, v =, and v =. Note that B = { v, v, v } is an orthogonal set. Also, let W be the subspace spanned by { v, v, v }. A = 8 a. [5 points] Find the orthogonal projection

### α = u v. In other words, Orthogonal Projection

Orthogonal Projection Given any nonzero vector v, it is possible to decompose an arbitrary vector u into a component that points in the direction of v and one that points in a direction orthogonal to v

### Section 6.1 - Inner Products and Norms

Section 6.1 - Inner Products and Norms Definition. Let V be a vector space over F {R, C}. An inner product on V is a function that assigns, to every ordered pair of vectors x and y in V, a scalar in F,

### MA106 Linear Algebra lecture notes

MA106 Linear Algebra lecture notes Lecturers: Martin Bright and Daan Krammer Warwick, January 2011 Contents 1 Number systems and fields 3 1.1 Axioms for number systems......................... 3 2 Vector

### Math 215 HW #6 Solutions

Math 5 HW #6 Solutions Problem 34 Show that x y is orthogonal to x + y if and only if x = y Proof First, suppose x y is orthogonal to x + y Then since x, y = y, x In other words, = x y, x + y = (x y) T

### Chapter 6. Orthogonality

6.3 Orthogonal Matrices 1 Chapter 6. Orthogonality 6.3 Orthogonal Matrices Definition 6.4. An n n matrix A is orthogonal if A T A = I. Note. We will see that the columns of an orthogonal matrix must be

### Linear algebra and the geometry of quadratic equations. Similarity transformations and orthogonal matrices

MATH 30 Differential Equations Spring 006 Linear algebra and the geometry of quadratic equations Similarity transformations and orthogonal matrices First, some things to recall from linear algebra Two

### 1 VECTOR SPACES AND SUBSPACES

1 VECTOR SPACES AND SUBSPACES What is a vector? Many are familiar with the concept of a vector as: Something which has magnitude and direction. an ordered pair or triple. a description for quantities such

### 12. Inner Product Spaces

1. Inner roduct Spaces 1.1. Vector spaces A real vector space is a set of objects that you can do to things ith: you can add to of them together to get another such object, and you can multiply one of

### Chapter 17. Orthogonal Matrices and Symmetries of Space

Chapter 17. Orthogonal Matrices and Symmetries of Space Take a random matrix, say 1 3 A = 4 5 6, 7 8 9 and compare the lengths of e 1 and Ae 1. The vector e 1 has length 1, while Ae 1 = (1, 4, 7) has length

### 1 Inner Products and Norms on Real Vector Spaces

Math 373: Principles Techniques of Applied Mathematics Spring 29 The 2 Inner Product 1 Inner Products Norms on Real Vector Spaces Recall that an inner product on a real vector space V is a function from

### MATH 551 - APPLIED MATRIX THEORY

MATH 55 - APPLIED MATRIX THEORY FINAL TEST: SAMPLE with SOLUTIONS (25 points NAME: PROBLEM (3 points A web of 5 pages is described by a directed graph whose matrix is given by A Do the following ( points

### NOTES ON LINEAR TRANSFORMATIONS

NOTES ON LINEAR TRANSFORMATIONS Definition 1. Let V and W be vector spaces. A function T : V W is a linear transformation from V to W if the following two properties hold. i T v + v = T v + T v for all

### Bindel, Spring 2012 Intro to Scientific Computing (CS 3220) Week 3: Wednesday, Feb 8

Spaces and bases Week 3: Wednesday, Feb 8 I have two favorite vector spaces 1 : R n and the space P d of polynomials of degree at most d. For R n, we have a canonical basis: R n = span{e 1, e 2,..., e

### Linear Algebra Review. Vectors

Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka kosecka@cs.gmu.edu http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length

### Notes on Orthogonal and Symmetric Matrices MENU, Winter 2013

Notes on Orthogonal and Symmetric Matrices MENU, Winter 201 These notes summarize the main properties and uses of orthogonal and symmetric matrices. We covered quite a bit of material regarding these topics,

### Math 4310 Handout - Quotient Vector Spaces

Math 4310 Handout - Quotient Vector Spaces Dan Collins The textbook defines a subspace of a vector space in Chapter 4, but it avoids ever discussing the notion of a quotient space. This is understandable

### 17. Inner product spaces Definition 17.1. Let V be a real vector space. An inner product on V is a function

17. Inner product spaces Definition 17.1. Let V be a real vector space. An inner product on V is a function, : V V R, which is symmetric, that is u, v = v, u. bilinear, that is linear (in both factors):

### 1 Introduction to Matrices

1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns

### Adding vectors We can do arithmetic with vectors. We ll start with vector addition and related operations. Suppose you have two vectors

1 Chapter 13. VECTORS IN THREE DIMENSIONAL SPACE Let s begin with some names and notation for things: R is the set (collection) of real numbers. We write x R to mean that x is a real number. A real number

### Elementary Linear Algebra

Elementary Linear Algebra Kuttler January, Saylor URL: http://wwwsaylororg/courses/ma/ Saylor URL: http://wwwsaylororg/courses/ma/ Contents Some Prerequisite Topics Sets And Set Notation Functions Graphs

### a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2.

Chapter 1 LINEAR EQUATIONS 1.1 Introduction to linear equations A linear equation in n unknowns x 1, x,, x n is an equation of the form a 1 x 1 + a x + + a n x n = b, where a 1, a,..., a n, b are given

### Au = = = 3u. Aw = = = 2w. so the action of A on u and w is very easy to picture: it simply amounts to a stretching by 3 and 2, respectively.

Chapter 7 Eigenvalues and Eigenvectors In this last chapter of our exploration of Linear Algebra we will revisit eigenvalues and eigenvectors of matrices, concepts that were already introduced in Geometry

### Metric Spaces. Chapter 7. 7.1. Metrics

Chapter 7 Metric Spaces A metric space is a set X that has a notion of the distance d(x, y) between every pair of points x, y X. The purpose of this chapter is to introduce metric spaces and give some

### Solutions to Math 51 First Exam January 29, 2015

Solutions to Math 5 First Exam January 29, 25. ( points) (a) Complete the following sentence: A set of vectors {v,..., v k } is defined to be linearly dependent if (2 points) there exist c,... c k R, not

### Recall that two vectors in are perpendicular or orthogonal provided that their dot

Orthogonal Complements and Projections Recall that two vectors in are perpendicular or orthogonal provided that their dot product vanishes That is, if and only if Example 1 The vectors in are orthogonal

### LINEAR ALGEBRA W W L CHEN

LINEAR ALGEBRA W W L CHEN c W W L Chen, 1997, 2008 This chapter is available free to all individuals, on understanding that it is not to be used for financial gain, and may be downloaded and/or photocopied,

### Linear Algebra I. Ronald van Luijk, 2012

Linear Algebra I Ronald van Luijk, 2012 With many parts from Linear Algebra I by Michael Stoll, 2007 Contents 1. Vector spaces 3 1.1. Examples 3 1.2. Fields 4 1.3. The field of complex numbers. 6 1.4.

### T ( a i x i ) = a i T (x i ).

Chapter 2 Defn 1. (p. 65) Let V and W be vector spaces (over F ). We call a function T : V W a linear transformation form V to W if, for all x, y V and c F, we have (a) T (x + y) = T (x) + T (y) and (b)

### Lectures notes on orthogonal matrices (with exercises) 92.222 - Linear Algebra II - Spring 2004 by D. Klain

Lectures notes on orthogonal matrices (with exercises) 92.222 - Linear Algebra II - Spring 2004 by D. Klain 1. Orthogonal matrices and orthonormal sets An n n real-valued matrix A is said to be an orthogonal

### Vector Math Computer Graphics Scott D. Anderson

Vector Math Computer Graphics Scott D. Anderson 1 Dot Product The notation v w means the dot product or scalar product or inner product of two vectors, v and w. In abstract mathematics, we can talk about

### Matrix Representations of Linear Transformations and Changes of Coordinates

Matrix Representations of Linear Transformations and Changes of Coordinates 01 Subspaces and Bases 011 Definitions A subspace V of R n is a subset of R n that contains the zero element and is closed under

### Mathematics Course 111: Algebra I Part IV: Vector Spaces

Mathematics Course 111: Algebra I Part IV: Vector Spaces D. R. Wilkins Academic Year 1996-7 9 Vector Spaces A vector space over some field K is an algebraic structure consisting of a set V on which are

### Math 333 - Practice Exam 2 with Some Solutions

Math 333 - Practice Exam 2 with Some Solutions (Note that the exam will NOT be this long) Definitions (0 points) Let T : V W be a transformation Let A be a square matrix (a) Define T is linear (b) Define

### CS3220 Lecture Notes: QR factorization and orthogonal transformations

CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss

### Chapter 20. Vector Spaces and Bases

Chapter 20. Vector Spaces and Bases In this course, we have proceeded step-by-step through low-dimensional Linear Algebra. We have looked at lines, planes, hyperplanes, and have seen that there is no limit

### Notes on Symmetric Matrices

CPSC 536N: Randomized Algorithms 2011-12 Term 2 Notes on Symmetric Matrices Prof. Nick Harvey University of British Columbia 1 Symmetric Matrices We review some basic results concerning symmetric matrices.

### [1] Diagonal factorization

8.03 LA.6: Diagonalization and Orthogonal Matrices [ Diagonal factorization [2 Solving systems of first order differential equations [3 Symmetric and Orthonormal Matrices [ Diagonal factorization Recall:

### BANACH AND HILBERT SPACE REVIEW

BANACH AND HILBET SPACE EVIEW CHISTOPHE HEIL These notes will briefly review some basic concepts related to the theory of Banach and Hilbert spaces. We are not trying to give a complete development, but

### Section 1.1. Introduction to R n

The Calculus of Functions of Several Variables Section. Introduction to R n Calculus is the study of functional relationships and how related quantities change with each other. In your first exposure to

### Vector Spaces. 2 3x + 4x 2 7x 3) + ( 8 2x + 11x 2 + 9x 3)

Vector Spaces The idea of vectors dates back to the middle 1800 s, but our current understanding of the concept waited until Peano s work in 1888. Even then it took many years to understand the importance

### October 3rd, 2012. Linear Algebra & Properties of the Covariance Matrix

Linear Algebra & Properties of the Covariance Matrix October 3rd, 2012 Estimation of r and C Let rn 1, rn, t..., rn T be the historical return rates on the n th asset. rn 1 rṇ 2 r n =. r T n n = 1, 2,...,

### 4.5 Linear Dependence and Linear Independence

4.5 Linear Dependence and Linear Independence 267 32. {v 1, v 2 }, where v 1, v 2 are collinear vectors in R 3. 33. Prove that if S and S are subsets of a vector space V such that S is a subset of S, then

### Applied Linear Algebra I Review page 1

Applied Linear Algebra Review 1 I. Determinants A. Definition of a determinant 1. Using sum a. Permutations i. Sign of a permutation ii. Cycle 2. Uniqueness of the determinant function in terms of properties

### Methods for Finding Bases

Methods for Finding Bases Bases for the subspaces of a matrix Row-reduction methods can be used to find bases. Let us now look at an example illustrating how to obtain bases for the row space, null space,

### 1 Norms and Vector Spaces

008.10.07.01 1 Norms and Vector Spaces Suppose we have a complex vector space V. A norm is a function f : V R which satisfies (i) f(x) 0 for all x V (ii) f(x + y) f(x) + f(y) for all x,y V (iii) f(λx)

### LS.6 Solution Matrices

LS.6 Solution Matrices In the literature, solutions to linear systems often are expressed using square matrices rather than vectors. You need to get used to the terminology. As before, we state the definitions

### Numerical Analysis Lecture Notes

Numerical Analysis Lecture Notes Peter J. Olver 5. Inner Products and Norms The norm of a vector is a measure of its size. Besides the familiar Euclidean norm based on the dot product, there are a number

### Høgskolen i Narvik Sivilingeniørutdanningen STE6237 ELEMENTMETODER. Oppgaver

Høgskolen i Narvik Sivilingeniørutdanningen STE637 ELEMENTMETODER Oppgaver Klasse: 4.ID, 4.IT Ekstern Professor: Gregory A. Chechkin e-mail: chechkin@mech.math.msu.su Narvik 6 PART I Task. Consider two-point

### Problem Set 5 Due: In class Thursday, Oct. 18 Late papers will be accepted until 1:00 PM Friday.

Math 312, Fall 2012 Jerry L. Kazdan Problem Set 5 Due: In class Thursday, Oct. 18 Late papers will be accepted until 1:00 PM Friday. In addition to the problems below, you should also know how to solve

### The Method of Partial Fractions Math 121 Calculus II Spring 2015

Rational functions. as The Method of Partial Fractions Math 11 Calculus II Spring 015 Recall that a rational function is a quotient of two polynomials such f(x) g(x) = 3x5 + x 3 + 16x x 60. The method

### Linear Programming Problems

Linear Programming Problems Linear programming problems come up in many applications. In a linear programming problem, we have a function, called the objective function, which depends linearly on a number

### 9 Multiplication of Vectors: The Scalar or Dot Product

Arkansas Tech University MATH 934: Calculus III Dr. Marcel B Finan 9 Multiplication of Vectors: The Scalar or Dot Product Up to this point we have defined what vectors are and discussed basic notation

### CITY UNIVERSITY LONDON. BEng Degree in Computer Systems Engineering Part II BSc Degree in Computer Systems Engineering Part III PART 2 EXAMINATION

No: CITY UNIVERSITY LONDON BEng Degree in Computer Systems Engineering Part II BSc Degree in Computer Systems Engineering Part III PART 2 EXAMINATION ENGINEERING MATHEMATICS 2 (resit) EX2005 Date: August

### MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix.

MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix. Nullspace Let A = (a ij ) be an m n matrix. Definition. The nullspace of the matrix A, denoted N(A), is the set of all n-dimensional column

### Linear Algebra: Vectors

A Linear Algebra: Vectors A Appendix A: LINEAR ALGEBRA: VECTORS TABLE OF CONTENTS Page A Motivation A 3 A2 Vectors A 3 A2 Notational Conventions A 4 A22 Visualization A 5 A23 Special Vectors A 5 A3 Vector

### minimal polyonomial Example

Minimal Polynomials Definition Let α be an element in GF(p e ). We call the monic polynomial of smallest degree which has coefficients in GF(p) and α as a root, the minimal polyonomial of α. Example: We

### 1 Sets and Set Notation.

LINEAR ALGEBRA MATH 27.6 SPRING 23 (COHEN) LECTURE NOTES Sets and Set Notation. Definition (Naive Definition of a Set). A set is any collection of objects, called the elements of that set. We will most

### Systems of Linear Equations

Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and

### Recall the basic property of the transpose (for any A): v A t Aw = v w, v, w R n.

ORTHOGONAL MATRICES Informally, an orthogonal n n matrix is the n-dimensional analogue of the rotation matrices R θ in R 2. When does a linear transformation of R 3 (or R n ) deserve to be called a rotation?

PYTHAGOREAN TRIPLES KEITH CONRAD 1. Introduction A Pythagorean triple is a triple of positive integers (a, b, c) where a + b = c. Examples include (3, 4, 5), (5, 1, 13), and (8, 15, 17). Below is an ancient

### 4 MT210 Notebook 4 3. 4.1 Eigenvalues and Eigenvectors... 3. 4.1.1 Definitions; Graphical Illustrations... 3

MT Notebook Fall / prepared by Professor Jenny Baglivo c Copyright 9 by Jenny A. Baglivo. All Rights Reserved. Contents MT Notebook. Eigenvalues and Eigenvectors................................... Definitions;

### MAT 200, Midterm Exam Solution. a. (5 points) Compute the determinant of the matrix A =

MAT 200, Midterm Exam Solution. (0 points total) a. (5 points) Compute the determinant of the matrix 2 2 0 A = 0 3 0 3 0 Answer: det A = 3. The most efficient way is to develop the determinant along the

### MATH1231 Algebra, 2015 Chapter 7: Linear maps

MATH1231 Algebra, 2015 Chapter 7: Linear maps A/Prof. Daniel Chan School of Mathematics and Statistics University of New South Wales danielc@unsw.edu.au Daniel Chan (UNSW) MATH1231 Algebra 1 / 43 Chapter

### Numerical Analysis Lecture Notes

Numerical Analysis Lecture Notes Peter J. Olver 6. Eigenvalues and Singular Values In this section, we collect together the basic facts about eigenvalues and eigenvectors. From a geometrical viewpoint,

### Cross product and determinants (Sect. 12.4) Two main ways to introduce the cross product

Cross product and determinants (Sect. 12.4) Two main ways to introduce the cross product Geometrical definition Properties Expression in components. Definition in components Properties Geometrical expression.

### Lecture 5 Principal Minors and the Hessian

Lecture 5 Principal Minors and the Hessian Eivind Eriksen BI Norwegian School of Management Department of Economics October 01, 2010 Eivind Eriksen (BI Dept of Economics) Lecture 5 Principal Minors and

### 15. Symmetric polynomials

15. Symmetric polynomials 15.1 The theorem 15.2 First examples 15.3 A variant: discriminants 1. The theorem Let S n be the group of permutations of {1,, n}, also called the symmetric group on n things.

### TOPIC 4: DERIVATIVES

TOPIC 4: DERIVATIVES 1. The derivative of a function. Differentiation rules 1.1. The slope of a curve. The slope of a curve at a point P is a measure of the steepness of the curve. If Q is a point on the

### Lecture L3 - Vectors, Matrices and Coordinate Transformations

S. Widnall 16.07 Dynamics Fall 2009 Lecture notes based on J. Peraire Version 2.0 Lecture L3 - Vectors, Matrices and Coordinate Transformations By using vectors and defining appropriate operations between

### 2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

### The Matrix Elements of a 3 3 Orthogonal Matrix Revisited

Physics 116A Winter 2011 The Matrix Elements of a 3 3 Orthogonal Matrix Revisited 1. Introduction In a class handout entitled, Three-Dimensional Proper and Improper Rotation Matrices, I provided a derivation

### PUTNAM TRAINING POLYNOMIALS. Exercises 1. Find a polynomial with integral coefficients whose zeros include 2 + 5.

PUTNAM TRAINING POLYNOMIALS (Last updated: November 17, 2015) Remark. This is a list of exercises on polynomials. Miguel A. Lerma Exercises 1. Find a polynomial with integral coefficients whose zeros include

### Notes on metric spaces

Notes on metric spaces 1 Introduction The purpose of these notes is to quickly review some of the basic concepts from Real Analysis, Metric Spaces and some related results that will be used in this course.

### MATH PROBLEMS, WITH SOLUTIONS

MATH PROBLEMS, WITH SOLUTIONS OVIDIU MUNTEANU These are free online notes that I wrote to assist students that wish to test their math skills with some problems that go beyond the usual curriculum. These

### MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors. Jordan canonical form (continued).

MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors Jordan canonical form (continued) Jordan canonical form A Jordan block is a square matrix of the form λ 1 0 0 0 0 λ 1 0 0 0 0 λ 0 0 J = 0

### by the matrix A results in a vector which is a reflection of the given

Eigenvalues & Eigenvectors Example Suppose Then So, geometrically, multiplying a vector in by the matrix A results in a vector which is a reflection of the given vector about the y-axis We observe that

### Geometry of Vectors. 1 Cartesian Coordinates. Carlo Tomasi

Geometry of Vectors Carlo Tomasi This note explores the geometric meaning of norm, inner product, orthogonality, and projection for vectors. For vectors in three-dimensional space, we also examine the

### MATH 4330/5330, Fourier Analysis Section 11, The Discrete Fourier Transform

MATH 433/533, Fourier Analysis Section 11, The Discrete Fourier Transform Now, instead of considering functions defined on a continuous domain, like the interval [, 1) or the whole real line R, we wish

### Notes on Determinant

ENGG2012B Advanced Engineering Mathematics Notes on Determinant Lecturer: Kenneth Shum Lecture 9-18/02/2013 The determinant of a system of linear equations determines whether the solution is unique, without

### Examination paper for TMA4115 Matematikk 3

Department of Mathematical Sciences Examination paper for TMA45 Matematikk 3 Academic contact during examination: Antoine Julien a, Alexander Schmeding b, Gereon Quick c Phone: a 73 59 77 82, b 40 53 99

### HOMEWORK 5 SOLUTIONS. n!f n (1) lim. ln x n! + xn x. 1 = G n 1 (x). (2) k + 1 n. (n 1)!

Math 7 Fall 205 HOMEWORK 5 SOLUTIONS Problem. 2008 B2 Let F 0 x = ln x. For n 0 and x > 0, let F n+ x = 0 F ntdt. Evaluate n!f n lim n ln n. By directly computing F n x for small n s, we obtain the following

### Dot product and vector projections (Sect. 12.3) There are two main ways to introduce the dot product

Dot product and vector projections (Sect. 12.3) Two definitions for the dot product. Geometric definition of dot product. Orthogonal vectors. Dot product and orthogonal projections. Properties of the dot

### Continued Fractions and the Euclidean Algorithm

Continued Fractions and the Euclidean Algorithm Lecture notes prepared for MATH 326, Spring 997 Department of Mathematics and Statistics University at Albany William F Hammond Table of Contents Introduction

### Finite dimensional C -algebras

Finite dimensional C -algebras S. Sundar September 14, 2012 Throughout H, K stand for finite dimensional Hilbert spaces. 1 Spectral theorem for self-adjoint opertors Let A B(H) and let {ξ 1, ξ 2,, ξ n

### Orthogonal Projections

Orthogonal Projections and Reflections (with exercises) by D. Klain Version.. Corrections and comments are welcome! Orthogonal Projections Let X,..., X k be a family of linearly independent (column) vectors

### Linearly Independent Sets and Linearly Dependent Sets

These notes closely follow the presentation of the material given in David C. Lay s textbook Linear Algebra and its Applications (3rd edition). These notes are intended primarily for in-class presentation

### Math 312 Homework 1 Solutions

Math 31 Homework 1 Solutions Last modified: July 15, 01 This homework is due on Thursday, July 1th, 01 at 1:10pm Please turn it in during class, or in my mailbox in the main math office (next to 4W1) Please

### ISOMETRIES OF R n KEITH CONRAD

ISOMETRIES OF R n KEITH CONRAD 1. Introduction An isometry of R n is a function h: R n R n that preserves the distance between vectors: h(v) h(w) = v w for all v and w in R n, where (x 1,..., x n ) = x

### 1 Lecture: Integration of rational functions by decomposition

Lecture: Integration of rational functions by decomposition into partial fractions Recognize and integrate basic rational functions, except when the denominator is a power of an irreducible quadratic.