# Vector and Matrix Norms

Size: px
Start display at page:

Transcription

1 Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty set of objects (called vectors) on which two binary operations, (vector) addition and (scalar) multiplication, are defined and satisfy the axioms below Addition: is a rule which associates a vector V with each pair of vectors x, y V and that member is called the sum x + y Scalar multiplication: is a rule which associates a vector V with each scalar, α F, and each vector x V, and it is called the scalar multiple αx For V to be called a Vector Space the two operations above must satisfy the following axioms x, y, w V : 1 Vector addition is commutative: x + y = y + x; 2 Vector addition is associative: x + (y + w) = (x + y) + w; 3 Vector addition has an identity: a vector in V, called the zero vector 0, such that x+0 = x; 4 Vector addition has an inverse: for each x V an inverse (or negative) element, ( x) V, such that x + ( x) = 0; 5 Distributivity holds for scalar multiplication over vector addition: α(x + y) = (αx + αy), α F; 4

2 6 Distributivity holds for scalar multiplication over field addition: (α+β)x = (αx+βx), α, β F; 7 Scalar multiplication is compatible with field multiplication: α(βx) = (αβ)x, α,β F; 8 Scalar multiplication has an identity element: 1x = x, where 1 is the multiplicative identity of F Any collection of objects together with two operations which satisfy the above axioms is a vector space In context of Numerical Analysis a vector space is often called a Linear Space Example 111 An obvious example of a linear space is R 3 with the usual definitions of addition of 3D vectors and scalar multiplication (Fig 11) Figure 11: A pictorial example of some vectors belonging to the linear space R 3 Example 112 Another example of a linear space is the set of all continuous functions f(x) on (, ) with the usual definition for the addition of functions and the scalar multiplication of functions, usually denoted by C(, ) If f(x) and g(x) C(, ) then f +g and αf(x) are also continuous on (, ) and the axioms are easily verified Example 113 A sub-space of C(, ) is P n : the space of polynomials of degree at most n [We will meet this vector space later in the course] Note, it must be of degree at most n so that addition (and subtraction) produce members of P n, eg (x x 2 ) P 2, x 2 P 2 and (x x 2 ) + (x 2 ) = x P 2 12 The Basis and Dimension of a Vector Space If V is a vector space and S = {x 1,x 2,,x n } is a set of vectors in V such that 5

3 S is a linearly independent set A set of vectors is linearly independent if none of them can be written as a linear combination of finitely many other vectors in the set and S spans V A span is a set of vectors V for which all other vectors V can be written as a linear combination of the vectors in the span then S is a basis for V The dimension of V is n Example 121 A basis for Example 111, where V = R 3, is i = (1,0,0), j = (0,1,0) and k = (0,0,1) and the dimension of the space is 3 Example 122 Example 112 has infinite dimension as no finite set spans C(, ) Example 123 A basis for Example 113, where V = P n, is the set of monomials 1, x, x 2,, x n The dimension of the space is n Normed Linear Spaces 131 Vector Norm We require some method to measure the magnitude of a vector for error analysis We generalise the concept of a length of a vector in 3D to n dimensions by defining a norm Given a vector/linear space V, then a norm, denoted by x for x V, is a real number such that x > 0, x 0, ( 0 = 0) (11) αx = α x, α R (12) x + y x + y (13) The norm is a measure of the size of the vector x where Equation (11) requires the size to be positive, Equation (12) requires the size to be scaled as the vector is scaled, and Equation (13) is known as the triangular inequality and has its origins in the notion of distance in R 3 Any mapping of an n-dimensional Vector Space onto a subset of R that satisfies these three requirements can be called a norm The space together with a defined norm is called a Normed Linear Space Example 131 For the vector space V = R n with x V given by x = (x 1,x 2,,x n ) an obvious 6

4 definition of a norm is x = ( x x x 2 n) 1/2 This is just a generalisation of the normed linear space V = R 3 with the norm defined as the magnitude of the vector x R 3 Example 132 Another norm on V = R n is x = max 1 i n { x i }, and it is easy to verify that the three axioms are obeyed (see Tutorial sheet 1) Example 133 Let V = C[a,b], the space of all continuous functions on the interval [a,b], and define This is also a normed linear space { 1/2 b f = (f(x)) dx} 2 a 132 Inner Product Space One of the most familiar norms is the magnitude of a vector, x = (x 1,x 2,x 3 ) R 3, (Example 131) This norm is commonly denoted x and equals x = x = x x2 2 + x2 3 = x x ie it is equal to the dot product of x The dot product of a vector is an example of an Inner Product Some other norms are also inner products In 131 we presented examples of Normed Linear Spaces Two of these can be obtained from the alternative route of Inner Product Spaces and it is useful to observe this fact as it gives access to an important result, namely the Cauchy-Schwarz Inequality For such spaces where and f,g = x,y = b a x = { x,x } 1/2 x i y i in Example 131 f(x)g(x)dx in Example 133 Hence, inner products give rise to norms but not all norms can be cast in terms of inner products The formal definition of an Inner Product is given on Tutorial Sheet 1 and it is easy to show that the above two examples satisfy the stated requirements The Cauchy-Schwarz Inequality is given by x,y 2 x,x y,y 7

5 and can be used to confirm the triangular inequality for norms For example, given x = { x,x } 1/2 x + y 2 = x + y,x + y = x,x + y,y + 2 x,y x,x + y,y + 2 x,x y,y (using C S inequality) x + y 2 ( x + y ) 2 (14) and hence the triangular inequality holds for all such norms A particular example is given on Tutorial Sheet Commonly Used Norms and Normed Linear Spaces In theory, any mapping of an n-dimensional space onto real numbers that satisfies (11)-(13) is a norm In practice we are only concerned with a small number of Normed Linear Spaces and the most frequently used are the following: 1 The linear space R n (Euclidean Space) where x = (x 1,x 2,,x i,,x n ) together with the norm ( n ) 1/p x p = x i p, p 1, is known as the L p normed linear space The most common are the one norm, L 1, and the two norm, L 2, linear spaces where p = 1 and p = 2, respectively 2 The other standard norm for the space R n is the infinity, or maximum, norm given by x = max 1 i n ( x i ) The vector space R n together with the infinity norm is commonly denoted L Example 134 Consider the vector x = (3, 1,2,0,4), which belongs to the vector space R 5 Then determine its (i) one norm, (ii) two norm and (iii) infinity norm (i) One norm: x 1 = = 10 (ii) Two norm: x 2 = = 30 (iii) Infinity norm: x = max( 3, 1, 2, 0, 4 ) = 4 Note: Each is a different way of measuring of the size of a vector R n For C[a,b] (Continuous functions) the corresponding norms are ( 1/p b f(x) p = f(x) dx) p, p 1, a 8

6 with p = 1 and p = 2 the usual cases, plus f(x) = max a x b f(x) 14 Sub-ordinate Matrix Norms We are interested in analysing methods for solving linear systems so we need to be able to measure the size of vectors in R n and any associated matrices in a compatible way To achieve this end, we define a Sub-ordinate Matrix Norm For the Normed Linear Space {R n, x }, where x is some norm, we define the norm of the matrix A n n which is sub-ordinate to the vector norm x as ( ) Ax A = max x =0 x Note, Ax is a vector, x R n Ax R n, so A is the largest value of the vector norm of Ax normalised over all non-zero vectors x Not surprisingly, the three requirements of a vector norm (11 13) are properties of A There are two further properties which are a consequence of the definition for A Hence, sub-ordinate matrix norms satisfy the following five rules: A > 0, A 0, ( 0 = 0),where 0 is the n n zero matrix, (15) αa = α A, α R, (16) A + B A + B, (17) Ax A x, (18) AB A B (19) These five rules, together with the three for vector norms (11 13), provide the means for an analysis of a linear system First, let us justify the above results: Rule 1 (15): For a matrix A n n where A 0, x R n where x 0 such that the vector Ax 0 So both Ax > 0 and x > 0, thus A > 0 Also, if A 0 then Ax = 0 x and A = 0 Rule 2 (16): Since Ax is a vector and therefore satisfies (12) we can show that { } { } { } αax α Ax Ax αa = max = max = α max = α A x =0 x x =0 x x =0 x 9

7 Rule 3 (17): { } (A + B)x A + B = max x =0 x where x m (x m R n ) maximises the right hand side = (A + B)x m x m Now, let us define Ax m = y m and Bx m = z m (y m, z m R n ) so that (A + B)x m = Ax m + Bx m = y m + z m y m + z m = Ax m + Bx m, therefore Hence A + B Ax m + Bx m x m max x =0 A + B A + B { } Ax + max x x =0 { } Bx x Rule 4 (18): By definition, A Ax x for anyx 0 Hence A x Ax x (includingx 0) Rule 5 (19): Finally, consider AB, where A and B are n n matrices Using a similar argument to that used in rule 3, we have { } ABx AB = max x =0 x where x m (x m R n ) maximises the right hand side = ABx m x m If we define Bx m = z m (z m R n ) and use rule 4 (18), then ABx m = Az m A z m = A Bx m A B x m Hence AB = ABx m x m A B x m x m = A B 141 Commonly Used Sub-ordinate Matrix Norms Clearly, the sub-ordinate matrix norm is not easy to calculate direct from its definition as in theory all vectors x R n should be considered Here we will consider the commonly used matrix norms and consider practical ways to calculate them 10

8 The easiest matrix norms to compute are matrix norms sub-ordinate to the L 1 and L vector norms These are A 1 = max x 0 ( n (Ax) ) i n x i A 1 which is equivalent to the maximum column sum of absolute values of A and max ( (Ax) i ) 1 i n A = max x 0 max ( x i ) 1 i n A which is equivalent to the maximum row sum of absolute values of A Proof that A 1 is equivalent to the maximum column sum of absolute values of A (proof of the relation for A is similar) The L 1 norm is x 1 = x i Let y = Ax (x, y R n and A n n ) and so y i = n j=1 a ijx j and Now Thus Ax 1 = y 1 = a ij x j j=1 Ax 1 Changing the order of summation then gives Ax 1 y i = a ij x j a ij x j j=1 j=1 a ij x j j=1 a ij x j j=1 j=1 a ij x j { n } x j a ij j=1 where n a ij is the sum of absolute values in column j Each column will have a sum, S j = n a ij, 1 j n and S j S m, where m is the column with the largest sum So and hence Ax 1 n x j S j S m x j = S m x 1 j=1 j=1 Ax 1 x 1 S m, 11

9 where S m is the maximum column sum of absolute values This is true for all non-zero x Hence, A S m We need to determine now if S m is the maximum value of Ax 1 x 1, or whether it is merely an upper bound In other words, is it possible to pick an x R n for which we reach S m? Try by choosing the following: x = [ 0,,0,1,0,,0] T 0 j m where x j = 1 j = m Now, substituting this particular x into Ax = y implies y i = n j=1 a ijx j = a im and thus Ax 1 = y 1 = y i = a im = S m So, Ax 1 = S m x 1 1 = S m This means that the bound can be reached with a suitable x and so ( ) Ax 1 A 1 = max = S m, x =0 x 1 where S m is the maximum column sum of absolute values Hence, S m is not just an upper bound, it is actually the maximum! Example 141: Determine the matrix norm sub-ordinate to (i) the one norm and (ii) the infinity norm for the following matrices: A = and B = A 1 = max(( ),( ),( ), = max(8,13,5) = 13 B 1 = max(( ),( ),( )) = max(6,5,2) = 6 A = max(( ),( ),( ), = max(11,8,7) = 11 B = max(( ),( ),( )) = max(6,5,2) = 6 Note, since B is symmetric, the maximum column sum of absolute values equals the maximum row sum of absolute values ie B 1 = B 12

10 15 Spectral Radius A useful, and important, quantity associated with matrices is the Spectral Radius of a matrix An n n matrix A has n eigenvalues λ i,i = 1,,n The Spectral Radius of A, which is denoted by ρ(a) is defined as Note, that ρ(a) 0 for all A 0 Furthermore, ρ(a) = max 1 i n ( λ i ) ρ(a) A, for all sub-ordinate matrix norms Proof Let e i be an eigenvector of A, so Ae i = λ i e i where λ i is the associated eigenvalue Hence, e i 0 and λ i 0 Then Ae i = λ i e i = λ i e i and so or Ae i A e i, λ i e i A e i, λ i A Now this result holds true λ i and hence ρ(a) = max 1 i n ( λ i ) A Note, the spectral radius is not a norm! This can be easily shown Let which gives However, A = 1 0 and B = ρ(a) = max( 1, 0 ) = 1 and ρ(b) = max( 0, 1 ) = 1 A + B = 1 2 and ρ(a + B) = max( 1, 3 ) = hence ρ(a + B) ρ(a) + ρ(b) So the spectral radius does not satisfy (13), rule 3 for a norm 13

11 16 The Condition Number and Ill-Conditioned Matrices Suppose we are interested in solving Ax = b where A n n, x n 1 & b n 1 and the elements of b are subject to small errors δb so that instead of finding x, we find x where A x = b + δb How big is the disturbance in x in relation to the disturbance in b? (ie How sensitive is this system of linear equations?) Ax = b and A x = b + δb so, A( x x) = δb and hence the critical change in x is given by ( x x) = A 1 δb We are interested in the size of the errors, so let us take the norm: x x = A 1 δb A 1 δb for some norm This gives a measure of the actual change in the solution However, a more useful measure is the relative change and so we also consider, b = Ax A x so Hence, the relative change in x, given by 1 x A b x x x is at worst the relative change in b times A A 1 A A 1 δb b, The quantity A A 1 = K(A) is called the condition number of A with respect to the norm chosen The condition number gives an indication of the potential sensitivity of a linear system of equations The smaller the conditional number, K(A), the smaller the change in x If K(A) is very large, the solution x, is considered unreliable We would like the condition number to be small, but it is easy to see that it will never be < 1 14

12 Note, AA 1 = I and thus AA 1 = I Using Rule 5 (19), we find AA 1 = I A A 1 = K(A) and using the definition of the subordinate matrix norm: ( ) ( ) Ix x I = max = max = 1 x =0 x x Hence, K(A) 1 If K(A) is modest then we can be confident that small changes to b do not seriously affect the solution However, if K(A) 1 (very large) then small changes to b may produce large changes in x When this happens, we say the system is ill-conditioned How large is large? This all depends on the system Remember, the condition number helps in finding the relative error: x x A A 1 δb x b If the relative error of δb / b = 10 6 and we wish to know x to within a relative error of b = 10 4 then provided the condition number K(A) < 100 then it would be considered small 161 An Example of Using Norms to Solve a Linear System of Equations Example 161: We will consider an innocent looking set of equations that result in a large condition number Consider the linear system Wx = b, where W is the Wilson Matrix, W = and the vector x = so that Wx = b = Now suppose we actually solve W x = b + δb = hence δb =

13 First, we must find the inverse of W, W = Then from W x = b + δb, we find 1 82 x = W 1 b + W δb = It is clear that the system is sensitive to changes: a small change to b has had a very large effect on the solution So the Wilson Matrix is an example of an ill-conditioned matrix and we would expect the condition number to be very large To evaluate the condition number, K(W) for the Wilson Matrix we need to select a particular norm First, we select the 1-norm (maximum column sum of absolute values) and estimate the error W 1 = 33 and W 1 1 = 136, and hence, K 1 (W) = W 1 W 1 1 = = 4488, which of course is considerably bigger than 1 Remember that such that the x x 1 x 1 K 1 (W) δb 1 b 1 error in x 4488 ( ) ( ) = So far, we have used the 1-norm but it is more natural to look at the -norm, which deals with the maximum values Since W is symmetric, W 1 = W and W 1 1 = W 1 so, x x x K (W) δb max( 01, 01, 01, 01 ) 4488 = b max( 32, 23, 33, 31 ) This is exactly equal to the biggest error we found, so a very realistic (reliable) estimate For this example, the bound provides a good estimate of the scale of any change in the solution 16

14 17 Some Useful Results for Determining the Condition Number 171 Finding Norms of Inverse Matrices To estimate condition numbers of matrices we need to find norms of inverses To find an inverse, we have to solve a linear system and if the linear system is sensitive, how reliable is the value we get for A 1? Ideally, we wish to estimate A 1 without needing to find A 1 One possible result that might help is the following: Suppose B is a non-singular matrix, Write B = I + A so BB 1 = I (I + A)(I + A) 1 = I, or so (I + A) 1 + A(I + A) 1 = I, (I + A) 1 = I A(I + A) 1 Taking norms and, using Rules 3 & 5 (17) and (19), we find (I + A) 1 = I A(I + A) 1 I + A(I + A) A (I + A) 1, so If A 1 then where B = I + A (I + A) 1 (1 A ) 1 B 1 = (I + A) A, Example 171: Consider the matrix 1 1/4 1/4 B = 1/4 1 1/4 1/4 1/4 1 17

15 then 0 1/4 1/4 A = B I = 1/4 0 1/4 1/4 1/4 0 Using the infinity norm, A = 1 2 < 1, so B /2 = 2, and B = 15 Hence, K (B) = B B or K (B) 3, which is quite modest 172 Limits on the Condition Number The Spectral Radius of a matrix states that sub-ordinate matrix norms are always greater than (or equal to) the absolute values of the eigenvalues This is true for all eigenvalues Suppose λ 1 λ 2 λ n 1 λ n > 0 ie there is a largest and a smallest eigenvalue Then, the spectral radius for all norms implies that, A λ 1 = max eigenvalues = ρ(a) Furthermore, if the eigenvalues of A are λ i and A is non-singular, then from Ae i = λe i we have 1 λ i e i = A 1 e i so the eigenvalues of A 1 are λ 1 (A non-singular λ 0) Hence, the spectral radius implies A 1 max 1 i λ i or Thus, the condition number for A, A 1 1 λ n K(A) = A A 1 λ 1 λ n, where λ1 λ n is the ratio of the largest to the smallest eigenvalue This ratio gives an indication of just how big the condition number can be, ie if the ratio is large, then K(A) will be large (and the matrix is ill-conditioned) Also, this result illustrates another simple result, namely that scaling a matrix dos not affect the condition number K(αA) = K(A) 18

16 18 Examples of Ill-Conditioned Matrices Example 181: An almost singular matrix is ill-conditioned Take A = then det(a) = 10 7, which is very small The eigenvalues of A are given by (1 λ) 2 ( ) = 0, or, using the expansion (1 + x) 1/ x x2 +, (1 λ) = ±( ) 1/2 ±( ) Hence, we find eigenvalues λ 1 2 and λ and hence the condition number K(A) = λ1 λ n So as you might expect, an almost singular matrix is ill-conditioned Example 182: The Hilbert Matrix is notorious for being ill-conditioned Consider the n n Hilbert matrix 1 1/2 1/3 1/n 1/2 1/3 1/4 H = 1/3 1/4 1/5 1/n 1/(2n 1) n n The condition numbers for various H n n matrices are given in the table below n K (H n n ) Typical single precision, floating point accuracy on a computer is approximately Apply Gaussian Elimination on a 6 6 Hilbert matrix on a computer with single precision arithmetic and the result is likely to be subject to significant error! Such error can be avoided with packages like 19

17 Maple which allows you to perform calculations using exact arithmetic, ie all digits will be carried in the calculations However, error can still occur if the matrix is not specified properly If the elements are not generated exactly, the wrong result can still be obtained 20

### Inner Product Spaces

Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and

### Linear Algebra Review. Vectors

Linear Algebra Review By Tim K. Marks UCSD Borrows heavily from: Jana Kosecka kosecka@cs.gmu.edu http://cs.gmu.edu/~kosecka/cs682.html Virginia de Sa Cogsci 8F Linear Algebra review UCSD Vectors The length

### Linear Algebra Notes for Marsden and Tromba Vector Calculus

Linear Algebra Notes for Marsden and Tromba Vector Calculus n-dimensional Euclidean Space and Matrices Definition of n space As was learned in Math b, a point in Euclidean three space can be thought of

### MATH 304 Linear Algebra Lecture 20: Inner product spaces. Orthogonal sets.

MATH 304 Linear Algebra Lecture 20: Inner product spaces. Orthogonal sets. Norm The notion of norm generalizes the notion of length of a vector in R n. Definition. Let V be a vector space. A function α

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### Au = = = 3u. Aw = = = 2w. so the action of A on u and w is very easy to picture: it simply amounts to a stretching by 3 and 2, respectively.

Chapter 7 Eigenvalues and Eigenvectors In this last chapter of our exploration of Linear Algebra we will revisit eigenvalues and eigenvectors of matrices, concepts that were already introduced in Geometry

### Introduction to Matrix Algebra

Psychology 7291: Multivariate Statistics (Carey) 8/27/98 Matrix Algebra - 1 Introduction to Matrix Algebra Definitions: A matrix is a collection of numbers ordered by rows and columns. It is customary

### 1 VECTOR SPACES AND SUBSPACES

1 VECTOR SPACES AND SUBSPACES What is a vector? Many are familiar with the concept of a vector as: Something which has magnitude and direction. an ordered pair or triple. a description for quantities such

### Similarity and Diagonalization. Similar Matrices

MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that

### 13 MATH FACTS 101. 2 a = 1. 7. The elements of a vector have a graphical interpretation, which is particularly easy to see in two or three dimensions.

3 MATH FACTS 0 3 MATH FACTS 3. Vectors 3.. Definition We use the overhead arrow to denote a column vector, i.e., a linear segment with a direction. For example, in three-space, we write a vector in terms

### Orthogonal Diagonalization of Symmetric Matrices

MATH10212 Linear Algebra Brief lecture notes 57 Gram Schmidt Process enables us to find an orthogonal basis of a subspace. Let u 1,..., u k be a basis of a subspace V of R n. We begin the process of finding

### Section 6.1 - Inner Products and Norms

Section 6.1 - Inner Products and Norms Definition. Let V be a vector space over F {R, C}. An inner product on V is a function that assigns, to every ordered pair of vectors x and y in V, a scalar in F,

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +

### October 3rd, 2012. Linear Algebra & Properties of the Covariance Matrix

Linear Algebra & Properties of the Covariance Matrix October 3rd, 2012 Estimation of r and C Let rn 1, rn, t..., rn T be the historical return rates on the n th asset. rn 1 rṇ 2 r n =. r T n n = 1, 2,...,

### BANACH AND HILBERT SPACE REVIEW

BANACH AND HILBET SPACE EVIEW CHISTOPHE HEIL These notes will briefly review some basic concepts related to the theory of Banach and Hilbert spaces. We are not trying to give a complete development, but

### MATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1.

MATH10212 Linear Algebra Textbook: D. Poole, Linear Algebra: A Modern Introduction. Thompson, 2006. ISBN 0-534-40596-7. Systems of Linear Equations Definition. An n-dimensional vector is a row or a column

### 3. INNER PRODUCT SPACES

. INNER PRODUCT SPACES.. Definition So far we have studied abstract vector spaces. These are a generalisation of the geometric spaces R and R. But these have more structure than just that of a vector space.

### 1 Introduction to Matrices

1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns

### Bindel, Spring 2012 Intro to Scientific Computing (CS 3220) Week 3: Wednesday, Feb 8

Spaces and bases Week 3: Wednesday, Feb 8 I have two favorite vector spaces 1 : R n and the space P d of polynomials of degree at most d. For R n, we have a canonical basis: R n = span{e 1, e 2,..., e

### T ( a i x i ) = a i T (x i ).

Chapter 2 Defn 1. (p. 65) Let V and W be vector spaces (over F ). We call a function T : V W a linear transformation form V to W if, for all x, y V and c F, we have (a) T (x + y) = T (x) + T (y) and (b)

### Systems of Linear Equations

Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and

### a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2.

Chapter 1 LINEAR EQUATIONS 1.1 Introduction to linear equations A linear equation in n unknowns x 1, x,, x n is an equation of the form a 1 x 1 + a x + + a n x n = b, where a 1, a,..., a n, b are given

### Lecture 2 Matrix Operations

Lecture 2 Matrix Operations transpose, sum & difference, scalar multiplication matrix multiplication, matrix-vector product matrix inverse 2 1 Matrix transpose transpose of m n matrix A, denoted A T or

### Linear Algebra I. Ronald van Luijk, 2012

Linear Algebra I Ronald van Luijk, 2012 With many parts from Linear Algebra I by Michael Stoll, 2007 Contents 1. Vector spaces 3 1.1. Examples 3 1.2. Fields 4 1.3. The field of complex numbers. 6 1.4.

### MATH 551 - APPLIED MATRIX THEORY

MATH 55 - APPLIED MATRIX THEORY FINAL TEST: SAMPLE with SOLUTIONS (25 points NAME: PROBLEM (3 points A web of 5 pages is described by a directed graph whose matrix is given by A Do the following ( points

### Matrix Representations of Linear Transformations and Changes of Coordinates

Matrix Representations of Linear Transformations and Changes of Coordinates 01 Subspaces and Bases 011 Definitions A subspace V of R n is a subset of R n that contains the zero element and is closed under

### Math 4310 Handout - Quotient Vector Spaces

Math 4310 Handout - Quotient Vector Spaces Dan Collins The textbook defines a subspace of a vector space in Chapter 4, but it avoids ever discussing the notion of a quotient space. This is understandable

### Notes on Determinant

ENGG2012B Advanced Engineering Mathematics Notes on Determinant Lecturer: Kenneth Shum Lecture 9-18/02/2013 The determinant of a system of linear equations determines whether the solution is unique, without

### Inner Product Spaces and Orthogonality

Inner Product Spaces and Orthogonality week 3-4 Fall 2006 Dot product of R n The inner product or dot product of R n is a function, defined by u, v a b + a 2 b 2 + + a n b n for u a, a 2,, a n T, v b,

### by the matrix A results in a vector which is a reflection of the given

Eigenvalues & Eigenvectors Example Suppose Then So, geometrically, multiplying a vector in by the matrix A results in a vector which is a reflection of the given vector about the y-axis We observe that

### Metric Spaces. Chapter 7. 7.1. Metrics

Chapter 7 Metric Spaces A metric space is a set X that has a notion of the distance d(x, y) between every pair of points x, y X. The purpose of this chapter is to introduce metric spaces and give some

### Mathematics Course 111: Algebra I Part IV: Vector Spaces

Mathematics Course 111: Algebra I Part IV: Vector Spaces D. R. Wilkins Academic Year 1996-7 9 Vector Spaces A vector space over some field K is an algebraic structure consisting of a set V on which are

### Linear Algebra Notes

Linear Algebra Notes Chapter 19 KERNEL AND IMAGE OF A MATRIX Take an n m matrix a 11 a 12 a 1m a 21 a 22 a 2m a n1 a n2 a nm and think of it as a function A : R m R n The kernel of A is defined as Note

### Linear Algebra: Vectors

A Linear Algebra: Vectors A Appendix A: LINEAR ALGEBRA: VECTORS TABLE OF CONTENTS Page A Motivation A 3 A2 Vectors A 3 A2 Notational Conventions A 4 A22 Visualization A 5 A23 Special Vectors A 5 A3 Vector

### Lecture 3: Finding integer solutions to systems of linear equations

Lecture 3: Finding integer solutions to systems of linear equations Algorithmic Number Theory (Fall 2014) Rutgers University Swastik Kopparty Scribe: Abhishek Bhrushundi 1 Overview The goal of this lecture

### MA106 Linear Algebra lecture notes

MA106 Linear Algebra lecture notes Lecturers: Martin Bright and Daan Krammer Warwick, January 2011 Contents 1 Number systems and fields 3 1.1 Axioms for number systems......................... 3 2 Vector

### Name: Section Registered In:

Name: Section Registered In: Math 125 Exam 3 Version 1 April 24, 2006 60 total points possible 1. (5pts) Use Cramer s Rule to solve 3x + 4y = 30 x 2y = 8. Be sure to show enough detail that shows you are

### MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix.

MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix. Nullspace Let A = (a ij ) be an m n matrix. Definition. The nullspace of the matrix A, denoted N(A), is the set of all n-dimensional column

### Solving Linear Systems, Continued and The Inverse of a Matrix

, Continued and The of a Matrix Calculus III Summer 2013, Session II Monday, July 15, 2013 Agenda 1. The rank of a matrix 2. The inverse of a square matrix Gaussian Gaussian solves a linear system by reducing

### Math 312 Homework 1 Solutions

Math 31 Homework 1 Solutions Last modified: July 15, 01 This homework is due on Thursday, July 1th, 01 at 1:10pm Please turn it in during class, or in my mailbox in the main math office (next to 4W1) Please

### 4.5 Linear Dependence and Linear Independence

4.5 Linear Dependence and Linear Independence 267 32. {v 1, v 2 }, where v 1, v 2 are collinear vectors in R 3. 33. Prove that if S and S are subsets of a vector space V such that S is a subset of S, then

### 15.062 Data Mining: Algorithms and Applications Matrix Math Review

.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop

### So let us begin our quest to find the holy grail of real analysis.

1 Section 5.2 The Complete Ordered Field: Purpose of Section We present an axiomatic description of the real numbers as a complete ordered field. The axioms which describe the arithmetic of the real numbers

### 8 Square matrices continued: Determinants

8 Square matrices continued: Determinants 8. Introduction Determinants give us important information about square matrices, and, as we ll soon see, are essential for the computation of eigenvalues. You

### University of Lille I PC first year list of exercises n 7. Review

University of Lille I PC first year list of exercises n 7 Review Exercise Solve the following systems in 4 different ways (by substitution, by the Gauss method, by inverting the matrix of coefficients

### Methods for Finding Bases

Methods for Finding Bases Bases for the subspaces of a matrix Row-reduction methods can be used to find bases. Let us now look at an example illustrating how to obtain bases for the row space, null space,

### MATH 304 Linear Algebra Lecture 9: Subspaces of vector spaces (continued). Span. Spanning set.

MATH 304 Linear Algebra Lecture 9: Subspaces of vector spaces (continued). Span. Spanning set. Vector space A vector space is a set V equipped with two operations, addition V V (x,y) x + y V and scalar

### 3. Reaction Diffusion Equations Consider the following ODE model for population growth

3. Reaction Diffusion Equations Consider the following ODE model for population growth u t a u t u t, u 0 u 0 where u t denotes the population size at time t, and a u plays the role of the population dependent

### NOTES ON LINEAR TRANSFORMATIONS

NOTES ON LINEAR TRANSFORMATIONS Definition 1. Let V and W be vector spaces. A function T : V W is a linear transformation from V to W if the following two properties hold. i T v + v = T v + T v for all

### Chapter 20. Vector Spaces and Bases

Chapter 20. Vector Spaces and Bases In this course, we have proceeded step-by-step through low-dimensional Linear Algebra. We have looked at lines, planes, hyperplanes, and have seen that there is no limit

### CHAPTER 5 Round-off errors

CHAPTER 5 Round-off errors In the two previous chapters we have seen how numbers can be represented in the binary numeral system and how this is the basis for representing numbers in computers. Since any

### Notes on Symmetric Matrices

CPSC 536N: Randomized Algorithms 2011-12 Term 2 Notes on Symmetric Matrices Prof. Nick Harvey University of British Columbia 1 Symmetric Matrices We review some basic results concerning symmetric matrices.

### MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors. Jordan canonical form (continued).

MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors Jordan canonical form (continued) Jordan canonical form A Jordan block is a square matrix of the form λ 1 0 0 0 0 λ 1 0 0 0 0 λ 0 0 J = 0

### I. GROUPS: BASIC DEFINITIONS AND EXAMPLES

I GROUPS: BASIC DEFINITIONS AND EXAMPLES Definition 1: An operation on a set G is a function : G G G Definition 2: A group is a set G which is equipped with an operation and a special element e G, called

### Chapter 17. Orthogonal Matrices and Symmetries of Space

Chapter 17. Orthogonal Matrices and Symmetries of Space Take a random matrix, say 1 3 A = 4 5 6, 7 8 9 and compare the lengths of e 1 and Ae 1. The vector e 1 has length 1, while Ae 1 = (1, 4, 7) has length

### Quotient Rings and Field Extensions

Chapter 5 Quotient Rings and Field Extensions In this chapter we describe a method for producing field extension of a given field. If F is a field, then a field extension is a field K that contains F.

### Review Jeopardy. Blue vs. Orange. Review Jeopardy

Review Jeopardy Blue vs. Orange Review Jeopardy Jeopardy Round Lectures 0-3 Jeopardy Round \$200 How could I measure how far apart (i.e. how different) two observations, y 1 and y 2, are from each other?

### Excel supplement: Chapter 7 Matrix and vector algebra

Excel supplement: Chapter 7 atrix and vector algebra any models in economics lead to large systems of linear equations. These problems are particularly suited for computers. The main purpose of this chapter

### 1 Sets and Set Notation.

LINEAR ALGEBRA MATH 27.6 SPRING 23 (COHEN) LECTURE NOTES Sets and Set Notation. Definition (Naive Definition of a Set). A set is any collection of objects, called the elements of that set. We will most

### Numerical Analysis Lecture Notes

Numerical Analysis Lecture Notes Peter J. Olver 6. Eigenvalues and Singular Values In this section, we collect together the basic facts about eigenvalues and eigenvectors. From a geometrical viewpoint,

### Recall that two vectors in are perpendicular or orthogonal provided that their dot

Orthogonal Complements and Projections Recall that two vectors in are perpendicular or orthogonal provided that their dot product vanishes That is, if and only if Example 1 The vectors in are orthogonal

### Notes on Orthogonal and Symmetric Matrices MENU, Winter 2013

Notes on Orthogonal and Symmetric Matrices MENU, Winter 201 These notes summarize the main properties and uses of orthogonal and symmetric matrices. We covered quite a bit of material regarding these topics,

### Brief Introduction to Vectors and Matrices

CHAPTER 1 Brief Introduction to Vectors and Matrices In this chapter, we will discuss some needed concepts found in introductory course in linear algebra. We will introduce matrix, vector, vector-valued

### 7 Gaussian Elimination and LU Factorization

7 Gaussian Elimination and LU Factorization In this final section on matrix factorization methods for solving Ax = b we want to take a closer look at Gaussian elimination (probably the best known method

### Vector Spaces. Chapter 2. 2.1 R 2 through R n

Chapter 2 Vector Spaces One of my favorite dictionaries (the one from Oxford) defines a vector as A quantity having direction as well as magnitude, denoted by a line drawn from its original to its final

### LINEAR ALGEBRA W W L CHEN

LINEAR ALGEBRA W W L CHEN c W W L Chen, 1997, 2008 This chapter is available free to all individuals, on understanding that it is not to be used for financial gain, and may be downloaded and/or photocopied,

### Least-Squares Intersection of Lines

Least-Squares Intersection of Lines Johannes Traa - UIUC 2013 This write-up derives the least-squares solution for the intersection of lines. In the general case, a set of lines will not intersect at a

### Lecture 5: Singular Value Decomposition SVD (1)

EEM3L1: Numerical and Analytical Techniques Lecture 5: Singular Value Decomposition SVD (1) EE3L1, slide 1, Version 4: 25-Sep-02 Motivation for SVD (1) SVD = Singular Value Decomposition Consider the system

### These axioms must hold for all vectors ū, v, and w in V and all scalars c and d.

DEFINITION: A vector space is a nonempty set V of objects, called vectors, on which are defined two operations, called addition and multiplication by scalars (real numbers), subject to the following axioms

### Section 1.1. Introduction to R n

The Calculus of Functions of Several Variables Section. Introduction to R n Calculus is the study of functional relationships and how related quantities change with each other. In your first exposure to

### Recall the basic property of the transpose (for any A): v A t Aw = v w, v, w R n.

ORTHOGONAL MATRICES Informally, an orthogonal n n matrix is the n-dimensional analogue of the rotation matrices R θ in R 2. When does a linear transformation of R 3 (or R n ) deserve to be called a rotation?

### Finite dimensional C -algebras

Finite dimensional C -algebras S. Sundar September 14, 2012 Throughout H, K stand for finite dimensional Hilbert spaces. 1 Spectral theorem for self-adjoint opertors Let A B(H) and let {ξ 1, ξ 2,, ξ n

### Linear Algebra: Determinants, Inverses, Rank

D Linear Algebra: Determinants, Inverses, Rank D 1 Appendix D: LINEAR ALGEBRA: DETERMINANTS, INVERSES, RANK TABLE OF CONTENTS Page D.1. Introduction D 3 D.2. Determinants D 3 D.2.1. Some Properties of

### Basics of Polynomial Theory

3 Basics of Polynomial Theory 3.1 Polynomial Equations In geodesy and geoinformatics, most observations are related to unknowns parameters through equations of algebraic (polynomial) type. In cases where

### α = u v. In other words, Orthogonal Projection

Orthogonal Projection Given any nonzero vector v, it is possible to decompose an arbitrary vector u into a component that points in the direction of v and one that points in a direction orthogonal to v

### LINEAR ALGEBRA. September 23, 2010

LINEAR ALGEBRA September 3, 00 Contents 0. LU-decomposition.................................... 0. Inverses and Transposes................................. 0.3 Column Spaces and NullSpaces.............................

### Solution to Homework 2

Solution to Homework 2 Olena Bormashenko September 23, 2011 Section 1.4: 1(a)(b)(i)(k), 4, 5, 14; Section 1.5: 1(a)(b)(c)(d)(e)(n), 2(a)(c), 13, 16, 17, 18, 27 Section 1.4 1. Compute the following, if

### Elementary Linear Algebra

Elementary Linear Algebra Kuttler January, Saylor URL: http://wwwsaylororg/courses/ma/ Saylor URL: http://wwwsaylororg/courses/ma/ Contents Some Prerequisite Topics Sets And Set Notation Functions Graphs

### Linear Programming. March 14, 2014

Linear Programming March 1, 01 Parts of this introduction to linear programming were adapted from Chapter 9 of Introduction to Algorithms, Second Edition, by Cormen, Leiserson, Rivest and Stein [1]. 1

### Notes on metric spaces

Notes on metric spaces 1 Introduction The purpose of these notes is to quickly review some of the basic concepts from Real Analysis, Metric Spaces and some related results that will be used in this course.

### Applied Linear Algebra I Review page 1

Applied Linear Algebra Review 1 I. Determinants A. Definition of a determinant 1. Using sum a. Permutations i. Sign of a permutation ii. Cycle 2. Uniqueness of the determinant function in terms of properties

### Mathematical Methods of Engineering Analysis

Mathematical Methods of Engineering Analysis Erhan Çinlar Robert J. Vanderbei February 2, 2000 Contents Sets and Functions 1 1 Sets................................... 1 Subsets.............................

### SYSTEMS OF EQUATIONS AND MATRICES WITH THE TI-89. by Joseph Collison

SYSTEMS OF EQUATIONS AND MATRICES WITH THE TI-89 by Joseph Collison Copyright 2000 by Joseph Collison All rights reserved Reproduction or translation of any part of this work beyond that permitted by Sections

### LEARNING OBJECTIVES FOR THIS CHAPTER

CHAPTER 2 American mathematician Paul Halmos (1916 2006), who in 1942 published the first modern linear algebra book. The title of Halmos s book was the same as the title of this chapter. Finite-Dimensional

### MAT 242 Test 2 SOLUTIONS, FORM T

MAT 242 Test 2 SOLUTIONS, FORM T 5 3 5 3 3 3 3. Let v =, v 5 2 =, v 3 =, and v 5 4 =. 3 3 7 3 a. [ points] The set { v, v 2, v 3, v 4 } is linearly dependent. Find a nontrivial linear combination of these

### Operation Count; Numerical Linear Algebra

10 Operation Count; Numerical Linear Algebra 10.1 Introduction Many computations are limited simply by the sheer number of required additions, multiplications, or function evaluations. If floating-point

### 4. How many integers between 2004 and 4002 are perfect squares?

5 is 0% of what number? What is the value of + 3 4 + 99 00? (alternating signs) 3 A frog is at the bottom of a well 0 feet deep It climbs up 3 feet every day, but slides back feet each night If it started

### Solving Linear Systems of Equations. Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.

Solving Linear Systems of Equations Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.edu These slides are a supplement to the book Numerical Methods with Matlab:

### 2.3 Convex Constrained Optimization Problems

42 CHAPTER 2. FUNDAMENTAL CONCEPTS IN CONVEX OPTIMIZATION Theorem 15 Let f : R n R and h : R R. Consider g(x) = h(f(x)) for all x R n. The function g is convex if either of the following two conditions

### December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B. KITCHENS

December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B KITCHENS The equation 1 Lines in two-dimensional space (1) 2x y = 3 describes a line in two-dimensional space The coefficients of x and y in the equation

### Lecture 5 Principal Minors and the Hessian

Lecture 5 Principal Minors and the Hessian Eivind Eriksen BI Norwegian School of Management Department of Economics October 01, 2010 Eivind Eriksen (BI Dept of Economics) Lecture 5 Principal Minors and

### A linear combination is a sum of scalars times quantities. Such expressions arise quite frequently and have the form

Section 1.3 Matrix Products A linear combination is a sum of scalars times quantities. Such expressions arise quite frequently and have the form (scalar #1)(quantity #1) + (scalar #2)(quantity #2) +...

### Algebra I Vocabulary Cards

Algebra I Vocabulary Cards Table of Contents Expressions and Operations Natural Numbers Whole Numbers Integers Rational Numbers Irrational Numbers Real Numbers Absolute Value Order of Operations Expression

### Section 4.4 Inner Product Spaces

Section 4.4 Inner Product Spaces In our discussion of vector spaces the specific nature of F as a field, other than the fact that it is a field, has played virtually no role. In this section we no longer

### Matrix Algebra. Some Basic Matrix Laws. Before reading the text or the following notes glance at the following list of basic matrix algebra laws.

Matrix Algebra A. Doerr Before reading the text or the following notes glance at the following list of basic matrix algebra laws. Some Basic Matrix Laws Assume the orders of the matrices are such that

### Unified Lecture # 4 Vectors

Fall 2005 Unified Lecture # 4 Vectors These notes were written by J. Peraire as a review of vectors for Dynamics 16.07. They have been adapted for Unified Engineering by R. Radovitzky. References [1] Feynmann,

### 1 if 1 x 0 1 if 0 x 1

Chapter 3 Continuity In this chapter we begin by defining the fundamental notion of continuity for real valued functions of a single real variable. When trying to decide whether a given function is or