# LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12,

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, Iterative methods 4.1 What a two year old child can do Suppose we want to find a number x such that cos x = x (in radians). This is a nonlinear equation, there is no explicit solution. Nevertheless a two-year old child can solve it with a calculator: just push the COS button many times... Suppose that x 0 = 0 is the initial number on the display, and x 1 =cosx 0,x 2 =cosx 1 etc. are the successive numbers. This is what you see x 0 = 0 x 1 = 1 x 2 = x 3 = x 4 = x 5 = x 6 = x 20 = x 21 = x 22 = and you see that at least the first three digits get stabilized. This will be the approximative solution to cos x = x. We also say that x is a fixed point of the function f(x) = cosx. In general the fixed point of a function f is a value x such that f(x) =x. Of course the method does not always work. Try to find a nonzero root of x 2 = x with this method. Unless you start exactly from x = 1, your iteration will either blow up or converge to zero. This means that x = 0 is a stable solution, while x = 1 is unstable. But in this case you at least got a root... Try now 1 = x. Here the situation is worse: the iteration blows up whatever close you x 2 started from the true root x = 1, unless if you started exactly from the root. This is because x = 1 is an unstable root of this equation.

2 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, Finally, try 1 x = x. Here nothing bad happens, just the procedure will never converge, the iteration alternates between two numbers: x 0 and x 1 =1/x 0. In this case the algorithm ended up in an infinite cycle. The conclusion is that whenever we try to solve an equation of the form f(x) =x (fixed-point equation), one can try the two-year-old-child stategy: start from some x 0 and iterate, i.e. generate the sequence x 1 = f(x 0 ), x 2 = f(x 1 ) x 3 = f(x 2 ) etc. If the sequence stabilizes (converges), then you found a solution. This very simple method is extremely useful, if it works... But we should emphasize, that if the sequence does not converge, then no conclusion can be made! You cannot say that the equation has no solution! Moreover, even if you found a solution, there could be other solutions. Some of them could be accessible by the same method starting from a different initial value x 0, some of them may not be accessible at all. It also could happen that for some initial value the iteration converges, for some other values does not. Whether the iteration converges or not is basically determined the derivative of f at the fixed point. If f (x fix ) < 1, then the iteration converges if you start it from a point close enough to the solution. The reason is roughly the following: From the iteration we have x (n) = f(x (n 1) ) and from the fixed point equation Subtract these equations from each other x fix = f(x fix ). x (n) x fix = f(x (n 1) ) f(x fix )

3 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, Now use Taylor expansion around x fix : f(x (n 1) ) f(x fix )+f (x fix ) ( x (n 1) x fix ) if x (n 1) is close to x fix. Hence, from these two last equations: x (n) x fix f (x fix ) x (n 1) x fix Hence, if f (x fix ) < 1, then we see that x (n) is closer to the fixpoint than x (n 1). After n iterations x (n) x fix f (x fix ) n x (0) x fix i.e. we see that the procedure converges exponentially fast if the number f (x fix ) is strictly smaller than 1. Moreover, we can give an estimate on the speed of convergence. We see that after n step the deviation of the approximate solution from the true one is reduced by a factor of f (x fix ) n. Hence if we want a certain precision ε, then we need at least n steps, where ε f (x fix ) n x (0) x fix i.e. log ε n log f (x fix ) if the error of the initial guess is at most of order one, i.e. one can assume that x (0) x fix 1 In this formula we can use logarithm of any base. For example if f (x fix ) =0.8, the error of the initial guess was smaller than 1, and we want 10 8 precision, then we need at least n lg 10 8 lg 0.8 = =82.55

4 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, To be on the safe side, one usually overestimates this number by one or two, so we need around iterations. The argument above is not quite correct, since we neglected the higher order terms in the Taylor expansion. If you start the iteration from a point close enough to the fixed point, then these higher order terms are really negligible, otherwise they could ruin the convergence. Recall that a similar phenomenon was found at the Newton s iteration. For a more rigorous discussion and nice pictures generated by a java applet, see bourbaki/1501/html/pdf/fixedpoints.pdf Finally notice that any equation of the form F (x) = 0 can be brought into a fixed-point equation just by writing it as x F (x) =x. Then the solution to F (x) = 0 is equivalent to the fixed point of f(x) :=x F(x). 4.2 Iterative methods for Ax = b Now we apply the same idea to solve Ax = b. Again, for simplicity, we assume that A is a regular square matrix, hence there is a unique solution. We have to bring the equation into a fixed-point equation and we do it by splitting the matrix A = B +(A B) with some cleverly chosen matrix B. The way to think about it is that B is the main part which will be chosen a nice matrix, and we view A as a small perturbation (by A B) of this nice matrix B. Clearly Ax = b is equivalent to Bx =(B A)x+b or x = B 1[ (B A)x + b ] =(I B 1 A)x+B 1 b

5 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, at least if B is invertible (hence we will have to choose B such that B 1 be simple). The algorithm is very simple: start with an initial vector x (0), and iteratively generate x (n) := (I B 1 A)x (n 1) + B 1 b and hope for the best... How to predict the convergence? At least is there some sufficient condition for convergence? Let δx (n) := x x (n) be the error after the n-th iteration. From Ax = b and Bx (n) = (B A)x (n 1) + b we easily see that B(δx (n) )=(B A)(δx (n 1) ) i.e. δx (n) =(I B 1 A)(δx (n 1) )=(I B 1 A) n (δx (0) ) where δx (0) is the difference of the initial value from the solution. Hence δx (n) (I B 1 A) n δx (0) (4.1) It is clear that if the norm of the matrix (I B 1 A) n converges to zero, then the iteration converges. Moreover, this norm gives an estimate on the speed of convergence. We need the following theorem: Theorem 4.1 Let H be a square matrix. Then H n 0if and only if all eigenvalues of H are less than one in absolute value. Proof: The theorem intuitively is clear; the eigenvalues are responsible for amplification of vectors. However the rigorous proof is not trivial for general matrices, so we restrict ourselves to the symmetric case.

6 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, If H is symmetric, then by the spectral theorem it can be written as H = QDQ t with some orthogonal matrix Q and diagonal matrix D. Clearly H n = QD n Q t,and H n = QD n Q t = D n (PROVE IT(*)). Since D contains the eigenvalues of H in the diagonal, the eigenvalues of D n are just the n-th power of the eigenvalues of H. The biggest of them clearly goes to zero if and only if the biggest eigenvalue of H is smaller than one in absolute value. 2 As a conclusion, we have Theorem 4.2 If A, B are both regular square matrices, then the iteration Bx (n) =(B A)x (n 1) + b converges to the the solution if all eigenvalues of I B 1 A are less than one in absolute value. The speed of convergence is exponential with a rate given by the largest (in modulus) eigenvalue of I B 1 A. REMARK 1.: The condition is not just sufficient, but also essentially necessary. But the important part is the sufficiency; it gives a criterion to decide in advance whether it is worth running iteration or not. REMARK 2.: The good news is that if the iteration converges, then it usually converges very fast. It means that there is a constant c<1 such that δx (n) c n δx 0 If I B 1 A is symmetric, then c clearly can be chosen as its biggest eigenvalue (WHY(*)?) Thenonsymmetriccaseisslightlyharder. It is well known that such an exponential estimate yields fast convergence even if c is very close to 1. For example, if c =0.99, then c 1000 e Also remember that it is usually very easy to implement an iterative algorithm. So this is a good algorithm whenever it works.

7 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, Jacobi iteration The art of iterative methods for Ax = b is the good decomposition. The simplest method is the Jacobi decomposition of a square matrix A =(a ij )as A=L+D+U where L is a lower triangular matrix, containing all elements of A strictly below the diagonal, U is its upper triangular counterpart and D is a diagonal matrix containing the diagonal elements of A. Let B = D and A B = L + U be the decomposition explained in Section 4.2. Since the diagonal matrix is easy to invert, we can easily write up the formulas: x (n) = D 1 (L + U)x (n 1) + D 1 b (4.2) or in coordinates x (n) i = 1 a ii ( bi j:j i a ij x (n 1) ) j The summation is taken over all j s except j = i. The initial vector x (0) can be chosen to be 0 unless we have a better a-priori guess for the true solution. Does this work? One could use Theorem 4.2 to check the eigenvalues of D 1 (L + U), but this may not be so easy. The following theorem is not optimal, but it gives a very easily verifiable condition for the convergence of the Jacobi iteration: Theorem 4.3 Suppose that A is diagonal dominant, that is a ii > a ij (4.3) for all i. Then the Jacobi iteration converges. j:j i

8 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, We do not give the rigorous proof of this theorem. We just remark that the condition (4.3) says that in every row the diagonal entry is bigger (in absolute value) than the sum of all the other entries (in absolute value) in the row. In other words, the matrix is close to a diagonal matrix in a sense that the matrix L + U containing the offdiagonal terms is smaller that the diagonal matrix D. The speed of convergence of the Jacobi iteration is given by the largest (in modulus) eigenvalue of D 1 (L + U). This directly follows Theorem 4.2. ( ) 1 2 EXERCISE: Consider the matrix A =. Show that it does not satisfy the condition 0 1 of Theorem 4.3, but the Jacobi iteration still converges. (Hint: Use Theorem 4.2). The condition (4.3) is quite restrictive, especially for big matrices. However it could be useful for sparse matrices, especially for matrices with special structure. When partial differential equations are solved by discretization, then the arising big matrix is usually almost diagonal in a sense that the only nonzero entries are one or two steps away from the diagonal, e.g., for the one step case a ij =0if i j >1 (this is also called tridiagonal matrix). In this case the condition (4.3) is not so restrictive, since it compares the diagonal entry a ii with only two other nonzero entries a i,i 1 and a i,i+1. Problem 4.4 Find the solution to ( ) ( x = 2 3 8) first with Gaussian elimination, then with Jacobi iteration up to two digits (10 2 precision) starting from the zero vector. Give an estimate on the number of steps to reach 8 digit precision. ( ) 1 SOLUTION: The true solution is x = either from Gauss, or just by looking at it. 2

9 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, ( ) 4 1 Then first notice that the matrix A = is diagonally dominant, hence Jacobi 2 3 ( ) 0 algorithm will converge from any initial vector. Let x (0) = and compute the next step 0 according to the Jacobi formula: (BE CAREFUL WITH INDICES) x (n) 1 = 1 (6 1 x(n 1) 2 ) 4 x (n) 2 = 1 (8 2 x(n 1) 1 ) 3 We compute each number up to three digits to detect the stabilization of the second digit x (1) 1 = 1 4 (6 1 0) = 3 2 =1.5 x (1) 2 = 1 3 (8 2 0) = 8 3 =2.33 The next iteration gives Next: Next Next x (2) 1 = 1 4 ( )=5 6 =0.833 x (2) 2 = 1 3 ( )=5 3 =1.66 x (3) 1 = 1 4 ( )=13 12 =1.08 x (3) 2 = 1 3 ( )=19 9 =2.1 x (4) 1 = 1 19 ( )=35 36 =0.97 x (4) 2 = 1 13 ( )=35 18 =1.94 x (5) 1 = 1 ( ) =

10 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, x (5) 2 = 1 ( ) = Next x (6) 1 = 1 ( ) = x (6) 2 = 1 ( ) = and x (7) 1 = 1 ( ) = x (7) 2 = 1 ( ) = Since we see that the error in the successive steps, x (6) x (7), is less than 1%, we can stop. There are many different rules to stop the algorithm, they usually differ from each other by a few steps. The important thing is that the true solution is unknown, so the true error, x (n) x true, must be estimated by the error in the successive steps, i.e. x (n) x (n+1), but usually this is reliable. To give an estimate on the number of steps to get 10 8 precision, we have to compute the largest eigenvalue of M := D 1 (L + U) = ( ) 1 ( ) 0 1 = 2 0 ( 0 ) 1/4 2/3 0 The eigenvalues are λ = ±1/ 6, hence the speed of convergence is determined by 1/ 6(you have to choose the one bigger in absolute value, but both eigenvalues have the same absolute value here) This means that the absolute error gets diminished roughly (not exactly since this is not the norm) by a factor of 1/ 6 after every step: x (n) x true ( 1 6 ) n x (0) x true

11 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, Here x (0) x true is of order one (this is the typical case), you can even forget about its exact value, replace it by 1. So to get a 10 8 precision you have to solve 10 8 ( 1 6 ) n i.e. n 8ln10/ln(1/ 6) As a rule of thumb you add 2-3 more steps to fix the order 1 type argument (one can make it more precise), so to get 10 8 digit precision one needs approximately steps Gauss-Seidel iteration There are several other (a bit more refined) iterative methods, relying on slightly different decomposition of A. Gauss-Seidel method uses B = L + D, andthen-th step of the iteration is x (n) =(L+D) 1( b Ux (n 1)) or (CHECK that equivalent) x (n) = D 1( b Lx (n) Ux (n 1)) or, in coordinates x (n) i = 1 ( i 1 bi a ij x (n) j a ii j=1 n j=i+1 a ij x (n 1) ) j (CHECK(*) that this is really the correct formula). This is very similar to the Jacobi iteration (see (4.2)), but in many cases it works better. Roughly speaking, Jacobi method considers A = L+D+U as a perturbation of its diagonal D, while Gauss-Seidel considers A = L+D+U as a perturbation of L + D. It turns out that Gauss-Seidel is also easier to implement on a computer (WHY?). From theoretical point of view we have the analogue of Theorem 4.3

12 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, Theorem 4.5 The Gauss-Seidel iteration converges for diagonal dominant matrices, i.e. for matrices satisfying (4.3). The speed of convergence of the Gauss-Seidel iteration is given by the largest (in modulus) eigenvalue of (L + D) 1 U. This directly follows Theorem 4.2. Problem 4.6 Consider the same problem as in the previous section, i.e. find the solution to ( ) ( x = 2 3 8) but now with Gauss-Seidel iteration up to two digits (10 2 precision) starting from the zero vector. Give an estimate on the number of steps to reach 8 digit precision. SOLUTION: We just follow the formula x (n) 1 = 1 (6 1 x(n 1) 2 ) 4 x (n) 2 = 1 (8 2 x(n) 1 ) 3 (Notice the deviation from Jacobi, the in the second line you use x (n) 1 instead of x (n 1) 1!) We generate the iteration: x (1) 1 = 1 4 (6 1 0) = 3 2 =1.5 Next: x (1) 2 = 1 3 ( )=5 3 =1.66 x (2) 1 = 1 4 ( )=13 12 =1.08 x (2) 2 = 1 13 ( )=35 18 =1.94

13 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, Next: x (3) 1 = 1 ( ) = x (3) 2 = 1 ( ) = Next x (4) 1 = 1 ( ) = x (4) 2 = 1 ( ) = and we can stop. Notice that the iteration was faster (around twice as fast) as Jacobi. To estimate the number of steps for large precision, we need to find the eigenvalues of M =(L+D) 1 U= ( ) ( ) ( ) /4 = /6 which are 0 and 1/6, hence the biggest absolute value is 1/6. To compute the number of steps we solve 10 8 (1/6) n i.e. n = 8ln10/ln(1/6) = 10.28, hence after steps we reach the 10 8 precision. Notice that the eigenvalue of the Gauss-Seidel algorithm was the square of the corresponding eigenvalue of Jacobi algorithm, i.e., the iteration will be twice as fast. This is not true for general matrices, but it is true for tridiagonal ones. There are several improvements of the Gauss-Seidel method, the most useful one is called Gauss-Seidel with relaxation. In general, relaxation is a method to avoid overshooting. Very intuitively; suppose that you run an iterative method in one dimension, the sequence of iteratively computed numbers x (n) is used to approximate the true solution x. The nicest situation is when you monotonically approach to x, from one direction: x (1) x (2) x (3) x. Numerically this situation is the most stable. But it could happen that you approach to the

14 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, solution in an alternating way; e.g., x (1) <x,thenx (2) >xand then again x (3) <xetc., and hopefully still x x (n) decreases to zero. Such a scheme carries the potential danger of overshooting; x (2) is on the other side of x and it could be farther away from x than x (1). In this case it looks wise to replace x (2), for example, by the average (x (1) + x (2) )/2 (or some weighted average) since this is clearly closer to x. Such a procedure is called relaxation and in particular this idea has been carefully implemented for the Gauss-Seidel iteration. It is worth to compare Jacobi and Gauss-Seidel iterations; this is the goal of one of the computer projects. Based upon a test on random matrices, it turns out that (i) Diagonal dominance is a restrictive sufficient condition: these iterative methods converge for much more matrices. (ii) In general one cannot directly compare the two methods. However, for tridiagonal matrices Jacobi and Gauss-Seidel converge or diverge simultaneously, and when they converge, then Gauss-Seidel is around twice as fast. This can also be proven rigorously. Here are some results of a program by Nolan Leaky. Solving 100 random 3 by 3 general system up to precision 10 3 Number of diagonally dominant: 1 Number of matrices for which Jacobi converged: 12 Average number of steps: Number of matrices for which Gauss-Seidel converged: 22 Average number of steps: Solving 100 random 4 by 4 general system up to precision 10 3 Number of diagonally dominant: 0 Number of matrices for which Jacobi converged: 2

15 LINEAR ALGEBRA: NUMERICAL METHODS. Version: August 12, Average number of steps: 124 Number of matrices for which Gauss-Seidel converged: 6 Average number of steps: As you can expect, the situation for tridiagonal matrices is much better: Solving 100 random 3 by 3 tridiagonal system up to precision 10 3 Number of diagonally dominant: 5 Number of matrices for which Jacobi converged: 30 Average number of steps: 49.4 Number of matrices for which Gauss-Seidel converged: 30 Average number of steps: 24.4 Solving 100 random 4 by 4 system up to precision 10 3 Number of diagonally dominant: 1 Number of matrices for which Jacobi converged: 14 Average number of steps: Number of matrices for which Gauss-Seidel converged: 14 Average number of steps: 33.9

### 10.2 ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS. The Jacobi Method

578 CHAPTER 1 NUMERICAL METHODS 1. ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS As a numerical technique, Gaussian elimination is rather unusual because it is direct. That is, a solution is obtained after

### General Framework for an Iterative Solution of Ax b. Jacobi s Method

2.6 Iterative Solutions of Linear Systems 143 2.6 Iterative Solutions of Linear Systems Consistent linear systems in real life are solved in one of two ways: by direct calculation (using a matrix factorization,

### Some Notes on Taylor Polynomials and Taylor Series

Some Notes on Taylor Polynomials and Taylor Series Mark MacLean October 3, 27 UBC s courses MATH /8 and MATH introduce students to the ideas of Taylor polynomials and Taylor series in a fairly limited

### 7 Gaussian Elimination and LU Factorization

7 Gaussian Elimination and LU Factorization In this final section on matrix factorization methods for solving Ax = b we want to take a closer look at Gaussian elimination (probably the best known method

### SOLVING LINEAR SYSTEMS

SOLVING LINEAR SYSTEMS Linear systems Ax = b occur widely in applied mathematics They occur as direct formulations of real world problems; but more often, they occur as a part of the numerical analysis

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### Diagonal, Symmetric and Triangular Matrices

Contents 1 Diagonal, Symmetric Triangular Matrices 2 Diagonal Matrices 2.1 Products, Powers Inverses of Diagonal Matrices 2.1.1 Theorem (Powers of Matrices) 2.2 Multiplying Matrices on the Left Right by

### DETERMINANTS. b 2. x 2

DETERMINANTS 1 Systems of two equations in two unknowns A system of two equations in two unknowns has the form a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 This can be written more concisely in

### Similarity and Diagonalization. Similar Matrices

MATH022 Linear Algebra Brief lecture notes 48 Similarity and Diagonalization Similar Matrices Let A and B be n n matrices. We say that A is similar to B if there is an invertible n n matrix P such that

### MATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1.

MATH10212 Linear Algebra Textbook: D. Poole, Linear Algebra: A Modern Introduction. Thompson, 2006. ISBN 0-534-40596-7. Systems of Linear Equations Definition. An n-dimensional vector is a row or a column

### Iterative Methods for Solving Linear Systems

Chapter 5 Iterative Methods for Solving Linear Systems 5.1 Convergence of Sequences of Vectors and Matrices In Chapter 2 we have discussed some of the main methods for solving systems of linear equations.

### Diagonalisation. Chapter 3. Introduction. Eigenvalues and eigenvectors. Reading. Definitions

Chapter 3 Diagonalisation Eigenvalues and eigenvectors, diagonalisation of a matrix, orthogonal diagonalisation fo symmetric matrices Reading As in the previous chapter, there is no specific essential

### Eigenvalues and eigenvectors of a matrix

Eigenvalues and eigenvectors of a matrix Definition: If A is an n n matrix and there exists a real number λ and a non-zero column vector V such that AV = λv then λ is called an eigenvalue of A and V is

### Iterative Methods for Linear Systems

Iterative Methods for Linear Systems We present the basic concepts for three of the most fundamental and well known iterative techniques for solving a system of linear equations of the form Ax = b. Iterative

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +

### Chapter 6. Orthogonality

6.3 Orthogonal Matrices 1 Chapter 6. Orthogonality 6.3 Orthogonal Matrices Definition 6.4. An n n matrix A is orthogonal if A T A = I. Note. We will see that the columns of an orthogonal matrix must be

### Linear Dependence Tests

Linear Dependence Tests The book omits a few key tests for checking the linear dependence of vectors. These short notes discuss these tests, as well as the reasoning behind them. Our first test checks

### Lecture 3: Finding integer solutions to systems of linear equations

Lecture 3: Finding integer solutions to systems of linear equations Algorithmic Number Theory (Fall 2014) Rutgers University Swastik Kopparty Scribe: Abhishek Bhrushundi 1 Overview The goal of this lecture

### Lecture 5: Singular Value Decomposition SVD (1)

EEM3L1: Numerical and Analytical Techniques Lecture 5: Singular Value Decomposition SVD (1) EE3L1, slide 1, Version 4: 25-Sep-02 Motivation for SVD (1) SVD = Singular Value Decomposition Consider the system

### Solution of Linear Systems

Chapter 3 Solution of Linear Systems In this chapter we study algorithms for possibly the most commonly occurring problem in scientific computing, the solution of linear systems of equations. We start

### Operation Count; Numerical Linear Algebra

10 Operation Count; Numerical Linear Algebra 10.1 Introduction Many computations are limited simply by the sheer number of required additions, multiplications, or function evaluations. If floating-point

### 10.3 POWER METHOD FOR APPROXIMATING EIGENVALUES

58 CHAPTER NUMERICAL METHODS. POWER METHOD FOR APPROXIMATING EIGENVALUES In Chapter 7 you saw that the eigenvalues of an n n matrix A are obtained by solving its characteristic equation n c nn c nn...

### We know a formula for and some properties of the determinant. Now we see how the determinant can be used.

Cramer s rule, inverse matrix, and volume We know a formula for and some properties of the determinant. Now we see how the determinant can be used. Formula for A We know: a b d b =. c d ad bc c a Can we

### Introduction to Matrix Algebra I

Appendix A Introduction to Matrix Algebra I Today we will begin the course with a discussion of matrix algebra. Why are we studying this? We will use matrix algebra to derive the linear regression model

### Linear Algebra Review Part 2: Ax=b

Linear Algebra Review Part 2: Ax=b Edwin Olson University of Michigan The Three-Day Plan Geometry of Linear Algebra Vectors, matrices, basic operations, lines, planes, homogeneous coordinates, transformations

### Solving Systems of Linear Equations

LECTURE 5 Solving Systems of Linear Equations Recall that we introduced the notion of matrices as a way of standardizing the expression of systems of linear equations In today s lecture I shall show how

### Inner Product Spaces

Math 571 Inner Product Spaces 1. Preliminaries An inner product space is a vector space V along with a function, called an inner product which associates each pair of vectors u, v with a scalar u, v, and

### We seek a factorization of a square matrix A into the product of two matrices which yields an

LU Decompositions We seek a factorization of a square matrix A into the product of two matrices which yields an efficient method for solving the system where A is the coefficient matrix, x is our variable

### 8 Square matrices continued: Determinants

8 Square matrices continued: Determinants 8. Introduction Determinants give us important information about square matrices, and, as we ll soon see, are essential for the computation of eigenvalues. You

### 1 Eigenvalues and Eigenvectors

Math 20 Chapter 5 Eigenvalues and Eigenvectors Eigenvalues and Eigenvectors. Definition: A scalar λ is called an eigenvalue of the n n matrix A is there is a nontrivial solution x of Ax = λx. Such an x

### AN INTRODUCTION TO NUMERICAL METHODS AND ANALYSIS

AN INTRODUCTION TO NUMERICAL METHODS AND ANALYSIS Revised Edition James Epperson Mathematical Reviews BICENTENNIAL 0, 1 8 0 7 z ewiley wu 2007 r71 BICENTENNIAL WILEY-INTERSCIENCE A John Wiley & Sons, Inc.,

### Notes on Orthogonal and Symmetric Matrices MENU, Winter 2013

Notes on Orthogonal and Symmetric Matrices MENU, Winter 201 These notes summarize the main properties and uses of orthogonal and symmetric matrices. We covered quite a bit of material regarding these topics,

### UNIT - I LESSON - 1 The Solution of Numerical Algebraic and Transcendental Equations

UNIT - I LESSON - 1 The Solution of Numerical Algebraic and Transcendental Equations Contents: 1.0 Aims and Objectives 1.1 Introduction 1.2 Bisection Method 1.2.1 Definition 1.2.2 Computation of real root

### Lecture 1: Schur s Unitary Triangularization Theorem

Lecture 1: Schur s Unitary Triangularization Theorem This lecture introduces the notion of unitary equivalence and presents Schur s theorem and some of its consequences It roughly corresponds to Sections

### MATH 2300 review problems for Exam 3 ANSWERS

MATH 300 review problems for Exam 3 ANSWERS. Check whether the following series converge or diverge. In each case, justify your answer by either computing the sum or by by showing which convergence test

### Lectures 5-6: Taylor Series

Math 1d Instructor: Padraic Bartlett Lectures 5-: Taylor Series Weeks 5- Caltech 213 1 Taylor Polynomials and Series As we saw in week 4, power series are remarkably nice objects to work with. In particular,

### Taylor Polynomials and Taylor Series Math 126

Taylor Polynomials and Taylor Series Math 26 In many problems in science and engineering we have a function f(x) which is too complicated to answer the questions we d like to ask. In this chapter, we will

### Au = = = 3u. Aw = = = 2w. so the action of A on u and w is very easy to picture: it simply amounts to a stretching by 3 and 2, respectively.

Chapter 7 Eigenvalues and Eigenvectors In this last chapter of our exploration of Linear Algebra we will revisit eigenvalues and eigenvectors of matrices, concepts that were already introduced in Geometry

### by the matrix A results in a vector which is a reflection of the given

Eigenvalues & Eigenvectors Example Suppose Then So, geometrically, multiplying a vector in by the matrix A results in a vector which is a reflection of the given vector about the y-axis We observe that

### Vector and Matrix Norms

Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty

### Notes on Determinant

ENGG2012B Advanced Engineering Mathematics Notes on Determinant Lecturer: Kenneth Shum Lecture 9-18/02/2013 The determinant of a system of linear equations determines whether the solution is unique, without

### Matrix Norms. Tom Lyche. September 28, Centre of Mathematics for Applications, Department of Informatics, University of Oslo

Matrix Norms Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo September 28, 2009 Matrix Norms We consider matrix norms on (C m,n, C). All results holds for

### 1. LINEAR EQUATIONS. A linear equation in n unknowns x 1, x 2,, x n is an equation of the form

1. LINEAR EQUATIONS A linear equation in n unknowns x 1, x 2,, x n is an equation of the form a 1 x 1 + a 2 x 2 + + a n x n = b, where a 1, a 2,..., a n, b are given real numbers. For example, with x and

### 15.062 Data Mining: Algorithms and Applications Matrix Math Review

.6 Data Mining: Algorithms and Applications Matrix Math Review The purpose of this document is to give a brief review of selected linear algebra concepts that will be useful for the course and to develop

### Solution to Homework 2

Solution to Homework 2 Olena Bormashenko September 23, 2011 Section 1.4: 1(a)(b)(i)(k), 4, 5, 14; Section 1.5: 1(a)(b)(c)(d)(e)(n), 2(a)(c), 13, 16, 17, 18, 27 Section 1.4 1. Compute the following, if

### Basic Terminology for Systems of Equations in a Nutshell. E. L. Lady. 3x 1 7x 2 +4x 3 =0 5x 1 +8x 2 12x 3 =0.

Basic Terminology for Systems of Equations in a Nutshell E L Lady A system of linear equations is something like the following: x 7x +4x =0 5x +8x x = Note that the number of equations is not required

### Abstract: We describe the beautiful LU factorization of a square matrix (or how to write Gaussian elimination in terms of matrix multiplication).

MAT 2 (Badger, Spring 202) LU Factorization Selected Notes September 2, 202 Abstract: We describe the beautiful LU factorization of a square matrix (or how to write Gaussian elimination in terms of matrix

### ALGEBRAIC EIGENVALUE PROBLEM

ALGEBRAIC EIGENVALUE PROBLEM BY J. H. WILKINSON, M.A. (Cantab.), Sc.D. Technische Universes! Dsrmstedt FACHBEREICH (NFORMATiK BIBL1OTHEK Sachgebieto:. Standort: CLARENDON PRESS OXFORD 1965 Contents 1.

### Elementary Number Theory We begin with a bit of elementary number theory, which is concerned

CONSTRUCTION OF THE FINITE FIELDS Z p S. R. DOTY Elementary Number Theory We begin with a bit of elementary number theory, which is concerned solely with questions about the set of integers Z = {0, ±1,

### Similar matrices and Jordan form

Similar matrices and Jordan form We ve nearly covered the entire heart of linear algebra once we ve finished singular value decompositions we ll have seen all the most central topics. A T A is positive

### SECTION 8.3: THE INVERSE OF A SQUARE MATRIX

(Section 8.3: The Inverse of a Square Matrix) 8.47 SECTION 8.3: THE INVERSE OF A SQUARE MATRIX PART A: (REVIEW) THE INVERSE OF A REAL NUMBER If a is a nonzero real number, then aa 1 = a 1 a = 1. a 1, or

### 5.3 Determinants and Cramer s Rule

290 5.3 Determinants and Cramer s Rule Unique Solution of a 2 2 System The 2 2 system (1) ax + by = e, cx + dy = f, has a unique solution provided = ad bc is nonzero, in which case the solution is given

### [1] Diagonal factorization

8.03 LA.6: Diagonalization and Orthogonal Matrices [ Diagonal factorization [2 Solving systems of first order differential equations [3 Symmetric and Orthonormal Matrices [ Diagonal factorization Recall:

### 2x + y = 3. Since the second equation is precisely the same as the first equation, it is enough to find x and y satisfying the system

1. Systems of linear equations We are interested in the solutions to systems of linear equations. A linear equation is of the form 3x 5y + 2z + w = 3. The key thing is that we don t multiply the variables

### Matrix Algebra and Applications

Matrix Algebra and Applications Dudley Cooke Trinity College Dublin Dudley Cooke (Trinity College Dublin) Matrix Algebra and Applications 1 / 49 EC2040 Topic 2 - Matrices and Matrix Algebra Reading 1 Chapters

### Factorization Theorems

Chapter 7 Factorization Theorems This chapter highlights a few of the many factorization theorems for matrices While some factorization results are relatively direct, others are iterative While some factorization

### Orthogonal Diagonalization of Symmetric Matrices

MATH10212 Linear Algebra Brief lecture notes 57 Gram Schmidt Process enables us to find an orthogonal basis of a subspace. Let u 1,..., u k be a basis of a subspace V of R n. We begin the process of finding

### The Inverse of a Matrix

The Inverse of a Matrix 7.4 Introduction In number arithmetic every number a ( 0) has a reciprocal b written as a or such that a ba = ab =. Some, but not all, square matrices have inverses. If a square

### Lecture 6. Inverse of Matrix

Lecture 6 Inverse of Matrix Recall that any linear system can be written as a matrix equation In one dimension case, ie, A is 1 1, then can be easily solved as A x b Ax b x b A 1 A b A 1 b provided that

### Cofactor Expansion: Cramer s Rule

Cofactor Expansion: Cramer s Rule MATH 322, Linear Algebra I J. Robert Buchanan Department of Mathematics Spring 2015 Introduction Today we will focus on developing: an efficient method for calculating

### MATH 4330/5330, Fourier Analysis Section 11, The Discrete Fourier Transform

MATH 433/533, Fourier Analysis Section 11, The Discrete Fourier Transform Now, instead of considering functions defined on a continuous domain, like the interval [, 1) or the whole real line R, we wish

### MATH 551 - APPLIED MATRIX THEORY

MATH 55 - APPLIED MATRIX THEORY FINAL TEST: SAMPLE with SOLUTIONS (25 points NAME: PROBLEM (3 points A web of 5 pages is described by a directed graph whose matrix is given by A Do the following ( points

### Numerical Methods I Solving Linear Systems: Sparse Matrices, Iterative Methods and Non-Square Systems

Numerical Methods I Solving Linear Systems: Sparse Matrices, Iterative Methods and Non-Square Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course G63.2010.001 / G22.2420-001,

### CURVE FITTING LEAST SQUARES APPROXIMATION

CURVE FITTING LEAST SQUARES APPROXIMATION Data analysis and curve fitting: Imagine that we are studying a physical system involving two quantities: x and y Also suppose that we expect a linear relationship

### 6. Cholesky factorization

6. Cholesky factorization EE103 (Fall 2011-12) triangular matrices forward and backward substitution the Cholesky factorization solving Ax = b with A positive definite inverse of a positive definite matrix

### Change of Continuous Random Variable

Change of Continuous Random Variable All you are responsible for from this lecture is how to implement the Engineer s Way (see page 4) to compute how the probability density function changes when we make

### The Characteristic Polynomial

Physics 116A Winter 2011 The Characteristic Polynomial 1 Coefficients of the characteristic polynomial Consider the eigenvalue problem for an n n matrix A, A v = λ v, v 0 (1) The solution to this problem

### Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur

Probability and Statistics Prof. Dr. Somesh Kumar Department of Mathematics Indian Institute of Technology, Kharagpur Module No. #01 Lecture No. #15 Special Distributions-VI Today, I am going to introduce

### The Phase Plane. Phase portraits; type and stability classifications of equilibrium solutions of systems of differential equations

The Phase Plane Phase portraits; type and stability classifications of equilibrium solutions of systems of differential equations Phase Portraits of Linear Systems Consider a systems of linear differential

### G.A. Pavliotis. Department of Mathematics. Imperial College London

EE1 MATHEMATICS NUMERICAL METHODS G.A. Pavliotis Department of Mathematics Imperial College London 1. Numerical solution of nonlinear equations (iterative processes). 2. Numerical evaluation of integrals.

### Chapter 17. Orthogonal Matrices and Symmetries of Space

Chapter 17. Orthogonal Matrices and Symmetries of Space Take a random matrix, say 1 3 A = 4 5 6, 7 8 9 and compare the lengths of e 1 and Ae 1. The vector e 1 has length 1, while Ae 1 = (1, 4, 7) has length

### FIXED POINT ITERATION

FIXED POINT ITERATION We begin with a computational example. solving the two equations Consider E1: x =1+.5sinx E2: x =3+2sinx Graphs of these two equations are shown on accompanying graphs, with the solutions

### 3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field

3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field 77 3.8 Finding Antiderivatives; Divergence and Curl of a Vector Field Overview: The antiderivative in one variable calculus is an important

### 6.8 Taylor and Maclaurin s Series

6.8. TAYLOR AND MACLAURIN S SERIES 357 6.8 Taylor and Maclaurin s Series 6.8.1 Introduction The previous section showed us how to find the series representation of some functions by using the series representation

### LS.6 Solution Matrices

LS.6 Solution Matrices In the literature, solutions to linear systems often are expressed using square matrices rather than vectors. You need to get used to the terminology. As before, we state the definitions

### SYSTEMS OF EQUATIONS AND MATRICES WITH THE TI-89. by Joseph Collison

SYSTEMS OF EQUATIONS AND MATRICES WITH THE TI-89 by Joseph Collison Copyright 2000 by Joseph Collison All rights reserved Reproduction or translation of any part of this work beyond that permitted by Sections

### Eigenvalues and Eigenvectors

Chapter 6 Eigenvalues and Eigenvectors 6. Introduction to Eigenvalues Linear equations Ax D b come from steady state problems. Eigenvalues have their greatest importance in dynamic problems. The solution

### Section 2.1. Section 2.2. Exercise 6: We have to compute the product AB in two ways, where , B =. 2 1 3 5 A =

Section 2.1 Exercise 6: We have to compute the product AB in two ways, where 4 2 A = 3 0 1 3, B =. 2 1 3 5 Solution 1. Let b 1 = (1, 2) and b 2 = (3, 1) be the columns of B. Then Ab 1 = (0, 3, 13) and

### Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model

Overview of Violations of the Basic Assumptions in the Classical Normal Linear Regression Model 1 September 004 A. Introduction and assumptions The classical normal linear regression model can be written

### Linear Algebra Notes for Marsden and Tromba Vector Calculus

Linear Algebra Notes for Marsden and Tromba Vector Calculus n-dimensional Euclidean Space and Matrices Definition of n space As was learned in Math b, a point in Euclidean three space can be thought of

### Chapter 31 out of 37 from Discrete Mathematics for Neophytes: Number Theory, Probability, Algorithms, and Other Stuff by J. M.

31 Geometric Series Motivation (I hope) Geometric series are a basic artifact of algebra that everyone should know. 1 I am teaching them here because they come up remarkably often with Markov chains. The

### Definition: A square matrix A is block diagonal if A has the form A 1 O O O A 2 O A =

The question we want to answer now is the following: If A is not similar to a diagonal matrix, then what is the simplest matrix that A is similar to? Before we can provide the answer, we will have to introduce

### Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components

Eigenvalues, Eigenvectors, Matrix Factoring, and Principal Components The eigenvalues and eigenvectors of a square matrix play a key role in some important operations in statistics. In particular, they

### Computational Optical Imaging - Optique Numerique. -- Deconvolution --

Computational Optical Imaging - Optique Numerique -- Deconvolution -- Winter 2014 Ivo Ihrke Deconvolution Ivo Ihrke Outline Deconvolution Theory example 1D deconvolution Fourier method Algebraic method

### a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2.

Chapter 1 LINEAR EQUATIONS 1.1 Introduction to linear equations A linear equation in n unknowns x 1, x,, x n is an equation of the form a 1 x 1 + a x + + a n x n = b, where a 1, a,..., a n, b are given

### 22 Matrix exponent. Equal eigenvalues

22 Matrix exponent. Equal eigenvalues 22. Matrix exponent Consider a first order differential equation of the form y = ay, a R, with the initial condition y) = y. Of course, we know that the solution to

### A note on companion matrices

Linear Algebra and its Applications 372 (2003) 325 33 www.elsevier.com/locate/laa A note on companion matrices Miroslav Fiedler Academy of Sciences of the Czech Republic Institute of Computer Science Pod

### CS3220 Lecture Notes: QR factorization and orthogonal transformations

CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss

### The last three chapters introduced three major proof techniques: direct,

CHAPTER 7 Proving Non-Conditional Statements The last three chapters introduced three major proof techniques: direct, contrapositive and contradiction. These three techniques are used to prove statements

### Numerical Methods I Eigenvalue Problems

Numerical Methods I Eigenvalue Problems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course G63.2010.001 / G22.2420-001, Fall 2010 September 30th, 2010 A. Donev (Courant Institute)

### Discrete Mathematics and Probability Theory Fall 2009 Satish Rao, David Tse Note 2

CS 70 Discrete Mathematics and Probability Theory Fall 2009 Satish Rao, David Tse Note 2 Proofs Intuitively, the concept of proof should already be familiar We all like to assert things, and few of us

### LINEAR SYSTEMS. Consider the following example of a linear system:

LINEAR SYSTEMS Consider the following example of a linear system: Its unique solution is x +2x 2 +3x 3 = 5 x + x 3 = 3 3x + x 2 +3x 3 = 3 x =, x 2 =0, x 3 = 2 In general we want to solve n equations in

### LINEAR ALGEBRA. September 23, 2010

LINEAR ALGEBRA September 3, 00 Contents 0. LU-decomposition.................................... 0. Inverses and Transposes................................. 0.3 Column Spaces and NullSpaces.............................

### 3. INNER PRODUCT SPACES

. INNER PRODUCT SPACES.. Definition So far we have studied abstract vector spaces. These are a generalisation of the geometric spaces R and R. But these have more structure than just that of a vector space.

### Limit processes are the basis of calculus. For example, the derivative. f f (x + h) f (x)

SEC. 4.1 TAYLOR SERIES AND CALCULATION OF FUNCTIONS 187 Taylor Series 4.1 Taylor Series and Calculation of Functions Limit processes are the basis of calculus. For example, the derivative f f (x + h) f

### 4. MATRICES Matrices

4. MATRICES 170 4. Matrices 4.1. Definitions. Definition 4.1.1. A matrix is a rectangular array of numbers. A matrix with m rows and n columns is said to have dimension m n and may be represented as follows:

### 9.2 Summation Notation

9. Summation Notation 66 9. Summation Notation In the previous section, we introduced sequences and now we shall present notation and theorems concerning the sum of terms of a sequence. We begin with a

### Lies My Calculator and Computer Told Me

Lies My Calculator and Computer Told Me 2 LIES MY CALCULATOR AND COMPUTER TOLD ME Lies My Calculator and Computer Told Me See Section.4 for a discussion of graphing calculators and computers with graphing