# Practical Numerical Training UKNum

Save this PDF as:

Size: px
Start display at page:

## Transcription

1 Practical Numerical Training UKNum 7: Systems of linear equations C. Mordasini Max Planck Institute for Astronomy, Heidelberg Program: 1) Introduction 2) Gauss Elimination 3) Gauss with Pivoting 4) Determinants 5) LU Decomposition 6) Inverse of a matrix with LU

2 1 Introduction

3 Task One of the most important numerical tasks: Solve a system of linear equations. a 11 x 1 + a 12 x 2 + a 13 x a 1N x N = b 1 a 21 x 1 + a 22 x 2 + a 23 x a 2N x N = b 2 a 31 x 1 + a 32 x 2 + a 33 x a 3N x N = b 3 a M1 x 1 + a M2 x 2 + a M3 x a MN x N = b M N unknowns: x j, j =1, 2,...,N M equations, =12 with the known coefficients = a ij with i =1, 2,...,M and j =1, 2,...,N =12 And known right hand side b i, i =1, 2,...,M.

4 (Non-)Singular systems If N=M: same number of equations and unknowns. Chance to find a unique solution. But not always: One or several of the M equations is a linear combination of the other ones (row degeneracy). All equations contain certain variables only in exactly the same linear combination (column degeneracy). For square matrixes, one implies the other. Such matrixes are called singular. This is an analytical problem. Additional conditions arise from the numerical treatment of the task.

5 Numerical issues While not exact linear combinations of each other, some of the equations may be so close to linearly dependent that roundoff errors in the machine render them linearly dependent at some stage in the solution process. In this case your numerical procedure will fail, even if there would be an analytical solution. Accumulated roundoff errors in the solution process can swamp the true solution. This problem particularly emerges if N is too large. The numerical procedure does not fail algorithmically. However, it returns a set of x s that are wrong, as can be discovered by direct substitution back into the original equations. The closer a set of equations is to being singular, the more likely this is to happen, since increasingly close cancellations will occur during the solution. In fact, the preceding item can be viewed as the special case where the loss of significance is unfortunately total. Press et al.

6 Matrix form Our master equation is usually written in the form of a matrix equation: A x = b A is the matrix of coefficients, and b is the right-hand side written as a column vector A = a 11 a a 1N a 21 a a 2N a M1 a M2... a MN b = b 1 b 2 b M By convention, the first index on an element aij denotes its row, the second index its column.

7 Special matrixes In this context, several special types of matrixes are important: 1) Unit matrix A diagonal matrix with all diagonal elements equal to one is called an identity/unit matrix. 2)Upper triangular matrix All the elements below the diagonal entries are zero.

8 Special matrixes II 3)Lower triangular matrix All the elements above the diagonal entries are zero. 4)Tridiagonal matrix A tridiagonal matrix is a square matrix in which all elements not on the following are zero - the major diagonal, the diagonal above the major diagonal, and the diagonal below the major diagonal.

9 Other special matrixes In many situations (e.g. solutions of partial differential equations in two or more dimensions), one is dealing with so called sparse linear systems, i.e. matrixes where only a small fraction of all matrix elements are nonzero. For such matrixes, one should use specialized solvers, as they can be much more rapid than general purpose solvers. zeros zeros zeros (a) (b) (c) (d) (e) (f ) For large systems (N~ ), one wants to consider pre-written specialized libraries, like LAPACK which is freely available. (g) (h) (i) ( j) (k) Press et al.

10 Methods to solve In this lecture, we consider the following methods: Naive Gaussian Elimination Gaussian Elimination with Partial Pivoting LU Decomposition (Gauss Seidel Iteration) For large systems (N~ ), one wants to consider pre-written specialized libraries, like LAPACK which is freely available. They have also been parallelized and ported onto GPUs.

11 2 Gaussian Elimination

12 Naive Gaussian elimination Two step procedure: Forward elimination: In this step, an unknown is eliminated in each equation starting with the first equation. This way, the equations are reduced to one unknown in each equation. Back Substitution: In this step, starting from the last equation, each of the unknowns is found.

13 Forward elimination goal The goal of forward elimination is to transform the coefficient matrix into an upper triangular matrix:

14 Initial equations A set of n equations and n unknowns To bring this to the form of an upper triangular matrix, (n-1) steps of forward elimination are needed.

15 Forward elimination step 1 a In the first step of forward elimination, the first unknown, x1 is eliminated from all rows below the first row. The first equation is selected as the pivot equation to eliminate x1. So, to eliminate x1 in the second equation, one divides the first equation by a11 (hence called the pivot element) and then multiplies it by a21. This is the same as multiplying the first equation by a21/a11 to give

16 Forward elimination step 1 b Now, this equation can be subtracted from the second equation to give Note that we have eliminated in the new second equation x1.

17 Forward elimination step 1 c We can write this new second equation also as where a " = a a 21 a a a " = a a 21 2n 2n a 11 a 1n

18 Forward elimination step 1 d Repeat this procedure of eliminating x1 for the remaining equations (3 to n) to reduce the set of equations as This is the end of step 1.

19 Forward elimination step 2 For the second step of forward elimination, we start with the second equation as the pivot equation and a 22 as the pivot element. To eliminate x2 in the third equation, one thus divides the second equation by a 22 (the pivot element) and then multiply it by a 32. Then we subtract this from the third equation. This makes the coefficient of x2 zero in the third equation. The same procedure is now repeated for the fourth equation until the n th equation to give......

20 Forward elimination further steps The next steps of forward elimination are conducted by using the third equation as a pivot equation and so on. Thus, there will be a total of n-1 steps of forward elimination. At the end of n-1 steps of forward elimination, we get a set of equations that look like......

21 Back substitution Example of a system of 3 equations: # & % ( % ( \$ % '( # % % \$ % x 1 x 2 x 3 & ( ( '( = # & % ( % 96.21( \$ % '( Solve each equation starting (obviously) from the last equation as it has only one unknown: We have now determined a first unknown quantity.

22 Back substitution II Then the second last equation, that is the (n-1)th equation, has two unknowns: xn and xn-1, but xn is already known from just before. This reduces the (n-1)th equation also to one unknown. We now iteratively work up to the first equation. Back substitution hence can be represented for all equations by the formula x i = b ( i 1 ) ( i 1 i a ) ( i 1 i,i+1 x i+1 a ) ( i 1 i,i+2 x i+2... a ) i,n x n ( i 1) for i = n 1,...,1 a ii x i = ( ) i 1 a ij b i i 1 n j =i+1 i 1 a ii ( ) x j ( ) for i = n 1,...,1

23 Example I The upward velocity of a rocket is given at three different times The velocity data is approximated by a polynomial as: Find the velocity at t=6 seconds.

24 Example II We need to determine the unknown coefficients ai. This results in a matrix of the form (Vandermonde matrix) Using data from the table, the matrix becomes:

25 Example III We can also write this as: 1) Forward elimination Number of steps of forward elimination is (n 1)=(3 1)=2

26 Example IV Forward elimination step 1 Divide Equation 1 by 25 and multiply it by 64,. [ ] 2.56 = [ ] Subtract the result from Equation 2 Substitute new equation for Eq. 2

27 Example V Forward elimination step 1 continued Divide Equation 1 by 25 and multiply it by 144,. [ ] 5.76 = [ ] Subtract the result from Equation 3 Substitute new equation for Equation 3

28 Example VI Forward elimination step 2 Divide Equation 2 by 4.8 and multiply it by 16.8,! [ ] 3.5 = [ ] Subtract the result from Equation 3 Substitute new equation for Equation 3

29 Example VII Back substitution # & % ( % ( \$ % '( # % % \$ % a 1 a 2 a 3 & ( ( '( = # & % ( % ( \$ % 0.76 '( Solving for a 3 Solving for a 2

30 Example VIII Back substitution continued # & % ( % ( \$ % '( # % % \$ % a 1 a 2 a 3 & ( ( '( = # & % ( % ( \$ % 0.76 '( Solving for a 1

31 Example IX Solution The solution vector is The polynomial that passes through the three data points is then:

32 Possible issues There are two pitfalls of the Naive Gauss elimination method. 1) Division by zero Here, we cannot even start the algorithm as we would have to devise by zero. This problem can also occur later on:

33 Possible issues II Here we can start the algorithm. However, at the first step of forward elimination, we get the following equations in matrix form Now at the beginning of the 2nd step of forward elimination, the coefficient of x2 in Equation 2 would be used as the pivot element. That element is zero and hence would create the division by zero problem. Division by zero is a possibility at any step of forward elimination.

34 Possible issues III 2) Large round-off errors The Naive Gauss elimination method is prone to round-off errors. This is true when there are large numbers of equations as errors propagate. Also, if there is subtraction of numbers from each other, it may create large errors. Example: Exact Solution

35 Possible issues III Solve it on a computer using 6 significant digits with chopping. One finds Solve it on a computer using 5 significant digits with chopping Is there a way to reduce the round off error? Obviously, increase the number of significant digits (always use double precision). This Decreases round-off error Does not avoid division by zero

36 3 Gaussian Elimination with Partial Pivoting

37 Partial pivoting Gaussian Elimination with Partial Pivoting Avoids division by zero Reduces round off error This method is an improvement of the algorithm we just saw before. What is Different About Partial Pivoting? At the beginning of the k th step of forward elimination, find the maximum of If the maximum of the values is in the p th row, then switch rows p and k. Otherwise, exactly the same algorithm as naive Gauss elimination except that we switch rows before each of the (n-1) steps of forward elimination.

38 Switching rows Matrix Form at Beginning of 2 nd Step of Forward Elimination Switched Rows

39 Example We solve the following set of equations by Gaussian elimination with partial pivoting Number of steps of forward elimination is again (n 1)=(3 1)=2.

40 Example II Forward Elimination: Step 1 Examine absolute values of first column, first row and below. Largest absolute value is 144 and exists in row 3. Switch row 1 and row 3. Begin forward elimination Divide Equation 1 by 144 and multiply it by 64,.

41 Example III Subtract the result from Equation 2 [ ] = [ ] Substitute new equation for Equation 2 Equation 3 Divide Equation 1 by 144 and multiply it by 25,. [ ] = [ ]

42 Example IV Subtract the result from Equation 3 Substitute new equation for Equation 3 Forward Elimination: Step 2 Examine absolute values of second column, second row and below Largest absolute value is and exists in row 3. Switch row 2 and row 3.

43 Example V Divide Equation 2 by and multiply it by 2.667, [ ] = [ ] Subtract the. result from Equation 3 Substitute new equation for Equation 3

44 Example VI Back Substitution Solve exactly as before for a 3 then for a 2 and finally for a 1

45 Round off errors Comparison with/without pivoting We studied earlier the system Exact Solution We found for Gaussian Elimination without pivoting, and 5 significant digits With pivoting, and 5 digits, one finds in contrast the exact solution. The fact that the round off error is fully removed here is by coincidence only, the general trend of a smaller error is however not a coincidence.

46 4 Determinants of a Square Matrix

47 Determinants Using naive Gauss Eliminations to find the determinant of a square matrix One of the more efficient ways to find the determinant of a square matrix is by taking advantage of the following two theorems on a determinant of matrices coupled with Naive Gauss elimination. Theorem 1 Theorem 2 If a multiple of one row of [A] nxn is added or subtracted to another row of [A] nxn to result in [B] nxn then det(a)=det(b). The same is true for column operations also. The determinant of an upper triangular, lower triangular or diagonal matrix [A] nxn is given by

48 Determinants II This implies that if we apply the forward elimination steps of the Naive Gauss elimination method, the determinant of the matrix stays the same according to Theorem 1. Then since at the end of the forward elimination steps, the resulting matrix is upper triangular, the determinant will be given by Theorem 2.

49 Example Find the determinant of (rocket example) For a 3 x 3 Matrix we of course know directly that But we can also use (in particular for higher dimensions) that with Naive Gaussian Elimination, we get the upper triangular matrix. In this example, we found earlier

50 Example II Therefore, according to theorem 1 and 2, the determinant is = 25 ( 4.8) 0.7

51 Determinants and pivoting What if one cannot find the determinant of the matrix using the Naive Gauss elimination method, for example, because one gets division by zero problems during the Naïve Gauss elimination method? In this case, one can apply Gaussian elimination with partial pivoting. However, the determinant of the resulting upper triangular matrix may differ by a sign. The following theorem applies in addition to the previous two to find the determinant of a square matrix. Theorem 3 Let [A] nxn be a n x n matrix. Then, if [B] nxn is a matrix that results from switching one row with another row, then det(a)=-det(b).

52 5 LU Decomposition

53 LU Decomposition LU Decomposition is another method to solve a set of simultaneous linear equations Which is better, Gauss Elimination or LU Decomposition? To answer this, a closer look at LU decomposition is needed.

54 LU Decomposition Method For most non-singular matrix [A] that one could conduct Naive Gauss Elimination forward elimination steps, one can write it as where [L] = lower triangular matrix [U] = upper triangular matrix [A] = [L][U]

55 LU Decomposition idea Note: [Z] is a n x 1 matrix, i.e. a vector

56 LU Decomposition algorithm How can this be used? Given [A][X] = [C] 1. Decompose [A] into [L] and [U] 2. Solve [L][Z] = [C] for [Z] 3. Solve [U][X] = [Z] for [X] As we will see below, both solving equation 2 and 3 is very simple. Eq. 2 is solved using forward substitution and then we use Eq. 3 to calculate the solution vector [X] by back substitution.

57 Computational time To solve [A][X] = [B] Gaussian Elimination LU Decomposition T=clock cycle time n = size of the matrix So both methods are equally efficient. However, for calculating the inverse of a matrix, LU can be much faster (cf below)

58 Method: Decompose [A] to [L] and [U] [ A] = [ L] [ U] = " 1 0 0% " u u u \$ ' \$ \$ ' \$ 0 u u # \$ &' # \$ 0 0 u 33 % ' ' &' [U] is the same as the coefficient matrix at the end of the forward elimination step during Naive Gauss elimination. [L] is obtained using the multipliers that were used in the forward elimination process

59 Example: Find the [U] matrix I Using the Forward Elimination Procedure like for Gauss Elimination Step 1:

60 Example: Find the [U] matrix II Matrix after Step 1: Step 2:

61 Example: Find the [L] matrix I Using the multipliers used during the Forward Elimination Procedure From the first step of forward elimination

62 Example: Find the [L] matrix II From the second step of forward elimination The [L] matrix is thus obtained for free if we do Naive Gaussian Elimination.

63 Does [L][U] = [A]? [ L] [ U] = =? This can always be checked by simple matrix multiplication.

64 LU Decomposition to solve SLEs I Solve the following set of linear equations using LU Decomposition Step 1 Use the algorithm we just saw for finding the [L] and [U] matrices [ A] = [ L] [ U] =

65 LU Decomposition to solve SLEs II Step 2 Set [L][Z] = [C] and then solve for [Z]. Due to the particular shape of [L], this is very simple by forward substitution starting from the first equation: z 1 = z 1 + z 2 = z z 2 + z 3 = 279.2

66 LU Decomposition to solve SLEs III Step 2 continued Complete the forward substitution to solve for [Z]

67 LU Decomposition to solve SLEs IV Step 3 Set [U][X] = [Z] Solve for [X] The 3 equations become Again, due to the particular shape of [U], this is very simple by backward substitution starting from the last equation.

68 LU Decomposition to solve SLEs V Step 3 continued From the 3 rd equation Substituting in a 3 and using the second equation Substituting in a 3 and a 2 in the first equation Hence the Solution Vector [X] is:

69 6 Inverse of a Matrix with LU Decomposition

70 Finding the inverse of a square matrix The inverse [B] of a square matrix [A] is defined as [A][B] = [I] = [B][A] How can LU Decomposition be used to find the inverse? Assume the first column of [B] to be [b 11 b 12 b n1 ] T. Use this and the definition of matrix multiplication First column of [B] Second column of [B] The remaining columns in [B] can be found in the same manner. Thus, we now determine [B] in one column after another, where such a column is taking the role of the [X] vector in the last examples.

71 Example: Inverse of a Matrix I Find the inverse of a square matrix [A] Step 1: Using the decomposition procedure, the [L] and [U] matrices are found to be [ A] = [ L] [ U] = These two matrixes will now be used for all columns of [B].

72 Example: Inverse of a Matrix II As before solving for the each column of [B] requires two furhter steps 2) Solve [L] [Z] = [C] for [Z] 3) Solve [U] [X] = [Z] for [X] Step 2: [ L] [ Z] = [ C] This generates the equations: z 1 z 2 z 3 = 1 0 0

73 Example: Inverse of a Matrix III Step 2: Solving [U][X] = [Z] for [X] This generates the equations: So the first column of the inverse of [A] is:

74 Example: Inverse of a Matrix IV Repeating for the second and third columns of the inverse Second Column Third Column Final result: The inverse of [A] is To check: [A][A] -1 = [I] = [A] -1 [A]

75 Inverse of a Matrix with LU For calculations of columns of the inverse of the matrix, the LU decomposition needs to be done only once, then the forward substitution n times, and the back substitution n times. In comparison, if Gaussian elimination method were used to find the inverse of a matrix, the forward elimination as well as the back substitution will have to be done n times. Time taken by Gaussian Elimination Time taken by LU Decomposition For large n, the term n 4 will become dominant. Comparing computational times of finding inverse of a matrix using LU decomposition and Gaussian elimination. n CT inverse GE / CT inverse LU

76 References This script is based on by Autar Kaw, Jai Paul and Numerical Recipes (2nd/3rd Edition) by Press et al., Cambridge University Press

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS. + + x 2. x n. a 11 a 12 a 1n b 1 a 21 a 22 a 2n b 2 a 31 a 32 a 3n b 3. a m1 a m2 a mn b m

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS 1. SYSTEMS OF EQUATIONS AND MATRICES 1.1. Representation of a linear system. The general system of m equations in n unknowns can be written a 11 x 1 + a 12 x 2 +

### MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS

MATRIX ALGEBRA AND SYSTEMS OF EQUATIONS Systems of Equations and Matrices Representation of a linear system The general system of m equations in n unknowns can be written a x + a 2 x 2 + + a n x n b a

### 7. LU factorization. factor-solve method. LU factorization. solving Ax = b with A nonsingular. the inverse of a nonsingular matrix

7. LU factorization EE103 (Fall 2011-12) factor-solve method LU factorization solving Ax = b with A nonsingular the inverse of a nonsingular matrix LU factorization algorithm effect of rounding error sparse

### SOLVING LINEAR SYSTEMS

SOLVING LINEAR SYSTEMS Linear systems Ax = b occur widely in applied mathematics They occur as direct formulations of real world problems; but more often, they occur as a part of the numerical analysis

### Abstract: We describe the beautiful LU factorization of a square matrix (or how to write Gaussian elimination in terms of matrix multiplication).

MAT 2 (Badger, Spring 202) LU Factorization Selected Notes September 2, 202 Abstract: We describe the beautiful LU factorization of a square matrix (or how to write Gaussian elimination in terms of matrix

### 7 Gaussian Elimination and LU Factorization

7 Gaussian Elimination and LU Factorization In this final section on matrix factorization methods for solving Ax = b we want to take a closer look at Gaussian elimination (probably the best known method

### Direct Methods for Solving Linear Systems. Matrix Factorization

Direct Methods for Solving Linear Systems Matrix Factorization Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University c 2011

### Lecture 6. Inverse of Matrix

Lecture 6 Inverse of Matrix Recall that any linear system can be written as a matrix equation In one dimension case, ie, A is 1 1, then can be easily solved as A x b Ax b x b A 1 A b A 1 b provided that

### Operation Count; Numerical Linear Algebra

10 Operation Count; Numerical Linear Algebra 10.1 Introduction Many computations are limited simply by the sheer number of required additions, multiplications, or function evaluations. If floating-point

### Solution of Linear Systems

Chapter 3 Solution of Linear Systems In this chapter we study algorithms for possibly the most commonly occurring problem in scientific computing, the solution of linear systems of equations. We start

### 6. Cholesky factorization

6. Cholesky factorization EE103 (Fall 2011-12) triangular matrices forward and backward substitution the Cholesky factorization solving Ax = b with A positive definite inverse of a positive definite matrix

### Linear Dependence Tests

Linear Dependence Tests The book omits a few key tests for checking the linear dependence of vectors. These short notes discuss these tests, as well as the reasoning behind them. Our first test checks

### We seek a factorization of a square matrix A into the product of two matrices which yields an

LU Decompositions We seek a factorization of a square matrix A into the product of two matrices which yields an efficient method for solving the system where A is the coefficient matrix, x is our variable

### by the matrix A results in a vector which is a reflection of the given

Eigenvalues & Eigenvectors Example Suppose Then So, geometrically, multiplying a vector in by the matrix A results in a vector which is a reflection of the given vector about the y-axis We observe that

### 2.1: MATRIX OPERATIONS

.: MATRIX OPERATIONS What are diagonal entries and the main diagonal of a matrix? What is a diagonal matrix? When are matrices equal? Scalar Multiplication 45 Matrix Addition Theorem (pg 0) Let A, B, and

### Solving Systems of Linear Equations

LECTURE 5 Solving Systems of Linear Equations Recall that we introduced the notion of matrices as a way of standardizing the expression of systems of linear equations In today s lecture I shall show how

### LINEAR SYSTEMS. Consider the following example of a linear system:

LINEAR SYSTEMS Consider the following example of a linear system: Its unique solution is x +2x 2 +3x 3 = 5 x + x 3 = 3 3x + x 2 +3x 3 = 3 x =, x 2 =0, x 3 = 2 In general we want to solve n equations in

### DETERMINANTS. b 2. x 2

DETERMINANTS 1 Systems of two equations in two unknowns A system of two equations in two unknowns has the form a 11 x 1 + a 12 x 2 = b 1 a 21 x 1 + a 22 x 2 = b 2 This can be written more concisely in

### 10.2 ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS. The Jacobi Method

578 CHAPTER 1 NUMERICAL METHODS 1. ITERATIVE METHODS FOR SOLVING LINEAR SYSTEMS As a numerical technique, Gaussian elimination is rather unusual because it is direct. That is, a solution is obtained after

### Solving Linear Systems, Continued and The Inverse of a Matrix

, Continued and The of a Matrix Calculus III Summer 2013, Session II Monday, July 15, 2013 Agenda 1. The rank of a matrix 2. The inverse of a square matrix Gaussian Gaussian solves a linear system by reducing

### The Inverse of a Matrix

The Inverse of a Matrix 7.4 Introduction In number arithmetic every number a ( 0) has a reciprocal b written as a or such that a ba = ab =. Some, but not all, square matrices have inverses. If a square

### Elementary Matrices and The LU Factorization

lementary Matrices and The LU Factorization Definition: ny matrix obtained by performing a single elementary row operation (RO) on the identity (unit) matrix is called an elementary matrix. There are three

### Solving Systems of Linear Equations

LECTURE 5 Solving Systems of Linear Equations Recall that we introduced the notion of matrices as a way of standardizing the expression of systems of linear equations In today s lecture I shall show how

### Matrices Worksheet. Adding the results together, using the matrices, gives

Matrices Worksheet This worksheet is designed to help you increase your confidence in handling MATRICES. This worksheet contains both theory and exercises which cover. Introduction. Order, Addition and

### 4. Matrix inverses. left and right inverse. linear independence. nonsingular matrices. matrices with linearly independent columns

L. Vandenberghe EE133A (Spring 2016) 4. Matrix inverses left and right inverse linear independence nonsingular matrices matrices with linearly independent columns matrices with linearly independent rows

### General Framework for an Iterative Solution of Ax b. Jacobi s Method

2.6 Iterative Solutions of Linear Systems 143 2.6 Iterative Solutions of Linear Systems Consistent linear systems in real life are solved in one of two ways: by direct calculation (using a matrix factorization,

### Introduction to Matrix Algebra I

Appendix A Introduction to Matrix Algebra I Today we will begin the course with a discussion of matrix algebra. Why are we studying this? We will use matrix algebra to derive the linear regression model

### University of Lille I PC first year list of exercises n 7. Review

University of Lille I PC first year list of exercises n 7 Review Exercise Solve the following systems in 4 different ways (by substitution, by the Gauss method, by inverting the matrix of coefficients

### Lecture Notes: Matrix Inverse. 1 Inverse Definition

Lecture Notes: Matrix Inverse Yufei Tao Department of Computer Science and Engineering Chinese University of Hong Kong taoyf@cse.cuhk.edu.hk Inverse Definition We use I to represent identity matrices,

### SYSTEMS OF EQUATIONS AND MATRICES WITH THE TI-89. by Joseph Collison

SYSTEMS OF EQUATIONS AND MATRICES WITH THE TI-89 by Joseph Collison Copyright 2000 by Joseph Collison All rights reserved Reproduction or translation of any part of this work beyond that permitted by Sections

### 9. Numerical linear algebra background

Convex Optimization Boyd & Vandenberghe 9. Numerical linear algebra background matrix structure and algorithm complexity solving linear equations with factored matrices LU, Cholesky, LDL T factorization

### 1. LINEAR EQUATIONS. A linear equation in n unknowns x 1, x 2,, x n is an equation of the form

1. LINEAR EQUATIONS A linear equation in n unknowns x 1, x 2,, x n is an equation of the form a 1 x 1 + a 2 x 2 + + a n x n = b, where a 1, a 2,..., a n, b are given real numbers. For example, with x and

### Factorization Theorems

Chapter 7 Factorization Theorems This chapter highlights a few of the many factorization theorems for matrices While some factorization results are relatively direct, others are iterative While some factorization

### Diagonal, Symmetric and Triangular Matrices

Contents 1 Diagonal, Symmetric Triangular Matrices 2 Diagonal Matrices 2.1 Products, Powers Inverses of Diagonal Matrices 2.1.1 Theorem (Powers of Matrices) 2.2 Multiplying Matrices on the Left Right by

### Vector and Matrix Norms

Chapter 1 Vector and Matrix Norms 11 Vector Spaces Let F be a field (such as the real numbers, R, or complex numbers, C) with elements called scalars A Vector Space, V, over the field F is a non-empty

### Using row reduction to calculate the inverse and the determinant of a square matrix

Using row reduction to calculate the inverse and the determinant of a square matrix Notes for MATH 0290 Honors by Prof. Anna Vainchtein 1 Inverse of a square matrix An n n square matrix A is called invertible

### LINEAR ALGEBRA. September 23, 2010

LINEAR ALGEBRA September 3, 00 Contents 0. LU-decomposition.................................... 0. Inverses and Transposes................................. 0.3 Column Spaces and NullSpaces.............................

### 1 Introduction to Matrices

1 Introduction to Matrices In this section, important definitions and results from matrix algebra that are useful in regression analysis are introduced. While all statements below regarding the columns

### Solving Linear Systems of Equations. Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.

Solving Linear Systems of Equations Gerald Recktenwald Portland State University Mechanical Engineering Department gerry@me.pdx.edu These slides are a supplement to the book Numerical Methods with Matlab:

### CS3220 Lecture Notes: QR factorization and orthogonal transformations

CS3220 Lecture Notes: QR factorization and orthogonal transformations Steve Marschner Cornell University 11 March 2009 In this lecture I ll talk about orthogonal matrices and their properties, discuss

### 1 Determinants and the Solvability of Linear Systems

1 Determinants and the Solvability of Linear Systems In the last section we learned how to use Gaussian elimination to solve linear systems of n equations in n unknowns The section completely side-stepped

### 1 Eigenvalues and Eigenvectors

Math 20 Chapter 5 Eigenvalues and Eigenvectors Eigenvalues and Eigenvectors. Definition: A scalar λ is called an eigenvalue of the n n matrix A is there is a nontrivial solution x of Ax = λx. Such an x

### Applied Linear Algebra I Review page 1

Applied Linear Algebra Review 1 I. Determinants A. Definition of a determinant 1. Using sum a. Permutations i. Sign of a permutation ii. Cycle 2. Uniqueness of the determinant function in terms of properties

### UNIT - I LESSON - 1 The Solution of Numerical Algebraic and Transcendental Equations

UNIT - I LESSON - 1 The Solution of Numerical Algebraic and Transcendental Equations Contents: 1.0 Aims and Objectives 1.1 Introduction 1.2 Bisection Method 1.2.1 Definition 1.2.2 Computation of real root

### Scientific Computing: An Introductory Survey

Scientific Computing: An Introductory Survey Chapter 3 Linear Least Squares Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction

### MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors. Jordan canonical form (continued).

MATH 423 Linear Algebra II Lecture 38: Generalized eigenvectors Jordan canonical form (continued) Jordan canonical form A Jordan block is a square matrix of the form λ 1 0 0 0 0 λ 1 0 0 0 0 λ 0 0 J = 0

### Helpsheet. Giblin Eunson Library MATRIX ALGEBRA. library.unimelb.edu.au/libraries/bee. Use this sheet to help you:

Helpsheet Giblin Eunson Library ATRIX ALGEBRA Use this sheet to help you: Understand the basic concepts and definitions of matrix algebra Express a set of linear equations in matrix notation Evaluate determinants

### 1 VECTOR SPACES AND SUBSPACES

1 VECTOR SPACES AND SUBSPACES What is a vector? Many are familiar with the concept of a vector as: Something which has magnitude and direction. an ordered pair or triple. a description for quantities such

### December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B. KITCHENS

December 4, 2013 MATH 171 BASIC LINEAR ALGEBRA B KITCHENS The equation 1 Lines in two-dimensional space (1) 2x y = 3 describes a line in two-dimensional space The coefficients of x and y in the equation

### Typical Linear Equation Set and Corresponding Matrices

EWE: Engineering With Excel Larsen Page 1 4. Matrix Operations in Excel. Matrix Manipulations: Vectors, Matrices, and Arrays. How Excel Handles Matrix Math. Basic Matrix Operations. Solving Systems of

### Introduction to Matrix Algebra

Psychology 7291: Multivariate Statistics (Carey) 8/27/98 Matrix Algebra - 1 Introduction to Matrix Algebra Definitions: A matrix is a collection of numbers ordered by rows and columns. It is customary

### SECTION 8.3: THE INVERSE OF A SQUARE MATRIX

(Section 8.3: The Inverse of a Square Matrix) 8.47 SECTION 8.3: THE INVERSE OF A SQUARE MATRIX PART A: (REVIEW) THE INVERSE OF A REAL NUMBER If a is a nonzero real number, then aa 1 = a 1 a = 1. a 1, or

### Systems of Linear Equations

Systems of Linear Equations Beifang Chen Systems of linear equations Linear systems A linear equation in variables x, x,, x n is an equation of the form a x + a x + + a n x n = b, where a, a,, a n and

### LU Decomposition. The original equation is to solve. Ax b =0ff At the end of the Gauss elimination, the resulting equations were

LU Decomposition LU decomposition is a better way to implement Gauss elimination, especially for repeated solving a number of equations with the same left-hand side That is, for solving the equation Ax

### 10.3 POWER METHOD FOR APPROXIMATING EIGENVALUES

58 CHAPTER NUMERICAL METHODS. POWER METHOD FOR APPROXIMATING EIGENVALUES In Chapter 7 you saw that the eigenvalues of an n n matrix A are obtained by solving its characteristic equation n c nn c nn...

### Solving Systems of Linear Equations. Substitution

Solving Systems of Linear Equations There are two basic methods we will use to solve systems of linear equations: Substitution Elimination We will describe each for a system of two equations in two unknowns,

### 13 MATH FACTS 101. 2 a = 1. 7. The elements of a vector have a graphical interpretation, which is particularly easy to see in two or three dimensions.

3 MATH FACTS 0 3 MATH FACTS 3. Vectors 3.. Definition We use the overhead arrow to denote a column vector, i.e., a linear segment with a direction. For example, in three-space, we write a vector in terms

### B such that AB = I and BA = I. (We say B is an inverse of A.) Definition A square matrix A is invertible (or nonsingular) if matrix

Matrix inverses Recall... Definition A square matrix A is invertible (or nonsingular) if matrix B such that AB = and BA =. (We say B is an inverse of A.) Remark Not all square matrices are invertible.

### Lecture 2 Matrix Operations

Lecture 2 Matrix Operations transpose, sum & difference, scalar multiplication matrix multiplication, matrix-vector product matrix inverse 2 1 Matrix transpose transpose of m n matrix A, denoted A T or

### Question 2: How do you solve a matrix equation using the matrix inverse?

Question : How do you solve a matrix equation using the matrix inverse? In the previous question, we wrote systems of equations as a matrix equation AX B. In this format, the matrix A contains the coefficients

### 8 Square matrices continued: Determinants

8 Square matrices continued: Determinants 8. Introduction Determinants give us important information about square matrices, and, as we ll soon see, are essential for the computation of eigenvalues. You

### Lecture 3: Finding integer solutions to systems of linear equations

Lecture 3: Finding integer solutions to systems of linear equations Algorithmic Number Theory (Fall 2014) Rutgers University Swastik Kopparty Scribe: Abhishek Bhrushundi 1 Overview The goal of this lecture

### a 11 x 1 + a 12 x 2 + + a 1n x n = b 1 a 21 x 1 + a 22 x 2 + + a 2n x n = b 2.

Chapter 1 LINEAR EQUATIONS 1.1 Introduction to linear equations A linear equation in n unknowns x 1, x,, x n is an equation of the form a 1 x 1 + a x + + a n x n = b, where a 1, a,..., a n, b are given

### Linearly Independent Sets and Linearly Dependent Sets

These notes closely follow the presentation of the material given in David C. Lay s textbook Linear Algebra and its Applications (3rd edition). These notes are intended primarily for in-class presentation

### 4. MATRICES Matrices

4. MATRICES 170 4. Matrices 4.1. Definitions. Definition 4.1.1. A matrix is a rectangular array of numbers. A matrix with m rows and n columns is said to have dimension m n and may be represented as follows:

### Iterative Methods for Linear Systems

Iterative Methods for Linear Systems We present the basic concepts for three of the most fundamental and well known iterative techniques for solving a system of linear equations of the form Ax = b. Iterative

### Math 313 Lecture #10 2.2: The Inverse of a Matrix

Math 1 Lecture #10 2.2: The Inverse of a Matrix Matrix algebra provides tools for creating many useful formulas just like real number algebra does. For example, a real number a is invertible if there is

### Linear Equations ! 25 30 35\$ & " 350 150% & " 11,750 12,750 13,750% MATHEMATICS LEARNING SERVICE Centre for Learning and Professional Development

MathsTrack (NOTE Feb 2013: This is the old version of MathsTrack. New books will be created during 2013 and 2014) Topic 4 Module 9 Introduction Systems of to Matrices Linear Equations Income = Tickets!

### MATH10212 Linear Algebra. Systems of Linear Equations. Definition. An n-dimensional vector is a row or a column of n numbers (or letters): a 1.

MATH10212 Linear Algebra Textbook: D. Poole, Linear Algebra: A Modern Introduction. Thompson, 2006. ISBN 0-534-40596-7. Systems of Linear Equations Definition. An n-dimensional vector is a row or a column

### Notes on Determinant

ENGG2012B Advanced Engineering Mathematics Notes on Determinant Lecturer: Kenneth Shum Lecture 9-18/02/2013 The determinant of a system of linear equations determines whether the solution is unique, without

### Matrices, Determinants and Linear Systems

September 21, 2014 Matrices A matrix A m n is an array of numbers in rows and columns a 11 a 12 a 1n r 1 a 21 a 22 a 2n r 2....... a m1 a m2 a mn r m c 1 c 2 c n We say that the dimension of A is m n (we

### Section 2.1. Section 2.2. Exercise 6: We have to compute the product AB in two ways, where , B =. 2 1 3 5 A =

Section 2.1 Exercise 6: We have to compute the product AB in two ways, where 4 2 A = 3 0 1 3, B =. 2 1 3 5 Solution 1. Let b 1 = (1, 2) and b 2 = (3, 1) be the columns of B. Then Ab 1 = (0, 3, 13) and

### Situation 23: Simultaneous Equations Prepared at the University of Georgia EMAT 6500 class Date last revised: July 22 nd, 2013 Nicolina Scarpelli

Situation 23: Simultaneous Equations Prepared at the University of Georgia EMAT 6500 class Date last revised: July 22 nd, 2013 Nicolina Scarpelli Prompt: A mentor teacher and student teacher are discussing

### Row Echelon Form and Reduced Row Echelon Form

These notes closely follow the presentation of the material given in David C Lay s textbook Linear Algebra and its Applications (3rd edition) These notes are intended primarily for in-class presentation

### MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix.

MATH 304 Linear Algebra Lecture 18: Rank and nullity of a matrix. Nullspace Let A = (a ij ) be an m n matrix. Definition. The nullspace of the matrix A, denoted N(A), is the set of all n-dimensional column

### Matrices Summary. To add or subtract matrices they must be the same dimensions. Just add or subtract the corresponding numbers.

Matrices Summary To transpose a matrix write the rows as columns. Academic Skills Advice For example: 2 1 A = [ 1 2 1 0 0 9] A T = 4 2 2 1 2 1 1 0 4 0 9 2 To add or subtract matrices they must be the same

### The Characteristic Polynomial

Physics 116A Winter 2011 The Characteristic Polynomial 1 Coefficients of the characteristic polynomial Consider the eigenvalue problem for an n n matrix A, A v = λ v, v 0 (1) The solution to this problem

### Matrix Algebra and Applications

Matrix Algebra and Applications Dudley Cooke Trinity College Dublin Dudley Cooke (Trinity College Dublin) Matrix Algebra and Applications 1 / 49 EC2040 Topic 2 - Matrices and Matrix Algebra Reading 1 Chapters

### Linear Algebra: Determinants, Inverses, Rank

D Linear Algebra: Determinants, Inverses, Rank D 1 Appendix D: LINEAR ALGEBRA: DETERMINANTS, INVERSES, RANK TABLE OF CONTENTS Page D.1. Introduction D 3 D.2. Determinants D 3 D.2.1. Some Properties of

### Matrices 2. Solving Square Systems of Linear Equations; Inverse Matrices

Matrices 2. Solving Square Systems of Linear Equations; Inverse Matrices Solving square systems of linear equations; inverse matrices. Linear algebra is essentially about solving systems of linear equations,

### Numerical Methods I Solving Linear Systems: Sparse Matrices, Iterative Methods and Non-Square Systems

Numerical Methods I Solving Linear Systems: Sparse Matrices, Iterative Methods and Non-Square Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course G63.2010.001 / G22.2420-001,

### 2x + y = 3. Since the second equation is precisely the same as the first equation, it is enough to find x and y satisfying the system

1. Systems of linear equations We are interested in the solutions to systems of linear equations. A linear equation is of the form 3x 5y + 2z + w = 3. The key thing is that we don t multiply the variables

### Au = = = 3u. Aw = = = 2w. so the action of A on u and w is very easy to picture: it simply amounts to a stretching by 3 and 2, respectively.

Chapter 7 Eigenvalues and Eigenvectors In this last chapter of our exploration of Linear Algebra we will revisit eigenvalues and eigenvectors of matrices, concepts that were already introduced in Geometry

### Methods for Finding Bases

Methods for Finding Bases Bases for the subspaces of a matrix Row-reduction methods can be used to find bases. Let us now look at an example illustrating how to obtain bases for the row space, null space,

Hill Cipher Project K80TTQ1EP-??,VO.L,XU0H5BY,_71ZVPKOE678_X,N2Y-8HI4VS,,6Z28DDW5N7ADY013 Directions: Answer all numbered questions completely. Show non-trivial work in the space provided. Non-computational

### 1 Lecture: Integration of rational functions by decomposition

Lecture: Integration of rational functions by decomposition into partial fractions Recognize and integrate basic rational functions, except when the denominator is a power of an irreducible quadratic.

### Generalized Inverse Computation Based on an Orthogonal Decomposition Methodology.

International Conference on Mathematical and Statistical Modeling in Honor of Enrique Castillo. June 28-30, 2006 Generalized Inverse Computation Based on an Orthogonal Decomposition Methodology. Patricia

### Basic Terminology for Systems of Equations in a Nutshell. E. L. Lady. 3x 1 7x 2 +4x 3 =0 5x 1 +8x 2 12x 3 =0.

Basic Terminology for Systems of Equations in a Nutshell E L Lady A system of linear equations is something like the following: x 7x +4x =0 5x +8x x = Note that the number of equations is not required

### We shall turn our attention to solving linear systems of equations. Ax = b

59 Linear Algebra We shall turn our attention to solving linear systems of equations Ax = b where A R m n, x R n, and b R m. We already saw examples of methods that required the solution of a linear system

### Lecture 1: Systems of Linear Equations

MTH Elementary Matrix Algebra Professor Chao Huang Department of Mathematics and Statistics Wright State University Lecture 1 Systems of Linear Equations ² Systems of two linear equations with two variables

### Lecture 5: Singular Value Decomposition SVD (1)

EEM3L1: Numerical and Analytical Techniques Lecture 5: Singular Value Decomposition SVD (1) EE3L1, slide 1, Version 4: 25-Sep-02 Motivation for SVD (1) SVD = Singular Value Decomposition Consider the system

### Solution to Homework 2

Solution to Homework 2 Olena Bormashenko September 23, 2011 Section 1.4: 1(a)(b)(i)(k), 4, 5, 14; Section 1.5: 1(a)(b)(c)(d)(e)(n), 2(a)(c), 13, 16, 17, 18, 27 Section 1.4 1. Compute the following, if

### Linear Algebra Notes

Linear Algebra Notes Chapter 19 KERNEL AND IMAGE OF A MATRIX Take an n m matrix a 11 a 12 a 1m a 21 a 22 a 2m a n1 a n2 a nm and think of it as a function A : R m R n The kernel of A is defined as Note

### Cofactor Expansion: Cramer s Rule

Cofactor Expansion: Cramer s Rule MATH 322, Linear Algebra I J. Robert Buchanan Department of Mathematics Spring 2015 Introduction Today we will focus on developing: an efficient method for calculating

### 2.5 Elementary Row Operations and the Determinant

2.5 Elementary Row Operations and the Determinant Recall: Let A be a 2 2 matrtix : A = a b. The determinant of A, denoted by det(a) c d or A, is the number ad bc. So for example if A = 2 4, det(a) = 2(5)

### Presentation 3: Eigenvalues and Eigenvectors of a Matrix

Colleen Kirksey, Beth Van Schoyck, Dennis Bowers MATH 280: Problem Solving November 18, 2011 Presentation 3: Eigenvalues and Eigenvectors of a Matrix Order of Presentation: 1. Definitions of Eigenvalues

### A Direct Numerical Method for Observability Analysis

IEEE TRANSACTIONS ON POWER SYSTEMS, VOL 15, NO 2, MAY 2000 625 A Direct Numerical Method for Observability Analysis Bei Gou and Ali Abur, Senior Member, IEEE Abstract This paper presents an algebraic method

### 1 Review of Least Squares Solutions to Overdetermined Systems

cs4: introduction to numerical analysis /9/0 Lecture 7: Rectangular Systems and Numerical Integration Instructor: Professor Amos Ron Scribes: Mark Cowlishaw, Nathanael Fillmore Review of Least Squares