1 Practical Numerical Training UKNum 7: Systems of linear equations C. Mordasini Max Planck Institute for Astronomy, Heidelberg Program: 1) Introduction 2) Gauss Elimination 3) Gauss with Pivoting 4) Determinants 5) LU Decomposition 6) Inverse of a matrix with LU
2 1 Introduction
3 Task One of the most important numerical tasks: Solve a system of linear equations. a 11 x 1 + a 12 x 2 + a 13 x a 1N x N = b 1 a 21 x 1 + a 22 x 2 + a 23 x a 2N x N = b 2 a 31 x 1 + a 32 x 2 + a 33 x a 3N x N = b 3 a M1 x 1 + a M2 x 2 + a M3 x a MN x N = b M N unknowns: x j, j =1, 2,...,N M equations, =12 with the known coefficients = a ij with i =1, 2,...,M and j =1, 2,...,N =12 And known right hand side b i, i =1, 2,...,M.
4 (Non-)Singular systems If N=M: same number of equations and unknowns. Chance to find a unique solution. But not always: One or several of the M equations is a linear combination of the other ones (row degeneracy). All equations contain certain variables only in exactly the same linear combination (column degeneracy). For square matrixes, one implies the other. Such matrixes are called singular. This is an analytical problem. Additional conditions arise from the numerical treatment of the task.
5 Numerical issues While not exact linear combinations of each other, some of the equations may be so close to linearly dependent that roundoff errors in the machine render them linearly dependent at some stage in the solution process. In this case your numerical procedure will fail, even if there would be an analytical solution. Accumulated roundoff errors in the solution process can swamp the true solution. This problem particularly emerges if N is too large. The numerical procedure does not fail algorithmically. However, it returns a set of x s that are wrong, as can be discovered by direct substitution back into the original equations. The closer a set of equations is to being singular, the more likely this is to happen, since increasingly close cancellations will occur during the solution. In fact, the preceding item can be viewed as the special case where the loss of significance is unfortunately total. Press et al.
6 Matrix form Our master equation is usually written in the form of a matrix equation: A x = b A is the matrix of coefficients, and b is the right-hand side written as a column vector A = a 11 a a 1N a 21 a a 2N a M1 a M2... a MN b = b 1 b 2 b M By convention, the first index on an element aij denotes its row, the second index its column.
7 Special matrixes In this context, several special types of matrixes are important: 1) Unit matrix A diagonal matrix with all diagonal elements equal to one is called an identity/unit matrix. 2)Upper triangular matrix All the elements below the diagonal entries are zero.
8 Special matrixes II 3)Lower triangular matrix All the elements above the diagonal entries are zero. 4)Tridiagonal matrix A tridiagonal matrix is a square matrix in which all elements not on the following are zero - the major diagonal, the diagonal above the major diagonal, and the diagonal below the major diagonal.
9 Other special matrixes In many situations (e.g. solutions of partial differential equations in two or more dimensions), one is dealing with so called sparse linear systems, i.e. matrixes where only a small fraction of all matrix elements are nonzero. For such matrixes, one should use specialized solvers, as they can be much more rapid than general purpose solvers. zeros zeros zeros (a) (b) (c) (d) (e) (f ) For large systems (N~ ), one wants to consider pre-written specialized libraries, like LAPACK which is freely available. (g) (h) (i) ( j) (k) Press et al.
10 Methods to solve In this lecture, we consider the following methods: Naive Gaussian Elimination Gaussian Elimination with Partial Pivoting LU Decomposition (Gauss Seidel Iteration) For large systems (N~ ), one wants to consider pre-written specialized libraries, like LAPACK which is freely available. They have also been parallelized and ported onto GPUs.
11 2 Gaussian Elimination
12 Naive Gaussian elimination Two step procedure: Forward elimination: In this step, an unknown is eliminated in each equation starting with the first equation. This way, the equations are reduced to one unknown in each equation. Back Substitution: In this step, starting from the last equation, each of the unknowns is found.
13 Forward elimination goal The goal of forward elimination is to transform the coefficient matrix into an upper triangular matrix:
14 Initial equations A set of n equations and n unknowns To bring this to the form of an upper triangular matrix, (n-1) steps of forward elimination are needed.
15 Forward elimination step 1 a In the first step of forward elimination, the first unknown, x1 is eliminated from all rows below the first row. The first equation is selected as the pivot equation to eliminate x1. So, to eliminate x1 in the second equation, one divides the first equation by a11 (hence called the pivot element) and then multiplies it by a21. This is the same as multiplying the first equation by a21/a11 to give
16 Forward elimination step 1 b Now, this equation can be subtracted from the second equation to give Note that we have eliminated in the new second equation x1.
17 Forward elimination step 1 c We can write this new second equation also as where a " = a a 21 a a a " = a a 21 2n 2n a 11 a 1n
18 Forward elimination step 1 d Repeat this procedure of eliminating x1 for the remaining equations (3 to n) to reduce the set of equations as This is the end of step 1.
19 Forward elimination step 2 For the second step of forward elimination, we start with the second equation as the pivot equation and a 22 as the pivot element. To eliminate x2 in the third equation, one thus divides the second equation by a 22 (the pivot element) and then multiply it by a 32. Then we subtract this from the third equation. This makes the coefficient of x2 zero in the third equation. The same procedure is now repeated for the fourth equation until the n th equation to give......
20 Forward elimination further steps The next steps of forward elimination are conducted by using the third equation as a pivot equation and so on. Thus, there will be a total of n-1 steps of forward elimination. At the end of n-1 steps of forward elimination, we get a set of equations that look like......
21 Back substitution Example of a system of 3 equations: # & % ( % ( $ % '( # % % $ % x 1 x 2 x 3 & ( ( '( = # & % ( % 96.21( $ % '( Solve each equation starting (obviously) from the last equation as it has only one unknown: We have now determined a first unknown quantity.
22 Back substitution II Then the second last equation, that is the (n-1)th equation, has two unknowns: xn and xn-1, but xn is already known from just before. This reduces the (n-1)th equation also to one unknown. We now iteratively work up to the first equation. Back substitution hence can be represented for all equations by the formula x i = b ( i 1 ) ( i 1 i a ) ( i 1 i,i+1 x i+1 a ) ( i 1 i,i+2 x i+2... a ) i,n x n ( i 1) for i = n 1,...,1 a ii x i = ( ) i 1 a ij b i i 1 n j =i+1 i 1 a ii ( ) x j ( ) for i = n 1,...,1
23 Example I The upward velocity of a rocket is given at three different times The velocity data is approximated by a polynomial as: Find the velocity at t=6 seconds.
24 Example II We need to determine the unknown coefficients ai. This results in a matrix of the form (Vandermonde matrix) Using data from the table, the matrix becomes:
25 Example III We can also write this as: 1) Forward elimination Number of steps of forward elimination is (n 1)=(3 1)=2
26 Example IV Forward elimination step 1 Divide Equation 1 by 25 and multiply it by 64,. [ ] 2.56 = [ ] Subtract the result from Equation 2 Substitute new equation for Eq. 2
27 Example V Forward elimination step 1 continued Divide Equation 1 by 25 and multiply it by 144,. [ ] 5.76 = [ ] Subtract the result from Equation 3 Substitute new equation for Equation 3
28 Example VI Forward elimination step 2 Divide Equation 2 by 4.8 and multiply it by 16.8,! [ ] 3.5 = [ ] Subtract the result from Equation 3 Substitute new equation for Equation 3
29 Example VII Back substitution # & % ( % ( $ % '( # % % $ % a 1 a 2 a 3 & ( ( '( = # & % ( % ( $ % 0.76 '( Solving for a 3 Solving for a 2
30 Example VIII Back substitution continued # & % ( % ( $ % '( # % % $ % a 1 a 2 a 3 & ( ( '( = # & % ( % ( $ % 0.76 '( Solving for a 1
31 Example IX Solution The solution vector is The polynomial that passes through the three data points is then:
32 Possible issues There are two pitfalls of the Naive Gauss elimination method. 1) Division by zero Here, we cannot even start the algorithm as we would have to devise by zero. This problem can also occur later on:
33 Possible issues II Here we can start the algorithm. However, at the first step of forward elimination, we get the following equations in matrix form Now at the beginning of the 2nd step of forward elimination, the coefficient of x2 in Equation 2 would be used as the pivot element. That element is zero and hence would create the division by zero problem. Division by zero is a possibility at any step of forward elimination.
34 Possible issues III 2) Large round-off errors The Naive Gauss elimination method is prone to round-off errors. This is true when there are large numbers of equations as errors propagate. Also, if there is subtraction of numbers from each other, it may create large errors. Example: Exact Solution
35 Possible issues III Solve it on a computer using 6 significant digits with chopping. One finds Solve it on a computer using 5 significant digits with chopping Is there a way to reduce the round off error? Obviously, increase the number of significant digits (always use double precision). This Decreases round-off error Does not avoid division by zero
36 3 Gaussian Elimination with Partial Pivoting
37 Partial pivoting Gaussian Elimination with Partial Pivoting Avoids division by zero Reduces round off error This method is an improvement of the algorithm we just saw before. What is Different About Partial Pivoting? At the beginning of the k th step of forward elimination, find the maximum of If the maximum of the values is in the p th row, then switch rows p and k. Otherwise, exactly the same algorithm as naive Gauss elimination except that we switch rows before each of the (n-1) steps of forward elimination.
38 Switching rows Matrix Form at Beginning of 2 nd Step of Forward Elimination Switched Rows
39 Example We solve the following set of equations by Gaussian elimination with partial pivoting Number of steps of forward elimination is again (n 1)=(3 1)=2.
40 Example II Forward Elimination: Step 1 Examine absolute values of first column, first row and below. Largest absolute value is 144 and exists in row 3. Switch row 1 and row 3. Begin forward elimination Divide Equation 1 by 144 and multiply it by 64,.
41 Example III Subtract the result from Equation 2 [ ] = [ ] Substitute new equation for Equation 2 Equation 3 Divide Equation 1 by 144 and multiply it by 25,. [ ] = [ ]
42 Example IV Subtract the result from Equation 3 Substitute new equation for Equation 3 Forward Elimination: Step 2 Examine absolute values of second column, second row and below Largest absolute value is and exists in row 3. Switch row 2 and row 3.
43 Example V Divide Equation 2 by and multiply it by 2.667, [ ] = [ ] Subtract the. result from Equation 3 Substitute new equation for Equation 3
44 Example VI Back Substitution Solve exactly as before for a 3 then for a 2 and finally for a 1
45 Round off errors Comparison with/without pivoting We studied earlier the system Exact Solution We found for Gaussian Elimination without pivoting, and 5 significant digits With pivoting, and 5 digits, one finds in contrast the exact solution. The fact that the round off error is fully removed here is by coincidence only, the general trend of a smaller error is however not a coincidence.
46 4 Determinants of a Square Matrix
47 Determinants Using naive Gauss Eliminations to find the determinant of a square matrix One of the more efficient ways to find the determinant of a square matrix is by taking advantage of the following two theorems on a determinant of matrices coupled with Naive Gauss elimination. Theorem 1 Theorem 2 If a multiple of one row of [A] nxn is added or subtracted to another row of [A] nxn to result in [B] nxn then det(a)=det(b). The same is true for column operations also. The determinant of an upper triangular, lower triangular or diagonal matrix [A] nxn is given by
48 Determinants II This implies that if we apply the forward elimination steps of the Naive Gauss elimination method, the determinant of the matrix stays the same according to Theorem 1. Then since at the end of the forward elimination steps, the resulting matrix is upper triangular, the determinant will be given by Theorem 2.
49 Example Find the determinant of (rocket example) For a 3 x 3 Matrix we of course know directly that But we can also use (in particular for higher dimensions) that with Naive Gaussian Elimination, we get the upper triangular matrix. In this example, we found earlier
50 Example II Therefore, according to theorem 1 and 2, the determinant is = 25 ( 4.8) 0.7
51 Determinants and pivoting What if one cannot find the determinant of the matrix using the Naive Gauss elimination method, for example, because one gets division by zero problems during the Naïve Gauss elimination method? In this case, one can apply Gaussian elimination with partial pivoting. However, the determinant of the resulting upper triangular matrix may differ by a sign. The following theorem applies in addition to the previous two to find the determinant of a square matrix. Theorem 3 Let [A] nxn be a n x n matrix. Then, if [B] nxn is a matrix that results from switching one row with another row, then det(a)=-det(b).
52 5 LU Decomposition
53 LU Decomposition LU Decomposition is another method to solve a set of simultaneous linear equations Which is better, Gauss Elimination or LU Decomposition? To answer this, a closer look at LU decomposition is needed.
54 LU Decomposition Method For most non-singular matrix [A] that one could conduct Naive Gauss Elimination forward elimination steps, one can write it as where [L] = lower triangular matrix [U] = upper triangular matrix [A] = [L][U]
55 LU Decomposition idea Note: [Z] is a n x 1 matrix, i.e. a vector
56 LU Decomposition algorithm How can this be used? Given [A][X] = [C] 1. Decompose [A] into [L] and [U] 2. Solve [L][Z] = [C] for [Z] 3. Solve [U][X] = [Z] for [X] As we will see below, both solving equation 2 and 3 is very simple. Eq. 2 is solved using forward substitution and then we use Eq. 3 to calculate the solution vector [X] by back substitution.
57 Computational time To solve [A][X] = [B] Gaussian Elimination LU Decomposition T=clock cycle time n = size of the matrix So both methods are equally efficient. However, for calculating the inverse of a matrix, LU can be much faster (cf below)
58 Method: Decompose [A] to [L] and [U] [ A] = [ L] [ U] = " 1 0 0% " u u u $ ' $ $ ' $ 0 u u # $ &' # $ 0 0 u 33 % ' ' &' [U] is the same as the coefficient matrix at the end of the forward elimination step during Naive Gauss elimination. [L] is obtained using the multipliers that were used in the forward elimination process
59 Example: Find the [U] matrix I Using the Forward Elimination Procedure like for Gauss Elimination Step 1:
60 Example: Find the [U] matrix II Matrix after Step 1: Step 2:
61 Example: Find the [L] matrix I Using the multipliers used during the Forward Elimination Procedure From the first step of forward elimination
62 Example: Find the [L] matrix II From the second step of forward elimination The [L] matrix is thus obtained for free if we do Naive Gaussian Elimination.
63 Does [L][U] = [A]? [ L] [ U] = =? This can always be checked by simple matrix multiplication.
64 LU Decomposition to solve SLEs I Solve the following set of linear equations using LU Decomposition Step 1 Use the algorithm we just saw for finding the [L] and [U] matrices [ A] = [ L] [ U] =
65 LU Decomposition to solve SLEs II Step 2 Set [L][Z] = [C] and then solve for [Z]. Due to the particular shape of [L], this is very simple by forward substitution starting from the first equation: z 1 = z 1 + z 2 = z z 2 + z 3 = 279.2
66 LU Decomposition to solve SLEs III Step 2 continued Complete the forward substitution to solve for [Z]
67 LU Decomposition to solve SLEs IV Step 3 Set [U][X] = [Z] Solve for [X] The 3 equations become Again, due to the particular shape of [U], this is very simple by backward substitution starting from the last equation.
68 LU Decomposition to solve SLEs V Step 3 continued From the 3 rd equation Substituting in a 3 and using the second equation Substituting in a 3 and a 2 in the first equation Hence the Solution Vector [X] is:
69 6 Inverse of a Matrix with LU Decomposition
70 Finding the inverse of a square matrix The inverse [B] of a square matrix [A] is defined as [A][B] = [I] = [B][A] How can LU Decomposition be used to find the inverse? Assume the first column of [B] to be [b 11 b 12 b n1 ] T. Use this and the definition of matrix multiplication First column of [B] Second column of [B] The remaining columns in [B] can be found in the same manner. Thus, we now determine [B] in one column after another, where such a column is taking the role of the [X] vector in the last examples.
71 Example: Inverse of a Matrix I Find the inverse of a square matrix [A] Step 1: Using the decomposition procedure, the [L] and [U] matrices are found to be [ A] = [ L] [ U] = These two matrixes will now be used for all columns of [B].
72 Example: Inverse of a Matrix II As before solving for the each column of [B] requires two furhter steps 2) Solve [L] [Z] = [C] for [Z] 3) Solve [U] [X] = [Z] for [X] Step 2: [ L] [ Z] = [ C] This generates the equations: z 1 z 2 z 3 = 1 0 0
73 Example: Inverse of a Matrix III Step 2: Solving [U][X] = [Z] for [X] This generates the equations: So the first column of the inverse of [A] is:
74 Example: Inverse of a Matrix IV Repeating for the second and third columns of the inverse Second Column Third Column Final result: The inverse of [A] is To check: [A][A] -1 = [I] = [A] -1 [A]
75 Inverse of a Matrix with LU For calculations of columns of the inverse of the matrix, the LU decomposition needs to be done only once, then the forward substitution n times, and the back substitution n times. In comparison, if Gaussian elimination method were used to find the inverse of a matrix, the forward elimination as well as the back substitution will have to be done n times. Time taken by Gaussian Elimination Time taken by LU Decomposition For large n, the term n 4 will become dominant. Comparing computational times of finding inverse of a matrix using LU decomposition and Gaussian elimination. n CT inverse GE / CT inverse LU
76 References This script is based on by Autar Kaw, Jai Paul and Numerical Recipes (2nd/3rd Edition) by Press et al., Cambridge University Press