LECTURE 5:Systems of linear equations

Transcription

1 1 LECTURE 5:Systems of linear equations 1 Naive Gaussian elimination Consider the following system of n linear equations with n unknowns: a 11 x 1 + a 12 x 2 + a 13 x a 1n x n = b 1 a 21 x 1 + a 22 x 2 + a 23 x a 2n x n = b 2 a 31 x 1 + a 32 x 2 + a 33 x a 3n x n = b a i1 x 1 + a i2 x 2 + a i3 x a in x n = b i..... a n1 x 1 + a n2 x 2 + a n3 x a nn x n = b n In compact form, this system can be written as n a i j x j = b i (1 i n) j=1 In these equations, a i j and b i are prescribed real numbers (data), and the unknowns x j are to be determined. The objective in this chapter is to find an efficient way for obtaining the numerical solution of such a set of linear equations. We begin by discussing the simplest form of a procedure known as Gaussian elimination. This form is not suitable for automatic computation, but serves as a starting point for developing a more robust algorithm.

2 2 Example We demonstrate the simplest form of the Gaussian elimination procedure using a numerical example. In this form, the procedure is not suitable for numerical computation, but it can be modified as we see later on. Consider the following four equations with four unknowns: 6x 1 2x 2 + 2x 3 + 4x 4 = 16 12x 1 8x 2 + 6x x 4 = 26 3x 1 13x 2 + 9x 3 + 3x 4 = 19 6x 1 + 4x 2 + x 3 18x 4 = 34 In the first round of the procedure, we eliminate x 1 from the last three equations. This is done by subtracting multiples of the first equation from the other ones. The result is 6x 1 2x 2 + 2x 3 + 4x 4 = 16 4x 2 + 2x 3 + 2x 4 = 6 12x 2 + 8x 3 + x 4 = 27 2x 2 + 3x 3 14x 4 = 18 Note that the first equation was not changed in the elimination procedure. It is called the pivot equation. The second step consists of eliminating x 2 from the last two equations. We ignore the first equation and use the second one as the pivot equation. We get 6x 1 2x 2 + 2x 3 + 4x 4 = 16 4x 2 + 2x 3 + 2x 4 = 6 + 2x 3 5x 4 = 9 4x 3 13x 4 = 21 Finally we eliminate x 3 from the last equation and obtain: 6x 1 2x 2 + 2x 3 + 4x 4 = 16 4x 2 + 2x 3 + 2x 4 = 6 + 2x 3 + 5x 4 = 9 3x 4 = 3 This system is said to be in the upper triangular form. It is equivalent to the system we started with. This completes the first phase of the Gaussian elimination, called forward elimination. The second phase is called back substitution. We solve the system in the upper triangular form, starting from the bottom. From the last equation we obtain x 4 = 3 3 = 1 Substituting this to the second last equation gives Eventually we obtain the solution 2x 3 5 = 9 x 1 = 3 x 2 = 1 x 3 = 2 x 4 = 1

3 3 Algorithm We now write the system of n linear equations in matrix form: a 11 a 12 a a 1n a 21 a 22 a a 2n a 31 a 32 a a 3n.... a i1 a i2 a i3... a in.... a n1 a n2 a n3... a nn x 1 x 2 x 3. x i. x n = b 1 b 2 b 3. b i. b n or Ax = b In the naive Gaussian elimination algorithm, the original data are overwritten with new computed values. In the forward elimination phase, there are n 1 principal steps. The first step consists of subtracting appropriate multiples of the first equation from the others. The first equation is called the first pivot equation and a 11 is called the first pivot element. For each of the other equations (2 i n), we compute ( a i j a i j ai1 a 11 )a 1 j 1 j n ( b i b i ai1 a 11 )b 1 The quantities a i1 /a 11 are the multipliers. The new coefficient for x 1 in all of the equations 2 i n will be zero because a i1 (a i1 /a 11 )a 11 = 0. After the first step, the system is in the form a 11 a 12 a a 1n 0 a 22 a a 2n 0 a 32 a a 3n a i2 a i3... a in a n2 a n3... a nn x 1 x 2 x 3. x i. x n = From now on, we ignore the first row (and effectively also the first column). We take the second row as the pivot equation and compute for the remaining equations (3 i n) ( a i j a i j ai2 a 22 )a 2 j 2 j n ( b i b i ai2 a 22 )b 2 This step eliminates x 2 from the equations 3 i n. b 1 b 2 b 3. b i. b n

4 4 In the kth step, we use the kth row as the pivot equation and for the remaining equations below the kth row (k + 1 i n), we compute ( a i j a i j aik a )a kk k j k j n ( b i b i aik a )b kk k Note that we must assume that all the divisors in this algorithm are nonzero!! In the beginning of the back substitution phase, the system is in the upper triangular form. Back substitution starts by solving the nth equation for x n : x n = b n a nn Then using the (n 1)th equation, x n 1 is solved: x n 1 = 1 a n 1,n 1 (b n 1 a n 1,n x n ) The formula for the ith equation is x i = 1 a ii (b i ) n a i j x j j=i+1 (i = n 1,n 2,...,1)

5 5 Code The following C program performs the Naive Gaussian elimination: #define N 4 int main(void) { int i, j, k, n; double sum, xmult; double a[n][n], b[n], x[n]; FILE *outfile; /* Read input */ GetEquations(a,b); /* Forward elimination */ for(k=0; k<n-1; k++) { for(i=k+1; i<n; i++) { xmult = a[i][k]/a[k][k]; a[i][k] = xmult; for(j=k+1; j<n; j++) { a[i][j] = a[i][j] - xmult*a[k][j]; b[i] = b[i] - xmult*b[k]; /* Back substitution */ x[n-1] = b[n-1]/a[n-1][n-1]; for(i=n-2; i>=0; i--) { sum = b[i]; for(j=i+1; j<n; j++) { sum = sum - a[i][j]*x[j]; x[i] = sum/a[i][i];

6 6 If we apply the code to the example case discussed before, we get the following output (only recalculated values): aij bi x[1] = x[2] = x[3] = x[4] = The result is the same as was previously obtained by hand. This simple algorithm does not work in many cases. The crucial computation is the replacement ( ) aik a i j a i j a k j a kk A round-off error in a k j is multiplied by the factor (a ik /a kk ). This factor is large if the pivot element a kk is small compared to a ik. The conclusion is that small pivot elements lead to worse round-off errors.

7 7 1.1 Condition number The condition number of a linear system Ax = b is defined as κ(a) = A A 1 Here, A is the matrix norm. There are several norms, e.g. l 1 -norm for an n n-matrix A is A 1 = max 1 j n n i=1 a i j. κ(a) gauges the transfer of error from the matrix A and the vector b to the solution x. The rule of thumb is that if κ(a) = 10 k, then one can expect to lose at least k digits of precision in solving the system Ax = b. A matrix A is called ill-conditioned if the linear system Ax = b is sensitive to perturbations in the elements of A, or to perturbations of the components of b. The larger the condition number is, the more ill-conditioned is the system. In computational studies, it is extremely important to estimate whether a numerical result can be trusted or not. The condition number provides some evidence regarding this question if one is solving linear systems. In mathematical software systems such as Matlab, an estimate of the condition number is often available so that one can estimate the reliability of the numerical result. Such a criterion may be used for example to choose a suitable solution method. 1.2 Residual vectors Without having the aid of sophisticated mathematical software, how can we determine whether a numerical solution to a linear system is correct? One check is to substitute the obtained numerical solution ˆx back into the original system. This means computing the residual vector ˆr = Aˆx b The situation is, however, more complicated than one would tend to imagine. A solution with a small residual vector is not necessarily a good solution. This can be the case if the original problem is sensitive to roundoff errors - ill-conditioned. The problem of whether a computed solution is good or not is difficult and mostly beyond the scope of this course.

8 8 2 Gaussian elimination with scaled partial pivoting 2.1 Failure of Naive Gaussian elimination Consider the following simple example { 0x1 + x 2 = 1 x 1 + x 2 = 2 The Naive Gaussian elimination program would fail here because the pivot element a 11 = 0. The program is also unreliable for small values of a 11. Consider the system { εx1 + x 2 = 1 x 1 + x 2 = 2 where ε is a small number other than zero. The forward elimination produces the following system { εx1 + x 2 = 1 (1 1/ε)x 2 = 2 1/ε In the back substitution, we obtain x 2 = 2 1/ε 1 1/ε x 1 = 1 x 2 ε If ε is small, then 1/ε is large, and in a computer with finite word length, both the values (2 1/ε) and (1 1/ε) would be computed as 1/ε. Thus the computer calculates that x 2 = 1 and x 1 = 0. But the correct answer is x 2 1 and x 1 1. The relative error in the computed solution for x 1 is 100%!! If we change the order of the equations, the Naive Gaussian algorithm works: { x1 + x 2 = 2 εx 1 + x 2 = 1 This becomes { x1 + x 2 = 2 (1 ε)x 2 = 1 2ε And the back substitution gives x 2 = 1 2ε 1 ε 1 x 1 = 2 x 2 1 This idea of changing the order of the equations is used in a procedure called scaled partial pivoting.

9 9 2.2 Scaled partial pivoting We have seen that the order in which we treat the equations significantly affects the accuracy of the algorithm. We now describe an algorithm where the order of the equations is determined by the system itself. We call the original order in which the equations are given the natural order of the system: {1,2,3,...,n. The new order in which the elimination is performed is denoted by the row vector l = [l 1,l 2,...,l n ] This is called the index vector. The strategy of interchanging the rows is called scaled partial pivoting. We begin by calculating a scale factor for each equation: s i = max 1 j n a i j (1 i n) These n numbers are recorded in a scale vector: s = [s 1,s 2,...,s n ]. The scale vector is not updated during the procedure because according to a common view, this extra computation is not worth the effort. Next we choose the pivot equation based on the scale factors. The equation for which the ratio a i1 /s i is the greatest is chosen to be the first pivot equation. Initially we set for the index vector l = [l 1,l 2,...,l n ] = [1,2,...,n]. If j is selected to be the index of the first pivot equation, we interchange l j with l 1 in the index vector. Thus we get l = [l j,l 2,...,l 1,...,l n ]. Next we use the multipliers a li 1 a l1 1 times the row l 1 and subtract these from the equations l i for 2 i n. (Note that now l 1 l j.) This is the first step of the forward elimination. Note that only the index vector is updated, not the equations! In the next step, we go through the ratios { ali 2 s li : 2 i n If j is now the index corresponding to the largest ratio, then interchange l j and l 2 in the index vector. Continue by subtracting the multipliers a li 1 a l2 1 times equation l 2 from the remaining equations l i for which 3 i n. The same procedure is repeated until n 1 steps have been completed.

10 10 Numerical example Consider the system: The index vector is initially l = [1,2,3,4]. The scale vector is s = [13, 18, 6, 12]. x 1 x 2 x 3 x 4 = Step 1. In order to determine the first pivot row, we look at the ratios: { { ali 1 3 : i = 1,2,3,4 = s li 13, 6 18, 6 6, 12 {0.23,0.33,1.0, The largest value of these ratios occurs first for the index j = 3. So the third row is going to be our pivot equation in step 1 of the elimination process. The entries l 1 and l j are interchanged in the index vector. Thus we have l = [3,2,1,4]. Now appropriate multiples of the pivot equation (l 1 = 3) are subtracted from the other equations so that zeros are created as coefficients of x 1. The result is x x x 3 = x 4 6 Step 2. In the next step (k = 2), we use the index vector l = [3,2,1,4] and scan the rows l 2 = 2, l 3 = 1 and l 4 = 4 for the largest ratio: { { ali 2 2 : i = 2,3,4 = s li 18, 12 13, 4 {0.11, 0.92, The largest is the second value (with index l 3 = 1). Thus the first row is our new pivot equation and the index vector becomes l = [3,1,2,4] after interchanging l 3 and l 2. Next, multiples of the first equation are subtracted from the two remaining equations l 3 = 2 and l 4 = 4. The result is x /3 83/6 x x 3 = 45/ /3 5/3 x 4 3

11 11 Step 3. In the third and final step, we first examine the ratios { { ali 3 13/3 : i = 3,4 = s li 18, 2/3 {0.24, The largest value is for l 3 = 2, and thus the 2nd row is used as the pivot equation. The index vector is unchanged because l k = l j (index of step is the same as the index of pivot equation). We have l = [3,1,2,4]. The remaining task is to subtract the appropriate multiple of equation 2 from equation 4. This gives /3 83/ /13 x 1 x 2 x 3 x 4 = 27 45/2 16 6/13 Back substitution The final index vector gives the order in which the back substitution is performed. The solution is obtained by first using l 4 = 4 to determine x 4, then l 3 = 2 to determine x 3, then l 2 = 1 to determine x 2 and finally l 1 = 3 to determine x 1. After the back substitution, we obtain the result 3 x = 1 2 1

12 12 Code In the following C implementation, the algorithm is programmed in such a way that it first carries out the forward elimination phase on the coefficient array (a i j ) only (procedure Gauss). The right-hand-side array (b i ) is treated in the next phase (procedure Solve). Since we are treating the array (b i ) later, we need to store the multipliers. These are easily stored in the array (a i j ) in positions where 0 entries would have been created otherwise. Here is the procedure Gauss for forward elimination with scaled partial pivoting. void Gauss(int n, double a[n][n], int l[n]) { int i, j, k, temp; double aa, r, rmax, smax, xmult; double *s; /* Allocate memory */ s = (double *) malloc((size_t) (n*sizeof(double))); /* Create the scaling vector */ for(i=0; i<n; i++) { l[i] = i; smax = 0; for(j=0; j<n; j++) { aa = fabs(a[i][j]); if(aa > smax) smax = aa; s[i] = smax; /* Loop over steps */ for(k=0; k<n-1; k++) { /* Choose pivot equation */ rmax = 0; j=k; for(i=k; i<n; i++) { r = fabs(a[l[i]][k]/s[l[i]]); if(r > rmax) { rmax = r; j = i; /* Interchange indeces */ temp = l[j]; l[j] = l[k]; l[k] = temp; /* Elimination for step k */ for(i=k+1; i<n; i++) { xmult = a[l[i]][k]/a[l[k]][k]; a[l[i]][k] = xmult; for(j=k+1; j<n; j++) { a[l[i]][j] = a[l[i]][j] - xmult*a[l[k]][j]; free(s);

13 13 And here is the procedure Solve for determining the array (b i ) and for the back substitution phase: void Solve(int n, double a[n][n], int l[n], double b[n], double x[n]) { int i, j, k; double sum; /* Determine array bi */ /* Elements a[l[i]][k] are the multipliers */ for(k=0; k<n-1; k++) { for(i=k+1; i<n; i++) { b[l[i]] = b[l[i]]-a[l[i]][k]*b[l[k]]; /* Back substitution phase */ x[n-1]= b[l[n-1]]/a[l[n-1]][n-1]; for(i=n-2; i>=0; i--) { sum = b[l[i]]; for(j=i+1; j<n; j++) { sum = sum - a[l[i]][j]*x[j]; x[i] = sum/a[l[i]][i]; If we use the program on the same problem as solved before by hand and using the naive Gaussian elimination program, we get the following results (note that in C the indexing goes from 0 to n 1, here it has been changed to run from 1 to n: Scale vector: s[1] = s[2] = s[3] = s[4] = Final index vector: l[1] = 1 l[2] = 3 l[3] = 4 l[4] = 2 Arrays (aij) and (bi) (multipliers in place of 0 s): ai1 ai2 ai3 ai4 bi Solution: x[1] = x[2] = x[3] = x[4] = We obtain the same solution as before but the arrays (ai j) and (bi) are different (due to different sequence of pivot equations).

14 Long operation count Solving large systems of linear equations on a computer can be expensive. We will now count the number of long operations involved in the Gaussian elimination with partial pivoting. By long operations we mean multiplications and divisions. We make no distinction between the two, although division is more time consuming than multiplication. i. Procedure Gauss Step 1: - Choice of pivot element: n divisions - Multipliers (one for each row): n 1 divisions - New elements: n 1 multiplications / row (n 1) (n 1) - Total: n + (n 1) + (n 1)(n 1) = n + n(n 1) = n 2 long operations Step 2: - Involves n 1 rows and n 1 columns - Total: (n 1) 2 long operations Whole procedure: -Total: ii. Procedure Solve n 2 + (n 1) 2 + (n 2) = n n3 (n + 1)(2n + 1) Forward processing of (b i ): - Involves n 1 steps - First step: n 1 multiplications - Second step: n 2 multiplications, etc. - Total: (n 1) + (n 2) = 2 n (n 1) Back substitution: - One operation in first step, two in the second and so on. - Total: n = n 2 (n + 1) Whole procedure: -Total: n 2 (n 1) + n 2 (n + 1) = n (n 1)(2n) = n2 2 Theorem on long operations. The forward elimination phase of the Gaussian elimination algorithm with scaled partial pivoting, if applied only to the n n coefficient array, involves approximately n 3 /3 long operations (multiplications or divisions). Solving for x requires an additional n 2 long operations.

15 15 3 LU factorization An n n system of linear equations can be written in matrix form Ax = b where the coefficient matrix A has the form a 11 a 12 a a 1n a 21 a 22 a a 2n A = a 31 a 32 a a 3n a n1 a n2 a n3... a nn We now want to show that the naive Gaussian algorithm applied to A yields a factorization of A into two products: A = LU where L has a lower triangular form: 1 l 21 1 L = l 31 l l n1 l n2 l n and U has a upper triangular form: u 11 u 12 u u 1n u 22 u u 2n U = u u 3n.... u nn This is called the LU factorization.

16 Benefit of LU factorization Using the LU factorization of A we can write Ax = (LU)x = L(Ux) = b This means that we can solve the system of linear equations in two parts: L z = b U x = z Solving these sets of equations is trivial since the matrices are in triangular form. We first solve for z using the first equation. Since L is lower triangular, we can use forward substitution: z 1 = b 1 l 11 z i = 1 i 1 [b i l ii j=1 l i j z j ] (i = 2,3,...,n) Once we have z, the second equation can be solved using back substitution: x n = z n u nn x i = 1 u ii [z i ] N u i j x j j=i+1 (i = n 1,n 2,...,1) The benefit of this method is that once we have the LU decomposition of A, we can use the same matrices U and L to solve for several different vectors b. The CPU time requirement for the above forward and back substitutions is O(n 2 ), whereas the Gauss elimination algorithm is O(n 3 ).

17 17 Numerical example The first example (in Naive Gaussian elimination) can be written in matrix form as: x x x 3 = x 4 34 We obtained the following upper triangular form for the system as a result of the forward elimination phase: x x x 3 = x 4 3 The forward elimination phase can be interpreted as starting from the first equation, Ax = b, and proceeding to MAx = Mb where M is a matrix chosen so that MA is the coefficient matrix in upper triangular form: MA = U The forward elimination phase of the naive Gaussian elimination algorithm consists of a series of steps. The first step gives us the following system: x x x 3 = x 4 18 The first step can be accomplished by multiplying by a lower triangular matrix M 1 : where M 1 = M 1 Ax = M 1 b Notice the special form of this matrix: the diagonal elements are all 1 s and the only other nonzero elements are in the first column. These are the negatives of the multipliers.

18 18 The second step results in the system: x 1 x 2 x 3 x 4 = which is equivalent to where M 2 M 1 Ax = M 2 M 1 b M 2 = Finally, step 3 gives the system in the upper triangular form x x x 3 = x which is equivalent to where M 3 M 2 M 1 Ax = M 3 M 2 M 1 b M 3 = Thus we obtain the matrix M as a product of the three multiplier matrices: M = M 3 M 2 M 1

19 19 We can now derive a different interpretation of the forward elimination phase. We see that A = M 1 U = M 1 1 M 1 2 M 1 3 U = LU The inverse of the M k s is obtained easily by changing the signs of the multipliers. Thus we have L = = This is a unit lower triangular matrix composed of the multipliers! We can easily verify that LU = = = A In LU factorization, the matrix A is factored or decomposed into a unit lower triangular matrix L and an upper triangular matrix U. The lower triangular elements of L are the multipliers located in the positions of the elements they annihilated from A. U is the final coefficient matrix obtained after the forward elimination phase. Note that the naive Gaussian elimination algorithm replaces the original coefficient matrix A with its LU factorization.

20 LU factorization theorem Theorem. Let A = (a i j ) be an n n matrix. Assume that the forward elimination phase of the naive Gaussian algorithm is applied to A without encountering any 0 divisors. Let the resulting matrix be denoted by à = (ã i j). If L = ã ã 31 ã ã n1 ã n2... ã n,n 1 1 and U = ã 11 ã 12 ã ã 1n 0 ã 22 ã ã 2n 0 0 ã ã 3n ã n,n then A = LU. Notice that we rely here on not encountering any zero divisors in the algorithm.

21 Solving linear systems using LU factorization Once the LU factorization of A is available, we can solve the system A x = b by writing L U x = b Then we solve two triangular systems: L z = b for z and for x. U x = z This is particularly useful if we have a problem that involves the same coefficient matrix A and several different right-hand side vectors b. z is obtained by the pseudocode (forward substitution): real array (b i ) n, (l i j ) n n, (z i ) n integer i, n z 1 b 1 for i = 2 to n do z i b i i 1 j=1 l i jz j end for And similarly x is obtained by the pseudocode (back substitution): real array (x i ) n, (u i j ) n n, (z i ) n integer i, n x n z n /u nn for i = n ( 1 to 1 step-1 ) do x i u 1 ii z i n j=i+1 u i jx j end for

22 Software packages Matlab and Maple produce factorizations of the form P A = L U where P is a permutation matrix corresponding to the pivoting strategy used. Example of such a matrix is P = A permutation matrix is an n n matrix which is obtained from the unit matrix by permuting its rows. Permuting the rows of any n n matrix A can be accomplished by multiplying A on the left by P. When Gaussian elimination with row pivoting is performed on A, the result is expressible as P A = L U. The matrix P A is A with rows rearranged. We can use the LU factorization of P A to solve the system A x = b in the following way: P A x = P b L U x = P b L y = P b This is easily solved for y. The solution is obtained by solving for x. U x = y