Solving Sets of Equations. 150 B.C.E., 九章算術 Carl Friedrich Gauss,

Transcription

1 Solving Sets of Equations 5 B.C.E., 九章算術 Carl Friedrich Gauss,

2 Gaussian-Jordan Elimination In Gauss-Jordan elimination, matrix is reduced to diagonal rather than triangular form Row combinations are used to eliminate entries above as well as below diagonal Numerical Methods Wen-Chieh Lin

3 Numerical Methods Wen-Chieh Lin Gaussian-Jordan Elimination (cont.) Elimination matrix used for given column vector a is of form n i a a m a a a a a a m m m m k i i k n k k k n k k,,, where M k a

4 Gaussian-Jordan Elimination (cont.) Gauss-Jordan elimination requires about n / multiplications and similar number of additions, 5% more expensive than LU factorization During elimination phase, same row operations are also applied to right-hand-side vector (vectors) of system of linear equations Once matrix is in diagonal form, components of solution are computed by dividing each entry of transformed right-hand side by corresponding diagonal entry of matrix Numerical Methods Wen-Chieh Lin 4

5 Using LU for multiple right-hand sides If LU factorization of a matrix A is given, we can solve Ax = b for different b vectors as follows: Ax = b LUx = b Solve Ly = b using forward substitution Then solve Ux = y using backward substitution Numerical Methods Wen-Chieh Lin 5

6 Row Interchanges Gaussian elimination breaks down if leading diagonal entry of remaining unreduced matrix is zero at any stage Easy fix: if diagonal entry in column k is zero, then interchange row k with some subsequent row having nonzero entry in column k and then proceed as usual If there is no nonzero on or below diagonal in column k, then there is nothing to do at this stage, so skip to next column Numerical Methods Wen-Chieh Lin 6

7 Row Interchanges (cont.) Zero on diagonal causes resulting upper triangular matrix to be singular, but LU factorization can still be completed Subsequent back-substitution will fail, however, as it should for singular matrix Numerical Methods Wen-Chieh Lin 7

8 Partial Pivoting In principle, any nonzero value will do as pivot, but in practice pivot should be chosen to minimize error propagation To avoid amplifying previous rounding errors when multiplying remaining portion of matrix by elementary elimination matrix, multipliers should not exceed in magnitude This can be accomplished by choosing entry of largest magnitude on or below diagonal as pivot at each stage Numerical Methods Wen-Chieh Lin 8

9 Partial Pivoting (cont.) Partial pivoting is necessary in practice for numerically stable implementation of Gaussian elimination for general linear system Numerical Methods Wen-Chieh Lin 9

10 LU Factorization with Partial Pivoting With partial pivoting, each M k is preceded by permutation P k to interchange rows to bring entry of largest magnitude into diagonal pivot position Still obtain MA = U, with U upper triangular, but now M = M n- P n- M P L=M - is not a triangular due to permutations L M ( M P T T nn MP MP ) P LP L P Numerical Methods Wen-Chieh Lin T n L n

11 Numerical Methods Wen-Chieh Lin Example: Pivoting Ax b x x x ] [ b A P M ] [ b A M P

12 Numerical Methods Wen-Chieh Lin Example: Pivoting (cont.) ] [ b A P M P ] [ b A M P M P

13 Numerical Methods Wen-Chieh Lin Example: Pivoting (cont.) ) ( L L P P M P M P M L T T U is still upper triangular but L is not lower triangular due to permutations

14 Complete Pivoting Complete pivoting is more exhaustive strategy in which largest entry in entire remaining unreduced submatrix is permuted into diagonal pivot position Requires interchanging columns as well as rows leading to factorization PAQ = LU with L unit lower triangular, U upper triangular, and P, Q permutations Numerical Methods Wen-Chieh Lin 4

15 Complete Pivoting (cont.) Numerical stability of complete pivoting is theoretically superior, but pivot search is more expensive than for partial pivoting Numerical stability of partial pivoting is more than adequate in practice, so it is almost always used in solving linear system by Gaussian elimination Numerical Methods Wen-Chieh Lin 5

16 Example: Pivoting and Precision Consider Without pivoting x ( E y x By Dx Ey C BD F E ) y C F x By CD BD B ( C C F CD CD BD B) C C C B Numerical Methods Wen-Chieh Lin 6

17 Numerical Methods Wen-Chieh Lin 7 Example: Pivoting and Precision With pivoting C By x F Ey Dx D F C y D E B F Ey Dx ) ( B C D E B D F C y BD CE BF D B C E F x ) (

18 Scaling Linear Systems In principle, solution to linear system is unaffected by diagonal scaling of matrix and right-hand-side vector Example: row scaling premultiplying both sides of system by nonsingular diagonal matrix D, the solution is unchanged DAx Db x ( DA) Db A b Numerical Methods Wen-Chieh Lin 8

19 Scaling Linear Systems (cont.) In practice, scaling affects both conditioning of matrix and selection of pivots in Gaussian elimination, which in turn affect numerical accuracy in finite-precision arithmetic It is usually best if all entries of matrix have about same size Numerical Methods Wen-Chieh Lin 9

20 Scaling Linear Systems (cont.) Sometimes it may be obvious how to accomplish this by choice of measurement units for variables, but there is no foolproof method for doing so in general Scaling can introduce error if not done carefully! Numerical Methods Wen-Chieh Lin

21 Given Example: Scale Partial Pivoting A, the exact solution is x = [,, ] T If only digits of precision is used we obtain a erroneous solution x = [.99,.9,.] T.67 5 b Numerical Methods Wen-Chieh Lin

22 Example: Scale Partial Pivoting (cont.) Premultiplying by a scaling matrix A S[ A b], b / S.5.5. / / Pivoting is required at the first column! Numerical Methods Wen-Chieh Lin

23 Example: Scale Partial Pivoting (cont.) In algorithm implementation, we don t scale equations explicitly Instead, we store the scale vector and row interchange information and only use them for pivot selection s partial pivoting no pivoting is required Numerical Methods Wen-Chieh Lin

24 Complexity of Solving Linear System LU factorization requires about n / floatingpoint multiplications and similar number of additions Forward and backward substitution for single right-hand side vector together require about n multiplications and similar number of additions Numerical Methods Wen-Chieh Lin 4

25 Complexity of Solving Linear System Can also solve linear system by matrix inversion: x = A - b Computing A - is equivalent to solve n linear systems, requiring LU factorization of A followed by n forward and backward substitutions, one for each column of identity matrix Operation count for inversion is about n, three times expensive as LU factorization Numerical Methods Wen-Chieh Lin 5

26 Inversion vs. Factorization x=a - b Needs to solve Ax = I LU factorization n forward and backward substitutions Multiplication of matrix and vector LUx = b LU factorization One forward and backward substitution Numerical Methods Wen-Chieh Lin 6

27 Inversion vs. Factorization (cont.) Inversion gives less accuracy answer; e.g., solving x = 8 by division gives x = 8/ = 6, but inversion gives x = - 8 =. 8 = 5.99 (using -digit arithmetic) Numerical Methods Wen-Chieh Lin 7

28 Inversion vs. Factorization (cont.) Matrix inverses often occurs as convenient notation in formulas, but explicit inverse is rarely required to implement such formulas For example, product A - B should be computed by LU factorization of A, followed by forward and backward substitution using each column of B Use factorization instead of inversion Numerical Methods Wen-Chieh Lin 8

29 Ill-Conditioned Systems Recall that A system is ill-conditioned if the solution is very sensitive to changes in the input Example: a near-singular coefficient matrix x.. y. x. y.. b. 98 x. y. 98 b. x y. We cannot test the accuracy of the computed solution merely by substituting the solution into equation to see whether the right-hand sides are reproduced Numerical Methods Wen-Chieh Lin 9

30 Condition Numbers and Norms The condition number of a matrix is defined in terms of norms We ll define the condition number of a matrix after introducing vector and matrix norms Numerical Methods Wen-Chieh Lin

31 Vector Norms Magnitude, modulus, or absolute value for scalars generalizes to norm for vectors We will use only p-norm, defined by x p i i for integer p > and n-vector x Important special cases x x n p n n x x x x max x i i i i i i -norm -norm p -norm Numerical Methods Wen-Chieh Lin

32 Properties of Vector Norms For any vector norm x and x if and only if x kx k x for any scalar k x y x y (triangular inequality) The definition of a vector norm needs to satisfies the above properties Numerical Methods Wen-Chieh Lin

33 Matrix Norms Matrix norm corresponding to given vector norm is defined by A max x Ax x Norm of a matrix measures maximum stretching that the matrix does to any vector in given vector norm Numerical Methods Wen-Chieh Lin

34 Matrix Norms Matrix norm corresponding to vector -nom is maximum absolute column sum A max Matrix norm corresponding to vector -norm is maximum absolute row sum A Handy way to remember these is that matrix norms agree with corresponding vector norms for n by matrix j max i n i n a j a ij ij Numerical Methods Wen-Chieh Lin 4

35 Properties of Matrix Norms Matrix norms we have defined satisfies A ka A B AB and k A A A A B for if B and only if any scalar Above are actually the required properties when a matrix norm is defined! k A Numerical Methods Wen-Chieh Lin 5

36 Condition Number Condition number of square nonsingular matrix A is defined by cond( A) A A By convention, cond(a) = if A is singular Large cond(a) means A is singular Since A x Ax A A max max x x x x condition number measures ratio of maximum stretching to maximum shrinking does to any nonzero vectors Numerical Methods Wen-Chieh Lin 6

37 Properties of Condition Number For any matrix A, cond(a) For identity matrix, cond(i) = For any matrix A and scalar k, cond(ka) = cond(a) Numerical Methods Wen-Chieh Lin 7

38 Computing Condition Number Definition of condition number involves matrix inverse, so it is nontrivial to compute Computing condition number from definition would require much more work than computing solution whose accuracy is to be assessed In practice, condition number is estimated inexpensively as byproduct of solution process Numerical Methods Wen-Chieh Lin 8

39 Computing Condition Number Matrix norm A is easily computed as maximum column sum (or row sum, depending on norm used) Estimating challenging A at low cost is more From properties of norms, if Az = y, then z A y A y A and bound is achieved for optimally chosen y z y Numerical Methods Wen-Chieh Lin 9

40 Computing Condition Number Efficient condition estimators heuristically pick y with large ratio z y, yielding good estimator for A Good software packages for linear systems provide efficient and reliable condition estimator Numerical Methods Wen-Chieh Lin 4

41 Error Bounds Condition number yields error bound for computed solution to linear system Let x be solution to Ax = b, and x approximate solution, r is residual be an r b Ax Ax Ax Ae AB A B r A e e A AB A B r e A r r A e A r Numerical Methods Wen-Chieh Lin 4

42 Error Bounds (cont.) Similarly, from b = Ax and x = A - b we obtain b x A A Combined with previous result r e A A We have r e A A b x b r A A r b Numerical Methods Wen-Chieh Lin 4

43 Error Bounds (cont.) Similarly, from b = Ax and x = A - b we obtain b x A A b Combined with previous result r e A r A condition number of A! We have r e r A A A A b x b Numerical Methods Wen-Chieh Lin 4

44 Error Bounds (cont.) cond( A) r b e x cond( A) r b The relative error in the computed solution vector is bounded by the relative residual divided/multiplied by the condition number When the condition number is large, the residual gives little information about the accuracy Numerical Methods Wen-Chieh Lin 44

45 Error Bounds Illustration In two dimensions, uncertainty in intersection point of two lines depends on whether lines are nearly parallel well-conditioned ill-conditioned Numerical Methods Wen-Chieh Lin 45

46 Residual Residual vector of approximate solution to linear system Ax = b is defined by r b Ax In theory, if A is nonsingular, then x x if, and only if, r but they are not necessarily small simultaneously Since e r cond( A) x b small relative residual implies small relative error in approximate solution only if A is wellconditioned Numerical Methods Wen-Chieh Lin 46

47 Iterative Refinement Given approximate solution x to linear system Ax = b, compute residual r b Ax Now solve linear system Az = r and take x x z as new and better approximate solution, since Ax A ( x z ) Ax Az (b r ) r b Numerical Methods Wen-Chieh Lin 47

48 Iterative Refinement (cont.) Process can be repeated to refine solution successively until convergence, potentially producing solution accurate to full machine precision Numerical Methods Wen-Chieh Lin 48

49 Error in Coefficients of Matrix Let A A matrix and system Using x A x x Ax b A A be E the perturbed coefficient x the solution to the perturbed b A and ( Ax) ( A Ax ( Ax) A A ) x b A ( Ax) Ax x A x Ex x x x A Ex Numerical Methods Wen-Chieh Lin 49

50 Error in Coefficients of Matrix (cont.) x x x x A A Ex E x A x x E cond( A) x A A E A x Error of the solution relative to the norm of the computed solution can be as large as the relative error in the coefficients of A multiplied by the condition number Numerical Methods Wen-Chieh Lin 5