Solving Linear Systems of Equations

Transcription

1 Solving Linear Systems of Equations Many practical problems could be reduced to solving a linear system of equations formulated as Ax = b This chapter studies the computational issues about directly and iteratively solving Ax = b A Linear System of Equations Vector and Matrix Norms Matrix Condition Number (Cond(A) = A A 1 ) Direct Solution for Ax = b LU-decomposition by Gaussian Elimination Gaussian Elimination with Partial Pivoting Cholesky Algorithm for A = LL t (A is positive definite) Iterative Solutions Jacobi method Gauss-Seidel method Other methods Applications

2 Overdetermined, Underdetermined, Homogeneous Systems a 11 x 1 + a 12 x a 1n x n = b 1 a 21 x 1 + a 22 x a 2n x n = b 2 a m1 x 1 + a m2 x a mn x n = b m Ax = b Definition: A linear system is said to be overdetermined if there are more equations than unknowns (m >n), underdetermined if m<n, homogeneous if b i =0, 1 i m x + y = 1 x + y = 3 x + y = 2 (A) x y = 3 (B) x y = 1 (C) 2x + 2y = 4 x + 2y = 2 2x + y = 5 x y = 2 (A) has no solution, (B) has unique solution, (C) has infinitely many solutions (D) x + 2y + z = 1 2x + 4y + 2z = 3 (E) x + 2y + z = 5 2x y + z = 3 (D) has no solution, (E) has infinitely many solutions

3 Some Special Matrices A =[a ij ] R n n Diagonal if a ij =0 i j Lower Δifa ij =0ifj>i Unit lower ΔifA is lower-δ with a ii =1 Lower Hessenberg if a ij =0forj>i+1 Band matrix with bandwidth 2k +1ifa ij =0for i j >k A band matrix with bandwidth 1isdiagonal A band matrix with bandwidth 3istridiagonal A band matrix with bandwidth 5ispentadiagonal A lower and upper Hessenberg matrix is tridiagonal A 1 = , A 2 = , A 3 = , A 4 =

4 A Direct Solution of Linear Systems A linear system 2x + y + z = 5 4x 6y = 2 2x + 7y + 2z = 9 A matrix representation Ax = b, or x y z = Solution using MATLAB A =[2, 1, 2; 4, 6, 0; 2, 7, 2]; b =[5, 2, 9] ; x = A\b (x = [1; 1; 2])

5 Elementary Row Operations (1) Interchange two rows: A r A s (2) Multiply a row by a nonzero real number: A r αa r (3) Replace a row by its sum with a multiple of another row: A s αa r + A s E 1 =, E 2 = 0 0 2, E 3 = 1 0 1, Example A = , E 1 A = , E 2 A = , E 3 A = Let L 3 = 0 1 1, L 2 = 1 0 1, L 1 = 2 1 0, A = Then L 3 L 2 L 1 A = = U (Upper Δ) A =(L 1 1 L 1 2 L 1 3 )U = LU, where L is unit lower Δ

6 Computing An Inverse Matrix By Elementary Row Operations A = = I E 1 A = = E 1 I E 2 E 1 A = = E 2 E 1 I E 3 E 2 E 1 A = = E 3 E 2 E 1 I E 4 E 3 E 2 E 1 A = = E 4 E 3 E 2 E 1 I = A 1 where the elementary matrices are E 1 = 1 1 0, E 2 = 2 0 1, E 3 = 1 2 0, E 4 = 1 0 3

7 LU-Decomposition If a matrix A can be decomposed into the product of a unit lower-δ matrix L and an upper-δ matrix U, then the linear system Ax = b can be written as LUx = b The problem is reduced to solving two simpler triangular systems Ly = b and Ux = y by forward and back substitutions A = , L 3 = 0 1 1, L 2 = 1 0 1, L 1 = Then where L 3 L 2 L 1 A = = U A = L 1 1 L 1 2 L 1 3 U = LU A = = = LU If b =[5, 2, 9] t,theny =[5, 12, 2] t and x =[1, 1, 2] t B = , b = x = 1 2 1

8 Analysis of Gaussian Elimination Algorithm for i =1, 2,,n 1 for k = i +1,i+2,,n m ki a ki /a ii if a ii 0 a ki a ki for j = i +1,i+2,,n a kj a kj m ki a ij The Worst Computational Complexity is O( 2 3 n3 ) 1 # of divisions are (n 1) + (n 2) + +1= n(n 1) 2 2 # of multiplications are (n 1) 2 +(n 2) = n(n 1)(2n 1) 6 3 # of subtractions are (n 1) 2 +(n 2) = n(n 1)(2n 1) 6

9 The Analysis of Gaussian Elimination and Back Substitution to solve Ax=b 2 3 n n2 7 n 6 R 1 : a 11 x 1 + a 12 x a 1n x n = b 1 R 2 : a 21 x 1 + a 22 x a 2n x n = b 2 R i : a i1 x 1 + a i2 x a in x n = b i R n : a n1 x 1 + a n2 x a nn x n = b n By Guassian Elimination, we need C 1 =[ n k=1 (k+1)(k 1) + n k=1 k(k 1)] flops to reduce the above linear system of equations equivalent to the following upper triangular system R 1 : u 11 x 1 + u 12 x u 1n x n = c 1 R i : u ii x i + + u in x n = c i R n : u nn x n = c n We need C 2 = n k=1 (2k 1) = n 2 flops to solve an upper triangular linear system of equations Therefore, the total number of flops of solving Ax = b is summarized as C 1 + C 2 = 2 3 n n2 7 6 n

10 PA=LU Let A R 4 4 and L 3 P 3 L 2 P 2 L 1 P 1 A = U by Gaussian Elimination with Partial Pivoting Denote L 1 = 0 α 1 α 2 α 3, L 2 = α α 5 0 1, L 3 = α 6 1 Without loss of generality, let P 2 = Then P 2 L 1 =(P 2 L 1 P 1 2 )P 2 = L (2) 1 P 2,where L (2) 1 = 0 α 3 α 2 α 1 Theorem: For any A R n n, there exists a permutation matrix P such that PA = LU, where L is unit lower ΔandU is upper Δ

11 Gaussian Elimination with partial Pivoting Algorithm for i =1, 2,,n p(i)=i for i =1, 2,,n 1 (a) select a pivotal element a p(j),i such that a p(j),i = max i k n a p(k),i (b) p(i) p(j) (c) fork = i +1,i+2,,n m p(k),i = a p(k),i /a p(i),i for j = i +1,i+2,,n a p(k),j = a p(k),j m p(k),i a p(i),j An example A = P 23 P 13 A = LU =

12 Matlab Codes for Gaussian Elimination with Partial Pivoting %% %% %% gaussppm - drive of Gaussian Elimination wit Partial Pivoting %% %% %% fin=fopen( gaussmatdat, r ); n=fscanf(fin, %d,1); A=fscanf(fin, %f,[n n]); A=A ; b=fscanf(fin, %f,n); X=gausspivot(A,b,n) %% %% %% gausspivotm - Gaussian elimination with Partial Pivoting %% %% %% function X=gausspivot(A,b,n) if (abs(det(a))<eps) disp(sprintf( A is singular with det=%f\n,det(a))) end C=[A, b]; % Gaussian Elimination with Partial Pivoting % for i=1:n-1 [pivot, k]=max(abs(c(i:n,i))); if (k>1) temp=c(i,:); C(i,:)=C(i+k-1,:); C(i+k-1,:)=temp; end m(i+1:n,i)= -C(i+1:n,i)/C(i,i); C(i+1:n,:)=C(i+1:n,:)+m(i+1:n,i)*C(i,:); end % Back substitution % X=zeros(n,1); %% Let X be a column vector of size n X(n)=C(n,n+1)/C(n,n); for i=n-1:-1:1 X(i)=(C(i,n+1)-C(i,i+1:n)*X(i+1:n))/C(i,i); end

13 Doolittle s LU Factorization Algorithm: A R n n, A = LU, L is unit lower-δ, U is upper-δ for k =0, 1,,n 1 L kk 1 for j = k, k +1,,n 1 U kj A kj k 1 s=0 L ksu sj for i = k +1,k+2,,n 1 L ik [ A ik k 1 s=0 L isu sk ] /Ukk A = = = LU

14 Crout s LU Factorization Algorithm: A R n n, A = LU, L is lower-δ, U is unit upper-δ for k =0, 1,,n 1 U kk 1 for i = k, k +1,,n 1 L ik A ik k 1 s=0 L isu sk for j = k +1,k+2,,n 1 U kj [ A kj k 1 s=0 L ksu sj ] /Lkk A = = = LU

15 Cholesky Algorithm Algorithm: A R n n, A = LL t, A is positive definite and L is lower-δ for j =0, 1,,n 1 L jj [ A jj ] j 1 1/2 k=0 L2 jk for i = j +1,j+2,,n 1 L ij [ A ij j 1 k=0 L ikl jk ] /Ljj A = = = LL t

16 Vector Norms Definition: A vector norm on R n is a function that satisfies τ : R n R + = {x 0 x R} (1) τ(x) > 0 x 0, τ(0) =0 (2) τ(cx) = c τ(x) c R, x R n (3) τ(x + y) τ(x)+τ(y) x, y R n Hölder norm (p-norm) x p =( n i=1 x i p ) 1/p for p 1 (p=1) x 1 = n i=1 x i (Mahattan or City-block distance) (p=2) x 2 =( n i=1 x i 2 ) 1/2 (Euclidean distance) (p= ) x = max 1 i n { x i } ( -norm)

17 Matrix Norms Definition: A matrix norm on R m n is a function that satisfies τ : R m n R + = {x 0 x R} (1) τ(a) > 0 A O, τ(o) =0 (2) τ(ca) = c τ(a) c R, A R m n (3) τ(a + B) τ(a)+τ(b) A, B R m n Consistency Property: τ(ab) τ(a)τ(b) A, B (a) τ(a) =max{ a ij 1 i m, 1 j n} (b) A F = [ mi=1 nj=1 a 2 ij] 1/2 (Fröbenius norm) Subordinate Matrix Norm: A = max x 0 { Ax / x } (1) If A R m n,then A 1 = max 1 j n ( m i=1 a ij ) (2) If A R m n,then A = max 1 i m ( nj=1 a ij ) (3) Let A R n n be real symmetric, then A 2 = max 1 i n λ i,whereλ i λ(a)

18 Matrix Condition Number Cond(A) = A A 1 For the matrix A = , A 1 = Cond 1 (A) = A 1 A 1 1 =6 45 =27 Cond (A) = A A 1 =8 35 =28 Cond 2 (A) = A 2 A 1 2 = =

19 Sensitivity and Error Bounds Let x and ˆx be the solutions to Ax = b and Aˆx = ˆb, respectively, where ˆx = x +Δx, ˆb = b +Δb Then b = Ax A x or x b A Thus Δx x Δx = A 1 Δb A 1 Δb A 1 Δb A b Cond(A) Δb b Similarly, suppose that (A + E) x = b and Ax = b, wehave Further derivations to get Δx x E Cond(A) A ( Δx Δb x Cond(A) b + E ) A

20 Iterative Methods for Solving Linear Systems 1 Iterative methods are msot useful in solving large sparse system 2 One advantage is that the iterative methods may not require any extra storage and hence are more practical 3 One disadvantage is that after solving Ax = b 1, one must start over again from the beginning in order to solve Ax = b 2 Jacobi Method Given Ax = b, write A = C M, wherec is nonsingular and easily invertible Then Ax = b (C M)x = b Cx = Mx + b x = C 1 Mx + C 1 b x = Bx + c, where B = C 1 M, c = C 1 b Suppose we start with an inital x (0),then x (1) = Bx (0) + c and x (k+1) = Bx (k) + c

21 Jacobi Iterative Method for Solving Linear Systems Suppose x is a solution to Ax = b, then x (1) x =(Bx (0) + c) (Bx + c) =B(x (0) x) x (k) x = B k (x (0) x) x (k) x B k x (0) x B k x (0) x x (k) x 0ask if B = C 1 M < 1 For simplest computations, 1-norm or -norm is used The simplest choice of C, M with A = C M is C = Diag(A), M = (A C) Theorem: Let x (0) R n be arbitrary and define x (k+1) = Bx (k) + c for k =0, 1, If x is a solution to Ax = b, then the necessary and sufficient condition for x (k) x is B < 1 Theorem: If A is diagonally dominant, ie, a ii > j i a ij for 1 i n Let A = C M and B = C 1 M,then n B = max 1 i n b ij = max a ij 1 i n j=1 j i a ii < 1 Example: Let Then 10 1 A = 2 10, b = 10 0 C = 0 10 [ 11 12, M = 0 01 B = C 1 M = 02 0 ], x (0) = 0 1, 2 0 [ 0 0, c = C 1 b = ] [ ] x (1) = Bx (0) + c = [ ], x (2) = [ ], x (3) = [ ]

22 Gauss-Seidel Method Given Ax = b, write A = C M, wherec is nonsingular and easily invertible Jacobi: C = diag(a 11,a 22,,a nn ), M = (A C) Gauss-Seidel: A =(D L) U = C M, wherec = D L, M = U for Jacobi Let x (0) R n be nonzero, then Cx (k+1) = Mx (k) + b implies that (D L)x (k+1) = Ux (k) + b Dx (k+1) = Lx (k+1) + Ux (k) + b x (k+1) 1 = 1 a 11 ( nj=2 a 1j x (k) j + b 1 ) x (k+1) ( i = 1 a ii i 1 j=1 a ij x (k+1) j n j=i+1 a ij x (k) ) j + b i, for i =2, 3,,n The difference between Jacobi and Gauss-Seidel iteration is that in the latter case, one is using the coordinates of x (k+1) as soon as they are calculated rather than in the next iteration The program for Gauss-Seidel is much simpler

23 Convergence of Gauss-Seidel Iterations Theorem: If A is diagonally dominant, ie, a ii > j i a ij for 1 i n Gauss-Seidel iteration converges to a solution of Ax = b Then Proof: Denote A =(D L) U and let j 1 α j = a ji, β j = n a ji, and R j = i=1 i=j+1 β j a jj α j Since A is diagonally dominant, then a jj >α j + β j,thus R j = Therefore, R = Max 1 j n R j < 1 β j a jj α j < a jj α j a jj α j =1 1 j n The remaining problem is to show that B = Max x =1 Bx R<1, where B = C 1 M =(D L) 1 U Let x =1andy = Bx, then y = Max 1 i n y i = y k for some k Then y = Bx =(D L) 1 Ux (D L)y = Ux Dy = Ly + Ux y = D 1 (Ly + Ux) Then y k = 1 a kk ( k 1 i=1 a ki y i n i=k+1 a ki x i ) y = y k 1 (α a kk k y + β k x ) which implies that for all x with x =1, Bx = y Thus, B = Max x =1 Bx R<1 β k a kk α k = R k < 1

24 Example for Gauss-Seidel Iterations Example: 10 1 A = 2 10 = b = [ ], x (0) = [ 0 0 ] 0 1 = D L U 0 0 Then Moreover, Thus x (1) 1 = 1 10 ( a 12x (0) 2 + b 1 )=11 x (1) 2 = 1 10 ( a 21x (1) 1 0+b 2 )=098 x (2) 1 = 1 10 ( a 12x (1) 2 + b 1 )=1002 x (2) 2 = 1 10 ( a 21x (2) 1 0+b 2 )=09996 x (1) =[11, 098] t x (2) =[1002, 09996] t x (3) =[100004, ] t