6. Cholesky factorization

Transcription

1 6. Cholesky factorization EE103 (Fall ) triangular matrices forward and backward substitution the Cholesky factorization solving Ax = b with A positive definite inverse of a positive definite matrix permutation matrices sparse Cholesky factorization 6-1

2 Triangular matrix a square matrix A is lower triangular if a ij = 0 for j > i A = a a 21 a a n 1,1 a n 1,2 a n 1,n 1 0 a n1 a n2 a n,n 1 a nn A is upper triangular if a ij = 0 for j < i (A T is lower triangular) a triangular matrix is unit upper/lower triangular if a ii = 1 for all i Cholesky factorization 6-2

3 Forward substitution solve Ax = b when A is lower triangular with nonzero diagonal elements a x 1 b 1 a 21 a 22 0 x = b 2. a n1 a n2 a nn x n b n algorithm: x 1 := b 1 /a 11 x 2 := (b 2 a 21 x 1 )/a 22 x 3 := (b 3 a 31 x 1 a 32 x 2 )/a 33. x n := (b n a n1 x 1 a n2 x 2 a n,n 1 x n 1 )/a nn cost: (2n 1) = n 2 flops Cholesky factorization 6-3

4 Back substitution solve Ax = b when A is upper triangular with nonzero diagonal elements a 11 a 1,n 1 a 1n x 1 b a n 1,n 1 a n 1,n x n 1 =. b n a nn x n b n algorithm: x n := b n /a nn x n 1 := (b n 1 a n 1,n x n )/a n 1,n 1 x n 2 := (b n 2 a n 2,n 1 x n 1 a n 2,n x n )/a n 2,n 2. x 1 := (b 1 a 12 x 2 a 13 x 3 a 1n x n )/a 11 cost: n 2 flops Cholesky factorization 6-4

5 Inverse of a triangular matrix triangular matrix A with nonzero diagonal elements is nonsingular Ax = b is solvable via forward/back substitution; hence A has full range therefore A has a zero nullspace, is invertible, etc. (see p.4-8) inverse can be computed by solving AX = I column by column A [ X 1 X 2 X n ] = [ e1 e 2 e n ] inverse of lower triangular matrix is lower triangular inverse of upper triangular matrix is upper triangular Cholesky factorization 6-5

6 Cholesky factorization every positive definite matrix A can be factored as A = LL T where L is lower triangular with positive diagonal elements cost: (1/3)n 3 flops if A is of order n L is called the Cholesky factor of A can be interpreted as square root of a positive define matrix Cholesky factorization 6-6

7 Cholesky factorization algorithm partition matrices in A = LL T as [ ] [ ][ ] a11 A T 21 l11 0 l11 L = T 21 A 21 A 22 L 21 L 22 0 L T 22 [ l11 = 2 l 11 L T 21 l 11 L 21 L 21 L T 21+L 22 L T 22 ] algorithm 1. determine l 11 and L 21 : 2. compute L 22 from l 11 = a 11, L 21 = 1 l 11 A 21 A 22 L 21 L T 21 = L 22 L T 22 this is a Cholesky factorization of order n 1 Cholesky factorization 6-7

8 proof that the algorithm works for positive definite A of order n step 1: if A is positive definite then a 11 > 0 step 2: if A is positive definite, then is positive definite (see page 4-23) A 22 L 21 L T 21 = A 22 1 a 11 A 21 A T 21 hence the algorithm works for n = m if it works for n = m 1 it obviously works for n = 1; therefore it works for all n Cholesky factorization 6-8

9 Example = l l 21 l 22 0 l 31 l 32 l 33 l 11 l 21 l 31 0 l 22 l l 33 first column of L = l l 32 l l 22 l l 33 second column of L [ ] [ [ ] [ ][ ] [ ] l22 0 l22 l 3 1 = 32 l 32 l 33 0 l 33 ] = [ ][ ] l 33 0 l 33 Cholesky factorization 6-9

10 third column of L: 10 1 = l 2 33, i.e., l 33 = 3 conclusion: = Cholesky factorization 6-10

11 Solving equations with positive definite A Ax = b (A positive definite of order n) algorithm factor A as A = LL T solve LL T x = b forward substitution Lz = b back substitution L T x = z cost: (1/3)n 3 flops factorization: (1/3)n 3 forward and backward substitution: 2n 2 Cholesky factorization 6-11

12 Inverse of a positive definite matrix suppose A is positive definite with Cholesky factorization A = LL T L is invertible (its diagonal elements are nonzero) X = L T L 1 is a right inverse of A: AX = LL T L T L 1 = LL 1 = I X = L T L 1 is a left inverse of A: XA = L T L 1 LL T = L T L T = I hence, A is invertible and A 1 = L T L 1 Cholesky factorization 6-12

13 Summary if A is positive definite of order n A can be factored as LL T the cost of the factorization is (1/3)n 3 flops Ax = b can be solved in (1/3)n 3 flops A is invertible with inverse: A 1 = L T L 1 Cholesky factorization 6-13

14 Sparse positive definite matrices a matrix is sparse if most of its elements are zero a matrix is dense if it is not sparse Cholesky factorization of dense matrices (1/3)n 3 flops on a current PC: a few seconds or less, for n up to a few 1000 Cholesky factorization of sparse matrices if A is very sparse, then L is often (but not always) sparse if L is sparse, the cost of the factorization is much less than (1/3)n 3 exact cost depends on n, #nonzero elements, sparsity pattern very large sets of equations (n 10 6 ) are solved by exploiting sparsity Cholesky factorization 6-14

15 Effect of ordering sparse equation (a is an (n 1)-vector with a < 1) [ ][ ] [ ] 1 a T u b = a I v c factorization [ 1 a T a I ] = [ 1 0 a L 22 ][ 1 a T 0 L T 22 ] where I aa T = L 22 L T 22 = factorization with 100% fill-in Cholesky factorization 6-15

16 reordered equation [ I a a T 1 ][ v u ] = [ c b ] factorization [ I a a T 1 ] = [ I 0 1 at a a T ][ I a 0 1 a T a ] = factorization with zero fill-in Cholesky factorization 6-16

17 Permutation matrices a permutation matrix is the identity matrix with its rows reordered, e.g., , the vector Ax is a permutation of x x 1 x x 3 = x 2 x 3 x 1 A T x is the inverse permutation applied to x = x 1 x 2 x 3 x 3 x 1 x 2 A T A = AA T = I, so A is invertible and A 1 = A T Cholesky factorization 6-17

18 Solving Ax = b when A is a permutation matrix the solution of Ax = b is x = A T b example x 1 x 2 x 3 = solution is x = ( 2.1,1.5,10.0) cost: zero flops Cholesky factorization 6-18

19 Sparse Cholesky factorization if A is sparse and positive definite, it is usually factored as A = PLL T P T P a permutation matrix; L lower triangular with positive diagonal elements interpretation: we permute the rows and columns of A and factor P T AP = LL T choice of P greatly affects the sparsity L many heuristic methods (that we don t cover) exist for selecting good permutation matrices P Cholesky factorization 6-19

20 Example sparsity pattern of A Cholesky factor of A pattern of P T AP Cholesky factor of P T AP Cholesky factorization 6-20

21 Solving sparse positive definite equations solve Ax = b via factorization A = PLL T P T algorithm 1. b := P T b 2. solve Lz = b by forward substitution 3. solve L T y = z by back substitution 4. x := Py interpretation: we solve (P T AP)y = b using the Cholesky factorization of P T AP Cholesky factorization 6-21