Matrix Norms. Tom Lyche. September 28, Centre of Mathematics for Applications, Department of Informatics, University of Oslo

Transcription

1 Matrix Norms Tom Lyche Centre of Mathematics for Applications, Department of Informatics, University of Oslo September 28, 2009

2 Matrix Norms We consider matrix norms on (C m,n, C). All results holds for (R m,n, R). Definition (Matrix Norms) A function : C m,n C is called a matrix norm on C m,n if for all A, B C m,n and all α C 1. A 0 with equality if and only if A = 0. (positivity) 2. αa = α A. (homogeneity) 3. A + B A + B. (subadditivity) A matrix norm is simply a vector norm on the finite dimensional vector spaces (C m,n, C) of m n matrices.

3 Equivalent norms Adapting some general results on vector norms to matrix norms give Theorem x 1. All matrix norms are equivalent. Thus, if and are two matrix norms on C m,n then there are positive constants µ and M such that µ A A M A holds for all A C m,n. 2. A matrix norm is a continuous function : C m,n R.

4 Examples From any vector norm V on C mn we can define a matrix norm on C m,n by A := vec(a) V, where vec(a) C mn is the vector obtained by stacking the columns of A on top of each other. A S := m i=1 A F := ( m i=1 n a ij, p = 1, Sum norm, j=1 n a ij 2) 1/2, p = 2, Frobenius norm, j=1 A M := max a ij, p =, Max norm. i,j (1)

5 The Frobenius Matrix Norm 1. A H 2 F = n m j=1 i=1 a ij 2 = m n i=1 j=1 a ij 2 = A 2 F.

6 The Frobenius Matrix Norm 2. m i=1 m i=1 n j=1 a ij 2 = m n j=1 a ij 2 = n j=1 i=1 a i 2 2 m i=1 a ij 2 = n j=1 a j 2 2.

7 Unitary Invariance. If A C m,n and U C m,m, V C n,n are unitary UA 2 F 2. = n j=1 Ua j 2 2 = n j=1 a j = A 2 F. AV F 1. = V H A H F = A H F 1. = A F.

8 Submultiplicativity Suppose A, B are rectangular matrices so that the product AB is defined. AB 2 F = n k ( ) i=1 j=1 a T 2 i b j k j=1 a i 2 2 b j 2 2 = A 2 F B 2 F. n i=1

9 Subordinance Ax 2 A F x 2, for all x C n. Since v F = v 2 for a vector this follows from submultiplicativity.

10 Explicit Expression Let A C m,n have singular values σ 1,..., σ n and SVD A = UΣV H. Then A F 3. = U H AV F = Σ F = σ σ2 n.

11 Consistency A matrix norm is called consistent on C n,n if 4. AB A B (submultiplicativity) holds for all A, B C n,n. A matrix norm is consistent if it is defined on C m,n for all m, n N, and 4. holds for all matrices A, B for which the product AB is defined. The Frobenius norm is consistent. The Sum norm is consistent. The Max norm is not consistent. The norm A := mn A M, A C m,n is consistent.

12 Subordinate Matrix Norm Definition Suppose m, n N are given, Let α on C m and β on C n be vector norms, and let be a matrix norm on C m,n. We say that the matrix norm is subordinate to the vector norms α and β if Ax α A x β for all A C m,n and all x C n. If α = β then we say that is subordinate to α. The Frobenius norm is subordinate to the Euclidian vector norm. The Sum norm is subordinate to the l 1 -norm. Ax A M x 1.

13 Operator Norm Definition Suppose m, n N are given and let α be a vector norm on C n and β a vector norm on C m. For A C m,n we define A := A α,β := max x 0 Ax β x α. (2) We call this the (α, β) operator norm, the (α, β)-norm, or simply the α-norm if α = β.

14 Observations A α,β = max x/ ker(a) Ax α x β = max x β =1 Ax α. Ax α A x β. A α,β = Ax α for some x C n with x β = 1. The operator norm is a matrix norm on C mn,. The Sum norm and Frobenius norm are not an α operator norm for any α.

15 Operator norm Properties The operator norm is a matrix norm on C m,n. The operator norm is consistent if the vector norm α is defined for all m N and β = α.

16 Proof In 2. and 3. below we take the max over the unit sphere S β. 1. Nonnegativity is obvious. If A = 0 then Ay β = 0 for each y C n. In particular, each column Ae j in A is zero. Hence A = ca = max x cax α = max x c Ax α = c A. 3. A + B = max x (A + B)x α max x Ax α + max x Bx α = A + B. 4. AB = max Bx 0 ABx α x α max y 0 Ay α y α = max Bx 0 ABx α Bx α max x 0 Bx α x α Bx α x α = A B.

17 The p matrix norm The operator norms p defined from the p-vector norms are of special interest. Recall x p := ( n j=1 x j p) 1/p, p 1, x := max 1 j n x j. Used quite frequently for p = 1, 2,. We define for any 1 p A p := max x 0 Ax p x p = max y p=1 Ay p. (3) The p-norms are consistent matrix norms which are subordinate to the p-vector norm.

18 Explicit expressions Theorem For A C m,n we have A 1 := max 1 j n m a k,j, (max column sum) k=1 A 2 := σ 1, (largest singular value of A) n A := max a k,j, (max row sum). 1 k m j=1 The expression A 2 is called the two-norm or the spectral norm of A. The explicit expression follows from the minmax theorem for singular values. (4)

19 Examples For A := 1 15 A 1 = A 2 = 2. A = A F = 5. A := [ ] A 1 = 6 A 2 = A = 7. A F = [ ] we find

20 The 2 norm Theorem Suppose A C n,n has singular values σ 1 σ 2 σ n and eigenvalues λ 1 λ 2 λ n. Then A 2 = σ 1 and A 1 2 = 1 σ n, (5) A 2 = λ 1 and A 1 2 = 1 λ n, if A is symmetric positive definite, (6) A 2 = λ 1 and A 1 2 = 1, if A is normal. (7) λ n For the norms of A 1 we assume of course that A is nonsingular.

21 Unitary Transformations Definition A matrix norm on C m,n is called unitary invariant if UAV = A for any A C m,n and any unitary matrices U C m,m and V C n,n. If U and V are unitary then U(A + E)V = UAV + F, where F = E. Theorem The Frobenius norm and the spectral norm are unitary invariant. Moreover A H F = A F and A H 2 = A 2.

22 Proof 2 norm UA 2 = max x 2 =1 UAx 2 = max x 2 =1 Ax 2 = A 2. A H 2 = A 2 (same singular values). AV 2 = (AV) H 2 = V H A H 2 = A H 2 = A 2.

23 Perturbation of linear systems x 1 + x 2 = 20 x 1 + ( )x 2 = The exact solution is x 1 = x 2 = 10. Suppose we replace the second equation by x 1 + ( )x 2 = , the exact solution changes to x 1 = 30, x 2 = 10. A small change in one of the coefficients, from to , changed the exact solution by a large amount.

24 Ill Conditioning A mathematical problem in which the solution is very sensitive to changes in the data is called ill-conditioned or sometimes ill-posed. Such problems are difficult to solve on a computer. If at all possible, the mathematical model should be changed to obtain a more well-conditioned or properly-posed problem.

25 Perturbations We consider what effect a small change (perturbation) in the data A,b has on the solution x of a linear system Ax = b. Suppose y solves (A + E)y = b+e where E is a (small) n n matrix and e a (small) vector. How large can y x be? To measure this we use vector and matrix norms.

26 Conditions on the norms will denote a vector norm on C n and also a submultiplicative matrix norm on C n,n which in addition is subordinate to the vector norm. Thus for any A, B C n,n and any x C n we have AB A B and Ax A x. This is satisfied if the matrix norm is the operator norm corresponding to the given vector norm or the Frobenius norm.

27 Absolute and relative error The difference y x measures the absolute error in y as an approximation to x, y x / x or y x / y is a measure for the relative error.

28 Perturbation in the right hand side Theorem Suppose A C n,n is invertible, b, e C n, b 0 and Ax = b, Ay = b+e. Then 1 e K(A) b y x x K(A) e b, K(A) = A A 1. (8) Proof: Consider (8). e / b is a measure for the size of the perturbation e relative to the size of b. y x / x can in the worst case be times as large as e / b. K(A) = A A 1

29 Condition number K(A) is called the condition number with respect to inversion of a matrix, or just the condition number, if it is clear from the context that we are talking about solving linear systems. The condition number depends on the matrix A and on the norm used. If K(A) is large, A is called ill-conditioned (with respect to inversion). If K(A) is small, A is called well-conditioned (with respect to inversion).

30 Condition number properties Since A A 1 AA 1 = I 1 we always have K(A) 1. Since all matrix norms are equivalent, the dependence of K(A) on the norm chosen is less important than the dependence on A. Usually one chooses the spectral norm when discussing properties of the condition number, and the l 1 and l norm when one wishes to compute it or estimate it.

31 The 2-norm Suppose A has singular values σ 1 σ 2 σ n > 0 and eigenvalues λ 1 λ 2 λ n if A is square. K 2 (A) = A 2 A 1 2 = σ 1 σ n K 2 (A) = A 2 A 1 2 = λ 1 λ n, A normal. It follows that A is ill-conditioned with respect to inversion if and only if σ 1 /σ n is large, or λ 1 / λ n is large when A is normal. K 2 (A) = A 2 A 1 2 = λ 1 λ n, A positive definite.

32 The residual Suppose we have computed an approximate solution y to Ax = b. The vector r(y :) = Ay b is called the residual vector, or just the residual. We can bound x y in term of r(y). Theorem Suppose A C n,n, b C n, A is nonsingular and b 0. Let r(y) = Ay b for each y C n. If Ax = b then 1 r(y) K(A) b y x x K(A) r(y) b. (9)

33 Discussion If A is well-conditioned, (9) says that y x / x r(y) / b. In other words, the accuracy in y is about the same order of magnitude as the residual as long as b 1. If A is ill-conditioned, anything can happen. The solution can be inaccurate even if the residual is small We can have an accurate solution even if the residual is large.

34 The inverse of A + E Theorem Suppose A C n,n is nonsingular and let be a consistent matrix norm on C n,n. If E C n,n is so small that r := A 1 E < 1 then A + E is nonsingular and If r < 1/2 then (A + E) 1 A 1 1 r. (10) (A + E) 1 A 1 A 1 2K(A) E A. (11)

35 Proof We use that if B C n,n and B < 1 then I B is nonsingular and (I B) B. Since r < 1 the matrix I B := I + A 1 E is nonsingular. Since (I B) 1 A 1 (A + E) = I we see that A + E is nonsingular with inverse (I B) 1 A 1. Hence, (A + E) 1 (I B) 1 A 1 and (10) follows. From the identity (A + E) 1 A 1 = A 1 E(A + E) 1 we obtain by (A + E) 1 A 1 A 1 E (A + E) 1 K(A) E A A 1 1 r. Dividing by A 1 and setting r = 1/2 proves (11).

36 Perturbation in A Theorem Suppose A, E C n,n, b C n with A invertible and b 0. If r := A 1 E < 1/2 for some operator norm then A + E is invertible. If Ax = b and (A + E)y = b then y x y y x x A 1 E K(A) E A, (12) 2K(A) E.. (13) A

37 Proof A + E is invertible. (12) follows easily by taking norms in the equation x y = A 1 Ey and dividing by y. From the identity y x = ( (A + E) 1 A 1) Ax we obtain y x (A + E) 1 A 1 A x and (13) follows.

38 Finding the rank of a matrix Gauss-Jordan cannot be used to determine rank numerically Use singular value decomposition numerically will normally find σ n > 0. Determine minimal r so that σ r+1,..., σ n are close to round off unit. Use this r as an estimate for the rank.

39 Convergence in R m,n and C m,n Consider an infinite sequence of matrices {A k } = A 0, A 1, A 2,... in C m,n. {A k } is said to converge to the limit A in C m,n if each element sequence {A k (ij)} k converges to the corresponding element A(ij) for i = 1,..., m and j = 1,..., n. {A k } is a Cauchy sequence if for each ɛ > 0 there is an integer N N such that for each k, l N and all i, j we have A k (ij) A l (ij) ɛ. {A k } is bounded if there is a constant M such that A k (ij) M for all i, j, k.

40 More on Convergence By stacking the columns of A into a vector in C mn we obtain A sequence {A k } in C m,n converges to a matrix A C m,n if and only if lim k A k A = 0 for any matrix norm. A sequence {A k } in C m,n is convergent if and only if it is a Cauchy sequence. Every bounded sequence {A k } in C m,n has a convergent subsequence.

41 The Spectral Radius ρ(a) = max λ σ(a) λ. For any matrix norm on C n,n and any A C n,n we have ρ(a) A. Proof: Let (λ, x) be an eigenpair for A X := [x,..., x] C n,n. λx = AX, which implies λ X = λx = AX A X. Since X 0 we obtain λ A.

42 A Special Norm Theorem Let A C n,n and ɛ > 0 be given. There is a consistent matrix norm on C n,n such that ρ(a) A ρ(a) + ɛ.

43 A Very Important Result Theorem For any A C n,n we have lim k Ak = 0 ρ(a) < 1. Proof: Suppose ρ(a) < 1. There is a consistent matrix norm on C n,n such that A < 1. But then A k A k 0 as k. Hence A k 0. The converse is easier.

44 Convergence can be slow A = , A 100 = , ɛ A 2000 = ɛ

45 More limits For any submultiplicative matrix norm on C n,n and any A C n,n we have lim k Ak 1/k = ρ(a). (14)