for x = (x 1,x 2,...,x n ) t C n. This norm satisfies the usual norm axioms:

Transcription

1 2: NORMS AND THE JACOBI METHOD Math 639 (updated: January 2, 2012) We shall be primarily interested in solving matrix equations with real coefficients. However, the analysis of iterative methods sometimes requires the use of complex matrices for reasons which shall become clear in the subsequent classes. We shall consider first vector norms on C n, the set of vectors with n components, each one a complex number. We start the Euclidean norm (standard vector length) defined by ( ) 1/2 2 = x i 2 i=1 for x = (x 1,x 2,...,x n ) t C n. This norm satisfies the usual norm axioms: (2.1) 2 0, for all x C n, (non-negativity) 2 = 0, only if x = 0, (definiteness) αx 2 = α 2, for α C,x C n, x+y y 2, for x,y C n, (triangle inequality). We shall sometimes refer to this norm as the l 2 norm. This norm obviously can be restricted to real vectors in R n. Norms give a notion of length to vectors. Consider the case of R 2. You can think of vectors in R 2 as points (the vector being from the origin to the point). The set of vectors having Euclidean norm less than or equal to one consists of points which are within the circle (including the boundary) of radius one centered about the origin. The conditions in (2.1) represent the set of axioms which a norm is required to satisfy. Many other norms are possible. For example, the l norm is defined by = max,...,n x j. It is not difficult to check that satisfies (2.1). The set of vectors in R 2 of length at most one using this norm is no longer a circle as was the case with the Euclidean norm. Find this set by tracing out its boundary; the set of vectors in R 2 whose l norm equals one. Another useful norm is the l 1 norm. It is given by 1 = x j. 1

2 2 Again, it is not hard to see that it satisfies all of the norm axioms (2.1). Sketch the set of vectors of R 2 of length less than or equal to one when measured in the norm. These norms can be generalized into a whole scale of norms, i.e., ( ) 1/p (2.2) p = x i p. These are norms for 1 p <. When 0 < p < 1, these expressions fail to satisfy the triangle inequality. Problem 1. For n = 2 and p = 1/2, find two vectors x,y R 2 satisfying x+y p > p + y p. We shall use norms to more precisely measure convergence of iterative methods. Our discussion of an error vector e i converging to the zero vector in Class 1 was somewhat intuitive. In contrast, we can be more rigorous if we show that for some norm on R n, e i 0 as i. Problem 2. Let M be an n n nonsingular matrix with complex coefficients and be a norm on C n. Show that M defined by M = Mx is also a norm on C n, i.e., show that this norm satisfies the axioms (2.1). Let G be an n n complex matrix. A vector norm on C n induces a matrix norm on G by Gx (2.3) G = sup x C n,x 0. The set of n n matrices is a vector space V (under matrix addition and the usual definition of scalar-matrix multiplication) and G satisfies the norm axioms (2.1) on V. This is sometimes called the operator norm and, in general, depends on the choice of vector norm. If G is an n n real matrix, we can define an alternative matrix norm by Gx (2.4) G r = sup x R n,x 0. It is clear that G r G. Most of our results below hold for the this matrix norm as well and we shall often use the same notation G for both. We shall sometimes refer to this as the real matrix norm.

3 Remark 1. Because the operator norm is a norm, we can make conclusions of the form A+ǫB A + ǫ B. Here A and B are n n matrices and ǫ is a scalar. We used two of the norm axioms for the above inequality. Remark 2. The spectral radius of a real or complex matrix A is defined by ρ(a) = max λ i where the maximum is over all eigenvalues λ i of A. We have that A ρ(a) no matter what norm on we choose to use on C n. Indeed, if λ i is an eigenvalue of A and x i is a corresponding eigenvector, then Ax i = λx i so Ax i x i = λ ix i x i = λ i. Now, A is the supremum of the quantity on the left hand side above over all non-zero vectors and so A ρ(a). The opposite inequality does not hold in general. The matrix ( ) 1 1 A = 0 1 has only the eigenvalue λ = 1 and so ρ(a) = 1. However, A = 2 as we shall see below (see, Proposition 1). Remark 3. For an n n real matrix, the real matrix norm given by (2.4) also satisfies G r ρ(g). This is no where near as obvious (we shall prove this in Class 4). As well as satisfying the norm hypothesis, the operator norm satisfies two important additional inequalities. We state and prove these for the real operator norm and real matrices. The proofs for the complex case are essentially identical. Specifically, if A,B are n n real matrices and y R n, (2.5) AB A B and Ay A y. We shall prove the second inequality first. It is obviously true if y = 0 as both the left and right hand side are zero. If y 0 then by the definiteness property of the norm, y 0 and Ay y sup Ax x R n,x 0 = A. This is just the second inequality of (2.5) in disguise. 3

4 4 For the first inequality in (2.5), if x R n with x 0, ABx = A(Bx) A Bx A B = A B where we used the second inequality of (2.5) twice above. The first inequality of (2.5) follows by taking the supremum over x R n, x 0. Now, consider a linear iterative method for solving Ax = b with A an n n matrix. Let x 1,x 2,... be the sequence of iterates generated by the method and x 0 be the starting iterate. Since the method is linear, the errors e i = x x i are related by e i+1 = Ge i for some n n matrix G. Applying (2.5) gives Repetitively applying this gives e i+1 G e i. e i ( G ) i e 0. Now if γ = G is less than one, then the -norm of the error converges to zero as i. Moreover, each step of the iteration reduces this norm of the error by at least a factor of γ. Thus, we have show: A linear iteration method converges for any starting iterate and any right hand side provided that there is a vector norm with corresponding matrix norm G < 1. Proposition 1. Let G be a n n matrix. Then G = max n { G ij }. i=1 Proof. Set γ = max n { G ij }. i=1 Let x be in C n with x 0 and define y = Gx. Then, y i = G ij x j G ij x j G ij γ. Taking the maximum over i = 1,...,n gives Gx = y γ.

5 Dividing by and taking the supremum over all such x give G γ. For the other direction, we first note that if G = 0, the identity is obvious. Otherwise, we let i be an index for which γ = G ij and set x C n by x j = { 0 : if G ij = 0 Ḡ ij / G ij : otherwise. Here Ḡij denotes the complex conjugate of G ij. Note that = 1 and Gx (Gx) i = G ij = γ. Thus γ Gx G. This completes the proof of the proposition. Example 1. (The Classical Jacobi Method). Let A be an n n matrix with nonzero diagonal entries and D be the diagonal matrix with entries D jj = A jj. The Jacobi Method is given by Dx i+1 = Dx i +(b Ax i ). Note that this method is cheap to implement as it only requires simple linear operations on vectors and the inversion of a diagonal matrix times a vector. Remark 4. The whole point of using iterative methods to solve matrix equations is to get a sufficiently accurate solution without doing an enormous amount of computation. Accordingly, each step of the iterative method should be relatively cheap (at least much cheaper than the amount of computational work required to invert the matrix using direct methods such as Gaussian Elimination). Thus, whenever we propose an iterative method, we will always consider the computational effort required per step. Of course, we shall also be interested in convergence/divergence and the rate of convergence. The error for the Jacobi Method satisfies (2.6) De i+1 = De i Ae i, (why?) i.e., e i+1 = (I D 1 A)e i. 5

6 6 Definition 1. A matrix A is called diagonally dominant if A jj > A ji for j = 1,...,n. i=1,i j Theorem 1. If A is diagonally dominant then the Jacobi method converges for any right hand side and initial iterate. Proof. Applying the previous proposition to the reduction matrix G = (I D 1 A) gives n G = max n i=1,i j A ji A jj which is less than one when A is diagonally dominant. Remark 5. The above proof illustrates how it may be more convenient to work with a specific norm to obtain a specific result. The l norm fits very well with the diagonally dominant assumption. We shall be applying iterative methods to sparse matrices. A matrix is sparse if it contains relatively few non-zero elements. An example is the tridiagonal n n matrix A 3 defined by 2 if i = j, (A 3 ) ij = 1 if i j = 1, 0 otherwise. A full n n matrix has n 2 non-zero entries. In contrast, A 3 only has 3n 2 nonzero entries. Dealing with sparse matrices efficiently involves avoiding computations involving the zero entries. To do this, the matrix must be stored in a scheme which only involves the nonzero entries. We shall use a modified Compressed Sparse Row (CSR) structure. A nice discussion of compressed sparse row structure can be found at dongarra/etemplates/node373.html. This structure is designed so that it is easy to access the entries in a row. Our modification is made so that it is also easy to access the diagonal entry in any row. The CSR structure involves three arrays, VAL, CIND and RIND. VAL is an array of real numbers and stores the actual (nonzero) entries of A. CIND is an integer array which contains the column indices for nonzero entries in

7 A. The length of VAL and CIND are equal to the number of nonzero entries in A. Finally, RIND is an integer array of dimension n+1 and contains the row offsets (into the arrays VAL and CIND). By convention, RIND(n+1) is set to the total number of nonzeroes plus one. This structure should become clear by examining the following example (see also the URL above). Consider A = 1/3 3 2/ /4 4 3/ /5 5 The modified CSR structure is as follows: VAL /3 2/3 4 1/4 3/4 5 1/5 CIND and RIND Note that the i th entry of RIND points to the start of the nonzero values (in VAL) for the i th row. It also points to the start of the column indices for that row. The modification is that we always put the diagonal entry at that location, i.e. VAL(RIND(i))=A i,i. The general CSR structure does not do this. Indeed, the general CSR storage scheme does not have a diagonal entry whenever the diagoal entry is zero. 7